Skip to main content

Ingestion code available in this github repo, clone repo onto a server with a large disk and you should be importing in no time

Reach out to us for credentials to try it out.
  1. Install Homebrew:
  1. Install amazon’s command line tool:
  1. Configure amazon’s command line tool:
  1. Set this new profile to be the default (or you can use --profile ... on all of your aws commands):
  1. List all the archive exports:
You’ll see some output that will look something like this (the timestamps will likely be different):
The filename format is ${DATABASE}-${TABLE}-${START_TIME}-${END_TIME}.parquet. The timestamps bound the updated_at column. You probably want to fetch the latest versions of each table the first time you build your database.
  1. List all the incremental exports:
  1. List all the files for a specific time range:
  1. Download a specific file:
  1. Download all the tables for a specific time range:
  1. Use the parquet cli:
  1. Check some data: