Indexer service requirements

• PostgreSQL database recommended specs

  • 1.6 TB disk space if ingesting full network data, real time
  • 16 cores
  • 128GB of RAM

• Credentials for PostgreSQL

  • Neynar’s role MUST have read and write on either the public schema or a dedicated schema for us. Replace “schema_name” and “username” to match your choice:

    cURL

    GRANT CREATE ON SCHEMA schema_name TO username;
    GRANT USAGE ON SCHEMA schema_name TO username;
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA schema_name TO username;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA schema_name TO username;
    GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA schema_name TO username;
    
    ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name
    GRANT ALL PRIVILEGES ON TABLES TO username;
    
    ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name
    GRANT ALL PRIVILEGES ON SEQUENCES TO username;
    
    ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name
    GRANT ALL PRIVILEGES ON FUNCTIONS TO username;
    

• Your database SHOULD be in AWS us-east-1 or equivalent for the lowest indexing lag. If not, additional data transfer charges will be needed.

• The database SHOULD require SSL on connections.

• The database MAY be in a private VPC, but the database MUST be “publicly accessible”.

• The database SHOULD limit access by IP address. We will give you our static IP during setup.

• 99.9% availability uptime on the database - we expect your database to be highly available for writes, or else it might increase the data lag

Important FYIs for managing your own disk

• Disk size: Start with a big enough disk! Changing disk sizes on Amazon EBS (which RDS uses) is limited by a 6 hour cooldown timer. This timer also applies to their “auto scaling” of drives. Whenever changing your disk type or size or iops, be sure to take this cooldown into consideration! Beware that this timer is shared by all three of those settings. So even if you change just the iops, you have to wait 6 hours to change the size or type!
• Read queries: If you need long running queries (especially ones that join multiple tables), know that they will block some syncing. This will manifest as spikes in your “buffered emits” graph. Fix this blocking by adding a replica and moving all of your reading to there. You will also probably need to enable hot_standby_feedback if your queries are taking too long with a replica.
• Be VERY CAREFUL with database triggers. They can break things in very surprising ways. This article covers some of the pain we’ve seen.