Guides

Indexer service - pipe Farcaster data

Pipe Farcaster data, or subsets of it, directly into your db

Suggest Edits

📘

Reach out for setup and pricing

A service that reads real-time data from hubs and indexes it into your Postgres database.

Benefits

  • Full control over a Farcaster dataset that is synced in real-time
    • custom indexes, derivative tables, and custom APIs
  • No need to maintain a hub
  • No need to maintain an indexer with new protocol updates
    • Neynar handles all protocol changes for newly available data

Requirements

See Requirements for indexer service

Steps

  • Contact for setup
  • Backfill
    • Once Neynar receives the credentials from you, we will verify access to the database and permissions.
    • We will set up the schema and start the backfill process.
    • Expect 24-48 hours for the backfill to complete
  • Livestream indexing
    • Post backfill, all data will be indexed from the live stream from the hub

Important FYIs for managing your own disk

  • Disk size: Start with a big enough disk! Changing disk sizes on Amazon EBS (which RDS uses) is limited by a 6 hour cooldown timer. This timer also applies to their “auto scaling” of drives. Whenever changing your disk type or size or iops, be sure to take this cooldown into consideration! Beware that this timer is shared by all three of those settings. So even if you change just the iops, you have to wait 6 hours to change the size or type!
  • Read queries: If you need long running queries (especially ones that join multiple tables), know that they will block some syncing. This will manifest as spikes in your “buffered emits” graph. Fix this blocking by adding a replica and moving all of your reading to there. You will also probably need to enable hot_standby_feedback if your queries are taking too long with a replica.
  • Be VERY CAREFUL with database triggers. They can break things in very surprising ways. This article covers some of the pain we’ve seen.

Notes

  • We read data from hubs directly, and hubs can differ from Warpcast from time to time: see here for more on this topic
  • By default, we pipe data in this schema: link , reach out if you want subsets of data or custom schemas

Questions

For any questions, message @manan or @rish on Warpcast

Updated 25 days ago