Summary
Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.
Motivation
This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.
Design
- After fetching batch N, spawn the DB write as a background task
- Immediately begin fetching batch N+1
- Before writing batch N+1, await completion of the batch N write
- Errors in either fetch or write should still propagate correctly
Key considerations:
- The reorder buffer (
BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder step
- The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
- SSE publishing (via
HeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timing
References
- tidx implementation:
sync/engine.rs — tick_realtime fetches batch N+1 while writing batch N
Summary
Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.
Motivation
This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.
Design
Key considerations:
BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder stepHeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timingReferences
sync/engine.rs—tick_realtimefetches batch N+1 while writing batch N