Skip to content

feat: pipelined fetch/write for indexer #55

@pthmas

Description

@pthmas

Summary

Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.

Motivation

This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.

Design

  1. After fetching batch N, spawn the DB write as a background task
  2. Immediately begin fetching batch N+1
  3. Before writing batch N+1, await completion of the batch N write
  4. Errors in either fetch or write should still propagate correctly

Key considerations:

  • The reorder buffer (BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder step
  • The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
  • SSE publishing (via HeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timing

References

  • tidx implementation: sync/engine.rstick_realtime fetches batch N+1 while writing batch N

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions