When building an indexer for a high-volume Solana program like Pumpfun and PumpSwap, system architecture is critical. In a previous post, we established a verified stream of decoded trades. This article examines a critical failure in the initial design of the Indexer. A single blocking RPC call caused silent data loss and dropped around two-thirds of the ingested stream.

Here is the technical breakdown of the bottleneck, the root cause, and the architectural fix.

The Gap in the Trade Payload

We index trade events to track wallet addresses and token amounts. However, the transaction payload lacks specific liquidity pool data. The pool address lives in the on-chain account state. Retrieving it requires a getAccountInfo RPC call to decode the pool state.

Blocking the Hot Path

The initial implementation executed this RPC call synchronously inside the event processor. For every incoming trade, the indexer paused to fetch the pool data before moving to the next event.

This network request takes around 300ms. While faster RPC providers can reduce this latency, the underlying architectural flaw remains. The ingestion framework (Carbon) processes events sequentially. Introducing synchronous network I/O to the hot path blocks the entire stream.

How the Channel Silently Overflows

When the processor blocks, incoming events accumulate in the internal channel. Carbon defaults to a channel capacity of 1,000 events.

Once this buffer fills, the system cannot queue new updates. Carbon attempts to push the event, fails immediately, and discards it. Because logging for this specific drop event is disabled by default, the loss occurs silently.

The system appeared perfectly healthy while dropping a massive percentage of the payload. I only identified the issue by implementing fixed-window telemetry.

Over a 25-second sample:

- Baseline (No RPC lookup): 442 trades processed
- RPC on buys only: 276 trades processed
- RPC on all trades: 147 trades processed

Synchronous I/O cost us two-thirds of our data.

Why Increasing Buffer Size Fails

A common initial reaction is to increase the channel size.

Buffers exist to absorb temporary bursts in traffic. They give the consumer time to catch up after a spike. They do not solve a fundamental mismatch between producer and consumer throughput. Given the sustained volume of Pumpfun, a larger buffer simply takes longer to fill. The outcome is the same.

Switching to an unbounded channel is worse. The queue grows indefinitely until the application runs out of memory and crashes. Losing some updates is preferable to crashing the entire indexer.

Queue capacity was a symptom. The root cause was executing network requests inside a sequential processing loop.

Unblocking the Stream with Async Caching

The solution requires removing the RPC call from the hot path and introducing a shared memory cache.

When a cache miss occurs, the processor dispatches the pool address to a background worker and immediately returns to the stream. The trade is emitted with an “unresolved” status. The background task executes the RPC call asynchronously and populates the cache. Subsequent trades for that pool resolve instantly from memory.

Trades emitted before the background lookup completes are not lost. They are temporarily stored, and a background repair pass enhances them once the pool data enters the cache.

We also implemented two optimizations to keep the system efficient.

  • First, when a pool creation event appears on the stream, we populate the cache directly from the event payload to skip the network entirely.
  • Second, we maintain a registry of active requests to prevent duplicate RPC calls for the same pool.

The ingestion processor now executes with zero blocking waits.

Defining the Drop Policies

By decoupling these processes, we can apply different backpressure rules to our channels.

Queue          Capacity     Policy
----------------------------------------------
Event Stream   10,000       Never Overflow
Pool Lookup    1,000        Drop and Count

The event queue must never overflow because a dropped trade is permanently lost. It only needs enough capacity to handle network spikes, as the processor now drains it faster than the stream fills it.

Conversely, dropping a background pool lookup has no negative impact. The pool will likely trade again in seconds, triggering a new lookup request. We allow this queue to shed load safely and simply record a metric. Load shedding is highly effective for repeatable operations but unacceptable for unique events.

Results

A 190-second production test demonstrates the improvement.

Metric            | Before                | After
------------------|-----------------------|-------------------------------
Events per second | 5.9                   | 118 sustained, 201 peak
Updates dropped   | Unknown               | 0
Cache hit rate    | 0%                    | 92%
RPC efficiency    | 1 call per uncached   | 457 calls for 1,486 misses
                    trade


The 300ms RPC latency still exists. It has simply been removed from the critical ingestion path.

The architectural rule is absolute: never place synchronous network waits inside a sequential data stream.