One Event, End to End

Assume a source emits a batch of log events, a synchronous transform changes them, and a network sink delivers them.

1. The source sends an EventArray

SourceSender accepts individual events, streams, or exact-size batches. Batch input is converted and chunked into EventArray values before an async send. If downstream capacity is unavailable, this future waits rather than blocking the Tokio worker thread.

Source: source_sender/output.rs:279

2. A pump isolates the source from topology fanout

Each source output has a pump task. The pump selects between source data and fanout control messages, adds source metadata, then awaits Fanout::send. Separating the source server from fanout permits multiple named outputs and lets reload commands be processed even while a source is idle.

Source: topology/builder.rs:1009

3. Fanout clones ownership, not merely bytes

Fanout arms one send future per downstream component. The last destination receives the original array; earlier destinations receive clones. Event metadata—including shared delivery finalizers—travels with those copies. Fanout completes only when the sends complete or a topology control operation changes their membership.

Source: fanout.rs:313

4. The transform owns its state

A synchronous transform task receives an array, invokes transform_all, fills a reusable TransformOutputsBuf, and awaits its output fanouts. A stateless, compute-heavy transform may instead clone the transform into bounded spawned work and use FuturesOrdered to preserve output order.

Sources: builder.rs:1325, builder.rs:1350

5. The sink buffer absorbs or propagates pressure

The sink’s input is a BufferReceiverStream<EventArray>. Its paired BufferSender applies the configured policy: await free space, drop the newest item, or overflow to another stage. Memory and disk implementations share this topology-facing contract.

Source: channel/sender.rs:241

6. Stream combinators build a request

A typical network sink filters and normalizes events, batches by size/time, encodes requests concurrently under a limit, discards request-build failures through explicit policy, and converts the resulting request stream into a generic Driver.

Source: elasticsearch/sink.rs:64

7. The driver respects service capacity

The driver polls Tower Service::poll_ready, calls ready services, and holds response futures in an unordered in-flight set. Its biased select! gives completed responses priority so acknowledgements free buffer and source capacity promptly.

Source: driver.rs:86

8. Delivery moves backward

After service-level retry is exhausted or succeeds, the driver maps the response to Delivered, Rejected, or another status and updates the request’s finalizers. When the last shared finalizer is dropped, its batch notifier resolves the source’s receiver.

Source: driver.rs:199

source → pump → fanout → transform → fanout → buffer → batch → service
   ↑                                                         │
   └──────────── aggregated finalizer status ────────────────┘