Build a Smaller Vector

Rebuild the data plane and its guarantees, not a toy configuration syntax.

Build a validated DAG whose tasks exchange owned events through bounded edges, fan out reliably, batch and retry sink requests, aggregate delivery status, and drain from sources to sinks.

1. Start with a synchronous concrete pipeline

Define Event { id, payload }, one generator, one uppercase transform, and one collecting sink. Pass a Vec<Event> through direct calls. Record expected order and which component owns each vector.

2. Represent and validate the graph

Store component IDs and input IDs in HashMaps. Reject missing inputs, duplicate IDs, incompatible event types, and cycles before constructing runtime state. Topologically sort the graph and test several invalid configurations.

3. Add bounded Tokio edges

Run source, transform, and sink as long-lived tasks connected by small mpsc::channels. Let receiver closure end downstream tasks naturally. Write a test where a slow sink visibly suspends the source.

4. Implement fanout

Clone events to two bounded destination senders and await both sends. Make one branch slow. Then add an explicit per-branch Block versus DropNewest policy and observe how it changes the system guarantee.

5. Separate stateless and stateful transforms

Define a cloneable per-event transform and a stream-owning stateful transform. For the stateless kind, process batches in bounded spawned tasks and release results in input order. Demonstrate that unordered release is faster in one test but observably different.

6. Build a sink service pipeline

Batch by count or timeout, encode a request, and send it through a small service trait:

trait Service<Req> {
    type Response;
    type Error;
    async fn ready(&mut self) -> Result<(), Self::Error>;
    async fn call(&mut self, req: Req) -> Result<Self::Response, Self::Error>;
}

Keep several calls in flight under a semaphore limit. Interleave response completion with new input rather than awaiting each request serially.

7. Add retry classification

Create Delivered, Retryable, and Rejected outcomes. Use bounded backoff and a maximum attempt count. Simulate “destination accepted, response lost” to show why at-least-once can duplicate data.

8. Attach delivery finalizers

Give each source batch a shared status object and one-shot receiver. Ensure fanout branches retain ownership. The source receives success only after all branches finalize. Test rejection dominance, transform filtering, and a dropped in-flight future.

9. Add memory and durable buffers

Begin with a bounded in-memory queue. Then implement a tiny append-only disk queue with record checksums and explicit acknowledgement before deletion. Crash between write, send, and acknowledgement, then document what is replayed.

10. Drain and reload

On shutdown, stop sources first, drop output senders, and join every task with a deadline. For reload, pause a fanout destination, replace its component and sender, then resume without restarting unchanged branches.

11. Compare with production

Map the reconstruction to TopologyPiecesBuilder, SourceSender, source pumps, Fanout, BufferSender, SyncTransform, TaskTransform, sink stream builders, Tower Service, Driver, and EventFinalizer.

The smaller system omits hundreds of integrations, production codecs, schema propagation, adaptive request concurrency, full disk-buffer recovery, internal telemetry, resource conflict detection, and rollback hardening. It should still make overload visible, avoid unbounded work, preserve chosen ordering, report delivery only after every branch, and shut down without abandoning ownership.