Vector: Orientation

Vector collects logs, metrics, and traces, transforms them, and delivers them to external systems. Its architectural center is not any particular integration. It is the machinery that compiles a configured directed acyclic graph into a running, bounded, observable data plane.

The architectural center

configuration DAG
       ↓ build and connect
Source task → pump → Fanout → Transform task → Fanout → sink buffer
                                                          ↓
                                               batch → request → service
                                                          ↓
                                                   destination

source acknowledgement ← event finalizers ← delivery result

The forward path carries owned EventArray values. The backward path is not a second channel through every component: shared finalizers attached to events aggregate delivery status and eventually resolve a source’s one-shot receiver.

Design thesis

Vector compiles configuration into a bounded graph of independently owned tasks, then carries delivery accountability backward through finalizers attached to the events moving forward.

• Bounded edges turn overload into backpressure instead of memory growth.
• Stateful components own their mutation inside long-lived tasks.
• Generic sink services separate batching and retry from integrations.
• Shutdown proceeds source-first so the graph becomes finite before draining.

Repository centers

• src/topology builds, connects, spawns, reloads, and stops the component DAG;
• lib/vector-core defines events, sources, transforms, sinks, fanout, and component-facing contracts;
• lib/vector-buffers implements bounded memory and persistent buffer edges;
• lib/vector-stream drives request streams into Tower services and records final delivery status;
• src/sinks/util provides batching, request building, retry, rate limiting, adaptive concurrency, and service composition;
• src/sources, src/transforms, and src/sinks contain concrete components.

The useful reading boundary

We will follow one batch from a generic source through a synchronous transform to an Elasticsearch-like network sink. The component is illustrative; the interesting code is shared infrastructure used by many integrations.

We will also trace fanout, overload, acknowledgement, reload, and shutdown. Reading only the successful network call would miss Vector’s real engineering.

What the reconstruction preserves

The smaller implementation will preserve a validated acyclic graph, one task per stateful component, bounded edges, fanout, optional ordered parallel transforms, batched service calls, classified retry, aggregated delivery status, and source-first graceful shutdown.