Generics and Component Boundaries

Vector combines static generic pipelines with dynamic component discovery. The interesting engineering lies in where it switches between them.

Configuration uses trait objects

At configuration time, one collection must contain many source, transform, and sink types selected from serialized type tags. SourceConfig, TransformConfig, and SinkConfig are object-safe, cloneable trait boundaries that build erased runtime components.

Source: config/source.rs:83

The dynamic boundary answers “which integration did the user configure?” Once a concrete sink is built, its internal hot path can recover static generic composition.

Runtime component enums erase outer differences

Transform stores boxed function, synchronous, or stream-task transforms. VectorSink stores either a futures Sink<EventArray> or a StreamSink. Topology code needs only these small runtime shapes, not every concrete Kafka, HTTP, file, or remap type.

Bufferable bundles required capabilities

BufferSender<T> is generic over items that support event counting, size measurement, finalization, and required thread-safety. Disk-capable Bufferable adds encoding and grouped-finalizer requirements. This lets memory and disk topology code remain reusable without accepting values it cannot account for or persist.

Source: vector-buffers/lib.rs:98

The sink driver is generic over stream and service

Driver<St, Svc> requires:

St: Stream,
St::Item: Finalizable + MetaDescriptive,
Svc: Service<St::Item>,
Svc::Future: Send + 'static,
Svc::Response: DriverResponse,

It knows nothing about HTTP or Elasticsearch. The request must expose metadata and finalizers; the response must explain delivery status. This is the smallest contract that lets the driver manage capacity, concurrency, telemetry, and acknowledgements.

Stream extension traits form a typed assembly language

A sink pipeline chains batching, partitioning, normalization, bounded request building, filtering, and into_driver. Every combinator changes the stream’s item type. Trait bounds prove that the next stage accepts the previous output.

Source: sinks/util/builder.rs:107

Tower makes delivery policy composable

The concrete service is wrapped in timeout, retry, rate limit, and fixed or adaptive concurrency layers. Generics preserve the full composition without a virtual call at each layer. Only selected storage boundaries erase types when heterogeneous values must coexist.

The pattern is worth copying: use trait objects for heterogeneous construction, enums for a small closed set of runtime modes, and generics for a hot homogeneous pipeline.