The Standard Library as Architecture
Arc<dyn ExecutionPlan>makes a plan node shareable by optimizers, parents, metrics, and partition streams without pretending there is one unique owner.Vec<Arc<dyn ExecutionPlan>>represents heterogeneous tree children; ordinary recursion and iterator transforms implement plan rewrites.Option,Result, and?make missing statistics, optional metrics, and fallible planning explicit instead of relying on sentinel values.HashMapandHashSethold catalogs, function registries, grouping state, and optimizer bookkeeping where identity rather than sequence matters.- atomics cheaply account for shared pool usage and metrics; mutexes protect compound policies such as fair sharing among spillable consumers.
Dropon aMemoryReservationshrinks the pool and eventually unregisters its consumer. Resource accounting follows ownership on success, error, and cancellation paths.Pin<Box<dyn Stream<Item = Result<RecordBatch>> + Send>>gives the executor a stable, uniform asynchronous output type while each stream keeps its own concrete state machine.
The architecture is not “Tokio all the way down.” Standard ownership types describe durable structure; async types describe when progress can pause.