Why Is It Designed This Way?

Why separate logical and physical plans?

A logical plan says what a query means: scan, filter, join, aggregate. A physical plan says how to execute it: which join algorithm, ordering, and partitioning. Keeping them separate lets semantic rewrites happen before machine and data-layout decisions.

Why return streams instead of completed tables?

An operator can produce a batch as soon as its inputs make progress. The consumer controls the pace by polling, intermediate results need not all be materialized, and dropping the output stream provides a natural cancellation boundary. collect is merely a terminal convenience that deliberately buffers.

Why columnar RecordBatch values?

Arrow arrays amortize dispatch across many values, improve cache locality, and form an interoperability boundary with file formats and other analytical systems. DataFusion moves batches between operators rather than Rust structs for individual rows.

Why are plan nodes trait objects?

The shape and concrete operators of a query are known only after planning. Arc<dyn ExecutionPlan> gives heterogeneous, shared plan trees and a public extension seam. Inside operators, generics still specialize reusable machinery where the concrete type is useful.

Why explicit memory reservations?

Rust prevents memory unsafety, not out-of-memory termination. Hash joins, aggregates, and sorts may retain input-proportional state, so they reserve bytes before growth and either spill or return an error when the pool refuses. Reservation drop returns accounting through RAII.

Why does async not mean every calculation is spawned?

Most operators are streams polled by their downstream consumer. Async is used where progress genuinely waits—object storage, repartition channels, spawned producers, or blocking bridges. Partitioning supplies parallel work; spawning every expression would add scheduling overhead without creating useful independence.