Errors, Cancellation, Memory, and Spill

DataFusionError carries planning, schema, execution, external, and contextual failures through one public result vocabulary. Optimizer EXPLAIN handling can capture a rule’s failure as part of the explanation rather than losing the phase that produced it.

Execution streams yield Result<RecordBatch>, so failures arrive at the same point as data. Task-backed receiver streams additionally supervise their JoinSet: ordinary errors enter the stream, panics are resumed rather than silently becoming an early end-of-stream, and dropping the stream cancels outstanding producers.

Large stateful operators use MemoryConsumer and MemoryReservation. A failed try_grow makes an operator spill and retry when supported, or terminate with an allocation error. The default pool is unbounded, while greedy, fair-spill, and consumer-tracking policies can be selected deliberately.

Cancellation is ownership-based. Dropping the final output stream stops demand and releases its children, task supervisors, channels, and reservations. External I/O still needs cancellation-safe futures and temporary files need their own cleanup guards; stream drop is the top-level signal, not magic preemption.

Metrics and EXPLAIN expose plan shape, partitioning, row counts, elapsed work, and spill behavior. In a query engine, observability is part of correctness: without it, a valid but catastrophically expensive plan is hard to distinguish from a hung one.