Build a Smaller DataFusion
- Define a tiny typed columnar
Batchwith integer and string columns. - Define logical
Scan,Filter,Project, andAggregateplan nodes. - Build a fluent DataFrame API; add a small expression parser only afterward.
- Write one logical rule that pushes a filter below a projection.
- Define an object-safe physical operator returning a stream of batches.
- Execute one partition end to end using poll-driven backpressure.
- Add partitioned scans and partial/final aggregation.
- Put a bounded channel at one repartition boundary and observe pressure.
- Add byte reservations to the hash aggregate; spill sorted runs when denied.
- Make dropping the root stream cancel producers and release reservations.
- Add an
EXPLAINtree and per-operator batch/row/time metrics.
Production DataFusion additionally provides Arrow’s full type system, SQL coverage, catalogs, file and object-store formats, statistics, many optimizer rules and join algorithms, window functions, UDF families, distributed extension points, spill implementations, and extensive compatibility testing.