Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Build a Smaller DataFusion

Reconstruction projectPlan, optimize, streamAnalysis first
  1. Define a tiny typed columnar Batch with integer and string columns.
  2. Define logical Scan, Filter, Project, and Aggregate plan nodes.
  3. Build a fluent DataFrame API; add a small expression parser only afterward.
  4. Write one logical rule that pushes a filter below a projection.
  5. Define an object-safe physical operator returning a stream of batches.
  6. Execute one partition end to end using poll-driven backpressure.
  7. Add partitioned scans and partial/final aggregation.
  8. Put a bounded channel at one repartition boundary and observe pressure.
  9. Add byte reservations to the hash aggregate; spill sorted runs when denied.
  10. Make dropping the root stream cancel producers and release reservations.
  11. Add an EXPLAIN tree and per-operator batch/row/time metrics.

Production DataFusion additionally provides Arrow’s full type system, SQL coverage, catalogs, file and object-store formats, statistics, many optimizer rules and join algorithms, window functions, UDF families, distributed extension points, spill implementations, and extensive compatibility testing.