Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DataFusion: Orientation

Query engineArrow-nativeRevision 9fdf144

Apache DataFusion is an extensible query engine. It accepts SQL or a DataFrame API, produces an immutable logical plan, rewrites it, chooses a physical plan, and pulls Arrow RecordBatch values through a partitioned execution graph.

SQL -> LogicalPlan -> optimized LogicalPlan -> ExecutionPlan
    -> one stream per partition -> RecordBatch consumer

Design thesis

DataFusion separates query meaning from execution strategy, then represents execution as poll-driven, partitioned batch streams with explicit memory reservations for operators that retain large state.