One SQL Query, Fully Traced

Consider SELECT city, count(*) FROM trips WHERE fare > 20 GROUP BY city.

1. SessionContext::sql snapshots session state and asks it to create a logical plan. Parsing yields a SQL AST; SqlToRel resolves tables, columns, functions, and types into DataFusion expressions and plan nodes.
2. The analyzer checks and coerces the plan. Logical optimizer rules can push the filter toward the scan, prune columns, and simplify expressions without choosing threads or file partitions.
3. The query planner maps logical nodes to Arc<dyn ExecutionPlan> nodes. A physical optimizer then enforces distribution and ordering requirements and may introduce repartition or coalesce operators.
4. Execution asks the root for an output stream. For several root partitions, callers can keep separate streams or place a coalescing node above them.
5. Polling the root recursively polls its children. The scan obtains a batch; the filter computes a selection mask; the partial aggregate updates per-partition state; a repartition boundary redistributes rows; the final aggregate emits grouped batches.
6. collect drains the stream into Vec<RecordBatch>. A streaming caller can instead process each batch and release it before requesting the next.

Nothing “runs the whole plan” when it is constructed. Plan creation describes the graph; polling an output stream drives the graph.