Concurrency, Async, and Backpressure

DataFusion expresses potential parallelism with partitions. An ExecutionPlan advertises output partitioning, and execute(partition, TaskContext) creates one SendableRecordBatchStream for that partition. Independent streams may be driven on multiple Tokio workers.

Within an ordinary operator chain, downstream polling supplies natural backpressure: if nobody polls the root, it stops polling its input. At an explicit concurrency boundary such as repartitioning, bounded channels connect producer tasks to output partitions. A full channel makes the producer await, carrying pressure upstream.

collect_partitioned demonstrates deliberate task parallelism: it creates all partition streams, spawns a collection future for each in a JoinSet, and restores partition order after completions arrive. Other execution paths need not spawn one Tokio task per node.

Blocking filesystem or compression work can use a blocking bridge. It must not occupy a Tokio worker that should keep polling network and object-store futures.

There are therefore three separate controls:

• partition count: available independent work;
• channel capacity: buffered batches across a task boundary;
• memory pool: retained operator state across the entire context.

They solve different problems and should not be presented as one “thread count.”