Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Generics, Traits, and Extension Boundaries

GenericsStatic inside, dynamic at plansRevision 9fdf144

DataFusion uses traits where third parties or runtime query shape require an open set: ExecutionPlan, PhysicalExpr, TableProvider, QueryPlanner, optimizer rules, scalar functions, and MemoryPool. These commonly appear as Arc<dyn Trait + Send + Sync> because one plan must contain unlike nodes.

Generics are strongest inside reusable implementations. Stream adapters are generic over their inner stream or produced item; expression kernels can be generic over Arrow array types; tree utilities accept closures that preserve the visitor operation without creating a new public object hierarchy.

Associated types and trait bounds tie values together when callers benefit from that relationship. Type erasure happens at the collection boundary:

concrete scan/filter/aggregate structs
             ↓ coercion
Vec<Arc<dyn ExecutionPlan>> and Arc<dyn PhysicalExpr>
             ↓ execution
Pin<Box<dyn RecordBatchStream + Send>>

This keeps user extensions possible and compile times manageable. Making the entire plan a deeply nested generic type would encode a runtime SQL statement in a compile-time type the caller cannot name.