Async, Concurrency, and Backpressure

Tokio schedules connection tasks, control-plane watches, timers, DNS, and request futures. Tower supplies the concurrency contract inside each path.

Connections and requests are nested concurrency

Listeners spawn independent connection lifecycles. A multiplexed HTTP/2 connection carries concurrent streams; a destination cache holds independent per-target services; a balancer distributes calls across concurrently ready endpoints. These are different levels, not one worker-count setting.

poll_ready is admission control

Tower separates “may I send?” from call. Queues, buffers, failfast, load shed, connection pools, and balancers compose by propagating readiness. A bounded NewQueue can wait through brief unavailability; failfast/load-shed turn sustained lack of capacity into an explicit error.

Watch channels carry latest control state

Policy, profiles, endpoint sets, and TLS credentials change while traffic flows. watch::Receiver<Arc<T>> gives many stacks the latest immutable value without queueing every intermediate configuration. Endpoint discovery uses a change stream because additions/removals must update a live pool.

Data-plane concurrency remains bounded by demand

Idle caches lazily build services and evict unused entries. Per-target queues have capacities and timeouts. Retry budgets cap amplification. Response body wrappers retain permits/metrics state until end-of-stream or drop, tying accounting to actual ownership.

No substantial CPU data parallelism is central to the proxy. Parallelism comes from Tokio polling independent network work on runtime workers; the engineering focus is controlled concurrency and predictable tail latency.