Async, Concurrency, and Distributed Parallelism

Garage uses concurrency at several independent levels. Collapsing them into “Tokio makes it parallel” hides the resource policy.

Request concurrency

The API server runs many connection and request futures on Tokio. A request usually suspends on body input, metadata RPCs, disk, or peer responses rather than blocking an executor thread.

Pipeline concurrency inside one PUT

read_and_put_blocks uses bounded MPSC channels between reading, checksum, encryption/hash, and write stages. futures::join! drives all four at once. Capacity 2/1/1 is a memory bound as well as a scheduling decision: only a few full blocks may wait between stages.

The checksummer is stateful, so blocks pass through it in order. Encryption and hashing are CPU work and cross through spawn_blocking. The async runtime remains available while the blocking pool performs those calculations.

Several block writes from one request

The writer owns a FuturesOrdered of in-flight block operations. It admits another block only below block_max_concurrent_writes_per_request. Completion is observed in input order, while the underlying writes can overlap. This prevents one large upload from creating an unbounded future set.

Parallel RPCs and quorum completion

A table insert sends updates to the placement nodes concurrently. It can return when every required write set has reached quorum. Remaining calls are driven in a spawned task, spreading the update without adding their full tail latency to the client response.

Block upload adds another budget: a semaphore counts buffered kilobytes held for peer transmission. Its owned permit travels inside RequestStrategy and is dropped only when the RPC set finishes.

Long-running background concurrency

Every table owns Merkle-update, synchronization, garbage-collection, and queued-insert workers. The block manager owns resync and scrub workers. A shared BackgroundRunner supervises them, records status, applies exponential error delay, and coordinates shutdown.

The important pattern is nested budgets:

many requests
  └─ bounded pipeline buffers per request
       └─ bounded block-write futures per request
            └─ quorum RPC fan-out
                 └─ global buffered-byte semaphore

Async supplies suspension. These explicit limits supply predictable resource use.