Errors, Consistency, Repair, and Shutdown

Garage does not try to make partial failure disappear. It represents where an operation has reached, returns once a stated consistency threshold is met, and keeps repair machinery running afterward.

Quorum is a success contract

Metadata placement returns one or more write sets. Table::insert sends the update to their nodes and succeeds only when each set has its required quorum. Reads collect a read quorum, decode each result, and merge differences with the entry’s CRDT rule.

This is not full immediate agreement. It is a contract that enough intersecting replicas have participated to provide the configured behavior.

Read repair is explicit

If a monotonic read sees missing or unequal replicas, Garage writes the merged entry back. Foreground observation becomes a repair opportunity. Merkle-based table synchronization separately finds divergence even when nobody reads a key.

Upload failure leaves evidence

Large PUT establishes Uploading object and version metadata before block transfer. The armed cleanup guard handles errors and cancellation. Publishing Complete happens last, so readers do not interpret a partly transferred version as a finished object.

Immutable blocks simplify retry

A block is named by the hash of its stored bytes. Re-sending the same hash is idempotent, and reads verify integrity. Reference-count changes enqueue delayed resync checks; a missing required block is fetched from another responsible node.

Deletion is deliberately conservative

Metadata tombstones must remain long enough to prevent deleted values from returning during reconciliation. Blocks whose reference count reaches zero are not removed immediately. GC and resync encode delay and propagation rules because premature deletion is harder to repair than temporary extra storage.

Worker failure and shutdown

A background worker error is logged, counted, and retried after exponential delay. A Tokio watch channel broadcasts stop intent. Workers are given time to finish their current unit; after an eight-second drain deadline, remaining worker futures are cancelled. API servers and cluster communications receive the same shutdown signal and are joined by the server orchestrator.

Graceful shutdown here is bounded cooperation, not a promise to wait forever.