Why Is It Designed This Way?

Why separate metadata tables from object blocks?

Metadata is small, structured, mutable, and must merge. Object bytes are large and expensive to copy but become simple once immutable and content-addressed. Using one protocol and representation for both would make each inherit the other’s worst constraints.

Why use CRDT tables instead of one primary node?

Garage targets multiple sites connected by ordinary, failure-prone networks. Mergeable entries let replicas accept and reconcile state without putting one leader on every metadata operation’s availability path.

Why acknowledge after a quorum rather than every replica?

Waiting for all replicas turns one slow or disconnected node into global unavailability. A quorum gives a precise threshold, while background synchronization and the remaining RPC futures continue convergence.

Why write Uploading metadata before streaming blocks?

The system needs durable evidence of incomplete work. Block references must not point to a version the metadata layer cannot identify, and cancellation must leave enough identity for cleanup.

Why mark the object Complete last?

Publication is a commit point. Readers should either see the prior complete version or the new complete version, never infer completeness from scattered blocks that happened to arrive first.

Why pipeline one upload with tiny bounded channels?

Reading, checksumming, CPU transformation, and network replication have different bottlenecks. Overlap improves throughput, while capacities 2/1/1 prevent a fast client or CPU stage from buffering an entire object in memory.

Why have both per-request and global limits?

A per-request write cap provides fairness between uploads. The global buffered-kilobyte semaphore protects node memory across all RPCs. Neither budget can substitute for the other.

Why are blocks content-addressed?

Immutable bytes named by a hash are easy to verify, retry, deduplicate, and repair. Mutable object meaning stays in the CRDT metadata that refers to those hashes.

Why treat repair as normal background work?

Replica divergence is an expected consequence of partial failure and changing cluster layouts. A distributed store is not reliable because failure never happens; it is reliable because it continuously discovers and repairs the allowed incomplete states.

Why is garbage collection so cautious?

Deleting a live block or forgetting a tombstone can destroy data or resurrect a deletion. Retaining extra bytes costs capacity; deleting too early can break the consistency model. Garage chooses the reversible side of that tradeoff.