Build a Smaller Garage

Rebuild the architectural center, not the S3 syntax. The goal is a small replicated blob store that genuinely preserves streaming, publication, quorums, merge, and repair.

What are we preserving?

• large values stream through bounded stages;
• metadata and immutable blocks have different representations;
• writes publish metadata only after block obligations finish;
• replica operations have explicit quorum thresholds;
• cancellation leaves cleanup work;
• repair can restore a missing block from another replica.

Stage 1: one-node content-addressed storage

Implement put(Read) that chunks input, hashes each block, writes it under its hash, and finally stores a manifest from object key to ordered hashes. Implement get by verifying and concatenating those blocks.

Stage 2: turn PUT into a bounded pipeline

Use bounded channels between chunking, hashing, and disk writing. Add a small concurrent write limit. Measure the maximum number of full blocks retained by the pipeline; it should follow channel capacities, not object size.

Stage 3: add explicit publication state

Represent metadata as:

enum VersionState {
    Uploading { id: VersionId },
    Complete { id: VersionId, blocks: Vec<Hash> },
    Aborted { id: VersionId },
}

Write Uploading first and Complete last. Arm a drop guard that queues an Aborted transition if the upload future disappears before publication.

Stage 4: create three in-process replicas

Give each replica its own metadata map and block directory. A deterministic placement function chooses three replicas. Send writes concurrently and return after two acknowledge. Inject latency and failure into the third.

Stage 5: make metadata mergeable

Give versions stable IDs and timestamps, retain concurrent versions, and define a deterministic merge. Implement a quorum read that merges responses. Do not use “last response wins”; make the conflict rule a type-level operation.

Stage 6: add read repair

When a read finds different metadata, send the merged value back. When a manifest names a locally missing block, fetch it from another placement node, verify the hash, and store it.

Stage 7: supervise background repair

Define a small Worker trait with work and wait_for_work. Run a repair queue with error backoff and a shutdown signal. Expose worker state so repair is observable rather than magical.

Failure exercises

• Cancel after writing two blocks but before publication.
• Lose one metadata response during a write.
• Corrupt one block and verify that its hash detects the fault.
• Remove one replica’s block and let repair fetch it.
• Partition one replica, perform writes, reconnect, and converge metadata.
• Begin shutdown while a repair unit is running; verify the bounded drain.

Compare with production Garage

Your version can teach the center in a few modules. Garage additionally provides S3 compatibility, authenticated streaming, encryption, multipart uploads, durable embedded databases, cluster layout transitions, Merkle table sync, conservative tombstone and block GC, observability, multiple APIs, data scrubbing, migrations, and operational tooling.

Those are not incidental extras. They are the hardening needed when the toy failure injector becomes real disks, networks, upgrades, and operators.