Rust by Example: Real Repositories

Small examples teach syntax. Mature repositories teach how Rust’s mechanisms fit together under real constraints.

This book walks through production Rust codebases from the outside in. It starts with what a user calls, follows that operation across crates and modules, and stops at the boundary where the work is actually performed. Along the way, it explains why the types, ownership relationships, tasks, channels, and error paths have the shapes they do.

public operation
       │
       ▼
API and type boundary
       │
       ▼
shared abstraction
       │
       ▼
concrete implementation
       │
       ▼
I/O, operating system, or external service

The goal is not to inventory every file. It is to develop a repeatable way to answer:

• Where does an operation enter the system?
• Which values own the state it needs?
• Which traits connect generic code to concrete behavior?
• Where can execution suspend, fail, or be cancelled?
• What must happen before a resource can be reused?

The current walkthroughs cover twenty-three contrasting systems:

• SQLx combines generics, protocol state machines, async I/O, pools, worker threads, and cancellation-safe cleanup;
• ripgrep uses synchronous parallelism, work stealing, worker-local state, and serialized output commits;
• Aikido uses bounded Tokio pipelines, task-owned mutation, desired state, and reconciliation;
• Axum adapts typed handlers into composable Tower services driven by Hyper and Tokio;
• Vector compiles a dataflow DAG into Tokio tasks and bounded edges with explicit durability and acknowledgement semantics;
• Helix serializes editor mutation while background jobs return owned results;
• uv combines async preparation, resource-specific limits, in-flight deduping, atomic caches, and target locking;
• rust-analyzer coordinates mutable inputs with cancellable read snapshots and demand-driven Salsa computation;
• Linkerd2-proxy composes target-specialized Tower stacks around readiness, discovery, streaming completion, and graceful drain;
• Nushell combines lazy structured iterators, explicit Rayon parallelism, and OS process lifecycle; and
• Apalis adapts ordinary async functions into backend-independent durable job services with typed extraction, middleware, retry, and tracked shutdown;
• Watchexec turns concurrent filesystem, signal, and keyboard observations into serialized policy and independently supervised child-process transitions;
• sccache separates high request concurrency from bounded compiler-process and jobserver capacity;
• Atuin turns offline local changes into encrypted append-only sync records;
• Garage coordinates replicated object storage, repair, and bounded background work;
• Zellij combines thread-owned session mutation with async PTY I/O and fair per-pane output handling;
• Tokio implements task scheduling, waking, I/O readiness, timers, cancellation, and a separate blocking boundary; and
• Deno connects V8 promises to typed Rust ops, native resources, Tokio-driven progress, and parallel worker isolates;
• Bevy derives safe parallel ECS scheduling from typed system access;
• godot-rust protects Rust ownership across an engine-controlled FFI lifecycle;
• Quinn joins a deterministic QUIC state machine to async sockets and targeted stream wakeups;
• DataFusion plans and executes partitioned Arrow batch streams under explicit memory accounting; and
• Rerun carries typed Arrow components through concurrent ingestion, temporal storage, immediate-mode querying, and GPU rendering.

How to Read a Repository

Do not begin by reading files in directory order. Begin with a concrete operation and construct the smallest map that can explain it.

Establish the revision

Record the commit, enabled features, and relevant runtime. Repository behavior can change while filenames remain familiar.

git rev-parse --short HEAD
cargo metadata --no-deps

Find the public boundary

Start with one thing a user can do:

sqlx::query("SELECT id FROM users")
    .fetch_one(&pool)
    .await?;

Identify the public function, the value it constructs, and the method that causes work to begin. This creates a path through the repository instead of a list of unrelated modules.

Build two maps

The static map shows crates and dependencies:

The dynamic map shows what happens over time:

Do not combine these too early. A crate dependency graph and a request sequence answer different questions.

Follow state, not only function calls

At each step, record:

• the important value;
• who owns it;
• whether it is borrowed mutably or shared;
• which state transition occurs;
• where an error or cancellation can interrupt the transition.

A function-call trace explains control flow. A state trace explains why the API has its shape.

Read bounds as architecture

Translate generic bounds into concrete statements. For example:

E: Executor<'c, Database = DB>

means the executor and query must agree on one database implementation. The bound is not incidental compiler syntax; it prevents a PostgreSQL query from being executed by a MySQL executor.

Verify the map

Search for every implementation of the important trait, inspect the cleanup path, and check at least one test or example. A trustworthy map includes the unhappy path and names the source locations that support it.

Build a Type-Driven API, One Decision at a Time

Rust API design becomes much easier when you stop asking, “Which advanced feature should I use?” and instead ask:

What relationship should the compiler preserve for the caller?

This chapter reconstructs the progress-bar API developed in Will Crichton’s Type-Driven API Design in Rust talk. The point is not the progress bar. The point is the sequence: begin with working concrete code, identify a restriction that is not essential, and remove exactly that restriction. Each step introduces a Rust feature because the API needs it.

The final design combines generics, associated types, trait bounds, blanket implementations, ownership-consuming methods, and typestate. You will not begin with any of those. You will arrive at each one because the previous API has a specific limitation.

The code below is a runnable reconstruction of the presentation’s edits, not a verbatim transcription or the speaker’s original source tree.

How to work through this chapter

Treat this as a coding session, not reference material.

1. Create a blank Rust binary called progress.rs.
2. Type the current checkpoint into it. Do not paste the final example.
3. Run it and confirm its behavior.
4. Stop at every Your turn prompt and answer before scrolling farther.
5. Make your own smallest change, even if you expect it to fail.
6. Read the compiler message before comparing with the next checkpoint.

Every version is intentionally incomplete. Do not “improve” three stages at once: the purpose is to feel which requirement causes which Rust feature to appear.

If you are using Cargo, the loop is:

edit progress.rs → cargo run → read the result → make one design decision

The completed reconstruction lives in examples/type-driven-progress, but do not open it until the typestate checkpoint.

Checkpoint 1: make the behavior real

The presentation starts with behavior directly inside main. Type this version before designing an API. The delay is shortened so the exercise stays quick:

use std::{thread::sleep, time::Duration};

const CLEAR: &str = "\x1B[2J\x1B[1;1H";

fn expensive_calculation(_n: &i32) {
    sleep(Duration::from_millis(100));
}

fn main() {
    let values: Vec<i32> = vec![1, 2, 3];
    let mut index: usize = 1;

    for value in values.iter() {
        println!("{}{}", CLEAR, "*".repeat(index));
        index += 1;
        expensive_calculation(value);
    }
}

It works. That matters. We now have observable behavior to preserve while the representation changes.

The code contains several accidental decisions:

• the collection must be a Vec;
• each element must be an i32;
• the progress state is mixed into main;
• the work and progress rendering are coupled together.

These are not all bad simultaneously. They are simply more specific than the problem requires. Good API evolution removes them one at a time, so every compiler error has a clear cause.

Your turn: Which lines are progress-display machinery rather than the application’s real work? Move only those lines into a function named progress while preserving the output.

Checkpoint 2: extract the first API

The first extracted function remains deliberately concrete and borrows the vector:

use std::{thread::sleep, time::Duration};

const CLEAR: &str = "\x1B[2J\x1B[1;1H";

fn progress(values: &Vec<i32>) {
    let mut index: usize = 1;

    for value in values.iter() {
        println!("{}{}", CLEAR, "*".repeat(index));
        index += 1;
        expensive_calculation(value);
    }
}

fn expensive_calculation(_n: &i32) {
    sleep(Duration::from_millis(100));
}

fn main() {
    let values: Vec<i32> = vec![1, 2, 3];
    progress(&values);
}

The behavior is unchanged, but the caller now has a named operation. Its signature still says exactly what it accepts:

fn progress(values: &Vec<i32>)

Your turn: List every restriction in this signature. Which restriction prevents the same progress logic from accepting a Vec<String>?

Checkpoint 3: remove the item-type restriction

The progress display never examines an i32. If that was the restriction you selected, make the smallest possible signature change that also accepts a Vec<String>.

Your turn: Change only the signature. What symbol will stand for the element type that the caller chooses?

After trying it, compare with this version:

fn progress<T>(values: &Vec<T>) {
    for (index, _value) in values.iter().enumerate() {
        println!("{}", "*".repeat(index + 1));
    }
}

fn main() {
    progress(&vec!["parse", "compile", "test"]);
}

T does not mean “some dynamically typed value.” It means that each call has one concrete element type selected at compile time. Rust normally generates a specialized version of the generic code for every concrete type used.

Run it with integers and strings. The useful design question is not “Where can I add generics?” It is:

Which facts does this implementation genuinely need to know?

This implementation needs to obtain items in sequence. It does not need to know their type or that they came from a Vec.

Checkpoint 4: remove the Vec restriction

Now try to call the function with (0..).take(3). It fails because a range iterator is not a Vec, even though it can provide everything the progress display needs.

Your turn: Describe the capability the function needs without naming a container. Does it need indexing, insertion, ownership of contiguous memory, or only one item after another?

Many APIs prematurely accept Vec<T> when they only need sequential access. Rust already has a trait for “produces one value after another”: Iterator. Change the function to accept a caller-chosen iterator:

fn progress<Iter>(iter: Iter)
where
    Iter: Iterator,
{
    for (index, _item) in iter.enumerate() {
        println!("{}", "*".repeat(index + 1));
    }
}

fn main() {
    progress([10, 20, 30].into_iter());
    progress((0..).take(3));
}

Read the bound as a promise:

Iter: Iterator

means, “the caller may choose any concrete Iter, provided it implements the Iterator contract.” That contract gives the implementation access to next() and to the iterator’s item type.

Pause on Iterator::Item

Before looking up the trait, predict what next must return. It cannot return an item every time because iteration eventually ends.

Your turn: Write the return type of next using Option and a placeholder item type. Then compare it with the essential portion of Iterator:

#![allow(unused)]
fn main() {
trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;
}
}

Item is associated with the iterator implementation. Once Rust knows the concrete iterator type, it can determine the one type that iterator yields:

std::vec::IntoIter<String>::Item = String
std::ops::Range<u64>::Item       = u64

That is different from a generic trait such as Convert<T>, where one type might implement the same trait several times for different Ts. An associated type says these facts belong together: this iterator has this item type.

This “keep related facts together” idea will recur throughout the design.

Checkpoint 5: give the loop back to the caller

Our function accepts arbitrary iterators, but it still owns the entire loop. That means the caller cannot naturally perform work on each yielded item.

Your turn: Write the call site you wish existed. Keep the caller’s for loop and add progress reporting without putting process(item) inside the library.

One desirable call site is:

for item in values.into_iter().progress() {
    process(item);
}

The caller owns the work. The library decorates iteration with a side effect.

Your turn: A value used by a for loop must be iterable. What could our API return that preserves the underlying items while storing an index?

This suggests an iterator adapter: a struct that owns another iterator and implements Iterator itself. First write only the state it must remember:

#![allow(unused)]
fn main() {
struct Progress<Iter> {
    iter: Iter,
    index: usize,
}

impl<Iter> Progress<Iter> {
    fn new(iter: Iter) -> Self {
        Self { iter, index: 0 }
    }
}

impl<Iter> Iterator for Progress<Iter>
where
    Iter: Iterator,
{
    type Item = Iter::Item;

    fn next(&mut self) -> Option<Self::Item> {
        let item = self.iter.next()?;
        self.index += 1;
        println!("{}", "*".repeat(self.index));
        Some(item)
    }
}
}

The key line is:

type Item = Iter::Item;

The wrapper preserves the underlying iterator’s item type. If it wraps an iterator of String, it is also an iterator of String. The progress layer changes behavior without changing the values flowing through it.

Notice the ownership model:

• Progress<Iter> owns Iter;
• next(&mut self) mutates the iteration position;
• each returned Iter::Item is moved out to the caller;
• no lifetime parameter is necessary because the wrapper owns the iterator.

An iterator may itself contain references, such as slice::Iter<'a, T>. In that case the lifetime is already carried inside Iter; Progress<Iter> does not need to duplicate it.

Checkpoint 6: earn the .progress() syntax

At this point, run:

for item in Progress::new(values.into_iter()) {
    process(item);
}

It composes correctly, but the constructor makes the progress implementation the visual center of the call. We want values.into_iter().progress().

Your turn: Can you write an inherent impl adding progress directly to std::vec::IntoIter<T> and every other iterator? Which Rust ownership rule prevents a library from freely adding inherent methods to foreign types?

We cannot add an inherent method to every iterator type: those types belong to the standard library or to callers.

An extension trait adds the method through trait resolution:

trait ProgressIteratorExt: Iterator + Sized {
    fn progress(self) -> Progress<Self> {
        Progress::new(self)
    }
}

impl<Iter> ProgressIteratorExt for Iter
where
    Iter: Iterator,
{}

The implementation is blanket: every type satisfying Iterator receives the extension trait implementation. Sized permits taking self by value and returning Progress<Self>.

Method-call syntax hides two useful compiler operations:

1. Rust finds an in-scope trait that provides progress.
2. It verifies that the receiver satisfies that trait’s implementation bounds.

This is why importing an extension trait makes methods appear, and why the method can be unavailable for inappropriate types without a runtime check.

The bound belongs where it improves the caller’s experience:

trait ProgressIteratorExt: Iterator + Sized {
    fn progress(self) {}
}

impl<Iter: Iterator> ProgressIteratorExt for Iter {}

fn main() {
    // `u32` is not an iterator, so it does not receive iterator extensions.
    1_u32.progress();
}

Compile this in your scratch project and inspect the diagnostic.

Checkpoint 7: distinguish bounded from unbounded

The simple display can count completed items, but a bar such as [*** ] also needs a total. Some iterators know their exact remaining length:

#![allow(unused)]
fn main() {
let finite = [10, 20, 30].into_iter();
assert_eq!(finite.len(), 3);
}

Others do not have a finite bound:

#![allow(unused)]
fn main() {
let unbounded = 0..;
let first_three: Vec<_> = unbounded.take(3).collect();
assert_eq!(first_three, vec![0, 1, 2]);
}

Your turn: Should every Iterator be required to provide a length? Test that idea against 0... If not, where should the stronger requirement live: on Progress itself or only on the method that needs it?

The standard library represents “knows its exact length” with ExactSizeIterator. We expose with_bound only when that capability is available:

impl<Iter> Progress<Iter>
where
    Iter: ExactSizeIterator,
{
    fn with_bound(mut self) -> Self {
        self.bound = Some(self.iter.len());
        self
    }
}

This is type-driven API design in a practical form. There is no if iterator_has_exact_size branch. The method exists for qualifying concrete types and does not exist for the others.

Two nuances matter:

• A trait bound is a compile-time capability requirement, not an inheritance hierarchy.
• ExactSizeIterator is stronger than Iterator; requiring it everywhere would unnecessarily reject streams and infinite ranges.

The concrete type matters more than what looks finite to a human. For example, an array’s IntoIter has an exact size. Some inclusive integer ranges do not implement ExactSizeIterator because their full length cannot always be represented safely. Let the trait bound express the actual guarantee instead of guessing from the syntax.

Put the stronger bound on the narrowest API that needs it.

Checkpoint 8: deliberately create a bad API

Suppose bounded progress bars show delimiters:

[***   ]

Add a builder-style method:

progress.with_delimiters(('<', '>'))

Now call it on both a bounded and an unbounded progress iterator. An unbounded display only prints a count or spinner, so delimiters have no visible effect:

(0..).progress().with_delimiters(('<', '>'))

Your turn: Do not reach for typestate yet. Explain the bug from the caller’s perspective, then propose at least two fixes. One may change the rendering behavior rather than the types.

This is an API-design smell: the caller can express a configuration that has no meaning. There are several legitimate fixes:

1. Make delimiters meaningful for unbounded output too.
2. Return a runtime error for an invalid configuration.
3. Make with_delimiters implicitly enable bounded display when possible.
4. Encode bounded versus unbounded state in the type and expose the method only in the bounded state.

The talk chooses option four to demonstrate typestate. That is not proof that typestate is always the best product decision. It is best when the states are few, transitions are clear, and preventing misuse justifies more complex types and compiler messages.

Checkpoint 9: encode the state transition

We will now explore the talk’s typestate solution. First represent the states as distinct types:

#![allow(unused)]
fn main() {
struct Unbounded;

struct Bounded {
    len: usize,
    delimiters: (char, char),
}
}

Your turn: Progress<Iter> currently records only the iterator type. Where could it record whether bounded configuration has occurred without a runtime bool?

Add the state as another type parameter:

#![allow(unused)]
fn main() {
struct Progress<Iter, State> {
    iter: Iter,
    index: usize,
    state: State,
}
}

Construction begins in the unbounded state. Write the return type of new before writing its body:

impl<Iter> Progress<Iter, Unbounded> {
    fn new(iter: Iter) -> Self {
        Self {
            iter,
            index: 0,
            state: Unbounded,
        }
    }
}

The crucial transition is with_bound.

Your turn: Fill in only its signature. It starts with Progress<Iter, Unbounded> and must prove bounded configuration in its return type. Should it take &self, &mut self, or self?

It consumes the old value and returns a value with a different type:

impl<Iter> Progress<Iter, Unbounded>
where
    Iter: ExactSizeIterator,
{
    fn with_bound(self) -> Progress<Iter, Bounded> {
        let len = self.iter.len();

        Progress {
            iter: self.iter,
            index: self.index,
            state: Bounded {
                len,
                delimiters: ('[', ']'),
            },
        }
    }
}

The transition consumes self because the implementation must move iter from the old wrapper into the new wrapper. The old unbounded value cannot be used afterward. Ownership and typestate reinforce each other.

Finally, place delimiter configuration on only one state:

impl<Iter> Progress<Iter, Bounded> {
    fn with_delimiters(mut self, delimiters: (char, char)) -> Self {
        self.state.delimiters = delimiters;
        self
    }
}

Now method order is part of the API contract:

let configured = (0..10)
    .progress()                         // Progress<_, Unbounded>
    .with_bound()                       // Progress<_, Bounded>
    .with_delimiters(('<', '>'));       // available for Bounded

Calling with_delimiters first fails because no such method is implemented for Progress<_, Unbounded>.

This is stronger than validating a boolean field at runtime. The type is a witness that with_bound already succeeded.

Checkpoint 10: read the API from its types

Do not run the following invalid order yet. First predict the error:

(0..10)
    .progress()
    .with_delimiters(('<', '>'));

Your turn: Which concrete receiver type is Rust searching for the method on? Which impl block contains the method? State the mismatch in words, then compile it and compare with the diagnostic.

Now consider the successful final type:

Progress<std::ops::Range<i32>, Bounded>

Without reading the implementation, we know:

• the wrapper owns a range iterator;
• its yielded item is i32, inherited through Iterator::Item;
• exact-length configuration has occurred;
• bounded-only methods are available;
• the compiler can reject calls inconsistent with that state.

The type is not merely storage layout. It records facts established by the program’s history.

The API-design lessons

1. Start with the caller

Write the desired call site before selecting traits or structs. The move from progress(iter) to iter.progress() revealed that we wanted a composable iterator adapter and therefore an extension trait.

2. Generalize one accidental restriction at a time

The progression was:

Vec<i32>
   ↓ generic item
Vec<T>
   ↓ generic capability
Iter: Iterator
   ↓ preserve caller's loop
Progress<Iter>: Iterator
   ↓ ergonomic method syntax
extension trait
   ↓ capability-specific configuration
ExactSizeIterator
   ↓ state-specific configuration
Progress<Iter, State>

Each feature answers a concrete limitation. None is present merely to showcase Rust.

3. Preserve relationships at the type boundary

The most important relationships are:

• Progress<Iter>::Item must equal Iter::Item;
• with_bound requires Iter: ExactSizeIterator;
• with_delimiters requires the bounded state;
• consuming transitions prevent continued use of stale states.

These are the same kinds of questions that appear in larger systems. A job runner might preserve Future::Output in JobHandle<T>. A database API might make a committed transaction unavailable for further queries. An HTTP builder might expose send only after a destination has been supplied.

4. Compile-time guarantees have a usability cost

Typestate can produce excellent autocomplete and prevent invalid calls, but it also creates longer types, more generic parameters, more implementation blocks, and sometimes intimidating diagnostics. API quality includes the failure experience, not just whether invalid code fails.

Ask:

• Is this misuse common or dangerous?
• Are there only a few meaningful states?
• Will callers understand the transition?
• Is a runtime Result clearer?
• Does the compiler error point toward the fix?

Use the type system as a design budget, not as a contest.

Continue the workshop

Only now open the completed reconstruction. Compare it with your version one impl block at a time, then work through these changes:

1. Change the progress wrapper so rendering happens after the caller processes an item. What information would the iterator adapter need that it does not currently have?
2. Add a .with_message(String) method that works in both states. On which impl block should it live?
3. Add a .with_delimiters(...) call before .with_bound() and read the full compiler error. Is the diagnostic good enough for a public library?
4. Replace typestate with Option<Bounded>. Which invalid programs now compile?
5. Decide whether delimiters truly require a bounded state. Could a different rendering design remove the invalid state instead of encoding it?

The runnable reconstruction is in examples/type-driven-progress/src/main.rs.

Interview summary

When discussing a Rust API, a strong explanation sounds like this:

I would begin with the intended call site and identify the invariants the implementation and caller must share. I would use ordinary generic bounds for capabilities, associated types when one implementing type determines another type, and typestate only when a small state machine prevents meaningful misuse. I would also evaluate compiler diagnostics and avoid making the public type surface more complicated than the guarantee warrants.

That answer is more valuable than simply naming traits, generics, or typestate. It explains why they belong in the design.

Toy System: An In-Process Job Runner

This first toy system begins from requirements rather than an existing API. We need one in-process runner for async I/O jobs with:

• one FIFO queue holding at most queue_capacity waiting jobs;
• at most concurrency_limit running jobs;
• async backpressure when the queue is full;
• a typed result handle for every submission;
• no cancellation when that handle is dropped; and
• shutdown that rejects new jobs, drains accepted jobs, then returns.

Persistence, retries, priorities, distributed workers, CPU scheduling, and exact byte-based memory limits are deliberately absent.

Design thesis

Use one bounded channel to route heterogeneous work into the runner, and one private typed oneshot channel per submission to route its result back.

That decomposition gives each mechanism one job:

many submitters → bounded MPSC<ErasedJob> → dispatcher → running futures
       caller ← oneshot<Result<T, E>> ←──── wrapper around its one future

The complete runnable crate lives at examples/toy-job-runner.

Start With the Caller

Before choosing channels or traits, write the experience we want:

let runner = JobRunner::builder()
    .queue_capacity(100)
    .concurrency_limit(3)
    .start()?;

let handle = runner.submit(async move {
    fetch_user(user_id).await
}).await?;

let user = handle.await?;
runner.shutdown().await;

The two awaits mean different things:

• submit(...).await waits for queue capacity. Success means the runner accepted ownership of the job.
• handle.await waits for job completion and returns its typed value or business error.

queue_capacity is clearer than max_jobs: it counts waiting jobs, not running jobs, lifetime submissions, or bytes. Arbitrary futures can own String, Vec, and Arc graphs whose total heap use is not described by size_of_val. Strict memory budgeting would require a different contract such as caller-supplied weights.

Version one accepts futures rather than requiring a Job trait. In-memory work needs no serialization or durable identity, and async move already captures precisely the owned input needed to execute later. A named trait can be layered on as an application convention.

Two Channels, Two Directions

The work channel

Tokio’s bounded mpsc channel supports multiple cloned senders and one dispatcher receiver. It packages FIFO storage, wakeups, closure, and async capacity waiting. A VecDeque supplies only storage; recreating the channel would also require a mutex, notifications, capacity accounting, and a closed state.

The result channel

Submission creates oneshot::channel::<Result<T, E>>(). The wrapper moved into the work queue owns Sender<Result<T, E>>; JobHandle<T, E> owns the receiver. send is synchronous: it moves the result into the channel’s single shared slot and optionally wakes the receiver.

If the caller waits ten seconds before awaiting, the result simply remains in that slot. The worker has already finished and released its concurrency slot. If the caller drops the handle, send returns the result as an error and the wrapper drops it. No worker waits for consumption.

This is why a global response queue would be worse: it would require IDs, type erasure for results, demultiplexing, and fairness between unrelated callers.

Erase Work, Preserve Results

One MPSC channel has one item type, but submissions may return unrelated types:

Future<Output = Result<User, HttpError>>
Future<Output = Result<u64, DbError>>

The runner erases the wrapper’s output to ():

type ErasedJob = Pin<Box<dyn Future<Output = ()> + Send + 'static>>;

Inside submit, the typed future and typed sender are captured together:

let (result_tx, result_rx) = oneshot::channel();

let erased = Box::pin(async move {
    let result: Result<T, E> = job.await;
    let _ = result_tx.send(result);
});

work_sender.send(erased).await?;
return JobHandle { receiver: result_rx };

The queue sees only Future<Output = ()>. The type relationship between T, E, and that caller’s handle remains inside the captured sender/receiver pair. We erase what the shared collection must homogenize and preserve what the caller needs.

The bounds explain ownership:

• 'static: queued work cannot borrow a caller stack frame that may disappear;
• Send: Tokio may move the future, result, or error between worker threads;
• no Sync is required merely to transfer exclusive ownership.

One Queue, Concurrent Execution

One queue does not imply one running job. It is like one line feeding several cashiers:

waiting: [C, D, E]       running: {A, B}
                         limit: 2

The dispatcher owns the MPSC receiver and a JoinSet. It receives a new job only when running.len() < concurrency_limit, then spawns it. Completion order is independent of FIFO admission order.

The queue and limit bound different resources:

• queue capacity bounds work accepted but not started;
• concurrency bounds futures currently performing I/O;
• retained, completed JobHandle results are a third memory category and are not globally bounded by either setting.

The implementation uses Sender::reserve().await before constructing the oneshot wrapper. This is backpressure: a full queue suspends the submitting future without blocking an OS thread. After obtaining the permit, submission rechecks shutdown state and either commits the job or returns SubmitError.

CPU-heavy work does not become cooperative merely because it is inside an async block. Such jobs should cross into spawn_blocking or Rayon; otherwise they can monopolize Tokio worker threads despite the runner’s count limit.

Errors and Graceful Shutdown

There are three distinct outcomes:

• BuildError: the runner configuration is invalid;
• SubmitError: no new work can be accepted;
• JobError<E>: either the user’s E, or infrastructure stopped before the result sender completed.

Keeping E generic preserves matching on domain failures. Converting every error into Box<dyn Error> would simplify internals the queue does not actually need to observe, while weakening the caller.

Graceful shutdown sets accepting = false, signals the dispatcher, closes its receiver, drains already buffered jobs under the same concurrency limit, joins running jobs, and notifies every shutdown waiter.

The acceptance boundary is the reserved MPSC permit followed by the shutdown recheck. A job committed before shutdown wins the race and must drain; a job whose permit resolves afterward is rejected. That makes “accepted” an observable ownership transfer rather than a vague timestamp.

Dropping JobHandle does not cancel work. Cancellation would require an additional explicit contract: a cancellation token, abort handle, or policy for what dropping means. Version one avoids making a destructor perform surprising business behavior.

Build and Extend It

Run the complete example:

cd examples/toy-job-runner
cargo test
cargo run

Read src/lib.rs in this order:

1. JobHandle<T, E> and its IntoFuture implementation;
2. JobRunner::submit and the per-job oneshot pair;
3. ErasedJob and why its output is ();
4. dispatch and its JoinSet admission condition;
5. shutdown and the accepting/stopped state transitions;
6. the three invariant-focused tests.

Then implement these extensions one at a time:

1. try_submit returning Full versus ShuttingDown without awaiting.
2. A Job trait adapter implemented in terms of the future API.
3. Caller-supplied job weights with a semaphore-based total budget.
4. Explicit cancellation whose handle drop still remains inert.
5. Multiple runner instances with different queue/concurrency policies.
6. Metrics for queued, running, completed, failed, and abandoned results.

Only after that compare the toy with Apalis. The toy’s futures are arbitrary in-memory work; Apalis payloads cross durable backend and codec boundaries. That one requirement explains much of the production library’s additional generic and lifecycle machinery.

SQLx: Orientation

SQLx is an asynchronous SQL toolkit. Its public API lets applications execute SQL directly while retaining Rust type integration, connection pooling, and optional compile-time query checking.

Design thesis

SQLx uses generic database families to share query machinery without erasing driver-specific types, and treats exclusive connection ownership as the unit that makes asynchronous protocol progress safe.

• A query is lazy until an executor drives it.
• One mutable connection serializes one wire protocol conversation.
• Pools bound concurrency and transfer temporary connection ownership.
• Runtime, driver, and blocking SQLite differences stay behind narrow traits.

Begin at the façade

The root package is named sqlx, but most implementation code lives in workspace crates. The root src/lib.rs mainly re-exports shared APIs and feature-selected drivers.

The same core also connects to sqlx-mysql and sqlx-sqlite. Separate macro crates handle code that runs during compilation:

• sqlx-macros exposes procedural-macro entry points;
• sqlx-macros-core performs query inspection and code generation;
• sqlx-cli manages migrations, databases, and offline query metadata.

Features assemble the product

The root Cargo.toml uses features to select databases, runtimes, TLS implementations, macros, migrations, and external Rust types.

[features]
runtime-tokio = ["_rt-tokio", "sqlx-core/_rt-tokio", "..."]
postgres = ["sqlx-postgres", "sqlx-macros?/postgres"]
macros = ["derive", "sqlx-macros/macros", "sqlx-core/offline", "..."]

This means sqlx is not one fixed binary arrangement. Two applications can compile substantially different portions of the workspace under the same public crate name.

The important shared contracts

sqlx-core/src/lib.rs exposes the concepts that drivers agree upon:

• Database bundles driver-specific associated types;
• Connection describes one physical database connection;
• Executor runs queries;
• Query stores SQL and encoded arguments;
• Pool<DB> manages multiple connections;
• Transaction<DB> ties ordered work to one connection.

The central relationship is:

Query<DB> must be executed by Executor<Database = DB>

A PostgreSQL query and a MySQL pool cannot accidentally meet because their types disagree.

Network drivers and SQLite diverge

PostgreSQL and MySQL speak network protocols over nonblocking sockets. Their connections are asynchronous protocol state machines.

SQLite is an in-process C library with blocking calls. SQLx gives each SQLite connection a worker thread and communicates with it through channels. Both implement the same high-level contracts, but their execution mechanisms are different.

That difference is a useful signal: shared traits describe observable capabilities, not identical implementations.

Scope of this walkthrough

The following chapters trace ordinary query execution, pooling, error handling, and compile-time query checking. Database type catalogs, migrations, TLS details, and every protocol message are outside the main path unless they explain an architectural decision.

Interactive System Map

Choose a backend process, then select any step to see what it owns, why it exists, and where it is implemented.

How to use the map

The arrows show execution order, not crate dependencies. Switching processes changes both the path and the kind of boundary being emphasized:

• Postgres query follows a public call to network responses.
• Pool contention follows admission, reuse, and release.
• SQLite query crosses from async code to a blocking worker thread.
• Cancellation follows invariant restoration rather than success output.
• Query macro follows compilation and rejoins the runtime query path.

The map is an index into the code, not a substitute for it. Later chapters explain the ownership and state transitions behind each path.

The Query Lifecycle

Consider an ordinary operation:

let user = sqlx::query("SELECT id, name FROM users WHERE id = $1")
    .bind(id)
    .fetch_one(&pool)
    .await?;

The compact chain crosses the public façade, generic core, pool, concrete driver, protocol encoder, socket, and row decoder.

Construction does not execute

sqlx_core::query::Query stores four important things:

pub struct Query<'q, DB: Database, A> {
    statement: Either<SqlStr, &'q DB::Statement>,
    arguments: Option<Result<A, BoxDynError>>,
    database: PhantomData<DB>,
    persistent: bool,
}

Calling query() creates this value. Calling bind(id) encodes the argument into DB::Arguments. Neither operation performs database I/O.

The value is marked #[must_use] because dropping it has no effect.

fetch_one delegates to an executor

The query method is mostly a typed handoff:

pub async fn fetch_one<'e, 'c: 'e, E>(self, executor: E)
    -> Result<DB::Row, Error>
where
    E: Executor<'c, Database = DB>,
{
    executor.fetch_one(self).await
}

The query knows its database type. The executor supplies the mechanism.

Executor defines a small set of primitive operations, including fetch_many, fetch_optional, and prepare_with. Default methods construct higher-level behavior:

• fetch filters rows out of fetch_many;
• fetch_all collects a row stream;
• fetch_one turns an absent row into Error::RowNotFound;
• execute collects query-result values.

Executing through a pool

The implementation of Executor for &Pool<DB> acquires a connection and then delegates again:

let mut conn = pool.acquire().await?;
let mut stream = conn.fetch_many(query);

while let Some(value) = stream.try_next().await? {
    yield value;
}

The checked-out connection remains owned by this stream until execution finishes or the stream is dropped.

PostgreSQL becomes concrete

sqlx-postgres/src/connection/executor.rs implements Executor for &mut PgConnection. Its fetch_many implementation extracts the generic query parts and calls PgConnection::run.

For a prepared query, run performs this sequence:

Messages are encoded into a write buffer and flushed together. The response loop then handles messages such as:

• RowDescription, which supplies column metadata;
• DataRow, which becomes PgRow;
• CommandComplete, which becomes PgQueryResult;
• ReadyForQuery, which marks the connection ready for another operation.

A state trace

The exclusive connection borrow is more than a borrowing detail. It protects a state machine whose buffers, prepared statements, transaction status, and pending responses must change in one order.

Generics as Architecture

SQLx uses generics to share behavior without erasing which database is in use. The key parameter is usually DB.

Database is a family of related types

The Database trait does not primarily describe methods. It associates one driver marker with all of the types belonging to that driver:

pub trait Database: 'static + Sized + Send + Debug {
    type Connection: Connection<Database = Self>;
    type TransactionManager: TransactionManager<Database = Self>;
    type Row: Row<Database = Self>;
    type QueryResult: Send + Sync + Default;
    type TypeInfo: TypeInfo;
    type Arguments: Arguments<Database = Self>;
    type Statement: Statement<Database = Self>;
}

For PostgreSQL, those choices include PgConnection, PgRow, PgArguments, and PgStatement. A generic pool can therefore name DB::Connection without carrying separate parameters for every related type.

Bounds express agreement

This bound appears throughout execution code:

E: Executor<'c, Database = DB>

Read it in two parts:

• E: Executor<'c> — E can execute work using a connection valid for 'c;
• Database = DB — its chosen database must match the query’s DB.

The equality constraint connects two otherwise generic values.

Encoding depends on the database

Binding requires:

T: Encode<'t, DB> + Type<DB>

Type<DB> describes the SQL type corresponding to T for this database. Encode<'t, DB> knows how to write the value into this driver’s argument buffer.

The DB parameter matters because databases do not share one representation or type catalog.

Lifetimes connect borrowed pieces

Query<'q, DB, A> may borrow SQL or a prepared statement for 'q. Executor<'c> may borrow a connection for 'c. Query methods introduce an execution lifetime 'e and require both borrows to remain valid long enough:

'q: 'e,
'c: 'e,

The annotations describe which borrowed inputs must outlive the returned future or stream. They do not measure how long execution takes.

Static and dynamic dispatch coexist

Ordinary database selection uses generics and associated types. The compiler knows that Pool<Postgres> contains PostgreSQL connections.

Database-reported errors use dynamic dispatch:

Error::Database(Box<dyn DatabaseError>)

The public error enum can store a PostgreSQL, MySQL, or SQLite error without making the entire enum generic. Callers retain common operations such as code() and kind(), and can downcast when driver-specific detail matters.

SQLx therefore chooses dispatch according to the relationship it needs:

queries and pools       → static database identity
heterogeneous errors    → runtime-selected implementation

Why boxed futures and streams appear

Async blocks and iterator adapters produce concrete types that are difficult or impossible to name in trait signatures. SQLx often returns:

BoxFuture<'e, Result<T, Error>>
BoxStream<'e, Result<DB::Row, Error>>

The box erases the concrete future or stream type while the generic parameters continue to preserve database identity and output relationships.

This is not “generics versus trait objects” as a repository-wide choice. SQLx uses each form at the boundary where it simplifies the contract.

Async and Concurrency

SQLx’s concurrency model has three layers:

nonblocking work within one query
multiple in-flight queries across pooled connections
physical parallelism in runtimes, workers, and database servers

These layers are related but not interchangeable.

One connection is serial

Network drivers implement Executor for an exclusive connection borrow:

impl<'c> Executor<'c> for &'c mut PgConnection { /* ... */ }

Two ordinary query futures cannot simultaneously borrow the same connection mutably. This matches the implementation: one connection owns one ordered protocol conversation.

PostgreSQL can pipeline protocol messages in some situations, but SQLx’s normal query API does not multiplex unrelated operations over one connection.

The pool creates query concurrency

Independent operations use the shared pool:

let (user, account) = tokio::try_join!(
    load_user(&pool),
    load_account(&pool),
)?;

Each operation attempts to acquire its own connection. PoolInner<DB> contains:

• an async semaphore limiting capacity;
• an ArrayQueue of idle connections;
• atomics tracking size, idle count, and closed state;
• shared configuration and lifecycle hooks.

When the pool reaches max_connections, another task awaits a permit. It does not block an operating-system thread.

Socket operations suspend

SQLx’s runtime-neutral Socket trait exposes readiness polling. A read future first tries a nonblocking read. When the socket reports WouldBlock, it asks the runtime to wake the task when the socket becomes readable and returns Poll::Pending.

The task is inactive during that wait. Tokio or another supported runtime can poll other tasks on the same thread.

SQLx wraps runtime-specific operations in sqlx-core/src/rt/mod.rs, including:

• spawning tasks and blocking work;
• sleeping and timeouts;
• yielding;
• runtime-specific socket and semaphore behavior.

Streams control result consumption

fetch() returns an asynchronous row stream. For PostgreSQL, there is no separate task eagerly collecting the full result set. Polling the stream drives the response loop and yields rows incrementally.

This limits application-side buffering, but it also means the stream holds its pool connection while the caller consumes it.

let mut rows = query.fetch(&pool);

while let Some(row) = rows.try_next().await? {
    process(row).await; // the connection remains checked out here
}

Slow per-row work can reduce pool availability. Collecting first releases the connection after reading finishes but uses memory proportional to the result set.

SQLite crosses a thread boundary

SQLite’s C API is blocking. Each SqliteConnection owns a dedicated worker thread created in sqlx-sqlite/src/connection/worker.rs.

The worker executes commands sequentially for one connection. Multiple pooled SQLite connections have multiple worker threads, although SQLite’s own locking rules still constrain concurrent writes.

Bounded command and row channels prevent unlimited buffering. A slow receiver eventually makes the producer wait.

Where parallelism may occur

• A multithreaded runtime may poll independent query tasks simultaneously.
• SQLite connections may execute on separate worker threads.
• PostgreSQL or MySQL may execute different connections in separate server processes or threads.
• A database query plan may itself use parallel workers.

None of these possibilities changes the one-mutable-owner rule for an SQLx connection.

Errors, Cancellation, and Cleanup

SQLx centralizes public failures in sqlx_core::error::Error, then separately decides whether the connection and pool accounting remain valid.

One public error surface

Important categories include:

• Database, containing an error reported by the server;
• Io, for transport failures;
• Protocol, for malformed or unexpected protocol state;
• Encode, Decode, and ColumnDecode;
• PoolTimedOut and PoolClosed;
• WorkerCrashed for a lost SQLite worker.

The enum is #[non_exhaustive], allowing the library to add variants without requiring every caller to update an exhaustive match.

Native errors retain their detail

Each driver implements DatabaseError for its native error type. PostgreSQL’s PgDatabaseError, for example, preserves SQLSTATE, detail, hint, table, column, and constraint fields.

The generic wrapper stores it as:

Error::Database(Box<dyn DatabaseError>)

Common constraint categories are normalized into ErrorKind, while callers can downcast when they need a PostgreSQL-specific field.

Drivers detect error messages centrally

PostgreSQL’s stream receive loop recognizes ErrorResponse, decodes a PgDatabaseError, and returns it as Error::Database. MySQL performs the same normalization when a packet begins with its error marker. SQLite converts C result codes into SqliteError.

Central detection keeps query loops focused on valid response messages.

Stream errors can arrive late

Constructing a stream does not prove that a query can execute. Errors can occur while acquiring, preparing, receiving the first row, decoding a later row, or processing the final completion message.

PostgreSQL’s fetch_optional continues consuming responses after finding its first row because deferred database constraints may fail near the end of the operation.

Cancellation is dropping a future

An async operation can be cancelled at an .await by dropping its future or stream. At that moment SQLx might have:

• acquired a semaphore permit;
• incremented the pool’s connection count;
• removed an idle connection from the queue;
• sent a query whose response is incomplete;
• begun a transaction.

Returning Err is not enough to restore those states.

RAII protects pool accounting

DecrementSizeGuard<DB> owns responsibility for a reserved pool slot. Unless explicitly cancelled, its Drop implementation decrements the size and releases the semaphore permit.

The guard is installed before connection work awaits. Every early return and cancellation therefore passes through the same compensation path.

reserve shared state
        │
        ▼
install drop guard
        │
        ▼
perform fallible async work
        │
        ├── success → disarm or transfer guard
        └── failure/cancel → Drop restores state

Connections must be synchronized before reuse

PostgreSQL tracks how many ReadyForQuery messages are pending. Before starting another operation, wait_until_ready flushes queued writes and drains responses until the protocol reaches its completion boundary.

When a pooled connection is returned, SQLx tests whether it can be restored to a usable state. A connection that fails the check is closed rather than placed back in the idle queue.

Drop cannot await

Dropping PoolConnection may spawn an async cleanup task because returning a connection can require I/O and Rust’s Drop::drop is synchronous.

Transactions use a related approach. Dropping an open transaction calls the driver’s start_rollback, which queues or signals rollback work so a later async operation completes it.

The architectural lesson is broader than SQLx:

Error propagation reports failure. Lifecycle code restores invariants and decides whether resources are reusable.

Compile-Time Query Checking

query!() looks like a runtime query API, but part of its work happens while the application is compiling.

Two timelines

compile time                         runtime
------------                         -------
read SQL                             construct Query
describe SQL or load cache           acquire connection
infer parameter/output types         execute protocol
generate Rust expression             decode rows

Keeping these timelines separate prevents a common misunderstanding: the macro does not replace the runtime executor or driver.

The procedural-macro entry point

sqlx-macros/src/lib.rs exposes expand_query. It parses the macro input and passes it to sqlx-macros-core with the enabled database drivers.

sqlx-macros-core/src/query/mod.rs then chooses a data source:

• a live database selected through DATABASE_URL; or
• cached query metadata from an offline .sqlx directory.

The URL scheme or cached database name selects the matching macro driver.

Description becomes generated types

For a live database, SQLx asks the driver to describe the SQL. The result contains parameter and column information. Macro expansion then:

1. verifies the number of arguments;
2. checks or generates parameter type expressions;
3. maps output columns to Rust types;
4. generates a record type when necessary;
5. emits an ordinary typed query expression.

Invalid SQL or mismatched types can therefore become compilation errors.

Runtime execution rejoins the normal path

The generated expression ultimately constructs the same Query family used by the non-macro functions. At runtime it still travels through:

Query → Executor → Pool/Connection → Driver → Database

Network failure, pool exhaustion, permission changes, constraint violations, and returned data remain runtime concerns.

Why separate crates exist

Procedural macros execute in the compiler’s process and have different build and dependency constraints from the runtime library. Keeping macro entry points, expansion logic, core traits, and drivers in separate crates lets Cargo assemble only the required pieces and avoids making the façade itself a procedural-macro crate.

Build a Smaller SQLx

Reimplementing all of SQLx is not a useful first target. A smaller project can preserve its central architecture without supporting three databases, several runtimes, TLS choices, migrations, macros, and years of compatibility behavior.

Rebuild the architectural center, not a toy imitation of the public syntax.

This chapter is the overview for a hands-on, multi-stage reconstruction. Each stage will become its own chapter and leave behind runnable code.

What are we preserving?

The smaller implementation must genuinely demonstrate:

• exclusive mutable connection access;
• lazy query execution;
• incremental row streaming;
• pool-limited concurrency;
• cancellation-safe resource accounting; and
• a worker-thread bridge for blocking operations.

It does not initially need TLS, real PostgreSQL authentication, dozens of SQL types, migrations, procedural macros, or optimized lock-free queues.

Stage 1: one concrete connection

Begin with one database and no generic traits:

struct Connection {
    socket: BufferedSocket,
    state: ProtocolState,
}

impl Connection {
    async fn execute(&mut self, query: Query) -> Result<QueryResult, Error>;
    fn fetch(&mut self, query: Query)
        -> impl Stream<Item = Result<Row, Error>> + '_;
}

Keep &mut self. It makes protocol serialization explicit and prevents a multiplexing problem before the decoder is correct.

Stage 2: framing and cancellation-safe reads

Build an incremental decoder that distinguishes an incomplete buffer from a complete message:

enum Decode<T> {
    Need(usize),
    Complete(T),
}

Do not remove bytes from the shared buffer until a full message has arrived. This lets a cancelled read resume without losing the message boundary.

Stage 3: stream rows

Return rows incrementally. Record the protocol completion marker separately from row messages so the connection cannot be declared reusable too early.

Test these cases:

• zero rows;
• many rows with a slow consumer;
• an error after some successful rows;
• dropping the stream early;
• the server closing the socket mid-message.

Stage 4: add a simple pool

Use understandable synchronization first:

struct Pool {
    idle: Mutex<VecDeque<Connection>>,
    permits: Semaphore,
}

The semaphore limits checked-out connections. The queue stores reusable ones. Only replace these structures with lock-free or specialized versions after measurement establishes a need.

Install an RAII guard before every await that follows shared-accounting changes.

Stage 5: define recovery

Decide what happens when a query future is dropped:

• drain the remaining response to a protocol boundary;
• send a database-specific cancellation request; or
• mark the connection broken and close it.

Closing is often the simplest correct first policy.

Stage 6: bridge a blocking implementation

To model SQLite, place blocking connection state on a worker thread. Use a bounded command channel and a bounded result channel. Define shutdown behavior before adding more commands.

Stage 7: extract generic contracts

After a second implementation exists, compare concrete types and extract only the behavior they genuinely share. Introduce a Database family of associated types and connect queries to executors with an equality constraint.

Stage 8: add compile-time tooling last

Compile-time SQL inspection is valuable but independent of basic execution. Add it after the runtime path has stable query, argument, row, and type-info contracts.

What the smaller version omits

At this point it will probably lack:

• robust TLS and authentication choices;
• statement-cache eviction;
• database-specific type coverage;
• optimized buffers and queues;
• hooks, tracing, migrations, and offline metadata;
• carefully tested behavior under every cancellation point.

Those omissions are useful. They expose which parts constitute the design and which parts harden it for production.

ripgrep: Orientation

Ripgrep is a recursive text-search application. A useful first model is not “grep, but in Rust.” It is a pipeline that discovers eligible files, searches several of them in parallel, and commits each file’s completed output without tearing it together with output from another worker.

The architectural center

The traversal is not merely a producer feeding a separate search pool. Each walk worker owns a callback, and that callback owns a cloned SearchWorker. The same operating-system thread discovers a searchable file and searches it.

Design thesis

Ripgrep parallelizes at the file boundary because files are independently searchable work units, while output becomes visible only through a short, serialized commit.

• Worker-local searchers keep hot mutable state out of locks.
• Purpose-built work stealing balances irregular directory trees.
• Synchronous threads match filesystem and CPU-heavy work better than async I/O.
• Per-file buffering trades global ordering for untorn, readable output.

The important crates

• The root ripgrep package and crates/core assemble CLI configuration and choose sequential or parallel execution.
• ignore walks directory trees, applies ignore and type rules, and owns the work-stealing scheduler.
• grep-searcher chooses an mmap, whole-file, or incremental line-buffer strategy and drives a Sink.
• grep-matcher defines the matching contract independently of a regex engine.
• grep-printer turns search events into standard, summary, or JSON output.

This is not Tokio or Rayon

The normal search path has no async runtime. Filesystem reads and searches are synchronous operations performed by scoped OS threads. Ripgrep also does not use Rayon for traversal: ignore builds its own scheduler from crossbeam_deque::Worker and Stealer.

That choice is the reason this case study belongs here. We are studying how an application chooses concurrency boundaries around real work—not how a general runtime implements them.

Scope

We will follow rg pattern directory through parallel traversal, ignore filtering, file searching, output serialization, errors, and early shutdown. Regex-engine construction, every flag, PCRE2, archive decompression, and the experimental index are secondary paths.

Interactive System Map

Choose a process, then select a step to inspect its ownership and design boundary.

The important surprise is where the boundary sits: traversal and searching are fused inside each worker, while output is buffered and committed separately.

Why Is It Designed This Way?

The most reusable lessons appear when we compare ripgrep’s choices with plausible alternatives.

Why not one producer and one shared work channel?

A central queue is simpler to sketch, but it naturally produces breadth-first behavior and concentrates contention. Ripgrep wants local depth-first traversal because wide trees can retain enormous numbers of paths and inherited ignore matchers. LIFO local deques preserve locality; stealing repairs imbalance.

Why not one shared SearchWorker behind a mutex?

Searching mutates scratch buffers and printer state. A mutex around one worker would serialize the expensive operation and erase useful parallelism. Cloning a worker per thread duplicates bounded scratch state while removing the hottest lock.

Why not write each matching line directly to stdout?

Line-sized locking permits output from multiple files to interleave and pays a synchronization cost for every line. Holding the lock for the entire search of a file prevents tearing but also holds it during file reads and matching. Private file buffers move the lock to the short commit phase.

Why not preserve sorted output with a result collector?

That is possible, but the collector must retain results for later files while waiting for earlier files to finish. A slow early file can make memory grow with all later output. Ripgrep chooses bounded, immediate output in parallel mode and stable order in sequential mode.

Why custom work stealing instead of Rayon?

Rayon is well suited to recursively divisible CPU work. Ripgrep’s traversal has domain-specific requirements: inherited ignore state, depth-first local order, visitor construction and cleanup per thread, cooperative Skip/Quit, and termination when dynamically generated directory work is exhausted.

The lesson is not that application-specific schedulers are generally better. It is that a concurrency abstraction should preserve the application’s unit of work and shutdown semantics. Here those semantics live inside ignore, which is itself reusable by applications other than ripgrep.

Why no async runtime?

An async rewrite would add future state, executor integration, and async-aware filesystem decisions without changing the basic need for CPU parallelism. The current worker count already bounds simultaneous blocking reads and searches. For this workload, threads make blocking and computation part of the same simple execution path.

One Parallel Search

Consider:

$ rg "unsafe" crates/

The production path is short enough to trace completely, but each step changes who owns the operation.

1. Select parallel mode

main parses low-level flags into HiArgs. run selects search_parallel unless the effective thread count is one. Sorting or searching one explicit file forces that count to one before this decision.

Source: crates/core/main.rs:78

2. Construct shared templates

search_parallel constructs:

• a HaystackBuilder for application-level file eligibility;
• a BufferWriter for atomic-ish file-sized output commits;
• atomic matched and searched flags;
• optional statistics behind a mutex; and
• one configured SearchWorker that will be cloned per traversal worker.

The initial SearchWorker owns a matcher, reusable Searcher scratch buffers, and a printer whose destination is a private output buffer.

Source: crates/core/main.rs:166

3. Build one visitor per OS thread

WalkParallel::run invokes its factory once per traversal thread. Each factory call clones SearchWorker, producing thread-local matcher, searcher, printer, and scratch state.

args.walk_builder()?.build_parallel().run(|| {
    let mut searcher = searcher.clone();
    Box::new(move |result| { /* this thread's callback */ })
});

There is no lock around Searcher. Mutable reusable buffers stay local to one worker, and synchronization is reserved for genuinely shared results.

4. Discover and filter entries

Each traversal worker processes directory Work. It reads a directory, extends the inherited ignore matcher, rejects ignored paths, maximum-size violations, and custom-filter failures, then pushes accepted children onto its local deque.

Source: crates/ignore/src/walk.rs:1769, generate_work

5. Convert an entry into a haystack

The application callback receives Result<DirEntry, ignore::Error>. HaystackBuilder reports traversal errors and rejects non-files, while still preserving special treatment for explicit paths and stdin.

This is a second filtering boundary: ignore answers “should traversal expose this entry?” while HaystackBuilder answers “should ripgrep search it?”

6. Search one file synchronously

SearchWorker::search configures binary detection, then chooses stdin, preprocessor, decompressor, or ordinary path search. Ordinary path search calls Searcher::search_path, which can use a memory map, a whole-file buffer for multiline mode, or an incremental rolling line buffer.

Source: SearchWorker::search, Searcher::search_path

The worker blocks while reading this file. Parallelism comes from other OS threads searching other files, not from suspending this operation.

7. Commit output

The printer writes all results for the file into its worker-local buffer. Only after the search completes does BufferWriter::print serialize that buffer to stdout. Files may appear in nondeterministic order, but their lines do not become arbitrarily interleaved with another worker’s file.

Source: crates/core/main.rs:189

State at the end

The worker clears and reuses its buffer for the next file. Shared atomics retain whether anything was searched or matched. When all local deques are empty and all workers have become inactive, a quit message propagates and scoped threads are joined before search_parallel returns.

Concurrency as a Product Decision

Ripgrep’s normal parallel search uses synchronous I/O plus several OS threads. That is not an implementation accident; it matches the unit of work.

The unit of parallelism is a file

Within one file, the searcher reads and matches sequentially. Across files, workers run in parallel. This gives each worker a substantial unit of work and keeps line ordering within a file straightforward.

worker 0: discover a.rs → search a.rs → commit a.rs output
worker 1: discover b.rs → search b.rs → commit b.rs output
worker 2: descend src/ → discover c.rs → search c.rs

Splitting individual lines among workers would introduce ordering, cross-line-match, context-line, and buffering problems for much smaller tasks.

Why fuse traversal and searching?

WalkParallel::run creates one callback per traversal worker. The callback searches the visited file immediately. There is no central path channel and no second search pool.

This removes a queue and avoids having separate thread counts for traversal and search. It also means a thread blocked on a slow file is temporarily unavailable for traversal, which work stealing mitigates by letting other workers take directories from its deque.

Local deques, global stealing

Every worker has a LIFO Crossbeam deque. Local LIFO processing encourages depth-first traversal, reducing the number of live paths and inherited ignore matchers. An idle worker attempts steal_batch_and_pop from its peers.

Source: Stack::new_for_each_thread, Stack::steal

This is dynamic load balancing. A directory tree is rarely balanced enough for static partitions: one root may contain ten files while another contains ten thousand.

Shared state is deliberately tiny

The parallel search shares only what must cross worker boundaries:

• AtomicBool for “searched anything?” and “matched anything?”;
• Mutex<Stats> only when aggregate statistics were requested;
• the output writer’s internal serialization;
• AtomicBool for immediate shutdown; and
• AtomicUsize for termination detection.

Everything expensive and frequently mutated—the matcher, line buffers, printer, and file output buffer—is cloned or constructed per worker.

Termination is a distributed state transition

An empty local deque does not mean the search is finished; another worker may discover more directories. A worker first marks itself inactive. Only the worker that observes the active count reach zero knows every deque was empty at the same coordination point. It injects a quit message, which other workers repeat as they exit.

Source: Worker::get_work

Concurrency versus async

Async would help if ripgrep needed to maintain huge numbers of mostly-idle operations. Its dominant operation is instead a bounded number of active file reads and CPU searches. A small OS-thread pool is a direct fit and permits the entire matching stack to remain synchronous.

Output, Errors, and Early Exit

Parallel work is only half of the design. Ripgrep must also decide what becomes observable when work completes out of order, fails, or is no longer needed.

Atomic output units

Each worker prints into a private termcolor::Buffer. After one file finishes, BufferWriter::print writes the completed buffer to stdout. The synchronization boundary is therefore one file’s rendered result, not one match and not the whole search.

This is a useful general pattern:

compute concurrently → stage result privately → commit as one visible unit

It preserves throughput without holding a global stdout lock throughout file I/O and matching.

Determinism costs parallelism

Parallel workers finish in scheduler- and filesystem-dependent order. Ripgrep therefore cannot promise stable file ordering in its normal parallel mode. When sorting is requested, HiArgs forces the effective thread count to one; the sequential path can then sort before searching.

Source: HiArgs thread selection, walk_builder

The important tradeoff is not “sorting is slow.” Global order requires knowing which result comes next, so the existing immediate-commit architecture cannot retain both its bounded buffering and arbitrary parallel completion.

Most file errors are partial failures

An unreadable directory or failed file search is reported, but other workers continue. A shared error indicator later influences the process exit code. This lets a search produce useful matches even when one subtree is inaccessible.

Fatal initialization errors still return through anyhow::Result; per-entry operational errors are logged and converted into WalkState::Continue.

Broken pipe is successful termination

If a downstream consumer closes the pipe—rg pattern | head, for example—an output write returns BrokenPipe. The parallel visitor returns WalkState::Quit, and top-level error handling treats a propagated broken pipe as exit code zero.

Source: search_parallel output handling, main error mapping

Quiet mode coordinates early exit

When one worker finds a match, it stores true in the shared matched atomic. If the selected mode permits stopping after a match, that worker returns WalkState::Quit. Worker::run sets the global quit flag, so peers stop taking ordinary work at their next coordination point.

This is cooperative cancellation. A worker already inside a synchronous file read is not forcibly interrupted; shutdown happens at explicit boundaries.

Panic cleanup

Traversal uses scoped threads and joins every handle. If a Worker is dropped while its thread is panicking, its Drop implementation sets the same global quit flag, encouraging peers to stop rather than continue unrelated work.

Source: Worker::drop

Build a Smaller ripgrep

The reconstruction should preserve the parallel search architecture, not copy ripgrep’s flags or output syntax.

Rebuild filtered traversal, worker-local search, work stealing, atomic output units, and coordinated shutdown.

1. Start with one concrete search

Accept a literal byte pattern and one root directory. Recursively visit files, read each through BufRead, and print matching lines. Keep this stage entirely single-threaded so filtering and errors are testable.

2. Separate traversal from eligibility

Introduce two decisions:

fn should_descend(entry: &DirEntry, rules: &Rules) -> bool;
fn should_search(entry: DirEntry) -> Option<Haystack>;

The first controls the directory tree. The second protects the searcher from directories, unsupported entries, and other application-level exclusions.

3. Make one reusable worker

Give SearchWorker its own read buffer and output Vec<u8>. Searching a file clears and reuses both. Return metadata separately from rendered bytes.

struct SearchWorker {
    needle: Vec<u8>,
    read_buf: Vec<u8>,
    output: Vec<u8>,
}

4. Add a fixed worker count

Begin with a bounded channel of file paths and N worker threads. This is not yet ripgrep’s fused traversal design, but it makes the concurrency boundary visible. Measure queue growth on a very wide directory.

5. Replace the central queue with local deques

Give every worker a LIFO deque of directory work and a list of peer stealers. Workers descend locally and steal only when idle. Carry the current ignore rules inside each directory work item.

Test an intentionally unbalanced tree: one root with a few files and one root with thousands. Confirm that workers steal from the busy root.

6. Stage then commit output

Search into the worker’s private buffer. Acquire the stdout lock only to write the completed buffer. Add a stress test that inserts yields between produced lines and confirms files never tear together.

7. Add distributed termination

An empty deque is not sufficient. Track active workers atomically and declare completion only when every worker is inactive while no deque contains work. Add a shared quit flag for broken pipes and “stop after first match.”

8. Extract generic contracts last

Once literal matching works, introduce a small matcher contract and an output sink. Keep traversal concrete unless a second traversal implementation creates a real abstraction pressure.

trait Matcher {
    type Error;
    fn find(&self, haystack: &[u8])
        -> Result<Option<Range<usize>>, Self::Error>;
}

Compare with production

The smaller version should now map onto these centers:

• WalkParallel and Stack for dynamic directory work;
• HaystackBuilder for application-level eligibility;
• SearchWorker for thread-local reusable state;
• Matcher, Searcher, and Sink for search composition;
• BufferWriter for visible output units; and
• atomics plus WalkState for completion and early shutdown.

Production ripgrep still supplies layered ignore precedence, Unicode-aware regex engines, mmap heuristics, encoding, multiline and context handling, binary detection, compressed files, several printers, platform behavior, and years of performance testing. Those are hardening around the same center—not the center itself.

Aikido: Orientation

Aikido is a policy operating system spanning historical research, frozen policy artifacts, simulation, deployment control, and live account execution. The repository is too large to learn by treating every crate as equally important. Its central production boundary is:

frozen policy input → desired exposure → broker action → observed account truth

Design thesis

Aikido separates concurrent observation and policy computation from mutation by assigning each live account to one executor task, then continuously reconciling desired exposure against broker truth.

• Bounded channels make pressure and ownership transfer explicit.
• One task owns each mutable broker client and account state machine.
• Generic broker traits keep policy independent of protocol adapters.
• Reconciliation, not an order-return value, closes the control loop.

Two connected systems

The repository has a slow decision loop and a fast runtime loop.

Research may use warehouse data to evaluate a candidate. Downstream runtime behavior should instead consume frozen artifacts and control-plane state. That separation prevents a live process from silently changing because a mutable research query changed underneath it.

Runtime workspace center

The runtime path crosses several crates:

• aikido-engine owns bars, strategies, actions, positions, and configuration;
• policy-kernel owns shared policy/accounting semantics;
• fleet-policy-engine decides routing and allocation across deployments;
• aikido-runtime wires Tokio tasks, channels, brokers, journals, and live reconciliation;
• projectx-rs and rithmic-rs provide broker/protocol clients.

The runtime task graph

One process hosts many Tokio tasks:

BarFetcher(s) → SharedBarStore → SignalEngine(s) → SignalRouter
                                                    ↓
                                           AccountExecutor(s)
                                                    ↓
                                                  broker
                                                    │
                        ExecutorStateUpdate ─────────┘

The key ownership rule is one AccountExecutor per deployment/client. That task owns one mutable broker client and dynamically keyed per-contract runtime state. Other tasks communicate with it through commands rather than borrowing or locking the broker.

Scope

We will follow one fresh bar through strategy evaluation, routing, guardrails, execution, and feedback. We will also study desired-state convergence and broker reconciliation because success-path order placement alone is not the system’s architectural center.

Legacy fleet simulation, every research script, every broker API method, TUI rendering, and deployment operations remain outside the main trace.

Interactive Runtime Map

Choose a runtime process, then inspect each ownership and policy boundary. Links target the committed revision bbb96924185fb6417fa1b8d165328808939cf7f0.

The arrows describe logical handoff order. Several steps run in independent tasks and are joined by bounded channels rather than one call stack.

Why Is It Designed This Way?

This chapter is the center of the case study. Each production mechanism exists because a simpler-looking alternative breaks a specific invariant.

Why one account executor instead of a task per order?

A task per order maximizes apparent concurrency but creates races between entries, exits, protection updates, manual commands, and reconciliation. One executor serializes mutation of the broker client and account state while the rest of the system remains concurrent.

The preserved invariant is: one authority decides the next account mutation from one ordered view of account state.

Why commands instead of Arc<Mutex<BrokerClient>>?

A shared mutex would prevent simultaneous method calls, but it would not define operation ordering, priority, deduplication, shutdown, or which state must be updated with each call. A command enum makes the ownership transfer and protocol explicit. The receiver can prioritize runtime-control commands ahead of signal traffic.

Why bounded channels instead of unbounded queues?

An unbounded queue converts a slow consumer into memory growth and increasing decision age. A bounded queue suspends producers, propagating overload.

The preserved invariant is not merely bounded memory: the runtime must not act on arbitrarily stale exposure-increasing intent. That is why bounded queues are paired with a stale-bar gate at the executor.

Why a singleton router instead of engines sending directly?

Direct engine-to-executor sends look simpler until routing depends on subscriptions, account availability, duplicate ownership, copy topology, guardrails, and policy allocation. A singleton router gives those decisions one ordered state cache and one place to enforce cross-deployment rules.

Why feed executor state back to signal engines?

Strategies need position context for exits and adjustments, but engines do not own real accounts. Feedback gives them synthetic views without granting broker authority. The executor remains the source of account mutation; engines remain proposal generators.

Why reconcile if local state was just updated?

Networks time out, brokers reject or partially fill, processes restart, and operators act outside the process. Local state records intent and recent knowledge. Broker state records the external fact the runtime must eventually match.

The preserved invariant is: observed account exposure outranks assumptions derived from submitted commands.

Why target exposure instead of imperative “buy one” policy output?

Imperative commands are difficult to retry: repeating “buy one” may double the position. A desired signed quantity is idempotent with respect to observation. After reconciliation, the runtime recomputes the remaining delta.

This turns recovery into repeated comparison:

desired − observed = next safe adjustment

Why generic broker functions instead of trait objects everywhere?

The executor owns one concrete adapter for a long time. Static dispatch retains exclusive mutable ownership, avoids a shared boxed client, and makes test brokers cheap to substitute. Trait objects would be justified where broker types must be selected heterogeneously at runtime inside one collection; they are not required at every helper boundary.

Why standard mutexes inside an async application?

Some critical sections are tiny, non-async memory operations. A standard mutex is appropriate when the guard is never held across .await; an async mutex would add scheduling overhead without benefit. Broker operations are different: they are not placed behind a mutex at all, but owned by one async task.

Why frozen policy packs instead of live warehouse reads?

A live query is mutable input: data corrections, publication changes, and query edits can change decisions without a deployment event. A frozen pack gives simulation, approval, staging, and runtime a versioned object to discuss and replay.

The preserved invariant is: deployment behavior is attributable to an exact reviewed input, not whatever the research warehouse returns now.

Why not fail the whole process when one task fails?

Some surrounding tasks are diagnostic or recoverable, while active account tasks may still need to protect or flatten positions. The supervisor records failure without immediately aborting every peer. This increases availability, but it also demands health reporting and explicit rules for which failures should block new exposure.

One Bar to One Broker Action

The representative operation is a fresh one-minute bar that produces an entry signal and eventually changes broker exposure.

1. The coordinator establishes the topology

launch_fleet creates a bounded SignalBatch channel, one bounded state-update channel per engine, and prebuilt command channels for executors. It then spawns one signal engine per instrument and one singleton router.

Source: fleet_coordinator.rs:82

The coordinator does not create broker clients. Executors are already running because broker construction, credentials, and connection policy belong to the binary’s setup phase.

2. A signal engine observes a new bar

run_signal_engine polls SharedBarStore, ignores already processed bars, updates session state and indicators, then asks StrategyRunner for actions. The strategy runner is locked only around evaluation and released before the next asynchronous channel send.

Source: signal_engine.rs:282

Even an empty action vector becomes a SignalBatch; executors still need bar ticks for maintenance such as end-of-day flatten checks.

3. The bounded send applies pressure

The engine awaits signal_tx.send(batch). Capacity is 64. If the singleton router falls behind, producers eventually suspend instead of building an unbounded queue of stale decisions. If every receiver is gone, the engine exits successfully.

4. The router chooses a destination

run_signal_router uses tokio::select! to alternate between new signal batches and executor state feedback. route_batch applies isolated, broadcast, or shared-dispatch topology, subscriptions, policy allocation, copy routing, duplicate claims, and entry guardrails.

Source: signal_router.rs:216, route_batch

The result is not a direct broker call. It is an ExecutorCommand::ProcessSignals sent to the chosen deployment’s bounded command channel.

5. One account executor serializes mutation

The executor receives the command, resolves or creates a ContractRuntimeState in its HashMap, rejects off-scope work, drops stale exposure-increasing actions, and deduplicates entry intent IDs.

Source: account_executor.rs:2322

Only this task mutates its broker client and per-contract position state. That single-writer rule is more important than whether Tokio happens to poll the task on one thread or another.

6. Generic action handling reaches a concrete broker

handle_bar_actions accepts Option<&mut (impl BrokerClient + Send)>. It runs capacity and risk checks, calculates allowed size, records attempts, and calls generic order helpers. At compile time, impl BrokerClient becomes the actual ProjectX, Rithmic, paper, or test client type.

Source: actions.rs:833, entry path

7. Observed state closes the loop

After an entry attempt, the executor accelerates reconciliation. Broker positions—not the mere fact that an order request returned—determine the account’s observed exposure. Position changes become ExecutorStateUpdate messages, which the router fans back to all engines for synthetic position tracking.

The complete operation is therefore a loop:

bar → proposal → routed command → guarded action → broker observation → feedback

Async and Concurrency

Aikido uses Tokio as an application runtime. It does not implement a scheduler; it chooses task and channel boundaries that encode domain ownership.

Task topology

• A BarFetcher task obtains bars for each leader instrument.
• A SignalEngine task evaluates strategies for each instrument.
• One SignalRouter task owns routing policy and its state cache.
• One AccountExecutor task owns each deployment’s broker client.
• Monitoring, command, WebSocket, persistence, and alert tasks surround the core path.

Tasks are concurrent. They may run in parallel on Tokio’s worker threads, but the design does not depend on a particular task remaining on a particular OS thread.

Bounded channels are asynchronous queues

The core channels use tokio::sync::mpsc::channel(64). A send that finds a full queue waits asynchronously. This provides two properties:

• memory cannot grow without bound merely because a consumer is slow;
• overload propagates upstream toward the producer.

This is backpressure, although it is not enough by itself. A queue full of old trading decisions may be bounded yet unsafe, so the executor independently rejects stale exposure-increasing signals.

select! merges independent event sources

The router waits for either engine batches or executor feedback:

tokio::select! {
    batch = signal_rx.recv() => { /* route forward */ }
    update = state_rx.recv() => { /* fan state backward */ }
}

This is concurrent waiting inside one task, not parallel execution. Keeping both branches in one router task also gives RouterStateCache a single mutable owner, avoiding a mutex around routing decisions.

The account executor is an actor-like single writer

An executor owns client: C, HashMap<String, ContractRuntimeState>, command receiver, reconcile schedule, and shutdown lifecycle. Callers send enum commands. They cannot concurrently mutate the broker or position maps.

The executor deliberately prioritizes queued runtime-control commands ahead of ordinary signal commands when draining its channel. A plain FIFO queue is not always the correct service policy for safety operations.

Source: account_executor.rs:2290

Avoiding locks across .await

Shared stores and UI snapshots use standard mutexes, but critical sections are kept short: clone or calculate the needed value, drop the guard, then await. The signal engine, for example, finishes strategy evaluation inside a block before awaiting its channel send.

Holding std::sync::MutexGuard across .await would both serialize unrelated tasks and can make a spawned future fail its Send requirement.

Supervision is not fail-fast

The binary collects task handles in a JoinSet and records task failures while other live tasks continue. It reports failure only after the task set drains. That is an explicit availability policy, not Tokio’s default behavior.

Source: runtime.rs:3457

The Standard Library as Architecture

Most of Aikido’s important runtime structure is expressed with ordinary Rust types rather than framework machinery.

HashMap: dynamic identity and owned state

HashMap<String, ContractRuntimeState> means a deployment executor can discover contracts dynamically while retaining exactly one owned state record per normalized contract ID. HashMap::entry performs initialize-if-absent without duplicating lookup logic.

Source: ensure_contract_runtime

The same collection expresses different invariants elsewhere:

• synthetic positions keyed by strategy, deployment, instrument, and lane;
• router accounts and open positions keyed by normalized identity;
• recently seen intent IDs keyed to time buckets for deduplication.

The key type is part of the architecture: it states what the system considers the same thing.

Arc: shared lifetime, not shared mutation

Arc<T> lets independently spawned tasks own the same long-lived service or shutdown flag. It does not make T mutable or thread-safe by itself.

Examples include Arc<AtomicBool> for shutdown and Arc<FleetGuardrails> for a shared service. The inner type still determines the mutation protocol.

Mutex and RwLock: snapshots with short critical sections

• Arc<Mutex<StrategyRunner>> permits hot-reloadable strategy state while one engine evaluates it exclusively.
• Arc<RwLock<RoutingSnapshot>> permits frequent read snapshots with rarer publication updates.
• UI state uses Arc<Mutex<DeploymentState>> because it is shared display and operator state, not the account’s primary execution owner.

Poisoning is sometimes treated as fatal with expect or unwrap, and sometimes converted into anyhow::Error. That choice reveals whether corrupted shared state is considered recoverable at that boundary.

Atomics: independent facts

AtomicBool represents the monotonic request to shut down. Engine statistics use atomic counters because each metric is independently observable and does not require a consistent multi-field transaction.

An atomic would be a poor replacement for positions or a routing plan: those values have relationships that must change together.

Enums: closed command protocols

ExecutorCommand models every request the account owner understands, including signals, reconcile, flatten, converge, manual operations and shutdown. ExecutorStateUpdate models the smaller feedback protocol.

The compiler forces each exhaustive match to confront new variants. This is stronger than passing strings or loosely shaped JSON between in-process tasks.

Option and Result: absence versus failure

• Option<&mut BrokerClient> means dry/deferred execution may legitimately have no active broker reference.
• Result<T> means an attempted operation failed.
• Option<oneshot::Sender<Result<(), String>>> means a command may optionally request a direct completion response.

Keeping these cases distinct prevents “not applicable,” “not yet connected,” and “failed” from collapsing into one ambiguous null or boolean.

Generics and Trait Boundaries

The most important generic boundary is BrokerClient.

One capability contract, several mechanisms

pub trait BrokerClient: Send {
    async fn place_market_order(&mut self, ...)
        -> Result<Option<i64>>;
    async fn search_positions(&mut self, ...)
        -> Result<Vec<BrokerPosition>>;
    async fn ensure_connected(&mut self) -> Result<()> { Ok(()) }
}

Source: broker_trait.rs:128

The trait describes what execution needs. It does not require ProjectX and Rithmic to share authentication, transport, bracket behavior, reconnection, or historical-data machinery.

Static dispatch in the hot stateful path

Functions use parameters such as:

client: &mut (impl BrokerClient + Send)

This is generic static dispatch. The compiler creates code for the concrete broker type. More importantly, the caller retains a concrete owned client with its full state and the callee receives one exclusive mutable borrow.

The executor does not need Arc<Mutex<dyn BrokerClient>>, heap allocation per call, or runtime method lookup.

Why Send appears

The executor future is spawned onto a multithreaded Tokio runtime. It may move between worker threads whenever it is suspended. State retained across an .await, including the concrete broker client, must therefore permit transfer between threads.

Send does not mean two tasks may use the broker concurrently. &mut still enforces exclusive access.

Default methods encode optional capabilities

place_entry_order defaults to a normal market order and reports that native brackets were not used. Always-connected brokers can inherit a no-op ensure_connected. History methods can default to empty results.

This keeps the common contract usable while allowing adapters with richer capabilities to override behavior. The risk is semantic ambiguity: a default empty history is not the same as “the broker proved there were no fills.” Such defaults need careful callers and documentation.

The forwarding blanket implementation

impl<T: BrokerClient> BrokerClient for &mut T forwards every operation. That makes nested mutable references produced by Option<&mut C>::as_mut() continue to satisfy generic helpers without manual dereferencing at each call site.

Source: broker_trait.rs:295

This is a small but powerful generic pattern: implement a capability for a borrow of every type that already has the capability.

Traits versus enums

Aikido uses both deliberately:

• an open set of broker implementations is modeled with a trait;
• a closed set of executor commands is modeled with an enum;
• strategy implementations may use trait objects where heterogeneous values must coexist in one collection;
• small, fixed runtime choices such as broker order status use enums.

Use a trait when downstream implementations should be extensible. Use an enum when the protocol variants should remain centrally known and exhaustively handled.

Errors, Reconciliation, and Shutdown

In a stateful external system, error handling is not complete when an error has been logged. The application must decide which state is now trustworthy.

Error layers

• anyhow::Result adds context and propagates startup or task-fatal failures.
• Operational failures may warn, update runtime incidents, block future entry, and continue serving safer work.
• Channel closure is often normal lifecycle information rather than an error.
• A command carrying a oneshot sender can report a local result directly to an operator or control-plane caller.

An accepted request is not observed truth

An order API response proves only that an interaction reached some broker boundary. It does not prove the intended position now exists. The executor therefore schedules fast reconciliation after entry-related work.

reconcile_account_exposure queries or consumes a broker snapshot, compares it with local positions, adopts broker-visible positions when appropriate, closes stale records, persists snapshots, and emits lifecycle events.

Source: account_reconcile.rs:734

Desired state is retried against observed state

For ConvergeToTarget, the executor computes:

delta = target_signed_qty - current_signed_qty

It submits only that delta, records the in-flight target, and waits for reconciliation to observe the effect before sending another adjustment. A new desired target can replace pending intent without assuming the prior broker action completed.

Source: maybe_drive_runtime_converge_intent

Safety failures change future policy

Some broker-account failures become persistent entry-block reasons. Existing positions can still require exits, protection, or reconciliation, so “stop all processing” would be less safe than “block exposure increases while preserving risk-reducing operations.”

This pattern separates availability from permission.

Cooperative shutdown

An Arc<AtomicBool> broadcasts a shutdown request. Tasks check it at loop boundaries. The account executor emits close-state feedback for its synthetic observers and persists snapshots before leaving its loop.

Channel ownership supplies another shutdown mechanism: when all engine senders are dropped, the router sees None, exits its loop, and sends explicit ExecutorCommand::Shutdown to executors.

Source: signal_router.rs:237, account_executor.rs:1935

Cancellation is not automatically transaction-safe

Dropping an arbitrary future while it is between broker submission, journal write, and reconciliation could leave uncertainty. The architecture reduces this risk through single-owner execution, explicit timeouts, persistent journals, idempotent intent IDs, and reconciliation. Those mechanisms—not Tokio cancellation alone—restore knowledge after interruption.

Build a Smaller Aikido Runtime

The reconstruction preserves the control loop, not trading sophistication.

Rebuild one proposal source, one router, one single-writer account executor, one broker simulator, desired-versus-observed reconciliation, and safe shutdown.

1. Start with concrete synchronous state

Represent one instrument and one account. Feed predetermined bars into one strategy function and produce Action::Enter, Action::Exit, or Action::Hold. Apply actions to an in-memory account with no traits or tasks.

2. Separate proposal from authority

The strategy returns proposals. A router decides whether a proposal is allowed and which account receives it. The account is the only component allowed to change observed exposure.

3. Introduce a command protocol

enum AccountCommand {
    ApplyProposal(Proposal),
    Converge { target: i32 },
    Reconcile,
    Shutdown,
}

Run one account loop that owns the account and receives commands synchronously. Test ordering and idempotency before adding Tokio.

4. Add bounded Tokio channels

Create one producer task, one router task, and one account task. Give each edge a small bounded channel so overload appears in tests quickly. Close senders and verify receivers finish rather than waiting forever.

5. Add a generic broker

Extract only the capabilities the account loop already uses:

trait Broker: Send {
    async fn adjust(&mut self, delta: i32) -> Result<()>;
    async fn observed_position(&mut self) -> Result<i32>;
}

Implement SimBroker first. Then add FlakyBroker that can accept an order but lose the response, delay observation, and report external position changes.

6. Make target state idempotent

Store desired and observed signed quantities. Submit only their difference. Never assume a successful return changed observed state; poll the broker and recompute.

7. Add stale and duplicate defenses

Give proposals timestamps and intent IDs. Reject stale exposure increases while still allowing exits. Remember recently applied intent IDs in a bounded window.

8. Add lifecycle and supervision

Use a shutdown signal, explicit Shutdown commands, sender-drop behavior, and a JoinSet. Persist a tiny journal before acknowledging mutations. Restart the account task from the journal, then reconcile against the broker.

9. Compare with production

Map the reconstruction back to:

• SignalEngine as proposal generation;
• SignalRouter as centralized policy and routing;
• ExecutorCommand as the ownership-transfer protocol;
• AccountExecutor as the single writer;
• BrokerClient as the generic external boundary;
• ConvergeToTarget as idempotent desired state; and
• reconciliation as recovery of external truth.

The smaller runtime intentionally omits real market data, multiple instruments, bracket orders, account lifecycle rules, policy packs, persistence databases, copy routing, operational UIs, and broker authentication. Its architectural center should still survive process restart, duplicate intent, delayed broker observation, channel pressure, and orderly shutdown.

Axum: Orientation

Axum is a useful study because its pleasant public API hides a rigorous set of adapters. An async function is not inherently an HTTP service. Axum connects that function to the network by converting every stage to a small set of shared protocols: HTTP request and response types, futures, and Tower’s Service.

The architectural center

TCP listener → Hyper connection → Tower Service → Router → Handler
                                                    ↓
                                     extract → await → respond

Axum does not implement an async runtime or HTTP parser. Tokio supplies the runtime and sockets. Hyper drives HTTP connections and request bodies. Tower supplies the composable service and middleware vocabulary. Axum’s center is the type-safe adaptation between an ergonomic handler and those lower-level parts.

Design thesis

Axum keeps handlers ordinary by adapting their typed arguments and futures into Tower services, making HTTP composition reuse one readiness-and-call vocabulary from routing down to middleware.

• Extractors turn request ownership into typed handler inputs.
• Response conversion lets domain-shaped returns satisfy HTTP uniformly.
• Tower layers add policy without changing handler signatures.
• Tokio and Hyper own scheduling and transport; Axum owns adaptation.

Crate boundaries

• axum-core defines foundational extraction, body, and response traits;
• axum provides routing, handlers, middleware, serving, and built-in extractors;
• axum-macros improves ergonomics and compiler diagnostics;
• axum-extra contains useful features that need not enlarge the core API;
• examples demonstrates production-shaped composition.

The most important question

Do not begin with Router::route. Begin with this:

How does async fn create_user(State(db), Json(input)) -> Result<...> become a cloneable service that Hyper can call concurrently?

Answering that question exposes the repository’s generics, macro-generated trait implementations, ownership rules, error model, and async boundaries.

What we will preserve

The reconstruction at the end will preserve route and method dispatch, sequential extraction with a single body consumer, handler futures, response conversion, middleware, explicit capacity limits, cancellation by dropping an in-flight request future, and graceful connection draining.

It will not reproduce HTTP parsing, every extractor, tuple arities, WebSockets, macros, or Tower’s entire type ecosystem.

Interactive Request Map

Choose a process and inspect each ownership, async, and policy boundary. Links target revision 151cd5c12325373b86daf405a6afc0a0086a6706.

The request path is a chain of nested futures. The connection task polls the outer future; each layer polls the next stage until a response is produced.

Why Is It Designed This Way?

This chapter is the center of the case study. Each mechanism protects a specific invariant.

Why build on Tower instead of inventing Axum middleware?

HTTP middleware is service transformation: receive a request, decide when and whether to call an inner service, then transform its response or error. Tower already defines that protocol and a reusable ecosystem. Axum gains composition without coupling routing to each timeout, trace, limit, or retry implementation.

Why are handlers converted to services?

Hyper should not understand Rust function arguments such as State or Json. The handler adapter translates one ergonomic function into the uniform service shape used by routing and middleware. The preserved invariant is: everything below the adapter speaks HTTP requests, futures, and responses.

Why can only the final extractor consume the body?

An HTTP request body is a stream, not clonable data. Two independent extractors cannot both consume it without buffering and replay policy. The split between FromRequestParts and FromRequest makes single ownership visible to the compiler.

Why run extractors sequentially?

Extractors can depend on request extensions produced by earlier middleware or extractors, and metadata extractors share mutable Parts. Sequential order gives deterministic short-circuiting and avoids locking one small per-request object. Independent external I/O belongs inside a purpose-built extractor or handler where join! can express that independence explicitly.

Why must failures become responses?

At the HTTP boundary, invalid input, authorization denial, not-found routing, and application failure all still require a valid response. An infallible outer service guarantees Hyper does not need application-specific error knowledge.

Why is Router always ready?

The router cannot ask the selected endpoint about capacity until it sees the request path and method. Waiting for every possible endpoint would let one unready route stall unrelated routes. Axum chooses always-ready routing and requires capacity policy to be installed at a meaningful layer.

Why clone services per request or connection?

Tower services often require mutable access to call, while the server must handle concurrent work. Cheaply cloned handles allow each in-flight operation to own the service value it polls. Shared expensive resources live behind those handles rather than inside a global router lock.

Why box routes after emphasizing generics?

Every combination of handler and middleware has a distinct concrete type. A router must store many unlike endpoints in one collection and keep compiler output manageable. Static types verify the boundary; internal type erasure provides heterogeneity and stable storage.

Why doesn’t Axum spawn each handler?

The caller already owns and polls a request future. Spawning again would detach cancellation and tracing context, require another join mechanism, and add scheduling overhead. Spawn only when work genuinely needs an independent lifecycle.

Why is graceful shutdown cooperative?

Forcibly stopping a future at an arbitrary instruction would violate resource and protocol invariants. The server stops admission, asks connections to drain, and waits for ownership to be released. Applications add deadlines when bounded shutdown matters more than completing every request.

One Request, Fully Traced

Consider this handler:

async fn create_user(
    State(db): State<Arc<Database>>,
    Path(team): Path<String>,
    Json(input): Json<CreateUser>,
) -> Result<(StatusCode, Json<User>), ApiError> { /* ... */ }

1. A connection gets its own task

serve awaits Listener::accept, constructs a service for the connection, adapts Tower’s service to Hyper, and asks its Executor to run the connection future. The default executor calls tokio::spawn.

Source: serve/mod.rs:563

2. Hyper calls the router as a service

Hyper owns HTTP parsing and connection behavior. When a request is available, the adapter calls Router<()> through Tower’s Service<Request<B>>. Axum normalizes the body to its own Body and enters call_with_state.

Source: routing/mod.rs:599

3. Routing refines the destination twice

PathRouter matches the URI path, stores captured parameters in request extensions, then forwards to an endpoint. MethodRouter selects the GET, POST, or other method service. A failed path or method becomes a fallback response rather than an escaped routing error.

Sources: path_router.rs:325, method_routing.rs:1200

4. The handler adapter splits the request

The macro-generated Handler implementation separates Parts from Body. It runs State and Path through FromRequestParts, in argument order. Each may inspect or mutate metadata, but cannot consume the body.

The final Json argument implements FromRequest and receives the reconstructed whole request. This type distinction makes “the body can be consumed once” a compile-time rule.

Source: handler/mod.rs:221

5. The handler future is awaited

Only after all extraction succeeds does Axum call create_user. Awaiting the database does not block the worker thread: the request future returns Pending, and Tokio may use that thread to poll other ready tasks.

No new task is created merely because the handler is async. It is one nested future inside Hyper’s connection machinery.

6. One concrete response type leaves the boundary

The handler may return many Rust types, but IntoResponse normalizes them. Both Ok((StatusCode, Json<User>)) and Err(ApiError) implement IntoResponse, so Result<T, E> can become the single HTTP Response expected by the service.

Source: into_response.rs:141

accept socket → spawn connection → Hyper parses request → Router::call
  → match path → match method → extract parts → consume body
  → await handler → IntoResponse → Hyper writes response body

This is sequential dependency inside one request. Concurrency appears because many connection and request futures can be suspended and polled independently.

Async, Concurrency, and Parallelism

Axum uses Tokio, but the layers divide responsibility carefully.

Where tasks actually appear

serve runs an accept loop. For each accepted connection it uses an Executor; the default implementation delegates to tokio::spawn. Hyper then drives that connection and may ask the executor to run internal work such as HTTP/2 connection management.

Axum does not manually spawn every handler. A handler produces a future, which becomes part of the request’s Tower service future and is driven by the connection machinery.

Tokio worker pool
├── accept-loop future
├── connection A future
│   ├── request A1 future
│   └── request A2 future when the protocol permits multiplexing
└── connection B future
    └── request B1 future

Concurrency is not parallelism

While one handler awaits a database response, another future can advance on the same OS thread. That is concurrency. Tokio’s multi-threaded scheduler may also poll ready tasks simultaneously on different workers; that is parallelism. Axum permits both, but neither makes CPU-heavy synchronous work non-blocking.

CPU-heavy parsing or computation must be moved to an appropriate bounded worker facility such as spawn_blocking, or designed as separate parallel work. A plain expensive loop inside a handler occupies a runtime worker.

Send + 'static explains many compiler errors

Spawned connection futures may move between runtime threads and outlive the stack frame that created them. Consequently serve, Executor, services, bodies, and handler futures carry Send and often 'static bounds. A handler that holds a non-Send guard across .await cannot satisfy this contract.

Shared state is explicit

Router is cheap to clone because its inner routing table is stored in an Arc. Application state must also be Clone + Send + Sync + 'static. Often the outer state contains Arc handles to pools or services rather than placing the entire application behind one mutex.

The key question is not “which mutex works with async?” but “what must actually have shared mutable ownership?” An immutable config can use Arc<Config>. A database pool already manages its own concurrency. Small synchronous state may use a standard mutex if its guard never crosses .await.

Concurrency limits are policy, not a default

The router reports itself ready. It does not guess the capacity of every downstream operation. Apply a ConcurrencyLimitLayer, timeout, queue, or load shed at the boundary whose scarce resource you understand.

• limit a costly inference endpoint separately from cheap health checks;
• let a database pool bound database connections;
• time out the whole request if downstream work shares one deadline;
• shed load before a queue if stale work has little value.

Scheduling creates concurrency; middleware turns it into explicit resource policy.

The Standard Library as Architecture

Axum’s advanced types are built from ordinary Rust ideas.

Arc makes configured routers cheap to clone

Router<S> contains Arc<RouterInner<S>>. Cloning a router increments shared ownership instead of copying its route tree. Builder methods use copy-on-write style internals while the configured application can be cloned per connection.

Source: routing/mod.rs:86

Infallible proves an error cannot escape

The top-level router’s Service::Error is std::convert::Infallible. Matching on an Infallible value has no branches. This is stronger than “we probably won’t return errors”: the type system proves that all application failures have already become HTTP responses.

Result separates extraction from rejection policy

Extractor traits return Result<Self, Self::Rejection>. Their associated rejection type must implement IntoResponse. The handler adapter uses ordinary matching and early return to short-circuit the remaining pipeline.

Interestingly, extracting Result<T, T::Rejection> itself is infallible. That lets a handler inspect a failed extraction and choose its own policy.

Familiar types compose protocols

• Option represents optional route methods and response metadata;
• tuples represent handler argument lists and layered response parts;
• associated types connect an extractor to its rejection and a service to its response, error, and future;
• marker types distinguish otherwise overlapping generic implementations;
• PhantomData records type relationships without runtime storage.

Pinning protects async state

Connection and route futures are pinned before polling. Once an async state machine may contain references into itself, moving it could invalidate those references. Pin expresses that its memory location is now stable.

Most application authors never manually pin a handler future. Framework code must, because it builds and delegates custom future state machines.

Ownership communicates lifecycle

The request is moved through the service chain. Parts is mutably borrowed by metadata extractors; the body remains separately owned until exactly one final extractor receives it. The response is then moved outward. The signatures are a lifecycle diagram even before any implementation is read.

Generics and Tower

Axum’s generics allow independently written handlers, extractors, bodies, listeners, middleware, and runtimes to agree on contracts.

Service<Request> is the common language

Tower’s conceptual interface is:

trait Service<Request> {
    type Response;
    type Error;
    type Future: Future<Output = Result<Self::Response, Self::Error>>;

    fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>>;
    fn call(&mut self, request: Request) -> Self::Future;
}

Layers transform one generic service type into another. Axum can therefore reuse timeouts, tracing, limits, and authorization without owning them.

How an async function implements Handler

Every async fn has an anonymous future type, and every argument list gives a different function type. Axum implements Handler<T, S> for functions whose arguments implement extraction traits, whose future is Send, and whose output implements IntoResponse.

The T parameter encodes the argument tuple and marker type. It is partly a coherence workaround that distinguishes blanket implementations which might otherwise overlap.

Source: handler/mod.rs:140

A macro expands tuple arities, not runtime magic

impl_handler! generates the same implementation for supported argument counts. At runtime it is ordinary code: split request, await each extractor, call the function, convert the result. The macro compensates for the lack of variadic generics.

Two extractor traits encode body ownership

FromRequestParts<S> receives mutable metadata and a shared state reference. FromRequest<S> owns the complete request. All but the final handler argument must use parts; the last may consume the body. A resource constraint becomes a generic bound.

State is a missing type

Router<S> means “a router still missing state S,” not “a router currently holding S.” Calling with_state supplies that value and can yield Router<()>, which is serveable. State<Inner> uses FromRef<Outer> to derive focused substate from the application’s outer state.

This catches incomplete wiring at compile time while letting libraries request only the state they need.

Static and dynamic dispatch meet in the middle

The public builder API retains concrete generic types for checking and composition. Internally, routes use a cloneable boxed service to store heterogeneous endpoints together. Good generic design chooses the boundary where type erasure makes the whole system usable.

Errors, Cancellation, and Backpressure

Axum distinguishes failures that are valid HTTP outcomes from failures that would otherwise escape the service.

Rejections are responses

Missing path data, invalid JSON, and absent headers are expected request failures. An extractor returns its rejection, and the handler adapter converts it with IntoResponse. The handler is never called.

Application errors follow the same model when their error type implements IntoResponse:

async fn handler() -> Result<Json<User>, ApiError> { /* ... */ }

This does not mean every internal error should be exposed. ApiError is the policy boundary that logs private context and chooses a safe status and body.

Tower errors must be handled before serving

Axum’s top-level router has Error = Infallible, because Hyper needs every request outcome to become a response. Middleware such as a generic Tower timeout may produce an error. HandleErrorLayer maps that error asynchronously to an IntoResponse, restoring the infallible outer contract.

Source: error_handling/mod.rs:115

Cancellation is usually dropping a future

If the peer disconnects and Hyper no longer needs the response, the request future may be dropped. Rust runs destructors for values currently owned by that future, but it does not roll back external effects.

• a database transaction guard can roll back on drop;
• a spawned child task may continue unless explicitly cancelled;
• an already-sent email or payment cannot be unsent;
• a multi-step mutation needs idempotency or a durable workflow boundary.

Cancellation safety is a property of each awaited operation, not a blanket guarantee supplied by Axum.

Readiness is deliberately subtle

Routing needs the request before it knows the destination. Axum therefore keeps routers always ready and drives destination readiness inside the returned future. A backpressure-sensitive service should be wrapped with an explicit load-shed, buffer, or limit policy—or placed around the entire router.

Source: middleware documentation

Graceful shutdown is a drain protocol

On shutdown, the server stops accepting new connections and signals connection tasks to begin Hyper’s graceful shutdown. A Tokio watch channel also accounts for live connection tasks. The server waits until their receivers are dropped.

Graceful does not imply bounded. A handler awaiting forever prevents complete drain, so production applications pair graceful shutdown with request or drain deadlines.

Source: serve/mod.rs:450

Build a Smaller Axum

The reconstruction should preserve the adaptation pipeline, not mimic Axum’s surface syntax.

Build one request type, one service protocol, path and method routing, typed extraction, response conversion, middleware, capacity policy, cancellation, and graceful draining.

1. Start with a concrete synchronous server core

Define Request { method, path, headers, body } and Response { status, body }. Route one path with a match, parse one body, call one function, and return one response. Write tests before adding networking.

2. Introduce the service protocol

trait Service<Req> {
    type Response;
    type Error;
    type Future: Future<Output = Result<Self::Response, Self::Error>>;
    fn call(&mut self, req: Req) -> Self::Future;
}

First implement it with Ready. Then return boxed async futures. Observe which lifetimes force the future to own its inputs.

3. Separate path and method routing

Create a PathRouter whose endpoints are MethodRouter values. Preserve 404 versus 405 behavior. Store captured path parameters in a type map or a simpler request-extension structure.

4. Build extraction without variadic generics

Implement a parts extractor and one body extractor. Manually support handlers with zero, one, and two arguments. This makes the reason for Axum’s tuple macro obvious before you imitate it.

trait FromParts<S>: Sized {
    type Rejection: IntoResponse;
    async fn from_parts(parts: &mut Parts, state: &S) -> Result<Self, Self::Rejection>;
}

Ensure the body-consuming extractor is last.

5. Normalize output and failure

Add IntoResponse for strings, status/body tuples, and Result<T, E>. Create a private application error that logs its source chain but returns a stable public message. Make the outer router error type Infallible.

6. Add middleware as a generic wrapper

Implement logging and timeout wrappers that each contain another Service. Trace the nested type produced by two wrappers. Then introduce one boxed route boundary and compare compile-time readability with runtime indirection.

7. Add Tokio at the edge

Run an accept loop and spawn one connection task per socket. You may use a tiny line-oriented protocol before integrating Hyper. The point is to see that the handler future is nested inside a connection task rather than automatically becoming its own task.

8. Make overload observable

Place a semaphore-backed concurrency limit around one slow route. Test that the limit is enforced. Then compare three explicit policies: wait for a permit, reject immediately, or wait only until a deadline.

There is no universally correct choice; the boundary must name the product policy.

9. Test cancellation and draining

Start a handler, drop its request future, and record which destructors run. Try again after spawning detached work. Finally stop admission, signal existing connections, and wait for an active-task counter to reach zero. Add a drain deadline so a forever-pending handler cannot block shutdown forever.

10. Compare with production

Map your pieces back to serve, Hyper’s connection driver, Router, MethodRouter, Handler, FromRequestParts, FromRequest, IntoResponse, Tower Layer, and HandleError.

The smaller version omits production HTTP correctness, protocol upgrades, HTTP/2 multiplexing, optimized matching, connect information, body utilities, macro diagnostics, and the full Tower ecosystem. Its architectural center should still explain ownership of the body, how handler functions become services, where futures are polled, how overload is bounded, and what happens when a request disappears.

Vector: Orientation

Vector collects logs, metrics, and traces, transforms them, and delivers them to external systems. Its architectural center is not any particular integration. It is the machinery that compiles a configured directed acyclic graph into a running, bounded, observable data plane.

The architectural center

configuration DAG
       ↓ build and connect
Source task → pump → Fanout → Transform task → Fanout → sink buffer
                                                          ↓
                                               batch → request → service
                                                          ↓
                                                   destination

source acknowledgement ← event finalizers ← delivery result

The forward path carries owned EventArray values. The backward path is not a second channel through every component: shared finalizers attached to events aggregate delivery status and eventually resolve a source’s one-shot receiver.

Design thesis

Vector compiles configuration into a bounded graph of independently owned tasks, then carries delivery accountability backward through finalizers attached to the events moving forward.

• Bounded edges turn overload into backpressure instead of memory growth.
• Stateful components own their mutation inside long-lived tasks.
• Generic sink services separate batching and retry from integrations.
• Shutdown proceeds source-first so the graph becomes finite before draining.

Repository centers

• src/topology builds, connects, spawns, reloads, and stops the component DAG;
• lib/vector-core defines events, sources, transforms, sinks, fanout, and component-facing contracts;
• lib/vector-buffers implements bounded memory and persistent buffer edges;
• lib/vector-stream drives request streams into Tower services and records final delivery status;
• src/sinks/util provides batching, request building, retry, rate limiting, adaptive concurrency, and service composition;
• src/sources, src/transforms, and src/sinks contain concrete components.

The useful reading boundary

We will follow one batch from a generic source through a synchronous transform to an Elasticsearch-like network sink. The component is illustrative; the interesting code is shared infrastructure used by many integrations.

We will also trace fanout, overload, acknowledgement, reload, and shutdown. Reading only the successful network call would miss Vector’s real engineering.

What the reconstruction preserves

The smaller implementation will preserve a validated acyclic graph, one task per stateful component, bounded edges, fanout, optional ordered parallel transforms, batched service calls, classified retry, aggregated delivery status, and source-first graceful shutdown.

Interactive Pipeline Map

Choose a process and inspect its ownership, concurrency, and reliability boundaries. Links target revision 1a274c0268c924f37b9a48234a638182a91c2442.

The graph shows logical ownership and waiting boundaries. Tokio may poll its tasks on different worker threads; it does not turn every arrow into a thread.

Why Is It Designed This Way?

Vector’s choices preserve operational properties under continuous load.

Why compile configuration into a DAG?

Acyclic flow gives the runtime a direction for type validation, startup, backpressure, reload, and shutdown. Cycles would require explicit feedback buffering and termination semantics. Rejecting them keeps component contracts local and draining understandable.

Why one task per stateful component?

The task owns mutable transform or sink state and processes an ordered stream. This avoids sharing that state behind locks. Independent components still run concurrently, so serialization exists only where the configuration names a stateful boundary.

Why give sources their own concurrency model?

A universal worker count would ignore the shape of the input. Files naturally partition by tailed file; sockets by connection; Kafka by partition. The source knows where independent progress is safe and where ordering or protocol state must remain coordinated.

Why separate the source task from output pumps?

Sources can expose multiple named outputs, while each output needs independent fanout control and pressure accounting. Pumps keep topology wiring out of integration code and let reload control messages progress even when no events arrive.

Why does fanout wait for every blocking branch?

If one branch could be skipped merely because it was slow, a configured output would silently stop receiving data. Waiting preserves reliable broadcast and propagates the slowest branch’s pressure. Operators who prefer availability over completeness must choose a drop or buffer policy explicitly.

Why parallelize only selected stateless transforms?

Cloning a stateful transform could split counters, windows, deduplication sets, or ordering state into inconsistent islands. Stateless cloneable work is safe to spread across cores. Even then, bounded in-flight work and ordered release prevent memory growth and surprising reorder.

Why put configurable buffers at sinks?

External destinations are the common long-latency and outage boundary. A sink buffer names exactly which destination owns the backlog and lets its durability and full behavior be chosen independently. Giant buffers between every pair of transforms would hide local bottlenecks and multiply memory planning.

Why use Tower services inside sinks?

Once events are encoded as requests, delivery has the same shape as other request systems: readiness, call, response, timeout, rate limit, retry, and concurrency. Tower composes these mechanisms while sink-specific code supplies request building and response classification.

Why prioritize completed service calls in select!?

Starting more requests while completed responses wait would retain finalizers and buffer records unnecessarily. Handling completions first releases delivery obligations and improves forward progress without forbidding new work.

Why implement acknowledgement through finalizers and Drop?

Events can be cloned, split, merged, filtered, buffered, retried, and dropped. A central acknowledgement task would need a parallel protocol for every one of those operations. Ownership already observes when the last branch disappears. Finalizers attach delivery accounting to the value whose lifecycle matters.

Why shut sources down first?

Stopping sinks first would make upstream components fail while still producing. Stopping admission and letting channel closure move downstream preserves the same direction as dataflow and gives queued work a chance to drain.

Why use the same machinery for startup and reload?

Reload is the harder path: resources must be reclaimed, fanouts paused or replaced, persistent buffers retained, and rollback possible. Exercising that machinery at initial startup reduces the chance that a rarely used secondary path has different invariants.

One Event, End to End

Assume a source emits a batch of log events, a synchronous transform changes them, and a network sink delivers them.

1. The source sends an EventArray

SourceSender accepts individual events, streams, or exact-size batches. Batch input is converted and chunked into EventArray values before an async send. If downstream capacity is unavailable, this future waits rather than blocking the Tokio worker thread.

Source: source_sender/output.rs:279

2. A pump isolates the source from topology fanout

Each source output has a pump task. The pump selects between source data and fanout control messages, adds source metadata, then awaits Fanout::send. Separating the source server from fanout permits multiple named outputs and lets reload commands be processed even while a source is idle.

Source: topology/builder.rs:1009

3. Fanout clones ownership, not merely bytes

Fanout arms one send future per downstream component. The last destination receives the original array; earlier destinations receive clones. Event metadata—including shared delivery finalizers—travels with those copies. Fanout completes only when the sends complete or a topology control operation changes their membership.

Source: fanout.rs:313

4. The transform owns its state

A synchronous transform task receives an array, invokes transform_all, fills a reusable TransformOutputsBuf, and awaits its output fanouts. A stateless, compute-heavy transform may instead clone the transform into bounded spawned work and use FuturesOrdered to preserve output order.

Sources: builder.rs:1325, builder.rs:1350

5. The sink buffer absorbs or propagates pressure

The sink’s input is a BufferReceiverStream<EventArray>. Its paired BufferSender applies the configured policy: await free space, drop the newest item, or overflow to another stage. Memory and disk implementations share this topology-facing contract.

Source: channel/sender.rs:241

6. Stream combinators build a request

A typical network sink filters and normalizes events, batches by size/time, encodes requests concurrently under a limit, discards request-build failures through explicit policy, and converts the resulting request stream into a generic Driver.

Source: elasticsearch/sink.rs:64

7. The driver respects service capacity

The driver polls Tower Service::poll_ready, calls ready services, and holds response futures in an unordered in-flight set. Its biased select! gives completed responses priority so acknowledgements free buffer and source capacity promptly.

Source: driver.rs:86

8. Delivery moves backward

After service-level retry is exhausted or succeeds, the driver maps the response to Delivered, Rejected, or another status and updates the request’s finalizers. When the last shared finalizer is dropped, its batch notifier resolves the source’s receiver.

Source: driver.rs:199

source → pump → fanout → transform → fanout → buffer → batch → service
   ↑                                                         │
   └──────────── aggregated finalizer status ────────────────┘

Async, Concurrency, and Parallelism

Vector is a Tokio application whose concurrency structure follows the dataflow graph rather than a single request tree.

Components are long-lived tasks

Sources, stateful transforms, and sinks run concurrently as independent tasks. Bounded channels let one component suspend without occupying a worker thread. Unlike an Axum request, these tasks may live for the lifetime of the process and process millions of values.

One configured source can create multiple tasks: the source server, an output pump supervisor, and one pump per named output. A network source may also spawn per-connection work internally. The source decides the natural unit of concurrency because a file tailer and socket listener have different shapes.

Async pipelining overlaps stages

At one instant the source can read batch C while the transform handles batch B and the sink sends batch A. This is pipeline concurrency. Each stage still owns its local mutable state and communicates through moved values.

time →
source      [A read][B read][C read]
transform          [A map ][B map ][C map ]
sink                      [A send][B send]

Stateless work may run in parallel

SyncTransform is cloneable. For selected compute-heavy transforms, the runner spawns bounded work tasks. Tokio’s multi-threaded scheduler may poll those tasks simultaneously on different cores. FuturesOrdered releases completed output in input order even if later computation finishes first.

This deliberately couples three policies:

• cloneable/stateless work is safe to distribute;
• an in-flight limit bounds memory and scheduling overhead;
• ordered completion preserves observable event order.

Stateful TaskTransforms instead own a stream and remain coordination points.

Sink concurrency is layered

A network sink can overlap work at several distinct boundaries:

• collect events into batches;
• encode/compress several batches with a bounded concurrent mapper;
• wait for Tower service readiness;
• keep multiple network response futures in flight;
• retry requests inside the service stack.

These limits should not be conflated. Encoding is CPU work; service readiness usually represents destination capacity; the input buffer represents tolerated pipeline backlog.

select! expresses service policy

The sink driver simultaneously watches completed requests, service readiness, and new input. Its biased ordering checks completions first. This is not a minor optimization: finalizing completed requests allows acknowledgements and buffer deletion to make forward progress under sustained traffic.

Cancellation must preserve accounting

Source sends may be cancelled while waiting on backpressure. The sender keeps an UnsentEventCount guard so dropped futures still report events that never crossed the boundary. Throughout the pipeline, finalizers use Drop to ensure abandoned event ownership contributes a terminal status rather than silently stranding a source waiter.

The Standard Library as Architecture

Vector’s runtime graph is built from ordinary collections, ownership, and drop semantics.

HashMap gives identity; IndexMap gives stable fanout order

Topology pieces use HashMap<ComponentKey, ...> to assemble tasks, inputs, outputs, and buffers by component identity. Fanout uses IndexMap so destination membership has stable iteration behavior while still supporting dynamic add, pause, replace, and removal.

Enums make operational policy exhaustive

Transform distinguishes function, synchronous, and task components. ControlMessage distinguishes add, remove, pause, and replace. WhenFull distinguishes block, drop-newest, and overflow. EventStatus distinguishes delivery outcomes.

Every match forces the implementation to state what a new policy variant means at the boundary that executes it.

Option represents lifecycle transitions

A paused fanout destination is an Option<Sender> whose sender has been taken. An optional overflow buffer changes the valid meaning of WhenFull::Overflow. The sink driver stores an optional current batch while alternating between input, readiness, and completion. These are small state machines encoded in data rather than scattered booleans.

Arc carries shared identity backward

Events duplicated by fanout share Arc<EventFinalizer> values. Each branch can update delivery status independently. The source is notified only after the last owner releases the shared finalizer. The shared allocation is not generic “global state”; it represents the identity of one delivery obligation.

Atomics aggregate without a central async task

EventFinalizer stores status in AtomicCell<EventStatus>. Updates combine outcomes monotonically, and Drop records the final value into the shared batch notifier. A one-shot channel wakes the source once all batch owners disappear.

Source: finalization.rs:176

VecDeque makes partial service admission explicit

When Tower becomes unready halfway through a ready chunk, the driver puts the remaining requests back into next_batch. A VecDeque supports taking from the front while preserving the remainder for the next readiness cycle.

Ownership closes the graph

Shutdown starts at sources. When they stop and their senders drop, downstream streams eventually observe closure. Transforms finish and drop their outputs; sinks then drain and finish. Channel ownership encodes much of the shutdown dependency graph without broadcasting a stop message to every stage.

Generics and Component Boundaries

Vector combines static generic pipelines with dynamic component discovery. The interesting engineering lies in where it switches between them.

Configuration uses trait objects

At configuration time, one collection must contain many source, transform, and sink types selected from serialized type tags. SourceConfig, TransformConfig, and SinkConfig are object-safe, cloneable trait boundaries that build erased runtime components.

Source: config/source.rs:83

The dynamic boundary answers “which integration did the user configure?” Once a concrete sink is built, its internal hot path can recover static generic composition.

Runtime component enums erase outer differences

Transform stores boxed function, synchronous, or stream-task transforms. VectorSink stores either a futures Sink<EventArray> or a StreamSink. Topology code needs only these small runtime shapes, not every concrete Kafka, HTTP, file, or remap type.

Bufferable bundles required capabilities

BufferSender<T> is generic over items that support event counting, size measurement, finalization, and required thread-safety. Disk-capable Bufferable adds encoding and grouped-finalizer requirements. This lets memory and disk topology code remain reusable without accepting values it cannot account for or persist.

Source: vector-buffers/lib.rs:98

The sink driver is generic over stream and service

Driver<St, Svc> requires:

St: Stream,
St::Item: Finalizable + MetaDescriptive,
Svc: Service<St::Item>,
Svc::Future: Send + 'static,
Svc::Response: DriverResponse,

It knows nothing about HTTP or Elasticsearch. The request must expose metadata and finalizers; the response must explain delivery status. This is the smallest contract that lets the driver manage capacity, concurrency, telemetry, and acknowledgements.

Stream extension traits form a typed assembly language

A sink pipeline chains batching, partitioning, normalization, bounded request building, filtering, and into_driver. Every combinator changes the stream’s item type. Trait bounds prove that the next stage accepts the previous output.

Source: sinks/util/builder.rs:107

Tower makes delivery policy composable

The concrete service is wrapped in timeout, retry, rate limit, and fixed or adaptive concurrency layers. Generics preserve the full composition without a virtual call at each layer. Only selected storage boundaries erase types when heterogeneous values must coexist.

The pattern is worth copying: use trait objects for heterogeneous construction, enums for a small closed set of runtime modes, and generics for a hot homogeneous pipeline.

Backpressure, Retries, and Acknowledgements

These mechanisms solve different failure windows. Treating them as synonyms is the fastest way to misunderstand Vector.

Backpressure limits admission

WhenFull::Block waits for buffer capacity. That wait propagates through fanout, transforms, pumps, and eventually the source. A pull-based source can stop polling; a socket source may stop reading; an upstream client may wait.

DropNewest preserves upstream responsiveness by sacrificing new data. Overflow tries a later buffer stage. These are product policies, not merely performance settings.

A memory buffer absorbs bursts, not crashes

A bounded memory buffer smooths differences between producer and consumer timing. It cannot survive process loss. A disk buffer changes the durability boundary by persisting data, but introduces storage capacity, corruption, and flush semantics that operators must monitor.

Durability answers “will queued data survive?” Backpressure answers “what happens when the queue is full?”

Batching trades latency for efficiency

Network sinks accumulate events until a size, count, or time condition is met. Encoding one batch amortizes headers, compression, and round trips. It also means one service request may own finalizers from many source events.

A bounded request-builder concurrency limit prevents CPU-heavy encoding and compression from turning backlog into unlimited spawned work.

Retry classifies outcomes

RetryLogic examines a response or error and returns Successful, Retry, RetryPartial, or DontRetry. A Fibonacci backoff policy and configured limits control when another attempt is made. The request remains unresolved while the Tower retry layer owns it.

Source: retries.rs:17

Only after retry policy produces a terminal result does the outer driver update event finalizers. This ordering prevents a temporary 503 from being reported to the source as final rejection before retries are exhausted.

Acknowledgement aggregates branches

A source creates a BatchNotifier and attaches cloned handles to events. When fanout duplicates an event, the finalizer’s Arc gains owners. Each sink sets a status and drops its owners after terminal delivery. The notifier’s one-shot resolves only after all relevant owners are gone.

Source: finalization.rs:257

Transforms must preserve, split, merge, or deliberately finalize this metadata when they change event cardinality. That is why finalization capabilities occur in core event and buffer trait bounds.

At-least-once permits duplicates

If a destination accepts a request but its response is lost, retry can deliver the same events again. Without a destination-supported idempotency key or transaction protocol, the sender cannot distinguish “not delivered” from “delivered but response lost.” Vector therefore does not claim exactly-once delivery from generic retry alone.

Shutdown drains from the source side

RunningTopology::stop signals sources first. Transforms and sinks finish when their upstream inputs close, allowing buffered events to move toward delivery. It tracks all task handles, reports lagging components, and can force shutdown after a configured deadline.

Source: running.rs:137

Build a Smaller Vector

Rebuild the data plane and its guarantees, not a toy configuration syntax.

Build a validated DAG whose tasks exchange owned events through bounded edges, fan out reliably, batch and retry sink requests, aggregate delivery status, and drain from sources to sinks.

1. Start with a synchronous concrete pipeline

Define Event { id, payload }, one generator, one uppercase transform, and one collecting sink. Pass a Vec<Event> through direct calls. Record expected order and which component owns each vector.

2. Represent and validate the graph

Store component IDs and input IDs in HashMaps. Reject missing inputs, duplicate IDs, incompatible event types, and cycles before constructing runtime state. Topologically sort the graph and test several invalid configurations.

3. Add bounded Tokio edges

Run source, transform, and sink as long-lived tasks connected by small mpsc::channels. Let receiver closure end downstream tasks naturally. Write a test where a slow sink visibly suspends the source.

4. Implement fanout

Clone events to two bounded destination senders and await both sends. Make one branch slow. Then add an explicit per-branch Block versus DropNewest policy and observe how it changes the system guarantee.

5. Separate stateless and stateful transforms

Define a cloneable per-event transform and a stream-owning stateful transform. For the stateless kind, process batches in bounded spawned tasks and release results in input order. Demonstrate that unordered release is faster in one test but observably different.

6. Build a sink service pipeline

Batch by count or timeout, encode a request, and send it through a small service trait:

trait Service<Req> {
    type Response;
    type Error;
    async fn ready(&mut self) -> Result<(), Self::Error>;
    async fn call(&mut self, req: Req) -> Result<Self::Response, Self::Error>;
}

Keep several calls in flight under a semaphore limit. Interleave response completion with new input rather than awaiting each request serially.

7. Add retry classification

Create Delivered, Retryable, and Rejected outcomes. Use bounded backoff and a maximum attempt count. Simulate “destination accepted, response lost” to show why at-least-once can duplicate data.

8. Attach delivery finalizers

Give each source batch a shared status object and one-shot receiver. Ensure fanout branches retain ownership. The source receives success only after all branches finalize. Test rejection dominance, transform filtering, and a dropped in-flight future.

9. Add memory and durable buffers

Begin with a bounded in-memory queue. Then implement a tiny append-only disk queue with record checksums and explicit acknowledgement before deletion. Crash between write, send, and acknowledgement, then document what is replayed.

10. Drain and reload

On shutdown, stop sources first, drop output senders, and join every task with a deadline. For reload, pause a fanout destination, replace its component and sender, then resume without restarting unchanged branches.

11. Compare with production

Map the reconstruction to TopologyPiecesBuilder, SourceSender, source pumps, Fanout, BufferSender, SyncTransform, TaskTransform, sink stream builders, Tower Service, Driver, and EventFinalizer.

The smaller system omits hundreds of integrations, production codecs, schema propagation, adaptive request concurrency, full disk-buffer recovery, internal telemetry, resource conflict detection, and rollback hardening. It should still make overload visible, avoid unbounded work, preserve chosen ordering, report delivery only after every branch, and shut down without abandoning ownership.

Roadmap

The library now contains twenty-three complete production case studies. The newest wave adds:

• Bevy: conflict-aware ECS scheduling, deferred structural mutation, and a separately owned render world;
• godot-rust: generated FFI registration, dynamic borrow guards, main-thread async, and host-owned lifecycle;
• Quinn: a runtime-independent QUIC state machine under async UDP drivers, targeted wakers, flow control, and drain;
• DataFusion: logical and physical plans, partitioned Arrow streams, backpressure, memory reservations, and spill; and
• Rerun: generated semantic types carried through byte-bounded ingestion, temporal chunk storage, immediate-mode query, and GPU rendering.

Future walkthroughs should be chosen for contrast rather than popularity alone. Strong next candidates are Serde for format-independent traits and code generation, a storage engine such as redb for transactions and unsafe boundaries, and Tauri for commands, IPC, capabilities, and native/web UI ownership.

Meilisearch was considered for this wave, but its repository instructions explicitly prohibit agent-generated analysis of that codebase. It is therefore intentionally excluded rather than represented by an unverified walkthrough.

Each new repository should introduce a different architectural center. Reusing the same walkthrough format makes the differences easier to see.

Helix: Orientation

Helix is a modal editor built around one strong ownership decision: the editor state is mutated in one foreground event loop, while I/O and other waiting work run asynchronously and return results to that loop.

The architectural center

terminal event ─┐
LSP message ────┼→ Application::event_loop → &mut Editor + &mut Compositor → render
timer/redraw ───┤                    ↑
job callback ───┘          async Job returns closure

command → Transaction → Document → Rope + selections + history + LSP change

Design thesis

Helix permits background work to run concurrently but reserves all visible editor mutation for one foreground loop; jobs return owned results and a one-use instruction for applying them.

• &mut Editor marks the single mutation authority.
• Async jobs capture snapshots instead of borrowing foreground state.
• Version checks distinguish completion from continued relevance.
• Transactional edits update text and dependent coordinates together.

The important code is spread across deliberately narrow crates:

• helix-core supplies ropes, selections, transactions, syntax, and editing primitives;
• helix-view owns documents, views, the editor model, and external-service registries;
• helix-term owns the application loop, commands, jobs, compositor, and terminal UI;
• helix-lsp owns JSON-RPC transport and language-server processes;
• helix-event supplies redraw, status, hook, cancellation, and debounce machinery.

What we will preserve

The reconstruction preserves transactional edits, a layered compositor, a single mutation authority, concurrent background jobs that return callbacks, coalesced redraws, stale-result checks, and bounded shutdown. It does not need tree-sitter, every modal command, real LSP, themes, or terminal escape handling.

Interactive Editor Map

Choose a process to see where waiting occurs and where mutation returns to the foreground loop.

The arrows show logical control flow. Tokio can move spawned futures between worker threads, but the editor model remains behind the event loop’s exclusive mutable borrow.

Why Is It Designed This Way?

Why keep editor mutation in one loop?

Text, selections, views, syntax state, history, diagnostics, and UI layers are coupled by user-visible invariants. Serial mutation makes each event an atomic transition without a lock hierarchy or partially updated render.

Why return callbacks from jobs?

A background future cannot safely retain &mut Editor across an arbitrary wait. Returning a closure lets it capture an owned result while the loop grants temporary mutation authority at the right moment.

Why model edits as transactions?

The same change description can update the rope, remap positions, compose with nearby edits, notify LSP, and generate an inverse. Direct string mutation would force every caller to reproduce those consequences.

Why use a rope and cheap snapshots?

Editors repeatedly modify large texts and send stable versions to background analysis. Structural sharing makes both operations affordable and avoids borrowing the live document during analysis.

Why is the compositor dynamically dispatched?

Overlays appear and disappear at runtime and have unrelated concrete types. A trait-object stack matches that product behavior while keeping the editor model independent of every prompt, picker, popup, and integration UI.

Why coalesce redraws?

Diagnostics, progress, typing, configuration, timers, and jobs can all request one. Rendering every request wastes terminal bandwidth and can starve input; rendering on a timer preserves responsiveness.

Why have waited and detached jobs?

Some work is advisory and may disappear at quit; saves and lifecycle work may carry durability obligations. One universal cancellation policy would either lose important work or make exit hostage to irrelevant work.

Why bound language-server shutdown?

Helix owes the protocol a shutdown/exit attempt, but it cannot transfer process liveness control to an unreliable child. Waiting for its own write, under a deadline, is the boundary Helix can actually guarantee.

One Keypress, Fully Traced

Consider an insert-mode key that changes the current document.

1. The event loop wins the terminal branch

Application::event_loop_until_idle uses a biased tokio::select! over signals, terminal input, job callbacks, status messages, waited jobs, and editor events. This is multiplexing: one task waits for many sources without dedicating a thread to each one.

2. The compositor routes the event

handle_terminal_events constructs a short-lived Context borrowing &mut Editor and &mut Jobs, then calls Compositor::handle_event. Layers are visited front to back until one consumes the event. Deferred layer callbacks run only after traversal, avoiding a second mutable borrow of the layer stack during iteration.

3. A command describes an edit

Editing code produces a Transaction, not a sequence of incidental string mutations. A transaction contains a ChangeSet and an optional new selection. It can be composed and inverted, which makes undo and coordinate mapping part of the editing model.

4. The document commits the consequences together

Document::apply_impl applies the changes to the rope, increments the version, maps every view’s selection and anchor through the changes, composes savepoint inverses, updates syntax state, and emits change notifications. A command cannot update text yet forget that cursor positions live in the old coordinate space.

5. Redraw is a result, not a second editor

If handling requests a redraw, the application renders the compositor into a surface and asks the terminal to draw it. Other redraw requests are coalesced through a timer in Editor::wait_event.

The complete invariant is: input may arrive concurrently, but each committed edit and render observes exclusive access to the model.

Async, Concurrency, and Parallelism

Helix uses Tokio, but it is not architected as “put the editor in an Arc<Mutex<_>> and spawn everything.”

One foreground mutation authority

Application owns Editor, Compositor, Jobs, terminal state, signals, and configuration. Its select! loop serializes observable mutations. This removes an enormous class of races between a keystroke, an LSP reply, a save completion, and a redraw.

Background jobs return capabilities

Job contains a 'static + Send future. It may finish with a boxed FnOnce callback that accepts &mut Editor, or both &mut Editor and &mut Compositor.

foreground snapshots owned request data
        ↓
spawned future waits for I/O
        ↓
channel carries Callback, not &mut Editor
        ↓
event loop invokes callback with exclusive access

The future must own or clone everything it keeps across .await. The callback does not own the editor; it receives a temporary borrow only when the event loop is ready to apply the result.

Concurrency is not automatically parallelism

Terminal input, timers, saves, debugger traffic, jobs, and language servers can all be in progress concurrently. Tokio may poll spawned Send jobs in parallel on multiple runtime workers. Yet editing and UI callbacks are serialized. CPU parallelism is not the organizing principle here; responsiveness and ownership are.

LSP transport is an independent state machine

The transport loop selects between outbound client messages, server output, initialization, and process lifecycle. Requests issued before initialization are queued. A shutdown flag is stored before flushing shutdown bytes so the reader cannot misclassify a server request in that narrow window. This is async code implementing protocol ordering, not merely avoiding blocked threads.

Cancellation safety is local and explicit

The transport pins a notification future outside its select! loop because recreating it could lose a permit. Jobs that must complete before exit enter a FuturesUnordered; ordinary jobs are detached but report their errors through the status channel. These choices state which work may be abandoned and which work is part of a clean exit.

The Standard Library as Architecture

Helix’s standard-library types expose its ownership model.

• Vec<Box<dyn Component>> is an ordered, heterogeneous UI layer stack.
• HashMap<ViewId, Selection> lets one document own per-view cursor state.
• Option<T> marks modes and resources that genuinely may not exist.
• Box<dyn FnOnce(...) + Send> is a one-use command carrying captured owned data.
• Arc<ArcSwap<Config>> gives readers cheap configuration snapshots while a reload replaces the shared value.
• std::mem::take temporarily moves accumulated changes or futures out of a borrowed owner so they can be processed safely.
• process handles, atomics, and Duration make external lifecycle and deadlines explicit.

The crucial distinction is between value snapshots and mutation rights. A rope clone is cheap structural sharing suitable for an async operation. An &mut Editor is deliberately scarce and never held across a detached task.

This is a recurring Rust design pattern: clone stable input, move it into work, then return a small result or command to the unique owner.

Generics and Trait Boundaries

Helix uses static polymorphism where a caller already knows the concrete type, and dynamic polymorphism where runtime composition demands heterogeneity.

Generics at construction and stream boundaries

Application::event_loop<S> is generic over any Stream<Item = io::Result<TerminalEvent>> + Unpin. Jobs::spawn<F> accepts any suitable future, and replace_or_push<T: Component> accepts a concrete component. These APIs are easy to inline and do not require users to name a boxed type.

Trait objects in long-lived collections

The compositor cannot have a Vec<TextEditor | Picker | Prompt | ...> enum that every extension must modify. It stores Box<dyn Component>. Jobs similarly erase distinct future and closure types into BoxFuture and boxed FnOnce callbacks so one queue can hold them.

Associated behavior stays cohesive

Component groups event handling, rendering, cursor selection, sizing, and identity. Dynamic dispatch is paid at a UI boundary where terminal rendering dwarfs the call overhead and runtime layering is the actual requirement.

The rule is practical: use generics to accept many implementations; erase the type when values must coexist, cross a channel, or outlive the generic call.

Errors, Stale Results, and Shutdown

Errors become editor-visible state

Job futures return anyhow::Result. A failed ordinary spawned job is reported through the event/status system; a failed waited job becomes Async job failed in the editor or propagates during final draining. The application loop remains alive because one optional feature failed.

Versioning separates completion from validity

Documents increment an LSP version on each real change. Async commands commonly capture a document ID, version, selection, or rope snapshot. Their callbacks must re-find the document and decide whether a result still applies. Completion only means “the service answered”; it does not prove the answer describes the current buffer.

Redraw pressure is coalesced

request_redraw wakes the editor, but wait_event schedules a redraw timer rather than rendering once for every producer notification. This converts a potential event storm into a bounded frame rate while still allowing urgent foreground event handling.

Shutdown distinguishes protocol courtesy from process control

close_language_servers enqueues shutdown and exit for every server, then waits only until those bytes are flushed to each server’s stdin. The wait has a timeout. It does not wait indefinitely for an external process to behave; process handles are configured to be reaped when dropped.

Jobs opt into wait_before_exiting. Final drain executes their callbacks when the necessary mutable references still exist and can enqueue follow-up waited jobs. Clean exit is therefore an explicit transitive obligation, not “sleep a little and hope.”

Build a Smaller Helix

Rebuild the ownership and event machinery, not a toy clone of modal syntax.

Build a tiny editor whose foreground loop owns all mutable state, applies invertible transactions, renders dynamic layers, and accepts asynchronous results only through version-aware callbacks.

1. Start with one concrete editor

Use String, a cursor index, and insert(char). Read scripted input events and print the buffer after each one. Write down the invariant that the cursor is on a UTF-8 boundary.

2. Introduce transactions

Represent retain/delete/insert operations, apply them only to a matching input length, map a cursor through them, compose adjacent edits, and construct an inverse from the pre-edit snapshot. Add undo before async code.

3. Separate document, view, and application

Let a document own text, version, and history; a view own viewport and selection; and the application own documents, views, and current focus. Open two views on one document and prove both selections map through an edit.

4. Add a dynamic compositor

Define Component { handle_event, render }, store boxed layers, and bubble events from the top layer down. Add a prompt overlay that consumes keys without teaching the base editor about prompts.

5. Replace the blocking loop with Tokio

Select over an input channel, redraw notification, timer, and job callback channel. Keep Editor as a plain owned field—do not introduce a mutex.

6. Add the job/callback bridge

Spawn a fake “language service” that receives an owned text snapshot and version, waits, and returns a boxed callback. Apply its diagnostics only if the document still exists and its version matches. Test an out-of-order result.

7. Coalesce rendering

Make ten background notifications request redraw. Render at most once per frame interval, but process input immediately. Count renders in a deterministic test.

8. Add shutdown obligations

Classify jobs as detachable or must-finish. Stop admission, await required jobs under a deadline, execute their returned callbacks, and abort or drop the rest. Simulate a child service that never answers.

9. Compare with production

Map your types to Application, Editor, Document, Transaction, Compositor, Component, Jobs, Callback, redraw events, and LSP transport. Helix adds a mature rope, syntax trees, multi-selection editing, protocols, terminal compatibility, plugins, debouncing, cancellation, and thousands of commands. Your smaller version should still preserve its architectural center: many things wait concurrently, one owner commits editor state.

uv: Orientation

uv turns a declarative Python environment into a consistent filesystem state. Its architectural center is a staged pipeline: discover and resolve packages, reuse or prepare artifacts, calculate an install plan, lock the target, and apply that plan using cache-backed files.

requirements → resolver ↔ concurrent metadata fetch
                         ↓ Resolution
installed state → Plan → prepare wheels → environment lock → uninstall/install
                                  ↕
                       append-oriented shared cache

The cache and target environment have different concurrency contracts. Cache entries are designed for concurrent readers and writers; mutation of one environment is guarded by a cross-process file lock.

Design thesis

uv overlaps slow preparation aggressively while assigning a separate limit and ownership protocol to each scarce resource, then exposes artifacts only after validation and atomic publication.

• Download, build, and install budgets are distinct semaphores.
• Same-key in-flight work is shared before a cache entry exists.
• Immutable cache artifacts support broad concurrency.
• One target lock protects the multi-file environment transition.

We will preserve staged resolution, per-resource concurrency limits, in-flight request deduplication, bounded blocking work, immutable artifacts, atomic cache publication, target locking, and structured failure context.

Interactive Sync Map

These are separate limits and coordination domains. “Concurrency = 8” is too coarse for a system where HTTP, builds, cache reads, installs, and subprocesses consume different resources.

Why Is It Designed This Way?

Why resolve before installing?

Mutation should not be used as a search procedure. A complete resolution lets uv report conflicts, create a reproducible lock, calculate removals, and prepare artifacts before disturbing the environment.

Why start many metadata requests?

Network latency dominates individual index lookups, while the resolver may need many independent facts. Overlap reduces wall time; the bounded request channel, deduplication, and deeper semaphores prevent unlimited resource use.

Why have separate concurrency budgets?

Fifty HTTP responses, fifty Python compilers, and fifty filesystem installers have radically different costs. One global permit pool would either starve cheap I/O or overload CPU, memory, child processes, and file handles.

Why share semaphore instances?

Nested builds and command helpers are part of the same process-wide load. If each layer interpreted “8 builds” independently, composition would multiply the promised limit.

Why deduplicate in-flight operations?

Dependency graphs converge: several packages may request the same metadata or artifact before the cache is populated. Waiting on one producer saves work and ensures all callers observe one result.

Why make the cache append-oriented and atomic?

Immutable entries are easy to share between threads, processes, and uv versions. Publishing with rename turns partial construction into a private detail and makes interruption recoverable.

Why lock the environment but not serialize the whole cache?

Independent artifacts can safely be prepared concurrently. site-packages is a shared multi-file namespace where overlapping uninstall/install operations would corrupt ownership. The narrow lock preserves both speed and correctness.

Why use external Python processes?

Python build backends and bytecode compilation are Python-defined behavior. Subprocess isolation respects that ecosystem boundary and limits failures, while bounded workers keep the orchestration under Rust’s control.

One uv sync, Fully Traced

1. Discover desired and actual state

The project command loads workspace metadata, Python requirements, settings, lock mode, interpreter, and the existing environment. This is mostly ordinary owned data and borrowing; async becomes important when discovery reaches the network, subprocesses, or locks.

2. Lock the environment before mutation

sync acquires the environment lock before the resolve-and-install path. Separate uv processes may populate the shared cache, but must not concurrently rewrite the same site-packages.

3. Produce or validate the lock

Lock modes encode frozen, locked, dry-run, and write behavior. The resolver drives a PubGrub-like decision process while metadata requests execute concurrently. Resolution is coordination-heavy CPU work fed by asynchronous evidence, not a parallel search that may accept inconsistent branches.

4. Derive an installation plan

Installed distributions, the resolution, reinstall policy, and cache state are compared to classify packages as already satisfied, removable, cached, or remote. Planning is separated from mutation so diagnostics and dry-run can describe the intended transition.

5. Prepare missing artifacts

Preparer starts distribution futures together. Downloads and builds are limited deeper in the database by shared semaphores. The largest artifacts start first to reduce the long tail.

6. Apply under exclusive target ownership

uv removes obsolete packages and installs prepared wheels using links, reflinks, or copies from its archive cache. Temporary files and renames avoid publishing partially written cache entries; the environment lock protects the multi-file target transition.

Async, Concurrency, and Parallelism

uv uses Tokio for waiting, Rayon and worker pools for CPU/file work, and subprocesses for Python builds. The distinctions matter.

Limits follow resources

Concurrency has independent download, build, install, and cache-read limits. Shared Arc<Semaphore> values make the download/build budgets global across nested operations; a recursive build cannot accidentally create its own full budget.

Resolution overlaps evidence gathering

The resolver consumes requests from a bounded channel and uses buffer_unordered to let metadata futures progress independently. The comment at Resolver::fetch is important: fine-grained database/build limits provide pressure below this wide logical fan-out.

Preparation is concurrent but deduplicated

Preparer::prepare_stream builds a FuturesUnordered. Before doing work, each distribution calls InFlight::register_or_wait. One future becomes producer; same-key callers await and clone its terminal result. Concurrency therefore does not mean duplicate network and build work.

Blocking work crosses an explicit bridge

Archive parsing, Git operations, file locks, and synchronous build-backend work use spawn_blocking or dedicated workers. Python bytecode compilation uses a bounded channel and a fixed number of child-process workers. This protects Tokio workers and bounds processes, file descriptors, and queued paths.

Cancellation and mutation have different costs

Dropping a metadata request is often cheap. Interrupting an environment update can leave a multi-file partial transition, so uv orders planning, preparation, locking, and mutation to minimize that window. Atomic cache publication makes cancelled producers leave either a complete entry or no visible entry.

The Standard Library as Architecture

• Path and PathBuf distinguish borrowed locations from owned paths moved into workers.
• Arc<T> shares immutable distributions, caches, reporters, and semaphore budgets.
• HashMap/HashSet encode package indexes, installed state, and plan membership.
• BTreeMap supplies deterministic serialization where reproducible lock output matters.
• NonZeroUsize and available_parallelism() connect configuration to a valid machine-derived default.
• Cow<'a, T> avoids cloning unchanged requirements while still allowing marker-adjusted owned variants.
• filesystem handles and RAII lock guards make process-level ownership end on drop.
• Command, ExitStatus, and pipes model Python/build tools as fallible external actors.

Most performance comes from choosing which values are immutable and reusable. An Arc<Dist> can cross many futures; the mutable environment cannot. A PathBuf moved into spawn_blocking is simple proof that the task does not retain a borrowed command context.

Generics and Trait Boundaries

uv’s internal crates use generics to specialize hot, strongly typed pipelines.

Preparer<'a, Context: BuildContext> borrows tags, cache, hashing policy, and a DistributionDatabase parameterized by the build context. Resolver fetching is generic over Provider: ResolverProvider. PEP 508 parsing is generic over a Pep508Url representation. Associated types keep errors and domain values connected to their provider.

Trait objects appear where optional runtime substitution is genuinely useful. Option<Arc<dyn Reporter>> lets CLI progress implementations observe the same pipeline without parameterizing every enclosing command. Errors are frequently boxed at domain boundaries to keep large recursive error enums manageable.

Enums carry closed domain variation: Dist::Built | Source, cache buckets, lock modes, link modes, and resolution outcomes. These are not trait objects because exhaustive policy decisions are valuable.

The pattern is: generics for compile-time collaborators in core algorithms, enums for closed protocol/domain states, and trait objects for leaf observers or heterogeneous errors whose exact type no longer affects control flow.

Cache Safety, Errors, and Interruption

Errors preserve domain context

Crates define specific thiserror enums rather than one process-wide error. For example, preparation distinguishes forbidden builds, forbidden binaries, distribution failures, cyclic build dependencies, and failures copied from the producer of a deduplicated request. Distribution errors carry the failing artifact and derivation chain so CLI reporting can explain why it was needed.

The cache is shared, append-oriented evidence

Downloaded and built artifacts become immutable archive entries. uv prepares a temporary directory, validates wheel records and hashes, then Cache::persist renames it into the archive. Other cache buckets can point at that immutable entry. Readers never need to observe construction in progress.

In-process duplicate work has one producer

uv-once-map registers a key or waits for the registered producer. The producer publishes either success or a cloneable error representation to all waiters. This is cache stampede control, not just memoization after completion.

Cross-process mutation uses locks

The environment is locked for installation. Cache-cleaning operations take a stronger cache lock and use a timeout so administrative work does not deadlock forever behind active commands. RAII releases locks on error or cancellation.

Output durability has layers

Atomic file replacement protects individual metadata and cache entries. A target lock protects the environment’s multi-file transition from competing writers. A lockfile records the resolved intent. None alone provides all three guarantees.

Build a Smaller uv

Build a package synchronizer that resolves a graph, concurrently prepares immutable artifacts into an atomic cache, and applies a plan under an exclusive target lock.

1. Resolve a concrete registry

Use an in-memory map from (name, version) to dependencies. Implement backtracking for version constraints and produce a complete Resolution before touching a target directory.

2. Add asynchronous metadata

Serve registry records through a fake latency-injecting provider. Feed requests through a bounded channel, overlap responses with FuturesUnordered, and keep the resolver’s decision state in one task.

3. Deduplicate in-flight keys

Implement register_or_wait around a map of shared completion cells. Request one package from five branches and assert the provider runs once. Propagate the same failure to every waiter and remove or finalize the entry deliberately.

4. Separate resource budgets

Add global semaphores for downloads and builds, plus a fixed install-worker count. Record peak counts in tests. Make a recursive source build use the same build semaphore.

5. Publish an immutable cache entry

Download into a temporary directory, verify a checksum and manifest, then rename into a content-addressed archive. Kill the task before and after rename; readers must see either absence or a complete artifact.

6. Calculate a plan

Compare desired resolution with installed manifests and classify keep, remove, install-from-cache, and prepare. Support dry-run by printing the plan without mutation.

7. Lock and apply

Take an OS file lock for the target, remove stale packages, then link or copy files from cache with atomic per-file replacement. Run two synchronizers against one environment and verify their mutation phases never overlap.

8. Bridge blocking work

Put archive extraction in spawn_blocking; run a fake build backend as a child process with captured output and a timeout. Ensure neither consumes unbounded Tokio workers or child processes.

9. Compare with production

Map the result to uv’s resolver/provider, InFlight, Concurrency, DistributionDatabase, Preparer, Plan, Installer, Cache, atomic fs helpers, and environment locks. The real uv adds Python’s full metadata and tag model, PubGrub diagnostics, registries, Git, build isolation, cross-platform link modes, authentication, and extensive cache invalidation. Preserve the center: concurrent evidence gathering, staged decisions, and narrow mutation.

rust-analyzer: Orientation

rust-analyzer keeps an in-memory model of Rust code responsive while files, project structure, compiler output, and editor requests change continuously. Its architectural center is a mutable coordinator that applies input changes and hands immutable analysis snapshots to cancellable worker computations.

LSP/VFS/Cargo events → main loop → GlobalState → AnalysisHost/Salsa inputs
                                      ↓ snapshot
                                worker request
                                      ↓
                              Task::Response → main loop → LSP

new input → new Salsa revision → old computations cancel/unwind

Design thesis

rust-analyzer serializes changes to ground truth in one coordinator while lending immutable database revisions to parallel, cancellable computations that derive only what the current editor request demands.

• Salsa snapshots give workers a coherent read world.
• New input cancels CPU work whose answer would already be stale.
• Memoized dependency graphs preserve unaffected computation.
• Process isolation contains proc-macro crashes and unstable compiler code.

The reconstruction preserves lossless parsing, a VFS/input boundary, incremental memoized queries, immutable snapshots, serial mutation, background read requests, revision cancellation, latency classes, and subprocess isolation.

Interactive Analysis Map

rust-analyzer is mainly a thread-and-channel application rather than a Tokio application. Concurrency comes from snapshots, worker pools, Rayon, actors, and isolated subprocesses; async syntax is not a prerequisite for responsiveness.

Why Is It Designed This Way?

Why one mutable GlobalState?

VFS changes, request queues, diagnostics generations, configuration, and workspace reloads require a total order. Serial coordination is cheaper and clearer than distributed locks; expensive read derivation still runs elsewhere.

Why snapshots rather than locks around analysis?

Workers need a stable semantic world while the user keeps typing. Snapshots allow parallel reads and make cancellation/revision checks explicit instead of holding a global read lock that delays fresh input.

Why incremental queries?

An IDE repeatedly asks related questions after tiny edits. Dependency-tracked memoization reuses unaffected facts and computes only what a feature demands.

Why cancel old work aggressively?

A perfectly computed completion for yesterday’s cursor position has negative value. Unwinding stale queries frees scarce CPU for the latest revision and lets callers retry under an explicit policy.

Why run typing requests on the main thread?

Very small on-enter, matching-brace, or selection operations are latency sensitive. Queueing behind large semantic requests could cost more than the computation, provided they remain read-only and predictably cheap.

Why isolate formatting in one thread?

The editor may synchronously wait for formatting, but rustfmt can block. Its own lane prevents both main-loop blocking and starvation behind analysis jobs.

Why keep syntax, semantics, IDE, and LSP types separate?

Each boundary has different stability and failure rules. Syntax should work without a project; semantics should not know paths; IDE values should describe editor concepts; wire compatibility belongs only to the LSP binary.

Why isolate proc macros in a process?

They execute arbitrary dynamic libraries and can crash or behave nondeterministically. A thread cannot contain a segfault or unload all bad process state; a child process can be restarted.

One Completion Request, Fully Traced

1. The main loop receives JSON-RPC

GlobalState::run selects events from LSP, worker tasks, VFS, flycheck, workspace loading, and test processes. One thread owns mutable GlobalState and establishes their order.

2. Typed dispatch claims the method

RequestDispatcher is a compile-time routing chain. Parsing turns raw JSON into the request type’s associated Params; the handler must return its associated Result. Typing-related requests may run synchronously for minimal scheduling latency; most read-only work uses the pool.

3. The server takes a snapshot

GlobalState::snapshot clones Arc configuration/workspace/VFS views and obtains an immutable Analysis snapshot from AnalysisHost. The worker never borrows mutable server state.

4. IDE façade lowers position to semantics

LSP coordinates become a file ID and offset. ide calls into HIR and Salsa queries: parse this file, resolve the enclosing syntax node, build scopes, infer types, and construct completion candidates. Only demanded facts compute; unchanged dependencies reuse memoized values.

5. The result crosses back as a task

The worker converts IDE POD values to LSP types and sends Task::Response. The main loop matches the request ID, records completion, and sends JSON-RPC. If a new input revision cancelled the analysis, dispatch maps that outcome to content-modified or retry policy instead of sending stale semantics.

Concurrency, Parallelism, and Cancellation

Serial input, concurrent derivation

The main loop exclusively applies VFS and configuration changes. Read-only requests use GlobalStateSnapshot on a custom thread pool. This resembles multi-version concurrency control: many readers derive answers from a stable revision while one coordinator advances ground truth.

Salsa supplies dependency-aware reuse

Each query records the input/query facts it read. A changed function body does not invalidate unrelated bodies or crate-wide facts unless dependency edges say it should. Parallel requests can share memoized results through the database rather than each reconstructing a compiler model.

Cancellation is revision invalidation

AnalysisHost::apply_change triggers cancellation before changing inputs. Old Salsa computations detect the new revision and unwind with salsa::Cancelled. RequestDispatcher catches the unwind at the read-only boundary and maps it to retry or LSP content-modified. The main loop remains intact.

Pools encode latency intent

stdx::thread::Pool queues boxed jobs with a ThreadIntent. A separate one-thread formatting pool prevents ordinary semantic work from delaying an editor-blocking rustfmt request. Cache priming and symbol indexing use bounded workers/Rayon for actual CPU parallelism.

Actors isolate I/O and failure

VFS watching, Cargo/flycheck, and proc-macro services communicate through channels. Proc macros run in another process because they may block, panic, segfault, or violate determinism. The language server treats partial project failure as degraded input, not permission to stop serving syntax features.

The Standard Library as Architecture

• Arc<T> makes configuration, workspace data, strings, syntax trees, and query outputs cheap snapshot values.
• Mutex/RwLock are confined to caches and VFS views that genuinely cross worker boundaries.
• PathBuf exists in project/VFS layers; semantic base-db uses opaque FileId, preventing filesystem leakage.
• Option<T> is fundamental because incomplete Rust syntax is ordinary IDE input.
• Result<T, E> represents operational failure; parser APIs instead return a tree plus an error list.
• Box<dyn FnOnce()> type-erases heterogeneous worker jobs.
• atomics count extant tasks and make lightweight revision/lifecycle observations.
• Command and child handles separate Cargo, rustfmt, flycheck, and proc macros from the analysis process.

The best lesson is negative capability: types make some dependencies impossible. The parser cannot query Salsa; semantic crates cannot discover paths; IDE return values do not leak syntax/HIR objects; only the binary crate serializes LSP.

Generics, Traits, and API Boundaries

rust-analyzer uses traits differently at each architectural layer.

Parser traits such as TokenSource and TreeSink keep the grammar generic over token and tree representations. Salsa’s Database trait is passed as &dyn salsa::Database through query-heavy compiler internals, giving generated query machinery one database boundary without parameterizing every HIR type.

RequestDispatcher::on<R> is generic over R: lsp_types::Request. Associated Params and Result types make mismatched handlers impossible, while const generic retry policy records cancellation behavior at the call site.

The public ide API deliberately exposes concrete POD-like domain values and an Analysis snapshot. HIR provides an object-oriented façade over ID-oriented internals. Closed variations—events, tasks, syntax kinds, completion kinds—use enums for exhaustive handling. Heterogeneous worker functions and VFS handles use trait objects because they must occupy one queue or field.

Generics protect local type relationships; façades protect crate relationships. Both matter more here than saving dynamic-dispatch nanoseconds.

Errors, Broken Code, and Lifecycle

Broken source is data

Parsing produces a syntax tree plus errors, never “no tree.” AST accessors return Option even where the grammar says a child should exist. IDE features can therefore operate on the incomplete file currently being typed.

Read failures are contained

Read-only handlers run inside catch_unwind. Ordinary errors become JSON-RPC errors; Salsa revision cancellation becomes retry/content-modified; unexpected panics are logged and isolated because the worker held only a snapshot. Mutable main-thread handlers are intentionally not recoverable in the same way.

External tools degrade independently

Cargo metadata, flycheck, rustfmt, and proc macros have dedicated messages and status. Workspace reload does not make the server wholly unavailable. Syntax highlighting can fall back while proc macros load; a broken build still permits local editing features.

Shutdown is a protocol state

The shutdown request marks shutdown_requested, drops proc-macro clients, cancels flycheck, and clears discovery handles. Later requests are rejected. The main loop exits only on the LSP exit notification; client disappearance without that sequence is reported as an error. Dropping pools closes their job channels before joining workers, an ordering documented in the field layout.

Build a Smaller rust-analyzer

Build a language service with lossless parsing, revisioned inputs, dependency-tracked queries, snapshot workers, and cancellation of stale work.

1. Parse incomplete expressions

Create tokens and a tiny immutable syntax tree. Return (tree, errors) and add error nodes rather than failing. Make every typed child accessor optional.

2. Add ground inputs

Store FileId → Arc<str> and a small crate graph. Keep paths in a separate VFS map. All semantic functions accept IDs, never read files.

3. Build a memoized query engine

Implement parse, definitions, name resolution, and type-of-expression queries. Record query dependencies and input revisions. Reuse a result only if its transitive inputs remain unchanged.

4. Separate host and snapshot

AnalysisHost owns mutable inputs. analysis() returns a cloneable read snapshot. Applying a change advances the revision and invalidates outstanding snapshots.

5. Add main-loop dispatch

Use channels for client, VFS, and worker events. Apply all mutations in one loop. Route typed read requests with a snapshot to a fixed thread pool and send responses back as tasks.

6. Make stale work cancel

Have long queries periodically compare their captured revision with the host’s current revision and unwind or return Cancelled. Map cancellation to retry for idempotent requests and content-modified otherwise.

7. Add latency classes

Keep trivial typing operations synchronous, semantic work on a worker pool, and one blocking formatter on a separate lane. Saturate the semantic pool and prove the typing and formatting paths still respond.

8. Isolate an extension

Run a fake macro expander as a subprocess over framed messages. Crash it on one request; preserve syntax features, report degradation, and restart it.

9. Compare with production

Map your design to VFS, GlobalState, AnalysisHost, Analysis, Salsa inputs and tracked functions, RequestDispatcher, TaskPool, HIR/IDE façades, Cargo, flycheck, and proc-macro server. The real repository adds the Rust language, macro hygiene, rich inference, project models, thousands of features, and careful memory tuning. Preserve its center: immutable derived worlds that can be abandoned the moment ground truth changes.

Linkerd2-proxy: Orientation

Linkerd2-proxy accepts long-lived TCP connections, detects protocols, enforces policy, discovers destinations, balances endpoints, proxies HTTP or opaque bytes, records metrics, and drains gracefully. Its architectural center is a pair of statically composed Tower service factories: inbound and outbound.

accept connection → detect TLS/protocol → authorize → per-target service
                                            ↓ HTTP request
route → discover/watch endpoints → queue/readiness → balance → connect/mTLS → proxy

Design thesis

Linkerd builds a concrete Tower stack for each target and uses readiness as the contract that transfers traffic only when downstream capacity exists, even while discovery and policy change concurrently.

• Generic stack construction specializes the hot data path.
• Bounded queues absorb bursts without hiding indefinite overload.
• Watches update long-lived services without rebuilding every request path.
• Drain stops admission before awaiting protocol-level completion.

We preserve target-typed stack construction, readiness/backpressure, dynamic discovery, cached per-target services, bounded queues, balancing, retries with replay limits and budgets, failure classification, and drain propagation.

Interactive Proxy Map

The stack is large at compile time but the runtime call path is ordinary Tower: poll readiness, move a request into call, await a response future.

Why Is It Designed This Way?

Why build stacks from targets?

Policy, metrics labels, TLS identity, routing, and connection behavior vary by port, route, destination, and endpoint. Instantiating a specialized service when the target becomes known moves lookup work out of every request.

Why make readiness separate from calls?

Capacity can disappear before a request is accepted. Polling readiness lets queues, balancers, pools, and overload policies cooperate without taking ownership of work they cannot serve.

Why use static Tower composition so heavily?

The data plane executes these wrappers for every request. Generics preserve request/response relationships, inline layers, and surface invalid ordering or missing target parameters during compilation.

Why cache services by target?

Discovery streams, connection pools, EWMA load, and retry budgets are stateful. Rebuilding them per request would discard learning and multiply control-plane subscriptions. Idle eviction bounds retained cardinality.

Why use watch for policy and credentials?

Consumers need current configuration, not every historical intermediate value. Latest-value delivery prevents slow data-plane consumers from creating an unbounded control-plane backlog.

Why bound queues and add failfast?

An unbounded queue turns an outage into memory growth and enormous tail latency. A small queue absorbs transients; a deadline exposes sustained unavailability so upstream systems can apply their own policy.

Why are retries route-aware and budgeted?

Retries multiply load precisely when a service is failing. Response classes express application semantics; replay limits protect memory; budgets prevent a local recovery mechanism from becoming a mesh-wide retry storm.

Why drain instead of aborting tasks?

Requests may already have passed policy, changed remote state, or begun a streaming response. Stopping admission and letting ownership reach natural end-of-stream gives protocols a coherent completion boundary.

One Proxied Request, Fully Traced

1. A connection becomes a typed target

The listener accepts a socket and captures original destination, peer address, transport labels, and drain state. Inbound detection peeks at bytes under a timeout to distinguish TLS, HTTP, and opaque transport. The result enriches a target value used to instantiate the next stack.

2. NewService specializes the stack

Layers do not inspect global configuration on every request. NewService<T> receives a target and constructs a concrete Service<Request> with cloned parameters, policy receivers, metrics labels, caches, and inner services. Param<T> statically declares which configuration a target supplies.

3. HTTP middleware transforms and authorizes

The inbound stack normalizes URIs, attaches tracing/metrics, applies server policy and route policy, classifies responses, and selects local forwarding or gateway/outbound behavior. Each layer wraps the same readiness/call contract.

4. Outbound discovery becomes a service set

Logical destination and route policy select a cached per-target router. Controller or DNS updates feed endpoint add/remove changes. Each endpoint service encapsulates connect, TLS identity, protocol client, timeouts, and metrics.

5. Readiness chooses capacity before ownership transfer

A queue absorbs only configured backlog. The balancer polls endpoint readiness and chooses an available endpoint using load. Service::call then owns the request future. For HTTP/2, many request futures may share one connection; HTTP/1 has different connection concurrency.

6. Completion walks wrappers outward

The endpoint response returns through classification, retry, timeout, tracing, and metrics bodies. End-of-stream matters: a successful response head is not necessarily successful delivery if its body later fails.

Async, Concurrency, and Backpressure

Tokio schedules connection tasks, control-plane watches, timers, DNS, and request futures. Tower supplies the concurrency contract inside each path.

Connections and requests are nested concurrency

Listeners spawn independent connection lifecycles. A multiplexed HTTP/2 connection carries concurrent streams; a destination cache holds independent per-target services; a balancer distributes calls across concurrently ready endpoints. These are different levels, not one worker-count setting.

poll_ready is admission control

Tower separates “may I send?” from call. Queues, buffers, failfast, load shed, connection pools, and balancers compose by propagating readiness. A bounded NewQueue can wait through brief unavailability; failfast/load-shed turn sustained lack of capacity into an explicit error.

Watch channels carry latest control state

Policy, profiles, endpoint sets, and TLS credentials change while traffic flows. watch::Receiver<Arc<T>> gives many stacks the latest immutable value without queueing every intermediate configuration. Endpoint discovery uses a change stream because additions/removals must update a live pool.

Data-plane concurrency remains bounded by demand

Idle caches lazily build services and evict unused entries. Per-target queues have capacities and timeouts. Retry budgets cap amplification. Response body wrappers retain permits/metrics state until end-of-stream or drop, tying accounting to actual ownership.

No substantial CPU data parallelism is central to the proxy. Parallelism comes from Tokio polling independent network work on runtime workers; the engineering focus is controlled concurrency and predictable tail latency.

The Standard Library as Architecture

• SocketAddr, IpAddr, and newtypes encode transport identity without strings.
• tuples and small target structs accumulate typed context as a connection crosses stacks.
• Arc<T> shares immutable policy, TLS configs, metrics families, and retry budgets.
• HashMap stores routers, endpoints, caches, and label-indexed metrics.
• Duration makes queue, detection, idle, failfast, and request timeouts distinct.
• Option<T> and Conditional<T, Reason> distinguish absence from an explicitly disabled capability with a reason.
• Pin<Box<dyn Future + Send>> erases futures only where runtime storage requires it.
• RAII body wrappers observe end-of-stream or early drop to finish metrics and permits.

Newtypes are especially important: local versus remote addresses, server versus client TLS, logical versus concrete destinations, and inbound versus outbound labels are compile-time distinctions despite similar underlying data.

Generics and Tower Stacks

Linkerd is an unusually strong example of generics as architecture.

Service<Req> associates a response, error, and future with one request type. NewService<T> associates a constructed service with one target type. Layer<S> maps an inner service type to a wrapped service type. Param<P> proves a target can supply a layer’s configuration. Together, they make invalid stack wiring a compile error.

Builder methods return impl Layer<..., Service = ...> and each .push() changes the enclosing Inbound<S>/Outbound<S> type. The enormous final type is optimized into direct calls. Enums such as Either<A, B> retain static dispatch for closed runtime branches.

Type erasure is applied at pressure points: buffered services need a uniform queued request type; recursive/dynamic routing needs cached heterogeneous boundaries; public application structs sometimes hide futures. BoxService, BoxCloneSyncService, boxed bodies, and the shared linkerd_error::Error stop type growth where flexibility outweighs inlining.

The design does not choose “generics or trait objects.” It composes statically for the hot path, then erases at queues, recursive boundaries, and application storage.

Failures, Retries, and Drain

Errors retain causal chains

Most stack boundaries normalize to a boxed shared error while concrete marker types identify policy rejection, failfast, timeout, TLS, routing, and discovery causes. Helpers inspect the source chain for metrics and HTTP/gRPC response mapping without requiring one monolithic error enum.

Retry requires replayability and permission

An HTTP request body may be a stream that cannot be cloned. Linkerd buffers only up to a configured replay limit; exceeding it disables replay. Route policy classifies failures, retry budgets limit additional traffic, and timeouts bound the whole operation. A retry is therefore a conjunction: policy says retryable, body is replayable, budget is available, and deadline remains.

Reconnect is a state machine

Reconnect moves through disconnected, backoff, connecting, and connected service states. Readiness drives reconnection rather than spawning an unbounded retry loop.

Shutdown drains ownership

SIGTERM completes a drain signal watched by servers. Listeners stop admitting new connections; active connection/request tasks receive drain context and are given time to finish. Readiness, queues, bodies, and connection handles already encode outstanding ownership, so graceful shutdown follows the same boundaries as normal traffic rather than maintaining a separate request registry.

Build a Smaller Linkerd2-proxy

Build a target-specialized Tower proxy with dynamic endpoint discovery, bounded readiness, balancing, replay-safe retries, and graceful drain.

1. Proxy one concrete HTTP service

Accept TCP, serve HTTP, rewrite the destination, and forward with Hyper. Record connection and request ownership before adding generic layers.

2. Define NewService<T> and target parameters

Create Target { logical, original_dst, labels }, a service factory, and Param<P>. Construct a service once per target instead of consulting global maps in every call.

3. Build middleware layers

Implement timeout, request counting, response classification, and authorization as Tower services/layers. Keep their errors concrete until an outer response mapping boundary.

4. Add discovery and a cached router

Use a watch/change channel to add and remove endpoints. Cache one router per logical destination and evict it after idle time. Preserve its endpoint state across requests.

5. Implement readiness and bounded pressure

Give endpoints capacity flags, poll only ready endpoints, and place a bounded queue with a failfast deadline before the balancer. Demonstrate queue, reject, and recovery under a paused endpoint.

6. Balance by load

Choose two ready endpoints randomly and send to the lower observed load. Hold a load guard until response body completion/drop, not merely response headers.

7. Add safe retries

Buffer request bodies up to a cap, classify retryable responses, enforce a token budget and overall deadline, and prove a non-replayable streaming body is never retried.

8. Add TLS and live credentials

Distribute Arc<ClientConfig> through a watch channel. New connections use the latest config while established connections retain the config that negotiated them.

9. Drain

Broadcast shutdown, stop accepts, reject new routing work, and wait for active bodies/connections under a deadline. Test a stream that finishes and one that never does.

10. Compare with production

Map to inbound/outbound stacks, NewService, Param, Tower Service, queues, failfast, idle caches, routers, endpoint discovery, P2C pools, retry replay, classifiers, mTLS, metrics, and drain. The real proxy adds mesh protocols, policies, opaque TCP, gateways, tap, observability, and extensive control-plane recovery. Preserve its center: capacity-aware typed services whose dynamic state follows the target hierarchy.

Nushell: Orientation

Nushell treats pipelines as transformations of structured values while still interoperating with byte-oriented Unix processes. Its architectural center is PipelineData: empty, one materialized Value, a lazy ListStream, or a ByteStream that may own a child process.

source text → parse/type shape → block/calls
                                  ↓
EngineState + mutable Stack → Command::run(PipelineData) → PipelineData
                                  ↕
                     external child pipes / plugins / Rayon

Design thesis

Nushell keeps structured pipelines lazy and synchronous by default, then introduces threads, Rayon, and child processes only at boundaries whose ownership and ordering consequences users can see.

• PipelineData distinguishes repeatable values from consumable streams.
• Pull-based iterators provide natural pressure and early termination.
• par-each makes parallelism and its ordering tradeoff explicit.
• ByteStream keeps child output and process completion in one lifecycle.

We preserve the value/stream distinction, lazy iterator pipelines, metadata and source spans, structured errors, cooperative signals, external-process bridges, bounded parallel mapping, explicit ordering policy, and cloneable command objects.

Interactive Pipeline Map

Nushell is mostly synchronous and pull-based. It uses threads and processes at specific boundaries rather than placing the whole evaluator on an async runtime.

Why Is It Designed This Way?

Why separate Value from streams?

A value is repeatably observable and cloneable. A stream is consumable state. Putting streams inside cloneable values would require locks and make reading one alias change what another alias observes.

Why distinguish list streams from byte streams?

Structured commands need typed rows; external programs speak bytes and have stdin/stdout/exit semantics. One universal stream would either erase structure or pretend a child process is just an iterator.

Why use synchronous iterators?

Most pipeline stages do local CPU work and benefit from pull-based laziness. Iterators provide early stop and bounded memory with far less machinery than an async runtime, while blocking boundaries are handled separately.

Why pass EngineState and &mut Stack?

Global declarations/configuration are shared read-mostly; variables and environment changes are call-local mutations. The signature makes this ownership split visible to every command implementation.

Why is parallelism a separate par-each command?

Parallel closure evaluation can reorder output, amplify memory, and interact with side effects. Naming it lets users opt into those semantics and choose thread count and order preservation.

Why use a dedicated Rayon pool?

Other commands use Rayon’s global pool, and streaming parallel work may block on channels. Sharing workers can create starvation deadlocks. Nested parallel calls need a private pool for the same reason.

Why attach signals to streams?

Lazy computation happens during later consumption, far from the command that created it. Carrying the interrupt capability with the iterator ensures Ctrl-C still reaches the actual work.

Why wrap child processes in ByteStream?

Output bytes, stderr, exit status, foreground job control, and cleanup describe one lifecycle. Keeping them together prevents downstream commands from losing the process result while consuming its data.

One Structured Pipeline, Fully Traced

Consider open people.json | where active | get name | first 10.

1. Parsing produces calls and typed shapes

The parser resolves declarations from a working set, checks command signatures, and produces a block. Spans connect each expression and error to original source. Parser/type errors can accumulate before evaluation begins.

2. Evaluation separates global and local state

EngineState holds declarations, blocks, files, configuration, signals, and long-lived services. Stack holds variables, environment changes, redirection, and call-local state. eval_call borrows the engine, mutably borrows the stack, and moves pipeline input.

3. Commands receive one closed data enum

Command::run accepts PipelineData and returns Result<PipelineData, ShellError>. Each stage can preserve streaming, intentionally collect, or reject an incompatible input.

4. Iterator adapters keep rows lazy

ListStream owns a boxed Iterator<Item = Value> + Send. where, get, and first can wrap it with filter/map/take behavior. Upstream produces only when downstream calls next; first 10 naturally prevents parsing the remainder.

5. The sink chooses materialization

Display consumes values incrementally; assignment or a command needing a whole list calls into_value and collects. PipelineMetadata and spans travel beside the data so rendering and errors retain provenance.

Concurrency, Parallelism, and Processes

Ordinary pipelines are lazy, not parallel

Most builtin pipelines are synchronous iterator chains. Stages overlap only in the sense that downstream demand drives upstream computation on the same thread. This yields bounded memory and early termination without scheduling, channels, or async state machines.

External processes create operating-system concurrency

run-external connects a ByteStream directly to child stdin when possible. If structured data must be encoded, a named worker thread copies it into the pipe. Child stdout/stderr remain lazy byte readers; another thread waits for exit status so pipe consumption and process completion can progress together.

par-each opts into CPU parallelism

par-each uses dedicated Rayon pools. Top-level calls reuse pools by requested thread count. Nested calls create a private pool: sharing the outer pool could deadlock when a streaming consumer blocks while occupying a worker needed by its producer.

Unordered mode streams results through a bounded channel of 64 items. Ordered mode attaches indices, collects, and parallel-sorts before returning—making memory/latency the explicit price of deterministic order.

Plugins bridge streams with protocol messages

Plugin interfaces send a header describing empty/value/list/bytes, then stream messages on separate flow-control machinery. Background writers prevent a bidirectional plugin protocol from deadlocking the evaluator while it awaits a response that depends on consumed input.

This repository demonstrates that concurrency should be introduced per semantic need: lazy pull for pipelines, OS processes for tools, Rayon for independent closures, and channels/threads for duplex bridges.

The Standard Library as Architecture

• Iterator is the core streaming protocol for structured rows.
• Box<dyn Iterator<Item = Value> + Send> erases arbitrary lazy adapter chains.
• std::process::Command, Stdio, child handles, and pipes implement Unix interoperability.
• Arc<AtomicBool> broadcasts interruption cheaply to streams and workers.
• mpsc::sync_channel bounds parallel result buffering; ordinary mpsc carries child status.
• Arc<Mutex<_>> protects rare shared job/status and cached-pool state.
• HashMap powers scopes, declarations, records, environment overlays, and pool caches.
• Cow<'a, str> avoids allocating borrowed command/signature text until ownership is needed.
• Span { start, end } is small provenance carried through values and errors.

Moving PipelineData between commands is important. A stream has one consumer; cloning it would create surprising alias-dependent observation. Materialized Values remain cloneable and safe to share.

Generics, Trait Objects, and Values

The command boundary uses dynamic dispatch because declarations are discovered by name at runtime. EngineState stores cloneable Box<dyn Command> values; CommandClone provides object-safe cloning. Each command advertises a concrete Signature, documentation, examples, and a common run method.

Inside commands, generics recover static composition. Iterator adapters are generic until ListStream::new erases them. Evaluation is generic over a DebugContext marker so normal and debug execution share code without a runtime branch at every operation. Conversion traits such as IntoPipelineData, IntoValue, and typed CallExt flag access connect Rust types to shell values.

Closed runtime diversity uses enums: Value, PipelineData, ShellError, OutDest, expressions, and byte-stream sources. Exhaustive matches are valuable because adding a new data mode must force every core boundary to consider it.

The pattern is layered: trait objects for a runtime registry, enums for the language’s closed data model, generics for implementation helpers, and boxed iterators exactly where arbitrary lazy pipelines must share one representation.

Errors, Interruption, and Process Lifecycle

Errors are language values with provenance

ShellError variants carry spans, labels, help, and nested sources. Commands return operational errors, while a Value::Error may travel inside a stream so lazy failures appear at the point of consumption. Collecting a ListStream unwraps the first error unless a debug path intentionally preserves it.

Signals are cooperative

Signals wraps an optional Arc<AtomicBool>. ListStream checks before every item; long-running algorithms check periodically; external job control can signal or kill processes. This is cancellation without async-future destruction.

Child output and exit are one resource

ByteStreamSource::Child retains stdout/stderr and shared exit-status state. Converting or draining it consumes pipes, waits for status, runs job-control callbacks, and turns nonzero exit into ShellError unless explicitly ignored. The shell cannot report success merely because stdout reached EOF.

Duplex pipes require concurrent draining

When both stdout and stderr are captured, Nushell consumes one on a helper thread to avoid the classic deadlock where a child blocks on a full pipe while the parent waits on the other. Plugin streams similarly run background writers and propagate connection loss to stream readers.

Build a Smaller Nushell

Build a structured shell whose lazy values interoperate with child-process byte streams and whose parallel map makes ordering and bounds explicit.

1. Define values and spans

Support int, string, record, list, nothing, and error. Give every value a source span. Implement field access and display without parsing source strings again.

2. Parse a concrete pipeline

Parse command names, flags, literals, and | into calls. Give commands signatures with accepted input/output types and report multiple spanned parse errors before execution.

3. Add PipelineData

Implement empty, value, list stream, and byte stream. Back list streams with Box<dyn Iterator<Item = Value> + Send>. Write lazy where, get, and first, plus an explicit collect.

4. Build the command registry

Define a cloneable object-safe Command with signature and run. Separate immutable EngineState from mutable Stack. Move input and output so a stream has one clear consumer.

5. Run external processes

Pipe a byte stream directly into stdin. Encode structured rows on a helper thread when necessary. Return a child-backed byte stream that owns stdout, stderr, an exit-status receiver, and cleanup.

6. Prevent pipe deadlocks

Capture large stdout and stderr simultaneously. Drain them concurrently and wait for exit only after both make progress. Turn nonzero status into a spanned shell error.

7. Add cooperative Ctrl-C

Share an atomic signal. Wrap list iterators and check it per item; check inside long loops; terminate foreground children. Verify an infinite source stops.

8. Add par-each

Use a dedicated fixed Rayon pool and bounded result channel. Stream unordered results first. Add --keep-order with indices and documented collection cost. Test nested parallel calls with a private inner pool.

9. Compare with production

Map to Value, PipelineData, ListStream, ByteStream, ChildProcess, Signals, Command, EngineState, Stack, evaluator, plugin streams, and par-each. Nushell adds a rich language, IR, job control, custom values, plugins, formats, table rendering, completion, and cross-platform process behavior. Preserve its center: structured values are repeatable; streams and processes are owned, lazy lifecycles.

Apalis: Orientation

Apalis turns an ordinary async function into the center of a durable job worker. Around that small function it composes a backend, typed extraction, Tower middleware, readiness, bounded concurrency, acknowledgement, retry, events, and cooperative shutdown.

async fn send_email(job: Email, state: Data<Arc<App>>) -> Result<(), MailError> {
    state.mailer.send(job).await
}

let worker = WorkerBuilder::new("email")
    .backend(storage)
    .data(app)
    .concurrency(16)
    .parallelize(tokio::spawn)
    .build(send_email);

The architectural center is not the syntax of send_email. It is this admission rule:

A backend may release the next durable task only when the composed service is ready to own it.

That connects storage semantics to Tower backpressure without making either side implement the other.

Design thesis

Apalis keeps application handlers ordinary while concentrating durable job policy in typed backend and Tower middleware boundaries.

• Payloads move into handlers so their futures can run independently.
• Service readiness is checked before durable work is claimed.
• Backends own encoding, polling, locking, and acknowledgement semantics.
• Generics describe the hot path; type erasure appears at heterogeneous edges.

We will preserve ordinary async handlers, owned task payloads, typed context extraction, backend-independent execution, readiness before dequeue, middleware-defined policy, tracked completion, and graceful drain.

Interactive Worker Map

Use the four traces to compare the happy path with the lifecycle obligations that surround it. In particular, notice that concurrency is enforced through service readiness before Backend::poll_next, not by fetching an unbounded batch and hoping the executor catches up.

Why Is It Designed This Way?

1. Why are handlers ordinary async functions?

They already have the essential shape: arguments in, future out. TaskFn adapts that shape to Tower Service, FromRequest constructs typed context arguments, and IntoResponse normalizes results. The caller writes application logic without a macro, base class, or runtime-specific handler trait.

2. What does the user have to own, borrow, or clone?

The worker owns its backend and handler. A job payload is moved into the handler. Extractors borrow the Task only while constructing owned arguments. Shared application data must be cloneable and is commonly an Arc<T>; mutable shared state needs an explicit lock, channel, or pool. Retry additionally requires a cloneable request because it must own another attempt.

3. Why separate storage/backend concerns from worker execution?

Durable queues differ in encoding, IDs, locks, acknowledgement, polling, and maintenance. Execution differs in readiness, concurrency, middleware, tracking, events, and shutdown. Backend connects the two through associated types and poll methods, while Backend::Layer inserts storage-specific lifecycle policy. Either side can evolve without copying the other.

4. How does the builder make invalid configurations harder to create?

WorkerBuilder changes type as the backend and layers are added. build is available only when backend types, task types, handler extraction, and the final Tower service agree. Missing or incompatible pieces become trait-bound errors instead of latent runtime branches.

5. Which types and traits appear publicly?

The common surface is WorkerBuilder, Worker, Monitor, Backend, Task, WorkerContext, Data, FromRequest, IntoResponse, TaskSink, retry/error types, and Tower Service/Layer. Backend capability traits expose optional operations without forcing every backend into one enormous interface.

6. How are job errors, retries, shutdown, and middleware represented?

Errors are typed at their source and erased only at composition boundaries. Retry, limits, timeouts, tracing, panic catching, and acknowledgement are Tower layers. Shutdown is a cloneable future and admission state. Tracking and Backend::poll_close turn shutdown into a drain protocol rather than a boolean checked only at loop exit.

7. Where does Apalis use generics versus trait objects?

Generics describe the hot, homogeneous pipeline: backend, codec, task, handler, extractors, service, layer stack, executor, and policy. Trait objects appear where runtime heterogeneity is intentional: boxed errors, listeners, monitor worker factories/futures, and selected erased streams. Static by default, dynamic at collection and failure boundaries.

8. How does the simplest caller experience compare with the internals?

The caller sees an async function and a short builder chain. Internally that function becomes a generic Tower service whose input is decoded from a backend, augmented by typed extensions, guarded by readiness, executed among unordered in-flight futures, observed by policy layers, acknowledged durably, tracked through cancellation, and drained by a monitor.

That asymmetry is the design achievement. The complexity was not deleted; it was concentrated behind contracts that let the ordinary path remain ordinary.

One Job, Fully Traced

1. The backend advertises readiness

CallAllUnordered first polls futures already in flight. It then asks the backend and the composed Tower service whether they are ready. Only after both say yes does it call Backend::poll_next.

This order matters. A concurrency-limit layer can return Pending, so the worker does not claim another durable job until it has capacity to own it.

2. Compact storage becomes typed input

The backend yields a Task<Compact, Connection, Id>. Its associated Codec decodes the compact representation into Args. Storage chooses identifiers, connections, serialization, queue semantics, and backend middleware; the worker does not know whether the task came from memory, Redis, SQL, or another broker.

3. The task enters the service stack

The worker composes worker context, user layers, backend middleware, readiness, and tracking around the handler service. TaskFn adapts the function to Service<Task<...>>.

For this handler:

async fn send(job: Email, state: Data<Arc<App>>, attempt: Attempt) -> Result<(), Error>

the payload Email is moved into job. Each additional argument implements FromRequest<Task<...>> and is produced from &Task before the function is called. The resulting values are owned across .await; no borrow of the worker or backend is held by the handler.

4. The future is polled

The function returns a future. parallelize(executor) can hand that future to an injected executor such as tokio::spawn; the core is not coupled to Tokio. The worker keeps the resulting service futures in FuturesUnordered, so completions are handled in completion order.

5. Completion becomes lifecycle state

IntoResponse normalizes the handler output. Tracking updates attempt and in-flight state. Backend-provided acknowledgement middleware observes the result and performs persistence-specific completion work. Its acknowledgement future is tracked too, so graceful shutdown cannot report completion while an acknowledgement is still pending.

The deepest invariant is:

capacity → dequeue → decode → extract → execute → acknowledge

Every arrow transfers responsibility; none is merely a convenient call order.

Async, Concurrency, and Backpressure

Apalis uses async for waiting on brokers and handlers, concurrency for multiple in-flight jobs, and an executor only when the application explicitly supplies one.

Ordinary async functions are already the right shape

An async function is a value implementing FnMut(Args...) -> Future. TaskFn supplies Tower’s poll_ready and call, the generated implementations extract up to sixteen typed arguments, and IntoResponse normalizes the output. No handler macro or runtime-owned callback interface is required.

There are two separate concurrency mechanisms

CallAllUnordered can retain many service futures in a FuturesUnordered. Actual admission is controlled by readiness. .concurrency(n) adds Tower’s ConcurrencyLimitLayer; when all permits are held, poll_ready becomes Pending, and the worker stops polling the backend for another task.

.parallelize(tokio::spawn) solves a different problem. Its generic executor turns a handler future into an independently scheduled future. Apalis accepts a function with the necessary input/output shape rather than naming Tokio in its core types.

backend ready? ─┐
service ready? ─┴─ yes → poll one job → push future ─┐
                                                     ├→ FuturesUnordered
shutdown? ───────── yes → stop admission → drain ───┘

Concurrency is not parallelism

Many network-bound jobs can make progress on one executor thread. They become parallel only if the executor runs them simultaneously on multiple threads. CPU-bound or blocking work still needs an appropriate blocking pool; merely making its function async does not make it cooperative.

Cancellation is resource accounting

WorkerContext::track wraps important futures. Its pinned drop logic decrements the task count even when a future is cancelled or dropped. Shutdown therefore tracks ownership, not just successful return paths.

Shutdown is a change in admission policy

Shutdown is a cloneable future backed by shared atomic state and a waker. Once signalled, readiness refuses new work. CallAllUnordered drains existing futures, then calls Backend::poll_close so buffered acknowledgements or broker resources can flush. Monitor coordinates several workers and may impose a final deadline.

The Standard Library and Ownership

The public ergonomics depend on ordinary Rust ownership rules being made visible at a few deliberate boundaries.

What the caller owns

• The backend value is moved into WorkerBuilder::backend, then into the worker.
• The handler function or service is moved into build.
• Each job’s Args is moved into the handler’s first argument.
• Values passed through .data(value) must be Clone + Send + Sync + 'static. The layer clones the value into task extensions; large shared state is usually Arc<State>.
• Extractors inspect &Task briefly, but return owned values for the handler.
• WorkerContext is cheap to clone because its shared internals use Arc.
• Retry policies clone Task, so retryable payload, connection, and identifier types must satisfy the corresponding Clone bounds.

The user normally does not lend &mut application state across .await. If shared mutation is necessary, the state explicitly contains a concurrency primitive such as Arc<Mutex<T>>, a pool, or a channel sender.

Small standard-library tools with architectural weight

Arc lets cloned tasks and contexts share execution metadata. Atomics represent monotonic shutdown, status, and in-flight counters without one giant lock. Mutex<Option<Waker>> stores the wakeup needed to turn shutdown state into a Future. PhantomData lets WorkerBuilder carry its not-yet-runtime type state without storing dummy payloads. Pin and pinned drop make task accounting safe when futures are cancelled.

Clone is a policy, not a convenience

Cloning an Arc<App> shares identity cheaply. Cloning a Task for retry means replaying its payload is semantically permitted. Apalis puts that distinction in trait bounds: a non-cloneable task can execute once, but cannot silently enter a retry mechanism that requires a second owned request.

Generics, Trait Objects, and the Public API

Apalis keeps the job hot path statically typed and erases types mainly where heterogeneous values must coexist.

Public types and traits readers meet

• Construction and execution: WorkerBuilder, Worker, Monitor, WorkerContext, and Shutdown.
• Job model: Task, TaskBuilder, ExecutionContext, TaskId, Attempt, Status, Data<T>, Metadata, and Extensions.
• Backend contract: Backend, TaskSink, and capability traits such as FetchById, Update, Reschedule, Vacuum, and RegisterWorker.
• Handler contract: TaskFn, FromRequest, and IntoResponse.
• Composition: Tower’s Service, Layer, Stack, and policy layers for retry, limits, timeout, tracing, and filtering.
• Failure/lifecycle: BoxDynError, AbortError, RetryAfterError, WorkerError, retry policies, and worker Event.

The prelude makes the common subset feel small; the individual modules retain the precise vocabulary for extension authors.

Where generics are used

Backend uses associated types for Args, Id, Connection, Codec, compact storage form, error, and middleware layer. Task<Args, Connection, Id> carries those choices forward. The handler, extractor tuple, executor, Tower service, layer stack, retry policy, codec, and acknowledgement service are generic too.

This gives direct calls and compile-time compatibility checks in the path that runs for every job.

The builder is a type-level state machine

It starts as:

WorkerBuilder<(), (), (), Identity>

.backend(storage) is available in that initial state and returns a builder whose Args, Connection, and Source come from storage: Backend. Every .layer(...) changes the middleware type to Stack<New, Old>. .build(handler) exists only when the backend and handler form a valid Service<Task<Args, Connection, Backend::Id>>.

Consequences:

• a missing backend cannot produce a runnable worker;
• the payload type cannot disagree with the backend;
• unavailable handler extractors fail compilation;
• incompatible middleware cannot be assembled;
• the final service order is encoded in the nested type.

This does not prevent semantic mistakes such as a bad retry count, but it makes structurally invalid configurations hard to represent.

Where trait objects appear

BoxDynError erases many error types at integration boundaries. Event listeners and monitor-owned worker factories/futures are boxed so different concrete workers can share one collection. Some stream/sink and future boundaries are boxed for the same reason.

The rule is practical: use generics where one concrete pipeline is known at build time; erase types where runtime heterogeneity is the feature.

Errors, Retries, Acknowledgement, and Shutdown

Apalis does not represent every failure as one undifferentiated Error.

Error layers

Handler failures flow through the service response and can be erased to BoxDynError where middleware needs one common type. AbortError means the failure should not be retried. RetryAfterError carries a delay request. CallAllError distinguishes backend polling, codec decoding, and service execution. Non-exhaustive WorkerError covers worker-level failures such as polling, readiness, codec, I/O, panic, state, and graceful exit.

That separation preserves the decision each layer must make: retry the job, report the worker, close the backend, or terminate the monitor.

Retry is middleware

RetryPolicy, BackoffRetryPolicy, and RetryIfPolicy implement Tower retry policy around the handler service. They can inspect the task and result, clone a request for another attempt, and schedule an immediate or delayed retry. A successful, killed, explicitly aborted, exhausted, or shutting-down task does not retry.

Because retry is a layer, placement matters. Tracing outside retry sees one logical job; tracing inside sees each attempt. The builder’s nested Stack makes that order concrete.

Acknowledgement belongs to the backend boundary

An Acknowledge layer awaits the handler result and then invokes backend-specific acknowledgement with the execution context. That operation may mark success, store failure, release a lock, or reschedule. The worker knows when acknowledgement belongs; the backend knows what durable transition it means.

The acknowledgement future is tracked as part of worker completion. Returning from the handler is therefore not confused with durably recording its outcome.

Graceful shutdown is staged

1. Shutdown is signalled and readiness stops admitting jobs.
2. Already claimed service futures keep running.
3. Tracking waits for handler and acknowledgement obligations to leave.
4. Backend::poll_close flushes or releases backend resources.
5. Monitor joins workers, applies restart policy where configured, reports errors, and can enforce a shutdown timeout.

An optional panic-catching layer converts a handler panic into an abort-style error. The worker monitor also contains panic at its worker-future boundary. These boundaries improve availability, but they do not make panic a normal domain error.

Build a Smaller Apalis

Rebuild the architectural center, not the spelling of the builder API.

What are we preserving?

• an ordinary async handler;
• an owned task payload plus typed shared data;
• a backend independent from execution;
• readiness before dequeue;
• bounded unordered execution;
• result-aware acknowledgement;
• retry as composable policy;
• shutdown that stops admission and drains ownership.

Stage 1: one task and one handler

Define Task<A, Id> and adapt FnMut(A) -> Future into a tiny service with ready and call. Move A into the future. Do not add persistence yet.

Stage 2: a backend contract

Define associated types for payload, compact form, ID, error, and codec. Add poll_ready, poll_next, and poll_close. Implement an in-memory backend with VecDeque and a waker.

Stage 3: the admission loop

Use FuturesUnordered. Poll completed jobs, then backend readiness, then service readiness, and only then dequeue one task. Add a test proving the backend is not polled while the service has no permit.

Stage 4: typed extraction

Add a task extension map keyed by TypeId, a Data<T> extractor, and a two-argument function adapter. Observe why extraction returns owned values rather than references held across .await.

Stage 5: failure and acknowledgement

Return Result from handlers. Wrap execution with an acknowledgement service that records success or failure only after the inner future resolves. Add an abort error and a retry policy that must clone the task.

Stage 6: bounded concurrency and executor injection

Implement readiness with a semaphore permit. Separately accept a generic spawn function. Demonstrate that admission limits and task scheduling solve different problems.

Stage 7: graceful drain

Signal shutdown, make readiness pending for new jobs, drain in-flight handlers and acknowledgements, then close the backend. Add a cancellation test proving a dropped tracked future decrements the count.

Compare with production Apalis

Your reconstruction now contains the same architectural argument. Production Apalis adds many backends and codecs, generated extractor arities, Tower interoperability, richer retry/backoff, worker events and restart policies, status metadata, observability, panic containment, feature gating, and backend maintenance capabilities.

The final exercise is diagnostic: replace the in-memory backend without editing the worker loop, then add middleware without editing the handler. If both work, you preserved the boundaries rather than merely imitating the surface syntax.

Watchexec: Orientation

Watchexec converts noisy operating-system events into deliberate child-process transitions. Its center is not filesystem watching alone: it is the policy boundary between event collection and a separately owned process state machine.

notify callback / signal / keyboard → priority event channel
                                      ↓ filter + throttle
                                 ActionHandler
                                      ↓ job controls
                              supervised child process

Design thesis

Watchexec turns unreliable bursts of observations into serialized policy decisions, while each child process is owned by an independent supervisor task that accepts explicit, prioritized controls.

• Bounded event admission makes overload observable.
• Debouncing coalesces implementation noise into one user-level action.
• Action handlers propose transitions and commit them by returning.
• Process groups, signals, timers, and reaping stay inside job supervision.

We preserve prioritized events, filtering, trailing-edge collection, action snapshots, job handles, busy-update policy, graceful signal escalation, error severity, and structured task shutdown.

Interactive Watch-and-Restart Map

Why Is It Designed This Way?

Why debounce after filtering?

Rejected paths should not consume the action window. The resulting batch describes relevant user work rather than raw watcher implementation noise.

Why do urgent events bypass filtering and throttling?

Signals are lifecycle control, not ordinary file observations. Ctrl-C must remain responsive even during a large event burst or an overly broad filter.

Why is the action handler serial?

Restart, queue, quit, and signal decisions conflict when applied concurrently. One ordered policy point makes the resulting process transition explainable.

Why does the handler return itself?

The return is a commit boundary. It separates a snapshot-based policy decision from mutation of the runtime’s job registry and shutdown state.

Why does every job have its own task?

Waiting for a child, receiving controls, and running a grace timer must progress concurrently with new event collection. Single ownership also removes locks from CommandState transitions.

Why use three control queues?

Graceful stopping intentionally suppresses stale normal work while preserving escalation and administrative control. Priority is part of process safety, not just throughput tuning.

Why distinguish runtime and critical errors?

A transient watcher or process error may be reported and tolerated. Loss of a core worker or an explicit upgrade changes whether the whole runtime can still honor its contract.

Why is configuration dynamically replaceable?

Watchexec is both a CLI and an embeddable long-lived runtime. Consumers can change handlers, paths, filters, and policy without reconstructing all source and supervisor tasks.

One File Change, Fully Traced

1. A callback crosses into async code

notify invokes a synchronous platform callback. The filesystem worker normalizes its result into Watchexec Event tags and uses the priority channel; the callback never owns the action loop or child process.

2. Admission, filtering, and coalescing

throttle_collect waits for the first accepted event, starts a window, and collects subsequent events. Normal events pass through Filterer; empty synthetic and urgent events bypass it. An urgent signal also bypasses the remaining throttle so Ctrl-C is not trapped behind filesystem noise.

3. Policy receives an owned snapshot

The collected Vec<Event> becomes Arc<[Event]>. ActionHandler receives that immutable batch plus cloned handles for the jobs known at invocation. Creating jobs or requesting quit records intent in the returned handler; the action worker incorporates it only after the callback returns.

4. A job receives controls

The CLI gets or creates one stable job, attaches a spawn hook for working directory and event-derived environment, then applies OnBusyUpdate: start, ignore, signal, restart, or queue one later run.

5. The supervisor mutates the process

Only the job task owns CommandState. It selects between child completion and prioritized control messages, spawns the command, signals or kills its process group, waits for exit, and publishes completion flags. The action loop owns policy; the supervisor owns process correctness.

Async, Concurrency, and Priorities

Watchexec::with_config creates one bounded priority event channel and spawns action, filesystem, signal, keyboard, and error workers in a Tokio JoinSet. A Notify start lock allows construction and configuration before those workers begin.

The action worker is deliberately serial: one event batch becomes one policy decision at a time. An async action handler is awaited in that loop, so a slow handler delays event consumption. The API documents this rather than implying unbounded handler concurrency is safe.

Job tasks provide the independent concurrency. Each owns one process state machine and receives controls through normal, high, and urgent queues. During a graceful-stop timer, normal controls are excluded while high/urgent controls and timer expiry remain selectable.

concurrent producers → bounded priority channel → serial decisions
                                                   ↓
                                  independent job supervisor tasks

This separates concurrency from ordering: sources may race, policy is ordered, and unrelated children can progress independently. LateJoinSet additionally keeps spawned job tasks attached to shutdown even if they finish before the owner begins joining them.

The Standard Library as Architecture

• Arc<Config> shares live configuration without transferring ownership to every worker.
• Arc<[Event]> is an immutable, cheaply cloned action snapshot.
• HashMap<Id, Job> owns the supervisor registry; Id avoids exposing task internals as identity.
• AtomicBool represents simple cross-task policy such as paused, queued, and running, while compound process state remains single-owner data.
• OnceLock<CriticalError> lets an error hook upgrade one runtime error exactly once without returning the hook object.
• Duration and Instant make debounce and escalation deadlines monotonic.

The crucial borrowing rule is that async workers own their inputs. A handler gets cloned job handles and an Arc event batch, not references into the action worker’s mutable registry. A spawn hook temporarily borrows the command before process creation; it cannot retain that mutable borrow across the process lifetime.

Generics, Traits, and Erased Hooks

The reusable library exposes concrete orchestration types—Watchexec, Config, ActionHandler, Event, Job, Command, Signal, Priority—and a small number of extension contracts.

Filterer is a trait object because applications select filtering policy at runtime. ChangeableFn<T, U> stores Arc<dyn Fn(T) -> U + Send + Sync> so handlers can be replaced while the runtime is alive. Async action handlers return boxed futures because each closure’s anonymous future type must fit that replaceable slot. Notify watchers are boxed for the same native-versus-polling choice.

Generics remain where one implementation is known locally: channel payloads, JoinSet tasks, helper futures, and callback construction. Process hooks use trait objects at the user-extensible boundary; CommandState stays a concrete enum in the hot ownership loop.

Watchexec is a useful counterpoint to Apalis and Axum: dynamic configuration is a product feature here, so more type erasure is appropriate than in a pipeline fully assembled at compile time.

Errors, Process Cleanup, and Shutdown

RuntimeError describes recoverable operating failures from sources, filtering, process operations, and full channels. The bounded error channel feeds a configurable hook. That hook may place a CriticalError into a one-assignment cell, explicitly upgrading severity.

Critical failure belongs to the worker set. A graceful exit closes the event channel; unexpected critical errors terminate the main task. When the action worker ends, the main task shuts down the remaining JoinSet workers.

Graceful process shutdown is a protocol:

1. send the configured signal to the process group;
2. install a monotonic grace timer;
3. continue admitting urgent/high controls;
4. convert timer expiry into forceful Stop or restart;
5. await and reap the child;
6. raise completion flags and delete the job.

The CLI escalates repeated quit requests from graceful signal, to immediate force-stop, to abort. Process completion is converted back into an event, so --exit-on-error and queued reruns re-enter the same serialized policy path.

Build a Smaller Watchexec

Rebuild the event-to-process contract, not a thin wrapper around notify.

What are we preserving?

• concurrent event producers and bounded admission;
• filtering plus trailing-edge debounce;
• urgent lifecycle priority;
• one serialized action policy;
• a task-owned child state machine;
• start, signal, restart, queued run, and graceful stop;
• explicit recoverable-versus-fatal errors;
• joined shutdown.

Reconstruction stages

1. Model Event, Priority, and a bounded input channel; use synthetic events.
2. Implement filter-then-debounce and test that urgent events flush immediately.
3. Give an action callback an immutable event batch and return an Outcome.
4. Build one supervisor task around tokio::process::Command and a control enum.
5. Add busy policies: ignore, signal, restart, or one coalesced queued run.
6. Add signal-then-deadline-then-kill, always awaiting the child afterward.
7. Add multiple jobs, runtime error reporting, and structured task joining.
8. Only then connect notify and platform process groups.

Production Watchexec still supplies portable watchers, ignore-file discovery, keyboard and OS signal sources, rich event tags, dynamic configuration, environment emission, process wrapping, shell interpretation, diagnostics, and many platform-specific failure paths.

sccache: Orientation

sccache is a local client/server compiler wrapper that turns a compiler invocation into a content-addressed lookup, falling back to local or distributed execution and asynchronously publishing successful artifacts.

build tool → short-lived client → resident server → compiler analysis + hash
                                               ↓
                                    cache hit or compilation
                                               ↓
                                  outputs now + cache write future

Design thesis

sccache separates the latency-sensitive compiler response from reusable cache publication, while a resident server shares compiler knowledge, storage, and a machine-wide execution budget across otherwise independent build clients.

• Compiler-specific parsing produces a common cacheable computation.
• Content identity, not invocation order, names reusable work.
• Trait objects select compilers and storage backends at runtime.
• Tokio overlaps IPC and remote I/O; blocking filesystem work and child processes cross explicit boundaries.
• A jobserver token limits compiler processes independently of connection count.

Interactive Compilation Map

Why Is It Designed This Way?

Why use a resident local server?

Short-lived compiler wrappers can share discovered compiler state, statistics, remote clients, storage connections, and a machine-wide process budget without requiring every build tool to embed them.

Why hash preprocessed inputs instead of command text alone?

The output depends on headers, environment, compiler identity, flags, and other implicit inputs. A fast but incomplete key would return incorrect artifacts.

Why separate compiler parsing from hashing and execution?

Recognition chooses the family; parsed invocation state then owns the expensive async path. Unsupported invocations can fall through without contaminating the shared server architecture with compiler-specific branches.

Why return a cache-write future with a miss?

The compiler result is latency-sensitive; making it reusable is a related background obligation. Carrying the future preserves tracking without forcing artifact upload into the compiler’s critical response path.

Why have a jobserver in addition to Tokio?

Tokio schedules futures, not CPU budgets consumed by rustc subprocesses that may themselves use threads. A token models the scarce external resource.

Why use trait objects for storage and compilers?

Their concrete types are selected from runtime configuration and executable discovery. Static monomorphization cannot place unrelated implementations in the same server maps without an enum that must change for every extension.

Why is shutdown ownership-based?

An incremented counter can miss an error path. Strong ActiveInfo ownership already exactly describes live services; the weak waiter derives completion from that fact.

One Compilation, Fully Traced

The wrapper connects to the resident server and sends Request::Compile with the executable, arguments, environment, and working directory. Each connection is a Tokio task, while SccacheService shares statistics, compiler discovery, storage, distributed state, and command creation through clones.

The server resolves the real compiler and caches a boxed Compiler. Argument parsing returns a boxed CompilerHasher: the compiler family decides whether the invocation is cacheable and what inputs influence identity.

generate_hash_key preprocesses or inspects dependencies, normalizes relevant paths and environment, and produces a key plus a prepared compilation. Storage then yields hit, miss, or error. A hit restores outputs and returns captured stdout/stderr without running the compiler.

On a miss, sccache acquires a jobserver token before starting expensive work. It may package and dispatch remotely or spawn locally. The process exit status decides whether outputs are eligible for caching.

CompileResult::CacheMiss contains both the compiler output and a boxed cache write future. The client-facing result can proceed while the server finishes artifact publication and updates statistics. The response and durable reuse point are related obligations, not one blocking step.

Async, Concurrency, and Resource Budgets

The server accepts connections continuously and spawns one Tokio task per connection. That is request concurrency, not permission to launch unlimited compilers.

The internal jobserver supplies the CPU/process budget. acquire() bridges the blocking jobserver helper thread to async Rust with a request channel and a per-acquisition oneshot; the returned Acquired value owns the token until drop. This is the same ownership pattern as the toy job runner, but here the permit represents machine compilation capacity rather than a typed result.

Remote storage and distributed scheduling are awaited normally. Disk archive I/O, dependency inspection, compression, and other synchronous library work use spawn_blocking so they do not occupy cooperative Tokio workers.

Multi-level storage introduces controlled fanout: levels are queried according to policy, slower hits may backfill faster levels, and write-error policy says which failures affect the compiler result. Concurrency is therefore split by resource and consequence rather than governed by one global semaphore.

Shutdown stops acceptance through explicit signal, RPC, or idle timeout. A WaitUntilZero future retains only a Weak pointer and resolves when all connection-owned ActiveInfo clones drop, with a ten-second outer deadline.

The Standard Library as Architecture

Arc<SccacheService> lets connection tasks share storage, compiler discovery, statistics, and execution infrastructure. Mutex protects asynchronously updated statistics; RwLock favors repeated compiler-cache reads over rare replacement after an executable changes.

HashMap<PathBuf, Option<CompilerCacheEntry<C>>> records both discovered compilers and negative discovery. Box<dyn Compiler<C>> gives each entry a runtime-selected compiler family, while its box_clone method makes the erased value explicitly cloneable.

Temporary directories own preprocessing and compilation intermediates. Their Drop cleanup is part of correctness: a future may fail or be cancelled at many awaits without leaving every partial file behind.

WaitUntilZero is a small ownership proof. Every connected service owns an ActiveInfo clone. The waiter keeps only Weak; when the last strong owner drops, Info::drop wakes the shutdown future. No separate connection counter can drift away from actual ownership.

The sharpest stdlib boundary is unsafe: inherited GNU jobserver file descriptors may keep a parent build alive, so sccache detects and closes them. The unsafety is isolated because closing the wrong raw descriptor violates process-wide invariants.

Generics, Trait Objects, and Compiler Families

The public binary has a dynamic problem: at runtime an executable may be rustc, Clang, GCC, MSVC, CUDA, or a proxy. Compiler<C>, CompilerProxy<C>, CompilerHasher<C>, and Storage are therefore trait-object boundaries. Collections can hold heterogeneous implementations and configuration can pick disk, Redis, S3, or multi-level storage.

The command creator C remains generic. Production uses real processes; tests substitute a deterministic mock without adding dynamic dispatch to every method. Associated behavior is erased only where heterogeneity is required.

Compiler::parse_arguments separates cheap compiler-family recognition from a per-invocation CompilerHasher. That hasher owns parsed arguments and can perform async dependency discovery, key generation, lookup, and fallback compilation.

CompileResult is an enum because callers must exhaustively distinguish hit, miss, uncacheable input, compiler failure, and infrastructure error. Its cache miss variant owns a boxed future for cache publication: only that future’s concrete implementation is irrelevant to the response layer.

The pattern is:

Generics preserve testable families; trait objects represent runtime plugin choices; enums preserve a closed set of product outcomes.

Errors, Cache Integrity, and Shutdown

A cache miss is not an error. MissType records normal absence, forced bypass, timeout, and read failure so statistics and policy retain the distinction. Compiler failure is also a valid completed result whose stdout, stderr, and exit status must reach the build tool.

Cache writes occur only after successful compilation and output collection. Entries are assembled through controlled archive and temporary-file paths; backend-specific publication decides when a key becomes visible. Multi-level backfill and write-error policy determine whether failure at a slower level should fail, warn, or be ignored.

Blocking filesystem and archive operations run through spawn_blocking; join failure is contextualized separately from the I/O error returned by the closure. Distributed compilation may fall back locally when policy permits, but authentication, packaging, compiler, and cache failures remain distinct.

Server shutdown has two phases. Explicit signal, shutdown RPC, or idle timeout stops accepting connections. Then WaitUntilZero observes connection ownership drain, capped by a ten-second deadline. The process does not pretend that closing the listener completed active compilations, nor does it surrender liveness indefinitely.

Build a Smaller sccache

Rebuild a concurrent command cache, not merely a HashMap<String, Vec<u8>>.

What are we preserving?

• a short-lived client and resident async server;
• deterministic keys over declared and discovered inputs;
• hit versus compile fallback;
• a separate child-process budget;
• captured output and exit status;
• publication after successful completion;
• runtime-selected storage;
• graceful connection drain.

Stages

1. Cache one fake compiler whose key includes source bytes and arguments.
2. Put the cache behind a local Tokio socket protocol.
3. Execute a real child command and capture declared output files.
4. Add a semaphore-like process permit independent of connections.
5. Implement disk storage through a Storage trait and atomic publication.
6. Return output before awaiting a deliberately slow secondary cache write.
7. Stop listener admission, drain connection tasks, then enforce a deadline.
8. Add simultaneous same-key requests and decide whether to deduplicate them.

Compare the result with production sccache: compiler argument semantics, preprocessing, dependency normalization, archives, many storage systems, distributed toolchains, authentication, multi-level backfill, jobserver interop, platform process handling, statistics, and corruption recovery are the hardening layers still missing.

Atuin: Orientation

Atuin records rich shell history locally in SQLite and optionally synchronizes encrypted records across machines. Its current V2 protocol is a generic record store used by history, key/value data, aliases, variables, and scripts.

local domain change → encrypted tagged record(host, index) → SQLite/WAL
                                      ↕ diff + pages
                              untrusted sync server
                                      ↓
                       rebuild local domain projections

Design thesis

Atuin treats the encrypted append-oriented record log as synchronization truth and local feature databases as rebuildable projections, so every machine remains useful offline and the server never needs plaintext.

• Per-host, per-tag indices turn sync into tail comparison.
• Typed builders make incomplete records and history entries harder to create.
• Envelope encryption binds metadata to ciphertext before upload.
• Sync is deliberately ordered today; correctness precedes parallel transfer.
• Projection failures are isolated so one feature does not block the others.

Interactive Offline-Sync Map

Why Is It Designed This Way?

Why keep a separate record store and history database?

The encrypted record log provides generic replication truth; the history database provides query-optimized local UX. Projections can change or rebuild without redesigning the wire protocol.

Why index independently by host and tag?

Offline machines can append without coordinating a global sequence. Each origin owns its monotonic tail, and unioning streams preserves all writes.

Why validate the key before sync?

Discovery of a wrong key after uploads could extend divergence with records this installation cannot read. Preflight keeps failure non-mutating.

Why download from the first gap?

Maximum index proves only that some high record exists, not that the prefix is complete. Pack expansion and interrupted pages make holes plausible.

Why is sync sequential?

Transfers could overlap, but packfile-before-history ordering and simpler local mutation currently provide more value than throughput. Concurrency is a policy choice, not an async default.

Why does the server see ciphertext?

Sync availability should not require trusting the service with shell commands, tokens, paths, or secrets users routinely type.

Why tolerate projection failures independently?

The record store has already converged. A broken script projection should not prevent history or aliases from becoming useful.

One Synchronization, Fully Traced

sync builds an authenticated API client, obtains local and remote RecordStatus, and computes a Diff for each (HostId, RecordTag) stream. Indices are owned by the originating host, avoiding one globally serialized counter across offline machines.

Before mutation, Atuin downloads one non-packfile record and attempts decryption. A wrong master key becomes SyncError::WrongKey; it does not first upload unreadable divergent state.

operations converts head comparisons into Upload, Download, or Noop and sorts deterministically. Packfile downloads precede loose history so expanded ranges can prevent redundant record downloads.

Uploads page from the first remote-missing index. Downloads begin at first_gap, not merely last + 1: a previously expanded pack may have inserted higher records while leaving a hole. Pages are idempotently pushed to SQLite.

Finally, downloaded record IDs drive projection rebuilding. History rebuild is incremental; aliases, variables, KV, and scripts rebuild independently. The encrypted log converges first, then caller-facing data catches up.

Async, Concurrency, and Offline Progress

Atuin’s CLI uses a current-thread Tokio runtime; the server uses a multi-thread runtime. Network requests and SQLx operations suspend cooperatively, while packing, unpacking, encryption-heavy blob work, and other synchronous CPU paths cross spawn_blocking.

The V2 sync loop explicitly says its operations could run in parallel but currently executes them sequentially. Ordering makes packfile expansion, per-host tails, progress, error attribution, and SQLite mutation easier to reason about. Async does not require maximizing concurrency.

SQLite uses WAL mode and pools for ordinary operations. Transactions group domain writes that must become visible together. The optional daemon introduces a broadcast event bus: history capture, sync, search indices, settings, and shutdown react through owned component tasks rather than sharing one mutable application object.

Concurrency boundaries follow consequences: hot shell hooks avoid database initialization for latency; sync pages bound network/memory work; background search loaders may be replaced; projection rebuild errors are contained. The system remains locally functional when network sync is slow or absent.

The Standard Library as Architecture

UUIDv7 newtypes—HistoryId, RecordId, and HostId—prevent identity domains from mixing while preserving time-ordered generation. RecordIdx expresses sequence position separately from globally unique identity.

Arc<CapClient> shares negotiated server capabilities; Arc<[...]> and owned vectors cross async boundaries without borrowing command-local stacks. Range<RecordIdx> makes packed coverage explicit. Option<RecordIdx> models a missing stream rather than inventing a sentinel index.

RecordStatus is a nested map from hosts and tags to heads. Comparing two immutable summaries produces a deterministic plan before mutation. SQLite transactions then own compound local changes.

Cryptographic keys are owned values loaded from restricted local paths. Each record receives a random content-encryption key wrapped by the master key; metadata becomes authenticated implicit assertion. The server owns ciphertext and ordering metadata, not the ability to inspect shell commands.

Generics, Builders, and Storage Boundaries

Client history uses a Database trait object because callers can swap or wrap local implementations at runtime. Server handlers instead use AppState<DB> with DB: atuin_server_database::Database, monomorphizing one chosen Postgres or SQLite backend through Axum state.

The V2 record model is generic over payload state: Record<DecryptedData> can be transformed into Record<EncryptedData>. The type prevents upload code from accidentally accepting plaintext and makes the cryptographic boundary visible without a runtime flag.

Builders for Record and history capture encode required fields in construction APIs. Newtypes keep host, record, session, and history IDs distinct. RecordTag is an extensible product discriminator that lets one sync engine carry several domain projections.

Atuin uses trait objects where local runtime substitution matters, generics where one server backend should remain statically checked, and payload-state types where an invalid security transition must be hard to express.

Errors, Encryption, and Repair

SyncError distinguishes local-store failure, remote request failure, operational setup, internal logic contradiction, suspected remote data loss, and wrong encryption key. These imply different remedies and should not become one string at the library boundary.

Sync validates the key before changing either side. Downloads locate the first gap and use saturating arithmetic, making partial previous runs recoverable. Idempotent record insertion tolerates re-downloading already present suffixes.

Packfile manifests are committed only after their covered history is expanded locally. Temporary network failure aborts; permanently unexpandable content is reported as data loss and skipped so unrelated streams can continue.

After convergence, each domain projection rebuilds independently and warns on failure. This prefers maximum locally usable state over one all-or-nothing transaction spanning unrelated databases. Offline operation is the ultimate failure mode: capture and search remain useful without the server.

Build a Smaller Atuin Sync

Preserve offline convergence, not the shell UI.

1. Define HostId, Tag, Index, and an append-only Record.
2. Store local streams in SQLite and compute per-stream heads.
3. Build a server that stores opaque records and returns status summaries.
4. Diff two summaries into deterministic upload/download/noop operations.
5. Transfer bounded pages idempotently and resume after injected failure.
6. Encrypt each payload locally and authenticate its host/tag/index metadata.
7. Rebuild one searchable history projection from downloaded record IDs.
8. Create two offline clients, append independently, and prove convergence.
9. Introduce a deliberate missing index and repair from first_gap.

Production Atuin adds multiple data domains, legacy migration, authenticated accounts, PASETO envelope encryption, packfiles and presigned object storage, daemon components, shell latency constraints, key rotation, rich SQLite queries, server backend portability, and extensive recovery diagnostics.

Garage: Orientation

Garage is a geo-distributed object store with an S3-compatible API. It splits small, mergeable metadata from large, immutable content-addressed blocks, then replicates both according to the cluster layout.

S3 request
    ↓
object/version metadata ── CRDT table ── quorum replicas
    ↓ block references
content-addressed blocks ── streamed RPC ── storage replicas
                                      ↓
                         background sync, repair, and GC

Design thesis

Garage makes distributed failure manageable by giving metadata and bytes different representations, making metadata mergeable and blocks immutable, and treating repair as a permanent background responsibility.

This case study focuses on one large PutObject, then follows the same data through quorum replication, failure cleanup, and repair. Garage also supports multipart uploads, K2V, websites, administration, several metadata databases, and many S3 endpoints; those are important product features, but not the architectural center we are rebuilding.

What to know first

• An async task can wait without occupying an OS thread.
• A bounded channel limits queued values and propagates backpressure.
• A content hash can name immutable bytes.
• A quorum means an operation succeeds after enough replicas respond.
• A CRDT defines how concurrent replicas merge without a single primary.

You do not need to know distributed-systems theory in advance. The chapters introduce each term where Garage makes it concrete.

Interactive Object-Storage Map

Why Is It Designed This Way?

Why separate metadata tables from object blocks?

Metadata is small, structured, mutable, and must merge. Object bytes are large and expensive to copy but become simple once immutable and content-addressed. Using one protocol and representation for both would make each inherit the other’s worst constraints.

Why use CRDT tables instead of one primary node?

Garage targets multiple sites connected by ordinary, failure-prone networks. Mergeable entries let replicas accept and reconcile state without putting one leader on every metadata operation’s availability path.

Why acknowledge after a quorum rather than every replica?

Waiting for all replicas turns one slow or disconnected node into global unavailability. A quorum gives a precise threshold, while background synchronization and the remaining RPC futures continue convergence.

Why write Uploading metadata before streaming blocks?

The system needs durable evidence of incomplete work. Block references must not point to a version the metadata layer cannot identify, and cancellation must leave enough identity for cleanup.

Why mark the object Complete last?

Publication is a commit point. Readers should either see the prior complete version or the new complete version, never infer completeness from scattered blocks that happened to arrive first.

Why pipeline one upload with tiny bounded channels?

Reading, checksumming, CPU transformation, and network replication have different bottlenecks. Overlap improves throughput, while capacities 2/1/1 prevent a fast client or CPU stage from buffering an entire object in memory.

Why have both per-request and global limits?

A per-request write cap provides fairness between uploads. The global buffered-kilobyte semaphore protects node memory across all RPCs. Neither budget can substitute for the other.

Why are blocks content-addressed?

Immutable bytes named by a hash are easy to verify, retry, deduplicate, and repair. Mutable object meaning stays in the CRDT metadata that refers to those hashes.

Why treat repair as normal background work?

Replica divergence is an expected consequence of partial failure and changing cluster layouts. A distributed store is not reliable because failure never happens; it is reliable because it continuously discovers and repairs the allowed incomplete states.

Why is garbage collection so cautious?

Deleting a live block or forgetting a tombstone can destroy data or resurrect a deletion. Retaining extra bytes costs capacity; deleting too early can break the consistency model. Garage chooses the reversible side of that tradeoff.

One Large PUT, Fully Traced

handle_put first parses metadata, checksum expectations, and encryption parameters. It converts the Hyper body into a stream and delegates to save_stream. The handler does not buffer the whole object.

save_stream reads the first block while fetching existing object metadata. If the body fits below the inline threshold, Garage encrypts it and stores it inside the object table. A large object takes the more interesting path.

Before transferring bytes, Garage writes an Uploading object version and an empty version-table entry. These are durable breadcrumbs: later block references always point at a known version, and interrupted work can be found and cleaned.

The body then passes through four cooperating futures:

HTTP body → fixed-size chunks → checksums → encrypt + content hash → replica writes
             channel(2)         channel(1)          channel(1)

Hashing and encryption use spawn_blocking; bounded Tokio channels keep those CPU stages from running arbitrarily far ahead of network writes. The final stage also caps the number of concurrent block writes for this request.

For every block, put_block_and_meta concurrently performs three obligations:

• replicate immutable block bytes to their placement set;
• add the block and offset to the version table;
• add a reference from the version to the block hash.

Only after all blocks finish, checksums and quotas pass, and metadata is ready does Garage merge a Complete object version. A Drop guard remains armed until that point. On error or cancellation it schedules cleanup for the incomplete upload.

The HTTP response therefore means more than “the body was received.” It means the final object metadata reached its configured write quorum after the byte and reference obligations completed.

Async, Concurrency, and Distributed Parallelism

Garage uses concurrency at several independent levels. Collapsing them into “Tokio makes it parallel” hides the resource policy.

Request concurrency

The API server runs many connection and request futures on Tokio. A request usually suspends on body input, metadata RPCs, disk, or peer responses rather than blocking an executor thread.

Pipeline concurrency inside one PUT

read_and_put_blocks uses bounded MPSC channels between reading, checksum, encryption/hash, and write stages. futures::join! drives all four at once. Capacity 2/1/1 is a memory bound as well as a scheduling decision: only a few full blocks may wait between stages.

The checksummer is stateful, so blocks pass through it in order. Encryption and hashing are CPU work and cross through spawn_blocking. The async runtime remains available while the blocking pool performs those calculations.

Several block writes from one request

The writer owns a FuturesOrdered of in-flight block operations. It admits another block only below block_max_concurrent_writes_per_request. Completion is observed in input order, while the underlying writes can overlap. This prevents one large upload from creating an unbounded future set.

Parallel RPCs and quorum completion

A table insert sends updates to the placement nodes concurrently. It can return when every required write set has reached quorum. Remaining calls are driven in a spawned task, spreading the update without adding their full tail latency to the client response.

Block upload adds another budget: a semaphore counts buffered kilobytes held for peer transmission. Its owned permit travels inside RequestStrategy and is dropped only when the RPC set finishes.

Long-running background concurrency

Every table owns Merkle-update, synchronization, garbage-collection, and queued-insert workers. The block manager owns resync and scrub workers. A shared BackgroundRunner supervises them, records status, applies exponential error delay, and coordinates shutdown.

The important pattern is nested budgets:

many requests
  └─ bounded pipeline buffers per request
       └─ bounded block-write futures per request
            └─ quorum RPC fan-out
                 └─ global buffered-byte semaphore

Async supplies suspension. These explicit limits supply predictable resource use.

The Standard Library as Architecture

Garage’s distributed algorithms are visible in ordinary ownership and collection choices, not only in networking code.

Arc<T>: stable shared services

Garage owns Arc<System>, Arc<BlockManager>, and typed Arc<Table<...>> values. Request handlers, RPC endpoints, and workers clone handles, not the database or cluster state. Arc answers “who may keep this service alive?”; interior synchronization answers “who may mutate it now?”

Enums: lifecycle states are data

An object version is explicitly Uploading or Complete; block RPCs and table RPCs are enums; workers report Busy, Throttled, Idle, or Done. Matches force code to account for each protocol and lifecycle state.

Drop: cancellation is an execution path

InterruptedCleanup begins armed around a large upload. Normal completion cancels it. Early return, task cancellation, or panic drops it while armed and schedules cleanup. This is RAII applied to distributed metadata obligations.

BTreeMap and BTreeSet: deterministic reconciliation

Range reads merge entries from several replicas by ordered encoded key. BTreeMap both deduplicates and restores the requested enumeration order; BTreeSet records which keys need read repair. The collection invariant is part of the consistency algorithm.

HashMap: batch network work by destination

insert_many groups encoded entries by replica node. Shared Arc<ByteBuf> values avoid re-encoding or copying the same update into every bookkeeping structure.

Ownership as resource accounting

The block sender acquires an owned semaphore permit proportional to buffered kilobytes and embeds it in RequestStrategy. The permit’s lifetime now equals the network obligation’s lifetime, including error and cancellation.

The recurring lesson is that Rust’s standard ownership tools become most useful when a type owns a real operational obligation: a service lifetime, an incomplete upload, a deterministic merge, or scarce buffer memory.

Generics, Traits, and Erased Workers

Garage uses generics where one algorithm should be specialized for many table schemas, and trait objects where a runtime collection must hold unlike worker types.

One table algorithm, many schemas

The central type is conceptually:

Table<F: TableSchema, R: TableReplication>

TableSchema supplies associated types for partition key, sort key, entry, and filter. TableReplication supplies placement and read/write quorum policy. The generic table can then implement insert, get, range reads, RPC encoding, Merkle synchronization, repair, and garbage collection once.

Associated types express a family invariant: an object table has one coherent combination of key, entry, and filter types. Callers cannot accidentally query it with a bucket-table key.

Behavioral bounds on entries

An Entry<P, S> must be cloneable, serializable, migratable, sendable, and a Crdt. Those bounds are not decoration. Distributed table code needs to move entries into async work, encode them for RPC, migrate stored versions, obtain keys, and merge divergent replicas.

Generic streaming handlers

save_stream<S> and read_and_put_blocks<S> accept any stream of byte chunks with the required item and pinning behavior. Tests, HTTP bodies, copies, and POST uploads can reuse the same storage pipeline without sharing one concrete body type.

Generic request policy owns a resource

RequestStrategy<T> is generic over a value dropped on completion. Most calls use T = (); block writes use an owned semaphore permit. This avoids a boxed callback and makes “release this exact resource when the RPC set ends” a compile-time ownership relationship.

Where dynamic dispatch wins

BackgroundRunner receives Box<dyn Worker>. Merkle, sync, GC, resync, scrub, and lifecycle workers have different concrete types but must coexist in one runtime collection. Their hot domain algorithms remain generic; only the heterogeneous supervision boundary is erased.

That division is a useful rule:

Use generics to preserve relationships inside reusable algorithms. Use a trait object at a collection or plugin boundary whose members are chosen at runtime.

Errors, Consistency, Repair, and Shutdown

Garage does not try to make partial failure disappear. It represents where an operation has reached, returns once a stated consistency threshold is met, and keeps repair machinery running afterward.

Quorum is a success contract

Metadata placement returns one or more write sets. Table::insert sends the update to their nodes and succeeds only when each set has its required quorum. Reads collect a read quorum, decode each result, and merge differences with the entry’s CRDT rule.

This is not full immediate agreement. It is a contract that enough intersecting replicas have participated to provide the configured behavior.

Read repair is explicit

If a monotonic read sees missing or unequal replicas, Garage writes the merged entry back. Foreground observation becomes a repair opportunity. Merkle-based table synchronization separately finds divergence even when nobody reads a key.

Upload failure leaves evidence

Large PUT establishes Uploading object and version metadata before block transfer. The armed cleanup guard handles errors and cancellation. Publishing Complete happens last, so readers do not interpret a partly transferred version as a finished object.

Immutable blocks simplify retry

A block is named by the hash of its stored bytes. Re-sending the same hash is idempotent, and reads verify integrity. Reference-count changes enqueue delayed resync checks; a missing required block is fetched from another responsible node.

Deletion is deliberately conservative

Metadata tombstones must remain long enough to prevent deleted values from returning during reconciliation. Blocks whose reference count reaches zero are not removed immediately. GC and resync encode delay and propagation rules because premature deletion is harder to repair than temporary extra storage.

Worker failure and shutdown

A background worker error is logged, counted, and retried after exponential delay. A Tokio watch channel broadcasts stop intent. Workers are given time to finish their current unit; after an eight-second drain deadline, remaining worker futures are cancelled. API servers and cluster communications receive the same shutdown signal and are joined by the server orchestrator.

Graceful shutdown here is bounded cooperation, not a promise to wait forever.

Build a Smaller Garage

Rebuild the architectural center, not the S3 syntax. The goal is a small replicated blob store that genuinely preserves streaming, publication, quorums, merge, and repair.

What are we preserving?

• large values stream through bounded stages;
• metadata and immutable blocks have different representations;
• writes publish metadata only after block obligations finish;
• replica operations have explicit quorum thresholds;
• cancellation leaves cleanup work;
• repair can restore a missing block from another replica.

Stage 1: one-node content-addressed storage

Implement put(Read) that chunks input, hashes each block, writes it under its hash, and finally stores a manifest from object key to ordered hashes. Implement get by verifying and concatenating those blocks.

Stage 2: turn PUT into a bounded pipeline

Use bounded channels between chunking, hashing, and disk writing. Add a small concurrent write limit. Measure the maximum number of full blocks retained by the pipeline; it should follow channel capacities, not object size.

Stage 3: add explicit publication state

Represent metadata as:

enum VersionState {
    Uploading { id: VersionId },
    Complete { id: VersionId, blocks: Vec<Hash> },
    Aborted { id: VersionId },
}

Write Uploading first and Complete last. Arm a drop guard that queues an Aborted transition if the upload future disappears before publication.

Stage 4: create three in-process replicas

Give each replica its own metadata map and block directory. A deterministic placement function chooses three replicas. Send writes concurrently and return after two acknowledge. Inject latency and failure into the third.

Stage 5: make metadata mergeable

Give versions stable IDs and timestamps, retain concurrent versions, and define a deterministic merge. Implement a quorum read that merges responses. Do not use “last response wins”; make the conflict rule a type-level operation.

Stage 6: add read repair

When a read finds different metadata, send the merged value back. When a manifest names a locally missing block, fetch it from another placement node, verify the hash, and store it.

Stage 7: supervise background repair

Define a small Worker trait with work and wait_for_work. Run a repair queue with error backoff and a shutdown signal. Expose worker state so repair is observable rather than magical.

Failure exercises

• Cancel after writing two blocks but before publication.
• Lose one metadata response during a write.
• Corrupt one block and verify that its hash detects the fault.
• Remove one replica’s block and let repair fetch it.
• Partition one replica, perform writes, reconnect, and converge metadata.
• Begin shutdown while a repair unit is running; verify the bounded drain.

Compare with production Garage

Your version can teach the center in a few modules. Garage additionally provides S3 compatibility, authenticated streaming, encryption, multipart uploads, durable embedded databases, cluster layout transitions, Merkle table sync, conservative tombstone and block GC, observability, multiple APIs, data scrubbing, migrations, and operational tooling.

Those are not incidental extras. They are the hardening needed when the toy failure injector becomes real disks, networks, upgrades, and operators.

Zellij: Orientation

Zellij is a terminal workspace whose server outlives any one attached client. It owns child pseudo-terminals, pane state, plugins, rendering, and session lifecycle. The client owns the user’s real terminal and transports input and rendered output across IPC.

keyboard → client IPC → route/action → Screen owner → PTY writer → child
                                                              ↓ output
client terminal ← rendered ANSI ← server ← debounce ← Screen/VTE parser

Design thesis

Zellij serializes mutation behind subsystem-owner threads, communicates with typed instructions, and uses async I/O only where waiting on many PTYs and timers benefits from it.

This is a hybrid design. “Uses Tokio” does not mean “everything is an async task.” The Screen, PTY command loop, plugin runtime, PTY writer, and background jobs each have dedicated synchronous owner threads. A shared four-thread Tokio runtime handles PTY readiness, timers, downloads, and action-completion waits.

What to know first

• A pseudo-terminal makes a child process behave as if it owns a terminal.
• ANSI/VT bytes describe text, cursor motion, colors, and terminal operations.
• A channel transfers owned messages between independently running owners.
• A oneshot carries one completion value back to one waiter.
• Debouncing combines many rapid requests into one later operation.

Interactive Terminal-Session Map

Why Is It Designed This Way?

Why is the session a server separate from the client?

Child processes and workspace state must survive detach, reconnect, and multiple viewers. The client owns one physical terminal; the server owns the durable interactive session.

Why use subsystem-owner threads instead of shared locks everywhere?

Screen layout, PTY lifecycle, plugin execution, and terminal writing have different blocking behavior and invariants. Typed queues serialize each domain’s mutations and make cross-domain causality visible.

Why use Tokio if the main owners are threads?

Waiting for readiness from many PTY file descriptors and timers is exactly where async multiplexing helps. It does not follow that pane geometry or VTE state should become concurrently mutable async state.

Why interpret keybindings on the server?

The server owns current modes, shared-session state, and runtime configuration. A thin client can reconnect or coexist with other clients without becoming a second authority on what a key means.

Why route terminal input through Screen before the PTY?

The same bytes may mean UI rename/search input, synchronized-pane broadcast, plugin interception, or focused-terminal input depending on session state. Screen owns that decision.

Why have a separate PTY writer thread?

Writing a program’s stdin while coupling it to stdout processing can deadlock real terminal applications. The writer also gives partial writes and fairness one explicit owner.

Why bound output admission but leave many control channels unbounded?

PTY output is high-volume and naturally pressure-sensitive. Control messages must often cross ownership boundaries without blocking an owner that is needed to service the destination. The tradeoff is intentional, though unbounded control traffic still requires disciplined producers.

Why debounce rendering?

Terminal programs emit bursts of many tiny updates. Rendering each one repeats layout and ANSI serialization work and can make Screen fall behind. A short window trades imperceptible latency for much less redundant work.

Why attach a oneshot completion token to selected actions?

Channels prove delivery, not logical completion. A token that travels to the last responsible owner lets callers order dependent actions without turning every instruction into heavyweight request/response RPC.

Why use trait objects for panes?

Terminal and plugin panes occupy the same runtime layout and need the same operations. The collection is heterogeneous by product design, so dynamic dispatch belongs exactly at that boundary.

One Keypress, Fully Traced

The client’s input handler receives a parsed terminal event and sends a ClientToServerMsg::Key over IPC. Keybindings are interpreted on the server, so live configuration and shared-session state determine the action.

The route thread turns that action into a typed subsystem instruction. For ordinary text it sends ScreenInstruction::WriteCharacter, carrying raw bytes, client identity, keyboard-protocol information, and a NotificationEnd.

Screen is the serial owner of tabs, focus, modes, and panes. It decides whether the bytes rename UI state, update search, go to one focused terminal, go to all synchronized panes, or become an intercepted plugin event.

For terminal input, the selected pane ultimately sends PtyWriteInstruction::Write. A dedicated writer thread owns a FIFO queue per terminal and performs nonblocking writes. It moves on when one kernel buffer returns EAGAIN, so a slow program cannot stall input to every pane.

NotificationEnd connects logical completion back to the route action. Its Drop implementation sends the result through a Tokio oneshot when the instruction reaches the end of its ownership path. The route thread normally waits no more than one second before allowing the next potentially racing action.

This is not request/response RPC between every thread. It is mostly one-way ownership transfer, with a small completion token attached only when ordering matters.

Threads, Async I/O, Concurrency, and Backpressure

Zellij’s concurrency follows state ownership.

Dedicated owner threads

One session starts threads for Screen, PTY commands, Wasm plugins, PTY writes, and background jobs. Each blocks on typed channel instructions and mutates its own state serially. Pane layout changes therefore do not require many fine- grained locks inside Screen.

Async work where readiness matters

Spawning a terminal returns an AsyncReader. A Tokio task per terminal awaits PTY readability using the platform implementation (AsyncFd on Unix). It sends ScreenInstruction::PtyBytes into a bounded Screen channel of capacity 50. Async I/O lets a few runtime threads wait on many child outputs.

The shared Tokio runtime also supplies timers, HTTP/download work, and oneshot completion waiting. It is not the owner of tabs or pane geometry.

A deliberate sync/async bridge

The internal instruction channels are synchronous. A PTY-reader task crosses that boundary with spawn_blocking when it sends to Screen. If the bounded channel fills, pressure reaches the blocking pool rather than stalling a Tokio worker directly.

Independent write fairness

PTY input uses a separate writer thread because reading and writing the same terminal in one execution path can deadlock some programs. It keeps a VecDeque per terminal, writes until the kernel would block, then tries the next terminal. Each pane’s pending bytes are capped at 10 MiB.

Render coalescing

Every PTY chunk updates pane state, but visible rendering is debounced through the background-jobs owner. Many bursts within roughly 10 ms become one RenderToClients. Screen renders only dirty state and serializes ANSI output per client.

The complete feedback loop is:

child readiness → bounded Screen admission → terminal state mutation
       → debounced render → client output → user input → fair PTY writer

Threads isolate mutable domains. Async tasks multiplex readiness. Bounds and debouncing prevent either mechanism from creating unlimited work.

The Standard Library as Architecture

Enums define subsystem protocols

ScreenInstruction, PtyInstruction, PtyWriteInstruction, PluginInstruction, and BackgroundJob make cross-thread commands explicit. An owner loop matches the complete vocabulary of operations it accepts. Data needed later travels inside the variant as owned values.

HashMap represents identity-indexed session state

Tabs, clients, terminal child PIDs, async task handles, pending events, and per-terminal write queues are maps because messages refer to stable IDs across thread boundaries. The ID is the shared language; a Rust reference cannot safely span those independent owners.

VecDeque makes per-pane FIFO visible

The PTY writer’s HashMap<u32, VecDeque<PendingWrite>> preserves byte order within each terminal while allowing round-robin progress between terminals. The data structure expresses both ordering and fairness policy.

Arc and atomics for narrow shared observations

PTY reader tasks and owners share small activity flags with Arc<AtomicBool>. Large pane state does not become globally shared merely because one activity bit must cross threads.

Drop closes protocols

Dropping the original NotificationEnd sends action completion. Dropping SessionMetaData broadcasts Exit to each owner and joins its threads. Dropping Pty closes its child panes. These types own lifecycle obligations, so RAII covers early return as well as the happy path.

OnceLock and process-wide facilities

The async runtime and selected shared configuration are initialized once. This is appropriate for truly process-wide infrastructure; per-session pane state remains owned by the session threads.

The pattern worth copying is restraint: share tiny facts, send owned commands, and keep large mutable models behind one owner.

Generics, Traits, and Runtime Polymorphism

Generic buses preserve instruction types

Bus<T> can receive one or more channels carrying exactly T, while sharing the same selection, timeout, OS-access, and sender plumbing. A Screen bus cannot accidentally receive a PTY instruction. The generic disappears after monomorphization; no dynamic dispatch is needed for the protocol loop.

Trait objects unify unlike panes

A tab stores Box<dyn Pane> because terminal panes and plugin panes coexist in the same tiled and floating collections. The common trait exposes geometry, rendering, input, focus, scrolling, and lifecycle behavior. Which concrete pane occupies a layout slot is runtime state.

This is a natural trait-object boundary: one heterogeneous collection needs uniform behavior. Making Tab<T: Pane> generic would allow only one pane type per tab or require a second enum mirroring all pane variants.

OS behavior is injected behind traits

ServerOsApi, ClientOsApi, and AsyncReader isolate Unix, Windows, tests, and held-pane behavior. Owners hold boxed trait objects because the selected platform implementation is runtime infrastructure, not a parameter readers should carry through every terminal method.

Closures carry one-off lifecycle policy

PTY spawning accepts callbacks for child exit. The closure owns the originating plugin information, hold-on-close policy, and typed senders needed later. This is generics and capture-based composition where defining a permanent trait implementation would add ceremony without creating a reusable domain type.

Typed messages versus erased services

Zellij keeps message payloads concrete in enums, uses generics for reusable plumbing, and uses trait objects at heterogeneous model/platform boundaries. That division keeps the concurrency graph readable while still supporting terminal panes, plugins, multiple operating systems, and test doubles.

Errors, Ordering, Slow Consumers, and Shutdown

Context follows instructions

Channel payloads include an ErrorContext. Each owner adds its own operation context before handling the message. Failures can therefore report the path through input, routing, Screen, PTY, or plugin code rather than only the final I/O error.

Errors are deliberately classified by call site. Structural owner-thread failure can be fatal; expected pane or OS-operation failures are often logged as non-fatal so the rest of the session remains usable.

Logical completion prevents action races

Some actions cross several threads before their effect is safe to follow. NotificationEnd travels with the instruction and resolves a oneshot when its owning path ends. The route thread normally caps the wait at one second; select CLI operations may wait indefinitely because their API promises a completed result.

Slow PTY output meets bounded admission

Terminal reader tasks send bytes through a capacity-50 channel to Screen. A backed-up Screen eventually slows readers instead of allowing unbounded output messages. Rendering is separately debounced, so parsing progress does not require repainting after every chunk.

Slow PTY input is isolated

The writer preserves partial writes with an offset and retries them. A stuck terminal does not block other terminal queues. At 10 MiB pending for one pane, Zellij logs and clears that queue: explicit degradation is preferable to unlimited session memory.

Shutdown follows ownership

SessionMetaData::drop sends Exit to PTY, Screen, plugin, writer, and background-job owners, then joins each thread. The PTY owner’s own Drop closes remaining children. Terminal-reader sends that fail during teardown are intentionally ignored at EOF because the consumer may already be gone.

This is a practical lesson: a closed channel during coordinated shutdown is a lifecycle signal, not automatically an application bug.

Build a Smaller Zellij

Build a one-session terminal multiplexer, not a parser that merely recognizes Zellij commands.

What are we preserving?

• a client/session process boundary;
• one owner for screen state;
• typed messages between input, screen, PTY, and output;
• async PTY reading and isolated nonblocking PTY writing;
• ANSI parsing into a persistent terminal grid;
• debounced dirty rendering;
• action completion and coordinated shutdown.

Stage 1: one child and one PTY

Spawn a shell under a pseudo-terminal. Forward the user terminal’s bytes to it and its bytes back. Restore the host terminal mode on every exit path.

Stage 2: model a terminal grid

Feed child output through a VTE parser. Store cells, cursor, dimensions, and a scrollback deque. Render the model rather than blindly echoing child bytes.

Stage 3: make Screen the owner

Move the grid into a Screen thread receiving:

enum ScreenInstruction {
    PtyBytes(PaneId, Vec<u8>),
    Key(PaneId, Vec<u8>, Completion),
    Resize(Size),
    Render,
    Exit,
}

Only this thread mutates panes, focus, and geometry.

Stage 4: add two panes

Give each pane an ID, PTY, grid, and write queue. Implement focus switching and a vertical split. The key route decides whether input changes focus or enters the active terminal.

Stage 5: async readers, fair writer

Run one async reader future per PTY and feed a bounded Screen channel. Create a dedicated writer owner with one VecDeque per pane and nonblocking partial writes. Cap pending bytes per pane.

Stage 6: debounce rendering

Mark panes dirty on VTE changes. Convert many render requests within 10–16 ms into one repaint. Record how many PTY chunks arrive per visible frame.

Stage 7: logical completion and shutdown

Attach a oneshot sender to focus/layout actions and resolve it by dropping an RAII token at the terminal owner. On shutdown, stop input admission, signal owners, close children, join threads, and restore the client terminal.

Failure exercises

• Make one child stop reading stdin; verify the other pane still receives keys.
• Flood stdout; verify bounded Screen admission.
• Emit one character per write; verify render coalescing.
• Kill a child mid-frame; keep the other pane alive.
• Detach the client while children run, then attach another client.
• Shut down with pending writes and verify every owner terminates.

Compare with production Zellij

Production adds rich layouts, floating and stacked panes, multiple clients and watchers, Wasm plugins, session serialization, images and hyperlinks, keyboard protocols, mouse handling, cross-platform PTYs, web sharing, configuration reload, nested sessions, extensive UI policy, and recovery behavior.

Your smaller version succeeds if its concurrency graph is genuine: each owner has a clear state boundary, PTY waits do not consume one thread each, a slow pane cannot freeze every pane, and repaint work is visibly coalesced.

Tokio: Orientation

Tokio turns Rust futures into a running asynchronous system. Its runtime bundles task scheduling, operating-system I/O readiness, timers, and a separate pool for blocking functions.

spawn future → runnable task queue → poll
                                  ├─ Ready(output) → JoinHandle
                                  └─ Pending + Waker
                                          ↓
                            I/O driver or timer fires
                                          ↓
                              wake → runnable queue

Design thesis

Tokio separates “may this future make progress?” from “which thread should poll it?”, connecting resource readiness to cooperative task scheduling with wakers.

Tokio does not run an async fn continuously in the background. Calling it creates a future. A task repeatedly polls that future only after it is spawned and initially scheduled or later woken.

Four terms to keep separate

• Asynchrony: work can pause at Pending and resume after a wake-up.
• Concurrency: many tasks can be in progress, even on one runtime thread.
• Parallelism: a multi-thread runtime can poll different tasks on several worker threads simultaneously.
• Blocking: an OS thread cannot run other work until an operation returns.

This chapter goes below the safe public API into Tokio’s unsafe task core. The goal is not to copy it casually. It is to understand which invariants a production executor must uphold so ordinary application code stays safe.

Interactive Runtime Map

Why Is It Designed This Way?

Why poll futures instead of giving each operation a thread?

Most network tasks spend most of their lifetime waiting. Storing suspended state in a future and polling only on readiness lets a small worker set manage many connections without an OS stack and scheduler entry per connection.

Why must a future return Pending with a Waker?

Pending alone would force the executor to poll every dormant task repeatedly. The waker gives the awaited resource a direct route to make exactly that task runnable again.

Why are tasks cooperatively scheduled?

Rust futures are ordinary state machines polled as function calls. Tokio cannot preempt arbitrary Rust code safely in the middle of a poll. Cooperative budgets and well-behaved async primitives create yield points at controlled boundaries.

Why use local queues, a LIFO slot, and a global injection queue?

Locally spawned follow-up work often shares hot data and should run quickly. Remote spawns need a synchronized entry point. Periodic global checks and a cap on repeated LIFO polls keep those locality optimizations from becoming starvation.

Why steal work?

Task arrival and wake-ups are uneven. A worker that exhausts its own queue can move runnable tasks from a busy worker, turning a fixed thread pool into useful parallelism without centralizing every pop behind one lock.

Why does I/O readiness wake tasks rather than perform their operations?

The OS reports that an operation may now succeed; it does not own the future’s buffer, error handling, or state machine. Waking re-enters the normal poll path, where the operation is retried and WouldBlock can clear stale readiness.

Why have a separate blocking pool?

A blocking syscall or CPU-heavy synchronous function can monopolize a runtime worker and delay every task assigned there. A separate pool contains that behavior. It is a bridge, not automatic CPU backpressure; callers may still need a semaphore or Rayon.

Why type-erase tasks internally?

spawn accepts every concrete future type, but one scheduler queue must hold them together. A small raw pointer plus type-specific vtable retains efficient generic construction and typed JoinHandle<T> while giving the queue one uniform runnable representation.

Why does cancellation happen at poll boundaries?

Tokio can atomically request cancellation and safely drop a future when the task harness next owns exclusive access. It cannot forcibly interrupt arbitrary user code between Rust instructions.

Why offer current-thread, local, and multi-thread runtimes?

Concurrency does not always require parallelism, and !Send futures must stay on one thread. Different schedulers make those constraints explicit instead of paying for or promising movement every application does not need.

One I/O Task, Fully Traced

Consider a spawned future awaiting readability from a TCP stream.

1. Spawn preserves the public output type

spawn<F> accepts a concrete Future + Send + 'static and returns JoinHandle<F::Output>. Internally, Tokio allocates one Cell<F, S> containing the future, scheduler handle, atomic task state, join waker, and intrusive-list links.

The scheduler cannot have a separate queue type for every F. RawTask retains only a pointer to the common header. A per-F, S vtable knows how to poll, schedule, deallocate, cancel, and read the output.

2. Initial notification enters a run queue

The task starts with NOTIFIED set and references for runtime ownership, the runnable notification, and the JoinHandle. A spawn from a worker normally uses local scheduling; a spawn from elsewhere enters the global injection queue and unparks a worker.

3. A worker polls the future

The worker selects local work, periodically checks global work, and otherwise steals from peers. Before polling, the task state atomically moves from idle to RUNNING, preventing concurrent poll or cancellation-drop access.

The future calls the TCP read implementation. No bytes are ready, so the resource’s ScheduledIo stores the task waker for readable interest and the future returns Poll::Pending. The task transitions back to idle.

4. The worker does something else

It polls another runnable task. If no work remains, a worker parks through the combined time/I/O driver instead of spinning.

5. Mio reports readiness

The OS poller returns an event. Tokio converts it to its readiness flags, updates the ScheduledIo atomic state, removes matching stored wakers under a lock, releases the lock, and wakes them.

6. Wake means schedule, not execute immediately

The raw waker atomically sets NOTIFIED. If the task was idle and not already queued, its scheduler submits one runnable reference and unparks a worker. Repeated wake-ups coalesce while NOTIFIED is already set.

7. The next poll makes progress

A worker polls the future again. The read syscall now succeeds. If the outer future completes, its output replaces the future in the task stage, state moves to COMPLETE, and the waker registered by JoinHandle is invoked.

The awaiting caller later takes F::Output through the typed join handle. The scheduler queue never needed to know that output type.

Scheduling, Concurrency, Parallelism, and Fairness

One task is polled by at most one worker

The atomic RUNNING bit is a lock around the future or completed output. A task may move between multi-thread workers, but two workers do not poll the same future simultaneously.

Runnable queues are not waiting-task queues

An I/O-waiting task remains owned by the runtime but is absent from run queues. Its waker is registered with a resource. This distinction is why thousands of sleeping tasks do not require repeatedly scanning thousands of queue entries.

Locality first, fairness periodically

Each multi-thread worker has a local queue and one LIFO slot for freshly scheduled local work. The LIFO path improves latency and cache locality in message-passing chains, but Tokio caps consecutive LIFO polls. Workers also check the global injection queue at a tuned interval.

When a local queue is empty, the worker steals from peers. External wake-ups or queue overflow use the synchronized injection path. Idle coordination unparks only enough workers to search for available work.

Cooperative scheduling has a budget

One poll is ordinary Rust code; Tokio cannot preempt it. Runtime-aware async operations consume a cooperative budget. When depleted, they yield so another task can run. A future that performs a long computation without awaiting still blocks its worker.

Parking integrates scheduling with resources

After exhausting and stealing runnable work, a worker parks through the driver. Mio waits for OS readiness with a timeout chosen by the timer wheel. Events wake resource waiters; those wakers schedule tasks back into runtime queues.

Current-thread versus multi-thread

A current-thread runtime uses one executor thread: tasks are concurrent but never poll in parallel. The multi-thread scheduler owns a fixed worker set and can poll different Send tasks simultaneously. LocalSet and the local runtime support !Send futures by keeping them on the owning thread.

Blocking is a separate workload class

spawn_blocking queues a closure to a dynamically managed blocking pool. Its large default thread limit is intended for blocking I/O as well as some CPU work; it is not a CPU-concurrency policy. Large CPU workloads should impose a smaller bound or use a dedicated executor.

The core lesson is that wake-ups create runnable work. Queue policies decide latency and fairness. Worker count decides potential parallelism. None of those by itself limits application-level resources such as database connections or HTTP requests.

The Standard Library as the Async Contract

Tokio’s public foundation comes from std, not from a special async language runtime.

Future, Poll, and Context

A future exposes one operation:

fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>

Ready(output) transfers a result. Pending promises that the future arranged for a relevant waker to be notified. Returning Pending without arranging a wake can leave a task dormant forever.

Pin

Compiled async state machines can contain references into their own stored state. Pin<&mut T> prevents safe code from moving such a future after polling begins. Tokio’s intrusive timer and waiter lists also rely on stable addresses.

Waker

A Waker is an owned, thread-safe callback-like handle. Tokio implements its RawWakerVTable using the task header: clone changes the reference count; wake changes notification state and schedules if necessary; drop releases a task reference.

Atomics encode the task lifecycle

One AtomicUsize packs running, complete, notified, cancelled, join-interest, join-waker, and reference-count state. Atomic read-modify-write transitions establish one order for races among polling, waking, joining, cancellation, and shutdown.

Arc, Mutex, and UnsafeCell

Shared scheduler and I/O structures use Arc; infrequent compound state uses locks; fields with access guaranteed by atomic protocol use UnsafeCell. unsafe does not remove synchronization—it lets Tokio express a verified synchronization rule the compiler cannot infer.

Drop

Dropping a future after cancellation runs its local destructors. Dropping registrations clears stored wakers to break cycles. Dropping runtime ownership initiates scheduler, driver, and blocking-pool teardown.

Application authors mostly see safe futures because Tokio concentrates these low-level invariants behind a narrow runtime task abstraction.

Generics, Associated Types, and Task Type Erasure

The public API preserves exact types

spawn<F> is generic over the future and returns JoinHandle<F::Output>. Channels are Sender<T> and Receiver<T>. Timeouts wrap a concrete future. Async I/O extension methods are generic over buffers and operations. Callers retain typed values without serialization or Any downcasts.

The Future associated type means one concrete future type has one output type. JoinHandle<T> can therefore expose Result<T, JoinError> even after the runtime has erased the future from its queues.

Generic allocation, uniform queue

At construction, Cell<T: Future, S> knows the exact future and scheduler types. Its stage is:

enum Stage<T: Future> {
    Running(T),
    Finished(Result<T::Output, JoinError>),
    Consumed,
}

The common first field is a non-generic header containing state and a pointer to a static vtable specialized for T, S. Scheduler queues store RawTask, a pointer to that header. Vtable functions recover layout offsets and invoke the right poll/drop/output code.

This is manual type erasure, closer to how a trait object works than to an enum. It avoids boxing a second dyn Future<Output = ...> abstraction while allowing outputs of every type.

Scheduler behavior is generic too

Internal task handles are parameterized by a Schedule implementation. Current-thread, multi-thread, and blocking tasks reuse the task harness while choosing different submission behavior.

Dynamic behavior stays behind small handles

At the public boundary, users normally compose concrete future types that the compiler can inline. At the scheduler boundary, Tokio deliberately erases them because heterogeneous queues require one representation.

That is the recurring pattern throughout this book:

Preserve types while they encode useful caller relationships; erase them at the exact runtime collection boundary that cannot remain homogeneous.

Cancellation, Panics, Shutdown, and Concurrency Safety

Cancellation is a state transition

JoinHandle::abort sets the task’s cancelled state and schedules it if needed. The next exclusive task-harness transition drops the future. Local variables run their destructors, but code after the suspended .await does not run.

Cancellation is therefore not rollback. Application invariants spanning multiple awaits need guards, transactions, idempotence, or an explicit shutdown protocol.

Panic and cancellation are join errors

The task harness catches a panic at the task boundary and stores a JoinError; cancellation produces another JoinError classification. The worker thread can continue running unrelated tasks. Business errors remain the future’s own Result inside the successful join output.

Dropping a join handle detaches

JoinHandle represents interest in the output, not ownership of execution. Dropping it clears join interest and lets the task continue. Explicit abort is the cancellation operation.

Blocking work has weaker cancellation

Once a spawn_blocking closure starts, Tokio cannot abort arbitrary synchronous code. Runtime shutdown may wait indefinitely unless the caller uses a timeout shutdown method, and the closure itself needs a cooperative stop mechanism for bounded termination.

Runtime shutdown closes admission before draining storage

The multi-thread scheduler closes both its owned-task collection and global injection queue, wakes workers, cancels owned tasks, then empties local and global runnable queues. Both structures need closed state to prevent a spawn race from leaving an unreachable reference cycle.

The I/O driver marks registrations shut down and wakes their waiters. The time driver advances outstanding timers into terminal error/wake behavior. Resource futures do not remain silently pending after their driver disappears.

Low-level races are tested systematically

Tokio has dedicated Loom tests for task-state combinations, queues, shutdown, oneshot channels, and schedulers. Loom explores legal thread interleavings against modeled atomics and locks. Ordinary unit tests rarely encounter the one ordering that breaks a custom waker or refcount protocol.

The production lesson is not “use more atomics.” It is that once correctness depends on a compact atomic state machine, transitions need written invariants and systematic interleaving tests.

Build a Smaller Tokio

Do not begin with work stealing, timers, networking, and lock-free task state at once. Preserve the architectural center in layers and keep each invariant observable.

What are we preserving?

• a future runs only when polled;
• Pending stores a route back to its task;
• wake makes an idle task runnable without polling every dormant future;
• one task is never polled concurrently;
• readiness and scheduling are separate concerns;
• cancellation drops suspended state;
• blocking work does not occupy the executor thread.

Stage 1: a single-thread executor

Store Pin<Box<dyn Future<Output = ()>>> tasks in a map and runnable task IDs in VecDeque. Poll until the runnable queue is empty. At this stage, accept the allocation and trait object: clarity is the goal.

Stage 2: implement a task waker

Give each task an Arc handle back to a synchronized runnable queue. Implement Wake so it inserts the task ID only if it is not already queued. Create a future that returns Pending, saves the waker, and can be completed from another thread.

Verify that an idle task is not repeatedly polled and ten wake calls before the next poll produce one runnable entry.

Stage 3: add a typed join result

Wrap each submitted typed future in an erased Future<Output = ()> that sends T through a oneshot. Return JoinHandle<T> containing the receiver. This teaches the same “erase work, preserve results” boundary without Tokio’s raw vtable.

Stage 4: integrate one readiness source

Use a nonblocking Unix stream or mio::Poll. A Readable future registers its waker by token and returns Pending. The driver thread waits for OS events and wakes only matching tasks. The executor still decides when polling occurs.

Stage 5: park instead of spin

When no runnable task exists, block on the driver. Add an executor unpark token so spawning from another thread interrupts the OS wait.

Stage 6: add timers

Start with a binary heap ordered by deadline. Make the driver wait until the earlier of I/O or the next timer. Expired timers wake their tasks. Only after this works should you compare the heap with Tokio’s multi-level timing wheel.

Stage 7: cancellation and shutdown

Give a task explicit Idle, Queued, Running, Complete, and Cancelled states under a mutex first. Abort marks cancellation and queues the task so the executor can exclusively drop its future. Shutdown closes spawn admission, cancels owned tasks, and drains runnable references.

Stage 8: a blocking bridge

Send FnOnce() jobs to one dedicated blocking thread and return typed results by oneshot. Demonstrate that a one-second synchronous sleep no longer delays a ready async timer on the executor.

Only then: parallel workers

Add one local queue per worker, a synchronized injection queue, and stealing. Write the invariant that one task can have only one queued notification and only one poll owner. Use Loom before replacing mutex state with packed atomics.

Failure exercises

• Lose a stored waker and observe permanent Pending.
• Wake during a poll and verify the task is scheduled again afterward.
• Abort while idle and while running.
• Drop a typed join handle before completion.
• Spawn concurrently with shutdown.
• Block an executor worker and measure unrelated timer delay.
• Overflow one local queue and preserve every runnable task.

Compare with production Tokio

Tokio adds optimized raw task allocation and type erasure, atomic refcount and lifecycle state, current-thread/local/multi-thread schedulers, work stealing, cooperative budgets, Mio resource registration, a hierarchical timing wheel, async synchronization primitives, platform networking and processes, blocking pool management, feature gating, metrics, tracing hooks, panic isolation, careful shutdown, and extensive Loom coverage.

Your smaller executor is successful when you can narrate one task’s exact ownership and runnable state from spawn to Pending, wake, completion, join, cancellation, and shutdown.

Deno: Orientation

Deno is a useful concurrency study because it connects two asynchronous worlds: JavaScript promises inside V8 and Rust futures driven by an event loop. It also has to expose operating-system capabilities without allowing JavaScript to own Rust objects directly.

JavaScript API
    ↓ calls a generated op
Rust permission check → async filesystem/network future
    ↓ pending op driver
Tokio + operating system readiness
    ↓ completed op
V8 event-loop turn → resolve/reject Promise → run microtasks

Design thesis

Deno keeps each V8 isolate locally owned, exposes native capabilities as typed Rust ops, and returns async completions to JavaScript through an explicit event loop.

That sentence contains the important boundaries:

• A worker owns a V8 isolate and its JavaScript state.
• An op is a typed Rust function exposed to JavaScript.
• OpState stores runtime services such as permissions and the resource table.
• A resource ID, or rid, lets JavaScript refer to an owned Rust resource.
• The op driver holds heterogeneous pending Rust futures behind one runtime interface.
• The event loop decides when completed ops, timers, modules, messages, and promise microtasks are advanced.

What to preserve while reading

Do not begin with every web API or command-line flag. Preserve these five architectural facts:

1. JavaScript cannot directly borrow a Rust socket across an await.
2. Permission checks happen at the native capability boundary.
3. Pending I/O does not block the isolate’s operating-system thread.
4. V8 is re-entered on the isolate owner, not from arbitrary completion threads.
5. Long-lived native objects have explicit identity and cleanup.

The runtime crate assembles workers, libs/core owns the V8/op/event-loop machinery, and ext implements capabilities such as files, networking, HTTP, and web APIs. That separation is the first map to keep in your head.

Interactive JavaScript-to-Rust Map

Click a process and then each numbered step. The first path is the main trace for this study. The others show where concurrency, durable native ownership, and cancellation differ from ordinary promise execution.

Why Is It Designed This Way?

Why not let JavaScript call Rust objects directly?

V8 values obey garbage collection and isolate rules; Rust values obey ownership and lifetimes. An op is a narrow conversion boundary between those systems. Its generated glue validates JavaScript inputs, invokes typed Rust, and maps a typed result or error back into V8.

Why keep the isolate single-owner?

V8 execution and most isolate state are not ordinary shared concurrent data. Keeping one owner avoids putting locks around the JavaScript heap and makes a single event-loop turn coherent. Parallelism is introduced by separate workers and by native work outside the isolate, not by concurrently mutating one heap.

Why have an event loop if Tokio already schedules futures?

Tokio answers when Rust futures can make progress. Deno must additionally decide when to resolve JavaScript promises, run microtasks, advance modules, fire timers, report rejections, and determine whether the JavaScript program is still alive. Those are language-runtime policies, not generic executor policy.

Why use resource IDs?

A TCP stream may outlive the connect op that created it. JavaScript needs a stable handle, while Rust must keep owning the concrete stream. ResourceTable stores Rc<dyn Resource> under a small integer. Each later op asks for the expected concrete type; a missing or mismatched ID becomes an error.

This is intentional type erasure at one collection boundary:

JavaScript: rid 7
               ↓ lookup as TcpStreamResource
Rust table: Rc<dyn Resource> → Rc<TcpStreamResource>

Why check permissions inside ops?

The JavaScript wrapper is not a security boundary. Native code is the last point before a filesystem or network side effect. Checking there keeps permission enforcement adjacent to the capability and protects alternate internal callers of the same op path.

Why both Rc<RefCell<_>> and thread-safe types?

Not all concurrency is shared-memory parallelism. Isolate-local OpState can be Rc<RefCell<_>> because it is accessed on one local execution context. Network drivers, worker-control paths, and genuinely cross-thread services use the appropriate Send, atomics, locks, or channels. Choosing synchronization from actual ownership is better than making every type thread-safe by default.

Why does cancellation use a resource?

An AbortSignal exists in JavaScript while the pending operation exists in Rust. A temporary CancelHandle in the resource table bridges those lifetimes. Closing it can wake/cancel the Rust future; every exit path removes it so the bridge does not leak.

The broader lesson is that Deno is not “JavaScript running on Tokio.” It is a language runtime that uses Tokio beneath carefully controlled semantic boundaries.

One readTextFile, Fully Traced

Consider:

const text = await Deno.readTextFile("config.json");

1. The public JavaScript wrapper prepares cancellation

runtime/js/90_deno_ns.js exposes the function implemented in ext/fs/30_fs.js. The wrapper normalizes the path. If the caller supplied an AbortSignal, it creates a temporary cancellation resource and installs an abort handler that closes that resource.

The wrapper then awaits op_fs_read_file_text_async(path, cancelRid) inside a try/finally. Cleanup removes the listener and rechecks the signal even when the native op fails.

2. Generated op glue converts the call

The Rust function is annotated #[op2(stack_trace)]. The macro-generated glue converts the JavaScript string and optional small integer into:

pub async fn op_fs_read_file_text_async(
    state: Rc<RefCell<OpState>>,
    path: String,
    cancel_rid: Option<ResourceId>,
) -> Result<FastString, FsOpsError>

The public language boundary is dynamic, but the implementation immediately regains concrete Rust types.

3. Rust validates authority before I/O

Before awaiting, the op borrows OpState, clones the configured filesystem service, optionally retrieves the CancelHandle, and asks PermissionsContainer::check_open for read access.

Notice the short borrow scope. The RefCell borrow is not held across the file future’s .await. Owned/cloned values cross the suspension point instead.

4. The file future enters the op driver

The generated async-op path associates the future with an op ID and JavaScript promise ID. FuturesUnorderedDriver first polls it once. If it completes immediately, Deno can return without another event-loop trip. Otherwise the driver erases the concrete future into its arena and adds it to a FuturesUnordered submission set.

The future is pending, but the V8 isolate is free to do other work.

5. Tokio and the filesystem implementation make progress

The concrete FileSystem implementation determines whether work uses async OS I/O or an appropriate blocking bridge. The important contract at this layer is the Rust Future: it returns Pending with a registered Waker, then becomes ready after the underlying operation can progress.

If cancellation was configured, or_cancel(cancel_handle) races completion against the cancellation signal.

6. Completion returns to the Deno event loop

The driver pushes a completed op into its local completion queue and wakes the outer event-loop future. On a later poll, Deno removes the completion, maps its FastString or FsOpsError into a V8 value, and settles the matching promise. The event loop performs a microtask checkpoint, allowing the suspended JavaScript async function to resume.

7. Both layers clean up

Rust removes and closes the temporary cancellation resource on its completion path. JavaScript’s finally removes the abort listener and gives an observed abort precedence as the public API specifies.

JS wrapper owns AbortSignal listener
Rust op owns filesystem future
Op driver owns pending execution
Promise ID owns result correlation
Event loop owns re-entry into V8

No response thread reaches into V8. Completion is a wake-up plus a later, ordered event-loop action.

Async, Concurrency, and Parallelism

One isolate, many concurrent obligations

One V8 isolate executes JavaScript on one owner at a time. That does not mean only one operation can be in progress. Several file reads, socket operations, timers, dynamic imports, and messages can all be pending together.

FuturesUnorderedDriver is the central example. It holds heterogeneous async ops and yields whichever completes next. Completion order need not match submission order.

isolate submits A ─┐
isolate submits B ─┼─ pending op set ─→ B ready ─→ settle Promise B
isolate submits C ─┘                  → A ready ─→ settle Promise A

This is concurrency: multiple lifecycles overlap. It may happen even if only one thread is executing JavaScript.

Where Tokio fits

Deno’s worker exposes its event loop as a Rust future. Tokio polls that future, and lower-level async I/O registers wakers with the relevant driver. When an op completion, timer, worker-control event, or I/O readiness matters, a wake makes the outer future runnable again.

Tokio provides scheduling and readiness. Deno layers language semantics on top:

• dispatch completed ops;
• resolve or reject promises;
• perform V8 microtask checkpoints;
• advance module evaluation;
• process timers, messages, and rejection events;
• decide whether refed work keeps the program alive.

Event-loop phases are fairness and semantic policy

JsRuntime::poll_event_loop_inner advances explicit phases: timers, pending work, idle/prepare callbacks, I/O, check/immediates, and close callbacks. Microtasks run at carefully selected boundaries.

The code deliberately avoids draining unbounded I/O in one turn: under sustained readiness, doing so could starve timers and other work and damage tail latency. “Ready” therefore does not mean “consume everything before yielding.”

What is actually parallel?

• Native I/O can progress independently while the isolate executes JavaScript.
• Tokio’s runtime may use multiple OS threads for Send work.
• Blocking operations may run on a separate blocking pool.
• Web Workers create separate V8 isolates with their own event loops and can run JavaScript in parallel.

Two JavaScript callbacks in one isolate do not run simultaneously. A Web Worker does not share arbitrary mutable V8 values with its parent; communication goes through messages and control channels.

Local futures are a feature

The op driver uses Rc, RefCell, Cell, and Deno’s unsynchronized task support. These pending ops can retain isolate-local state and need not all be Send. The design avoids pretending isolate-local work may migrate freely between threads.

That yields a useful rule:

Make a future Send only when its ownership really may cross threads. Use a local executor boundary for futures tied to a single-threaded subsystem.

Backpressure is not one global queue

Deno has several distinct pressure points: pending ops, resource-specific buffers, worker message channels, HTTP bodies, and subprocess streams. A single “concurrency limit” would not express all of them. Each subsystem needs a limit matching the resource it protects.

The event loop’s liveness accounting is another kind of bound. Refed work keeps the program alive; unrefed work may make progress during an active turn but does not independently prevent exit.

The Standard Library as Architecture

Deno’s runtime code is sophisticated, but much of its architecture is visible through ordinary standard-library types.

Future, Poll, Context, and Waker

The entire bridge is built on Rust’s polling contract:

fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Output>

Pending means “I cannot finish now, and I arranged for a wake when that may change.” The Deno event loop itself follows this same interface, so Tokio does not need special knowledge of V8.

Rc<T> records isolate-local shared ownership

ResourceTable stores Rc<dyn Resource>, and async ops often receive Rc<RefCell<OpState>>. Rc makes a precise statement: several local owners may retain a value, but it is not being advertised as thread-safe.

A pending read can clone an Rc<TcpStreamResource>. Removing its rid prevents new lookups, while existing operations may still hold the resource. Its close() hook cancels those operations where required.

RefCell<T> gives short, checked mutation windows

An op borrows OpState to retrieve permissions, services, and resources, then drops the borrow before awaiting. RefCell moves enforcement from compile time to runtime, but it does not remove the borrowing rule. Holding a mutable borrow across arbitrary re-entrant JavaScript or .await points would still be a bug.

BTreeMap<ResourceId, Rc<dyn Resource>>

This standard collection is a capability registry. The key is stable enough to cross the JavaScript boundary; the erased value lets files, sockets, subprocess pipes, and cancellation handles coexist. Generic get::<T> restores the expected concrete type or returns BadResourceId.

VecDeque and FuturesUnordered

The op driver uses a completion VecDeque because completed items are locally queued and removed from the front. Pending operations use FuturesUnordered because readiness order, not insertion order, should determine which result is processed next.

These structures solve different problems:

FuturesUnordered: which pending future is ready?
VecDeque:         which already-ready completion is dispatched next?

Option makes lifecycle state explicit

An optional cancellation ID, optional inspector state, optional pending timer, or an enum like the op driver’s MaybeTask makes absence and transitions visible. Production async code frequently consists of making these lifecycle states precise enough that cleanup can cover every path.

The lesson is not “use more locks.” Deno first asks who owns the isolate, who owns a resource, and which values cross a suspension or thread boundary. The chosen standard type follows from those answers.

Generics, Macros, and Type Erasure

Deno uses generics where concrete types improve implementation safety, then erases them where heterogeneous runtime collections require one representation.

Ops recover static types at a dynamic boundary

JavaScript supplies dynamically typed values. An #[op2] declaration describes the Rust contract with String, Option<ResourceId>, borrowed buffers, serde values, and Result<R, E>. Generated glue performs conversion and maps errors.

This keeps repetitive V8 conversion machinery out of each implementation while leaving the native body ordinary typed Rust.

The op driver is generic over result mapping

OpDriver<C: OpMappingContext> is generic over the environment into which an op result is mapped. Production uses V8OpMappingContext; tests can use a simpler context without constructing a full V8 runtime.

Its submission methods remain generic over each future’s output and error:

fn submit_op_fallible<R, E, const LAZY: bool, const DEFERRED: bool>(
    op: impl Future<Output = Result<R, E>>,
    rv_map: C::MappingFn<R>,
)

The concrete mapping function travels with the future. At the shared pending-op collection, the implementation erases that future and mapping information into a uniform allocation. Completion later restores the correct operation.

Resource lookup combines trait objects and generics

The table stores:

BTreeMap<ResourceId, Rc<dyn Resource>>

Insertion and retrieval are generic:

add<T: Resource>(value: T) -> ResourceId
get<T: Resource>(rid: ResourceId) -> Result<Rc<T>, ResourceError>

The trait object answers “how can unlike resources share one map?” The generic method answers “which concrete resource does this op require?” This is a strong example of using both forms of polymorphism rather than treating them as rivals.

Extension composition hides implementation families

Runtime assembly combines many extensions, ops, state initializers, and JavaScript modules. Generic builders and trait bounds preserve concrete service types while assembling them; erased op declarations give the runtime one list it can register with V8.

Public simplicity versus internal complexity

The caller sees:

const conn = await Deno.connect({ hostname, port });

Internally, that crosses generated conversion, permission policy, configurable services, error classification, async future erasure, promise correlation, and resource type erasure. The complexity belongs behind the boundary because it exists to preserve a simple, safe capability.

Use this heuristic in your own APIs:

• Keep concrete types and generics along a single compile-time composition path.
• Introduce a trait object where unlike values must share storage or runtime dispatch.
• Erase only what the shared boundary requires; preserve typed results for the caller whenever possible.

Errors, Cancellation, Resources, and Shutdown

Errors cross an intentional mapping boundary

An op commonly returns Result<R, E> where E implements Deno’s JavaScript error classification contract. The generated/runtime mapping chooses the JavaScript error class, message, and stack behavior. A filesystem error is not merely formatted text; callers should receive a useful JavaScript exception.

The runtime also distinguishes larger failure scopes:

• one op rejects one promise;
• an unhandled promise rejection follows language policy;
• module evaluation can fail;
• a worker can report a terminal error to its host;
• an isolate/runtime failure can end the process-level operation.

Permission denial is an ordinary operation failure

Filesystem and network ops check permission immediately before capability use. For TCP connection, Deno checks the requested hostname and then checks resolved IP candidates as well. This prevents a name-resolution path from bypassing network policy.

Cancellation is cooperative and cross-layer

For readTextFile({ signal }):

1. JavaScript creates a CancelHandle resource.
2. Rust retrieves it before the relevant await.
3. or_cancel observes either operation completion or cancellation.
4. Closing the resource signals the pending operation.
5. Rust removes the temporary resource on every exit path.
6. JavaScript removes its listener in finally.

Cancellation does not preempt arbitrary Rust instructions. It is observed at a future boundary designed to handle it.

Resource close is stronger than map removal

Removing rid → resource stops future JavaScript lookup. It may not destroy the resource immediately because pending ops can hold cloned Rcs. The Resource trait’s close method lets a type cancel outstanding work as part of closure.

TcpStreamResource explicitly cancels read/write operations. The source even documents a subtle method-resolution failure that once made close call the trait’s no-op default instead of the concrete cancellation behavior. This is a good production lesson: cleanup paths deserve focused tests because code that looks like ownership release may not yet release the operating-system resource.

Event-loop completion is a liveness proof

The runtime tracks pending ops, refed ops, timers, module evaluation, dynamic imports, background tasks, promise events, external operations, and live native handles. Returning Ready(Ok(())) means none of the work that should keep the program alive remains.

That is stronger than “the queue was empty during this poll.”

Worker shutdown closes admission and wakes ownership

A Web Worker has a control path and termination waker. Its event-loop poll first checks termination, registers for later termination, and delegates ordinary progress to JsRuntime. The host receives structured close or terminal-error events.

The general pattern is:

record stop intent → wake blocked event loop → stop admitting work
→ cancel/close owned resources → report terminal outcome

Reliable async systems represent shutdown as an ownership protocol, not merely as a boolean checked occasionally.

Build a Smaller Deno Runtime Boundary

Do not begin by embedding V8. First rebuild the architectural center with a small command language and a local event loop.

What are we preserving?

• a dynamic caller-facing command becomes a typed native operation;
• native services and permissions live in explicit runtime state;
• several operations can be pending concurrently;
• results return through correlation IDs on the event-loop owner;
• long-lived native objects live behind typed resource IDs;
• cancellation and close wake pending operations;
• a second worker owns separate state and communicates by messages.

Suggested construction sequence

1. Define one concrete operation

Accept Command::ReadText { path }. Validate it, check a simple ReadPermissions value, and return Result<String, OpError>.

2. Separate submission from completion

Assign a PromiseId, put the typed read future into FuturesUnordered, and store a callback or oneshot sender that knows how to deliver that ID’s result. Drive the set from a poll_fn-based event loop.

3. Add a resource table

Implement:

type ResourceId = u32;

trait Resource: Any {
    fn close(&self) {}
}

Store unlike resources behind Rc<dyn Resource>. Provide generic add<T> and get<T> methods with checked downcasting. Add an in-memory duplex resource before using a real socket.

4. Add cancellation

Create a cancellation resource for one read, race it against a delay, and prove that every success, error, abort, and dropped-caller path removes the temporary entry.

5. Make event-loop liveness explicit

Track refed pending operations separately from unrefed background work. Write tests showing when the event loop may exit and when it must stay alive.

6. Add a worker boundary

Run a second local runtime with independent state. Exchange owned messages over bounded channels. Do not share the resource table. This demonstrates why parallel language workers need a message boundary.

7. Compare with production Deno

Only now inspect what Deno adds: V8 handle safety, op2 code generation, zero-copy buffers, many extension families, JavaScript error classes, permission prompts, module loading, inspector integration, web compatibility, platform-specific I/O, and carefully tuned event-loop phase semantics.

Tests that matter more than features

• Two ops complete out of submission order but reach the right callers.
• A permission failure performs no I/O.
• Looking up a resource as the wrong type returns an error.
• Closing a resource wakes a pending read.
• Dropping one result observer does not stop unrelated operations.
• Unrefed work does not keep an otherwise idle loop alive.
• Worker termination wakes a parked event loop.

Rebuild the op/resource/event-loop relationship, not a toy imitation of Deno’s public JavaScript syntax.

Bevy: Orientation

Bevy is a data-oriented game engine built from ordinary Rust crates. Its most important architectural unit is the entity-component-system world: entities identify things, components hold typed data, and systems declare how they access that data.

runner → App::update → ordered schedules
                         ↓
                dependency + access graph
                         ↓
              compatible systems in parallel
                         ↓
                 apply deferred Commands
                         ↓
             extract into render sub-application

Design thesis

Bevy turns typed function parameters into a runtime data-access graph, then uses that graph to run independent systems in parallel without allowing conflicting access to the same world data.

The main crates to retain are bevy_app for lifecycle and plugins, bevy_ecs for worlds and scheduling, bevy_tasks for execution pools, bevy_asset for background loading, and bevy_render for extraction and GPU work.

This study follows one frame. It does not inventory every engine subsystem.

Interactive Frame Map

Why Is It Designed This Way?

Why ECS instead of a tree of game objects?

Components group data by capability rather than inheritance. A movement system can ask for every (Velocity, &mut Transform) without knowing which concrete game-object classes exist. This makes behavior composable and exposes the exact data access the scheduler needs.

Why infer parallelism from parameters?

Requiring users to spawn every task would make world safety their problem. Bevy converts Res<T>, ResMut<T>, Query<&T>, and Query<&mut T> into access metadata. Two systems may overlap only when those sets are compatible.

Why defer structural changes?

Spawning an entity or adding a component can move archetype storage and invalidate iteration assumptions. Commands records mutations while systems run against a stable world, then applies them at a synchronization point.

Why retain explicit ordering?

Non-conflicting does not mean semantically independent. Schedules, sets, before, after, and chained configurations let applications state causal requirements that data access alone cannot infer.

Why a separate render world?

Simulation data is broad and mutable; rendering needs a prepared snapshot of a smaller set of values. Extraction creates a boundary where simulation can advance independently of later render preparation and GPU submission. Bevy can also pipeline these sub-applications.

Why several task pools?

Frame-critical compute, long-running async compute, and I/O have different latency and saturation characteristics. Separate pools prevent background loading from consuming the same scheduling budget as systems that must finish this frame.

One Frame, Fully Traced

1. The runner owns the outer loop

App::run transfers the built application into a runner. A windowed build normally installs the winit runner; a headless schedule runner can call App::update itself. The runner, not App, defines when another frame begins.

2. Updating advances every sub-application

App::update delegates to SubApps::update. The main world runs first. Each configured sub-app can extract from the main world and run its own schedule.

3. Main expands into ordered schedules

Main::run_main runs startup schedules once, then First, PreUpdate, the fixed-step loop, Update, scene spawning, PostUpdate, and Last. These are coarse semantic barriers; each inner schedule may run many systems in parallel.

4. The schedule prepares an executable graph

Systems have explicit dependency edges and access sets generated from their parameters. MultiThreadedExecutor::init precomputes conflicting-system bitsets. A writer conflicts with readers or writers of the same component or resource; compatible readers do not.

5. Ready compatible systems enter the compute pool

The executor tracks remaining dependencies, ready, running, completed, skipped, and unapplied systems with bitsets. It scopes tasks on ComputeTaskPool and starts systems whose dependencies are satisfied and whose access does not conflict with currently running work. Non-Send and exclusive systems remain on constrained execution paths.

6. Systems receive checked views

A function system’s SystemParam state constructs values such as Query, Res, local state, messages, and Commands. The system sees only the access it declared. Unsafe world access stays inside the scheduler/parameter machinery whose compatibility checks uphold the public borrowing contract.

7. Deferred buffers become world mutations

Systems can enqueue spawns, despawns, component inserts, and custom commands without taking exclusive world access immediately. ApplyDeferred or the schedule’s final apply drains the relevant command queues sequentially into &mut World.

8. Rendering extracts a snapshot

The render sub-app temporarily exposes the main world during ExtractSchedule. Extraction systems copy or move render-relevant components and resources into the render world. Later render schedules prepare assets, queue work, and submit GPU commands without treating the simulation world as GPU-owned state.

The critical invariant is: parallel systems borrow stable storage; structural mutation happens only after those borrows have ended.

Concurrency, Parallelism, and Frame Budgets

Bevy has three distinct forms of concurrency.

Parallel schedule execution

The multithreaded executor combines two graphs:

• dependency edges say what must finish first;
• component/resource access says what may safely overlap.

Query<&Position> can run beside another reader. Query<&mut Position> conflicts with both. Disjoint filters may allow more precise compatibility. Ready systems execute on ComputeTaskPool; completion events unlock dependents.

Parallel iteration inside one system

A system can use parallel query iteration when one system owns the relevant access but the entity set can be partitioned safely. This is nested data parallelism, distinct from running several systems.

Background async work

Assets use IoTaskPool; longer async computation has AsyncComputeTaskPool. These tasks may outlive a frame and return owned results or update thread-safe service state. They must not retain ordinary world borrows across suspension.

Barriers are deliberate

Exclusive systems, non-Send parameters, dependencies, and ApplyDeferred reduce parallelism because they protect real invariants. A faster schedule that observes half-applied entity structure would be incorrect.

The scheduler therefore optimizes a constrained problem:

maximize ready compatible work
subject to dependency, access, thread-affinity, and mutation barriers

Frame latency also matters more than aggregate throughput. An unbounded system or background-result integration step can still miss the frame budget even if all memory access is safe.

The Standard Library as Architecture

• TypeId identifies component, resource, system, and schedule types at runtime.
• HashMap and dense/sparse tables connect those IDs to storage and metadata.
• FixedBitSet makes dependency, readiness, conflict, and completion sets cheap to combine.
• Mutex protects executor state shared by scoped worker tasks, while the world itself is accessed through scheduler-proven disjointness.
• Arc makes services such as AssetServer cheaply cloneable across background tasks.
• PhantomData preserves generic type/lifetime relationships in zero-sized parameter and handle machinery.
• Any and downcasting support heterogeneous registries at deliberate runtime boundaries.
• Result lets systems and commands participate in configurable error policy.

The striking point is that &T versus &mut T is not just local syntax. Bevy turns that standard borrowing distinction into global scheduling metadata.

Generics, System Parameters, and Type Erasure

Bevy’s pleasant API is powered by a conversion pipeline:

ordinary function
→ IntoSystem
→ FunctionSystem<Marker, F>
→ SystemParam tuple state
→ boxed ScheduleSystem

SystemParam uses generic associated types to distinguish long-lived parameter state from the short-lived item borrowed for one run. Query<'w, 's, D, F> is generic over selected data and filtering policy; D also determines whether access is read-only.

Bundles recursively describe groups of components. Plugins are generic composition at application construction time. Assets retain Handle<A> and Assets<A> so an image handle cannot accidentally retrieve a mesh.

At the schedule boundary, unlike concrete function and parameter tuples must share a collection. Bevy erases them behind dyn System, retaining virtual operations for access metadata, initialization, running, deferred application, and type identity.

This is the recurring pattern:

Use generics to derive exact behavior and safety; erase the final system only when heterogeneous runtime scheduling requires it.

Errors, Change Detection, and Lifecycle

System and command errors flow through configurable handlers. The fallback can panic, but applications can install policy appropriate to a game, editor, or server. Executor panics are captured from worker tasks and resumed on the coordinating thread after scoped work is reconciled.

Change detection uses ticks attached to component mutations and system runs. Changed<T> means “changed relative to this system’s last observation,” not a global event queue. Tick aging and wraparound handling are therefore part of correctness.

Entity identifiers include generations so stale handles do not silently refer to newly allocated entities. Typed asset handles similarly separate stable identity from asynchronous availability; failed loads become observable load state and events.

Deferred commands create an explicit failure boundary: the producing system may finish successfully while applying a queued command can fail later. Error context must preserve which system and command created the obligation.

Application exit is data-driven through AppExit, but cleanup still belongs to the runner, sub-apps, task pools, windows, render resources, and platform event loop. A coherent final frame requires stopping new work, applying or discarding known deferred work according to policy, and releasing GPU/OS ownership.

Build a Smaller Bevy Scheduler

Preserve the architectural center, not Bevy’s game API.

1. Create a World with typed resource storage and two component columns.
2. Adapt ordinary functions into a System trait with initialize/run methods.
3. Have each parameter register read and write TypeId access.
4. Build a dependency DAG and reject cycles.
5. Run ready, access-compatible systems with scoped worker tasks.
6. Add a per-system command queue and one explicit apply barrier.
7. Add a non-Send system that must execute on the coordinator thread.
8. Add an AssetServer-like background task returning an owned result.
9. Copy render-relevant state into a second World and process it independently.

Tests should prove that readers overlap, writers exclude, dependencies override compatibility, commands cannot invalidate live queries, a panic is surfaced, and stale entity generations are rejected.

Production Bevy additionally supplies archetypes, sparse storage, query caching, change ticks, observers, reflection, scenes, platform runners, asset pipelines, render graphs, GPU lifetime management, and years of scheduler optimization.

godot-rust: Orientation

godot-rust exposes Godot 4’s GDExtension API as an idiomatic Rust library. Unlike Bevy, Rust does not own the engine loop or object model. Godot creates, calls, reference-counts, and destroys objects; the binding must preserve Rust’s rules across that foreign lifecycle.

Godot loads extension → register generated class metadata
→ Godot invokes C callback → recover Rust instance
→ dynamic bind guard → typed method → convert return to Godot

Design thesis

godot-rust concentrates unsafe engine interoperation behind generated glue, typed object handles, runtime borrow guards, and explicit thread/lifecycle checks so application code can look like ordinary Rust.

Study godot-ffi, godot-codegen, godot-macros, godot-cell, and godot-core as successive layers rather than one undifferentiated binding.

Interactive Engine-Boundary Map

Why Is It Designed This Way?

Why does Gd<T> exist instead of Box<T>?

Godot owns object identity and may use manual or reference-counted memory. Gd<T> is a typed view of an engine object pointer whose clone/drop behavior follows the class’s memory strategy. A Rust box would falsely claim sole allocation and destruction authority.

Why are Rust user classes dynamically borrowed?

Godot and GDScript can re-enter Rust through signals and virtual callbacks. Compile-time lifetimes cannot describe all foreign call sequences. bind and bind_mut therefore return guards enforcing shared/exclusive access at runtime, similar to RefCell but attached to engine object storage.

Why macros and code generation?

Godot publishes a large versioned API description. Generated engine classes and procedural macros keep function signatures, inheritance markers, registration, argument conversion, and virtual trampolines synchronized. Handwritten wrappers would multiply unsafe boilerplate and drift.

Why is inheritance represented with traits and composition?

Rust has no class inheritance. GodotClass::Base, Inherits<T>, generated deref behavior, and a stored Base<T> preserve the useful relationships while keeping the Rust user value structurally explicit.

Why are most objects main-thread-bound?

Godot’s scene tree and object lifecycle are not generally thread-safe. The safe API refuses to mark Gd<T> freely Send/Sync. Reviewed value operations and explicit thread-safe callables opt in separately.

Why catch panics at callbacks?

Rust unwinding across an extern "C" boundary is invalid. Trampolines catch and report panics, clean task/callback state, and return through the engine ABI.

One ready Callback, Fully Traced

1. Macros describe the user class

#[derive(GodotClass)] generates GodotClass, base/inheritance relationships, configuration, construction hooks, and registration shards. A #[godot_api] implementation of an engine interface records virtual methods such as ready.

2. Extension initialization registers the class

Godot loads the dynamic library and calls initialization for Core, Servers, Scene, and Editor levels. The binding loads compatible method tables and auto_register_classes gathers distributed registration shards. It supplies Godot with create, free, reference, notification, and virtual lookup callbacks.

3. Godot asks for the virtual trampoline

When the scene lifecycle reaches _ready, Godot uses the registered virtual function pointer. Generated code chooses the callback for this class/API version and receives raw engine pointers and argument slots.

4. The callback validates and borrows the Rust instance

The trampoline recovers InstanceStorage<T> associated with the Godot object. A mutable virtual acquires the same exclusive dynamic borrow represented by GdMut<T>. Re-entrant shared or mutable access fails instead of creating aliased &mut T.

5. Typed Rust code runs

Arguments have already crossed GodotFfi/conversion traits. The user method receives &mut self; its Base<Node> provides controlled access to inherited engine behavior. OnReady fields are initialized before the callback and validated for required editor assignments.

6. The result returns through the ABI

Return conversion writes the correct Godot representation. The borrow guard is dropped before control returns. The trampoline catches a panic so unwinding cannot cross into C++, reports context, and supplies ABI-appropriate failure behavior.

The engine owns invocation; the binding owns validation and the temporary Rust borrow; user code owns only its class state.

Concurrency, Async, and Thread Affinity

Godot may use internal threads, but ordinary scene-tree objects are main-thread objects. Gd<T> deliberately does not promise unrestricted cross-thread use. The binding exposes a reviewed subset of thread-safe value lifecycle functions and requires Send + Sync + 'static for explicitly thread-safe closures.

godot::task::spawn is a local async executor integrated with the engine. It accepts non-Send futures, stores them on the main thread, and uses a custom Waker to arrange another main-thread poll. Signal futures can wake from other threads, but actual object-facing polling is redirected to the owner thread.

A task must not hold GdRef or GdMut across .await. The suspended task could be re-entered by _process, a signal, or GDScript, and the stale guard would block valid access. Strict safeguards detect and warn at the suspension point.

Parallel work should operate on owned, thread-safe Rust data, then schedule a small result integration back on the engine thread. This is the same single-writer principle seen in UI applications, imposed here by a foreign engine rather than a Rust event loop.

The Standard Library as Architecture

• Raw pointers represent the ABI fact that Godot supplies object and value addresses; safe wrappers validate before exposing behavior.
• PhantomData attaches class, ownership, and non-Send meaning without changing the pointer layout.
• TypeId and HashMap support class and dynamic-trait registries.
• AtomicBool and atomics record initialization/hot-reload state shared with callbacks.
• Cell stores isolate/main-thread lifecycle state; mutex-backed globals are used for registration paths and fail fast on unexpected concurrency.
• catch_unwind prevents a panic from leaving a Rust-controlled trampoline.
• Future, Poll, Waker, and thread-local storage implement the local async bridge.
• guard Drop releases dynamic borrows even on early returns and panics.

The standard library supplies mechanisms; the binding’s real work is assigning them semantics matching Godot’s ownership contract.

Generics, Generated APIs, and Runtime Checks

Gd<T: GodotClass> statically carries the engine class. Bounds such as T: Inherits<Node> admit only operations valid for that class hierarchy. Associated types on GodotClass select Base, declaration domain, and memory strategy, allowing clone/drop behavior to differ for manually managed and reference-counted objects.

Conversion traits describe whether a Rust value travels by value or reference, its Godot Variant representation, and its low-level ABI layout. Generated methods use these traits rather than dynamically guessing signatures.

Procedural macros generate implementations for user-defined classes and methods; build-time code generation creates the enormous engine API. At runtime, registries erase class registration functions and Godot supplies untyped pointers. Exact generic types are recovered only after class, liveness, and borrow validation.

This is a boundary-heavy form of generics: the type system prevents invalid calls before FFI, while runtime checks defend facts controlled by C++ after compilation.

Errors, Panics, Liveness, and Hot Reload

There are several independent failure domains:

• conversion can reject an incompatible Variant or argument shape;
• a Gd<T> can reference an object Godot already freed;
• dynamic borrowing can detect re-entrant aliasing;
• a method may be unavailable for the loaded engine version;
• initialization metadata can be incompatible with the engine binary;
• user callbacks and async polls can panic.

The library checks object validity and runtime type before dangerous pointer use. Reference-counted and manual-memory classes follow different lifecycle rules, and explicit free rejects active borrows. Placeholder instances in the editor are represented as a distinct limitation rather than pretending Rust state exists.

Initialization and deinitialization occur at engine-defined levels. Class registries are loaded and unloaded accordingly, with special handling for editor hot reload. Fatal startup incompatibility is reported or terminates in a context-sensitive way rather than continuing into undefined behavior.

The deepest lesson is that a safe wrapper cannot make foreign lifecycle facts compile-time facts. It combines static restrictions with fast, contextual runtime validation at every point the engine can invalidate an assumption.

Build a Smaller Godot Binding

Use a tiny fake C engine rather than Godot itself.

1. Export C functions that create, call, retain, and destroy opaque objects.
2. Wrap the pointer as EnginePtr<T> with an explicit ownership strategy.
3. Add a generation/liveness registry and checked downcasting.
4. Store a Rust user value behind a runtime borrow cell; expose bind guards.
5. Generate one callback trampoline that converts arguments and catches panics.
6. Register a class and virtual callback through a static descriptor.
7. Add a main-thread-only local future awakened by a foreign callback.
8. Demonstrate and reject a guard held across suspension.
9. Unregister classes and invalidate handles during simulated reload.

Tests should cover double free, dead handles, wrong dynamic type, re-entrant mutable borrow, callback panic, thread misuse, and unload with live objects.

Production godot-rust adds the full versioned Godot API, procedural macros, inheritance metadata, signals, properties, RPCs, editor integration, generated documentation, platform ABI differences, and extensive integration/sanitizer testing.

Quinn: Orientation

Quinn is a pure-Rust QUIC implementation split into three layers: quinn-proto is a deterministic protocol state machine with no I/O, quinn-udp handles efficient platform datagrams, and quinn turns the state machine into futures and streams for Tokio or smol.

UDP readiness → EndpointDriver → proto::Endpoint
→ ConnectionDriver → proto::Connection
→ stream event → wake exactly that application future

Design thesis

Quinn isolates protocol decisions from asynchronous I/O, then uses explicit drivers and targeted wakers to connect one UDP socket to many independent connections and streams.

Interactive QUIC Map

Why Is It Designed This Way?

Why separate quinn-proto from sockets?

The protocol is a function of datagrams, time, configuration, and prior state. Keeping it I/O-free enables deterministic simulated-time tests, fuzzing, custom event loops, and reasoning about loss without real sleeps or packet timing.

Why one endpoint driver and many connection drivers?

One UDP socket carries packets for many QUIC connections. The endpoint owns routing and stateless responses; each connection owns encryption, streams, acknowledgement, loss, congestion, and timers. Separate drivers reflect that ownership while channels carry protocol events between them.

Why individual stream wakers?

A packet may make one stream readable without helping hundreds of others. Maps keyed by StreamId wake only the blocked reader or writer affected by a protocol event, preventing a thundering herd.

Why abstract the runtime?

QUIC needs spawn, clock, timer, UDP receive, and UDP send—not Tokio specifically. Small traits let Tokio and smol supply those capabilities and let tests control time.

Why explicit loop budgets?

A busy UDP socket or connection can remain continuously ready. Datagram-count and time bounds force driver polls to yield so unrelated tasks retain latency.

Why does dropping a handle not instantly forget protocol state?

QUIC must send close/reset frames, release connection IDs, and drain peer-visible state. Drop triggers protocol actions; endpoint wait_idle observes when all connections have actually drained.

One Stream Read, Fully Traced

1. Socket readiness wakes the endpoint

EndpointDriver polls AsyncUdpSocket::poll_recv. The runtime-specific socket registers its waker with the reactor when no datagram is available.

2. The endpoint routes without owning connection semantics

Received bytes enter proto::Endpoint::handle. Connection IDs select an existing ConnectionHandle, produce a new incoming attempt, or cause a stateless response. An existing packet becomes ConnectionEvent::Proto on that connection’s channel.

3. The connection driver advances pure state

ConnectionDriver drains endpoint events into proto::Connection, processes the packet, advances cryptography/acknowledgement/flow-control state, and polls protocol events. StreamEvent::Readable { id } identifies the affected stream.

4. Exactly one blocked reader is woken

The high-level connection state stores blocked reader wakers by StreamId. Forwarding the readable event removes and wakes the matching entry.

5. The application future reads buffered chunks

RecvStream::poll_read_generic locks connection state, obtains the protocol receive stream, and consumes ordered or unordered chunks. Finalizing the chunk view returns flow-control credit; if that requires a control frame, it wakes the connection driver for transmission.

6. Backpressure closes the loop

If no bytes are available, the reader stores its current waker and returns Pending. If bytes are consumed, QUIC can advertise more credit. Thus slow application consumption constrains the peer through protocol flow control rather than an unbounded local queue.

Async, Multiplexing, and Fairness

One endpoint may serve many connections, and each connection may contain many bidirectional and unidirectional streams. These lifecycles overlap without one task per packet.

Endpoint and connection state are mutex-protected because application futures and runtime drivers may run on different worker threads. Critical sections call the synchronous protocol core, update waker maps, and leave quickly; .await does not occur while holding the lock.

Stream flow control bounds unconsumed bytes per stream and connection. QUIC congestion control separately bounds bytes in flight on the network. Socket buffer capacity, stream credit, connection credit, and congestion windows are different budgets and must not be collapsed into one “concurrency limit.”

The drivers cap receive/event/transmit work per poll. A self-wake requests another turn when work remains. This preserves throughput while giving the executor a scheduling boundary.

Parallelism comes from runtime tasks on several threads; protocol ordering remains serialized per connection under its state lock.

The Standard Library as Architecture

• Arc gives endpoint and connection handles shared ownership independent of any one future.
• Mutex serializes each protocol state machine while allowing handles on many executor threads.
• AtomicUsize distinguishes public live handles from the driver itself.
• HashMap<StreamId, Waker> correlates readiness with exactly one blocked stream operation.
• Option<Waker> represents at most one driver notification claim.
• Pin<Box<dyn Future>> and trait objects erase runtime-specific timers and sockets at the narrow integration boundary.
• Bytes shares immutable packet/stream storage cheaply.
• enums preserve transport, application, reset, closed, 0-RTT, and local-close errors as separate states.

The types make the protocol’s ownership graph visible: socket → endpoint → connection → stream, with wake paths traveling back upward only when needed.

Generics, Traits, and Runtime Independence

Most protocol logic uses concrete types and generics: packet number spaces, stream directions, crypto sessions, congestion controllers, and read helpers retain exact compile-time structure.

The runtime boundary uses trait objects deliberately:

Arc<dyn Runtime>
Box<dyn AsyncUdpSocket>
Pin<Box<dyn UdpSender>>
Pin<Box<dyn AsyncTimer>>

The endpoint must choose an executor/socket implementation at runtime and store it uniformly. The cost is insignificant beside network I/O, and it prevents runtime generics from infecting every public connection and stream type.

Read and write polling helpers remain generic over closures and output shapes, letting one checked state transition power several ergonomic methods without virtual dispatch.

This is good boundary placement: erase platform scheduling; keep protocol data and application results typed.

Errors, Cancellation, Timers, and Drain

Quinn distinguishes connection errors from stream-local reset, peer stop, closed-stream misuse, and rejected 0-RTT. This matters because retrying a write after flow-control blocking is normal, while retrying rejected early data may duplicate application effects.

Dropping an unfinished send stream resets it; dropping a receive stream stops it, unless the operation already reached a terminal state. Async convenience methods document cancellation safety: a cancelled write_all may already have accepted a prefix, so callers cannot assume transactional behavior.

The connection driver maintains the next protocol timeout using an abstract timer and checks the runtime clock directly on each poll. Loss recovery, handshake expiry, keep-alive, and idle timeout therefore advance through the same deterministic core as packet events.

Termination stores one ConnectionError, wakes every blocked class of waiter, and reports a drained event to the endpoint. Endpoint::close stops new work; wait_idle waits until the connection routing set is empty. That is graceful shutdown as a protocol-obligation proof, not merely socket drop.

Build a Smaller Quinn

1. Build a deterministic Connection state machine accepting Datagram and Timeout events and yielding Transmit and StreamEvent values.
2. Test it with a simulated clock and two in-memory endpoints.
3. Add an async UDP driver that feeds datagrams into the same core.
4. Route connection IDs from one socket to several connection states.
5. Expose RecvStream and SendStream futures with per-stream wakers.
6. Add stream and connection flow-control credit.
7. Add retransmission deadlines and a simple congestion window.
8. Bound work per driver poll and test that a hot connection cannot starve one quiet connection.
9. Implement close, reset, drop behavior, and endpoint drain.

Production Quinn additionally supplies the full QUIC/TLS specifications, packet protection, migration, ECN, MTU discovery, 0-RTT, datagrams, batching, platform socket tuning, runtime adapters, fuzzing, and extensive simulations.

DataFusion: Orientation

Apache DataFusion is an extensible query engine. It accepts SQL or a DataFrame API, produces an immutable logical plan, rewrites it, chooses a physical plan, and pulls Arrow RecordBatch values through a partitioned execution graph.

SQL -> LogicalPlan -> optimized LogicalPlan -> ExecutionPlan
    -> one stream per partition -> RecordBatch consumer

Design thesis

DataFusion separates query meaning from execution strategy, then represents execution as poll-driven, partitioned batch streams with explicit memory reservations for operators that retain large state.

Interactive Query-Engine Map

Use the tabs to follow one query, partition fan-out, memory pressure, and cancellation. Select a step to see what it owns, why that boundary exists, and where to read the pinned source.

Why Is It Designed This Way?

Why separate logical and physical plans?

A logical plan says what a query means: scan, filter, join, aggregate. A physical plan says how to execute it: which join algorithm, ordering, and partitioning. Keeping them separate lets semantic rewrites happen before machine and data-layout decisions.

Why return streams instead of completed tables?

An operator can produce a batch as soon as its inputs make progress. The consumer controls the pace by polling, intermediate results need not all be materialized, and dropping the output stream provides a natural cancellation boundary. collect is merely a terminal convenience that deliberately buffers.

Why columnar RecordBatch values?

Arrow arrays amortize dispatch across many values, improve cache locality, and form an interoperability boundary with file formats and other analytical systems. DataFusion moves batches between operators rather than Rust structs for individual rows.

Why are plan nodes trait objects?

The shape and concrete operators of a query are known only after planning. Arc<dyn ExecutionPlan> gives heterogeneous, shared plan trees and a public extension seam. Inside operators, generics still specialize reusable machinery where the concrete type is useful.

Why explicit memory reservations?

Rust prevents memory unsafety, not out-of-memory termination. Hash joins, aggregates, and sorts may retain input-proportional state, so they reserve bytes before growth and either spill or return an error when the pool refuses. Reservation drop returns accounting through RAII.

Why does async not mean every calculation is spawned?

Most operators are streams polled by their downstream consumer. Async is used where progress genuinely waits—object storage, repartition channels, spawned producers, or blocking bridges. Partitioning supplies parallel work; spawning every expression would add scheduling overhead without creating useful independence.

One SQL Query, Fully Traced

Consider SELECT city, count(*) FROM trips WHERE fare > 20 GROUP BY city.

1. SessionContext::sql snapshots session state and asks it to create a logical plan. Parsing yields a SQL AST; SqlToRel resolves tables, columns, functions, and types into DataFusion expressions and plan nodes.
2. The analyzer checks and coerces the plan. Logical optimizer rules can push the filter toward the scan, prune columns, and simplify expressions without choosing threads or file partitions.
3. The query planner maps logical nodes to Arc<dyn ExecutionPlan> nodes. A physical optimizer then enforces distribution and ordering requirements and may introduce repartition or coalesce operators.
4. Execution asks the root for an output stream. For several root partitions, callers can keep separate streams or place a coalescing node above them.
5. Polling the root recursively polls its children. The scan obtains a batch; the filter computes a selection mask; the partial aggregate updates per-partition state; a repartition boundary redistributes rows; the final aggregate emits grouped batches.
6. collect drains the stream into Vec<RecordBatch>. A streaming caller can instead process each batch and release it before requesting the next.

Nothing “runs the whole plan” when it is constructed. Plan creation describes the graph; polling an output stream drives the graph.

Concurrency, Async, and Backpressure

DataFusion expresses potential parallelism with partitions. An ExecutionPlan advertises output partitioning, and execute(partition, TaskContext) creates one SendableRecordBatchStream for that partition. Independent streams may be driven on multiple Tokio workers.

Within an ordinary operator chain, downstream polling supplies natural backpressure: if nobody polls the root, it stops polling its input. At an explicit concurrency boundary such as repartitioning, bounded channels connect producer tasks to output partitions. A full channel makes the producer await, carrying pressure upstream.

collect_partitioned demonstrates deliberate task parallelism: it creates all partition streams, spawns a collection future for each in a JoinSet, and restores partition order after completions arrive. Other execution paths need not spawn one Tokio task per node.

Blocking filesystem or compression work can use a blocking bridge. It must not occupy a Tokio worker that should keep polling network and object-store futures.

There are therefore three separate controls:

• partition count: available independent work;
• channel capacity: buffered batches across a task boundary;
• memory pool: retained operator state across the entire context.

They solve different problems and should not be presented as one “thread count.”

The Standard Library as Architecture

• Arc<dyn ExecutionPlan> makes a plan node shareable by optimizers, parents, metrics, and partition streams without pretending there is one unique owner.
• Vec<Arc<dyn ExecutionPlan>> represents heterogeneous tree children; ordinary recursion and iterator transforms implement plan rewrites.
• Option, Result, and ? make missing statistics, optional metrics, and fallible planning explicit instead of relying on sentinel values.
• HashMap and HashSet hold catalogs, function registries, grouping state, and optimizer bookkeeping where identity rather than sequence matters.
• atomics cheaply account for shared pool usage and metrics; mutexes protect compound policies such as fair sharing among spillable consumers.
• Drop on a MemoryReservation shrinks the pool and eventually unregisters its consumer. Resource accounting follows ownership on success, error, and cancellation paths.
• Pin<Box<dyn Stream<Item = Result<RecordBatch>> + Send>> gives the executor a stable, uniform asynchronous output type while each stream keeps its own concrete state machine.

The architecture is not “Tokio all the way down.” Standard ownership types describe durable structure; async types describe when progress can pause.

Generics, Traits, and Extension Boundaries

DataFusion uses traits where third parties or runtime query shape require an open set: ExecutionPlan, PhysicalExpr, TableProvider, QueryPlanner, optimizer rules, scalar functions, and MemoryPool. These commonly appear as Arc<dyn Trait + Send + Sync> because one plan must contain unlike nodes.

Generics are strongest inside reusable implementations. Stream adapters are generic over their inner stream or produced item; expression kernels can be generic over Arrow array types; tree utilities accept closures that preserve the visitor operation without creating a new public object hierarchy.

Associated types and trait bounds tie values together when callers benefit from that relationship. Type erasure happens at the collection boundary:

concrete scan/filter/aggregate structs
             ↓ coercion
Vec<Arc<dyn ExecutionPlan>> and Arc<dyn PhysicalExpr>
             ↓ execution
Pin<Box<dyn RecordBatchStream + Send>>

This keeps user extensions possible and compile times manageable. Making the entire plan a deeply nested generic type would encode a runtime SQL statement in a compile-time type the caller cannot name.

Errors, Cancellation, Memory, and Spill

DataFusionError carries planning, schema, execution, external, and contextual failures through one public result vocabulary. Optimizer EXPLAIN handling can capture a rule’s failure as part of the explanation rather than losing the phase that produced it.

Execution streams yield Result<RecordBatch>, so failures arrive at the same point as data. Task-backed receiver streams additionally supervise their JoinSet: ordinary errors enter the stream, panics are resumed rather than silently becoming an early end-of-stream, and dropping the stream cancels outstanding producers.

Large stateful operators use MemoryConsumer and MemoryReservation. A failed try_grow makes an operator spill and retry when supported, or terminate with an allocation error. The default pool is unbounded, while greedy, fair-spill, and consumer-tracking policies can be selected deliberately.

Cancellation is ownership-based. Dropping the final output stream stops demand and releases its children, task supervisors, channels, and reservations. External I/O still needs cancellation-safe futures and temporary files need their own cleanup guards; stream drop is the top-level signal, not magic preemption.

Metrics and EXPLAIN expose plan shape, partitioning, row counts, elapsed work, and spill behavior. In a query engine, observability is part of correctness: without it, a valid but catastrophically expensive plan is hard to distinguish from a hung one.

Build a Smaller DataFusion

1. Define a tiny typed columnar Batch with integer and string columns.
2. Define logical Scan, Filter, Project, and Aggregate plan nodes.
3. Build a fluent DataFrame API; add a small expression parser only afterward.
4. Write one logical rule that pushes a filter below a projection.
5. Define an object-safe physical operator returning a stream of batches.
6. Execute one partition end to end using poll-driven backpressure.
7. Add partitioned scans and partial/final aggregation.
8. Put a bounded channel at one repartition boundary and observe pressure.
9. Add byte reservations to the hash aggregate; spill sorted runs when denied.
10. Make dropping the root stream cancel producers and release reservations.
11. Add an EXPLAIN tree and per-operator batch/row/time metrics.

Production DataFusion additionally provides Arrow’s full type system, SQL coverage, catalogs, file and object-store formats, statistics, many optimizer rules and join algorithms, window functions, UDF families, distributed extension points, spill implementations, and extensive compatibility testing.

Rerun: Orientation

Rerun is a logging, storage, query, and visualization stack for multimodal robotics and physical-AI data. A small SDK call can cross generated semantic types, Arrow serialization, micro-batching, byte-bounded transport, a temporal chunk store, per-frame queries, and a wgpu renderer.

Points3D -> Arrow components -> Chunk -> sink/transport
         -> EntityDb/ChunkStore -> time query -> visualizer -> GPU

Design thesis

Rerun preserves one typed, Arrow-native data model across language SDKs, transport, storage, query, and rendering, while placing byte-aware ownership boundaries wherever producers can outrun consumers.

Interactive Logging-to-Rendering Map

Use the tabs to trace one Points3D log, byte pressure, a viewer frame, and the native/WebAssembly execution split.

Why Is It Designed This Way?

Why archetypes made of components?

Points3D is an ergonomic semantic bundle, but positions, colors, labels, and radii remain independently queryable component columns. New optional components can extend a visualization without replacing an opaque point-cloud payload.

Why use Arrow everywhere?

Arrow provides columnar arrays shared by Rust, Python, C++, transport, storage, and analytical query code. A single representation avoids repeatedly translating every value into row objects at subsystem boundaries.

Why micro-batch synchronous log calls?

The user should be able to log from ordinary code without making their whole application async. A dedicated batching pipeline coalesces tiny rows into efficient chunks, while byte budgets stop that convenience from becoming an unbounded memory queue.

Why identify data by entity path, component, timeline, and row ID?

Multimodal streams arrive at different rates and may be updated independently. These axes let a viewer ask “what was the latest color at this time?” while maintaining deterministic ordering and component-level history.

Why an immediate-mode viewer?

Each frame derives visible state from the current blueprint, time, and stored data. This avoids a large web of mutation callbacks whose cached UI state can drift from the selected time. Memoization and the renderer retain expensive work beneath that conceptually fresh frame.

Why separate native and WebAssembly scheduling?

Native builds can use threads, blocking channels, Tokio, and direct files. Browser builds generally own a single local event loop and browser APIs. cfg-selected capability wrappers preserve the higher-level pipeline without claiming both platforms have identical blocking or Send behavior.

One Points3D Log, Fully Traced

1. The caller constructs Points3D::new(positions).with_colors(colors) and passes a shared reference to RecordingStream::log.
2. Generated archetype code exposes SerializedComponentBatch values through AsComponents. Positions and colors are already Arrow arrays with semantic component descriptors.
3. The stream creates a time-bearing RowId, snapshots thread-local timeline state, builds a PendingRow, and pushes it into a cloneable ChunkBatcher.
4. A batching thread linearizes commands, groups compatible rows, and emits Arrow-backed Chunk values when time or size thresholds fire. Input and output queues share a configured byte-in-flight budget.
5. A forwarding thread converts chunks to log messages and hands them to a file, memory, stdout, gRPC, or other dyn LogSink. Transport moves Arrow record batches rather than reconstructing every point.
6. The receiver validates the chunk and EntityDb inserts it into ChunkStore, updating temporal/component indexes, query caches, entity metadata, and subscribers from the resulting events.
7. On a viewer frame, the spatial view asks Points3DVisualizer which components it requires. A range/latest-at query returns position chunks and optional colors, radii, labels, and annotations for the selected time.
8. The visualizer memoizes CPU interpretation, fills point/line builders, and hands draw data to re_renderer, which uploads or reuses GPU resources and records wgpu commands.

The convenient call does not hide a second semantic model. The same component descriptors remain recognizable throughout the pipeline.

Concurrency, Async, and Pipeline Pressure

RecordingStream is Send + Sync; clones from many application threads feed one linearized batching pipeline. Per-producer order is preserved, while a global order between unrelated threads is intentionally not promised.

The batcher and sink-forwarder have dedicated OS threads. This isolates row coalescing, Arrow work, compression, and potentially blocking sinks from the caller’s hot path. Rerun also uses Rayon for parallel CPU transformations, which is why its public docs warn against calling log while holding a standard mutex: work stealing plus an external lock can create a dependency cycle.

The important capacity is bytes, not message count. One image may outweigh thousands of scalar logs. re_quota_channel computes SizeBytes, admits data under a shared byte budget, and wakes blocked senders when receive releases those bytes. An oversized single message is allowed only after the channel empties.

Viewer networking and background services use async Rust on native builds. re_async::AsyncRuntimeHandle spawns Send futures on a supplied Tokio runtime, but uses the browser’s local executor for possibly non-Send futures under WebAssembly.

The UI itself is immediate-mode and primarily owner-threaded. Concurrency feeds it immutable-ish chunks, caches, and GPU work rather than concurrently mutating egui state from every producer.

The Standard Library as Architecture

• Arc makes Arrow buffers, chunks, stores, and runtime services cheap to pass across readers without copying image or point data.
• Weak lets handles observe shared recording state without forcing it to stay alive forever.
• Mutex and RwLock distinguish small mutable control state from stores with many readers and serialized insertion.
• AtomicI64 assigns recording ticks without acquiring the larger stream lock.
• HashMap, range maps, and ordered indexes encode the multiple lookup axes of entity, timeline, component, chunk ID, and time.
• std::thread::JoinHandle proves ownership of native batcher/forwarder lifecycle. Drop sends shutdown and joins after disconnecting bounded output so teardown cannot deadlock on a full queue.
• std::sync::mpsc::sync_channel(0) appears as a tiny acknowledgement rendezvous for operations such as flush; it is not confused with the bulk data path.
• cfg(target_arch = "wasm32") makes absent platform capabilities explicit at compile time.

The standard library defines who owns long-lived state. Tokio, Crossbeam, Rayon, egui, Arrow, and wgpu fill narrower execution and representation roles.

Generics, Generated Types, and Erasure

Generated archetypes such as Points3D give callers concrete constructors, builders, descriptors, and documentation. Components implement serialization traits that map typed Rust values into Arrow arrays. The generic RecordingStream::log<AS: AsComponents + ?Sized> accepts built-ins, custom bundles, and trait objects through the same API.

AsComponents is deliberately object-safe. Arrays, Vec<AS>, references, and Box<dyn AsComponents> flatten into Vec<SerializedComponentBatch>. That is the erasure point: semantic identity remains in each ComponentDescriptor, while concrete bundle shape no longer matters to batching or transport.

The viewer returns to strong types. Generic query and visualization helpers such as process_archetype::<Points3D, _, _> use descriptors to request required and optional components, then typed Arrow slicing exposes values like [f32; 3], u32, or String.

Trait objects handle open runtime sets: Box<dyn LogSink>, visualizer systems, data sources, and registries. Generics handle repeated algorithms whose concrete types improve inference or representation. Code generation handles the large, cross-language product schema that hand-written generic abstractions alone would not keep consistent.

Errors, Ordering, Shutdown, and Memory

Public logging is fallible even where today’s local path rarely fails; the result contract leaves room for compatibility and serialization failures. High-frequency or teardown paths often log an error once and continue, because observability code should not normally crash the application it observes.

Rows receive globally distinctive IDs and explicit timepoints. The chunk store rejects unsorted rows, treats empty chunks and duplicate chunk IDs as no-ops, records compaction lineage, and emits store events only after coherent index mutation. Those events invalidate caches and update subscribers.

Shutdown is ordered. A flush marker acknowledges all prior commands from that producer; final drop disconnects the batcher output before sending shutdown and joining, preventing a full bounded channel from trapping teardown. The forwarding layer then flushes the active sink.

Memory is managed at several scopes: byte-bounded ingestion channels prevent producer backlog, chunk-store accounting and garbage collection bound retained history, memoizers reuse expensive query/visualization work, and GPU caches own device resources. A limit at one stage cannot bound all later stages.

Native and browser overload behavior differs intentionally: native senders can block on a condition variable; WebAssembly cannot block the browser thread and therefore records the over-budget condition while continuing. Reliability means describing platform truth, not forcing identical APIs to lie.

Build a Smaller Rerun

1. Define EntityPath, Timeline, RowId, and two typed components.
2. Create one Points2D archetype implementing an object-safe component-bundle trait.
3. Serialize its components into a tiny columnar batch without copying them back into rows.
4. Accept synchronous log calls from several threads and micro-batch them on one owner thread.
5. Implement a byte-weighted MPSC queue with fair blocked senders.
6. Add file and in-memory sinks behind a small object-safe trait.
7. Index chunks by entity, component, and timeline; implement latest_at.
8. Build an immediate-mode HTML or egui view that reruns the query each frame.
9. Memoize unchanged query interpretation and separate it from renderer-owned buffers.
10. Implement flush acknowledgement, drop ordering, disconnect, and store GC.
11. Add a single-threaded target adapter and identify which native guarantees cannot be preserved there.

Production Rerun additionally supplies a generated multi-language data model, Arrow/Sorbet schemas, gRPC and .rrd protocols, a large temporal chunk store, blueprints, DataFusion integration, many visualizers, native and web viewers, GPU rendering, importers, server infrastructure, and compatibility tooling.

Walkthrough Template

Use this structure when adding a repository. Every case study has an Understand X half and a Build a Smaller X half.

Rebuild the architectural center, not a toy imitation of the public syntax.

Understand X

Revision and scope

Record the repository URL, commit, workspace version, enabled features, and which subsystems are deliberately out of scope.

What the repository does

Describe its responsibility in one paragraph. Name its external boundaries and the unit of work it processes.

Workspace map

Explain the role of each architecturally important crate. Omit support crates until they appear in an execution trace.

Representative operation

Show a small public API call or CLI command. Follow it through exact files and symbols until it reaches I/O or another concrete effect.

Deeply annotated execution trace