One Job, Fully Traced

1. The backend advertises readiness

CallAllUnordered first polls futures already in flight. It then asks the backend and the composed Tower service whether they are ready. Only after both say yes does it call Backend::poll_next.

This order matters. A concurrency-limit layer can return Pending, so the worker does not claim another durable job until it has capacity to own it.

2. Compact storage becomes typed input

The backend yields a Task<Compact, Connection, Id>. Its associated Codec decodes the compact representation into Args. Storage chooses identifiers, connections, serialization, queue semantics, and backend middleware; the worker does not know whether the task came from memory, Redis, SQL, or another broker.

3. The task enters the service stack

The worker composes worker context, user layers, backend middleware, readiness, and tracking around the handler service. TaskFn adapts the function to Service<Task<...>>.

For this handler:

async fn send(job: Email, state: Data<Arc<App>>, attempt: Attempt) -> Result<(), Error>

the payload Email is moved into job. Each additional argument implements FromRequest<Task<...>> and is produced from &Task before the function is called. The resulting values are owned across .await; no borrow of the worker or backend is held by the handler.

4. The future is polled

The function returns a future. parallelize(executor) can hand that future to an injected executor such as tokio::spawn; the core is not coupled to Tokio. The worker keeps the resulting service futures in FuturesUnordered, so completions are handled in completion order.

5. Completion becomes lifecycle state

IntoResponse normalizes the handler output. Tracking updates attempt and in-flight state. Backend-provided acknowledgement middleware observes the result and performs persistence-specific completion work. Its acknowledgement future is tracked too, so graceful shutdown cannot report completion while an acknowledgement is still pending.

The deepest invariant is:

capacity → dequeue → decode → extract → execute → acknowledge

Every arrow transfers responsibility; none is merely a convenient call order.