Errors, Retries, Acknowledgement, and Shutdown

Apalis does not represent every failure as one undifferentiated Error.

Error layers

Handler failures flow through the service response and can be erased to BoxDynError where middleware needs one common type. AbortError means the failure should not be retried. RetryAfterError carries a delay request. CallAllError distinguishes backend polling, codec decoding, and service execution. Non-exhaustive WorkerError covers worker-level failures such as polling, readiness, codec, I/O, panic, state, and graceful exit.

That separation preserves the decision each layer must make: retry the job, report the worker, close the backend, or terminate the monitor.

Retry is middleware

RetryPolicy, BackoffRetryPolicy, and RetryIfPolicy implement Tower retry policy around the handler service. They can inspect the task and result, clone a request for another attempt, and schedule an immediate or delayed retry. A successful, killed, explicitly aborted, exhausted, or shutting-down task does not retry.

Because retry is a layer, placement matters. Tracing outside retry sees one logical job; tracing inside sees each attempt. The builder’s nested Stack makes that order concrete.

Acknowledgement belongs to the backend boundary

An Acknowledge layer awaits the handler result and then invokes backend-specific acknowledgement with the execution context. That operation may mark success, store failure, release a lock, or reschedule. The worker knows when acknowledgement belongs; the backend knows what durable transition it means.

The acknowledgement future is tracked as part of worker completion. Returning from the handler is therefore not confused with durably recording its outcome.

Graceful shutdown is staged

1. Shutdown is signalled and readiness stops admitting jobs.
2. Already claimed service futures keep running.
3. Tracking waits for handler and acknowledgement obligations to leave.
4. Backend::poll_close flushes or releases backend resources.
5. Monitor joins workers, applies restart policy where configured, reports errors, and can enforce a shutdown timeout.

An optional panic-catching layer converts a handler panic into an abort-style error. The worker monitor also contains panic at its worker-future boundary. These boundaries improve availability, but they do not make panic a normal domain error.