Errors, Reconciliation, and Shutdown

In a stateful external system, error handling is not complete when an error has been logged. The application must decide which state is now trustworthy.

Error layers

• anyhow::Result adds context and propagates startup or task-fatal failures.
• Operational failures may warn, update runtime incidents, block future entry, and continue serving safer work.
• Channel closure is often normal lifecycle information rather than an error.
• A command carrying a oneshot sender can report a local result directly to an operator or control-plane caller.

An accepted request is not observed truth

An order API response proves only that an interaction reached some broker boundary. It does not prove the intended position now exists. The executor therefore schedules fast reconciliation after entry-related work.

reconcile_account_exposure queries or consumes a broker snapshot, compares it with local positions, adopts broker-visible positions when appropriate, closes stale records, persists snapshots, and emits lifecycle events.

Source: account_reconcile.rs:734

Desired state is retried against observed state

For ConvergeToTarget, the executor computes:

delta = target_signed_qty - current_signed_qty

It submits only that delta, records the in-flight target, and waits for reconciliation to observe the effect before sending another adjustment. A new desired target can replace pending intent without assuming the prior broker action completed.

Source: maybe_drive_runtime_converge_intent

Safety failures change future policy

Some broker-account failures become persistent entry-block reasons. Existing positions can still require exits, protection, or reconciliation, so “stop all processing” would be less safe than “block exposure increases while preserving risk-reducing operations.”

This pattern separates availability from permission.

Cooperative shutdown

An Arc<AtomicBool> broadcasts a shutdown request. Tasks check it at loop boundaries. The account executor emits close-state feedback for its synthetic observers and persists snapshots before leaving its loop.

Channel ownership supplies another shutdown mechanism: when all engine senders are dropped, the router sees None, exits its loop, and sends explicit ExecutorCommand::Shutdown to executors.

Source: signal_router.rs:237, account_executor.rs:1935

Cancellation is not automatically transaction-safe

Dropping an arbitrary future while it is between broker submission, journal write, and reconciliation could leave uncertainty. The architecture reduces this risk through single-owner execution, explicit timeouts, persistent journals, idempotent intent IDs, and reconciliation. Those mechanisms—not Tokio cancellation alone—restore knowledge after interruption.