Separate the case from the worker running it
Consider an onboarding case that waits three days for a signed schedule. The computer process that first handled the upload need not remain running for three days. The case must retain its identity, required evidence, current owner and permitted next action so another worker can continue later.
Do not treat an in-memory variable or an informal, uncontrolled spreadsheet note as a durable checkpoint. A controlled persistent register can store checkpoint state if it meets the required consistency and recovery checks. Define what is persisted and how a restarted worker discovers unfinished cases. The stored state must explain the next step without reconstructing it from a person’s inbox.
Choose checkpoints at business boundaries
| Checkpoint | Persisted evidence | Permitted continuation |
|---|---|---|
| Packet registered | Client, checklist version, original file references | Request missing evidence |
| Review waiting | Reviewer, evidence versions, response deadline | Accept a matching decision |
| Account attempted | Logical operation, payload and target reference | Confirm or reconcile creation |
| Handoff confirmed | Accepted target account and handoff record | Close the onboarding task |
Checkpoint a completed business fact, not merely the current line of code. “Account created” requires confirmation from the target. If the call timed out, the checkpoint should say that creation was attempted and remains uncertain. A restart then knows it must reconcile before issuing another request.
Persist progress between handoffs
- Capture
- Case and source versions
- Wait
- Durable decision state
- Confirm
- External result recorded
- Resume
- Only unresolved work
| Evidence | Meaning |
|---|---|
| Capture | Case and source versions |
| Wait | Durable decision state |
| Confirm | External result recorded |
| Resume | Only unresolved work |
Uncertain effects require target reconciliation.
Persist deadlines and the relevant calendar assumptions. On restart, compare the stored deadline with current time and choose the overdue route if necessary. Recalculating “three days from now” on every restart would extend the commitment indefinitely.
Understand what replay does
Microsoft documents that Durable orchestration uses recorded history and replay, requiring deterministic orchestration code. Its guidance places restrictions on direct time APIs and other nondeterministic operations. Read the orchestrator constraints. These are platform-specific rules; check the recovery model of the engine you actually choose.
At the business level, replay reconstructs what has already been decided. It should not ask an AI model to reinterpret an old document or perform an external write again simply because a worker restarted. Store the accepted interpretation and execute new interpretation as an explicit new activity when required.
Even with a durable engine, an external effect can outlive a lost response. Design the boundary with an operation reference and a reconciliation path. A replayable orchestration does not make every connected application’s write automatically idempotent.
Accept each response against its current context
In ON-42, a reviewer accepts packet revision 3. Before that response arrives, the client submits revision 4 with a changed service scope. Store both events. Match the decision to revision 3 and evaluate whether it remains applicable under the policy; do not attach it to whichever file is currently newest.
Use an expected state or version when applying the response. Two workers receiving the same event must not independently advance the case. A database transaction or comparable concurrency control should protect the state change and its associated event record.
Retain late, repeated and unmatched responses for investigation without letting them create a second completion. Operators need a clear distinction between “received” and “accepted for this checkpoint”.
Plan cancellation and workflow changes
A cancellation should stop future work while preserving confirmed effects. If an account already exists, closing the case does not remove that account. Create a separately authorized cleanup task where appropriate and retain the cancellation reason.
For workflow updates, decide whether existing cases remain on their original version or undergo an explicit migration. A case waiting under an old checklist may not satisfy a new rule. Record the chosen migration, affected cases and review requirements rather than silently loading current code into old decisions.
Test recovery at the awkward moment
Stop the worker before persisting an attempt, after the target accepts a write, while waiting for review and immediately after accepting a response. Restart it, then deliver the same response twice. Check that the case retains one logical result and every unresolved action has an owner.
Include those interruption points in the recovery acceptance tests for Trion’s cross-system handoffs.