A support agent gets approval to issue a credit. The billing API accepts the request, but the connection drops before the response comes back. Then the worker restarts.
Should it send the request again?
The model can read the ticket, understand the policy, and recommend the right amount. None of that tells it whether the first credit went through.
The billing system has that answer. Until the workflow checks, another attempt could pay the customer twice.
That is where durable execution becomes useful. It keeps a record of a workflow's progress outside the process running it, so work can continue after a crash or a pause. The application still has to decide what is safe to repeat.
When this needs an AI agent build
If your agent produces a draft that someone can regenerate, you may not need much beyond a job queue and a stored result. If it changes customer records, waits for approval, or coordinates work across systems, the recovery path deserves as much attention as the successful one.
For an AI agent development project, I would start there: pick an action and ask what happens if the worker disappears immediately after sending it. The answer usually reveals more about the architecture than a demo does.
There is no useful rule that everything running longer than thirty seconds needs a durable agent framework. A two-second write can be expensive to duplicate. A twenty-minute read-only task may be cheap to restart.
The recovery workflow starts with an unanswered request
Take the credit example. Before contacting billing, the workflow should save an identifier for that particular operation.
If the provider supports idempotency keys, each retry of the same credit should use the same key, within the provider's documented rules. Giving every attempt a fresh identifier defeats the protection.
When a response goes missing, the next step is to look up the operation or reconcile against an authoritative billing receipt. A local log saying "request sent" is not enough. It records what your application tried to do, not what the other system committed.
Billing Receipt as Recovery Evidence
Some APIs do not offer reliable deduplication or a way to query the outcome. For a consequential action, the workflow may need to stop with an unknown result and ask an operator to reconcile it. Retrying until something returns a success code can make the problem worse.
This also changes how you model completion. The credit can be confirmed while the customer notification is still pending. Keep those as separate steps. Otherwise, retrying a failed notification can accidentally run the billing operation again.
Before choosing a runtime, write down the recovery rules for this one action. A small table is enough to expose the gaps:
| What the workflow knows | What it should do next | What lets it proceed |
|---|---|---|
| Credit proposed; approval pending | Wait, with an expiry and a named reviewer | Approval for this account, amount, and proposal |
| Billing request timed out | Look up the operation; hold further writes | A receipt or a reliable confirmation that no credit was created |
| Credit confirmed; notification failed | Retry only the notification, with its own duplicate protection | The billing receipt remains attached to the run |
| Billing outcome cannot be established | Assign reconciliation to an operator | A recorded resolution before any further credit attempt |
The last row needs an owner. An operator needs an account reference, the attempted amount, an operation ID, and a record of checks already made. A generic "agent failed" alert leaves them to repeat the investigation.
What the runtime actually saves
The tooling is useful now, but the details matter more than the label "durable."
Temporal puts external work in Activities. An Activity can run again, so a billing call still needs a safe retry strategy. Temporal's Activity documentation explains the retry and idempotency responsibilities.
Model calls belong behind an execution boundary too. Recovery should use a recorded result where available, rather than quietly asking the model to reconsider an earlier decision.
LangGraph persists graph state through checkpointers. Its in-memory examples do not survive a process restart; you need persistent storage for that. The persistence documentation explains the distinction.
There is another detail worth reading before adding approvals: when an interrupted node resumes, it starts again from the beginning. Code before the interrupt can run again, so a side effect there could duplicate work. See LangGraph's interrupt documentation.
For a team already using AWS, Step Functions Standard is another option. It supports long waits and callbacks, with executions lasting up to a year. Express has a five-minute limit and does not support the callback integration pattern, according to AWS's comparison.
The distinction matters if a person might take a day to respond. AWS also documents different execution guarantees across workflow types. Those do not remove the need to reason about retries at the external API boundary.
I would choose based on the application and the team's operating experience. Adding two orchestration layers means deciding which one owns retries, deadlines, and cancellation. Without that decision, a failed call can be retried by the tool client, the agent runtime, and the outer workflow.
Yesterday's approval may no longer apply
Suppose the manager approves the credit on Friday, but the workflow resumes on Monday. A colleague could have resolved the ticket over the weekend. The approved amount might no longer be appropriate.
The saved approval is still there. Its assumptions may not be.
Store what the person actually approved: the account, amount, reason, and any material conditions, along with who approved it and when it expires. An "approved" flag attached to a run is too broad.
Before acting, check the current record and permissions. If the proposal has materially changed, send it back for review.
This is application logic. A runtime can wait for the right callback and recover the right state without knowing that someone already issued the credit manually.
The execution record needs to make this understandable to an operator. They should be able to see the proposal, approval, attempted action, external receipt, and outstanding work without reading an entire chat transcript.
Keep model and tool versions with the decision where they help explain it. Apply access and retention rules to the saved evidence; copying every sensitive payload into a trace is not a sound audit policy.
A paused run also has to survive your next release
A workflow that waits for days will eventually wait through a deployment. The new code has to understand the old state, or the old run needs a supported way to finish on its original code.
Temporal's current Worker Versioning model includes pinned workflows and workflows that automatically upgrade; the latter still need replay-safe changes. That is a concrete deployment choice, not something persistence solves by itself. Temporal documents those behaviors here.
Whatever runtime you use, test this with a real paused run before a release. A renamed step, changed state field, or different tool contract can break recovery even when the stored data is intact.
Give the run an end condition as well: a deadline, retry limits, and a spending budget. Cancellation should stop new work while leaving any in-flight action available for reconciliation. Killing your worker does not prove the billing provider stopped processing its request.
Limits worth settling before you build
Durable execution adds storage, operational support, and work to keep old runs compatible with new code. If your existing queue already handles a safely repeatable task, replacing it may buy little. Start by identifying a recovery requirement it cannot meet.
The billing integration can also set a hard limit. If nobody can establish whether a credit happened, changing orchestrators will not supply the missing evidence. Keep that action under human control until there is a reliable reconciliation path.
Finally, recovery says nothing about whether the proposed credit was justified. You still need to evaluate the model against representative tickets and check its evidence.
A workflow can reliably finish an incorrect decision. Test the recommendation and the execution separately, with someone accountable for the unresolved cases.
What Quellix would build
For an AI agent development engagement, I would keep the first build narrow: one support workflow, one billing integration, an approval screen, and a view of runs that need attention. The model proposes the credit; application checks and the approval decision determine whether it can proceed. The billing receipt determines whether it completed.
Before expanding it, I would deliberately break it. Stop the worker after billing accepts a request but before the receipt is saved. Send the approval callback twice.
Then change the ticket while approval is pending, and deploy new code during the wait. Each test should end with either the intended credit confirmed once or a clearly assigned reconciliation task.
Track unresolved outcomes and duplicate effects alongside completion rate and cost per resolved request. Agent observability should make those failures visible, rather than just show that the model returned an answer.
You may find that most of this workflow is ordinary automation. Use the model for the part that benefits from interpreting a messy ticket, and keep the billing operation constrained.
The customer needs the correct credit and a clear response. They do not benefit from giving the agent more freedom over either.
Talk to an AI Engineer about a workflow you need to make recoverable.
