The Retry That Paid The Invoice Twice

An accounts-payable agent pays a $4,200 invoice. Two seconds later the pod is evicted mid-run. Your orchestrator does the obvious, correct-looking thing: it retries.

ACME gets paid twice, and nothing in the system knows.

This is not an exotic failure. At-least-once execution is the default everywhere — Kubernetes restarts, queue redelivery, provider timeouts, a human clicking “run again,” the replay-based resume in your own approval gate. Any agent that performs a side effect and then crashes before recording it will, on retry, perform it again.

TL;DR

Two mechanisms, and teams build the first and skip the second:

Checkpoints make resumption cheap — resume from the last completed turn instead of the beginning.
Idempotency keys make it safe — the same logical action is recognised on replay, and a genuinely different action is not.

The checkpoint tells you where you were. It does not tell you what already happened to the outside world. That is the gap the second mechanism fills, and it is the one that sends the customer two refunds.

Ordering matters: record intent BEFORE performing the side effect. Record after, and a crash in between leaves an action that happened with no record that it did.

OWASP: ASI08. Runnable code: patterns/harness/durable_execution/

The double payment, prevented

uv run python -m patterns.harness.durable_execution.demo

=== WITHOUT the pattern ===
  payment ledger after one crash and one retry:
    $4,200 -> INV-9
    $4,200 -> INV-9
  ^ ACME was paid twice, and nothing in the system knows

=== WITH the pattern ===
  attempt 1: pays, then the pod is evicted
    ledger: ['$4,200 -> INV-9']

  attempt 2: the retry, in a fresh process
    DENIED by policy: already performed in this run (idempotency key 8156aa4b...);
      previous result: paid $4,200 for INV-9
    ledger: ['$4,200 -> INV-9']

=== the case you cannot resolve automatically ===
    decision: PAUSE
    reason: a previous attempt at send_payment was interrupted before completing;
            a human must confirm whether it took effect

The bug I hit, which is the most useful part of this article

My first implementation keyed idempotency on (run_id, turn, tool, arguments).

That is wrong, and the end-to-end test caught it: the turn number does not survive a resume. A resumed trajectory that differs by one step lands the same logical payment on a different turn, so the key moves, so nothing deduplicates, so the invoice is paid twice — with an idempotency system installed and reporting healthy.

The provider’s tool-call id is worse: it changes on every attempt.

What actually identifies the action is (run, tool, arguments, occurrence), where occurrence counts how many times this exact call has been attempted in this process:

def idempotency_key(run_id: str, call: ToolCall, occurrence: int = 1) -> str:
    material = json.dumps({"run": run_id, "tool": call.name,
                           "arguments": call.arguments,
                           "occurrence": occurrence}, sort_keys=True)
    return hashlib.sha256(material.encode()).hexdigest()[:20]

occurrence keeps this honest in both directions. An agent that legitimately sends the same payment twice in one run is doing two things, so they get ...:1 and ...:2. A replay maps to the same two keys, in order, and both are recognised. Deduplication that cannot tell “again” from “twice” is a bug either way.

And because the counter resets per process, a resumed run lines up with the keys the original wrote — with nothing to reconstruct. No turn counters, no ids.

The pattern

guard = DurableExecution(
    checkpoints=CheckpointStore(directory=Path("/var/agent/checkpoints")),
    effects=SideEffectLog(path=Path("/var/agent/effects.jsonl")),
    side_effecting=frozenset({"send_payment", "send_email", "create_ticket"}),
)

An interrupted effect PAUSES; it does not guess. Intent written, no completion: we genuinely cannot know whether the payment landed. Both automatic answers are wrong, so the run parks for a human.

Only side-effecting tools need keys. Replaying a read is free. The side_effecting set is a deliberate, reviewable list.

When to use it

Any agent that moves money, sends messages to people, provisions infrastructure, or writes to a system of record.
Any agent whose runs are long enough to be interrupted, or that pauses for human approval.
Any agent behind a queue, cron, or orchestrator with retry semantics you did not write. You probably have one and have not checked its retry policy.
The moment someone says “we’ll just re-run the failed ones.”

When NOT to use it

Read-only agents. A retry costs tokens and nothing else.
When the operation is already idempotent by nature. If your tool is set_status(id, "closed"), calling it twice is harmless. Prefer making the downstream operation idempotent over making the agent remember — the vendor’s Idempotency-Key header is strictly better than your log, because it is enforced where the effect happens.
Do not build this if you already run a durable workflow engine. Temporal, Durable Functions, and Step Functions do checkpointing and replay properly, including leases and timers this pattern hand-waves. This is the shape of what they give you, useful when you cannot adopt one.
Do not checkpoint transcripts containing data you may not persist. A checkpoint is a durable copy of the whole context window, so every decision from the redaction boundary applies to it too.

Trade-offs and failure modes

Checkpoints are large and grow. Every turn writes the whole transcript. Combined with compaction that is tolerable; without it a long run writes megabytes.
Resume is not replay-identical against a live model. Restoring the transcript restores the inputs, not the decisions. Correctness still holds — the idempotency log protects side effects — but anyone expecting determinism will be surprised.
The occurrence counter is per-process. Two concurrent processes on the same run id would both see occurrence 1 and both proceed. Concurrent execution needs a lease, which is exactly the machinery a real workflow engine provides.
Argument instability breaks keys. A timestamp, UUID, or re-worded description in the arguments means every retry is a new key and deduplication silently stops working. Key on a business identifier where you can.
The effects log is now safety-critical. Losing it means losing replay protection; truncating it re-enables double payments.

Frequently asked questions

Isn’t this what my message queue’s exactly-once delivery is for?

Exactly-once delivery is largely a myth over an unreliable network; what systems actually provide is at-least-once delivery plus idempotent processing. This pattern is the idempotent-processing half, applied at the agent’s tool boundary rather than at the message boundary.

Should I checkpoint every turn or less often?

Every turn is the simple default and it is what this implementation does. If checkpoint size becomes a problem, checkpoint less often and accept redoing more work — the correctness guarantee comes from the idempotency log, not from checkpoint frequency, which is a useful property to internalise.

How do I resolve the “interrupted, unknown outcome” case?

Query the downstream system. That is the only correct answer, and it is why the pattern pauses for a human rather than guessing: someone has to check whether the payment exists. If your downstream exposes a lookup by business key, automate the check — that is strictly better than either default.

Does this work with streaming responses?

The checkpoint side does. The idempotency side is unaffected, because it guards tool execution rather than token generation. Streaming introduces its own problem — partial output already delivered to a user cannot be retracted — which is a genuine gap I have not solved.

References

OWASP GenAI Security Project, Top 10 for Agentic Applications 2026 — ASI08
Microsoft, Agent Framework — durable agents, checkpointed workflows, saga orchestration for compensating actions
Temporal and Azure Durable Functions — the mature form of this pattern
Stripe’s idempotent requests — the argument for enforcing idempotency at the effect rather than the caller
Runnable code and tests: patterns/harness/durable_execution/

Part of the agent harness and governance series. Next: verification loops — why “I’ve fixed it” is a claim, not a completion signal.