The Retry Loop Your Circuit Breaker Cannot See

The CRM returns 503. The agent reads the error, reasons about it, and tries again. And again. Eight times, at full token cost, sounding confident throughout — while the CRM, already struggling, takes eight more requests it cannot serve.

If you have been building distributed systems for a while, you know the answer: circuit breaker. And you would be half right, which is the interesting part.

TL;DR

Two failure modes look identical in a dashboard and need different controls:

A classic circuit breaker misses the first one entirely: three failures against a healthy service with a bad argument never trip a threshold, so the agent loops until your budget guard kills the run.

That is the failure mode a conventional retry policy was never designed for, because a conventional caller does not reason about the error and decide to try again. An LLM does, and it will happily retry a deterministic failure forever.

OWASP: ASI08 (Cascading Failures). Runnable code: patterns/harness/failure_containment/

Both, as program output

uv run python -m patterns.harness.failure_containment.demo

=== WITHOUT the pattern ===
  calls that reached the failing CRM: 8

=== WITH the pattern: thrash detection ===
  calls that reached the CRM: 3
  circuit state: closed (never opened!)
  crm_lookup has already failed 3 times with these exact arguments; repeating it
  will not help. Change your approach or report what you could not do.
  final: I could not reach the CRM; here is what I know from the ticket itself.

=== WITH the pattern: circuit breaker ===
  calls that reached the CRM: 3 (different arguments each time,
  so thrash detection could not help — only the circuit could)
  circuit state: open
  crm_lookup is unavailable (circuit open, retry in 30s). Continue without it
  and say so in your answer.

=== recovery: one probe, not a stampede ===
  after cooldown: half_open
  probe succeeded, circuit state: closed

Note circuit state: closed (never opened!) in the second block. The dependency was healthy the whole time. No breaker would have fired.

The pattern

guard = FailureContainment(
    policy=BreakerPolicy(failure_threshold=3, window_seconds=60, cooldown_seconds=30),
    thrash_threshold=3,
)

1. Thrash is checked before the circuit. A wrong plan against a healthy dependency never opens a circuit, so if the circuit check came first this would loop forever. Ordering is the control here.

2. Thrash is scoped to exact arguments. crm_lookup(4521) failing three times says nothing about crm_lookup(4522). Penalising the tool would punish a correct next step.

3. A success clears the thrash counter, so a transient failure does not permanently blacklist a good plan.

4. Half-open allows exactly one probe. This prevents the recovery stampede where every parallel call rushes a service that just came back. A failed probe restarts the full cooldown rather than counting toward a fresh threshold — a sick dependency should not be probed on a shrinking interval.

5. Denials carry an instruction, not just a refusal. “Circuit open, retry in 30s — continue without it and say so in your answer” produces graceful degradation. Bare DENIED produces a model that keeps guessing, which is the behaviour you were trying to stop.

6. Time is injected. FakeClock means every state transition is tested deterministically with no sleep anywhere. A circuit breaker whose tests sleep is a circuit breaker with untested transitions.

When to use it

• Any agent calling a network dependency it does not own — CRM, ticketing, payments, search, another team’s service, an MCP server.
• Any agent that can call the same tool repeatedly within a run. That is nearly all of them, and thrash detection is the cheapest guard you can add.
• Multi-agent systems, where one agent’s failure becomes another’s input and a retry storm compounds at every hop.
• Alongside cost budgeting: budgets bound the damage, this prevents it.

When NOT to use it

• Local, in-process, deterministic tools — a calculator, a date parser, a regex. If it cannot fail intermittently, a breaker is dead code that confuses the next reader.
• Where the platform already does it. If your calls go through a service mesh or gateway with retry and breaker policy, adding another layer gives you two policies with different opinions. Keep the thrash half — no mesh can see it, because to the mesh those are eight successful HTTP 200s carrying application errors — and drop the circuit.
• On something whose failure should stop the run outright. If the agent cannot function without the CRM, failing fast and loudly beats degrading into a confidently uninformed answer.
• Do not set thrash_threshold=1. Some tools legitimately fail once and succeed on retry — token refresh, cold start, lock contention — and refusing the second attempt turns a self-healing system into a broken one.

Trade-offs and failure modes

• Per-process state is not a per-service circuit. Twenty replicas each keep their own counters, so a service can take 20× your threshold before anything opens. Shared state fixes the arithmetic and adds a dependency whose failure mode you now own. Most teams should accept per-process and set thresholds accordingly.
• Thresholds are guesses until you have data. Too low and you open circuits during normal jitter; too high and you never protect anything. Alarm on open-circuit frequency rather than treating each one as an incident.
• Argument-scoped thrash is defeated by noisy arguments. If the model varies a timestamp or request id, every call has a new signature and thrash detection silently stops working — the same fragility as idempotency keys in durable execution. Canonicalise the fields that identify the action.
• Graceful degradation changes the answer. A run that lost the CRM produces a worse answer that still looks complete. Make the agent say so, and treat degraded runs as a distinct outcome in your metrics rather than a success.
• This does not implement timeouts, and timeouts matter more. A tool that hangs for 120 seconds is worse than one that fails in 50ms, and nothing here helps. Timeouts belong in the tool client — and they are what feed this breaker’s failure signal.
• Breakers can mask a real outage. An open circuit means your agent stops complaining, which means your alerts go quiet at exactly the wrong moment. Export circuit state as a metric.

Frequently asked questions

Is thrash detection just deduplication?

No — deduplication would refuse the second identical call unconditionally. Thrash detection only counts failures, allows a configurable number of retries, and resets on success. The distinction matters because retrying a transient failure is correct behaviour that you want to preserve.

Why not let the model figure out that retrying is pointless?

Sometimes it does. Often it does not, because “try again with a small variation” is a genuinely reasonable strategy that happens to be wrong here, and the model has no way to know the failure is deterministic. Making that judgment in code is cheaper and more reliable than hoping.

How does this interact with the model’s own error handling?

Complementarily, as long as the denial tells the model what to do instead. The denial message is doing real work: it converts “this failed again” into “stop trying this and report honestly,” which is the behaviour you want and which the raw error does not produce.

Should the circuit be shared across agents?

If they share a dependency, ideally yes — the dependency does not care which agent is hammering it. In practice per-agent breakers plus per-dependency rate limits at the gateway is a reasonable split, and much simpler to operate.

References

• OWASP GenAI Security Project, Top 10 for Agentic Applications 2026 — ASI08
• Michael Nygard, Release It! — the canonical circuit breaker and why half-open exists
• Microsoft, Agent Governance Toolkit — “Agent SRE”: circuit breakers and error budgets for agent runtimes
• Runnable code and tests: patterns/harness/failure_containment/

Part of the agent harness and governance series. Next: sandboxed execution — and why a Python function is not a sandbox.