If you’ve run anything on a model API for a few weeks, you’ve probably met at least one of these. At the busiest hour of the day, calls start coming back with a 429. An answer that streamed nicely on your laptop arrives all at once in staging, after a long pause. Or an email says the model pinned in your config will be retired in two months.
So here’s my question. Is any of this new?
Mostly, no. A 429 is a rate limit. A stalled stream is a buffering proxy. A retirement notice is a dependency reaching end of life. You’ve handled all three before, just not with a model on the other end. An agent, a harness, a whole AI application, is still a distributed system, and this is where your years of experience count the most.
Draw it like any other service
Take your agent and draw the boxes. Your backend, the model provider, a few MCP servers, your database, the APIs your tools call. Every arrow is a network call with its own latency, quota and way of failing. The model provider is a third-party dependency, like a payment gateway, and deserves the same suspicion. Then write down the NFRs as usual. Latency, throughput, availability and cost, with cost much higher up the list than you’re used to.
The model is new; the runtime still needs familiar distributed-system controls.
Stateless or stateful?
In 02a we saw that the model call is stateless. An agent isn’t. It has a conversation, tool results, maybe an approval it’s waiting for. You know the answer from web apps. Keep the processes stateless and put the state in a store. Session state, the conversation in progress, can sit somewhere fast with an expiry. Durable state, what must survive a crash or a deploy, goes in a real database: checkpoints, what the agent has already done, approvals. Then any worker can pick up any request.
MCP went through exactly this. Its current spec, 2026-07-28, removed the initialization handshake and protocol-level sessions. Every request now carries what the server needs, and a server that needs state across calls hands out an explicit handle. Each request stands on its own, so plain replicas behind a load balancer work without sticky sessions.
Event-driven agents
Not every agent waits for someone to type. Many start from a webhook, a queue or a schedule. The Claude Code GitHub Action, for example, runs when someone mentions @claude in an issue or pull request, or on a cron schedule.
Then everything you know about queue consumers applies. Events arrive twice, and out of order, and you need to stop loops. That GitHub Action rejects bot actors by default so bots can’t keep triggering Claude. When agents start triggering agents, you want the same guard.
Idempotency, retries and recovery
In 05b we had the refund that ran twice after a crash, and why checkpoints need idempotency keys. What I’ll add is that an agent has two retry loops.
The first is yours. The call to the model fails with a 429, a 500 or an overloaded error, and your code retries. Exponential backoff with jitter, honour the retry-after header, cap the attempts. Anthropic’s SDK already retries these twice by default, so check before you wrap it in your own retry and multiply the attempts.
A checkpoint can replay a step; the idempotency key prevents the side effect from happening twice.
The second loop is the model’s. A tool fails, and the model tries again a slightly different way, for as long as you let it. Your circuit breaker sees six different calls and never trips. I wrote about that in The Retry Loop Your Circuit Breaker Cannot See, and about the double payment in The Retry That Paid The Invoice Twice.
When a multi-step task fails halfway, after booking the hotel and before booking the flight, you’re back to sagas. Each side effect needs a compensating action written in code. Don’t ask the model to improvise the undo.
The parts that really are new
Token cost is runtime cost. In a normal service a request costs about the same each time. In an agent, every turn resends the conversation so far, so a twenty-turn task costs far more than twenty times the first call. Caching makes a repeated prefix much cheaper, and batch APIs cost about half for work that can wait. The bill is set by what your loop does at runtime, so check a budget per run and per tenant before each call. More in You Will Find Out About The Runaway Agent From The Invoice.
Rate limits are counted in tokens too. Anthropic and OpenAI both limit requests per minute and tokens per minute, so ten users with huge contexts can hit the wall before a hundred with small ones. OpenAI’s docs note that failed requests count toward the limit, so hammering a 429 makes it worse. On most Claude models only uncached input counts, so caching buys you throughput as well as savings. When traffic jumps 10x, this is often the first thing to break. A queue in front of model calls with per-tenant limits is the usual fix, and it stops one noisy tenant from starving the rest.
Streams go through every proxy. In 02b I said a buffering proxy can hold the whole stream. Nginx, for example, buffers proxied responses by default and drops a connection after 60 seconds without data. A model that thinks for a while before its first token can trip that. Turn off buffering on streaming routes, check idle timeouts on every hop, and decide what the client does when a stream breaks halfway. Anthropic’s docs also warn that an error can arrive mid-stream after a 200, so error handling can’t stop at the status code.
Models get retired. Anthropic’s deprecation page lists each model with a retirement date, and requests to a retired model fail. On 30 September 2026, for example, Anthropic gave Claude Sonnet 4.5 a retirement date of 30 November 2026. They promise at least 60 days’ notice. And as we saw in 02a, OpenAI shut down the whole Assistants API in August 2026. Treat a model like any dependency with an end-of-life date. Pin the exact version, track the sunset dates, and keep a test set so a model swap is a measured change.
The whole mapping in one place.
| In your agent | The pattern you already know |
|---|---|
| model API with quotas | third-party dependency with rate limits |
| conversation and checkpoints | session store and durable store |
| webhook or schedule trigger | queue consumer, at-least-once delivery |
| tool with a side effect | idempotency key |
| task fails halfway | saga with compensation |
| model retrying forever | retry storm, circuit breaker |
| model retirement | dependency end of life |
When not to
Don’t build all of this on day one. An internal tool for twenty people doesn’t need a queue, a saga or an AI gateway. Start with a timeout, a sane retry and a spend limit. Add the rest when traffic or side effects ask for it, as you would for any other service.
Next: how do you know any of this is working? Evals and observability.
