If you work anywhere near customers, you’ve heard some version of this. A director says “we want AI to improve operations.” Or a demo goes really well, and three months later nothing is in production. Engineers who sit with the customer and build this are now often called forward deployed engineers. If you’re the one in that meeting, this is for you.
So, the director has said it. What do you do first?
Most of us want to open a laptop and show something. I’d hold off. “Improve operations” isn’t a requirement yet. In system design, before you draw a box, you ask who the users are and what good looks like. Here you do that work with the customer.
Narrow it to one task
I try to get to one task, one group of users, and one number. Which work is slow, expensive or error-prone today? Who does it, how often, and how do they know when it’s done right? “Improve operations” might become “first-line support staff spend most of their time answering the same order-status questions.” That’s something you can build and measure.
When AI isn’t the answer
Then ask honestly whether it needs a model at all. OpenAI’s practical guide to building agents gives three signs that it does. The work needs judgment with lots of exceptions, the rules have become too tangled to maintain, or it depends on reading unstructured things like emails and documents. If none of those fit, the guide says a deterministic solution may suffice. Google’s Rules of Machine Learning made the same point years ago. Rule one is “Don’t be afraid to launch a product without machine learning.”
Order status is a good example. If the answer is a lookup in the order system, a form and a query will beat an agent on cost, speed and reliability. The model earns its place when customers describe problems in their own words and the answer depends on a policy with exceptions. Same lesson as 05a. Start with the simplest thing that works.
Agree the numbers before you build
Write the success metrics down with the customer before anyone builds anything. Anthropic’s docs on success criteria ask for criteria that are specific, measurable, achievable and relevant, and they list latency and price next to accuracy.
The one I’d add is cost per resolved task. Accuracy alone hides too much. A support agent that answers well but takes twenty turns and a big model to do it can cost more than the person it was meant to help. Take everything you spend in a period, model calls, retries and the human handovers, and divide by the tasks that were actually resolved. The market already thinks this way. Intercom, for example, prices its Fin support agent per outcome, which mostly means a conversation the customer didn’t need more help with. If your customer is comparing you to that, you need the same number.
Is the data ready?
Four quick questions save weeks. Does the data exist? Can we reach it from where the agent will run? Is it current, or is the knowledge base two years old? And who is allowed to see what? The last one matters most. The agent mustn’t show one customer’s ticket to another, the permission-aware retrieval point from 07.
From demo to production
A demo is the happy path with clean data and someone friendly at the keyboard. Production is every odd input, real permissions, rate limits at peak, and a bill. Articles 10 and 11 are mostly about that gap.
So stage it. A demo shows it’s possible. A pilot runs with real users on a small slice of traffic, with a person who takes over when the agent is unsure. OpenAI’s guide puts it as start small, validate with real users and grow over time, and it plans for human handover from the start. Production is when the pilot’s numbers hold and your evals block a release that makes them worse. I wrote about that gate in An Eval Suite That Never Blocks Is A Dashboard.
The enterprise questions, on day one
Ask these in the first week, or they kill the project late.
- Identity. Will users sign in with the company’s SSO, and does the agent act with their permissions or its own?
- Network. Is traffic to the model allowed over the public internet? Cloud providers offer private paths. AWS, for example, documents PrivateLink endpoints that connect your VPC to Bedrock privately.
- Data residency. Where may prompts be processed and stored? Anthropic’s API, for example, has a setting to keep inference in the US, and on Bedrock or Google Cloud the region comes from the endpoint you choose. Check what your provider offers for the region your customer needs.
- Retention and training. Is the customer’s data kept by the provider, and is it used for training? Get that in writing from the provider’s docs and the contract.
Governance from 12 starts right here too. Who approves risky actions, and where is it logged?
Be honest about risk
Here’s a hypothetical I’d expect to meet one day. A customer wants a fully autonomous support agent in two weeks, and half the knowledge base is out of date. What do you say?
I’d say what I can deliver. In two weeks, an agent that drafts answers for a human to send, on the questions where the data is good, measured against the numbers we agreed. Fully autonomous comes later, if the pilot earns it. And say what will go wrong. It will sometimes be confidently wrong, the cost moves with usage, and it’s only as good as the data underneath. A customer who hears this early trusts you more when something breaks.
When not to do all this? For an internal tool for a small team with no customer data, a short conversation and a quick pilot are enough. The full process is for when real users, real money or real data are involved.
Next, we’ll put the whole series together in a few end-to-end case studies.
