# Change the prompt, the context, or the model?

AI for Software Engineers, part 6 (14 of 22)

> When an AI system gives poor answers, measure first, then change the cheapest thing: the prompt, then the context you pass in, then the model choice, and fine-tune only as a last resort. Most wrong answers about your own systems are missing context, not a weak model.

You've had this happen. You ask your assistant how your team deploys, and it confidently describes a process you don't use. Or the agent you built in the last two articles keeps returning JSON with one field missing. Most of us do one of two things next. We open the model picker and choose the biggest model. Or someone in the meeting says "we should fine-tune it on our data".

So before reading on, think about it. The assistant gets your deploy process wrong. Do you teach the model, or do you show it the runbook?

Show it the runbook. And to see why, it helps to line up everything you could change, from the cheapest to the most expensive.

**First, measure**

Remember the five prompts from article 04c, each with a good answer written down? This is where they pay off. If you can't tell whether a change made things better, every change is a guess. OpenAI's model optimization guide puts evals first, before prompt engineering and long before fine-tuning, and Anthropic's prompting docs say the same.

Keep a couple of regression cases too, things that work today. A fix that breaks them isn't a fix.

**Start with the prompt**

The cheapest change takes minutes. Say what you want more clearly. Give the format. Add two or three examples of a good answer. Say what to do when the information isn't there. OpenAI's accuracy guide says prompt engineering is typically the best place to start, and often the only method needed for things like summarisation, translation and code generation.

A missing JSON field is very often fixed right here. Many providers now also offer structured outputs, where the API holds the model to your schema, so check that before anything heavier.

**Then the context**

Now back to the deploy process. Was the model ever going to know it? It's your team's process. It isn't on the public internet, and it changed last month. No wording fixes that. The model needs the facts.

You already know every tool for this. From article 04a and 04b, the facts go into an instructions file if every session needs them, a skill if you need them sometimes, a tool if they live in a system, or retrieval if there are too many to send every time.

OpenAI's accuracy guide draws the line in a way I like. Optimise the context when the model lacks knowledge, because it wasn't in training, it's out of date, or it's proprietary. Optimise the model when the behaviour is inconsistent, like the format, the tone or the way it reasons. Most wrong answers I see in companies are the first kind. We'll go deep on retrieval in the next article.

**Then the model choice**

Sometimes the facts are there and the model still can't do the job. Then change which model you call, or how.

Start inside the model you have. Many models have a thinking or effort setting, and Anthropic's model selection guide says tuning effort is often a better lever than switching models. Then try a bigger tier, or a different provider, against your test set.

And go the other way too. Most work isn't hard. Classifying a ticket, pulling fields out of an email, summarising a log. A small model does these faster and for a fraction of the price. Anthropic's guide offers two starting points. Start small and move up when your tests show a gap, or start with the most capable model and move down once it works.

Remember routing from article 05a? You classify the request and send it down a path. Anthropic's Building effective agents gives exactly this example, easy questions to a small model and hard ones to a bigger one. It's a load balancer with a policy. You can also pair models inside one task. A cheap model does most of the turns and asks a bigger one for advice at the hard points, or a big model plans and hands the bulk reading to cheap workers. Anthropic's [cost and intelligence guide](https://platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence) walks through both, including when one model is still cheaper. With several providers, teams often put a gateway in front so routing and fallback live in one place.

Cheap models don't stop a runaway loop from running up a bill, though. I wrote about budgets that halt before the expensive call in [You Will Find Out About The Runaway Agent From The Invoice](https://www.allsrc.dev/posts/pattern-cost-and-tool-budgeting/).

**Last, the model itself**

Fine-tuning means you take a model and train it further on your own examples, so its weights change. You get your own version of the model.

Here I want to be candid. Fine-tuning is good at behaviour. It's poor at knowledge. Provider guides list the same kinds of wins. A format the model must hit every time. A tone or style. Classification. Domain language that differs a lot from what the model saw in training, which Google's Vertex AI docs call out. If you fine-tune on your deploy runbook, you get a model that sounds like your runbook, frozen on the day you trained it. Change the process next week and you're training again. Put the runbook in context and you just edit a file.

The one I find most useful is distillation. You get a big model working well on a narrow task, save its good answers, and use them to fine-tune a small model. OpenAI's docs describe exactly this. It's a cost move, and a good one for a high-volume task.

There are newer methods too. Preference fine-tuning learns from pairs of better and worse answers, and reinforcement fine-tuning scores answers with a grader you write. Both are advanced, and OpenAI's [reinforcement fine-tuning guide](https://developers.openai.com/api/docs/guides/reinforcement-fine-tuning) is a good place to start if you're curious.

When not to fine-tune? When you don't have an eval yet. When you have a handful of examples. OpenAI suggests starting with about 50 good ones and says that if 50 make no difference, rethink the task or prompt before adding more. When the facts change often. And remember the cost after training. You're tied to one base model, only some models can be tuned at all, and every model upgrade means doing it again. And if your training data holds customer data, that data now lives inside a model.

Here's the ladder in one place.

| Change | Fixes | Cost to try |
|---|---|---|
| Prompt and examples | unclear task, format, tone | minutes |
| Context (files, skills, tools, retrieval) | missing or stale facts | hours to days |
| Model choice or routing | too weak, too slow, too costly | hours, plus testing |
| Fine-tuning | stubborn behaviour, cost at volume | weeks, and ongoing |

If you think of it as a service, the prompt is config, the context is the data you pass in, the model choice is the hardware you run on, and fine-tuning is recompiling the binary. You wouldn't recompile to fix a missing row in the database.

Next we go deeper on the context rung. Why retrieval, when you need it, and why vectors are only one way to do it.

## References

- Building Effective AI Agents (https://www.anthropic.com/engineering/building-effective-agents)