# Beyond text

AI for Software Engineers, part 8 (16 of 22)

> Images, documents and audio go into the same request as content blocks and take up context like text does. Voice agents either chain speech-to-text, a model and text-to-speech or use one speech-to-speech model, trading control and visibility for lower latency.

You've probably done this without thinking twice. You hit an odd error in a dialog box, take a screenshot and paste it into ChatGPT or Claude, and it reads the error and tells you what's wrong. Or you've talked to the voice mode in ChatGPT, or Gemini Live in the Gemini app, on a walk. And at some point you've phoned a company's support line and talked to a bot that paused just a beat too long before every answer.

Everything in this series so far has been text going in and text coming out. So what actually changes when the input is a picture or a voice?

**Images, documents and video**

Start with the screenshot. Remember the request from Part 2, a list of messages with roles. A message doesn't have to hold only text. It holds content blocks, so you send an image block next to a text block in the same user message. Anthropic, OpenAI and Google all work this way, each with its own field names.

And from Part 3 you know what happens next. The image becomes tokens and takes up room in the context window. Anthropic's [vision docs](https://platform.claude.com/docs/en/build-with-claude/vision) say the model sees an image as small square patches, each one a token, so a big screenshot can cost more than a page of text. Gemini models take video too, sampled at about a frame per second plus the audio, so a ten-minute clip is a lot of context. Resize and crop before you send.

Most of these models read images but answer in text. Making images or video is often a different model, like Google's Veo. And vision is good at reading a chart, a form or a messy invoice, but only approximate at counting things or saying exactly where something sits on the screen. Anthropic's docs list both as limits.

**Voice, and the pause**

Now the support bot with the pause. Every part of it might be fast on its own. So why does the whole thing feel slow?

Because in a conversation the parts add up. There are two common ways to build a voice agent.

The first is a chain of three. Speech-to-text turns the caller's audio into text, your model reads it and writes an answer, and text-to-speech turns that back into audio. If you've built streaming pipelines, it's one of those, with a latency budget every stage spends a bit of. The caller hears the total.

The second is a speech-to-speech model. One model takes the caller's audio and answers in audio. OpenAI's Realtime API and Google's Live API work this way, over a WebSocket or WebRTC connection that stays open in both directions. Compare that with SSE from Part 2, which only goes server to client. In a call, the caller can talk while the model is still speaking, so you need a two-way pipe.

OpenAI's voice agent docs describe the trade-off well, and here's how I'd summarise it.

| | Chained (speech-to-text, model, text-to-speech) | Speech-to-speech |
|---|---|---|
| Latency | higher, the stages add up | lower, one model, one hop |
| Control | you see the text in the middle and can check it before it's spoken | the middle is hidden inside one model |
| Swapping parts | change any of the three on its own | take the model as it comes |
| Cost | priced per stage, compare on your own traffic | audio tokens cost well above text tokens on OpenAI's realtime models |

So which one? If the call has to feel like a natural conversation, speech-to-speech. If you need to check every answer before it's spoken, or pick your own model and voice, the chain. OpenAI's docs suggest the chain when each stage needs to be visible or replaceable. And don't trust anyone's latency numbers. Measure your own, on your own phone lines.

Voice also has a problem text never had. Who's talking? The system has to notice when the caller has finished and stop speaking the moment they interrupt, which people call barge-in. The realtime APIs handle this for you, and it's still worth testing with real accents and real background noise.

**A call centre, end to end**

Let me put it together. Say a customer calls their bank about a card payment they don't recognise. What does the harness actually do?

![A flow with six steps: the caller phones in, the voice agent says it's an AI and listens, tools verify the caller and fetch transactions, it acts within limits such as blocking the card, it hands off to a human with a summary if needed, and after the call it writes notes and follow-ups.](assets/08-call-centre-flow.svg)

*What the harness does on one support call.*

The call comes in over the phone network. OpenAI's Realtime API, for example, can take calls over SIP, the protocol phone systems use over the internet. The agent first says the caller is talking to an AI. In the EU, the AI Act's transparency rules require people to be told when they're interacting with an AI system, and recording consent rules apply wherever you are.

Then it's the loop from Part 5 with audio at both ends. The agent listens, works out the intent and calls tools to verify the caller and fetch recent transactions. Then it acts, within limits. Blocking a card might be fine. Refunding money might not. Those rules belong in the harness, where the model can't talk its way past them.

When it's out of its depth, or the caller asks for a person, it hands off to a human with a short summary so the customer doesn't repeat everything. After the call, it writes up notes and flags any follow-up.

One thing to watch. A transcript is full of personal data, card numbers and addresses included. Decide what the model sees, what gets logged and what gets masked. I wrote about that in [One Redaction Function Is Always Wrong](https://www.allsrc.dev/posts/pattern-redaction-boundary/).

This is a preview. We'll come back to this call centre when we look at multiple agents, distributed systems and governance.

**Computer use, and an honest check**

Last one. You may have seen an agent drive a browser or a desktop. Anthropic and OpenAI both offer computer use in their APIs. It's the same tool loop again. The model looks at a screenshot, asks for a click or some typing, your code does it on a machine, takes a new screenshot and sends it back. The model never touches the machine. Your harness does.

It's useful when there's no API, like an old internal app with only a screen. But be realistic. Every step is a round trip with an image in it, so it's slow and fills the context fast. Long tasks still go wrong more often than you'd like. And every page it reads is untrusted. Anthropic's docs say instructions on a web page can override yours, and both providers recommend an isolated VM or browser, an allowlist of sites, and a human to confirm anything hard to undo. My post [A Regex Is Not A Sandbox](https://www.allsrc.dev/posts/pattern-sandboxed-execution/) covers the isolation side.

If an API exists, use the API. It's faster, cheaper and far easier to test than clicking through a screen.

Next: what happens when one agent isn't enough.