2 min readBuilding AI systems
AI phone agents that callers don't hate
Voice is where AI meets customers with the least patience. How to design phone agents that resolve calls, hand off well, and handle data carefully.
Voice is where AI assistants meet customers with the least patience. A caller can't skim, can't scroll back, and can tell within seconds whether they're being helped or handled. Done well, an AI phone agent answers every call, books appointments, and routes urgent issues straight away. Done badly, it's a phone tree with a nicer voice.
Keep it fast and short
Latency matters more on the phone than anywhere else. Long pauses feel like a dropped call, and long answers are hard to follow. Keep responses to a sentence or two, confirm one thing at a time, and let callers interrupt.
Design the handoff first
Decide when a person takes over before you design anything else: upset or distressed callers, anything urgent or safety-related, repeated misunderstandings, and requests outside the agent's scope. Transfer with context, so the caller never has to repeat themselves.
Connect it to the systems that matter
An agent that can only take a message adds little. The value comes from checking availability and booking, looking up an order or account, creating a ticket, or sending a confirmation text. Each of those is a tool with clear permissions, and anything that commits the business should be confirmed with the caller first.
Be upfront and careful with data
- Tell callers they're speaking with an AI assistant, and offer a person.
- Follow the call recording and consent rules that apply where you and your callers are.
- Collect only what the task needs, and don't take payment details by voice unless your payment setup is built for it.
- Treat transcripts with the same care as any other customer record.
The best phone agent is the one callers forget about, because their problem got solved.
Measure what callers experience
- Calls resolved without a transfer, and whether those callers rang back.
- Transfer rate, and the reasons behind it.
- Time to resolution compared with before.
- A weekly sample of calls reviewed by a person.