AI Voice Agents for Customer Service: What They Can and Can't Do

If you run support for a small or mid-sized business, you have probably had a vendor tell you that an AI voice agent for customer service can "answer every call". Some of that is true now, some of it is a demo that falls apart on a real phone line, and the difference is not obvious until you have paid for a month of minutes.

This explainer is for the founder, operations lead or support manager who already runs chat support and is wondering whether voice is the next step. It covers how a voice agent works, what latency and interruption handling mean in practice, what voice agents do well and badly, what a minute costs, the two compliance rules you cannot skip, and a checklist to run any vendor through.

It is honest about limits, including PepoChat's own: its voice option runs on Vapi through your own Vapi account, and the voice assistant does not yet share the chat agent's knowledge base or show up in the team inbox. The last section says exactly what that means.

What is an AI voice agent for customer service?

An AI voice agent is a program that holds a spoken conversation with a caller in real time: it listens, decides what to say using a large language model, and speaks the answer back. That is different from the phone tree you already know. An IVR (interactive voice response) system plays fixed recordings and routes on keypad presses or a handful of keywords; a voice agent understands free-form sentences and can answer them, ask follow-up questions and call tools such as a calendar or an order-lookup API.

It is also different from a chat agent with a microphone bolted on. In text chat a two-second pause before a reply is fine and the visitor can re-read. On a call, a two-second silence sounds like a dropped line. Almost every engineering decision in voice comes from that one constraint.

In practice, "AI voice agent for customer service" covers two deployment shapes. A phone agent answers a real telephone number, so it lives inside telephony rules about consent and disclosure. A web voice agent lets a visitor click a button on your site and talk to the assistant through their browser microphone, which is how PepoChat's voice option works. The pipeline underneath is the same; the compliance surface is not.

How does the voice agent pipeline work?

The classic architecture is three models in a row, and most platforms, Vapi included, still build on it and let you pick a provider for each stage.

1. Speech-to-text (STT) converts the caller's audio into a transcript as they talk. Speech recognition, as Wikipedia defines it, covers "methods and technologies that translate spoken language into text", and its accuracy is measured as a word error rate.
2. The language model (LLM) reads the transcript plus your instructions, decides what to do (answer, ask, look something up, hand off) and produces text. This is the same kind of model that powers a chat agent, and in a good setup it answers from your knowledge base rather than from memory.
3. Text-to-speech (TTS) turns that text into audio. Speech synthesis is the artificial production of human speech, and its quality is judged on naturalness (does it sound human) and intelligibility (is it easy to understand).

Each stage adds delay and can introduce errors the next one cannot see. If STT hears "cancel" as "council", the LLM confidently answers the wrong question and TTS reads it beautifully.

Speech-to-speech models: one model instead of three

The newer approach collapses the pipeline. A speech-to-speech model takes audio in and produces audio out directly, without a separate transcript in the middle. OpenAI's Realtime API guide describes a "speech-to-speech voice agent" that "works directly with audio, maintains conversation state, and can call tools", connecting over WebRTC in the browser or WebSocket on a server, with built-in voice activity detection and support for barge-in.

The advantages are lower first-audio latency and access to things a transcript throws away: tone, hesitation, whether the caller sounds annoyed. The trade-offs are less control (you cannot swap the STT vendor for one language, or inspect the exact transcript the model acted on), fewer voices, and usually a higher per-minute price than a tuned three-stage pipeline. Many 2026 deployments still run the pipeline for cost and control.

Why latency and interruption handling decide whether a voice agent is usable

Latency in a voice agent is the gap between the caller finishing a sentence and the agent starting to speak. Vapi's introduction page advertises "sub-600ms response times with natural turn-taking", and that figure is a reasonable bar: under roughly half a second feels natural, one second feels like a slow line, two seconds and callers start saying "hello?".

That budget covers the network hop, the STT finalising the last words, the model's time to first token and the TTS time to first audio. Streaming is what makes it work: a good pipeline synthesises the first clause of the answer while the model is still writing the rest.

Endpointing: knowing when the caller has finished

The hard problem is not speed but deciding when to speak. Endpointing is the logic that decides a caller has finished their turn rather than paused to think. Too eager and the agent talks over people; too patient and every reply feels slow.

Vapi's speech configuration docs show what this looks like in practice: a "start speaking plan" with a default wait of 0.4 seconds after the caller pauses, and smart endpointing options that use audio cues (prosody), transcript cues (punctuation, silence after a question) or a combination to guess whether the turn is over. Reading a 16-digit order number aloud, with pauses between groups, is the classic case that trips naive endpointing.

Barge-in: letting the caller interrupt

Barge-in (interruption handling) is the agent's ability to stop talking when the caller starts. Without it, a caller who says "no, the other order" waits through a 20-second monologue. The same Vapi docs describe a "stop speaking plan" with a word-count threshold before the agent stops, a voice-activity window that defaults to 0.2 seconds, and a one-second back-off before the agent resumes.

A word threshold of one means a cough or "mm-hm" cuts the agent off; a threshold of three means the caller must say a full phrase before the agent yields. There is no universally right setting, so a platform that lets you tune it is worth more than one that promises it "just works".

What AI voice agents do well in 2026

Voice agents are at their best on calls that are frequent, short, and answerable from a document or a database.

FAQs and policy questions. Opening hours, return windows, whether you ship to Ireland, how to reset a password. These are the questions your chat agent already answers from its knowledge base, and a voice agent grounded on the same content gets them right for the same reason.

Appointment booking and rescheduling. "Can I move Thursday's appointment to next week?" is a well-bounded task: check availability, offer slots, confirm. Tool calling makes it reliable because the model asks a calendar rather than guessing. PepoChat's chat agent does this against a real calendar; on the voice side the equivalent is configured on the voice platform.

After-hours coverage. The most common first deployment. Overnight and weekend calls that would otherwise hit voicemail get answered, common questions get resolved, and everything else is captured as a message with a name, a number and a summary for the morning. The bar is low and the risk is contained.

Status lookups. "Where is my order?", "Has my ticket been assigned?", "When does my subscription renew?". With a read-only tool wired to Shopify, a helpdesk or Stripe, the agent reads the answer from the system of record instead of guessing.

What AI voice agents still can't do reliably

Complex judgement. "My order arrived damaged and I'm leaving for a trip tomorrow, what can you do?" needs someone weighing policy, cost, goodwill and the specifics. An agent can collect the facts and offer the documented options, but deciding to break policy is a human job. Wire it so the agent escalates rather than improvises; the human handoff post covers how that should work on chat, and the principles are the same on a call.

Accents, dialects and background noise. Speech recognition still degrades on exactly the inputs real support lines are full of: strong regional accents, code-switching, a caller on a train, a speakerphone in a warehouse. Wikipedia's speech recognition article puts it bluntly: "Vocalizations vary in terms of accent, pronunciation, articulation, roughness, dialect, nasality, pitch, volume, and speed. Speech is distorted by background noise, echoes, and recording characteristics." Every downstream stage inherits those errors, so test with your real callers' audio, not the vendor's demo voice.

Emotional escalations. An angry or distressed caller wants to be heard by a person and can tell within seconds that they are not. A voice agent that keeps cheerfully offering the FAQ answer to someone who is crying makes things worse. Detecting frustration and handing off quickly is a feature to demand; some platforms do it on keywords, some on tone, and speech-to-speech models have an edge because they can hear it.

Regulatory and financial disclosures. Anything where the exact words matter: cancellation terms, medical or legal information, payment authorisations, identity verification. A language model paraphrases, and a paraphrased disclosure is not a disclosure. Keep these on scripted recordings or a human, and keep payments and refunds out of the agent's tool set entirely; misheard digits plus a financial side effect is the worst combination voice offers.

What does an AI voice agent cost per minute?

Voice pricing is per minute of conversation, and the minute is usually assembled from separately billed parts. Understanding the parts is more useful than any headline number, because vendors bundle them differently.

As of September 2026, Vapi's pricing page lists its platform fee at $0.05 per minute on the usage-based tier, with the other components billed at the chosen provider's rates. Its own examples put Deepgram transcription at roughly $0.0095 to $0.0099 per minute, OpenAI models at $0.0077 to $0.0452 per minute depending on the model, and ElevenLabs voices at $0.0146 to $0.0238 per minute, and its calculator estimates $82 to $129 per month for 1,000 minutes. Check their pricing page for current numbers; provider rates change often.

Two qualitative points hold whichever platform you pick. A fully loaded minute in 2026 is priced in cents, not dollars, so the economics favour many short calls. And the model choice is the biggest swing in the bill: a small fast model can be several times cheaper per minute than a frontier one, and for FAQ-style calls it is usually good enough. Speech-to-speech models tend to price audio by the token rather than the minute, which is harder to forecast, so run a pilot before committing.

PepoChat does not resell voice minutes. You connect your own Vapi account, so Vapi bills you for voice usage; PepoChat's own plans (every feature on the free plan, limits removed on Pro; see pricing) cover the chat agent, knowledge base and team inbox.

Compliance basics: say it's AI, and get consent to record

Two rules cover most of what a small business needs to get right. This is not legal advice, and you should check the rules for every country you take calls from, but they are the baseline any buyer should insist on.

Disclose that the caller is talking to an AI. In the European Union, Article 50(1) of the AI Act requires providers to ensure that AI systems "intended to interact directly with natural persons" are designed so that people "are informed that they are interacting with an AI system", unless it is obvious from the context; the obligation applies from 2 August 2026. In the United States, the FCC's February 2024 declaratory ruling confirmed that calls made with AI-generated voices are "artificial" under the Telephone Consumer Protection Act, so outbound AI calls fall under the same prior-express-consent rules as robocalls. The practical answer is a one-line opener: "Hi, you've reached Acme. I'm an automated assistant; I can help with orders and bookings, or say 'agent' to reach a person."

Get consent before recording. Voice platforms record and transcribe by default, because that is how you review quality. Some jurisdictions require only one party (you) to know; others require everyone on the call to consent. The Wikipedia summary of call recording laws lists the US states that require all-party consent, including California, Florida, Illinois, Pennsylvania and Washington, and notes that in Germany recording without the consent of all parties is a criminal offence. The safe default is to announce recording at the start of every call and let the caller hang up or ask to opt out.

Beyond those two, apply the same data hygiene you would to chat: where transcripts and recordings are stored and for how long, who at the vendor can access them, and how you delete a caller's data on request. The questions in GDPR and AI chatbots apply unchanged to voice, with recordings as an extra category of personal data.

How to evaluate an AI voice agent: a checklist

Run every vendor, and your own pilot, through this table. Score each row before you look at the price.

Where PepoChat's voice option fits, honestly

PepoChat is a chat-first AI support agent. The voice option exists, and it is worth being precise about what it is and is not.

What it is: an optional voice conversation inside the PepoChat chat widget, powered by Vapi. You connect your own Vapi account with your public key and an assistant id, and visitors get a voice button in the widget. The voice assistant itself (prompt, voice, model, tools, speech settings) is configured on Vapi's platform, which also bills you for the minutes.

What it is not yet: the voice assistant does not share the chat agent's knowledge base, and voice calls do not appear in the team inbox. Content you train the chat agent on does not automatically reach the voice assistant, and operators will not see a voice call in the inbox, so the handoff path for voice has to be built on the Vapi side, for example by ending the call with a message capture or a transfer.

What to do next

If you are new to AI support, start with chat: it is cheaper per conversation, easier to review, and it forces you to build the knowledge base that any voice agent will need later. The guide to training an AI chatbot on your website and PDFs is the place to begin, and the human handoff post linked above explains the escalation design both channels depend on.

If you are ready to test voice, set up a Vapi assistant for one narrow job (after-hours message capture is the classic first one), run the checklist above against it with real callers, and only then connect it to the PepoChat widget. You can create a free workspace with every feature on the free plan, and the contact page is there if you want a second opinion on whether voice is the right next step for your volume.