Most teams evaluate an AI voice agent the wrong way. They listen to a two-minute demo, hear a pleasant voice, and assume the hard part is the model. The hard part is everything around the model: the phone network, the speech pipeline, the calendar, the CRM, and the half-second of silence that makes a caller hang up.
KeenCraft builds and operates more than ten live AI platforms, including VoiceCake, Rawk.ai, and Dynaris. This article is the production checklist we use before an agent is allowed to sit on a real phone number.
What a production voice agent actually is
An inbound agent answers a missed or after-hours call, identifies intent, collects the facts it needs, and either completes the job or hands off cleanly. An outbound agent places follow-up or reminder calls with the same discipline. Both are phone systems first. The LLM is one component.
A working stack usually looks like this:
- Telephony: Twilio or a comparable SIP/PSTN layer so the business keeps a real number, recording policy, and failover path.
- Speech-to-text: Deepgram or equivalent, tuned for the accent mix and vocabulary of the practice (provider names, insurance terms, service codes).
- Reasoning: an LLM behind LangChain or LangGraph so the conversation can branch, retry, and recover instead of following a rigid tree.
- Text-to-speech: ElevenLabs or a similar voice, with barge-in so the caller can interrupt.
- Orchestration: Vapi or Retell AI when they fit, plus custom glue when they do not.
- Systems of record: the calendar, CRM, and notification tools the staff already trusts.
The latency budget is the product
Callers do not forgive lag the way chat users do. A 900ms gap after every sentence feels like a bad connection. People talk over the agent, the agent talks over them, and the call collapses into noise.
On Rawk.ai, a live voice-agent builder platform, KeenCraft led a latency pass that dropped end-to-end response time from roughly 900ms to roughly 320ms. That number is not a marketing flourish. It is the difference between a call that feels robotic and one that feels human.
A practical budget, measured from the end of the caller's utterance to the start of audible speech:
- STT partials and endpointing: keep streaming; do not wait for a perfect transcript before thinking.
- LLM time-to-first-token: stream the reply; never buffer a full paragraph.
- TTS time-to-first-byte: start audio as soon as the first clause is ready.
- Tool calls: prefetch calendar availability before you need it, and cache what does not change mid-call.
- Network: colocate STT, LLM, and TTS as close as the providers allow. Extra regions add tens of milliseconds you cannot buy back with a better prompt.
Sub-400ms is the range we aim for on client builds. You will not hit it on day one if the agent is doing three sequential tool calls on a cold calendar API. That is an architecture problem, not a prompt problem.
Conversation design that survives a real caller
Production callers do not follow the happy path. They give a date without a time. They change their name spelling twice. They ask a question the prompt never mentioned. They have a crying child in the background.
Collect facts in a stable order
For appointment booking, the agent needs a small, explicit slot list: who is calling, what service, which location if there is more than one, preferred window, and a reachable callback number. Confirm each high-stakes field out loud before writing it. Do not invent a slot the calendar did not offer.
Define failure, not just success
Every agent needs an escape hatch: transfer to a human, take a message, or schedule a callback. Infinite retry loops are how callers decide the business is closed forever. On Dynaris, the front desk is not voice-only: chat and email sit in the same customer thread, so a failed call can continue as a message without losing context.
Isolate tools per workspace
Multi-location and multi-tenant systems cannot share tool servers casually. Dynaris runs voice agents and tools as separate MCP-based services so each workspace has its own tool surface. That isolation is a security and correctness requirement, not a nice-to-have.
CRM and calendar wiring is the booking
VoiceCake is a live 24/7 inbound platform for dental, healthcare, fitness, and mortgage clients. The proof that matters is not that the agent can talk. It is that a dental engagement books real appointments. That only happens when the agent can read open slots, write the booking, and trigger the same confirmation the staff already sends.
- Read availability from the system of record, not from a spreadsheet export.
- Write the appointment with the same fields a receptionist would: patient, provider, duration, notes, source.
- Send confirmation on the channel the business already uses (SMS, email, or both).
- Log the call outcome even when nothing was booked, so follow-up is not guesswork.
If the CRM is HubSpot, Salesforce, Zoho, or a custom system, the same rule applies. KeenCraft's automation work exists because a voice agent that cannot write a record creates a second, worse inbox.
A ship checklist we actually use
- Latency on live PSTN, not on a browser demo: measure p50 and p95 of time-to-first-audio.
- Barge-in: the caller can interrupt without the agent finishing a paragraph.
- Booking write-path: create, reschedule, and cancel against the real calendar.
- Identity: name, phone, and location disambiguation when two records look similar.
- Handoff: warm transfer or a complete message, never a dead air drop.
- After-hours vs in-hours behavior: different prompts, same system of record.
- Recording and retention: match the industry. Healthcare work, such as OptimateMD.health, needs data-handling designed in from architecture, not bolted on.
- Observability: transcripts, tool-call traces, and a way to replay a failed call without guessing.
When you should not build a voice agent yet
If the business has no calendar of record, no one who owns the front desk process, or a phone tree that already loses callers before a human answers, a voice agent will amplify the mess. Fix the workflow, then automate it. That is the same stance we take on custom software: ship the smallest system that owns the outcome.
If you do have missed calls, after-hours demand, or a receptionist drowning in routine booking, a production agent is one of the few AI products that shows up on the P&L in weeks rather than quarters. KeenCraft prices that work as a fixed Statement of Work, with conversation design and CRM wiring in scope before any code is written.