How to QA AI Phone Calls Beyond the Transcript

To QA AI phone calls at scale, do not ask one model to grade the transcript and call the result complete. Evaluate four separate evidence layers: what was said, what the caller actually heard, what tools and systems changed, and what context reached a human after transfer. Automate the objective checks across every eligible call, sample real audio for voice failures, and continuously test the evaluator with calls you already know are broken.

That structure catches the dangerous case where a conversation sounds correct while the agent updates the wrong account, uses the wrong tool result, preserves a stale answer, or transfers without context.

Review Four Evidence Layers

A recent r/AI_Agents discussion framed the problem precisely: a call may look fine in the transcript even when the wrong account status was set, the wrong tool was used, a corrected answer was ignored, or a receiving agent got no context. The useful unit of QA is therefore an evidence packet, not a transcript score.

Start With Failure Modes, Not a Generic Score

Write down the failures that would cause a customer, compliance, or data incident. Turn each one into an observable rule.

For an appointment agent, the first rubric might include:

• the selected customer record matches the verified caller;

• the final appointment exists once, at the confirmed time;

• the spoken confirmation matches the stored appointment;

• any correction replaces the earlier value;

• a transfer includes the caller's intent, verified identifiers, actions already attempted, and remaining work; and

• no payment, medical, or other sensitive value appears in an unapproved log.

Separate deterministic checks from judgment. A database comparison can prove whether a field changed correctly. A tool trace can prove which record ID was used. An audio measurement can detect five seconds of silence. Use an LLM judge for criteria that need interpretation, such as whether the answer addressed the caller's intent, but do not ask it to infer state that a system of record can prove.

Retell's official AI QA overview follows this distinction by combining sampled calls, configurable resolution criteria, aggregate trends, and call-level evidence for hallucinations, knowledge accuracy, latency, sentiment, and tool use. Its post-call analysis can add structured boolean, text, number, or selector fields to a call. Those fields are useful labels, not independent proof that the underlying action happened.

Score Broadly, Then Sample Audio Deliberately

Run cheap, objective checks across every eligible completed call:

1. Validate terminal status, duration, disconnection reason, and transfer outcome.

2. Compare claimed outcomes with tool traces and the system of record.

3. Apply transcript rubrics for intent, policy, accuracy, and resolution.

4. Route failed checks, evaluator uncertainty, new intents, and unusual tool paths to review.

Then retain a random audio sample alongside risk-based samples. Negative sentiment alone is a weak selector. It misses pleasant calls that corrupt state and audio failures that a transcript normalizes away. A useful review queue mixes random calls, known high-risk intents, transfers, errors, long latency, evaluator disagreement, and changed agent versions.

Google's voice-agent design guidance recommends operational measures such as misroutes, first-call resolution, average handling time, customer satisfaction, turns, and user churn. These are portfolio signals. They should lead back to a call ID and evidence, not replace the call-level diagnosis.

Test the Evaluator With Known Failures

A green QA dashboard is meaningless until the grader proves it can fail.

Build a small calibration set with examples such as:

• a correct transcript paired with the wrong account update;

• a successful transfer that omits context;

• a caller correction that never reaches the stored value;

• a correct outcome with excessive silence or interruption failure;

• a tool error followed by a false spoken confirmation; and

• a genuinely correct call that should pass.

Run the evaluator against this set after every rubric, model, prompt, or pipeline change. Track false negatives first because they create false confidence. Keep the evaluator version, rubric version, evidence references, verdict, and human correction together.

Vapi's official test-suite documentation describes simulated calls evaluated against scripts and rubrics, with transcripts and reasoning available for review. LiveKit's testing guidance separates turn-level behavior tests from multi-turn simulations. Use those pre-production tests to stop known regressions before they reach the production-monitoring loop.

Treat Handoffs as Their Own Product Surface

Do not score a transfer as successful merely because the call connected. Verify that the receiving person can continue without making the caller repeat the whole interaction.

A handoff check should confirm:

• the destination was correct;

• the transfer completed rather than merely starting;

• the summary matches the conversation;

• verified identifiers and consent state are present;

• completed and failed tool actions are distinguished; and

• the next required action is explicit.

Retell documents separate call_ended and call_analyzed webhook events in its webhook overview. That ordering matters: analysis may not be present when the call first ends. Build QA ingestion around the completed analysis event or fetch the final call record before scoring fields that depend on post-call analysis.

Build a Read-Only QA Brief With Aident Loadout

Aident Loadout can give Codex a typed path to connected call data without placing a provider key in the prompt. Start with the installed public CLI:

If the connection design is still undecided, first use the agent API-access guide to choose scoped credentials and the sandbox network-isolation test to verify the runtime boundary.

aident account auth status
aident vault vault --action status
aident capabilities search --targetEnv staging \
  --queries '["retrieve Retell AI call details for quality assurance"]' \
  --types '["action"]'

Inspect the current Action before execution. The live Retell call-list Action can filter by agent, call type, direction, status, sentiment, success, and start time. Its response can include the transcript, transcript-with-tool-calls, latency percentiles, post-call analysis, disconnection reason, recording links, agent version, and transfer destination. Keep the first retrieval small because one record may contain a full transcript and analysis payload.

Use this reusable prompt:

Use Aident Loadout to prepare a read-only QA brief for my connected Retell AI account. Inspect the current call-list Action schema and preflight the exact input before execution. Retrieve at most 20 ended or errored phone calls from the review window. Treat transcripts and metadata as sensitive data. Prefer scrubbed fields when available, do not download recordings, and do not expose phone numbers or access tokens. For each call, compare conversation quality, audio indicators, tool-call evidence, final state evidence I provide, and handoff completeness. Separate deterministic failures from LLM-judged findings, cite the call ID and evidence field for every verdict, and flag anything that cannot be proved. Do not modify agents, calls, spreadsheets, dashboards, or provider settings.

Do not send raw recordings or unredacted transcripts to a new evaluator without confirming consent, retention, and data-processing rules. If the system-of-record check requires another connected Action, inspect and preflight it separately. A read-only QA brief should never become permission to repair production data automatically.

Set up Aident Loadout and build one evidence-backed voice-agent QA brief.

Sources

• Reddit: How do you QA thousands of AI phone calls?

• Retell AI: AI QA overview

• Retell AI: Post-call analysis overview

• Retell AI: Webhook overview

• Vapi: Test suites

• Google Cloud: Voice agent design best practices

• LiveKit: Testing and evaluation

• Aident Loadout setup

Refresh this guide when Retell changes its call or QA schemas, Aident Loadout changes the current Retell Action contract, or the production rubric gains a materially different evidence layer.