AI Call Quality Control System
An AI pipeline that listens to every customer call and tells a moving company whether the quote can be trusted
We built an automated system that transcribes every inbound call with speaker diarization, compares it against the quote sent to the customer using Gemini 2.5 Pro, and returns a structured pass/fail verdict — in about 15 minutes, with no human listening to a single recording.
Cases in database
940
Customer moves tracked end to end
Cases AI-analysed
122
Fully transcribed, scored, verdicted
REST endpoints
36
Express API, Joi-validated
Call to verdict
~15m
End-to-end pipeline latency
Direct answer
An AI call quality control system automatically transcribes, diarizes, and analyses every customer service phone call with an LLM, comparing what was said against the resulting quote and returning a structured pass/fail verdict in about 15 minutes — without a human listening to the recording.
The call and the quote sometimes told different stories, and nobody could tell which
The client runs a moving company in France with a small team of agents handling inbound enquiries by phone. A customer calls, describes their move — dates, volume of furniture, floors, whether there's a lift — and the agent takes notes. A few hours later, an email quote goes out.
The problem was that the quote and the call sometimes disagreed. An agent would mishear a detail: a customer says "second floor," the quote says "fourth." Packing materials discussed on the call would vanish from the quote. These discrepancies led to disputes, unhappy customers, and crews that turned up under-equipped because the quote didn't match reality.
There was also no coaching data. With no recording review process, the client had no way of knowing which agents were consistently asking the right questions and which ones were missing things — no visibility, no way to flag a problem before a quote went out.
There's no point training agents if you don't know what's actually happening on the calls.
A fully automated pipeline from call end to structured verdict, with no human in the loop
The system connects four external services — Ringover, RunPod, Gemini, and the client's WordPress back-office — into a single automated pipeline. From the moment a call ends to the moment a verdict appears in the dashboard, nothing requires a human to press a button.
The hardest bugs are the ones where the error message is completely accurate but tells you nothing useful about the actual problem.
1. Poll & match (every 15 minutes)
A Bull job polls the Ringover VOIP API for newly completed calls with recordings. Phone numbers are normalised to E.164 and matched against cases pulled from the client's WordPress back-office.
2. Transcribe with speaker diarization
Matched recordings are sent to WhisperX running serverlessly on RunPod. The output isn't a text blob — it's a labelled conversation showing exactly who said what, agent versus customer.
3. Analyse with Gemini 2.5 Pro
The labelled transcript is compared against the quote data using a carefully engineered, versioned prompt. Groq's Llama 3.3 70B is wired in as a faster alternative when reasoning depth isn't the priority.
4. Structured verdict
The model returns YES, NO, ESTIMATED_QUOTE, or MULTIPLE_QUOTES, with a written, human-readable explanation the operations team can act on directly.
What 122 fully-analysed calls actually found
Of 940 cases in the system, 122 have been through the full AI pipeline so far.
| Status | Count | Meaning |
|---|---|---|
| YES | 30 | Quote matches the call — approved to send |
| NO | 47 | Discrepancies found — corrections needed |
| ESTIMATED_QUOTE | 35 | All calls unanswered or voicemail |
| MULTIPLE_QUOTES | 10 | Client requested multiple variants |
A strict layered backend with async processing at every slow step
Routes only validate input and call services. Services contain all business logic and never touch the database. Models handle every database query through Knex — no raw SQL, no business logic in the wrong layer. It's the only way a solo-built system stays maintainable by someone else, or future me.
Ingestion
Processing Queue
Back-office
Why each piece of the stack earned its place
Node.js 20 + Express
The client's existing back-office was already in Node. Staying in the same runtime meant shared infrastructure and easier deployment, with Express thin enough to structure properly — no magic, no conventions hiding how things work.
PostgreSQL 15 + Knex
Relational data with real foreign keys and proper transactions. Knex gives readable, composable queries without a full ORM's overhead or raw SQL's footguns.
Redis + Bull
Transcription takes 30–90 seconds; analysis can take 2–3 minutes. Doing that in a request-response cycle means gateway timeouts and no retry logic. Bull gives delayed jobs, priority, retries, and a dashboard out of the box.
Gemini 2.5 Pro
The analysis prompt compares a multi-page quote against a multi-turn transcript and produces structured JSON with reasoning. Gemini 2.5 Pro handles the context window and reasoning depth this needs; Groq Llama 3.3 70B is wired in as a faster alternative.
WhisperX on RunPod
Standard Whisper gives a single transcript blob. WhisperX adds word-level timestamps and speaker diarization — the difference between "here's what was said" and "here's what the agent specifically said."
Ringover polling, not webhooks
Polling over webhooks was a deliberate reliability choice: if the server is down during a webhook delivery, that event is lost permanently. Polling every 15 minutes means you might be late, but you never miss a call.
Five bugs worth writing about
Most of the hard problems in a system like this aren't in the happy path — they're in the gap between what a symptom looks like and what's actually causing it.
WhisperX hallucinating on VOIP recordings, breaking the LLM downstream
Repeated filler tokens silently poisoned Gemini's input
Phone audio is compressed, sometimes noisy, and often has one-sided pauses where the customer is thinking. WhisperX occasionally hallucinated repeated filler tokens in that silence — one transcript contained " un un un un..." repeated 189 times, adding 378 characters of noise.
Gemini's API timed out on every analysis attempt for that case. The LLM wasn't failing to understand the content — it was choking on an absurdly repetitive input. Five job attempts failed in a row before the raw transcript was pulled and read by hand.
Fix: a post-processing step collapses any sequence of three or more repeated short tokens into a single instance, before the transcript ever reaches the analysis queue. One-line fix; a morning to find it.
- Regex-based token collapse
- Runs before analysis queue
- Corrected transcript persisted to DB
- Root cause found via raw transcript read
Bull's job timeout racing Gemini's HTTP timeout
Two “failures” logged per attempt, for one already-retired job
Bull marked a job failed after 180 seconds. Gemini's axios call was configured with a 240-second timeout. Bull would fail and retry the job — then, ~60 seconds later, the still-running Gemini response would arrive and the old catch block would log a second, unrelated-looking error for a job that had already been retired.
It looked like two separate failures per attempt, which made debugging deeply confusing.
Fix: Bull's job timeout was raised above Gemini's HTTP timeout, and an AbortController cancels the axios call the moment Bull actually cuts the job — no more phantom errors.
- Bull timeout raised above HTTP timeout
- AbortController tied to Bull cancellation
- No more duplicate error logs
French phone number formats breaking call-to-case matching
+33 6 12 34 56 78, 0612345678, 06.12.34.56.78 — all the same number
The system matches calls to cases by phone number. WordPress stores whatever format the agent typed. Ringover returns E.164. Without normalisation, that's zero matches and the whole system falls over.
Fix: a phone normalisation utility strips all non-digits, applies French prefix rules (06/07 numbers get a +33 prefix; numbers already in +33 format stay as-is), and rejects known junk formats the WP form accepts but shouldn't. All comparisons happen on the normalised form.
- Strips all non-digit characters
- French 06/07 → +33 prefix rule
- Unit-tested per format variant
The analysis prompt took weeks, not hours, to get reliable
Early versions hallucinated discrepancies; later versions missed real ones
Getting a structured, reliable verdict across voicemails, two-minute quick calls, and 40-minute detailed consultations required many iterations, and without infrastructure to test prompts, every change was a blind guess.
Fix: a training interface built into the admin dashboard — define test scenarios (transcript + expected verdict), run them against any stored prompt version, compare outputs side by side. Prompt versions are immutable database records; the active prompt is a setting. This turned prompt engineering from guesswork into something resembling a test suite.
- Versioned, immutable prompt records
- Scenario runner in admin dashboard
- Side-by-side output comparison
- One-click rollback to any prior version
313 “unmatched” calls that turned out not to be a bug
Ringover logs every call on the line — not just customer enquiries
The webhook logs dashboard showed hundreds of calls that hadn't matched a case. It looked like the phone matching was broken. It wasn't: Ringover logs supplier calls, agent-to-agent calls, and junk numbers too — none of which match a case, and all of which pile up in pending_calls. The system was working exactly as designed.
The 313 broke down as 153 voicemails/hangups, 125 short calls (<2 min), 12 medium calls (2–5 min), and 23 real calls (5min+) worth investigating.
- 153 voicemail / hangup
- 125 short calls (<2 min)
- 12 medium calls (2–5 min)
- 23 real calls (5+ min, worth reviewing)
Six milestones, delivered solo
Project bootstrap & infrastructure
Node.js skeleton, PostgreSQL schema and migrations, Redis + Bull, auth middleware, env config, and the full layer structure enforced from day one. WordPress sync endpoint so the client could start backfilling cases immediately.
Ringover integration
Polling job every 15 minutes, phone normalisation, call-to-case matching, recording download, transcription job queuing. Idempotency check so an already-processed call_id is skipped silently.
AI engine — transcription & analysis
WhisperX transcription via RunPod with speaker-diarization post-processing, LLM analysis service with structured JSON output, the initial prompt, and retry logic. The most technically dense milestone — everything downstream depends on it.
Full API & back-office dashboard
All 36 REST endpoints, Joi validation, rate limiting, EJS dashboard with Alpine.js, case detail view, verdict display, webhook logs. The system could be fully operated by a non-technical user from here.
Exceptional situations
Handling calls that don't fit the standard pattern — all voicemail, multiple quotes requested, estimated quotes without a proper conversation. Each gets its own status, dashboard display, and message to the ops team.
Training interface & prompt management
Admin interface for prompt versions, test scenarios, and side-by-side comparison. Agent performance dashboard showing score trends per agent over time.
Every call gets checked now, and agent coaching finally has data
122 cases have been fully analysed. Of those, 47 came back NO — real discrepancies between the call and the quote that would previously have gone completely undetected. The system runs unattended: PM2 keeps it alive, GitHub Actions handles deploys, and the polling job matches calls without manual intervention. The only thing a human has to do is action the NO verdicts — exactly where a human should be.
47 real discrepancies surfaced
Revenue risk and customer-satisfaction risk that had zero detection mechanism before this system existed — each with a written breakdown of exactly what didn't match.
Agent coaching has data for the first time
Agents are scored on a consistent rubric — floor numbers, furniture volume, special items. The operations manager can now see, per agent, how scores trend over time.
35 estimated quotes flagged
Jobs where a quote went out without a proper conversation — a new data point the client didn't have before, so those cases can be followed up on deliberately.
2pm call, 2:15pm verdict
The pipeline runs continuously with no manual triggering. A call that ends at 2pm has a structured QC verdict roughly 15 minutes later, every time.
What held up, and what I'd do differently
The architecture held up well. The layering discipline applied from M0 meant there was never a need to go back and refactor a service because it was doing database queries it shouldn't. The migration pattern kept the database clean. Queue-based processing meant failures were contained and retryable rather than cascading.
If starting over, I'd invest more time upfront in the prompt-testing infrastructure — it was built in M5 but should have been M1 or M2. Having a way to test and compare prompts early would have saved a lot of the trial-and-error that happened mid-build.
I'd also think harder about the WhisperX hallucination problem earlier. The fix was trivial once I knew what to look for, but finding it took real debugging time because the symptom (an LLM timeout) was so far removed from the cause (a bad transcript). Logging the raw transcript on failure would have surfaced this immediately.
The thing I'm most satisfied with is that the system is genuinely useful — not a proof of concept. The operations team uses the dashboard, the QC verdicts are actioned, and the agent scores are reviewed. This one earns its place.
Questions about this system
What is an AI call quality control system?
An automated pipeline that listens to every customer service phone call, transcribes it, identifies who said what, and uses a large language model to verify whether the information given by the agent was accurate — returning a pass/fail verdict before any quote or contract goes to the customer.
How does WhisperX speaker diarization work?
WhisperX extends OpenAI Whisper with forced alignment and speaker diarization: it identifies different speakers in an audio file and assigns each spoken segment a speaker label (e.g. SPEAKER_00, SPEAKER_01). For call quality control, this is what lets the system distinguish what the agent said from what the customer said.
Why use Gemini 2.5 Pro for call analysis instead of GPT-4 or Claude?
Gemini 2.5 Pro was chosen for its long context window, its structured-output feature (which enforces the JSON schema at the API level rather than hoping the model returns valid JSON), and its reasoning depth on comparison tasks. The system is model-agnostic by design — Groq Llama 3.3 70B is also wired in and can be swapped in via a settings change.
Why poll Ringover every 15 minutes instead of using webhooks?
Webhook delivery requires the server to be running at the exact moment the event fires. If the server is deploying, restarting, or crashed during delivery, that event is lost permanently. Polling every 15 minutes guarantees every completed call is eventually seen, with a maximum delay of one poll interval — the right trade-off when accuracy matters more than real-time speed.
How do you prevent the same call from being processed twice?
Every Ringover call has a unique call_id. Before queuing any processing job, the system checks whether that call_id already exists in the database, and skips the record silently if so. This idempotency check means the poll job can run as often as needed with no risk of duplicate transcription or analysis.
Can this AI call QC system be adapted for other industries?
Yes. The core pipeline — VOIP integration, WhisperX transcription, LLM analysis with a versioned prompt, structured verdict output — is industry-agnostic. The industry-specific part is the analysis prompt and the verdict schema; the pipeline stays the same while the prompt and schema change.
What happens when WhisperX produces a bad transcript?
A post-processing step runs on every transcript before it reaches the analysis queue, cleaning common artefacts like hallucinated repeated tokens, empty segments, and non-speech noise markers. If the cleaned transcript is still below a minimum useful length, the case is flagged for manual review instead of generating a low-confidence verdict.
How is the LLM prompt managed in production?
Prompt versions are immutable database records, with the active prompt controlled by a configuration setting. Any admin can define test scenarios, run them against any stored prompt version, and compare outputs side by side before promoting a new version to production. Rolling back is a single database update.
Need every call checked automatically, not spot-checked by hand?
I build production AI pipelines that turn calls, documents, and forms into structured, actionable data — wired into the systems you already use.
Start a conversation