Keepford: an AI phone receptionist where code, not the LLM, decides

Hear it and read it: the two-call recording below · the full case study (PDF) · keepford.com

Context

Keepford is an AI phone receptionist for home-service businesses, and an independent project of mine: not a client engagement, and not a product with customers. Phase 1 built the voice receptionist: a Retell voice agent on a Twilio number, backed by an application built with Python, FastAPI and PostgreSQL. Built with Python (Keepford); primary stack Ruby on Rails.

The most important result is a design rule that came out of real test calls rather than being assumed up front: each time a real call exposed a model failure or a vendor limitation, the fix moved a decision or action out of the model and into deterministic application code.

Challenge

A chatbot that gets something wrong produces a message the user can reread or retry. A voice agent acts in real time on a caller who cannot see what the system is doing, and some of its mistakes cannot be taken back.

• Safety: a caller who smells gas needs exact instructions and then needs to hang up and leave. A helpful follow-up question is the wrong answer.
• Booking correctness: "you're booked" is a commitment. If the appointment doesn't exist, nobody arrives.
• Transfers: these happen on the telephony leg, which the voice platform may not control.
• Provider failures: a calendar write can time out after it has succeeded.
• Speech recognition: the model reads a transcript, not the caller, and transcripts drop words and garble addresses.
• Opaque identifiers: LLMs are unreliable at copying long strings back exactly.
• Truthful confirmation: every "done" the caller hears must correspond to something that happened.

The central question is: what should the model be allowed to decide? My answer: the model owns the conversation, and the application owns anything with consequences.

Approach: where the LLM talks and where code decides

Twilio sends every call to the application first. The application registers the call with Retell and connects the audio, so it stays in the call path and can later redirect the live call itself.

• The model handles the conversation: understanding speech, keeping context, asking questions, speaking to the caller, and deciding when to call a tool.
• The application decides and acts: safety classification, service-area and supported-service decisions, availability, slot references, booking validation and persistence, transfer mechanics, failure handling, and classifying and storing every call's outcome. The model reaches it through seven tools behind a signed endpoint, and each tool result tells the model what happened and what it may say.

Business details are configuration, not code; a second, deliberately different fictional business (test-only, not live) runs its own scenarios through the same test kit with no changes to the core.

This is not a claim that the model cannot make mistakes. It still decides whether and when to call a tool, and a voice platform cannot force a tool call. Some behaviours, such as how it ends a spam call, still depend on the prompt. What the split guarantees is narrower: once a decision reaches the application, the decision itself is made by code, not by the model.

Key decisions

Life-safety calls are ended by the phone system, not the model

Safety classification was fixed-rule from the first build, and by the time of the real-call tests it covered gas, carbon monoxide and fire. The model passes the caller's words to an assess_situation tool, which classifies them with fixed rules. Given the same recognised words, the classification is deterministic, and no business configuration can switch it off. What the first design still left to the model was delivery: the tool returned the approved script with instructions to read it word for word and end the call, and trusted the model to do both.

The gas-leak scenario failed twice on real test calls. On the first attempt, speech recognition dropped the word "gas"; the fixed rules correctly did not treat an unnamed smell as a gas leak, so the model asked a clarifying question, and when the caller repeated themselves the script was read correctly but the model added speech of its own. On the second, the script was read verbatim, immediately, and then the model did not end the call.

Classification was not the problem, and the fix did not change it. The fix moved delivery and termination into the application. On a life-safety classification, the application saves the outcome, safety event, owner alert and exact script in one transaction. Only then does it ask Twilio to redirect the live call to a safety endpoint, which speaks the saved script with <Say> and ends the call with <Hangup/>. The model's call leg ends at the redirect, so it gets no further conversational turn, and the words come from the database, never from the model. If Twilio refuses the redirect, the model receives the script and the end-call instruction as before: a weaker guarantee, but the outcome and alert are already saved. The trade-off is deliberate: the caller hears the phone system's voice, not the assistant's, in exchange for exact words and a hang-up that doesn't depend on the model.

The third gas attempt passed, and so did a fresh re-verification call on a later build; the carbon-monoxide scenario passed first time on the same design. This is deterministic termination after classification, not a claim that every emergency will be detected. Two things upstream of the classifier remain limits: speech recognition can lose the trigger word, and the model has to pass the caller's words to the tool. Detection is also keyword-based, so unusual phrasing can be missed. Fire and smoke are implemented as a third category with automated tests, but fire was not tested on a real call. The scripts are approved defaults each business signs off; they are not safety advice, and Keepford is not an emergency service.

Don't make an LLM copy opaque IDs

The first booking design exposed opaque slot IDs: 62-character strings encoding technician, service and start time, which the model had to copy back. On an early build, the model sent back a corrupted copy three times; the application couldn't decode it and refused, but at that build it wrongly told the caller the time was unavailable. That failure led to two changes: a truthful message for an ID that doesn't decode, and a new rule that now anchors booking integrity - book only an ID that appears, by exact string match, in the list of slots this call was offered. Never decode an ID to decide whether it is bookable, because a mis-copied ID can decode to a different, perfectly valid time.

A later test call produced exactly that, and was the first time the new check caught the model: the caller chose 7:30, and three times the model sent the ID for 7:00, one character different. The check refused each one and nothing wrong was booked, but the conversation never recovered.

The second fix removed the copying. check_availability now labels each offered time with a per-call reference (S1, S2, S3) that is never reassigned. The model sends back only the reference; the application resolves it to the ID it stored for that call, then still applies the exact offered-slot check. An unknown reference such as S99 fails closed as slot_not_offered. The model now selects from a small constrained set while the application keeps authority over what each selection means. Three real calls passed on this build, one of them booking in a single tool call.

Say "booked" only after the write

After the offered-slot check, a booking takes a hold in PostgreSQL (an exclusion constraint on each technician's time rejects an overlapping hold outright, and holds expire quickly), re-checks the calendar for that exact range, reserves an ExternalRef record for the write, writes the appointment, and only then, in one transaction, stores the job, confirms the hold and stores the confirmation.

The caller hears "You're all set", followed by "We'll text a confirmation shortly", only after the write succeeds. If the write fails or times out, the hold is released and the result is pending_confirmation: the caller is told the office will text them to confirm the time, and the pending ExternalRef plus a retained lead become a reconciliation record for the office. On the demo line, neither text is actually sent.

Concurrency protection is enforced by the database and verified by automated race tests: 2 and 10 simultaneous bookings for one slot, and exactly one succeeds. Retries don't create a second appointment, and the pending_confirmation path is tested. Real calls demonstrated the success path only; the real two-phone race could not be run, because no second phone was available.

Transfers the application controls

Calls arrive through Twilio and reach Retell over SIP, and Retell's built-in transfer could not act on that Twilio leg. So the application took over: transfer_call claims the transfer and saves the destination, Twilio redirects the live call, the phone system speaks the business's emergency guidance (if any) and then "Connecting you now", and it dials the on-call number with a ring timeout. If nobody answers, the caller is told plainly what happens next and can leave a voicemail; the call is classified and one urgent owner alert is generated.

A real call shaped the ordering: on the first burst-pipe attempt the model spoke the guidance and requested the transfer in the same turn, and the redirect cut the guidance off mid-sentence. The application now saves the guidance and the phone system speaks it before the announcement. On the demo line the on-call number is an automated test line, not a technician, and owner alerts are generated and stored, not sent.

What real calls exposed

The real calls were not a demonstration at the end; they were part of the engineering loop. Each failed attempt was recorded with the build it ran on, and the ones that mattered changed the implementation:

• A dropped "gas" happens before the application sees the words; it is now a documented limit, not an unknown.
• A model that didn't end a safety call led to application-owned termination.
• Wrong slot IDs led first to the offered-slot check, then to per-call references.
• Guidance cut off by a redirect moved the guidance into the phone system.

Evidence

• Contract and scenario tests: every scenario passes the tool-contract layer for both fictional businesses (27 of 27 and 6 of 6), without telephony. About 1,200 automated tests run in CI, with linting, type checking and a business-data leak check; ten are deliberate expected failures that document known limits. No coverage percentage has been measured.
• Simulation: one scoped simulated-conversation run on the voice platform passed 8 of 8, covering the scenarios a simulation without a phone leg can exercise.
• Real-phone acceptance: 23 scenarios; 20 passed, 3 could not be run, none were failing at sign-off. The full per-scenario summary is in the appendix below.
• Performance (point-in-time, from the voice platform's own measurements, one tester calling the US demo line internationally): median about 1.1 s from the caller stopping to the agent speaking, 90th percentile about 2.5 s; the slowest booking tool call took 931 ms.

Scope: live vs planned

Live in Phase 1	Planned (not in Phase 1)
Live phone line and voice conversation (Twilio, Retell)	Google Calendar and GoHighLevel scheduling
Service, service-area and price-range handling	CRM sync and follow-ups, including missed-call text-back
Fixed-rule life-safety handling, application-owned termination	Invoicing and payments (QuickBooks, Stripe)
Application-controlled transfers with voicemail fallback	Automation flows (n8n, Zapier, Make.com)
Availability and booking with database holds	Vapi as a second voice platform; an MCP owner assistant
Call outcomes and missed-call reasons for every call	Sending SMS (messages are generated and stored today)
Platform-failure fallback audio (automated tests only)

How it was built

I set the architecture and engineering rules, defined the acceptance criteria, made the design decisions and ran the real-call testing. The code was written by an AI coding agent working under those rules, with CI enforcing them.

What this does not show

• One business: one fictional business is live; the demo is not a client deployment and has no customers.
• Callers: one tester, calling the US demo number from a non-US mobile. The evidence does not establish behaviour across multiple callers, accents, background-noise conditions or US-originating callers.
• Current build: only three real calls ran on the current booking build; other passes come from earlier Phase 1 builds, some re-verified and some not re-run.
• Concurrency: no real concurrent-phone test.
• SMS: none sent; confirmation texts and owner alerts are generated and stored, and sending them is planned, not implemented.
• Fallback: the platform-failure fallback audio is built and covered by automated tests, but hasn't been exercised on a real call.
• Speech recognition: it still changes words, including addresses and safety trigger words.
• Infrastructure: a single shared server without backups or rate limiting, not a hardened production environment.

What this demonstrates for client work

The interesting part of a voice AI system isn't getting an LLM to talk. It's deciding where the LLM stops having authority. The same approach carries to any workflow where a model talks to customers but the business carries the consequences:

• Don't let an LLM own decisions with irreversible consequences. Keep the side effects in code.
• Give models constrained references, not opaque identifiers. Selecting is more reliable than copying, and the application keeps authority over what a selection means.
• Treat vendor behaviour as an engineering constraint, not an assumption.
• Test with real phone calls, which expose failures that simulations miss, such as dropped words and redirect timing.
• Prefer a truthful "we'll confirm" to an optimistic "booked".

Appendix: real-call acceptance summary

20 PASS / 3 BLOCKED / 0 FAIL. 39 real calls from one tester, 19-22 September 2026.

What the evidence labels mean:

• Early Phase 1 build: the first acceptance builds.
• Re-verified: an earlier pass repeated with a fresh call on a later build.
• Late Phase 1 build: after booking hardening, before slot references.
• Current booking build: after slot references.
• Judged unaffected: the slot-reference change does not touch that row's path.

#	Scenario	Expected outcome	Result	Attempts	Evidence generation
1	Routine booking, new customer	Booked	PASS	3 (1 FAIL)	Current booking build
2	Existing customer booking	Booked	BLOCKED	0	Test setup would have overwritten a customer record from an earlier test call
3	Price question only	Details captured	PASS	2 (1 FAIL)	Late Phase 1 build; judged unaffected
4	Address in service area?	Callback details captured	PASS	1	Late Phase 1 build; judged unaffected
5	Outside service area	Out-of-area details captured	PASS	3 (1 FAIL regression)	Re-verified
6	Service not offered	Declined; details captured	PASS	3 (2 FAIL)	Early Phase 1 build; not re-verified
7	Slot taken by a competing caller	Booked after conflict	BLOCKED	0	No second phone for a simultaneous call; covered by automated race tests
8	After-hours booking by the AI	Booked	PASS	4 (3 FAIL)	Current booking build
9	Burst pipe in hours	Transferred, answered	PASS	3 (1 FAIL)	Re-verified
10	Burst pipe after hours	Transferred, answered	PASS	1	Early Phase 1 build; not re-run
11	Suspected gas leak	Safety script, call ends	PASS	4 (2 FAIL)	Re-verified
12	Carbon-monoxide symptoms	Safety script, call ends	PASS	1	Early Phase 1 build; not re-run
13	Transfer unanswered, voicemail left	Failed transfer, details captured	PASS	1	Early Phase 1 build; not re-run
14	Transfer unanswered, caller hangs up	Failed transfer, missed	PASS	1	Early Phase 1 build; not re-run
15	Caller demands a person, in hours	Transferred, answered	PASS	1	Early Phase 1 build; not re-run
16	Caller demands a person, after hours	Callback details captured	PASS	2	Re-verified
17	Hang-up during greeting	Abandoned early, missed	PASS	1	Early Phase 1 build; not re-run
18	Call drops mid-conversation	Dropped, missed	PASS	1	Early Phase 1 build; not re-run
19	Wrong number	Logged as wrong number	PASS	2	Re-verified
20	Sales / spam call	Logged as spam	PASS	2 (recorded deviations)	Re-verified
21	Withheld caller ID	Logged as withheld	BLOCKED	0	Carrier could not withhold caller ID
22	Fast talker gives address	Address read back; booked	PASS	2	Current booking build
23	Caller goes silent	Dropped, missed	PASS	1	Early Phase 1 build; not re-run. The platform did not end the call; the caller hung up