I'm Learning Voice AI by Building an AI Receptionist for a Restaurant

I don’t have a background in voice AI, and I’m not going to pretend this post is expert advice. What I have is a habit of learning things by building them badly first, then a little less badly. So when I got curious about what it actually takes to build an AI phone agent — the kind that answers a restaurant’s calls and takes an order — I decided to find out by doing it, in public, mistakes included.

This isn’t a launch. There’s no restaurant using this. It’s a working prototype: a real Rails backend connected to a real voice AI platform, capable of taking a full order over a real phone call — but very much still a learning project, not a product. If you’re looking for a production-ready blueprint, this isn’t that. If you’re curious what actually goes wrong when someone without voice-AI experience tries to build one of these, that’s what this is.

Code’s here if you want to poke at it: github.com/amitkssolanki/ai-receptionist.

The one architecture call I’m fairly confident about

Early on I had to decide: try to build the actual voice pipeline — speech-to-text, the model loop, text-to-speech, turn-taking — or lean on a platform that already does it. I leaned on a platform (Vapi), and even as a beginner at this, that felt like the right instinct. Real-time voice AI is genuinely hard and already solved by people who do only that. My part was everything behind the call: the actual business of running a restaurant over the phone.

Rails owns the menu, the orders, the customer records, the call logs, an admin dashboard. Vapi owns making the phone call sound like a phone call. That split is probably the single reason this got anywhere at all instead of becoming a rabbit hole in audio engineering I wasn’t equipped for.

A bug that had nothing to do with AI

Before any of the interesting stuff, I lost a chunk of time to something with zero glamour: Ruby gems failing to compile correctly on Apple Silicon. The Ruby install being used was an x86_64 build running under Rosetta, and every native gem extension kept compiling for the wrong architecture and crashing on load. The actual cause, once found: the Ruby install’s own config had its architecture flag set to an empty string, so the compiler silently defaulted to the machine’s native chip instead of matching the Ruby interpreter. One line fixed it for good. Not an AI lesson, just a reminder that “learning to build with AI tools” still means learning to debug toolchains like anyone else.

The platform didn’t work the way I assumed

I designed the webhook API — the endpoints the voice platform calls mid-conversation for menu lookups, adding items to a cart, submitting an order — based on an assumption I hadn’t actually verified: that the platform would bake a call’s identity into each tool’s URL. It doesn’t. Vapi sends every event to one single webhook URL, with the call’s identity inside the request body instead, and expects responses back in its own specific shape. Different transport model than I’d guessed.

Rather than reworking everything, I added one adapter — a translation layer that speaks Vapi’s actual shape but calls into the same underlying logic everything else uses. I don’t know if that’s the “correct” way an experienced voice-AI engineer would solve it. It’s the way I could explain to myself, and it worked.

The moment that actually taught me something

During a real test call, I (as the customer) asked for a Margherita pizza. Somewhere in the back-and-forth, the assistant said: “I’ll add garlic knots.”

It never actually did. No function got called. It just said it, confidently, in exactly the tone you’d want from a phone host.

What caught it: the design requires the assistant to pull the real cart from the database and read it back out loud before finalizing anything, then wait for an explicit yes. When it did that, the real data came back — one pizza, no garlic knots — and it corrected itself before anything wrong got submitted.

I didn’t design that safeguard because I knew this specific failure mode was coming. I added it because “always confirm before you finalize” seemed like obviously sensible advice for anything customer-facing. It happened to catch something I wouldn’t have predicted. That feels like the actual, unglamorous shape of building reliable systems around a component that can be confidently wrong: not clever prediction, just not skipping the boring verification step.

A bug I only found by clicking around

Separately — building an admin page to edit the restaurant’s timezone, I used a standard Rails helper for a timezone dropdown, and it silently selected the wrong timezone every single time, no error at all. The cause: that helper matches against one naming convention, and the database column stored a different one. Nothing in an automated test would have caught this unless I’d specifically thought to check it. It surfaced because I opened the page and looked at it. Small thing, but it’s shaping how skeptical I’m learning to be of “the tests pass” as a substitute for actually looking.

Keeping this free while I figure it out

One practical thing, since I’m doing this on a hobby budget, not a company card: every voice AI platform gives free trial credit, and it disappears fast if you’re not deliberate — real per-minute voice testing burns through it in under an hour once you count the model, transcription, and voice costs stacked together. What worked: test every piece of logic for free first (automated tests, plain HTTP requests against my own webhook, no phone call involved), then use the platform’s own browser-based test-call feature (still free, no telephony) to validate the actual conversation before spending a single real phone-call minute. By the time real minutes were on the line, only “does this sound natural on an actual call” was left to check.

Where this actually stands

To be direct about it: this is a prototype I built to learn, not a product anyone should deploy. It hasn’t been used by a real restaurant. It hasn’t been load-tested, security-reviewed, or run past an actual voice-AI or ML engineer for a sanity check. The order-taking logic has a known simplification (no real concept of “modifier groups,” so nothing stops someone from technically selecting three drinks on a combo meal). It’s not deployed anywhere permanent. The “real inbound phone call, real caller ID” path is configured but hasn’t been stress-tested with real call volume.

What I can say is that it works end to end for a single real test order — menu lookup, cart building, confirmation, submission — verified in the database, not just in a transcript. For someone learning this space from close to zero, that felt like a real milestone, even if it’s a small one by the standards of anyone who does this professionally.

If you’ve built something similar, or you do this for a living and can see something I got wrong or could do better, I’d genuinely like to hear it — that’s the whole point of building this in public.

Code: github.com/amitkssolanki/ai-receptionist