The Assistant That Couldn't Say Hello: Building a Grounded RAG Menu Chat for Spree Commerce

This is Part 4 of a four-part series on spree-restaurant, a Spree Commerce platform built so Square owns the menu. Part 1 covers the platform architecture. Part 2 covers spree_square, the open-source sync engine. Part 3 covers spree_doordash, delivery dispatch. This post is about the last piece: a chat widget that answers real questions about the menu — grounded in the same data everything else in this platform already syncs from Square, not a hand-maintained FAQ document that drifts the moment someone changes a price at the counter.

Code’s here: github.com/amitkssolanki/spree_menu_chat · rubygems.org/gems/spree_menu_chat.

Retrieval, not a system prompt stuffed with the whole menu

The lazy version of “add AI chat to a storefront” is a system prompt with the entire menu pasted into it. That works until the menu changes, at which point either someone remembers to update the prompt too — a second source of truth, exactly the problem Part 1 already solved once for the storefront itself — or the assistant confidently describes a dish that’s been 86’d since Tuesday.

spree_menu_chat grounds every answer in retrieval instead: each Spree::Product and Spree::Policy gets embedded as its own chunk (name, description, category, and — when spree_square is installed — a real summary of its modifier lists, e.g. “Left Half: Cheese, Pepperoni, Margherita, Veggie, BBQ Chicken (+$2.00)”), stored as a vector in Postgres via pgvector. A customer’s question gets embedded the same way, the closest chunks come back by cosine distance, and only those real chunks — never general knowledge — get handed to the model as context. Change a price in Square, and the next sync that already runs for the storefront re-embeds the product automatically. No second content pipeline to maintain.

The rest of the stack follows the path of least new infrastructure, on purpose:

• Generation: Gemini 2.5 Flash-Lite, via Google’s free developer tier — no billing account needed, which keeps this feature’s running cost at $0. Referenced by the gemini-flash-lite-latest alias rather than a pinned dated model string, after the pinned version 404’d for a fresh API key within days of building this ("no longer available to new users" — while still showing up in the model-listing endpoint, which apparently doesn’t reflect real per-account availability). The alias resolves to whatever Google currently recommends instead.
• Embeddings: Voyage AI’s voyage-4-lite, 1024 dimensions, covered by the first 200M free tokens — a restaurant menu is a few thousand tokens to embed, so this never gets close to that ceiling. The tighter real constraint turned out to be rate, not volume: without a payment method on file, Voyage caps requests at 3/minute and 10K tokens/minute, which a naive per-record embedding loop over even a 40-item menu blows through in the first few calls. EmbeddingClient now tracks its own recent request timestamps and sleeps proactively before the next call would exceed that limit, rather than firing calls as fast as the caller loops and hoping job-level retries eventually catch up.
• Storage: pgvector on the same Postgres the rest of the app already runs on — no new service, the same reasoning that already has Solid Queue running inside Postgres rather than a separate Redis.
• Retrieval and generation both happen in Rails, not the Next.js storefront. The browser calls a same-origin Next.js route, which proxies server-side to a Rails endpoint authenticated with the store’s ordinary Storefront publishable key — the same pattern every other Spree API call in this storefront already uses, and it means the Gemini and Voyage API keys never ship to the browser at all.

The read-only guardrail, enforced structurally

The assistant can answer questions. It cannot place an order, add anything to a cart, or touch an account — and that’s not just a line in the system prompt asking it nicely not to. AnswerGenerator never builds or passes a tools/function-declarations argument to the Gemini call, and no write-capable Spree model — Spree::Order, Spree::LineItem, Spree::Cart — is referenced anywhere in the request path. Gemini has no mechanism to invoke anything beyond generating text, so a prompt-injection attempt smuggled into a question, or even into the embedded catalog content itself, has nothing to hijack into an action. At worst it can talk the model into an off-topic reply, which the system prompt is asked to refuse.

That’s checked by a spec that inspects the real outgoing request body for the literal absence of a tools key, plus a full grep across the app for any write-capable model reference — not an assumption that it’s clean, confirmation of it, the same standard Part 2 held the Square credential review to before that gem shipped publicly.

A spec suite that was lying the whole time

The best bug this project found — better than any of Square’s or DoorDash’s — didn’t show up as a failing test. It showed up as a passing test suite next to a completely broken feature.

Gemini’s streaming endpoint sends Server-Sent Events, data: <json> frames separated by a blank line. The parser I wrote looked for a literal "\n\n" to split frames — which matched the hand-written WebMock fixtures in the spec suite perfectly, because I wrote those fixtures with the same assumption. Gemini’s real wire format uses \r\n line endings. The parser never found a match against the real response, so every real streaming request silently yielded zero chunks and zero errors — no exception, nothing logged. AnswerGenerator.stream’s “Gemini never yielded any text” guard quietly substituted the canned fallback for every single real question, including ones with a perfect context match. And because the fixtures shared the exact same wrong assumption as the code being tested against them, the suite stayed green the entire time.

This is precisely the failure mode “verify against real infrastructure, not just specs” exists to catch, and it did — one curl -N against the actual streaming endpoint was enough to show three chunks of literal silence where a real answer should have been. Fixed by normalizing \r\n to \n before splitting, and the fixtures got corrected too, so they now test the real format instead of re-testing the same wrong assumption back at themselves.

Two smaller bugs came from the same “run it against something real” discipline: neighbor’s t.vector migration helper is only a transitive gemspec dependency, so Bundler.require never auto-required it — invisible until an actual migration ran. And the similarity floor that decides whether retrieved content is “close enough” to answer from was originally set to 0.7, picked before ever querying a real embedding. Real cosine distances for genuinely correct matches against the live catalog ranged 0.32–0.63; a 0.7 floor rejected nearly all of them, so “what comes with the buffalo wings?” — a real menu item with a real matching description — returned “I don’t have information about that.” A real off-topic question in the same test batch landed at 0.76, comfortably outside the corrected 0.35 floor. Numbers picked from a spec pass and numbers picked from a real question turned out to be very different numbers.

It could answer anything — except “hi”

By the time all of the above was working, the assistant handled real, specific questions well. Ask it what comes with an item, whether something’s vegetarian, what the delivery area covers — accurate, grounded answers, sourced entirely from the real synced catalog and policies.

Then I tried the two things an actual customer types first.

> hi
"I don't have information about that. Please reach out to
[email protected] and we'll be happy to help."

> what can I order?
"...the provided text does not contain a specific list of what
items you can order..."

Both are the retrieval mechanism working exactly as designed, applied to a question it was never designed for. A bare greeting has nothing meaningful for vector search to match against, so nothing clears the similarity floor, so the assistant falls back to “I don’t have that” — for a hello. And “what can I order?” is deliberately vague; no single embedded product chunk represents “the entire menu,” so top-K similarity search over one embedded question surfaces a handful of loosely related items at best, and the assistant — correctly, by its own grounding rules — refuses to invent the rest rather than guess.

The fix is a small, regex-based SpreeMenuChat::Intent classifier — not a model call, so it costs nothing and adds no latency. A detected greeting short-circuits straight to a canned, store-branded response before the token budget check or any Voyage/Gemini call even happens:

The classifier only fires on a whole-message greeting, deliberately — “hi, what are your hours?” still gets a real, grounded answer to the real question in it, not just “hello back.”

A detected browsing question (“what can I order?”, “what options do I have?”, “show me the menu”) routes to a new Retriever.menu_overview, which bypasses similarity search entirely and hands the model every currently-embedded product at once, sorted by name. With real content to summarize instead of a handful of near-misses, the model does exactly what you’d want:

Building that exposed a real, separate data bug: the query backing menu_overview returned 95 embeddings for what should have been 45 real products. Fifty of them pointed at product rows a much earlier catalog cleanup had already hard-deleted via a raw SQL statement — which bypasses every ActiveRecord callback, including the one that would normally clean up an embedding when its product goes away. Those fifty stale chunks had been getting fed to the model as real context on every retrieval the whole time, silently degrading answer quality with content about menu items that no longer existed. Fixed three ways at once: the orphaned rows got deleted; Spree::Product gained a real after_commit ... on: :destroy hook so a normal single-product delete cleans up after itself going forward; and a new spree_menu_chat:prune_orphaned_embeddings rake task covers the bulk raw-SQL case that hook structurally can’t reach, alongside menu_overview itself defensively skipping any orphaned row it finds rather than trusting the index blindly.

What it does today

• Grounded Q&A over the real, synced menu and policies — retrieval-augmented, never general knowledge, with an honest “I don’t have that, here’s how to reach us” fallback when nothing relevant exists.
• Handles greetings and broad browsing questions as a real conversational surface, not just a search box that only accepts precise queries.
• The read-only guardrail is structural, not prompted — no tool-calling ever reaches the model, no write-capable Spree model is reachable from the request path.
• Rate limiting and a daily cost cap, both Postgres-backed with no new infrastructure — 20 messages per session per hour by default, and a daily token budget (tracked from Gemini’s own reported usage, not an estimate) that degrades to a plain “we’ve reached today’s limit” response rather than spending without bound.
• Admin visibility — every conversation logged, with a real per-conversation detail view rendering the model’s markdown as actual formatting instead of literal asterisks, so support staff can see exactly what customers asked and what they were told.

All of it verified against the real running app at every step — a real synced 45-item, 8-category catalog and 7 real policies, real curl’d questions against the real streaming endpoint, real admin pages loaded against real Postgres — not just the gem’s own SQLite dummy-app spec suite, which is precisely where three of the bugs above were invisible until something real ran through it.

If you want this for your own store

The extension is public and free to use — MIT licensed, on GitHub and RubyGems, installable in any Spree store. It takes a soft dependency on spree_square for modifier-list grounding when that’s installed, but works fine on a plain Spree catalog otherwise.

If you’re running a Spree store and want customers to actually get useful answers about your menu instead of a search box that only works for exact phrasing, or if you want a Spree platform built around this kind of grounded assistant from the start, reach out — [email protected] or @amitkssolanki on GitHub.