The demo looked solid—so why does production feel riskier?
A demo usually runs on clean inputs, a steady script, and a teammate ready to “nudge” the prompt if it drifts. Production doesn’t give you that. Real users paste messy text, switch languages mid-sentence, and ask for things you didn’t plan for—at 4:55 p.m. when support is thin.
That’s why the same model can look sharp in a meeting and still trigger escalations a week after launch. The gap isn’t mysterious; it’s the unspoken assumptions baked into the feature: what counts as a “good” answer, which errors are acceptable, and who absorbs the cost when it’s wrong.
What outcome are we actually optimizing when we say “success”?
“Success” usually gets defined as “the model answered,” and the dashboard backs it up: high completion rates, happy-looking thumbs, fewer tickets. Then a frontline manager sends a screenshot where the model gave a confident answer that was technically on-topic but operationally useless—like telling a customer to “contact support” when the whole point was to avoid that handoff.
If you don’t name the outcome, the system will optimize whatever is easiest to measure. In a triage flow, do you want speed (shorter handle time), accuracy (fewer wrong routes), or safety (never downgrade urgent cases)? You can’t get all three at the same level. Tightening safety rules often means more “I’m not sure” responses, which pushes work back to humans at peak times.
Pick one primary metric and two guard metrics, then write down what you will accept. The hard part is admitting which failures you’ll tolerate—because that answer decides who gets paged later.
Whose reality is this model assuming about users and context?
That “who gets paged later” question often traces back to a quieter one: whose day-to-day reality the model assumes. In a demo, the “user” is usually patient, clear, and working in the happy path. In production, it might be a rushed rep toggling between tools, a customer on a shaky mobile connection, or a bilingual user mixing terms the model treats as separate concepts.
Those assumptions show up as invisible defaults. Does the model assume users know your internal product names? That they can copy-paste long error messages? That time zones, holidays, and local rules don’t matter? If it’s generating steps for a refund, one wrong assumption—like “the buyer has the original card”—turns a helpful answer into a dead end and a second contact.
Make teams write a “user and context contract” before launch: who’s using it, under what pressure, with what missing info. Then test the ugliest versions of those inputs. The next problem is what happens when the model can’t meet that contract.
The data looked representative—until the first weird week
That “can’t meet the contract” moment often arrives as a week that looks nothing like your sample data. A partner changes a form, a policy email goes out, a product label gets renamed, or a seasonal surge brings in new language and tone. The model didn’t “get worse.” Your inputs shifted, and the training mix you trusted stops matching what users are now sending.
This is where “representative data” quietly means “representative of last quarter.” If your evaluation set came from resolved tickets, you likely under-tested angry customers, partial screenshots, and brand-new error codes. In a routing flow, a single new category can force the model to guess; it will still produce an answer, and the first signal you see may be a spike in reopens, refunds, or manual overrides.
Before launch, run a “weird week” drill: hold out the newest two weeks, inject policy changes, and test on the messiest 5% of inputs. Then set a check that flags new terms and sudden topic shifts, so you notice drift before customers do.
When the model is unsure, what does it do—and who pays for that choice?
Catching drift early matters because uncertainty doesn’t show up as a polite “not sure” unless you force it to. In many customer workflows, the default behavior under uncertainty is to guess with a confident tone, because that’s what looks like a “helpful” answer. In routing, that can mean a customer lands in the wrong queue. In refund guidance, it can mean a rep follows steps that sound plausible but violate policy.
You need to decide what the model should do when it’s outside the safe lane: abstain, ask a clarifying question, or hand off. Each choice has a bill. Abstaining increases handle time and pushes load onto humans—often at the exact peak hours you were trying to protect. Asking questions can annoy users and drop completion rates. Guessing keeps flows moving, but the cost lands later as reopens, chargebacks, or compliance reviews.
Make uncertainty a product behavior, not an accident: define “don’t know” thresholds, require citations or tool checks for high-stakes claims, and log every fallback path so you can price it. Once you pick that behavior, you can design the day-one guardrails that keep it consistent.
You can’t ship “AI magic”: what guardrails must exist on day one?
Those fallback paths only help if they’re fenced in from day one. In the first release, teams often let the model “just answer” everywhere because it boosts completion rate. Then the first edge case hits—an account locked mid-flow, a policy exception, a customer asking for a refund reason code—and the model fills the gap with a confident guess.
Start with hard boundaries: which intents are allowed, which data sources the model may use, and which actions it must never suggest (like bypassing identity checks). Add a “must cite” rule for anything that can trigger money movement, policy enforcement, or legal exposure. If you can’t provide a source, force a handoff. Put input filters in front (PII scrubbing, profanity, jailbreak patterns) and output checks behind (blocked phrases, required disclaimers, format constraints).
This costs time and UX polish. You’ll ship more refusals and more handoffs, and support will feel it. That’s acceptable only if you also instrument it: every refusal, every handoff reason, and a sampled review queue, so you can see where your guardrails are too tight—or not tight enough.
After launch, what signals will tell you the assumptions are breaking?
Once you instrument refusals and handoffs, the fastest way to spot broken assumptions is to watch where the “fallback” paths start clustering. If the same intent suddenly drives more abstains, clarifying questions, or human transfers, your inputs likely shifted (new policy, new UI copy, new error codes) or users are asking for something the model wasn’t built to handle.
Pair that with outcome signals, not just model stats: reopens, repeat contacts within 24 hours, manual overrides, and downstream corrections (refund reversals, reroutes, compliance flags). Sample the “confident but wrong” cases by reviewing answers that users accepted quickly but later triggered fixes. The practical pain: this takes real reviewer time, and without a weekly cadence, you’ll notice drift only after escalations stack up.
When those signals move, freeze scope, tighten the safe lane, and retest the ugliest 5% before you add features.
