Why your AI demo breaks in week one

It's a familiar week. You wired a model into your product, it worked in the demo, everyone was thrilled, and you shipped. Seven days later the bug reports start: blank responses, a spinner that never stops, a bill three times what you expected. Nobody changed the model. What changed is that real people started using it.

A demo proves the thing can work once, under conditions you control. Production is making it work the thousandth time, under conditions you don't. The gap is almost never the model. It's everything around the model. Here are the three gaps that show up first, roughly in the order they'll hit you.

In a demo you type the one prompt you know works. Real users do none of that. They paste a 40-page PDF into a chat box built for one line. They send an empty message and expect an answer. They write in Turkish when your prompt was tuned for English. They type in ALL CAPS with three emojis and a misspelling per word. One of them tries to jailbreak it for fun on the second day. Each of those hits a code path your prototype never ran.

The cost side is worse than the crash side. A 40-page paste on a per-token model isn't a bug. It's a bill. One user, one careless copy-paste, and you've spent your daily budget in a single request. The demo never showed you this because in the demo the input was always the same 200 characters.

The fix isn't glamorous. Validate before the model, not after. Cap input length at whatever your model can actually reason over, and reject anything above that with a real error message. Run a cheap classifier or a regex pass first for the obvious cases: empty strings, garbage bytes, prompt-injection giveaways like "ignore previous instructions." Then treat the model's output the same way. If you expect JSON, parse it against a schema. If the schema fails, retry once with a stricter prompt, then fall back to a safe default.

Model APIs time out. They rate-limit you at the worst moment. They stall for three seconds and then return a 500 for no reason you'll ever learn. Twilio drops a webhook. Your STT provider degrades quietly and starts returning half-transcribed audio. In the demo you got lucky once. In production you're rolling the dice thousands of times a day, and the unlucky rolls are what your users remember.

The naive version calls the API and hopes. The version that survives its first busy day wraps every call in three things: a hard timeout so a stalled request can't hold a user hostage, a bounded retry with exponential backoff and jitter so a transient blip doesn't become a page, and a fallback response so the failure lands as a sentence instead of a stack trace. Eight seconds is a good starting timeout for a chat completion. Three retries is usually the ceiling before you're just making the outage worse.

// generate.ts
 
// the demo version. one call, one prayer.
const out = await model(prompt);
 
// the version that survives week one.
async function withRetry<T>(fn: () => Promise<T>, max = 3): Promise<T> {
  for (let attempt = 1; attempt <= max; attempt++) {
    try {
      return await Promise.race([
        fn(),
        new Promise<T>((_, reject) =>
          setTimeout(() => reject(new Error("timeout")), 8_000)
        ),
      ]);
    } catch (err) {
      if (attempt === max) throw err;
      const backoff = 200 * 2 ** attempt + Math.random() * 200;
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
  throw new Error("unreachable");
}
 
const out = await withRetry(() => model(prompt)).catch(() => fallback);

The jitter matters more than it looks. Without it, every retry from every stuck request fires at the same moment after the same backoff, and you turn a two-second blip into a retry storm that hits your own rate limit. With jitter, retries spread out and the queue drains.

There's a second half people skip. A circuit breaker. If the provider is returning 500s for the last thirty seconds, stop calling it. Serve the fallback directly. This is what keeps a bad five minutes at your vendor from becoming a bad five minutes at yours.

In a demo, you're the only user, and you're being gentle. Production is a room full of strangers who aren't.

The demo ran on your laptop, where you watched every log line scroll past. In production, a user tells you "it gave me a weird answer" and you have nothing. No request. No prompt. No idea which model version answered, how long it took, or how much it cost. You can't fix what you can't see, and most "the AI got it wrong" complaints are really "I have no idea what the AI did."

Trace every generation from the first day. Structured logs, one line per turn, with the same shape every time. Something like:

agent.turn.completed conversation_id=c_8fa2 intent=booking confidence=0.94 model=claude-3.5 tokens_in=412 tokens_out=88 cost_usd=0.031 duration_ms=812 status=ok

That single line lets you answer, in one query, "why was this call slow" and "which intent is costing me the most" and "which conversation had that weird answer the user complained about." A conversation ID that threads through every turn is the difference between debugging in an hour and debugging in a week.

Then wire two alerts on top of the log stream. One on latency (p95 above your target for five minutes straight), one on cost per conversation (a sudden spike usually means someone found the 40-page paste). You don't need Datadog on day one. A structured log pipe into any store you can query, plus two alert rules, catches the failure modes that matter.

None of this is hard. It's just the work that doesn't demo, so it's the work that gets skipped, right up until the week it can't be. Do it before launch and week one is quiet.

That's the whole goal. A launch week where nothing interesting happens.

Anas