AI Features Fail Silently

One day every AI tool on this site stopped working. Nothing paged me. No error page appeared. A few of the tools had fallbacks, so they kept producing something that looked like an answer. I found out by using one myself and getting a response that was clearly not from a model.

The cause was boring. The code named a specific dated model version, the provider retired it, and every call started failing. The failure was boring too, and that's the problem: AI features don't crash, they quietly stop being AI.

When I went looking across about 30 of my own apps, I found the same failure in four different costumes.

Four ways an AI feature dies quietly

A retired model id. Pin a dated model version and you've scheduled an outage for the day it's retired. Use the provider's alias for the model family where you can, and add a test that fails if a dated id appears in the code.

A fallback that does its job too well. One app had a client-side fallback for when its recommendation call failed. The call failed, the fallback worked, and users got generic suggestions for weeks. A fallback is designed to hide failure. That's fine for the user in the moment, and terrible for you if nobody counts how often it fires.

A cheap path that masks the expensive one. Another tool used a quick keyword match first and a model classifier second. The classifier died, the keyword path kept answering most questions, and the tool looked healthy on every surface I checked.

An error swallowed in the background. A third app ran its AI synthesis in a background task after the response was sent. When the model call threw, the error went to a console log nobody reads. The feature "shipped." It never produced a single synthesis in production.

None of these threw an error a user would see. All of them would have been caught by the same small piece of plumbing.

The fix: a live probe for every AI route

For each AI feature, run a scheduled check in production that:

1. Calls the real route with a known input, the way a user would.
2. Asserts a real model answered, not the fallback. Have the route report which path produced the answer, and check that field.
3. Alarms when it doesn't run. A probe that silently stops is just another quiet failure. It needs a heartbeat that turns red when it's missing.

Then make the failures visible where they happen:

• Log every fallback as an event and alarm on its rate. Ten fallbacks in an hour is an outage wearing a disguise.
• Store AI failures in a column, on the row the feature was supposed to fill, not in a console. Then "how many of these failed this week" is a query.
• Treat a credential or model change as a release. When you rotate a key or change a model, list every route and job that uses it and probe each one the same day.

This is the AI version of a rule I hold for every scheduled job: alarm on "didn't happen", not just on "failed". I wrote more about that in Building Small Apps.

The quieter failure: eval numbers that don't add up

The second way AI work fails silently is on paper. Teams run evals, record the results and make a ship call, and the record looks complete. It isn't always true.

I reviewed an eval record for an AI feature that read as finished: a baseline, a list of fixes, a final score and a "Ship" decision. Then I recomputed it.

• The baseline was a weighted score of 2.65 out of 5, reported as 64%. But 2.65 ÷ 5 = 53%.
• The logged fixes summed to a final weighted score of 3.55. The record said 3.85.
• The ship call was made at 82%, while the same document set 85% as the gate for that level of autonomy.

None of these were lies. They were the ordinary drift that happens when numbers are typed rather than computed, across several sessions, by people moving fast. But a ship decision rested on them, and nobody had added them up.

The rules that would have caught it are simple:

1. Every eval percentage names its denominator. "64%" means nothing. "2.65 of 5 (53%)" can be checked by anyone.
2. Every iteration record sums to its final score. If the fixes don't add up to the claimed result, the record is wrong somewhere, and you find out where before you ship.
3. The ship call is checked against the gate in the same document. If they disagree, the gate wins or the gate gets changed on purpose, in writing.
4. Every AI feature keeps a small golden set, ten real cases is enough to start, that you rerun on every prompt or model change.

If evals are new to you, what an AI eval is covers the basics and how to design AI evals walks through building one. Eval-driven development is where the golden set starts paying for itself.

Try this today

Pick one AI feature you own. Ask your team two questions: "If the model stopped answering right now, how would we find out?" and "If I recompute our last eval score from the logged results, will I get the number we reported?" If either answer is "we'd notice eventually", you've found this week's work.