The demo almost always works. That is what a demo is for — a chosen input, a favourable path, and a room that wants to be impressed. The rollout is where the same system meets inputs nobody curated, and that is usually where the trouble starts.
The usual diagnosis is that the technology was not ready. More often the project never agreed what a good output was, so nobody could tell the difference between a system that was working and a system that was merely running.
Why does the pilot look fine and the rollout not?
A pilot is graded by impression. Someone senior looks at the output, recognises it as roughly the sort of thing they asked for, and calls it promising. That is a real signal, but it measures plausibility rather than correctness, and the two come apart exactly where the consequences live.
At rollout the volume rises and the attention paid to each individual output falls. Nobody reads every one. So the failures that survive are the ones that look right: the reconciliation that balances but misclassifies, the summary that quietly drops the exception, the confident answer with nothing behind it. Plausibility is the thing a language model is built to produce, which makes the errors that get through the hardest ones to notice.
A pilot rarely catches that, because a pilot is small enough to be watched.
What does defining good first actually change?
Writing the evaluation before the build is unglamorous, and it changes three things.
It forces someone to state what the output is for and who carries the consequence when it is wrong. That is an awkward conversation in week one and a considerably worse one in month six.
It turns quality into something checkable by a person who was not in the room. An evaluation written afterwards tends to describe what the system already does. Written first, it constrains what the system is allowed to do.
And it gives you a defensible way to stop. Without an agreed bar, cancelling a project that demos well is a matter of opinion, and opinion loses to momentum.
What does a usable evaluation look like?
Not a score out of ten. A small set of real cases with agreed correct answers, including the awkward ones where correct is a judgement rather than a number, because those are the cases that decide whether the system is safe to run at all.
Then criteria, weighted, so that a system cannot pass by being fluent. And a critical-fail rule: certain errors fail an output outright, however good the rest of it is. Inventing a figure. Dropping an exception. Producing a number that cannot be traced back to where it came from. We maintain a rubric library for finance-operations work built on that shape, because averaging those failures into an otherwise respectable score is precisely how a system that looks fine does something indefensible.
Where this sits
Our operating model is vet, disclose, verify, and evaluation is what makes verify more than a good intention. You cannot verify an output against a standard nobody wrote down.
It is also the same question we put to people rather than systems. What we ask a candidate for AI work is how they would know a workflow had worked, and the answer sorts them faster than any build task will. The organisational version is identical. If you cannot define good before building, you will not be able to say afterwards whether the thing works. You will only be able to say that it runs.
None of this makes the demo worse. It makes the demo mean something.
If you are moving an AI pilot towards a rollout and the evaluation is the part nobody has written yet, that is a conversation we have often — [email protected]
The vetting bar — the same question asked of a person instead of a system.
The 15-point gate we run before any AI touches a real person — largely a list of ways to define good before building.
We ran our own governance audit on ourselves. It failed. — what an unevaluated output looks like when it is ours.
TL;DR: AI projects rarely fail at the demo — they fail at evaluation. A pilot is graded by impression, which measures plausibility rather than correctness, and at rollout the errors that survive are the ones that look right. Writing the evaluation before the build forces someone to say what the output is for, who carries the consequence when it is wrong, and what counts as a critical failure regardless of how good the rest of the output is.