The pattern is consistent enough to be predictable. A pilot is funded, a demo lands well, everyone agrees it is impressive — and then nothing reaches production. Six months later the same capability is proposed again, by a different team, as a new pilot.
The cause is rarely the model. It is that the pilot was scoped to produce a demo, and a demo is not evidence. A demo proves the system can succeed once, on an example chosen after the fact. Production requires knowing how often it fails, on which inputs, and what that costs.
What a pilot has to establish
An evaluation set that existed before the build
If the examples were collected after the system was working, they were selected — consciously or not — for the cases it handles. That is a demo, not a measurement.
Write down 100 to 300 real cases first, sampled from actual traffic including the messy tail: the malformed inputs, the ambiguous requests, the ones a human had to escalate. Fix the expected outcomes before anyone tunes a prompt. Everything after that is measured against a target that cannot move.
This is the single highest-leverage thing in the list, and the most frequently skipped — because it is unglamorous work that produces no visible progress in week one.
A baseline that is not the AI
Whatever the process is today — a rules engine, a spreadsheet, three people in an operations team — measure it on the same evaluation set. Two numbers matter: how accurate it is, and what it costs per case.
Without a baseline, "92% accurate" is unreadable. If the existing rules engine scores 94%, the project is a regression. If the humans score 88% and take four minutes a case, it is a substantial win. The number alone tells you nothing.
The cost per task at real volume
Pilot economics do not survive contact with production traffic. Work out the fully loaded cost of one task — model calls including retries and any multi-step chains, retrieval infrastructure, and the human review time the workflow still requires — then multiply by actual monthly volume.
Retries and human review are the two most commonly excluded, and they are usually the two largest. A workflow with a 15% escalation rate is paying for the model and a person on 15% of cases.
The failure mode, not just the failure rate
Two systems at 90% accuracy can carry completely different risk. Ask what the 10% looks like:
- Does it fail loudly — refusing, erroring, flagging low confidence — or silently, returning a confident, plausible, wrong answer?
- Is the wrong answer detectable downstream, or does it enter a system of record where nobody will check it?
- Is the failure correlated? A system that fails randomly is an inconvenience. One that fails on every invoice from a particular supplier is a systemic problem discovered by that supplier.
Silent, correlated, undetectable failure is the combination that stops a project at the production review — and it is usually first examined at that review, far too late.
Access to the real data, on real infrastructure
Pilots run on an extract: a clean CSV, a snapshot, a copy with the awkward records removed. Production runs against the live system, with its permissions model, its rate limits, its downtime and its personal data obligations.
Whether the pilot can legally and technically read production data — under the actual access controls, with the actual retention rules — is a question worth answering in week one. It has killed more otherwise-successful projects than model quality ever has.
A named owner for the day it degrades
Model providers deprecate versions. Upstream data schemas change. Input distributions drift as the business changes. An AI feature is not a delivery; it is a system with an ongoing error rate that somebody has to watch.
If no team owns the evaluation set after launch, quality decays invisibly until a customer finds it.
Scope the pilot to answer questions, not to impress
A pilot worth funding is written as a set of questions with thresholds attached, agreed before work starts:
- On the 200-case evaluation set, does it beat the current baseline on accuracy?
- What is the fully loaded cost per case at projected monthly volume?
- What proportion of failures are silent rather than flagged?
- Can it read production data under the real access controls?
- What does it cost per month to keep running, including the review time?
Each of those has a number attached and a threshold agreed in advance. If the answers come back below threshold, the pilot has succeeded — it produced a decision for a fraction of the cost of a production build. That framing matters, because a pilot that can only succeed by proceeding is not an experiment; it is a commitment with extra steps.
The corollary: a pilot should be cheap and short enough that stopping is a real option. Once it grows large enough that cancelling would be embarrassing, it stops producing honest answers.
Where this usually lands
In our experience the projects that reach production are rarely the most technically ambitious. They are the ones with a narrow, well-measured task, a clear baseline to beat, and a human review path for the tail. Ambition is easier to add to a system that already runs than to retrofit reliability onto a demo.
For how we structure discovery and delivery around this, see our services — or start a conversation about a specific workflow.
AI pilots — common questions.
How long should an AI pilot run?
Long enough to answer its questions and short enough that stopping remains a real option — typically four to eight weeks. Once a pilot is large enough that cancelling would be embarrassing, it stops producing honest answers.
What accuracy does an AI system need to reach production?
There is no universal figure. What matters is whether it beats the existing baseline on the same evaluation set, and what the failures look like. A 90% system that fails loudly and detectably is often deployable where a 95% system that fails silently into a system of record is not.
Why do AI pilots demo well but fail in production?
Because a demo is evaluated on examples chosen after the system was built, while production faces the full input distribution including the messy tail. Building a fixed evaluation set before development is what makes the two comparable.
What is usually missing from AI cost estimates?
Retries and human review. Both scale with volume and are commonly excluded from pilot economics, yet a workflow with a 15% escalation rate pays for both the model and a person on 15% of cases.