- Most pilots fail for organisational reasons — no baseline, no owner, no path through security — not because the model is weak.
- Agree the business metric and the success threshold before writing any code.
- Build on real data from week two, and start the security review on day one.
- End every pilot with a written go/no-go decision. A fast “no” is a good outcome.
A widely cited 2025 MIT report, The GenAI Divide, found that about 95% of enterprise generative AI pilots produced no measurable impact on profit and loss. The same research found that companies working with specialised partners reached production far more often than those building alone. The technology works; the way most pilots are run does not.
We see the same pattern when companies ask us to rescue a stalled AI initiative. The demo impressed the leadership team, a budget was approved, and then — months later — nothing is in front of real users. Here is why that happens and how to avoid it.
The pilot trap
A pilot is supposed to answer one question: should we invest in putting this into production? Most pilots never answer it, because they were designed to show that AI is impressive rather than to prove that a specific workflow gets measurably better. The result is “pilot purgatory” — a working demo that nobody is willing to switch off or to fund properly.
Five reasons pilots stall
- No baseline. If nobody measured how long the task took, or how often it went wrong, before the pilot, there is nothing to compare the pilot against — and no business case.
- Demo data instead of real data. Clean sample documents hide the messy scans, edge cases, and missing fields that make up a real workload.
- No business owner. When the pilot belongs only to IT or an innovation team, nobody in operations is accountable for adopting it.
- Security and integration left until the end. The pilot runs on a laptop or a public API, and the security review that happens afterwards sends the team back to the start.
- No way to measure quality. Without an evaluation set of real cases with known answers, every prompt change is a guess, and confidence never builds.
The 6-week path that works
The structure below is the one we use in our AI Pilot Program. It is deliberately short: long enough to build working software on real data, short enough that the decision happens while everyone still cares.
| When | Focus | What you get |
|---|---|---|
| Week 1 | Scope | One workflow, a measured baseline, an agreed success threshold, data access, and the security review started |
| Weeks 2–5 | Build on real data | Working software in your own cloud account, an evaluation set of real cases, and a demo every week |
| Week 6 | Decide | Results against the baseline, a cost-per-case estimate, risks, and a written go/no-go recommendation |
Building inside your AWS, Azure, or Google Cloud environment from the start means the pilot already meets your data rules. If the answer is “go,” there is no rebuild — you extend the same system to production in two-week sprints.
What “production-ready” actually requires
Whatever approach you take, a generative AI system is not ready for real users until it has:
- Single sign-on and role-based access that mirrors your existing permissions
- Logging of every request and response, with sensitive data handled according to your policy
- An evaluation suite that runs before every change to prompts, models, or retrieval
- Monitoring for quality drift, latency, errors, and cost per transaction
- A human review path for low-confidence or high-impact outputs
- A runbook: who is alerted, how to roll back, and how to switch the feature off
- A named business owner who reports on the metric the pilot was meant to move
Production is also not the finish line. Models change, data drifts, and usage grows. Plan for ongoing monitoring and improvement from the start — either in-house or through a managed AI operations service.
Who should own an AI pilot
The most reliable predictor of success we see is a business owner who feels the pain of the current process — the head of claims, the operations director, the support manager — paired with a technical lead who can make architecture decisions quickly. Executive sponsorship helps, but the day-to-day owner is what turns a pilot into a habit.
Frequently asked questions
How should we measure the ROI of a generative AI pilot?
Measure the workflow, not the model. Capture a baseline before the pilot — handling time per case, cost per case, error or rework rate, backlog — and compare the same numbers on real cases during the pilot. Model accuracy matters only insofar as it moves those business numbers.
Should we build our own AI solution or buy one?
Buy when an off-the-shelf product fits the workflow and your data rules. Build, or partner to build, when the workflow is specific to your business, involves sensitive data that must stay in your environment, or needs deep integration with your systems. Many companies end up with a mix of both.
What happens if the pilot does not hit its targets?
Then the pilot did its job. A clear no-go decision after six weeks — with the findings documented — is far cheaper than a project that drifts for a year. Often the findings point to a fixable gap, such as missing data, which becomes the next step.
How much data do we need for a generative AI pilot?
Less than most teams expect. For document and knowledge use cases, a few hundred representative real examples with known correct outcomes are usually enough to build an evaluation set and measure performance honestly.