Why AI projects stall between the demo and production
None of these reasons is a model problem. They all appear the moment you leave demo data for real data.
In this article6 sections
- The demo is the easy part
- Ten demo documents say nothing about ten thousand real ones
- Working on real data means working on personal data
- AI doesn't fix messy data, it puts it in the spotlight
- The price of an AI project isn't what it costs to build
- Without a success criterion written beforehand, a project can neither succeed nor stop
The demo is the easy part
On a few well-chosen cases, almost everything works, and everyone leaves enthusiastic.
The rest: the awkward cases, the real load, access rights, the people who have to change the way they work: that's where it gets stuck, and where budgets stop. Nearly half of proofs of concept never go into service: the bill is real, the benefit never became so.
The remedy is a building habit: taking into account, from the prototype on, what going into production requires. Access rights, edge cases, cost per use, what will need monitoring and who will have to do it.
Ten demo documents say nothing about ten thousand real ones
A demo is prepared with what's at hand: a dozen clean PDFs, a few pages long, chosen because they read well. In production, it's statements hundreds of pages long, crooked scans, files arriving in batches every morning.
This change of scale doesn't just require more patience: it brings up constraints the prototype didn't know. Server memory, processing time, timeouts, cost per page. On one of our extraction projects, a single document of nearly 500 pages was enough to jam the machine that effortlessly processed the test files. On another, a 190-page document took more than two hours to process.
Prototyping simply remains the right approach: it's the fastest way to show what's possible. But you need to get a sample of real data very quickly: the biggest files, the worst scanned, the oldest, and the volume of a normal month. That sample, not the demo, tells you what production will demand.
Working on real data means working on personal data
As soon as you leave demo data, you're handling names, addresses, client or patient files. Sending them as they are to a model hosted elsewhere means entrusting them to a third party, with everything data protection law then requires.
The most solid answer is to anonymise before sending: replace names and identifiers with markers, let the model work on the neutralised text, then reinsert the real values on return. On a medical project, we set the requirement before building: no patient name may get through. The check covered more than two hundred real cases, all detected. And if detection fails, the tool stops rather than sending the text in the clear.
A prototype can do without all that. A system in production, never. And real data tends to stay where it was used: in a test set, a code repository, a demo account never deleted. It's handled with production rules from the first day it enters the project.
AI doesn't fix messy data, it puts it in the spotlight
The day you ask a model to learn from your history, you discover the duplicates, the half-filled fields, the three naming conventions that have coexisted for ten years.
It's not a technical problem: it's the reality of every company, and nobody had faced it because people compensated without saying so.
A project that doesn't plan for this work doesn't plan for half its budget. Hence the value of opening up the data before promising a model: the real volume, the real quality, what's usable and what isn't. This step often shows that you need to tidy up first, or that simpler processing would do.
The price of an AI project isn't what it costs to build
What weighs is what comes after: the cost of every call, monitoring, models changing under your feet, new cases to handle.
Many projects are approved on a development budget and then abandoned eighteen months later over a running cost nobody had quantified.
A system you can't afford to run was never really delivered. That cost is estimated before building, not after.
Without a success criterion written beforehand, a project can neither succeed nor stop
"Seeing what AI can do for us" isn't a goal: it's spending with no finish line.
Without a threshold defined at the start, nobody can say whether the result is good, so another round is launched, then another, until someone cuts the budget out of weariness. It's the leading cause of abandonment, far ahead of technical difficulties.
Writing this threshold at the start is uncomfortable: a deadline, an error rate, a volume processed, and what triggers stopping. It's much less costly there than at the end.
An AI project is badly judged on its demo and well judged on its running cost. The points above all come back to the same question, asked before starting: what, in six months, will let us say it was worth it, or that we should stop?
Ce sujet touche à notre expertise Custom AI project.
We start with a conversation, not a quote.
Voir l'expertiseRead next
