ai
Why AI Pilots Stall
There’s a familiar progression in AI projects: an impressive demo, an enthusiastic pilot, a suggestion to revisit it next quarter, and then silence. By the time someone asks what happened, the team has moved on.
A model can fail to do the job, of course. But I’ve also watched projects stall when the capability was useful and the demonstration worked. Getting it into daily use required data the tool couldn’t reach, or a definition of success nobody had agreed on. A better model wouldn’t have resolved either problem.
Those are the two things I want to examine here. People and adoption deserve their own discussion, and they can stop a technically sound project too.
Where the data comes from
For a demo, someone can gather the right information in advance. They find the files, clean up the example, and put everything where the model can use it. That’s reasonable for demonstrating a capability. It leaves open the question of how the information will get there next Tuesday.
In an organization, the inputs may live in someone’s inbox, a spreadsheet updated when its owner remembers, a SaaS tool whose export requires a ticket, and a database only one team can access. The demo may have depended on a person collecting all of that by hand. If the pilot needs the same preparation every time, account for that work before claiming it saves anyone time.
Follow the inputs back to their sources. Find out who owns them, how current they are, and what permission you need to use them. An export that arrives once a quarter can’t support a decision that needs this week’s numbers. A feed maintained by one person needs a plan for when that person is unavailable.
Connecting systems and keeping their data usable may consume more of the project than the model itself. Discovering that early lets you budget for it, narrow the task, or choose a different starting point.
What would count as working?
A demo shows that the model can do something under the conditions you’ve arranged. A pilot has to establish whether it can do a useful job repeatedly, with your data and inside your process. Without an agreed standard, the team can spend months improving outputs and still be unable to say whether the project is worth continuing.
Before starting, I want answers to three questions:
- What task will this change? Name the work and the person doing it now. Does the tool replace a step, prepare a draft, or help someone make a decision? “It makes us more efficient” doesn’t tell you enough to design or evaluate it.
- Can it get the inputs it needs? Trace the data to its owner and work out how it will arrive during ordinary use. If that depends on someone who “usually pulls the report,” include that dependency in the plan.
- How will we judge the result? Choose a measure relevant to the task: time saved, error rate, throughput, cost, or something else you can assess. Record how the work performs today so you have a comparison.
It’s also fine to experiment without a business outcome in mind. Learning what the technology can do is a reasonable use of a small budget. Give that experiment a learning goal and evaluate it accordingly. Trouble starts when a project funded as a business investment can only show that people enjoyed trying the tool.
Keep the first task manageable
An ambitious first pilot often needs information from several systems and cooperation from several teams. Each dependency adds something you have to resolve before you can test the idea. A broad objective also makes a disappointing result harder to diagnose.
I’d start with one task, reachable data, and a result people can recognize. That gives the team a chance to learn how the technology behaves in its own environment. It also gives you something specific to discuss with the people who will use it: whether it helps, where it fails, and what they still have to do themselves.
A small scope doesn’t guarantee success. It makes the failure easier to understand and less expensive to learn from. If the task works well, you have evidence for the next investment.
Someone has to keep it running
The pilot also needs a plan for what happens if it succeeds. Who owns it after the project team moves on? How will they notice a broken data feed or a decline in output quality? What happens to work the model gets wrong?
Monitoring, maintenance, and a human fallback all cost time. Include them in the evaluation. A tool that saves an hour of drafting but creates an hour of checking may still improve quality, but you need to know what benefit you’re getting.
The surrounding process has to change as well. People need to know when to use the tool and what to do with its output. Otherwise it can remain available for months while the work continues around it.
Before approving the next phase, I’d want the team to show how an ordinary task will move through the system, including an error and the handoff to whoever handles it. That tells me much more about readiness than another carefully prepared example.