Why your AI pilot never shipped

Nearly every operator we meet has already run an AI pilot. Usually more than one. Very few have anything running in production that a customer touches.

That gap is not a technology gap. The demo genuinely worked. What killed it was structural, and the same seven causes come up almost every time.

1. The pilot was scoped to succeed

A pilot built to impress a steering committee gets curated inputs, a friendly evaluator and a happy path. It proves the model can do the task under favourable conditions — which was never in doubt.

Production is the unfavourable conditions. Malformed inputs, missing data, the customer who writes in three languages in one message, the order that exists in the ERP but not in the storefront, the policy exception nobody documented.

The pilots that convert to production are scoped to fail informatively. Run against a random sample of real traffic including the ugly tail, measure honestly, and let the failure rate tell you what to build. A pilot that reports 94% on curated data has told you nothing. One that reports 71% on real traffic, with a breakdown of the 29%, has given you a build plan.

2. Nobody owned the outcome

The pilot belonged to innovation, or data, or an agency. Production belongs to whoever runs the function — support, marketing, finance — and they were not in the room.

Predictable result: the thing that was built doesn't fit how the work is actually done, and the person who'd have to change their process has no stake in making it succeed. It quietly doesn't get adopted, and eventually gets described as "not quite there yet."

The fix is unglamorous. The person accountable for the metric the system is supposed to move owns the project from the first meeting, and their name is on the spec. If they won't sign it, that's information — build something else.

3. There was no definition of "good enough"

Without a threshold agreed in advance, evaluation becomes an opinion, and opinion always finds a reason to wait.

The conversation goes: it's promising, but it got that one wrong. Compared to what? Human agents get things wrong at a measurable rate that nobody has ever measured, because nobody audits humans the way they audit software.

Before building, write down: this system handles X% of category Y at Z accuracy, escalating the rest with a written summary. Measure the human baseline for the same task so the comparison is real. Then the launch decision is arithmetic instead of a debate.

The teams that ship measure their humans first. It reframes everything — usually in automation's favour, and always in the direction of a decision you can actually make.

4. The data work was underestimated

The model was never the hard part. The hard part was that the returns policy exists as a PDF, three Slack threads and one person's memory. That product attributes are inconsistent across 8,000 SKUs. That order status means four different things in four systems.

This is the single most underestimated line in every AI project, and it doesn't look like AI work, so it doesn't get budgeted.

Two things help. First, budget it explicitly and honestly — in our experience data preparation is 40–60% of a first build. Second, recognise that the work is a permanent asset: structured product attributes pay into support, PDP content, ads and search simultaneously. Fund it as infrastructure, not as project overhead.

5. It was built somewhere it could never live

The pilot ran on a laptop, or a vendor's sandbox, or a cloud account nobody in IT approved. Going live meant a security review that was never scheduled, a data processing agreement that was never drafted, and an integration nobody scoped.

Nine months of momentum against a procurement queue.

The teams that ship build in the production environment from week one. Their own cloud, their own keys, their own repo, security involved at kickoff rather than at launch. It's slower for the first fortnight and dramatically faster after that — and it turns the security team from a gate into a stakeholder.

6. It shipped and then nobody maintained it

The subtlest failure, because it looks like success for about two quarters.

The agent goes live, works well, and everyone moves on. Then the business changes — new products, a new returns policy, a new market, a new fulfilment partner, a model deprecation. Accuracy erodes gradually. Escalations creep up. Someone eventually notices it's giving out-of-date information, trust collapses, and it gets switched off.

Agents drift because businesses change. The operating model has to include: weekly review of escalations, monthly retraining against new edge cases, a named owner, and a budget line. Plan for 15–25% of build cost annually. A system nobody maintains is a system with an expiry date.

7. The org change never happened

The system worked and the process around it didn't move.

Support agents kept manually checking every automated response "just to be safe" — so handle time went up, not down. Marketing kept the four-day approval loop the agent was built to compress. The reporting agent produced the dashboard and everyone kept building their own spreadsheet.

Automating a step inside an unchanged process yields close to nothing. The process has to be redesigned around the new capability, which means someone has to change how their team works — and that's a management problem, not a technical one.

This is why we do a process teardown before automating anything, and why the honest recommendation is sometimes delete this workflow rather than automate it.

What the teams that ship do differently

Reading across the deployments that made it, five habits recur:

They start narrow and boring. One workflow, high volume, low judgement, measurable. Not the most impressive candidate — the most tractable one. They earn the accuracy record and the organisational trust, then move up.

They measure the human baseline. Before automating, they know what their team's current accuracy, speed and consistency actually are. Every subsequent conversation is grounded.

They build the escalation path first. Not as a fallback but as the primary safety design. Good handoff is what lets you deploy conservatively, and conservative deployment is what keeps the first launch from being the last.

They ship in weeks, not quarters. Something real in production inside a month, then iterate against live traffic. Long build cycles die of organisational entropy long before they die of technical difficulty.

They own it. Their repo, their infrastructure, their keys, their eval set. Not because vendors are untrustworthy, but because a system you can't modify is one you can't maintain — and maintenance is where the value actually lives.

The uncomfortable diagnosis

If you've run three pilots and shipped none, the constraint is almost certainly not model capability. Models have been good enough for the workflows most operators want to automate for a while now.

The constraint is that shipping requires: a named owner with real accountability, an agreed threshold, honest measurement, budgeted data work, production infrastructure from day one, a maintenance commitment, and someone willing to change how a team works.

That's an operating problem wearing a technology costume. It's also completely solvable — and it's solvable faster than a fourth pilot.

---

Our audit produces a ranked map of automatable workflows with hours and euros attached, plus the honest list of what you shouldn't automate. Yours to keep either way. Book one.