An AI agent that nails a demo can still fall apart on real data. Here is where the last 10 percent hides, what it costs to run, and how to ship one that holds.
A team walks into a meeting with a demo. The agent reads an email, pulls the right record, drafts a reply, and everyone in the room nods. Budget gets approved that afternoon. Six weeks later the same agent is quietly forwarding refund requests to the wrong queue, and nobody can say why.
We see this pattern most weeks. A prototype that impressed a room, sitting on top of five hand-picked examples, meets the five thousand real ones and folds. Since 2024 we have shipped 30-plus agents to production, and the honest summary is this: the demo is the easy 90 percent. The last 10 percent is the whole job.
A demo is a controlled environment. You chose the inputs. You ran it until it looked good. That tells you the happy path exists. It tells you nothing about the inputs you did not think of, which is where real users live.
Here is the mechanism. Language models are probabilistic. Give one the same messy invoice twice and you can get two different answers. On a demo with clean data, that variance hides. At scale it shows up as the 1 in 50 case that routes a $9,000 refund to the wrong team. Ninety percent reliability sounds close to done. For a workflow that runs a thousand times a day, ninety percent means a hundred failures a day. That is not a product. That is a support ticket generator with a chat interface.
Moving from 90 to 99 percent is where the engineering actually lives, and it looks nothing like prompt tweaking.
Most of our build time on an agent goes to work the demo never needed:
None of this is glamorous. All of it is the difference between a thing that demos and a thing you can trust at 2am when nobody is watching.
From the field. For HrefStack, a martech company, we built an autonomous SEO content agent. The first version wrote decent articles. The version that mattered ran 24/7 with zero manual uploads, generated 300-plus leads a month, and cut their cost per acquisition 60 percent against paid channels. That gap, between "writes articles" and "runs the pipeline unattended", took the back half of a 10 week build. The writing was never the hard part. The reliability was.
Two costs get quoted wrong all the time.
The first is inference. A production agent typically runs $50 to $2,000 a month depending on volume and model. The number people miss is that good engineering cuts it 3 to 10 times. Caching repeated calls, routing easy requests to a smaller model and hard ones to a larger one, and designing prompts that do not re-send the same context on every turn are the levers. A team that skips this pays five times more for the same output and calls the technology expensive.
The second is the build. We quote fixed price, not hourly, so you know the number before you commit. A single-purpose agent runs $8,000 to $20,000 and ships in 3 to 4 weeks. A multi-step workflow agent is $20,000 to $45,000 over 5 to 7 weeks. An enterprise platform is $45,000 and up, phased over 8 to 12 weeks. Most teams do not need the platform. They need three boring workflows automated well.
A few rules we would give a friend spending real money on this.
Buy the outcome, not the model. Model names change every quarter. The owned software that wraps the model is what compounds. If a proposal is mostly about which model it uses, it is selling you the part that will be obsolete by winter.
Demand the repo. If an agency will not hand you the code, walk away. Code ownership is not a premium feature. It is the deal. Every client we work with gets 100 percent ownership and 30 days of post-launch support, and we say that up front because the alternative is a rental you can never leave.
Ask to see the eval suite. Not the demo. The test set. If the answer is a blank look, you are looking at a prototype with a good sales day, not a system.
Be suspicious of no-code stacks that became load-bearing. Zapier, Make, and n8n are great until the person who built the zap leaves and nobody can debug the thing your operations now depend on. Fine for the experiment. Dangerous as the foundation.
Why does my agent work in testing but fail in production? Because testing used inputs you picked and production uses inputs you did not. The fix is an evaluation suite built from real cases, plus retry and fallback handling for the failures a demo never triggers.
How long does it take to move an AI agent from prototype to production? For most single-purpose agents, 3 to 4 weeks. Multi-step workflows run 5 to 7 weeks. The prototype is often already done. The production hardening is the work being scheduled.
How much does it cost to run an AI agent each month? Usually $50 to $2,000 depending on volume and model. Caching, model routing, and disciplined prompt design typically cut that 3 to 10 times, so the same workload can cost very different amounts.
Do we own the code? Yes. Every engagement includes full code ownership and 30 days of post-launch support. If a vendor will not give you the repository, treat that as the answer to whether you should hire them.
Can you fix an agent we already started? Often, yes. A common engagement is taking a demo that stalled on real data and doing the reliability engineering it never got. That is most of what we do.
If you have an agent that wowed a room and then met real data, that is a normal place to be, not a failure. The reliability work is a known quantity. We take on two engagements a quarter, we scope in a free 30-minute call, and we reply within two business days.
Tell us what you are building on the start a project page, or read how we approach AI agent development end to end.
