A 96% test score can hide a worthless model. Here is how we scope, measure, and price custom computer vision so it survives real images.
Your model scored 96% on the test set and your client is unhappy. Both things are true at once, and the gap between them is the entire discipline of production computer vision.
We see this on almost every vision project that arrives already half-built. Somebody trained a detector, it looked excellent on the images they had, and then it met the loading dock at 6am in November. The accuracy number did not lie. It answered a different question than the business was asking.
Accuracy is the share of predictions that were correct. On a defect line where 3% of parts are bad, a model that says "fine" to everything scores 97%. It is also worthless, and it will pass a demo if nobody looks past the headline figure.
This is the vision version of an opinion we hold about all of this work: prototypes lie. A demo that works on five hand-picked examples tells you nothing about the five thousand real ones. Moving a system from 90% to 99% is where the engineering actually lives, and it is usually most of the budget.
The fix is not a better model first. It is a better measurement first.
The first deliverable on a vision project should be a labelled set of your own images, chosen to include the cases that matter rather than the cases that were easy to collect.
That means the badly lit ones. The ones where the part is half out of frame. The ones photographed on a phone at an angle by somebody in a hurry. The ones where two defects overlap. If your evaluation set is all clean studio shots, you have built a test that your model cannot fail and that predicts nothing.
A useful evaluation set is usually smaller than teams expect. A few hundred well-chosen and correctly labelled images beat tens of thousands of scraped ones, because you can actually inspect what went wrong. The labelling is the expensive part, and it is not a task to hand to whoever is free. Two people labelling the same 200 images and disagreeing on 30 of them is not a labelling problem, it is a specification problem, and you want to find it in week one rather than week nine.
Every vision system trades these two against each other, and the right trade is a business decision rather than a technical one.
Precision asks: when the system flags something, how often is it right. Recall asks: of everything it should have flagged, how much did it catch.
A pharmaceutical inspection line wants recall. Missing a contaminated unit costs incomparably more than pulling a good one for review, so you tune for catching everything and accept that humans will clear a pile of false alarms.
An automated invoice pipeline usually wants precision. A wrong value written into the accounting system silently is worse than a document routed to a person, so you tune to be confident when you act and hand off when you are not.
Ask which mistake actually costs money before anyone touches a training script. The answer sets the threshold, and the threshold sets the experience of using the system far more than the model architecture does.
The default should be a strong pre-trained model or a vision-language model, evaluated against your set on day two. Sometimes that is already good enough, and the project becomes an integration project rather than a machine learning project. That is a good outcome and we say so when we find it.
Fine-tune when your images are genuinely unusual: proprietary parts, a specific camera rig, medical or industrial imagery that looks nothing like the public web. Fine-tune when the baseline gets you to 85% and the business needs 97%. Do not fine-tune because it feels more like real work.
The same logic applies further up the stack. Buy the outcome, not the model. Model names change every quarter. The evaluation set, the labelled data, and the pipeline around them are what you still own in three years.
Our fixed-price bands run $4,000 to $30,000, and a vision project sits inside them the same way an agent project does. A single-purpose detector with a clear accuracy target runs $4,000 to $10,000 across three to four weeks. A multi-step pipeline, meaning capture, detection, validation, human review, and writing results into another system, runs $10,000 to $22,000 across five to seven weeks. An enterprise deployment across multiple sites or cameras runs $22,000 to $30,000 and up, phased, over eight to twelve weeks.
Inference is cheaper than people expect and rarely the constraint. Production runs typically land between $50 and $2,000 a month, and sensible engineering, meaning batching, resolution control, and running the cheap model first and escalating only on uncertainty, cuts that three to ten times over.
The labelling and the evaluation work is the line item teams forget to budget. It is also the line item that decides whether the rest of the money was well spent.
A vision model degrades because the world moves. A supplier changes packaging. Someone repositions a camera by four inches during a cleaning shift. The winter light through the skylight is different from the summer light. None of this is a defect in your model and all of it will reduce your accuracy.
Monitoring for this is not optional and it is not expensive. Track the confidence distribution over time, track the share of predictions landing in the uncertain band, and sample a handful of production images weekly to re-label. When the uncertain share climbs, something changed upstream, and you usually find it faster by asking the operators than by retraining.
We build the same reflex into agent work, and the reasoning is identical there: a system without an evaluation suite is a liability with an interface on it.
Teams sometimes treat the review queue as an embarrassment, a sign the automation did not fully work. It is the opposite. A system that routes its uncertain cases to a person is a system that knows what it does not know, and that is the only kind safe to put in front of a real process.
Design it deliberately. Set the confidence band that triggers review, put the queue somewhere people already work rather than in a new tool nobody opens, and feed every human correction back into the evaluation set. Done well, the review load falls month over month and you have a compounding asset. Done badly, the queue silently grows until somebody starts clicking approve on everything, which is worse than no automation because now there is a log claiming it was checked.
If an off-the-shelf API reads your documents at the accuracy you need, use it and spend the money elsewhere. If you have fewer than a couple of hundred examples of the thing you want to detect and no way to collect more, the honest answer is that the project is not ready. If nobody can articulate what a mistake costs, the threshold conversation has no answer and the system will get tuned to whatever felt good in the demo.
We turn down vision work on those grounds a few times a year. We take two engagements a quarter, so a project that is not ready is not a project we can carry.
Less than most vendors imply if you start from a pre-trained model. A few hundred well-labelled images per class is often enough to reach a working baseline, and the quality and variety of those images matters far more than the count. You need enough of the awkward cases to measure honestly, which is a different requirement from having enough to train on.
Three to four weeks for a single-purpose detector, five to seven for a full pipeline with review and integration, eight to twelve phased for a multi-site deployment. Those match our general engagement range of four to eight weeks from kickoff to live deployment, with the longer end reserved for work that touches several locations.
Yes, and it often should. Latency, privacy rules, and unreliable connectivity all push work to the edge, onto cameras, gateways, or phones. The trade is that edge hardware constrains model size, so the accuracy target and the deployment target have to be decided together rather than in sequence.
You do, on our engagements. You get 100% code ownership, the trained weights, and the evaluation set, plus 30 days of post-launch support. If an agency will not hand over the repo and the model, that tells you what the arrangement actually is.
Written by the Codestreaks team. The price bands are our own published fixed-price tiers and the inference cost range comes from production systems we run and pay for. The evaluation-set and human-review approach described here is the process we use on vision engagements, and the specification-disagreement example comes from labelling passes on real client images, retold without naming the client. We are an agency that sometimes argues a project is not ready to build, which is worth weighing accordingly. Drafting is AI-assisted with a human editing pass over our own project record.
If you have a set of images and a decision you want made from them, that is enough to scope on a call. Bring twenty examples including the ugly ones. We can usually tell you within the half hour whether this is an API integration, a fine-tune, or not ready yet. Free 30 minutes, reply within two business days.
More on how we build these on our computer vision development service page, or start a project.
Related reading: what AI readiness assessments actually measure, what changes when you build for the cloud, and where automation pays off on the factory floor.
A serial maker of SaaS products and AI agents, I’ve built and launched 10+ tools, grown products to thousands of users, and taken multiple ventures. I share the process what works, what breaks, and how builders can ship faster and smarter. Previously a data scientist at a Big Four consulting firm.
