A working checklist for vetting agentic AI companies: eval suites, repo ownership, audit logs, honest pricing, and red flags. Written by one of them, bias declared.
Codestreaks Team
You have a shortlist. Three or four agentic AI companies, each with a polished demo, a case studies page, and a proposal that uses the word "autonomous" at least six times. The demos all worked. The prices differ by 5x. And nothing in the decks tells you which of these teams will still be useful to your business eight months from now.
Bias declared up front: we are one of these companies. Codestreaks builds AI agents for a living, so this is not a neutral survey. It is the checklist we would hand a friend who was never going to hire us, drawn from 30+ projects we've shipped to production since 2024 and from the wreckage of projects other vendors shipped before us. Every test in it can disqualify us too. Use it on everyone, including us.
Prototypes lie. A demo that works on five hand-picked examples tells you nothing about the five thousand real ones your business will throw at it. Moving an agent from 90% to 99% reliability is where the actual engineering lives, and none of that engineering is visible in a thirty-minute screen share.
That has a practical consequence: stop scoring the demo. Every serious vendor has a demo that works. Score the things that predict whether the system survives contact with your real data, your real permissions, and your real edge cases. That is what the six checks below are for.
From the field. The most common way an engagement starts for us: a team arrives with a demo that impressed everyone in a meeting and then fell apart on real data. The pilot ran on clean, hand-picked inputs. Production had duplicates, scanned PDFs, half-migrated records, and edge cases nobody wrote down. The vendor who built the demo was gone, and nothing was logged, so nobody could even say which answers had been wrong. Most of the work we get paid for is reliability engineering, not the first demo. Whoever you hire, make sure they know that is the job.
An agent without an evaluation suite is a liability with a chat interface. The eval suite is a set of real historical cases with known correct outcomes, run automatically against every prompt change, model swap, and code deploy. If the pass rate drops, the release blocks.
Ask the vendor to show you one from a past engagement (redacted is fine). A real answer looks like a versioned dataset and a pass-rate report. A fake answer sounds like "we test extensively before launch". Testing before launch is not the point. The point is catching the regression that shows up eight weeks after launch, when a model gets retired on a published deprecation schedule and quietly swapped underneath your workflow.
Ask one question: "If we part ways at week six, what do we walk away with?" The only good answer is everything. The repository, the prompts, the eval dataset, the infrastructure accounts in your name.
If an agency won't give you the repo, walk away. That is not a nice-to-have, it is the deal. Some agentic AI companies are structured as thin products: your "custom agent" is a configuration inside their platform, and leaving means starting over. That can be a legitimate business model, but it should be priced like a subscription, not like custom development. We hand over 100% code ownership on every project. Ask your other candidates to match that in writing.
Your agent will be wrong sometimes. The question is whether anyone can find out what happened. Ask to see the audit trail from a live deployment: every step logged, tool calls, retrieved documents, model and prompt versions, all tied to a trace ID, readable by a compliance officer without an engineer translating.
Then ask the follow-up that actually filters: "When the agent gives a wrong answer in production, walk me through what you do." Teams that have operated agents answer with a replay workflow, where they re-run the exact decision and watch what the agent saw. Teams that have only demoed agents answer with a shrug and a prompt tweak.
Agentic AI development services cluster into three tiers, and a quote that ignores them deserves suspicion. Our own fixed prices, published: a single-purpose agent runs $8,000-$20,000 over 3-4 weeks. A multi-step workflow agent runs $20,000-$45,000 over 5-7 weeks. An enterprise platform with approvals and governance runs $45,000-$60,000+ over 8-12 weeks, delivered in phases.
A quote far below these ranges is usually a no-code stack wearing a custom-development suit. Those stacks are great until they become load-bearing, and then nobody can debug them. A quote far above deserves one question: "which part of this is hard, specifically". Ask about running costs too. Production inference typically lands between $50 and $2,000 a month, and good engineering (caching, model routing, prompt design) cuts that bill 3-10x. A vendor who quotes without asking about your volumes is guessing.
Models get deprecated. APIs change under you. An agent that ships and gets abandoned degrades quietly until someone notices the numbers drifting. Ask what post-launch support is included (we include 30 days on every engagement), and more importantly, ask whether the handover is designed so any competent engineering team can take over: documentation, runbooks, and the eval suite as a safety net for whoever makes the next change.
Watch for the opposite design, systems built so only the vendor can maintain them, with a retainer attached. Dependency is not a service.
A studio that says yes to everything is telling you either that their calendar is empty or that their delivery is thin. Ask when they would actually start and what they are delivering right now, and listen for specifics. Ask what project they recently turned down and why. A team with real delivery discipline has a concrete, recent answer.
This is also where fake urgency shows up. Discounts that expire Friday and "one slot left" pressure are trust signals, all of them negative. Our version of this: we take two engagements per quarter, we publish that number, and we say no when the scope doesn't fit. Slow is fine. Pressure is not.
Any one of these is a reason to stop being polite:
Ask each finalist for one client whose system has been in production for six months or more, then ask that client two things: what broke, and how fast it got fixed. Every production system breaks. A reference who claims otherwise wasn't paying attention.
Ask for outcomes with numbers attached. When we point to our HrefStack build, the numbers are a 60% reduction in customer acquisition cost versus paid channels and 300+ leads a month from agent-generated content, running unattended. Whatever numbers your candidates offer, ask how they are measured. A vendor whose case studies are all adjectives is telling you something.
They design, build, and deploy software agents that use language models to execute multi-step work inside your existing tools: triage, data entry between systems that don't talk to each other, first drafts, status chasing. The good ones deliver an evaluated, logged, permission-aware system you own outright. The weak ones deliver a demo with a monthly fee attached.
For custom, fixed-scope work: roughly $8,000-$20,000 for a single-purpose agent, $20,000-$45,000 for a multi-step workflow agent, and $45,000-$60,000+ for an enterprise platform, plus $50-$2,000 a month in inference. Quotes far outside those ranges deserve scrutiny in both directions.
Run the same six checks on both. Big firms bring headcount and compliance familiarity; small studios bring senior attention and speed. Our founder started as a data scientist at a Big Four consulting firm and watched too many engagements end at the strategy deck instead of shipped software. Judge either kind of firm by what actually shipped, not by the logo.
If you have engineers with production LLM experience and the calendar room, yes, and you should consider it. The model APIs are the easy part. The scarce skill is evaluation and reliability discipline. A common hybrid: hire a firm to ship v1 with a full eval suite, then take it in-house. That only works if you own the code, which is check number two.
Take the six checks and the red-flag list into every vendor call, ours included. If you want to run them on us, book a free 30-minute scoping call. You'll get a straight read on scope, cost, and fit, and we reply within two business days. If we're not the right team for it, we'll say so, and the checklist is yours to keep either way.
Book a scoping call or see how we run these engagements at AI agent development.
