How to choose an AI development company: 12 questions that separate production teams from demo builders

Accuracy: will it actually work?

1. "How will we know how often it is right?"

Good answer: "We build an evaluation set from your real cases — questions, documents or tasks with known correct results — and score every version against it before release." Worry: "We test it thoroughly" with no mention of a test set, or "the model is very accurate".

2. "What happens when it does not know?"

Good answer: a concrete fallback — it says so, asks a clarifying question, or hands off to a person with context. Worry: any suggestion that a better prompt removes errors entirely.

3. "Can we see something you shipped to real users, and how it is measured today?"

A case study should describe the problem, the decisions and how the system is monitored — not only a screenshot. Many clients cannot be named; a good team can still explain what they built in detail.

Data and security: what happens to your information?

4. "Where does our data go, and which provider settings will you use?"

They should name the model providers, explain whether data is retained or used for training under the plan they will use, and redact data the model does not need. Worry: "it's all secure" with no specifics.

5. "How do you stop one user seeing another user’s data?"

For any assistant over private documents or customer records, permission checks have to happen in retrieval and in your systems — not in the prompt. If the answer is "we tell the model not to", walk away.

6. "If it can take actions, what stops it doing the wrong thing?"

Look for scoped tools, the user's own permissions, confirmation before anything irreversible, and a log of every action. This matters most for AI agents and MCP servers.

Cost: what will it cost to build and to run?

7. "What will it cost per month to run at our volume?"

A competent team can estimate cost per conversation or per task from token counts, and will measure it on a prototype before you commit to the full build. See our worked examples for AI agents and chatbots. Worry: no answer until after launch.

8. "What would you do first, and what would it cost?"

Good answer: a small, fixed-price first step — a discovery sprint or a pilot on one workflow — that produces a working result and real numbers. Worry: a large fixed quote for a broad "AI platform" before anyone has looked at your data.

9. "When would you tell us not to use AI for this?"

Some problems are better solved with rules, search or a plain form. A team that never says this is selling, not advising.

Ownership: what do you keep if you part ways?

10. "Who owns the code, prompts, evaluation set and cloud accounts?"

You should — from the first commit, in repositories and accounts in your name. The prompts and evaluation set are as valuable as the code; make sure they are in the repository, versioned.

11. "Are we locked to one model provider?"

Models improve and prices change every few months. A provider-agnostic model layer lets you switch when a cheaper or better model passes your evaluation set. Ask how long a switch would take.

12. "What does hand-over look like?"

Documentation, a runbook for incidents, dashboards showing quality and cost, and a session with whoever maintains it next. Ask to see a hand-over document from a previous project.

Freelancer, small team, large agency or in-house?

We are a small senior team, and we are honestly the wrong choice for a programme that needs twenty engineers next month. We are a good choice for taking one AI workflow from idea to production, or making an existing prototype reliable — see from AI proof of concept to production.

A simple scorecard

Score each vendor 0–2 on each question (0 = no real answer, 1 = vague, 2 = specific with an example). Anything below 16 out of 24 is a demo builder. A 0 on question 1, 5 or 10 is disqualifying on its own, whatever the total.