From AI proof of concept to production: what demos skip

Demo vs production at a glance

1. Build an evaluation set before you tune anything

The single biggest difference between teams that ship AI features and teams that stall is an evaluation set: a collection of real inputs, with what a correct output looks like, that you can score automatically. Without one, every prompt change is a guess and every model upgrade is a gamble.

• Use real data, not invented examples. Pull cases from production logs, support tickets or documents — including the messy ones.
• Keep a held-out set. Tune prompts on one split and report results on another, so you are not grading your own homework.
• Score what matters. For classification, accuracy per class. For extraction, field-level correctness. For generated text, a rubric of specific yes/no questions ("does the reply invent facts not in the source?") rather than a vague 1–10 score.
• Add every production failure to the set. The evaluation set should grow with the feature.

2. If an LLM grades the output, grade the grader

For open-ended output, a second model scoring the first against a rubric ("LLM-as-judge") is often the only scalable option. It is also easy to get wrong: an uncalibrated judge happily passes bad answers or fails good ones, and you end up optimising toward its quirks.

The method we use, including in our own testing: have people label a sample of outputs against the same rubric, run the judge on that sample, and compare. Track false passes and false fails separately — a judge that passes everything looks great on average. Iterate on the rubric and judge prompt until agreement with human labels is high, and only then trust the judge on the full set. We have also found that a model's self-reported confidence is a poor stand-in for this; measure against labels instead.

3. Make the output fit your software

If code consumes the model's output, "usually valid JSON" is a production incident waiting to happen. Major model APIs support schema-constrained (structured) output: you supply a schema and the response is generated to match it. Validate it anyway before it touches your database.

In an AI assessment and reporting app we built for a leadership consultancy, templates are generated as typed JSON from a free-text topic, and executive summaries follow a fixed section contract. We also gave the summariser labelled findings instead of raw scores, which removed a whole class of confident, numerically wrong statements. Giving the model less — only what it needs — often improves accuracy and lowers cost at the same time.

4. Control what it can do, and make actions safe to repeat

Prototypes typically run with an admin API key. Production AI that takes actions must act as a specific user with that user's permissions, so it can never do more than the person on whose behalf it runs.

• Route actions through your existing validated API, not direct database writes, so business rules are enforced in one place.
• Make writes idempotent. Network retries, model retries and duplicate tool calls happen. An invoice must not be sent twice because a request timed out.
• Require confirmation for irreversible actions, especially when the AI reads untrusted content (emails, documents) that could contain injected instructions.
• Log every action with the user, inputs and result.

These are the defaults behind the remote MCP server we built so a consultancy's staff could run operations from Claude: each tool proxies an existing authenticated endpoint and inherits the signed-in user's permissions.

5. Plan for the model being slow, wrong or unavailable

• Timeouts and retries with backoff, distinguishing per-minute rate limits from daily quotas and from real errors — each needs a different response.
• Surface retry timing to users instead of failing silently when a provider returns a quota error.
• A non-AI fallback. If summarisation fails, show the source; if classification fails, route to a person. The feature degrades instead of breaking.
• Background jobs for slow work. Do not hold a web request open for a 30-second model call; queue it and stream or notify.

6. Know what each request costs

Track tokens and cost per feature, per customer and per model from day one. It turns pricing decisions — included, metered, add-on — into arithmetic, and it catches runaway loops early. It also lets you safely move easy steps to a smaller model: if the evaluation score holds, the saving is real. Our AI agent cost breakdown walks through the token arithmetic at current prices.

7. Trace everything

When a customer reports a strange answer, support needs to see exactly what the model received, which tools it called, what came back and how long each step took. LLM-observability tools such as Langfuse, or traces exported through OpenTelemetry, give you that per-request history. Store prompts and outputs only where your data agreements allow, and redact what you do not need.

8. Roll out gradually

1. Shadow mode. The AI runs on real traffic but only proposes; people still act. Compare its proposals with what they did.
2. Feature flag to a small group. Real users, real feedback, easy rollback.
3. Supervised autonomy. The AI acts on the cases the evaluation set shows it handles reliably; everything else goes to a person.
4. Wider rollout, with the evaluation suite running on every prompt or model change and production failures feeding back into it.

When the right answer is to stop

Sometimes the evaluation set shows the idea does not work well enough on real data, and no amount of prompt tuning closes the gap. That is a successful outcome of a proof of concept, not a failure — it cost weeks instead of a launch. The honest options are then to narrow the scope, fix the underlying data, keep a person in the loop permanently, or drop the feature.

If you have a prototype that needs to become a dependable feature, that is the core of our AI integration services. For agents that take actions across systems, see AI agent development.