How to Tell If Your AI Feature Actually Works

Conventional software is tested by assertion. Given this input, the function returns this value. The test passes or it does not, and the same input produces the same result every time.

AI features break that model in two ways. The output is not deterministic, so the same input can produce different results across runs. And there is frequently no single correct answer — a summary can be good in several different ways and bad in several others.

Teams reach this point and often conclude that evaluation is impractical, so they ship on impressions: a few people try it, it seems fine, it goes live. Then it degrades, or it fails for a category of user nobody tested, and there is no way to tell what changed.

Evaluation is practical. It is just a different discipline from unit testing, and it has to be built deliberately.

Start with the failure, not the metric

The common mistake is to start by choosing a metric. Teams adopt a published benchmark score or a general quality rating and discover it does not move when the system improves, or moves when nothing meaningful changed.

Start instead by collecting actual failures. Run the feature against real inputs, look at the outputs, and write down what is wrong with the bad ones in plain language. After thirty or forty, the failures sort themselves into categories: it misses information that exists in the source, it invents specifics, it ignores the requested format, it answers in the wrong language, it is confidently wrong about a particular class of question.

Those categories are what to measure. A metric derived from observed failures tracks the thing you are trying to fix. A metric adopted from elsewhere measures something that may not be your problem at all.

Build the dataset from real inputs

An evaluation set is a collection of inputs with known expected behaviour. Its quality determines everything downstream.

Use real inputs, not invented ones. Inputs a team writes to test a feature are cleaner, better formed and more polite than what users send. A system tuned against invented inputs performs worse on real ones in ways that are invisible until launch.

Include the hard cases deliberately. Missing data, ambiguous phrasing, inputs in a second language, questions that have no answer in the available material, conflicting sources. These are the conditions production supplies constantly and demos never contain. A dataset of easy cases produces a score that is high and meaningless.

Have a qualified person define the expected behaviour. For domain tasks, that means someone who knows the domain, not the engineer who built the feature. Where the correct answer is contested, record the disagreement rather than forcing a single key.

Keep it modest and grow it. A carefully built set of a hundred cases is more useful than a thousand assembled carelessly. Add every production failure to it as they occur, which turns incidents into permanent regression coverage.

Hold some back. Keep a portion of the set unseen during development, so improvements can be checked against cases that were not used to make them.

Four ways to score, and when each applies

Exact and structural checks. Where the output has a defined shape — valid JSON, a required field present, a code that exists in a reference list, a number within a range — assert it directly. These checks are cheap, deterministic and catch a surprising share of real failures. Run them first.

Reference comparison. Where a correct answer exists, compare against it. String similarity is weak for free text, because a correct answer phrased differently scores badly. Semantic similarity is better but blurs meaningful distinctions. These are useful as a rough signal and poor as a sole criterion.

Model-as-judge. A second model scores the output against written criteria. This scales, which is its main advantage, and it has well-known weaknesses: judges favour longer answers, favour outputs from similar models, and drift when the criteria are vague. It works when the criteria are specific and binary — does this answer cite a source for every factual claim, yes or no — and poorly when asked for an overall quality rating. Calibrate the judge against human judgement on a sample before relying on it, and recalibrate when you change the judge model.

Human review. The most accurate and least scalable. Reserve it for calibrating the automated methods and for the cases they disagree on, rather than as the routine path.

Most working systems use all four: structural checks on everything, reference comparison where keys exist, a judge for the rest, and human review on a sample.

Measure the components separately

A single end-to-end score hides where a system is failing, and the hiding is systematic rather than random.

For a retrieval-based system, ask two questions in order. Was the correct material retrieved at all? And given what was retrieved, was the answer faithful to it? A system can score well on final answers while its retrieval is weak, and you will not see that until the document set grows or the question distribution shifts.

For an agent, evaluate the trajectory as well as the outcome. A run that reached the right answer through an invalid path will fail on the next input that differs slightly. Knowing which step failed is what makes a failure actionable; a binary outcome is not.

For anything multilingual or multi-segment, break the score down by language, by customer, by document type, by question category. An aggregate average is dominated by whichever segment has the most cases, and a system can be failing badly in one language while the headline number looks healthy.

Variance is a metric

Run the same evaluation set several times. The outputs will differ, and the spread is itself information.

A system with high variance is not ready regardless of its average score, because users experience individual runs rather than averages. A feature that is excellent on most attempts and badly wrong on some is worse, operationally, than one that is consistently adequate — the first requires every output to be checked, the second does not.

Variance also invalidates single-run comparisons. If two configurations differ by a small margin on one pass, that difference may be noise. Run both several times before concluding anything.

When to run it

Evaluation that runs only before launch answers one question once. The useful version runs continuously.

On every change to the prompt, the model, the retrieval configuration or the tools. All four change behaviour, and prompt changes in particular have effects far outside the thing being adjusted.

On model version updates. A provider updating the underlying model changes your system's behaviour without you deploying anything. The evaluation set is what detects it.

On a schedule in production, against live traffic samples. The input distribution shifts as users learn what the feature does, and a system tuned for last quarter's questions degrades quietly against this quarter's.

Treat a drop in score the way a failing test suite is treated: as a blocker, not as a data point to discuss later.

Measure cost and latency alongside quality

Quality metrics in isolation push a system toward configurations that are better and unusable — more retrieved context, more reasoning steps, a larger model, every improvement paid for in seconds and tokens.

Record cost per task and latency per task in the same evaluation run. The decision is then explicit: this change improves accuracy by a given amount and costs a given amount more. That is a trade-off a team can make deliberately rather than discovering it on the invoice.

Production feedback is the other half

An evaluation set is a fixed sample. Production is the live distribution, and it moves.

Capture the signals that already exist: where users reject or edit an output, where they rephrase and ask again, where they escalate to a human, where a conversation ends abruptly. None of these is a quality score, and all of them point at the cases worth examining.

Review a sample of real outputs regularly, with a person reading them. The failures found this way are the ones nobody predicted, and they are the ones that belong in the evaluation set next.

Why teams skip this

Evaluation is skipped for three reasons, all understandable.

It requires effort before any value is visible. Building a dataset is slow work with nothing to demonstrate at the end of it, competing against feature work that shows something.

It requires domain expertise that the engineering team does not have. Defining correct behaviour for trade classification, clinical summarisation or contract analysis means getting time from people whose time is scarce.

And it produces uncomfortable information. A team that measures properly learns that the feature is wrong more often than the demo suggested, which is a harder conversation than not knowing.

That third reason is the actual one in most cases. It is also the strongest argument for doing it, because the information exists whether or not it is measured — the only question is whether the team learns it before the customer does.

How we approach it

Accuracy is a property of a corpus and a question set, not a property of a model. A configuration that performs well on one organisation's documents can perform badly on another's, because the vocabulary, languages and document structure differ — which means evaluation belongs next to each customer's data rather than in a general benchmark.

In Clavix360, our AI CRM and ERP platform, each organisation runs on its own instance with its own data, so the evaluation question is asked per organisation rather than answered once centrally. For Ara, our cross-lingual matching research programme, evaluation is built from published classification decisions where the reasoning is on record, so both the result and the path to it can be checked — a system that reaches the right answer by the wrong route will break on the next case.

The short version for anyone starting: collect real failures, write the categories down, build a small dataset of real inputs including the hard ones, score the components separately, and run it on every change. Everything else is refinement.