Evals: Testing AI Features With No Right Answer

Exact-string assertions compare output to one reference string; unit tests can also check properties and structured behaviour. A language model has no single correct answer to compare against, so the usual test passes nonsense or fails a correct paraphrase. Test everything around the model that has one right answer, score the rest against a hand-scored list of test questions, and commit those numbers as a baseline CI can fail.

That hand-scored list is what the field calls an eval set, and the run that scores it is an eval. Everything below is the shape of a setup that survives contact with a real product.

Why do normal tests fail on AI output?

Three reasons, and only the first is obvious.

An equality assertion needs one right answer. "Code 10750 applies, see page 768" and "Page 768 points to 10750" are the same answer and different strings. Assert on the string and you fail correct work; assert on a substring and you pass an answer that got the number right for the wrong reason.

Temperature 0 is not determinism. The same prompt to the same model name can route to a different revision, a different serving stack, or a different batch, and come back different. Tests written as if the output were fixed go flaky in week three and get skipped by week five.

The worst reason: the failure mode is a confident, well-formatted, wrong answer. Nothing crashes. In May 2024 Stanford RegLab ran a preregistered study of 202 legal queries against commercial legal research tools and found Lexis+ AI wrong or misgrounded on more than 17% of queries and Westlaw's AI-Assisted Research on about 33%. The 58–82% figures for general chatbots come from earlier research summarised in the Stanford paper, rather than a direct comparison on this entire query set. Both products were sold on retrieval — the technique of feeding the model your own documents — and marketed as hallucination-free.

Note the word misgrounded. The answer carries a citation, and the cited text does not actually say what the answer claims. A check that only verifies "a citation is present and the document exists" is blind to it, which is exactly why so many teams believe their retrieval feature is clean.

What can you still test the normal way?

More than most teams assume. Wrap the model in a shell of deterministic behaviour and test the shell to death — it is free, fast, and catches the bugs that actually reach users.

Output shape. JSON parses, the schema validates, required fields exist. Ordinary.

Tool-call markup never reaches the user. Some models write their tool call as plain text instead of returning it in the structured field the API expects. In one system the model answered "let's assemble the list", then printed its own internal invoke markup straight into the chat. That is a pure string problem with three assertions: the leaked call is recovered and its arguments parse, the markup is stripped so the user sees only prose, and a normal grounded answer passes through byte-identical. Written once, it guards every model swap afterwards.

Every citation resolves. Each source id the answer emits must exist in the corpus. Set membership, no model involved.

Retrieval is measurable with forty-year-old metrics. Rank-1 accuracy, recall@10, mean reciprocal rank. Hand-write gold cases as {query, primary, topN, mustNotInclude} — and treat mustNotInclude as the valuable half. The plausible near-miss is what retrieval actually gets wrong: a query about producing food surfacing the code for manufacturing food-processing machines. In one classification assistant over a public business-code catalogue, 110 retrieval cases hold recall@10 at 1.0 and MRR at 1.0 with zero must-not-include violations — and that suite failed loudly the day a scoring rule started demoting a correct code.

Application guardrails fire. Deterministic rate limits, access checks, and redaction paths can be asserted. Model-generated refusals still require evaluation.

How do you build an eval set that is actually useful?

Start with hand-curated cases based on your own failure modes. Thirty relevant cases can be more useful than three thousand unchecked synthetic ones; inspected synthetic cases can still broaden coverage.

Give every case a query, the expected answer, a rank tolerance, and a list of answers that must not appear. Then put a test on the test set itself: assert that the thirty cases span at least ten distinct categories. Eval sets rot toward whatever the team was debugging last quarter, and a one-line coverage assertion stops that drift for free.

Then build a second, separate set that contains no right answer at all.

The abstention set. Eight deliberately useless inputs — "I want to start a business", "what's the most profitable industry", "how do I apply for a bank loan", a bare greeting. The metric is not accuracy. It is false confidence rate: how often the system produced a definite answer when the honest reply was "you haven't told me enough yet". Target zero. The baseline on the system above sits at zero false confidence with 37.5% of inputs triggering an abstention flag in the scoring logic. That flag is not a measured rate of model-generated clarification responses.

This set matters more than its size suggests. Kalai and Nachum at OpenAI argued in September 2025 that models hallucinate partly because evaluations score answers 0 or 1, so a model that always guesses outscores one that admits uncertainty. The same trap works one level up: score your own eval purely on accuracy and every prompt revision you keep will be the one that guesses harder. Penalise a confident error more than an honest "I don't know", or you will optimise your product into a bluffer. If you are also building the refusal path itself, the mechanics are in LLM hallucination guardrails.

Why does a perfect score sometimes mean nothing?

Because judged metrics have a denominator, and an abstention is not in it.

A misgrounding rate is unsupported claims divided by claims that carried a citation. When the system honestly declines, it makes no claims, so the case contributes nothing to the denominator. Fill your set with hard, ambiguous, heavily-regulated questions — the ones that feel like a serious test — and the system correctly abstains on nearly all of them. The report comes back with a beautiful rate computed over a single claim.

The fix is a deliberate mix. Twelve hard cases stay, because they test whether the system fabricates under ambiguity. Eight light ones join them: single-topic questions whose answer maps to exactly one record with complete source coverage, verified by inspecting the corpus first. Those produce confident, grounded, gradeable answers. The rebalanced baseline reads 29 claims with citations, 1 unsupported, 3 partial, a stored misgrounding rate of 0.045, and an abstention rate of 0.35. The scoring code subtracts 7 abstention flags from 29 cited claims, yielding 1/22, rounded to 0.045. The raw unsupported-claim ratio of 1/29 is approximately 0.0345; it uses a different denominator and is not the stored metric. This custom denominator also mixes claim counts with per-answer abstention flags, so review its definition before comparing other systems. Report the three partially supported claims separately.

Design the case mix so the metric has a population. A denominator of one is not a pass.

When do you need an AI judge, and how far do you trust it?

Use a judge only for what no assertion can reach: whether a claim is supported by the excerpt it cites. There is no gold answer to compare against — you judge the claim against the model's own retrieved source. Reference-free grading is the entire point.

Judges are also models, with their own defects. A large preprint submitted on 17 June 2026 ran 21 judge models from nine providers over three benchmarks — roughly 541,000 individual judgments — and found that raw agreement flatters them badly: Cohen's kappa, which discounts agreement you would get by chance, lands 33 to 41 percentage points below exact-match agreement on MT-Bench. Two judges already running in production showed severe position bias, preferring whichever answer was shown first. Verbosity bias was small, under 0.011, under that study's single pairwise rubric. Judge rankings moved by up to 14 positions depending on the benchmark, so "best judge model" is not a portable claim.

What survives that in practice:

• Grade one claim at a time. Single-item grading against a source excerpt avoids choosing between first and second answers, though other biases remain.
• Force a verbatim quote. The judge must return the span that supports the claim, or the label unsupported. Verify that the returned quote exists in the source; a plausible explanation alone is insufficient evidence.
• Three or four labels, no 1–10 scale. Supported, partial, unsupported, no evidence. Fine-grained scales produce fine-grained noise.
• Calibrate once against humans. Hand-label thirty rows, measure agreement, and redo it whenever you change the judge model. A judge upgrade silently moves your baseline.

Budget for it. A judged run calls the provider once to generate the answer, then once per cited claim to grade it. Build two escape hatches into the runner from the start — a --dry mode with a stubbed model and stubbed judge so CI can verify the wiring offline for free, and --limit N to cap a live run while you are iterating.

How does this become a gate and not a report?

By writing the numbers to a file that CI reads.

Commit the baseline as JSON with four things in it: when it was generated, how many cases ran, the metrics, and the thresholds those metrics must hold. Keeping thresholds beside the metrics means a policy change shows up in the diff.

{
  "generatedAt": "2026-06-24T07:04:51.345Z",
  "caseCount": 20,
  "metrics": { "misgroundingRate": 0.045, "unsupported": 1, "abstentionRate": 0.35 },
  "thresholds": { "misgroundingRate": 0.05, "maxFailedRows": 0 }
}

Run the layers at different speeds. The deterministic suite runs on every commit — it avoids paid model calls and catches deterministic regressions. The judged suite runs on a schedule and on every prompt change, behind a --gate flag that exits non-zero when a threshold breaks.

Treat the prompt as code, because it is. A prompt edit changes production behaviour exactly like a code edit, and it is the change most likely to move groundedness. Version it, and re-run the judged suite before merge.

Store per-row results, not just aggregates. When a rate moves from 0.045 to 0.12, the aggregate tells you that something broke and the rows tell you which question, which claim, and which excerpt.

A minimal setup worth copying

1. Schema and guardrail assertions in your normal test runner, every commit.
2. One string test per known output-format defect, starting with tool markup leaking into user text.
3. Twenty to thirty gold cases with a mustNotInclude list and a coverage assertion over the set.
4. A separate abstention set scored on false confidence, target zero.
5. A reference-free judge on cited claims, one claim per call, with --dry and --limit.
6. A committed baseline with thresholds, gated in CI, rows and all.

Deterministic tests can check output formats, source identifiers, and application boundaries. They do not prove every answer correct or prevent every unsupported guess. Measure failures on curated cases and monitor actual use. If retrieval is still being designed, start with RAG for business data.

What should you ask before accepting an AI feature?

Prepare actual customer questions, including ambiguous and unanswerable ones. Agree which must be correct, when the system should clarify, and when it should refer to a person. Request case-level results before and after a model change; an average can hide expensive errors.