A Fortune 100 insurer selected a model ranked first on MMLU for an adjudication assistant, and within six weeks p95 latency had climbed to 4.1 seconds while hallucination on policy-coverage questions was running at 7.3%. The failure was not the model. The failure was leaderboard-driven model selection — the anti-pattern of substituting aggregate academic benchmarks for task-specific, domain-calibrated evaluation — and it is far more prevalent in enterprise AI programs than practitioners admit.
The Production Failure That Started This
In Q3 2024, a Fortune 100 property-and-casualty insurer was evaluating LLMs for an internal adjudication assistant — a system that would help claims analysts determine coverage applicability against a 400-page policy corpus. The evaluation committee ran MMLU, HellaSwag, and TruthfulQA against five candidate models, ranked them, and selected the top scorer. Procurement moved. The model was deployed behind an API gateway with a straightforward retrieval-augmented generation stack.
Six weeks post-launch, the operational telemetry was damning: p95 latency was 4.1 seconds per query (acceptable threshold was 2.0 seconds), analyst adoption was at 31% of target, and an internal audit of 200 randomly sampled outputs found a 7.3% hallucination rate on coverage-determination questions. The model was fabricating policy exclusion clauses that did not exist. Analysts caught the errors — this time — but the risk surface was unacceptable in a regulated context.
The model had scored 87.4% on MMLU. It had never been evaluated on a single insurance adjudication task before deployment.
Why Leaderboard-Driven Model Selection Fails at Enterprise Scale
MMlu, introduced by Hendrycks et al. (arXiv:2009.03300), is a 57-subject multiple-choice benchmark covering academic knowledge from elementary mathematics to professional medicine. It is a useful coarse filter for general reasoning capability. It is not an evaluation of production fitness for any specific enterprise task, and conflating the two is the anti-pattern I will call leaderboard substitution.
The mechanism of failure is straightforward. MMLU measures recognition of correct answers within a closed set. Enterprise adjudication, clinical summarization, financial analysis, and legal review all require generation under domain constraints, calibrated uncertainty about information that may not be in the context, and consistent formatting across thousands of queries. These properties have no direct correlate in a multiple-choice benchmark.
Further, MMLU leaderboards are increasingly contaminated. A 2024 analysis by Roberts et al. (arXiv:2311.04607) documented systematic benchmark data leakage into training sets across major model releases. A model ranked first in April may have been trained on data that overlaps with the benchmark's test split. The score tells you something about that model's training data curation. It tells you nothing reliable about your insurance policy corpus.
The latency problem is a separate but related failure. Leaderboard comparisons rarely report tail latency under realistic load. A model that achieves median 800ms latency can easily produce p95 latency above 4 seconds when the prompt includes 3,000-token policy excerpts, retrieval context, and a structured output requirement. The insurer had tested latency with 150-token prompts. Production prompts averaged 2,800 tokens.
Martin Fowler's concept of an [evaluation harness](https://martinfowler.com/articles/building-an-enterprise-ai.html) is instructive here: an evaluation is only valid if it exercises the system under conditions that approximate production. A benchmark that does not include your document schema, your query distribution, your output format requirements, and your latency SLA is not a production evaluation. It is a vendor scorecard.
Chip Huyen makes the same point in Designing Machine Learning Systems (O'Reilly, 2022): offline metrics and online metrics diverge, often sharply, and the divergence is largest in domains with high specificity. Insurance adjudication is about as domain-specific as enterprise NLP gets.
The Architecture I Recommend Instead
Task-specific evaluation must precede model selection. The evaluation harness should be constructed before any model is shortlisted, and it should include the following components:
1. A golden dataset of 300–500 labeled examples drawn from actual production queries. For the adjudication case, this means real analyst questions paired with verified coverage determinations signed off by senior adjusters. This dataset is your ground truth. It takes four to six weeks to construct properly. That time investment is non-negotiable.
2. Domain-calibrated metrics, not aggregate accuracy. For coverage determination, the relevant metrics are: factual precision (does the output cite a real policy clause?), factual recall (does it identify all relevant exclusions?), and abstention rate (does the model decline to answer when it lacks sufficient context?). ROUGE and BLEU are irrelevant here. LLM-as-judge scoring using a specialized evaluation model (e.g., GPT-4o with a structured evaluation rubric) combined with human review on a 10% sample is a workable production approach. Eugene Yan's [writing on LLM evaluation](https://eugeneyan.com/writing/llm-evaluators/) is the most practical treatment I have read on this subject.
3. Latency profiling under realistic load. Run candidate models at p50, p95, and p99 latency with prompts drawn from your actual token-length distribution. Profile under concurrent load matching your expected QPS. If your p95 requirement is 2 seconds at 50 QPS, test at that load.
4. Adversarial probes specific to your domain. For insurance, this includes questions about policy terms that do not exist, questions designed to elicit confident answers on ambiguous exclusion language, and multi-hop questions that require cross-referencing two policy sections. OWASP LLM Top 10 ([owasp.org/www-project-top-10-for-large-language-model-applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)) categorizes overreliance and hallucination as LLM09; your adversarial suite should produce a measurable hallucination rate for each candidate.
5. Retrieval-integrated evaluation. Do not evaluate the language model in isolation if it will be deployed with RAG. Evaluate the full pipeline: retrieval quality (nDCG@10 on your corpus), generation quality on retrieved context, and generation quality on misretrieved context (what does the model do when the retrieved chunk is wrong?).
On model size: for domain-specific tasks with structured outputs, a 13B–34B parameter model fine-tuned on domain data frequently outperforms a 70B+ general model at lower latency and cost. The insurer's corrective path was to fine-tune a 13B model on 12,000 labeled adjudication examples and run it against a hybrid retrieval pipeline. Hallucination dropped to 1.1%. P95 latency fell to 1.4 seconds.
# Minimal evaluation harness skeleton
from datasets import load_dataset
from eval_utils import llm_judge_score,
factual_citation_check
def evaluate_pipeline(pipeline, golden_dataset,
sample_size=500):
results = []
for
example in golden_dataset.select(range(sample_size)):
output = pipeline.run(example["query"],
example["context"])
results.append({
"factual_precision": factual_citation_check(output,
example["policy_text"]),
"judge_score": llm_judge_score(output,
example["gold_answer"]),
"latency_ms": output.latency_ms,
"abstained": output.abstained,
})
return
aggregate_metrics(results)The AWS Well-Architected Framework's Machine Learning Lens ([docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/wellarchitected-machine-learning-lens.html)) explicitly calls out the need for business-context evaluation prior to model selection. The NIST AI Risk Management Framework ([nist.gov/system/files/documents/2023/01/26/AI RMF 1.0.pdf](https://airc.nist.gov/RMF)) categorizes benchmark reliance without task validation as a model risk under the GOVERN function. Both frameworks are correct. Neither is operationally specific enough for engineering teams to implement without a concrete harness.
Production Readiness Checklist
1. Golden dataset exists and is signed off by domain SMEs — minimum 300 labeled examples covering the full query distribution, including edge cases and adversarial inputs.
2. Evaluation metrics are task-specific — no ROUGE, no BLEU, no aggregate accuracy. Factual precision, calibrated recall, abstention rate, and format compliance are defined and measurable.
3. LLM-as-judge scoring is validated — the evaluation model's verdicts are spot-checked against human review on at least 50 examples. Inter-rater agreement is above 0.80 (Cohen's kappa).
4. Latency profiling is done at production token lengths and QPS — p95 and p99 latency are within SLA at 1.5x expected peak load.
5. Adversarial hallucination suite is executed — every candidate model has a measured hallucination rate on domain-specific adversarial prompts before selection.
6. Retrieval-integrated evaluation is complete — pipeline evaluation includes misretrieval scenarios; the model's behavior on wrong context is documented.
7. Benchmark contamination risk is assessed — training data cutoffs and known contamination reports are checked for each candidate model against your evaluation dataset's source material.
8. A/B deployment plan exists — model is not promoted to 100% traffic without a shadow-mode or canary deployment phase with production traffic.
9. Regression suite is defined — evaluation is re-run on every model version update, with a defined threshold for regression that blocks promotion.
10. Compliance and regulatory sign-off is documented — for regulated industries, the evaluation methodology and results are archived as part of model governance documentation.
What I Would Build Differently
The harness I described above assumes you have 300–500 labeled examples before you begin. In practice, enterprises rarely have clean labeled data at the start of an AI program. The labeling work is often underestimated by a factor of three in project plans — it is slow, expert-dependent, and politically contentious when the domain experts disagree with each other. I have seen golden datasets that took four months to produce, not four weeks. If I were starting over, I would budget labeling effort as a first-class project workstream, with its own PM and a defined schema before any model evaluation begins.
The second gap in this architecture is temporal validity. A golden dataset constructed from 2023 queries will drift in coverage against 2025 production queries as products, policies, and regulations change. I have not yet implemented a satisfying automated drift-detection mechanism for evaluation datasets — the best I have managed is a quarterly human review of query clusters against the golden set, which is better than nothing but not rigorous. This is an open problem.
Finally, LLM-as-judge scoring introduces a meta-evaluation problem: who evaluates the evaluator? The judge model has its own biases and failure modes. For high-stakes regulated applications, human-in-the-loop validation of at least 10% of outputs is not optional, and that cost should be included in the total cost of ownership calculation from day one.
DIAGRAM_HINT: Side-by-side comparison: left panel shows leaderboard-driven selection flow (MMLU score → model selection → deployment → production failure metrics); right panel shows task-specific evaluation harness flow (golden dataset construction → domain metrics → latency profiling → adversarial probes → retrieval-integrated eval → model selection), with failure rates and latency annotations at each stage.
Figure 1. Side-by-side flow diagram: leaderboard-driven model selection path (MMLU → selection → deployment → 7.3% hallucination, 4.1s p95) vs. task-specific evaluation harness path (golden dataset → domain …
Comments (0)
Join the conversation!
No comments yet. Be the first to share your thoughts!