For most of computing history, benchmarks ran in one direction: humans designed tests, machines took them. The benchmark measured machine capability against a human-defined standard.
Something has inverted. AI systems now routinely expose the limits of human evaluation. When a model produces an output that expert reviewers cannot reliably distinguish from human work, the benchmark has stopped measuring the machine and started measuring the reviewer. The question shifts from "can the model pass the test?" to "is the test still a test?"
The inversion followed capability. When models were weak, any competent human could evaluate their outputs. As models improved, evaluation became harder. Now, in domains like code generation, legal reasoning, and literary prose, model outputs frequently exceed the evaluation capacity of non-expert reviewers — and sometimes expert ones.
The result: bad benchmarks got exploited. Models trained to score well on capability tests without necessarily acquiring the underlying capability. The benchmark became a target, and Goodhart's law applied. Once a measure becomes a target, it ceases to be a good measure.
The more interesting effect: good benchmarks became diagnostic of human evaluation quality, not just model quality. A benchmark that a capable model saturates tells you the benchmark was too easy. A benchmark where human raters disagree sharply tells you evaluation is the bottleneck, not capability.
Evaluation infrastructure is now a first-class problem. Building systems that can reliably assess AI outputs is as important as building AI systems. The organizations that figure this out first have a durable advantage — not because they have better models, but because they can tell which models are better.
Human judgment is load-bearing in new ways. Not as a gold standard for correctness (models often know more than the evaluators), but as a filter for the things that matter: coherence, usefulness, alignment with unstated goals. These require human judgment precisely because they can't be fully specified in advance.
The capability curve makes this worse before it gets better. As models improve further, the evaluation problem compounds. The gap between model output quality and human evaluation quality will widen in most domains before better evaluation tools close it.
See also: evaluation-infrastructure.html