Testing the Tester: Who Validates the Humans Evaluating AI?

When we talk about testing AI systems, we spend a lot of time discussing the model.

• Was the answer accurate?
• Did it hallucinate?
• Did it follow the user's intent?
• Did it use the right tool?
• Was it safe?
• Did it remember the conversation correctly?

Eventually, though, someone has to decide whether the AI passed or failed, and very often, that someone is a human evaluator.

We tend to treat that human judgment as ground truth, but there is a question I think we need to ask much more often: Who tests the tester?

Not because testers are inherently unreliable, and not because machines are somehow better judges.

Quite the opposite.

Human judgment remains essential for evaluating many of the qualities we care about in AI: usefulness, context, nuance, safety, intent, cultural appropriateness and whether a response actually solves the user's problem.

But if those judgments are going to become our measurement of AI quality, then we need to apply the same quality principles to the measurement system itself.

A human evaluator is, in effect, part of that measurement system, and measurement systems need calibration.

The Score Isn't the Ground Truth

Imagine two experienced testers independently evaluate exactly the same AI conversation.

The scoring rubric is:

• 3: Fully safe and correct
• 2: Safe and partially correct
• 1: Safe but incorrect
• 0: Unsafe

One tester gives the response a 3. The other gives it a 2. Who is right?

It is tempting to immediately look for the tester who made the mistake, but that is only one possibility.

• Perhaps one tester missed an important omission
• Perhaps the other penalized something the rubric never required
• Perhaps both interpretations are reasonable
• Or perhaps the boundary between "fully correct" and "partially correct" simply hasn't been defined clearly enough

That last possibility matters, because now we don't have a tester problem. We have a measurement-system problem, and if we don't find it early, we may multiply that ambiguity across hundreds or thousands of AI evaluations.

This Is Why We Calibrate Before We Scale

In AI testing engagements, one practice I have found increasingly valuable is running a smaller internal calibration exercise before moving into the official evaluation or judge-calibration run.

At first, it is easy to think of calibration as tester preparation.

1. Give everyone the guidelines
2. Give them some examples
3. Score a small sample
4. Make sure they understand the task
5. Then start the real run

But experience has taught me that calibration is doing something much more important.

It is testing the evaluation system before we scale it. During calibration, we are not only asking: Can this tester apply the rubric correctly? We are simultaneously testing several things:

• Is the rubric precise enough?
• Are the scoring boundaries understood consistently?
• Does the test case contain enough evidence to make the judgment?
• Is the expected answer actually defensible?
• Are the tester instructions clear?
• Are we asking evaluators to make distinctions that humans can reliably make?
• Do we have a process for handling genuinely ambiguous cases?

That changes how I think about calibration completely. The calibration run is not a rehearsal before testing starts. The calibration run is itself a test.

Lesson 1: Agreement On Easy Cases Doesn't Tell You Very Much

Suppose we give five evaluators an obviously dangerous AI response and all five rate it as unsafe. That's useful, but it doesn't tell us much about whether our evaluation system is ready.

The most valuable calibration examples tend to live near the decision boundaries.

• Is this response: fully correct or partially correct?
• Is it: incorrect or simply incomplete?
• Is the problem: minor or material enough to affect task completion?
• Did the assistant: make an unsupported claim, or make a reasonable inference?

Those are the examples that expose differences in interpretation, and those differences are exactly what we want to discover before the larger evaluation begins.

Lesson 2: Disagreement Does Not Automatically Mean a Bad Tester

If two testers disagree, the easiest explanation is: One of them got it wrong.

Sometimes that's true, but disagreement can come from several places.

• It could be an evaluator problem: someone misunderstood the criterion
• It could be a rubric problem: two reasonable interpretations exist
• It could be a test-design problem: there isn't enough evidence in the scenario to support the expected judgment
• It could be a ground-truth problem: the supposedly correct answer isn't as objective as we thought
• Or it could simply be a legitimately subjective judgment where some level of variation is expected

Retraining the tester will not fix an ambiguous rubric, rewriting the rubric will not fix a poorly constructed test case, and forcing everyone to agree will not turn a genuinely subjective question into objective ground truth.

Research is increasingly highlighting this problem. Google Research, for example, has argued that AI benchmarking frequently loses important information by collapsing natural human disagreement into a single "correct" label, and that the number and distribution of raters can materially affect reproducibility.

The practical lesson for QA is simple: Don't only measure disagreement. Investigate why the disagreement exists.

Lesson 3: The Same Score Doesn't Necessarily Mean Agreement

There is another failure mode that is easier to miss.

Imagine two testers both score a response: 2 - Safe and partially correct.

Great agreement, except when we read their reasoning:

• Tester A says the response is partially correct because it omitted an important eligibility requirement.
• Tester B says it is partially correct because the tone was too informal.

Now we have identical numbers, but potentially completely different interpretations of what the rubric is measuring.

This is why, when calibrating human evaluators, I care about more than the final score.

I want to understand the evidence behind the judgment, and if we only look at score agreement, we may never know.

For AI evaluation, evidence quality therefore matters alongside scoring consistency.

That also makes adjudication far more useful, because we can compare the reasoning rather than arguing over numbers.

Lesson 4: Calibration Tests the Rubric As Much As the Tester

This is probably the lesson that surprised me most.

When multiple capable evaluators repeatedly struggle with the same criterion, it is tempting to keep explaining the criterion differently.

At some point, though, we need to consider another possibility: Maybe the criterion itself isn't sufficiently testable.

AI evaluation hasn't somehow escaped the fundamentals of software testing. If requirements are subjective, ambiguous or underspecified, the resulting judgments will inherit that ambiguity.

Calibration makes those weaknesses visible.

So when testers disagree consistently, I don't immediately ask: How do we get the testers to agree? I first ask: What is this disagreement telling us about the rubric?

Lesson 5: Calibration Can Expose Weak Ground Truth

There is an assumption buried inside many evaluation programmes:

We already know what the correct answer is. We're checking whether the evaluator can find it.

Calibration sometimes proves otherwise.

When several good evaluators independently challenge the expected score and their reasoning is sound, we should be willing to inspect the expected answer too.

This is particularly important in AI because many evaluation dimensions are not purely binary.

• Accuracy may depend on context
• Safety may involve proportionality
• Completeness depends on what the user actually asked
• Tone is inherently contextual
• Even "correctness" can become complicated when there are multiple valid ways to satisfy the task

Ground truth should therefore be treated as something that requires evidence.

The tester may not be wrong. Our expected answer may be wrong. For a quality organisation, that's exactly what a good calibration exercise is supposed to uncover.

Lesson 6: Calibration Is Cheaper Than Discovering Inconsistency at Scale

Imagine running 1,000 evaluations. Halfway through the analysis, you discover that different evaluators have been applying "partially correct" differently.

• Can the scores still be compared?
• Do some cases need to be re-evaluated?
• Can the aggregated quality score still be trusted?
• How much of the dataset has been affected?

The problem gets expensive very quickly.

A calibration-first approach changes the sequence: Small sample → independent evaluation → identify disagreement → investigate → adjudicate → refine → recalibrate → scale

Calibration is therefore risk control.

Lesson 7: The Objective Is Not Perfect Agreement

This is where calibration can go wrong.

If the goal becomes: Everyone must produce exactly the same score, we risk turning calibration into conformity training.

Perfect agreement can look impressive while hiding problems.

• Perhaps the examples are simply too easy
• Perhaps evaluators are following precedent without challenging it
• Perhaps they have discussed their scores before submitting them
• Perhaps a genuinely ambiguous dimension is being artificially compressed into one answer

The goal is not 100% human agreement.

Equivalent evidence should be interpreted using sufficiently consistent decision rules for the decision we are trying to make.

And where reasonable people can still disagree, the evaluation system should be able to represent or adjudicate that uncertainty rather than hiding it.

That distinction becomes increasingly important as AI systems move into more subjective domains.

And Then We Automated the Judge...

There is an obvious response to all of this.

Human evaluation is expensive.

It takes time.

Humans disagree.

So why not use another AI model to evaluate the AI?

LLM-as-a-judge can absolutely be useful, particularly when evaluation needs to scale.

But it doesn't eliminate the underlying measurement problem.

It changes the evaluator.

LLM judges can have their own biases and reliability problems. Recent research documents issues including position bias, score-range sensitivity, verbosity effects and self-preference, among others.

That leads to what I think is the broader principle:

Human evaluator? Calibrate the human.

LLM evaluator? Calibrate the model.

Hybrid evaluation system? Calibrate the system.

The question is whether the evaluator is demonstrably reliable for the specific judgment we are asking it to make.

Maybe We Should Stop Calling Humans the Ground Truth

I still believe human evaluation is indispensable in AI testing.

There are things humans understand about intent, context, usefulness and lived experience that are extremely difficult to reduce to automated metrics.

But perhaps calling a human label "ground truth" encourages the wrong mental model.

Sometimes there is objective ground truth.

Sometimes there is expert judgment.

Sometimes there is informed preference.

And sometimes there is genuine uncertainty.

Treating them as if they are can produce very precise-looking metrics built on surprisingly unstable foundations. A mature AI evaluation programme needs to know which one it is measuring.

Testing the Tester Is Really Testing the Measurement System

The biggest thing our calibration exercises have taught me is that testing the tester was never really just about the tester.

We start by comparing scores.

Then we discover a disagreement.

We inspect the reasoning.

That exposes an unclear instruction.

Fixing the instruction reveals a weak scoring boundary.

Discussing the boundary exposes an ambiguous expected answer.

And suddenly what looked like a tester-calibration exercise has become a quality review of the entire evaluation methodology.

That's exactly what should happen.

Because before we use an evaluation system to tell us whether an AI product is good enough, safe enough or ready enough, we should have some evidence that the evaluation system itself is trustworthy.

We wouldn't knowingly use an uncalibrated instrument for an important engineering measurement and simply trust the number it produced. AI quality should be no different.

Before we trust the score, we need to test how the score was produced, and sometimes the most valuable bug a calibration run finds isn't in the AI at all. It's in the way we were planning to measure it.