Evaluation as a Control System for AI Products

An evaluation score is descriptive. A control system is operational: it observes a system, compares evidence with policy, decides what may happen next, and verifies the consequence.

Start from decisions

Before designing a benchmark, list the decisions evidence must support:

• May this model or prompt enter production?
• Which requests require retrieval, a tool, a stronger model, or human review?
• What failure rate triggers rollback or incident response?
• Which capability claim may be shown to users or executives?
• What data must never enter prompts, traces, or evaluator inputs?

If a metric cannot influence one of these decisions, it may still be useful for research, but it is not yet part of the operating control loop.

Build an evaluation portfolio

Deterministic checks

Use parsers, schemas, invariants, permissions, citation existence, tool preconditions, and exact calculations wherever the requirement is deterministic. Do not replace a software assertion with an LLM judge.

Reference and rubric evaluations

Use labeled examples and explicit rubrics for quality dimensions that require interpretation. Separate factual support, task completion, style, policy compliance, and user usefulness rather than collapsing them into one score.

Adversarial and boundary evaluations

Test injection, exfiltration, conflicting instructions, missing context, long inputs, malformed tools, stale data, cancellation, retries, and dependency failures. High averages do not compensate for an unbounded failure with high consequence.

Human evaluation

Use domain experts when correctness depends on tacit practice, consequence, or professional judgment. Measure agreement and document ambiguity. Human review is evidence with its own variance, cost, and potential bias.

Production evidence

Observe outcome, latency, cost, error, escalation, correction, and complaint signals without indiscriminately storing sensitive inputs. Production monitoring detects distribution changes and failure combinations that offline sets miss; it does not replace pre-release testing.

Tie thresholds to consequence

| Risk class | Example policy | |---|---| | Low consequence and reversible | Monitor quality and allow rapid correction | | Material but reviewable | Require threshold, source evidence, and sampled human audit | | High consequence | Require domain validation, bounded authority, escalation, and rollback readiness | | Prohibited consequence | Prevent the action regardless of model confidence |

Thresholds should include uncertainty and sample size. A change from 91% to 93% on a tiny convenient set is not automatically progress. Segment results by task, user, language, input length, risk, and failure type so aggregate gains cannot hide regressions.

Release control

A release manifest should identify the exact model, prompt, retrieval configuration, tools, policies, evaluator versions, test-set version, and system image. Approval applies to that artifact, not to a mutable project name.

Deploy the tested image to preview, verify its actual routes and dependencies, record approval against its digest, and promote the same artifact. Rebuilding after approval breaks the evidence-to-artifact link.

Common anti-patterns

1. Optimizing a public benchmark that does not represent the product.
2. Using one opaque judge for every quality dimension.
3. Changing prompts, data, evaluator, and model together without attribution.
4. Reporting an average without slices, uncertainty, or failure severity.
5. Collecting extensive traces without a retention and privacy decision.
6. Monitoring failures without assigning an owner and response.

Operating principle

Evaluation is not the final gate after engineering. It is the feedback architecture that connects product intent, system behavior, risk policy, deployment, and organizational learning.