AI Evals Are Product Design, Not QA

Most AI evaluation starts too late. A team builds a promising workflow, collects a set of examples, argues about what a “good” answer looks like and then asks product to turn the resulting score into a launch decision. That sequence treats evaluation as quality assurance. For an AI product, evaluation is part of the product definition.

An eval is useful when it tells the team whether the product helped a user make a better decision—not merely whether the model produced plausible text.

One Score Cannot Describe A System

A RAG or agentic product is not a model endpoint. It is a chain: retrieval, context construction, reasoning, tool use, presentation and a user action. A failure at any layer can create the same visible symptom. “The answer was wrong” might mean the right source was never retrieved, the source was retrieved but ignored, the tool call failed, or the interface encouraged the user to trust an uncertain answer.

Collapsing that system into one aggregate accuracy number makes the dashboard simpler and the product harder to improve. The team needs separate measures for separate product decisions.

Start With What The AI Is Allowed To Decide

Before writing an evaluation dataset, define the decision the product is allowed to influence. A research assistant may summarize evidence but should not silently decide strategy. A qualification agent may identify fit but should hand a high-value or ambiguous prospect to a person. The boundary determines what “correct” means.

This is also where human-in-the-loop design becomes concrete. “A human reviews the output” is not a product design. Which outputs? At what confidence? With what evidence visible? What happens when the reviewer disagrees? Those questions produce evaluation cases and interface requirements at the same time.

Build The Dataset From Product Risk

Random examples usually over-represent the happy path. Useful eval sets are intentionally inconvenient. They include ambiguous intent, conflicting sources, missing data, long context, stale documents, tool timeouts and requests that sit just outside policy. They also reflect the distribution that matters commercially—not only what is easy to label.

For a RAG-backed workflow, I would deliberately separate at least 4 failure classes:

• Retrieval miss: the necessary evidence exists but does not enter context.
• Grounding failure: the evidence is present but the answer contradicts or ignores it.
• Abstention failure: the system answers confidently when the evidence is insufficient.
• Decision failure: the answer is factually acceptable but does not help the user take the next action.

Offline Evals Need Online Consequences

Offline evaluation makes iteration fast. Online behavior tells you whether the metric deserves trust. A response can score well with reviewers and still increase user effort. It can be accurate and arrive too late. It can improve task completion while creating an unacceptable escalation rate for operations.

Connect the layers. If groundedness improves, does acceptance improve? If acceptance improves, does the downstream decision improve? If conversion rises, what happens to quality or retention? This is ordinary product thinking applied rigorously to a probabilistic system.

The Product Manager’s Job

The PM does not need to invent every grader or own the evaluation infrastructure. The PM does need to define the behavior worth rewarding, the failures worth preventing and the business trade-off the team is willing to make. That work cannot be delegated to a generic benchmark.

When evaluation is designed early, it improves the architecture, the fallback, the interface and the launch plan. When it is added at the end, it mostly produces a number people negotiate around. That is why AI evals belong in product design.