Use this when you've built (or are about to ship) an AI-powered feature and need to define what "good" looks like before it reaches users. Covers quality rubrics, golden datasets, eval pipeline design, and pass/fail thresholds. If you're assessing an entire engagement's AI health, use /ai-health-check instead -- this skill is for evaluating a specific feature's output quality.

Related skills: Eval criteria originate in /ai-product-spec. For engagement-level AI assessment, use /ai-health-check. For ongoing production monitoring, see /llm-observability-plan -- eval thresholds become monitoring thresholds. For clinical/regulated AI validation (more rigorous than standard eval), use /clinical-validation-protocol. For evaluating agent trajectories (multi-step, tool-calling) rather than single outputs, use /agent-eval-harness.

The hard part most teams miss

Eval design looks like a measurement problem and is actually three judgment problems a generalist skips:

1. The pass bar is not a number you pick, it is a number you derive. "We set the threshold at 85 percent" is where most eval systems quietly fail, because the 85 came from nowhere. The bar is stakes times reversibility times baseline times human parity (Step 2). A misread support reply and a misread lab value do not share a threshold, and pretending they do is how a safe-looking eval ships an unsafe feature.

2. The aggregate score hides the failure that matters. A single weighted quality number averages away the one catastrophic output, the unauthorized refund, the hallucinated dose, the agreement with a customer who is wrong. That is the entire reason hard-fail dimensions sit outside the weighted score (Step 2): some failures are not "lower quality," they are a different kind of event. See examples.md § Worked eval runs, Run 2, for an output that clears the weighted bar and fails anyway.

3. The dataset decays unless someone owns it. Evals do not rot because the method was wrong. They rot because nobody adds the new production failure as test case 51, so the suite slowly stops resembling reality. The highest-leverage decision in the whole plan is naming that owner (Step 8), not choosing LLM-as-judge versus human review.

Hold the plan to these three. Everything below is the mechanism; this is the point.

Process

Step 1: Define what "good" means

Ask the user:

1. What does this AI feature do? (One-line description of the capability)
2. What does a great output look like? (Ask for 2-3 real examples of ideal outputs)
3. What does a bad output look like? (Ask for 2-3 examples of outputs that would be unacceptable)
4. What are the failure modes you worry about? (Hallucination, wrong tone, harmful content, irrelevance, too slow, too expensive)
5. Who judges quality today? (Users via feedback, internal reviewers, automated checks, nobody yet)
6. What's the consequence of a bad output? (User annoyance, lost revenue, legal risk, safety risk)

The consequence of failure determines eval rigor. A chatbot that sometimes gives a mediocre answer needs lighter evals than an AI that generates medical summaries.

Step 2: Design the quality rubric

Create a scoring framework with explicit dimensions:

Customize dimensions to the feature. A code generation tool needs "correctness" and "security." A writing assistant needs "tone" and "originality." Not every dimension applies to every feature.

CARATS quality dimensions as starting columns: Use knowledge/ai-safety-evals-reference.md § behavior-spec-canvas for the full reference. The CARATS dimensions (Consistency, Accuracy, Reliability, Alignment, Tone, Security) provide a starting rubric -- choose the dimensions that match the feature's risk profile. Not every feature needs all six.

Threshold formula: There is no universal threshold. Each scenario's pass threshold is determined by four factors: stakes (cost of being wrong) x reversibility (can the user recover?) x baseline (what are you comparing against?) x human parity (how well would a person do this?). See knowledge/ai-safety-evals-reference.md § threshold-formula for the full framework. Show the math visibly -- don't hide behind "we set the threshold at 85%."

Topic tree for eval coverage: To ensure your rubric covers the full surface area, build a topic tree: start with the feature's top-level purpose, then branch into sub-capabilities, then branch into dimensions per sub-capability. Each leaf is an eval scenario. This prevents the common failure of testing only the happy path.

Binary pass/fail vs Likert scales: Prefer binary pass/fail for individual eval scenarios. Binary forces clearer criteria and more consistent labeling. Reserve Likert scales (1-5) for aggregate rubric dimensions where you need to track improvement over time. See knowledge/ai-safety-evals-reference.md § testing-vs-evals-bridge for the rationale.

Confusion matrix for classification features: When the AI component is a classifier (router, categorizer, tagger), add a confusion matrix to the rubric:

For safety-critical features, weight false negatives much higher than false positives. A missed safety signal is harm; a false alarm is annoyance.

Stakeholder-specific dimensions: Beyond generic quality dimensions, ask which stakeholders have skin in the game and add a dimension for each:

Not every feature needs all four. Ask: "Who gets paged if this goes wrong?" -- that person's concern is an eval dimension.

Scores 2 and 4 fall between adjacent anchors -- use them when output is better than "fail" but not quite "acceptable," or better than "acceptable" but not "excellent."

Scoring rules:

• Weight dimensions by importance (must sum to 100%). Start by ranking dimensions by consequence of failure -- the dimension where a bad score hurts most gets the highest weight.
• Set a minimum passing score (e.g., weighted average >= 3.5)
• Set hard-fail dimensions (e.g., Safety score of 1 = automatic fail regardless of other scores)

Step 3: Build the golden dataset

A golden dataset is a curated set of inputs with known-good expected outputs. This is the foundation of repeatable evaluation.

Dataset design:

For each entry:

• Input: The exact prompt/query/context sent to the model
• Expected output: What a good response looks like (or key elements it must contain)
• Evaluation criteria: Which rubric dimensions matter most for this input
• Source: Where this test case came from (user research, bug report, adversarial design)

Sourcing test cases: Ask the user what data they can draw from -- production logs, user research sessions, support tickets, competitor examples, or synthetic inputs. Real user data makes the strongest golden datasets.

Start with 30-50 entries. Grow the dataset as you find new failure modes in production. Version-control the dataset alongside your prompts -- when the prompt changes, you need to know which dataset version established the baseline.

Step 4: Choose the eval method

Where to screen -- input vs output:

Evals don't just run on model outputs. Decide where in the pipeline to evaluate:

Most teams start with output screening (catch bad responses) and offline evals (regression testing). Add input screening when specific query patterns are known to cause failures.

Recommended approach for most teams:

1. Automated first pass: Exact match or keyword checks for structural requirements (output format, required fields, length)
2. LLM-as-judge second pass: For quality dimensions that need judgment (relevance, tone, completeness)
3. Human review sample: Spot-check 10-20% of outputs weekly to calibrate the automated eval

LLM-as-judge setup:

• Write a clear judging prompt with your rubric embedded
• Include 3-5 calibration examples with scores and reasoning
• Test the judge against human-scored samples -- it should agree 80%+ of the time
• Use a different (and ideally more capable) model for judging than the model being evaluated -- a weaker judge can't reliably score a stronger model's output

Step 5: Define tradeoff boundaries

AI features live on three axes. You can't maximize all three:

Document explicit tradeoff decisions:

• "We'll accept slightly lower quality (3.5 vs 4.0 rubric score) to stay under $0.02 per interaction"
• "We'll use a more expensive model for safety-critical outputs and a cheaper model for low-stakes ones"
• "Latency above 5 seconds is unacceptable even if quality improves"

Step 6: Generate the eval plan

Compile into a structured document:

# Eval Plan: (Feature name)

**Generated:** (date)
**Feature:** (brief description)
**Quality bar:** (minimum passing rubric score)
**Hard-fail criteria:** (dimensions where score of 1 = automatic fail)

## Quality Rubric
(Table from Step 2 -- dimensions, score definitions, weights)

## Golden Dataset
(Summary from Step 3 -- categories, counts, sources)
(Link to or embed the actual dataset)

## Eval Pipeline
(Method from Step 4 -- automated checks, LLM-as-judge config, human review cadence)

### Automated checks
- (Check 1: e.g., output must be valid JSON)
- (Check 2: e.g., response must be under 500 tokens)
- (Check 3: e.g., must not contain PII patterns)

### LLM-as-judge configuration
- Judge model: (model name)
- Judging prompt: (summary or link)
- Calibration accuracy: (% agreement with human scores)

### Human review
- Frequency: (e.g., weekly sample of 20 outputs)
- Reviewer: (who)
- Process: (how disagreements are resolved)

## Tradeoff Boundaries
(Table from Step 5 -- quality, latency, cost targets and limits)

## When to Run Evals
- **Pre-deploy:** Run full golden dataset before every prompt or model change
- **Nightly/weekly:** Run on a sample of recent production outputs to detect drift
- **On regression reports:** When users report quality issues, add the failing input to the golden dataset and re-run

## Alerting
- **Threshold breach:** When eval scores drop below minimum passing score, notify (team/channel)
- **Hard-fail trigger:** When any hard-fail dimension scores 1, notify immediately
- **Drift detection:** When scores trend downward over 3+ eval runs, flag for investigation

## Implementation Checklist
- [ ] **(P0)** Build golden dataset with initial 30-50 entries
- [ ] **(P0)** Implement automated structural checks
- [ ] **(P0)** Set up LLM-as-judge with calibrated prompt
- [ ] **(P1)** Run baseline eval against current model output
- [ ] **(P1)** Establish human review cadence
- [ ] **(P2)** Set up regression testing in CI/CD
- [ ] **(P2)** Build dashboard for eval scores over time

## Open Questions
- (Unresolved eval decisions)
- (Things that need baseline data to determine)

Step 7: Generate demo-ready eval narrative (optional)

When the user needs to present eval progress to stakeholders, generate a plain-language narrative alongside the eval plan. This bridges the gap between having eval data and being able to explain it to people who expect feature demos.

Narrative template:

## Eval Progress Summary -- {{feature name}}, {{date}}

### Where we started
{{1-2 sentences describing baseline quality and what "bad" looked like in week 1}}

### Where we are now
{{1-2 sentences describing current quality with specific score improvements}}

### What moved
| Dimension | Before | Now | What changed |
|-----------|--------|-----|--------------|
| {{dimension}} | {{score}} | {{score}} | {{plain-language explanation of what the team did}} |

### What didn't move (and why)
{{Honest acknowledgment of dimensions that stayed flat, with explanation of prioritization}}

### What this means for the product
{{Business impact translation: fewer errors, less manual review, faster throughput, etc.}}

### What we're targeting next
{{1-2 dimensions, target scores, and what the team plans to do}}

Guidelines for the narrative:

• Lead with examples, not numbers. Show a before/after output side by side, then give the score.
• Translate scores to business impact: "Score went from 2.8 to 3.6" becomes "30% fewer outputs need manual correction."
• Don't hide regressions. Name them, explain the tradeoff, show the plan.

Related skills: For a full demo script built around eval progress, use /ai-progress-demo.

Step 8: Review and finalize

Ask the user:

• Does the rubric capture what matters most about output quality?
• Is the golden dataset representative of real user inputs?
• Is the eval method practical given team size and budget?
• Are the tradeoff boundaries honest or aspirational?
• How will eval results feed back into model/prompt improvements?
• Who owns maintaining the golden dataset over time? (This is the #1 reason eval systems decay -- nobody adds new test cases.)

Adjust based on feedback.

Clinical validation variant (for safety-critical or regulated AI)

When the AI feature affects patient safety, clinical decisions, or regulated data, apply clinical laboratory QC thinking on top of the standard eval:

Daily controls (continuous QC):

• Run a set of known-good inputs through the model on a defined schedule (daily or per-deployment)
• Compare outputs against established acceptable ranges (not just "pass/fail" -- track the actual scores)
• Apply Westgard-style rules: a single bad result is a warning; trending results or consecutive failures mean stop and investigate
• Document QC results even when they pass -- the trend matters more than any single point

Proficiency testing (external validation):

• Periodically evaluate the model using inputs from an external source (not the team that built it)
• Compare results against peer performance or expert panel consensus
• Proficiency testing reveals blind spots that internal QC cannot -- your golden dataset may have systematic gaps
• Failed proficiency testing requires root cause analysis and CAPA (see /capa-design)

False-negative cost weighting:

• For safety-critical AI, weight false negatives (missed adverse events, missed contraindications) much higher than false positives
• A false positive is an annoyance; a false negative is a harm. Adjust rubric weights accordingly
• Define hard-fail criteria: "If the model misses a safety signal, the eval fails regardless of all other scores"

When to use clinical validation instead of standard eval:

• The model's output directly informs treatment decisions
• The output is included in regulatory submissions
• Patient safety could be affected by a wrong output
• The system processes data subject to 21 CFR Part 11 or ICH-GCP
• Use /clinical-validation-protocol for the full validation plan

Output location

Present the eval plan as formatted text in the conversation for the user to copy into their docs tool.