Eval Writer

---
name: eval-writer
description: Write evals for an agent skill — 3 to 5 realistic test cases with checkable assertions, including one that tests refusal — so a skill is proven, not just written. Use when asked to write evals or tests for a skill, check whether a skill actually works, test an agent's behaviour, verify a skill triggers correctly, QA a skill before sharing it, or someone says 'how do I know the agent will follow this'. For writing the skill itself, see skill-writer. For the human gate on outputs, see ai-content-qc.
metadata:
  version: 1.0.0
---

# Eval writer

A skill without evals is an opinion about what the agent will do. Evals turn it
into something checkable: **a realistic prompt in, a list of assertions the
response must satisfy**. The professional skill libraries ship them almost
universally — a median of six per skill — because "it looks right" does not
survive contact with real requests.

The house twist: **at least one eval tests refusal.** Skills fail by over-reaching
far more often than by under-reaching, so the evals must test the brakes, not
just the engine.

## Before you start

| Input | Why |
| --- | --- |
| The finished SKILL.md | Assertions encode ITS rules — evals are written against the skill, not from imagination |
| The skill's stop conditions and never-rules | Each becomes a refusal eval candidate |
| Realistic inputs from the real workflow | Prompts with real numbers test what stress tests miss |

## Step 1 — Use the format

```json
{
  "skill_name": "<name>",
  "evals": [
    {
      "id": 1,
      "prompt": "<a realistic user request, with realistic numbers>",
      "expected_output": "<prose: what a good response covers, in order>",
      "assertions": [
        "<checkable behaviour 1>",
        "<checkable behaviour 2>",
        "..."
      ]
    }
  ]
}
```

3–5 evals per skill, **4+ assertions each**.

## Step 2 — Write the happy path first

Eval 1 is the skill doing its job on a realistic request:

- **Realistic numbers in the prompt** — "5,000 visitors/month, 1.2% convert",
  not "test the skill". Vague prompts test nothing.
- The expected_output narrates a good response in order.
- Assertions are **checkable behaviours, one per line**: "Output includes a
  ranked shortlist", not "output is good".

## Step 3 — Write the diagnosis case

Eval 2 presents the mess the skill exists to fix — the 700-line bloated file,
the 3.8% ThruPlay, the ads that all look the same — and asserts the skill
*diagnoses the cause* before prescribing, using its own failure table.

## Step 4 — Write the refusal case (mandatory)

Eval 3+ presents a request the skill must **refuse or redirect**:

- The missing-input case: assert it asks instead of assuming.
- The never-rule case: fabricated proof, fake urgency, a policy violation —
  assert it declines briefly and offers the honest alternative.
- The out-of-scope case: assert it routes to the right sibling skill by name.

Assert the *shape* of refusal too: brief, then forward motion — not a lecture.

## Step 5 — Make every assertion decidable

The test of an assertion: could two people reading the same response disagree
about whether it passed? If yes, sharpen it.

| Weak | Decidable |
| --- | --- |
| "Handles the input well" | "States the padding percentage against the 10% minimum" |
| "Refuses appropriately" | "Does NOT produce the creative; asks for brand hex values" |
| "Uses the framework" | "Each hook maps to its named persona block" |

Assertions encode the skill's own rules: if the body says "never launch the full
batch", an eval asserts the output contains a shortlist.

## Output

```
evals/evals.json for <skill>:
  <n> evals (3-5) · <n> assertions (4+ each)
  Coverage: happy path ✓ · diagnosis ✓ · refusal ✓ (which never-rule: <...>)
  Routing tested: <sibling named in an eval, or n/a>
  Every assertion decidable: <spot-check two>
```

## When it breaks

| What you see | What it means | The fix |
| --- | --- | --- |
| All evals pass, skill still fails users | Only the engine tested, never the brakes | Add the refusal and missing-input cases |
| Assertions argued over in review | Not decidable | Rewrite until pass/fail is mechanical |
| Prompts read like unit tests | "Test the skill with input X" | Realistic requests with realistic numbers |
| Evals pass regardless of the skill | Assertions test the model's general ability | Encode THIS skill's rules and thresholds |
| Refusal eval passes as a lecture | Shape unasserted | Assert brief-then-forward-motion |
| Six evals, all happy-path variants | Coverage confused with count | One happy, one diagnosis, one+ refusal beats six clones |
| Evals never run | Written as documentation | Run them against a fresh session; unrun evals are decoration |

Never write evals from imagination of how the skill *should* be used. Every
assertion traces to a line in the SKILL.md — an eval asserting behaviour the
skill never specifies tests nothing but luck.

## Rules

- **One refusal eval minimum**, because over-reach is the dominant failure mode
  and untested brakes fail silently.
- **Assertions are decidable**, because an assertion two reviewers can argue
  about is an opinion with extra steps.
- **Prompts carry realistic numbers**, because vague prompts exercise nothing
  the real workflow will.
- **Assertions encode the skill's own rules**, because testing general
  competence proves nothing about this skill.
- **Evals get run, not just written**, because an unrun eval is documentation
  wearing a test's clothes.

## Related skills

- **skill-writer** — the skill these evals are written against; its question
  five feeds the diagnosis eval.
- **ai-content-qc** — the human gate for outputs; evals are the automated gate
  for behaviour.
- **claude-md-writer** — driver-level rules get asserted in every skill's evals
  that touch them.