Chat UIs train a bad habit: you skim a fluent answer and feel complete. Agents make that habit expensive. Fluency is not a ship criterion.
Evaluation is how you answer: did this run satisfy the spec, with evidence? If you cannot say yes or no in a minute, you do not have a workflow. You have a vibe.
Write tests before you run
Steal this shape into every compiled spec:
| # | Test | How you check | Pass? |
|---|---|---|---|
| 1 | Deliverable format | File exists, right type | |
| 2 | Constraints | Must-not list grepped / read | |
| 3 | Sources | Claims map to the pack | |
| 4 | Length / structure | Word count, headings, table columns | |
| 5 | Side effects | Diff is only where allowed |
You will skip tests when you are tired. That is why they belong in the spec, not in your head.
Four kinds of evidence
- Mechanical — tests, linters, link checkers, “price is a number,” “CSV has 12 rows.”
- Trace — tool calls, URLs fetched, commands run. No trace, no trust on agents.
- Diff — what changed in git or the doc. Read it.
- Human — taste, legal, brand, “would I send this to a customer?”
Mechanical tests are cheap. They will not catch a politely wrong strategy. Humans catch that — if they look. Human in the loop.
“Looks good” is not a test
The model will agree with you that it looks good. Of course it will.
Replace:
- “Make it better” → “Fail test 3: the table is missing the TLS column.”
- “Are you sure?” → “Quote the sentence in SOURCES.md that supports paragraph 2.”
- “Ship it” → checklist complete and you opened the artifact, not the chat summary.
If the agent summarizes its own work, open the artifact anyway. Summaries are where files fail to save.
Eval for each mode
| Mode | Minimum eval |
|---|---|
| Chat | Did it answer this question? Close the tab. |
| Workflow | Spec checklist + you skim the artifact |
| Agent | Trace + diff + checklist + retry budget |
Research jobs need source-level eval: every non-obvious claim has a URL or file. Pattern: research.
Money jobs need policy eval: amount, destination, merchant allowlist. That is not this page’s job — Pay By Prompt.
Retry vs rewrite the spec
When a run fails:
- Retry if the test is still right and the output missed it
- Change the spec if you realized the test was wrong
- Stop if you are about to lower the bar so the agent can “win”
Lowering the bar to match the output is how slop farms get built. No eval.
Sampling
You cannot read every token of a 40-file agent run. Sample on purpose:
- First and last commits
- Any file matching
prod,bill,secret - Random 10% of the rest
- Every failing mechanical test, none of the passing ones
Write the sample rule down so you do not “trust it this once.”
Done when
A run is done when all are true:
- Acceptance tests pass or are explicitly waived in writing
- Trace exists for tool-using jobs
- Retry budget not exceeded without a human note
- You (or a named reviewer) opened the artifact
“The agent said it finished” is not in that list.