The agent that invented an email address
Before FIXER went live, an eval run passed 17 of 18 cases. Reading the replies showed the score was wrong about what mattered: one answer gave a recruiter an email address that does not exist.
WHAT HAPPENED
FIXER, the terminal agent, ran on Claude Haiku 4.5 to keep the site inside a $3 monthly budget. The first eval run scored 17/18. Reading every reply turned up three defects the score never saw:
- An answer about the logistics platform told the visitor to email [email protected]. The real address has a 099 in it.
- An answer credited a feature to the wrong project.
- A Turkish answer contained a phrase that made no sense.
ROOT CAUSE
Two layers. The grader only checked each case for its own expected and forbidden phrases, so a wrong email in an unrelated answer was invisible to it. And the smaller model drifted on facts at the margins of a long profile.
A third finding changed the cost argument: Haiku 4.5 only caches prompts of 4,096 tokens or more. FIXER's prompt was about 3,300, so nothing was ever cached and every answer paid for the whole prompt. The cheap model was not actually cheaper.
FIX
- Global checks on every case: any email in a reply must be exactly the real one, and replies must not contain markdown links.
- Prompt tightened: stay literal, never attribute a feature to a different project, copy the email character for character.
- Model moved to Claude Sonnet 5.5, measured side by side on the same cases. With its prompt cached, it came out at about $0.004 per answer against Haiku's $0.0053.
VERIFICATION
19 of 19 cases passed after the change. The suite has grown since, and every run is published at /evals.
ALSO FOUND IN THE SAME REVIEW
The eval file held a private product name in plain text, inside a folder the site serves publicly. It never shipped: the check now reads the name from an environment variable.
LESSONS
- A passing score is not a review. Read the outputs.
- Put invariants in the grader. Anything that must never be wrong gets checked on every case.
- Check the cache minimum before picking a smaller model for cost.