Write the evals before you tune the prompt
Without evals, every prompt change is a guess that fixes one answer and quietly breaks three. A small eval set, built first, turns it into a test run.
Most AI features get tuned by vibes. Someone tries five questions, tweaks the prompt until those five look good, and ships. A week later a user hits the sixth question, and the fix for that one quietly breaks two of the original five. Nobody notices, because nobody runs the five again.
Evals are just tests for model behaviour. The trick is to write them before you start tuning, the same way acceptance criteria come before code.
Start with twenty real questions
You don't need a benchmark. You need a list of inputs your users will actually send, with a note on what a good answer looks like. Twenty is enough to start. Pull them from support tickets, sales calls, search logs or, before launch, from the people who will use the feature.
For each one, write down what must be true of the answer, not the exact wording:
- cites at least one source, and the source contains the claim
- refuses or says "I don't know" when the answer isn't in the data
- returns valid JSON that matches the schema
- never mentions another customer's data
- stays under the length limit
Check what you can with code
Many of those checks need no model at all. Schema validation (Zod or Pydantic), "does the cited chunk ID exist in what we retrieved", length limits, banned phrases and "did it call the right tool with the right arguments" are all plain assertions. They are fast, free and never disagree with themselves.
Use a model as the judge only for what code can't check, like "is this answer faithful to the retrieved text" or "does it actually address the question". Give the judge a short rubric and ask for a pass or fail with a reason, not a score out of ten. Spot-check its verdicts by hand early on; judges have blind spots too.
Run it on every change
The eval set earns its keep when it runs automatically: on every prompt edit, model upgrade, retrieval change or chunking tweak. A change ships when the pass rate holds or improves, and when nothing that used to pass now fails. That second condition matters more than the headline number.
Tools like Langfuse, Braintrust and LangSmith store the dataset, run experiments and keep the results side by side, but a script and a CSV is a perfectly good first version.
Feed it from production
Once the feature is live, trace every call: input, retrieved context, prompt version, output, latency and cost. When a user gets a bad answer, that trace becomes a new eval case. Over a few months the set grows from twenty guesses into a record of every way the feature has actually failed.
If you can't say how often your AI feature gives a good answer, you don't know whether last week's prompt change helped. Twenty eval cases written on day one are worth more than a hundred written after launch.
Scout, the chatbot on this site, is being built this way: an eval set of real visitor questions that has to pass before any prompt change goes out.
Got a demo that needs to survive real users?
Tell me what you have and what has to ship, on a call or by email with a link to the repo or the live app. I'll tell you honestly whether I can help, and what I'd look at first.