The shape of the problem
Prompts are edited to fix a specific complaint. The edit works for that case. It also changes behaviour on cases nobody re-checked, because a prompt is global state and every instruction competes with every other instruction.
Traditional tests do not catch this, because the output is not deterministic and the failure is a matter of degree rather than an exception. So the change ships, and the regression is discovered weeks later by someone who assumed the feature had always been mediocre.
What an eval suite actually is
A fixture set — inputs paired with what should be true about the output — and evaluators that check those properties. Run the same fixtures and evaluators against every proposed change, and degradation becomes visible at the pull request rather than in production.
The fixtures should come from real traffic rather than imagination. Cases you invent reflect what you think users do; cases from logs reflect what they actually do, which is consistently stranger.
Choose the gating property before the quality metric
The most common mistake is starting with a general quality score. It feels thorough and it cannot gate a build, because a small movement in an aggregate score is not adjudicable — nobody can say whether 0.82 down from 0.84 should block a release.
Start instead with the single property whose violation would destroy trust in the feature. Groundedness for anything that reports figures. Tenant scoping for anything multi-tenant. Format compliance for anything another system parses. Refusal behaviour for anything with a safety requirement.
These are mechanically checkable and binary, which is exactly what a release gate needs. Subjective quality can be tracked alongside as a trend without being the thing that blocks the build.
Handling noise honestly
LLM evals are noisy. The same input can pass on one run and fail on the next, and a suite that fails randomly gets disabled within a fortnight — after which you have the cost of maintaining evals and none of the protection.
Practical mitigations: run the ambiguous checks more than once and treat the distribution rather than the single result; weight individual severe failures more heavily than small movements in an average; and reserve hard build failures for the deterministic properties, letting the fuzzier signals inform rather than block.
The suite has to be trustworthy before it can be authoritative. An eval nobody believes is worse than no eval, because it consumes the political capital you would need to introduce a good one.
The model-change case
The highest-value moment for an eval suite is a model swap — whether you initiate it to reduce cost, or a provider does it on your behalf. This is a behavioural change to your product introduced by something outside your repository.
With a suite in place, a swap becomes a measurable decision: run the fixtures against both, compare, decide. Without one, the choice is made on price and discovered on quality.
What good looks like after a month
A modest fixture set drawn from real traffic. Two or three mechanical evaluators covering the properties that matter. CI integration with an explicit pass/fail policy. A documented process for adding a fixture whenever a bug is found — so each incident permanently increases coverage.
That is a small amount of work, and it converts a class of silent failures into a class of visible ones. It is consistently the highest-leverage engineering available on a generative feature, and consistently the last thing teams do.