Evaluation harnesses wired into CI.

Why this is the highest-leverage thing you are not doing

Every generative feature has a moment where it silently gets worse. A prompt is edited for one case and regresses three others. A provider updates a model and behaviour shifts underneath you. Without evals in CI, the detection mechanism is a customer complaint, and the diagnosis is archaeology.

Evaluation in CI means the same dataset and the same evaluators run against every proposed change, so degradation is caught at the pull request rather than in production.

Pick the property that matters first

Broad subjective quality scoring is appealing and usually premature. The first eval should target the single property whose violation would destroy trust in the feature — groundedness for anything that reports numbers, tenant-scoping for anything multi-tenant, format compliance for anything another system consumes.

Those properties are mechanically checkable, which means they can gate a build. Subjective quality can be tracked alongside; it should not be what blocks a release, because it cannot be adjudicated automatically.

Eval noise is real

LLM evals are noisy, and a single aggregate score is a poor release gate. Repeated runs, attention to confidence, and weighting individual severe failures over small average movements all matter. A test suite that fails randomly gets disabled within two weeks, so the harness has to be trustworthy before it can be authoritative.