The stall
Teams know retrieval quality should be measured, know that requires labelled data, estimate the labelling effort, and postpone. Months later changes are still being shipped on impression, and nobody can say whether last quarter's work helped.
The way out is to accept that the first baseline does not need to be authoritative. It needs to make changes comparable.
Start from real queries
Sample actual user queries from logs — a few dozen is enough to begin, weighted toward frequency but including the strange tail, since that is where retrieval usually fails.
Invented queries reflect assumptions about usage. Real ones reflect usage, which is consistently less tidy and more informative.
Judge what came back
For each query, capture the retrieved set and judge relevance. Doing fifty by hand takes an afternoon and teaches more about the corpus than any dashboard. A capable model can extend the labelling afterwards, with a human-checked sample to establish whether its judgements track yours.
The output is a baseline. It is imperfect and it is enough — from here, a change can be shown to have helped or hurt, which was the actual goal.
Cheap signals that need no labels
Several useful measures require no ground truth at all. How often does retrieval return nothing? How often is the top result identical across very different queries — a classic sign of a collapsed embedding space? How often does the generated answer cite nothing retrieved, indicating the model fell back on parametric knowledge?
That last one is particularly valuable, because it detects RAG that is silently not doing its job while still producing plausible answers.
Grow the set from incidents
Every reported bad answer becomes a fixture. This is the cheapest possible growth mechanism, it directs coverage at cases that demonstrably matter, and it means the suite gets stronger precisely where the system has already proven weak.
After a few months of that discipline, you have the golden set you postponed building — assembled as a side effect of handling problems you had to handle anyway.