A golden set is a small, curated reference set used to check whether a system still behaves as intended. It is not simply a random test split. Each example has a reason to exist, a reviewed expectation and enough context to explain why an answer is acceptable or unsafe.
For LLM and RAG systems, a golden set is often the most useful bridge between a qualitative concern and a repeatable regression test.
1. What belongs in a golden set
Include examples that exercise:
- common high-value tasks,
- known failure modes,
- ambiguous and incomplete evidence,
- stale or conflicting documents,
- adversarial instructions inside retrieved content,
- minority languages, formats or asset types that matter operationally.
The set should be small enough to review carefully and broad enough to expose trade-offs.
2. Adjudication is part of the data
Each item should record the prompt, approved context, expected answer properties, unacceptable claims, risk level and the reason for the label. If two experts disagree, keep the disagreement visible and resolve it with an explicit rule.
For an item , a useful record is:
where is the query, the allowed context, the reference, the forbidden behavior, the risk level and the version.
3. Prevent the set from becoming a target
If the model or prompt is tuned directly against a fixed golden set, the score can improve without generalization. Keep a private holdout, rotate a portion of the cases and add newly observed failures only after review. Track near-duplicates to reduce leakage.
The goal is not to hide the test forever. It is to prevent a narrow score from replacing understanding.
4. Evaluate properties, not only exact wording
Security answers may be valid with different phrasing. Score grounded claims, missing caveats, unsafe actions, citation correctness and escalation behavior. For free-form responses, pair automated checks with human adjudication on a sample.
5. Version and report changes
A golden set changes as the threat model, policy and corpus change. Publish the version, inclusion criteria, domain coverage and known blind spots. When a score moves, report whether the change came from the model, retrieval, prompt, policy or test set.
A small, honest reference set is more valuable than a large benchmark whose labels no one can defend.
This note is an original synthesis of curated evaluation-set practice for LLM systems.
