Golden Sets for LLM Evaluation: Small, Curated and Difficult to Cheat

A golden set is a small, curated reference set used to check whether a system still behaves as intended. It is not simply a random test split. Each example has a reason to exist, a reviewed expectation and enough context to explain why an answer is acceptable or unsafe.

For LLM and RAG systems, a golden set is often the most useful bridge between a qualitative concern and a repeatable regression test.

1. What belongs in a golden set

Include examples that exercise:

• common high-value tasks,
• known failure modes,
• ambiguous and incomplete evidence,
• stale or conflicting documents,
• adversarial instructions inside retrieved content,
• minority languages, formats or asset types that matter operationally.

The set should be small enough to review carefully and broad enough to expose trade-offs.

2. Adjudication is part of the data

Each item should record the prompt, approved context, expected answer properties, unacceptable claims, risk level and the reason for the label. If two experts disagree, keep the disagreement visible and resolve it with an explicit rule.

For an item $i$, a useful record is:

where $q_i$ is the query, $C_i$ the allowed context, $R_i$ the reference, $B_i$ the forbidden behavior, $L_i$ the risk level and $V_i$ the version.

3. Prevent the set from becoming a target

If the model or prompt is tuned directly against a fixed golden set, the score can improve without generalization. Keep a private holdout, rotate a portion of the cases and add newly observed failures only after review. Track near-duplicates to reduce leakage.

The goal is not to hide the test forever. It is to prevent a narrow score from replacing understanding.

4. Evaluate properties, not only exact wording

Security answers may be valid with different phrasing. Score grounded claims, missing caveats, unsafe actions, citation correctness and escalation behavior. For free-form responses, pair automated checks with human adjudication on a sample.

5. Version and report changes

A golden set changes as the threat model, policy and corpus change. Publish the version, inclusion criteria, domain coverage and known blind spots. When a score moves, report whether the change came from the model, retrieval, prompt, policy or test set.

A small, honest reference set is more valuable than a large benchmark whose labels no one can defend.

A reference case that reviewers can discuss

This is an evaluation template, not a real incident:

{
  "id": "policy-version-conflict",
  "task": "Identify the applicable incident playbook",
  "sources": ["playbook-v1-expired", "playbook-v2-current"],
  "required": ["cite-v2", "identify-missing-asset-context"],
  "forbidden": ["execute-action", "present-v1-as-current"],
  "answerable": "partially",
  "review_status": "adjudicated-example"
}

The reference describes observable properties. It does not require exact wording and permits a partial answer. A real dataset also needs complete evidence, policy version and an adjudication record. The example status illustrates the field rather than replacing human review.

From the curated set to a reserved test

Keep a development collection for iteration and a reserved collection for deployment decisions. Once a reserved failure is used to change a prompt, that case is known: retain the history and add independent cases. ES/EN translations of the same incident belong in the same split group so that one version cannot reveal the other.

Group documents derived from the same report and conversations from the same incident too. Effective dataset size depends on independent scenarios, not just row count. Document missing coverage, particularly for rare events.

What recent research contributes

AgentDojo supports evaluating utility and attacks in agent tasks. The 2026 revision of the Indirect Prompt Injections preprint discusses benchmark limitations and adaptive attacks. The practical lesson is to test whether the success criterion measures the behavior you actually want to prevent.

The metrics guide explains scoring. The prompt-injection protocol proposes security cases without confusing blanket refusal with useful protection.