LLM evaluation becomes misleading when one number is asked to summarize an entire system. A security assistant can be fluent but ungrounded, correct but too slow, safe but unhelpful, or accurate on common cases while failing on rare high-impact incidents.
The answer is a measurement stack whose dimensions remain visible.
1. Task quality
Start with the task itself. Depending on the workflow, measure extraction F1, classification quality, answer correctness, ranking quality or human-rated usefulness. Match the metric to the decision the system supports.
2. Evidence and grounding
For RAG, measure retrieval recall and precision, citation correctness, claim support and answer completeness. A response can be relevant to the question while still making an unsupported claim.
For claims and evidence , report at least:
If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.Aggregate the result carefully and keep uncertain cases visible rather than forcing them into a binary score.
3. Safety and robustness
Security evaluation should include refusal quality, prompt-injection resistance, sensitive-data exposure, unsafe-action rate and behavior under contradictory or manipulated context. Red-team cases are useful only when the scenario and success criterion are defined clearly.
4. Calibration and abstention
If a model emits confidence, test whether it correlates with correctness. A well-calibrated assistant should know when to ask for more evidence or escalate. Abstention is not failure when the cost of an unsupported action is high.
5. Operations and human use
Measure latency, token and retrieval cost, memory footprint, availability, analyst correction time and the percentage of recommendations accepted without modification. These metrics reveal whether a model is useful in the actual loop rather than only in a benchmark.
6. Report the vector, not a vanity score
A dashboard can show a vector:
If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.for task quality, grounding, safety, calibration, latency and human utility. The components should be reported with the dataset, policy and model version that produced them.
The most credible evaluation makes trade-offs explicit. It says what improved, what regressed and which failures remain possible.
Define denominators before measuring
Two rates with the same name can measure different things. For retrieval, specify whether relevance concerns documents, chunks or independent evidence. For citations, separate claim coverage from correctness of emitted citations. An answer with one correctly cited claim can still omit most requested information.
| Measure | Denominator to declare |
|---|---|
| Retrieval recall | Relevant evidence in the annotated set |
| Citation coverage | Verifiable claims in the answer |
| Unsafe action rate | Evaluated cases or action attempts, per protocol |
| Useful success | Evaluable legitimate tasks |
| Abstention | Presented queries, separating answerable cases |
Uncertainty and version comparisons
Compare versions on the same cases to study paired differences. When queries come from one incident, estimate uncertainty while respecting that grouping. Do not treat them as independent observations. Retain individual failures and accompany averages with distributions by language, task and risk.
For stochastic outputs, repeat part of the set to measure run-to-run variation. Zero observed failures in a small sample does not establish a zero failure probability. State sample size and inference limits.
Automated judges and calibration
RAGChecker supports fine-grained diagnostics, but any automated judge requires comparison with human domain labels. Check length, order and style biases. Verbal LLM confidence is not a calibrated probability either: validate confidence against correctness before setting thresholds.
The preprint Safety, or Just Capability? reinforces specifying the behavior a benchmark measures. For case construction, see golden sets; for tool failures, see the agent-security protocol.
