RAG evaluation: building a representative test set without skewing the decision

A representative RAG test set does not try to confirm that the system can answer a few well-chosen questions. It must reproduce the tasks, roles, authorised sources, excluded sources and uncertain situations of the scope you are about to open. Each case sheet must make it possible to explain a regression, then decide whether a version can move forward, must be restricted or must be fixed.

In short

• Build cases from business tasks and the corpus actually concerned, not only from the model's capabilities.
• Also describe the expected refusals: unauthorised user, withdrawn source, contradictory document or no usable evidence.
• Freeze the version of the corpus, the rights and the review rule before comparing two versions. Otherwise the observed gap cannot be attributed.

This method helps a team prepare an evaluation decision. It provides no universal threshold and no guarantee of quality, security or compliance. Criteria and owners must be defined for the use case concerned.

Why does a demo set skew the decision?

A demo set often contains the easiest questions, the cleanest documents and a single user role. It can show a technical capability without representing the situation the team will have to operate.

In its work on test collections, TREC describes the three elements of an information retrieval evaluation infrastructure: documents, topics or tasks, and relevance judgements. For an enterprise RAG, this principle must be complemented by access rules and the expected behaviour when no authorised evidence can support an answer.

NIST's generative AI profile recommends adapting evaluation to the system's context and risks. It does not turn a series of successful questions into a deployment authorisation. The point is to make visible what the system handles well, what it correctly refuses and what remains out of scope.

What should a RAG test set represent?

Representativeness is not a matter of volume alone. It comes from covering the situations that matter for the decision. Before writing cases, define the scope: task, users, corpus, rights, the consequence of a wrong answer and the expected out-of-scope behaviour.

A coverage matrix makes this preparation readable:

Dimension	Question to cover	Elements to keep in the case
Business task	Which decision, search or action must the user complete?	Realistic request, useful result and criterion for uselessness
Role and rights	Who asks the question and which sources may they consult?	Role, required attributes, authorised and forbidden sources
Corpus	Which documents represent real cases?	Identifiers, version, owner, freshness and exclusions
Difficulty	Which cases reveal a limit rather than a demo?	Ambiguity, contradiction, outdated document or missing source
Expected answer	What should the system do with the available evidence?	Expected elements, citations, abstention or escalation
Review	How does a person qualify the result?	Scoring rule, review owner and reason for the gap

Scoping an enterprise RAG project lets you gather this scope before choosing a metric. Without this step, a test set mostly measures what the team was able to collect, not what decides whether the system opens.

How do you write a usable case sheet?

A sheet must be readable without reconstructing the context from memory. Use the same format for every case:

1. The user task. Write the question as it is asked at work, with the essential context but without superfluous data.
2. The test role. State the simulated identity or role and the attributes that govern access.
3. The reference corpus. List the documents or passages that can support the answer, with their version. Also list the sources the system must not expose.
4. The answer contract. Specify what must be present, what must be cited and the situations where the system must report a limit rather than infer.
5. The review rule. Describe the reasons for success, failure and escalation. A vague judgement like "correct answer" does not let you compare two versions.
6. The traces needed. Keep what will let you analyse the gap: system version, retrieval, filters, context sent, answer and review decision, according to the applicable data rules.

This sheet distinguishes an accurate answer obtained from an unauthorised source from a useful, grounded answer accessible to the role concerned. It is also what lets you investigate a regression after an update to the corpus, model, prompt or retrieval.

To turn this method into an exportable sheet, use our RAG test set builder (in French). It prepares a template for an authorised, refused or documentary case without asking for any business data. The real sources, roles, versions and criteria still have to be filled in within the team's test file.

Which hard cases should be included from the start?

Hard cases are not exceptions to add after the first success. They are needed to avoid drawing conclusions about a system that only meets its best scenario. The exact choice depends on the business, but the matrix should include situations such as:

• the same question asked by two roles with different rights;
• an applicable but old, superseded or contradictory document;
• an ambiguous question that requires clarification or a limited answer;
• a request whose evidence is missing from the authorised corpus;
• a source whose access was removed or whose deletion must be taken into account;
• an indexed document that contains an instruction irrelevant to the task;
• a request that must be refused rather than answered from out-of-scope data.

These scenarios do not presume that an architecture is faulty. They give a way to check the expected behaviour when one element of the chain leaves the ideal scenario. To prepare the authorisation rules that support them, also see our guide on access control architecture choices in a RAG.

How do you keep the test set from drifting with the system?

A test set becomes of little use if the corpus, the rights or the system change without the case saying so. Freeze a baseline before each comparison: version of the index or corpus, access rules, retrieval configuration, model, prompt and execution date.

Microsoft's documentation on evaluating and monitoring RAG applications points out that a system's components interact and that a change in the data can affect the retrieved chunks and overall quality. This dependency makes traceability more useful than an isolated score: the team must be able to see whether the difference comes from the corpus, retrieval, generation or a control rule.

When a case is no longer representative, do not delete it silently. Record the reason: task abandoned, source withdrawn, access policy change or scope change. Keep the cases that revealed a real error as regression cases, unless their context is no longer legally or technically usable.

How do you tie the test set to a version decision?

Compare a candidate version with a reference on the same scope. A decision review can follow five steps:

1. Freeze the cases, roles, sources and review rules for the comparison.
2. Run the reference and the candidate version, keeping the necessary traces.
3. Examine retrieval, answers, refusals, latency and cost separately.
4. Have the gaps reviewed that affect a critical task, sensitive data or an expected out-of-scope behaviour.
5. Record a decision: proceed, restrict the scope, fix or revert to the reference, with the reason and the owner.

Our guide to evaluating a RAG in production details the dimensions to separate during this review. This page focuses on the quality of the starting material: if the cases do not represent the decision, even the most rigorous metrics remain hard to interpret.

When is the test set representative enough?

There is no universal number of cases that makes a test set representative. The decision question is more concrete: are the priority tasks, distinct roles, sensitive sources, expected refusals and situations that could block the opening explicitly covered?

A test set can be incomplete and still useful, as long as its limits are written down. This transparency is better than a general conclusion drawn from a clean demo. Before widening the scope, also prepare the deployment decision with the go/no-go criteria for an LLM system.

FAQ

• What is the difference between a RAG test set and a public benchmark?

  A public benchmark can help understand a method or compare approaches within its own frame. An enterprise RAG test set must above all represent the tasks, documents, rights and expected behaviours of the scope actually opened. It cannot be replaced by a score obtained on an external corpus.

• Should you use real questions in a RAG test set?

  Yes, when data rules allow it. Real tasks, or tasks reviewed by people who know the business, reduce the risk of testing artificial wording. They must be handled under the applicable confidentiality and retention rules.

• How do you test a RAG when no answer should be given?

  Create a case where the source is missing, insufficient or unauthorised, then describe the expected limitation, refusal or escalation answer. Also check that the traces identify the control decision without exposing more data.

• Should you change the test set after a corpus change?

  Yes. Identify the affected cases, keep the corpus version and state the reason for the change. Compare the change with a reference on the cases that remain relevant before widening the new scope.

Methodology and sources

• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, consulted on 17 September 2026. The profile describes risk management to adapt to the context of generative AI systems.
• NIST, The 34th Text REtrieval Conference, March 2026, consulted on 17 September 2026. The publication describes the information retrieval evaluation infrastructure, including document collections, tasks and relevance judgements.
• Microsoft Learn, Introduction to evaluation & monitoring RAG applications, consulted on 17 September 2026. The documentation presents the relationships between data, RAG components, evaluation and monitoring.

The method proposed is a design grid for evaluating enterprise RAG systems. It contains no client results, no scores, no timelines and no promise of security, compliance or performance. Test cases, retention rules and decision criteria must be validated by the business, security, data and operations functions concerned.

---

Do you need to compare a RAG with a new version without confusing a demo with production evidence? Describe your scope to IgnitionAI. A first conversation can help organise the tasks, refusal cases and evidence needed for a production decision.