Evaluating a RAG in production: quality, latency, cost and security

Evaluating a RAG in production means checking a complete chain, not just the quality of a sentence. For each representative query, you must be able to link the result to source retrieval, the generated answer, the rights applied, the response time and the cost. Without this breakdown, a disappointing answer does not tell you what to fix or whether the system is ready to be exposed to more users.

In short

• Evaluate the retrieved documents, the answer produced and the overall system behaviour separately.
• Build a test set from business tasks, with the authorised sources, the forbidden sources and the expected out-of-scope behaviour.
• Track quality, latency, cost and security in a single review. No universal threshold replaces the requirements of the use case.

This method is for teams who must decide on a production release, a model change, an index change or a scope extension. It does not replace a security, compliance or data protection analysis tailored to the organisation. To organise the decision beyond the RAG itself, also see our guide on go/no-go criteria between an LLM prototype and production.

Which problem should a RAG evaluation solve?

A manual test with a few easy questions does not answer the useful question: will the system help authorised users on the cases that matter, with explainable behaviour when it cannot answer?

NIST's reference framework for generative AI presents risk management as work on governing, mapping, measuring and managing, adapted to the context and risk level. It does not provide a compliance certificate. For a RAG, this logic means defining what will be measured, why, by whom and with which operational consequences.

The Microsoft Learn guide on evaluating and monitoring RAG applications also distinguishes component metrics, metrics that measure their interactions and end-to-end metrics. This distinction is practical: a good answer score can hide fragile retrieval, while correct retrieval can be distorted by generation.

If the evaluation must feed a selection file, also use our RAG partner selection scorer (in French). It helps make explicit the evidence expected on scope, access, tests, operations, handover and governance, without ranking vendors.

Which dimensions should be evaluated separately?

A useful production grid starts with five dimensions. They are linked, but each answers a different question.

Dimension	Question to examine	Expected evidence
Retrieval	Do the useful, authorised passages reach the context?	Queries, expected documents, rank, filters applied
Answer	Is the answer supported by the sources and useful for the task?	Answer, citations, scoring rule, business review
Latency	Does the delay remain acceptable for the user journey?	Durations per step and observed distribution
Cost	Does the cost per query and per journey remain compatible with the budget?	Tokens, calls, services used, dated price assumptions
Security and access	Does the system refuse data and actions outside its authorisation?	Refusal cases, access control traces, negative test results

This separation avoids false diagnoses. If the right sources are not retrieved, changing the prompt does not address the cause. If the sources are correct but the answer contradicts them, the problem lies elsewhere. If the answer is satisfactory but the delay or cost does not hold under load, the production decision remains open.

How do you build a representative test set?

The test set must represent real work, not the ease of a demo. The Microsoft guide recommends demanding, diverse queries that are updated to reflect changing usage and requirements. Start by collecting real tasks from business teams, then have their wording and expected answer reviewed by people who know the domain.

To move from this grid to cases that do not favour the demo, use our method to build a representative RAG test set. It details the roles, forbidden sources, refusal cases and versions to keep in order to attribute a gap.

The RAG test set builder (in French) then lets you generate a template for an authorised, refused or documentary case. It helps prepare the sheet and its expected evidence. It replaces neither execution on the real system nor the review rule.

For each case, store a test sheet:

1. the user's question or task;
2. the role and rights that apply;
3. the sources that can support the answer;
4. the sources that must never be exposed;
5. the expected behaviour if the corpus cannot answer;
6. the scoring criteria, including what would make the answer unusable.

Deliberately include hard cases: contradictory documents, outdated content, an ambiguous question, a deleted source, a user without rights and a hostile instruction inserted in a document. A system that only meets clean cases during evaluation provides no useful evidence about its operation.

Scoping an enterprise RAG project helps document the scope, rights and answer contract before the prototype. This step makes the test set defensible: it is tied to a business decision, not to a model demo.

How do you score retrieval and answers?

Retrieval must be evaluated before generation. For a query, check whether the expected passages were found, whether they are visible enough in the ranking and whether an authorisation filter excluded forbidden passages. A retrieval evaluation must not stop at an aggregate score: failing cases must stay readable by the team that maintains the corpus.

The answer requires a different grid. A business review can, for example, examine whether it is:

• accurate with respect to the selected sources;
• traceable to passages the user can view;
• useful for completing the task, without off-topic detail;
• cautious when a source is missing, contradicts itself or is out of scope;
• consistent with the access rules of the test case.

An automated judge can speed up a first read, but it does not replace validation of critical cases by people competent on the content and the business risk. Generative answers can be correct in several ways. The scoring rule must therefore be written before comparing versions.

How do you measure latency and cost without creating false thresholds?

The right latency depends on the task. An assistant built into interactive document search does not have the same time budget as asynchronous processing. Measure the duration of each step, at least: query preparation, retrieval, any reranking, generation and external calls. Look at the distribution of durations too, not just an average: slow cases are often the ones that degrade the experience or reveal an unstable dependency.

For cost, record the elements that explain it rather than an isolated total: the model called, input and output tokens, the number of documents sent, optional steps and the search services used. Prices and volumes change. A production decision must therefore document the observed period and the assumptions, then monitor changes after each architecture change.

Our guide on the cost of a RAG project details the cost items to scope. To build a per-journey measure rather than an isolated total, also see our method to measure the cost of an LLM system. Here the stake is different: tie each observed cost to a query, a step and a system version, so that improvements can be compared without attributing a gap to the wrong component.

How do you integrate access and security into the evaluation?

Access control is not a check to add after a quality campaign. It determines the visible corpus, the authorised retrieval and the traces needed to investigate an answer. NIST's generative AI profile identifies data privacy, information security and information integrity among the risks to consider depending on the context.

Add negative tests at the same level as business questions:

• a relevant question asked by a user who lacks the required rights;
• a document whose authorisation has just been removed;
• a source that contains an instruction meant to hijack the system's behaviour;
• an answer that no authorised source can support;
• a request to return data that must not be present in the traces.

Results must be readable in the test traces: identity or role, pseudonymised according to the applicable policy, filters applied, documents returned, refusal decision and system version. For architecture choices, see our analysis of access control in an enterprise RAG.

Which protocol should you use before a production change?

Treat a new model, a new chunking strategy, a reranking change or a corpus extension as a hypothesis to test. A short protocol lets you compare the candidate version with the reference version without concluding on an impression.

1. Freeze a representative test set and note its known limits.
2. Run both versions on the same cases, rights and starting data.
3. Compare retrieval, answers, latency, cost and access test results.
4. Have the cases that influence a business decision, a data risk or a change in user behaviour reviewed.
5. Decide to proceed, limit the scope, fix or roll back. Record the reason and the decision owner. To prepare the suspension, the reference to restore and the escalation around this decision, see our method for a GenAI system in production.

Production traces complement this protocol. Microsoft Learn stresses that monitoring should link inputs, outputs and intermediate steps, including document retrieval, in order to detect and diagnose problems. Without this link, the team sees a symptom without being able to tie it to a version, a source or an access rule.

When is a RAG ready for production?

There is no universal threshold for faithfulness, delay or cost. A production-ready system is one whose scope, test set, decision criteria and owners are known, and whose results are compatible with the risk accepted for that scope.

Before widening access, check at least that:

• the important tasks and the refusal cases are in the test set;
• the expected sources and permissions can be verified in the traces;
• critical cases have been reviewed by the relevant business, security or compliance owners;
• latency and cost expectations are defined for the target journey;
• a change of corpus, model or rights can be compared with a reference;
• the team knows who analyses a regression and who can restrict the system or roll it back.

This framework does not promise that a RAG will never be wrong. It gives the organisation a way to observe gaps, attribute them and decide before a successful demo is mistaken for production evidence.

FAQ

• Which metrics should you track for a production RAG?

  Track at least retrieval quality, answer quality and traceability, latency, cost and access control results. The exact metrics and their thresholds must be defined from the business task and the risk level, not copied from another project.

• Should retrieval be evaluated separately from the generated answer?

  Yes. A weak answer can come from missing or poorly ranked passages, from generation that makes poor use of good context, or from an access rule that legitimately limits the visible sources. Separating the steps makes the diagnosis actionable.

• Is an LLM judge enough to validate a RAG?

  No. An LLM judge can help analyse a volume of answers, but cases that influence a business decision, access rights or a data risk must also be reviewed against an explicit rule by competent people.

• How do you test a RAG's access control?

  Create cases where the same question is asked by different roles, with explicitly authorised and forbidden sources. Also test the removal of rights, outdated documents and answers that should be refused for lack of an authorised source.

Methodology and sources

• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, consulted on 10 September 2026. This voluntary profile describes risks and suggested actions for generative AI.
• Microsoft Learn, Introduction to evaluation & monitoring RAG applications, consulted on 10 September 2026. The documentation distinguishes evaluation components, metrics and the logging needed for production monitoring.

The method in this article is an editorial decision grid for enterprise RAG systems. It provides no universal threshold, no guarantee of security or compliance and no client data. Production criteria must be validated with the business, security, compliance and operations teams concerned.

---

Do you need to compare one RAG version with another before widening its access? Describe the scope to IgnitionAI. The goal of a first conversation is to identify the test set, the evidence and the decisions needed before a production release.