Chatbot on your internal documents: 7 criteria to avoid buying a demo
Before buying a chatbot on your internal documents, check seven points: citations, access rights, refusals, isolation, hosting, connectors and full cost.
Transparency note. This article follows IgnitionAI's editorial policy. The criteria rely on the RAG guide of the French Directorate General for Enterprise (DGE) and on OWASP publications, consulted on 22 October 2026. IgnitionAI also publishes a RAG product, IgnitionRAG. This grid applies to it as to any other solution, and the article compares no vendor. It contains no statistic and no numerical estimate. See the Methodology and sources section at the end of the article.
A chatbot demo on internal documents almost always shows the same thing: a well-chosen question, a fluent answer, a prepared corpus. It rarely shows what decides a deployment. Does the answer cite the exact sentence? Does it respect the rights of the person asking? Can it say it does not know? Seven criteria take you from the demo to a decision. Each one is checked on your documents, with evidence the vendor can produce.
In short
- Require passage-level citations, verifiable in one click, for every answer.
- Check that access rights apply from the passage search onwards, before the model receives anything.
- Test questions with no answer: the system must say so instead of filling in from memory.
- Ask how spaces or organisations are isolated, and have it tested.
- Locate each layer (documents, model, ancillary services, logs) and the law that applies to each provider.
- List the connectors actually available, their rights and how deletions propagate.
- Work out the full cost over time: subscription or infrastructure, model usage, maintenance, evaluation.
Why is a demo not enough?
A demo is built to succeed. The corpus is clean, the questions are known, access rights are missing or simplified. The real system will face contradictory versions, scanned documents, users with different rights and questions whose answer is not in the corpus.
The DGE RAG guide puts it in its own way. It recommends objectively evaluating the performance of an installed system with dedicated tools, and notes that this evaluation weighs on employee adoption. It also reminds readers that RAG reduces hallucinations without eliminating them. The purchase should therefore rest on verifiable behaviour, and the ease of a demo is not one of them.
Criterion 1: do answers cite a verifiable passage?
A useful citation points to the document, its version and the precise passage that supports the answer. A file name alone does not let you check the claim.
In its RAG Security Cheat Sheet, OWASP asks for every answer to return the retrieved documents, the passages used and their provenance metadata. The DGE guide lists explainability among the topics to discuss with a provider: give clear explanations of the answers, including sources and references.
Evidence to request. Take ten questions whose answer you know. For each one, open the citation and check that the displayed passage says what the answer claims. Ask the vendor to cite the exact sentence.
Criterion 2: do access rights apply before retrieval?
An internal chatbot reads documents with varying rights: HR, finance, legal, management. The decisive question is when those rights are enforced.
OWASP is explicit. Access control metadata must accompany every vector chunk. The check must happen at retrieval time, since rights may have changed since indexing. The cheat sheet adds: do not rely on the language model to enforce access control, which must take place before content reaches the model.
Also ask which identity the connector uses to read the source. If the system reads with a broad technical account, everything that account sees becomes searchable by the members of the space. Our article on access control architecture choices in a RAG details the options.
Evidence to request. Two test accounts with different rights, the same question, and a document reserved for only one of them. The second account must see neither the answer nor the citation.
Criterion 3: can the system say "I don't know"?
A chatbot that always answers is a risk. When the corpus does not contain the answer, the expected behaviour is to say so.
OWASP applies a related logic to failures, under the name fail-closed design: if retrieval fails, the system must not answer from model memory alone but indicate that the knowledge base is unavailable. The same principle applies to a failed access control check: return nothing rather than a filtered subset.
Evidence to request. Add plausible questions whose answer is not in the documents to your test set. Count how often the system makes up an answer. Our method to build a representative test set helps choose these questions.
Criterion 4: how are spaces isolated?
As soon as several teams, subsidiaries or clients share one platform, leakage from one space to another becomes the main risk. OWASP ranks it in its Top 10 under LLM08:2025: in a multi-tenant environment sharing a vector database, embeddings from one group may be retrieved in response to another group's queries.
The countermeasures published by OWASP are concrete: separate namespaces or indices per tenant, filtering at query time, and cross-tenant test queries replayed regularly. A serious vendor should be able to describe its isolation model and accept that you test it.
Evidence to request. Two test spaces, a marked document in each, and questions that target the other space's document. Expected result: zero out-of-scope passages.
Criterion 5: where does the data live, and under which law?
"Hosted in France" tells you where the servers are. It leaves three questions open: the law that applies to each provider, where the model runs, and the ancillary services that process data (authentication, email, logs).
The DGE guide gives two criteria to assess data leakage risk: control of the infrastructure and control of the model. It presents on-premise hosting as the most controlled architecture, provided you have the skills to maintain it over time. It cites SecNumCloud-qualified cloud services as an alternative. ANSSI, the French national cybersecurity agency, states that this qualification aims to protect sensitive data against cybercriminal threats and the application of extraterritorial laws. It adds that the qualification does not prejudge the security of the services a customer builds on the qualified offer.
Evidence to request. The subprocessor list, with country and role. Where model calls are executed. The exact scope of any qualification announced.
Criterion 6: which connectors, with which rights and freshness?
Brochures often list dozens of integrations. What matters: the connectors actually available for your sources, the rights they require, the sync frequency and what happens when a document is deleted.
OWASP asks for least privilege on ingestion connectors: read access to specific folders, rather than admin access to an entire drive. The cheat sheet also asks for cascading deletion: removing a source document must remove its chunks, its embeddings and the cached answers. Our method to version, update and withdraw a source document describes the matching tests.
Evidence to request. Delete a test document in the source, wait for the sync, then ask a question that targeted it. It must no longer appear, neither in the answer nor in the citations.
Criterion 7: what is the full cost over time?
The subscription price is only one line. The DGE guide notes that costs depend heavily on usage volume. Outside SaaS, computing power must be added: GPU purchase and electricity for on-premise hosting, usage-based variable cost in the cloud. On maintenance, the guide notes that SaaS providers generally handle it within the subscription. With a custom solution, it falls to the integrator or to your teams.
Add two items that are often forgotten: continuous evaluation, which the DGE recommends to detect drift, and the time business teams spend reviewing answers and maintaining documents. Our article on the cost of an enterprise RAG project details the items.
Evidence to request. A quote that separates subscription or infrastructure, model usage, connectors, maintenance and evaluation, over the contract term.
The seven criteria in one table
| Criterion | Question to the vendor | Test to run on your documents |
|---|---|---|
| Citations | Does the answer point to the exact passage? | Ten known questions, citation opened each time |
| Access rights | Do rights apply before retrieval? | Two accounts, one restricted document |
| "I don't know" | What does the system do without a source? | Questions with no answer in the corpus |
| Isolation | How are spaces isolated? | Two spaces, crossed queries |
| Hosting | Where does each layer live, under which law? | Subprocessor list and model location |
| Connectors | Which rights, which sync, which deletion? | Delete a document, then ask about it |
| Full cost | What does the price include, over time? | Quote broken down by item |
Sources: DGE, RAG guide, November 2024; OWASP, RAG Security Cheat Sheet and LLM08:2025. Test wording is our method.
Decision grid
| Your situation | Recommended approach |
|---|---|
| Low-sensitivity corpus, one team, quick need | SaaS solution, checking criteria 1, 3 and 7 during a trial |
| Documents with mixed rights (HR, finance, legal) | Criteria 2 and 6 are blocking: no purchase without a two-account test and a deletion test |
| Several subsidiaries or clients on one platform | Criterion 4 is blocking: isolation test before each new space |
| Sensitive documents, sovereignty requirements | Criterion 5 is blocking: qualification and location of each layer checked with the CISO |
| Team able to run infrastructure over time | Compare SaaS and hosted solution on criterion 7, maintenance included |
Grid basis: DGE criteria (pp. 9, 14-15 and 19) and the OWASP countermeasures cited above. Ranking the criteria by situation is our reading.
Before comparing vendors, define the assisted task and the data scope. Our RAG project scoping method helps set those decisions.
FAQ
-
How many questions do you need to test a document chatbot?
There is no universal number. The test set must cover real tasks, questions with no answer in the corpus, cases with different rights and documents of varying quality. A small, well-built set teaches more than a large set of easy questions.
-
Can a chatbot that cites its sources still be wrong?
Yes. The DGE guide notes that RAG reduces hallucinations without eliminating them. A citation lets you check the answer. It does not guarantee the answer is right, which is why you should open the citation during the trial.
-
Should you require a SecNumCloud qualification?
It depends on the sensitivity of the documents and your obligations. ANSSI presents SecNumCloud as protection against cybercriminal threats and the application of extraterritorial laws. Check the exact scope of any announced qualification, layer by layer.
-
Can the language model filter forbidden documents by itself?
No. OWASP recommends enforcing access control before content reaches the model. A forbidden passage that reaches the model's context can end up in the answer.
Methodology and sources
Institutional sources
- DGE, Guide de la génération augmentée par récupération (RAG) (RAG guide, in French), November 2024, consulted on 22 October 2026. Pages used: 7 (summary sheet: costs depend heavily on usage volume), 9 (data leakage risk criteria), 14-15 (hosting modes), 19 (evaluation and maintenance), 20 (limits: persisting hallucinations), 21 (topics to discuss with a provider). The costs quoted by the DGE were assessed in June 2024 and are not reused here.
- ANSSI, Cloud: SecNumCloud qualification and security visa (in French), consulted on 22 October 2026.
Technical sources
- OWASP, RAG Security Cheat Sheet, consulted on 22 October 2026. Sections used: 4 (access control inheritance, deletion), 5 (source attribution), 6 (chunk isolation), 12 (testing), 13 (connectors), 14 (fail-closed design).
- OWASP, LLM08:2025 Vector and Embedding Weaknesses, consulted on 22 October 2026.
Interpretation
The seven criteria, the proposed tests and the decision grid are our reading of these sources. They are not quotes from the DGE, ANSSI or OWASP. This article contains no statistic and no numerical estimate.
Correction policy
If you spot a factual error or a source that has become outdated, report it to contact@ignitionai.fr. IgnitionAI's editorial policy provides for a review of each report and, for a substantial error, a visible correction note at the top of the article.
Last source review
22 October 2026.
Are you comparing chatbot solutions for your internal documents? Describe your context to IgnitionAI. A first conversation can help turn these seven criteria into a trial protocol fitted to your documents and access rights.