The situation
You want people to ask questions of your organisation’s documents and get answers they can trust: policies, contracts, product specifications, support articles, project files. The demo is easy. A weekend prototype can embed some PDFs and answer questions about them.
What stops it reaching real users is less visible. Documents live in several systems with different access rules. Some are superseded but never deleted. Some users must never see certain content. And when an answer is wrong, nobody can tell whether the system retrieved the wrong passage, ranked the right one too low or simply invented something.
This page is for commissioning the system properly: designed and built as a bounded delivery, with access rights, citations and freshness as acceptance criteria rather than afterthoughts.
What the work involves
Sources and permissions first. Before any embedding, I inventory every source in scope: where it lives, who owns it, how often it changes, and how access is expressed (groups, folders, client codes, tenant IDs). The hardest design decision in most enterprise retrieval is how to carry those rules into the index so they can be enforced in the query. Post-filtering after retrieval leaks, and it also starves the model of context.
A relevance set before a pipeline. With two or three of your domain experts, I build a set of real questions, the passages that should answer them, and passages that must never appear for given roles. This becomes the measure for every choice that follows.
Retrieval designed for your content. Chunking that respects document structure, hybrid lexical-plus-vector search where exact terms matter (part numbers, clause references, product codes), metadata filters for version and access, and re-ranking where it measurably helps.
Answers that show their sources. Each answer cites the passage and document version used. If nothing relevant is retrieved, the assistant says so rather than improvising.
Freshness as an operational property. Ingestion runs on changes or a schedule, with alerts when a source stops updating and removal of withdrawn documents.
The signature deliverable
You receive the retrieval and knowledge-system implementation: ingestion, index, retrieval service and answer interface in your environment, plus the evaluation harness, permission test set, source inventory and runbook. Illustrative example of an evaluation summary at acceptance:
| Question category | Questions | Correct passage in top 5 | Cited correctly | Permission leaks |
|---|---|---|---|---|
| HR policy | 40 | 37 | 36 | 0 |
| Product specifications | 55 | 49 | 48 | 0 |
| Client contracts (restricted) | 30 | 26 | 26 | 0 |
Illustrative example. Figures are placeholders showing the format, not results from a client.
How acceptance is judged
After the source assessment and baseline, we agree thresholds in writing: retrieval recall per question category, citation correctness, zero leaks on the permission set, and a freshness check that passes for each source. The pilot adds real-user failures to the evaluation set. Your named owner signs off against those measures. Where a category falls short, the cause is recorded with options rather than hidden in an average.
Ownership and handover
Everything runs in your accounts and repositories. The owner receives the harness, the test sets, the source inventory, the re-indexing and access-change runbook, and a prioritised backlog of known limits. I pair with at least one of your engineers during the build so the system is not a black box at handover. I do the work personally; any specialist help is disclosed and approved by you first.
When to choose something else
If you already run an assistant and its answers have drifted, RAG quality rescue starts from a failure analysis of your live system. If your team owns retrieval and needs senior capacity under its own manager, hire a RAG engineer on contract. If the blocker is that sources and access rules are unknown or scattered, start with AI data readiness. If you need to know whether a retrieval method holds in general, that is a research question for dipankar.cc.