How to build a RAG solution

Collect fifty to two hundred representative questions with the answers an expert would give, including the expected sources. This is your evaluation set, your regression test for AI, which you run on every change. Without measured quality, every improvement is guesswork.

Inventory the documents: owner, currency, duplicate versions. Archive what is outdated before you index.

Chunk on document structure (headings, articles, clauses) with overlap. Index in Azure AI Search with both vectors and keywords, and turn on semantic ranking. Include metadata: source, version, publication date, audience.

Test hybrid search against your evaluation set before a model is involved: is the right piece of text in the top five results? If not, this is your problem, not the generation.

Choose a model in Foundry and write a system prompt with three hard rules: answer only from the supplied passages, always cite the source, and say so explicitly when the information is missing.

Run your evaluation set on every change (Foundry evaluations helps with this). Retrieval, groundedness and relevance are built-in evaluators; refusal behavior is not, so you build that as a custom evaluator, which is still preview. In production, log every interaction with retrieved chunks and scores.

Documents change, so indexes do too. Arrange reindexing on publication, monitor unanswered questions as a backlog and repeat the evaluation monthly. RAG is not a project with an end date; it is a product with a data chain.