Finance: RAG for compliance questions with Microsoft Foundry

Underwriters and claims handlers at an insurer searched daily through thousands of versions of policy terms, internal guidelines and regulator policy. Wrongly quoted terms led to complaints and remediation costs. A public chatbot was out of the question: answers must be demonstrably traceable to source documents.

A RAG solution on Microsoft Foundry. Documents are chunked and indexed in Azure AI Search on publication (hybrid: vector plus keyword, with semantic ranking). The application answers questions solely from retrieved passages, always cites source and version number, and refuses to answer when the retrieval score is below a threshold. Every interaction is logged with prompt, retrieved chunks and answer for audit and evaluation.

Grounding with mandatory source citation is the difference between "a chatbot" and a defensible tool in a regulated sector. Foundry offers the model-choice freedom (a small model for reformulation, a larger one for synthesis) and the evaluation tooling needed to prove quality, not just claim it.

Microsoft 365 Copilot with a declarative agent would have been simpler, but gave insufficient control over retrieval thresholds, refusal behavior and logging. A fully self-built stack (open-source vector database plus custom orchestration) gave maximum control, but at structurally higher operational cost.

• Strict grounding means more often "I do not know", deliberately accepted; a wrong answer is more expensive here than no answer.
• Semantic ranking and vectorization cost money per query; caching of frequent questions was needed to make the business case work.
• Document version control suddenly became a hard requirement: RAG mercilessly exposes that seven versions of the same policy are circulating.

Microsoft Foundry (model catalog, evaluations, content safety), Azure AI Search, Azure OpenAI in Microsoft Foundry Models, Application Insights for tracing, Entra ID.

• Chunk on document structure (articles, clauses), not on fixed character counts.
• Evaluate with a fixed question set per release; groundedness is a metric, not a feeling.
• Log retrieval scores, most "hallucinations" turn out to be retrieval problems.

Eighty percent of the quality gain came from the data chain: cleaning up documents, archiving duplicate versions and adding metadata. Switching the model gave a few percent improvement; improving the indexing gave tens of percent.