Healthcare: Clinical knowledge assistant with Microsoft Foundry

A RAG agent on Microsoft Foundry searches exclusively through authorized protocols and guidelines, indexed in Azure AI Search with hybrid retrieval and semantic ranking. Simple questions, such as looking up a dosage table, go to a small, fast model. More complex questions, where multiple guidelines need to be weighed against each other, go to a stronger model with more reasoning capability. Every answer cites the source and version number of the protocol, and the system refuses to answer once the retrieval score drops below a configured threshold.

In healthcare, an ungrounded answer is not an inconvenience but a risk. By making groundedness evaluation mandatory before production, and by having the system refuse rather than guess, the assistant remains a tool that points to the source instead of an oracle that confidently makes mistakes. Choosing the model per task keeps costs and latency under control without sacrificing quality on complex questions.

One large, expensive model for every question was simpler to build, but unnecessarily drove up cost and latency for the majority of simple questions. A generic chatbot without grounding on the hospital's own protocols was considered, but offered no guarantee that the answer matched current, hospital-specific policy.

• A strict groundedness threshold means the assistant sometimes refuses to answer where a doctor had a reasonable assumption; that is a deliberate choice, not a shortcoming.
• Two models side by side (small for simple, strong for complex) require a routing layer that needs maintenance as new models become available.
• Provisioned Throughput during peak moments increases predictability, but also fixed costs outside peak hours.

Microsoft Foundry (Foundry Agent Service, model catalog, evaluations, content safety), Azure AI Search (grounding), Microsoft Entra ID (identity and Conditional Access), Microsoft Purview (sensitivity labels, DSPM for AI).

• Make groundedness evaluation and a mandatory citation requirement part of the CI/CD gate, not a separate quality check afterward.
• Route simple and complex questions to different models instead of using one model for everything.
• Let sensitivity labels travel through to the answer, so confidentiality is not lost when citing a source.

In the first weeks, the retrieval threshold turned out to be too strict: the assistant refused more often than necessary on questions that could in fact be answered with grounding. After refining the chunking strategy on the protocol documents, using smaller, semantically coherent sections, the number of unwarranted refusals dropped significantly without lowering the groundedness threshold.