Building grounded agents with Microsoft Foundry: model choice, permissions and evaluations

Not every step in an agent flow needs the heaviest, most expensive model. Simple classification or extraction tasks run perfectly well on a smaller, faster model, while complex reasoning with multiple trade-offs deserves a stronger model. Using one model for every step is the simplest choice to build, but unnecessarily drives up cost and latency for the majority of interactions that never needed a heavy model in the first place.

An answer that sounds convincing but is not based on the retrieved context is not a minor mistake in an enterprise context, it is a hallucination that undermines trust in the entire solution. Groundedness evaluation therefore belongs as a mandatory gate in the CI/CD flow, alongside relevance, coherence and safety evaluations, with a defined quality threshold before a new version reaches production.

User input, and certainly input from external parties such as customers, is an untrusted source. Prompt shields detect and block attempts to reason an agent outside its instructions through carefully crafted input, groundedness detection flags ungrounded output, and PII filters prevent sensitive personal data from ending up unfiltered in an answer. These guardrails should be on before the first user reaches the agent, not as a reaction to the first incident.

An agent without its own identity is not auditable: nobody can later determine which action was performed by which agent, in which context. An Entra Agent ID gives every agent a traceable identity, and least-privilege tool permissions ensure an agent only has access to exactly the actions its task requires, nothing more.

Serverless capacity fits variable volume and keeps costs low during quiet periods, but offers no guarantees during unexpected spikes. Provisioned Throughput reserves throughput for predictable latency under constant load, at a fixed cost base. The wrong choice results in throttling where predictability was needed, or unnecessary cost where variability is the norm.

A problem in a multi-step agent run is only reproducible with end-to-end tracing of prompts, tool calls and runs. Without that visibility, an incident remains an anecdote instead of a diagnosable problem. Online evaluation on production traffic adds to this by continuously monitoring quality, safety and cost instead of only at release time.

The most common and most damaging mistake is bringing an agent to production without groundedness evaluation, assuming a well-written system instruction is enough to prevent hallucination. In practice, organizations only discover this once a user passes on a convincing but incorrect answer to a customer or colleague. Groundedness evaluation as a mandatory gate, with an instruction to refuse below the retrieval threshold, prevents this scenario from playing out in production instead of in a test environment.