The research question
What is Microsoft Foundry really, beyond model access, and how do you use it to design an enterprise AI platform with governance, cost, and observability in order?
Why it matters
The biggest misconception is seeing Foundry as Azure OpenAI with a new name. It is the platform for the full lifecycle of AI apps and agents: build, ground with RAG, deploy, observe, govern, and optimize cost. Anyone who treats it as a model API misses the architecture and gets surprised by tokens, tools, retrieval, and runtime cost.
What the research says
The research base under a RAG platform is by now solid. Lewis et al. (2020) introduced retrieval-augmented generation at NeurIPS and showed that a model retrieving external knowledge answers more specifically and factually than one that must draw everything from its weights. Gao et al. (2023) mapped the architectural choices in a survey. Liu et al. (2023) published the "lost in the middle" effect in TACL: models use information at the start and end of their context better than information in the middle, which means supplying more context does not automatically work better. Zhang et al. (2023) described the mechanisms behind hallucination. Zheng et al. (2023) showed at NeurIPS that a strong model acting as judge reaches over eighty percent agreement with human raters, but with demonstrable position, verbosity, and self-preference bias.
The product layer: Microsoft Learn describes Foundry as a layered architecture with a Foundry resource for governance, networking, security, and model deployments, projects as the development boundary, and connected services such as Storage, Key Vault, and Azure AI Search each with their own governance boundary. The Foundry Agent Service provides prompt agents and hosted agents with a dedicated endpoint and Entra identity. The model router routes prompts to suitable models in Balanced, Cost, or Quality mode. Foundry IQ, underpinned by Azure AI Search, is the managed knowledge layer for RAG with agentic retrieval and permission-aware grounding. The control plane gives central visibility into governance, observability, cost, and token usage.
What the research does not prove
The line between what the studies actually establish and what we infer from them.
There is no independent research on Foundry, Foundry IQ, or the model router. That these services produce better outcomes than a self-built RAG stack has not been measured. The research findings above concern the patterns, not Microsoft's implementation of them.
Whether long context makes RAG redundant is an open question. "Lost in the middle" was measured on mid-2023 models and newer models demonstrably handle long context better, but nobody has established where the crossover lies. Anyone claiming RAG is dead, or conversely indispensable, is ahead of the evidence.
LLM-as-judge, the method underlying automated evaluation, has documented bias. Use it to flag regression, not to establish absolute quality, and keep human sampling alongside it. Our recommendations on project structure, model routing, and evaluation setup derive from this literature plus product documentation; they have not been tested as a whole.
Technical context
Think in platform layers. The resource is the governance boundary (policy, networking, model deployments). Projects isolate teams and workloads. Foundry Models provide model access through pay-per-token or provisioned deployments. The model router picks a model per prompt based on complexity. Foundry IQ and Azure AI Search form the retrieval and knowledge layer: embeddings, vector and hybrid search, agentic retrieval that breaks a complex question into parallel subqueries with citations. The Agent Service adds the runtime: prompt agents, hosted agents, tools, sessions, and identity. Observability runs through evaluators, Azure Monitor, Application Insights, and OpenTelemetry tracing.
Architecture implications
- Separate production, test, and experiment with projects instead of one resource; put networking and private endpoints on the resource, not per project again.
- Choose deliberately between prompt agents (no compute to manage) and hosted agents (containerized, dedicated endpoint and identity) based on control versus operational work.
- Position RAG as the default for grounding in changing or proprietary knowledge; model Azure AI Search and Foundry IQ as their own layer with embeddings, chunking, and agentic retrieval.
- Put observability in place from day one: evaluations before deployment, tracing and monitoring after. Agents without evaluation and tracing cannot be accounted for.
Security implications
Present Foundry security as a layered control model: identity, access, data, network, runtime guardrails, observability, and compliance. Azure RBAC separates control plane from data plane and assigns roles at resource and project scope, with managed identities at both. For data, use permission-aware retrieval with source ACLs and OBO patterns, complemented by Microsoft Purview DSPM. Prompt injection deserves explicit attention: malicious input through users, retrieved content, tool output, or memory can steer model behavior, and context pooling, persistent memory, and agentic tool execution increase the blast radius. Apply content filters and Prompt Shields, guardrail policies, and human review on high-risk actions.
Cost implications
Model Foundry cost service by service and meter by meter, not as a platform fee. The major drivers: model inference (token-metered, rate per model and deployment), input and output tokens (system prompts, context, history, tool definitions, retries), hosted agents (on container compute per hour; prompt agents carry no extra Agent Service charge), tools (File Search, Code Interpreter, Web Search, Foundry IQ), RAG and retrieval (search capacity, embeddings, index storage, agentic retrieval), monitoring, and networking. Optimize on cost per successful outcome, not per message: one request can fan out into retrieval, model calls, tools, guardrails, retries, and logs. The model router is a FinOps lever, but needs evaluation gates; choose provisioned capacity only after measured throughput.
Adoption implications
Production Foundry requires a platform operating model, not just a proof of concept. The control plane is intended for organizations managing multiple agents across projects with central compliance, policy enforcement, and cost and token tracking. Typical pilot-to-production blockers: no clear platform owner for resources, projects, quotas, and networking; no evaluation maturity (golden datasets, groundedness, safety checks, regression gates); no FinOps telemetry (token usage, cost per request and per outcome); no security baseline (RBAC, managed identities, private endpoints, guardrails, prompt-injection tests); and no lifecycle process (versioning, release gates, rollback, incident response).
Trade-offs
- Foundry versus Microsoft 365 Copilot: custom apps, agents, and model control versus ready-made productivity over Microsoft 365 data within Copilot's compliance boundaries.
- Foundry versus Copilot Studio: pro-code and model and runtime choice versus low-code, graphical design, and Power Platform integration.
- Prompt agents versus hosted agents: less operational work versus more control, a dedicated endpoint, and identity.
- Single model versus model routing: reproducibility versus cost and latency optimization with evaluation gates.
- RAG versus fine-tuning: current, citable knowledge versus stable behavior and shorter prompts.
Common mistakes
- Treating Foundry as Azure OpenAI with a new name, without explaining the resource, project, control plane, and agent model.
- Building RAG without modeling Azure AI Search, embeddings, chunking, agentic retrieval, and citation quality.
- Measuring messages instead of successful outcomes, while one request fans out into model, retrieval, tool, and logging events.
- Deploying hosted agents without right-sizing CPU, memory, and concurrency.
- Skipping evaluations, traces, red teaming, and drift monitoring before production.
For architects
Explain Foundry as a layered AI platform: resource for governance and networking, projects for isolation, Foundry Models and the model router for model strategy, Foundry IQ and Azure AI Search for RAG, the Agent Service for the runtime, and the control plane plus observability for steering. Model cost per service and per outcome, and set security as a layered control model with Prompt Shields, RBAC, and Purview from the start.
References
Grouped by source hierarchy. Research carries the reasoning, product documentation carries the implementation. Verify any of it yourself.
Methodology & confidence
The knowledge and evaluation layer rests on RAG research: Lewis et al. (2020) for the pattern, Gao et al. (2023) for the overview, Liu et al. (2023) for the context limitations, Zhang et al. (2023) for hallucination, and Zheng et al. (2023) for the limits of automated evaluation (tier 1). Three of these are peer-reviewed at NeurIPS and in TACL, two are preprints.
Microsoft Learn and the Foundry pricing page supply the product facts on architecture, Agent Service, model router, agentic retrieval, observability, Prompt Shields, and Purview DSPM (tier 3). The evidence level is mixed and freshness is fast: the model router and agentic retrieval carry preview elements and limits that shift with each release.
Peer-reviewed research
Journals, systematic reviews, meta-analyses, and reputable conference proceedings. This is the substantive basis.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis, P., Perez, E., Piktus, A. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS 2020)
- Lost in the Middle: How Language Models Use Long ContextsLiu, N.F., Lin, K., Hewitt, J. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics
- Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng, L., Chiang, W.-L., Sheng, Y. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS 2023)
- Retrieval-Augmented Generation for Large Language Models: A SurveyGao, Y., Xiong, Y., Gao, X. et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint (arXiv:2312.10997)
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsZhang, Y., Li, Y., Cui, L. et al. (2023). Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint (arXiv:2309.01219)
Official technical documentation
How you build and configure it. Answers the implementation question, not the evidence question.
- Microsoft Foundry architectureMicrosoft Foundry architecture. Microsoft Learn
- What is Microsoft Foundry Agent Service?What is Microsoft Foundry Agent Service?. Microsoft Learn
- Hosted agents in Foundry Agent ServiceHosted agents in Foundry Agent Service. Microsoft Learn
- Model router for Microsoft Foundry conceptsModel router for Microsoft Foundry concepts. Microsoft Learn
- Retrieval augmented generation (RAG) and indexes in Microsoft FoundryRetrieval augmented generation (RAG) and indexes in Microsoft Foundry. Microsoft Learn
- Agentic retrieval overview (Azure AI Search)Agentic retrieval overview (Azure AI Search). Microsoft Learn
- Observability in Generative AIObservability in Generative AI. Microsoft Learn
- Prompt Shields in Azure AI Content SafetyPrompt Shields in Azure AI Content Safety. Microsoft Learn
- Considerations for deploying Microsoft Purview Data Security Posture Management (DSPM) for AIConsiderations for deploying Microsoft Purview Data Security Posture Management (DSPM) for AI. Microsoft Learn
- Microsoft Foundry pricingMicrosoft Foundry pricing. Microsoft Azure
Continue across TechExplained
The same research, applied in other ways.