Microsoft Foundry: The AI App & Agent Platform

The biggest misconception is seeing Foundry as Azure OpenAI with a new name. It is the platform for the full lifecycle of AI apps and agents: build, ground with RAG, deploy, observe, govern, and optimize cost. Anyone who treats it as a model API misses the architecture and gets surprised by tokens, tools, retrieval, and runtime cost.

Microsoft Learn describes Foundry as a layered architecture: a Foundry resource for governance, networking, security, and model deployments; projects as the development boundary for teams, agents, evaluations, and files; and connected services such as Storage, Key Vault, and Azure AI Search, each with its own governance boundary. The Foundry Agent Service provides prompt agents (no compute of your own to manage) and hosted agents (containerized, dedicated endpoint and Entra identity, billed on active CPU and memory consumption). The model router routes prompts to suitable models in Balanced, Cost, or Quality mode. Foundry IQ, underpinned by Azure AI Search, is the managed knowledge layer for RAG with agentic retrieval and permission-aware grounding. The control plane gives central visibility into governance, observability, cost, and token usage across the whole AI estate.

Think in platform layers. The resource is the governance boundary (policy, networking, model deployments). Projects isolate teams and workloads. Foundry Models provide model access through pay-per-token or provisioned deployments. The model router picks a model per prompt based on complexity. Foundry IQ and Azure AI Search form the retrieval and knowledge layer: embeddings, vector and hybrid search, agentic retrieval that breaks a complex question into parallel subqueries with citations. The Agent Service adds the runtime: prompt agents, hosted agents, tools, sessions, and identity. Observability runs through evaluators, Azure Monitor, Application Insights, and OpenTelemetry tracing.

• Separate production, test, and experiment with projects instead of one resource; put networking and private endpoints on the resource, not per project again.
• Choose deliberately between prompt agents (no compute to manage) and hosted agents (containerized, dedicated endpoint and identity) based on control versus operational work.
• Position RAG as the default for grounding in changing or proprietary knowledge; model Azure AI Search and Foundry IQ as their own layer with embeddings, chunking, and agentic retrieval.
• Put observability in place from day one: evaluations before deployment, tracing and monitoring after. Agents without evaluation and tracing cannot be accounted for.

Present Foundry security as a layered control model: identity, access, data, network, runtime guardrails, observability, and compliance. Azure RBAC separates control plane from data plane and assigns roles at resource and project scope, with managed identities at both. For data, use permission-aware retrieval with source ACLs and OBO patterns, complemented by Microsoft Purview and DSPM for AI. Prompt injection deserves explicit attention: malicious input through users, retrieved content, tool output, or memory can steer model behavior, and context pooling, persistent memory, and agentic tool execution increase the blast radius. Apply content filters and Prompt Shields, guardrail policies, and human review on high-risk actions.

Model Foundry cost service by service and meter by meter, not as a platform fee. The major drivers: model inference (token-metered, rate per model and deployment), input and output tokens (system prompts, context, history, tool definitions, retries), hosted agents (on container compute per hour; prompt agents carry no extra Agent Service charge), tools (File Search, Code Interpreter, Web Search, Foundry IQ), RAG and retrieval (search capacity, embeddings, index storage, agentic retrieval), monitoring, and networking. Optimize on cost per successful outcome, not per message: one request can fan out into retrieval, model calls, tools, guardrails, retries, and logs. The model router is a FinOps lever, but needs evaluation gates; choose provisioned capacity only after measured throughput.

Production Foundry requires a platform operating model, not just a proof of concept. The control plane is intended for organizations managing multiple agents across projects with central compliance, policy enforcement, and cost and token tracking. Typical pilot-to-production blockers: no clear platform owner for resources, projects, quotas, and networking; no evaluation maturity (golden datasets, groundedness, safety checks, regression gates); no FinOps telemetry (token usage, cost per request and per outcome); no security baseline (RBAC, managed identities, private endpoints, guardrails, prompt-injection tests); and no lifecycle process (versioning, release gates, rollback, incident response).

• Foundry versus Microsoft 365 Copilot: custom apps, agents, and model control versus ready-made productivity over Microsoft 365 data within Copilot's compliance boundaries.
• Foundry versus Copilot Studio: pro-code and model and runtime choice versus low-code, graphical design, and Power Platform integration.
• Prompt agents versus hosted agents: less operational work versus more control, a dedicated endpoint, and identity.
• Single model versus model routing: reproducibility versus cost and latency optimization with evaluation gates.
• RAG versus fine-tuning: current, citable knowledge versus stable behavior and shorter prompts.

• Treating Foundry as Azure OpenAI with a new name, without explaining the resource, project, control plane, and agent model.
• Building RAG without modeling Azure AI Search, embeddings, chunking, agentic retrieval, and citation quality.
• Measuring messages instead of successful outcomes, while one request fans out into model, retrieval, tool, and logging events.
• Deploying hosted agents without right-sizing CPU, memory, and concurrency.
• Skipping evaluations, traces, red teaming, and drift monitoring before production.