ISV: Embedded multi-tenant AI copilot with Microsoft Foundry

Multi-tenant RAG agents on Foundry Agent Service use an isolated index or tenant scoping within Azure AI Search per tenant, so retrieval never searches beyond a tenant's boundaries. Tracing and cost attribution are set up per tenant, so token usage and costs can be traced back to the customer that caused them. Evaluations run mandatorily per release, before a new version rolls out to all tenants.

Tenant isolation in the retrieval layer itself, rather than relying on careful prompt instructions, is the only way to give hard guarantees to customers who are themselves under oversight. Model choice flexibility and standardized governance scale across customers without every new customer requiring an entirely new architecture.

A single shared index with tenant filtering only in the prompt was cheaper to build, but left the guarantee against tenant data leaks dependent on correct prompt logic instead of a hard separation in the retrieval layer. A fully separate deployment per customer guaranteed isolation, but would not scale to hundreds of customers without unmanageable operational overhead.

• Tenant scoping within a single index is more efficient to manage than a separate index per tenant, but requires the scoping logic to be flawless with every change.
• Shared Provisioned Throughput is more cost efficient than per-tenant serverless capacity, but reintroduces a form of shared infrastructure that must be carefully monitored for cross-tenant noise.
• Evaluations per release lengthen the release cycle, but prevent a quality regression from spreading to all customers at once.

Microsoft Foundry (Foundry Agent Service, evaluations, tracing), Azure AI Search (tenant-scoped grounding), Microsoft Entra ID (Entra Agent ID per agent instance).

• Enforce tenant isolation in the retrieval layer itself, not only through instructions in the system prompt.
• Set up tracing and cost attribution per tenant from the first release, not as a later addition once the first customer asks for a cost breakdown.
• Make evaluations a mandatory gate per release, so a quality regression is caught before rollout to all tenants.

In the first version, tenant scoping through metadata filters on a shared index was sometimes slower than expected under high concurrent load. A revision of the index structure, using tenant ID as the primary partition criterion instead of a separate filter, brought retrieval time back to a level that stayed predictable even under peak load.