How to choose an LLM

Formulate what the model concretely has to do: summarize, classify, reason over multiple steps, generate code, or hold a conversation with tool use. Public benchmarks measure averages over everything; you only need one task that works well.

• Data residency and compliance: which regions are allowed? This often removes more models than any benchmark.
• Latency: a customer conversation demands different response times than a nightly batch.
• Context length: how much really has to go into the prompt? More context is slower and more expensive, and often a sign that retrieval should be better.
• Cost per thousand interactions at expected volume, not per token in a demo.

Pick from the Foundry model catalog: one small and cheap model, one mid-range, one top model. The question is not "which model is best" but "what is the smallest model that does the task demonstrably well enough".

Build an evaluation set of real examples from your context and run all three models through it with Foundry evaluations. Measure task-specific. Correctness, groundedness and relevance have built-in evaluators; format compliance and refusal behavior do not, so you build those as a custom evaluator, which is still preview. Ten real examples say more than any leaderboard.

Extrapolate to production volume. A model that scores twice as well but is ten times as expensive is rarely the right choice for high-volume tasks. A small model is easily an order of magnitude cheaper per token than a frontier model; our own rule of thumb is ten to thirty times, but Microsoft does not publish that ratio, so check it against the current price list. Those ratios shift with every model generation. So think in a portfolio instead of one model: small and fast where it can, large or reasoning where it must. Using one large model for everything is using a Formula 1 car to deliver the mail: technically impressive, economically indefensible.

Provisioned deployment for high, predictable volume with a tight latency requirement; pay-as-you-go for variable or experimental traffic. Choose the deployment type on data residency (Data Zone if data must stay within, say, the EU), plan quota per region and per model, and know that not every model is available in every region.

Models get new versions and are retired. Record which version you use and test a new version against your evaluation set before you switch, otherwise you get a silent quality regression. Abstract the model choice in your architecture (Foundry makes switching easy), keep your evaluation set current and repeat the comparison every quarter. Model choice is not a one-time decision but a recurring process.