The research question
What actually drives the cost of an AI agent, and how do you model it before going to production?
Why it matters
An agent that solves a task in five steps makes five model calls, each with context, tools, and sometimes a retry. Cost scales not with the number of users but with the number of decisions the agent makes. Without a model, you structurally underestimate the bill.
What the research says
The cost of an agent does not sit in the price list but in the systems layer underneath. Kwon et al. (2023) showed at SOSP that naive memory management in LLM serving wastes most GPU memory through KV-cache fragmentation, and that paging that cache raises throughput several times over at equal latency. Yu et al. (2022) showed at OSDI with Orca that scheduling weighs at least as heavily as hardware: admitting new requests mid-generation instead of waiting for a full batch raises throughput dramatically. Dettmers et al. (2022) showed at NeurIPS that 8-bit quantisation halves the memory footprint up to 175 billion parameters without quality loss. Chen et al. (2023) described with FrugalGPT a cascade strategy routing requests through progressively more expensive models until quality suffices, reporting comparable quality at a fraction of the cost. Hoffmann et al. (2022) showed that a smaller, well-trained model can beat a larger, undertrained one, which structurally lowers inference cost.
The product layer: the FinOps Foundation defines token economics as metering, attributing, and connecting token consumption to business outcomes, and identifies system prompt overhead, context and memory, model selection, output length, retries, and orchestration overhead as the major consumption variables. Microsoft Foundry documentation states that cost depends on the Azure services used and that you should estimate it, validate with test workloads, reconcile against meter charges, and monitor with Cost Management. Microsoft 365 Copilot combines per-seat licensing with consumption for agents.
What the research does not prove
The line between what the studies actually establish and what we infer from them.
None of these studies measures a real agent loop. FrugalGPT tests classification, question answering, and summarisation, not multi-step tool use with feedback. vLLM and Orca measure single-turn inference. The economics of an agent that goes back and forth with tools fifteen times, with highly variable output lengths, has not been studied in peer-reviewed work at the time of writing. That is a genuine gap, not a detail.
The absolute figures in these papers are dated. They rest on 2022 and 2023 API pricing and hardware. What holds is the structural findings: batching and memory management drive throughput, quantisation trades memory against precision, and cascades trade latency against cost. Take the mechanisms, not the numbers.
Orca's throughput gains assume requests with comparable output lengths. Precisely in agentic scenarios, where one step produces two tokens and the next two thousand, that efficiency can degrade. And our own cost models per step, per RAG call, and per orchestration layer are calculation frameworks we built, not measured results.
Technical context
Token cost is only one part. A realistic model includes: the token model (input plus output per call), the RAG model (embeddings and searches), tools and workflow (number of steps), storage and data, M365 licensing, and operational cost such as monitoring and networking. Model selection is the strongest lever: a large model per step is often overkill.
Architecture implications
- Route per step: use a small model for simple decisions and escalate only where needed.
- Limit context: memory and system prompt travel with every call and count again each time.
- Cache where answers are repeatable, and set a hard limit on the number of agent steps.
- Measure per outcome, not per call, so a cheap agent that often fails does not count as a win.
Security implications
Cost and security meet at tool scope: an agent with broad rights does more, fails more often, and generates more retries. Tight scopes are cheaper and safer at the same time.
Cost implications
The hidden cost sits in retries, overlong output, and orchestration overhead. Reason with a realistic number of steps per task, not with an average call. Always reconcile your Foundry estimate against real meter charges before scaling.
Adoption implications
Cost management is not a separate finance topic but a design variable. Teams that look at cost only afterward build agents that work technically but do not add up economically. Put the cost question next to the architecture question.
Trade-offs
- Large model versus model routing: more consistent but more expensive versus cheaper with more design work.
- More context versus lower cost: richer prompts give better answers but count on every call.
- Autonomy versus predictability: free-roaming agents solve more but make the bill hard to forecast.
Common mistakes
- Reporting cost per token instead of cost per successful outcome.
- Forgetting retries and orchestration overhead in the estimate.
- Using the largest model everywhere because it gives the best answer.
- Never reconciling the Foundry estimate against real meter charges.
For architects
Treat cost as an architecture dimension alongside security and performance. Build a small cost model per agent with the real variables (steps, model choice, context, retries) and validate it with a test workload before scaling.
References
Grouped by source hierarchy. Research carries the reasoning, product documentation carries the implementation. Verify any of it yourself.
Methodology & confidence
The cost mechanics of inference rest on systems and ML research: Kwon et al. (2023) at SOSP, Yu et al. (2022) at OSDI, Dettmers et al. (2022) at NeurIPS, Hoffmann et al. (2022) on scaling laws, and Chen et al. (2023) on cascades (tier 1). Four of these are peer-reviewed at top venues; FrugalGPT is a preprint later accepted at TMLR.
The FinOps framework supplies the cost structure for token economics (tier 2). Microsoft Learn supplies the product facts on Foundry cost management and Copilot licensing (tier 3). The evidence level is mixed because the underlying mechanisms are well researched but the agent loop itself is not.
Peer-reviewed research
Journals, systematic reviews, meta-analyses, and reputable conference proceedings. This is the substantive basis.
- Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon, W., Li, Z., Zhuang, S. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM Symposium on Operating Systems Principles (SOSP 2023)DOI 10.1145/3600006.3613165
- Orca: A Distributed Serving System for Transformer-Based Generative ModelsYu, G.-I., Jeong, J.S., Kim, G.-W. et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX Symposium on Operating Systems Design and Implementation (OSDI 2022)
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleDettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Advances in Neural Information Processing Systems (NeurIPS 2022)
- Training Compute-Optimal Large Language ModelsHoffmann, J., Borgeaud, S., Mensch, A. et al. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems (NeurIPS 2022)
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformanceChen, L., Zaharia, M., Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv preprint (arXiv:2305.05176); later in Transactions on Machine Learning Research
Academic and institutional
Research institutes and standards bodies such as NIST, ISO, IEEE, and ACM.
Official technical documentation
How you build and configure it. Answers the implementation question, not the evidence question.
Continue across TechExplained
The same research, applied in other ways.