Skip to main content
TechExplainedTechExplained
|
Best PracticeLevel: Intermediate

Keeping the token cost of an AI solution under control

Where the money actually leaks in Foundry workloads: context length, model choice, retries and a retrieval step that sends too much along.

TechExplained 3 min readPublished: 4 August 2026Last updated: 4 August 2026
#cost optimization#tokens#rag#enterprise ai
Architect presents token cost management for an AI solution to the team, with cost per conversation and per application on the screen
01

Calculate per conversation, not per token

A price per thousand tokens tells you nothing about your monthly bill. Convert it to the unit your organisation thinks in: what does one conversation cost, one processed document, one resolved ticket. That number makes the conversation with the business possible and immediately exposes which step dominates.

The calculation has four factors: the number of interactions, the tokens in the prompt, the tokens in the answer, and the number of times the chain runs again. That last one is almost always forgotten and is often the largest with agents.

02

Context length is the most expensive dial

In a RAG solution most of the cost sits in the prompt, not in the answer. Sending ten chunks because "more context is better" triples the price of every question and usually does not make the answer better; a model forced to dig through noise answers more vaguely.

What helps:

  • Fewer but better chunks. Reranking down to five results almost always beats ten unfiltered ones, in quality and in price.
  • Chunk size that matches the source. Four hundred to eight hundred tokens works for prose; a table or a product list belongs in its own shape.
  • Short system instructions. That text ships with every call. A thousand-token instruction at a hundred thousand conversations a month costs more than most people expect.
03

Use the large model where it makes a difference

Not every step in a chain deserves the same model. Classification, rewriting a question and choosing a tool are cheap tasks where a small model does fine. The final answer to the customer is the step where quality counts.

So the biggest saving rarely sits in "pick a cheaper model" but in "pick the right model per step". Do measure it: only route once you have seen on your own evaluation set that the small model handles that step.

04

Retries and agent loops are a cost item

An agent that gets stuck and keeps trying burns tokens without result. So always put a hard limit on the number of iterations and on the total duration of a run, and log how often that limit is hit. That number is a quality signal and a cost item at the same time.

The same goes for error handling that automatically retries. A retry on a timeout is reasonable; a retry on a disappointing answer is a second call in disguise.

05

Cache what repeats

Many enterprise questions are not unique. A cache on the combination of question and sources used absorbs repeat questions without a model being involved. Important: make the cache respect authorisation, otherwise an answer leaks to someone who is not allowed to see the source. That is one of the few places where a cost measure can introduce a security risk.

06

Make consumption visible per application

One shared deployment for all applications makes it impossible to see where the money goes. Give every application its own deployment, or at minimum its own key, and log per call which application, which model and how many tokens. Without those three fields any cost analysis afterwards is a reconstruction.

07

Pitfalls

  • Steering on price per token. A model that costs half but needs three attempts is more expensive.
  • Only costing out the demo. Demos have short conversations. Production has conversation history, and that grows into the prompt with every turn.
  • Forgetting embeddings. Re-indexing a large source is a one-off but substantial item, and it happens more often than planned once the chunking strategy changes.
  • Measuring cost without measuring quality. Without an evaluation set you do not know whether a saving was a regression.

The process at a glance

Click a step for its key decision

Summary

Calculate per conversation, not per token

A price per thousand tokens tells you nothing about your monthly bill. Convert it to the unit your organisation thinks in: what does one conversation cost, one processed document, one resolved ticket. That number makes the conversation with the business possible and immediately exposes which step dominates.

Related content

Keeping the token cost of an AI solution under control | TechExplained