Measuring the cost of an LLM system: a scoping method for CTOs

Measuring the cost of an LLM system is not about multiplying a token price by an estimated volume. A defensible scoping decision links a complete business request to the components actually used: model, context, retrieval, tools, traces and infrastructure. It separates what varies with usage from what is billed even without traffic, then compares the assumptions with the observed counters.

In short

• Start from a representative business journey and measure its steps, not just the model call.
• Keep the units that explain cost separate: input and output tokens, tool calls, retrieved documents, infrastructure time and storage.
• Compare estimates and observed costs on the same technical scope. Vendors point out that billing data can arrive with a delay.
• Choose the model, the context and execution limits based on quality and cost evidence specific to the use case. There is no universal cost per request.

This method helps prepare a technical and financial decision. It provides no vendor prices, no project budget and no promise of savings. Prices, volumes, contractual constraints and data rules must be checked for each environment.

Why is the model's list price not enough?

The model is a major cost item, but it does not sum up an LLM system. A request can also involve a document retrieval layer, embeddings, reranking, tool calls, a database, traces, a security gateway and permanently available infrastructure.

AWS distinguishes managed models, generally billed by input and output tokens, from self-hosted models, whose cost depends more on infrastructure, storage and network. Microsoft Foundry's documentation also points out that an application can use several Azure services and that the view of a single model does not necessarily cover the cost of the whole solution.

The scoping question therefore becomes: "what does a useful journey cost for this user, at this quality level and with these controls?" This wording avoids comparing an isolated unit price with a bill that includes other components.

Which journey should you measure before putting a figure on it?

Choose one or more journeys that match the initial scope. For a document assistant, this can be a simple question with sources, a question that requires several documents, an out-of-scope request and a request that must be refused under the access rights. For an agent, add the journeys that call a tool and those that stop before an impactful action.

For each journey, keep a measurement sheet:

1. the user task, the role and the expected result;
2. the version of the system, model, prompt and index;
3. the authorised sources or tools;
4. the steps executed and their duration;
5. the units consumed per step;
6. the expected behaviour on failure, refusal or limit overrun.

The same breakdown makes evaluation easier. Our guide on evaluating a RAG in production explains how to separate retrieval, answer, latency, cost and access so that a gap is not attributed to the wrong component.

Which cost items should be separated in the cost model?

The following table is a collection grid, not a price list. The relevant items depend on the chosen architecture.

A single unit can inform several decisions. Context size, for example, affects input cost, latency and sometimes quality. AWS recommends optimising prompt length, response length and the variables that directly drive consumption, while checking that the expected functionality is still met.

How do you build an estimate that can be checked?

A useful estimate states its assumptions instead of producing a single total. Start by defining a period, a user scope and a distribution of journeys. Then keep the prices applicable on the date of the estimate, the region or hosting mode, and the units billed by each component.

For the variable parts, use a formula per journey:

variable cost of the journey
= inference cost
+ retrieval and data cost
+ cost of the tools called
+ trace and evaluation cost attributable to the journey

Add fixed or semi-fixed costs separately: reserved capacity, available infrastructure, minimum storage, security services, supervision or support. The goal is not to allocate every cent artificially. It is to make visible what changes with traffic, what changes with request complexity and what remains due when traffic is zero.

Microsoft Foundry recommends keeping the same resource scope when comparing an estimate with an actual cost and recording gaps per service. The documentation also states that cost and usage records can arrive with a delay: a reconciliation must rely on a trend window, not on a minute-by-minute reading.

How do you instrument the measurement without making traces unusable?

A trace useful for cost analysis links a request to its steps without storing more business data than necessary. In practice, you must be able to find a system version, usage counters, the services used, the duration of each step and the execution result. The full content of a request or answer is not essential in every context and must follow the organisation's data protection rules.

Microsoft's observability documentation describes signals such as token consumption, latency, error rates, quality scores and traces of LLM or tool calls. Use these signals to explain a variation, not as a substitute for a business decision rule.

For a RAG system, also add what distinguishes a costly but grounded answer from a costly answer caused by unnecessary retrieval: number of documents, context size, reranking, filters applied and the answer's result. Access rights must remain a test dimension, as in our method to scope an enterprise RAG project.

Which design decisions can reduce cost without degrading the use case?

The right lever depends on the observed evidence. It is not about reducing every counter without looking at the result.

1. Size the model to the task. AWS recommends evaluating smaller models before scaling up, then comparing the quality obtained with the real need. This decision must be tested on a representative set of cases.
2. Reduce context to useful content. Measure retrieval quality before mechanically increasing the number of documents sent to the model. A longer context can increase cost without making the answer more verifiable.
3. Bound agentic workflows. Define the authorised actions, the maximum number of steps, the stop conditions and the escalation path. An execution that runs without an exit condition should not be treated as a mere billing problem.
4. Evaluate caching as a hypothesis. AWS states that prompt caching can reduce input token costs when long, repeated components are reused, subject to the conditions specific to the model and the vendor. Measure hit and miss rates, as well as quality and latency, before adopting this lever.
5. Review after each significant change. A new model, a new retrieval policy, a corpus or tool change alters the units consumed. Compare the candidate version with a reference rather than inferring the effect from a single dashboard.

These choices must remain compatible with quality, security, rights and operations objectives. NIST's GenAI profile proposes adapting risk management to the organisation's context, requirements and risk tolerance. It does not prescribe a budget or a universal architecture.

Which decision table should you prepare for the CTO?

A short table is often more useful than a complex projection. It links each assumption to a measurement and a possible decision.

The expected result is not a number that claims to stay true in the long term. It is a basis for comparison that the team can update when volume, model, data or scope change. For the broader decision to move from a prototype to an operated system, also see the go/no-go criteria for an LLM system.

FAQ

How do you calculate the cost per request of an LLM system?

Measure a complete request on a representative journey. Separate inference, context and retrieval, tools, traces and the relevant share of infrastructure. Keep the units and dated prices so you can explain the gap between an estimate and the observed costs.

Should you track tokens only?

No. Tokens often explain part of inference, but an LLM system can also generate search, storage, tool, compute, security and observability costs. The items to track depend on the architecture and the business journey.

How do you reduce the cost of a RAG without degrading the answer?

Start by measuring retrieval and answers on the same test cases. Then reduce context or change the model only if the evidence shows that the result remains suited to the use case and the access rules. Do not replace an evaluation with a token reduction.

Is real-time cost data always final?

No. Microsoft states that cost and usage records can be delayed depending on ingestion. Use near-real-time data to follow an operational trend, and billing data over a consistent window for financial reconciliation.

Methodology and sources

• NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, published in July 2024, consulted on 15 September 2026. This profile accompanies the AI RMF and proposes actions to adapt to the context, requirements and risk tolerance.
• Microsoft Foundry, Plan and manage costs, consulted on 15 September 2026. The documentation describes estimation, tracking, the comparison scope between forecasts and observed costs, and possible delays in cost data.
• Microsoft Foundry, Observability in Generative AI, consulted on 15 September 2026. The documentation describes token, latency, error and quality metrics as well as execution traces.
• AWS Well-Architected Framework, Generative AI Lens: Cost optimization, consulted on 15 September 2026. The guide covers model selection, resource consumption and workflow limits.
• AWS Well-Architected Framework, Right-size model selection to optimize inference costs, consulted on 15 September 2026.
• AWS Well-Architected Framework, Implement prompt caching to reduce token costs, consulted on 15 September 2026.

The method presented is a scoping grid for enterprise LLM systems. It contains no client figures, vendor prices, timelines, SLAs or promises of savings, security or compliance. Assumptions, responsibilities and trace retention rules must be validated for each organisation.

Do you need to link an LLM prototype to a defensible cost and operations model? Describe your scope to IgnitionAI. A first conversation can help identify the journeys, the units to measure and the decisions to document before widening usage.