LLM Cost Management: Real Cost Structures in Production

When enterprises evaluate AI adoption, the first question they almost always ask is: how much will LLM API fees cost per month? The question itself is reasonable, but most companies use an estimation method with a systemic flaw — one that leads to underbudgeting and cost overruns after launch.

LLM costs in production are determined by architectural decisions, not simply usage volume multiplied by unit price.

Two Common Flaws in Cost Estimation

Most enterprises estimate costs by multiplying the expected token count per query by the API unit price, then multiplying by projected usage volume. This method has two fundamental flaws.

Flaw 1: Assuming a fixed cost per query. In a conversational AI system, every time a user sends a new message, the system prompt, conversation history, and RAG retrieval results must all be passed into the model together. As the conversation progresses, the input token count per round grows continuously. Without management, the last few turns of a ten-round conversation can cost four to five times more than the first turn.

Flaw 2: Not distinguishing between repeated and dynamic content. System prompts, knowledge base context, and conversation summaries are all passed into the model repeatedly in every API call. If the API provider supports caching, this stable content only needs to be billed once, with subsequent calls at a lower rate — which can significantly reduce total costs.

The Engineering Prerequisites for Prompt Caching

Prompt caching is an existing API feature, officially supported by Anthropic Claude and some other providers. The prerequisite for a cache hit is that the system prompt and the earlier portions of the conversation remain stable and are not dynamically reorganized with each query.

If the system reshuffles prompt content on every round, caching cannot take effect. This is a decision that must be made at the architecture design stage, not an optimization that can be retrofitted after launch.

Context Window Management: Another Cost Lever

In conversational systems, another common source of cost expansion is unbounded context window growth.

Most systems truncate based on message count (e.g., keeping only the last ten messages). The problem is that message length varies enormously: a reply containing a full document summary may be longer than ten ordinary messages combined. Truncating by message count does not effectively control costs.

Using token count as the truncation basis, combined with Incremental Summarisation — automatically compressing older history into a summary when the conversation approaches the window limit, which is then inherited at the start of the next session — is an effective method for controlling long-conversation costs without sacrificing conversational continuity. This architecture supports complete session continuity across multiple summarization cycles with minimal impact on user experience.

Multi-Model Strategy: Managing Both Cost and Availability

Depending on a single LLM provider carries two risks: pricing risk (the provider changes its billing structure) and availability risk (service outage).

In practice, the more robust approach is to test multiple major model providers — across three dimensions: translation consistency, hallucination rate, and output format — and select the default production provider based on test results rather than vendor marketing materials.

Configuring a fallback chain via environment variables — so that if the primary provider fails, the system automatically switches to a backup — is both an engineering safety measure and a practical bargaining lever that maintains optionality across providers. The most cost-effective provider for translation and summarization versus long-form reasoning versus short replies may differ; multi-model configuration allows selecting the best option by task type.

Monthly Cost Ranges Enterprises Should Expect

Taking a medium-sized Hong Kong company's internal AI query system as an example — approximately 200 to 500 queries per day with a RAG knowledge base:

The cost difference comes from architectural design, not usage control. With the same query volume, an optimized system can cost as little as 25–30% of the unoptimized version per month.

Cost Transparency Is a Basic Requirement in Project Evaluation

Enterprises should require any AI consultant or system vendor to provide the following information before the project begins:

• Estimated token usage per query
• Caching strategy and expected cache hit rate
• Context management approach (including truncation mechanism)
• Fallback plan for model providers

If the vendor cannot provide these specific numbers, what they are likely delivering is a demo system rather than a production-grade system.

Further reading: The True Cost Structure of AI Automation: Four Layers Beyond the Quote · ChatGPT API Pricing 2026: Claude and Gemini Comparison with HKD Rates