LLM Cost Optimization: Cut Tokens by Design, Not by Cutting Corners

Most AI bills I review have the same profile: a large model, a long prompt, called on every step, with no record of which step costs what. The fix is not a cheaper provider. It is design.

1. Measure cost per step

Log model, input tokens, output tokens and cached tokens for every call, tagged by step. Within a day you will see that two or three steps account for most of the spend. Optimise those and ignore the rest.

2. Send less context

Input tokens are usually the majority of cost, because every call resends instructions, history and documents. Retrieve a few precise entries instead of whole documents, summarise old turns, and trim tool results. This is context engineering, and it is the biggest lever.

3. Route to smaller models

Classification, extraction, formatting and intake conversations rarely need a large model. Route them to a small one and test the pass rate.

4. Cache stable prefixes

Put instructions and tool definitions first and keep them identical between calls so prompt caching bills them at a discount.

5. Cap and shape output

Output tokens cost more than input tokens on most models. Ask for structured output with only the fields you need, set a maximum length, and do not ask the model to explain its reasoning in the response unless something reads it.

6. Batch what can wait

Many providers offer batch processing at a discount for work that does not need an immediate answer, such as nightly enrichment, bulk classification or content drafts. Move anything non-interactive there.

7. Replace model calls with code

Every step that can be written as a rule should be code: date parsing, routing on a known field, arithmetic, lookups, validation. Code costs nothing per run.

What not to cut

• Checks. Removing validation to save a call means bad outputs reach customers.
• The model on the hardest step. If the planning step needs a large model, keep it and save elsewhere.
• Logging. Without it you cannot tell whether a saving hurt quality.

A realistic result

Applying the first five moves to a single-prompt system often cuts spend by half or more, with equal or better quality, because each call is smaller and more focused. The principle underneath is the one in AI system design: every token should be there for a reason.

How to calculate LLM cost per task

For each step: (input tokens minus cached tokens) times input price, plus cached tokens times cache price, plus output tokens times output price. Add the steps of one task, including retries, and you have cost per task. Multiply by tasks per month for the budget. Doing this once per system usually changes which optimisation looks most important.

A worked example

An illustrative support agent before optimisation: one large-model call per reply, 25,000 input tokens, 600 output tokens, no caching. After optimisation:

• Classification moved to a small model: a few hundred tokens.
• Knowledge loaded by index instead of in full: input on the main call down to about 4,000 tokens.
• Stable instructions cached: most of those 4,000 billed at the cache rate.
• Output capped with a schema: about 300 tokens.

The cost per reply drops to a fraction of the original, and replies are more consistent because each call is focused.

LLM token cost: the levers ranked

How I keep costs down

Cost per task is a design target on every system on my systems page, because most of them run at volume. The principles of AI system design are, in large part, a cost discipline.

LLM API cost comparison: compare per task, not per token

Comparing providers by price per million tokens is misleading. A cheaper model that needs two retries, or a longer prompt to behave, can cost more per finished task. Run your real test set on each candidate, record total tokens including retries, and compare cost per passed task. That number, not the price list, decides which model is cheaper for you.

Set budgets, not just targets

• A spend cap per run, so one runaway task cannot burn the budget.
• A daily cap per user or account.
• Alerts when a step's average cost rises above its baseline.

Frequently asked questions

How can I reduce LLM costs?

Measure cost per step, send less context per call, route simple steps to smaller models, cache stable prompt prefixes, cap output length, batch non-urgent work, and replace model calls with code where a rule works.

What uses the most tokens in an LLM application?

Usually input context resent on every call: long system prompts, conversation history, retrieved documents and tool output. Trimming and caching these usually saves more than any other change.

Does using a smaller model reduce quality?

Not for every task. Classification, extraction and formatting often pass at the same rate on small models. Test each step on a set of real examples before deciding, and keep large models for steps that need them.

How is LLM cost calculated?

Per call: uncached input tokens times the input price, plus cached tokens times the cache price, plus output tokens times the output price. Sum all calls in a task, including retries, to get cost per task.

Are output tokens more expensive than input tokens?

On most models, yes, often several times more per token. Capping output length and asking for only the fields you need is an easy saving.

How do I compare LLM API costs fairly?

Run the same real test set on each model and compare cost per passed task, including retries and the prompt length each model needs, rather than comparing list prices per million tokens.