LLM System Design: The Model Is One Component, Not the System

When a team says their LLM app is unreliable, they usually mean the model. When I look at the system, the model is almost never the problem. The problem is that the model was treated as the whole application instead of one part of it.

An LLM is a component with unusual properties: it is slow, it costs money per call, its output is not guaranteed to match any format, and it can be confidently wrong. A good LLM system design wraps that component so the rest of the system never has to trust it blindly.

The seven components of an LLM system

1. Input handling. Clean, classify and size the incoming request. Reject or route what the model should not see.
2. Context retrieval. Find the few documents, records or knowledge entries this request needs. This is where most quality comes from.
3. Prompt assembly. Combine stable instructions, retrieved context and the request in a fixed order, stable parts first so they can be cached.
4. The model call. Choose the model for this step, set limits on output length, and ask for structured output where possible.
5. Tool execution. When the model asks to call a tool, code runs it with checked arguments and narrow permissions.
6. Output validation. Code checks the output against a schema and business rules before anything downstream uses it.
7. Storage and monitoring. Log inputs, outputs, token counts, cost, latency and check results for every call.

Only one of the seven is the model. The other six are ordinary engineering, and they decide whether the system holds up.

Design the data flow before the prompt

I draw the flow as boxes before writing a word of prompt. For each box I write three things: what goes in, what comes out, and what happens when it fails. For the model box, the failure column is long: wrong format, missing field, hallucinated value, refusal, timeout. Each failure needs a planned response, usually a retry with the error message, a fallback model, or a hand-off to a person.

This exercise also shows where you do not need a model. Date parsing, routing on a known field, totals and lookups are code. Every step moved from model to code is faster, free and deterministic.

Synchronous or queued

A chat reply has to be fast, so the path from request to answer must be short: one retrieval, one model call, one check. Anything slower, such as generating a report, crawling a website or writing ten pages, belongs in a background queue with a job record the user can poll. Mixing the two is a common cause of timeouts. Decide per feature which path it takes.

Where LLM systems usually break

• No output validation. The app assumes the JSON is valid and crashes the first time it is not.
• Unbounded context. Conversation history grows until every call is slow and expensive. Summarise or trim it.
• No per-step logging. When quality drops nobody can tell whether retrieval or the model changed.
• One model for everything. The largest model runs simple classification calls all day.

Each of these is a design gap, not a model gap. The principles of AI system design cover the fix for each, and if the work splits naturally into separate jobs, see how to split work across agents.

A minimal production checklist

• Every model output is parsed and validated by code before use.
• Every prompt has a stable prefix and a variable suffix.
• Every call logs model, input tokens, output tokens, cost and latency.
• Every step has a timeout, a retry limit and a fallback.
• Conversation history is capped, with older turns summarised.
• There is a spend cap per user and per day.

A worked example: a support reply flow

Here is how the seven components map onto a support reply feature I would build today.

1. Input handling: strip signatures and quoted history from the email, detect language, reject attachments over a size limit.
2. Retrieval: find the customer's account and last three tickets in the helpdesk, and the two policy entries that match the classified topic.
3. Prompt assembly: stable instructions first, then policies, then the account summary, then the email.
4. Model call: ask for JSON with a reply, a confidence level and a list of policy codes used.
5. Tools: the model may request an order lookup; code runs it with the customer's ID only.
6. Validation: JSON parses, the reply is under 250 words, every policy code exists, no refund promised above the allowed amount.
7. Logging: tokens, cost, latency, the policy codes used and check results, per reply.

Low-confidence replies and any failed check go to a person as a draft. Everything else can be sent automatically once the logs show the pass rate is stable.

What to monitor once it is live

• Check pass rate per step. A drop points at the step that changed.
• Cost per completed task, not cost per call. Retries hide in per-call numbers.
• Latency at the slow end. The slowest five percent of requests is what users complain about.
• Human override rate. How often a person edits or rejects the draft is the most honest quality metric you have.

From the build side

Most of my work on AI application development is in the six components that are not the model. It is also why the systems on my systems page keep running after launch: the model can be swapped, the surrounding system stays.

LLM system design interview questions, answered from the build side

If you are here preparing for an LLM system design interview, the structure above is a strong answer frame. Interviewers usually probe the same areas:

• How do you stop hallucinations? Ground answers in retrieved sources, ask for citations to entry codes, and check claims against the source with code where possible.
• How do you control cost? Smaller models per step, less context per call, cached prefixes, capped output, and spend limits per user.
• How do you handle scale? Queues for background work, rate-limit aware concurrency, and separate budgets for interactive and batch traffic.
• How do you know it works? Test sets per step, code checks on every output, and production logs of pass rates.

The honest answer to most of them is the same: treat the model as one component and design everything around it.

Frequently asked questions

What is LLM system design?

LLM system design is the architecture of an application built around a large language model. It covers input handling, retrieval, prompt assembly, model selection, tool execution, output validation, storage and monitoring, with the model treated as one component inside a normal software system.

What are the main components of an LLM application?

Input handling, context retrieval, prompt assembly, the model call, tool execution, output validation, and logging with monitoring. The model is one of these. The others decide most of the system's reliability and cost.

How do you make an LLM application reliable?

Validate every output with code, keep context small and relevant, log every call with cost and latency, set timeouts and retries with fallbacks, and move any step that can be written as a rule out of the model and into code.

How do you handle invalid JSON from an LLM?

Parse and validate every response against a schema. On failure, retry once with the validation error included in the prompt, and if it fails again, fall back to another model or route the item to a person.

What should I log for each LLM call?

The step name, model, prompt version, input tokens, cached tokens, output tokens, cost, latency, check results and an ID linking the call to the user task it belongs to.

How do you prevent hallucinations in an LLM application?

Retrieve relevant sources and instruct the model to answer only from them, ask it to cite the source entries it used, validate claims against those sources with code where possible, and route low-confidence answers to a person.