AI System Design: Classic System Design Principles, Applied to AI

Most AI projects I am asked to fix have the same shape. One model, one enormous prompt, every document the company owns pasted into it, and a hope that the model will sort it out. It works in the demo. Then the bill arrives, the answers drift, and nobody can tell which part of the prompt caused which mistake.

The fix is rarely a better model. It is the same thing that fixed this problem in ordinary software forty years ago: design the system before you write the prompt.

What classic system design actually teaches

Strip away the interview questions about load balancers and the discipline of system design comes down to a handful of principles that have held for decades:

• Separation of concerns. Each part does one job and owns its own data.
• Clear interfaces. Parts talk through small, explicit contracts, not by reaching into each other.
• Load only what you need. Indexes, lazy loading and pagination exist because reading everything is slow and expensive.
• Cache what repeats. Work that produces the same result twice should be paid for once.
• Isolate failure. One broken part should not take the whole system down, and you should be able to see which part broke.
• Measure, then optimise. You cannot fix latency or cost you have not instrumented.

None of these were invented for AI. All of them apply to it directly, and they map onto the problems people hit with language models almost one to one.

The same principles, applied to AI

Why one big prompt fails

A model reads every token you send it on every call. If your prompt carries 40,000 tokens of policy, product data and examples, you pay for all 40,000 on every turn, even when the user asked a question that needed 300 of them. Worse, the useful 300 are now buried. Anthropic's own engineering team describes context as a finite resource with diminishing returns, and recommends finding the smallest set of high-signal tokens that gets the job done.

The one-prompt design also makes failure invisible. When the output is wrong you cannot tell whether the model misread the instructions, missed the relevant document, or reasoned badly. Everything happened in one call, so there is one place to look and no way to see inside it.

What a designed AI system looks like

Here is the shape I use for a landing page builder I run. The model never sees the whole knowledge base. It sees a short core prompt, about 550 words, that is always loaded and never changes, which makes it cheap to cache. Everything else lives in a knowledge base of roughly 850 entries, each tagged with a code and a one line "use when" description, listed in a single index file.

When the builder needs to write a hero section, it searches the index, pulls the two or three entries about hero sections, and works from those. A small, fast model handles the intake conversation with the customer and hands a structured brief to a larger model that does the building. The large model never pays to read the small talk. What each brand told it last time sits in a memory store and comes back as a short summary, not as the old transcript.

Every part has one job. Every handoff is a small structured object. Nothing is loaded unless the current step needs it.

The design questions to answer before writing a prompt

1. What are the separate jobs? Write them down as verbs: classify, retrieve, draft, check, send. Each verb is a candidate for its own step.
2. Which jobs need a model at all? Anything you can write as a rule should be code. Code is free, instant and never hallucinates.
3. What does each step need to see? List the inputs per step. If a step's list says "everything", it is two steps.
4. What does each step hand on? Define the output as a schema. If you cannot write the schema, the step is not defined yet.
5. Which model is the cheapest one that passes? Classification and extraction rarely need the largest model. Planning and writing often do.
6. How will you know a step failed? Decide the code check before you write the prompt.

What it buys you

The gains are boring and large. Token spend usually drops by more than half, because each call carries a fraction of the context. Latency drops because small models handle the small steps. Quality goes up because each prompt is about one thing and the model is not choosing between forty instructions. And when something breaks, the logs tell you which step, with which input, produced the bad output.

The rest of this series takes each principle in turn: how to split work across agents, context engineering, memory, routing, caching and evaluation, with the patterns I use in production.

A worked example: the same job, designed two ways

Take a common request: answer inbound sales emails using the company's product catalogue, pricing rules and past conversations with that customer. Here is the same job built two ways. The numbers are illustrative, but the proportions are typical of what I see.

Design A: one prompt

One large model. The prompt carries the full catalogue, every pricing rule, the last twenty emails with the customer and the new email. Call it 30,000 input tokens per reply. At 500 replies a day that is 15 million input tokens a day on the most expensive model tier, and when a reply quotes the wrong price nobody can tell whether the rule was missing, buried or misread.

Design B: a designed system

1. Code finds the customer in the CRM and pulls their plan and open deals. No model.
2. A small model classifies the email: pricing question, support issue, meeting request, unsubscribe. A few hundred tokens.
3. Code fetches only the catalogue entries and pricing rules for the products mentioned, plus a short memory summary of the customer.
4. A large model drafts the reply from that brief. Perhaps 3,000 input tokens, most of them a cached, stable instruction prefix.
5. Code checks every price in the draft against the price table and blocks the send if one does not match.

Design B sends roughly a tenth of the tokens to the expensive model, runs the cheap classification on a cheap model, and catches the most costly error, a wrong price, with a check that costs nothing. When something does go wrong, the log shows which step and which input.

Common mistakes when teams first try this

• Splitting too far. Ten agents for a five-step job. Every boundary adds a schema, a check and latency. Split only where context, model size, timing or permissions differ.
• Splitting the prompt but not the context. Three agents that all receive the full history are one agent with a bigger bill.
• Leaving rules in the model. Asking a model to calculate a discount or check a date range when one line of code would do it exactly.
• No logging per step. The design only pays off if you can see cost and failure per step.

How I apply this in my own systems

Every system on my systems page is built on these principles. The landing page builder separates a small intake model from a large builder model and keeps its knowledge in an indexed base the builder searches. The self-selling conversational AI bot is assembled by a pipeline in which code crawls a website, a model turns it into a structured knowledge base, and code creates the voice agent through an API. The content engine runs research, angle selection and writing as separate steps with their own context. Same principles, different jobs. You can read more about me and how I work.

Frequently asked questions

What is AI system design?

AI system design is the practice of structuring an AI application as separate parts with one job each, clear structured handoffs, and only the context each part needs. It applies classic system design principles such as separation of concerns, caching and failure isolation to models, prompts, tools and agents.

How is AI system design different from prompt engineering?

Prompt engineering improves the wording of a single call to a model. AI system design decides how many calls there are, what each one sees, which model runs it, how outputs are checked, and how results move between steps. Good prompts matter, but the design usually decides cost and reliability.

Does AI system design reduce token costs?

Yes. When each step loads only the context it needs, prompts shrink, stable instructions can be cached, and small steps can run on cheaper models. In practice a designed system often uses less than half the tokens of a single large prompt doing the same work.

Do I need multiple agents to use AI system design?

No. Many well designed AI systems use one model with a good retrieval index, structured outputs and code checks. Splitting into multiple agents only makes sense when the jobs need different context, different models or can run in parallel.

What are the core principles of AI system design?

Separation of concerns, clear structured interfaces between steps, loading only the context each step needs, caching stable prompt content, isolating failures with code checks, and measuring cost and quality per step before optimising.

Where should I start with AI system design?

Write the process as a list of verbs, mark which steps are rules for code and which need judgement, define the input and output of each step, and only then write prompts for the steps that need a model.