Seven System Design Principles Every AI Builder Should Borrow
Seven classic system design principles carry directly into AI systems: separation of concerns, explicit interfaces, least privilege, caching, idempotency, graceful degradation, and observability. Applied to AI, they mean one job per agent, schema-defined handoffs, narrow tool permissions, cached prompt prefixes, safe retries, fallbacks when a model fails, and per-step logging of tokens, cost and quality.
Many of the problems teams hit with AI systems were solved decades ago in ordinary distributed systems. The solutions just need translating. These are the seven principles I lean on most.
1. Separation of concerns
Classic: each module does one thing and owns its data.
In AI: each agent or model call has one job and a short prompt about that job. Mixed responsibilities in one prompt produce mixed results.
2. Explicit interfaces
Classic: modules communicate through defined contracts.
In AI: agents hand each other structured objects with a schema, not transcripts. The schema is the contract, and code validates it at every boundary.
3. Least privilege
Classic: each component gets only the access it needs.
In AI: each agent gets only the tools it needs, and code enforces permissions on every tool call. An agent that drafts emails should not hold the tool that sends them.
4. Caching
Classic: do not recompute what has not changed.
In AI: keep prompt prefixes stable so prompt caching applies, and store results of expensive steps so reruns skip them.
5. Idempotency
Classic: repeating an operation has the same effect as doing it once.
In AI: retries are common because models time out or fail checks. Make actions safe to retry: check whether the email was already sent or the record already written before acting again.
6. Graceful degradation
Classic: when a part fails, the system keeps working at reduced capacity.
In AI: when a model call fails or returns invalid output, fall back to another model, a template answer, or a hand-off to a person, instead of breaking the flow.
7. Observability
Classic: logs, metrics and traces for every component.
In AI: log the prompt, the output, tokens, cost, latency, model and check result for every call. When quality drops you can find the step and the input that caused it.
Putting them together
None of these need special AI tooling. They need the habit of treating a model as a component with an interface, a cost and a failure rate, rather than as a magic box at the centre of the system. That habit is what AI system design means, and it is why the model is one component, not the system.
Three more that matter
Loose coupling
Components should depend on each other's contracts, not internals. In AI systems, keep prompts, schemas and tools in your own code so you can swap a model, a provider or a framework without rewriting everything.
Backpressure
When downstream is slower than upstream, slow the input down instead of falling over. Queue model calls, limit concurrency, and back off on rate limits.
Single source of truth
Each fact should live in one place. The CRM owns contact data, the knowledge base owns policy, memory owns preferences. Agents read from those sources through tools instead of carrying copies in prompts that drift out of date.
A system design primer for AI builders
If you are coming to AI from outside software engineering, the classic system design material is worth reading, especially on caching, queues, consistency and failure handling. Skip the parts about scaling databases to millions of users until you need them. The ideas that transfer immediately are the ones above: small parts, clear contracts, narrow permissions, caching, safe retries, fallbacks and logs.
Applying them in practice
| Principle | Question to ask in a design review |
|---|---|
| Separation of concerns | Does any prompt cover more than one job? |
| Explicit interfaces | Is every handoff defined by a schema? |
| Least privilege | Does any agent hold a tool it does not need? |
| Caching | Is the start of every prompt stable? |
| Idempotency | What happens if this step runs twice? |
| Graceful degradation | What happens if the model is down? |
| Observability | Can we see cost and failures per step? |
These questions shaped every system on my systems page. More about my background is on my about page.
Where classic principles need adjusting for AI
Two habits need care. First, determinism: classic systems assume the same input gives the same output, and models do not guarantee that. Design checks and tests around acceptable ranges rather than exact matches. Second, testing: unit tests still matter for code steps, but model steps need test sets and pass rates instead of single assertions. The principles hold; the way you verify them changes.
A one-page design review template
- Job of the system in one sentence.
- Steps, marked code, model or person.
- Schema at each boundary.
- Tools and permissions per agent.
- Cached prefix per prompt.
- Retry, fallback and idempotency per action.
- Logs and metrics per step.
Frequently asked questions
What are the key principles of system design?
Common core principles are separation of concerns, explicit interfaces, least privilege, caching, idempotency, graceful degradation and observability. They make systems easier to change, cheaper to run and safer when parts fail.
Do system design principles apply to AI systems?
Yes, directly. One job per agent, schema-defined handoffs, narrow tool permissions, cached prompts, safe retries, fallbacks and per-call logging are the AI versions of classic principles.
Why is idempotency important for AI agents?
Model calls fail and get retried often. If the actions an agent takes are idempotent, a retry will not send a second email or create a duplicate record.
What is separation of concerns in AI systems?
Giving each agent or model call one job with a short prompt about that job, instead of one prompt that handles intake, research, writing and checking together.
What is graceful degradation for AI?
Planning what happens when a model call fails or returns invalid output: retry, fall back to another model or a template, or hand off to a person, so the overall process keeps working.
How do you test non-deterministic AI steps?
With test sets of real examples and pass rates rather than single exact-match assertions. Code checks verify format and rules on every output, and the pass rate across the set shows whether a change helped or hurt.
Want an AI system designed this way?
I design and build AI systems where each agent has one job, sees only what it needs, and hands clean context to the next. Five are running in production right now. See what I have shipped, or book a call and bring the process you want automated.