Generative AI System Design: A Practical Checklist Before You Build
Generative AI system design is planning how a system that generates text, images or actions will be built: what it is for, how work is split into steps, what context each step sees, which models run where, how outputs are checked, what safety limits apply, what it will cost, and how it will be rolled out. A short checklist answered before building prevents most production problems.
Generative AI projects tend to start with a prompt that works in a chat window. The system gets built around that prompt afterwards. This checklist reverses the order. Answer it first, and the prompts become much easier to write.
Scope
- What exact job does the system do, in one sentence?
- Who uses the output, and what do they do with it?
- What does a wrong output cost? A typo, a lost lead, a legal problem?
- What volume is expected per day?
Steps
- What are the steps, written as verbs?
- Which steps are rules that code should do?
- Which steps need judgement or generation?
- Can any steps run in parallel?
Context
- What does each model step need to see, and nothing more?
- Where does that knowledge live, and how is it indexed?
- What is remembered between sessions, and how is it forgotten?
- Which parts of each prompt are stable enough to cache?
Models
- Which is the smallest model that passes each step on a test set?
- What is the fallback if a model is down or fails a check?
Validation
- What schema does each output follow?
- What code checks run on each output?
- Where does a person review or approve, and for how long?
Safety and permissions
- Which tools can each agent call?
- Which actions are irreversible, and are they gated?
- How is one customer's data kept out of another's context?
- What are the step limits and spend caps per run and per day?
Cost
- Tokens per call, calls per run, runs per day, price per token: what is the monthly estimate?
- Which steps can be batched at a discount?
Operations
- What is logged per call?
- How will you know quality has dropped?
- What is the rollout: shadow mode, approval mode, then autonomous?
Each section of this checklist maps to a principle covered in this series, starting with AI system design and LLM cost optimization.
A worked example: answering the checklist for a content system
| Question | Answer |
|---|---|
| Job in one sentence | Turn a weekly topic into one SEO article and three social posts in the brand's voice. |
| Cost of a wrong output | Embarrassment or a false claim in public, so a person approves before publishing. |
| Steps | Research, angle choice, write article, write social posts, check, approve, publish. |
| Rule steps | Scheduling, formatting, publishing, length and heading checks. |
| Context per step | Research gets the topic; writer gets the angle, research summary and brand voice entries. |
| Models | Mid model for research summaries, large model for the article, small model for social posts. |
| Validation | Schema, word counts, heading structure, banned claims list, links resolve. |
| Rollout | Approval required on every piece for the first month. |
Answering takes an hour. Skipping it costs weeks of rework when the first draft of the system turns out to be one giant prompt.
GenAI system design interview vs real projects
If you came here preparing for a generative AI system design interview, the checklist above is also a good answer structure: clarify the job and constraints, break it into steps, design context and retrieval, choose models per step, explain validation and safety, estimate cost, and describe rollout and monitoring. Real projects add one thing interviews rarely cover: the integrations, which take most of the build time.
How I use it
I run this checklist at the start of every project. The content engine on my systems page is close to the worked example above.
Generative AI architecture: the minimum viable version
For a first release, the smallest sound generative AI architecture has five parts: an input handler, a retrieval step over a labelled knowledge base, one model step with structured output, a code check, and a log. Approval by a person sits between the check and any public or customer-facing output. Everything else, such as routing, multiple agents and memory, can be added when the logs show it is needed.
Red flags during design
- Nobody can say what a wrong output costs.
- The plan has no test set.
- Every step uses the largest model.
- Outputs go straight to customers from day one.
- There is no estimate of monthly cost.
Any one of these is fixable in an afternoon at design time and expensive to fix after launch.
Frequently asked questions
What is generative AI system design?
It is the planning of a system that uses generative models: defining its job, splitting work into steps, designing context and retrieval, choosing models, validating outputs, setting safety limits, estimating cost and planning rollout.
What should I decide before building a generative AI system?
The exact job, the steps and which need a model, what each step sees, which model runs each step, how outputs are checked, what actions need approval, the cost estimate and the rollout plan.
How do you roll out a generative AI system safely?
Start in shadow mode where outputs are logged but not used, move to approval mode where a person signs off, and only then run autonomously on steps whose logs show they are reliable.
How do you answer a generative AI system design question?
Clarify the job and constraints, break the task into steps, decide which steps use models, design the context and retrieval for each, choose models per step, explain validation and safety, estimate cost, and describe rollout and monitoring.
What is the biggest mistake in generative AI system design?
Building the whole system around a single large prompt, which makes cost high, quality inconsistent and failures impossible to trace to a step.
What is the simplest generative AI architecture for a first release?
An input handler, retrieval over a labelled knowledge base, one model step with structured output, a code check, logging, and human approval before anything reaches customers.
Want an AI system designed this way?
I design and build AI systems where each agent has one job, sees only what it needs, and hands clean context to the next. Five are running in production right now. See what I have shipped, or book a call and bring the process you want automated.