ML System Design vs AI System Design: What Changes When You Stop Training Models

Ask for ML system design material and you get a well established curriculum: data collection, labelling, feature engineering, training pipelines, offline evaluation, serving infrastructure and drift monitoring. It is excellent, and most of it does not apply to the AI systems most businesses are building now.

Those systems do not train a model. They call one through an API. The design problem moves from "how do we produce a good model" to "how do we get good, cheap, reliable work out of a model we did not build".

What ML system design focuses on

• Collecting and labelling training data.
• Feature pipelines and feature stores.
• Training, tuning and versioning models.
• Offline metrics such as precision and recall.
• Serving models at low latency and high throughput.
• Detecting data drift and retraining.

What AI system design focuses on

• Choosing which hosted model runs each step, and routing between them.
• Context engineering: what the model sees on each call.
• Retrieval from knowledge bases and business systems.
• Tool design and permissions.
• Splitting work across agents with structured handoffs.
• Validating outputs with code.
• Token cost, caching and spend limits.

Side by side

Where they overlap

Both disciplines insist on evaluation before release and monitoring after it. Both treat the model as one component in a larger system. Both punish teams that skip logging. If you come from ML, your instincts about test sets, baselines and monitoring transfer directly. What is new is that the model is fixed and the levers are everything around it.

Which one do you need?

If you have a narrow, high-volume prediction task with lots of labelled data, such as fraud scoring or demand forecasting, ML system design is the right frame. If you are automating work that involves reading, writing, deciding or talking, such as support, sales, document processing or content, you are doing AI system design. Most business automation I build is the second kind, and the principles that matter are the ones in this overview.

The same problem, solved both ways

Consider routing support tickets to the right team.

The ML system design approach

Collect tens of thousands of past tickets with their final team as the label. Train a classifier. Build a feature pipeline, evaluate precision and recall per team, deploy a model server, monitor for drift as products change, and retrain every quarter. Excellent at very high volume with stable categories.

The AI system design approach

Write a short prompt describing each team's scope, give a small hosted model the ticket, and ask for a team name from a fixed list in a schema. Validate the output with code. Build a test set of a hundred real tickets to measure accuracy. When a new team is created, edit the prompt. Live in days, and good enough for most volumes.

Which wins?

At millions of tickets a day with stable categories, the trained classifier is cheaper per item. At thousands a day with categories that change, the hosted model approach is cheaper to build and maintain. Many mature systems end up with both: a trained model for the bulk and a language model for the hard or new cases.

Skills that transfer

• Evaluation discipline: test sets, baselines and regression checks apply directly.
• Data quality instincts: retrieval quality is a data problem.
• Monitoring: watching for quality drops after a change is the same habit as watching for drift.

Skills that are new

• Designing context per call and keeping it small.
• Writing tools and their permissions.
• Splitting work across agents with structured handoffs.
• Managing token cost, caching and routing.

I work almost entirely on the AI system design side, building on hosted models. You can see the results on my systems page and read what an AI developer actually does.

ML system design interview vs AI system design interview

An ML system design interview usually asks you to design a recommender, a ranking system or a fraud detector: data, features, model choice, training, offline and online evaluation, serving and monitoring. An AI or LLM system design interview more often asks you to design a support agent, a document assistant or a coding helper: retrieval, context, tools, agent structure, guardrails, evaluation and cost. Prepare for whichever the role is actually about, and expect overlap on evaluation and monitoring.

A combined architecture

Some of the best systems use both. A trained classifier scores a million records cheaply; a language model handles the few thousand edge cases the classifier is unsure about, with richer reasoning and an explanation. Design the hand-off between them like any other boundary: a confidence threshold, a schema, and logs of which path each record took.

Frequently asked questions

What is ML system design?

ML system design is the architecture of systems that train and serve machine learning models: data pipelines, feature stores, training, evaluation, deployment and drift monitoring.

How is AI system design different from ML system design?

AI system design usually builds on hosted foundation models instead of training new ones. Its levers are model selection, context engineering, retrieval, tool use, agent orchestration, output validation and token cost, rather than training data and model training.

Do I need to train a model to build an AI system?

Usually not. Most business AI systems use hosted models through APIs, with quality coming from good context, retrieval and checks. Training or fine-tuning is worth it only for narrow, high-volume tasks with plenty of labelled data.

Is ML system design still relevant with LLMs?

Yes. Many high-volume prediction tasks are still cheaper and more accurate with trained models, and the evaluation and monitoring habits of ML system design apply directly to LLM systems.

What should an ML engineer learn to build AI agents?

Context engineering, tool design and permissions, agent orchestration patterns, structured outputs and validation, and token cost management with caching and model routing.

Can ML models and LLMs be used together?

Yes. A common design uses a cheap trained model for high-volume scoring and sends only uncertain or complex cases to a language model, with a confidence threshold deciding which path each item takes.