RAG System Design: Retrieval Is an Index Problem Before It Is an AI Problem

Retrieval-augmented generation sounds like an AI technique. Most of it is information architecture. When a RAG system gives bad answers, the cause is almost always that it retrieved the wrong text, or too much text, before the model ever started writing.

The four parts of a RAG system

1. Chunking. Splitting source material into pieces that each answer one kind of question.
2. Labelling. Giving each chunk metadata: topic, type, date, source, and when it should be used.
3. Indexing. Making chunks findable, by vector similarity, keywords, metadata filters, or a mix.
4. Retrieval and assembly. Fetching a few chunks for the request and placing them in the prompt.

Chunk by meaning, not by length

The default in many tutorials is to cut documents every 500 tokens. That splits answers in half and glues unrelated topics together. Chunk along the structure of the content: one chunk per topic, rule, product or procedure. A chunk should make sense on its own.

Label every chunk with when to use it

The most useful change I have made to a knowledge base was adding a one-line "use when" field to every entry. In the page builder I run, each of roughly 850 entries has a code, a tag saying whether it is constant, brand-specific or time-sensitive, and a "use when" line. The model searches those lines, not the full text. It finds the right entry far more reliably than similarity search over raw paragraphs, and the index is small enough to read cheaply.

Use more than vectors

Vector search is good at fuzzy matches and bad at exact ones. Product codes, names and dates are better found by keyword or metadata filter. Most production retrieval I build is hybrid: filter by metadata first, then rank by keywords and vectors together.

Retrieve less

Sending the top twenty chunks feels safe and is usually worse. The model has to sort through them, the relevant one gets diluted, and you pay for all twenty. Retrieve two to five well-labelled chunks. If the answer is not in them, let the model ask for more through a search tool. That is the idea behind agentic RAG: retrieval becomes a tool the model calls when it needs to, rather than a fixed dump before every answer.

Mark what expires

Prices, policies and platform rules change. Tag time-sensitive chunks with a verified date and a source, and re-check them on a schedule. Stale retrieval produces confident wrong answers, which is worse than no answer.

A RAG design checklist

• Chunks follow content structure and stand alone.
• Every chunk has metadata and a "use when" description.
• Retrieval combines metadata filters, keywords and vectors.
• Few chunks per request, with a search tool for more.
• Time-sensitive chunks carry dates and are re-verified.
• Retrieval is logged so you can see what was fetched for each answer.

RAG is one form of context engineering, and the same rule from AI system design applies: load only what you need.

A worked example: fixing a support bot's retrieval

A support bot answered about seventy percent of questions correctly. Its knowledge was the help centre, cut into 500-token chunks, searched by vector similarity, top ten chunks into the prompt. The fixes, in order of impact:

1. Re-chunked by article section, so each chunk answered one question.
2. Added metadata: product, plan, and a "use when" line written for each chunk.
3. Filtered by the customer's product and plan first, then searched within that subset.
4. Added keyword search alongside vectors for error codes and feature names.
5. Cut retrieval to the top three chunks and gave the model a search tool for more.

Accuracy rose and the prompt shrank. None of the changes touched the model or the prompt wording.

RAG architecture at a glance

Where I use it

The self-selling conversational AI bot on my systems page builds its knowledge base by crawling a business's website and having a model extract services, pricing, hours and FAQs into structured entries, which is chunking and labelling done at setup time. The page builder uses an indexed base of around 850 entries, searched by "use when" lines.

Agentic RAG system design

In agentic RAG, retrieval becomes a set of tools: search the index, fetch an entry by ID, list a section, filter by metadata. The agent decides when to call them and can search again with a better query if the first results miss. Design the tools to return compact results with IDs, so the agent can cite exactly what it used. Cap the number of searches per answer so a vague question cannot trigger a hundred lookups.

Evaluating retrieval separately

Build a small set of questions with the entries that should be retrieved for each. Measure whether those entries appear in the results, separately from whether the final answer is good. When answer quality drops, this tells you immediately whether retrieval or generation is at fault.

Frequently asked questions

What is RAG system design?

RAG system design is the architecture of retrieval-augmented generation: how knowledge is chunked, labelled and indexed, how relevant chunks are retrieved for each request, and how they are passed to a language model to ground its answer.

Why do RAG systems give wrong answers?

Usually because retrieval fetched the wrong chunks or too many loosely related ones. Poor chunking, missing metadata, vector-only search and stale content are the common causes, more often than the model itself.

What is agentic RAG?

Agentic RAG gives the model a search tool so it can retrieve knowledge when it decides it needs it, and search again if the first results are not enough, instead of receiving a fixed set of chunks before every answer.

What is the best chunk size for RAG?

There is no single best size. Chunk by meaning: one section, rule, product or procedure per chunk, so each chunk can answer a question on its own. Fixed-length chunking often splits answers and mixes topics.

Should RAG use vector search or keyword search?

Usually both. Vector search handles paraphrased questions, keyword search handles exact terms like codes and names, and metadata filters narrow the candidates before either runs.

How do you evaluate a RAG system?

Measure retrieval and generation separately. For retrieval, check whether the expected entries appear in the results for a set of test questions. For generation, check whether answers are correct and supported by the retrieved entries.