RAG Tutorial for Developers: Build a Reliable Retrieval-Augmented Generation Workflow

Overview

Retrieval-augmented generation, or RAG, connects a language model to a collection of external documents. Instead of asking the model to answer entirely from its training data, an application first searches a knowledge base, places relevant passages in the prompt, and asks the model to answer using that context.

A practical RAG pipeline usually contains six stages:

1. Ingestion: collect documents and extract usable text.
2. Preparation: clean text, preserve useful metadata, and remove duplicates.
3. Chunking: divide documents into searchable passages.
4. Indexing: create embeddings and store them in a vector index or compatible search system.
5. Retrieval: find passages related to a user’s question.
6. Generation: provide the selected passages to an LLM and produce an answer, ideally with citations.

RAG does not automatically make answers accurate. A model can still receive the wrong passages, misunderstand a relevant passage, or answer beyond the evidence. Reliability comes from treating retrieval, prompt design, structured output, and evaluation as separate engineering concerns. For prompt construction, see this practical prompt engineering guide. If your application must return machine-readable results, pair retrieval with structured output prompts.

How to estimate

Estimate a RAG system in layers rather than trying to assign one number to the entire application. This makes the calculation easier to update and helps reveal which design decision is driving the result.

1. Estimate the indexed corpus

Start with the amount of text that will be searchable. Record the number of source files, average document length, expected document growth, and how often documents will be reprocessed. A simple planning formula is:

Indexed text = current text + expected new text during the planning period

Then estimate the number of chunks:

Chunk count ≈ total text units ÷ effective chunk size

The exact unit may be characters, words, or tokens, depending on the tools in your stack. Overlap increases the number of stored text units, so include it in the estimate rather than treating it as free capacity.

2. Estimate embedding work

Embedding work is usually tied to the number of chunks processed. Separate the initial indexing workload from ongoing updates:

Total embedding workload = initial chunks + changed or new chunks

If documents are frequently edited, avoid re-embedding the entire corpus by tracking document identifiers, versions, and content hashes. Re-embed only the chunks affected by a change when your ingestion design supports that approach.

3. Estimate query work

For query traffic, record expected questions per day, average retrieval attempts per question, and the number of passages returned per attempt. A basic workload estimate is:

Monthly retrieval operations = questions per day × active days × retrieval attempts per question

Generation workload should be estimated separately because the final prompt may contain the user question, system instructions, conversation history, and retrieved context. Track both input and output tokens, or the equivalent usage unit required by your model provider.

4. Estimate total operating cost

Use a worksheet with separate rows for embedding, vector storage, document storage, retrieval infrastructure, model input, model output, logging, monitoring, and scheduled re-indexing. The general formula is:

Total cost = fixed infrastructure + indexing cost + query retrieval cost + generation cost + observability cost

Do not fill in provider prices until you have selected the services and region. Instead, keep each price as an editable input. This prevents an old estimate from looking precise after pricing, traffic, or model choices change.

Inputs and assumptions

A useful RAG estimate depends on assumptions that can be inspected and challenged. Document these inputs in version control or in the same project workspace as the pipeline.

• Corpus size: number of files, estimated text volume, formats, and expected growth.
• Chunk size: the target size of each passage and the amount of overlap.
• Metadata: fields such as title, section, author, date, product, access level, and source URL.
• Update frequency: how often files change and whether updates are incremental or full re-indexes.
• Query volume: expected questions per day, peak traffic, retries, and multi-step retrieval.
• Retrieval depth: the number of candidate and final passages returned for each question.
• Prompt size: system instructions, conversation history, retrieved context, and response budget.
• Quality target: whether the application prioritizes recall, concise answers, citations, low latency, or strict abstention.
• Access controls: whether retrieval must filter documents by user, team, tenant, or permission level.

Chunking should follow the structure of the source material where possible. A policy section, API method, or troubleshooting procedure is often more useful as a complete unit than an arbitrary slice of text. Test several chunk sizes on representative questions instead of assuming that smaller chunks are always better.

Use metadata filters before semantic search when the user’s request contains a clear constraint, such as a product, date range, or document type. Hybrid retrieval can also be useful when exact terms, identifiers, or code symbols matter as much as semantic similarity. The right approach depends on the corpus and query patterns, so record the choice as an experiment rather than a permanent rule.

For evaluation, create a small dataset of realistic questions with expected source documents, acceptable answers, and known unanswerable cases. The guide to creating evaluation datasets can help organize this process. Review retrieval and generation separately: a correct answer based on an irrelevant passage may be fragile, while a wrong answer from a clearly relevant passage points to prompting or model behavior.

Worked examples

Example A: internal documentation assistant

Assume an internal knowledge base contains product documentation, operating procedures, and support notes. The team expects a moderate number of daily questions and updates a portion of the documents each week.

Begin with these worksheet fields:

1. Current document count and estimated text volume.
2. Target chunk size and overlap.
3. Percentage of documents changed during an update cycle.
4. Questions per day and average retrieved passages per question.
5. Average prompt input and response output size.
6. Storage, embedding, retrieval, and model rates supplied by the selected providers.

If only changed documents are re-indexed, the recurring embedding estimate is based on the changed portion rather than the full corpus. If every answer includes several long passages, generation input usage may become the dominant variable. The team can then compare two options: return fewer, better-ranked passages, or use a reranking step to improve selection before generation. Measure answer quality and latency for both options before choosing.

Example B: frequently updated support knowledge base

Now assume the corpus is smaller but changes throughout the day. A full re-index may be simple to implement, yet its repeated embedding workload could exceed the workload of a larger but mostly static corpus. Estimate each update pattern separately:

Update workload = documents changed per cycle × cycles per day × active days

Track failed ingestion jobs, duplicate content, and documents that have been removed. A deletion that is not propagated to the index can produce stale answers even when the generation prompt is well designed. Include a deletion and replacement test in the evaluation set.

For both examples, add a fallback rule: when retrieval scores are weak, sources conflict, or no passage answers the question, the application should say that the available knowledge is insufficient or ask a clarifying question. This is generally safer than filling gaps with an unsupported response.

When to recalculate

Recalculate the RAG estimate whenever a major input changes, not only when an invoice arrives. Review it when document volume grows, chunking changes, traffic increases, a model or embedding service changes, retrieval depth is adjusted, or the application adds conversation history and tool calls.

Also revisit the design after evaluation exposes a quality problem:

• Poor retrieval: inspect parsing, chunk boundaries, metadata, query rewriting, filters, and ranking.
• Hallucinated answers: tighten the grounding instructions, require source references, and test abstention behavior.
• Stale knowledge: add version tracking, update monitoring, and deletion handling.
• High latency: measure ingestion, search, reranking, model time, and network overhead independently.
• Rising cost: compare prompt size, retrieval depth, update frequency, retries, and model selection before reducing quality blindly.

Keep a lightweight change log containing the date, assumptions, measured query volume, quality results, and provider inputs used in each estimate. This turns the calculation into an operational tool rather than a one-time planning document. Finally, test every change against the same evaluation dataset and review a sample of real interactions with sensitive information handled appropriately. RAG systems become more dependable when retrieval quality, prompt behavior, and cost are measured together.

For the next implementation step, define one narrow use case, prepare a representative document sample, build the smallest ingestion and retrieval loop, and record baseline results before adding complexity. You can then use a prompt testing framework to compare grounding instructions and retrieval settings with repeatable evidence.