What Is RAG? Retrieval-Augmented Generation Explained Simply
RAG (retrieval-augmented generation) makes an AI system search a set of documents first and answer from what it finds. Named in a 2020 paper by Lewis et al.
Short answer
RAG (retrieval-augmented generation) is a technique in which an AI system first searches a set of documents for passages relevant to a question, then writes its answer from those passages rather than from memory. The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research. It keeps answers current and checkable without retraining, but cuts hallucinations without ending them. PepoChat is a RAG system for customer support, free for 100 AI replies a month.
You have probably seen the acronym on a product page or in a meeting: "our chatbot uses RAG". It is one of the most used terms in applied AI, and it names a simple idea that is easy to get wrong.
This guide is for business owners, support leads, students and anyone who wants the plain version. You will learn what the letters stand for and where they came from, how a RAG system answers a question step by step, how it compares with fine-tuning and long context windows, where it fails, and what to check before you trust one.
What is RAG in AI?
RAG in AI means retrieval-augmented generation: before a language model answers, a search step finds the most relevant passages in a chosen set of documents and adds them to the model's instructions. The model then writes its reply from that material. Retrieval finds, generation writes, and the documents can change without touching the model.
| Term | Stands for | In one sentence | Everyday example | Not the same as |
|---|---|---|---|---|
| RAG | Retrieval-augmented generation | Search first, then answer from what was found | A help-center bot quoting your delivery times | Fine-tuning, which changes the model itself |
| LLM | Large language model | The model that writes the answer | The model behind a chat assistant | A search engine |
| Retriever | No acronym | The search step that picks passages | Finds the "Shipping" page for "when will it arrive?" | The index, which only stores |
| RAG status | Red, amber, green | A traffic-light rating in project reports | "Budget: amber" | Anything to do with AI |
Read the name backwards and it explains itself: generation (the model writes text) augmented (given extra material) by retrieval (a search that finds that material). A project report's "RAG red" is the traffic-light status, not AI.
What does RAG stand for, and where does the term come from?
RAG stands for retrieval-augmented generation, a name introduced by Patrick Lewis and eleven co-authors from Facebook AI Research, University College London and New York University in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, first posted to arXiv on 22 May 2020 and accepted at NeurIPS 2020.
The paper paired two models. A retriever called Dense Passage Retrieval (DPR) searched about 21 million passages of 100 words each, cut from a December 2018 copy of Wikipedia, and a generator, BART-large, wrote answers from the passages it was handed. The authors called knowledge stored in a model's weights parametric memory and the searchable index non-parametric memory, and their abstract named the two problems the index solves: "providing provenance for their decisions and updating their world knowledge remain open research problems."
They tested the updating claim directly: with a 2016 Wikipedia index the model answered questions about world leaders as of 2016, and with a 2018 index its answers changed to match, with no retraining. Facebook AI's announcement of 28 September 2020, which also released RAG in the Hugging Face Transformers library, put it simply: "we control what it knows simply by swapping out the documents it uses for knowledge retrieval."
Question-answering systems retrieved passages before 2020; the paper added a general recipe that fine-tuned retriever and generator together, and the name. Today the word is used loosely: almost any system that searches text and pastes the results into a prompt is called RAG, usually with an off-the-shelf model that is never retrained.
Why does AI need RAG?
AI needs RAG because a language model only knows what was in its training data, which stops at a fixed date and never included your private documents.
A large language model (LLM) is an AI model trained on a very large amount of text to predict the next word or word piece, which is what lets it write fluent answers. Its training cutoff is the date after which it has seen no new data. That leaves three gaps: it has never read your handbook or help center, it does not know what changed after the cutoff, and it cannot say where a fact came from, because knowledge in its weights has no page number.
When a model lacks a fact, it does not stay silent; it predicts something plausible. A hallucination is a fluent, confident statement that is false or that no source supports. RAG addresses all of these at once: the model is handed the relevant page, the page can be edited today, and the system knows which passage it used.
How does RAG work? The pipeline step by step
A RAG system works in two phases: indexing, which prepares the documents once and again whenever they change, and answering, which runs for every question.
Larkfield Design is a made-up 40-person company with a 60-page employee handbook and a small HR wiki, about 40,000 words in all. An employee types: "I've been here eight months. How much parental leave do I get?"
1. Ingest: collect and clean the text
Every source becomes plain text: web pages lose menus and footers, PDFs give up their text layer, and scanned pages go to a model that reads images. Larkfield's handbook and 30 wiki pages become one pile of clean text.
2. Chunk: split the text into passages
A chunk is a passage of a document, usually a paragraph or a few, that is stored and retrieved as one unit. Search works better on focused passages than on a whole manual. Cut into chunks of about 200 words, Larkfield's 40,000 words become roughly 200 chunks, and section 7.1 ("Eligibility: six months' service") and section 7.2 ("Parental leave: 16 weeks paid") land in separate ones. Sizes vary: the 2020 paper used 100-word passages, and PepoChat cuts text into chunks of about 1,000 characters.
3. Embed: turn each chunk into numbers
An embedding is a list of numbers that represents the meaning of a piece of text. OpenAI's embeddings guide calls it "a vector (list) of floating point numbers" and says "the distance between two vectors measures their relatedness"; one of its embedding models returns 1,536 numbers per text by default. Larkfield's chunks about leave end up close together, far from the chunks about expenses.
4. Store: put the embeddings in a vector index
A vector database is a database built to store embeddings and quickly return the ones nearest to a new embedding. Most also keep each chunk's text and labels such as file name, section and date, so one of Larkfield's 200 entries reads "Employee handbook, section 7.2, updated March 2026".
5. Retrieve: find the passages that match the question
The question is embedded with the same model, and the index returns the nearest chunks. Semantic search is finding text by meaning rather than exact words, which is how "parental leave" matches a section titled "Family leave". Hybrid search runs a keyword search alongside and merges the two lists; Microsoft's Azure AI Search documentation, updated August 2026, notes that product codes, jargon, dates and people's names "perform better with keyword search because it can identify exact matches." Reranking is a second, slower scoring pass over the shortlist; Azure's semantic ranker rescores the top 50 results.
For Larkfield, the top five chunks are sections 7.1, 7.2 and 7.4, a wiki page called "Leave FAQ", and an old 2023 policy PDF nobody deleted. Keep that last one in mind.
6. Augment: put the passages into the prompt
The retrieved chunks are added to the model's request with instructions such as "answer only from the passages below; if they do not answer the question, say so; name the section you used". The context window is the most text a model can consider in one request, which is why only the best few chunks go in.
7. Generate: the model writes a grounded answer
Grounding is tying every statement in an answer to a source the model was given. Larkfield's assistant replies: "You qualify. After six months' service, employees get 16 weeks of paid parental leave (Handbook 7.1 and 7.2)." The model never knew Larkfield's policy; it read it a second ago.
8. Cite: show where the answer came from
Good systems attach sources from the retrieval step's labels, so the employee can open section 7.2 and check. A model can invent a citation as easily as a fact, so a reference is only trustworthy when the search layer produced it.
What is a RAG system made of?
A RAG system is made of five working parts plus the instructions that tie them together, and each part can fail on its own, which is why "the AI got it wrong" is rarely the model's fault alone.
| Part | Its job | Typical choice | What goes wrong |
|---|---|---|---|
| Loader | Turns files and pages into text | PDF, web and Office parsers | Scans, scripts and tables come out empty |
| Chunker | Cuts text into passages | Fixed size or by heading | A rule and its exception land apart |
| Embedding model | Turns passages into numbers | A hosted embedding API | Exact codes match poorly |
| Vector index | Stores and searches embeddings | A vector database or search service | Stale entries never removed |
| Language model | Writes the answer | Any capable LLM | Mixes passages or fills gaps |
What are the types of RAG?
The main types of RAG are basic vector RAG, hybrid RAG, RAG with reranking, GraphRAG and agentic RAG, and they differ mostly in how the search step finds passages.
| Type | How it retrieves | Best for | Trade-off |
|---|---|---|---|
| Basic (vector) RAG | Nearest embeddings only | Plain-language questions over prose | Misses exact codes and names |
| Hybrid RAG | Keyword plus vector, merged | Questions with product names | Two indexes to keep in sync |
| RAG with reranking | Broad search, then precise rescoring | Large collections | Extra latency and cost |
| GraphRAG | A knowledge graph and summaries built in advance | "What are the main themes?" | Expensive to build and refresh |
| Agentic RAG | An AI agent plans several searches | Multi-part questions | Slower, less predictable |
Hybrid search
Hybrid RAG runs keyword and vector search on the same question and merges the ranked lists, in Azure's case with a method called Reciprocal Rank Fusion. It is the usual fix when a system finds "how do I reset my password" but misses "error E-4012".
GraphRAG
GraphRAG is Microsoft Research's method, introduced in February 2024, in which a language model first extracts the people, places and concepts in a collection into a knowledge graph and summarizes related groups of them. The paper by Darren Edge and colleagues (April 2024) reports better answers than plain RAG to whole-collection questions such as "what are the main themes in the dataset?"
Agentic RAG
Agentic RAG puts an AI agent in charge of retrieval: it decides whether to search, splits a hard question into smaller searches, checks the results and searches again. A January 2025 survey by Aditi Singh and colleagues describes it as adding reflection, planning, tool use and multi-agent collaboration to the pipeline.
RAG vs fine-tuning vs long context: which should you use?
Use RAG when the answers live in documents that change or that the model has never seen; use fine-tuning to change how a model writes, a long context window for a few documents read once, and plain prompting for general knowledge.
Fine-tuning is continuing to train a model on your own examples so its internal weights change. A long context approach skips search and pastes whole documents into every request.
| Approach | How your knowledge gets in | Cost per question | Freshness | Use this when |
|---|---|---|---|---|
| Plain prompting | It doesn't; the model uses its training | Lowest | Frozen at the cutoff | The question is general knowledge |
| RAG | A few retrieved passages per question | Low | As fresh as the index | Facts change or are private |
| Long context | Whole documents in every request | High for big files | As fresh as what you paste | A few documents, read once |
| Fine-tuning | Training changes the model's weights | Low per question, high to train | Frozen until you retrain | You need a style, format or skill |
Google's Gemini long-context documentation, checked 12 November 2026, says many Gemini models accept "1 million or more tokens", about eight average-length English novels, where earlier models handled 8,000. The same page warns that when the model must find several pieces of information at once it "does not perform with the same accuracy", and says techniques such as RAG "remain valuable in specific scenarios". A 2023 study, "Lost in the Middle" by Nelson Liu and colleagues, found models used information at the start or end of a long input better than information buried in the middle. For a help center that changes weekly, resending every page with every question rarely pays.
What is RAG good at, and where does it fail?
RAG is good at answering from specific, private and changing documents and showing where an answer came from; it fails when the search step returns the wrong passages or the documents themselves are wrong.
Updates are edits, not training runs, and the system can decline when nothing relevant comes back. Its failures are mostly retrieval failures.
Retrieval misses
The right passage exists but is not in the top results, because the question uses words the document never does, or the answer sits in an image or a scanned page with no text. Our list of 12 reasons a chatbot says "I don't know" covers the fixes.
Stale sources
The index holds last year's version. Larkfield's 2023 policy PDF was retrieved beside the current handbook; if it said 12 weeks, the model now has two answers.
Conflicting documents
When two passages disagree, the model may pick one, merge them or hedge. Retrieval finds similar text, not true text, so keep one current page per fact.
Answers stitched from the wrong chunk
A chunk cut mid-section can carry a rule without its exception, or a table row without its header, and the model answers confidently from half a fact. Writing so that every chunk stands alone prevents most of it.
Whole-collection questions
"Summarize every complaint this year" needs the whole collection, not the top five passages. That is the gap GraphRAG was built for.
RAG reduces hallucination; it does not remove it. A preregistered study by Varun Magesh and colleagues, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools" (May 2024), found that RAG-based legal research tools from LexisNexis and Thomson Reuters hallucinated less than GPT-4 used as a general chatbot, yet still "between 17% and 33% of the time". Our guide to stopping chatbot hallucinations covers the defenses.
What are some examples of RAG?
Everyday examples of RAG include support agents that answer from a help center, internal assistants that answer from company files, the AI answers in search engines, and tools that let you chat with a PDF.
A support agent answering from a help center
A visitor asks a store's chat widget about delivery times, and the agent retrieves the shipping page before replying. PepoChat works this way.
An internal policy assistant
Microsoft's documentation, updated 10 November 2026, says Microsoft 365 Copilot with the Premium license grounds responses in organizational data such as "emails, files, meetings, calendars, teams" through Microsoft Graph, with access scoped by each user's permissions.
AI answers in search engines
Google's Search Central documentation says AI Overviews and AI Mode may use a "query fan-out" technique, "issuing multiple related searches across subtopics and data sources" to develop a response, and show links to supporting pages. Perplexity's help center says it searches the internet in real time and gives every answer numbered citations linking to the original sources.
Chat with your PDF
Google's Gemini Notebook, called NotebookLM until Google renamed it in July 2026, lets you upload PDFs, websites and Google Docs and "chat with your notebook to get grounded information based on your sources with clear in-line citations".
The original: trivia from Wikipedia
The 2020 RAG model answered open-domain trivia and wrote Jeopardy-style questions from its Wikipedia index, setting the state of the art on three open-domain question-answering tasks.
How do you judge a RAG system?
Judge a RAG system on three things first: whether it shows its sources, whether it says "I don't know" when the documents are silent, and how fresh its index is.
| Check | How to test it | Good sign | Red flag |
|---|---|---|---|
| Sources | Ask five questions, open each cited page | The page says what the answer says | Links that don't contain the claim |
| "I don't know" | Ask about something your documents never cover | It declines or offers a person | A confident invented answer |
| Freshness | Change a fact, ask again after the next sync | The new fact appears | The old fact weeks later |
| Exact terms | Ask using a product code or error number | Finds the right page | A vague answer about a similar topic |
| Conflicts | Ask about a fact two old pages disagree on | One consistent answer | A different answer each time |
Build a test set before comparing tools: 20 to 40 real questions from your inbox, including five your documents cannot answer, and rerun it after every big content change. How a system handles the unanswerable five tells you more than any demo.
How do you build or buy a RAG system?
You can build a RAG system from parts, use a managed retrieval service from a cloud or model vendor, or buy a finished product made for one job, such as customer support.
Building means choosing each part in the table above, then writing the code for syncing, prompts and evaluation. Managed services remove the plumbing: OpenAI's retrieval documentation says a file added to one of its vector stores "will be automatically chunked, embedded, and indexed", and Azure AI Search adds hybrid search and reranking. Finished products add a chat interface, website and app sync, analytics and a way to hand off to people.
Build when retrieval is the product you sell, your permissions are complex, or your documents need special parsing. Buy when the job is standard and your time is better spent on the content.
What are the common misconceptions about RAG?
The most common misconception is that RAG trains the model on your data; it does not, and most other myths follow from that one.
"RAG trains the AI on my documents." The model's weights never change. Your documents sit in a separate index and are read at answer time, like notes handed to a speaker.
"It learns from every conversation." Not by itself. Answers change when someone changes the documents; our explainer on what a knowledge base chatbot is covers what does improve over time.
"RAG stops hallucinations." It reduces them, as the legal-research study shows.
"RAG needs a vector database." Retrieval can be a keyword index, a database query or a web search.
"Long context makes RAG obsolete." Not for large or changing collections, where cost, speed and accuracy favor sending a few good passages.
"More documents mean better answers." Every outdated or duplicate page is a candidate for retrieval. A small, current knowledge base beats a large, messy one.
How PepoChat uses RAG
PepoChat is a RAG system packaged for customer support. You add knowledge sources: a website crawl of up to 200 pages and 3 levels deep, a sitemap import, single links, files (PDF including scanned pages, DOCX, Markdown, HTML, TXT, CSV, JSON and images), text snippets and Q&A pairs, Notion and Google Drive connectors that sync daily, and a built-in Help Center whose published articles feed the knowledge base. Each workspace gets its own private index.
For every visitor message, the AI support agent searches that index with the visitor's question, keeps the best eight passages, up to 12,000 characters, and answers only from them, with the retrieved text marked as untrusted data, which makes it harder for an instruction planted in a web page to steer the agent. When it finds nothing relevant, it says so and offers a person, and the conversation is marked escalated in the team inbox for human handoff. On paid plans, URL sources are re-fetched weekly and only changed pages are re-indexed; on Free you refresh with "Sync website pages".
The honest limits: search is by meaning only, with no separate keyword pass, so write product codes into sentences; replies do not show source links, so you check answers in the team inbox; and you do not choose the AI model. You can start free with 100 AI replies a month and 5 knowledge sources, no credit card needed.
What to do next
- Read RAG for customer support, explained for the support version of this pipeline and how to write docs that retrieve well.
- See how answers work in PepoChat, from retrieval to handoff.
- Compare plans on the pricing page: Free has 100 AI replies a month, Starter is $39 a month for 1,000.
- Create a free PepoChat workspace, add your help center or website, and ask it the five questions your documents cannot answer.
- Related explainers: What is an LLM?, What is an AI hallucination? and What is a knowledge base?.
Frequently asked questions
- Is RAG in AI the same as a RAG status (red, amber, green)?
- No. In AI, RAG means retrieval-augmented generation, a way of making a language model answer from documents it searches first. In project management, a RAG status is a red, amber or green rating that shows whether a task, budget or deadline is on track. The two share only the letters, so check the context before assuming which one a document means.
- Is ChatGPT a RAG system?
- Sometimes. When ChatGPT answers from memory, it uses only what its model learned in training, so that is not RAG. When it searches the web or reads a file you upload and then answers from what it found, it is doing retrieval-augmented generation in practice. The difference shows in the answer: retrieved answers can point to sources, memory answers cannot.
- Does RAG need a vector database?
- No. A vector database is the most common way to store embeddings for semantic search, but any search step that finds relevant text counts as retrieval: a keyword index, a database query, a web search or a mix of them. Small collections can even be searched in memory. Choose the store after you know what questions people ask and how often the documents change.
- What is the difference between RAG and semantic search?
- Semantic search finds and ranks passages by meaning and stops there, so you get a list of results to read yourself. RAG uses a search step, often semantic search, and then hands the top passages to a language model that writes one answer from them. Semantic search is a part of many RAG systems, not an alternative to them.
- Can RAG answer questions from PDFs, spreadsheets and websites?
- Yes, once the text is extracted. PDFs, Word files, web pages, Markdown and CSV files all become plain text that is chunked and indexed, and scanned pages need a model that can read images first. PepoChat, for example, accepts PDFs including scanned pages, DOCX, CSV, JSON and images, and crawls up to 200 pages of a website. Key facts in tables answer best when also written as sentences.
- How much does it cost to use RAG?
- It depends on whether you build or buy. Building means paying for embeddings, a vector store, a model call for every answer and the engineering time to keep it running. Packaged tools charge per reply or per seat instead. PepoChat, a RAG-based AI support agent, is free for 100 AI replies a month, and its Starter plan is $39 a month for 1,000 replies.
Try this on your own site in ten minutes
PepoChat includes every chat feature on the free plan — 100 AI replies and 5 knowledge sources a month, no credit card.
Keep reading
Chatbase vs PepoChat: Which AI Support Agent Fits Your Team in 2026?
An honest side-by-side of Chatbase and PepoChat on pricing, free plan or trial, sources, handoff, booking, actions, channels and voice, and a pick by team type.
Intercom Fin Pricing Explained, and What a Flat-Plan Alternative Costs
What Intercom Fin's $0.99 per outcome adds up to once you count resolutions and seats, with a worked 2,000-conversation example and when a flat plan wins.