RAG: Teaching the Model About Your Company

The model knows nothing about your company

Ask a model "what's our expense reimbursement limit?" and it will make something up. It has to — your internal handbook was never in its training data.

Two ways out. You can fine-tune a model on your documents, which is expensive, slow, and has to be redone every time a document changes. Or you can look up the relevant passages and send them along with the question.

The second approach is called RAG — retrieval-augmented generation. It's how roughly all production AI applications handle private knowledge.

It's really just one idea

Retrieve first, then paste what you retrieved into the prompt.

retrieved_docs = "Taxi fares are reimbursable up to $40 per trip with a receipt."
question = "Can I expense a taxi?"

prompt = f"""Answer using only the material below. If it isn't there, say you don't know.

Material:
{retrieved_docs}

Question: {question}"""

print(prompt)

Print that and you see the whole technique: an ordinary prompt with a paragraph pasted in. Send it to a model and you get a correct, grounded answer — not because the model learned anything, but because you handed it the fact.

All of RAG's complexity lives in one place: how do you find the right few paragraphs?

Why not keyword search?

A user asks "how do I expense a taxi?" Your handbook says "ground transportation receipt submission procedure." Not a single word overlaps. Keyword search returns nothing.

Vector search solves this by comparing meaning rather than characters.

The mechanism: an embedding model converts a piece of text into a list of numbers — a vector, typically a few hundred to a few thousand of them. Texts with similar meanings land near each other in that space. At query time you embed the question too and find its nearest neighbors.

"Taxi" and "ground transportation" end up close together because they appeared in similar contexts across the embedding model's training data. That's the whole reason this works, and also the reason it sometimes doesn't: if your domain uses jargon the embedding model never saw, similarity gets noisy.

The four steps

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

# 1. Load
docs = TextLoader("handbook.md", encoding="utf-8").load()

# 2. Split
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)
print(len(chunks), chunks[0].page_content[:60])

# 3. Embed and store
vectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())

# 4. Retrieve
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

Steps 1 and 2 need no API key, and the print shows you how many chunks you got and what the first one starts with. Run those two alone first — if your chunk count is 1, your document didn't load the way you think it did, and there's no point embedding anything yet.

Steps 3 and 4 do call the embeddings API. Install with pip install langchain-community faiss-cpu. FAISS is a local vector store: no server, no configuration, and fine up to a few hundred thousand chunks. Note that FAISS.from_documents embeds every chunk, which costs money and time proportional to your corpus — build the index once and save it with vectorstore.save_local(...) rather than rebuilding on every run.

Splitting decides whether RAG works

This is the most underrated step in RAG. Split badly and no amount of prompt tuning later will save you.

• chunk_size=500 — about 500 characters per chunk. Too large and you pay for long irrelevant context that dilutes the model's attention. Too small and you slice sentences apart and retrieve fragments. 300–800 is a reasonable starting range for prose.
• chunk_overlap=50 — neighboring chunks share 50 characters. This keeps a complete thought from being cut in half exactly at a chunk boundary. Without overlap, the sentence "Taxi fares are reimbursable up to $40" can become "Taxi fares are reimbur" in one chunk and "sable up to $40" in the next, and neither matches the question well.
• RecursiveCharacterTextSplitter tries to break at natural boundaries first — paragraphs, then sentences, then words — falling back to a hard cut only when it must. Much better than slicing every 500 characters blindly.

If your documents have real structure — Markdown headings, numbered policy clauses — use a structure-aware splitter like MarkdownHeaderTextSplitter. Splitting by heading means every chunk carries its own context, so a chunk about taxis still knows it lives under "Travel Expenses."

Assembling the full chain

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

template = """Answer using only the material below.
If the material doesn't cover it, reply "That isn't in the documentation." Do not guess.

Material:
{context}

Question: {question}"""

prompt = ChatPromptTemplate.from_template(template)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(rag_chain.invoke("What do I need to expense a taxi?"))

This prints an answer grounded in your handbook — or the "That isn't in the documentation" line if the handbook is silent, which is exactly the behavior you want.

The dictionary at the top is the part that looks strange. Read it as two parallel branches from the same input: the question goes through retriever | format_docs to become the context string, and simultaneously passes through untouched to become question. RunnablePassthrough() means "hand the input along unchanged."

A dict inside an LCEL chain always means this: run each value on the same input, collect the results into a dict, pass it on. Once you've seen it twice it stops looking odd.

When results are bad, debug in this order

Beginners always reach for the prompt first. It's almost always the wrong move. Do this instead:

1. Print the retrieved chunks. print(retriever.invoke("your question")). If the answer isn't in what came back, the model is not the problem and no prompt will ever fix it. This one check separates "retrieval problem" from "generation problem," and those have completely different fixes.
2. Adjust chunk_size and your splitting strategy. This is where most retrieval problems actually live.
3. Adjust k. Too few chunks and you miss the answer; too many and you bury it in noise.
4. Only then, change the prompt.

This ordering will save you days. Write it on a sticky note.

Always return your sources

Ship a RAG system without citations and you've built something users can't verify:

docs = retriever.invoke("taxi expenses")
for d in docs:
    print(d.metadata)   # {'source': 'handbook.md'}

Every Document carries a metadata dict, and loaders populate it with the source path. Surface that in your UI. AI systems get things wrong; an answer with a traceable source is one a user can check, and that's the difference between a tool people trust and a demo.

Ten-second check

Your RAG system gives a wrong answer. What's step one?

Print the retrieved chunks and check whether the correct answer is in them at all. Find out whether retrieval or generation is broken before you change anything.