HOST_A: Okay, so I want to start with a story. And it's not a hypothetical — this actually happened. A lawyer in New York, 2023, used ChatGPT to help draft a legal brief. The AI confidently cited six cases. Real-sounding case names, real-sounding quotes, real-sounding court rulings. Except none of them existed. The cases were completely fabricated.
HOST_B: Mata v. Avianca. That's the one you're talking about. Yeah, it made headlines everywhere. The judge was absolutely furious. The attorneys had to file a public apology. And the wild thing is — the model didn't say "I'm not sure." It was completely, utterly confident.
HOST_A: And that's what scares me. Like, wrong is bad enough, but wrong with total confidence? That's a different level of dangerous.
HOST_B: Right, and here's the thing most people don't understand about why that happens. These language models — they're not retrieving facts from some internal database. They're not looking anything up. They're predicting the next most plausible token based on everything they were trained on. So if a plausible-sounding legal citation fits the pattern, the model produces it, even if it never existed.
HOST_A: It's like the model learned the shape of a legal brief but not the actual law.
HOST_B: Exactly. It learned the grammar of legal writing, the rhythm of case citations, the way legal arguments flow. But whether those specific cases are real? That's not what the training objective cares about. The training objective was "predict the next token well."
HOST_A: So the model is essentially confabulating. Like how humans with certain memory disorders will fill in gaps with invented but plausible-sounding memories.
HOST_B: That's a really apt analogy actually. Confabulation is the right word. The model has a world model — it has patterns, relationships, concepts — but when you ask it something specific that wasn't deeply encoded in training, it fills the gap with something that fits the pattern.
HOST_A: And this happens in medicine too, right? People using AI for medical advice?
HOST_B: Constantly. There are documented cases of AI chatbots giving dangerous medication dosage advice, suggesting drug combinations that are contraindicated, completely making up clinical trial results. Not because the AI is malicious, but because plausible-sounding medical information is exactly the kind of pattern these models are good at generating.
HOST_A: So this is the hallucination problem. And supposedly, RAG is the solution. Today we're going into the deep end on Retrieval-Augmented Generation — what it actually is, why it works, where it falls short, and whether it's really the silver bullet everyone says it is.
HOST_B: And I have some opinions about that. Spoiler: it's not a silver bullet. But it is a genuinely important and useful technique. So let's actually start from first principles, because I think a lot of people have heard the term RAG without really understanding the mechanics.
HOST_A: Yeah let's do that. So what is RAG, at the most basic level?
HOST_B: Okay so at its core, RAG — Retrieval-Augmented Generation — is a two-step pipeline. Step one: retrieve. Before the language model generates anything, you go find relevant information. Step two: generate. You take that retrieved information and you stuff it into the prompt — the context window — and then ask the model to generate an answer based on what you just provided.
HOST_A: So instead of relying on what the model baked in during training, you're feeding it fresh, specific, relevant information right at inference time.
HOST_B: Exactly. And that changes the game fundamentally. The model doesn't have to have memorized your company's internal documentation, your legal database, your product catalog. You retrieve the relevant bits on the fly and give them to the model.
HOST_A: And the original paper on this is what — Lewis et al. 2020? Facebook AI?
HOST_B: Yes! Patrick Lewis and colleagues at Facebook AI Research — now Meta AI — published the paper in 2020. It was actually called "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." And it was genuinely groundbreaking. They showed that you could combine a parametric memory — that's the neural network, the language model — with a non-parametric memory, which is the retrieved documents, and get dramatically better performance on knowledge-intensive tasks.
HOST_A: Right, so "parametric" meaning the knowledge is baked into the weights of the model itself, and "non-parametric" meaning it's stored externally and looked up at runtime.
HOST_B: Perfect. That distinction is really important. Parametric memory is what causes hallucinations when it goes wrong — because the model is generating from patterns, not from explicit lookup. Non-parametric memory is more like a database — it's either there or it's not.
HOST_A: Okay so how does the retrieval actually work? Because when I think "retrieval" I think of, like, keyword search. Like Ctrl+F. But that's not what's happening here, is it?
HOST_B: Not quite, no. Modern RAG uses semantic search, which is fundamentally different from keyword search. And to understand semantic search, we need to talk about embeddings. So an embedding is a way of representing text — a word, a sentence, a paragraph, a document — as a vector of numbers. A high-dimensional vector, typically like 768 or 1536 dimensions.
HOST_A: And what does that vector actually represent?
HOST_B: It represents the meaning of the text in that high-dimensional space. The key property is that texts with similar meaning will have vectors that are close to each other in that space. So "dog" and "puppy" will have nearby vectors. "Machine learning" and "neural networks" will be nearby. "Legal brief" and "court document" will be nearby. Even across languages, if you use a multilingual embedding model.
HOST_A: And "nearby" means — what exactly? Low distance between the vectors?
HOST_B: Right. The most common measure is cosine similarity. You measure the angle between two vectors — if they're pointing in roughly the same direction in that high-dimensional space, they're similar. Cosine similarity of 1 means identical direction, 0 means completely unrelated, negative means opposite.
HOST_A: Okay so I have my documents, I turn them all into embedding vectors, and then when a user asks a question, I embed the question too, and find which document vectors are most similar?
HOST_B: That's it exactly. That's the core of vector search. You compute the embedding of the query, then find the k most similar document embeddings — that's top-k retrieval — and you retrieve those k documents to put in the context.
HOST_A: And k is typically what, like five, ten?
HOST_B: Varies a lot by use case. Three to ten is common. You want enough context to answer the question, but not so much that you overwhelm the model or hit the context window limit. Though as context windows have gotten larger, people are experimenting with k equals twenty, fifty, even more.
HOST_A: Now, where are all these embedding vectors stored? Because I'm imagining if you have millions of documents, you can't just loop through all of them and compute similarities one at a time.
HOST_B: Right, that's where vector databases come in. A vector database is specifically optimized to store and search embedding vectors at scale. The big ones in the space right now are Pinecone — which is fully managed cloud service, very popular for startups. Weaviate — which is open source and has a lot of interesting features. Chroma — which is lightweight, great for local development and prototyping. And then there's pgvector, which is a Postgres extension. So if you're already running Postgres, you can just add vector search without a whole separate system.
HOST_A: Oh interesting, so you don't have to have a separate vector database — you can extend your existing SQL database?
HOST_B: Exactly. pgvector is wildly popular in production because it removes operational complexity. You already have Postgres, you already know how to manage it, you already have backups set up. Just add the extension. The search performance doesn't match specialized vector databases at massive scale, but for most production use cases, it's more than good enough.
HOST_A: And these vector databases use approximate nearest neighbor search? Not exact search?
HOST_B: Yes, for scale. Exact nearest neighbor search — brute force comparing your query vector to every stored vector — is O(n) and gets slow at millions of vectors. Approximate nearest neighbor, ANN, uses index structures like HNSW — Hierarchical Navigable Small World graphs — or IVF — Inverted File Index — to trade a tiny bit of recall accuracy for massively faster search. You might miss the actual closest vector one percent of the time, but you get your results in milliseconds instead of seconds.
HOST_A: Okay, so I now have this mental model: documents get embedded, stored in a vector DB, query comes in, gets embedded, nearest neighbors are retrieved, those go into the prompt along with the query, and the LLM generates an answer.
HOST_B: That's naive RAG. And it works! It actually works surprisingly well for a lot of use cases. But it's also where most of the simplistic blog posts stop, and the interesting nuances start.
HOST_A: Right, because everyone says RAG is the answer to hallucinations. And I've seen so many demos where someone builds a RAG system over their documentation and it looks great. But then I've also seen plenty of production failures. So let's talk about why everyone's building RAG before we get to why it sometimes doesn't work.
HOST_B: Yeah. So the mainstream case for RAG is actually quite strong. The first big thing is knowledge freshness. A model trained in, say, early 2024 doesn't know about anything that happened after that. But with RAG, you can continuously add new documents to your vector database. Your system always has access to current information without retraining.
HOST_A: Which retraining is extremely expensive. We're talking tens of millions of dollars for a frontier model.
HOST_B: Hundreds of millions if you're OpenAI or Google. Even fine-tuning a smaller model takes significant compute. RAG sidesteps that entirely. Your knowledge store is separate from your model weights — you update the knowledge store, not the model.
HOST_A: That's a huge deal for anything in a fast-moving domain. Like news, or financial data, or legal updates, or medical guidelines.
HOST_B: Exactly. And the second big thing is domain specificity. A general-purpose LLM knows a lot, but it doesn't know your specific company's internal policies, your proprietary documentation, your internal Slack conversations, your customer support tickets. RAG lets you build AI systems grounded in your specific organizational knowledge.
HOST_A: So instead of a model that knows everything about medicine in general, you can build a RAG system that's grounded in your specific hospital's protocols and patient records. That's a much more useful thing for a specific clinical decision support tool.
HOST_B: Right. And third — and this is the one enterprises care about most — is auditability and trust. When the model answers a question, you can show which source documents were retrieved. You can cite your sources. "This answer is based on Policy Document 42B, Section 3." That's huge for regulated industries — financial services, healthcare, legal. You need to be able to audit why the system said what it said.
HOST_A: Okay and that connects back to the original problem, right? The lawyer thing. If the AI had been grounded in an actual legal database and cited actual retrieved case documents, the hallucinated cases wouldn't have been possible. You're constraining the model to information that actually exists in your database.
HOST_B: That's the theory, yes. Though — and I'll come back to this — the practice is more complicated. But yes, that's the core promise.
HOST_A: So why is RAG everywhere right now? LangChain, LlamaIndex — these frameworks essentially exist to make RAG pipelines easy to build.
HOST_B: LangChain especially became the de facto standard for building RAG applications quickly. You can spin up a RAG pipeline in like twenty lines of Python. Connect to a vector store, set up an embedding model, wire up an LLM, done. LlamaIndex — formerly GPT Index — is more specialized for document ingestion and indexing. Both are huge because they abstract away a lot of the plumbing.
HOST_A: And every enterprise AI project seems to start with "let's do RAG over our internal docs."
HOST_B: It's become the default first step. Which honestly makes sense as a starting point. But I see a lot of teams stop there, declare victory, and then wonder why things go wrong in production. So let me share some contrarian perspectives.
HOST_A: Please. Because I've been nodding along going "yes, RAG is great" and I suspect you're about to complicate my life.
HOST_B: Ha. So here's the first thing I want to push back on: the idea that RAG solves hallucinations. It shifts the problem, it doesn't eliminate it.
HOST_A: Okay, explain.
HOST_B: So with naive RAG, if the retrieval is bad — if you retrieve the wrong documents, or retrieve documents that are tangentially related but don't actually answer the question — the model now has bad context. And what does the model do with bad context?
HOST_A: It tries to answer using that bad context?
HOST_B: Often yes. Or it confabulates to fill the gaps. "Based on the provided context" — and then it makes something up anyway. There's a paper by Shi et al. that showed adding irrelevant context to a prompt can actually make model performance worse than having no context at all, because the model gets confused and tries to incorporate irrelevant information.
HOST_A: Oh that's bad. So "garbage in, garbage out" doesn't just apply to the data you ingest — it applies to what you retrieve.
HOST_B: Exactly. Your retrieval quality is now the bottleneck. Your chunking strategy matters enormously. If you've chunked your documents in a way that splits up important context, you retrieve a fragment that's missing the key information. The model can't answer well from a fragment.
HOST_A: So you've moved the problem from "the model doesn't know" to "the retrieval system didn't find the right thing." Which is progress, but it's not a complete solution.
HOST_B: Right. And the second contrarian point I want to make is about long-context models. Gemini 1.5 Pro has a one million token context window. Gemini 1.5 Flash can do a million tokens. Claude 3 has two hundred thousand. These are massive context windows. And the question I've been asking myself is: at what point does retrieval become unnecessary?
HOST_A: Because if your entire knowledge base fits in the context window, you just... stuff it all in?
HOST_B: That's the argument. If you have a ten-thousand-page technical manual and the model can fit all of it in context, why bother with a vector database at all? Just load the whole thing.
HOST_A: But doesn't that get incredibly expensive? You're paying for all those tokens on every query.
HOST_B: Yes, and that's the counterargument. Token costs at scale are significant. RAG is more efficient because you're only sending the relevant chunks — maybe two thousand tokens out of a million-token document — rather than the whole thing. Plus inference time gets slower with huge contexts.
HOST_A: So it's a cost and latency trade-off.
HOST_B: Mostly. But there's also a quality argument. Research on "lost in the middle" — a paper by Liu et al. — found that language models actually perform worse when the relevant information is in the middle of a very long context. They attend better to the beginning and end. So retrieval, by putting the most relevant information first in the context, can actually improve answer quality over naive full-context stuffing.
HOST_A: Wait, that's interesting. So even if the model technically has access to everything, there are attention dynamics that make retrieval still valuable?
HOST_B: Exactly. That's one of those things I think the "long context makes RAG obsolete" crowd underestimates. The attention mechanism isn't perfectly uniform across a million tokens.
HOST_A: Okay but even so, do you think long-context models will eventually make RAG unnecessary?
HOST_B: For small to medium corpora? Probably yes, within a few years. For massive enterprise knowledge bases — billions of documents, petabytes of data? No chance. You'll always need retrieval. The question is where the crossover point is.
HOST_A: I actually want to push back on your earlier contrarian point though. Because yes, RAG shifts the problem to retrieval quality — but isn't a retrieval quality problem more tractable than a model knowledge problem? Like, I can fix my chunking strategy without retraining a model.
HOST_B: That's a fair point actually. Retrieval failures are often diagnosable and fixable. Model knowledge failures — the baked-in stuff — are much harder to address. You either retrain, fine-tune, or accept it. So in that sense, moving the problem to retrieval is actually progress.
HOST_A: Right. So maybe the narrative isn't "RAG doesn't solve hallucinations" — it's "RAG makes the problem more tractable."
HOST_B: I'll accept that reframing. It's more accurate.
HOST_A: Okay what about fine-tuning versus RAG? Because I've heard arguments both ways. Some people say fine-tuning is the real answer for domain-specific performance.
HOST_B: So this is a genuine debate. Fine-tuning changes the weights of the model itself — you're essentially teaching the model new knowledge or new behavior. It can produce very smooth, natural responses about domain-specific topics. But it's expensive, it requires quality training data, it takes time, and crucially — it doesn't update automatically. You retrain for new knowledge.
HOST_A: Whereas RAG is cheap to update, easy to iterate on, but depends on retrieval quality.
HOST_B: Right. The current consensus — and I broadly agree with this — is that fine-tuning and RAG are complementary, not competing. Fine-tuning for style, tone, and domain-specific reasoning patterns. RAG for factual knowledge that changes over time. Use both when you can afford to.
HOST_A: But most teams can't afford to fine-tune, so they start with RAG.
HOST_B: Exactly. RAG is the accessible entry point. And I've seen teams build really sophisticated RAG systems that outperform naive fine-tuning anyway.
HOST_A: What about GraphRAG? I keep seeing that term pop up.
HOST_B: Oh, great topic. GraphRAG is Microsoft's approach — they published a paper and released a system — that instead of just chunking documents and embedding them, builds a knowledge graph from the documents. Entities, relationships, communities of related concepts. And then when you query, you traverse the graph to find related information.
HOST_A: So instead of "find chunks that are semantically similar to my question," it's "find entities and relationships that are relevant to my question."
HOST_B: Right. And it's particularly good for questions that require synthesizing information across many documents. Like "what are the themes across all our customer feedback?" — that's not a question that works well with standard RAG because no single chunk answers it. But if you've built a graph of entities and communities, you can answer it.
HOST_A: That makes sense. Standard RAG is good for lookup — "what's the policy for X?" GraphRAG is better for synthesis — "what patterns exist across all of this?"
HOST_B: That's a clean distinction. The downside is GraphRAG is significantly more complex and slower to build. For most use cases, standard RAG is perfectly adequate.
HOST_A: Okay so we've been talking about naive RAG and some of its problems. But there's a whole world of techniques to improve RAG that I want to dig into. Because you mentioned chunking earlier and I want to get into the weeds on that.
HOST_B: Yes, chunking is so underappreciated as a lever. It seems like a boring implementation detail — how do you split your documents into pieces — but it might be the most impactful thing you do in your RAG pipeline.
HOST_A: So what are the main approaches?
HOST_B: Okay so starting with the simplest: fixed-size chunking. You split every document into chunks of, say, five hundred tokens, with maybe fifty tokens of overlap between consecutive chunks. Simple, fast, deterministic. The downside is you split based on token count, not based on where meaning actually breaks.
HOST_A: So you might cut a paragraph mid-sentence.
HOST_B: Exactly. Or you might cut right before the critical piece of information. A question about "what's the side effect of Drug X?" might retrieve a chunk that ends right before the side effect is listed.
HOST_A: Ouch. That's the kind of thing that looks fine in development and blows up in production.
HOST_B: Classic. So the next level is semantic chunking — you split based on semantic similarity. Consecutive sentences that are topically related stay together, and you split when the topic shifts. Greg Kamradt popularized this approach. It produces much more coherent chunks but it's more expensive computationally — you're doing embedding comparisons to find split points.
HOST_A: And then there's hierarchical chunking?
HOST_B: Right, sometimes called parent-document retrieval. The idea is you have a hierarchy of chunk sizes — say, small chunks for retrieval and large chunks for context. When you retrieve a small chunk that matches the query well, you actually return its parent — the larger context it came from. So you get precise retrieval but broad context.
HOST_A: Oh that's clever. So you use the small chunk to find the needle, but you give the model the haystack around the needle.
HOST_B: Exactly. LlamaIndex has good built-in support for this. And it dramatically improves answer quality for questions where context matters.
HOST_A: Okay, what about re-ranking? Because I've heard this is important but I'm fuzzy on it.
HOST_B: Re-ranking is one of the most important improvements you can make to a naive RAG system. So here's the thing: when you do your initial vector search, you get your top-k results based on embedding similarity. But embedding models are trained to be fast — they embed your query and document chunks independently, with no interaction between them. That's called a bi-encoder.
HOST_A: Because query and document are encoded separately.
HOST_B: Right. A cross-encoder, on the other hand, takes the query and a document together as input and produces a relevance score. Because it sees both at once, it can model the interaction between them much more accurately. But it's too slow to run on your entire database — you'd never finish.
HOST_A: So you use the fast bi-encoder to get a candidate set, and then the accurate cross-encoder to re-rank that candidate set.
HOST_B: Exactly. You do the initial retrieval with your vector database — maybe get the top twenty results — and then run a cross-encoder re-ranker to re-score and re-sort those twenty results. Then you take the top three or five for the actual prompt.
HOST_A: And Cohere Rerank is a popular service for this?
HOST_B: Yes, Cohere's Rerank API is widely used. You send them your query and your candidate documents, they return relevance scores, you sort by those scores. It's genuinely impressive how much it can improve RAG quality. There's also the open-source option — you can run cross-encoder models yourself, BGE Reranker is a popular one.
HOST_A: What about HyDE? I saw that mentioned somewhere and I had no idea what it meant.
HOST_B: HyDE — Hypothetical Document Embeddings — is a fascinating technique from a paper by Gao et al. The observation is that query embeddings and document embeddings come from different distributions. When you ask a question, the embedding of the question might not be close to the embedding of the document that answers it, even if the document does answer it, because questions look different from answers in embedding space.
HOST_A: Oh interesting. Like "what is the capital of France?" might not be close to "Paris is the capital of France" in embedding space?
HOST_B: In practice those are fine, but yes that's the intuition. The more complex the question, the worse this mismatch can get. So HyDE flips it: you ask the LLM to generate a hypothetical answer to your question — an answer that might not be accurate but that is in the right format — and then you embed that hypothetical answer and use it for retrieval.
HOST_A: Wait, so you use the LLM to hallucinate a plausible answer, and then use that hallucination to search for real answers?
HOST_B: Exactly! You're leveraging the model's ability to produce document-like text to bridge the distribution gap. The hypothetical answer looks more like the target documents in embedding space than the original question did. You then retrieve real documents similar to the hypothetical answer, and use those real documents to generate the actual final answer.
HOST_A: That's kind of genius. You're using hallucination as a feature, not a bug.
HOST_B: Ha, yes. And empirically it works quite well, especially for complex or abstract queries.
HOST_A: What about ColBERT? I know that's another retrieval approach.
HOST_B: ColBERT — Contextualized Late Interaction over BERT — is by Omar Khattab and Matei Zaharia, 2020, out of Stanford. It's a late interaction model, which is a middle ground between bi-encoders and cross-encoders. In ColBERT, you independently encode the query and document into sequences of token-level vectors, not just one single vector for the whole input. Then at query time, you do a MaxSim operation — for each query token, you find its most similar document token, and you sum those similarity scores.
HOST_A: So you get more fine-grained interaction than a bi-encoder, but you precompute the document side so it's faster than a full cross-encoder.
HOST_B: Exactly. The document token vectors are precomputed and stored. Only the query vectors are computed at runtime. And you can do this efficiently with PLAID, the ColBERT engine. It's genuinely excellent for retrieval quality. The downside is storage — you're storing token-level vectors for every document, which is much more than a single embedding per chunk.
HOST_A: So it's a quality versus storage trade-off.
HOST_B: Right. For most production systems, bi-encoder plus re-ranker is the sweet spot. ColBERT is great when you need the highest possible retrieval quality and can afford the storage.
HOST_A: What about hybrid search? I've heard that term.
HOST_B: Hybrid search combines dense retrieval — the vector/embedding-based search we've been talking about — with sparse retrieval, which is keyword-based search, typically BM25. BM25 is essentially a fancy TF-IDF — term frequency inverse document frequency — it scores documents based on exact keyword matches, accounting for document length and term frequency.
HOST_A: So it's closer to old-school information retrieval.
HOST_B: Right. And here's the thing: sometimes you want exact keyword matching. If someone asks about "Article 47B subsection 3," dense retrieval might find semantically similar legal text that isn't the right one. BM25 will find the exact phrase. Dense retrieval is better for conceptual questions; sparse retrieval is better for exact lookup.
HOST_A: So you run both and combine the scores?
HOST_B: Yes. Reciprocal Rank Fusion is a common way to combine the rankings. And empirically, hybrid search almost always outperforms either approach alone. Weaviate has great built-in hybrid search. Most modern RAG frameworks support it.
HOST_A: Okay let me bring up something I've been wanting to ask about — agentic RAG. Because I feel like "RAG" is evolving and it's not just a simple pipeline anymore.
HOST_B: Yeah, agentic RAG is where things get really interesting. In naive RAG, it's one shot — the user asks, you retrieve, you generate. In agentic RAG, the retrieval becomes part of a broader agent loop. The LLM can decide to retrieve multiple times, from multiple sources, with different queries. It can reflect on whether the retrieved information is sufficient. It can decide it needs more context and issue another retrieval.
HOST_A: So the model is driving its own research process.
HOST_B: Exactly. And this solves a lot of the cases where naive RAG fails — questions that require multi-hop reasoning, where you need to retrieve something, use that to inform another retrieval, and so on. Like "what's the standard of care for a patient with condition X who is also taking drug Y?" — that might require retrieving the condition treatment guidelines, then retrieving the drug interaction data, then synthesizing both.
HOST_A: That's much more like how a human expert would work. Look one thing up, use it to know what else to look for.
HOST_B: Right. Frameworks like LangChain and LlamaIndex both have agent abstractions that support this. And it's become a significant trend in production AI systems.
HOST_A: At the cost of latency, presumably?
HOST_B: Yes. Multi-hop retrieval adds latency. But for complex enterprise use cases where accuracy matters more than speed, it's worth it.
HOST_A: Okay so how do you actually evaluate whether your RAG system is working? Because "it gives good-sounding answers" is not a rigorous evaluation.
HOST_B: This is so important and so often skipped. The RAGAS framework — Retrieval-Augmented Generation Assessment — is the most widely used evaluation approach for RAG systems. And it defines several distinct metrics that you should be measuring separately.
HOST_A: What are the main ones?
HOST_B: So RAGAS defines four core metrics. First is faithfulness — is the generated answer actually grounded in the retrieved context? Does it say things that are supported by the retrieved documents, or is it making stuff up?
HOST_A: So you could answer the question but still fail on faithfulness if the model goes beyond the context.
HOST_B: Exactly. The model might give a correct answer by using its parametric knowledge, but if that answer isn't in the retrieved context, the system isn't behaving as intended — you've got a hallucination risk even if it happened to be right this time.
HOST_A: And the second metric?
HOST_B: Answer relevancy — how directly does the answer address the question? You could be perfectly faithful to the context but still give an irrelevant answer if your context didn't contain the right information.
HOST_A: So faithfulness is about context-answer alignment, and relevancy is about question-answer alignment.
HOST_B: Right. Then there's context precision — of the retrieved chunks, how many of them are actually relevant? If you retrieve ten chunks and only two are useful, your context precision is low. And context recall — of all the information needed to answer the question, how much of it did you actually retrieve? You might have high precision — everything you retrieved is relevant — but low recall — you missed some important pieces.
HOST_A: So you want all four to be high, and they can fail independently.
HOST_B: Yes. And that's why evaluation is so important — because your system can look great on one metric and be terrible on another. I've seen RAG systems with great faithfulness but terrible context recall, meaning they give answers that are grounded in what they retrieved, but they missed the most important documents.
HOST_A: How do you run RAGAS evaluation? Do you need a labeled dataset?
HOST_B: You need a question-answer test set. Ideally you have questions where you know the right answer and the expected source documents. You can create these manually, or you can use LLMs to generate synthetic evaluation sets from your documents. RAGAS has good tools for this.
HOST_A: Okay, let's get practical. Someone's building a new system and they're deciding between RAG, fine-tuning, and just using a long-context model. How do you think about that decision?
HOST_B: I have a pretty clear decision tree for this. Start with the question: does the information you need change frequently? If yes — it updates weekly, monthly — then RAG is almost certainly the right answer. Fine-tuning and pretraining on static knowledge that goes stale is expensive and painful.
HOST_A: Makes sense. Dynamic knowledge base equals RAG.
HOST_B: Second question: how large is the corpus? If it's small enough to fit in context and cost isn't a concern, long-context might be simplest — no infrastructure needed. But if you're dealing with terabytes of documents, retrieval is the only option regardless.
HOST_A: And the third question?
HOST_B: Do you need to change how the model reasons, not just what it knows? If you want the model to always respond in a specific format, follow domain-specific reasoning patterns, be calibrated to a particular style — that's where fine-tuning shines. RAG doesn't change the model's behavior, it just changes what information it has access to.
HOST_A: So in practice, a mature enterprise system probably uses all three? Fine-tuned model with RAG over a live knowledge base, and possibly using long context for certain document-heavy subtasks?
HOST_B: That's the state of the art for sophisticated deployments. But for a first system, RAG is the fastest and most flexible starting point.
HOST_A: How do you pick a vector database?
HOST_B: For most teams I'd say: start with pgvector if you're already on Postgres and your scale is reasonable — millions of vectors, not billions. Zero additional infrastructure, excellent ecosystem, battle-tested. If you need something fully managed and cloud-native, Pinecone is extremely popular — great documentation, very low operational overhead. If you're open-source-first and want flexibility, Weaviate is excellent — it has built-in hybrid search, great schema flexibility. Chroma is my go-to for local development and prototyping — it runs in memory or as a local server, no setup needed.
HOST_A: And for enterprises that are already on a cloud provider?
HOST_B: Every major cloud has a vector database now. AWS has OpenSearch with k-NN, Google Cloud has Vertex AI Matching Engine, Azure has Azure AI Search. If you're deeply committed to one cloud, using their native offering simplifies auth and networking.
HOST_A: Let's talk about production failures. Because I think this is where people get burned. What are the most common ways RAG systems fail in production?
HOST_B: Oh, I have a list. First one: chunks too small. You chunk at, say, a hundred tokens because you want precise retrieval. But now each chunk has lost all context. A paragraph about drug side effects that references "the drug" with no mention of its name because the name was in the previous chunk. You retrieve a fragment that doesn't make sense in isolation.
HOST_A: The model gets "it causes headaches in 5% of patients" with no antecedent.
HOST_B: Exactly. The fix is usually bigger chunks, or parent document retrieval. The second failure: chunks too large. You swing the other way and chunk at two thousand tokens. Now each chunk has so much information that when you retrieve it, the model has to dig through a wall of text to find the relevant part, and it gets confused or misses it.
HOST_A: Lost in the middle problem, at the chunk level.
HOST_B: Right. Third failure, and this one's insidious: wrong embedding model. You embed your domain-specific technical documents with a general-purpose embedding model that wasn't trained on that vocabulary. The embeddings don't capture the domain-specific semantics well. "Myocardial infarction" and "heart attack" might not be close in a general embedding model, but would be in a medical one.
HOST_A: So domain-specific embeddings matter.
HOST_B: A lot. There are specialized embedding models for legal, medical, financial text. It's worth evaluating whether a domain-specific model outperforms the general one for your use case.
HOST_A: What else?
HOST_B: Fourth: not filtering by access control. In enterprise RAG, different users should have access to different documents. If your RAG system retrieves from a pool that includes HR records, financial projections, and customer PII — and not everyone should see all of that — you need metadata filtering at the retrieval layer. This is a security failure waiting to happen and I've seen it overlooked in demos that go to production.
HOST_A: That one is genuinely scary.
HOST_B: Yes. Fifth: no evaluation, ever. You build the system, it looks good in a few demos, you ship it. Six months later you're not sure why it's giving bad answers on some queries and no one has baseline metrics to compare against.
HOST_A: So you need to build an eval set from day one.
HOST_B: From day one. Before you optimize anything, establish your baseline metrics so you know whether your changes actually help.
HOST_A: Let me ask about real companies. Because I think concrete examples help. What are some real companies building RAG systems and what can we learn from them?
HOST_B: Notion AI is a great example. They use RAG to let you ask questions about your own Notion workspace. The challenge they face is that Notion documents are highly personal and contextual — a document called "Q3 plans" means something different for every user. Their retrieval has to be personalized and access-controlled per user.
HOST_A: And GitHub Copilot?
HOST_B: GitHub Copilot does something sophisticated: when you're writing code, it retrieves relevant context from your open files, recently opened files, and the broader repository. The "retrieval" there isn't traditional vector search — it's more of a context window management problem. They've done a lot of work on which code context to include to get the best completions.
HOST_A: What about Perplexity?
HOST_B: Perplexity is essentially a RAG system where the retrieval is live web search. They take your query, search the web, retrieve the top results, and then generate a synthesized answer with citations. It's a great demo of RAG at scale with fresh data. Their challenge is that web content is noisy and contradictory, so their synthesis step has to handle conflicting sources.
HOST_A: And they show you the sources, which is that auditability point you mentioned.
HOST_B: Right. You can click through to verify every claim. That's the ideal RAG UX in my opinion. Not just the answer, but the answer plus the receipts.
HOST_A: Okay I want to pull back and do a synthesis, because we've covered a lot of ground. I want to know where you land on the big questions and where my thinking has actually shifted.
HOST_B: Let's do it. The first big question: is RAG the solution to hallucinations?
HOST_A: I came in thinking yes, RAG is the hallucination solution. I leave thinking it's a partial solution that shifts the problem to retrieval quality — which is more tractable, but it's not solved. And the crucial thing you said that stuck with me: you need to evaluate retrieval quality specifically, not just end-to-end answer quality. Because you can have a system with great-sounding outputs that has terrible retrieval under the hood, and you won't know until it fails on something important.
HOST_B: That's right. And I want to add to that — the retrieval quality problem is actually a content quality problem. If your source documents are wrong, poorly written, contradictory, or incomplete, RAG faithfully retrieves and grounds answers in bad information. Garbage in, garbage out is real.
HOST_A: The second big question: will long-context models make RAG obsolete?
HOST_B: I came into this conversation thinking that long context is a serious threat to RAG's relevance. I still think that for small corpora, long context will displace retrieval for many use cases within a few years. But you challenged me on the cost argument and the attention dynamics, and I think I was understating those. At scale — millions of documents, cost-sensitive deployment, latency requirements — RAG will remain essential.
HOST_A: And actually that "lost in the middle" paper really changed my thinking. I hadn't thought about the fact that having all the information doesn't mean the model attends to it uniformly. Retrieval isn't just about finding information — it's about organizing attention.
HOST_B: That's a really nice way to put it. Retrieval is curating what the model pays attention to.
HOST_A: Third question: RAG versus fine-tuning?
HOST_B: Not a competition. They solve different things. Fine-tuning for behavior and style, RAG for knowledge. Use both when you can. Start with RAG when resources are limited.
HOST_A: What's changed most in your thinking through this conversation?
HOST_B: Honestly? The RAGAS evaluation stuff. I've been in too many demos where someone builds a RAG system, shows five impressive examples, and declares victory. The discipline of measuring faithfulness and context recall separately — and building an eval set before you start optimizing — is something I want to be more rigorous about.
HOST_A: For me it's the chunking. I had no idea how much the chunking strategy matters. I thought it was a boring implementation detail, and you've convinced me it might be the highest-impact lever in the whole pipeline.
HOST_B: It really is. And it's so overlooked.
HOST_A: Okay, last thing: for someone who's never built a RAG system and wants to start — what's the one-sentence advice?
HOST_B: Start simple: use LangChain or LlamaIndex, chunk your documents at five hundred tokens with a hundred tokens overlap, use OpenAI or Cohere embeddings, use pgvector or Chroma for storage, build a twenty-question eval set before you start, and then iterate on the things RAGAS tells you are failing. Don't try to implement HyDE and ColBERT and hybrid search on day one.
HOST_A: That's more than one sentence but I'll let it go. For me: the thing I'll carry away is that RAG is not a product, it's a process. You're constantly iterating on chunking, retrieval, re-ranking, prompting. Teams that treat it as a one-time build will have mediocre systems. Teams that treat it as an ongoing engineering discipline will have great ones.
HOST_B: I love that framing. RAG as process, not product.
HOST_A: Well that was a properly nerdy forty minutes. Ryan, this was great.
HOST_B: Thanks Emma. I always enjoy these conversations where we actually disagree about stuff and have to work it out.
HOST_A: Agreed. Thanks everyone for listening to Clawd Talks. If you want to dig deeper into anything we covered today — the Lewis et al. 2020 RAG paper, the RAGAS evaluation framework, the ColBERT paper by Khattab and Zaharia — I'll put all the links in the show notes. Until next time.
HOST_B: Take care, everyone. Build something good.
HOST_A: Okay wait, I have one more thing I want to say before we sign off.
HOST_B: Ha, go for it.
HOST_A: I just want to name the thing that we danced around for the whole episode: the fundamental tension in RAG is that you're using a system that was known to hallucinate, to process information that was retrieved by a system that might retrieve the wrong thing, to generate an answer for a user who might not know enough to verify it. Every step has failure modes, and they compound.
HOST_B: Yeah. That's the honest picture. RAG is better than not RAG. But "better" doesn't mean "safe."
HOST_A: And I think that's actually the most important thing for practitioners to hold onto. Don't let the existence of grounding documents lull you into thinking the system is reliable. You still need evaluation, human oversight, and epistemic humility about what your AI system actually knows.
HOST_B: Beautifully put. Okay, now we sign off.
HOST_A: Now we sign off. See you all next time.
HOST_B: Cheers.
HOST_A: Oh actually, one genuinely final thing — can we just appreciate how far we've come? In 2020, Lewis et al. published the original RAG paper and it was a research breakthrough. In 2026, RAG is so mainstream that it's considered table stakes for enterprise AI. That's a remarkable pace of adoption for a research technique.
HOST_B: Six years from academic paper to table stakes. That is fast. For context, most academic CS research takes a decade to make it to production. RAG made it in six years partly because the infrastructure caught up so quickly — the vector databases, the embedding model APIs, the frameworks. It wasn't just the idea, it was the tooling ecosystem.
HOST_A: And we're not done. Agentic RAG, multi-modal RAG where you're retrieving images and audio, knowledge graphs — there's still a lot of evolution coming.
HOST_B: Multi-modal RAG is its own whole episode. Remind me about that one.
HOST_A: Noted. Okay, truly signing off now. Thanks for listening.
HOST_B: Thanks all. Cheerio.
HOST_A: You know that's not actually how British people talk, right?
HOST_B: I'm leaning into the stereotype. It's a podcast tradition at this point.
HOST_A: Fair enough. Bye everyone.
HOST_B: Goodbye!
HOST_A: So let's talk about something that came up in my prep that I thought was fascinating — the question of document freshness within a RAG corpus. Because you mentioned that RAG lets you update knowledge without retraining. But actually keeping your vector database up to date is its own operational challenge, right?
HOST_B: Oh, absolutely. This is a real production pain point. When a source document gets updated — say your company policy changes, or a medical guideline gets revised — you need to: detect that the document changed, delete the old chunks from your vector database, re-chunk the new document, re-embed the new chunks, and re-insert them. If you skip any of those steps, you end up with stale information in your retrieval index.
HOST_A: And if your RAG system confidently cites an outdated policy, that's a real problem.
HOST_B: Potentially a liability. And this is why I always tell teams: your RAG system is only as fresh as your ingestion pipeline. You need continuous ingestion — webhooks, polling, event-driven updates — not a one-time batch import.
HOST_A: So the operational complexity of RAG isn't just the vector database, it's the whole document lifecycle management.
HOST_B: Right. LlamaIndex has some nice abstractions for this with their document managers and refresh pipelines. But ultimately someone has to own the process: which documents are in scope, how they're identified when they change, how they get re-ingested.
HOST_A: And in enterprises, the documents are often scattered across SharePoint, Confluence, Google Drive, Salesforce — different systems with different APIs. Connectors to all of those are their own project.
HOST_B: Yes. There's a whole category of companies — Unstructured.io, Glean, Guru — that are essentially building document ingestion and knowledge management infrastructure. Because that problem turns out to be harder than the retrieval itself.
HOST_A: Interesting. So the non-sexy data plumbing is often the real bottleneck.
HOST_B: As it usually is. The demo is always "here's the cool LLM doing cool things." The production reality is "here's the ETL pipeline breaking at three AM."
HOST_A: Ha. Okay, and what about document formats? Because enterprise documents aren't all clean text. PDFs, scanned images, spreadsheets, PowerPoints...
HOST_B: PDF parsing is a whole saga. PDFs can be text-based, image-based scans, or some horrible hybrid. Table structures in PDFs are notoriously hard to extract cleanly. An earnings report with complex financial tables — trying to get meaningful chunks out of that for RAG is genuinely hard.
HOST_A: What are teams using for this?
HOST_B: PyMuPDF and pdfplumber for text-based PDFs. Unstructured.io's library for more complex documents — it does OCR, table extraction, layout analysis. For really critical documents, some teams are using multi-modal models directly — feed the PDF page as an image to a vision model for extraction.
HOST_A: So document preprocessing is itself an AI pipeline now.
HOST_B: Increasingly, yes. Vision language models are getting good enough that you can skip the traditional extraction and just ask the model "what's in this document page?" But it's expensive to do at scale.
HOST_A: We should also probably talk about security and data privacy, because this comes up constantly in enterprise.
HOST_B: Yes. Two big concerns. First: data residency. If you're a European company, you might not be able to send your documents to a US-based embedding API. So you need on-premise embedding models — which exist, there are great open-source embedding models like BGE, E5, nomic-embed — but it adds infrastructure.
HOST_A: And second?
HOST_B: Second: the vector database itself. Your embedding vectors encode information about your documents. If you're storing confidential documents, those embeddings are derived from that content. Storing them on a third-party cloud vector database is a real security question. Some companies require on-premise vector storage for sensitive data.
HOST_A: So the full air-gapped enterprise RAG stack — everything on premise — is a thing.
HOST_B: It is. And it's harder and more expensive but increasingly available. Ollama for local LLMs, open-source embedding models, pgvector on your own Postgres, a self-hosted Weaviate cluster. You can build a fully local RAG stack now. Two years ago that was much harder.
HOST_A: That's actually a meaningful shift. The democratization of the whole stack.
HOST_B: Right. And speaking of democratization — I want to circle back to something practical. There's been a real maturation in how teams think about RAG evaluation. Early on, everyone was vibes-based: "does it seem right?" Now there's much more rigor, and not just RAGAS. Companies like Arize AI and Weights & Biases have evaluation and observability tooling specifically for RAG. LangSmith from LangChain does tracing and evaluation. You can see, for every query, what was retrieved, what the prompt looked like, what the model generated, what the latency was.
HOST_A: Observability for RAG, not just the model.
HOST_B: Yes. And I think that's actually the marker of maturity for any AI system — when you have sufficient observability to debug and improve it systematically. We're getting there with RAG.
HOST_A: Okay I think that genuinely wraps it. We went deep on a lot of things we didn't plan to discuss, which is how you know the topic is rich.
HOST_B: That's the sign of a good episode. You can always tell when the hosts are genuinely thinking through something versus just reciting what they prepared.
HOST_A: Well I feel like I've genuinely updated my views a few times today, which is the goal.
HOST_B: Same. Really enjoyed it.
HOST_A: Clawd Talks. Go build things. Bye.
HOST_B: Bye!