If you’ve spent any time around GenAI engineering, you’ve built a Retrieval-Augmented Generation pipeline . Probably with LangChain. Probably in an afternoon. And if you’re anything like me, you probably walked away with a working demo and a nagging feeling that you didn’t actually understand what just happened.
That feeling is what sent me back to basics. No frameworks. No abstractions doing the thinking for me. Just PDFs, Python, a local LLM, and the actual math.
Here’s what building it the hard way taught me — and why I think every engineer working with RAG in production should do this at least once.
In this article
The problem with learning RAG through frameworks
LangChain is genuinely great software. But when you call
VectorStoreRetriever.from_documents()
,
a dozen decisions get made for you silently: how text gets split, how much overlap exists between chunks, how similarity gets computed, and what actually gets stuffed into the prompt.
When your RAG pipeline in production gives a wrong or hallucinated answer, you need to know exactly where in that chain things went sideways. If you’ve never built it without the framework, you’re debugging a black box.
Chunking is where most of the real decisions live
I expected embeddings to be the hard part. They weren’t. Chunking was.
My first instinct — and I suspect most people’s first instinct — was to slice text every N characters. It works. It also quietly wrecks retrieval quality because chunks start mid-sentence and end mid-thought. The model has no coherent unit of meaning to embed.
Switching to sentence-aware chunking — packing complete sentences up to a target size and carrying a little overlap into the next chunk — changed retrieval quality more than any other decision in the project. It’s a small implementation detail with an outsized effect, and debugging my own broken chunks made that visceral.
Cosine similarity is just a dot product
This was the moment I finally saw behind the curtain. Once embedding vectors are normalized, finding the most similar chunk reduces to a matrix multiplication between the query vector and the matrix of chunk vectors. That’s it: linear algebra applied to language.
scores = chunk_vectors @ query_vector
Building the vector store as a plain Python list and a NumPy matrix made this click in a way that reading about approximate nearest-neighbor search never did. Real vector databases solve a scaling problem, not a conceptual one. At a few thousand chunks, matrix multiplication is plenty fast. Once you see that, database choice becomes an engineering tradeoff instead of a tutorial checkbox.
The model will lie confidently — only your prompt stops it
The most humbling moment was asking my pipeline a question with no answer in the source PDFs — and watching a local 8B model invent a fluent, convincing, entirely wrong response.
The fix wasn’t clever. It was one explicit line in the prompt:
“If the context doesn’t contain the answer, say you don’t know.”
That single instruction was the difference between a system I could trust and one that looked impressive until it quietly lied to someone. Building the prompt by hand, instead of trusting a framework’s default template, made me test for that failure deliberately.
What I’d tell someone starting today
Frameworks like LangChain earn their place once you’re moving fast and the fundamentals are second nature. But there’s durable value in building the naive version first. It’s the difference between using a tool and understanding what the tool is actually doing for you.
That difference matters in production.
It separates engineers who can debug a RAG system from engineers who can only rebuild the demo.
Explore the complete source code, implementation, and project documentation on GitHub.
View the GenAI RAG Pipeline on GitHub →