RAG Systems Engineer

A basic RAG system is easy to build. Embed some documents, store vectors, retrieve the top-k at query time, stuff them in the context window, and generate. The problem is that basic RAG fails in predictable ways: it retrieves documents that are semantically similar to the query but not actually relevant, it loses coherence when the retrieved chunks are out of context, and it degrades as the knowledge base grows because nearest-neighbor search stops being discriminative at scale.

Production RAG requires solving each of these failure modes explicitly. Hybrid retrieval - combining dense vector search with sparse BM25 keyword search - dramatically outperforms either approach alone, especially for domain-specific terminology that embeddings do not always capture well. Cross-encoder reranking adds a second pass that catches cases where the first-pass retrieval ranked results by surface similarity rather than relevance. Context window management ensures that retrieved chunks are assembled in an order that is coherent for the model to read, not just a bag of relevant paragraphs.

The most important metric for a RAG system is not retrieval recall in isolation - it is end-to-end answer quality on the actual questions users ask. Building evaluation pipelines that measure this directly, rather than proxying with embedding similarity scores, is what separates a RAG system that works in production from one that works in a demo.

LUNA's memory system is a practical example of production RAG design. The core requirement was simple: conversational context should persist across sessions and survive model swaps. Switching from a local Ollama model to Claude's API mid-project should not lose the context of everything discussed so far.

This required a retrieval layer that is completely decoupled from the inference layer. Memory is stored in a structured format that any model can read - not tied to any specific model's context window format or token counting method. At query time, relevant memories are retrieved based on the current conversation context, ranked by relevance and recency, and injected into the prompt before the LLM generates a response.

The design also required managing what not to retrieve. Injecting every memory into every prompt would quickly exhaust context window limits and degrade response quality as the model tries to attend to too much irrelevant history. The retrieval system filters aggressively, bringing in only memories that are likely to be relevant to the current turn. This is the same challenge that every production RAG system faces - retrieval precision matters as much as retrieval recall.