Overview
Retrieval-augmented generation (RAG) looks deceptively simple in a demo: embed documents, retrieve top-k chunks, ask the model to answer. In production, quality collapses when corpora grow, permissions matter, and users ask ambiguous questions.
Introduction
Treat RAG as a data product plus an application, not a prompt experiment. The retrieval layer, chunking strategy, ranking, and evaluation harness decide whether answers are useful—or confidently wrong.
Chunking and metadata
Chunk by semantic boundaries, not fixed token windows alone. Preserve headings, document IDs, ACL tags, and freshness dates so retrieval can filter before ranking. Bad chunks make even the best models hallucinate.
Hybrid retrieval
Combine dense vector search with keyword/BM25 for identifiers, error codes, and rare terms. Re-rank with a cross-encoder or LLM judge when latency budgets allow. Hybrid pipelines consistently beat single-retriever setups on enterprise corpora.
Grounding and citations
Force answers to cite retrieved passages. If retrieval confidence is low, refuse or escalate instead of inventing. Users trust systems that know when they do not know.
Security and tenancy
Enforce document-level ACL filters in the retrieval query—never only in the UI. Isolate indexes by tenant where required, and redact sensitive fields before embedding.
Evaluation
Track retrieval recall, answer groundedness, latency, and cost per query. Run golden-question suites on every change to chunking, embeddings, or prompts. Without evaluation, RAG improvements are vibes.
Key Takeaways
Ship a narrow domain first, instrument quality, then expand corpus coverage. Production RAG is an iterative platform capability—not a one-week integration.