What Is RAG and Why Does It Matter
Large language models like GPT-4 are trained on data up to a cutoff date. They know a lot — but they don't know your company's latest documentation, your product specs, your internal policies, or what happened last week. Ask a plain LLM about your business and it will either guess (hallucinate) or say it doesn't know.
RAG — Retrieval-Augmented Generation — solves this by giving the LLM access to a curated, up-to-date knowledge base at query time. Instead of relying solely on training data, the model first retrieves the most relevant documents, then generates an answer grounded in those documents.
The result: an AI system that answers questions about your specific domain accurately, with sources, without hallucinating.
How RAG Works: The Architecture
A RAG pipeline has three main components:
### 1. Ingestion (Building the Knowledge Base) Your source documents — PDFs, web pages, database records, support tickets, product manuals — are chunked into segments and converted into vector embeddings (numerical representations of meaning) using an embedding model like OpenAI's text-embedding-3 or open-source alternatives like Sentence Transformers.
These embeddings are stored in a vector database: Pinecone, Weaviate, pgvector (PostgreSQL extension), or Qdrant. The vector database makes semantic similarity search fast at scale.
### 2. Retrieval (Finding Relevant Context) When a user asks a question, their query is also converted to a vector embedding. The vector database is searched for the chunks most semantically similar to the query — not just keyword-matching, but meaning-matching. "How do I cancel my subscription?" would retrieve chunks about cancellation policy even if they use the word "terminate" rather than "cancel".
The top-k most relevant chunks (typically 3–8) are retrieved and assembled into a context window.
### 3. Generation (Answering with Context) The retrieved context is passed to the LLM along with the user's query in a structured prompt: "Using only the following documents, answer this question..."
The LLM generates a response grounded in the retrieved documents. With proper prompting, it will cite sources and refuse to speculate beyond what the documents say.
Chunking Strategy: Often the Most Important Decision
How you split documents into chunks has more impact on RAG quality than model choice. Get this wrong and retrieval is noisy; get it right and accuracy is dramatically higher.
Fixed-size chunking — Split text into chunks of N tokens with overlap. Simple, works well for dense technical text.
Semantic chunking — Split at natural semantic boundaries (paragraphs, sections). Preserves context better for narrative content.
Hierarchical chunking — Store both summary-level and detail-level chunks. Retrieve summaries first, then drill down. Better for long documents with complex structure.
Sentence-window chunking — Index individual sentences but retrieve surrounding context. High precision for factual Q&A.
For most business applications, semantic chunking with 512–1024 token chunks and 20% overlap is a strong starting point.
Choosing the Right Vector Database
| Database | Best For | Hosting | Scale | |---|---|---|---| | Pinecone | Managed, production, fastest setup | Cloud | Millions of vectors | | pgvector | Already using PostgreSQL, want to minimise infra | Self-hosted | Up to ~1M vectors | | Weaviate | Multi-modal, GraphQL queries | Cloud + self-hosted | Large scale | | Qdrant | Open-source, high performance | Self-hosted | Large scale | | Chroma | Prototyping, local dev | Local | Small scale |
For most business RAG applications with < 500,000 document chunks, pgvector on your existing Postgres instance is the simplest production path. For larger scale or managed infrastructure, Pinecone.
Evaluation: How to Know if Your RAG System Is Working
The failure mode most teams miss: the RAG system seems to work in demos but gives wrong answers in production. Proper evaluation catches this.
Retrieval evaluation: - Recall@k — Does the correct chunk appear in the top-k results? - Precision — Are the retrieved chunks actually relevant?
Generation evaluation: - Faithfulness — Does the answer stay within the retrieved context, or does it hallucinate? - Answer relevance — Does the response actually answer the question? - Context utilisation — Is the model using the retrieved context or ignoring it?
Tools like RAGAS provide automated evaluation pipelines. Running evaluation on a labelled test set before launching is non-negotiable for production systems.
Common RAG Failure Modes
Hallucination despite RAG — The model goes beyond the retrieved context. Fix: stricter system prompts, temperature tuning, output validation.
Wrong chunks retrieved — Embedding model doesn't capture domain-specific semantics well. Fix: domain-specific embedding fine-tuning, hybrid retrieval (vector + BM25 keyword).
Chunk boundary issues — A key piece of information is split across two chunks, neither of which retrieves fully. Fix: overlap strategy, larger chunk size for that document type.
Stale knowledge base — Documents updated but embeddings not re-indexed. Fix: incremental ingestion pipeline with change detection.
Too much context — Retrieving too many chunks overwhelms the model's context window. Fix: relevance scoring threshold, reranking.
Hybrid Retrieval: Combining Vector and Keyword Search
Pure vector search has a weakness: for exact matches (product codes, names, dates), keyword search (BM25) often outperforms semantic similarity. Production RAG systems often use hybrid retrieval — combining vector similarity scores with BM25 scores using Reciprocal Rank Fusion (RRF).
This handles queries like "What is the return policy for SKU-48293?" far better than vector-only retrieval.
Building a Production RAG System
A production RAG deployment at NeuragenceAI typically includes:
- Ingestion pipeline — Scheduled document ingestion with change detection, chunking, embedding, and upsert to vector store
- Retrieval layer — Hybrid search with reranking (Cohere Rerank or cross-encoder model)
- LLM layer — GPT-4o or Claude 3.5 Sonnet with structured output and citation extraction
- API layer — FastAPI or Express backend, streaming responses, conversation history management
- Evaluation pipeline — Automated RAGAS evaluation on test set, alerting on quality regression
- Analytics — Query logging, unanswered question tracking, chunk utilisation analysis
The hardest part isn't the technology — it's the data quality. Clean, well-structured, consistently updated source documents are the foundation of a RAG system that actually works.
If you're evaluating whether RAG is right for your use case, the deciding question is: does your application require answering questions about a specific, evolving body of knowledge? If yes, RAG is almost certainly the right architecture.