The infrastructure behind retrieval-augmented generation in production

A retrieval-augmented generation prototype takes an afternoon: load some PDFs, split them, embed the chunks, put them in a vector store, and pass the top results to a model with the question. It answers well in the demo.

Production is a different system. Documents change every day and the answers must reflect that. Some documents are confidential and must never reach users who cannot open them. The index holds millions of chunks, not hundreds. And when an answer is wrong, someone has to find out whether retrieval or generation failed. Most of that work is infrastructure, not prompting.

Ingestion is a data pipeline

Treat ingestion like any other data pipeline, with the same expectations about reliability and observability:

• Connectors per source. Wikis, ticket systems, file shares and databases each need their own extraction, with credentials, rate limits and pagination.
• Extraction and cleaning. Converting PDFs, HTML and office documents to clean text is where quality is quietly lost. Tables, headings and lists carry meaning; flattening them into one paragraph hurts retrieval.
• Stable document identity. Every chunk must trace back to a source document ID and version, so it can be updated or deleted later.
• Queue-based processing. Put work on a queue so a burst of changes, or a slow embedding API, does not lose updates. Make every step idempotent.

Store the extracted text separately from the index. When you change the chunking strategy or the embedding model, you will re-process everything, and re-extracting from the sources is the slowest and most fragile part.

Chunking

Chunks are the unit of retrieval, so their size is a trade-off. Small chunks match precisely but lose context. Large chunks keep context but dilute the match and consume the model's context window.

Practical defaults that hold up well:

• Split on structure first: headings, sections, list items, table rows. Fall back to fixed token windows only within long sections.
• A few hundred tokens per chunk, with modest overlap, is a reasonable starting point to tune from.
• Carry context into the chunk: prepend the document title and section heading, so a chunk that says "the limit is 30 days" still says what the limit is about.
• Keep metadata with every chunk: source, URL, last modified time, language, and the access control information discussed below.

Embeddings

The embedding model turns each chunk and each query into a vector. A few decisions follow from the choice:

• Dimensions drive storage and memory. A 1,536-dimension float vector is about 6 KB before index overhead. Multiply by the number of chunks to size the store.
• Hosted or self-hosted. Hosted APIs such as OpenAI's text-embedding-3 models or Cohere Embed are simple to start with. Self-hosted open models remove per-token cost and keep data in your environment, at the price of running GPU or CPU inference.
• The model version is part of the index. Vectors from different models are not comparable. Record the model name with every vector, and plan for re-embedding the whole corpus when you change it, by building a new index alongside the old one and switching over.

Choosing a vector store

For most teams, the choice is between adding vectors to a database they already run and adopting a search engine. Dedicated vector databases exist and scale well, but each one is another stateful system to operate.

With pgvector, the schema is ordinary SQL:

CREATE EXTENSION IF NOT EXISTS vector;
 
CREATE TABLE chunks (
  id               bigserial PRIMARY KEY,
  document_id      text NOT NULL,
  document_version text NOT NULL,
  content          text NOT NULL,
  allowed_groups   text[] NOT NULL,
  embedding_model  text NOT NULL,
  embedding        vector(1536) NOT NULL,
  source_updated_at timestamptz NOT NULL
);
 
CREATE INDEX chunks_embedding_hnsw
  ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX chunks_allowed_groups ON chunks USING gin (allowed_groups);

Approximate indexes and filters interact badly: the index returns the nearest candidates, and the filter then removes most of them, leaving too few results. pgvector 0.8 added iterative index scans, enabled with SET hnsw.iterative_scan = relaxed_order, which keep scanning until enough rows pass the filter. On OpenSearch, use an engine that supports filtering during the search rather than after it.

Hybrid search, combining keyword and vector results, is worth enabling from the start. Vector search misses exact identifiers such as error codes, product SKUs and names, which keyword search finds trivially.

Freshness

Users trust a RAG system until it confidently quotes last quarter's policy. Freshness needs to be designed:

• Change-driven updates. Use webhooks or change data capture from sources where available, and incremental polling by modification time elsewhere.
• Replace, do not append. When a document changes, delete its old chunks and write the new ones in one transaction, keyed by document ID.
• Handle deletions. A document removed at the source must disappear from the index. Periodic reconciliation, comparing the set of IDs in the index with the source, catches missed deletes.
• Measure lag. Track the time between a source change and the index update, and alert when it grows.

Access control

This is where production RAG most often goes wrong. If the index contains a document, any user's question can retrieve it, and the model will happily summarize it.

Enforce permissions at retrieval time, inside the query, before results reach the model:

SET hnsw.iterative_scan = relaxed_order;
SELECT id, document_id, content
FROM chunks
WHERE allowed_groups && $2::text[]
ORDER BY embedding <=> $1
LIMIT 8;

Here the second parameter is the requesting user's groups, resolved from the identity provider on each request, never from anything the user or the model supplies. Sync permissions from sources as carefully as content: a document that becomes restricted must lose its old groups in the index promptly. For strict tenant separation, separate indexes or databases per tenant are simpler to reason about than filters.

Evaluation

Without measurement, every change to chunking, embeddings or prompts is a guess. Evaluate retrieval and generation separately:

• Retrieval: build a set of real questions with the documents that should answer them. Measure whether those documents appear in the top results (recall at k) and how high they rank.
• Generation: check whether answers are supported by the retrieved context and actually address the question. Frameworks such as Ragas automate these checks with a model acting as judge; spot-check the judge against human review.
• Production signals: log queries, retrieved chunk IDs, the answer and user feedback, with care for sensitive content. Questions with no good retrieval results show where the corpus has gaps.

Run the evaluation set in CI whenever the pipeline, the index or the prompt changes.

Where to start

1. Pick one source with clear ownership and permissions, and build ingestion with stable document IDs and stored extracted text.
2. Chunk by structure, with titles and headings carried into each chunk.
3. Start with the database you already operate: pgvector if you run Postgres, OpenSearch if you run it already.
4. Put access control in the retrieval query from the first day, using groups from your identity provider.
5. Wire up change-driven updates and a nightly reconciliation for deletes.
6. Write fifty real questions with expected sources, and run them on every change.