An enterprise RAG architecture has two paths. The ingestion pipeline parses, chunks, enriches, embeds and indexes your documents. The query path retrieves, reranks and assembles context for a language model. The vector database is one component sitting in the middle, and in our experience it's rarely where production goes wrong. Most problems come from the parts around it: chunking, permissions, freshness, evaluation and operations.
We make HyperLake, a sovereign AI operating platform. It isn't a vector database or a RAG framework, and the section near the end explains how it relates to the architecture below.

What is enterprise RAG architecture?
Retrieval-augmented generation (RAG) pairs a language model with an external store of knowledge. The original paper by Lewis and colleagues joined a pretrained sequence-to-sequence model with a dense vector index of Wikipedia, searched by a neural retriever (Lewis et al., 2020). The authors framed it as a way to tackle limited knowledge access, weak provenance and knowledge in model weights that's hard to update.
"Enterprise" piles on requirements that paper never had to face: many document sources, per-user permissions, audit, data residency, uptime and a cost budget. Microsoft's Azure Architecture Center describes the same basic shape, with a request flow (query, orchestrator, search, prompt, response) and a separate data pipeline that chunks, enriches, embeds and persists content (Microsoft Learn).
What are the components of a RAG architecture?
| Stage | What it does | Common failure |
|---|---|---|
| Source connectors | Pull documents, tickets, wikis, tables | Stale or partial sync |
| Parsing | Turn PDFs, slides, HTML into clean text | Tables and headings lost |
| Chunking | Split text into retrievable units | Chunks with no context |
| Enrichment | Add title, summary, keywords, ACLs | Missing permission metadata |
| Embedding | Convert chunks to vectors | Model mismatched to domain |
| Index and store | Vector index plus text index | Slow or low-recall search |
| Retrieval | Dense, sparse or hybrid search | Right passage never returned |
| Reranking | Re-score top candidates | Added latency, no gain |
| Generation | Prompt with context, produce answer | Ignores or misuses context |
| Evaluation | Measure each stage and the end result | No baseline, so no progress |
The Microsoft guide recommends evaluating each step on its own, while keeping in mind that users only ever see the end result. Here's why that matters. If you only measure final answers, you can't tell whether a bad one came from parsing, retrieval or generation.
How do vector databases fit into RAG?
A vector database stores embeddings and returns the ones closest to a query embedding. For a plain-language explanation of indexing and storage, see Vector Databases and Semantic Search Explained and Vector Storage: How Semantic Search Works. For index types compared, see Vector Indexing Algorithms: HNSW vs IVF-PQ vs ScaNN.
The first decision isn't which product. It's which kind of deployment:
- A vector extension in a database you already run. pgvector adds vector search to PostgreSQL with HNSW and IVFFlat indexes (pgvector README). Vectors stay next to the rest of your data and its access controls, which is simpler to run at modest scale.
- A dedicated vector engine. Engines such as Qdrant support dense and sparse vectors, full-text matching and multi-stage queries (Qdrant docs). It makes sense when vector search is a core workload with its own scaling needs.
- A managed cloud search service, if you want someone else to run the index. Check where data is stored and who can get at it.
Our upcoming guide to open-source vector databases for enterprise RAG compares specific engines. For architecture, the point is simpler: keep your chunking, metadata and evaluation outside the vector store, and the store stays replaceable.
What are the main challenges in enterprise RAG, and how do you solve them?
Why do relevant chunks fail to get retrieved?
Chunking can strip out the very thing that makes a chunk understandable. Anthropic's example is a chunk saying a company's revenue grew by 3%, with no hint of which company or which period. Its Contextual Retrieval method adds short, chunk-specific context before embedding and before building the keyword index. In Anthropic's own tests, the top-20 retrieval failure rate fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when combined with contextual BM25, and to 1.9% when reranking was added (Anthropic). Those are one vendor's results on its own datasets. Treat them as a direction to test, not a guarantee.
The practical fixes, roughly in the order teams try them:
- Chunk along document structure (headings, sections) instead of fixed character counts. See Semantic Chunking vs. Hierarchical and Structure-Aware.
- Attach metadata to every chunk: title, section, date, source, owner.
- Add context to each chunk before embedding, as described above.
- Pull back the parent section when a small chunk matched; see Multi-Vector and Parent Document Retrieval Patterns.
How do hybrid search and reranking help?
Dense vectors capture meaning but can miss exact terms like part numbers, error codes and names. Keyword search (BM25) has the opposite problem. Qdrant's documentation describes combining dense, sparse and full-text searches and fusing them with Reciprocal Rank Fusion, which scores each document by its rank in each result list. The same page warns against blending raw dense and BM25 scores with a simple weighted sum, since their scales differ. pgvector's README describes combining vector search with PostgreSQL full-text search and points to Reciprocal Rank Fusion or a cross-encoder for merging the rankings.
A reranker then re-scores the top candidates with a more accurate, slower model. See Contextual Compression: ColBERT vs. Cross-Encoders. Weigh the latency you add against the recall you gain. Sometimes the gain is small.
How do you enforce permissions in RAG?
This is the one enterprise teams underestimate. OWASP lists vector and embedding weaknesses as LLM08:2025 and notes that when several user groups share one vector database, one group's embeddings can turn up in another group's queries. Its mitigations include strict logical and access partitioning of datasets, validating knowledge sources, classifying data and keeping immutable logs of retrieval activity (OWASP).
Two patterns are common:
- Filter at query time. Store the allowed groups on each chunk and filter on the caller's groups.
- Partition by tenant or sensitivity. Use separate collections or indexes, so a mistake in a filter can't cross the boundary.
Here's a generic illustration of the first pattern in SQL with pgvector. The table, columns and group names are hypothetical.
-- Illustration only: filter by the caller's groups, then rank by distance
SELECT id, content
FROM chunks
WHERE allowed_groups && ARRAY['finance', 'all-staff']
ORDER BY embedding <=> '[0.12, -0.40, 0.88]' -- query embedding (3 dims for brevity)
LIMIT 5;There's a catch. pgvector's README notes that with approximate indexes, filters run after the index scan, so a WHERE clause can return fewer rows than the LIMIT asks for. It describes iterative index scans, partial indexes and partitioning as remedies. Every engine with filtered search has its own version of this trade-off, so test filtered queries with realistic permission mixes, not just unfiltered ones. Related controls are in Row Filtering and Column Masking for Data Security and LLM Access Control: RBAC for Models and AI Agents.
How do you keep the index fresh?
Documents change. They get deleted, or someone loses access to them. A RAG system that only ever appends will keep answering from superseded or revoked content. Plan for:
- change detection at the source (timestamps, or change data capture for databases; see Debezium Change Data Capture);
- deletions and permission changes that reach the index, not just new documents;
- a stored document version and ingestion date on every chunk, so answers can cite them.
What happens when the model ignores retrieved context?
Stuffing in more retrieved text doesn't always help. The "Lost in the Middle" study found that models tended to answer best when relevant information sat at the start or end of a long input, and did worse when it was buried in the middle (Liu et al., 2023). So retrieve fewer, better passages, order them on purpose, and tell the model to say so when the context doesn't contain the answer.
How do you evaluate a RAG system?
Build a test set before you tune anything. The Microsoft guide recommends gathering representative documents and test queries, including queries your documents don't cover, and evaluating each phase. Measure retrieval (did the right passage show up in the top results?) separately from generation (was the answer grounded, complete and relevant?). See LLM Evals: How to Test Models Before Production and LLM Observability: Traces, Quality, Cost, and Drift.
What does it cost to scale?
Raw vector storage is easy to estimate. Here's the arithmetic on a hypothetical example: 10 million chunks, each embedded as a 1,024-dimension vector stored as 32-bit floats (4 bytes per number).
- 1,024 dimensions x 4 bytes = 4,096 bytes per vector
- 10,000,000 vectors x 4,096 bytes = 40,960,000,000 bytes, about 41 GB
That's just the raw vectors. Indexes, text, metadata, replicas and backups all come on top. pgvector's README notes that indexes don't have to fit in memory but usually perform better when they do, and that HNSW builds take longer and use more memory than IVFFlat. Dimension count, chunk count and replica count drive storage and memory, so measure them before you pick hardware. For serving costs, see Why AI Inference Is So Expensive.
When should you move beyond standard RAG?
A standard pipeline runs a fixed sequence: query, search, assemble context, generate. Microsoft's guide says this works well for queries that map to a single search against a single index. For multistep reasoning, dynamic source selection, and combining retrieval with actions, it suggests agentic RAG, where an agent treats retrieval as a tool it can call. Agents make the permission problem wider, because every tool call is one more access path. See Task-Based Access Control for AI Agents and Agent Grounding: The Missing Discipline in Enterprise AI.
If the real question is whether you need retrieval at all, read RAG vs. Fine-Tuning: How to Choose. If relationships between entities matter, see Vector Database vs Graph Database vs Knowledge Graphs for AI (2026).
Where does HyperLake fit?
HyperLake doesn't replace a vector database or an embedding pipeline. It's a platform for assembling, deploying and governing the data, models and applications behind AI in a customer's own environment or a client's. According to HyperLake's product description, the engines it works with follow the workload, and these include Qdrant or Milvus for vectors. Its stated approach to private AI is to ground models in governed enterprise context, with policies on data and tool access, and with people and agents reaching data through authenticated, governed services. HyperLake is relevant if you need that governed, self-hosted setting around RAG. For a prototype over a handful of documents, a library and a single database will be simpler.
Frequently asked questions
What is the difference between RAG and a vector database?
RAG is an architecture: retrieve relevant content, then hand it to a language model to generate an answer. A vector database is one component that can do the retrieval step, by storing embeddings and finding the nearest ones. A RAG system can also use keyword search, SQL or a graph instead of a vector database, or alongside one.
Do I need a dedicated vector database for enterprise RAG?
Not always. A vector extension such as pgvector keeps vectors in PostgreSQL, which may be enough at moderate scale and makes permissions and operations simpler. A dedicated engine gets attractive when vector search needs its own scaling, filtering or hybrid-search features, so test both against your own data and query mix.
What is the biggest challenge in enterprise RAG?
It varies. The two we see missing most often are permission-aware retrieval and evaluation. Without per-user filtering you risk exposing restricted content, and without a test set you can't tell whether a change helped.
How do I improve RAG accuracy?
Measure retrieval and generation separately, then change one thing at a time: chunking, metadata, hybrid search, reranking, then the prompt. Anthropic's published results suggest adding context to chunks and reranking can reduce retrieval failures on its test data, so verify that on yours.
How should I deploy RAG when data cannot leave my environment?
Run the embedding model, vector store, reranker and language model inside infrastructure you control, and keep logs and keys there as well. Check each component's licence and whether it works offline. For deployment context, see On-Premises AI: Self-Hosted and Hybrid Deployment.
Is RAG still relevant with long-context models?
Yes. A long context doesn't remove the need to choose what goes into it. The "Lost in the Middle" findings suggest models don't use all of a long input equally well, and retrieval also gives you permission filtering, freshness and source citation.



