Vector Databases Explained for RAG

By Kokil Thapa | Last reviewed: September 2026

You need vector databases explained for RAG because most teams hit the same wall: the LLM knows general facts, but it does not know your product docs, legal forms, or order history. Retrieval-Augmented Generation (RAG) fixes that by fetching relevant chunks at query time and passing them into the prompt. The vector database is the engine that makes that retrieval fast and accurate. If you already read our RAG explained guide, this page goes deeper on the storage and search layer that sits behind it.

What is a vector database and why does RAG need one?

A vector database stores high-dimensional numeric arrays called embeddings. Each embedding represents the semantic meaning of a chunk of text, an image, or another object. Traditional SQL indexes excel at exact matches and ranges. They fail when a user asks “How do I cancel a booking?” and your docs say “refund policy for trek deposits.” The phrases differ, but the meaning overlaps.

Vector search closes that gap. You embed the user question, compare it against stored document vectors, and return the nearest neighbours. That is the retrieval step in RAG. Without it, you either stuff the entire knowledge base into the prompt (expensive and impossible at scale) or hope the model guesses correctly (risky for legal, medical, or financial content).

On legal-tech portals I have worked on, RAG is attractive because you can update a notary checklist or court-marriage FAQ without retraining a model. You re-index the changed pages. The vector store holds the searchable memory. The LLM handles language. That split keeps costs predictable and answers traceable to source documents.

Embeddings: the unit vector databases actually store

An embedding is a fixed-length float array. OpenAI’s text-embedding-3-small returns 1536 dimensions by default. Cohere, Voyage, and open-source models like nomic-embed-text use different sizes. You must embed documents and queries with the same model and dimension count. Mixing models breaks similarity math entirely.

Distance metrics matter. Cosine similarity is the default for normalized embeddings. Inner product (dot product) works when vectors are not normalized. Euclidean (L2) distance appears in some libraries. PostgreSQL’s pgvector extension supports all three. Pick one metric and stay consistent across index creation and query code.

Metadata filters alongside vectors

Production RAG rarely runs pure vector search alone. You filter by tenant ID, document type, language, or publication date. A law firm portal might scope search to “client-visible” files only. A multi-vendor marketplace might restrict results to one seller’s catalogue. Store metadata in the same row as the vector. Apply SQL or JSON filters before or after the approximate nearest-neighbour step depending on the engine.

How does the RAG ingestion and retrieval pipeline work?

RAG has two distinct phases: offline indexing and online retrieval. Teams that treat them as one script usually regret it. Indexing is batch-oriented, idempotent, and safe to rerun. Retrieval is latency-sensitive and runs on every user message.

1. Extract — Pull text from HTML, Markdown, PDF, database rows, or API responses.
2. Chunk — Split text into overlapping segments, typically 300–800 tokens with 10–20% overlap.
3. Embed — Call an embedding API or local model for each chunk.
4. Upsert — Write vector + text + metadata into the vector store.
5. Query — Embed the user question, fetch top-K neighbours, optionally re-rank.
6. Generate — Pass retrieved chunks into the LLM system prompt and return the answer with citations.

Chunk quality drives retrieval quality more than index tuning does. A common mistake is splitting on fixed character counts and cutting sentences in half. Prefer structure-aware chunking: headings, paragraphs, or logical sections from your CMS. On a Laravel documentation site, chunking by heading hierarchy often beats naive token windows.

Approximate nearest neighbour (ANN) indexes

Exact brute-force comparison against every vector works for thousands of rows. It fails at millions. ANN algorithms trade a small accuracy loss for large speed gains. HNSW (Hierarchical Navigable Small World) is the most common choice in 2026. IVFFlat partitions vectors into lists and searches a subset. pgvector supports both. Managed services like Pinecone and Weaviate build indexes automatically.

Index build time and memory use are operational costs people forget. HNSW indexes can consume several times the raw vector storage. Plan disk and RAM on your PostgreSQL server before you index 500k legal document chunks. Our pgvector vs Pinecone comparison walks through those trade-offs on real hosting budgets.

Re-ranking after vector retrieval

Top-K vector results are fast but imperfect. A cross-encoder re-ranker scores each query-chunk pair more accurately than bi-encoder embeddings alone. The pattern: retrieve 20–50 candidates by vector search, re-rank to the best 5, then send those to the LLM. Latency increases by 100–300 ms. Answer quality often jumps noticeably on technical documentation and long-form legal guides.

Which vector database should you choose for production RAG?

There is no universal winner. The right choice depends on data volume, existing stack, team skills, and latency budget. PHP and Laravel teams often already run MySQL 9.7 or PostgreSQL 18. Adding pgvector to PostgreSQL keeps one database for transactional data and vectors. That simplifies backups, replication, and access control you already understand from multi-tenant database design.

For most client projects I evaluate, pgvector inside PostgreSQL wins when vector count stays under a few million and the team already runs Laravel with PostgreSQL. Managed Pinecone makes sense when you need sub-50 ms search at scale and no DBA time. Read RAG vs fine-tuning before you pay for either a large vector cluster or model training.

When a plain relational table is enough

Not every “AI search” feature needs a dedicated vector engine on day one. Below roughly 10,000 chunks, a PostgreSQL table with a sequential scan and proper caching can serve internal admin tools. Once p95 query latency crosses your threshold or CPU spikes during search, add an HNSW index. Incremental adoption beats over-provisioning a managed vector cluster for a brochure site with forty FAQ entries.

How do you implement vector search in a Laravel or PHP stack?

I integrate LLM APIs into Laravel applications; I do not train custom embedding models. That is the realistic path for most Nepal agencies and product teams in 2026. You call an embedding endpoint, store results in pgvector, and query from a service class or queued job. Our pgvector and Laravel practical setup covers migration details. Here is the core schema pattern.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,
    tenant_id UUID NOT NULL,
    source_url TEXT NOT NULL,
    chunk_index INT NOT NULL,
    content TEXT NOT NULL,
    embedding vector(1536) NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX document_chunks_embedding_idx
ON document_chunks
USING hnsw (embedding vector_cosine_ops);

Laravel 13 on PHP 8.3+ can use raw DB queries or packages like pgvector/pgvector-php. Keep embedding calls in a queued job so HTTP requests stay fast. Batch embed where the API allows it. OpenAI’s embedding endpoints accept array inputs, which cuts round trips during nightly re-index jobs.

use Illuminate\Support\Facades\DB;
use OpenAI\Laravel\Facades\OpenAI;

$response = OpenAI::embeddings()->create([
    'model' => 'text-embedding-3-small',
    'input' => $chunks,
]);

foreach ($response->embeddings as $i => $item) {
    DB::table('document_chunks')->insert([
        'tenant_id' => $tenantId,
        'content' => $chunks[$i],
        'embedding' => '[' . implode(',', $item->embedding) . ']',
        'metadata' => json_encode(['lang' => 'en']),
    ]);
}

Similarity query with metadata filter

Always parameterize tenant and visibility filters. A missing WHERE tenant_id = ? clause in multi-tenant RAG is a data leak waiting to happen. This query pattern uses cosine distance via the <-> operator in pgvector.

$results = DB::select('
    SELECT id, content, source_url,
           1 - (embedding <=> ?::vector) AS similarity
    FROM document_chunks
    WHERE tenant_id = ?
      AND metadata->>\'visibility\' = \'public\'
    ORDER BY embedding <=> ?::vector
    LIMIT 8
', [$queryVector, $tenantId, $queryVector]);

Validate JSON payloads with a JSON formatter during development. Bad metadata shapes silently break filters at runtime. For production chat UIs, see building a RAG chatbot for product documentation and AI-powered search for Laravel products.

PHP developers without Laravel can follow the same SQL patterns from our vector databases for PHP developers guide. Symfony 8.1 apps on PHP 8.4.1 use identical PostgreSQL integration through Doctrine DBAL raw queries or native PDO.

How do you evaluate, secure, and operate vector databases for RAG?

Vector search fails quietly. The API returns 200 OK. The answer just sounds confident and wrong. You need evaluation metrics and observability from week one, not after a client complaint.

• Recall@K — Did the correct source appear in the top K results for a labelled question set?
• Answer faithfulness — Does the LLM response stick to retrieved text without inventing facts?
• Latency p95 — Embedding + search + LLM must fit your UX budget, often under 3–5 seconds total.
• Index freshness — Track last indexed timestamp per document; stale chunks cause outdated answers.
• Cost per query — Embedding and token usage add up; see AI rate limits and cost optimization.

Security and compliance basics

Vectors are not encryption. Anyone with database access can read stored chunk text. Treat the vector table like any sensitive content table. Use row-level security in PostgreSQL for tenant isolation. Redact PAN numbers, passport details, or client identifiers before indexing on legal portals. If you host on a VPS you manage, fold vector backups into the same cron dumps described in our Linux system administration workflows.

For regulated content, log which chunk IDs were retrieved for each answer. That audit trail matters when a user challenges automated guidance on a notary or immigration workflow. A portal like Translation Nepal or Mijar Law Associates benefits from cite-back links in the UI, not just a chat bubble with plain text.

Operational tasks your cron should handle

Schedule these jobs explicitly:

1. Nightly full or incremental re-index of changed CMS pages.
2. Weekly vacuum and index health check on pgvector HNSW indexes.
3. Embedding model version audit — never silently upgrade the model without re-embedding all chunks.
4. Dead chunk cleanup when source documents are deleted or unpublished.

Managed vector services shift index maintenance to the vendor. You still own ingestion logic, chunk quality, and prompt design. Either way, RAG is an application feature, not a one-time database install.

Key Takeaways

• Vector databases store embeddings and run similarity search so RAG retrieves relevant chunks before the LLM answers.
• Chunking quality and metadata filters usually matter more than picking the fanciest ANN algorithm on day one.
• pgvector on PostgreSQL 18 fits most Laravel and PHP projects until you prove you need a dedicated managed vector service.
• Always scope queries by tenant and visibility; vector search without filters is a common multi-tenant data leak.
• Measure recall@K and index freshness — bad retrieval returns fluent but wrong answers.
• Pin embedding model versions and re-index when you change models; never mix vectors from different embedders.

People Also Ask

Is a vector database the same as a regular database?

No. A regular relational database stores rows and columns with exact-match indexes. A vector database—or a vector extension like pgvector—adds similarity search over float arrays. Many teams use both in one PostgreSQL instance: orders in normal tables, document embeddings in a vector column with an HNSW index.

Do you need a vector database if you use RAG?

You need vector search capability, not necessarily a separate product. RAG requires storing embeddings and finding nearest neighbours at query time. That can live in pgvector, Redis vector sets, OpenSearch, or a managed service. The architecture demands similarity retrieval; the vendor is flexible.

How many documents can pgvector handle for RAG?

pgvector handles hundreds of thousands to a few million vectors comfortably on a well-sized server with HNSW indexes. Beyond that, query latency, index build time, and RAM pressure may push you toward a dedicated ANN service or sharding strategy. Start with Postgres unless benchmarks say otherwise.

What embedding model should I use for RAG in 2026?

Use one production-grade embedding model and keep it consistent. OpenAI text-embedding-3-small is a common default for English and multilingual business content. Open-source models work when data cannot leave your server. Match the model across indexing and query paths, and document the version in your deployment config.

Ship RAG with the right vector layer

Vector databases explained for RAG boil down to a simple division of labour: embeddings and similarity search find the facts; the LLM formats the answer. Pick pgvector when you already run PostgreSQL. Reach for managed ANN when scale and latency demand it. Invest in chunking, filters, and evaluation before you invest in exotic infrastructure.

If you want RAG on a Laravel portal, legal knowledge base, or internal documentation site, I help teams design the ingestion pipeline, database schema, and AI integration without over-engineering the stack. See related work in our portfolio or start a scoped discussion via contact us. For API design around your retrieval service, our API development and custom software development pages outline how we typically structure those projects.