Build an Embeddings Pipeline

By Kokil Thapa | Last reviewed: August 2026

If you need to add semantic search or retrieval-augmented generation (RAG) to a PHP application, you must build an embeddings pipeline that reliably converts text into vector representations and stores them for fast similarity retrieval. Many developers attempt this by calling external APIs synchronously during user requests, which creates latency bottlenecks and unpredictable costs. A production-grade approach decouples embedding generation from ingestion using background queues and local or dedicated inference services.

This approach mirrors patterns I use when building complex data systems, such as those described in my guide on database-driven website development in Nepal, where reliability and cost-efficiency are paramount. For legal-tech portals handling sensitive case files, keeping embedding generation on-premise or within a private VPC is often a strict compliance requirement rather than just a performance optimization.

How do you architect a scalable embeddings pipeline?

A robust embeddings pipeline separates three distinct concerns: ingestion, inference, and storage. Treating these as a single synchronous operation is the most common failure mode for PHP applications attempting vector search. In practice, your web server should never wait for a transformer model to process a document; it should only dispatch a job.

The diagram above illustrates the recommended async topology. Your Laravel application accepts raw content and immediately dispatches a GenerateEmbeddingJob. A dedicated queue worker picks up this job and calls an inference service. This service can be a separate Python container running Sentence Transformers or a managed endpoint. Crucially, the heavy lifting happens outside the HTTP request cycle.

For projects where I have implemented Laravel API best practices, this pattern also simplifies error handling. If the inference service is temporarily unavailable, the job fails and retries automatically via Laravel's queue backoff strategy, preventing data loss. The database write only occurs once the vector is successfully generated, ensuring your index never contains partial or null embeddings.

Which open-source embedding models work best for production?

Choosing the right model determines your search quality, storage costs, and inference speed. While OpenAI’s text-embedding-3-small remains popular, open-weight models now match or exceed its performance on multilingual benchmarks while eliminating per-token fees. For Nepal-based projects dealing with mixed English and Nepali text, multilingual support is non-negotiable.

I currently recommend BGE-M3 for most production embeddings pipelines. It supports dense, sparse, and multi-vector representations, making it exceptionally versatile for RAG systems that need both keyword and semantic matching. Its 1024 dimensions offer a sweet spot between retrieval accuracy and pgvector storage overhead. For strictly English applications with tight memory constraints, nomic-embed-text-v1.5 at 768 dimensions reduces index size by roughly 40% compared to 1024-dim models with minimal quality loss.

When deploying these models, avoid loading them directly in PHP. Instead, serve them via a lightweight Python API using FastAPI or TEI (Text Embeddings Inference). This keeps your PHP workers stateless and allows you to scale inference independently. On a typical Ubuntu 24.04 server with an NVIDIA T4 GPU, BGE-M3 processes approximately 300 chunks per second—more than sufficient for batch-indexing large document repositories overnight.

How do you configure PostgreSQL pgvector for efficient storage?

PostgreSQL with the pgvector extension is the pragmatic choice for Laravel applications already using relational databases. It eliminates operational complexity of maintaining a separate vector database like Pinecone or Weaviate for small-to-medium datasets. As of 2026, pgvector 0.8+ includes significant performance improvements for HNSW indexing that make it viable for millions of vectors.

Installing and configuring pgvector

On Ubuntu 24.04 with PostgreSQL 16 or 17, install the extension via the official PGDG repository:

sudo apt install postgresql-17-pgvector
# Then in psql:
CREATE EXTENSION IF NOT EXISTS vector;

Create your embeddings table with appropriate constraints. Always store the original chunk text alongside the vector for debugging and re-ranking:

CREATE TABLE document_embeddings (
    id BIGSERIAL PRIMARY KEY,
    document_id UUID NOT NULL REFERENCES documents(id),
    chunk_text TEXT NOT NULL,
    embedding VECTOR(1024) NOT NULL, -- Match your model dimensions
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Create HNSW index for cosine similarity search
CREATE INDEX ON document_embeddings 
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);

The m and ef_construction parameters control the trade-off between index build time, memory usage, and recall. For most embeddings pipeline workloads, m=16 and ef_construction=200 provide excellent recall (>95%) without excessive RAM consumption. Increase ef_construction to 400+ only if benchmarking shows insufficient recall for your specific dataset.

Always prefer HNSW over IVFFlat for new projects unless you have tens of millions of vectors and severe RAM constraints. HNSW provides consistent sub-10ms query times for datasets under 5 million vectors on modest hardware, while IVFFlat requires careful tuning of lists and probes parameters to avoid catastrophic recall degradation.

How do you implement chunking and embedding generation in Laravel?

Effective chunking is more important than model selection for RAG quality. Fixed-size character splitting destroys semantic boundaries. Use recursive text splitters that respect paragraph and sentence structure, and always preserve metadata like section headers or document titles in each chunk.

Chunking strategy implementation

Install a PHP text splitter package or implement a simple recursive splitter. For legal documents on projects like Court Marriage In Nepal, I use section-aware chunking that keeps entire clauses together:

<?php

namespace App\Services;

class DocumentChunker
{
    public function __construct(
        private int $maxTokens = 512,
        private int $overlap = 64
    ) {}

    public function chunk(string $text, array $metadata = []): array
    {
        $paragraphs = preg_split('/\n\s*\n/', $text);
        $chunks = [];
        $currentChunk = '';
        
        foreach ($paragraphs as $paragraph) {
            $candidate = trim($currentChunk . "\n\n" . $paragraph);
            
            if ($this->estimateTokens($candidate) > $this->maxTokens) {
                if (!empty($currentChunk)) {
                    $chunks[] = [
                        'text' => trim($currentChunk),
                        'metadata' => $metadata,
                    ];
                }
                // Handle overlap for context continuity
                $words = explode(' ', $currentChunk);
                $overlapText = implode(' ', array_slice($words, -$this->overlap));
                $currentChunk = $overlapText . "\n\n" . $paragraph;
            } else {
                $currentChunk = $candidate;
            }
        }
        
        if (!empty(trim($currentChunk))) {
            $chunks[] = ['text' => trim($currentChunk), 'metadata' => $metadata];
        }
        
        return $chunks;
    }
    
    private function estimateTokens(string $text): int
    {
        // Rough approximation: 1 token ≈ 4 chars for English
        // Adjust for Nepali/multilingual content
        return (int) ceil(mb_strlen($text) / 3.5);
    }
}

Queued embedding generation job

Dispatch embedding generation asynchronously. This job handles batching, API calls, and database writes:

<?php

namespace App\Jobs;

use App\Models\DocumentEmbedding;
use App\Services\EmbeddingService;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class GenerateDocumentEmbeddings implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;
    public int $backoff = 60;

    public function __construct(
        private string $documentId,
        private array $chunks
    ) {}

    public function handle(EmbeddingService $service): void
    {
        // Batch embed for efficiency (most APIs support batching)
        $texts = array_column($this->chunks, 'text');
        $vectors = $service->embedBatch($texts);

        $records = [];
        foreach ($this->chunks as $i => $chunk) {
            $records[] = [
                'document_id' => $this->documentId,
                'chunk_text' => $chunk['text'],
                'embedding' => '[' . implode(',', $vectors[$i]) . ']',
                'metadata' => json_encode($chunk['metadata']),
                'created_at' => now(),
            ];
        }

        // Bulk insert for performance
        DocumentEmbedding::insert($records);
    }
}

For the EmbeddingService, wrap your HTTP client with retry logic and circuit breaking. When calling a local TEI or FastAPI endpoint, set reasonable timeouts (30s for batch requests) and log failures for monitoring. If you are integrating this into a larger system, consider reading about mastering Laravel queues to handle backpressure during bulk indexing operations.

How do you perform semantic search queries with pgvector?

Once your embeddings pipeline has populated the database, querying uses standard SQL with the cosine distance operator. Always combine vector similarity with traditional filters to narrow the search space before computing distances.

-- Find top 5 similar chunks for a query embedding
SELECT 
    chunk_text,
    metadata,
    1 - (embedding <=> $1::vector) AS similarity
FROM document_embeddings
WHERE document_id IN (
    SELECT id FROM documents WHERE category = 'family-law'
)
ORDER BY embedding <=> $1::vector
LIMIT 5;

In Laravel, use raw expressions for the vector operator since Eloquent doesn’t natively support pgvector syntax:

$results = DocumentEmbedding::query()
    ->selectRaw("chunk_text, metadata, 1 - (embedding <=> ?::vector) as similarity", [$queryVectorString])
    ->whereHas('document', fn($q) => $q->where('category', 'family-law'))
    ->orderByRaw("embedding <=> ?::vector", [$queryVectorString])
    ->limit(5)
    ->get();

Set hnsw.ef_search at the session level before running queries to tune recall/speed trade-offs dynamically. For interactive search, SET hnsw.ef_search = 100; provides fast responses. For batch processing or high-accuracy requirements, increase to 200–400. Remember that higher values increase CPU usage proportionally.

Build an Embeddings Pipeline That Scales Reliably

When you build an embeddings pipeline for production, prioritize architectural simplicity over novelty. Use PostgreSQL with pgvector unless you have proven scale requirements exceeding 10 million vectors. Choose open-weight models like BGE-M3 for multilingual support and cost control. Decouple inference from your web tier using Laravel queues. These decisions compound into systems that are cheaper to operate, easier to debug, and simpler to hand off to future maintainers.

If you are planning a semantic search or RAG feature for your Laravel application and need guidance on architecture, model selection, or pgvector optimization, reach out to discuss your embeddings pipeline requirements. I regularly help teams in Nepal and globally ship production vector search systems that actually work under real-world load.