Run LLMs Locally with Ollama and vLLM

By Kokil Thapa | Last reviewed: September 2026

You do not need a cloud API key to ship useful AI features. When you run LLMs locally with Ollama and vLLM, prompts, documents, and user data stay on hardware you control. That matters for legal portals, internal tools, and any workflow where PII must not leave your network. Ollama gets a model running in minutes on a laptop or small server. vLLM targets production throughput on a GPU box. This guide covers install commands, API wiring, hardware sizing, and a clear decision path between the two.

Why should you run LLMs locally with Ollama and vLLM?

Cloud LLM APIs are convenient. They also send every prompt to a third party. For a law-firm portal or document workflow, that is often unacceptable. Local inference keeps client names, case notes, and uploaded PDFs inside your VPC or office LAN.

Cost is the second driver. A busy support bot can burn Rs 50,000–150,000 per month (~USD 375–1,125) on hosted tokens. A one-time GPU server at Rs 400,000–800,000 (~USD 3,000–6,000) pays back quickly at moderate volume. You trade CapEx for predictable OpEx.

Latency and offline use round out the case. A Kathmandu office on unreliable power still gets answers from a local box on UPS. No round trip to US-East means sub-second first tokens on a 7B model with a decent GPU.

I integrate hosted LLM APIs on production Laravel apps regularly. I do not train models. For privacy-sensitive work, local inference is the pattern I reach for first. See our AI integration and automation service for how that fits client projects.

Common local use cases I see on client work:

• Internal document Q&A over contracts and SOPs without uploading to OpenAI.
• Dev-time code review and log summarisation in CI, covered in our Ollama DevOps workflows guide.
• Structured JSON extraction from forms, paired with a JSON formatter for debugging output.
• Chat widgets on intranets where outbound HTTPS to AI vendors is blocked.

How do you install and run Ollama for local LLM inference?

Ollama is the fastest path from zero to a working local model. One binary, a model registry, and an OpenAI-compatible API on port 11434. It runs on macOS, Linux, and Windows with optional NVIDIA or AMD GPU acceleration.

Install Ollama on Linux

On Ubuntu 22.04 or 24.04 — the stack I use on most servers — run the official installer:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version
sudo systemctl enable ollama
sudo systemctl start ollama

Pull a model and start chatting from the shell:

ollama pull llama3.2:3b
ollama run llama3.2:3b

The 3B variant fits 8 GB RAM machines. It is fine for classification, short summaries, and routing prompts. For better reasoning, pull llama3.1:8b or mistral:7b. Quantised GGUF weights keep VRAM demand manageable.

Call Ollama from HTTP

Ollama exposes a REST API compatible with common client libraries. Test with curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Summarise GDPR in three bullet points.",
  "stream": false
}'

For OpenAI SDK compatibility, use the /v1/chat/completions endpoint:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Point any OpenAI client at http://localhost:11434/v1 with a dummy API key. That single change often unlocks existing app code. Read our guide on streaming LLM responses with SSE when you need token-by-token UI updates.

Ollama production tips

Ollama is not a load balancer. Bind it to localhost unless you trust every device on the network. Put Nginx or Caddy in front with TLS and basic auth for remote access. Set OLLAMA_HOST=0.0.0.0:11434 only on isolated VLANs.

Keep models on fast disk. A 7B Q4 model is roughly 4–5 GB. NVMe beats spinning rust for cold starts. For Open WebUI frontends, see our Ollama plus Open WebUI setup.

When should you choose vLLM instead of Ollama?

vLLM is a high-throughput inference engine built for production. It uses PagedAttention to batch requests efficiently on NVIDIA GPUs. When ten users hit your app at once, vLLM keeps latency stable. Ollama queues serially and stalls.

Pick vLLM when all of these apply:

1. You have an NVIDIA GPU with 16 GB VRAM or more.
2. Concurrent users or API clients exceed a handful.
3. You need continuous batching, tensor parallelism, or LoRA hot-swapping.
4. You want an OpenAI-compatible server that scales like a microservice.

Stay on Ollama when you are prototyping, running on CPU-only hardware, or serving one developer workstation. Our Ollama vs LM Studio comparison covers desktop GUI alternatives. vLLM has no friendly desktop app — it is server software.

Install vLLM with pip

vLLM requires Python 3.10+ and CUDA. On a clean Ubuntu GPU box:

python3 -m venv /opt/vllm-venv
source /opt/vllm-venv/bin/activate
pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype auto \
  --max-model-len 8192

Test the OpenAI-compatible endpoint:

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain vLLM in one sentence."}]
  }'

Run vLLM behind systemd or Docker on a dedicated inference node. Do not colocate it with MySQL on a 16 GB RAM VPS — you will OOM both services. For GPU Kubernetes patterns, see running AI workloads on Kubernetes with GPUs.

Verdict: start every project on Ollama. Promote the same prompt templates to vLLM once load testing proves you need it. That two-stage path avoids over-engineering day one.

How do you connect a Laravel app to local Ollama or vLLM APIs?

Most PHP teams already use Guzzle or Laravel's HTTP client. Point it at your local OpenAI-compatible base URL. No vendor SDK required.

Laravel service class example

<?php

namespace App\Services;

use Illuminate\Support\Facades\Http;

class LocalLlmClient
{
    public function __construct(
        private string $baseUrl = 'http://127.0.0.1:11434/v1',
        private string $model = 'llama3.2:3b',
    ) {}

    public function chat(string $userMessage): string
    {
        $response = Http::timeout(120)
            ->post("{$this->baseUrl}/chat/completions", [
                'model' => $this->model,
                'messages' => [
                    ['role' => 'system', 'content' => 'Reply concisely.'],
                    ['role' => 'user', 'content' => $userMessage],
                ],
                'temperature' => 0.2,
            ])
            ->throw()
            ->json();

        return $response['choices'][0]['message']['content'] ?? '';
    }
}

Store the base URL in .env so staging hits Ollama and production hits vLLM:

LOCAL_LLM_BASE_URL=http://inference.internal:8000/v1
LOCAL_LLM_MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct

Queue long calls. A 70B model on weak hardware can exceed PHP-FPM timeouts. Dispatch a job, stream progress over websockets or polling, and return the result async. That mirrors how I handle document summarisation on legal-tech portals like Mijar Law Associates.

For JSON-only responses, add a response-format hint and validate server-side. Never trust raw model output for database writes. Our structured outputs guide covers schema enforcement. Test payloads with the regex tester when cleaning messy text before parsing.

Wire function calling carefully. Local 7B models hallucinate tool names more often than GPT-4 class models. Validate every tool invocation in PHP before execution. See function calling with LLMs for guardrail patterns.

What hardware do you need to run LLMs locally in 2026?

VRAM is the bottleneck for GPU inference. RAM matters for CPU-only Ollama runs. Disk speed affects model load time, not steady-state tokens.

Practical tiers for Nepal and global small-team budgets:

• Laptop / NUC (8–16 GB RAM, no GPU): Ollama with 3B Q4 models. Expect 5–15 tokens/sec. Fine for demos and personal scripts.
• Workstation (RTX 4060 Ti 16 GB): 7B–8B models at 30–60 tokens/sec. Sweet spot for a five-person dev team.
• Server (RTX 4090 / L40S 24–48 GB): 8B fast, 70B quantised, or vLLM with batching for production APIs.
• Multi-GPU node: 405B-class models or high-QPS vLLM. Overkill until metrics prove you need it.

Budget roughly Rs 350,000–600,000 (~USD 2,600–4,500) for a sensible 4090 build in Kathmandu import scenarios. Cloud GPU rentals from AWS or RunPod can beat ownership below ~500 GPU-hours per month. Our self-hosting cost breakdown walks through the math.

CPU inference on a 16-core Xeon without GPU works for batch overnight jobs. It is a poor fit for interactive chat. If you must go CPU-only, stick to 3B models and aggressive quantisation.

Linux server admin skills matter as much as hardware. GPU drivers, CUDA versions, and systemd units break silently after kernel upgrades. If your team lacks that capacity, factor in Linux system administration support or managed hosting abroad with GPU instances.

How do you monitor and secure local LLM deployments?

Local does not mean safe by default. An exposed Ollama port on a public IP becomes a free GPU mine within hours. Treat inference servers like databases.

Security checklist

1. Bind to private IPs. Use VPN or Tailscale for remote dev access.
2. Terminate TLS at a reverse proxy. Issue Let's Encrypt certs the same way you would for Laravel.
3. Rate-limit at Nginx. Local models can still be DoS'd into uselessness.
4. Log prompts without storing raw PII longer than needed. Redact before log shipping.
5. Keep models updated. Pull new Ollama tags or pin Hugging Face revisions in vLLM.

Monitoring belongs in your existing stack. Export request counts, latency p95, and tokens per minute. Alert when queue depth spikes — that usually means you have outgrown Ollama. Read LLMOps monitoring and guardrails for production patterns.

Run evals before swapping models. A newer 8B model can regress on Nepali-language prompts or your JSON schema. Our LLM evaluation guide shows a lightweight golden-set workflow. Pair evals with hallucination reduction techniques when answers feed user-facing legal content.

For privacy-first architecture rationale, see local LLMs for privacy-sensitive apps. Operational maturity maps to LLMOps: ship and operate LLM apps.

Official references worth bookmarking: the Ollama API documentation, the vLLM documentation, and OpenAI API reference for endpoint parity when porting clients.

Key Takeaways

• Install Ollama first for fast local prototyping; both tools expose OpenAI-compatible HTTP APIs your existing SDKs can reuse.
• Move to vLLM on an NVIDIA GPU when concurrent users, batching, or sustained API load exceed Ollama's serial queue.
• Queue LLM calls in Laravel, validate every structured response server-side, and never execute model-suggested tools without PHP checks.
• Budget hardware by VRAM: 8 GB for 3B models, 16 GB for 7B–8B, 24 GB+ for vLLM production or larger weights.
• Lock inference servers behind private networking, TLS, and rate limits — an open port 11434 is an open invitation.
• Run evals and monitor latency before promoting a new local model to production user flows.

People Also Ask

Can Ollama and vLLM use the same models?

Not directly. Ollama consumes GGUF bundles via its registry and Modelfile format. vLLM loads Hugging Face safetensors checkpoints. You can run the same model family — Llama 3.1 8B, for example — but you download separate artefacts for each runtime.

Does Ollama work without a GPU?

Yes. Ollama falls back to CPU inference with AVX-capable processors. Speed drops sharply on large models. Stick to 3B quantised weights on CPU-only machines and expect interactive latency, not chat-app snappiness.

Is vLLM free for commercial use?

vLLM is open source under the Apache 2.0 licence. You pay for hardware, electricity, and ops time — not per-token fees. Model licences are separate; Meta Llama weights require accepting their community licence before download.

Which is easier for a Laravel developer?

Ollama wins on simplicity. One install script, one port, immediate curl tests. vLLM needs Python virtualenvs, CUDA alignment, and VRAM tuning. Laravel integration is identical once both expose /v1/chat/completions — the difference is ops overhead, not PHP code.

Ship local AI without sending data to the cloud

You now have a concrete path to run LLMs locally with Ollama and vLLM: prototype on Ollama today, load-test, then promote to vLLM when concurrency demands it. Keep prompts on your network, wire OpenAI-compatible clients in Laravel, and treat inference nodes with the same security discipline as your database tier.

If you want help designing a private AI layer for a portal, booking system, or internal tool, review our API development services and custom software development offerings. For a scoped architecture review, contact us with your model size, user count, and privacy constraints — we will recommend the smallest stack that actually fits.