Self-Host LLMs with Ollama and Open WebUI

By Kokil Thapa | Last reviewed: August 2026

You want to run large language models locally but need a practical path that avoids cloud API costs and keeps proprietary data off third-party servers. Learning to self-host LLMs with Ollama and Open WebUI gives you a private, production-grade inference stack on your own hardware. This guide covers the exact installation, configuration, and operational patterns I use when deploying local AI infrastructure for development workflows and internal business tools.

Before diving into terminal commands, understand that this stack replaces two separate concerns: the inference engine (Ollama) and the user interface (Open WebUI). For teams evaluating whether to build custom AI tooling versus buying SaaS, understanding the technical trade-offs between custom and managed solutions helps frame the investment in local hardware correctly. Self-hosting makes sense when data privacy, latency, or recurring API costs outweigh the operational overhead of maintaining your own GPU server.

How do you install and configure Ollama for local model serving?

Ollama is the backend runtime that downloads, caches, and serves quantized GGUF models through a REST API. On Ubuntu 24.04 LTS, the official install script handles binary placement and systemd unit creation automatically. Run the installer with curl, then verify the service is active before pulling any models.

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama
systemctl status ollama

The default installation binds to localhost:11434. For multi-user environments or when Open WebUI runs in a separate Docker network, you must explicitly bind to all interfaces and configure environment variables for GPU selection and concurrent request handling. Edit the systemd override file rather than modifying the vendor unit directly, so package upgrades do not clobber your changes.

sudo systemctl edit ollama

# Add to the [Service] section:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="CUDA_VISIBLE_DEVICES=0"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

After saving the override, reload the daemon and restart the service. The OLLAMA_NUM_PARALLEL variable controls how many requests a single loaded model can handle simultaneously; setting this too high on limited VRAM causes silent failures where requests queue indefinitely. Start with 2–4 for consumer GPUs and test under realistic load. The OLLAMA_MAX_LOADED_MODELS parameter prevents memory exhaustion when users switch between models frequently; each loaded model reserves its full context window in VRAM regardless of active usage.

Pull a baseline model to validate the installation end-to-end. Llama 3.1 8B Instruct at Q4_K_M quantization offers the best balance of quality and VRAM footprint for most development tasks. The pull command streams progress to stdout; subsequent loads skip download if the digest matches.

ollama pull llama3.1:8b-instruct-q4_K_M
ollama list
curl http://localhost:11434/api/tags

What is the correct Docker Compose configuration for Open WebUI?

Open WebUI provides the chat interface, RAG pipeline, user management, and model administration layer that sits in front of Ollama. Running it in Docker isolates Node.js and Python dependencies from your host system and simplifies upgrades. The critical configuration details are the volume mount for persistent data, the correct Ollama base URL (which differs inside Docker networking), and GPU device reservation if you plan to use Open WebUI's built-in embedding models.

version: '3.8'
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - open-webui-data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=false
      - DEFAULT_USER_ROLE=admin
    extra_hosts:
      - "host.docker.internal:host-gateway"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  open-webui-data:

The host.docker.internal hostname resolves to the Docker host from inside the container, but only when you include the extra_hosts directive. Without it, Open WebUI cannot reach Ollama bound to localhost. On Linux systems where this alias fails, replace it with the host's LAN IP or use a custom Docker network with explicit service linking. The named volume open-webui-data stores SQLite databases, uploaded files, RAG vector indices, and user settings; losing this volume means losing all chat history and configured knowledge bases.

Set WEBUI_AUTH=true and ENABLE_SIGNUP=false for any deployment beyond personal use. The first account created becomes the administrator; subsequent users require admin invitation. Disabling open signup prevents unauthorized access on shared networks. For teams managing multiple client projects, understanding current security practices for web applications ensures your local AI instance does not become an unmonitored attack surface.

How does GPU acceleration affect inference performance and model selection?

Running LLMs on CPU alone produces 1–3 tokens per second for 8B parameter models, which is unusable for interactive chat. NVIDIA GPU acceleration via CUDA is effectively mandatory for any real workload. The relationship between VRAM capacity, model quantization, and context length determines what you can actually run without swapping to system RAM, which collapses performance back to CPU speeds.

These numbers assume the entire model fits in VRAM with headroom for KV cache. When VRAM fills, Ollama silently offloads layers to CPU, and throughput drops non-linearly. Monitor actual GPU memory usage during inference with nvidia-smi -l 1 rather than trusting theoretical calculations. Context length consumes VRAM quadratically for some attention implementations; doubling context can consume far more than double the memory.

For teams without dedicated GPU hardware, consider renting cloud GPU instances temporarily for evaluation before purchasing. A single RTX 4090 workstation in Nepal costs roughly NPR 350,000–450,000 (~USD 2,600–3,300) including PSU and cooling upgrades, while equivalent cloud rental runs USD 0.40–0.80/hour. Break-even for continuous use hits around 4–6 months. If your usage is sporadic or experimental, cloud rental remains cheaper. For businesses evaluating infrastructure investments, comparing local hosting providers and pricing against owned hardware clarifies the true total cost.

How do you implement RAG and document ingestion securely?

Open WebUI includes a built-in RAG pipeline using sentence-transformers embeddings stored in ChromaDB by default. Upload PDFs, markdown files, or plain text through the workspace interface, and the system chunks, embeds, and indexes them automatically. However, the default embedding model (all-MiniLM-L6-v2) produces mediocre retrieval quality for domain-specific content. Replace it with bge-m3 or nomic-embed-text for significantly better recall on technical documentation and legal texts.

# Inside Open WebUI Admin Settings → RAG
Embedding Model: nomic-ai/nomic-embed-text-v1.5
Chunk Size: 1000
Chunk Overlap: 200
Top K: 5
Relevance Threshold: 0.3

Chunk size and overlap matter more than most operators realize. Legal documents and technical specifications contain cross-references that span paragraph boundaries; aggressive chunking severs these relationships. Test retrieval quality with known questions against your corpus before declaring the RAG pipeline production-ready. The relevance threshold filters out low-confidence matches; setting it too low injects noise, while setting it too high causes false negatives. Start at 0.3 and adjust based on observed retrieval precision.

Security for ingested documents follows the same principles as any application handling sensitive data. Open WebUI stores uploaded files and vector indices in the mounted volume with filesystem permissions inherited from the Docker daemon. Restrict volume ownership to the container user, disable public sharing links, and audit access logs regularly. For legal-tech deployments where attorney-client privilege applies, confirm that no telemetry or analytics endpoints transmit document content externally; Open WebUI is fully offline-capable when configured correctly, but verify network traffic with tcpdump during initial setup.

What operational monitoring and maintenance routines prevent silent failures?

Local LLM stacks fail silently more often than cloud APIs. Ollama may stop responding to requests while the systemd service reports active; GPU memory may fragment causing intermittent slowdowns; Docker containers may restart without surfacing errors to users. Establish monitoring before you trust the system for real work.

• Create a health-check cron job that queries /api/tags every five minutes and alerts on failure or response time exceeding 10 seconds.
• Log Ollama inference metrics (tokens/sec, queue depth, model load times) to Prometheus or a simple rotating log file for trend analysis.
• Schedule weekly ollama rm cleanup for unused model tags to reclaim disk space; quantized models accumulate quickly during experimentation.
• Monitor GPU temperature and fan curves; sustained inference loads push consumer GPUs to thermal limits that trigger throttling after 30–60 minutes.
• Back up the Open WebUI data volume nightly; SQLite corruption from unexpected power loss has caused complete data loss in production deployments I have reviewed.

Docker Compose deployments should include a watchdog sidecar or use Docker's built-in healthcheck directive with restart policies. Relying solely on restart: unless-stopped masks underlying issues; the container restarts successfully but continues failing the same way. Pair restart policies with external validation that confirms the application actually serves requests after restart.

When Should You Self-Host LLMs with Ollama and Open WebUI Instead of Using Cloud APIs?

The decision to self-host LLMs with Ollama and Open WebUI hinges on three factors: data sensitivity, usage volume, and tolerance for operational responsibility. If your workload involves client-confidential legal documents, internal business strategy, or regulated healthcare data, local inference eliminates third-party data exposure entirely. For high-volume use exceeding 2 million tokens per month, owned hardware typically breaks even within 6–12 months compared to OpenAI or Anthropic API pricing at 2026 rates.

However, self-hosting demands ongoing maintenance time. Model updates require manual testing against your specific prompts; GPU drivers need periodic upgrades; security patches for Docker and the host OS cannot be deferred indefinitely. Teams without dedicated DevOps capacity should budget 4–8 hours monthly for maintenance or consider hybrid approaches where sensitive workloads run locally and general-purpose queries use cloud APIs. For organizations planning broader automation initiatives, reviewing available AI automation tools and integration strategies helps position local inference as one component of a larger workflow rather than an isolated experiment.

Start with a single GPU workstation running the configuration above. Validate that inference quality meets your requirements before scaling to multi-GPU or clustered deployments. Most teams discover that 8B–12B models handle 80% of internal use cases adequately; chasing 70B parameter models without measured quality gaps wastes capital and complicates operations unnecessarily. Build incrementally, measure continuously, and treat your local LLM stack as production infrastructure deserving the same rigor as any other business-critical system.

If you need help architecting a private AI inference stack, integrating RAG pipelines with existing Laravel or WordPress systems, or evaluating whether self-hosting makes financial sense for your specific workload, reach out to discuss your requirements.