Run llama.cpp for Local Inference

By Kokil Thapa | Last reviewed: September 2026

You want to run llama.cpp for local inference when cloud APIs are too slow, too expensive, or off-limits for client data. llama.cpp is a plain C/C++ stack that loads GGUF weights and serves text on CPU or GPU with no Python runtime in the hot path. This guide walks through build, model choice, CLI prompts, and an OpenAI-compatible server you can wire into a Laravel or API workflow.

Why should you run llama.cpp for local inference instead of a cloud API?

Cloud LLM APIs are easy to start. They get expensive at volume. They also send prompts outside your network.

Local inference keeps documents, code, and customer messages on hardware you control. That matters for legal-tech portals, internal admin tools, and any workflow where data residency is non-negotiable.

llama.cpp sits at the low-level end of the local LLM stack. Ollama wraps similar engines for convenience. vLLM targets GPU clusters. llama.cpp gives you direct control: pick quantizations, tune context length, and ship a single binary.

I integrate LLM APIs into production apps regularly. For privacy-sensitive features, a local llama-server behind your app is often the simplest compliant path. See our guide on local LLMs for privacy-sensitive apps for the broader pattern.

Typical use cases include draft generation for CMS content, log summarisation on a dev machine, and offline coding assistants on flights or unreliable power.

• Privacy: prompts never leave the server.
• Cost: no per-token bill after hardware is paid for.
• Latency: no round trip to a remote region.
• Control: you choose model size, quant level, and context window.

For Nepal-based teams on budget hardware, a 7B–8B model at Q4 quant often runs well on 16 GB RAM. GPU builds are faster but not mandatory.

How do you install and build llama.cpp on Linux?

Official source lives on GitHub at ggml-org/llama.cpp. Clone it on Ubuntu 22/24—the same servers I use for Linux production hosting.

Install build dependencies

sudo apt update
sudo apt install -y git build-essential cmake curl

# Optional: NVIDIA CUDA toolkit for GPU builds
# sudo apt install -y nvidia-cuda-toolkit

Clone and compile

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j $(nproc)

Binaries land in build/bin/. The names you use most are llama-cli and llama-server. Older docs mention main and server; upstream renamed them in recent releases.

GPU-enabled builds

For NVIDIA GPUs, pass CUDA flags at configure time:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)

Apple Silicon uses Metal automatically on macOS. Vulkan and SYCL backends exist for other hardware. Check the project README for your target.

A common mistake is mixing an old binary with a new GGUF file. Rebuild after pulling upstream when models fail to load with schema errors.

How do you choose and download GGUF models for llama.cpp?

llama.cpp reads GGUF files—not original PyTorch or Safetensors weights. Hugging Face hosts thousands of converted models. Filter by tags like gguf and pick a quant from a trusted publisher.

Quantization trades quality for RAM and speed. Our article on model quantization explains the theory. For daily work, these levels are a solid starting point:

Download with huggingface-cli

pip install -U huggingface_hub

huggingface-cli download \
  bartowski/Llama-3.2-3B-Instruct-GGUF \
  Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --local-dir ./models

Or fetch directly with curl -L -O from the file URL on Hugging Face. Store models outside your git repo. A 4 GB file does not belong in version control.

Match model family to task. Instruct-tuned weights behave better for chat and summarisation. Base models suit completion-style prompts.

How do you run llama.cpp for local inference from the command line?

Start with llama-cli before you expose a network port. It confirms the model loads and gives a feel for tokens per second.

Basic interactive chat

./build/bin/llama-cli \
  -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -cnv \
  -p "You are a helpful assistant." \
  -n 512 \
  --temp 0.7

Flag cheat sheet:

• -m — path to GGUF model file.
• -cnv — conversational mode with chat template.
• -n — max tokens to generate.
• -c — context size (default varies; raise for long documents).
• -ngl — GPU layers to offload (CUDA/Metal); 99 often means all.
• --temp — sampling temperature; lower for factual tasks.

Single-shot prompt (no chat loop)

./build/bin/llama-cli \
  -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  -p "Summarise nginx reverse proxy in three bullets." \
  -n 256 \
  --no-display-prompt

Paste JSON through our JSON formatter when you pipe structured output into scripts. Broken JSON from sloppy prompts is a frequent integration bug.

Run llama-server for HTTP API access

Production integrations should call llama-server. It speaks an OpenAI-compatible REST API documented in the upstream repo.

./build/bin/llama-server \
  -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -c 8192 \
  -ngl 99

Test with curl:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Explain GGUF in one paragraph."}],
    "max_tokens": 200
  }'

Bind to 127.0.0.1 unless you put a reverse proxy and auth in front. An open LLM port on a public VPS is an abuse magnet within hours.

On a production Laravel application, point an HTTP client at that local endpoint the same way you would OpenAI. Cache responses in Redis 8.10 when inputs repeat. Queue long jobs through Laravel's worker stack.

How does llama.cpp compare to Ollama, LM Studio, and vLLM?

All four run models locally. They optimise for different operators. Pick based on who maintains the stack and how much control you need.

Read Ollama vs LM Studio for GUI-focused workflows. Read Ollama and vLLM when throughput matters more than binary size.

My default recommendation: start with Ollama to validate use cases. Move to llama.cpp when you need a pinned build, custom compile flags, or direct embedding in a C++ service. Keep vLLM for multi-GPU batch serving on Kubernetes—see AI/ML workloads on Kubernetes with GPUs.

Ollama actually bundles llama.cpp under the hood for many models. Running llama.cpp directly removes the middle layer and one moving part.

How do you harden llama.cpp for dev and staging workflows?

Local inference is not production-ready by default. Treat it like any other internal service.

1. Process supervision: run llama-server under systemd or supervisord with restart on failure.
2. Resource limits: cap CPU and memory so a large context request cannot freeze the host.
3. Network isolation: listen on localhost; expose only through nginx with TLS and API keys.
4. Model versioning: symlink current.gguf to the active file; swap atomically on upgrade.
5. Observability: log prompt length, latency, and tokens/sec; alert on error rates.
6. Fallback: queue requests when the server is down; never block checkout or form submission on LLM uptime.

For CI pipelines, a small model can power AI-assisted code review without sending proprietary diffs to the cloud. Pair that pattern with local LLMs in DevOps workflows on larger teams.

On legal-tech portals I have shipped, document summarisation runs on-premises. Client uploads never hit a third-party inference API. That architecture aligns with how those firms think about confidentiality.

Hardware budgeting for a small office in Kathmandu: a used workstation with 32 GB RAM and a mid-range NVIDIA card (~Rs 180,000–250,000, ~USD 1,350–1,875) often beats a year of cloud tokens for a five-person dev team. CPU-only inference on a 16 GB laptop still works for 3B models at Q4.

Embedding in application code

PHP/Laravel apps should not load GGUF files natively. Call llama-server over HTTP from a service class. Keep prompts in config, not controllers. Validate and sanitise user input before it reaches the model.

For structured extraction, ask for JSON and parse defensively. Use the same idempotency patterns you would for any external API integration. Retry with backoff on 503 responses when the model is busy.

The GGUF specification is documented on Hugging Face at huggingface.co/docs/hub/gguf. Refer to it when a download looks corrupt or a quant label is unfamiliar.

If you already run Docker for local dev—see Docker Compose for Laravel—you can containerise llama.cpp too. Mount models as a volume. Pin the image digest. GPU passthrough needs the NVIDIA Container Toolkit on the host.

Key Takeaways

• Build llama.cpp with CMake; use llama-cli for tests and llama-server for OpenAI-compatible HTTP.
• Download GGUF quantizations from Hugging Face; Q4_K_M is the best default for 16 GB machines.
• Bind llama-server to localhost, add auth at the proxy, and never expose an unauthenticated LLM port.
• Choose llama.cpp over Ollama when you need pinned binaries, custom backends, or minimal dependencies.
• Integrate from Laravel or other backends via HTTP; cache, queue, and fallback like any external API.
• Match hardware to model size: 3B on CPU for dev, 7B–8B on GPU for office-grade throughput.

People Also Ask

Do you need a GPU to run llama.cpp?

No. llama.cpp runs on CPU with AVX2/AVX512 acceleration. A GPU speeds up larger models dramatically. For 3B–7B weights, many developers start on CPU and add -ngl GPU offload when a CUDA or Metal device is available.

What is the difference between GGUF and GGML?

GGML was the older binary format. GGUF replaced it as the standard container for llama.cpp models. Always download GGUF files. Legacy GGML weights require conversion or re-download from a current repository.

Can llama.cpp replace OpenAI API calls in production?

For many internal tools, yes—with caveats. Smaller local models hallucinate more on edge cases. Keep cloud APIs as fallback for critical paths. Run evaluation sets before you cut over customer-facing features.

How much RAM do you need for a 7B model?

A 7B model at Q4_K_M needs roughly 5 GB for weights plus overhead for context and KV cache. Plan 8 GB minimum system RAM. A 16 GB machine leaves headroom for the OS and your IDE.

Ship local inference without guessing

You now have a full path to run llama.cpp for local inference: build, pick a GGUF quant, test with llama-cli, and serve through llama-server. The stack is boring, fast, and keeps sensitive data off third-party APIs.

If you want help wiring local LLMs into a Laravel portal, eCommerce flow, or internal tool, review our client portal work and custom software development services. For hands-on integration planning, contact us with your hardware specs and use case.