Model Quantization: Run Bigger Models on Less VRAM

By Kokil Thapa | Last reviewed: September 2026

Model Quantization: Run Bigger Models on Less VRAM is the fastest way to fit a 70B-class LLM onto a single consumer GPU. Full FP16 weights for a 7B model alone need roughly 14 GB of video memory before you add KV cache and runtime buffers. Quantization stores those same weights in INT4, INT8, or FP8, which cuts footprint by 50–75% and often speeds up inference on modern GPUs. If you already run local models with Ollama and vLLM, you have seen quantization in action—even when the UI never labels it that way.

What is model quantization and why does it reduce VRAM usage?

Quantization maps floating-point weight values into a smaller set of discrete levels. A 16-bit float uses 65,536 distinct values per weight. INT4 uses sixteen. The model file shrinks because each parameter occupies fewer bits on disk and in GPU memory.

VRAM consumption has three major parts during inference. Weights dominate for large models. The KV cache grows with context length and batch size. Activations and temporary buffers add overhead on top. Quantization attacks the weight term first, which is why a 70B model that needs ~140 GB in FP16 can fit into ~40 GB at Q4.

Two quantization styles matter in practice. Post-training quantization (PTQ) converts a trained model without retraining. Quantization-aware training (QAT) simulates low precision during training for tighter accuracy. Most open-weight LLMs you download today use PTQ because it is cheap and fast.

Per-channel or per-group scaling preserves quality better than a single global scale factor. Tools like GPTQ and AWQ compute those scales from calibration data. That detail explains why a sloppy INT4 export can feel broken while a tuned Q4_K_M GGUF file feels nearly identical to FP16 on everyday prompts.

How bits map to bytes on real hardware

Use this rough formula: VRAM for weights ≈ (parameter count × bits per weight) / 8. A 7B model at FP16 needs about 14 GB. The same model at Q4 needs about 3.5 GB for weights alone. Always add 20–40% headroom for KV cache, CUDA context, and framework overhead.

Which quantization formats should you choose for local LLM inference?

Format choice depends on your runtime, GPU generation, and tolerance for quality loss. There is no universal winner. A format that excels on NVIDIA Tensor Cores may be useless in CPU-only llama.cpp builds on a budget VPS.

For local development on a laptop with 8–12 GB VRAM, start with Q4_K_M or Q5_K_M in GGUF. For production GPU serving, AWQ or GPTQ INT4 paired with KServe model serving on Kubernetes is a common stack. FP8 shines on H100-class hardware where native FP8 Tensor Cores are available.

I integrate LLM APIs into client apps rather than training models myself. On production Laravel portals, the inference layer is almost always a pre-quantized checkpoint behind an API gateway. Choosing the right format upstream saves hosting cost downstream. See our AI integration and automation services for that full pipeline.

How do you quantize an LLM with llama.cpp or Hugging Face?

Quantization is a build step, not a runtime toggle. You export or convert once, then serve the artifact repeatedly. Pick one toolchain and stick with it for a given deployment path.

llama.cpp and GGUF conversion

Download a base model in Hugging Face format, then convert and quantize with llama.cpp. This workflow is ideal for running local LLMs for DevOps workflows on modest hardware.

# Clone llama.cpp and build with CUDA if you have a GPU
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make LLAMA_CUDA=1

# Convert Hugging Face weights to FP16 GGUF
python convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf

# Quantize to Q4_K_M (strong quality/size balance)
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

# Smoke test
./llama-cli -m model-q4_k_m.gguf -p "Explain model quantization in one paragraph."

Ollama wraps a similar flow. Running ollama pull llama3:8b downloads a pre-quantized GGUF variant. For custom fine-tunes, import with a Modelfile pointing at your local GGUF path.

Hugging Face GPTQ and AWQ

For NVIDIA serving with vLLM, GPTQ and AWQ remain the workhorse formats. Both need calibration text—typically 128–512 samples from your target domain.

# Install tooling (Python 3.10+, CUDA recommended)
pip install autoawq transformers accelerate

# AWQ example (see AutoAWQ docs for model-specific flags)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3-8B-Instruct"
quant_path = "llama3-8b-awq"

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

quant_config = {"zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM"}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

Serve the AWQ checkpoint with vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model ./llama3-8b-awq \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Official references: Hugging Face quantization overview and llama.cpp quantize documentation.

Calibration data that actually matters

Calibration tells the quantizer which activation ranges matter. Generic Wikipedia snippets work for general chat. Legal, medical, or code-heavy apps need in-domain samples. I have seen Nepali-English mixed prompts change perplexity scores on bilingual fine-tunes because the calibrator never saw Devanagari Unicode during export.

1. Collect 128–512 representative prompts from production logs (redact PII first).
2. Match sequence length to your real max_model_len where possible.
3. Run the same eval set at FP16 and at target bit depth before you ship.
4. Keep the FP16 baseline artifact in your registry for rollback.

Store both versions using patterns from model versioning and registries. Quantized weights are not interchangeable even when the base model name matches.

What quality trade-offs happen when you run bigger models on less VRAM?

Lower bit depth means coarser weight resolution. Most general prompts survive Q4 well. Long-chain reasoning, rare-token languages, and precise JSON output break first. The failure mode is subtle: the model sounds confident but omits constraints or hallucinates field names.

Rule of thumb from community benchmarks and my own integration testing: Q5_K_M is often indistinguishable from FP16 on short tasks. Q4_K_M is the practical default. Q3_K_M and below show measurable perplexity jumps. Q2 is an emergency option when VRAM is the hard ceiling, not a quality target.

Evaluate with task-specific benchmarks, not vibes. For JSON APIs, validate schema conformance rate. For RAG assistants, measure answer faithfulness against retrieved chunks. Track drift after quantization the same way you would after a model swap. The guide on monitoring ML models in production for drift applies directly.

KV cache quantization is a separate lever. Some runtimes offer INT8 or FP8 KV caches to stretch context length on the same card. That helps long-document Q&A on legal-tech portals where 32K context is common. It does not replace weight quantization; it complements it.

How do you pick the right GPU and runtime for quantized inference?

VRAM math drives hardware choice. A 24 GB RTX 4090 handles a Q4 34B model with moderate context. A 12 GB card tops out around Q4 7B–8B unless you offload layers to system RAM—which kills tokens-per-second.

Cloud GPU pricing makes this tangible. An A100 80 GB instance in a US region might run USD 2–3 per hour on demand. Quantizing from FP16 to INT4 can drop you from two GPUs to one. That is roughly USD 1,400/month saved at 24/7 load—a figure that matters for Nepal startups billing in NPR (Rs 185,000+ at typical 2026 rates).

For Kubernetes deployments, pin GPU memory limits and use serving ML models with GPU on Kubernetes patterns. Mixed-precision nodes need driver and CUDA versions that match your container base image. Mismatch here produces silent CPU fallback and angry users.

How do you deploy quantized models in production safely?

Treat quantized artifacts as first-class release objects. Hash the file, record bit depth, calibrator version, and eval scores in your model registry. Wire them through the same CI/CD pipeline for machine learning models you use for FP16 builds.

A minimal production checklist:

• Export FP16 and quantized variants from the same base commit.
• Run automated eval gates before promotion (accuracy, latency, JSON validity).
• Expose model_version and quantization fields in your API response metadata.
• Load-test at expected concurrent requests—INT4 is faster until memory bandwidth saturates.
• Document rollback to FP16 if a regression appears in production.

When you wrap inference behind a Laravel API, validate responses server-side. Do not trust quantized output for payment amounts, legal dates, or statutory deadlines without schema checks. I follow the same pattern on document-heavy client portals like Mijar Law Associates where a wrong date breaks trust instantly.

For OpenAI-compatible endpoints consumed by agents, pair quantization with Model Context Protocol (MCP) tool boundaries. Smaller models plus strict tool schemas often beat a raw 70B free-form reply. Use the JSON formatter tool to inspect API payloads during integration testing.

NVIDIA documents FP8 and INT8 inference patterns in their Transformer Inference performance guide. Cross-check vendor claims against your own benchmarks on your hardware.

Key Takeaways

• Model quantization cuts weight memory by 50–75%, letting you run bigger models on less VRAM on the same GPU.
• Start with Q4_K_M (GGUF) locally and AWQ/GPTQ INT4 for vLLM GPU serving unless eval data says otherwise.
• Always calibrate with in-domain prompts and compare against an FP16 baseline before you ship.
• Account for KV cache and activations—weight math alone underestimates total VRAM need.
• Version quantized artifacts separately and monitor quality drift after deployment.
• Pair smaller quantized models with strict output validation for business-critical apps.

People Also Ask

Does quantization affect inference speed or just memory?

Both. INT4 and INT8 reduce memory bandwidth pressure, which often raises tokens per second on GPU. Extreme low-bit formats can hurt accuracy enough that you retry prompts, wiping out any speed gain. Benchmark end-to-end latency on your hardware, not theoretical FLOPs.

Can you quantize a model you already fine-tuned?

Yes. Post-training quantization runs on fine-tuned checkpoints the same way it runs on base models. Re-calibrate with domain-specific text after fine-tuning. Quantizing before fine-tune (QAT) is rare in LLM workflows unless you control the full training stack.

Is Ollama quantization the same as llama.cpp quantization?

Ollama uses llama.cpp under the hood for most local models. The GGUF variants it pulls are pre-quantized upstream. Custom models you import should use the same Q4_K_M or Q5_K_M labels for predictable VRAM use.

What VRAM do I need for a Q4 70B model?

Weights alone need roughly 35–40 GB at Q4. Add KV cache for your context window—another 4–8 GB at 8K context is common. Plan for 48 GB total or use multi-GPU tensor parallelism with vLLM.

Ship smarter inference without buying new hardware

Model Quantization: Run Bigger Models on Less VRAM is not a hack—it is standard practice for 2026 LLM deployments. Pick the format your runtime supports, calibrate with real prompts, eval before release, and treat every quantized file as a distinct model version. That workflow lets a 12 GB laptop run useful 8B assistants and lets production teams defer the next GPU purchase by months.

If you want help wiring quantized inference into a Laravel app, Kubernetes cluster, or client portal, review our custom software development services or read how large language models actually work for architectural context. For performance tuning beyond VRAM, see speed optimization services and deploying a machine learning model as an API. Ready to plan your stack? Contact us with your model size, target latency, and GPU budget.