Serve LLMs in Production: Throughput and Latency

By Kokil Thapa | Last reviewed: September 2026

Serving LLMs in production forces you to trade throughput against latency every day. A chat widget on a legal portal and a batch document pipeline on the same GPU cluster need opposite tuning. I integrate LLM APIs on production Laravel apps — I do not train models — and the gap between a demo and a stable service is almost always measured in tokens per second and time-to-first-token. Our AI integration and automation service handles this stack end to end. This guide covers how to Serve LLMs in Production: Throughput and Latency with concrete patterns you can ship this week.

What metrics matter when you Serve LLMs in Production: Throughput and Latency?

Throughput and latency are not one number. They describe different moments in the same request lifecycle. Confusing them leads to wrong hardware buys and angry users staring at blank screens.

Time to first token (TTFT) is the delay between your API call and the first streamed chunk. Users feel this as "snappiness." On interactive chat, TTFT above 800 ms feels broken on a fast connection. Legal-tech portals with document Q&A often target 400–600 ms TTFT for the first visible word.

Inter-token latency is the gap between consecutive tokens during generation. Smooth streaming needs roughly 20–40 ms per token for readable output. Spikes here cause stutter even when TTFT looks fine.

Tokens per second (TPS) measures aggregate generation speed. Higher TPS means more throughput per GPU dollar. Batch summarisation jobs care about sustained TPS. Live chat cares about TTFT first and per-token smoothness second.

Requests per second (RPS) counts completed API calls. It drops when context windows grow or batch sizes shrink. Track RPS alongside queue depth so you spot saturation before timeouts cascade.

Instrument every stage. A Server-Sent Events streaming setup lets you log TTFT at the HTTP layer while your inference server logs GPU-side batch stats. Without both views, you cannot tell whether slowness lives in the network, the queue, or the model.

Baseline targets by workload type

• Interactive chat: TTFT under 600 ms, streaming from token one, cap output at 512–1024 tokens unless the user opts in.
• RAG Q&A: TTFT under 800 ms including retrieval; precompute embeddings where possible.
• Batch summarisation: maximise TPS; latency per document can be minutes if jobs finish overnight.
• Function calling: TTFT matters less than total round-trip; see function calling patterns for timeout design.

Use a JSON log schema your team can grep in production. A JSON formatter helps during local debugging before logs hit your aggregator.

{
  "event": "llm_inference",
  "request_id": "req_8f2a",
  "model": "llama-3.1-8b-instruct",
  "ttft_ms": 412,
  "tokens_out": 187,
  "duration_ms": 2340,
  "tps": 79.9,
  "queue_wait_ms": 38,
  "gpu_id": 0
}

How do you choose between API providers and self-hosted inference?

The throughput-versus-latency trade-off starts at deployment mode. Managed APIs shift capacity risk to the vendor. Self-hosted inference on your own GPU gives control but adds ops burden you cannot ignore on a small team.

On client projects I usually start with a managed API for the MVP. I move to self-hosted vLLM when monthly token spend crosses roughly Rs 80,000–120,000 (~USD 600–900) or when data residency rules block third-party inference. The break-even point depends on model size and utilisation — run the numbers in our LLM cost optimization guide before buying hardware.

For self-hosting details, read self-hosting an LLM: options, costs, and GPU requirements. For local dev parity, Ollama vs vLLM locally covers what transfers to production and what does not.

How do you optimize throughput without killing user-facing latency?

Throughput gains often come from batching multiple requests on one GPU forward pass. Batching raises average TPS but adds queue wait for every request in the batch. The fix is to split traffic classes and tune the scheduler — not to batch everything blindly.

Continuous batching with vLLM

vLLM uses PagedAttention and continuous batching to merge decode steps across concurrent requests. That is the default choice for self-hosted production in 2026. Start the server with explicit memory limits so KV cache exhaustion does not crash the process mid-peak.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype auto \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-chunked-prefill

--max-num-seqs caps concurrent sequences. Raise it for throughput on batch jobs. Lower it when TTFT spikes on chat. --enable-chunked-prefill interleaves long prefills with decode work so one huge prompt does not block an entire GPU for seconds.

Separate queues for interactive and batch work

Never put overnight summarisation in the same Redis queue as live chat. On Laravel apps I use two queue names with different worker counts and timeouts — the same pattern as Laravel queues with Redis in production.

1. interactive-llm — concurrency 2–4 per GPU, job timeout 30 s, retries 0 (fail fast, show a message).
2. batch-llm — concurrency 8–16, timeout 600 s, retries 2 with backoff.
3. Route by middleware or job class; never infer priority from user role alone.
4. Expose queue depth to your health check; pause batch workers when interactive depth exceeds a threshold.

Rate limiting at the API edge stops one client from starving others. Our guide on API rate limiting and abuse prevention applies directly to LLM endpoints priced per token.

Streaming is non-negotiable for chat

Streaming does not reduce total generation time. It cuts perceived latency by showing TTFT immediately. Buffer the first token if you run output guardrails, but keep the buffer under 100 ms. Anything longer feels like a hang. Pair streaming with LLMOps monitoring and guardrails so safety checks do not silently add seconds.

How do you architect a production LLM serving stack?

A production stack has four layers: edge API, orchestration, inference, and observability. Skip any layer and you will debug blind at 2 a.m. when Dashain traffic spikes.

Edge API layer

Your Laravel or Symfony app should not call the GPU directly from HTTP workers. Put a thin gateway in front — even a dedicated microservice — that handles auth, rate limits, prompt templates, and request IDs. This mirrors how I build REST APIs for production clients: validate input, enqueue work, return a stream or job ID.

# Laravel Job — dispatch inference, stream via SSE controller
class RunLlmInference implements ShouldQueue
{
    public string $queue = 'interactive-llm';
    public int $timeout = 30;

    public function handle(LlmGateway $gateway): void
    {
        $gateway->stream(
            model: 'llama-3.1-8b-instruct',
            messages: $this->messages,
            max_tokens: 512,
            temperature: 0.2,
        );
    }
}

Keep prompts and system instructions in version-controlled templates. Hot-reload them without redeploying weights. On a legal-tech portal, document-type-specific prompts reduced average output length by 30% — which directly improved TPS without new hardware.

Inference layer

Run vLLM behind an internal load balancer. One GPU per process is the safe default. Multi-GPU tensor parallelism helps 70B+ models; 8B models rarely need it. Place inference in the same region as your app server. Cross-region hops add 100–200 ms before prefill even starts — the same lesson as hosting for a Nepal audience with latency tuning.

Observability layer

Track GPU utilisation, KV cache usage, queue depth, TTFT p50/p95, and TPS per model. Alert on p95 TTFT, not averages — averages hide tail latency that drives support tickets. Wire this into your broader LLMOps ship-and-operate workflow and MLOps production path.

On document portals like Mijar Law Associates, inference sits behind authenticated routes with audit logging. Latency targets matter, but so does PII protection in LLM apps — redact before the prompt hits the model.

What goes wrong when throughput and latency targets conflict?

Most production incidents I see are configuration mistakes, not model quality issues. The patterns repeat across eCommerce chatbots and legal document assistants alike.

KV cache exhaustion

Long contexts fill GPU memory with KV cache entries. When VRAM fills, vLLM evicts sequences or rejects new ones. Throughput collapses and TTFT jumps to seconds. Fix: lower --max-model-len, truncate retrieved chunks in RAG, or add GPUs before marketing pushes a "unlimited context" feature.

One queue for everything

A batch job holding 16 slots blocks chat users. Split queues and cap batch concurrency during business hours. Nepal businesses often peak around evening local time — schedule heavy jobs for off-peak windows.

Ignoring prefill cost

RAG pipelines that dump 20 retrieved chunks into every prompt inflate prefill time. Precompute summaries, use rerankers to cut chunk count, and cache frequent queries in Redis 8.10. Prefill is often 60–80% of TTFT on knowledge-base chat.

No autoscaling signal

GPU scaling is slow compared to PHP-FPM workers. Scale on queue depth and p95 TTFT, not CPU. Keep a warm standby instance if your SLA promises sub-second chat. Linux system administration for GPU nodes includes driver pinning, nvidia-smi monitoring, and disk space for model weights — a full 70B quantised model can exceed 40 GB on disk.

Skipping evals under load

Optimising TPS can degrade answer quality when temperature and top-p drift across deployments. Run LLM output evals after every scheduler tuning change. Throughput without quality checks ships regressions faster.

Before major releases, mirror production load in staging. Our staging environment guide applies to inference endpoints too — load-test with the same context lengths your RAG pipeline produces, not toy prompts.

For teams that want load testing and latency audits without building an ML team, testing and optimization services cover profiling, queue tuning, and regression baselines.

Key Takeaways

• Measure TTFT, inter-token latency, TPS, and RPS separately — one dashboard number hides user-facing pain.
• Split interactive and batch traffic into different queues with different concurrency limits on the same GPU pool.
• Use vLLM continuous batching for self-hosted inference; tune --max-num-seqs against your TTFT budget.
• Stream all chat responses; total generation time stays the same but perceived latency drops sharply.
• Cap context length and RAG chunk size before buying more GPUs — prefill often dominates TTFT.
• Run evals after every scheduler or batch-size change so throughput gains do not silently erode output quality.

People Also Ask

What is a good tokens-per-second rate for production LLMs?

For 7B–8B models on a single A10 or L4 GPU, expect 80–150 TPS aggregate under continuous batching with mixed chat load. Interactive single-user chat often sees 40–70 TPS per stream. Compare against your cost per million tokens and TTFT SLA — raw TPS alone does not justify hardware spend.

Does streaming improve LLM latency?

Streaming reduces perceived latency by surfacing TTFT immediately. Total wall-clock time for the full completion stays roughly the same. Always stream user-facing chat; buffer only for sub-100 ms safety checks.

How many concurrent users can one GPU serve?

It depends on model size, context length, and output cap. An 8B model on 24 GB VRAM with 4K context often handles 20–40 concurrent decode streams with acceptable TTFT. Load-test your exact prompt templates — RAG prefill changes the answer completely.

Should I use CPU or GPU for LLM inference in production?

Use GPU for any user-facing workload in 2026. CPU inference via llama.cpp suits dev laptops and offline scripts. Production chat on CPU routinely delivers TTFT above 5 seconds on 8B models — unacceptable for support or sales use cases.

Ship inference you can measure and trust

Serve LLMs in Production: Throughput and Latency is an engineering discipline, not a one-time benchmark. Log the four core metrics, split your queues, stream chat output, and tune batch sizes against real prompts — not synthetic tests. When you need help wiring LLM gateways into Laravel, Redis queues, and GPU infrastructure, contact us or explore AI integration for production apps. Start with one workload class, measure for a week, then expand — that beats over-provisioning GPUs on day one.