Serve ML Models with GPU on Kubernetes

By Kokil Thapa | Last reviewed: September 2026

You need low-latency inference, not another notebook demo. To serve ML models with GPU on Kubernetes, you expose a stable HTTP or gRPC endpoint backed by NVIDIA GPUs, with scheduling, health checks, and autoscaling wired in from day one. Most teams already run apps on Kubernetes; the hard part is GPU drivers, device plugins, and serving runtimes that actually use the hardware you pay for. This guide walks through a production path—from cluster prep through rollout and cost control—that mirrors what I have seen on AI integration and automation projects where inference sits beside Laravel APIs and business workflows.

What do you need before you can serve ML models with GPU on Kubernetes?

GPU inference on Kubernetes is an infrastructure problem first. The control plane does not understand GPUs out of the box. You need compatible hardware, a container runtime, and a device plugin that advertises GPUs as schedulable resources.

Hardware and cluster baseline

Every GPU node needs an NVIDIA driver matched to your GPU generation. Datacenter cards (A100, L4, T4) differ from consumer RTX units in memory, ECC, and multi-instance GPU (MIG) support. For a small team in Nepal or abroad, a single cloud GPU node often beats buying hardware upfront—expect roughly Rs 45,000–120,000/month (~USD 340–900) for one managed GPU worker, depending on region and card type.

Your Kubernetes version should be within the supported window of the NVIDIA GPU Operator. Ubuntu 22.04 or 24.04 worker nodes with containerd are the common choice. If you are new to node layout, read Kubernetes worker node architecture before adding GPU pools.

Software stack checklist

• NVIDIA GPU Operator — installs drivers, container toolkit, device plugin, and optional DCGM exporters.
• Serving runtime — KServe, NVIDIA Triton Inference Server, TorchServe, or vLLM for LLMs.
• Model artifact store — S3-compatible object storage or a ReadWriteMany PVC; see OpenEBS for Kubernetes storage for local options.
• Observability — Prometheus metrics, request latency histograms, and GPU utilisation from DCGM.
• Security — RBAC, network policies, and image scanning; start with Kubernetes RBAC.

If your main product is a PHP or Laravel API, treat the GPU cluster as a separate inference tier. Your app calls it over HTTP; that pattern keeps GPU complexity out of your web tier and matches how REST API development teams usually split concerns.

How do you install and verify the NVIDIA GPU Operator?

Without the operator, pods requesting nvidia.com/gpu stay Pending forever. The operator bundles driver management and the device plugin so you do not SSH into every node manually.

Install with Helm

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true

Label GPU nodes so only inference workloads land there:

kubectl label nodes gpu-node-01 nvidia.com/gpu.present=true
kubectl label nodes gpu-node-01 node-role.kubernetes.io/gpu-worker=true

Verify the device plugin advertises capacity:

kubectl get nodes -o json | jq '.items[] | {name:.metadata.name, gpu:.status.capacity["nvidia.com/gpu"]}'

You should see a numeric GPU count per labelled node. If the count is zero or missing, check operator pod logs before deploying any model. For deeper scheduling behaviour, read GPU scheduling on Kubernetes.

Smoke-test with a CUDA pod

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  restartPolicy: OnFailure
  nodeSelector:
    nvidia.com/gpu.present: "true"
  containers:
  - name: cuda
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
    resources:
      limits:
        nvidia.com/gpu: 1

Apply the manifest and confirm the pod reaches Running. A failed test here saves hours of debugging model containers later. When pods crash repeatedly, use CrashLoopBackOff debugging steps before touching model code.

Which serving stack should you use for GPU inference on Kubernetes?

Pick the runtime based on model format, batching needs, and team skills—not hype. All options below can serve ML models with GPU on Kubernetes when configured correctly.

For most greenfield inference in 2026, KServe or Triton covers the majority of cases. KServe integrates cleanly with Istio or Knative-style scale-to-zero; Triton wins when you need one GPU serving many optimised models. See KServe model serving on Kubernetes for predictor-specific details.

Example: KServe InferenceService with GPU

Store your model in a versioned bucket. Pin versions in production; ad-hoc latest tags break rollbacks. Track artifacts using ideas from model versioning and registries.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sentiment-gpu
  namespace: ml-serving
spec:
  predictor:
    sklearn:
      storageUri: s3://ml-models/sentiment/v3
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "1"
          memory: 2Gi
          nvidia.com/gpu: "1"
      nodeSelector:
        nvidia.com/gpu.present: "true"

Expose the service through your Ingress controller. TLS termination at the edge keeps certificates out of model pods. Compare controller options in Kubernetes Ingress controllers explained.

Example: Triton with TensorRT on GPU

Triton loads a config.pbtxt per model and can run TensorRT engines for lower latency. Mount models from PVC or S3 via init containers.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-gpu
spec:
  replicas: 1
  selector:
    matchLabels:
      app: triton-gpu
  template:
    metadata:
      labels:
        app: triton-gpu
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:24.08-py3
        args: ["tritonserver", "--model-repository=/models"]
        ports:
        - containerPort: 8000
        - containerPort: 8001
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 8Gi
          requests:
            nvidia.com/gpu: 1
            memory: 4Gi
        volumeMounts:
        - name: models
          mountPath: /models
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: triton-models-pvc

Validate payloads with a JSON formatter during integration testing. Small schema mismatches cause 400 storms that look like GPU failures.

How do you deploy and expose GPU inference safely in production?

A working pod is half the job. Production means resource guarantees, health probes, secrets, and network isolation.

Set requests and limits correctly

GPUs are exclusive by default: one pod with limits.nvidia.com/gpu: 1 gets a whole card unless you use MIG or time-slicing. Always set equal requests and limits for GPU count. Under-provisioned CPU or memory can OOM-kill the container while the GPU sits idle. Read Kubernetes resource limits and requests before tuning.

Health checks that reflect real inference

Do not rely on TCP socket checks alone. Hit a lightweight /health or /v2/health/ready endpoint that loads minimal GPU context where possible.

livenessProbe:
  httpGet:
    path: /v2/health/live
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /v2/health/ready
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10

Large models need longer initialDelaySeconds. A too-aggressive probe kills pods during weight load and triggers CrashLoopBackOff loops.

Network and secrets

Restrict east-west traffic with NetworkPolicy so only your API tier reaches inference pods. Pull model weights using short-lived credentials from Vault or cloud IAM—not baked into images. For broader ML platform context, see run AI/ML workloads on Kubernetes with GPUs and Kubeflow ML pipelines on Kubernetes.

1. Create a dedicated ml-serving namespace with RBAC scoped to deployers.
2. Mount model storage read-only where the runtime allows it.
3. Enable audit logging on Ingress for request tracing.
4. Schedule nightly PVC snapshots; see Velero backup and restore for Kubernetes.
5. Document rollback: previous image tag plus previous model version URI.

How do you autoscale and control cost for GPU model serving?

GPUs are expensive when idle. Autoscaling on CPU percentage alone fails for inference—GPU utilisation or request concurrency drives better decisions.

Horizontal Pod Autoscaler with custom metrics

Install Prometheus Adapter or KEDA. Scale on requests per second, queue depth, or DCGM GPU utilisation.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: triton-gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: triton-gpu
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Pods
    pods:
      metric:
        name: inference_requests_per_second
      target:
        type: AverageValue
        averageValue: "30"

KServe can scale to zero when traffic drops. Cold starts for large LLMs may take 30–90 seconds—acceptable for batch, painful for interactive chat. Match scale-to-zero to SLA expectations. Details live in horizontal pod autoscaling in Kubernetes.

Cost levers that actually work

• Right-size the GPU — an L4 often suffices where teams default to A100.
• Batch inference — queue overnight jobs on shared nodes instead of 24/7 replicas.
• Spot/preemptible nodes — fine for fault-tolerant batch; risky for sole replica serving.
• TensorRT / ONNX optimisation — smaller memory footprint means fewer nodes.
• Monitor drift — bad models generate retries and wasted GPU cycles; see monitor ML models in production for drift.

Tune cluster-wide performance with guidance from Kubernetes performance tuning. If you operate mixed CPU and GPU pools, Linux system administration support keeps driver upgrades from blocking releases.

What breaks most often when teams serve ML models with GPU on Kubernetes?

Most incidents are plumbing, not math. These patterns show up repeatedly across client environments.

Pod stuck Pending

Cause: no node with free nvidia.com/gpu, missing tolerations for GPU taints, or image pull failures. Run kubectl describe pod and read Events first. Confirm the GPU Operator daemonsets are healthy on the target node.

CUDA out of memory

Cause: batch size too large, multiple processes on one GPU, or loading full FP32 weights on a small card. Fix with smaller batches, TensorRT quantisation, or a larger GPU SKU—not blind replica increases.

High latency under load

Cause: insufficient dynamic batching, CPU bottleneck in pre/post-processing, or storage latency loading models from slow NFS. Profile before buying more GPUs. On one production deployment, moving weights to local NVMe cut cold-start time more than adding a second replica.

Driver and kernel drift

Cause: unattended node upgrades without pinning operator versions. Treat GPU nodes like pets: cordon, drain, upgrade driver stack, validate cuda-test, uncordon. Document the sequence in your runbook alongside support and maintenance procedures.

When inference backs a customer-facing product—similar to platforms like Gulfbizlist or internal AI features on a Laravel app—define SLOs for p95 latency and error rate before launch. GPU clusters fail loudly; your users should not be the first alert channel.

Key Takeaways

• Install the NVIDIA GPU Operator and verify nvidia.com/gpu capacity before deploying any model server.
• Choose KServe for unified rollouts, Triton for multi-model GPU throughput, or vLLM for LLM APIs.
• Always set equal GPU requests and limits, and use inference-aware readiness probes.
• Autoscale on RPS or GPU metrics—not CPU alone—and accept cold-start trade-offs with scale-to-zero.
• Version model artifacts, automate backups, and canary new weights before sending 100% traffic.
• Most outages are scheduling, memory, or driver issues; fix plumbing before retraining models.

People Also Ask

Can Kubernetes share one GPU across multiple model pods?

By default, one GPU limit maps to one exclusive device per pod. NVIDIA time-slicing and MIG partitions let you split cards, but you must configure the GPU Operator accordingly. Shared setups trade isolation for density—fine for dev, risky for latency-sensitive production unless you monitor contention closely.

Do you need Istio to serve models with KServe?

KServe historically integrated with Istio or Knative for traffic splitting and scale-to-zero. You can run predictors without a service mesh, but you lose built-in canary and revision routing. Many teams use a plain Ingress for internal APIs and add Istio when rollout complexity justifies it.

How is serving an LLM on GPU different from a small sklearn model?

LLMs need far more VRAM, longer load times, and runtimes like vLLM with continuous batching. A T4 may host a quantised 7B model; larger models need L4 or A100-class hardware. Plan autoscaling around cold starts and token throughput, not request count alone.

Is GPU inference on Kubernetes overkill for a startup?

Not always. Managed single-node GPU plus KServe or Triton beats bespoke VMs when you already run Kubernetes for other services. For a lone model and minimal traffic, a managed API (OpenAI, Bedrock, or a regional provider) may cost less until volume grows. Re-evaluate when monthly inference spend exceeds roughly Rs 65,000 (~USD 490) and you need data residency or custom weights.

Ship GPU inference you can operate tomorrow

To serve ML models with GPU on Kubernetes, treat GPUs as a scheduled resource, pick a serving runtime matched to your model family, and wire observability plus autoscaling before launch—not after the first outage. Start with one GPU node, prove cuda-test and a canary InferenceService, then expand the pool based on measured p95 latency and utilisation. If you want help connecting inference to a Laravel or WordPress product, defining API contracts, or hardening the cluster layer, contact us to plan the integration. For related reading, browse technical articles on Kubernetes and ML ops or explore more about the engineering approach behind these production patterns.