Run AI/ML Workloads on Kubernetes with GPUs

By Kokil Thapa | Last reviewed: August 2026

To successfully run AI/ML workloads on Kubernetes with GPUs, you must move beyond simple node labeling and implement the NVIDIA GPU Operator to automate driver installation, container runtime configuration, and device plugin management. While my daily work focuses on Laravel and web infrastructure, I frequently consult on backend systems that integrate ML inference APIs, where the underlying compute layer determines whether a model serves requests in 50ms or times out. This guide covers the exact configuration patterns, scheduling constraints, and operational checks required to make GPU-accelerated pods reliable in production, avoiding the common pitfalls of manual driver management and resource fragmentation.

How do you configure the NVIDIA GPU Operator for production clusters?

The most common mistake when attempting to build scalable and efficient systems with GPU acceleration is manually installing NVIDIA drivers on host nodes. This approach breaks during kernel upgrades, creates version drift between nodes, and makes cluster scaling painful. The NVIDIA GPU Operator solves this by containerizing the entire GPU software stack—drivers, container toolkit, device plugin, and DCGM exporter—as managed DaemonSets.

Installation and validation workflow

For Kubernetes 1.29+ running in 2026, the GPU Operator v24.x is the current stable baseline. Always install via Helm to enable atomic upgrades and rollback capabilities:

<!-- Add NVIDIA Helm repository -->
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

<!-- Install GPU Operator with default production values -->
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.version=570.124.06 \
  --set toolkit.version=v1.17.4-ubuntu22.04 \
  --set dcgmExporter.enabled=true \
  --wait

After installation, verify that all operator components are healthy before scheduling any workload. The nvidia-device-plugin pods must be Running on every GPU node, and the validation pod should complete successfully:

• Check operator status: kubectl get pods -n gpu-operator
• Verify GPU visibility: kubectl exec -it <device-plugin-pod> -- nvidia-smi
• Confirm allocatable resources: kubectl describe node <gpu-node> | grep nvidia.com/gpu
• Run validation suite: kubectl get pods -n gpu-operator -l app=nvidia-operator-validator

If the device plugin fails to start, check for conflicting manual installations. Remove any host-level nvidia-docker2 packages and ensure the node’s container runtime is configured to use the nvidia-container-runtime. The operator handles this automatically on fresh nodes, but pre-configured hosts often have leftover artifacts that cause initialization loops.

How do you schedule GPU resources and prevent fragmentation?

Requesting GPUs in Kubernetes is straightforward (nvidia.com/gpu: 1), but naive scheduling leads to severe fragmentation. A single A100 requested by a lightweight inference pod wastes 70GB of VRAM that could serve dozens of models. Production clusters require explicit partitioning strategies aligned with workload characteristics.

Multi-Instance GPU (MIG) for inference workloads

MIG physically partitions an A100 or H100 into isolated instances with dedicated memory, cache, and compute cores. This is essential for multi-tenant inference serving where latency guarantees matter. Configure MIG profiles through the GPU Operator’s device plugin config:

<!-- Example: Split A100-80GB into 7x 1g.10gb instances -->
apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - device-filter: ["0x20B210DE", "0x20B510DE"]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7

Pods then request specific MIG profiles instead of whole GPUs:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

Time-slicing for development and batch workloads

For non-latency-sensitive tasks like experimentation or overnight batch processing, time-slicing allows multiple pods to share a physical GPU without MIG’s rigid partitioning. Enable this via the device plugin configuration:

plugin:
  config:
    name: time-slicing-config
    default: time-slicing
---
# time-slicing-config ConfigMap
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4  # Allow 4 pods per physical GPU

Time-slicing does not provide memory isolation. Pods can still OOM each other. Use it only for trusted workloads or development namespaces where cost efficiency outweighs strict performance guarantees.

How do you monitor GPU utilization and detect underprovisioning?

GPU metrics are not exposed by default in Kubernetes. Without DCGM (Data Center GPU Manager) exporter, you’re flying blind—unable to distinguish between a saturated GPU and one idling at 5% while pods queue. The GPU Operator deploys DCGM exporter automatically when enabled, exposing Prometheus-compatible metrics on port 9400.

Critical metrics to alert on

Configure ServiceMonitor resources if using Prometheus Operator, or scrape annotations directly. These four metrics reveal real operational health:

1. DCGM_FI_DEV_GPU_UTIL: Actual SM (streaming multiprocessor) activity. Sustained <30% indicates over-provisioning or CPU bottlenecks.
2. DCGM_FI_DEV_FB_USED / FREE: Framebuffer memory pressure. Alert when used >90% for >5 minutes to prevent OOM kills.
3. DCGM_FI_DEV_GPU_TEMP: Thermal throttling threshold. A100/H100 throttle at 85°C; sustained temps above 80°C signal cooling issues.
4. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: Tensor core utilization specifically. Low tensor activity with high GPU util means inefficient code (e.g., excessive data loading).

Create alerts that trigger before user impact. A practical rule: if DCGM_FI_DEV_GPU_UTIL averages below 20% for 30 minutes across all GPUs in a node pool, scale down or consolidate workloads. Conversely, if pending GPU pods exceed available capacity for more than 10 minutes during business hours, trigger autoscaling or notify ops.

Integrating with application-level observability

GPU metrics alone don’t explain why inference latency spiked. Correlate DCGM metrics with your application’s request traces. For teams building REST APIs backed by ML models, instrument the inference endpoint to emit custom metrics alongside GPU telemetry. When p99 latency increases, check if it correlates with GPU memory pressure or thermal throttling—not just application code changes.

What security and multi-tenancy controls protect GPU workloads?

GPUs are expensive shared resources. Without proper isolation, a misbehaving pod can consume all GPU memory, crash neighboring workloads, or access sensitive model weights. Security for GPU workloads extends beyond standard Kubernetes RBAC.

Namespace-level resource quotas

Always enforce ResourceQuotas for GPU resources per namespace. Prevents any single team from monopolizing cluster capacity:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team-a
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"
    requests.nvidia.com/mig-1g.10gb: "8"

Pod security standards and runtime restrictions

GPU workloads historically required privileged containers. Modern GPU Operator versions support unprivileged execution with appropriate capabilities. Apply Pod Security Standards at the namespace level:

• Baseline profile: Sufficient for most inference workloads using MIG or time-slicing.
• Restricted profile: Requires additional seccomp/AppArmor tuning but provides strongest isolation for multi-tenant environments.
• Privileged exceptions: Only for training jobs requiring direct device access or custom kernel modules. Document and audit these explicitly.

Model weights and training data are often proprietary or regulated. Store them in encrypted PVCs or object storage with IAM-based access, never baked into container images. Use init containers to fetch credentials at runtime from vault services rather than environment variables.

How do you optimize costs when running AI/ML workloads on Kubernetes with GPUs?

GPU compute dominates ML infrastructure budgets. Optimizing costs isn’t about cheaper hardware—it’s about maximizing useful work per dollar spent. Three levers deliver immediate ROI without sacrificing reliability.

Right-sizing through profiling

Before committing to instance types, profile actual resource consumption. Many teams provision A100-80GB for models that fit comfortably in A10G or T4 memory. Run representative workloads with DCGM metrics collection for 24 hours, then analyze peak vs. average utilization. Downgrade instances where peak memory stays below 40% of provisioned capacity.

Spot/preemptible instances for fault-tolerant workloads

Training jobs with checkpointing tolerate interruption. Schedule these on spot instances at 60-90% discount. Implement graceful shutdown handlers that save checkpoints on SIGTERM. Never run stateful inference servers on spot unless you have redundant replicas and automatic failover.

Autoscaling aligned to demand patterns

Cluster Autoscaler supports GPU-aware scaling policies. Configure separate node groups for training (spot, scale-to-zero overnight) and inference (on-demand, minimum reserve capacity). Set scale-down delays longer than typical job durations to avoid thrashing. For Nepal-based teams managing global clients, align scaling schedules with client timezone business hours rather than local time.

Track cost-per-inference or cost-per-training-run as your primary efficiency metric, not raw GPU utilization. High utilization on oversized instances is still wasteful. Regularly review billing dashboards alongside technical metrics to catch drift before it compounds.

Next steps for reliable GPU operations

To reliably run AI/ML workloads on Kubernetes with GPUs, treat the GPU stack as first-class infrastructure: automate driver management with the GPU Operator, enforce allocation strategies matching workload profiles, instrument observability before launching production traffic, and apply defense-in-depth security controls. Start with MIG for inference serving and time-slicing for development, then graduate to whole-GPU allocations only when profiling proves necessity. Monitor DCGM metrics continuously and right-size quarterly based on actual usage patterns, not assumptions.

If your team needs help designing GPU infrastructure that integrates cleanly with existing web platforms or requires guidance on cloud hosting options suitable for hybrid deployments, reach out to discuss your specific requirements. Practical experience matters more than theoretical best practices when GPUs cost thousands per month.