GPU Scheduling on Kubernetes

By Kokil Thapa | Last reviewed: August 2026

GPU scheduling on Kubernetes fails silently when device plugins are misconfigured or resource requests don’t match physical hardware topology. Unlike CPU and memory, GPUs are extended resources that require explicit vendor-specific setup before the scheduler can allocate them. Getting this right prevents expensive idle hardware and failed training jobs.

How does GPU scheduling on Kubernetes actually work?

Kubernetes doesn’t natively understand GPU hardware. It relies on the device plugin framework to discover accelerators and report them as extended resources. When a node starts, the device plugin daemonset registers available GPUs with the kubelet. The scheduler then sees nvidia.com/gpu: 4 (for example) in the node’s allocatable resources and can satisfy pod requests against that inventory.

Critical detail: GPUs are allocated as whole units by default. You cannot request 0.5 GPU unless you enable Multi-Instance GPU (MIG) or time-slicing. A pod requesting nvidia.com/gpu: 2 will only land on a node with at least two unallocated physical GPUs. This is why node labeling and affinity rules matter as much as the resource request itself.

How do you install and configure the NVIDIA device plugin in 2026?

The official NVIDIA device plugin Helm chart is the standard deployment method for Kubernetes 1.29+ clusters running in 2026. Avoid legacy static manifest deployments; they lack MIG support and proper upgrade paths.

Prerequisites

• NVIDIA driver ≥ 550 installed on all GPU nodes
• nvidia-container-toolkit ≥ 1.14 configured as the container runtime
• Kubernetes cluster ≥ 1.29 with feature gate DevicePlugins enabled (default)
• Helm 3.14+ installed locally

Installation steps

1. Add the NVIDIA Helm repository:
   

   helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
   helm repo update

2. Create a values file for production configuration:
   

   # nvidia-device-plugin-values.yaml
   version: v0.16.2
   migStrategy: mixed
   compatWithCPUManager: true
   failOnInitError: false
   nodeSelector:
     nvidia.com/gpu.present: "true"
   resources:
     limits:
       memory: 256Mi
     requests:
       cpu: 100m
       memory: 128Mi

3. Install the plugin:
   

   helm install nvidia-device-plugin nvdp/nvidia-device-plugin \
     --namespace kube-system \
     --values nvidia-device-plugin-values.yaml \
     --wait

4. Verify GPU detection:
   

   kubectl get nodes -o json | jq '.items[] | select(.status.allocatable["nvidia.com/gpu"] != null) | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}'

If GPUs don’t appear in allocatable resources within 60 seconds, check the device plugin pod logs in kube-system. Common failures include missing NVIDIA driver, incorrect container runtime config, or SELinux/AppArmor blocking device access. On Ubuntu 24.04 nodes I’ve managed, the most frequent issue was nvidia-container-runtime not being set as the default runtime in /etc/containerd/config.toml.

How do you request GPUs correctly in pod specifications?

GPU resources must be specified in both requests and limits, and the values must match exactly. Kubernetes rejects pods where GPU requests differ from limits because GPUs aren’t compressible resources.

apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.08-py3
    resources:
      requests:
        nvidia.com/gpu: 2
        memory: 32Gi
        cpu: "8"
      limits:
        nvidia.com/gpu: 2
        memory: 32Gi
        cpu: "8"
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: "all"
  nodeSelector:
    nvidia.com/gpu.product: "NVIDIA-A100-SXM4-80GB"
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Key points from real deployments:

• Never omit limits. Pods without GPU limits may be scheduled but fail at runtime with “device not found” errors.
• Use node selectors or affinity to target specific GPU models. Mixing A100s and T4s in the same pool causes unpredictable performance.
• Set tolerations if GPU nodes have taints (common practice to prevent non-GPU workloads from landing on expensive hardware).
• Avoid NVIDIA_VISIBLE_DEVICES=all in multi-tenant clusters. Let the device plugin handle isolation; overriding it breaks MIG partitioning.

When should you use MIG versus time-slicing for GPU sharing?

Multi-Instance GPU (MIG) physically partitions A100/H100/A30 GPUs into isolated slices with dedicated memory, cache, and compute cores. Time-slicing virtually shares a single GPU across multiple pods using CUDA MPS or time-sharing. Choose based on workload isolation requirements, not just utilization metrics.

In practice, I enable MIG mixed strategy on production clusters serving legal-tech document processing pipelines where latency SLAs matter. For internal ML experimentation environments, time-slicing with replicas: 4 per GPU gives researchers faster iteration without waiting for exclusive access. Never use time-slicing for latency-sensitive inference; one noisy neighbor stalls all co-located pods.

How do you debug GPU scheduling failures in production?

GPU scheduling issues manifest as pods stuck in Pending state with events like “Insufficient nvidia.com/gpu”. Systematic debugging requires checking four layers in order.

Diagnostic checklist

1. Verify node GPU capacity:
   

   kubectl describe node <gpu-node> | grep -A5 "Allocatable:"

   If nvidia.com/gpu is missing, the device plugin isn’t running or failed initialization.
2. Check device plugin health:
   

   kubectl logs -n kube-system -l app=nvidia-device-plugin --tail=200

   Look for “Failed to initialize NVML” or “permission denied” errors.
3. Inspect pending pod events:
   

   kubectl describe pod <pod-name> | grep -A10 Events:

   “Unschedulable” with GPU resource shortage means capacity exists but affinity/tolerations block placement.
4. Validate runtime configuration:
   

   ssh <gpu-node> 'cat /etc/containerd/config.toml | grep -A3 nvidia'

   Missing [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia] section causes silent allocation failures.

On a recent client project involving computer vision model training, pods remained pending despite four free A100s showing in node status. The root cause was a stale node label nvidia.com/gpu.memory=40960 left over from a previous T4 deployment. The pod’s node selector targeted 81920 MB, so the scheduler correctly refused placement even though raw GPU count was sufficient. Always validate labels match actual hardware after node replacements or upgrades.

What are the operational best practices for GPU clusters?

Running GPUs in production requires discipline beyond initial setup. These patterns come from maintaining CI/CD pipelines and infrastructure for ML workloads across multiple environments.

• Label nodes by GPU model and memory. Use nvidia.com/gpu.product and nvidia.com/gpu.memory consistently. Automated labeling via NVIDIA GPU Operator is preferred over manual annotation.
• Taint GPU nodes. Apply nvidia.com/gpu=true:NoSchedule to prevent CPU-only workloads from consuming expensive instances. Add matching tolerations only to GPU-aware workloads.
• Monitor allocation, not just utilization. High GPU utilization with zero pending pods indicates good packing. Low utilization with pending pods signals fragmentation or overly restrictive affinity rules.
• Pre-pull large container images. PyTorch/TensorFlow images exceed 8GB. Use ImagePullPolicy: IfNotPresent and consider a local registry mirror to avoid 5-minute pull times during scaling events.
• Test MIG reconfiguration offline. Changing MIG profiles requires GPU reset and takes 2–5 minutes per device. Never reconfigure production nodes during active workloads; drain first.

For teams evaluating whether to manage GPU infrastructure in-house versus using managed services, the break-even point typically arrives around 8–12 persistent GPUs. Below that threshold, cloud spot/preemptible instances with proper checkpointing often cost less than dedicated hardware plus engineering time. Above it, owned hardware with local hosting or reserved cloud capacity becomes economical, especially for steady-state inference workloads common in Nepal-based legal-tech and e-commerce platforms.

Implementing Reliable GPU Scheduling on Kubernetes

GPU scheduling on Kubernetes succeeds when you treat device plugins, resource specs, and node topology as interconnected system components rather than isolated configuration items. Install the NVIDIA device plugin via Helm with explicit version pinning, enforce matching requests and limits in every pod spec, use node labels and taints to control placement, and choose MIG or time-slicing based on isolation needs rather than convenience. Debug systematically from node capacity upward through plugin health, pod events, and runtime configuration.

If your team is setting up GPU workloads and needs hands-on implementation support, reach out to discuss your specific infrastructure requirements. Whether you’re deploying ML pipelines, inference services, or compute-intensive applications, getting the scheduling foundation right prevents costly rework later.