NVIDIA Container Toolkit for Docker GPU

By Kokil Thapa | Last reviewed: September 2026

Your ML container starts fine on CPU, then dies the moment it needs CUDA. The host has an NVIDIA GPU and nvidia-smi works on bare metal, but Docker still cannot see the device. That gap is exactly what NVIDIA Container Toolkit for Docker GPU closes—it wires the host driver into the container runtime so GPU workloads run the same way inside containers as on the host. If you already followed our Docker installation guide for Ubuntu, this page picks up where that leaves off. I maintain production Ubuntu servers for client workloads, and GPU passthrough is one of the first things I verify before any AI or batch job ships.

What is the NVIDIA Container Toolkit for Docker GPU?

The toolkit is NVIDIA's supported bridge between Linux GPU drivers and OCI runtimes like Docker and containerd. It is not a replacement for the driver. You still install proprietary or open-kernel NVIDIA drivers on the host first. The toolkit then injects userspace libraries and device nodes into containers at start time.

Three packages matter in practice:

• libnvidia-container — low-level library that mounts GPU devices and driver files into a container namespace.
• nvidia-container-toolkit — CLI tools including nvidia-ctk for runtime configuration.
• nvidia-container-runtime — OCI-compliant runtime hook Docker invokes when you request GPU access.

Docker does not talk to the GPU directly. It delegates to the configured runtime, which calls NVIDIA hooks before your container process starts. That design keeps containers portable while still giving CUDA apps what they need. For background on how runtimes fit together, see our explainer on containerd and the container runtime stack.

Modern toolkit releases also support CDI (Container Device Interface). CDI generates device specs under /etc/cdi/ so runtimes can attach GPUs without legacy hook paths. Docker 25+ and recent toolkit versions prefer CDI on fresh installs. Both paths work; pick one and stay consistent across your fleet.

If you run mixed infrastructure—bare Docker on Ubuntu plus Kubernetes elsewhere—the same driver and toolkit versions should match. Our GPU scheduling on Kubernetes guide covers the cluster side; this page focuses on single-host Docker.

How do you install the NVIDIA Container Toolkit on Ubuntu?

Installation assumes Ubuntu 22.04 or 24.04 LTS, root or sudo access, and a 64-bit x86_64 or ARM64 host with a supported NVIDIA GPU. Verify the driver before touching Docker GPU settings.

Step 1: Confirm the host driver

Run nvidia-smi on the host. You should see driver version, CUDA version, and at least one GPU listed. If the command fails, install drivers from NVIDIA's repository or Ubuntu's ubuntu-drivers metapackage first. Containers inherit driver capabilities from the host—they cannot upgrade a missing kernel module.

Step 2: Add the NVIDIA Container Toolkit repository

NVIDIA publishes apt packages through their libnvidia-container stable channel. Use the signed-by keyring pattern so apt verifies package integrity:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Package names may bundle runtime and toolkit together on current releases. Confirm with dpkg -l | grep nvidia-container after install.

Step 3: Configure Docker to use the NVIDIA runtime

The nvidia-ctk helper writes the correct snippet into Docker's config. This is cleaner than hand-editing JSON:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Inspect /etc/docker/daemon.json. You should see default-runtime still set to runc, with nvidia listed under runtimes. Do not set NVIDIA as the global default unless every container on that host needs a GPU. Most production hosts run mixed CPU and GPU workloads.

Step 4: Smoke-test GPU access

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

The output should mirror host nvidia-smi. If it does, NVIDIA Container Toolkit for Docker GPU is working. Pin CUDA base image tags in production rather than using latest. CUDA minor versions must stay compatible with your host driver per NVIDIA's CUDA compatibility matrix.

For servers you do not manage yourself, our Linux system administration service covers driver installs, Docker hardening, and GPU host prep on Ubuntu 22/24.

How do you run Docker containers with GPU access?

Docker exposes GPUs through the --gpus flag on docker run. Under the hood, Docker sets the NVIDIA runtime and passes device requests to the toolkit hooks.

Common docker run patterns

# All GPUs
docker run --rm --gpus all my-ml-image:latest

# Specific GPU indices
docker run --rm --gpus '"device=0,2"' my-ml-image:latest

# One GPU with memory and CPU limits
docker run --rm --gpus '"device=0"' \
  --memory=16g --cpus=4 my-ml-image:latest

GPU isolation is index-based, not fractional like Kubernetes MIG unless you configure MIG profiles on supported datacenter cards. Combine GPU flags with standard resource limits documented in our guide on limiting Docker container resources so one training job cannot starve the host.

Runtime flags and environment

Most CUDA images expect NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES. The toolkit sets sensible defaults—typically compute,utility. Add graphics or video only when you need OpenGL or NVENC inside the container.

docker run --rm --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics \
  my-opengl-app:latest

Persist model weights and datasets on bind mounts or named volumes. GPU memory is ephemeral; checkpoint files belong on disk. See Docker volumes and persistent data for layout patterns that survive container restarts.

Pick one method per environment and document it in your runbooks. Mixing legacy --runtime=nvidia with modern --gpus on the same host causes confusion during handoffs.

How does Docker Compose configure GPU devices?

Compose v2 on Linux supports GPU access without Docker Swarm. The simplest form uses the top-level gpus service key:

services:
  inference:
    image: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime
    gpus: all
    volumes:
      - ./models:/models:ro
    command: python /models/serve.py
    ports:
      - "8080:8080"

For explicit device counts and capabilities—closer to production Swarm stacks—use the deploy resources block:

services:
  trainer:
    image: my-trainer:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

When you maintain several services—API, worker, Redis—wire them together as shown in our Docker Compose multi-container apps guide. Add a GPU profile so CPU-only developers can start the stack without hardware:

services:
  worker-gpu:
    profiles: ["gpu"]
    gpus: all
    image: my-worker:latest

Start with docker compose --profile gpu up. Validate your Compose YAML with a JSON formatter when merging override files—trailing commas break parsing silently in some editors.

Self-hosting LLMs locally pushes this setup hard. GPU memory dictates which models fit; our self-hosting an LLM cost and GPU requirements breakdown helps you size hardware before you buy cards. For production inference pipelines, AI integration and automation covers API wiring, queue workers, and deployment patterns beyond raw GPU access.

How do you troubleshoot NVIDIA Container Toolkit GPU errors?

Most failures fall into four buckets: missing driver, stale Docker config, wrong CUDA image, or permission issues. Work through them in order instead of reinstalling everything.

Error: could not select device driver "" with capabilities: [[gpu]]

Docker cannot find the NVIDIA runtime. Re-run sudo nvidia-ctk runtime configure --runtime=docker and restart Docker. Confirm /etc/docker/daemon.json contains the nvidia runtime entry. On hosts migrated from legacy nvidia-docker2, remove old packages—they conflict with the current toolkit.

Error: Failed to initialize NVML / unknown error

The container started but driver userspace inside does not match the host kernel module. Check host nvidia-smi first. If the host works, your CUDA base image may be too new for the installed driver. Downgrade the image tag or upgrade the host driver to a compatible branch.

Error: no GPUs found / CUDA unavailable

Verify you passed --gpus or Compose GPU keys. A plain docker run without GPU flags will start a CUDA image on CPU-only paths and fail at runtime. Inside the container, run ls -la /dev/nvidia*—you should see device nodes.

Rootless Docker and GPU access

Rootless Docker GPU support improved in recent releases but still trails rootful setups. Map user namespaces and cgroup settings carefully. For production ML hosts, I still recommend rootful Docker with strict firewall and rootless patterns reserved for CI agents without GPUs.

Scan GPU images before deploy—the CUDA runtime is still a full OS layer with packages worth auditing. Our Trivy container scanning guide fits CI pipelines that build custom ML images. For Kubernetes migration paths, read serving ML models with GPU on Kubernetes once single-host Docker is stable.

Production checklist

1. Pin driver, toolkit, Docker, and CUDA image versions in internal docs.
2. Monitor GPU utilisation with nvidia-smi dmon or DCGM exporters.
3. Set restart policies and memory limits so runaway kernels do not hang the host.
4. Schedule driver reboots during maintenance windows—kernel module updates require it.
5. Back up compose files and daemon.json alongside application code.

On client projects I've shipped, the expensive mistakes happen after install—unpinned images, missing limits, no rollback plan. Treat GPU hosts like database servers: change control, backups, and monitored alerts. Networking between GPU workers and APIs still follows standard Docker patterns covered in Docker networking explained.

Application teams building on Laravel often use Sail for local dev without GPUs, then promote to GPU servers for inference. That split is normal—see local Laravel dev with Sail and Docker for the CPU side and promote only GPU services to toolkit-enabled hosts. Custom ML pipelines fall under custom software development when you need more than off-the-shelf containers.

Hardware costs in Nepal vary widely—a used RTX 3090 might run Rs 120,000–150,000 (~USD 900–1,100) while datacenter A100 rentals on cloud exceed Rs 500/hour (~USD 3.70/hour). For budget planning, factor electricity and cooling; GPUs draw sustained load a typical office UPS cannot handle. We've documented similar trade-offs on the Gulfbizlist platform project where infrastructure choices shaped deployment architecture.

Official references worth bookmarking: the NVIDIA Container Toolkit install guide, Docker GPU resource constraints documentation, and your GPU's driver release notes on NVIDIA's developer portal.

Key Takeaways

• Install and verify the NVIDIA host driver with nvidia-smi before installing NVIDIA Container Toolkit for Docker GPU.
• Run nvidia-ctk runtime configure --runtime=docker, restart Docker, then smoke-test with a pinned CUDA base image.
• Use --gpus all or Compose gpus / deploy.resources.devices—never legacy runtime flags on new installs.
• Match CUDA image tags to your host driver using NVIDIA's compatibility matrix to avoid NVML errors.
• Combine GPU access with CPU, memory, and volume limits so training jobs cannot take down the host.
• Document pinned versions and scan ML images in CI before production deploy.

People Also Ask

Do I need nvidia-docker2 in 2026?

No. NVIDIA deprecated nvidia-docker2 in favour of NVIDIA Container Toolkit with the --gpus flag. Remove old packages if present—they override daemon.json and break modern Docker GPU requests.

Can Docker containers use multiple GPUs?

Yes. Pass --gpus '"device=0,1"' or set count: 2 in Compose device reservations. All requested GPUs must be free on the host; the toolkit does not queue workloads across devices.

Does WSL2 support NVIDIA Container Toolkit for Docker GPU?

Yes on Windows 11 with WSL2 GPU paravirtualization and current Windows NVIDIA drivers. Install the toolkit inside the WSL2 Linux distro, not on Windows host Docker Desktop alone. Native Linux production hosts remain simpler to debug.

How is CDI different from the legacy NVIDIA runtime?

CDI generates standardized device JSON under /etc/cdi/ that any CDI-aware runtime consumes. Legacy hooks still work, but CDI is the forward path for Docker 25+ and reduces custom runtime configuration drift across hosts.

Ship GPU Workloads With Confidence

NVIDIA Container Toolkit for Docker GPU is the standard way to expose CUDA hardware to containers on Linux in 2026. Install the driver, add the toolkit, configure Docker once, and validate with nvidia-smi inside a pinned CUDA image. From there, Compose stacks, inference APIs, and batch trainers behave predictably—as long as you pin versions and set resource limits. Need help preparing Ubuntu GPU servers, wiring ML services, or hardening production Docker hosts? Contact us or read more on the about page about how we deploy and maintain production infrastructure. Related reading: multi-stage Docker builds, containerizing a Laravel app, and the homepage for services overview.