CUDA and Container GPU Basics

By Kokil Thapa | Last reviewed: September 2026

You need CUDA and Container GPU Basics when a model training job or inference API works on bare metal but fails the moment you wrap it in Docker. The GPU is present on the host, yet the container sees no devices, wrong driver versions, or a cryptic libcuda.so error. That gap sits between the NVIDIA kernel driver, user-space CUDA libraries inside the image, and the container runtime hooks that wire them together. This guide walks through that stack on Ubuntu with Docker, the checks I run on production Linux servers, and the failure modes that waste hours if you skip them. For the toolkit install path alone, see the companion piece on the NVIDIA Container Toolkit for Docker GPU access.

What is CUDA and how does GPU access work inside a container?

CUDA is NVIDIA's parallel computing platform. Your application calls CUDA APIs; those APIs load user-space libraries that talk to the kernel driver, which schedules work on the physical GPU. Containers do not embed a GPU. They receive bind-mounted device nodes and injected libraries from the host at start time.

Think of four layers stacked on the server:

• Hardware GPU — the physical card (datacenter A100/H100 or a workstation RTX).
• Kernel driver — nvidia.ko plus device files under /dev/nvidia*.
• User-space driver / CUDA runtime — libcuda.so, cuDNN, NCCL inside the image or mounted from the host.
• Container runtime — Docker or containerd plus the NVIDIA Container Toolkit CDI or legacy hook.

On Linux system administration engagements, the first question is always whether the problem is host-side or image-side. Host-side means driver missing or toolkit not configured. Image-side means CUDA 12 libraries against a driver that only supports CUDA 11.

The Linux namespaces that form the basis of containers isolate process, mount, and network views. GPU passthrough is an deliberate exception. The toolkit mounts /dev/nvidia0, /dev/nvidiactl, and /dev/nvidia-uvm into the container namespace and sets LD_LIBRARY_PATH so the app finds libcuda.so.

Understanding containerd as the container runtime matters because Docker on modern Ubuntu often delegates to containerd. The NVIDIA toolkit must register with whichever runtime actually launches your pods or containers.

How do you install the NVIDIA driver and Container Toolkit on Ubuntu?

Start on the host, never inside an unprivileged container. If nvidia-smi fails on bare metal, no amount of Docker flags will help.

Step 1: Install the host driver

On Ubuntu 22.04 or 24.04 LTS, the recommended path is NVIDIA's package repository or Ubuntu's ubuntu-drivers meta-package for workstation cards. Datacenter teams usually pin a specific driver branch tested against their GPU fleet.

sudo apt update
sudo apt install -y linux-headers-$(uname -r)
sudo ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot

After reboot, confirm the driver:

nvidia-smi

You should see the driver version, CUDA driver API version, and listed GPUs. Record the driver version. You will need it when picking container base images.

Step 2: Install the NVIDIA Container Toolkit

The toolkit adds a Docker (or containerd) runtime hook. Follow the official install guide at NVIDIA Container Toolkit documentation. On Ubuntu:

1. Add the NVIDIA GPU repository GPG key and apt source.
2. Install nvidia-container-toolkit.
3. Run sudo nvidia-ctk runtime configure --runtime=docker.
4. Restart Docker: sudo systemctl restart docker.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

For production hosts I maintain under support and maintenance contracts, I document the driver and toolkit versions in the server runbook. Upgrades without that note cause silent regressions weeks later.

How do you run a GPU-enabled Docker container?

Once the host is configured, exposing GPUs to a container is a single flag in modern Docker. The Docker documentation on GPU resource constraints describes the current CLI syntax.

Basic GPU passthrough

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

The --gpus all flag tells Docker to attach every visible GPU. For a single card on a multi-GPU server:

docker run --rm --gpus '"device=0"' nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

To cap visible devices by UUID — useful when GPU 0 is reserved for display:

docker run --rm --gpus '"device=UUID-GPU-XXXX"' my-ml-image:latest python train.py

Docker Compose with GPUs

Compose v2 supports the deploy.resources.reservations.devices block. A minimal service definition:

services:
  trainer:
    image: my-pytorch:2.6-cuda12
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

See Docker Compose for multi-container apps for networking and volume patterns that pair with GPU services. Inference and training stacks often split a CPU-only API gateway from a GPU worker container.

Environment variables that matter

• NVIDIA_VISIBLE_DEVICES — comma-separated indices, UUIDs, or all.
• NVIDIA_DRIVER_CAPABILITIES — typically compute,utility; add graphics only if you need OpenGL inside the container.
• CUDA_VISIBLE_DEVICES — application-level mask; respected by CUDA-aware frameworks after devices are mounted.

When wiring AI integration and automation pipelines for clients, I set these explicitly in Compose or Kubernetes manifests. Defaults work until they do not — especially on shared GPU nodes.

What is the difference between host driver CUDA and image CUDA versions?

This confusion causes more production incidents than any other GPU topic. The CUDA version printed by nvidia-smi on the host is the maximum CUDA driver API the installed driver supports. The CUDA version in your container image tag (for example cuda:12.6) is the user-space toolkit bundled for compilation and linking.

Rule of thumb: the image CUDA minor version must be less than or equal to what the host driver supports. A container built with CUDA 12.6 libraries runs on a driver that reports CUDA 12.8 in nvidia-smi. The reverse fails — CUDA 12.6 image on a driver capped at CUDA 11.8 produces errors about insufficient driver version.

NVIDIA publishes compatibility matrices in the CUDA Toolkit release notes. Pin them in your internal docs the same way you pin PHP or Node versions on web stacks.

For orchestrated workloads, read GPU scheduling on Kubernetes and serving ML models with GPU on Kubernetes. The driver/toolkit prep on each node is identical to Docker standalone. Kubernetes adds device plugin allocation and fractioning concerns on top.

When building custom images, prefer NVIDIA's official nvidia/cuda bases (base, runtime, or devel). The devel variant includes nvcc if you compile CUDA inside the build stage. Multi-stage builds copy only the runtime artifacts into the final slim layer.

How do you troubleshoot GPU access failures in containers?

Work top-down. Confirm the host, then the toolkit, then the container image, then the application.

Host checks

nvidia-smi
ls -l /dev/nvidia*
docker info 2>/dev/null | grep -i nvidia

If /dev/nvidia* nodes are missing, the driver install is incomplete. If Docker info shows no NVIDIA runtime, rerun nvidia-ctk runtime configure and restart Docker.

Container smoke test

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

When this passes but your app image fails, diff the environments:

docker run --rm --gpus all my-app:latest env | grep -E 'NVIDIA|CUDA'
docker run --rm --gpus all my-app:latest ls -l /dev/nvidia*

Parse structured output with a JSON formatter when you export nvidia-smi --query-gpu=... results into monitoring pipelines.

Common errors and fixes

• could not select device driver "" with capabilities: [[gpu]] — toolkit not configured or Docker not restarted after install.
• Unknown flag: gpus — Docker Engine too old; upgrade to 19.03+ or use nvidia-docker2 legacy wrapper on ancient hosts.
• CUDA driver version is insufficient — upgrade host driver or downgrade container CUDA tag.
• Out of memory — unrelated to passthrough; reduce batch size or set Docker container resource limits alongside GPU flags so CPU RAM does not starve the job.

For stuck containers, apply techniques from debugging a running container — exec in with the same GPU flags, strace failing library loads, verify ldconfig -p | grep cuda.

What security and operations practices apply to GPU containers?

GPU containers are not a security boundary. Any process with access to /dev/nvidia0 can execute arbitrary kernels on the card. Treat GPU nodes like root-equivalent workloads.

• Run training jobs as non-root inside the image where the framework allows it.
• Scan images with tools covered in container image scanning with Trivy before deploying to GPU fleets.
• Segregate multi-tenant GPU servers with MIG (on supported datacenter GPUs) or separate physical nodes — not just different containers sharing one card.
• Monitor temperature, power, and memory via nvidia-smi dmon or DCGM exporters.

GPU servers in Nepal often run on imported hardware with limited local vendor support. Budget Rs 15,000–40,000/month (~USD 110–295) for a cloud GPU instance versus capital expense on a local RTX or A-series card. Factor electricity and cooling into that decision.

For enterprise deployments, pair this stack with enterprise application development practices — CI pipelines that build CUDA images on GPU runners, promote immutable tags, and roll back on failed health checks. I've seen teams commit multi-gigabyte CUDA layers without layer caching; build times balloon and deploy windows slip.

Read AI governance and responsible AI basics when end users interact with model output your GPU containers serve. Infrastructure correctness does not replace policy.

Performance tuning on the host — PCIe gen, power limits, persistence mode — overlaps with Linux performance tuning basics. Enable persistence mode on datacenter GPUs that launch many short-lived containers; it removes driver re-init latency per job.

sudo nvidia-smi -pm 1

On shared CI runners, document which runners carry GPUs. A docker build without --gpus still succeeds but never exercises CUDA code paths until production — a nasty surprise.

Examples from shipped work: a business listing directory platform needed CPU-only containers while search embeddings ran on a separate GPU worker. Splitting services kept the public web tier simple and isolated expensive inference. That pattern maps cleanly to custom software development projects where budgets cannot fund a GPU on every environment.

Key Takeaways

• Install and verify the NVIDIA host driver with nvidia-smi before touching Docker GPU flags.
• Configure the NVIDIA Container Toolkit, restart Docker, and smoke-test with an official nvidia/cuda image.
• Match container CUDA tags to the driver API version nvidia-smi reports — newer images on old drivers always fail.
• Use --gpus all or Compose device reservations explicitly; set NVIDIA_VISIBLE_DEVICES on shared nodes.
• Troubleshoot host → toolkit → smoke test → app image in order; change one variable per attempt.
• Treat GPU containers as privileged compute; scan images, monitor thermals, and segregate tenants on production fleets.

People Also Ask

Do I need CUDA installed on the host if I use GPU containers?

You need the NVIDIA kernel driver on the host, not the full CUDA toolkit. The container image carries user-space CUDA libraries for your application. The driver must be new enough to support the CUDA version inside the image. Host-side CUDA SDK installs are only required if you compile CUDA code directly on bare metal outside containers.

Can Docker containers share one GPU?

Yes. Multiple containers can mount the same /dev/nvidia0 device concurrently. CUDA time-slices kernels at the driver level, but memory is not isolated unless you use MIG on supported datacenter GPUs or enforce limits at the orchestrator. Heavy concurrent jobs contend for VRAM and can OOM each other without coordination.

What is the difference between nvidia-docker2 and --gpus?

nvidia-docker2 was the legacy wrapper package that registered a separate Docker runtime binary. Modern Docker Engine integrates GPU support natively through the NVIDIA Container Toolkit and CDI specs. New deployments should install the toolkit and use --gpus; do not install nvidia-docker2 on greenfield hosts unless you maintain ancient Docker versions.

Does rootless Docker support GPUs?

GPU passthrough under rootless Docker requires additional CDI configuration and is less mature than rootful setups. Most production GPU fleets still run Docker or containerd as root on dedicated inference or training nodes. Evaluate rootless containers for security on CPU workloads first; add GPU only where your runtime version documents supported paths.

Deploy GPU workloads with confidence

CUDA and Container GPU Basics are the foundation for every containerised ML pipeline you run in 2026. Driver on the host, toolkit wired into Docker, CUDA tags matched to reality — three steps that prevent most first-day outages. Once the smoke test passes, move upstack into Kubernetes scheduling, model serving, and application integration. If you want help hardening GPU nodes, building CUDA-aware CI pipelines, or connecting inference APIs to a Laravel or WordPress front end, contact us or review testing and optimization services. You can also browse the home page and about page for broader DevOps and web platform work.