Nvidia GPU Acceleration for Docker Containers

Installing Nvidia GPU acceleration for docker containers on Ubuntu 22.04 Server. This does not install any X11 components, only the headless packages required for running AI/ML applications inside docker containers.

This method also works on Ubuntu 22.04 Desktop, but under the condition that the Nvidia GPU is not used for graphics. For example, X11 uses iGPU (Intel/AMD) for HDMI video out and the Nvidia GPU is exclusively used for GPU accelerated computing.

Test Hardware

OS	Hardware	Display
Ubuntu 22.04 Server (headless)	Nvidia RTX 3060 12GB	No display connected
Ubuntu 22.04 Desktop	AMD 3600G CPU, Nvidia RTX 3060 12GB	Display connected to iGPU HDMI output

Step 1: Uninstall the Open Source `Novueau` Driver

The open source Novueau Nvidia driver kernel module is incompatible and must first be blacklisted and removed.

# Check if nouveau is auto loaded
lsmod | grep nouveau

# If auto loaded, blacklist the novueau kernel module, rebuild initramfs and reboot

cat > /etc/modprobe.d/blacklist-nvidia-nouveau.conf
blacklist nouveau
options nouveau modeset=0

update-initramfs -u
reboot

# Ensure that nouveau module is not auto loaded anymore
lsmod | grep nouveau

Step 2: Install the Nvidia Drivers

# Install nvidia CUDA drivers and utils for headless servers
apt-get install nvidia-headless-535 nvidia-utils-535

Step 3: Install the Nvidia Container Toolkit

# Enable nvidia-container-toolkit repositiory

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
    gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    tee /etc/apt/sources.list.d/nvidia-container-toolkit.list 

apt update

# Install nvidia container toolkit and enable it
apt-get install nvidia-container-toolkit

# See /etc/docker/daemon.json after running this command
nvidia-ctk runtime configure --runtime=docker

systemctl restart docker

Step 4: Verify GPU Availability Inside Docker

# Verify that the GPU is availble inside docker
root@catin:~# docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Tue Oct  3 13:28:51 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        Off | 00000000:01:00.0 Off |                  N/A |
|  0%   51C    P8              17W / 170W |      2MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

# Checking GPU usage using gpustat
# See: https://github.com/wookayin/gpustat

#TODO how to use Nvidia docker containers to run GPU accelerated code.