Purpose
This article explains how to prepare a VM for GPU-enabled workloads by installing NVIDIA drivers, CUDA, Fabric Manager, and the NVIDIA Container Toolkit.
Overview
Before running GPU workloads in Giotto, the VM must be configured so that:
- the NVIDIA GPU is correctly recognized by the operating system
- CUDA libraries and tools are available
- Fabric Manager is enabled when required by the hardware
- Docker can expose the GPU to containers through the NVIDIA Container Toolkit
This article shows one working installation example based on Ubuntu 22.04 and CUDA 12.5 / NVIDIA driver 555. Other compatible CUDA and driver versions can also work, as long as they are aligned with the GPU, the host OS, and the container/runtime requirements.
When to use this article
Use this article when:
- you are preparing a fresh GPU VM
- you need CUDA available on the host
- you want Docker containers to access the NVIDIA GPU
- you are setting up the base machine before installing Giotto
Important note on versions
The commands below use:
- CUDA 12.5
- NVIDIA driver 555
- Ubuntu 22.04
These are example versions known to work in this setup. Other versions may also be valid. The important point is that:
- the driver is compatible with the GPU
- the CUDA toolkit is compatible with the intended workloads
- the container runtime is configured to expose GPUs correctly
If your environment standardizes on a different CUDA or driver version, you can adapt the commands accordingly.
What this installation does
This setup performs the following actions:
- installs the CUDA repository and toolkit
- installs the NVIDIA GPU driver
- installs and enables NVIDIA Fabric Manager
- updates the shell environment with CUDA paths
- installs the NVIDIA Container Toolkit
- configures Docker to use the NVIDIA runtime
- restarts Docker
- reboots the machine
Example installation script
The following script is one working example for Ubuntu 22.04:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.5.1/local_installers/cuda-repo-ubuntu2204-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-5-local_12.5.1-555.42.06-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-5-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update && sudo apt-get -y install cuda-toolkit-12-5
sudo add-apt-repository ppa:graphics-drivers/ppa --yes
sudo apt update && sudo apt install -y nvidia-driver-555
sudo apt-get install -y nvidia-fabricmanager-555
sudo systemctl enable --now nvidia-fabricmanager
echo 'export PATH=/usr/local/cuda-12.5/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc
# install nvidia container toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# reboot
sudo reboot
Step-by-step explanation
1. Add the CUDA repository and install the toolkit
The first part of the script adds the NVIDIA CUDA repository and installs the CUDA toolkit.
This provides the CUDA compiler tools, libraries, and runtime components required by many GPU-enabled applications.
2. Install the NVIDIA driver
The next part installs the NVIDIA driver.
The driver is what allows the operating system to communicate with the GPU hardware. Without a working driver, CUDA workloads will not run correctly.
3. Install and enable Fabric Manager
Fabric Manager is installed and enabled as a system service.
This is important on some GPU systems, especially where the hardware or topology expects NVIDIA fabric management support.
4. Export CUDA paths
The script appends CUDA paths to .bashrc:
PATHLD_LIBRARY_PATH
This makes the CUDA binaries and libraries available in shell sessions.
5. Install NVIDIA Container Toolkit
The NVIDIA Container Toolkit allows Docker containers to access NVIDIA GPUs.
This is required when workloads are run in containers and need GPU visibility.
6. Configure Docker runtime
The command:
sudo nvidia-ctk runtime configure --runtime=docker
updates Docker so it can use the NVIDIA runtime.
7. Restart Docker and reboot
Docker is restarted so the runtime configuration takes effect, and the machine is rebooted to ensure that the driver and related services are fully initialized.
How to verify the installation
After the machine comes back up, verify the setup step by step.
Verify the driver
Run:
nvidia-smi
Expected output:
Verify CUDA
Run:
nvcc --version
Expected output:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Jun__6_02:18:23_PDT_2024
Cuda compilation tools, release 12.5, V12.5.82
Build cuda_12.5.r12.5/compiler.34385749_0Verify also CUDA runtime
python3 - <<'PY'
import ctypes
ctypes.CDLL("libcudart.so")
print("CUDA runtime OK")
PYExpected output is "CUDA runtime OK".
Verify Docker GPU access
A final useful validation is to check that Docker can see the GPU through the NVIDIA runtime. Run (you may need sudo privileges):
docker info | grep -i runtimeExpected:
Runtimes: io.containerd.runc.v2 nvidia runcThen:
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiThis is the first critical Docker test. You should see the exact same GPUs as on the host.
Common issues
nvidia-smi does not work
This usually indicates a driver installation problem, an incomplete reboot, or a mismatch between the driver and the environment.
CUDA is installed but commands are not found
This usually means the CUDA PATH or LD_LIBRARY_PATH has not been applied correctly in the shell.
Docker cannot access the GPU
This usually indicates that the NVIDIA Container Toolkit was not installed or Docker was not reconfigured and restarted correctly.
Fabric Manager service fails
This may happen if the selected driver and fabric manager versions do not match, or if the hardware does not require the installed setup.
Best practices
Keep driver and CUDA choices consistent
Pick a driver and CUDA version pair that is validated for your environment.
Reboot after installation
A reboot is strongly recommended after driver installation and Docker runtime configuration.
Validate in stages
Check:
- driver visibility with
nvidia-smi - CUDA visibility with
nvcc --version - container GPU access through Docker
Document the chosen version set
If this VM is part of a repeatable deployment process, keep a record of the CUDA and driver versions used.
Comments
0 comments
Article is closed for comments.