Building a custom Red Hat Enterprise Linux kernel for NVIDIA DGX Spark

The NVIDIA DGX Spark platform, powered by the NVIDIA GB10 Grace Blackwell Superchip, requires a custom kernel build to enable compatibility with Red Hat Enterprise Linux (RHEL) 10. This guide walks through building the latest RHEL 10 development kernel and the corresponding NVIDIA graphics processing unit (GPU) driver from source. By the end, you'll have an NVIDIA DGX Spark Founder's Edition running CentOS Stream 10 with a custom RHEL kernel, NVIDIA drivers, and CUDA toolkit installed.

Prerequisites

To build the kernel and GPU driver, you need:

Build environment:

• An ARM64 (aarch64 architecture) system running RHEL 10, or
• An NVIDIA DGX Spark Founder's Edition with CentOS Stream 10 installed

Note that released RHEL 10.y versions do not currently install on the GB10 platform, which is why CentOS Stream 10 is an alternative for building directly on the target hardware. Any ARM64 system running RHEL 10 will work as a build host, but building on the GB10 itself can simplify testing and deployment.

Baseline knowledge:

• Familiarity with RPM package builds and spec files
• Working knowledge of kernel compilation workflows
• Comfort with the command line and basic git operations

Tools and dependencies: The build system requires the following dependencies :

If using RHEL 10 as the build system, enable the codeready-builder repository :

sudo subscription-manager repos --enable codeready-builder-for-rhel-10-aarch64-rpms   

Install the base required packages for kernel building :

sudo dnf -y install git make

Obtain the kernel source

Starting in your home directory, clone the RHEL 10 kernel repository customized for the GB10 platform:

git clone https://gitlab.com/redhat/edge/kernel/nvidia-gb10.git
cd nvidia-gb10

Create an empty localversion file to prevent the build system from appending unwanted version suffixes:

touch localversion

This step matters because the kernel build system typically appends git commit information to the kernel version string. For a clean, reproducible build that matches the RHEL kernel packaging expectations, you want the version string to reflect only the official kernel version without local modifications.

Install build dependencies

The kernel source includes a target that identifies and installs all required build dependencies:

sudo dnf -y install $(make dist-get-buildreqs | grep "Missing dependencies:" | cut -d":" -f2-)

This command queries the kernel build system for missing dependencies, extracts the package names, and installs them in one pass. Depending on your system's current state, this may install compilers, build tools, headers, and other libraries needed for kernel compilation.

Build the kernel

Run the kernel build:

make dist-rpms

This target builds binary RPM packages using the kernel's built-in packaging infrastructure. The process compiles the kernel, modules, and supporting files, then packages them into distributable RPMs. Depending on your hardware, this step can take considerable time—expect anywhere from 30 minutes to several hours.

When the build completes, the kernel RPMs are located in:

~/nvidia-gb10/redhat/rpm/RPMS/aarch64/kernel-64k-*

The kernel-64k variant provides better general performance on an ARM64 system in exchange for increased memory consumption. You should validate that your workload performs acceptably in this configuration.

Validation checkpoint: Verify that the kernel RPMs were created:

ls -lh ~/nvidia-gb10/redhat/rpm/RPMS/aarch64/kernel-64k-*.rpm

You should see several RPM files, including at minimum kernel-64k, kernel-64k-core, kernel-64k-modules, and kernel-64k-devel.

Build the NVIDIA GPU driver

The GB10 platform requires NVIDIA's open source GPU kernel modules. These modules must be built against the same kernel version you just compiled.

Install kernel development headers

Install the kernel-64k-devel package from your build output:

sudo dnf install redhat/rpm/RPMS/aarch64/kernel-64k-devel-6*.rpm

This package provides the kernel headers and build configuration necessary to compile out-of-tree modules like the NVIDIA driver.

Validation checkpoint: Verify the kernel headers are in place:

ls /usr/src/kernels/*gb10*

You should see a directory matching your kernel version with .gb10 in the name.

Create the GPU driver spec file

Create a file in your home directory named kmod-nvidia-open.spec with the following content:

%define driver_version 595.58.03
%define kversion %(ls /usr/src/kernels/ | grep '\.gb10' | sort -V | tail -n 1)
%define kversion_bare %(echo "%{kversion}" | sed -e 's/+64k$//' -e 's/\\.el[0-9]*[^.]*\\.[^.]*$//' -e 's/-/./g')
%global debug_package %{nil}

Name:           kmod-nvidia-open
Version:        %{kversion_bare}
Release:        1%{?dist}
Summary:        NVIDIA open source GPU kernel modules
License:        MIT and GPLv2
URL:            https://github.com/NVIDIA/open-gpu-kernel-modules
Source0:        https://github.com/NVIDIA/open-gpu-kernel-modules/archive/refs/tags/%{driver_version}.tar.gz#/open-gpu-kernel-modules-%{driver_version}.tar.gz

BuildRequires:  kernel-devel-uname-r = %{kversion}
BuildRequires:  make
BuildRequires:  gcc
Requires:       kernel-uname-r = %{kversion}

Provides:       kmod-nvidia-open = 3:%{driver_version}
Provides:       nvidia-kmod = 3:%{driver_version}
Conflicts:      kmod-nvidia-open-dkms
Conflicts:      kmod-nvidia-latest-dkms

%description
This package provides the NVIDIA open source GPU kernel modules built from
the official NVIDIA open-gpu-kernel-modules repository.

%prep
%setup -q -n open-gpu-kernel-modules-%{driver_version}

%build
unset LDFLAGS
make modules %{?_smp_mflags} SYSSRC=/usr/src/kernels/%{kversion}

%install
mkdir -p %{buildroot}/lib/modules/%{kversion}/extra/nvidia
install -m 644 kernel-open/nvidia-peermem.ko %{buildroot}/lib/modules/%{kversion}/extra/nvidia/
install -m 644 kernel-open/nvidia-modeset.ko %{buildroot}/lib/modules/%{kversion}/extra/nvidia/
install -m 644 kernel-open/nvidia-drm.ko %{buildroot}/lib/modules/%{kversion}/extra/nvidia/
install -m 644 kernel-open/nvidia-uvm.ko %{buildroot}/lib/modules/%{kversion}/extra/nvidia/
install -m 644 kernel-open/nvidia.ko %{buildroot}/lib/modules/%{kversion}/extra/nvidia/

%post
/sbin/depmod -a %{kversion} || :

%postun
/sbin/depmod -a %{kversion} || :

%files
/lib/modules/%{kversion}/extra/nvidia/nvidia-peermem.ko
/lib/modules/%{kversion}/extra/nvidia/nvidia-modeset.ko
/lib/modules/%{kversion}/extra/nvidia/nvidia-drm.ko
/lib/modules/%{kversion}/extra/nvidia/nvidia-uvm.ko
/lib/modules/%{kversion}/extra/nvidia/nvidia.ko

%changelog

This spec file automates several critical steps:

• Driver version pinning: The driver_version macro locks the build to NVIDIA driver version 595.58.03, which is validated for the GB10 platform.
• Kernel version detection: The kversion macro dynamically discovers the installed kernel version by scanning /usr/src/kernels/ for directories matching .gb10, ensuring the driver builds against your custom kernel.
• Version normalization: The kversion_bare macro strips suffixes and reformats the kernel version into a clean package version string.
• Module installation: The spec installs five kernel modules (nvidia.ko, nvidia-uvm.ko, nvidia-drm.ko, nvidia-modeset.ko, nvidia-peermem.ko) into the kernel's extra directory and runs depmod to update module dependencies.

The unset LDFLAGS in the %build section prevents system linker flags from interfering with the NVIDIA driver's build process.

Build the GPU driver RPM

Create the RPM build directory structure and run the build:

cd ~
mkdir -p ~/rpmbuild/SOURCES
rpmbuild -bb --define "_disable_source_fetch 0" kmod-nvidia-open.spec

The mkdir command creates the directory where rpmbuild will download source files. The _disable_source_fetch 0 definition tells rpmbuild to download the NVIDIA driver source tarball directly from GitHub during the build, placing it in ~/rpmbuild/SOURCES/.

The build compiles the NVIDIA kernel modules against your custom kernel and packages them into an RPM. This process is faster than the kernel build, typically completing in 5-15 minutes depending on your hardware.

Validation checkpoint: Verify the GPU driver RPM was created:

ls -lh ~/rpmbuild/RPMS/aarch64/kmod-nvidia-open-*.rpm

You should see a single RPM file with a version matching your kernel build.

Installing the custom kernel and driver

Once you have both the kernel and GPU driver RPMs, you can install them on the target GB10 system:

cd ~/nvidia-gb10/redhat/rpm/RPMS/aarch64/
sudo dnf install \
    kernel-64k-6.12* \
    kernel-64k-core-6.12* \
    kernel-64k-modules-6.12* \
    kernel-64k-modules-core-6.12* \
    kernel-64k-modules-extra-6.12*
sudo dnf install ~/rpmbuild/RPMS/aarch64/kmod-nvidia-open-*.rpm

After installation, reboot the system and select the new kernel from the GRUB menu. Once booted, verify the kernel version:

uname -r

Confirm the NVIDIA modules loaded successfully:

lsmod | grep nvidia

You should see the NVIDIA modules listed: nvidia, nvidia_drm and nvidia_modeset.

Installing NVIDIA® CUDA®

Once the kernel and GPU drivers are installed, you may install NVIDIA CUDA to be able to use CUDA workloads on your device The NVIDIA CUDA packages require dependencies that are available via the Extra Packages for Enterprise Linux (EPEL) repo.

You must enable the EPEL repo before you can successfully install NVIDIA CUDA. Follow the instructions at: How to install EPEL on RHEL and CentOS Stream in order to enable the EPEL repo.

Then proceed with the following steps to install NVIDIA CUDA:

sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel10/sbsa/cuda-rhel10.repo
sudo dnf clean all
sudo dnf -y install cuda

If the installation of CUDA fails, verify that you have successfully installed and enabled the EPEL repo.

Reboot after installing CUDA to ensure the NVIDIA systemd hooks are triggered correctly.

Updating the kernel

Because the kernel repository is regularly rebased, standard git pull operations will fail. To update your local repository and rebuild:

cd nvidia-gb10
git fetch origin
git reset --hard origin/main

This discards any local changes and resets your working directory to match the upstream main branch. After resetting, repeat the kernel build and GPU driver build steps to produce updated RPMs.

Note that this approach intentionally discards uncommitted changes. If you have local modifications you want to preserve, create a separate branch or stash your changes before running git reset --hard.

What's next

With a custom kernel and GPU driver installed, you can:

• Test GPU workloads using CUDA or other NVIDIA frameworks
• Integrate the custom kernel into automated image builds or deployment pipelines
• Monitor kernel updates from the upstream repository and rebuild as needed to pick up fixes and new features

For production deployments, consider automating the kernel and driver builds using continuous integration/continuous delivery (CI/CD) pipelines to ensure consistent, reproducible artifacts across your fleet.