This text describes the necessary steps to update a Debian netboot image with a more recent kernel (e.g. a kernel from backports or a custom kernel). It is written to be generic as possible, in the hope that it's useful for all Debian users (There's a part below which is specific to the production environment of the Wikimedia Foundation, but it can be skipped/adapted to your deployment)

Backport Buster 5.10 kernel to Buster

The specific example used here is the integration of the backport of the updated 5.10 kernel for Buster into a netboot image of buster (which is based on 4.19).

Download the kernel you want to integrate. In this case we're using the Linux 5.10 kernel for Buster (source package name linux-5.10), available from http://security.debian.org/pool/updates/main/l/linux-5.10/ (Ofc you can also build a custom kernel and use that)

Extract the deb of the kernel package you want to use:

ar x $KERNELDEB
tar xf data.tar.xz (or possibly .bz or .gz)

Now download the netinst image you want to modify. For Buster this is the netboot.tar.gz from http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/

Unpack the existing initrd image you want to modify

mkdir netboot
sudo unmkinitramfs initrd.gz netboot

Copy the content of lib/modules/$KERNELVERSION from the unpacked kernel deb to netboot/lib/modules (the complete directory including the modules.order and modules.buildin files).

Now rebuild the initrd:

cd netboot
find . 2>/dev/null | cpio --quiet -H newc -o | gzip > ../initrd-new.gz

It's very likely that the netboot image will also need to be updated with firmware, in particular for PXE boots:

For that, download one of the firmware bundles which are available per Debian release: http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/$DISTRO/current/firmware.tar.gz I.e. for our revised Buster image we need to download http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz It's important to get the right firmware.tar.gz release, as some firmwares are only compatible with specific versions of the Linux kernel.

A Debian initramfs can consist of one or several gzipped CPIO archives, as such we need to convert the firmware tarball to a CPIO archive:

mkdir firmware
tar -C firmware -zxf firmware.tar.gz
pax -x sv4cpio -s'%firmware%/firmware%' -w firmware | gzip -c >firmware.cpio.gz

Next we need to append this our previously generated initrd:

cat firmware.cpio.gz >> initrd-new.gz

Now that we have an updated initrd, but you'll also need the kernel itself from the kernel deb extracted above: It's in the boot/ directory of the data.tar.xz extracted above and named like vmlinuz-KERNELVERSION-amd64

You need to add this initrd and kernel image in your PXE boot environment.

WMF-specific instructions for this below:

The tftpboot environment is distributed via puppet/volatile. Log into puppetmaster1001 and enter the /var/lib/puppet/volatile/tftpboot/ directory. Make a copy of the existing installer environment and e.g. copy it to e.g. installer-buster510.

Now enter the jessie49-installer/debian-installer/amd64 directory and replace linux with the vmlinuz-KERNELVERSION-amd64 file extracted from the kernel and replace the initrd.gz with the one we rebuilt.

Now force a puppet run on the install servers via Cumin:

sudo cumin A:installserver 'run-puppet-agent'

Updating production point release

The process to update the prodcution point release images has been scripted so follow theses simple instuctions but fall back on the notes above if you hit any errors

from the puppetmasteres run the the following command to built the debian-installer folder

$ sudo update-netboot-image buster                                                                          
--2021-03-29 09:20:55--  http://ftp.us.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 37534443 (36M) [application/x-gzip]
Saving to: ‘netboot.tar.gz’

netboot.tar.gz                        100%[========================================================================>]  35.79M  31.9MB/s    in 1.1s    

2021-03-29 09:20:56 (31.9 MB/s) - ‘netboot.tar.gz’ saved [37534443/37534443]

--2021-03-29 09:20:56--  http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz [following]
--2021-03-29 09:20:56--  http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Reusing existing connection to [webproxy.eqiad.wmnet]:8080.
Proxy request sent, awaiting response... 200 OK
Length: 84721761 (81M) [application/x-gzip]
Saving to: ‘firmware.tar.gz’

firmware.tar.gz                       100%[========================================================================>]  80.80M  33.0MB/s    in 2.5s    

2021-03-29 09:20:59 (33.0 MB/s) - ‘firmware.tar.gz’ saved [84721761/84721761]

The updated netboot environment can be found in /tmp/tmp.LeRoc5mJaL/buster-installer, if everyone looks fine, move it to /var/lib/puppet/volatile/tftpboot
and make sure to remove /tmp/tmp.LeRoc5mJaL

as the output indicates the installer has been built into a temporary folder. Before moving this folder to volatile first make a backup of the current installer

$ mv /var/lib/puppet/volatile/tftpboot/buster-installer{,-10.8}
$ mv /tmp/tmp.LeRoc5mJaL/buster-installer /var/lib/puppet/volatile/tftpboot

Once this has completed you should reimage a server to ensure everything runs as expected