Updating netboot image with newer kernel
This text describes the necessary steps to update a Debian netboot image with a more recent kernel (e.g. a kernel from backports or a custom kernel). It is written to be generic as possible, in the hope that it's useful for all Debian users (There's a part below which is specific to the production environment of the Wikimedia Foundation, but it can be skipped/adapted to your deployment)
Backport Buster 5.10 kernel to Buster
The specific example used here is the integration of the backport of the updated 5.10 kernel for Buster into a netboot image of buster (which is based on 4.19).
- Download the kernel you want to integrate. In this case we're using the Linux 5.10 kernel for Buster (source package name linux-5.10), available from http://security.debian.org/pool/updates/main/l/linux-5.10/ (Ofc you can also build a custom kernel and use that)
- Extract the deb of the kernel package you want to use:
ar x $KERNELDEB
tar xf data.tar.xz (or possibly .bz or .gz)
- Now download the netinst image you want to modify. For Buster this is the netboot.tar.gz from http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/
- Unpack the existing initrd image you want to modify
mkdir netboot
sudo unmkinitramfs initrd.gz netboot
Copy the content of lib/modules/$KERNELVERSION from the unpacked kernel deb to netboot/lib/modules (the complete directory including the modules.order and modules.buildin files).
- Now rebuild the initrd:
cd netboot
find . 2>/dev/null | cpio --quiet -H newc -o | gzip > ../initrd-new.gz
It's very likely that the netboot image will also need to be updated with firmware, in particular for PXE boots:
For that, download one of the firmware bundles which are available per Debian release: http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/$DISTRO/current/firmware.tar.gz I.e. for our revised Buster image we need to download http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz It's important to get the right firmware.tar.gz release, as some firmwares are only compatible with specific versions of the Linux kernel.
A Debian initramfs can consist of one or several gzipped CPIO archives, as such we need to convert the firmware tarball to a CPIO archive:
mkdir firmware
tar -C firmware -zxf firmware.tar.gz
pax -x sv4cpio -s'%firmware%/firmware%' -w firmware | gzip -c >firmware.cpio.gz
- Next we need to append this our previously generated initrd:
cat firmware.cpio.gz >> initrd-new.gz
Now that we have an updated initrd, but you'll also need the kernel itself from the kernel deb extracted above: It's in the boot/ directory of the data.tar.xz extracted above and named like vmlinuz-KERNELVERSION-amd64
You need to add this initrd and kernel image in your PXE boot environment.
WMF-specific instructions for this below:
The tftpboot environment is distributed via puppet/volatile. Log into puppetmaster1001 and enter the /var/lib/puppet/volatile/tftpboot/ directory. Make a copy of the existing installer environment and e.g. copy it to e.g. installer-buster510.
Now enter the jessie49-installer/debian-installer/amd64 directory and replace linux with the vmlinuz-KERNELVERSION-amd64 file extracted from the kernel and replace the initrd.gz with the one we rebuilt.
Now force a puppet run on the install servers via Cumin:
sudo cumin A:installserver 'run-puppet-agent'
Updating production point release
The process to update the prodcution point release images has been scripted so follow theses simple instuctions but fall back on the notes above if you hit any errors
- from the puppetmasteres run the the following command to built the debian-installer folder
$ sudo update-netboot-image buster
--2021-03-29 09:20:55-- http://ftp.us.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 37534443 (36M) [application/x-gzip]
Saving to: ‘netboot.tar.gz’
netboot.tar.gz 100%[========================================================================>] 35.79M 31.9MB/s in 1.1s
2021-03-29 09:20:56 (31.9 MB/s) - ‘netboot.tar.gz’ saved [37534443/37534443]
--2021-03-29 09:20:56-- http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz [following]
--2021-03-29 09:20:56-- http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Reusing existing connection to [webproxy.eqiad.wmnet]:8080.
Proxy request sent, awaiting response... 200 OK
Length: 84721761 (81M) [application/x-gzip]
Saving to: ‘firmware.tar.gz’
firmware.tar.gz 100%[========================================================================>] 80.80M 33.0MB/s in 2.5s
2021-03-29 09:20:59 (33.0 MB/s) - ‘firmware.tar.gz’ saved [84721761/84721761]
The updated netboot environment can be found in /tmp/tmp.LeRoc5mJaL/buster-installer, if everyone looks fine, move it to /var/lib/puppet/volatile/tftpboot
and make sure to remove /tmp/tmp.LeRoc5mJaL
as the output indicates the installer has been built into a temporary folder. Before moving this folder to volatile first make a backup of the current installer
$ mv /var/lib/puppet/volatile/tftpboot/buster-installer{,-10.8}
$ mv /tmp/tmp.LeRoc5mJaL/buster-installer /var/lib/puppet/volatile/tftpboot
Once this has completed, run puppet agent on the installservers
sudo cumin A:installserver 'run-puppet-agent'
You should then reimage a server to ensure everything runs as expected