K8s on RPi 4 Pt. 2 – Proven Steps to PXE Boot RPi 4 with Ubuntu

Table of Contents

• Overview
• What is PXE boot?
• Configure PXE Boot
  • Configure PXE Boot Raspberry Pi 4
  • Configure the TFTP Server
• Install Ubuntu Server on Raspberry Pi
  • Copy /boot Directory to rpi-tftpboot NFS folder
  • Touch Up Some of The Files on the NFS Share Drive
  • Copy Ubuntu OS files to rpi-pxe NFS Drive
• Test The PXE Boot Raspberry Pi
• Bumps to Watch Out
• Summary
• References

Overview

Running Raspberry Pi 4 with SD card (a good fast SD card) is fine, if speed and resiliency is not a concern. SD cards always have speed limitation and there are chances SD card maybe corrupted during incidents such as power interruption. I am going to use this to host my websites and data. For this, I need a better solution with proper storage backup for the OS, binaries and data. The best is the solution can provide reusable golden base OS image to deploy multiple Kubernetes nodes. This is where I am looking at PXE boot Raspberry Pi 4 from my Synology storage server.

I am using Ubuntu OS for my Raspberry Pi here. You can refer to the Part 1 for the reason of why I choose Ubuntu.

Note: You can also boot your RPi 4 from USB drive which provides better performance on disk IO. You can find out how to do this from this great post – I’m booting my Raspberry Pi 4 from a USB SSD by Jeff Curling.

What is PXE boot?

The person that can explain this better is no other than Wikipedia. The Preboot eXecution Environment, PXE (most often pronounced as /ˈpɪksiː/ pixie, often called PXE Boot/pixie boot.) specification describes a standardized client–server environment that boots a software assembly, retrieved from a network, on PXE-enabled clients. 

On the client side it requires only a PXE-capable network interface controller (NIC), and uses a small set of industry-standard network protocols such as DHCP and TFTP.

In short, you can boot your Raspberry Pi OS remotely over network from a central server that is configured to support PXE booting.

Configure PXE Boot

You need to install Raspberry Pi OS and configure the PXE boot using the OS. You can then proceed to re-install the Ubuntu OS on your RPi 4. I have not found a simple way to configure PXE boot on Ubuntu OS directly at the moment. Please let me know if you aware of how to do this.

I will not be covering the Raspberry Pi OS installation here. This can be easily done with Raspberry Pi Imager.

Configure PXE Boot Raspberry Pi 4

Once you have the Raspberry Pi OS install on the SD Card, proceed to boot it and ssh into the OS. Do the following command to update the OS and install the rpi-eeprom.

sudo apt-get update -y && sudo apt-get full-upgrade -y && sudo apt install -y rpi-eeprom

Run the following command to get the latest installed EEPROM and make a copy in your local directory inside the RPi 4.

LATEST_PIEEPROM=$(ls -1 -r /lib/firmware/raspberrypi/bootloader/stable/pieeprom*  | head -n1)

cp $LATEST_PIEEPROM pieeprom.bin

Create a EEPROM configuration with the following command. In here it is named as bootconf.txt. The only configuration that you need to change is the TFTP_IP. This is your server where the RPi 4 will be booting from. We will cover this TFTP server configuration setup in the next section.

cat <<EOF | tee bootconf.txt
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
DHCP_FILE_TIMEOUT=30000
TFTP_IP=<tftp server ip>
TFTP_PREFIX=0
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0
EOF

Next you will need to run the command to update this new configuration and write them into the RPi 4 EEPROM. This will complete the PXE Boot configuration at the RPi 4. 

rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt pieeprom.bin

sudo rpi-eeprom-update -d -f ./pieeprom-new.bin

You will need to reboot the OS to allow the changes to be effective. There are one more thing we need to do before we proceed to reboot the OS. 

Let’s execute the following command to find out the RPi serial number. Take down this serial number. We will need to use it for our PXE boot folder at the TFTP server site.

vcgencmd otp_dump | grep 28: | sed s/.*://g

You can now try to reboot the RPi 4 with the SD card unplugged. Connect a monitor to the RPi 4 while you are doing this. If you see messages that RPi 4 is trying to network boot from the configured TFTP IP address, means the configuration is working. The boot will fail at this stage because we have not completed the server site configurations.

Configure the TFTP Server

In here, I am using my existing Synology storage server for the NFS share required for the PXE boot and the necessary TFTP server configuration. 

The first thing we need to do is to create 2 NFS share folders. I named them as rpi-pxe and rpi-tftpboot. rpi-tftpboot will store all the RPi boot loader files and rpi-pxe will store all the RPi OS files. 

You can refer the following for the configurations that I had done. You can just apply the same configurations and permissions for both the share folders or tweak them to your preferences.

• 

• 

• 

Note: TFTP stands for Trivial File Transfer Protocol. Please refers wikipedia for more explanation.

Synology storage server includes the DHCP server which also includes the TFTP server that you can configure easily for this PXE boot purpose. The following are the configurations that I have done.

• 

• 

• 

• 

In order to enable TFTP server on Synology, you need to first enable the DHCP server. By enabling the DHCP server, you need to configure a range of subnet IP addresses. 

In my case I am going to use this subnet IP addresses for my Raspberry Pi nodes. I am using the Synology as my home DNS server, so the Primary DNS is configured to the Synology IP address. In your case, you should configure this to your existing DNS server.

In the Vendor screen, create a new vendor settings with 43 for Raspberry Pi Boot values.

In the PXE screen, click on the checkbox to enable PXE and select the Local TFTP Server option. You can ignore the Boot Loader value. We do not need to place bootcode.bin in the rpi-tftpboot root directory. The PXE will lookup for the bootcode.bin from each of the folder that we will create named after the RPi serial number later on.

Install Ubuntu Server on Raspberry Pi

Download the Ubuntu for Raspberry Pi installer from Ubuntu website. I am using Ubuntu Server 20.04. You can use the Raspberry Pi Imager to burn the OS onto your SD card and boot the Raspberry Pi as usual. 

Refer the Ubuntu Server Raspberry Pi installation guide here.

Copy /boot Directory to rpi-tftpboot NFS folder

Once you have booted up the Ubuntu, ssh into the server to copy the /boot folder onto the NFS share drive we created earlier.

Note: Booting Ubuntu on RPi 4 requires patient. It will take some time for the system to boot completely and ready because of the cloud-init. You really need to wait until it is fully completed as per described here.

ssh into the Ubuntu and install the nfs-common to allow us to mount to the NFS shared folder using NFS client.

sudo apt-get install nfs-common

At the storage server side, we need to create a sub-directory in the NFS share drive for the specific Raspberry Pi node. We will create a folder name after the RPi 4 serial number under the rpi-tftpboot share drive. 

At the end of the day, we will have multiple folders with each corresponding serial numbers at the rpi-tftpboot shared driver. 

For example if the serial number for the specific Raspberry Pi is b153d618. We will then create the folder and mount the NFS shared drive on the local Ubuntu SD card and start copying the /boot directory to the directory named b153d618.

# create the local mount directory
sudo mkdir /media/boot 

# mount the NFS drive
sudo mount <storage server ip>:/volume1/rpi-tftpboot/b153d618 /media/boot

# copy the boot dir
sudo rsync -xa --progress /boot /media/boot/

Touch Up Some of The Files on the NFS Share Drive

Because of the way how Ubuntu boot loading is different from Raspberry Pi OS, we need to adjust a little bit on the boot loading files in the shared drive we had just created. You can run the following command inside the /media/bootdirectory.

# Restructure some of the files location.
cp -a firmware/start4.elf ./start4.elf && \ 
cp -a firmware/bcm2711-rpi-4-b.dtb ./bcm2711-rpi-4-b.dtb \ 
&& cp -a firmware/config.txt ./config.txt \ 
&& cp -a firmware/usercfg.txt ./usercfg.txt \ 
&& cp -a firmware/syscfg.txt ./syscfg.txt \ 
&& cp -a firmware/cmdline.txt ./cmdline.txt \ 
&& cp -a firmware/fixup4.dat ./fixup4.dat

# make sure the ownership is root:root
sudo chown root:root start4.elf bcm2711-rpi-4-b.dtb config.txt usercfg.txt syscfg.txt cmdline.txt fixup4.dat

We also need to uncompress the vmlinuz kernel file.

# Double check the the compressed file is with offset `0000000`
sudo od -A d -t x1 vmlinuz-5.4.0-1045-raspi | grep '1f 8b 08 00'

0000000 1f 8b 08 00 00 00 00 00 02 03 ec 5c 0d 74 14 d7

# Uncompressed the `vmlinuz-5.4.0-1045-raspi`
zcat vmlinuz-5.4.0-1045-raspi > vmlinux-5.4.0-1045-raspi

Modify the config.txt with the following changes. Make sure the kernel value is configured with the correct name and version number from the above.

[pi4]                            
# kernel=uboot_rpi_4.bin
max_framebuffers=2
                                 
[pi2]                            
# kernel=uboot_rpi_2.bin
                                 
[pi3]                            
# kernel=uboot_rpi_3.bin
                                 
[all]                            
arm_64bit=1                      
device_tree_address=0x03000000
kernel=vmlinux-5.4.0-1045-raspi  
initramfs initrd.img-5.4.0-1045-raspi followkernel

Change the cmdline.txt with the following details. Note the rpi-pxe/kube1 shared folder, this will be the shared folder that we are going to create in the next step.

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=<storage server ip>:/volume1/rpi-pxe/kube1 elevator=deadline rootwait fixrtc rw

Copy Ubuntu OS files to rpi-pxe NFS Drive

We need to create a folder named kube1 in the NFS share folder rpi-pxe. This could be any name that make sense for you, it does not have to be kube1 as in my case. Make sure this folder name is the same as configured in cmdline.txt above.

Copy the complete Ubuntu OS filesystems to this newly created share folder.

# create the local mount directory
sudo mkdir /media/root 

# mount the NFS drive
sudo mount <storage server ip>:/volume1/rpi-pxe/kube1 /media/root

# copy the root dir. Will take some time to complete. Grab a coffee!
sudo rsync -xa --progress / /media/root/

Update the fstab file in the share folder just copied over for kube1. 

Updated 20-03-2023: Added the /boot mount entry required in fstab

<storage server ip>:/volume1/rpi-pxe/kube1   /       nfs     defaults,rw,nolock   0  0
tmpfs   /tmp    tmpfs   defaults    0   0
tmpfs   /var/tmp    tmpfs   defaults    0   0
tmpfs   /var/run    tmpfs   defaults    0   0
<storage server ip>:/volume1/rpi-tftproot/bc6ea0c3             /boot  nfs    defaults,rw,nolock        0       0

Note: I think the entry for <storage server ip>:/volume1/rpi-pxe/kube1 is not required in the fstab since we have already define this in the cmdline.txt earlier.

Test The PXE Boot Raspberry Pi

Once you have done, shutdown the Raspberry Pi and power off. Remove the SD card and boot the Raspberry Pi over the network.

At this moment, it is best that you have monitor connected to the Raspberry Pi so that you will be able to know what is going on with the booting process. This also give your visibility of what is the error message on screen is something is not right.

Once everything is fine. You can repeat the same process for all the other Raspberry Pi nodes that you intended to boot from network drive. 

I basically just make a copy of the first node for each folder under rpi-pxe and rpi-tftpboot. I use that as a golden copy for all the sub-sequence nodes configuration.

Bumps to Watch Out

Mistake happens. Some tips to make sure you would not repeat the mistakes that I have gone through.

• Copy the /boot and the OS filesystem using -a parameter to maintain all the file permissions and attributes. Use -x to avoid crossing filesystem boundaries while copying files.
• Make sure you have copied all the required boot loader files into the root directory of the rpi-tftpboot/<serial> folder, as mentioned in this section.
• Ensure the NFS server IP address is correct and the mount points for each rpi-pxe and rpi-tftpboot is correct for each node. When you have more nodes coming in, mistake tends to happen. I once boot node A into node B boot loader. Make sure you double check the Raspberry Pi serial number is correct. Label your Raspberry Pi with IP address, serial number and node name if possible.
• If you intended to use this PXE boot approach for Kubernetes (which I will be covering in the next post), make sure you have tried the first node with completed Kubernetes installation. There will be additional network and node configuration that we need to do for Kubernetes installation. Once you have a stable OS with Kubernetes installation, then it will be more meaningful to create the golden copies for the boot loader and OS filesystems. 

Summary

This concludes the steps required to configure PXE boot for Ubuntu on Raspberry Pi 4. In the next post, I am going to cover Kubernetes installation on Raspberry Pi 4.

References

• PXE Boot Diskless Raspberry Pi 4 With Ubuntu, Ubiquiti and Synology (2): Config TFTP and NFS mounts
• PXE Booting Raspberry PI 4 with Synology and Ubiquiti Edgerouter