Written by Sam McGeown on 9/3/2020 · Read in about 6 min (1090 words)
Published under Cloud Native

I love Raspbery Pis - I have done since they first released them, all the way up to the present iteration, the Raspberry Pi 4. They’re phenominal little bits of kit, endlessly hackable and because of their really low price, they open up computing to a huge number of people who otherwise wouldn’t get the opportunity.

One of the irritating things, though, about running Raspberry Pis is that they typically boot from an SD card. SD cards are notorious for having high attrition rates, especially if you’re reading/writing to them regularly. Running a Kubernetes cluster on my Raspberry Pis is especially hard on the SD card because you’re not only running the system’s read and writes, but you’re also running containers and their I/O on the card.

PXE booting the Raspberry Pis will allow me to back the OS and containers with my Synology NAS, and to forget the headache of SD card fatigue! There are three steps to configuring PXE boot for my Pis:

  1. Configure the DHCP Server
  2. Configure the NFS and TFTP Server
  3. Configure the Raspberry Pi

Configure DHCP Server

In order for the Raspberry Pi to know where to boot from, we need too configure the DHCP server with some specific options. In my home network DHCP is running from my Ubiquiti EdgeRouter Pro 8. DHCP Option 43 is used to exchange “vendor-specific information” - in this case we need to tell the Raspbery Pi that we have a PXE boot available for it. On my network I’ve configured a DHCP server for RPi called VLAN020.

1# Define option 43
2set service dhcp-server global-parameters "option option-43 code 43 = string;"
3# Set option 43 to "Raspberry Pi Boot " - the spaces are to fix a legacy bug with the old boot controller
4set service dhcp-server shared-network-name VLAN020 subnet subnet-parameters "option option-43 "Raspberry Pi Boot ";"
5# Configure the bootfile-server and tftp-server-name
6set service dhcp-server shared-network-name VLAN020 subnet bootfile-server
7set service dhcp-server shared-network-name VLAN020 subnet tftp-server-name

In the WebUI, it looks like this:

EdgeRouter DHCP Options
EdgeRouter DHCP Options

Configure NFS and TFTP Server

I’ve configured two shared folders on my Synology NAS - rpi-pxe and rpi-tftpboot.

rpi-pxe is shared as an NFS export, with read/write permissions to mount from my Raspbery Pi subnet. Since we’re using this as a Linux boot device, we must not enable root squash (No mapping) and it should be synchronous.

Export NFS Options
Export NFS Options

Inside the rpi-pxe share I have created a new folder for each of my Pis and used chmod 777 to enable full permissions on the share. Each Pi will copy it’s root file system into it’s respective folder, and boot from there.

RPi root folders
RPi root folders

rpi-tftproot is shared as an NFS share in the same way, but also as the TFTP server using the File Services settings on the Synology. The folders and content required will be created later on.

TFTP Share
TFTP Share

Configure Raspberry Pi Boot

Now that the infrastucture is in place to PXE boot the Pis, it’s time to configure the Pis themselves. I’m assuming you’ve got a Pi already running and configured - this can be a fresh install of Rasbian or one that’s been running a while - it doesn’t much matter. I performed these steps on all of my Pis, from the SD booted install.

Several of the steps need to use the Pi’s serial number, so make a note of the output:

1# Get the Serial
2vcgencmd otp_dump | grep 28: | sed s/.*://g
3# output was: c42356f3

Configure EEPROM

Firstly, we need to configure the EEPROM to boot from the network - by default it will only boot from the SD card.

 1# Update
 2sudo apt-get update
 3# Upgrade
 4sudo apt-get full-upgrade
 5# Install the latest EEPROM binaries
 6sudo apt install rpi-eeprom
 7# Copy the latest EEPROM binary
 8cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2020-01-17.bin pieeprom.bin
 9# Export the default settings to a file
10rpi-eeprom-config pieeprom.bin > bootconf.txt
11# Edit the settings file and change the BOOT_ORDER to 0x21
12vim bootconf.txt
Configure the boot settings
Configure the boot settings
1# Create a new EEPROM binary with the settings embedded
2rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt pieeprom.bin
3# Install the new EEPROM binary
4sudo rpi-eeprom-update -d -f ./pieeprom-new.bin
5# Reboot
6sudo reboot

Copy Root to NFS server

Once the Pi has rebooted we need to copy the root partition to the NFS server. I created a directory in the rpi-pxe share for the Pi rpi4-01 and ensured the write permissions were 777. You might need to install rsync using sudo apt-get install rsync.

1mkdir -p /nfs/rpi4-01
2sudo mount /nfs/rpi4-01
3# Copy the root OS partition to the NFS share, excluding the NFS mount
4sudo rsync -xa --progress --exclude /nfs / /nfs/rpi4-01/

Once the copy has completed (grab a coffee, it’ll take a while) edit the /etc/fstab in the NFS share (so /nfs/rpi4-01/etc/fstab for me) and remove the existing configuration (only the proc line should be left). Mount the TFTP boot share as the /boot partition - the fstab file should look like this:

1proc            /proc           proc    defaults          0       0
2192.168.4.10:/volume1/rpi-tftpboot/c42356f3 /boot nfs defaults,vers=3,proto=tcp 0 0

Copy Boot files to TFTP server

To create a bootable TFTP folder, we need to copy the contents of /boot to the TFTP server. To do this I simply mounted the rpi-tftpboot folder as an NFS share in my Pi. The Raspberry Pi 4 bootloader will look for bootable files in a folder corresponding to its serial in the TFTP root directory. In the example below I am creating a folder based on the serial and copying the boot files there. This allows a cmdline.txt for each Pi, which lets me specify where the root of the OS is. It also means that I can update the boot files for each Pi individually.

1sudo mount /nfs/rpi-tftpboot
2mkdir -p /nfs/rpi-tftpboot/c42356f3
3sudo cp -r /boot/* /nfs/pxe-tftpboot/c42356f3

Next edit the cmdline.txt file in the new TFTP boot directory and replace the root partition with the NFS partition.

1console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=,vers=3 rw ip=dhcp elevator=deadline rootwait

Boot your Pi!

All being well, you can now shutdown the Pi, remove the SD card and boot via PXE!

Both root '/' and boot '/boot' are mount NFS
Both root '/' and boot '/boot' are mount NFS

If you encounter problems booting, double check the steps have been completed correctly:

  • Validate the EEPROM configuration (you’ll have to boot the Pi with the SD card again) using vcgencmd bootloader_config and make sure the BOOT_ORDER is set to 0x21
  • Watch the DHCP logs on the EdgeRouter by logging to file
  • View the TFTP logs on the Synology using tail -f /var/log/opentftp.log
Share this post