Raspberry Pi NFS Root

Raspberry Pi can be diskless booted using multiple methods.

This approach uses an SD card for loading the Linux kernel, the /boot/firmware partition is loaded from SD card. Then /boot/firmware/cmdline.txt sets the kernel to boot from NFS rather than use the local disk. Reverting back to SD Card boot is trivial as only the cmdline.txt file needs to be changed.

Advantages

1. Only external dependency is the NFS server. DHCP, bootp, tftp servers are not required.
2. It's trivial to revert the RPi from NFS root to local boot by just changing the cmdline.txt file.

Installation

Environment Setup

This setup uses RPi model: 4B+ on Debian Bookworm.

root@rpi:~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

Step 0: Clean Install RPi on SD Card

Follow the official documentation and install Raspbian, Debian, Ubuntu or the Linux flavor of your choice. Use a suitably size SD card and perform local installation of the OS to the SD card.

Step 1: Setup NFS Server

Follow the alpine-nfs documentation to setup an NFS server in the network. The nfs server root must be 10GB or greater and should export /export/rpi as the NFS mount point which will be exported from the server.

Step 2: Transfer SD Card Root to NFS

In the RPi, mount the nfs export locally and copy/rsync the root file system files from SD card to the NFS mount point.

rpi:~# mount -t nfs 10.31.1.231:/export/rpi /mnt

# Run "mount" and confirm that NFS export is mounted on /mnt 

# Run rsync multiple times to ensure a clean copy of the rootfs if copied to NFS 
rpi:~# rsync -ax           / /mnt/
rpi:~# rsync -axv --delete / /mnt/
rpi:~# rsync -axv --delete / /mnt/

# While rsync is running, on alpine-nfs-server watch the progress 
alpine-nfs-server:~# watch -n1 du -sh /export/rpi

Step 3: Change Kernel Command Line

In the RPi, backup the default cmdline.txt file it is required to disable diskless boot and revert to booting from local SD card.

# Backup original cmdline.txt
rpi:~# cp /boot/firmware/cmdline.txt /boot/firmware/cmdline.txt-local

Modify set the root, nfsroot and ip parameters in the cmdline.txt file.

Command line: root=/dev/nfs nfsroot=10.13.1.231:/export/rpi,tcp,vers=3 ip=10.13.1.252::10.13.1.1:255.255.255.0:rpi:eth0:off

rpi:~# cat /boot/firmware/cmdline.txt   # Note: the below must be on a single line, not multiple lines

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=10.13.1.231:/export/rpi,tcp,vers=3 ip=10.13.1.252::10.13.1.1:255.255.255.0:rpi:eth0:off rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=IN

Step 4: Copy /boot/firmware to NFS Server

In Raspberry, the /boot/firmware folder is on a separate vfat filesystem that can be read by the RPi boot loader. The rsync command in Step 2 will not have copied this folder to the NFS sever. Copy it manually.

# Copy /boot/firmware to NFS
cp -a /boot/firmware/* /mnt/boot/firmware

Step 5: Reboot and Verify

Reboot the RPi and run the mount command to verify that the rootfs is indeed on NFS.

root@rpi:~ # mount | grep nfs
10.13.1.231:/export/rpi on / type nfs (rw,noatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,port=2049,timeo=600,retrans=3,sec=sys,local_lock=all,addr=10.13.1.231)

Step 6: Performance Testing

RPi will provide between 80MB/s to 100MB/s of read/write throughput. This is almost saturating the wired 1Gb/s NIC.

root@rpi:/tmp# time dd if=/dev/urandom bs=1M count=1000 of=random
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 11.8835 s, 88.2 MB/s

real    0m11.894s
user    0m0.005s
sys     0m7.588s

root@rpi:/tmp# time dd if=/dev/zero bs=1M count=1000 of=zeroes
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 9.94405 s, 105 MB/s

real    0m9.951s
user    0m0.000s
sys     0m2.150s

# Clean up
root@rpi:/tmp# rm random zeroes

Step 7: Reverting to SD Card Boot

To revert NFS root and mount the SD Card again as the rootfs, just revert the /boot/firmware/cmdline.txt file back to the original one.

# Boot to local root fs on SD Card
root@rpi:~# cat /boot/firmware/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=1d3aa4bc-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=IN

Tips

1. Save the SD Card root settings as cmdline.txt-local and NFS root settings as cmdline.txt-nfs. Switching between local and NFS root is as simple as copying the required file as cmdline.txt.
2. To improve NFS read/write performance, ensure Proxmox server hosting alpine-nfs-server VM has zpool sync disabled (zfs set sync=disabled rpool) in the ZFS pool hosting the NFS server VM. See proxmox-best-practices, ZFS zpool Flags.
3. In alpine-nfs-server, use a separate /export volume that will host the RPi NFS root. Do not combine it with the alpine root fs. This will provide easier data management.

Files

SD Card Boot cmdline

root@rpi:~# cat /boot/firmware/cmdline.txt-local
console=serial0,115200 console=tty1 root=PARTUUID=1d3aa4bc-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=IN

NFS Boot cmdline

root@rpi:~# cat /boot/firmware/cmdline.txt-nfs
console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=10.13.1.231:/export/rpi,tcp,vers=3 ip=10.13.1.252::10.13.1.1:255.255.255.0:rpi:eth0:off rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=IN