Raspberry Pi OS • Pi5-16GB performance issue with NVME data disk

My 16G Pi5 showed up, and I've been putting it to use. I replaced my 8G Pi5 workstation with the 16G Pi, and tried sdm, which some of you may know is my favorite workload.

TLDR; I'm seeing a significant performance issue in this real-world scenario when using the NVME as a data drive. I'll cover the test systems and then the results.

Test Systems

• Pi5-8GB boots from a Kingston A400 connected via a Startech USB SATA adapter. Data disk is a Kingston NV3 500GB NVME drive on a Geekworm X1001
• Pi5-16GB boots from a Kingston A400 connected via a Startech USB SATA adapter. Data disk is a Kingston NV3 500GB NVME drive on a GeeekPi N04

I've had very good luck with both of these hats with the Kingston drives on other Pi5s.

Both systems:

• Have identical /boot/firmware/config.txt
• Have identical /boot/firmware/cmdline.txt except, of course, for PARTUUID
• Have Active Coolers on them
• Were built using my "standard" sdm-built configuration (IOW essentially identical, produced from a fixed configuration)
• Are fully updated
  
  Raw NVME drive performance
  

  Code:

  #Pi5-8GBhdparm -t /dev/nvme0n1/dev/nvme0n1: Timing buffered disk reads: 2324 MB in  3.00 seconds = 774.02 MB/sectime {  rm -f 2024-11-19-raspios-bookworm-arm64-lite.img ; cp /ssdy/work/vanilla/2024-11-19-raspios-bookworm-arm64-lite.img . ; }real    0m2.513suser    0m0.008ssys     0m1.036s

  Code:

  #Pi5-16GBhdparm -t /dev/nvme0n1/dev/nvme0n1: Timing buffered disk reads: 2638 MB in  3.00 seconds = 878.82 MB/sectime { rm -f 2024-11-19-raspios-bookworm-arm64-lite.img ; cp /ssdy/work/vanilla/2024-11-19-raspios-bookworm-arm64-lite.img . ; }real    0m1.738suser    0m0.000ssys     0m1.700s

  Code:

  #Pi5-8GB with N04 hat (for comparison)hdparm -t /dev/nvme0n1/dev/nvme0n1: Timing buffered disk reads: 2472 MB in  3.00 seconds = 823.95 MB/sec

  Pretty amazing raw NVME disk performance on the 16GB Pi! My hopes were high as I started testing.
  
  The first time I ran sdm on Pi5-16GB using the NVME data disk it seemed like it was taking a long time, so I brought up Pi5-8GB and ran the same test. Indeed, the 16GB was SIGNIFICANTLY slower than the 8GB.
  
  I put together an easily reproducible test.
  
  Repro steps
  • Install sdm: curl -L https://raw.githubusercontent.com/gitbl ... mInstaller | bash
  • Grab a fresh .IMG (I used 2024-11-19-raspios-bookworm-arm64-lite.img)
  • Run this script passing it the IMG file:

    Code:

    #!/bin/bashfunction errexit() {    echo -e "$1"    exit 1}[ $EUID -eq 0 ] && sudo="" || sudo="sudo"img="$1"[ "$img" == "" ] && errexit "? No IMG specified"[ "$(type -t sdm)" == "" ] && errexit "? sdm is not installed"assets="."[ -f $assets/my.plugins ] && rm -f $assets/my.plugins.1 && mv $assets/my.plugins $assets/my.plugins.1(cat <<EOF# Plugin List generated $(date +"%Y-%m-%d %H:%M:%S")user:adduser=bls|password=passworddisables:triggerhappy|bluetooth|piwizapt-filetrim-enablequietness:consoleblank=300|noquiet=keep|nosplash=keep|noplymouth# Nothing to see here...Just a random collection of appsapps:apps=keychain vim emacs zile python3-pip qemu-user-static binfmt-support systemd-container lsof iperf3 zip dnsutils lftp ncftp tmux socat xterm xterm x11-apps xfonts-base xfonts-100dpi xfonts-75dpi xfonts-scalable xdm icewm elpa-yaml-mode putty aptitude file qpdfview qrencode avahi-utils uuid-runtime jq mmv btop iftop arping traceroute whois telnet nmap wavemon feh links2 zerofree pv f3 xxd yad inxi byobu libheif-examples pdfgrep mc nnn calc lsscsi sg3-utils firefoxsshhostkey:generate-keysL10n:host|wificountry=usEOF    ) | bash -c "cat >|$assets/my.plugins"$sudo sdm --customize --plugin @$assets/my.plugins --reboot 10 --expand-root \      --extend --xmb 3072 \      --batch \      --logwidth 192 $img

  NVME Results
  
  Pi5-8GB: 2min 20sec
  Pi5-16GB: 3min 35sec
  
  System disk swap
  
  I swapped the system disk between the two Pis and the poor performance remained on the Pi5-16GB when using the NVME data disk.
  
  A performance surprise!
  
  I reran the test on both systems with the IMG file on the USB SATA disk rather than the NVME. The performance in this scenario is nearly identical. In fact, the 16GB is slightly faster.
  
  The Big Question
  
  What the heck is going on with NVME performance on the 16GB system?
  
  I'd appreciate any suggestions or pointers to what I can do to speed up the 16GB system when using the NVME as a data disk. I've included full details in case anyone else wants to try it. Would be great to get some test data from some other folks with similar Pi5-16GB configurations.

Statistics: Posted by bls — Wed Jan 15, 2025 10:00 pm

---