/dev/null: Permission denied on Alpine Linux

I upgraded two virtual machines to Alpine Linux 3.24 last night. One worked perfectly. The other came back up after reboot, but the services it runs didn’t start up (reminder to self: this is why you don’t run upgrades late at night), and when I logged in (fortunately SSH was still working!) I saw that some of the scripts were throwing errors like this: -ash: can't create /dev/null: Permission denied Wait, what? Sure enough: ls -l /dev/null crw-rw---- 1 root root 1, 3 Jun 10 10:52 /dev/null The obvious solution was to fix the permissions on /dev/null: doas chmod a+rw /dev/null But that didn’t persist when I rebooted, because devtmpfs isn’t on disk, it’s in memory. Besides, I needed to check for other devices that might have broken permissions. I re-ran the upgrade, followed by repair… doas apk upgrade --available doas apk fix --directory-permissions doas apk fix -x But that didn’t work either. Because /dev isn’t built from packages, it’s built by a device manager. I found a bunch of…