The
Raspberry Pi server that I built following
the pi-hosted instructions has crashed hard. It appears the external boot disk may have become corrupted, or worse, that the USB bus in the
Raspberry Pi is suddenly having problems with the external drive. Fortunately, I think I've recovered what data that I have, but I am attempting to recover the boot partition before I rebuild this server from scratch. When I boot I see "Firmware not found" on the diagnostic screen.
#It took some work, but I managed to recover the
Raspberry Pi server. Writing down some quick notes here to remember what I did. First off, the problem occurred while I was doing an apt full-upgrade via Shell In A Box. Not sure whether the problem was caused because I was logged in using Shell In A Box or just bad luck. Something happened to cause files in the boot partition to be deleted. What I did to fix was the following:
#
- Flashed a current version of 64-bit Raspberry Pi OS Lite to an SD card. Ran apt update & apt upgrade. Disconnected the SSD, plugged in the SD Card and booted.
#
- Connected the external SSD to the USB port to confirm it was visible. Ran lsblk and determined the boot partition was on sda1; root on sda2.
#
- I created a couple of mount points:
mkdir /mnt/usb1 and mkdir /mnt/usb2
#
- Learned the file system type by running blkid. I mounted the root partition:
sudo mount -t ext4 /dev/sda2 /mnt/usb2
#
- I browsed the file system and copied the /portainer directory and contents to the SD card for backup. (Yes, I did not have backups of this server, which I need to fix!)
#
- I mounted the boot partition:
sudo mount -t vfat /dev/sda1 /mnt/usb1
#
- I ran
dosfsck -t -a -w /dev/sda1 that reported a dirty bit being set that it cleared but it did not find any corrupted files
#
- I compared /boot on the SD Card to the /mnt/usb1 and noted that /mnt/usb1 had fewer files by far. Having read on the RaspberryPi.com forum of a person repairing by coping the contents of /boot from a fresh install of RaspberryPi OS, that is what I did.
#
- After I completed step 8 the boot up went further than previously but did not complete. After reviewing further information it occurred to me that I overwrote cmdline.txt from the SD Card to the SSD and that the root PARTUUID would be set to the wrong value, being the one from the SD Card rather than the SSD.
#
- Shutdown and booted from the SD Card, connected the SSD to the USB port and ran sudo blkid to display the PARTUUID for sda2 and wrote that down. I mounted /dev/sda1 and edited cmdline.txt with the correct PARTUUID.
#
- Umount /dev/sda1, shutdown, removed the SD Card and then boot with the SSD attached to the USB port. Finally, successful boot!
#The troubleshooting was not as direct as suggested above, I did several other things in the attempt to resolve that did not work, the key ones being:
#
- Ran sudo raspi-config to set the EEPROM to default then set back to the more recent stable version thinking that would resolve "Firmware not found" but that did not work.
#
- Ran sudo rpi-update that downloaded and flashed another version of the EEPROM, which is the version currently installed as of now. I did this because someone who had a similar problems to what I was experiencing reported this fixed the problem, despite the warnings that running it should only be done by direction of engineering.
#I don't want to believe that today is the last day of August. I have a emotional aversion to Labor Day weekend, it marked the end of summer and the return to school during my childhood. Even now as a 56 year old I am still emotionally tied to the passage of time during my childhood.
#