NastyNAS/psu-failure-vm-recovery-2026-07.md

PSU Failure → VM & Docker Recovery

Date: 2026-07-14 Server: NastyNAS (192.168.0.5) Incident: PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded).

All diagnosis and repair was done over SSH (root@192.168.0.5) because the webGUI was unusable.

---

Root causes found

1. "VMs nuked" — actually just the definitions were offline, not lost.

   • libvirt.img (holds the VM XML definitions) lived on disk1, which is offline (damaged SATA port, and no parity to emulate it).
   • At array start, Unraid couldn't find libvirt.img at /mnt/user/system/libvirt/libvirt.img and auto-created a blank 1 GB replacement on disk4 → VM Manager showed nothing.
   • All vdisks were always safe on the cache pool (/mnt/cache/domains/). No XML backup existed anywhere on the flash drive.

2. Unusable web UI — three compounding causes:

   • Kopia + Tdarr + Syncthing all rescanning /mnt/user at boot, pinning shfs (load avg ~20).
   • A wedged NVIDIA kernel thread: libvirtd deadlocked (futex_do_wait) when the RTX 3060 (0000:2b:00.0) driver was unloaded mid-probe — triggered by the original Windows VM's GPU passthrough. The stuck thread (nv_drm_dev_unload) spun at 99.9% CPU and was unkillable (survived kill -9, became a zombie). This is why every VM-related page hung. Only a reboot clears it.
   • docker.img reported ~376k btrfs checksum errors (see #3).

3. "docker.img corruption" — trivial, not structural.

   • The 376k errors were all one 4 KB block in one file: binhex-prowlarr's container stdout log (…-json.log, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time.
   • No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss).

---

Actions taken

• Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot).
• Controlled reboot to clear the wedged NVIDIA thread. Because a normal reboot would hang on the "stop libvirt" step, forced it via SysRq (sync → remount-RO u → reboot b) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound.
• Rebuilt all VM definitions from the intact vdisks (originals were lost with disk1's libvirt.img). Detected each disk's boot type first to avoid guesswork:
  • Windows 11 — GPT + EFI, viostor.sys present → q35 + OVMF (-tpm loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID 6f9a9a94-4518-fa05-6df1-c7dc1a1c4350 so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB).
  • Manifold — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
  • forge — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
  • Linux — empty 80 GB scratch disk (never installed); defined but left off.
  • All bridged on br0, plain VNC console (no GPU passthrough — that config was in the lost XML).
• Verified by VNC screenshot that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts.
• Set autostart on Windows 11, Manifold, forge.
• Fixed docker "corruption": stopped prowlarr, deleted the corrupted -json.log, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. No docker.img rebuild needed.
• Docker cleanup — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. Left the 65 stopped containers (~7.75 GB) untouched — those are dormant apps, a per-app decision.

---

Reference details (for next time)

• Cache pool is a 2-device btrfs pool (single profile, ~2.7 TB): nvme1n1p1 (931 GB SanDisk) + nvme0n1p1 (1.86 TB HP EX900). uuid 6b6a9c2f-0c7c-4e61-961e-16aa12567538. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the live pool — don't.
• Array data disks: disk2–6 present (sdb–sdf, XFS). disk1 is offline (DISK_NP_DSBL).
• VM definitions now persist in the (formerly blank) libvirt.img on disk4.
• No screenshot/convert tools on the host; virsh screenshot <dom> /tmp/x.ppm outputs PNG despite the extension — pull and view directly.
• dmesg -T timestamps are unreliable right after boot (NTP clock-step); trust btrfs device stats counters over dmesg wall-clock.

---

Open follow-ups

• Repair disk1's SATA port (hardware). Once reconnected + mounted (Unassigned Devices), the original libvirt.img on it holds the original VM XMLs — including the RTX 3060 GPU passthrough for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh.
• Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced.
• Decide which of the 65 stopped containers are truly dead and can be removed.
• Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.