tonym/NastyNAS
1
0
Fork 0
Code Issues 70 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
125cd58672f16bd27398c675d5b8bd25571e3280
NastyNAS/psu-failure-vm-recovery-2026-07.md
Tony M 125cd58672 Add 2026-07 PSU failure VM & Docker recovery note 
Documents the PSU-failure recovery: VMs rebuilt from intact vdisks after
libvirt.img went offline with disk1, the wedged NVIDIA-thread libvirt
deadlock cleared by a controlled reboot, the docker.img 'corruption' that
was one throwaway container log, and ~77GB of docker cleanup. disk1 SATA
repair remains the open hardware follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 22:25:32 -05:00

5.3 KiB
Raw Blame History

PSU Failure → VM & Docker Recovery

Date: 2026-07-14 Server: NastyNAS (192.168.0.5) Incident: PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded).

All diagnosis and repair was done over SSH (root@192.168.0.5) because the webGUI was unusable.

Root causes found

  1. "VMs nuked" — actually just the definitions were offline, not lost.

    • libvirt.img (holds the VM XML definitions) lived on disk1, which is offline (damaged SATA port, and no parity to emulate it).
    • At array start, Unraid couldn't find libvirt.img at /mnt/user/system/libvirt/libvirt.img and auto-created a blank 1 GB replacement on disk4 → VM Manager showed nothing.
    • All vdisks were always safe on the cache pool (/mnt/cache/domains/). No XML backup existed anywhere on the flash drive.

  2. Unusable web UI — three compounding causes:

    • Kopia + Tdarr + Syncthing all rescanning /mnt/user at boot, pinning shfs (load avg ~20).
    • A wedged NVIDIA kernel thread: libvirtd deadlocked (futex_do_wait) when the RTX 3060 (0000:2b:00.0) driver was unloaded mid-probe — triggered by the original Windows VM's GPU passthrough. The stuck thread (nv_drm_dev_unload) spun at 99.9% CPU and was unkillable (survived kill -9, became a zombie). This is why every VM-related page hung. Only a reboot clears it.
    • docker.img reported ~376k btrfs checksum errors (see #3).

  3. "docker.img corruption" — trivial, not structural.

    • The 376k errors were all one 4 KB block in one file: binhex-prowlarr's container stdout log (…-json.log, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time.
    • No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss).

Actions taken

  • Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot).
  • Controlled reboot to clear the wedged NVIDIA thread. Because a normal reboot would hang on the "stop libvirt" step, forced it via SysRq (sync → remount-RO u → reboot b) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound.
  • Rebuilt all VM definitions from the intact vdisks (originals were lost with disk1's libvirt.img). Detected each disk's boot type first to avoid guesswork:
    • Windows 11 — GPT + EFI, viostor.sys present → q35 + OVMF (-tpm loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID 6f9a9a94-4518-fa05-6df1-c7dc1a1c4350 so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB).
    • Manifold — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
    • forge — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
    • Linux — empty 80 GB scratch disk (never installed); defined but left off.
    • All bridged on br0, plain VNC console (no GPU passthrough — that config was in the lost XML).
  • Verified by VNC screenshot that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts.
  • Set autostart on Windows 11, Manifold, forge.
  • Fixed docker "corruption": stopped prowlarr, deleted the corrupted -json.log, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. No docker.img rebuild needed.
  • Docker cleanup — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. Left the 65 stopped containers (~7.75 GB) untouched — those are dormant apps, a per-app decision.

Reference details (for next time)

  • Cache pool is a 2-device btrfs pool (single profile, ~2.7 TB): nvme1n1p1 (931 GB SanDisk) + nvme0n1p1 (1.86 TB HP EX900). uuid 6b6a9c2f-0c7c-4e61-961e-16aa12567538. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the live pool — don't.
  • Array data disks: disk2–6 present (sdb–sdf, XFS). disk1 is offline (DISK_NP_DSBL).
  • VM definitions now persist in the (formerly blank) libvirt.img on disk4.
  • No screenshot/convert tools on the host; virsh screenshot <dom> /tmp/x.ppm outputs PNG despite the extension — pull and view directly.
  • dmesg -T timestamps are unreliable right after boot (NTP clock-step); trust btrfs device stats counters over dmesg wall-clock.

Open follow-ups

  • Repair disk1's SATA port (hardware). Once reconnected + mounted (Unassigned Devices), the original libvirt.img on it holds the original VM XMLs — including the RTX 3060 GPU passthrough for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh.
  • Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced.
  • Decide which of the 65 stopped containers are truly dead and can be removed.
  • Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.