# PSU Failure → VM & Docker Recovery

**Date:** 2026-07-14
**Server:** NastyNAS (192.168.0.5)
**Incident:** PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded).

All diagnosis and repair was done over SSH (`root@192.168.0.5`) because the webGUI was unusable.

---

## Root causes found

1. **"VMs nuked" — actually just the definitions were offline, not lost.**
   - `libvirt.img` (holds the VM XML definitions) lived on **disk1**, which is offline (damaged SATA port, and no parity to emulate it).
   - At array start, Unraid couldn't find `libvirt.img` at `/mnt/user/system/libvirt/libvirt.img` and **auto-created a blank 1 GB replacement** on disk4 → VM Manager showed nothing.
   - **All vdisks were always safe** on the cache pool (`/mnt/cache/domains/`). No XML backup existed anywhere on the flash drive.

2. **Unusable web UI — three compounding causes:**
   - Kopia + Tdarr + Syncthing all rescanning `/mnt/user` at boot, pinning `shfs` (load avg ~20).
   - A **wedged NVIDIA kernel thread**: libvirtd deadlocked (`futex_do_wait`) when the RTX 3060 (`0000:2b:00.0`) driver was unloaded mid-probe — triggered by the *original* Windows VM's GPU passthrough. The stuck thread (`nv_drm_dev_unload`) spun at 99.9% CPU and was **unkillable** (survived `kill -9`, became a zombie). This is why every VM-related page hung. **Only a reboot clears it.**
   - `docker.img` reported ~376k btrfs checksum errors (see #3).

3. **"docker.img corruption" — trivial, not structural.**
   - The 376k errors were all **one 4 KB block** in **one file**: `binhex-prowlarr`'s container stdout log (`…-json.log`, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time.
   - No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss).

---

## Actions taken

- Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot).
- **Controlled reboot** to clear the wedged NVIDIA thread. Because a normal `reboot` would hang on the "stop libvirt" step, forced it via SysRq (`sync` → remount-RO `u` → reboot `b`) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound.
- **Rebuilt all VM definitions** from the intact vdisks (originals were lost with disk1's `libvirt.img`). Detected each disk's boot type first to avoid guesswork:
  - **Windows 11** — GPT + EFI, `viostor.sys` present → q35 + OVMF (`-tpm` loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID `6f9a9a94-4518-fa05-6df1-c7dc1a1c4350` so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB).
  - **Manifold** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
  - **forge** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
  - **Linux** — empty 80 GB scratch disk (never installed); defined but left off.
  - All bridged on `br0`, plain VNC console (no GPU passthrough — that config was in the lost XML).
- **Verified by VNC screenshot** that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts.
- Set autostart on Windows 11, Manifold, forge.
- **Fixed docker "corruption"**: stopped prowlarr, deleted the corrupted `-json.log`, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. **No docker.img rebuild needed.**
- **Docker cleanup** — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. **Left the 65 stopped containers (~7.75 GB) untouched** — those are dormant apps, a per-app decision.

---

## Reference details (for next time)

- **Cache pool** is a **2-device btrfs pool** (single profile, ~2.7 TB): `nvme1n1p1` (931 GB SanDisk) + `nvme0n1p1` (1.86 TB HP EX900). uuid `6b6a9c2f-0c7c-4e61-961e-16aa12567538`. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the *live* pool — don't.
- Array data disks: disk2–6 present (sdb–sdf, XFS). **disk1 is offline** (`DISK_NP_DSBL`).
- VM definitions now persist in the (formerly blank) `libvirt.img` on **disk4**.
- No screenshot/convert tools on the host; `virsh screenshot <dom> /tmp/x.ppm` outputs PNG despite the extension — pull and view directly.
- `dmesg -T` timestamps are unreliable right after boot (NTP clock-step); trust `btrfs device stats` counters over dmesg wall-clock.

---

## Open follow-ups

- [ ] **Repair disk1's SATA port** (hardware). Once reconnected + mounted (Unassigned Devices), the **original** `libvirt.img` on it holds the original VM XMLs — including the **RTX 3060 GPU passthrough** for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh.
- [ ] Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced.
- [ ] Decide which of the 65 stopped containers are truly dead and can be removed.
- [ ] Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.