125cd58672
Documents the PSU-failure recovery: VMs rebuilt from intact vdisks after libvirt.img went offline with disk1, the wedged NVIDIA-thread libvirt deadlock cleared by a controlled reboot, the docker.img 'corruption' that was one throwaway container log, and ~77GB of docker cleanup. disk1 SATA repair remains the open hardware follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 lines
5.3 KiB
Markdown
62 lines
5.3 KiB
Markdown
# PSU Failure → VM & Docker Recovery
|
||
|
||
**Date:** 2026-07-14
|
||
**Server:** NastyNAS (192.168.0.5)
|
||
**Incident:** PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded).
|
||
|
||
All diagnosis and repair was done over SSH (`root@192.168.0.5`) because the webGUI was unusable.
|
||
|
||
---
|
||
|
||
## Root causes found
|
||
|
||
1. **"VMs nuked" — actually just the definitions were offline, not lost.**
|
||
- `libvirt.img` (holds the VM XML definitions) lived on **disk1**, which is offline (damaged SATA port, and no parity to emulate it).
|
||
- At array start, Unraid couldn't find `libvirt.img` at `/mnt/user/system/libvirt/libvirt.img` and **auto-created a blank 1 GB replacement** on disk4 → VM Manager showed nothing.
|
||
- **All vdisks were always safe** on the cache pool (`/mnt/cache/domains/`). No XML backup existed anywhere on the flash drive.
|
||
|
||
2. **Unusable web UI — three compounding causes:**
|
||
- Kopia + Tdarr + Syncthing all rescanning `/mnt/user` at boot, pinning `shfs` (load avg ~20).
|
||
- A **wedged NVIDIA kernel thread**: libvirtd deadlocked (`futex_do_wait`) when the RTX 3060 (`0000:2b:00.0`) driver was unloaded mid-probe — triggered by the *original* Windows VM's GPU passthrough. The stuck thread (`nv_drm_dev_unload`) spun at 99.9% CPU and was **unkillable** (survived `kill -9`, became a zombie). This is why every VM-related page hung. **Only a reboot clears it.**
|
||
- `docker.img` reported ~376k btrfs checksum errors (see #3).
|
||
|
||
3. **"docker.img corruption" — trivial, not structural.**
|
||
- The 376k errors were all **one 4 KB block** in **one file**: `binhex-prowlarr`'s container stdout log (`…-json.log`, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time.
|
||
- No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss).
|
||
|
||
---
|
||
|
||
## Actions taken
|
||
|
||
- Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot).
|
||
- **Controlled reboot** to clear the wedged NVIDIA thread. Because a normal `reboot` would hang on the "stop libvirt" step, forced it via SysRq (`sync` → remount-RO `u` → reboot `b`) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound.
|
||
- **Rebuilt all VM definitions** from the intact vdisks (originals were lost with disk1's `libvirt.img`). Detected each disk's boot type first to avoid guesswork:
|
||
- **Windows 11** — GPT + EFI, `viostor.sys` present → q35 + OVMF (`-tpm` loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID `6f9a9a94-4518-fa05-6df1-c7dc1a1c4350` so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB).
|
||
- **Manifold** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
|
||
- **forge** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
|
||
- **Linux** — empty 80 GB scratch disk (never installed); defined but left off.
|
||
- All bridged on `br0`, plain VNC console (no GPU passthrough — that config was in the lost XML).
|
||
- **Verified by VNC screenshot** that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts.
|
||
- Set autostart on Windows 11, Manifold, forge.
|
||
- **Fixed docker "corruption"**: stopped prowlarr, deleted the corrupted `-json.log`, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. **No docker.img rebuild needed.**
|
||
- **Docker cleanup** — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. **Left the 65 stopped containers (~7.75 GB) untouched** — those are dormant apps, a per-app decision.
|
||
|
||
---
|
||
|
||
## Reference details (for next time)
|
||
|
||
- **Cache pool** is a **2-device btrfs pool** (single profile, ~2.7 TB): `nvme1n1p1` (931 GB SanDisk) + `nvme0n1p1` (1.86 TB HP EX900). uuid `6b6a9c2f-0c7c-4e61-961e-16aa12567538`. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the *live* pool — don't.
|
||
- Array data disks: disk2–6 present (sdb–sdf, XFS). **disk1 is offline** (`DISK_NP_DSBL`).
|
||
- VM definitions now persist in the (formerly blank) `libvirt.img` on **disk4**.
|
||
- No screenshot/convert tools on the host; `virsh screenshot <dom> /tmp/x.ppm` outputs PNG despite the extension — pull and view directly.
|
||
- `dmesg -T` timestamps are unreliable right after boot (NTP clock-step); trust `btrfs device stats` counters over dmesg wall-clock.
|
||
|
||
---
|
||
|
||
## Open follow-ups
|
||
|
||
- [ ] **Repair disk1's SATA port** (hardware). Once reconnected + mounted (Unassigned Devices), the **original** `libvirt.img` on it holds the original VM XMLs — including the **RTX 3060 GPU passthrough** for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh.
|
||
- [ ] Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced.
|
||
- [ ] Decide which of the 65 stopped containers are truly dead and can be removed.
|
||
- [ ] Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.
|