Add 2026-07 PSU failure VM & Docker recovery note

Documents the PSU-failure recovery: VMs rebuilt from intact vdisks after
libvirt.img went offline with disk1, the wedged NVIDIA-thread libvirt
deadlock cleared by a controlled reboot, the docker.img 'corruption' that
was one throwaway container log, and ~77GB of docker cleanup. disk1 SATA
repair remains the open hardware follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -10,30 +10,119 @@
 - **Username:** tonym
 - **API Token:** 4a155bf2a69852c805f3ba1b900e47056e5f7f0c (full permissions)
 
-## Current Setup
+## Current Rebuild Status (September 2025)
 
-### Docker Containers
+### Cache Pool Failure Recovery ✅ IN PROGRESS
-- **Recently Installed:** ollama/ollama (AI/LLM inference engine)
+- **Incident:** Complete cache pool failure requiring appdata rebuild
+- **Recovery Method:** Using "Previous Apps" to restore container configurations
+- **Total Containers:** 80+ containers across all categories
+- **Documentation:** All rebuild progress tracked in Gitea issues
 
-### Known Fixes Applied
+### Docker Container Tracking System ✅ COMPLETED
-- Disabled readmore.js on Docker page to fix slow performance issues
+- **Total Issues Created:** 80+ individual container configuration issues
+- **Issues Updated:** 15+ core services with detailed actionable steps
+- **Format:** Step-by-step checklists with exact configurations
+- **Location:** http://192.168.0.5:3022/tonym/NastyNAS/issues
+- **Status:** Ready for systematic configuration
+
+### Services Configured & Online ✅
+- **Pushover-On-Start** - Boot notifications working
+- **ArchiveBox** - Web archiving operational (port TBD)
+- **Change Detection** - Website monitoring active (port TBD)
+- **13ft Ladder** - Paywall bypass functional (port TBD)
+- **README.md** - Service directory created and updating
 
 ## Project Structure
-This directory tracks Unraid server configuration, issues, and changes:
+This directory tracks Unraid server configuration and rebuild:
-- `issues.md` - Known issues and their fixes (migrated to Gitea issues)
+- `rebuild-log.md` - Container restoration progress tracking
-- `todo.md` - Running log of changes and modifications (migrated to Gitea issues)
+- `backup-strategy.md` - Comprehensive backup plan (CA Backup + Kopia + B2)
+- `indexer-recovery-complete.md` - Recovered Prowlarr API keys and configurations
+- `container-list.md` - Complete list of containers by category
+- `containers-not-restored.md` - Deliberately skipped containers with reasoning
+- `future-work.md` - Tracking future improvements from closed issues
+- `unraid-7.0.1-docker-setup.md` - Complete Docker setup guide with paths and ports
+- `issues-update-summary.md` - Summary of updated Gitea issues with detailed steps
+- `psu-failure-vm-recovery-2026-07.md` - 2026-07 PSU failure: VM rebuild, libvirt/NVIDIA deadlock fix, docker log-corruption fix (disk1 still offline)
+- `README.md` - Live service directory (updates as services come online)
 - `CLAUDE.md` - This file, providing context for AI assistance
 
-## Gitea Integration ✅ COMPLETED
+## Planned Implementation Phases
-- ✅ All projects migrated to Gitea server (cancertherapies, kilncontroller, markdownconverter, med-tracker, spikeclone)
-- ✅ Issues migrated from markdown files to Gitea issue tracking
-- ✅ Git repositories initialized and pushed for all projects
-- ✅ Project documentation converted to actionable issue tracking
 
-## Current Issue Tracking (Migrated to Gitea)
+### Phase 1: Container Restoration ⏳ CURRENT
-- **Unraid-Server repo**: Docker performance fixes, documentation tasks
+- Complete Docker container configuration using Gitea issue tracking
-- **All project repos**: Development roadmaps, completed features, future enhancements
+- Core media stack in progress (Sonarr/Radarr/Lidarr importing libraries)
+- Prowlarr ready for indexer import (all API keys recovered)
+- Download clients need configuration (SABnzbd import issue noted)
+- 4 services online, 76+ to configure
+
+### Phase 2: Backup Implementation ⏳ PENDING
+- Install CA Backup plugin for local appdata backups (daily)
+- Configure Kopia container for deduplicated backups
+- Set up dual backup schedules and retention policies
+- Test restoration procedures from both backup methods
+
+### Phase 3: Offsite Protection ⏳ PENDING  
+- Connect Kopia to Backblaze B2 for offsite storage
+- Set up Gitea repository mirroring to external services
+- Document complete disaster recovery procedures
+- Test full disaster recovery workflow
+
+### Phase 4: Web Services & Access ⏳ PENDING
+- Configure Nginx Proxy Manager for reverse proxy
+- Set up subdomains for all services
+- Deploy dashboard solution (Homepage vs Homarr evaluation)
+- Implement SSL certificates with Let's Encrypt
+
+## Future Documentation Plans
+
+### README.md Service Directory ⏳ PLANNED
+- **Purpose:** Live directory of available services for friends/users
+- **Update Process:** As Gitea issues are closed → add services to README
+- **Format:** Organized by category with direct links and descriptions
+- **Status:** Will be created during Phase 1 as services come online
+
+### Welcome Guide for Friends ⏳ PLANNED
+- **Purpose:** User-friendly guide for accessing server services
+- **Content:** Service descriptions, access instructions, usage tips
+- **Target Audience:** Non-technical friends and family
+- **Format:** Simple, illustrated guide with direct links
+- **Location:** Will be hosted on server (possibly BookStack or simple HTML)
+
+## Backup Recovery Information
+
+### Recovered Prowlarr Indexers ✅ AVAILABLE
+- **DOGnzb:** API key available in recovery documentation
+- **NZBFinder:** API key available in recovery documentation
+- **OZnzb, Nzb.su, NZBCat:** All API keys recovered and documented
+- **Total:** 7 active indexers ready for import
+
+### Backup Strategy Implementation
+- **Local Backups:** CA Backup plugin → daily to array (7 days retention)
+- **Deduplicated:** Kopia → continuous snapshots with compression
+- **Offsite:** Kopia → Backblaze B2 (monthly archives, 3 month retention)
+- **Cost Estimate:** $10-20/month for complete backup coverage
+
+## Container Categories (80+ Total)
+- **Media Management:** Sonarr, Radarr, Plex, Tautulli, Overseerr, etc.
+- **LLM & AI:** ollama, AnythingLLM, LocalAI, big-AGI, Invoke-AI
+- **Photos:** Immich, Immich-Power-Tools
+- **System Tools:** Nginx Proxy Manager, Uptime Kuma, Glances, Prometheus
+- **Databases:** MariaDB, PostgreSQL, Redis
+- **Household Management:** Grocy, Homebox, Monica, LubeLogger
+- **Development:** Gitea, BookStack, Jira Software
+- **Gaming:** Minecraft Bedrock Server, Epic Games Free Games
+- **Security & IoT:** Frigate, ESPHome, AirConnect
+- **And many more...**
 
 ## Important Notes
-- Server performance has been optimized by disabling unnecessary JavaScript features
+- All container configurations tracked as individual Gitea issues
-- Focus on documenting changes for future reference and troubleshooting
+- Prowlarr API keys successfully recovered from database backup
+- Dual backup strategy planned to prevent future data loss
+- Service directory and user guides planned for Phase 4
+- Focus on systematic, documented approach to prevent future rebuild needs
+
+## Known Fixes Applied ✅ STILL RELEVANT
+- **Disabled readmore.js on Docker page** to fix slow performance issues
+  - Location: Docker settings page → "Enable container table readmore-js" → Disabled
+  - Result: Significantly improved Docker page load times
+  - Status: Applied and working

psu-failure-vm-recovery-2026-07.md

@@ -0,0 +1,61 @@
+# PSU Failure → VM & Docker Recovery
+
+**Date:** 2026-07-14
+**Server:** NastyNAS (192.168.0.5)
+**Incident:** PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded).
+
+All diagnosis and repair was done over SSH (`root@192.168.0.5`) because the webGUI was unusable.
+
+---
+
+## Root causes found
+
+1. **"VMs nuked" — actually just the definitions were offline, not lost.**
+   - `libvirt.img` (holds the VM XML definitions) lived on **disk1**, which is offline (damaged SATA port, and no parity to emulate it).
+   - At array start, Unraid couldn't find `libvirt.img` at `/mnt/user/system/libvirt/libvirt.img` and **auto-created a blank 1 GB replacement** on disk4 → VM Manager showed nothing.
+   - **All vdisks were always safe** on the cache pool (`/mnt/cache/domains/`). No XML backup existed anywhere on the flash drive.
+
+2. **Unusable web UI — three compounding causes:**
+   - Kopia + Tdarr + Syncthing all rescanning `/mnt/user` at boot, pinning `shfs` (load avg ~20).
+   - A **wedged NVIDIA kernel thread**: libvirtd deadlocked (`futex_do_wait`) when the RTX 3060 (`0000:2b:00.0`) driver was unloaded mid-probe — triggered by the *original* Windows VM's GPU passthrough. The stuck thread (`nv_drm_dev_unload`) spun at 99.9% CPU and was **unkillable** (survived `kill -9`, became a zombie). This is why every VM-related page hung. **Only a reboot clears it.**
+   - `docker.img` reported ~376k btrfs checksum errors (see #3).
+
+3. **"docker.img corruption" — trivial, not structural.**
+   - The 376k errors were all **one 4 KB block** in **one file**: `binhex-prowlarr`'s container stdout log (`…-json.log`, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time.
+   - No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss).
+
+---
+
+## Actions taken
+
+- Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot).
+- **Controlled reboot** to clear the wedged NVIDIA thread. Because a normal `reboot` would hang on the "stop libvirt" step, forced it via SysRq (`sync` → remount-RO `u` → reboot `b`) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound.
+- **Rebuilt all VM definitions** from the intact vdisks (originals were lost with disk1's `libvirt.img`). Detected each disk's boot type first to avoid guesswork:
+  - **Windows 11** — GPT + EFI, `viostor.sys` present → q35 + OVMF (`-tpm` loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID `6f9a9a94-4518-fa05-6df1-c7dc1a1c4350` so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB).
+  - **Manifold** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
+  - **forge** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio.
+  - **Linux** — empty 80 GB scratch disk (never installed); defined but left off.
+  - All bridged on `br0`, plain VNC console (no GPU passthrough — that config was in the lost XML).
+- **Verified by VNC screenshot** that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts.
+- Set autostart on Windows 11, Manifold, forge.
+- **Fixed docker "corruption"**: stopped prowlarr, deleted the corrupted `-json.log`, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. **No docker.img rebuild needed.**
+- **Docker cleanup** — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. **Left the 65 stopped containers (~7.75 GB) untouched** — those are dormant apps, a per-app decision.
+
+---
+
+## Reference details (for next time)
+
+- **Cache pool** is a **2-device btrfs pool** (single profile, ~2.7 TB): `nvme1n1p1` (931 GB SanDisk) + `nvme0n1p1` (1.86 TB HP EX900). uuid `6b6a9c2f-0c7c-4e61-961e-16aa12567538`. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the *live* pool — don't.
+- Array data disks: disk2–6 present (sdb–sdf, XFS). **disk1 is offline** (`DISK_NP_DSBL`).
+- VM definitions now persist in the (formerly blank) `libvirt.img` on **disk4**.
+- No screenshot/convert tools on the host; `virsh screenshot <dom> /tmp/x.ppm` outputs PNG despite the extension — pull and view directly.
+- `dmesg -T` timestamps are unreliable right after boot (NTP clock-step); trust `btrfs device stats` counters over dmesg wall-clock.
+
+---
+
+## Open follow-ups
+
+- [ ] **Repair disk1's SATA port** (hardware). Once reconnected + mounted (Unassigned Devices), the **original** `libvirt.img` on it holds the original VM XMLs — including the **RTX 3060 GPU passthrough** for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh.
+- [ ] Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced.
+- [ ] Decide which of the 65 stopped containers are truly dead and can be removed.
+- [ ] Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.