diff --git a/CLAUDE.md b/CLAUDE.md index 9675757..dc1a5bd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,30 +10,119 @@ - **Username:** tonym - **API Token:** 4a155bf2a69852c805f3ba1b900e47056e5f7f0c (full permissions) -## Current Setup +## Current Rebuild Status (September 2025) -### Docker Containers -- **Recently Installed:** ollama/ollama (AI/LLM inference engine) +### Cache Pool Failure Recovery ✅ IN PROGRESS +- **Incident:** Complete cache pool failure requiring appdata rebuild +- **Recovery Method:** Using "Previous Apps" to restore container configurations +- **Total Containers:** 80+ containers across all categories +- **Documentation:** All rebuild progress tracked in Gitea issues -### Known Fixes Applied -- Disabled readmore.js on Docker page to fix slow performance issues +### Docker Container Tracking System ✅ COMPLETED +- **Total Issues Created:** 80+ individual container configuration issues +- **Issues Updated:** 15+ core services with detailed actionable steps +- **Format:** Step-by-step checklists with exact configurations +- **Location:** http://192.168.0.5:3022/tonym/NastyNAS/issues +- **Status:** Ready for systematic configuration + +### Services Configured & Online ✅ +- **Pushover-On-Start** - Boot notifications working +- **ArchiveBox** - Web archiving operational (port TBD) +- **Change Detection** - Website monitoring active (port TBD) +- **13ft Ladder** - Paywall bypass functional (port TBD) +- **README.md** - Service directory created and updating ## Project Structure -This directory tracks Unraid server configuration, issues, and changes: -- `issues.md` - Known issues and their fixes (migrated to Gitea issues) -- `todo.md` - Running log of changes and modifications (migrated to Gitea issues) +This directory tracks Unraid server configuration and rebuild: +- `rebuild-log.md` - Container restoration progress tracking +- `backup-strategy.md` - Comprehensive backup plan (CA Backup + Kopia + B2) +- `indexer-recovery-complete.md` - Recovered Prowlarr API keys and configurations +- `container-list.md` - Complete list of containers by category +- `containers-not-restored.md` - Deliberately skipped containers with reasoning +- `future-work.md` - Tracking future improvements from closed issues +- `unraid-7.0.1-docker-setup.md` - Complete Docker setup guide with paths and ports +- `issues-update-summary.md` - Summary of updated Gitea issues with detailed steps +- `psu-failure-vm-recovery-2026-07.md` - 2026-07 PSU failure: VM rebuild, libvirt/NVIDIA deadlock fix, docker log-corruption fix (disk1 still offline) +- `README.md` - Live service directory (updates as services come online) - `CLAUDE.md` - This file, providing context for AI assistance -## Gitea Integration ✅ COMPLETED -- ✅ All projects migrated to Gitea server (cancertherapies, kilncontroller, markdownconverter, med-tracker, spikeclone) -- ✅ Issues migrated from markdown files to Gitea issue tracking -- ✅ Git repositories initialized and pushed for all projects -- ✅ Project documentation converted to actionable issue tracking +## Planned Implementation Phases -## Current Issue Tracking (Migrated to Gitea) -- **Unraid-Server repo**: Docker performance fixes, documentation tasks -- **All project repos**: Development roadmaps, completed features, future enhancements +### Phase 1: Container Restoration ⏳ CURRENT +- Complete Docker container configuration using Gitea issue tracking +- Core media stack in progress (Sonarr/Radarr/Lidarr importing libraries) +- Prowlarr ready for indexer import (all API keys recovered) +- Download clients need configuration (SABnzbd import issue noted) +- 4 services online, 76+ to configure + +### Phase 2: Backup Implementation ⏳ PENDING +- Install CA Backup plugin for local appdata backups (daily) +- Configure Kopia container for deduplicated backups +- Set up dual backup schedules and retention policies +- Test restoration procedures from both backup methods + +### Phase 3: Offsite Protection ⏳ PENDING +- Connect Kopia to Backblaze B2 for offsite storage +- Set up Gitea repository mirroring to external services +- Document complete disaster recovery procedures +- Test full disaster recovery workflow + +### Phase 4: Web Services & Access ⏳ PENDING +- Configure Nginx Proxy Manager for reverse proxy +- Set up subdomains for all services +- Deploy dashboard solution (Homepage vs Homarr evaluation) +- Implement SSL certificates with Let's Encrypt + +## Future Documentation Plans + +### README.md Service Directory ⏳ PLANNED +- **Purpose:** Live directory of available services for friends/users +- **Update Process:** As Gitea issues are closed → add services to README +- **Format:** Organized by category with direct links and descriptions +- **Status:** Will be created during Phase 1 as services come online + +### Welcome Guide for Friends ⏳ PLANNED +- **Purpose:** User-friendly guide for accessing server services +- **Content:** Service descriptions, access instructions, usage tips +- **Target Audience:** Non-technical friends and family +- **Format:** Simple, illustrated guide with direct links +- **Location:** Will be hosted on server (possibly BookStack or simple HTML) + +## Backup Recovery Information + +### Recovered Prowlarr Indexers ✅ AVAILABLE +- **DOGnzb:** API key available in recovery documentation +- **NZBFinder:** API key available in recovery documentation +- **OZnzb, Nzb.su, NZBCat:** All API keys recovered and documented +- **Total:** 7 active indexers ready for import + +### Backup Strategy Implementation +- **Local Backups:** CA Backup plugin → daily to array (7 days retention) +- **Deduplicated:** Kopia → continuous snapshots with compression +- **Offsite:** Kopia → Backblaze B2 (monthly archives, 3 month retention) +- **Cost Estimate:** $10-20/month for complete backup coverage + +## Container Categories (80+ Total) +- **Media Management:** Sonarr, Radarr, Plex, Tautulli, Overseerr, etc. +- **LLM & AI:** ollama, AnythingLLM, LocalAI, big-AGI, Invoke-AI +- **Photos:** Immich, Immich-Power-Tools +- **System Tools:** Nginx Proxy Manager, Uptime Kuma, Glances, Prometheus +- **Databases:** MariaDB, PostgreSQL, Redis +- **Household Management:** Grocy, Homebox, Monica, LubeLogger +- **Development:** Gitea, BookStack, Jira Software +- **Gaming:** Minecraft Bedrock Server, Epic Games Free Games +- **Security & IoT:** Frigate, ESPHome, AirConnect +- **And many more...** ## Important Notes -- Server performance has been optimized by disabling unnecessary JavaScript features -- Focus on documenting changes for future reference and troubleshooting \ No newline at end of file +- All container configurations tracked as individual Gitea issues +- Prowlarr API keys successfully recovered from database backup +- Dual backup strategy planned to prevent future data loss +- Service directory and user guides planned for Phase 4 +- Focus on systematic, documented approach to prevent future rebuild needs + +## Known Fixes Applied ✅ STILL RELEVANT +- **Disabled readmore.js on Docker page** to fix slow performance issues + - Location: Docker settings page → "Enable container table readmore-js" → Disabled + - Result: Significantly improved Docker page load times + - Status: Applied and working \ No newline at end of file diff --git a/psu-failure-vm-recovery-2026-07.md b/psu-failure-vm-recovery-2026-07.md new file mode 100644 index 0000000..bf5d9ef --- /dev/null +++ b/psu-failure-vm-recovery-2026-07.md @@ -0,0 +1,61 @@ +# PSU Failure → VM & Docker Recovery + +**Date:** 2026-07-14 +**Server:** NastyNAS (192.168.0.5) +**Incident:** PSU failed and was replaced. During the swap, disk1's SATA port was re-damaged. A New Config was done (all disks left in original slots, intending to re-add disk1 in slot 1). Parity is not in use. Afterward: VM configs appeared to be gone and the web UI was unusable (pages never loaded). + +All diagnosis and repair was done over SSH (`root@192.168.0.5`) because the webGUI was unusable. + +--- + +## Root causes found + +1. **"VMs nuked" — actually just the definitions were offline, not lost.** + - `libvirt.img` (holds the VM XML definitions) lived on **disk1**, which is offline (damaged SATA port, and no parity to emulate it). + - At array start, Unraid couldn't find `libvirt.img` at `/mnt/user/system/libvirt/libvirt.img` and **auto-created a blank 1 GB replacement** on disk4 → VM Manager showed nothing. + - **All vdisks were always safe** on the cache pool (`/mnt/cache/domains/`). No XML backup existed anywhere on the flash drive. + +2. **Unusable web UI — three compounding causes:** + - Kopia + Tdarr + Syncthing all rescanning `/mnt/user` at boot, pinning `shfs` (load avg ~20). + - A **wedged NVIDIA kernel thread**: libvirtd deadlocked (`futex_do_wait`) when the RTX 3060 (`0000:2b:00.0`) driver was unloaded mid-probe — triggered by the *original* Windows VM's GPU passthrough. The stuck thread (`nv_drm_dev_unload`) spun at 99.9% CPU and was **unkillable** (survived `kill -9`, became a zombie). This is why every VM-related page hung. **Only a reboot clears it.** + - `docker.img` reported ~376k btrfs checksum errors (see #3). + +3. **"docker.img corruption" — trivial, not structural.** + - The 376k errors were all **one 4 KB block** in **one file**: `binhex-prowlarr`'s container stdout log (`…-json.log`, inode 1239, root 5). Something read the log every ~70 s and re-hit the same bad block each time. + - No image, container filesystem, or Docker metadata was damaged. Disk I/O errors were zero (data-at-rest bit-rot from the unclean power loss). + +--- + +## Actions taken + +- Stopped Kopia + Tdarr to relieve load (they auto-start on boot; stopped again post-reboot). +- **Controlled reboot** to clear the wedged NVIDIA thread. Because a normal `reboot` would hang on the "stop libvirt" step, forced it via SysRq (`sync` → remount-RO `u` → reboot `b`) so it couldn't stall. Array auto-started clean; wedged thread gone; GPU cleanly bound. +- **Rebuilt all VM definitions** from the intact vdisks (originals were lost with disk1's `libvirt.img`). Detected each disk's boot type first to avoid guesswork: + - **Windows 11** — GPT + EFI, `viostor.sys` present → q35 + OVMF (`-tpm` loader) + swtpm TPM 2.0 + virtio disks. Reused the surviving nvram UUID `6f9a9a94-4518-fa05-6df1-c7dc1a1c4350` so Secure Boot/TPM state paired up. 2 vdisks (raw, 400 GB + 1000 GB). + - **Manifold** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio. + - **forge** — Debian 13, qcow2, GPT+EFI → q35 + OVMF + virtio. + - **Linux** — empty 80 GB scratch disk (never installed); defined but left off. + - All bridged on `br0`, plain VNC console (no GPU passthrough — that config was in the lost XML). +- **Verified by VNC screenshot** that each real VM boots its OS: Windows → lock screen (no BitLocker lockout), Manifold/forge → Debian login prompts. +- Set autostart on Windows 11, Manifold, forge. +- **Fixed docker "corruption"**: stopped prowlarr, deleted the corrupted `-json.log`, restarted (back to healthy), zeroed the btrfs error counter. Verified 0 new errors. **No docker.img rebuild needed.** +- **Docker cleanup** — reclaimed ~77 GB: build cache 71 GB → 5.6 GB; removed all unreferenced images (pytorch cuda 7.2 GB, node:20/22-bookworm, docker:27-cli, hotio/lidarr, invidious, Gitea Actions leftovers) → images reclaimable 11.9 GB → 0. **Left the 65 stopped containers (~7.75 GB) untouched** — those are dormant apps, a per-app decision. + +--- + +## Reference details (for next time) + +- **Cache pool** is a **2-device btrfs pool** (single profile, ~2.7 TB): `nvme1n1p1` (931 GB SanDisk) + `nvme0n1p1` (1.86 TB HP EX900). uuid `6b6a9c2f-0c7c-4e61-961e-16aa12567538`. Mounting one device separately (e.g. read-only rescue) shows the whole tree but is a device of the *live* pool — don't. +- Array data disks: disk2–6 present (sdb–sdf, XFS). **disk1 is offline** (`DISK_NP_DSBL`). +- VM definitions now persist in the (formerly blank) `libvirt.img` on **disk4**. +- No screenshot/convert tools on the host; `virsh screenshot /tmp/x.ppm` outputs PNG despite the extension — pull and view directly. +- `dmesg -T` timestamps are unreliable right after boot (NTP clock-step); trust `btrfs device stats` counters over dmesg wall-clock. + +--- + +## Open follow-ups + +- [ ] **Repair disk1's SATA port** (hardware). Once reconnected + mounted (Unassigned Devices), the **original** `libvirt.img` on it holds the original VM XMLs — including the **RTX 3060 GPU passthrough** for the Windows VM. Copy it to recover exact configs, or re-add passthrough fresh. +- [ ] Re-add disk1 to the array once the port is fixed; rebuild parity if/when parity is reintroduced. +- [ ] Decide which of the 65 stopped containers are truly dead and can be removed. +- [ ] Kopia/Tdarr are currently stopped — restart from the Docker tab when ready.