Wiki

# Proxmox 192.168.10.20

Type-1 hypervisor (Proxmox VE) running the home lab. The as2.no host (Server4-Ubuntu2) is itself VM 104 on this hypervisor. Administered from as2.no over SSH as root (key auth).

VM memory reads high in Proxmox — that's cache, not a leak. Proxmox's per-VM memory gauge counts everything the guest has touched (incl. Linux page cache), so it shows ~85–90% for a VM that's actually using far less. The real figure is on the Linux Server page (from inside the guest, cache excluded) — e.g. VM 104 shows ~87% in Proxmox but ~41% real. The qemu-guest-agent is installed (fixes graceful shutdown, fs-freeze backups, guest-IP reporting) but on PVE 9.x it does not change this memory gauge. So: trust the guest/Linux-Server number for real usage, Proxmox's for reserved/touched.

Virtual machines (qm list):

Storage (pvesm status):

All seven passthrough drives report SMART overall-health PASSED (live on the Proxmox page) — but that threshold-based verdict hides two latent issues (last deep-checked 2026-06-21):

• sdb (Z8401V6Z, 8 TB ST8000AS0002) carries latent damage — a short self-test ends in a read failure at LBA 273408, with 16 pending / 16 offline-uncorrectable sectors and 21 UDMA CRC errors. SMART still says PASSED because pending sectors don't trip the threshold; treat it as failing. It only holds the backup copy of H: (healthy), so no primary data is at risk — replace when convenient.
• Both ST8000DM004 drives (sdd/sde) have ~8.5 years (74,100+ h) of runtime — plan replacements.
• Passthrough disks are excluded from Proxmox backups (backup=0).

Backups & offsite (Jottacloud). VM images are dumped with vzdump (snapshot mode, zstd) to the Server4-Win-backup CIFS share — //192.168.10.3/Proxmox-backup on the Windows fileserver — then Jottacloud on that box uploads them offsite nightly at 02:00. Run manually for now (no scheduled job). Consistency: 101/104 get a guest-agent fs-freeze (101 = Windows/VSS-consistent, 104 = fs-consistent); 105/106 are consistent while stopped; 102 now has the guest agent too (added 2026-07-07), so its backups are fs-consistent.

⚠️ Run the Proxmox backup before 02:00. It must finish the .vma.dat → .vma.zst rename before Jottacloud starts — otherwise Jottacloud opens the freshly-written .dat to upload it and Proxmox's rename fails with a sharing violation (unable to rename … .vma.dat to … .vma.zst). This silently broke every backup from ~mid-2025 until 2026-07-07; the fix is scheduling Jottacloud (not real-time) and keeping the two from overlapping.

# Re-check one drive's health + attributes
ssh root@192.168.10.20 smartctl -H -A /dev/sdb

# Manual VM backup -> Windows share (run BEFORE 02:00 / Jottacloud)
ssh root@192.168.10.20 vzdump 104 --storage Server4-Win-backup --mode snapshot --compress zstd

# Manage VMs (list / start / stop / config)
ssh root@192.168.10.20 qm list
ssh root@192.168.10.20 qm start 106

# Server4-Win VM 101 · 192.168.10.3

The Windows 11 file server. Six physical drives are passed straight through from the host (raw NVMe/SATA passthrough), so Windows owns and formats them directly. The QEMU guest agent is enabled, so the disks can be queried remotely from as2.no. Files are shared over FTPS via FileZilla Server (explicit TLS required, min TLS 1.2).

Physical disk → Windows drive-letter map:

• Backup pairs: I: backs up H:, and D: backs up E:.
• I: (failing) only holds the backup of H: (healthy) — no primary data at risk, but the H: → I: backup is unreliable until replaced.
• G: is a separate external USB drive (Seagate Expansion).

# Server4-Win11-test VM 102 · 192.168.10.14

Scratch Windows 11 VM for testing.

# Server4-Ubuntu2 VM 104 · 192.168.10.15

The Ubuntu host serving as2.no — VM 104 on the Proxmox box. Live metrics on the Linux page.

# CML VM 105 · 192.168.10.16

Cisco Modeling Labs (CML 2.10.0) — Cisco's official network simulation platform. Runs virtual topologies of real Cisco node types (IOS, IOS-XE, IOS-XR, NX-OS) for building and testing network designs — certification study (CCNA / CCNP / CCIE) and pre-deployment validation. Normally powered off; start it when needed.

✅ Upgraded 2.8 → 2.10.0 on 2026-07-08. Gotcha for next time: CML 2.10+ pulls docker-ce from download.docker.com (via the new cml-docker-shim package), but the repo line in /etc/apt/sources.list.d/docker.list ships commented out. Result: the Cockpit upgrade reports SUCCESS (code 0) but silently stays on the old version, because docker-ce is "not installable" so the whole cml2 upgrade is held. Fix: uncomment the deb …download.docker.com… noble stable line, sudo apt-get update (confirm apt-cache policy docker-ce shows a candidate), then re-run the Cockpit upgrade and reboot. Two attempts failed silently before we found this.

# eve-ng VM 106 · 192.168.10.18

EVE-NG Community Edition (6.2.0) — a multi-vendor network emulation platform. Runs virtual devices from many vendors (Cisco, Juniper, Arista, Fortinet and more) in a single topology, complementing CML for multi-vendor labs and certification practice. Normally powered off; start it when needed.

# Jellyfin CT 111 · 192.168.10.11

Self-hosted media server for movies & TV, in a privileged LXC on Proxmox. Hardware transcoding runs on the host's Intel iGPU (QuickSync); the media library is mounted read-only over CIFS from Server4-Win. Reachable at https://jellyfin.as2.no (valid cert; nginx → container :8096) — on the LAN, over WireGuard, and publicly on the internet (family access, no VPN) since 2026-07-31.

• QuickSync passthrough: /dev/dri bound into the CT (cgroup2 device allows in /etc/pve/lxc/111.conf); the jellyfin user is in render+video. intel-opencl-icd installed for HDR tone-mapping.
• Media mounts: host CIFS mounts (ro, creds root-only) bind-mounted into the CT at /media/Movies and /media/Series.
• Reverse proxy: nginx vhost on 192.168.10.15 → 192.168.10.11:8096 (websockets + proxy_buffering off); on the shared as2.no Let's Encrypt cert (see HTTPS). Jellyfin KnownProxies = .15.
• Public exposure: ASUS router forwards TCP 443 → .15; hardened at nginx with a login rate-limit (60r/m on /Users/AuthenticateByName), a fail2ban jail (5 fails / 10 min → 1 h ban; LAN/WG/docker whitelisted), and HSTS/security headers.
• Up/down + version probed by the summary poller (see Host pollers) and shown on the Services page.

# Manage from the Proxmox host (192.168.10.20)
pct exec 111 -- systemctl status jellyfin
pct exec 111 -- systemctl restart jellyfin

# Container console
pct enter 111

# Networking

Flat single-VLAN L2 on 192.168.10.0/24 — one subnet, no inter-VLAN routing. Full address map on the IP Plan page.

• Gateway / WAN: ASUS router on 192.168.10.1 — see Router.
• Switching: 4× Cisco Catalyst 2960CX — see Switches.
• Wireless: 4× Cisco AP4800 + embedded WLC — see Wireless.
• Firewall: UFW — ports 80/443 restricted to 192.168.10.0/24 (local only).
• DNS: Domeneshop — as2.no and arildsorli.no A records point to 85.165.99.110.
• HTTPS / TLS: Let's Encrypt (DNS-01) — see the HTTPS section.
• LAN DNS resolver: dnsmasq on 192.168.10.15 — see dnsmasq.

# Switches — Cisco Catalyst 2960CX 4 × WS-C2960CX-8PC-L

Uniform four-switch access layer, one per room. SSH key auth on all four; IOS version, health (CPU / mem / flash / temp / uptime) and up/down status are polled live (SNMP v2c) and shown on the Topology page. All four currently up on IOS 15.2(7)E14 (fleet-upgraded from 15.2(7)E8).

• Model WS-C2960CX-8PC-L (8-port PoE) across the fleet.
• Monitored via SNMP v2c; each switch forwards syslog to 192.168.10.15:514 → Loki (see Syslog).

# Wireless — Cisco AP 4800 + WLC AireOS 8.10.196.0

Four Cisco Aironet AP4800s, one per room, joined to an embedded WLC (Mobility Express) running on AP4800-1. AP4800-1 (Stue) is the wireless root and backhauls to each room's AP; the controller is reachable at the VIP 192.168.10.7. Live WiFi client counts per radio are on the WiFi page.

• Embedded WLC (Mobility Express), AireOS 8.10.196.0 — version read live from the poller.
• Controller VIP 192.168.10.7; the WLC also forwards syslog under host label AP4800-1.

# Router — ASUS 192.168.10.1

The ASUS router is the LAN gateway and WAN edge. It serves DHCP, holds the public IP (dynamic, kept current by the Domeneshop updater), and forwards only WireGuard 51820/udp inbound. SSH-polled every 30s for per-interface throughput and WiFi-client counts.

# dnsmasq LAN DNS

Local DNS resolver for the home network — forwards queries upstream and supports custom A records.

• Point the router's DNS server to 192.168.10.15 to use it for all LAN clients.
• Add custom hostnames with address=/hostname.local/192.168.x.x.
• *.as2.no resolves to 192.168.10.15 on the LAN.
• UFW allows port 53 from 192.168.10.0/24 only.

# Start / restart
cd ~/dnsmasq
docker compose up -d

# View logs
docker logs -f dnsmasq

# Test a query from the LAN
dig @192.168.10.15 google.com

# WireGuard VPN

Self-hosted VPN via wg-easy — a Docker image that bundles WireGuard with a web UI for managing clients.

• WireGuard config stored in Docker volume etc_wireguard.
• Web UI at https://wg.as2.no 🔒 (valid cert; nginx → container :51821) — manage clients, download configs.
• Requires kernel capabilities NET_ADMIN and SYS_MODULE.
• IP forwarding enabled via net.ipv4.ip_forward=1.

# Start / restart
cd ~/wg-easy
docker compose up -d

# View logs
docker logs -f wg-easy

# HTTPS — Let's Encrypt DNS-01

TLS certificates issued by Let's Encrypt using the DNS-01 challenge — no public port 80 required.

• DNS-01 works by creating a temporary _acme-challenge TXT record at Domeneshop.
• Plugin credentials stored in /etc/letsencrypt/renewal/as2.no.conf.
• Nginx uses fullchain.pem and privkey.pem from the cert path.

# Check certificate expiry
sudo certbot certificates

# Force renewal (for testing)
sudo certbot renew --dry-run

# Check renewal timer
systemctl status certbot.timer

# Domeneshop IP Updater

Monitors the server's public IP and automatically updates the as2.no A record at Domeneshop when it changes.

• Domains: as2.no, arildsorli.no.
• Script: ~/domeneshop-ip-updater/updater.py.
• Credentials: /etc/domeneshop-ip-updater/config (chmod 600).
• State file: /var/lib/domeneshop-ip-updater/last_ip.
• Runs every 5 minutes via domeneshop-ip-updater.timer.
• GitHub: github.com/arildsorli/domeneshop-ip-updater (private).

# Check logs
journalctl -u domeneshop-ip-updater.service -f

# Check timer status
systemctl status domeneshop-ip-updater.timer

# Monitoring stack Grafana · InfluxDB · Loki

Two pipelines, both Docker Compose at /home/arild/monitoring on 192.168.10.15 (its own git repo). Metrics: Telegraf → InfluxDB v2 → Grafana. Logs: devices → AxoSyslog (514) → Loki → Grafana. Embedded dashboards on the Monitoring page; the v2 overview cards come from the host pollers.

What Telegraf collects (every 30s): SNMP v2c from the four 2960CX switches and the AireOS WLC at 192.168.10.7 — interface counters/errors, CPU, memory, flash, temperature, uptime, plus each switch's IOS sysDescr and the WLC's swVersion (these drive the live version badges on Topology) and per-AP client counts. It also reads this server's own CPU / memory / disk / NIC via a read-only /hostfs mount.

• Grafana access: nginx vhost grafana.as2.no → 127.0.0.1:3000 (also :3000 direct on the LAN), on the as2.no Let's Encrypt cert. Anonymous Viewer + embedding are enabled so the dashboard iframes render; admin password is set in the UI (.env holds only the bootstrap value).
• InfluxDB data model: measurements asus_netdev (WAN/wired/wireless rx/tx), snmp_interface (switch + WLC interface counters/errors), switch_cpu / switch_mem / switch_env / switch_flash, wlc_ap_clients, and host cpu/mem/disk/net. Throughput = non-negative derivative × 8 (bits/s).
• Loki: single-binary, filesystem store, 30-day retention; labels host · program · severity · facility.
• Secrets (InfluxDB token, Grafana pw, SNMP communities) live in ~/monitoring/.env (chmod 600, never committed).

# Syslog pipeline AxoSyslog → Loki

Switches, the WLC and the ASUS router forward syslog to 192.168.10.15:514. AxoSyslog normalises every format (including Cisco's non-standard frame), maps source IPs to friendly host names and ships to Loki — searchable in Grafana and on the Syslog page by host, severity, program and facility.

# Host pollers systemd timers

Small scripts on 192.168.10.15 (in ~/monitoring/scripts, run by systemd timers) that feed data the SNMP path can't. Two write straight to InfluxDB; two emit the JSON this v2 dashboard reads — which is why the cards refresh roughly every minute.

• as2_summary.py assembles the overview: WAN throughput, switches + parsed IOS, WLC AireOS version, WiFi totals, Proxmox VM count + SMART disk health, server CPU/mem/disk, syslog rate.
• The ASUS pollers SSH in with a dedicated Dropbear key (~/.ssh/asus_poller_ed25519), independent of the router's single web-admin session.
• Version-controlled unit files live in ~/monitoring/systemd/; the live copies are in /etc/systemd/system/.

# Poller status / logs
systemctl status as2-summary.timer
journalctl -u as2-summary.service -f

# Run a poller once by hand
~/monitoring/scripts/as2_summary.py

# Device-side SNMP/syslog config switches · WLC · router

What was applied on each device so it reports to the stack on 192.168.10.15. SNMP is read-only and locked to 192.168.10.15; syslog is sent to 192.168.10.15:514. Community strings live in ~/monitoring/.env (shown as placeholders below).

Cisco 2960CX switches (cx-1…4) — in config mode, saved to startup-config:

access-list 99 permit 192.168.10.15
snmp-server community <SNMP_COMMUNITY> RO 99
logging host 192.168.10.15
logging trap informational
! end & write memory

WLC (192.168.10.7, AireOS Mobility Express) — SNMP RO community locked to 192.168.10.15 + syslog at notifications level. Applied once via ~/monitoring/scripts/wlc_phase2_config.exp <community>:

config snmp community create <WLC_SNMP_COMMUNITY>
config snmp community accessmode ro <WLC_SNMP_COMMUNITY>
config snmp community ipaddr 192.168.10.15 255.255.255.255 <WLC_SNMP_COMMUNITY>
config snmp community mode enable <WLC_SNMP_COMMUNITY>
config snmp version v2c enable
config logging syslog host 192.168.10.15
config logging syslog level notifications
save config

ASUS router (192.168.10.1) — no SNMP, so it only sends syslog. The helper ~/monitoring/scripts/asus_set_syslog.sh logs into the web UI and sets nvram log_ipaddr=192.168.10.15, applied with restart_logger (no reboot). Its metrics come from the SSH pollers instead of Telegraf.

• The WLC needs a more complex community string than the switches — kept separate as WLC_SNMP_COMMUNITY.
• All three forward syslog to AxoSyslog on 192.168.10.15:514 (see Syslog pipeline); only the switches and WLC are polled over SNMP by Telegraf.

# Inventory DB & API SQLite · FastAPI · single source of truth

One asset table on 192.168.10.15 is the single source of truth behind the v2 Clients, IP-Plan, Hardware and Network (switch + AP tables, also WiFi) pages — each page is a filtered view of it, so an IP or note reads the same everywhere and editing it once updates every view. Curated fields are hand-edited in the browser; live fields are owned by the poller.

Data flow: client_inventory.py UPSERTs live state by MAC → inventory.db ← browser edits via the API; the inventory-api container serves the DB as JSON to the pages; render_crt.py regenerates the read-only CRT tables from the same DB.

Asset model — the merge key is the MAC (stable; IPs move). ip_static is the assigned IP (NULL = DHCP); a view shows ip_static or, if unset, the live ip_current. Notes are per-page — notes_ip_plan / notes_hardware / notes_network / notes_clients, so each view carries its own independent note (e.g. the Proxmox RAM-ceiling note lives only on Hardware). type (host / desktop / laptop / spare / given_away / client / switch / ap) selects which view a row appears in — switch/ap drive the Network & WiFi tables, and a hardware form-factor outranks host so e.g. HT-PC shows on Hardware while its ip_static keeps it on IP-Plan. Live fields (online, ip_current, last_seen, switch_port, vendor) are poller-owned and not editable.

• Auth: GET is open on the LAN; writes need the shared password — nginx limit_except GET HEAD basic-auth on /v2/api/, user arild, htpasswd at /etc/nginx/.as2_inventory_htpasswd. The API itself listens on 127.0.0.1 only.
• Editing: the ✎ Edit button on IP-Plan / Hardware / Network makes rows editable (inline edit · + add · 🗑 delete); the Clients roster has an ✎ Edit notes toggle (notes only — it's a discovered roster, so no add/delete). The password is prompted once per session and sent as basic-auth.
• All IP labels on v2 are DB-driven: the structured tables, the Topology diagram labels (network.js), the Proxmox VM table (live qm list from as2_summary.py with the IP joined from the DB), and even inline prose/header labels — those use <span data-host-ip="Device Name"> filled by app.js fillHostIPs() from /assets (literal IP inside = fallback). Change a device's IP in the DB and every label follows. v2 uses full 192.168.10.x addresses throughout; only the /24 subnet, DHCP-pool ranges and attribute/href URLs stay literal.
• Concurrency: WAL mode; the container runs as uid 1000 so its -wal/-shm sidecars stay shared with the host poller.
• CRT inventory pages are read-only, regenerated from the DB each poller cycle (clients.html is generated directly by client_inventory.py).

# rebuild / restart the API
cd ~/monitoring && docker compose up -d --build inventory-api

# re-seed the DB from current data (destructive) · regenerate CRT
python3 ~/monitoring/inventory/migrate.py --write
python3 ~/monitoring/inventory/render_crt.py

# change the edit password
printf 'arild:%s\n' "$(openssl passwd -apr1)" | sudo tee /etc/nginx/.as2_inventory_htpasswd
sudo systemctl reload nginx

# Services

# Nginx web server · reverse proxy

Nginx (1.24.0) is the front door for everything on as2.no — it serves the static homepage and reverse-proxies the three other *.as2.no services. Every vhost redirects :80 → :443 and terminates TLS on the shared Let's Encrypt wildcard cert. UFW limits 80/443 to 192.168.10.0/24, so it's reachable on the LAN (or over WireGuard) only.

• Vhosts live in /etc/nginx/sites-available/, enabled via symlinks in /etc/nginx/sites-enabled/ (one file per *.as2.no host).
• Shared CSP / security headers come from /etc/nginx/snippets/ — security-headers.conf (main site, script-src 'none'), -v2.conf (v2 dashboard + games allow script-src 'self') and -games.conf.
• Theme switch: the as2.no vhost reads a theme cookie at / — v2 redirects to /v2/; /theme/v2 and /theme/crt set the cookie. Works even for the no-JS CRT site.
• Live data proxy: /v2/grafana/api/ds/query is a same-origin reverse proxy to 127.0.0.1:3000/api/ds/query so the v2 dashboard fetches interface lists / uptime without CORS (read-only query endpoint only).
• Inventory API: /v2/api/ proxies to the inventory-api on 127.0.0.1:8090; limit_except GET HEAD puts basic-auth on writes only.
• Caching: HTML and everything under /v2/ is no-cache (so ?v= cache-busters take effect); css/woff2/svg/ico cache for 7 days.

# Test config, then reload (zero-downtime)
sudo nginx -t
sudo systemctl reload nginx

# Enable a new vhost
sudo ln -s /etc/nginx/sites-available/<name> /etc/nginx/sites-enabled/

# Tail access / error logs
sudo tail -f /var/log/nginx/access.log /var/log/nginx/error.log

# Dev Stack

• Web server: Nginx (serves the site + reverse-proxies code/portainer/grafana)
• Containers: Docker + Docker Compose
• Container UI: Portainer (Docker container on port 9000)
• Observability: Grafana · InfluxDB · Loki · Telegraf · AxoSyslog — see Monitoring stack
• IDE: code-server v4 (port 8080, proxied by Nginx)
• Runtimes: Node.js v24 LTS, Python 3.12, pip
• Version control: git + GitHub CLI (gh)

# Docker Workflows

Most services on 192.168.10.15 run as Docker containers, managed with Docker Compose across a few project directories.

# List running containers
docker ps

# View resource usage
docker stats

# View logs for a container
docker logs -f <container-name>

# Restart a container
docker compose -f ~/wg-easy/docker-compose.yml restart

# Pull latest image and recreate container
docker compose pull
docker compose up -d

# Remove unused images / volumes
docker system prune

# as2-homepage

This website. Two maintained themes — the classic CRT terminal look and this modern v2 NOC dashboard — built with plain HTML, CSS and a little vanilla JS. No frameworks.

• Served directly from ~/as2-homepage/ by Nginx.
• HTTPS via Let's Encrypt DNS-01 challenge.
• Theme switch is cookie-based; the v2 dashboard pulls live values from the pollers.

cd ~/as2-homepage
# edit files...
git add .
git commit -m "describe change"
git push