// as2.no — server wiki

Documentation and setup notes for the homelab server.

# Proxmox

Type-1 hypervisor (Proxmox VE) running the home lab. The as2.no host (Server4-Ubuntu2) is itself VM 104 on this hypervisor. Administered from as2.no over SSH as root (key auth). Last updated 2026-06-15.

VM memory reads high in Proxmox — that's cache, not a leak. Proxmox's per-VM gauge counts everything the guest has touched (incl. Linux page cache), so it shows ~85–90% for a VM using far less. Real usage is on the Linux Server page (from inside the guest, cache excluded) — e.g. VM 104 reads ~87% in Proxmox but ~41% real. qemu-guest-agent is installed (fixes graceful shutdown, fs-freeze backups, guest-IP reporting) but on PVE 9.x it does not change this gauge. Trust the guest/Linux-Server number for real usage.

SMART status of the physical drives passed through to VM 101 (Server4-Win). Last checked 2026-06-21.

• sdb (Z8401V6Z) is failing — a SMART short self-test (2026-06-05) ended in a read failure at LBA 273408, a confirmed bad sector unrepaired since ~hour 5,427 (~1.6 yr). Also 16 pending / 16 offline-uncorrectable sectors and 21 UDMA CRC errors. Re-checked 2026-06-21 (now ~hour 20,161): identical — counts unchanged and the fault has not spread. *SMART overall still reports PASSED because pending sectors don't trip the threshold — treat it as failing anyway. It holds only the backup copy of H: (healthy), so no primary data is at risk; replace it when convenient
• The two ST8000DM004 drives have ~8.5 years of runtime — plan replacements
• None of these passthrough disks are included in Proxmox backups (backup=0)

## Backups & offsite (Jottacloud)

VM images are dumped with vzdump (snapshot mode, zstd) to the Server4-Win-backup CIFS share — //192.168.10.3/Proxmox-backup on the Windows fileserver — then Jottacloud on that box uploads them offsite nightly at 02:00. Run manually for now (no scheduled job). Consistency: 101/104 get a guest-agent fs-freeze (101 = Windows/VSS-consistent, 104 = fs-consistent); 105/106 are consistent while stopped; 102 now has the guest agent too (added 2026-07-07), so its backups are fs-consistent.

Run the Proxmox backup BEFORE 02:00. It must finish the .vma.dat → .vma.zst rename before Jottacloud starts — otherwise Jottacloud opens the freshly-written .dat to upload it and Proxmox's rename fails with a sharing violation (unable to rename ... .vma.dat to ... .vma.zst). This silently broke every backup from ~mid-2025 until 2026-07-07; the fix is scheduling Jottacloud (not real-time) and keeping the two from overlapping.

# Re-check one drive's health + attributes
ssh root@192.168.10.20 smartctl -H -A /dev/sdb

# Manual VM backup -> Windows share (run BEFORE 02:00 / Jottacloud)
ssh root@192.168.10.20 vzdump 104 --storage Server4-Win-backup --mode snapshot --compress zstd

# Manage VMs (list / start / stop / config)
ssh root@192.168.10.20 qm list
ssh root@192.168.10.20 qm start 106

# Server4-Win (VM 101)

The Windows 11 file server — VM 101 on the Proxmox host. Six physical drives are passed straight through from the host (raw NVMe/SATA passthrough), so Windows owns and formats them directly. The QEMU guest agent is enabled, so the disks can be queried remotely from as2.no. Files are shared over FTPS via FileZilla Server (explicit TLS required, min TLS 1.2).

Physical disk → Windows drive-letter map. Windows reports generic QEMU HARDDISK serials, so drives are matched by SATA port order (cross-checked against size and the QEMU serial sequence).

• Backup pairs: I: backs up H:, and D: backs up E:
• I: (failing, Z8401V6Z) only holds the backup copy of H:, which is healthy — so no primary data is at immediate risk. OK for now; replace I: when convenient, but until then the H: → I: backup is unreliable
• G: is a separate external USB drive (Seagate Expansion)

# Server4-Win11-test (VM 102)

Scratch Windows 11 VM for testing — VM 102 on the Proxmox host.

# Server4-Ubuntu2 (VM 104)

The Ubuntu host serving as2.no — VM 104 on the Proxmox box (see Proxmox).

# CML (VM 105)

Cisco Modeling Labs (CML 2.10.0) — Cisco's official network simulation platform. Runs virtual topologies of real Cisco node types (IOS, IOS-XE, IOS-XR, NX-OS) for building and testing network designs — certification study (CCNA / CCNP / CCIE) and pre-deployment validation. VM 105, normally powered off; start it when needed.

Upgraded 2.8 → 2.10.0 on 2026-07-08. Gotcha for next time: CML 2.10+ pulls docker-ce from download.docker.com (via the new cml-docker-shim package), but the repo line in /etc/apt/sources.list.d/docker.list ships commented out. Result: the Cockpit upgrade reports SUCCESS (code 0) but silently stays on the old version, because docker-ce is "not installable" so the whole cml2 upgrade is held. Fix: uncomment the deb ...download.docker.com... noble stable line, sudo apt-get update (confirm apt-cache policy docker-ce shows a candidate), then re-run the Cockpit upgrade and reboot. Two attempts failed silently before we found this.

# eve-ng (VM 106)

EVE-NG Community Edition (6.2.0) — a multi-vendor network emulation platform. Runs virtual devices from many vendors (Cisco, Juniper, Arista, Fortinet and more) in a single topology, complementing CML for multi-vendor labs and certification practice. VM 106, normally powered off; start it when needed.

# Jellyfin (CT 111)

Self-hosted media server for movies & TV, in a privileged LXC on Proxmox (host). Hardware transcoding runs on the host's Intel iGPU (QuickSync); the library is mounted read-only over CIFS from Server4-Win. Served at https://jellyfin.as2.no (valid cert; nginx → container :8096) — reachable on the LAN, over WireGuard, and publicly on the internet (family access, no VPN) as of 2026-07-31.

• QuickSync passthrough: /dev/dri bound into the CT (device allows in /etc/pve/lxc/111.conf); jellyfin user in render+video; intel-opencl-icd for HDR tone-mapping
• Media: host CIFS mounts (ro, root-only creds) bind-mounted into the CT at /media/Movies and /media/Series
• Reverse proxy: nginx on 192.168.10.15 → 192.168.10.11:8096 (websockets); shared as2.no Let's Encrypt cert (see HTTPS); Jellyfin KnownProxies = .15
• Public exposure: ASUS router forwards TCP 443 → .15; hardened at nginx with a login rate-limit (60r/m on /Users/AuthenticateByName) + a fail2ban jail (5 fails / 10 min → 1 h ban, LAN/WG whitelisted) + HSTS/security headers
• Up/down + version on the Services page via the summary poller

# Networking

• Firewall: UFW — ports 80/443 restricted to 192.168.10.0/24 (local only)
• DNS: Domeneshop — as2.no and arildsorli.no A records point to 85.165.99.110
• HTTPS / TLS: Let's Encrypt (DNS-01) — see the HTTPS section
• LAN DNS resolver: dnsmasq on 192.168.10.15 — see dnsmasq

# Switches — Cisco Catalyst 2960CX

Uniform four-switch access layer, one per room (model WS-C2960CX-8PC-L, 8-port PoE). SSH key auth on all four; IOS version, health (CPU / mem / flash / temp / uptime) and up/down status are polled live over SNMP v2c and shown on the network topology page. All four currently up on IOS 15.2(7)E14 (fleet-upgraded from 15.2(7)E8).

• Monitored via SNMP v2c by Telegraf; each switch forwards syslog to 192.168.10.15:514 → Loki (see Syslog pipeline)
• SSH config and how it's applied are documented under device-side config

# Wireless — Cisco AP 4800 + WLC

Four Cisco Aironet AP4800s, one per room, joined to an embedded WLC (Mobility Express) running on AP4800-1. AP4800-1 (Stue) is the wireless root and backhauls to each room's AP; the controller is reachable at the VIP 192.168.10.7, AireOS 8.10.196.0 (version read live from the poller). Live WiFi client counts per radio are on the monitoring page.

• Controller VIP 192.168.10.7; the WLC also forwards syslog under host label AP4800-1
• SNMP RO community is locked to .15 — see device-side config

# Router — ASUS (192.168.10.1)

The ASUS router is the LAN gateway and WAN edge. It serves DHCP, holds the public IP (dynamic, kept current by the Domeneshop updater), and forwards only WireGuard 51820/udp inbound. SSH-polled every 30s for per-interface throughput and WiFi-client counts.

# dnsmasq (LAN DNS)

Local DNS resolver for the home network — forwards queries upstream and supports custom A records.

• Point router's DNS server to 192.168.10.15 to use for all LAN clients
• Add custom hostnames in dnsmasq.conf with address=/hostname.local/192.168.x.x
• *.as2.no resolves to 192.168.10.15 on the LAN
• UFW allows port 53 from 192.168.10.0/24 only

# Start / restart
cd ~/dnsmasq
docker compose up -d

# View logs
docker logs -f dnsmasq

# Test a query from the LAN
dig @192.168.10.15 google.com

# WireGuard VPN

Self-hosted VPN via wg-easy — a Docker image that bundles WireGuard with a web UI for managing clients.

• WireGuard config stored in Docker volume etc_wireguard
• Web UI at https://wg.as2.no (valid cert; nginx -> container :51821) — manage clients, download configs
• Requires kernel capabilities NET_ADMIN and SYS_MODULE
• IP forwarding enabled via net.ipv4.ip_forward=1

# Start / restart
cd ~/wg-easy
docker compose up -d

# View logs
docker logs -f wg-easy

# HTTPS — Let's Encrypt (DNS-01)

TLS certificates issued by Let's Encrypt using the DNS-01 challenge — no public port 80 required.

• DNS-01 works by creating a temporary _acme-challenge TXT record at Domeneshop
• Plugin credentials stored in /etc/letsencrypt/renewal/as2.no.conf
• Nginx uses fullchain.pem and privkey.pem from the cert path

# Check certificate expiry
sudo certbot certificates

# Force renewal (for testing)
sudo certbot renew --dry-run

# Check renewal timer
systemctl status snap.certbot.renew.timer

# Domeneshop IP Updater

Monitors the server's public IP and automatically updates the as2.no A record at Domeneshop when it changes.

• Domains: as2.no, arildsorli.no
• Script: ~/domeneshop-ip-updater/updater.py
• Credentials: /etc/domeneshop-ip-updater/config (chmod 600)
• State file: /var/lib/domeneshop-ip-updater/last_ip
• Runs every 5 minutes via domeneshop-ip-updater.timer
• GitHub: github.com/arildsorli/domeneshop-ip-updater (private)

# Check logs
journalctl -u domeneshop-ip-updater.service -f

# Check timer status
systemctl status domeneshop-ip-updater.timer

# Monitoring stack

Two pipelines, both Docker Compose at ~/monitoring on .15 (its own git repo). Metrics: Telegraf → InfluxDB v2 → Grafana. Logs: devices → AxoSyslog (514) → Loki → Grafana. Embedded dashboards are on the monitoring page; the v2 overview cards come from the host pollers.

What Telegraf collects (every 30s): SNMP v2c from the four 2960CX switches and the AireOS WLC at .7 — interface counters/errors, CPU, memory, flash, temperature, uptime, plus each switch's IOS sysDescr and the WLC's swVersion (these drive the live version badges on the topology page) and per-AP client counts. It also reads this server's own CPU / memory / disk / NIC via a read-only /hostfs mount.

• Grafana access: nginx vhost grafana.as2.no → 127.0.0.1:3000 (also :3000 direct on the LAN), on the as2.no Let's Encrypt cert. Anonymous Viewer + embedding are enabled so the dashboard iframes render; admin password is set in the UI (.env holds only the bootstrap value)
• InfluxDB data model: measurements asus_netdev (WAN/wired/wireless rx/tx), snmp_interface (switch + WLC interface counters/errors), switch_cpu / switch_mem / switch_env / switch_flash, wlc_ap_clients, and host cpu/mem/disk/net. Throughput = non-negative derivative × 8 (bits/s)
• Loki: single-binary, filesystem store, 30-day retention; labels host · program · severity · facility
• Secrets (InfluxDB token, Grafana pw, SNMP communities) live in ~/monitoring/.env (chmod 600, never committed)

# Syslog pipeline (AxoSyslog → Loki)

Switches, the WLC and the ASUS router forward syslog to 192.168.10.15:514. AxoSyslog normalises every format (including Cisco's non-standard frame), maps source IPs to friendly host names and ships to Loki — searchable in Grafana and on the monitoring page by host, severity, program and facility.

# Host pollers

Small scripts on .15 (in ~/monitoring/scripts, run by systemd timers) that feed data the SNMP path can't. Two write straight to InfluxDB; two emit the JSON the v2 dashboard reads — which is why the cards refresh roughly every minute.

• as2_summary.py assembles the overview: WAN throughput, switches + parsed IOS, WLC AireOS version, WiFi totals, Proxmox VM count + SMART disk health, server CPU/mem/disk, syslog rate
• The ASUS pollers SSH in with a dedicated Dropbear key (~/.ssh/asus_poller_ed25519), independent of the router's single web-admin session
• Version-controlled unit files live in ~/monitoring/systemd/; the live copies are in /etc/systemd/system/

# Poller status / logs
systemctl status as2-summary.timer
journalctl -u as2-summary.service -f

# Run a poller once by hand
~/monitoring/scripts/as2_summary.py

# Device-side SNMP/syslog config

What was applied on each device so it reports to the stack on .15. SNMP is read-only and locked to .15; syslog is sent to .15:514. Community strings live in ~/monitoring/.env (shown as placeholders below).

Cisco 2960CX switches (cx-1…4) — in config mode, saved to startup-config:

access-list 99 permit 192.168.10.15
snmp-server community <SNMP_COMMUNITY> RO 99
logging host 192.168.10.15
logging trap informational
! end & write memory

WLC (.7, AireOS Mobility Express) — SNMP RO community locked to .15 + syslog at notifications level. Applied once via ~/monitoring/scripts/wlc_phase2_config.exp <community>:

config snmp community create <WLC_SNMP_COMMUNITY>
config snmp community accessmode ro <WLC_SNMP_COMMUNITY>
config snmp community ipaddr 192.168.10.15 255.255.255.255 <WLC_SNMP_COMMUNITY>
config snmp community mode enable <WLC_SNMP_COMMUNITY>
config snmp version v2c enable
config logging syslog host 192.168.10.15
config logging syslog level notifications
save config

ASUS router (.1) — no SNMP, so it only sends syslog. The helper ~/monitoring/scripts/asus_set_syslog.sh logs into the web UI and sets nvram log_ipaddr=192.168.10.15, applied with restart_logger (no reboot). Its metrics come from the SSH pollers instead of Telegraf.

• The WLC needs a more complex community string than the switches — kept separate as WLC_SNMP_COMMUNITY
• All three forward syslog to AxoSyslog on .15:514 (see Syslog pipeline); only the switches and WLC are polled over SNMP by Telegraf

# Inventory DB & API

One asset table on .15 is the single source of truth behind the v2 clients, IP-plan, hardware and network (switches + APs, also WiFi) pages — each is a filtered view, so an IP or note reads the same everywhere and an edit updates every view. Curated fields are hand-edited in the browser; live fields belong to the poller. These CRT pages are read-only renderings of the same data.

Flow: client_inventory.py UPSERTs live state by MAC → inventory.db ← browser edits via the API; the inventory-api container serves the DB as JSON; render_crt.py regenerates the CRT tables between <!--GEN:*--> markers.

• Merge key is the MAC (stable; IPs move). ip_static = assigned IP (blank = DHCP); a view shows ip_static or else the live ip_current. Notes are per-page — notes_ip_plan / notes_hardware / notes_network / notes_clients, each view its own note
• type (host/desktop/laptop/spare/given_away/client/switch/ap) picks the view; switch/ap feed the network & WiFi tables, a hardware form-factor outranks host so HT-PC shows on hardware while ip_static keeps it on IP-plan
• Live fields (online, ip_current, last_seen, switch_port, vendor) are poller-owned, never editable
• Auth: GET open on the LAN; writes (POST/PATCH/DELETE on /v2/api/assets) need the shared password — nginx limit_except GET HEAD basic-auth, user arild, htpasswd /etc/nginx/.as2_inventory_htpasswd. API binds 127.0.0.1 only
• Edit on v2 via the ✎ Edit button on IP-Plan / Hardware / Network (inline edit · add · delete); the Clients roster has an ✎ Edit notes toggle (notes only); password prompted once per session
• All IP labels on v2 are DB-driven: the structured tables, the Topology diagram (network.js), the Proxmox VM table (live qm list + IP from the DB), and inline prose labels via <span data-host-ip="Device Name"> filled by app.js fillHostIPs(). Change a device's IP in the DB and every v2 label follows

# rebuild API / re-seed DB / regenerate CRT
cd ~/monitoring && docker compose up -d --build inventory-api
python3 ~/monitoring/inventory/migrate.py --write
python3 ~/monitoring/inventory/render_crt.py

# change the edit password
printf 'arild:%s\n' "$(openssl passwd -apr1)" | sudo tee /etc/nginx/.as2_inventory_htpasswd
sudo systemctl reload nginx

# Services

# Nginx

Nginx (1.24.0) is the front door for everything on as2.no — it serves the static homepage and reverse-proxies the three other *.as2.no services. Every vhost redirects :80 → :443 and terminates TLS on the shared Let's Encrypt wildcard cert. UFW limits 80/443 to 192.168.10.0/24, so it's reachable on the LAN (or over WireGuard) only.

• Vhosts live in /etc/nginx/sites-available/, enabled via symlinks in /etc/nginx/sites-enabled/ (one file per *.as2.no host)
• Shared CSP / security headers come from /etc/nginx/snippets/ — security-headers.conf (main site, script-src 'none'), -v2.conf (v2 dashboard + games allow script-src 'self') and -games.conf
• Theme switch: the as2.no vhost reads a theme cookie at / — v2 redirects to /v2/; /theme/v2 and /theme/crt set the cookie. Works even for the no-JS CRT site
• Live data proxy: /v2/grafana/api/ds/query is a same-origin reverse proxy to 127.0.0.1:3000/api/ds/query so the v2 dashboard fetches interface lists / uptime without CORS (read-only query endpoint only)
• Caching: HTML and everything under /v2/ is no-cache (so ?v= cache-busters take effect); css/woff2/svg/ico cache for 7 days

# Test config, then reload (zero-downtime)
sudo nginx -t
sudo systemctl reload nginx

# Enable a new vhost
sudo ln -s /etc/nginx/sites-available/<name> /etc/nginx/sites-enabled/

# Tail access / error logs
sudo tail -f /var/log/nginx/access.log /var/log/nginx/error.log

# Dev Stack

• Web server: Nginx (serves the site + reverse-proxies code/portainer/grafana)
• Containers: Docker + Docker Compose
• Container UI: Portainer (Docker container on port 9000)
• Observability: Grafana · InfluxDB · Loki · Telegraf · AxoSyslog — see Monitoring stack
• IDE: code-server v4 (port 8080, proxied by Nginx)
• Runtimes: Node.js v24 LTS, Python 3.12, pip
• Version control: git + GitHub CLI (gh)

# Docker Workflows

Most services on .15 run as Docker containers, managed with Docker Compose across a few project directories.

# List running containers
docker ps

# View resource usage
docker stats

# View logs for a container
docker logs -f <container-name>

# Restart a container
docker compose -f ~/wg-easy/docker-compose.yml restart

# Pull latest image and recreate container
docker compose pull
docker compose up -d

# Remove unused images / volumes
docker system prune

# as2-homepage

This website. Two maintained themes — the classic CRT terminal look (this page) and a modern v2 NOC dashboard — built with plain HTML, CSS and a little vanilla JS. No frameworks.

• Served directly from ~/as2-homepage/ by Nginx (v2 lives in v2/)
• HTTPS via Let's Encrypt DNS-01 challenge
• Theme switch is cookie-based; CRT keeps the scanlines, vignette, blinking cursor and ASCII header

cd ~/as2-homepage
# edit files...
git add .
git commit -m "describe change"
git push

// as2.no — powered by too much coffee