Rebuild dotfiles as one branch with per-host layers

Replaces the previous repo, which had split into two histories that never
met (mainframe on a dead GitLab remote, the laptops on Gitea) with 146
dirty files across three machines and the NAS never enrolled at all.

Branch-per-machine is gone. One main, with host differences expressed as
small files under hosts/<hostname>/ rather than as branches, so there is
nothing to merge. The reconciled zsh layer reduces 15-33 line forks to
1-7 effective lines per host; distro differences (oh-my-zsh prefix,
syntax-highlighting path, fd vs fdfind) are probed in common/ instead.

Fresh history: the old one carried six plaintext credentials, 45 MB of
mail caches, browser caches and vendored binaries. 5,096 tracked files
and 144 MB become 462 files and 2.6 MB. The .gitignore is now an
allowlist, which is what keeps that true.

Root cause of the rot: ~/.local/bin was a symlink to scripts/ with GOPATH
inside it, so every go install wrote into version control (2.2 GB on the
work laptop). PATH now points at the repo instead of the reverse.

Also: Hyprland replaces sway and is sourced in two halves so $browser is
defined before use; singleton automations carry ConditionHost= alongside
host-layer-only placement; ddns moves from cron to a guarded timer;
package manifests and pkg-snapshot/pkg-restore replace the X11-era
install_scripts/; networkmanager-dmenu added to system76 (the binding
always existed, the package never did).

hosts/mainframe/sbin/backup-db-dump

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+#
+# Dump every database to a consistent on-disk file so restic snapshots a
+# restorable copy rather than a live file mid-write.
+#
+# Design notes:
+#
+#  * SQLite dumps run from the HOST, not inside the containers. Vaultwarden
+#    and SFTPGo images ship no sqlite3 binary and traggo is distroless with no
+#    shell at all, so `podman exec` cannot work there. The DB files are on host
+#    bind mounts and `.backup` is safe against a live writer.
+#
+#  * The Immich dump goes to the NFS mount (-> the NAS), NOT /var/backups.
+#    Immich's blobs live on the NAS and its database on mainframe; writing the
+#    dump beside the blobs means one NAS restic snapshot captures both halves.
+#    A database restored against missing photos is not a restore.
+#
+#  * .env is PARSED, never sourced. OPENVPN_PASSWORD contains characters that
+#    break `.` under bash, and quoting it changes what podman-compose passes to
+#    the container -- so the file is left exactly as compose expects and read
+#    with a plain parser instead.
+#
+#  * Every file is written to .tmp and renamed, so a concurrent backup never
+#    snapshots a truncated dump.
+
+set -euo pipefail
+umask 077
+
+DEST=/var/backups/db
+IMMICH_DEST=/home/connor/photo/immich/backups
+ENVFILE=/home/connor/.env
+
+log() { printf '%s backup-db-dump: %s\n' "$(date -Is)" "$*"; }
+env_get() { sed -n "s/^$1=//p" "$ENVFILE" | head -1 | sed -E "s/^'(.*)'\$/\1/; s/^\"(.*)\"\$/\1/"; }
+
+mkdir -p "$DEST"
+IMMICH_DB_USERNAME=$(env_get IMMICH_DB_USERNAME)
+HEDGEDOC_DB_ROOT_PASSWORD=$(env_get HEDGEDOC_DB_ROOT_PASSWORD)
+[ -n "$IMMICH_DB_USERNAME" ] || { log "ERROR: could not read IMMICH_DB_USERNAME"; exit 1; }
+
+# --- Postgres (Immich) -----------------------------------------------------
+if mountpoint -q /home/connor/photo && mkdir -p "$IMMICH_DEST" 2>/dev/null; then
+    tgt="$IMMICH_DEST"
+else
+    log "WARN: /home/connor/photo not mounted; writing immich dump locally instead"
+    tgt="$DEST"
+fi
+log "immich postgres -> $tgt"
+podman exec connor_immich_db_1 sh -c "pg_dumpall -U '$IMMICH_DB_USERNAME'" \
+    | zstd -q -o "$tgt/immich-pgdump.sql.zst.tmp"
+mv -f "$tgt/immich-pgdump.sql.zst.tmp" "$tgt/immich-pgdump.sql.zst"
+
+# --- MariaDB (HedgeDoc) ----------------------------------------------------
+log "hedgedoc mariadb"
+podman exec connor_hedgedocdb_1 sh -c \
+    "mariadb-dump --single-transaction -u root -p'$HEDGEDOC_DB_ROOT_PASSWORD' hedgedoc" \
+    | zstd -q -o "$DEST/hedgedoc.sql.zst.tmp"
+mv -f "$DEST/hedgedoc.sql.zst.tmp" "$DEST/hedgedoc.sql.zst"
+
+# --- SQLite ----------------------------------------------------------------
+for spec in \
+    "vaultwarden:/home/connor/data/bitwarden/db.sqlite3" \
+    "gitea:/home/connor/data/gitea/gitea/gitea.db" \
+    "traggo:/home/connor/data/traggo/traggo.db" \
+    "sftpgo:/home/connor/docs/sftpgo.db" \
+    "shanty:/usr/local/shanty/shanty.db" \
+; do
+    name=${spec%%:*}; path=${spec#*:}
+    if [ ! -f "$path" ]; then log "WARN: $name db missing at $path, skipping"; continue; fi
+    log "$name sqlite"
+    rm -f "$DEST/$name.sqlite3.tmp"
+    sqlite3 "$path" ".backup '$DEST/$name.sqlite3.tmp'"
+    # Verify before promoting. A corrupt dump that looks like a file is worse
+    # than a missing one: it restores without complaint.
+    if [ "$(sqlite3 "$DEST/$name.sqlite3.tmp" 'pragma integrity_check;' 2>&1 | head -1)" != "ok" ]; then
+        log "ERROR: $name failed integrity_check, keeping previous dump"
+        rm -f "$DEST/$name.sqlite3.tmp"; continue
+    fi
+    mv -f "$DEST/$name.sqlite3.tmp" "$DEST/$name.sqlite3"
+done
+
+log "done"

hosts/mainframe/sbin/podman-stack-update

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+#
+# Weekly image refresh for the compose stack in /home/connor.
+#
+# Why this exists: every service in compose.yml is pinned to `:latest` (bar the
+# two Immich images, which carry digests), podman-auto-update.timer is disabled
+# and nothing else pulled. The tag made the stack look current while the
+# running images were up to twelve months old -- jellyseerr, the personal-site
+# pair and searxng's valkey had not moved since 2025. `:latest` without a
+# puller is not a rolling tag, it is a snapshot with a misleading name.
+#
+# Recreation is deliberate rather than pull-only. Stale images are the larger
+# standing risk, `restart: unless-stopped` plus the nightly restic snapshot
+# make a bad pull recoverable, and podman keeps the previous image so a
+# rollback is `podman tag` away. Set APPLY=0 below to downgrade this to
+# pull-and-notify if that trade ever stops being worth it.
+set -euo pipefail
+umask 077
+
+APPLY=1
+PROJECT_DIR=/home/connor
+NTFY_URL=https://ntfy.rcjohnstone.com/infra
+NTFY_ENV=/etc/ntfy/publish.env
+
+notify() {  # notify <priority> <tags> <title> <body>
+    local u p
+    [ -r "$NTFY_ENV" ] || return 0
+    # PARSED, not sourced: the bot password contains ` and &, so sourcing it
+    # dies with a syntax error. Same reason the restic scripts use sed.
+    u=$(sed -n 's/^NTFY_USER=//p' "$NTFY_ENV" | head -1)
+    p=$(sed -n 's/^NTFY_PASS=//p' "$NTFY_ENV" | head -1)
+    [ -n "$u" ] && [ -n "$p" ] || return 0
+    curl -fsS --max-time 20 -u "$u:$p" \
+         -H "Title: $3" -H "Priority: $1" -H "Tags: $2" \
+         -d "$4" "$NTFY_URL" >/dev/null || true
+}
+
+# Trap installed before anything can fail, so a crash still reports. The
+# restic script learned this the hard way: its failure-log code sat above the
+# first thing that could die, and the one failure it was written to capture
+# happened before it was reached.
+LOG=$(mktemp /tmp/podman-stack-update.XXXXXX)
+FAILLOG=/var/log/podman-stack-update.failed.log
+cleanup() {
+    local rc=$?
+    if [ "$rc" -ne 0 ]; then
+        cp -f "$LOG" "$FAILLOG" 2>/dev/null || true
+        notify urgent rotating_light "Stack update FAILED on $(uname -n)" \
+            "exit $rc"$'\n\n'"$(tail -n 25 "$LOG")"
+    fi
+    rm -f "$LOG"
+}
+trap cleanup EXIT
+
+cd "$PROJECT_DIR"
+
+# Image IDs before, so the report names what actually moved rather than
+# everything that was pulled.
+declare -A before
+while read -r ref id; do before["$ref"]=$id; done < <(
+    podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' 2>/dev/null)
+
+echo "=== pull ===" >>"$LOG"
+podman-compose pull >>"$LOG" 2>&1
+
+changed=()
+while read -r ref id; do
+    [ "$ref" = "<none>:<none>" ] && continue
+    if [ "${before[$ref]:-none}" != "$id" ]; then changed+=("$ref"); fi
+done < <(podman images --format '{{.Repository}}:{{.Tag}} {{.ID}}' 2>/dev/null)
+
+if [ "${#changed[@]}" -eq 0 ]; then
+    echo "no image changes" >>"$LOG"
+    notify low white_check_mark "Stack update: no changes" "All images already current."
+    exit 0
+fi
+
+if [ "$APPLY" -ne 1 ]; then
+    notify default package "Stack update: ${#changed[@]} image(s) available" \
+        "$(printf '%s\n' "${changed[@]}")"$'\n\n'"APPLY=0, not recreated."
+    exit 0
+fi
+
+echo "=== up -d ===" >>"$LOG"
+podman-compose up -d >>"$LOG" 2>&1
+
+# A container that is not running 30s after recreation is the failure mode
+# worth shouting about -- a bad image starts, crashes, and without this the
+# run still looks like a success.
+sleep 30
+notrunning=$(podman ps -a --filter 'label=io.podman.compose.project=connor' \
+             --format '{{.Names}} {{.Status}}' 2>/dev/null | grep -v '^\S* Up' || true)
+
+if [ -n "$notrunning" ]; then
+    notify urgent rotating_light "Stack update: containers DOWN on $(uname -n)" \
+        "Updated:"$'\n'"$(printf '%s\n' "${changed[@]}")"$'\n\n'"Not running:"$'\n'"$notrunning"
+    exit 1
+fi
+
+notify default package "Stack update OK on $(uname -n)" \
+    "${#changed[@]} image(s) updated:"$'\n'"$(printf '%s\n' "${changed[@]}")"

hosts/mainframe/sbin/restic-backup

@@ -0,0 +1,127 @@
+#!/usr/bin/env bash
+#
+# Nightly restic backup of mainframe's LOCAL data to the Hetzner Storage Box.
+#
+# Scope: this host only. The six directories under /home/connor that are NFS
+# mounts from the NAS (audio books docs downloads photo video) are excluded
+# automatically by --one-file-system and are backed up by the NAS's own copy
+# of this script, straight off the XFS array. Pulling 500G over NFS every
+# night to hand it back to the same machine would be absurd.
+#
+# Order matters: dump-databases-then-snapshot. backup-db-dump writes the
+# Immich pg dump onto the NAS mount so the NAS's 03:30 run picks it up an hour
+# later. If that run ever overtakes this one it snapshots yesterday's dump --
+# degraded, not broken, and Immich also writes its own daily dump to the same
+# directory.
+
+set -euo pipefail
+umask 077
+
+export RESTIC_CACHE_DIR=/var/cache/restic
+set -a; . /etc/restic/hetzner.env; set +a
+
+# `hostname` is not installed on the NAS and is not on systemd's PATH in
+# general; uname -n always is.
+HOST=$(uname -n); HOST=${HOST%%.*}
+EXCLUDES=/etc/restic/excludes.txt
+NTFY_URL=https://ntfy.rcjohnstone.com/backup
+NTFY_ENV=/etc/ntfy/publish.env
+LOG=$(mktemp /tmp/restic-backup.XXXXXX)
+# Keep the log when the run FAILS. Without this the EXIT trap deleted the only
+# record of restic's actual error, leaving nothing to diagnose from but the 25
+# lines that made it into the ntfy body -- which is exactly what happened on
+# 2026-08-24 when forget/prune died and the cause could not be recovered.
+for d in /var/log "${HOME:-/nonexistent}/.local/state" /tmp; do
+    [ -d "$d" ] && [ -w "$d" ] && { FAILLOG=$d/restic-backup.failed.log; break; }
+done
+cleanup() {
+    local rc=$?
+    # Explicit if, not `[ $rc -ne 0 ] && cp ...`: a failing test as the last
+    # statement of a trap is the kind of set -e landmine that has bitten this
+    # codebase before.
+    if [ "$rc" -ne 0 ] && [ -n "${FAILLOG:-}" ]; then
+        cp -f "$LOG" "$FAILLOG" 2>/dev/null || true
+    fi
+    rm -f "$LOG"
+}
+trap cleanup EXIT
+
+PATHS=(
+    /etc
+    /root
+    # /usr/local rather than /usr/local/bin: shanty's live database is bind
+    # mounted from /usr/local/shanty, not from anywhere under /home/connor.
+    /usr/local
+    /var/backups/db
+    # Five named podman volumes live here. Four are tiny and one (model-cache)
+    # is excluded. Compose bind-mounts almost everything from /home/connor, so
+    # this is easy to forget -- and forgetting it silently drops syncthing's
+    # and searxng's state from every snapshot.
+    /var/lib/containers/storage/volumes
+    /home/connor
+)
+
+log()  { printf '%s restic-backup: %s\n' "$(date -Is)" "$*" | tee -a "$LOG"; }
+
+notify() {  # notify <priority> <tags> <title> <body>
+    local pri=$1 tags=$2 title=$3 body=$4 u p
+    [ -r "$NTFY_ENV" ] || return 0
+    # PARSED, not sourced. The bot password contains ` and &, so `. $NTFY_ENV`
+    # dies with a syntax error -- and it cannot simply be quoted either,
+    # because movie_recs_notify reads the same file with a literal split on
+    # "=" and would then send the quotes as part of the password.
+    u=$(sed -n 's/^NTFY_USER=//p' "$NTFY_ENV" | head -1)
+    p=$(sed -n 's/^NTFY_PASS=//p' "$NTFY_ENV" | head -1)
+    [ -n "$u" ] && [ -n "$p" ] || return 0
+    curl -fsS --max-time 20 \
+         -u "$u:$p" \
+         -H "Title: $title" -H "Priority: $pri" -H "Tags: $tags" \
+         -d "$body" "$NTFY_URL" >/dev/null || true
+}
+
+fail() {
+    log "FAILED: $1"
+    notify urgent "rotating_light" "Backup FAILED on $HOST" \
+        "$1"$'\n\n'"$(tail -n 25 "$LOG")"
+    exit 1
+}
+
+# --- 1. consistent database dumps ------------------------------------------
+log "dumping databases"
+/usr/local/bin/backup-db-dump >>"$LOG" 2>&1 || fail "backup-db-dump failed"
+
+# --- 2. snapshot -----------------------------------------------------------
+log "backing up: ${PATHS[*]}"
+rc=0
+nice -n 10 ionice -c2 -n7 restic backup \
+    --one-file-system \
+    --exclude-file="$EXCLUDES" \
+    --exclude-caches \
+    --tag "$HOST" \
+    --verbose=1 \
+    "${PATHS[@]}" >>"$LOG" 2>&1 || rc=$?
+# restic exits 3 when it could not read *some* files but the snapshot was
+# still written. That is worth a warning, not a failure -- a nightly job that
+# hard-fails on one transiently-locked file stops being a backup.
+if [ "$rc" -ne 0 ]; then
+    [ "$rc" -eq 3 ] || fail "restic backup exited $rc"
+    log "WARN: restic exited 3 (some files unreadable); snapshot was written"
+fi
+
+# --- 3. retention ----------------------------------------------------------
+log "forget + prune"
+# --group-by host, NOT the default host+paths. With the default, changing the
+# PATHS list above starts a fresh retention group and the snapshots taken under
+# the old path list are kept forever -- every group gets its own
+# daily/weekly/monthly/yearly allowance. One host per repo, so one group.
+restic forget --prune \
+    --group-by host \
+    --tag "$HOST" \
+    --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 3 \
+    >>"$LOG" 2>&1 || fail "restic forget/prune failed"
+
+# --- 4. report -------------------------------------------------------------
+summary=$(grep -E '^(Added to the repository|processed|snapshot [0-9a-f]{8} saved)' "$LOG" | tail -3)
+stats=$(restic stats --mode raw-data latest 2>/dev/null | grep -E 'Total Size' || true)
+log "done"
+notify default "floppy_disk" "Backup OK on $HOST" "${summary:-(no summary)}"$'\n'"$stats"

hosts/mainframe/sbin/restic-notify-failure

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# OnFailure= handler for the restic units. Invoked by systemd with the failed
+# unit's name as $1.
+#
+# Why this exists: restic-backup notifies from inside the script, but
+# restic-check calls /usr/bin/restic directly and restic-seed had no notify at
+# all. Both could therefore fail completely silently -- and on 2026-08-21 the
+# seed did exactly that, dying on a dropped SFTP connection and leaving a stale
+# lock that then failed the Sunday check, with nothing said for three days.
+# "Silence is the alarm" only works if every unit can actually raise one.
+set -uo pipefail
+UNIT=${1:-unknown.service}
+NTFY_URL=https://ntfy.rcjohnstone.com/backup
+
+for f in /etc/ntfy/publish.env /home/connor/.config/ntfy/publish.env; do
+    [ -r "$f" ] && { NTFY_ENV=$f; break; }
+done
+[ -n "${NTFY_ENV:-}" ] || exit 0
+
+# PARSED, not sourced -- the bot password contains ` and &, so sourcing dies
+# with a syntax error. Same reason restic-backup's notify() uses sed.
+u=$(sed -n 's/^NTFY_USER=//p' "$NTFY_ENV" | head -1)
+p=$(sed -n 's/^NTFY_PASS=//p' "$NTFY_ENV" | head -1)
+[ -n "$u" ] && [ -n "$p" ] || exit 0
+
+HOST=$(uname -n); HOST=${HOST%%.*}
+RESULT=$(systemctl show -p Result --value "$UNIT" 2>/dev/null)
+BODY=$(printf '%s failed on %s (Result=%s)\n\n%s\n' \
+       "$UNIT" "$HOST" "$RESULT" \
+       "$(journalctl -u "$UNIT" -n 20 --no-pager -o cat 2>/dev/null | tail -c 1200)")
+
+curl -fsS --max-time 20 -u "$u:$p" \
+     -H "Title: $HOST: $UNIT FAILED" -H "Priority: urgent" -H "Tags: rotating_light" \
+     -d "$BODY" "$NTFY_URL" >/dev/null || true