Two independent reasons the nightly backup was not producing a snapshot.
zstd refuses to overwrite its -o target. A dump that fails midway therefore
wedges the job permanently rather than just losing one night: pg_dumpall dies
(the immich db is not up yet right after a boot), zstd still writes its empty
frame, pipefail aborts before the mv, and the orphaned .tmp then makes every
later run fail instantly on "already exists; not overwritten". Three nights
were lost that way from 2026-09-15. The sqlite branch already rm -f'd its tmp;
these two did not. -f closes it.
sftpgo moved its state onto the /var/lib/sftpgo bind mount, so the old
/home/connor/docs/sftpgo.db only tripped the "db missing, skipping" WARN --
sftpgo went unbacked-up from 2026-09-09 while the run still reported success.
Left as a WARN deliberately: a decommissioned service should not break the
nightly run. Read the WARNs.
42c4277 split the ssh config into common plus a host layer and restored
system76's aliases verbatim, but stubbed nas and mainframe out as having no
machine-specific aliases. Those are the two machines that run restic, and
RESTIC_REPOSITORY is sftp:storagebox:/home/restic -- so the NAS backup failed
every night from 2026-09-15 with "Could not resolve hostname storagebox",
three nights with no offsite copy.
mainframe was unaffected only by accident: its job runs as root and reads
/root/.ssh/config, which dotbot does not manage. Worth remembering before
assuming an ssh change here reaches every backup.
sub2, not sub1 -- the hosts share one Storage Box account but are chrooted
into separate sub-account homes.
RECS was the absolute path /home/connor/.local/bin/movie_recs. That worked
while ~/.local/bin was a dotbot link into hosts/<host>/bin, but base.conf.yaml
now *creates* it as a real directory owned by the package managers, so the
movie_recs link went away with it. Every timer run since died on
FileNotFoundError before reaching LiteLLM, which is why the ntfy push went
quiet rather than reporting a failure.
The unit already invokes this script as %h/.dotfiles/hosts/mainframe/bin/...,
so deriving the sibling path from __file__ cannot drift from however the unit
is spelled, and does not care what ~/.local/bin becomes next.
The file covered the inference stack in detail and said almost nothing about
the repo it now lives in, so every session rediscovered the same handful of
conventions.
Adds the four hosts with their addresses, roles and package managers --
including that sctfw004 is US-government adjacent and stays on OpenAI and
local models only -- and the five things here that actually cost an hour: the
allowlist .gitignore, and why a new directory being invisible to git is
expected rather than broken; secrets living only in an untracked
~/.secrets.zsh and never in a tracked zshenv; ConditionHost as the guard that
keeps a one-machine timer from firing on all four; and dots / pkg-snapshot as
the maintenance pair, with the deliberate silence of dots --nag explained.
Also notes that ~/.claude/CLAUDE.md is a symlink into this repo, since
editing it silently edits tracked config, and sharpens the .pre-<thing> note:
that convention is for the untracked ~/config and ~/docs trees, where it is
the only backup those files get. Inside a tracked repo git already is the
backup and .gitignore excludes the pattern anyway.
eb6a42c set HEADLESS-1 from SUNSHINE_CLIENT_WIDTH/HEIGHT/FPS but never
actually ran: Sunshine execs a prep-cmd directly, with no shell, so ${...}
was passed through literally. swaymsg received "mode ${SUNSHINE_CLIENT_WIDTH}
x..." as its argument, exited 2, and Sunshine reports any failed prep-cmd to
the client as the one opaque "failed to start the specified application".
The variable names were right and Sunshine does export them; the shell was
the missing part, which is why upstream's own example wraps the command.
Verified live at 2560x1440@60 and 1920x1080@120.
Worth keeping for the next time this breaks: swaymsg exits 2 when the socket
is reachable and the command is rejected, and 1 when it cannot find the
socket at all. That distinction is what ruled out a SWAYSOCK problem and
confirmed the malformed mode string without guessing.
The work laptop drives two 3840x2160 panels, so a fixed 1920x1080 capture
was being upscaled 2x on the client — fine for Big Picture's large UI,
visibly soft for terminal text on the Desktop app.
A headless wlroots output has no physical size, so it can take any mode.
Both apps now set HEADLESS-1 to SUNSHINE_CLIENT_WIDTH/HEIGHT/FPS in their
prep-cmd and restore 1920x1080 on undo. SWAYSOCK joins the environment
import so Sunshine's prep-cmds can reach the compositor at all.
Audited every non-shell context in the repo — hypr, waybar, systemd,
desktop entries — for repo scripts named without a path. Two real hits
beyond the scratchpad bind:
hyprctl_modify_wkspc1 runs from exec-once at every Hyprland startup, so it
has been failing silently on both laptops since ~/.local/bin stopped being
a symlink into the repo. Nothing surfaces an exec-once that cannot find its
binary.
system76's playlistBuilder bind had the same bare hypr_workspace. Its inner
command already used `zsh -lc`, so `play` resolved; the outer call did not.
Scratchpad terminal. The original config called hypr_workspace by absolute
path; splitting the config into common and host layers I rewrote those to
bare names, assuming PATH would resolve them. It does not: Hyprland's exec
runs with whatever PATH the session inherited at startup, which here is the
system default — no ~/.local/bin, no repo directories. That was survivable
while ~/.local/bin was a symlink into the repo, but the cutover made it a
real directory and removed the 29 duplicated scripts, so the binding broke.
Repo scripts are now named via $bin, defined per host in local-env.conf
because the home directory differs.
Same commit fixes toggleFloating.sh, which the shared config referenced as
/home/connor/.config/... — correct on two machines, wrong on the work
laptop. Now ~.
rbw. The old install.conf.yaml linked ~/.config/rbw on every host; I put it
in the mainframe and system76 host configs and forgot the work laptop, so
rbw there had no config at all. Moved to base, where it was always meant to
be.
The stream rendered but ignored keyboard and mouse. Sunshine injects input
through uinput exclusively — it has no support for the Wayland virtual
keyboard/pointer protocols — so the compositor has to have a libinput
backend to see those devices. WLR_BACKENDS=headless creates only the
headless backend, so sway had zero input devices against the kernel's 24.
libinput needs a seat, and a systemd --user service belongs to no logind
session, so libseat needs seatd. seatd is now enabled on this host.
Its socket defaults to group `seat`, which does not help here: the user
manager's group list is fixed when it starts, Linger=yes means that is at
boot, and so adding connor to `seat` would not take effect until a reboot —
re-logging in changes nothing. The drop-in points seatd at `video`, a group
connor already holds for GPU access, which makes this deterministic instead
of dependent on group-propagation timing.
sway now reports 7 input devices including libvirtualhid Mouse and
Keyboard, which are Sunshine's.
hosts/mainframe/etc/ is a new tree for system-level config; install flags
it the same way it flags sbin.
The previous fix added DISPLAY to the import but left it as a separate
exec line. sway runs exec lines asynchronously and does not order them, so
Sunshine won the race every time — it started in the same second as the
compositor, before DISPLAY existed in the manager environment, and passed
no DISPLAY to the apps it launched. Steam still failed with "Unable to
open a connection to X" while Xwayland was up and the manager environment
looked correct, which is what made it confusing.
One exec with && instead of two.
The earlier verification was wrong for the same reason: I launched Steam
by hand with DISPLAY=:0 set, which tested Xwayland rather than the path
Sunshine actually uses. Re-verified properly by launching Steam in a shell
carrying only the environment read out of Sunshine's own /proc entry.
Two reasons both Sunshine apps streamed a blank screen.
Steam is an X client, and sway starts Xwayland lazily. DISPLAY therefore
did not exist at the moment the config imported the environment for
Sunshine, so Steam came up with "Unable to open a connection to X" and
rendered nothing. `xwayland force` starts it at compositor launch instead,
and DISPLAY joins the import.
The Desktop app was not broken at all — the desktop was genuinely empty.
swaybg was painting #101920 on an output with no windows and no bar, which
over a stream is indistinguishable from a failed capture. There are now
keybinds for a terminal and a launcher, and kitty starts with the session
so a working capture looks obviously working.
Verified: Xwayland on :0, DISPLAY exported, xdpyinfo and zenity connect,
and Steam Big Picture Mode renders as a window on HEADLESS-1.
Reaching the web UI from another machine sends an Origin header naming
this host rather than localhost, and Sunshine's CSRF check rejects it:
Web UI: [10.0.0.8] -- CSRF protection blocked request from origin:
https://10.0.0.2:47990
csrf_allowed_origins takes a comma-separated list of bare origins, NOT a
JSON array — brackets and quotes are rejected entry by entry, which is how
the first attempt failed. Both the IP and hostname spellings are listed
because the browser sends whichever was typed.
origin_web_ui_allowed=lan set explicitly alongside it, so the two settings
that gate remote access to the UI are stated in one place.
Moonlight went on the work laptop as a flatpak, which neither apt-mark nor
pacman can see — so a rebuild from the manifests would have silently left
it out. pkg-restore adds the flathub user remote before installing, since
Ubuntu ships flathub as a system remote that --user cannot resolve.
System-scope flatpaks are deliberately not recorded: those belong to
whoever administers the machine, not to these dotfiles.
Measured rather than assumed: the compositor and Sunshine together hold
220 MiB on the 3060 — sway ~112, Sunshine ~108. That card also runs the
local LLM, and with an 11.3 GB model resident on a 12 GB card the free
headroom goes from 992 MiB to 772. Losing a fifth of what is left matters
for KV cache growth, so leaving it running was the wrong default.
sway-headless and sunshine no longer start at boot. `stream on` brings them
up, `stream status` reports state and VRAM, and sunshine-idle-stop.timer
checks every five minutes whether any Moonlight client still holds a
connection to 47984/47989/48010 — stopping the stack after 15 idle minutes
so a forgotten session does not sit on memory the inference stack wants.
Note for future edits: `systemctl --user disable` on a unit dotbot has
symlinked into ~/.config/systemd/user deletes the symlink itself, because
systemd treats it as a linked unit. Re-run ./install after disabling one.
mainframe has no desktop and the only connected output hangs off the Intel
iGPU, while the RTX 3060 that encodes drives no monitor. Capturing the real
output would copy every frame between GPUs. So: a minimal sway session on
the headless wlroots backend, pinned to card0, with one 1920x1080 virtual
output for Sunshine to capture. Verified end to end — Sunshine finds
zwlr_screencopy_manager_v1 and creates h264_nvenc and hevc_nvenc.
(av1_nvenc fails because Ampere has no AV1 encoder; that is expected.)
Two things had to be fixed to get there. The installed sunshine-git was
built against boost 1.89 against a system now on 1.92 and could not load
five libraries — which is why the service had been enabled but never once
run. Replaced with sunshine-bin, which is current and prebuilt.
And Sunshine now gets a unit of ours rather than a drop-in: the source AUR
package ships sunshine.service while -bin ships
app-dev.lizardbyte.app.Sunshine.service, so a drop-in silently stops
applying the moment you switch between them.
apps.json replaces the stock file, whose "Low Res Desktop" entry shelled
out to xrandr and would have done nothing on Wayland.
Connor's call: the work laptop's commented-out copilot is the state he
wants on all four machines, so the host conditional added in 8ddbbe2 is
unnecessary machinery. Back to a plain plugin list.
zshenv is read by every zsh, so a nested shell prepended the repo's bin
directories a second time. The NAS showed each of them twice after one
`zsh -l -i`. typeset -U keeps the first occurrence, which is also the
ordering this file intends.
The NAS is a minimal Arch install with no inetutils, so hostname(1) does
not exist and install died on its first line of real work. uname -n is
POSIX and always present. The same assumption was in dots, pkg-snapshot,
pkg-restore and zshenv's fallback.
Taking the work laptop's init.lua as canonical carried its commented-out
copilot line to every machine. Copilot was enabled on mainframe, system76
and the NAS before the refit — only the work laptop had it off. Sharing the
file silently disabled it on the other three.
The plugin list is now built in a table and copilot appended unless the
host is sctfw004. Keyed on vim.fn.hostname() rather than $DOTFILES_HOST,
because nvim launched from a desktop entry does not inherit the shell's
environment.
Promoting the work laptop's copy to common/ carried its hardcoded fdfind,
which is the Debian/Ubuntu binary name. On Arch the binary is fd, so the
script would have failed on system76 — quietly, since it runs from a timer.
The work laptop's first install left five files dirty, all of them wallust
targets. Tracking generated colour files means each machine rewrites the
others' palette on every theme change — exactly the churn a shared repo
should not have.
hypr, waybar and swaync now get their colours as real files written into
real directories, so nothing is linked and nothing is tracked. kitty and
qutebrowser land inside directories this repo links wholesale, so those two
are ignored by path instead. swaync switched from a directory link to two
file links to make room for its generated colors.css.
Regeneration is automatic: hypr_refresh runs every 15 seconds on both
laptops, so a fresh machine has colours within a quarter of a minute.
Both laptops run hypr_refresh and their unit files had diverged. The work
laptop's is the refined one — Type=oneshot, which is correct for a script
that exits, a 15s interval with AccuracySec, and a comment explaining the
wallpaper-daemon guard. system76 had Type=simple with Restart=on-failure,
which respawns a oneshot script continuously. Shared version wins.
Three more units existed only on the work laptop and were untracked:
backlight-probe, kanshi and mopidy.
Two passes, both aimed at a consistent experience rather than a faithful
snapshot of each machine's accidents.
Standardisation. Audited every host-layer config present on more than one
machine and asked whether the difference was intent or drift:
- mpv was byte-identical; swayimg differed only because system76 still
had the shipped default and the work laptop had the real config; bob
differed by one setting the work laptop was simply missing.
- user-dirs.dirs disagreed with BOTH machines and with zshenv, which has
said $HOME/downloads all along. Standardised on that.
- mimeapps.list is now the union of both machines' associations.
- swaync's config and style are shared; only wallust's generated
colors.css stays per host, alongside hypr's and waybar's.
All of the above moved to common/.
Dropped as pure state rather than config: gtk-2.0 (file-chooser geometry),
pavucontrol.ini (window size), zoom.conf (a per-install random_key) and
zoomus.conf (panel widths). gtk-3.0 stays per host — its bookmarks encode
the two different home directories.
App config. GIMP, inkscape, FreeCAD, cura, OrcaSlicer, LightBurn and OBS
all hold real settings, so a curated subset of each is now tracked:
settings, keybindings and user content, but not session state, tool
options, device lists or window geometry. kicad is skipped — both library
tables hold a single entry, i.e. untouched defaults.
Content lives once in common/ even when only one machine has the app; the
host configs decide who gets the link. That is the distinction that lets
53% of link sources be shared while the NAS still does not get GIMP.
Two exclusions on credential grounds: obs-studio/plugin_config holds a
websocket server_password, and gh is dropped entirely because `gh auth
login` writes a token into hosts.yml — now also blocked by name.
Five directories swept in held no files — matplotlib, three gtk dirs and
one mpv. Git cannot represent an empty directory, so they existed on
mainframe where they were created but vanished on clone, and system76's
install failed with "Nonexistent target".
install now checks every link source exists before running dotbot, so this
surfaces on the machine that introduced it rather than the next one to
pull.
Reactive capture missed wallust entirely, so this is a systematic pass over
~/.config on all four machines: every entry that is not already a symlink,
judged on whether it is hand-authored config or app-generated state.
Shared and identical everywhere -> common/: lazygit, ncmpcpp, neofetch,
matplotlib. Everything else differs per machine and goes in
hosts/<host>/config/: gtk-2.0/3.0/4.0, mimeapps.list, user-dirs.dirs,
mpv, swaync, swayimg, bob, pavucontrol.ini, zoom; plus beets, bottom,
glow and mpd on system76; fastfetch, gh, uv, traggo, tiling-assistant and
xdg-desktop-portal on the work laptop; inbox-tidy and llmfit on mainframe.
Deliberately excluded, all verified to hold live credentials: the work
laptop's Claude OAuth tokens (~/.config/anthropic/credentials), the NAS
restic repo password and hetzner.env, subsonic passwords in subtui, stmp
and termsonic, mopidy, supersonic, the Proton bridge env and the LiteLLM
virtual keys. The ignore list now names those shapes directly — a bare
`password` file would not have been caught before.
Also excluded as generated state or too large to be meaningful: GIMP,
kicad, BambuStudio, OrcaSlicer, calibre, LightBurn, obs-studio, feishin,
libreoffice, VirtualBox, FortiClient; and mainframe's dead X11-era
leftovers (i3, openbox, lxpanel, lxsession, pcmanfm).
The router resolves it to 10.0.0.2 at the DNS level and the remote URL
already carries :2222, so the alias was redundant everywhere and wrong on
mainframe, where it pointed at localhost. Verified: connecting with
-F /dev/null succeeds from both mainframe and system76.
pkg-snapshot and dots used plain sort, which is locale-dependent: the same
package set ordered differently on two machines, so dots reported drift
that did not exist. A maintenance tool that always cries wolf is the exact
failure mode dots exists to avoid.
The laptops are on different Hyprland versions — system76 on 0.55.4,
the work laptop on 0.54.3 — and the option was removed in 0.55. Taking the
work laptop's config as canonical therefore introduced a config error on
system76 that did not exist before.
It is redundant either way: 0.54.3 reports the default as already enabled.
This is the first case of the shared config needing to tolerate two
compositor versions; it will not be the last.
Git reads both, and ~/.gitconfig wins. Linking only the XDG path left the
stale ~/.gitconfig in charge, so system76 kept signing commits as a
previous employer even after the cutover appeared to succeed — the symlink
was correct and had no effect.
Putting the whole ssh config in common/ was wrong twice over. It mapped
git.rcjohnstone.com to localhost:2222, which is only true on mainframe
where Gitea runs -- on system76 it broke git entirely. And it replaced a
77-line file holding 13 personal host aliases (storagebox, several
forwarded localhost ports, older work hosts) with a five-entry fleet list.
common/ssh/config now Includes ~/.ssh/config.local first, so host entries
win on the first-value-obtained rule, and carries only the fleet aliases.
Each host keeps its own list; system76's is restored verbatim.
The laptops are themed by wallust, whose config and five templates lived
only in ~/.config/wallust on each machine. A fresh install would have had
no colours at all: hyprland.conf sources colors.conf on its first line and
waybar's style.css imports colors.css on its first line, and nothing in
the repo generated either. Both generated files are now linked per host so
bootstrap works, and wallust writes back through the symlinks.
update_colors.sh was its predecessor and is now actively harmful: it
writes sway/config and wofi/style.css, neither of which exists on any
machine any more, and it regenerates waybar/style.css from a template that
was never carried across -- so running it would have overwritten the
shared Hyprland waybar style with sway-era output.
common/colors (255 theme files, 1.1 MB) was only ever its input.
The earlier survey grepped for the exact package name and concluded the
binding in hyprland.conf was dead. It is not: system76 has it from AUR as
networkmanager-dmenu-git, and the work laptop carries the upstream script
in its own bin. The $mainMod+W binding already works on both.
Adding the repo package to system76.pacman would have made pkg-restore
install it alongside the AUR one — both ship /usr/bin/networkmanager_dmenu,
so pacman would have refused with a file conflict.
dotbot does not carry a `defaults` block between config files, so the
host configs ran without force/relink and refused to replace paths that
already existed as real files — the opencode profiles, the Claude files,
the sunshine config and the old systemd units all failed to link. Each
host config now restates the block.
ddns.timer used OnCalendar=*-*-*/2, which systemd rejects: a day-of-month
step needs an explicit start value. The retired cron line was
`0 2 */2 * *`, so the correct spelling is *-*-1/2.
Replaces the previous repo, which had split into two histories that never
met (mainframe on a dead GitLab remote, the laptops on Gitea) with 146
dirty files across three machines and the NAS never enrolled at all.
Branch-per-machine is gone. One main, with host differences expressed as
small files under hosts/<hostname>/ rather than as branches, so there is
nothing to merge. The reconciled zsh layer reduces 15-33 line forks to
1-7 effective lines per host; distro differences (oh-my-zsh prefix,
syntax-highlighting path, fd vs fdfind) are probed in common/ instead.
Fresh history: the old one carried six plaintext credentials, 45 MB of
mail caches, browser caches and vendored binaries. 5,096 tracked files
and 144 MB become 462 files and 2.6 MB. The .gitignore is now an
allowlist, which is what keeps that true.
Root cause of the rot: ~/.local/bin was a symlink to scripts/ with GOPATH
inside it, so every go install wrote into version control (2.2 GB on the
work laptop). PATH now points at the repo instead of the reverse.
Also: Hyprland replaces sway and is sourced in two halves so $browser is
defined before use; singleton automations carry ConditionHost= alongside
host-layer-only placement; ddns moves from cron to a guarded timer;
package manifests and pkg-snapshot/pkg-restore replace the X11-era
install_scripts/; networkmanager-dmenu added to system76 (the binding
always existed, the package never did).
connor
Connor Johnstone