Working notes for Claude on mainframe

LLM / inference stack

Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode on top. Before touching anything under ~/config/litellm/, ~/.config/opencode/, or the model routing, read ~/guides/llm/robots.md. It carries the design decisions, the measured data, and — more importantly — the specific wrong conclusions this project has already reached and corrected. ~/guides/llm/README.md is the short human version.

The five things most likely to waste an hour if you don't know them:

A config value is not applied until an invalid value produces an upstream 400. LiteLLM silently discards parameters it doesn't route (store: false bare in litellm_params returns 200 and never reaches OpenAI; it needs extra_body).
Editing bind-mounted config does nothing without sudo podman restart <container>. Changing an env var needs a full sudo podman-compose up -d <svc> — a restart is not enough. The reverse trap is worse and bites silently: podman-compose up -d will NOT restart a container when only a bind-mounted file changed, because the compose definition is unchanged. It prints the container name and does nothing. This applies to Grafana provisioning as much as to litellm/config.yaml.
If every hosted model hangs (not errors) while the host has internet, it's Docker: dockerd sets ip filter FORWARD to policy drop and kills all podman egress. DNS still resolves, which makes it look like a broken vendor. Fix: sudo bash ~/config/host/podman-docker-forward.sh. Don't remove Docker — the Gitea Actions runners need it.

3a. podman ps saying running is not evidence a container is alive. After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden, authelia, gitea, grafana, prometheus among them — reported Up 9 hours with a State.Pid that no longer existed. ntfy's healthcheck had failed 539 consecutive times while podman ps called it up. Cause: podman-restart.service is Type=oneshot and forks a conmon per container into its own cgroup; ONE container failing to create fails the unit, and systemd's default KillMode=control-group then SIGKILLs every conmon it just started. conmon is what updates container state, so killed it updates nothing and the state stays running forever.

Check with bash ~/config/host/podman-restart-killmode.sh --check, which compares each running container's pid against /proc. That script also applies the fix (KillMode=process), applied here 2026-09-19 — it takes effect on the next boot, so a reboot before then can still produce this. Recover one with sudo podman restart -t 5 <name> && sudo podman start <name>; the restart is EXPECTED to error with conmon process killed, and the start afterwards is what works.

⚠ Do not test liveness with kill -0 <pid>. The containers run as root, so as a normal user it fails with EPERM, not ESRCH — it reports every container dead and reads as a total outage. That misdiagnosis was made once already. /proc/<pid> needs no privilege.

HTTP 200 with empty content is not necessarily a bug. Three causes: an Anthropic refusal, a reasoning model eating its whole max_tokens budget before emitting text, or a dropped parameter. Local models do it too. Give reasoning models real headroom before concluding anything.
OpenCode Zen and Z.ai GLM were deleted from the LiteLLM config on 2026-11-06 — Zen's ToS block was never reversed, and Connor cancelled the GLM subscription. Don't add zen/* or glm/* back; the history (including the z.ai two-endpoint trap) is in ~/guides/llm/robots.md section 8 and 14.1b-k, and the pre-removal config is ~/config/litellm/config.yaml.pre-glm-zen-removal.
The OpenCode sidebar under-reports cost — badly. It omits delegated child sessions, where most agent work happens: it showed ~$5.80 against a vendor-verified $17.39. Use opencode stats or the LiteLLM spend log.
Open WebUI gives every model its full built-in tool suite by default (meta.builtinTools defaults to True when absent). Models that are weak at tool discipline loop on query_knowledge_files and friends even with no knowledge base configured. Symptom: many round trips with 25-30 token completions. Disable per-model in Admin → Models; there is no global switch.
On this box, local inference is RAM-bandwidth-bound, not VRAM-bound. Decode speed is set by a model's active parameters, not its total, so an 80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking which wall you're actually against.
OMO (oh-my-openagent) was removed from every OpenCode profile on 2026-10-29. It injected its own skills and prompts into non-OMO agents and used a lot of context. The agent/* and personal/atlas-* style aliases in LiteLLM were for OMO's roster and are unused by OpenCode now; OMO rewrote them at request time, so they 400 if called directly.

Cost work: read model_group, never model, to attribute a request. Cost is driven by cache-read volume and context size far more than by model choice.

The dotfiles repo and the fleet

~/.dotfiles is one branch linked onto four machines by dotbot. There are deliberately no per-host branches: the layout this replaced was three machines with two histories that had never met, and host layers cost a few small files and remove the merge entirely.

Host	Address	Role	Packages
`mainframe`	10.0.0.2	the server — containers, inference stack, Moonlight host	pacman + AUR
`nas`	10.0.0.3	bulk storage; NFS-exports `audio books docs downloads photo video`; own 03:30 backup run	pacman + AUR
`system76`	10.0.0.4	personal laptop	pacman + AUR
`sctfw004`	10.0.0.8	work laptop — `work_laptop` in ssh config, user `connorjohnstone`	apt + flatpak

sctfw004 is US-government adjacent: OpenAI and local models only, never Anthropic. Don't propose an Anthropic-backed workflow for that machine.

Five things about this repo that will cost you an hour:

The .gitignore is an allowlist, and it is load-bearing. It opens with **/* and then names what is allowed back in. A new top-level directory is invisible to git until it is explicitly un-ignored, so "I added the file and git doesn't see it" is expected, not a bug — fix it by adding the ! entry, never by relaxing the **/*. The denylist it replaced accumulated browser caches, mail caches and vendored binaries. It also re-excludes credential shapes (**/credentials, **/*.env, **/hosts.yml, **/password, .secrets.zsh); every one of those has held a live credential on these machines. Leave them alone.
Secrets never live in a tracked file, and specifically not in zshenv. common/zsh/zshenv ends by sourcing ~/.secrets.zsh, which is untracked and mode 600 — common/zsh/secrets.zsh.example is the shape to copy. Keys for OpenCode are read out of rbw at launch and passed in, never written to disk. A new secret goes in ~/.secrets.zsh, never anywhere under ~/.dotfiles.
Any unit that must run on exactly one machine carries ConditionHost=<host>. One branch is linked everywhere, so an unguarded timer fires on all four. inbox-tidy, movie-recs, rent-utilities and ddns mutate a live mailbox, send notifications, send bills and update live DNS records, so they are guarded twice: the units exist only in hosts/mainframe/systemd/ and never in common/, and each carries the condition. Genuinely per-machine things like hypr_refresh are correctly unguarded — don't "fix" them.
dots is the one command that answers "is this machine clean". Committed, pushed, and does the package manifest still match reality. It is wired into zshrc as dots --nag, which prints nothing when everything is clean. That silence is the point — a login banner that always fires is one you stop reading.
pkg-snapshot writes what is installed into packages/<host>.* — pacman, AUR, apt and flatpak as separate files. Run it after installing something worth keeping and commit the diff; the diff doubles as the log of what was added and when. pkg-restore goes the other way.

This file is a symlink. ~/.claude/CLAUDE.md → ~/.dotfiles/common/claude/CLAUDE.md. Editing it edits the repo, so a change here wants a commit and reaches every machine at the next install.

OpenCode reads this file too, as its global instructions, because ~/.config/opencode/AGENTS.md does not exist. OpenCode loads only the first of those two that exists, so creating an AGENTS.md there silently drops this file from every OpenCode session. Keep one file, and keep it here.

Rust libraries

The crates here (ordinary-diffeq, astrokit) share a working rhythm. Follow it in a new one.

One coherent change at a time, and the commit is Connor's to make. Finish the piece of work, leave it in the working tree, show what changed, and stop. Don't commit, push, tag or cargo publish unless the request says to. Tests passing is not the same as reviewed.
Zero warnings, zero fmt diff, every time. The bar is the whole tree, not the lines you touched:
- cargo fmt --check with no output.
- cargo clippy --all-targets -- -D warnings.
- cargo doc --no-deps with no warnings at all.
- cargo test, which runs the doctests too. Include the readme as the crate's front page with #![doc = include_str!("../readme.md")] so its examples are compiled and cannot drift from the API.
- cargo test --no-default-features if the crate has feature flags.
A warning that was already there still counts. Fix it in the same pass and say so. Three rustdoc warnings sat in ordinary-diffeq across four releases because each session read them as someone else's problem.
Run cargo fmt and keep the tree clean. If a repo isn't clean yet, clean it in its own commit rather than letting formatting ride along with a change — but clean it. Where rustfmt would wreck something laid out on purpose, a Butcher tableau or a table of interpolation coefficients, put #[rustfmt::skip] on that item with a one-line reason; --check honors it, so the tree stays clean and the table stays readable. Checking one file is rustfmt --check --edition 2021 <file>, because cargo fmt --check -- <file> ignores the file argument and does the whole workspace.
Hooks, split by what they cost. Measure before deciding where a check goes; on ordinary-diffeq's warm tree cargo fmt --check is 0.1s, clippy 0.6s and cargo test --lib 0.6s, but the full cargo test is 16s because every doctest links its own binary. So pre-commit gets the first three and pre-push gets the doctests and cargo doc. Keep the benches and anything networked out of both — a hook that takes a minute is one that gets --no-verify. Git does not track where it looks for hooks, so commit them to hooks/ and point each clone at it once with git config core.hooksPath hooks.
Benchmarking, when a change could plausibly cost something. criterion, and measure rather than assert — the method and mainframe's noise floor are in the rust-crate-release skill.
In 0.x, a breaking change is a minor bump. Changing a trait method signature took ordinary-diffeq from 0.5.0 to 0.6.0.
The release process — branch, bump Cargo.toml alone, merge --no-ff, annotated tag, publish. The exact sequence is in the rust-crate-release skill. A publish cannot be undone; a bad version can only be yanked.
Each feature keeps a plan that outlives it, as an issue on the Gitea remote. Don't create roadmap/features/ files in a repo that doesn't already have them. Comment on the issue when the work lands: what was done, what was measured, what is still a known limitation, and which approach was tried first and why it was wrong. The last of those is what saves the next session. Leave the issue open until given permission to close it.

General

~/config/ and ~/docs/ are deliberately untracked. Back up edits there by hand as <file>.pre-<thing> snapshots — the established convention, and the only backup those trees get. Inside a tracked repo git already is that backup, so a .pre-* file there is redundant (and .gitignore excludes the pattern anyway).
~/guides/ is tracked in the ~ repo and is where reference material goes, ~/.dotfiles is its own repo. Check which one you are in before committing.
Don't create project-level CLAUDE.md files in ~ — Connor keeps guidance at the ~/.claude level and reference material in ~/guides/.
Measure before concluding, and when a previous conclusion turns out to have been reached from bad data, record both the correction and why the original reasoning was unsound — otherwise someone re-opens it.
US English spelling everywhere — code, comments, docs, commit messages, prose. meters, normalize, stabilize, recognize, license, modeling, maneuver, behavior, analyze. British spellings keep slipping into generated text, so check before handing work over.
Plain prose, not persuasive. Write the fact and stop. One line of comment where one line does it. Commit messages say what changed, not why it was a good idea. No bold for emphasis, no aphorisms, no "X is not Y" constructions, no sentence whose only job is to make a point land. Concrete examples that explain something are welcome; salesmanship is not.
No AI attribution in commit messages. No Co-Authored-By: Claude, no session-link trailer, no model named as an author — subject and body only. This overrides the harness default, which asks for the trailer.

Code comments

Comment only where the why isn't obvious from the code. Never add a comment that restates what the code plainly does.
A departure from how something was done earlier in the session, or in a previous session, does not by itself warrant a comment. Explain that in the chat, not in the code.
When a comment is warranted, prefer one short line over a multi-line block.
No semicolons or em-dashes in prose comments. Use more common punctuation.
Deferred work always carries a literal TODO: marker so it can be grepped.
Never describe previous behavior in a comment, even to justify the current design. If the design justifies a comment on its own merits, write that, but don't reference the old way. Git has the history.
Default to no comment. Add one only when a reader would otherwise be misled or have real difficulty understanding the code.

Process

Explain what the problem is and what the plan is before starting to implement.
Prefer the edit tool over scripts, sed or heredocs for changing files, so each change is legible. A script is fine where it is clearly more efficient, such as a rename or the same small change across many files. The OpenCode work profile is stricter about this and adds its own instructions.
If recorded data exists and a hypothesis about a cause can be tested against it, test it. Be sure of a root cause before fixing, even when that takes longer.
Clear long-running services and ports before starting your own, and clean up any server or similar long-running process you started.

Shell tools

Prefer rg over grep and fd over find. Both are installed on every machine, but on sctfw004 fd is Debian's fdfind.
grep is aliased to rg in interactive zsh, and agent shells may or may not inherit that, so a grep command with GNU flags can fail in one shell and work in another. Use rg explicitly, or command grep when GNU grep is needed.

16 KiB Raw Blame History