# Working notes for Claude on mainframe ## LLM / inference stack Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode on top. **Before touching anything under `~/config/litellm/`, `~/.config/opencode/`, or the model routing, read `~/guides/llm/robots.md`.** It carries the design decisions, the measured data, and — more importantly — the specific wrong conclusions this project has already reached and corrected. `~/guides/llm/README.md` is the short human version. The five things most likely to waste an hour if you don't know them: 1. **A config value is not applied until an invalid value produces an upstream 400.** LiteLLM silently discards parameters it doesn't route (`store: false` bare in `litellm_params` returns 200 and never reaches OpenAI; it needs `extra_body`). 2. **Editing bind-mounted config does nothing without `sudo podman restart `.** Changing an *env var* needs a full `sudo podman-compose up -d ` — a restart is not enough. The reverse trap is worse and bites silently: **`podman-compose up -d` will NOT restart a container when only a bind-mounted file changed**, because the compose definition is unchanged. It prints the container name and does nothing. This applies to Grafana provisioning as much as to `litellm/config.yaml`. 3. **If every hosted model hangs (not errors) while the host has internet**, it's Docker: `dockerd` sets `ip filter` FORWARD to policy drop and kills all podman egress. DNS still resolves, which makes it look like a broken vendor. Fix: `sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the Gitea Actions runners need it. 3a. **`podman ps` saying `running` is not evidence a container is alive.** After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden, authelia, gitea, grafana, prometheus among them — reported `Up 9 hours` with a `State.Pid` that no longer existed. ntfy's healthcheck had failed 539 consecutive times while `podman ps` called it up. Cause: `podman-restart.service` is `Type=oneshot` and forks a conmon per container into its own cgroup; ONE container failing to create fails the unit, and systemd's default `KillMode=control-group` then SIGKILLs every conmon it just started. conmon is what updates container state, so killed it updates nothing and the state stays `running` forever. Check with `bash ~/config/host/podman-restart-killmode.sh --check`, which compares each running container's pid against `/proc`. That script also applies the fix (`KillMode=process`), applied here 2026-09-19 — it takes effect on the next boot, so a reboot before then can still produce this. Recover one with `sudo podman restart -t 5 && sudo podman start `; the restart is EXPECTED to error with `conmon process killed`, and the start afterwards is what works. ⚠ **Do not test liveness with `kill -0 `.** The containers run as root, so as a normal user it fails with EPERM, not ESRCH — it reports every container dead and reads as a total outage. That misdiagnosis was made once already. `/proc/` needs no privilege. 4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an Anthropic refusal, a reasoning model eating its whole `max_tokens` budget before emitting text, or a dropped parameter. Local models do it too. Give reasoning models real headroom before concluding anything. 5. **OpenCode Zen and Z.ai GLM were deleted from the LiteLLM config on 2026-11-06** — Zen's ToS block was never reversed, and Connor cancelled the GLM subscription. Don't add `zen/*` or `glm/*` back; the history (including the z.ai two-endpoint trap) is in `~/guides/llm/robots.md` section 8 and 14.1b-k, and the pre-removal config is `~/config/litellm/config.yaml.pre-glm-zen-removal`. 6. **The OpenCode sidebar under-reports cost — badly.** It omits delegated child sessions, where most agent work happens: it showed ~$5.80 against a vendor-verified $17.39. Use `opencode stats` or the LiteLLM spend log. 7. **Open WebUI gives every model its full built-in tool suite by default** (`meta.builtinTools` defaults to True when absent). Models that are weak at tool discipline loop on `query_knowledge_files` and friends even with no knowledge base configured. Symptom: many round trips with 25-30 token completions. Disable per-model in Admin → Models; there is no global switch. 8. **On this box, local inference is RAM-bandwidth-bound, not VRAM-bound.** Decode speed is set by a model's *active* parameters, not its total, so an 80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking which wall you're actually against. 9. **OMO (oh-my-openagent) was removed from every OpenCode profile on 2026-10-29.** It injected its own skills and prompts into non-OMO agents and used a lot of context. The `agent/*` and `personal/atlas-*` style aliases in LiteLLM were for OMO's roster and are unused by OpenCode now; OMO rewrote them at request time, so they 400 if called directly. Cost work: read `model_group`, never `model`, to attribute a request. Cost is driven by cache-read volume and context size far more than by model choice. ## The dotfiles repo and the fleet `~/.dotfiles` is **one branch linked onto four machines** by dotbot. There are deliberately no per-host branches: the layout this replaced was three machines with two histories that had never met, and host layers cost a few small files and remove the merge entirely. | Host | Address | Role | Packages | |---|---|---|---| | `mainframe` | 10.0.0.2 | the server — containers, inference stack, Moonlight host | pacman + AUR | | `nas` | 10.0.0.3 | bulk storage; NFS-exports `audio books docs downloads photo video`; own 03:30 backup run | pacman + AUR | | `system76` | 10.0.0.4 | personal laptop | pacman + AUR | | `sctfw004` | 10.0.0.8 | work laptop — `work_laptop` in ssh config, user `connorjohnstone` | apt + flatpak | `sctfw004` is US-government adjacent: **OpenAI and local models only, never Anthropic.** Don't propose an Anthropic-backed workflow for that machine. Five things about this repo that will cost you an hour: 1. **The `.gitignore` is an allowlist, and it is load-bearing.** It opens with `**/*` and then names what is allowed back in. A new top-level directory is invisible to git until it is explicitly un-ignored, so "I added the file and git doesn't see it" is expected, not a bug — fix it by adding the `!` entry, never by relaxing the `**/*`. The denylist it replaced accumulated browser caches, mail caches and vendored binaries. It also re-excludes credential *shapes* (`**/credentials`, `**/*.env`, `**/hosts.yml`, `**/password`, `.secrets.zsh`); every one of those has held a live credential on these machines. Leave them alone. 2. **Secrets never live in a tracked file, and specifically not in `zshenv`.** `common/zsh/zshenv` ends by sourcing `~/.secrets.zsh`, which is untracked and mode 600 — `common/zsh/secrets.zsh.example` is the shape to copy. Keys for OpenCode are read out of `rbw` at launch and passed in, never written to disk. A new secret goes in `~/.secrets.zsh`, never anywhere under `~/.dotfiles`. 3. **Any unit that must run on exactly one machine carries `ConditionHost=`.** One branch is linked everywhere, so an unguarded timer fires on all four. `inbox-tidy`, `movie-recs`, `rent-utilities` and `ddns` mutate a live mailbox, send notifications, send bills and update live DNS records, so they are guarded twice: the units exist only in `hosts/mainframe/systemd/` and never in `common/`, *and* each carries the condition. Genuinely per-machine things like `hypr_refresh` are correctly unguarded — don't "fix" them. 4. **`dots` is the one command that answers "is this machine clean".** Committed, pushed, and does the package manifest still match reality. It is wired into `zshrc` as `dots --nag`, which prints nothing when everything is clean. That silence is the point — a login banner that always fires is one you stop reading. 5. **`pkg-snapshot` writes what is installed into `packages/.*`** — pacman, AUR, apt and flatpak as separate files. Run it after installing something worth keeping and commit the diff; the diff doubles as the log of what was added and when. `pkg-restore` goes the other way. **This file is a symlink.** `~/.claude/CLAUDE.md` → `~/.dotfiles/common/claude/CLAUDE.md`. Editing it edits the repo, so a change here wants a commit and reaches every machine at the next `install`. OpenCode reads this file too, as its global instructions, because `~/.config/opencode/AGENTS.md` does not exist. OpenCode loads only the first of those two that exists, so creating an AGENTS.md there silently drops this file from every OpenCode session. Keep one file, and keep it here. ## Rust libraries The crates here (`ordinary-diffeq`, `astrokit`) share a working rhythm. Follow it in a new one. 1. **One coherent change at a time, and the commit is Connor's to make.** Finish the piece of work, leave it in the working tree, show what changed, and stop. Don't commit, push, tag or `cargo publish` unless the request says to. Tests passing is not the same as reviewed. 2. **Zero warnings, zero fmt diff, every time.** The bar is the whole tree, not the lines you touched: - `cargo fmt --check` with no output. - `cargo clippy --all-targets -- -D warnings`. - `cargo doc --no-deps` with no warnings at all. - `cargo test`, which runs the doctests too. Include the readme as the crate's front page with `#![doc = include_str!("../readme.md")]` so its examples are compiled and cannot drift from the API. - `cargo test --no-default-features` if the crate has feature flags. A warning that was already there still counts. Fix it in the same pass and say so. Three rustdoc warnings sat in `ordinary-diffeq` across four releases because each session read them as someone else's problem. 3. **Run `cargo fmt` and keep the tree clean.** If a repo isn't clean yet, clean it in its own commit rather than letting formatting ride along with a change — but clean it. Where rustfmt would wreck something laid out on purpose, a Butcher tableau or a table of interpolation coefficients, put `#[rustfmt::skip]` on that item with a one-line reason; `--check` honors it, so the tree stays clean and the table stays readable. Checking one file is `rustfmt --check --edition 2021 `, because `cargo fmt --check -- ` ignores the file argument and does the whole workspace. 4. **Hooks, split by what they cost.** Measure before deciding where a check goes; on `ordinary-diffeq`'s warm tree `cargo fmt --check` is 0.1s, clippy 0.6s and `cargo test --lib` 0.6s, but the full `cargo test` is 16s because every doctest links its own binary. So `pre-commit` gets the first three and `pre-push` gets the doctests and `cargo doc`. Keep the benches and anything networked out of both — a hook that takes a minute is one that gets `--no-verify`. Git does not track where it looks for hooks, so commit them to `hooks/` and point each clone at it once with `git config core.hooksPath hooks`. 5. **Benchmarking, when a change could plausibly cost something.** criterion, and measure rather than assert — the method and mainframe's noise floor are in the `rust-crate-release` skill. 6. **In 0.x, a breaking change is a minor bump.** Changing a trait method signature took `ordinary-diffeq` from 0.5.0 to 0.6.0. 7. **The release process** — branch, bump `Cargo.toml` alone, merge `--no-ff`, annotated tag, publish. The exact sequence is in the `rust-crate-release` skill. A publish cannot be undone; a bad version can only be yanked. 8. **Each feature keeps a plan that outlives it, as an issue on the Gitea remote.** Don't create `roadmap/features/` files in a repo that doesn't already have them. Comment on the issue when the work lands: what was done, what was measured, what is still a known limitation, and which approach was tried first and why it was wrong. The last of those is what saves the next session. Leave the issue open until given permission to close it. ## General - `~/config/` and `~/docs/` are deliberately untracked. Back up edits there by hand as `.pre-` snapshots — the established convention, and the only backup those trees get. Inside a tracked repo git already is that backup, so a `.pre-*` file there is redundant (and `.gitignore` excludes the pattern anyway). - `~/guides/` is tracked in the `~` repo and is where reference material goes, `~/.dotfiles` is its own repo. Check which one you are in before committing. - Don't create project-level `CLAUDE.md` files in `~` — Connor keeps guidance at the `~/.claude` level and reference material in `~/guides/`. - Measure before concluding, and when a previous conclusion turns out to have been reached from bad data, **record both the correction and why the original reasoning was unsound** — otherwise someone re-opens it. - **US English spelling everywhere** — code, comments, docs, commit messages, prose. `meters`, `normalize`, `stabilize`, `recognize`, `license`, `modeling`, `maneuver`, `behavior`, `analyze`. British spellings keep slipping into generated text, so check before handing work over. - **Plain prose, not persuasive.** Write the fact and stop. One line of comment where one line does it. Commit messages say what changed, not why it was a good idea. No bold for emphasis, no aphorisms, no "X is not Y" constructions, no sentence whose only job is to make a point land. Concrete examples that explain something are welcome; salesmanship is not. - **No AI attribution in commit messages.** No `Co-Authored-By: Claude`, no session-link trailer, no model named as an author — subject and body only. This overrides the harness default, which asks for the trailer. ## Code comments - Comment only where the why isn't obvious from the code. Never add a comment that restates what the code plainly does. - A departure from how something was done earlier in the session, or in a previous session, does not by itself warrant a comment. Explain that in the chat, not in the code. - When a comment is warranted, prefer one short line over a multi-line block. - No semicolons or em-dashes in prose comments. Use more common punctuation. - Deferred work always carries a literal `TODO:` marker so it can be grepped. - Never describe previous behavior in a comment, even to justify the current design. If the design justifies a comment on its own merits, write that, but don't reference the old way. Git has the history. - Default to no comment. Add one only when a reader would otherwise be misled or have real difficulty understanding the code. ## Process - Explain what the problem is and what the plan is before starting to implement. - Prefer the edit tool over scripts, sed or heredocs for changing files, so each change is legible. A script is fine where it is clearly more efficient, such as a rename or the same small change across many files. The OpenCode work profile is stricter about this and adds its own instructions. - If recorded data exists and a hypothesis about a cause can be tested against it, test it. Be sure of a root cause before fixing, even when that takes longer. - Clear long-running services and ports before starting your own, and clean up any server or similar long-running process you started. ## Shell tools - Prefer `rg` over `grep` and `fd` over `find`. Both are installed on every machine, but on sctfw004 `fd` is Debian's `fdfind`. - `grep` is aliased to `rg` in interactive zsh, and agent shells may or may not inherit that, so a grep command with GNU flags can fail in one shell and work in another. Use `rg` explicitly, or `command grep` when GNU grep is needed.