dotfiles/common/claude/CLAUDE.md

# Working notes for Claude on mainframe

## LLM / inference stack

Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU
model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode on
top. **Before touching anything under `~/config/litellm/`, `~/.config/opencode/`,
or the model routing, read `~/guides/llm/robots.md`.** It carries the design
decisions, the measured data, and — more importantly — the specific wrong
conclusions this project has already reached and corrected. `~/guides/llm/README.md`
is the short human version.

The five things most likely to waste an hour if you don't know them:

1. **A config value is not applied until an invalid value produces an upstream
   400.** LiteLLM silently discards parameters it doesn't route (`store: false`
   bare in `litellm_params` returns 200 and never reaches OpenAI; it needs
   `extra_body`).

2. **Editing bind-mounted config does nothing without
   `sudo podman restart <container>`.** Changing an *env var* needs a full
   `sudo podman-compose up -d <svc>` — a restart is not enough. The reverse trap
   is worse and bites silently: **`podman-compose up -d` will NOT restart a
   container when only a bind-mounted file changed**, because the compose
   definition is unchanged. It prints the container name and does nothing. This
   applies to Grafana provisioning as much as to `litellm/config.yaml`.

3. **If every hosted model hangs (not errors) while the host has internet**, it's
   Docker: `dockerd` sets `ip filter` FORWARD to policy drop and kills all podman
   egress. DNS still resolves, which makes it look like a broken vendor. Fix:
   `sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the
   Gitea Actions runners need it.

3a. **`podman ps` saying `running` is not evidence a container is alive.**
   After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden,
   authelia, gitea, grafana, prometheus among them — reported `Up 9 hours`
   with a `State.Pid` that no longer existed. ntfy's healthcheck had failed
   539 consecutive times while `podman ps` called it up. Cause:
   `podman-restart.service` is `Type=oneshot` and forks a conmon per
   container into its own cgroup; ONE container failing to create fails the
   unit, and systemd's default `KillMode=control-group` then SIGKILLs every
   conmon it just started. conmon is what updates container state, so killed
   it updates nothing and the state stays `running` forever.

   Check with `bash ~/config/host/podman-restart-killmode.sh --check`, which
   compares each running container's pid against `/proc`. That script also
   applies the fix (`KillMode=process`), applied here 2026-09-19 — it takes
   effect on the next boot, so a reboot before then can still produce this.
   Recover one with `sudo podman restart -t 5 <name> && sudo podman start
   <name>`; the restart is EXPECTED to error with `conmon process killed`,
   and the start afterwards is what works.

   ⚠ **Do not test liveness with `kill -0 <pid>`.** The containers run as
   root, so as a normal user it fails with EPERM, not ESRCH — it reports
   every container dead and reads as a total outage. That misdiagnosis was
   made once already. `/proc/<pid>` needs no privilege.

4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an
   Anthropic refusal, a reasoning model eating its whole `max_tokens` budget
   before emitting text, or a dropped parameter. Local models do it too. Give
   reasoning models real headroom before concluding anything.

5. **OpenCode Zen and Z.ai GLM were deleted from the LiteLLM config on
   2026-11-06** — Zen's ToS block was never reversed, and Connor cancelled the
   GLM subscription. Don't add `zen/*` or `glm/*` back; the history (including
   the z.ai two-endpoint trap) is in `~/guides/llm/robots.md` section 8 and
   14.1b-k, and the pre-removal config is
   `~/config/litellm/config.yaml.pre-glm-zen-removal`.

6. **The OpenCode sidebar under-reports cost — badly.** It omits delegated child
   sessions, where most agent work happens: it showed ~$5.80 against a
   vendor-verified $17.39. Use `opencode stats` or the LiteLLM spend log.

7. **Open WebUI gives every model its full built-in tool suite by default**
   (`meta.builtinTools` defaults to True when absent). Models that are weak at
   tool discipline loop on `query_knowledge_files` and friends even with no
   knowledge base configured. Symptom: many round trips with 25-30 token
   completions. Disable per-model in Admin → Models; there is no global switch.

8. **On this box, local inference is RAM-bandwidth-bound, not VRAM-bound.**
   Decode speed is set by a model's *active* parameters, not its total, so an
   80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free
   VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking
   which wall you're actually against.

9. **OMO (oh-my-openagent) was removed from every OpenCode profile on
   2026-10-29.** It injected its own skills and prompts into non-OMO agents
   and used a lot of context. The `agent/*` and `personal/atlas-*` style
   aliases in LiteLLM were for OMO's roster and are unused by OpenCode now;
   OMO rewrote them at request time, so they 400 if called directly.

Cost work: read `model_group`, never `model`, to attribute a request. Cost is
driven by cache-read volume and context size far more than by model choice.

## The dotfiles repo and the fleet

`~/.dotfiles` is **one branch linked onto four machines** by dotbot.
There are deliberately no per-host branches: the layout this replaced was three
machines with two histories that had never met, and host layers cost a few small
files and remove the merge entirely.

| Host | Address | Role | Packages |
|---|---|---|---|
| `mainframe` | 10.0.0.2 | the server — containers, inference stack, Moonlight host | pacman + AUR |
| `nas` | 10.0.0.3 | bulk storage; NFS-exports `audio books docs downloads photo video`; own 03:30 backup run | pacman + AUR |
| `system76` | 10.0.0.4 | personal laptop | pacman + AUR |
| `sctfw004` | 10.0.0.8 | work laptop — `work_laptop` in ssh config, user `connorjohnstone` | apt + flatpak |

`sctfw004` is US-government adjacent: **OpenAI and local models only, never
Anthropic.** Don't propose an Anthropic-backed workflow for that machine.

Five things about this repo that will cost you an hour:

1. **The `.gitignore` is an allowlist, and it is load-bearing.** It opens with
   `**/*` and then names what is allowed back in. A new top-level directory is
   invisible to git until it is explicitly un-ignored, so "I added the file and
   git doesn't see it" is expected, not a bug — fix it by adding the `!` entry,
   never by relaxing the `**/*`. The denylist it replaced accumulated browser
   caches, mail caches and vendored binaries. It also re-excludes credential
   *shapes* (`**/credentials`, `**/*.env`, `**/hosts.yml`, `**/password`,
   `.secrets.zsh`); every one of those has held a live credential on these
   machines. Leave them alone.

2. **Secrets never live in a tracked file, and specifically not in `zshenv`.**
   `common/zsh/zshenv` ends by sourcing `~/.secrets.zsh`, which is untracked
   and mode 600 — `common/zsh/secrets.zsh.example` is the shape to copy. Keys
   for OpenCode are read out of `rbw` at launch and passed in, never written to
   disk. A new secret goes in `~/.secrets.zsh`, never anywhere under
   `~/.dotfiles`.

3. **Any unit that must run on exactly one machine carries
   `ConditionHost=<host>`.** One branch is linked everywhere, so an unguarded
   timer fires on all four. `inbox-tidy`, `movie-recs`, `rent-utilities` and
   `ddns` mutate a live mailbox, send notifications, send bills and update live
   DNS records, so they are guarded twice: the units exist only in
   `hosts/mainframe/systemd/` and never in `common/`, *and* each carries the
   condition. Genuinely per-machine things like `hypr_refresh` are correctly
   unguarded — don't "fix" them.

4. **`dots` is the one command that answers "is this machine clean".**
   Committed, pushed, and does the package manifest still match reality. It is
   wired into `zshrc` as `dots --nag`, which prints nothing when everything is
   clean. That silence is the point — a login banner that always fires is one
   you stop reading.

5. **`pkg-snapshot` writes what is installed into `packages/<host>.*`** —
   pacman, AUR, apt and flatpak as separate files. Run it after installing
   something worth keeping and commit the diff; the diff doubles as the log of
   what was added and when. `pkg-restore` goes the other way.

**This file is a symlink.** `~/.claude/CLAUDE.md` →
`~/.dotfiles/common/claude/CLAUDE.md`. Editing it edits the repo, so a change
here wants a commit and reaches every machine at the next `install`.

OpenCode reads this file too, as its global instructions, because
`~/.config/opencode/AGENTS.md` does not exist. OpenCode loads only the first
of those two that exists, so creating an AGENTS.md there silently drops this
file from every OpenCode session. Keep one file, and keep it here.

## Rust libraries

The crates here (`ordinary-diffeq`, `astrokit`) share a working rhythm. Follow it
in a new one.

1. **One coherent change at a time, and the commit is Connor's to make.** Finish
   the piece of work, leave it in the working tree, show what changed, and stop.
   Don't commit, push, tag or `cargo publish` unless the request says to. Tests
   passing is not the same as reviewed.

2. **Zero warnings, zero fmt diff, every time.** The bar is the whole tree, not
   the lines you touched:
   - `cargo fmt --check` with no output.
   - `cargo clippy --all-targets -- -D warnings`.
   - `cargo doc --no-deps` with no warnings at all.
   - `cargo test`, which runs the doctests too. Include the readme as the crate's
     front page with `#![doc = include_str!("../readme.md")]` so its examples are
     compiled and cannot drift from the API.
   - `cargo test --no-default-features` if the crate has feature flags.

   A warning that was already there still counts. Fix it in the same pass and say
   so. Three rustdoc warnings sat in `ordinary-diffeq` across four releases
   because each session read them as someone else's problem.

3. **Run `cargo fmt` and keep the tree clean.** If a repo isn't clean yet, clean
   it in its own commit rather than letting formatting ride along with a change —
   but clean it. Where rustfmt would wreck something laid out on purpose, a
   Butcher tableau or a table of interpolation coefficients, put
   `#[rustfmt::skip]` on that item with a one-line reason; `--check` honors it, so
   the tree stays clean and the table stays readable. Checking one file is
   `rustfmt --check --edition 2021 <file>`, because `cargo fmt --check -- <file>`
   ignores the file argument and does the whole workspace.

4. **Hooks, split by what they cost.** Measure before deciding where a check goes;
   on `ordinary-diffeq`'s warm tree `cargo fmt --check` is 0.1s, clippy 0.6s and
   `cargo test --lib` 0.6s, but the full `cargo test` is 16s because every doctest
   links its own binary. So `pre-commit` gets the first three and `pre-push` gets
   the doctests and `cargo doc`. Keep the benches and anything networked out of
   both — a hook that takes a minute is one that gets `--no-verify`. Git does not
   track where it looks for hooks, so commit them to `hooks/` and point each clone
   at it once with `git config core.hooksPath hooks`.

5. **Benchmarking, when a change could plausibly cost something.** criterion,
   and measure rather than assert — the method and mainframe's noise floor are
   in the `rust-crate-release` skill.
6. **In 0.x, a breaking change is a minor bump.** Changing a trait method
   signature took `ordinary-diffeq` from 0.5.0 to 0.6.0.

7. **The release process** — branch, bump `Cargo.toml` alone, merge `--no-ff`,
   annotated tag, publish. The exact sequence is in the `rust-crate-release`
   skill. A publish cannot be undone; a bad version can only be yanked.
8. **Each feature keeps a plan that outlives it, as an issue on the Gitea
   remote.** Don't create `roadmap/features/` files in a repo that doesn't
   already have them. Comment on the issue when the work lands: what was done,
   what was measured, what is still a known limitation, and which approach was
   tried first and why it was wrong. The last of those is what saves the next
   session. Leave the issue open until given permission to close it.

## General

- `~/config/` and `~/docs/` are deliberately untracked. Back up edits there by
  hand as `<file>.pre-<thing>` snapshots — the established convention, and the
  only backup those trees get. Inside a tracked repo git already is that
  backup, so a `.pre-*` file there is redundant (and `.gitignore` excludes the
  pattern anyway).
- `~/guides/` is tracked in the `~` repo and is where reference material goes,
  `~/.dotfiles` is its own repo. Check which one you are in before committing.
- Don't create project-level `CLAUDE.md` files in `~` — Connor keeps guidance at
  the `~/.claude` level and reference material in `~/guides/`.
- Measure before concluding, and when a previous conclusion turns out to have
  been reached from bad data, **record both the correction and why the original
  reasoning was unsound** — otherwise someone re-opens it.
- **US English spelling everywhere** — code, comments, docs, commit messages,
  prose. `meters`, `normalize`, `stabilize`, `recognize`, `license`, `modeling`,
  `maneuver`, `behavior`, `analyze`. British spellings keep slipping into
  generated text, so check before handing work over.
- **Plain prose, not persuasive.** Write the fact and stop. One line of comment
  where one line does it. Commit messages say what changed, not why it was a
  good idea. No bold for emphasis, no aphorisms, no "X is not Y" constructions,
  no sentence whose only job is to make a point land. Concrete examples that
  explain something are welcome; salesmanship is not.
- **No AI attribution in commit messages.** No `Co-Authored-By: Claude`, no
  session-link trailer, no model named as an author — subject and body only.
  This overrides the harness default, which asks for the trailer.

## Code comments

- Comment only where the why isn't obvious from the code. Never add a comment
  that restates what the code plainly does.
- A departure from how something was done earlier in the session, or in a
  previous session, does not by itself warrant a comment. Explain that in the
  chat, not in the code.
- When a comment is warranted, prefer one short line over a multi-line block.
- No semicolons or em-dashes in prose comments. Use more common punctuation.
- Deferred work always carries a literal `TODO:` marker so it can be grepped.
- Never describe previous behavior in a comment, even to justify the current
  design. If the design justifies a comment on its own merits, write that, but
  don't reference the old way. Git has the history.
- Default to no comment. Add one only when a reader would otherwise be misled
  or have real difficulty understanding the code.

## Process

- Explain what the problem is and what the plan is before starting to
  implement.
- Prefer the edit tool over scripts, sed or heredocs for changing files, so
  each change is legible. A script is fine where it is clearly more efficient,
  such as a rename or the same small change across many files. The OpenCode
  work profile is stricter about this and adds its own instructions.
- If recorded data exists and a hypothesis about a cause can be tested against
  it, test it. Be sure of a root cause before fixing, even when that takes
  longer.
- Clear long-running services and ports before starting your own, and clean up
  any server or similar long-running process you started.

## Shell tools

- Prefer `rg` over `grep` and `fd` over `find`. Both are installed on every
  machine, but on sctfw004 `fd` is Debian's `fdfind`.
- `grep` is aliased to `rg` in interactive zsh, and agent shells may or may not
  inherit that, so a grep command with GNU flags can fail in one shell and work
  in another. Use `rg` explicitly, or `command grep` when GNU grep is needed.