dotfiles/common/claude/CLAUDE.md

Working notes for Claude on mainframe

LLM / inference stack

Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode + OMO on top. Before touching anything under ~/config/litellm/, ~/.config/opencode/, or the model routing, read ~/guides/llm/robots.md. It carries the design decisions, the measured data, and — more importantly — the specific wrong conclusions this project has already reached and corrected. ~/guides/llm/README.md is the short human version.

The five things most likely to waste an hour if you don't know them:

1. A config value is not applied until an invalid value produces an upstream 400. LiteLLM silently discards parameters it doesn't route (store: false bare in litellm_params returns 200 and never reaches OpenAI; it needs extra_body).

2. Editing bind-mounted config does nothing without sudo podman restart <container>. Changing an env var needs a full sudo podman-compose up -d <svc> — a restart is not enough. The reverse trap is worse and bites silently: podman-compose up -d will NOT restart a container when only a bind-mounted file changed, because the compose definition is unchanged. It prints the container name and does nothing. This applies to Grafana provisioning as much as to litellm/config.yaml.

3. If every hosted model hangs (not errors) while the host has internet, it's Docker: dockerd sets ip filter FORWARD to policy drop and kills all podman egress. DNS still resolves, which makes it look like a broken vendor. Fix: sudo bash ~/config/host/podman-docker-forward.sh. Don't remove Docker — the Gitea Actions runners need it.

3a. podman ps saying running is not evidence a container is alive. After a reboot on 2026-09-19, eleven containers — ntfy, bitwarden, authelia, gitea, grafana, prometheus among them — reported Up 9 hours with a State.Pid that no longer existed. ntfy's healthcheck had failed 539 consecutive times while podman ps called it up. Cause: podman-restart.service is Type=oneshot and forks a conmon per container into its own cgroup; ONE container failing to create fails the unit, and systemd's default KillMode=control-group then SIGKILLs every conmon it just started. conmon is what updates container state, so killed it updates nothing and the state stays running forever.

Check with bash ~/config/host/podman-restart-killmode.sh --check, which compares each running container's pid against /proc. That script also applies the fix (KillMode=process), applied here 2026-09-19 — it takes effect on the next boot, so a reboot before then can still produce this. Recover one with sudo podman restart -t 5 <name> && sudo podman start <name>; the restart is EXPECTED to error with conmon process killed, and the start afterwards is what works.

⚠ Do not test liveness with kill -0 <pid>. The containers run as root, so as a normal user it fails with EPERM, not ESRCH — it reports every container dead and reads as a total outage. That misdiagnosis was made once already. /proc/<pid> needs no privilege.

4. HTTP 200 with empty content is not necessarily a bug. Three causes: an Anthropic refusal, a reasoning model eating its whole max_tokens budget before emitting text, or a dropped parameter. Local models do it too. Give reasoning models real headroom before concluding anything.

5. Zen (zen/*) is under a ToS block — make no calls to it, including "harmless" ones. Entries stay in config in an access group no key holds.

6. The OpenCode sidebar under-reports cost — badly. It omits delegated child sessions, where most agent work happens: it showed ~$5.80 against a vendor-verified $17.39. Use opencode stats or the LiteLLM spend log.

7. Open WebUI gives every model its full built-in tool suite by default (meta.builtinTools defaults to True when absent). Models that are weak at tool discipline loop on query_knowledge_files and friends even with no knowledge base configured. Symptom: many round trips with 25-30 token completions. Disable per-model in Admin → Models; there is no global switch.

8. On this box, local inference is RAM-bandwidth-bound, not VRAM-bound. Decode speed is set by a model's active parameters, not its total, so an 80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking which wall you're actually against.

9. The OMO agent aliases embed their model family on purpose — personal/atlas-kimi-k2.7-code, not personal/atlas. OMO picks its per-model prompt by string-matching the last path segment of the model name, so a "tidy-up" rename silently disables every Kimi/GPT-specific prompt and guard with no error. Never shorten them. local/librarian and local/explore are deliberately left un-suffixed — isGptModel matches a bare "gpt" substring and gpt-oss-20b would trip it.

Cost work: read model_group, never model, to attribute a request. Cost is driven by cache-read volume and context size far more than by model choice.

The dotfiles repo and the fleet

~/.dotfiles is one branch linked onto four machines by dotbot. Shared config is in common/, per-machine config in hosts/<hostname>/, and what gets linked where is meta/base.conf.yaml plus meta/hosts/<hostname>.conf.yaml. There are deliberately no per-host branches: the layout this replaced was three machines with two histories that had never met, and host layers cost a few small files and remove the merge entirely.

Host	Address	Role	Packages
mainframe	10.0.0.2	the server — containers, inference stack, Moonlight host	pacman + AUR
nas	10.0.0.3	bulk storage; NFS-exports audio books docs downloads photo video; own 03:30 backup run	pacman + AUR
system76	10.0.0.4	personal laptop	pacman + AUR
sctfw004	10.0.0.8	work laptop — work_laptop in ssh config, user connorjohnstone	apt + flatpak

sctfw004 is US-government adjacent: OpenAI and local models only, never Anthropic. Don't propose an Anthropic-backed workflow for that machine.

Five things about this repo that will cost you an hour:

1. The .gitignore is an allowlist, and it is load-bearing. It opens with **/* and then names what is allowed back in. A new top-level directory is invisible to git until it is explicitly un-ignored, so "I added the file and git doesn't see it" is expected, not a bug — fix it by adding the ! entry, never by relaxing the **/*. The denylist it replaced accumulated browser caches, mail caches and vendored binaries. It also re-excludes credential shapes (**/credentials, **/*.env, **/hosts.yml, **/password, .secrets.zsh); every one of those has held a live credential on these machines. Leave them alone.

2. Secrets never live in a tracked file, and specifically not in zshenv. common/zsh/zshenv ends by sourcing ~/.secrets.zsh, which is untracked and mode 600 — common/zsh/secrets.zsh.example is the shape to copy. Keys for OpenCode are read out of rbw at launch and passed in, never written to disk. A new secret goes in ~/.secrets.zsh, never anywhere under ~/.dotfiles.

3. Any unit that must run on exactly one machine carries ConditionHost=<host>. One branch is linked everywhere, so an unguarded timer fires on all four. inbox-tidy, movie-recs, rent-utilities and ddns mutate a live mailbox, send notifications, send bills and update live DNS records, so they are guarded twice: the units exist only in hosts/mainframe/systemd/ and never in common/, and each carries the condition. Genuinely per-machine things like hypr_refresh are correctly unguarded — don't "fix" them.

4. dots is the one command that answers "is this machine clean". Committed, pushed, and does the package manifest still match reality. It is wired into zshrc as dots --nag, which prints nothing when everything is clean. That silence is the point — a login banner that always fires is one you stop reading.

5. pkg-snapshot writes what is installed into packages/<host>.* — pacman, AUR, apt and flatpak as separate files. Run it after installing something worth keeping and commit the diff; the diff doubles as the log of what was added and when. pkg-restore goes the other way.

This file is a symlink. ~/.claude/CLAUDE.md → ~/.dotfiles/common/claude/CLAUDE.md. Editing it edits the repo, so a change here wants a commit and reaches every machine at the next install.

General

• ~/config/ and ~/docs/ are deliberately untracked. Back up edits there by hand as <file>.pre-<thing> snapshots — the established convention, and the only backup those trees get. Inside a tracked repo git already is that backup, so a .pre-* file there is redundant (and .gitignore excludes the pattern anyway).
• ~/guides/ is tracked in the ~ repo and is where reference material goes, ~/.dotfiles is its own repo. Check which one you are in before committing.
• Don't create project-level CLAUDE.md files in ~ — Connor keeps guidance at the ~/.claude level and reference material in ~/guides/.
• Measure before concluding, and when a previous conclusion turns out to have been reached from bad data, record both the correction and why the original reasoning was unsound — otherwise someone re-opens it.