# Working notes for Claude on mainframe

## LLM / inference stack

Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU
model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode + OMO on
top. **Before touching anything under `~/config/litellm/`, `~/.config/opencode/`,
or the model routing, read `~/guides/llm/robots.md`.** It carries the design
decisions, the measured data, and — more importantly — the specific wrong
conclusions this project has already reached and corrected. `~/guides/llm/README.md`
is the short human version.

The five things most likely to waste an hour if you don't know them:

1. **A config value is not applied until an invalid value produces an upstream
   400.** LiteLLM silently discards parameters it doesn't route (`store: false`
   bare in `litellm_params` returns 200 and never reaches OpenAI; it needs
   `extra_body`).

2. **Editing bind-mounted config does nothing without
   `sudo podman restart <container>`.** Changing an *env var* needs a full
   `sudo podman-compose up -d <svc>` — a restart is not enough. The reverse trap
   is worse and bites silently: **`podman-compose up -d` will NOT restart a
   container when only a bind-mounted file changed**, because the compose
   definition is unchanged. It prints the container name and does nothing. This
   applies to Grafana provisioning as much as to `litellm/config.yaml`.

3. **If every hosted model hangs (not errors) while the host has internet**, it's
   Docker: `dockerd` sets `ip filter` FORWARD to policy drop and kills all podman
   egress. DNS still resolves, which makes it look like a broken vendor. Fix:
   `sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the
   Gitea Actions runners need it.

4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an
   Anthropic refusal, a reasoning model eating its whole `max_tokens` budget
   before emitting text, or a dropped parameter. Local models do it too. Give
   reasoning models real headroom before concluding anything.

5. **Zen (`zen/*`) is under a ToS block — make no calls to it**, including
   "harmless" ones. Entries stay in config in an access group no key holds.

6. **The OpenCode sidebar under-reports cost — badly.** It omits delegated child
   sessions, where most agent work happens: it showed ~$5.80 against a
   vendor-verified $17.39. Use `opencode stats` or the LiteLLM spend log.

7. **Open WebUI gives every model its full built-in tool suite by default**
   (`meta.builtinTools` defaults to True when absent). Models that are weak at
   tool discipline loop on `query_knowledge_files` and friends even with no
   knowledge base configured. Symptom: many round trips with 25-30 token
   completions. Disable per-model in Admin → Models; there is no global switch.

8. **On this box, local inference is RAM-bandwidth-bound, not VRAM-bound.**
   Decode speed is set by a model's *active* parameters, not its total, so an
   80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free
   VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking
   which wall you're actually against.

9. **The OMO agent aliases embed their model family on purpose** —
   `personal/atlas-kimi-k2.7-code`, not `personal/atlas`. OMO picks its
   per-model prompt by string-matching the *last path segment* of the model
   name, so a "tidy-up" rename silently disables every Kimi/GPT-specific prompt
   and guard with no error. **Never shorten them.** `local/librarian` and
   `local/explore` are deliberately left un-suffixed — `isGptModel` matches a
   bare `"gpt"` substring and `gpt-oss-20b` would trip it.

Cost work: read `model_group`, never `model`, to attribute a request. Cost is
driven by cache-read volume and context size far more than by model choice.

## General

- Repos here use an **allowlist** `.gitignore` (`**/*` plus explicit `!` entries).
  `~/config/` and `~/docs/` are deliberately untracked — back up config edits by
  hand as `<file>.pre-<thing>` snapshots, which is the established convention.
- Don't create project-level `CLAUDE.md` files in `~` — Connor keeps guidance at
  the `~/.claude` level and reference material in `~/guides/`.
- Measure before concluding, and when a previous conclusion turns out to have
  been reached from bad data, **record both the correction and why the original
  reasoning was unsound** — otherwise someone re-opens it.