connor
e6644d0616
Replaces the previous repo, which had split into two histories that never met (mainframe on a dead GitLab remote, the laptops on Gitea) with 146 dirty files across three machines and the NAS never enrolled at all. Branch-per-machine is gone. One main, with host differences expressed as small files under hosts/<hostname>/ rather than as branches, so there is nothing to merge. The reconciled zsh layer reduces 15-33 line forks to 1-7 effective lines per host; distro differences (oh-my-zsh prefix, syntax-highlighting path, fd vs fdfind) are probed in common/ instead. Fresh history: the old one carried six plaintext credentials, 45 MB of mail caches, browser caches and vendored binaries. 5,096 tracked files and 144 MB become 462 files and 2.6 MB. The .gitignore is now an allowlist, which is what keeps that true. Root cause of the rot: ~/.local/bin was a symlink to scripts/ with GOPATH inside it, so every go install wrote into version control (2.2 GB on the work laptop). PATH now points at the repo instead of the reverse. Also: Hyprland replaces sway and is sourced in two halves so $browser is defined before use; singleton automations carry ConditionHost= alongside host-layer-only placement; ddns moves from cron to a guarded timer; package manifests and pkg-snapshot/pkg-restore replace the X11-era install_scripts/; networkmanager-dmenu added to system76 (the binding always existed, the package never did).
79 lines
4.4 KiB
Markdown
79 lines
4.4 KiB
Markdown
# Working notes for Claude on mainframe
|
|
|
|
## LLM / inference stack
|
|
|
|
Connor runs a self-hosted inference stack: LiteLLM proxy in front of a local GPU
|
|
model (llama-swap), OpenAI, Anthropic, and Moonshot/Kimi, with OpenCode + OMO on
|
|
top. **Before touching anything under `~/config/litellm/`, `~/.config/opencode/`,
|
|
or the model routing, read `~/guides/llm/robots.md`.** It carries the design
|
|
decisions, the measured data, and — more importantly — the specific wrong
|
|
conclusions this project has already reached and corrected. `~/guides/llm/README.md`
|
|
is the short human version.
|
|
|
|
The five things most likely to waste an hour if you don't know them:
|
|
|
|
1. **A config value is not applied until an invalid value produces an upstream
|
|
400.** LiteLLM silently discards parameters it doesn't route (`store: false`
|
|
bare in `litellm_params` returns 200 and never reaches OpenAI; it needs
|
|
`extra_body`).
|
|
|
|
2. **Editing bind-mounted config does nothing without
|
|
`sudo podman restart <container>`.** Changing an *env var* needs a full
|
|
`sudo podman-compose up -d <svc>` — a restart is not enough. The reverse trap
|
|
is worse and bites silently: **`podman-compose up -d` will NOT restart a
|
|
container when only a bind-mounted file changed**, because the compose
|
|
definition is unchanged. It prints the container name and does nothing. This
|
|
applies to Grafana provisioning as much as to `litellm/config.yaml`.
|
|
|
|
3. **If every hosted model hangs (not errors) while the host has internet**, it's
|
|
Docker: `dockerd` sets `ip filter` FORWARD to policy drop and kills all podman
|
|
egress. DNS still resolves, which makes it look like a broken vendor. Fix:
|
|
`sudo bash ~/config/host/podman-docker-forward.sh`. Don't remove Docker — the
|
|
Gitea Actions runners need it.
|
|
|
|
4. **HTTP 200 with empty content is not necessarily a bug.** Three causes: an
|
|
Anthropic refusal, a reasoning model eating its whole `max_tokens` budget
|
|
before emitting text, or a dropped parameter. Local models do it too. Give
|
|
reasoning models real headroom before concluding anything.
|
|
|
|
5. **Zen (`zen/*`) is under a ToS block — make no calls to it**, including
|
|
"harmless" ones. Entries stay in config in an access group no key holds.
|
|
|
|
6. **The OpenCode sidebar under-reports cost — badly.** It omits delegated child
|
|
sessions, where most agent work happens: it showed ~$5.80 against a
|
|
vendor-verified $17.39. Use `opencode stats` or the LiteLLM spend log.
|
|
|
|
7. **Open WebUI gives every model its full built-in tool suite by default**
|
|
(`meta.builtinTools` defaults to True when absent). Models that are weak at
|
|
tool discipline loop on `query_knowledge_files` and friends even with no
|
|
knowledge base configured. Symptom: many round trips with 25-30 token
|
|
completions. Disable per-model in Admin → Models; there is no global switch.
|
|
|
|
8. **On this box, local inference is RAM-bandwidth-bound, not VRAM-bound.**
|
|
Decode speed is set by a model's *active* parameters, not its total, so an
|
|
80B MoE with 3B active runs at the same speed as a 30B with 3B active. Free
|
|
VRAM buys prefill, not decode. Don't recommend a bigger GPU without checking
|
|
which wall you're actually against.
|
|
|
|
9. **The OMO agent aliases embed their model family on purpose** —
|
|
`personal/atlas-kimi-k2.7-code`, not `personal/atlas`. OMO picks its
|
|
per-model prompt by string-matching the *last path segment* of the model
|
|
name, so a "tidy-up" rename silently disables every Kimi/GPT-specific prompt
|
|
and guard with no error. **Never shorten them.** `local/librarian` and
|
|
`local/explore` are deliberately left un-suffixed — `isGptModel` matches a
|
|
bare `"gpt"` substring and `gpt-oss-20b` would trip it.
|
|
|
|
Cost work: read `model_group`, never `model`, to attribute a request. Cost is
|
|
driven by cache-read volume and context size far more than by model choice.
|
|
|
|
## General
|
|
|
|
- Repos here use an **allowlist** `.gitignore` (`**/*` plus explicit `!` entries).
|
|
`~/config/` and `~/docs/` are deliberately untracked — back up config edits by
|
|
hand as `<file>.pre-<thing>` snapshots, which is the established convention.
|
|
- Don't create project-level `CLAUDE.md` files in `~` — Connor keeps guidance at
|
|
the `~/.claude` level and reference material in `~/guides/`.
|
|
- Measure before concluding, and when a previous conclusion turns out to have
|
|
been reached from bad data, **record both the correction and why the original
|
|
reasoning was unsound** — otherwise someone re-opens it.
|