dotfiles/common/claude/memory/-home-connor/local-code-autocomplete.md

name, description, metadata

name	description	metadata
local-code-autocomplete	RETIRED 2026-10-29. Local FIM autocomplete on mainframe (minuet + qwen3-coder-30b): what was measured, kept in case it comes back	type reference

Retired 2026-10-29: Connor found the completions not worth having. minuet was removed from nvim and qwen3-coder-30b from llama-swap, LiteLLM and disk. The full config is archived in ~/guides/llm/archive/local-tier-2026-10.md. What follows is the state before removal.

Measured 2026-09-18 (llmbench suite 6b, fim).

Local code autocomplete is viable on this box. Route: a raw /v1/completions through LiteLLM with the model's FIM control tokens in the prompt. ⚠ /infill is 404 through the proxy — not exposed.

FIM is a vocabulary feature, so only two local models can do it at all:

model	FIM tokens	exact	TTFT warm
qwen3-coder-30b	yes	0.56	62 ms
qwen3-next-80b-128k	yes	0.25	149 ms
gpt-oss-20b, gemma3-12b	none	—	—

⚠ Check the GGUF VOCAB, not the metadata keys. None of these four declare tokenizer.ggml.fim_*_token_id; the tokens are in the vocab regardless. A metadata check says "no local model can infill" and is a false negative.

⚠ qwen3-coder-30b's "SKIPPED FOR CAPACITY" verdict does not apply here. That was recorded against the agent tier, where 128k is the floor. Autocomplete needs a few thousand tokens around the cursor. The verdict was nearly inherited without checking its scope.

⚠ Exact match is a FLOOR. Of its four misses, one is correct code (a valid match arm both models independently prefer over upstream's) and one is defensible. Read completions before quoting the number: uv run python tools/inspect_fim.py --only differs.

Open question is not capability but eviction: a local autocomplete tier fires per keystroke and llama-swap runs one llama-server — see podman-compose-bind-mount-restart context and robots.md §11.

Related: litellm-extra-body-reasoning for how parameters reach the model.

---

2026-09-29 — the failure mode to check first, and a tuning trap.

Two independent things broke autocomplete on the same day; symptoms are identical (no ghost text), so check them in this order.

1. rbw locked → minuet never sends a request at all. There is no LITELLM_KEY_NVIM in ~/.secrets.zsh; the key comes from rbw only, and minuet caches "no key" for the whole nvim session. Tell-tale: ZERO requests in the llama-swap and LiteLLM logs, not slow ones. Confirm in one command: nvim --headless +MinuetStatus +qa prints key: NOT FOUND. Fix: rbw unlock, then restart nvim.

2. ⚠ Do not tune qwen3-coder-30b for decode throughput. A servefit sweep set -b 256 -ub 128 for +21% decode. Its only consumer is FIM autocomplete, which emits 32 tokens but prefills the whole cursor context — TTFT is prefill-bound and decode is nearly irrelevant. Small batches cut prefill from ~1150 to ~445 tok/s, putting a 3.7k-token context at 5.1 s against minuet's 3 s request_timeout: every completion aborted client-side. Restored to -b 2048 -ub 512 (keeping -ncmoe 20 and q4_0 KV — the A/B isolated -b/-ub). Costs ~92 MiB of VRAM headroom, 541 → 449 MiB spare.

   The sweep was not wrong, it was scoped to the wrong metric — the same error shape as inheriting this model's "skipped for capacity" verdict above. Benchmark the role, not the model.

Warm completions measure 0.98 s, matching the 0.7-1.3 s on record. A cold first request on a large file is still ~3.8 s and will exceed the 3 s timeout; raising request_timeout in common/nvim/init.lua is the open option.

---

2026-10-22 — Viseron took VRAM; -ncmoe 20 -> 22; request_timeout 3 -> 8.

Viseron (viseron + ffmpeg_camera_1) now holds ~666 MiB of VRAM, more than the 449 MiB spare the tuned entry had, so qwen3-coder-30b failed to load at all ("upstream command exited prematurely"). 21 still OOMs; 22 fits with ~440 MiB spare. Prefill fell to ~860 tok/s from ~1,100 (not A/B'd), so a fresh ~3k-token context took 4.8 s, past minuet's 3 s default. request_timeout = 8 now set in common/nvim/init.lua. Warm, same-context requests are ~0.8 s.

A cold load is NOT lost when minuet times out: llama-swap finishes loading after the client disconnects (tested: ready ~5 s after a 3 s-aborted request). Cost is one missed completion. A BufEnter preload was offered and declined as too much; don't re-propose it.

llama-swap watches its config (poll, 2 s) — editing ~/config/llama-swap/config.yaml needs no container restart.