dotfiles/common/claude/memory/-home-connor/local-code-autocomplete.md

name, description, metadata

name	description	metadata
local-code-autocomplete	Local code autocomplete (FIM) works on mainframe; qwen3-coder-30b is the best model for it despite being written off for capacity	type reference

Measured 2026-09-18 (llmbench suite 6b, fim).

Local code autocomplete is viable on this box. Route: a raw /v1/completions through LiteLLM with the model's FIM control tokens in the prompt. ⚠ /infill is 404 through the proxy — not exposed.

FIM is a vocabulary feature, so only two local models can do it at all:

model	FIM tokens	exact	TTFT warm
qwen3-coder-30b	yes	0.56	62 ms
qwen3-next-80b-128k	yes	0.25	149 ms
gpt-oss-20b, gemma3-12b	none	—	—

⚠ Check the GGUF VOCAB, not the metadata keys. None of these four declare tokenizer.ggml.fim_*_token_id; the tokens are in the vocab regardless. A metadata check says "no local model can infill" and is a false negative.

⚠ qwen3-coder-30b's "SKIPPED FOR CAPACITY" verdict does not apply here. That was recorded against the agent tier, where 128k is the floor. Autocomplete needs a few thousand tokens around the cursor. The verdict was nearly inherited without checking its scope.

⚠ Exact match is a FLOOR. Of its four misses, one is correct code (a valid match arm both models independently prefer over upstream's) and one is defensible. Read completions before quoting the number: uv run python tools/inspect_fim.py --only differs.

Open question is not capability but eviction: a local autocomplete tier fires per keystroke and llama-swap runs one llama-server — see podman-compose-bind-mount-restart context and robots.md §11.

Related: litellm-extra-body-reasoning for how parameters reach the model.

---

2026-09-29 — the failure mode to check first, and a tuning trap.

Two independent things broke autocomplete on the same day; symptoms are identical (no ghost text), so check them in this order.

1. rbw locked → minuet never sends a request at all. There is no LITELLM_KEY_NVIM in ~/.secrets.zsh; the key comes from rbw only, and minuet caches "no key" for the whole nvim session. Tell-tale: ZERO requests in the llama-swap and LiteLLM logs, not slow ones. Confirm in one command: nvim --headless +MinuetStatus +qa prints key: NOT FOUND. Fix: rbw unlock, then restart nvim.

2. ⚠ Do not tune qwen3-coder-30b for decode throughput. A servefit sweep set -b 256 -ub 128 for +21% decode. Its only consumer is FIM autocomplete, which emits 32 tokens but prefills the whole cursor context — TTFT is prefill-bound and decode is nearly irrelevant. Small batches cut prefill from ~1150 to ~445 tok/s, putting a 3.7k-token context at 5.1 s against minuet's 3 s request_timeout: every completion aborted client-side. Restored to -b 2048 -ub 512 (keeping -ncmoe 20 and q4_0 KV — the A/B isolated -b/-ub). Costs ~92 MiB of VRAM headroom, 541 → 449 MiB spare.

   The sweep was not wrong, it was scoped to the wrong metric — the same error shape as inheriting this model's "skipped for capacity" verdict above. Benchmark the role, not the model.

Warm completions measure 0.98 s, matching the 0.7-1.3 s on record. A cold first request on a large file is still ~3.8 s and will exceed the 3 s timeout; raising request_timeout in common/nvim/init.lua is the open option.