dotfiles/common/claude/memory/-home-connor/llamacpp-offload-method.md

name, description, metadata

name	description	metadata
llamacpp-offload-method	-ngl alone is the wrong knob for partial offload; full offload is a cliff, and -ot/KV-quant/MTP are the levers llmbench never used	type project

Tuning -ngl alone is the naive method, and every speed number recorded on mainframe before 2026-09-28 used it. llmbench's ctxfit ladders -ngl and nothing else, so each recorded value is the optimum of the wrong parameter.

Full offload is a CLIFF, not a slope. Qwen3.8-27B, 65 blocks, measured 2026-09-28 (short/medium/long t/s):

UD-Q4_K_XL   18.0 GB  -ngl 36    6.58 / 5.44 / 3.20
GSQ IQ3_S    11.8 GB  -ngl 56    6.67 / 5.45 / 3.34
GSQ IQ3_XXS   9.8 GB  -ngl 99   13.31 / 11.75 / 8.52

Going 36 -> 56 layers resident bought ~0%. Going 56 -> 65 (all of them) bought +155% at long context. With ANY layer on CPU every token pays a PCIe round trip; removing the last one removes the whole penalty rather than a proportional share. A prediction of ~35% from a linear "3.5x per CPU layer" model was badly wrong. ttft collapsed too: 24,255 ms -> 313 ms warm.

Three levers llama.cpp offers that were never used here:

• --override-tensor "blk\.(...)\.ffn_.*=CPU" with -ngl 99. Spills only FFN tensors, which are large and latency-tolerant, keeping ALL attention resident -- attention is small and used constantly. -ngl N instead spills whole blocks, attention included. This is the right knob whenever a model does NOT fully fit.
• -ctk q4_0 -ctv q4_0 quantized KV. Once weights fit, this buys CONTEXT rather than speed.
• --spec-type draft-mtp --spec-draft-n-max 2 with a -mtp GGUF. Reported 88% acceptance, 2.76 mean accepted tokens -- near 2x decode, independent of offload. Both Qwen3.8's MTP head and Glimmer's DFlash drafter sat unused.

Why: a public report runs Qwen3.8-27B at ~9.7 t/s with 96K context on the same RTX 3060 12GB using Q4_K_S -- a LARGER quant than ones dismissed here as too slow. "Too big to be fast" was never established; only "too big to be fast when you spill whole layers".

How to apply: when a model does not fully fit, use -ngl 99 plus an -ot FFN regex and tune the block count, rather than lowering -ngl. Re-test anything previously disqualified on speed -- the verdict may be an artifact of the method. ctxfit needs to search FFN placement, not just -ngl. See npu-setting-clamps-memory.