Update for minuet
This commit is contained in:
1 parent
226ac906a2
commit
f6921f3049
4 files changed
+30
-5
No files matched your search
@@ -7,7 +7,7 @@
|
||||
- [Reasoning effort per model](litellm-extra-body-reasoning.md) — GLM-5.3 defaults to reasoning_effort=max and cannot disable thinking; gpt-oss needs chat_template_kwargs via extra_body
|
||||
- [Reasoning vs completion tokens](reasoning-tokens-vs-completion-tokens.md) — reasoning comes out of max_tokens, and providers disagree about counting it
|
||||
- [Open WebUI image generation](owui-image-generation.md) — works via LiteLLM; the default 512x512 size is rejected by every image model
|
||||
- [Local code autocomplete](local-code-autocomplete.md) — FIM via /v1/completions; if it dies check rbw lock first, and never tune the coder model for decode
|
||||
- [Local code autocomplete](local-code-autocomplete.md) — FIM via /v1/completions; check rbw first; never tune for decode; ncmoe 22 + 8 s timeout since Viseron took VRAM
|
||||
- [NPU setting clamps memory](npu-setting-clamps-memory.md) — MSI NPU/AI BIOS options cost 17% DRAM bandwidth and the NPU is unusable by llama.cpp anyway.
|
||||
- [mainframe bad core](mainframe-bad-core.md) — APIC 20 (cpu10, an E-core) throws fatal MCEs; offlined by a unit, RMA candidate.
|
||||
- [crowdsec bans own devices](crowdsec-bans-own-devices.md) — "no external access" is usually crowdsec banning his own phone; http_path includes the query string
|
||||
|
||||
@@ -11,7 +11,10 @@ The LiteLLM stack serves two profiles with different permitted providers:
|
||||
**Anthropic is not an employer-approved vendor**, so `anthropic/*` is
|
||||
excluded there by policy, not by preference. PRC-origin models are not
|
||||
formally banned by his office but should be treated as prohibited until the
|
||||
security office gives written approval.
|
||||
security office gives written approval. That restriction is about HOSTED
|
||||
PRC-origin models (Moonshot/Kimi etc.). A PRC-origin model running locally
|
||||
(qwen3-coder-30b on llama-swap) is fine for work, since nothing leaves the
|
||||
machine. Connor's ruling, 2026-10-22.
|
||||
- **Personal** — `anthropic/*`, `openai/*`, and local.
|
||||
- `zen/*` is blocked for both, see [[zen-tos-hold]].
|
||||
|
||||
|
||||
@@ -69,3 +69,22 @@ identical (no ghost text), so check them in this order.
|
||||
Warm completions measure 0.98 s, matching the 0.7-1.3 s on record. A cold
|
||||
first request on a large file is still ~3.8 s and will exceed the 3 s timeout;
|
||||
raising `request_timeout` in `common/nvim/init.lua` is the open option.
|
||||
|
||||
---
|
||||
|
||||
**2026-10-22 — Viseron took VRAM; -ncmoe 20 -> 22; request_timeout 3 -> 8.**
|
||||
|
||||
Viseron (viseron + ffmpeg_camera_1) now holds ~666 MiB of VRAM, more than the
|
||||
449 MiB spare the tuned entry had, so qwen3-coder-30b failed to load at all
|
||||
("upstream command exited prematurely"). 21 still OOMs; 22 fits with ~440 MiB
|
||||
spare. Prefill fell to ~860 tok/s from ~1,100 (not A/B'd), so a fresh ~3k-token
|
||||
context took 4.8 s, past minuet's 3 s default. `request_timeout = 8` now set in
|
||||
`common/nvim/init.lua`. Warm, same-context requests are ~0.8 s.
|
||||
|
||||
A cold load is NOT lost when minuet times out: llama-swap finishes loading after
|
||||
the client disconnects (tested: ready ~5 s after a 3 s-aborted request). Cost is
|
||||
one missed completion. A BufEnter preload was offered and declined as too much;
|
||||
don't re-propose it.
|
||||
|
||||
llama-swap watches its config (poll, 2 s) — editing
|
||||
`~/config/llama-swap/config.yaml` needs no container restart.
|
||||
Reference in new issue
Block a user