Phase 5: triage feed and the tuning surface

/ is the labelling machine: ranked clusters, one keystroke per judgement
(j/k move, i/n label, o open), advancing automatically so a run of judgements is
a run of single keys. Feedback volume is the whole point -- the old system
collected 32 labels because its buttons did not persist and its UI could only be
clicked.

Browser feedback posts to /ui/feedback/, deliberately NOT under /api/feedback/,
which Caddy routes around Authelia for the ntfy buttons. The browser path stays
behind the identity gateway and so needs no signature.

/tune drives the same route() the live service calls, so what it shows is what
would have been sent. The old replay harness hardcoded its parameters and could
never sweep the thresholds it existed to tune.

Building it corrected a claim I had already made twice. I said the budgets bind
before the thresholds, then wrote that the subject quota binds before the
budget. Measured on this corpus, neither is generally true: at the shipped
budgets (3 interest, 2 importance) the BUDGET binds and the quota is inert --
moving it 1 to 3 changes nothing. Raise budgets to 8 and the QUOTA takes over,
1/2/3 giving 10/11/13 alerts. The page now says exactly that, with the numbers,
rather than a rule of thumb that happened to be wrong.
This commit is contained in:
connor committed 2026-09-08 17:18:13 -04:00
1 parent e99b7c9fb7
commit 42c38d759d
4 files changed
+454 -3

No files matched your search

+14 -3
View File
@@ -84,8 +84,19 @@ Re-run the calibration in `profile/CALIBRATION.md` before trusting them.
`simulate` is the tuning surface. The old system's replay harness hardcoded its `simulate` is the tuning surface. The old system's replay harness hardcoded its
parameters and so could never sweep the thresholds it existed to tune. parameters and so could never sweep the thresholds it existed to tune.
## Web
/ triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
/tune move the thresholds and budgets, see what would have been sent
/digest the day's briefing, mobile-first
/healthz
Everything except `/api/feedback/*` sits behind the identity gateway. That one
path is exempt because ntfy runs notification buttons from the phone and a
native app cannot complete an interactive login; it is HMAC-signed instead.
## Status ## Status
Ingest, embedding, clustering, two-axis scoring and routing are built and Running in production. Ingest, clustering, two-axis scoring, routing, ntfy
running against live feeds. Delivery (ntfy, the digest page) and the web UI are delivery with working feedback buttons, the digest page and the triage UI are
not yet written. all live.
+153
View File
@@ -0,0 +1,153 @@
<!doctype html>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Triage</title>
<style>
:root { color-scheme: light dark; --fg:#16181d; --bg:#fdfdfc; --muted:#6b7280;
--rule:#e4e4e7; --accent:#7c3aed; --card:#fff; --sel:#f3f0ff;
--pos:#15803d; --neg:#b91c1c; --warn:#b45309; }
@media (prefers-color-scheme: dark) {
:root { --fg:#e8e8ea; --bg:#141416; --muted:#9ca3af; --rule:#2a2a2e;
--accent:#a78bfa; --card:#1b1b1f; --sel:#241f36;
--pos:#4ade80; --neg:#f87171; --warn:#fbbf24; }
}
* { box-sizing:border-box; }
body { margin:0; background:var(--bg); color:var(--fg);
font:15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif; }
main { max-width:52rem; margin:0 auto; padding:1.25rem 1rem 6rem; }
header { display:flex; flex-wrap:wrap; gap:.75rem; align-items:baseline;
justify-content:space-between; border-bottom:1px solid var(--rule);
padding-bottom:.6rem; margin-bottom:1rem; }
h1 { font-size:1.15rem; margin:0; letter-spacing:-.01em; }
.stats, .filters { color:var(--muted); font-size:.83rem; }
.filters a { color:var(--accent); text-decoration:none; margin-left:.6rem; }
.filters a.on { text-decoration:underline; }
article { background:var(--card); border:1px solid var(--rule); border-radius:9px;
padding:.7rem .85rem; margin-bottom:.5rem; scroll-margin-top:1rem; }
article.sel { border-color:var(--accent); background:var(--sel); }
article.done { opacity:.4; }
.top { display:flex; gap:.6rem; align-items:baseline; justify-content:space-between; }
h2 { font-size:.98rem; line-height:1.35; margin:0 0 .3rem; font-weight:600; }
h2 a { color:inherit; text-decoration:none; }
h2 a:hover { text-decoration:underline; }
.tags { display:flex; flex-wrap:wrap; gap:.4rem; margin:.35rem 0; }
.tag { font-size:.7rem; text-transform:uppercase; letter-spacing:.04em;
color:var(--muted); border:1px solid var(--rule); border-radius:99px;
padding:.05rem .45rem; white-space:nowrap; }
.tag.lane-alert\:important { color:var(--neg); border-color:currentColor; }
.tag.lane-alert\:interest { color:var(--accent); border-color:currentColor; }
.tag.demote { color:var(--warn); border-color:currentColor; }
.srcs { list-style:none; padding:0; margin:.35rem 0 0; }
.srcs li { font-size:.83rem; margin:.12rem 0; color:var(--muted); }
.srcs a { color:inherit; text-decoration:none; border-bottom:1px solid var(--rule); }
.acts { display:flex; gap:.4rem; margin-top:.5rem; }
button { font:inherit; font-size:.83rem; padding:.22rem .7rem; cursor:pointer;
border:1px solid var(--rule); background:transparent; color:inherit;
border-radius:6px; }
button.yes:hover { border-color:var(--pos); color:var(--pos); }
button.no:hover { border-color:var(--neg); color:var(--neg); }
button[disabled] { opacity:.45; cursor:default; }
kbd { font:inherit; font-size:.72rem; border:1px solid var(--rule);
border-bottom-width:2px; border-radius:4px; padding:0 .25rem; color:var(--muted); }
.help { position:fixed; bottom:0; left:0; right:0; background:var(--card);
border-top:1px solid var(--rule); padding:.45rem 1rem; font-size:.78rem;
color:var(--muted); display:flex; gap:1rem; justify-content:center; }
.empty { color:var(--muted); font-style:italic; padding:2rem 0; }
@media (max-width:34rem) { .help { display:none; } }
</style>
<main>
<header>
<h1>Triage</h1>
<div class="stats">{{ rows|length }} shown · {{ labelled }} labelled · {{ total }} clusters</div>
<div class="filters">show
<a class="{{ 'on' if show == 'unlabelled' }}" href="?show=unlabelled">unlabelled</a>
<a class="{{ 'on' if show == 'all' }}" href="?show=all">all</a>
· <a href="/digest">digest</a>
</div>
</header>
{% for r in rows %}
<article id="c{{ loop.index0 }}" data-id="{{ r.id }}"
class="{{ 'done' if r.labelled }}">
<h2><a href="{{ r.sources[0].url if r.sources else '#' }}" target="_blank"
rel="noopener">{{ r.headline }}</a></h2>
<div class="tags">
<span class="tag lane-{{ r.lane }}">{{ r.lane }}</span>
<span class="tag">{{ r.topic }} · int {{ '%.2f'|format(r.interest) }}</span>
<span class="tag">{{ r.importance_topic }} · imp {{ '%.2f'|format(r.importance) }}</span>
<span class="tag">{{ r.origins }} origin{{ '' if r.origins == 1 else 's' }}</span>
{% if r.demote %}<span class="tag demote">demoted: {{ r.demote }}</span>{% endif %}
</div>
<ul class="srcs">
{% for s in r.sources %}
<li><a href="{{ s.url }}" target="_blank" rel="noopener">{{ s.title }}</a>
— {{ s.name }}</li>
{% endfor %}
</ul>
<div class="acts">
<button class="yes" data-kind="interested">Interested <kbd>i</kbd></button>
<button class="no" data-kind="not_interested">Not interested <kbd>n</kbd></button>
</div>
</article>
{% else %}
<p class="empty">Nothing left to label. Try <a href="?show=all">show all</a>.</p>
{% endfor %}
</main>
<div class="help">
<span><kbd>j</kbd>/<kbd>k</kbd> move</span>
<span><kbd>i</kbd> interested</span>
<span><kbd>n</kbd> not interested</span>
<span><kbd>o</kbd> open</span>
</div>
<script>
const cards = [...document.querySelectorAll('article')];
let cur = 0;
const select = i => {
if (!cards.length) return;
cur = Math.max(0, Math.min(i, cards.length - 1));
cards.forEach(c => c.classList.remove('sel'));
cards[cur].classList.add('sel');
cards[cur].scrollIntoView({block: 'nearest', behavior: 'smooth'});
};
async function label(card, kind) {
const btns = card.querySelectorAll('button');
btns.forEach(b => b.disabled = true);
try {
const r = await fetch(`/ui/feedback/${card.dataset.id}/${kind}`, {method: 'POST'});
if (!r.ok) throw new Error(r.status);
card.classList.add('done');
// Advance, so a run of judgements is one keystroke each.
if (cards[cur] === card) select(cur + 1);
} catch (e) {
btns.forEach(b => b.disabled = false);
card.style.borderColor = 'var(--neg)';
console.error('feedback failed', e);
}
}
cards.forEach((card, i) => {
card.addEventListener('click', () => select(i));
card.querySelectorAll('button').forEach(b =>
b.addEventListener('click', ev => { ev.stopPropagation(); label(card, b.dataset.kind); }));
});
document.addEventListener('keydown', ev => {
if (ev.metaKey || ev.ctrlKey || ev.altKey) return;
const card = cards[cur];
switch (ev.key) {
case 'j': select(cur + 1); break;
case 'k': select(cur - 1); break;
case 'i': if (card) label(card, 'interested'); break;
case 'n': if (card) label(card, 'not_interested'); break;
case 'o': if (card) card.querySelector('h2 a')?.click(); break;
default: return;
}
ev.preventDefault();
});
select(0);
</script>
+136
View File
@@ -0,0 +1,136 @@
<!doctype html>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Tune</title>
<style>
:root { color-scheme: light dark; --fg:#16181d; --bg:#fdfdfc; --muted:#6b7280;
--rule:#e4e4e7; --accent:#7c3aed; --card:#fff; --neg:#b91c1c; }
@media (prefers-color-scheme: dark) {
:root { --fg:#e8e8ea; --bg:#141416; --muted:#9ca3af; --rule:#2a2a2e;
--accent:#a78bfa; --card:#1b1b1f; --neg:#f87171; }
}
* { box-sizing:border-box; }
body { margin:0; background:var(--bg); color:var(--fg);
font:15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif; }
main { max-width:54rem; margin:0 auto; padding:1.25rem 1rem 4rem; }
header { display:flex; gap:1rem; align-items:baseline; justify-content:space-between;
border-bottom:1px solid var(--rule); padding-bottom:.6rem; margin-bottom:1rem; }
h1 { font-size:1.15rem; margin:0; }
a { color:var(--accent); }
form { background:var(--card); border:1px solid var(--rule); border-radius:9px;
padding:.9rem 1rem; margin-bottom:1.1rem; }
.grid { display:grid; grid-template-columns:repeat(auto-fit,minmax(13rem,1fr)); gap:.9rem; }
label { display:block; font-size:.8rem; color:var(--muted); margin-bottom:.2rem; }
input[type=range] { width:100%; accent-color:var(--accent); }
input[type=number] { width:5rem; font:inherit; background:transparent; color:inherit;
border:1px solid var(--rule); border-radius:5px; padding:.15rem .35rem; }
.val { font-variant-numeric:tabular-nums; color:var(--fg); font-size:.85rem; }
.headline { display:flex; gap:1.5rem; flex-wrap:wrap; align-items:baseline;
margin:.9rem 0 1.2rem; }
.big { font-size:1.9rem; font-weight:650; letter-spacing:-.02em; }
.big small { font-size:.8rem; font-weight:400; color:var(--muted); margin-left:.3rem; }
h2 { font-size:.82rem; text-transform:uppercase; letter-spacing:.05em;
color:var(--muted); margin:1.4rem 0 .5rem; }
li { list-style:none; padding:.3rem 0; border-bottom:1px solid var(--rule);
font-size:.9rem; display:flex; gap:.6rem; }
ul { padding:0; margin:0; }
.sc { font-variant-numeric:tabular-nums; color:var(--muted); min-width:3rem; }
.lane { color:var(--accent); font-size:.72rem; text-transform:uppercase;
letter-spacing:.04em; min-width:5.5rem; }
.lane.imp { color:var(--neg); }
.note { color:var(--muted); font-size:.82rem; }
</style>
<main>
<header>
<h1>Tune</h1>
<div class="note"><a href="/">triage</a> · <a href="/digest">digest</a></div>
</header>
<form method="get">
<div class="grid">
<div>
<label>interest threshold <span class="val" id="v1">{{ routing.interest_threshold }}</span></label>
<input type="range" name="interest_threshold" min="0" max="6" step="0.1"
value="{{ routing.interest_threshold }}" oninput="v1.textContent=this.value">
</div>
<div>
<label>importance threshold <span class="val" id="v2">{{ routing.important_threshold }}</span></label>
<input type="range" name="important_threshold" min="0" max="6" step="0.1"
value="{{ routing.important_threshold }}" oninput="v2.textContent=this.value">
</div>
<div>
<label>digest floor <span class="val" id="v3">{{ routing.digest_floor }}</span></label>
<input type="range" name="digest_floor" min="-2" max="4" step="0.1"
value="{{ routing.digest_floor }}" oninput="v3.textContent=this.value">
</div>
<div>
<label>interest alerts / day</label>
<input type="number" name="interest_per_day" min="0" max="20"
value="{{ routing.interest_per_day }}">
</div>
<div>
<label>importance alerts / day</label>
<input type="number" name="important_per_day" min="0" max="20"
value="{{ routing.important_per_day }}">
</div>
<div>
<label>digest size</label>
<input type="number" name="digest_max_items" min="1" max="100"
value="{{ routing.digest_max_items }}">
</div>
<div>
<label>max alerts per subject / day</label>
<input type="number" name="max_alerts_per_group_per_day" min="1" max="10"
value="{{ routing.max_alerts_per_group_per_day }}">
</div>
</div>
<p style="margin:.9rem 0 0"><button>Simulate</button>
<span class="note">· over the last {{ '%.0f'|format(days * 24) }}h of clusters</span></p>
</form>
<div class="headline">
<div><div class="big">{{ '%.1f'|format(per_day) }}<small>notifications/day</small></div></div>
<div class="note">
{{ counts['alert:important'] }} importance · {{ counts['alert:interest'] }} interest ·
{{ counts['digest'] }} digest · {{ counts['drop'] }} dropped
</div>
</div>
<h2>Would notify</h2>
<ul>
{% for a in alerts %}
<li><span class="lane {{ 'imp' if 'important' in a.lane }}">{{ a.lane.replace('alert:','') }}</span>
<span class="sc">{{ '%.2f'|format(a.score) }}</span>
<span>{{ a.headline }}</span></li>
{% else %}
<li class="note">Nothing clears these thresholds.</li>
{% endfor %}
</ul>
<h2>Would go to the digest</h2>
<ul>
{% for d in digest %}
<li><span class="sc">{{ '%.2f'|format(d.score) }}</span>
<span>{{ d.headline }}</span>
<span class="note">{{ d.reason if d.reason != 'digest' else '' }}</span></li>
{% endfor %}
</ul>
<p class="note" style="margin-top:1.5rem">
Sliders are a simulation only — nothing here writes to <code>config.toml</code>.
</p>
<p class="note">
Three constraints compete, and which one actually binds is not obvious.
Measured on this corpus: at the shipped budgets (3 interest, 2 importance)
the <strong>budget</strong> binds and the subject quota is inert — moving the
quota from 1 to 3 changes nothing. Raise the budgets to 8 and the
<strong>quota</strong> takes over, at which point 1 / 2 / 3 yields 10 / 11 /
13 alerts. The <strong>threshold</strong> binds last in both cases: there is
almost always more above-threshold news than there are slots, so it works as
a quality floor for slow days rather than as a volume control.
</p>
<p class="note">
So to get <em>more</em> notifications, raise the budgets first. To get more
<em>variety</em> at the same volume, lower the budgets and raise the quota.
</p>
</main>
+151
View File
@@ -28,6 +28,22 @@ def _con():
return store.connect(Config().db_path) return store.connect(Config().db_path)
# Scoring a page load is cheap (matrix ops over ~1,300 clusters), but building
# ProfileVectors loads the embedding model. Cached so only the first request
# pays for it.
_VECTORS = None
def _scored(con, cfg):
global _VECTORS
from .embed import Embedder
from .score import ProfileVectors, score_clusters
if _VECTORS is None:
_VECTORS = ProfileVectors(con, cfg.profile(), Embedder.from_config(cfg))
return score_clusters(con, _VECTORS, cfg.scoring, cfg.clustering)
async def healthz(request): async def healthz(request):
return PlainTextResponse("ok") return PlainTextResponse("ok")
@@ -60,6 +76,138 @@ async def feedback(request):
return JSONResponse({"ok": True, "cluster_id": cluster_id, "kind": kind}) return JSONResponse({"ok": True, "cluster_id": cluster_id, "kind": kind})
async def triage(request):
"""The labelling machine.
This is where feedback volume comes from, so it is built to be fast to skim
and label: ranked, keyboard-driven, one keystroke per judgement. The old
system's UI could only be clicked, and its clicks did not persist.
"""
from datetime import UTC, datetime
from .route import route
cfg = Config()
con = _con()
scores = _scored(con, cfg)
decisions = {d.cluster_id: d for d in
route(con, scores, cfg.routing, cfg.delivery,
datetime.now(UTC), cfg.block_groups())}
labelled = {r["cluster_id"] for r in con.execute("SELECT DISTINCT cluster_id FROM feedback")}
show = request.query_params.get("show", "unlabelled")
limit = min(int(request.query_params.get("limit", 60)), 200)
ranked = sorted(scores, key=lambda s: -max(s.interest_score, s.importance_score))
rows = []
for s in ranked:
if s.veto:
continue
if show == "unlabelled" and s.cluster_id in labelled:
continue
d = decisions.get(s.cluster_id)
members = con.execute(
"""SELECT i.title, i.url, so.name FROM cluster_members cm
JOIN items i ON i.id = cm.item_id
JOIN sources so ON so.id = i.source_id
WHERE cm.cluster_id = ? LIMIT 5""", (s.cluster_id,)).fetchall()
rows.append({
"id": s.cluster_id, "headline": s.lead_title,
"topic": s.interest.block, "importance_topic": s.importance.block,
"interest": s.interest_score, "importance": s.importance_score,
"origins": s.corroboration.origins, "demote": s.demote,
"lane": d.lane if d else "?", "reason": d.reason if d else "",
"labelled": s.cluster_id in labelled,
"sources": [{"name": m["name"], "title": m["title"], "url": m["url"]}
for m in members],
})
if len(rows) >= limit:
break
return TEMPLATES.TemplateResponse(
request, "triage.html",
{"rows": rows, "show": show, "total": len(scores), "labelled": len(labelled)})
async def web_feedback(request):
"""Feedback from the browser.
Deliberately NOT under /api/feedback/, which Caddy routes around Authelia
for the ntfy buttons. This path stays behind the identity gateway, so it
needs no signature.
"""
from datetime import UTC, datetime
kind = request.path_params["kind"]
if kind not in ("interested", "not_interested"):
return JSONResponse({"error": "bad kind"}, status_code=400)
con = _con()
con.execute(
"INSERT INTO feedback (cluster_id, kind, source, created_at) VALUES (?,?,?,?)",
(request.path_params["cluster_id"], kind, "web", datetime.now(UTC).isoformat()))
return JSONResponse({"ok": True})
async def tune(request):
"""Threshold tuning against real history.
The old system's replay harness hardcoded its parameters, so it could never
sweep the thresholds it existed to tune. Here the sliders drive the same
`route()` the live service calls, so what you see is what it would have sent.
"""
from datetime import UTC, datetime
from .route import DIGEST, DROP, IMPORTANT, INTEREST, route
cfg = Config()
con = _con()
scores = _scored(con, cfg)
routing = dict(cfg.routing)
for key in ("interest_threshold", "important_threshold", "digest_floor"):
if (raw := request.query_params.get(key)) is not None:
try:
routing[key] = float(raw)
except ValueError:
pass
for key in ("interest_per_day", "important_per_day", "digest_max_items",
"max_alerts_per_group_per_day"):
if (raw := request.query_params.get(key)) is not None:
try:
routing[key] = int(raw)
except ValueError:
pass
now = datetime.now(UTC)
decisions = route(con, scores, routing, cfg.delivery, now, cfg.block_groups())
by_id = {s.cluster_id: s for s in scores}
lanes: dict[str, list] = {}
for d in decisions:
lanes.setdefault(d.lane, []).append(d)
days = routing["max_age_hours"] / 24.0
counts = {lane: len(lanes.get(lane, [])) for lane in (IMPORTANT, INTEREST, DIGEST, DROP)}
alerts = [
{"lane": d.lane, "reason": d.reason, "score": d.score,
"headline": by_id[d.cluster_id].lead_title,
"topic": by_id[d.cluster_id].interest.block}
for d in sorted(lanes.get(IMPORTANT, []) + lanes.get(INTEREST, []),
key=lambda d: -d.score)
]
digest = [
{"reason": d.reason, "score": d.score,
"headline": by_id[d.cluster_id].lead_title,
"topic": by_id[d.cluster_id].interest.block}
for d in sorted(lanes.get(DIGEST, []), key=lambda d: -d.score)[:25]
]
return TEMPLATES.TemplateResponse(
request, "tune.html",
{"routing": routing, "counts": counts, "alerts": alerts, "digest": digest,
"per_day": (counts[IMPORTANT] + counts[INTEREST]) / days, "days": days,
"defaults": cfg.routing})
async def digest_page(request): async def digest_page(request):
day = request.path_params.get("day") day = request.path_params.get("day")
try: try:
@@ -107,6 +255,9 @@ async def digest_page(request):
routes = [ routes = [
Route("/", triage),
Route("/ui/feedback/{cluster_id}/{kind}", web_feedback, methods=["POST"]),
Route("/tune", tune),
Route("/healthz", healthz), Route("/healthz", healthz),
Route("/api/feedback/{cluster_id}/{kind}/{token}", feedback, methods=["POST"]), Route("/api/feedback/{cluster_id}/{kind}/{token}", feedback, methods=["POST"]),
Route("/digest", digest_page), Route("/digest", digest_page),