Browser feedback uses the signed public endpoint; add the config page
Connor pointed out he may use the site from a laptop away from home. The gated /ui/feedback/ path was worse than just inconvenient: fetch() follows redirects, so an expired Authelia session would go 302 -> login page -> HTTP 200, and the UI would mark the card labelled having recorded nothing. Silent data loss on the one signal the system learns from. Both surfaces now post to the same HMAC-signed public endpoint, with `redirect: 'error'` so a redirect fails loudly instead of looking like success. The gated route is gone -- one mechanism, not two. /config shows what every block actually matches right now, and a box to try a filter before saving it: describe what you want caught, see what it catches and what sits just below the line, move the threshold, then save it as a veto or a demote. That boundary view is the point -- a crypto filter at 0.55 catches 131 clusters including "Security updates for Monday", and at 0.62 catches 4, which you can only know by looking. Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries the reasoning behind every constant and a comment-stripping round-trip would quietly destroy the record of why each number is what it is. Verified: 97 top-level comment lines in, 97 out. The config mount had to stop being :ro, and ~/config/news2 is now owned 10001:1000 group-writable so the container can save while hand-editing still works. `:U` would have fixed the container and locked the user out.
This commit is contained in:
1 parent
42c38d759d
commit
5f4bc8d890
8 files changed
+1481
-26
No files matched your search
@@ -88,6 +88,7 @@ parameters and so could never sweep the thresholds it existed to tune.
|
|||||||
|
|
||||||
/ triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
|
/ triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
|
||||||
/tune move the thresholds and budgets, see what would have been sent
|
/tune move the thresholds and budgets, see what would have been sent
|
||||||
|
/config what each block matches, plus a box to try a new filter and save it
|
||||||
/digest the day's briefing, mobile-first
|
/digest the day's briefing, mobile-first
|
||||||
/healthz
|
/healthz
|
||||||
|
|
||||||
|
|||||||
+5
-2
@@ -36,7 +36,8 @@ file: `~/compose.yml.pre-news-v2`):
|
|||||||
# as uid 10001, not root, so without it SQLite fails with
|
# as uid 10001, not root, so without it SQLite fails with
|
||||||
# "attempt to write a readonly database".
|
# "attempt to write a readonly database".
|
||||||
- ./data/news2:/data:U
|
- ./data/news2:/data:U
|
||||||
- ./config/news2:/config:ro
|
# NOT :ro -- the /config page writes back to profile.toml.
|
||||||
|
- ./config/news2:/config
|
||||||
healthcheck:
|
healthcheck:
|
||||||
# NOT curl: the v0.2 image shipped none, so its healthcheck failed forever
|
# NOT curl: the v0.2 image shipped none, so its healthcheck failed forever
|
||||||
# while the service was fine and `podman ps` reported it unhealthy for days.
|
# while the service was fine and `podman ps` reported it unhealthy for days.
|
||||||
@@ -53,7 +54,9 @@ The image is tagged `localhost/news-py:latest` rather than reusing
|
|||||||
rollback path; the old database at `./data/news/db` is likewise untouched.
|
rollback path; the old database at `./data/news/db` is likewise untouched.
|
||||||
Rolling back is a two-line edit plus `podman-compose up -d news-backend`.
|
Rolling back is a two-line edit plus `podman-compose up -d news-backend`.
|
||||||
|
|
||||||
Config lives in `~/config/news2/` (copied from `profile/*.toml`).
|
Config lives in `~/config/news2/`, owned `10001:1000` and group-writable so the
|
||||||
|
container (uid 10001) can save from the `/config` page while you can still edit
|
||||||
|
the files by hand. `:U` would have worked for the container and locked you out.
|
||||||
|
|
||||||
The `caddy` service still bind-mounts `./data/news/web:/srv/news:ro`, which is
|
The `caddy` service still bind-mounts `./data/news/web:/srv/news:ro`, which is
|
||||||
now unused since the Caddyfile no longer serves a static SPA. Harmless; remove
|
now unused since the Caddyfile no longer serves a static SPA. Harmless; remove
|
||||||
|
|||||||
@@ -0,0 +1,210 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Config</title>
|
||||||
|
<style>
|
||||||
|
:root { color-scheme: light dark; --fg:#16181d; --bg:#fdfdfc; --muted:#6b7280;
|
||||||
|
--rule:#e4e4e7; --accent:#7c3aed; --card:#fff; --neg:#b91c1c;
|
||||||
|
--warn:#b45309; --pos:#15803d; }
|
||||||
|
@media (prefers-color-scheme: dark) {
|
||||||
|
:root { --fg:#e8e8ea; --bg:#141416; --muted:#9ca3af; --rule:#2a2a2e;
|
||||||
|
--accent:#a78bfa; --card:#1b1b1f; --neg:#f87171; --warn:#fbbf24;
|
||||||
|
--pos:#4ade80; }
|
||||||
|
}
|
||||||
|
* { box-sizing:border-box; }
|
||||||
|
body { margin:0; background:var(--bg); color:var(--fg);
|
||||||
|
font:15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif; }
|
||||||
|
main { max-width:54rem; margin:0 auto; padding:1.25rem 1rem 5rem; }
|
||||||
|
header { display:flex; gap:1rem; align-items:baseline; justify-content:space-between;
|
||||||
|
border-bottom:1px solid var(--rule); padding-bottom:.6rem; margin-bottom:1.1rem; }
|
||||||
|
h1 { font-size:1.15rem; margin:0; }
|
||||||
|
a { color:var(--accent); }
|
||||||
|
h2 { font-size:.82rem; text-transform:uppercase; letter-spacing:.05em;
|
||||||
|
color:var(--muted); margin:1.6rem 0 .55rem; }
|
||||||
|
.note { color:var(--muted); font-size:.83rem; }
|
||||||
|
.card { background:var(--card); border:1px solid var(--rule); border-radius:9px;
|
||||||
|
padding:.7rem .85rem; margin-bottom:.5rem; }
|
||||||
|
.row { display:flex; gap:.7rem; align-items:baseline; flex-wrap:wrap; }
|
||||||
|
.id { font-weight:600; font-size:.95rem; }
|
||||||
|
.meta { color:var(--muted); font-size:.79rem; }
|
||||||
|
.top { margin:.35rem 0 0; padding:0; list-style:none; }
|
||||||
|
.top li { font-size:.83rem; color:var(--muted); padding:.08rem 0; }
|
||||||
|
.sc { font-variant-numeric:tabular-nums; }
|
||||||
|
input[type=number], input[type=text], textarea {
|
||||||
|
font:inherit; background:transparent; color:inherit; border:1px solid var(--rule);
|
||||||
|
border-radius:5px; padding:.2rem .4rem; }
|
||||||
|
input[type=number] { width:4.6rem; }
|
||||||
|
textarea { width:100%; min-height:4.5rem; resize:vertical; }
|
||||||
|
button { font:inherit; font-size:.85rem; padding:.28rem .8rem; cursor:pointer;
|
||||||
|
border:1px solid var(--rule); background:transparent; color:inherit;
|
||||||
|
border-radius:6px; }
|
||||||
|
button.primary { border-color:var(--accent); color:var(--accent); }
|
||||||
|
.probe { background:var(--card); border:1px solid var(--rule); border-radius:9px;
|
||||||
|
padding:.9rem 1rem; }
|
||||||
|
.hits { display:flex; gap:1.2rem; align-items:baseline; margin:.7rem 0 .3rem; }
|
||||||
|
.big { font-size:1.6rem; font-weight:650; }
|
||||||
|
.caught li { color:var(--fg); }
|
||||||
|
.nearmiss { border-left:2px solid var(--warn); padding-left:.7rem; margin-top:.6rem; }
|
||||||
|
.ok { color:var(--pos); } .err { color:var(--neg); }
|
||||||
|
.gate { color:var(--warn); }
|
||||||
|
</style>
|
||||||
|
<main>
|
||||||
|
<header>
|
||||||
|
<h1>Config</h1>
|
||||||
|
<div class="note"><a href="/">triage</a> · <a href="/tune">tune</a> ·
|
||||||
|
<a href="/digest">digest</a> · {{ clusters }} clusters scored</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<h2>Try a filter</h2>
|
||||||
|
<div class="probe">
|
||||||
|
<p class="note" style="margin-top:0">
|
||||||
|
Describe what you want caught, as a sentence — this is compared by meaning,
|
||||||
|
not keywords, so write it the way an article would read. You see what it
|
||||||
|
catches before anything is saved.
|
||||||
|
</p>
|
||||||
|
<textarea id="text" placeholder="e.g. Cryptocurrency price movements and token launches"></textarea>
|
||||||
|
<div class="row" style="margin-top:.55rem">
|
||||||
|
<label class="note">threshold
|
||||||
|
<input type="number" id="thr" min="0.3" max="0.95" step="0.01" value="0.60"></label>
|
||||||
|
<button class="primary" id="run">Preview</button>
|
||||||
|
<span class="note" id="status"></span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="out" hidden>
|
||||||
|
<div class="hits">
|
||||||
|
<div class="big" id="count">0</div>
|
||||||
|
<div class="note">clusters caught of {{ clusters }}</div>
|
||||||
|
</div>
|
||||||
|
<ul class="top caught" id="matches"></ul>
|
||||||
|
<div class="nearmiss" id="nearbox" hidden>
|
||||||
|
<div class="note">just below the line — raise the threshold to exclude these,
|
||||||
|
lower it to include them</div>
|
||||||
|
<ul class="top" id="near"></ul>
|
||||||
|
</div>
|
||||||
|
<div class="row" style="margin-top:.8rem">
|
||||||
|
<input type="text" id="newid" placeholder="block id, e.g. crypto-prices">
|
||||||
|
<button id="save-veto">Save as veto</button>
|
||||||
|
<button id="save-demote">Save as demote</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2>Interests</h2>
|
||||||
|
{% for b in interests %}
|
||||||
|
<div class="card">
|
||||||
|
<div class="row">
|
||||||
|
<span class="id">{{ b.id }}</span>
|
||||||
|
<span class="meta">group {{ b.group }} · {{ b.probes }} probes</span>
|
||||||
|
{% if b.requires_any %}
|
||||||
|
<span class="meta gate">gated on {{ b.requires_any|length }} place terms</span>
|
||||||
|
{% endif %}
|
||||||
|
<span style="margin-left:auto"></span>
|
||||||
|
<label class="meta">weight
|
||||||
|
<input type="number" step="0.05" min="0" max="2" value="{{ b.weight }}"
|
||||||
|
data-kind="interest" data-id="{{ b.id }}" class="w"></label>
|
||||||
|
</div>
|
||||||
|
<ul class="top">
|
||||||
|
{% for t in b.top %}<li><span class="sc">{{ '%.2f'|format(t.score) }}</span> — {{ t.title[:78] }}</li>{% endfor %}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
<h2>Vetoes</h2>
|
||||||
|
{% for b in vetoes %}
|
||||||
|
<div class="card">
|
||||||
|
<div class="row">
|
||||||
|
<span class="id">{{ b.id }}</span>
|
||||||
|
<span class="meta">threshold {{ b.threshold }} · catches {{ b.hits }} clusters</span>
|
||||||
|
</div>
|
||||||
|
<ul class="top">
|
||||||
|
{% for t in b.top %}<li><span class="sc">{{ '%.2f'|format(t.score) }}</span> — {{ t.title[:78] }}</li>{% endfor %}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
<h2>Demotes</h2>
|
||||||
|
{% for b in demotes %}
|
||||||
|
<div class="card">
|
||||||
|
<div class="row">
|
||||||
|
<span class="id">{{ b.id }}</span>
|
||||||
|
<span class="meta">penalty {{ b.penalty }} · {{ b.probes }} probes</span>
|
||||||
|
</div>
|
||||||
|
<ul class="top">
|
||||||
|
{% for t in b.top %}<li><span class="sc">{{ '%.2f'|format(t.score) }}</span> — {{ t.title[:78] }}</li>{% endfor %}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
<p class="note" style="margin-top:1.6rem">
|
||||||
|
A <strong>veto</strong> excludes outright; a <strong>demote</strong> only
|
||||||
|
subtracts a penalty. Prefer a demote whenever the category sits next to
|
||||||
|
something you actually want — a hard veto on a sports-adjacent region once
|
||||||
|
killed a story about a mayor barring an Israeli football club, which is
|
||||||
|
Middle East politics.
|
||||||
|
</p>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const $ = id => document.getElementById(id);
|
||||||
|
let last = null;
|
||||||
|
|
||||||
|
async function preview() {
|
||||||
|
const text = $('text').value.trim();
|
||||||
|
if (!text) return;
|
||||||
|
$('status').textContent = 'embedding…';
|
||||||
|
try {
|
||||||
|
const r = await fetch('/config/preview', {
|
||||||
|
method: 'POST', headers: {'Content-Type': 'application/json'},
|
||||||
|
redirect: 'error',
|
||||||
|
body: JSON.stringify({text, threshold: parseFloat($('thr').value)}),
|
||||||
|
});
|
||||||
|
const d = await r.json();
|
||||||
|
last = {text, threshold: parseFloat($('thr').value)};
|
||||||
|
$('count').textContent = d.total;
|
||||||
|
$('matches').innerHTML = d.matches.map(m =>
|
||||||
|
`<li><span class="sc">${m.score.toFixed(2)}</span> — ${m.title}</li>`).join('')
|
||||||
|
|| '<li class="note">nothing above this threshold</li>';
|
||||||
|
$('near').innerHTML = (d.near_miss || []).map(m =>
|
||||||
|
`<li><span class="sc">${m.score.toFixed(2)}</span> — ${m.title}</li>`).join('');
|
||||||
|
$('nearbox').hidden = !(d.near_miss || []).length;
|
||||||
|
$('out').hidden = false;
|
||||||
|
$('status').textContent = '';
|
||||||
|
} catch (e) {
|
||||||
|
$('status').innerHTML = '<span class="err">preview failed</span>';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function save(kind) {
|
||||||
|
const id = $('newid').value.trim();
|
||||||
|
if (!id || !last) { $('status').innerHTML = '<span class="err">needs an id</span>'; return; }
|
||||||
|
const payload = {action: 'add', kind, id, probes: [last.text]};
|
||||||
|
if (kind === 'veto') payload.threshold = last.threshold; else payload.penalty = 1.0;
|
||||||
|
const r = await fetch('/config/save', {
|
||||||
|
method: 'POST', headers: {'Content-Type': 'application/json'},
|
||||||
|
redirect: 'error', body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
const d = await r.json();
|
||||||
|
$('status').innerHTML = d.ok
|
||||||
|
? '<span class="ok">saved — reload to see it applied</span>'
|
||||||
|
: `<span class="err">${d.error}</span>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
$('run').addEventListener('click', preview);
|
||||||
|
$('text').addEventListener('keydown', e => {
|
||||||
|
if (e.key === 'Enter' && (e.metaKey || e.ctrlKey)) preview();
|
||||||
|
});
|
||||||
|
$('save-veto').addEventListener('click', () => save('veto'));
|
||||||
|
$('save-demote').addEventListener('click', () => save('demote'));
|
||||||
|
|
||||||
|
document.querySelectorAll('input.w').forEach(inp => {
|
||||||
|
inp.addEventListener('change', async () => {
|
||||||
|
const r = await fetch('/config/save', {
|
||||||
|
method: 'POST', headers: {'Content-Type': 'application/json'}, redirect: 'error',
|
||||||
|
body: JSON.stringify({action: 'weight', kind: inp.dataset.kind,
|
||||||
|
id: inp.dataset.id, value: parseFloat(inp.value)}),
|
||||||
|
});
|
||||||
|
const d = await r.json();
|
||||||
|
inp.style.borderColor = d.ok ? 'var(--pos)' : 'var(--neg)';
|
||||||
|
});
|
||||||
|
});
|
||||||
|
</script>
|
||||||
@@ -69,6 +69,7 @@
|
|||||||
|
|
||||||
{% for r in rows %}
|
{% for r in rows %}
|
||||||
<article id="c{{ loop.index0 }}" data-id="{{ r.id }}"
|
<article id="c{{ loop.index0 }}" data-id="{{ r.id }}"
|
||||||
|
data-yes="{{ r.tok_yes }}" data-no="{{ r.tok_no }}"
|
||||||
class="{{ 'done' if r.labelled }}">
|
class="{{ 'done' if r.labelled }}">
|
||||||
<h2><a href="{{ r.sources[0].url if r.sources else '#' }}" target="_blank"
|
<h2><a href="{{ r.sources[0].url if r.sources else '#' }}" target="_blank"
|
||||||
rel="noopener">{{ r.headline }}</a></h2>
|
rel="noopener">{{ r.headline }}</a></h2>
|
||||||
@@ -93,6 +94,10 @@
|
|||||||
{% else %}
|
{% else %}
|
||||||
<p class="empty">Nothing left to label. Try <a href="?show=all">show all</a>.</p>
|
<p class="empty">Nothing left to label. Try <a href="?show=all">show all</a>.</p>
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
|
|
||||||
|
{% if not signed %}
|
||||||
|
<p class="empty">NEWS_FEEDBACK_SECRET is unset, so labelling is disabled.</p>
|
||||||
|
{% endif %}
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<div class="help">
|
<div class="help">
|
||||||
@@ -117,8 +122,16 @@
|
|||||||
const btns = card.querySelectorAll('button');
|
const btns = card.querySelectorAll('button');
|
||||||
btns.forEach(b => b.disabled = true);
|
btns.forEach(b => b.disabled = true);
|
||||||
try {
|
try {
|
||||||
const r = await fetch(`/ui/feedback/${card.dataset.id}/${kind}`, {method: 'POST'});
|
const token = kind === 'interested' ? card.dataset.yes : card.dataset.no;
|
||||||
|
// Same public, HMAC-signed endpoint the phone uses. `redirect: 'error'`
|
||||||
|
// matters: without it an expired session would follow a redirect to the
|
||||||
|
// login page, return 200, and this would report success having recorded
|
||||||
|
// nothing.
|
||||||
|
const r = await fetch(`/api/feedback/${card.dataset.id}/${kind}/${token}?src=web`,
|
||||||
|
{method: 'POST', redirect: 'error'});
|
||||||
if (!r.ok) throw new Error(r.status);
|
if (!r.ok) throw new Error(r.status);
|
||||||
|
const body = await r.json();
|
||||||
|
if (!body.ok) throw new Error(body.error || 'rejected');
|
||||||
card.classList.add('done');
|
card.classList.add('done');
|
||||||
// Advance, so a run of judgements is one keystroke each.
|
// Advance, so a run of judgements is one keystroke each.
|
||||||
if (cards[cur] === card) select(cur + 1);
|
if (cards[cur] === card) select(cur + 1);
|
||||||
|
|||||||
+157
-22
@@ -12,6 +12,7 @@ import os
|
|||||||
from datetime import UTC, date, datetime, timedelta
|
from datetime import UTC, date, datetime, timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
from starlette.applications import Starlette
|
from starlette.applications import Starlette
|
||||||
from starlette.responses import JSONResponse, PlainTextResponse
|
from starlette.responses import JSONResponse, PlainTextResponse
|
||||||
from starlette.routing import Route
|
from starlette.routing import Route
|
||||||
@@ -19,7 +20,7 @@ from starlette.templating import Jinja2Templates
|
|||||||
|
|
||||||
from . import store
|
from . import store
|
||||||
from .config import Config
|
from .config import Config
|
||||||
from .notify import verify
|
from .notify import sign, verify
|
||||||
|
|
||||||
TEMPLATES = Jinja2Templates(directory=str(Path(__file__).parent / "templates"))
|
TEMPLATES = Jinja2Templates(directory=str(Path(__file__).parent / "templates"))
|
||||||
|
|
||||||
@@ -94,6 +95,7 @@ async def triage(request):
|
|||||||
route(con, scores, cfg.routing, cfg.delivery,
|
route(con, scores, cfg.routing, cfg.delivery,
|
||||||
datetime.now(UTC), cfg.block_groups())}
|
datetime.now(UTC), cfg.block_groups())}
|
||||||
|
|
||||||
|
secret = os.environ.get("NEWS_FEEDBACK_SECRET")
|
||||||
labelled = {r["cluster_id"] for r in con.execute("SELECT DISTINCT cluster_id FROM feedback")}
|
labelled = {r["cluster_id"] for r in con.execute("SELECT DISTINCT cluster_id FROM feedback")}
|
||||||
show = request.query_params.get("show", "unlabelled")
|
show = request.query_params.get("show", "unlabelled")
|
||||||
limit = min(int(request.query_params.get("limit", 60)), 200)
|
limit = min(int(request.query_params.get("limit", 60)), 200)
|
||||||
@@ -118,6 +120,13 @@ async def triage(request):
|
|||||||
"origins": s.corroboration.origins, "demote": s.demote,
|
"origins": s.corroboration.origins, "demote": s.demote,
|
||||||
"lane": d.lane if d else "?", "reason": d.reason if d else "",
|
"lane": d.lane if d else "?", "reason": d.reason if d else "",
|
||||||
"labelled": s.cluster_id in labelled,
|
"labelled": s.cluster_id in labelled,
|
||||||
|
# Signed here so the browser posts to the SAME public endpoint the
|
||||||
|
# phone uses. The alternative -- an Authelia-gated path -- fails
|
||||||
|
# silently when a session expires: fetch() follows the redirect to
|
||||||
|
# the login page, gets a 200, and the UI reports success while
|
||||||
|
# nothing was recorded.
|
||||||
|
"tok_yes": sign(secret, s.cluster_id, "interested") if secret else "",
|
||||||
|
"tok_no": sign(secret, s.cluster_id, "not_interested") if secret else "",
|
||||||
"sources": [{"name": m["name"], "title": m["title"], "url": m["url"]}
|
"sources": [{"name": m["name"], "title": m["title"], "url": m["url"]}
|
||||||
for m in members],
|
for m in members],
|
||||||
})
|
})
|
||||||
@@ -126,26 +135,8 @@ async def triage(request):
|
|||||||
|
|
||||||
return TEMPLATES.TemplateResponse(
|
return TEMPLATES.TemplateResponse(
|
||||||
request, "triage.html",
|
request, "triage.html",
|
||||||
{"rows": rows, "show": show, "total": len(scores), "labelled": len(labelled)})
|
{"rows": rows, "show": show, "total": len(scores), "labelled": len(labelled),
|
||||||
|
"signed": bool(secret)})
|
||||||
|
|
||||||
async def web_feedback(request):
|
|
||||||
"""Feedback from the browser.
|
|
||||||
|
|
||||||
Deliberately NOT under /api/feedback/, which Caddy routes around Authelia
|
|
||||||
for the ntfy buttons. This path stays behind the identity gateway, so it
|
|
||||||
needs no signature.
|
|
||||||
"""
|
|
||||||
from datetime import UTC, datetime
|
|
||||||
|
|
||||||
kind = request.path_params["kind"]
|
|
||||||
if kind not in ("interested", "not_interested"):
|
|
||||||
return JSONResponse({"error": "bad kind"}, status_code=400)
|
|
||||||
con = _con()
|
|
||||||
con.execute(
|
|
||||||
"INSERT INTO feedback (cluster_id, kind, source, created_at) VALUES (?,?,?,?)",
|
|
||||||
(request.path_params["cluster_id"], kind, "web", datetime.now(UTC).isoformat()))
|
|
||||||
return JSONResponse({"ok": True})
|
|
||||||
|
|
||||||
|
|
||||||
async def tune(request):
|
async def tune(request):
|
||||||
@@ -208,6 +199,148 @@ async def tune(request):
|
|||||||
"defaults": cfg.routing})
|
"defaults": cfg.routing})
|
||||||
|
|
||||||
|
|
||||||
|
def _centroids(con):
|
||||||
|
rows = con.execute(
|
||||||
|
"""SELECT c.id, c.centroid, (SELECT title FROM items WHERE id=c.lead_item_id) AS lead
|
||||||
|
FROM clusters c ORDER BY c.last_seen_at DESC LIMIT 2000""").fetchall()
|
||||||
|
if not rows:
|
||||||
|
return [], np.zeros((0, 1), dtype=np.float32)
|
||||||
|
mat = np.stack([store.from_blob(r["centroid"]) for r in rows])
|
||||||
|
mat /= np.maximum(np.linalg.norm(mat, axis=1, keepdims=True), 1e-12)
|
||||||
|
return rows, mat
|
||||||
|
|
||||||
|
|
||||||
|
async def config_page(request):
|
||||||
|
"""What each block currently matches, and a box to try a new filter.
|
||||||
|
|
||||||
|
The point is to make narrowing down cheap. The old system's config view was
|
||||||
|
read-only and showed no consequences, so tuning meant editing a file,
|
||||||
|
restarting, and waiting a poll cycle to find out what changed.
|
||||||
|
"""
|
||||||
|
cfg = Config()
|
||||||
|
con = _con()
|
||||||
|
profile = cfg.profile()
|
||||||
|
rows, mat = _centroids(con)
|
||||||
|
|
||||||
|
from .embed import Embedder
|
||||||
|
from .score import ProfileVectors
|
||||||
|
|
||||||
|
global _VECTORS
|
||||||
|
if _VECTORS is None:
|
||||||
|
_VECTORS = ProfileVectors(con, profile, Embedder.from_config(cfg))
|
||||||
|
|
||||||
|
def summarise(kind: str, key: str, default: float):
|
||||||
|
out = []
|
||||||
|
for block in profile.get(kind, []):
|
||||||
|
sims = None
|
||||||
|
for bid, probes, _ in _VECTORS.blocks[kind]:
|
||||||
|
if bid == block["id"]:
|
||||||
|
sims = (mat @ probes.T).max(axis=1) if len(mat) else np.zeros(0)
|
||||||
|
thr = float(block.get(key, default))
|
||||||
|
hits = int((sims >= thr).sum()) if sims is not None else 0
|
||||||
|
top = []
|
||||||
|
if sims is not None and len(sims):
|
||||||
|
for i in np.argsort(-sims)[:3]:
|
||||||
|
top.append({"title": rows[i]["lead"], "score": float(sims[i])})
|
||||||
|
out.append({"id": block["id"], "weight": block.get("weight", 1.0),
|
||||||
|
"group": block.get("group", block["id"]),
|
||||||
|
"probes": len(block["probes"]), "threshold": thr,
|
||||||
|
"penalty": block.get("penalty"), "hits": hits, "top": top,
|
||||||
|
"requires_any": block.get("requires_any", [])})
|
||||||
|
return out
|
||||||
|
|
||||||
|
return TEMPLATES.TemplateResponse(
|
||||||
|
request, "config.html",
|
||||||
|
{"interests": summarise("interest", "___", 0.60),
|
||||||
|
"vetoes": summarise("veto", "threshold", 0.60),
|
||||||
|
"demotes": summarise("demote", "___", 0.55),
|
||||||
|
"clusters": len(rows), "scoring": cfg.scoring, "routing": cfg.routing})
|
||||||
|
|
||||||
|
|
||||||
|
async def config_preview(request):
|
||||||
|
"""Embed an arbitrary sentence and show what it would catch, right now."""
|
||||||
|
body = await request.json()
|
||||||
|
text = (body.get("text") or "").strip()
|
||||||
|
threshold = float(body.get("threshold", 0.60))
|
||||||
|
if not text:
|
||||||
|
return JSONResponse({"matches": [], "total": 0})
|
||||||
|
|
||||||
|
cfg = Config()
|
||||||
|
con = _con()
|
||||||
|
rows, mat = _centroids(con)
|
||||||
|
if not len(mat):
|
||||||
|
return JSONResponse({"matches": [], "total": 0})
|
||||||
|
|
||||||
|
from .embed import Embedder
|
||||||
|
|
||||||
|
vec = Embedder.from_config(cfg).encode([text])[0]
|
||||||
|
sims = mat @ vec
|
||||||
|
order = np.argsort(-sims)
|
||||||
|
matches = [{"title": rows[i]["lead"], "score": round(float(sims[i]), 3)}
|
||||||
|
for i in order[:25] if sims[i] >= threshold]
|
||||||
|
return JSONResponse({
|
||||||
|
"matches": matches,
|
||||||
|
"total": int((sims >= threshold).sum()),
|
||||||
|
"clusters": len(rows),
|
||||||
|
# The boundary is where judgement is needed, so show what sits just
|
||||||
|
# below the line as well as what is caught.
|
||||||
|
"near_miss": [{"title": rows[i]["lead"], "score": round(float(sims[i]), 3)}
|
||||||
|
for i in order if threshold - 0.06 <= sims[i] < threshold][:5],
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
async def config_save(request):
|
||||||
|
"""Write a change back to profile.toml, preserving comments.
|
||||||
|
|
||||||
|
tomlkit rather than tomllib+dump: this file carries the reasoning behind
|
||||||
|
every constant, and a round-trip that strips comments would quietly destroy
|
||||||
|
the record of why each number is what it is.
|
||||||
|
"""
|
||||||
|
import tomlkit
|
||||||
|
|
||||||
|
body = await request.json()
|
||||||
|
cfg = Config()
|
||||||
|
path = cfg.profile_path
|
||||||
|
doc = tomlkit.parse(path.read_text())
|
||||||
|
|
||||||
|
action = body.get("action")
|
||||||
|
if action == "weight":
|
||||||
|
for block in doc.get(body["kind"], []):
|
||||||
|
if block["id"] == body["id"]:
|
||||||
|
block["weight"] = float(body["value"])
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
return JSONResponse({"error": "block not found"}, status_code=404)
|
||||||
|
elif action == "add":
|
||||||
|
kind = body["kind"]
|
||||||
|
if kind not in ("veto", "demote"):
|
||||||
|
return JSONResponse({"error": "kind must be veto or demote"}, status_code=400)
|
||||||
|
table = tomlkit.table()
|
||||||
|
table["id"] = body["id"]
|
||||||
|
if kind == "veto":
|
||||||
|
table["threshold"] = float(body.get("threshold", 0.60))
|
||||||
|
else:
|
||||||
|
table["penalty"] = float(body.get("penalty", 1.0))
|
||||||
|
arr = tomlkit.array().multiline(True)
|
||||||
|
for probe in body["probes"]:
|
||||||
|
arr.append(probe)
|
||||||
|
table["probes"] = arr
|
||||||
|
doc.setdefault(kind, tomlkit.aot()).append(table)
|
||||||
|
else:
|
||||||
|
return JSONResponse({"error": "unknown action"}, status_code=400)
|
||||||
|
|
||||||
|
try:
|
||||||
|
path.write_text(tomlkit.dumps(doc))
|
||||||
|
except OSError as exc:
|
||||||
|
return JSONResponse(
|
||||||
|
{"error": f"cannot write {path}: {exc}. Is /config mounted read-only?"},
|
||||||
|
status_code=500)
|
||||||
|
|
||||||
|
global _VECTORS
|
||||||
|
_VECTORS = None # probes changed; re-embed on the next request
|
||||||
|
return JSONResponse({"ok": True})
|
||||||
|
|
||||||
|
|
||||||
async def digest_page(request):
|
async def digest_page(request):
|
||||||
day = request.path_params.get("day")
|
day = request.path_params.get("day")
|
||||||
try:
|
try:
|
||||||
@@ -256,8 +389,10 @@ async def digest_page(request):
|
|||||||
|
|
||||||
routes = [
|
routes = [
|
||||||
Route("/", triage),
|
Route("/", triage),
|
||||||
Route("/ui/feedback/{cluster_id}/{kind}", web_feedback, methods=["POST"]),
|
|
||||||
Route("/tune", tune),
|
Route("/tune", tune),
|
||||||
|
Route("/config", config_page),
|
||||||
|
Route("/config/preview", config_preview, methods=["POST"]),
|
||||||
|
Route("/config/save", config_save, methods=["POST"]),
|
||||||
Route("/healthz", healthz),
|
Route("/healthz", healthz),
|
||||||
Route("/api/feedback/{cluster_id}/{kind}/{token}", feedback, methods=["POST"]),
|
Route("/api/feedback/{cluster_id}/{kind}/{token}", feedback, methods=["POST"]),
|
||||||
Route("/digest", digest_page),
|
Route("/digest", digest_page),
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ dependencies = [
|
|||||||
"jinja2>=3.1",
|
"jinja2>=3.1",
|
||||||
"uvicorn>=0.30",
|
"uvicorn>=0.30",
|
||||||
"python-multipart>=0.0.9",
|
"python-multipart>=0.0.9",
|
||||||
|
"tomlkit>=0.15.1",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
|
|||||||
+29
-1
@@ -105,9 +105,12 @@ def client(tmp_path, monkeypatch):
|
|||||||
published_at,first_seen_at,content_hash) VALUES
|
published_at,first_seen_at,content_hash) VALUES
|
||||||
('i1','s1','Council votes on transit','','http://b/1','http://b/1',?,?,'h')""",
|
('i1','s1','Council votes on transit','','http://b/1','http://b/1',?,?,'h')""",
|
||||||
(now.isoformat(), now.isoformat()))
|
(now.isoformat(), now.isoformat()))
|
||||||
|
# 384 dims: the triage page scores against the real profile probes, so a
|
||||||
|
# short centroid blows up in the matmul.
|
||||||
|
import numpy as np
|
||||||
con.execute("""INSERT INTO clusters (id,centroid,size,first_seen_at,last_seen_at,lead_item_id)
|
con.execute("""INSERT INTO clusters (id,centroid,size,first_seen_at,last_seen_at,lead_item_id)
|
||||||
VALUES ('c1',?,1,?,?,'i1')""",
|
VALUES ('c1',?,1,?,?,'i1')""",
|
||||||
(store.to_blob(__import__("numpy").zeros(3, dtype="float32")),
|
(store.to_blob(np.zeros(384, dtype="float32")),
|
||||||
now.isoformat(), now.isoformat()))
|
now.isoformat(), now.isoformat()))
|
||||||
con.execute("INSERT INTO cluster_members VALUES ('c1','i1',?)", (now.isoformat(),))
|
con.execute("INSERT INTO cluster_members VALUES ('c1','i1',?)", (now.isoformat(),))
|
||||||
con.execute("""INSERT INTO notification_log (cluster_id,lane,score,created_at)
|
con.execute("""INSERT INTO notification_log (cluster_id,lane,score,created_at)
|
||||||
@@ -116,6 +119,7 @@ def client(tmp_path, monkeypatch):
|
|||||||
VALUES (?,'The council met.','m',?)""",
|
VALUES (?,'The council met.','m',?)""",
|
||||||
(now.date().isoformat(), now.isoformat()))
|
(now.date().isoformat(), now.isoformat()))
|
||||||
from news import web
|
from news import web
|
||||||
|
web._VECTORS = None # module-level cache must not leak across tests
|
||||||
return TestClient(web.app), now
|
return TestClient(web.app), now
|
||||||
|
|
||||||
|
|
||||||
@@ -204,3 +208,27 @@ def test_feedback_survives_a_concurrent_writer(tmp_path, monkeypatch):
|
|||||||
finally:
|
finally:
|
||||||
stop.set()
|
stop.set()
|
||||||
t.join(timeout=5)
|
t.join(timeout=5)
|
||||||
|
|
||||||
|
|
||||||
|
def test_browser_labels_use_the_same_signed_public_endpoint(client):
|
||||||
|
"""The browser must not post to an Authelia-gated path.
|
||||||
|
|
||||||
|
fetch() follows redirects, so with an expired session a gated POST would go
|
||||||
|
302 -> login page -> HTTP 200, and the UI would report success having
|
||||||
|
recorded nothing. Both surfaces use the signed public endpoint instead.
|
||||||
|
"""
|
||||||
|
c, _ = client
|
||||||
|
html = c.get("/").text
|
||||||
|
assert "/api/feedback/" in html
|
||||||
|
assert "/ui/feedback/" not in html, "the gated browser route must be gone"
|
||||||
|
assert "redirect: 'error'" in html, "a redirect must fail loudly, not look like success"
|
||||||
|
assert html.count("data-yes=") > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_labelling_is_disabled_when_no_secret_is_configured(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("NEWS_DB", str(tmp_path / "n.db"))
|
||||||
|
monkeypatch.delenv("NEWS_FEEDBACK_SECRET", raising=False)
|
||||||
|
store.connect(tmp_path / "n.db")
|
||||||
|
from news import web
|
||||||
|
web._VECTORS = None
|
||||||
|
assert "labelling is disabled" in TestClient(web.app).get("/").text
|
||||||
Reference in new issue
Block a user