The pages were only reachable by typing URLs, and the digest had no navigation
at all. A single _nav.html partial now appears on all four, marking the current
page, so Triage / Digest / Tune / Config are one click apart.
Parameterised test asserts every page links to the others and marks itself
active, so a page added later cannot ship stranded.
Connor pointed out he may use the site from a laptop away from home. The gated
/ui/feedback/ path was worse than just inconvenient: fetch() follows redirects,
so an expired Authelia session would go 302 -> login page -> HTTP 200, and the
UI would mark the card labelled having recorded nothing. Silent data loss on the
one signal the system learns from.
Both surfaces now post to the same HMAC-signed public endpoint, with
`redirect: 'error'` so a redirect fails loudly instead of looking like success.
The gated route is gone -- one mechanism, not two.
/config shows what every block actually matches right now, and a box to try a
filter before saving it: describe what you want caught, see what it catches and
what sits just below the line, move the threshold, then save it as a veto or a
demote. That boundary view is the point -- a crypto filter at 0.55 catches 131
clusters including "Security updates for Monday", and at 0.62 catches 4, which
you can only know by looking.
Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries
the reasoning behind every constant and a comment-stripping round-trip would
quietly destroy the record of why each number is what it is. Verified: 97
top-level comment lines in, 97 out.
The config mount had to stop being :ro, and ~/config/news2 is now owned
10001:1000 group-writable so the container can save while hand-editing still
works. `:U` would have fixed the container and locked the user out.
Connor spotted a Missouri redistricting story labelled ohio-state-politics.
Measuring showed that was not an isolated slip: 66% of the winners of a
place-based block named no place at all. Glastonbury, Anthony Albanese and a
Tim Montgomerie story were all scoring as italy-mexico; a singing-apraxia story
was ohio-state-politics.
Embeddings model meaning well and named entities poorly -- "Italian politics"
and "Australian politics" are nearly the same vector. Place is identity, not
meaning, so it is matched lexically, which is the same division of labour that
makes lexical comparison right for syndication detection and wrong for
clustering. Blocks carry `requires_any`; a block cannot win for an item whose
text names none of its terms.
Word boundaries, not substrings: "rome" is inside "chrome" and "mali" inside
"normalise". That is the trap the old blocklist fell into with "nfl" inside
"conflict".
The first implementation was wrong in a way worth recording, because it looked
correct and measured almost unchanged (66% -> 30%). It wrote a sentinel into the
RAW score before normalisation -- but when most items are gated, that sentinel
becomes the block's own mean, so a gated item z-scored to roughly zero and could
still win. Normalisation erased the gate. Applied after normalisation instead:
66% -> 0%, and place-block winners fell from 140 to 10, so 130 items now land in
the block they actually belong to.
Found in production, not in tests. SQLite's default busy timeout is ZERO, so
when a poll held the write lock while storing ~1,500 embeddings, a concurrent
feedback POST failed instantly with "database is locked" rather than waiting.
Polls take minutes and run every fifteen, so this was a wide failure window on
the single endpoint that must never drop a write -- feedback is the only way the
taste model learns anything, and the previous system's entire failure was
collecting 32 labels.
Connections now set busy_timeout=15s. Regression test included, and its first
draft was wrong in an instructive way: the competing thread held the write lock
continuously, which no finite timeout can survive. It now writes in bursts with
gaps, like a real poll cycle, which is the contention actually being tested.
Also cleans up the static-site plumbing left over from the Rust service: the
caddy /srv/news bind mount, the built SPA under data/news/web, and the
news-update systemd units -- which turned out to be DANGLING symlinks into
docs/projects/news/deploy/, the path the old repo used before being renamed to
news_triage, and had been failing every ten minutes unnoticed ever since.
ntfy push is deliberately short: a title, one line, and a Click through to the
story. The digest is a WEB PAGE, not a push payload -- the old system sent the
whole report as one long markdown message and ntfy turned it into a .md
attachment, which is what made it miserable to read on a phone.
Feedback buttons finally work. The old ones never fired once: NEWS_API_BASE_URL
was unset so they were dropped behind a single boot WARN, and /api/feedback had
no auth anyway. ntfy runs http actions FROM THE PHONE, so the endpoint must be
public -- which is why each button now carries an HMAC signed over
(cluster_id, kind). A leaked URL authorises one vote on one story rather than
acting as a bearer token for the whole model. Missing config drops the buttons
loudly instead of silently.
Digest prose is one LLM call per day through LiteLLM, and that is the entire
generative-model surface of this system -- clustering, relevance, importance and
vetoes are all embeddings and arithmetic. Feed text is passed as JSON data,
never interpolated into prompt prose, so a headline cannot read as an
instruction. Any failure falls back to the plain list; the digest never depends
on a model being reachable. Verified against the live proxy: 20s, themed
paragraphs, no invented facts.
Digest page is mobile-first, dark-mode aware, and renders from notification_log
so it is a record of what was actually sent.
deploy/README.md documents the Caddy vhost and env the buttons require, and uses
a python healthcheck rather than curl -- the v0.2 image shipped no curl, so its
healthcheck failed forever while the service was fine.
62 tests.
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
Connor's point: "several outlets carried this" covers situations that carry
different amounts of information, and origin_count() flattened them all to one
number. Splitting on same/different newsroom and same/different text gives four
cases:
same newsroom, same text -> pure duplicate (BBC World + BBC top). Zero
signal; the front-page placement it does carry is
already captured by sources.prominence.
same newsroom, diff text -> DEPTH. A newsroom filing four pieces on one story
is a real editorial judgement, previously
discarded entirely.
diff newsroom, same text -> PICKUPS. Syndicated copy; each carrier chose to
run it. Weak but non-zero.
diff newsroom, diff text -> ORIGINS. Independent judgement and independent
newsgathering. Strongest.
score = origins + 0.3*log1p(pickups) + 0.2*log1p(depth), weights in config.
Origins stay dominant; the others break ties among the many two- and
three-origin clusters, and log damping stops a high-volume feed inflating depth.
Measuring first changed the design. Depth outnumbers pickups 78 to 11 across 91
real clusters, because these sources are mostly national outlets writing their
own copy rather than local papers running wire -- so the more valuable half was
the one not originally proposed. UK settlements scores 10.55 from 10 origins
plus 15 same-newsroom pieces; AfD scores 5.55 from 5 origins plus 15.
Tests assert the four-case ordering directly, and that twenty pieces from one
newsroom still rank below two independent ones. Both needed a helper generating
vectors with exact pairwise cosine: the band between the 0.82 clustering
threshold and the 0.97 syndication threshold is narrow, and two earlier drafts
of these tests used ad-hoc vectors that fell outside it and asserted nothing.
Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.
Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).
On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.
Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.
First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
Reading real clustered output caught a config error the tests could not:
Ohio Capital Journal and Kentucky Lantern are both States Newsroom outlets
syndicating the same national desk, and carried 10 byte-identical headlines in a
single poll -- more than any other source pair. Configured as separate buckets
they would have doubled the corroboration signal on every syndicated story,
which is exactly the failure the bucket concept exists to prevent.
Found by checking identical headlines across all source pairs rather than
spot-fixing the one noticed by eye. That sweep also confirmed the buckets
already set deliberately are right: BBC Top/World, Ars Technica/Science and
NPR News/Politics each share a bucket and each showed shared headlines.
Regression test added.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a
cycle; `news sources` and `news clusters` inspect the result. Replay and the
live service share run_cycle so tuning and production cannot diverge -- the old
system's replay harness hardcoded default params and could never sweep the
thresholds it existed to tune.
Calibrated, not guessed:
- Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup
capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76
items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The
old system produced 515 clusters of which 514 were singletons.
- Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting
one wire story is one confirmation rather than ten.
Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of
separation on titles and -0.2 with summaries -- news duplicates are paraphrases,
which is the same lexical trap that sank the old Jaccard clustering.
Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every
feed URL times out from this host) and Tech Policy Press (serves HTML at every
documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss
works). Reddit moved to .rss because .json now 403s without OAuth, and its
intervals raised because it 429s after three quick requests. Deutsche Welle is
back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled
quick-xml parser could not.
connor