Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Ground-up replacement for news_triage, whose signal quality failed because it
had no representation of what its user likes -- three keywords fed to BM25 plus
a two-term substring blocklist.
profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto
blocks, 5 importance blocks, each a set of probe sentences to be embedded.
The [scoring] constants are fitted, not guessed. CALIBRATION.md records the
measurements against all 3,885 items in the v0.2 database and 40 real feedback
labels, including two findings that changed the design:
- compare blocks by z-score within block, never raw cosine; raw cosine let
geopolitics win 78% of items and buried known-liked items at median rank 1246
of 3885 (293 after normalizing)
- veto needs a margin over the interest signal, not a threshold; a plain 0.55
cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent
in Europa League", which the old substring blocklist also killed
connor