Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Second elicitation round. The significant correction is Connor's own framing:
"my actual war interest is low, but unfortunately all wars are important." The
click history looked war-heavy only because world-news feeds are war-heavy, so
geopolitics-at-weight-1.0 was reading the corpus, not the reader. Statecraft is
now its own block and armed conflict is five weighted bands (anti-imperial,
US-involved, Mideast/Israel, undercovered, general) routed mainly by importance.
New: space-aerospace at 0.95, the highest-weighted block -- Connor is an
aerospace engineer. Plus science-discovery, environment-ecology,
ohio-state-politics, left-movements, anticolonial-sovereignty, far-right-watch,
political-theory. Linux/ricing/neovim folded into software-systems.
Introduces `demote` blocks, distinct from hard vetoes, for categories adjacent to
real interests: crypto/AI hype sits next to AI regulation, culture-war outrage
next to far-right-watch, true crime next to criminal-justice policy. Hard-vetoing
those would take real stories with them -- the lesson from the Europa League
story that both a substring blocklist and a cosine threshold wrongly killed.
Sources 39 -> 56, all fetched and verified: space (SpaceNews, Spaceflight Now,
NASASpaceflight, NASA, ESA, Phys.org), Linux (Phoronix, LWN, It's FOSS,
r/unixporn), left/anticolonial (Intercept, Drop Site, Tribune, African
Arguments). Rejected on inspection: nature.rss is journal metadata ("Author
Correction: ...") and sciencedaily is a press-release mill.
Embedding stays English-only, measured rather than assumed: the same story in
Spanish and English scored 0.758 and 0.600 cosine against a random-pair p99 of
0.634, so one pair was indistinguishable from an unrelated article. Italy and
Mexico are covered by English-language sources instead.
Also fixes clustering performance: centroids are held in memory across a batch
instead of one SELECT plus a numpy rebuild per item, which had taken ~15 minutes
for 1,258 items.
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a
cycle; `news sources` and `news clusters` inspect the result. Replay and the
live service share run_cycle so tuning and production cannot diverge -- the old
system's replay harness hardcoded default params and could never sweep the
thresholds it existed to tune.
Calibrated, not guessed:
- Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup
capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76
items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The
old system produced 515 clusters of which 514 were singletons.
- Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting
one wire story is one confirmation rather than ten.
Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of
separation on titles and -0.2 with summaries -- news duplicates are paraphrases,
which is the same lexical trap that sank the old Jaccard clustering.
Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every
feed URL times out from this host) and Tech Policy Press (serves HTML at every
documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss
works). Reddit moved to .rss because .json now 403s without OAuth, and its
intervals raised because it 429s after three quick requests. Deutsche Welle is
back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled
quick-xml parser could not.
Ground-up replacement for news_triage, whose signal quality failed because it
had no representation of what its user likes -- three keywords fed to BM25 plus
a two-term substring blocklist.
profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto
blocks, 5 importance blocks, each a set of probe sentences to be embedded.
The [scoring] constants are fitted, not guessed. CALIBRATION.md records the
measurements against all 3,885 items in the v0.2 database and 40 real feedback
labels, including two findings that changed the design:
- compare blocks by z-score within block, never raw cosine; raw cosine let
geopolitics win 78% of items and buried known-liked items at median rank 1246
of 3885 (293 after normalizing)
- veto needs a margin over the interest signal, not a threshold; a plain 0.55
cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent
in Europa League", which the old substring blocklist also killed
connor