Phase 2: ingest, embed, cluster
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a cycle; `news sources` and `news clusters` inspect the result. Replay and the live service share run_cycle so tuning and production cannot diverge -- the old system's replay harness hardcoded default params and could never sweep the thresholds it existed to tune. Calibrated, not guessed: - Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76 items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The old system produced 515 clusters of which 514 were singletons. - Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting one wire story is one confirmation rather than ten. Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of separation on titles and -0.2 with summaries -- news duplicates are paraphrases, which is the same lexical trap that sank the old Jaccard clustering. Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every feed URL times out from this host) and Tech Policy Press (serves HTML at every documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss works). Reddit moved to .rss because .json now 403s without OAuth, and its intervals raised because it 429s after three quick requests. Deutsche Welle is back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled quick-xml parser could not.
This commit is contained in:
1 parent
c89a723603
commit
64690ecfe0
18 files changed
+1338
No files matched your search
@@ -203,6 +203,13 @@ probes = [
|
||||
# 3,885 items in the v0.2 database and Connor's 40 real feedback labels.
|
||||
# See profile/CALIBRATION.md for the measurements.
|
||||
|
||||
[delivery]
|
||||
# Settled with Connor 2026-09-08.
|
||||
digest_hour = "08:45"
|
||||
quiet_hours = { start = "22:00", end = "07:00" }
|
||||
# The importance lane MAY override quiet hours. The interest lane never may.
|
||||
importance_may_wake = true
|
||||
|
||||
[scoring]
|
||||
# Compare blocks by z-score WITHIN each block, never by raw cosine. Raw cosine
|
||||
# is not comparable across blocks -- "geopolitics" averages 0.501 similarity to
|
||||
|
||||
Reference in new issue
Block a user