d4631ec13d1c9fad97bcd78f1e328c9c49fa7c35
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 connor
|
d4631ec13d |
Stop one subject taking every notification slot
Connor noticed 5 of 5 alerts were Middle East. Measuring showed that was not proportionate: Middle East blocks are ~11% of the eligible corpus while software-systems alone is 25%, yet Middle East took 35% of the top 20 and software took none. Two mechanisms, both measured: 1. Five conflict-* blocks could each claim their own alert slot. Interest blocks now carry a `group`, and routing enforces max_alerts_per_group_per_day, so related blocks share one quota. This alone moved the mix from 5/5 Middle East to three distinct groups. 2. Z-scoring subtracts each block's MEAN, which silently encodes how much of the corpus that block matches. software-systems has the HIGHEST mean raw similarity of any block (0.537) and the LOWEST max z (2.51); conflict-mideast-israel has a lower mean (0.460) and reaches 3.47. A topic is penalised for being well covered in your own feed list. Added `normalize = "top-anchored"` as an alternative that rescales each block by its own upper tail instead of its mean, leaving `weight` to express preference rather than corpus composition. Also ruled out a plausible-sounding cause rather than assuming it: HN and Lobsters items are title-only and much shorter than news articles, so terse titles seemed likely to match prose probes poorly. Measured the opposite -- title-only HN items score 0.618 against the software block, ABOVE Phoronix/LWN with summaries (0.610) and Ars Technica (0.540). Short text is not the problem; prevalence is. |
||
|
connor
|
23966a2d63 |
Phase 3: two-axis scoring and routing
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
|
||
|
connor
|
a2c812794d |
Tiered corroboration: origins, pickups and depth scored separately
Connor's point: "several outlets carried this" covers situations that carry
different amounts of information, and origin_count() flattened them all to one
number. Splitting on same/different newsroom and same/different text gives four
cases:
same newsroom, same text -> pure duplicate (BBC World + BBC top). Zero
signal; the front-page placement it does carry is
already captured by sources.prominence.
same newsroom, diff text -> DEPTH. A newsroom filing four pieces on one story
is a real editorial judgement, previously
discarded entirely.
diff newsroom, same text -> PICKUPS. Syndicated copy; each carrier chose to
run it. Weak but non-zero.
diff newsroom, diff text -> ORIGINS. Independent judgement and independent
newsgathering. Strongest.
score = origins + 0.3*log1p(pickups) + 0.2*log1p(depth), weights in config.
Origins stay dominant; the others break ties among the many two- and
three-origin clusters, and log damping stops a high-volume feed inflating depth.
Measuring first changed the design. Depth outnumbers pickups 78 to 11 across 91
real clusters, because these sources are mostly national outlets writing their
own copy rather than local papers running wire -- so the more valuable half was
the one not originally proposed. UK settlements scores 10.55 from 10 origins
plus 15 same-newsroom pieces; AfD scores 5.55 from 5 origins plus 15.
Tests assert the four-case ordering directly, and that twenty pieces from one
newsroom still rank below two independent ones. Both needed a helper generating
vectors with exact pairwise cosine: the band between the 0.82 clustering
threshold and the 0.97 syndication threshold is narrow, and two earlier drafts
of these tests used ad-hoc vectors that fell outside it and asserted nothing.
|
||
|
connor
|
d57b9e092c |
Detect syndication from text, not just from configured ownership
Connor's observation: two outlets running the same story can be caught by comparing the text, without anyone having to know who owns whom. Corroboration is now origin_count(), a union-find where two items share an origin if EITHER they are in the same bucket, OR their text is near-identical (normalised title match, or cosine >= 0.97). The rules are complementary: buckets are same-owner/different-text (NYT filing five distinct AfD pieces), text matching is different-owner/same-text (a local station reprinting wire). On real data it fires exactly where it should: the vote-by-mail cluster collapses 2 buckets -> 1 origin, catching the States Newsroom syndication automatically -- the case previously fixed by hand in config -- and a Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no static bucket can express since WCPO is genuinely independent on local stories. The eight Renoir-heist reports, each independently written, correctly stay at 8. Across 91 multi-item clusters: 174 buckets -> 163 true origins. Note this is the one place a lexical comparison is right. Clustering fails on lexical methods because paraphrases share no wording, which is why the SimHash stage was measured and deleted -- but syndicated copy is literally the same bytes. First formulation was wrong and the data caught it: summing syndication groups and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only ever reduce corroboration, which the union-find guarantees and a test asserts. |
||
|
connor
|
d047c58d8a |
Merge States Newsroom affiliates into one independence bucket
Reading real clustered output caught a config error the tests could not: Ohio Capital Journal and Kentucky Lantern are both States Newsroom outlets syndicating the same national desk, and carried 10 byte-identical headlines in a single poll -- more than any other source pair. Configured as separate buckets they would have doubled the corroboration signal on every syndicated story, which is exactly the failure the bucket concept exists to prevent. Found by checking identical headlines across all source pairs rather than spot-fixing the one noticed by eye. That sweep also confirmed the buckets already set deliberately are right: BBC Top/World, Ars Technica/Science and NPR News/Politics each share a bucket and each showed shared headlines. Regression test added. |
||
|
connor
|
863f5e20ac |
Make the system portable: split config from profile, remove hard-coding
Audit found three things compiled in that shouldn't have been: the User-Agent carried the author's domain, the cluster threshold and window were module constants the pipeline never overrode, and the embedding model was pinned -- which blocked the one change most likely to be wanted, swapping in a multilingual model for native-language sources. Configuration is now three files with distinct roles: config.toml (machinery), profile.toml (who the reader is), sources.toml (what to read). All are relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name, feed URL, or personal identifier. tests/test_config.py keeps it that way: it builds a configuration for an invented reader and asserts none of the author's identity or preferences reaches the running code. Writing it immediately found a real bug -- CONFIG_DIR was resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after the module was first imported. Now resolved per instantiation. Documented the coupling that portability creates: [clustering].threshold and [scoring].veto_absolute are cosine constants calibrated against one corpus with one model, so changing [embedding].model invalidates both. |
||
|
connor
|
6dc3694260 |
Profile v2: split conflict from statecraft, add space, expand vetoes
Second elicitation round. The significant correction is Connor's own framing:
"my actual war interest is low, but unfortunately all wars are important." The
click history looked war-heavy only because world-news feeds are war-heavy, so
geopolitics-at-weight-1.0 was reading the corpus, not the reader. Statecraft is
now its own block and armed conflict is five weighted bands (anti-imperial,
US-involved, Mideast/Israel, undercovered, general) routed mainly by importance.
New: space-aerospace at 0.95, the highest-weighted block -- Connor is an
aerospace engineer. Plus science-discovery, environment-ecology,
ohio-state-politics, left-movements, anticolonial-sovereignty, far-right-watch,
political-theory. Linux/ricing/neovim folded into software-systems.
Introduces `demote` blocks, distinct from hard vetoes, for categories adjacent to
real interests: crypto/AI hype sits next to AI regulation, culture-war outrage
next to far-right-watch, true crime next to criminal-justice policy. Hard-vetoing
those would take real stories with them -- the lesson from the Europa League
story that both a substring blocklist and a cosine threshold wrongly killed.
Sources 39 -> 56, all fetched and verified: space (SpaceNews, Spaceflight Now,
NASASpaceflight, NASA, ESA, Phys.org), Linux (Phoronix, LWN, It's FOSS,
r/unixporn), left/anticolonial (Intercept, Drop Site, Tribune, African
Arguments). Rejected on inspection: nature.rss is journal metadata ("Author
Correction: ...") and sciencedaily is a press-release mill.
Embedding stays English-only, measured rather than assumed: the same story in
Spanish and English scored 0.758 and 0.600 cosine against a random-pair p99 of
0.634, so one pair was indistinguishable from an unrelated article. Italy and
Mexico are covered by English-language sources instead.
Also fixes clustering performance: centroids are held in memory across a batch
instead of one SELECT plus a numpy rebuild per item, which had taken ~15 minutes
for 1,258 items.
|
||
|
connor
|
64690ecfe0 |
Phase 2: ingest, embed, cluster
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a cycle; `news sources` and `news clusters` inspect the result. Replay and the live service share run_cycle so tuning and production cannot diverge -- the old system's replay harness hardcoded default params and could never sweep the thresholds it existed to tune. Calibrated, not guessed: - Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76 items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The old system produced 515 clusters of which 514 were singletons. - Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting one wire story is one confirmation rather than ten. Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of separation on titles and -0.2 with summaries -- news duplicates are paraphrases, which is the same lexical trap that sank the old Jaccard clustering. Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every feed URL times out from this host) and Tech Policy Press (serves HTML at every documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss works). Reddit moved to .rss because .json now 403s without OAuth, and its intervals raised because it 429s after three quick requests. Deutsche Welle is back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled quick-xml parser could not. |
||
|
connor
|
c89a723603 |
Seed the rewrite: profile, calibration, and repo scaffolding
Ground-up replacement for news_triage, whose signal quality failed because it had no representation of what its user likes -- three keywords fed to BM25 plus a two-term substring blocklist. profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto blocks, 5 importance blocks, each a set of probe sentences to be embedded. The [scoring] constants are fitted, not guessed. CALIBRATION.md records the measurements against all 3,885 items in the v0.2 database and 40 real feedback labels, including two findings that changed the design: - compare blocks by z-score within block, never raw cosine; raw cosine let geopolitics win 78% of items and buried known-liked items at median rank 1246 of 3885 (293 after normalizing) - veto needs a margin over the interest signal, not a threshold; a plain 0.55 cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent in Europa League", which the old substring blocklist also killed |