connor
64690ecfe0
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a cycle; `news sources` and `news clusters` inspect the result. Replay and the live service share run_cycle so tuning and production cannot diverge -- the old system's replay harness hardcoded default params and could never sweep the thresholds it existed to tune. Calibrated, not guessed: - Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76 items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The old system produced 515 clusters of which 514 were singletons. - Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting one wire story is one confirmation rather than ten. Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of separation on titles and -0.2 with summaries -- news duplicates are paraphrases, which is the same lexical trap that sank the old Jaccard clustering. Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every feed URL times out from this host) and Tech Policy Press (serves HTML at every documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss works). Reddit moved to .rss because .json now 403s without OAuth, and its intervals raised because it 429s after three quick requests. Deutsche Welle is back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled quick-xml parser could not.
27 lines
980 B
Python
27 lines
980 B
Python
from news.util import canonical_url, content_hash, stable_id
|
|
|
|
|
|
def test_canonical_url_strips_tracking_and_www():
|
|
a = canonical_url("https://www.bbc.co.uk/news/world-123?utm_source=rss&utm_medium=feed")
|
|
b = canonical_url("https://bbc.co.uk/news/world-123/")
|
|
assert a == b == "https://bbc.co.uk/news/world-123"
|
|
|
|
|
|
def test_canonical_url_keeps_meaningful_query():
|
|
url = canonical_url("https://news.ycombinator.com/item?id=4242&utm_campaign=x")
|
|
assert url == "https://news.ycombinator.com/item?id=4242"
|
|
|
|
|
|
def test_canonical_url_drops_fragment():
|
|
assert canonical_url("https://x.com/a#section") == "https://x.com/a"
|
|
|
|
|
|
def test_stable_id_is_deterministic_and_distinct():
|
|
assert stable_id("source", "u") == stable_id("source", "u")
|
|
assert stable_id("source", "u") != stable_id("source", "v")
|
|
|
|
|
|
def test_content_hash_matches_only_identical_text():
|
|
assert content_hash("a b") == content_hash("a b")
|
|
assert content_hash("a b") != content_hash("a b")
|