connorandClaude Opus 5
80533f2824
CLUSTERING. Merging on similarity to the running centroid alone let the centroid walk. Every merge pulls the mean toward what has accumulated, so a cluster that keeps growing converges on its SUBJECT rather than its event -- and a subject centroid is within 0.82 of nearly everything on that subject. Measured on the 110-item "Losing control of AI is the plan" cluster: 10 members within 0.82 of the founding story, 110 within 0.82 of the final centroid. Two new bars. `anchor_threshold` requires similarity to the founding item, not just the mean; it is clamped to `threshold` so lowering that in config is not silently overridden. `discussion_threshold` holds a merge to a near-duplicate bar when BOTH sides are discussion sources -- HN, Reddit, Lobsters, arXiv, GitHub -- which publish independent posts about a subject rather than reports of an event, and were being counted as corroboration for each other. Measured: discussion merges that are the same link cross-posted sit at 0.990 similarity, topical ones at 0.769; 0.90 keeps 89% of the former and rejects 98% of the latter, and the latter were 2,745 of 2,876. Re-clustering the full corpus through the real code: zero clusters of 50+, down from 16 holding 2,269 items; largest is 33, down from 667. The arXiv blob goes 665 -> 1, the AI one 110 -> 15, and a 108-item Houthi blob splits into two distinct events. Known-good events are intact (Space Force 33 -> 30, Turkey LGBTQ+ raids 25 -> 23). THE PLACE GATE was a union across a quota group where it should have been an intersection. `local` holds three blocks with three disjoint gazetteers, so an item had to name a term from all three to survive. "Flash flood relief: Cincinnati City Council advances new stormwater plan" scored 0.697 against the cincinnati probes -- the highest of any block on either axis, z 4.04 against urbanism's 2.10 -- and was zeroed for not saying "Kentucky". That, not the gazetteer, is why the block took 0 of 196 digest slots in a week. Cincinnati now wins 12 clusters and Kentucky 5, both reaching the digest. `local_sources` still earns its place for the second-order cause: only 16% of clusters led by a Cincinnati newsroom name any gazetteer term, and widening the list to the whole metro reached just 31%. Provenance settles what a headline does not say. DIGEST PROSE now receives each story's cleaned summary, not only its headline. With headlines alone an opinion piece arrived as nothing but its own provocation. Measured: 23 of 27 summaries carry 15+ words absent from the headline; 8 of 27 needed a trailer stripped, 3 a byline block, 2 HTML entities. SPACE. Three probes added for satellites, launch policy and agency contracts -- the block covered only civil and scientific spaceflight, so anything about orbital operations had nothing to match and lost to a war block. Deliberately no defence-space probe: that is the one that would win the Space Force story, and whether military space belongs here is Connor's call. The `war-escalation` probe that named "weapons of mass destruction" was matching the word "weapons"; it now describes the act of escalating. That story's importance falls 5.22 -> 4.40. Five space sources added, all verified live: European Spaceflight, Payload, SpacePolicyOnline, The Planetary Society, NASA Science. The old three returned 17 items in a week against 255 from one Cincinnati station, so even a well-scoring space story rarely had anything to score. Also: a profile with no [[veto]] blocks raised UnboundLocalError on the first item scored. ALGORITHM.md records the whole pipeline and why each decision is what it is, including the conclusions this project has reached and since corrected, and the defects that are known and still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
4.9 KiB
news
A personal news aggregator: many sources in, two to five well-chosen push notifications and one readable daily digest out.
This is a ground-up rewrite of news_triage, which is still
running. The rewrite exists because the old system had no representation of what
its user actually likes — its whole preference model was three keywords fed to
BM25 plus a two-term substring blocklist, and everything else was ranking
machinery with nothing meaningful to rank.
Full pipeline reference, and why each decision is what it is: ALGORITHM.md. Read it before changing a scoring constant — several obvious-looking improvements have already been tried, measured, and found worse, and it records which.
The core idea
Taste is a set of vectors, not a bag of keywords.
Every interest, veto and importance category in profile/profile.toml
is written as prose and embedded with a small CPU model (bge-small-en-v1.5,
~130 MB, ONNX, no torch, no GPU — so it never contends with the llama-swap stack).
An item's score is the max cosine similarity across a block's probes, never the
mean: averaging "Middle East geopolitics" with "grid-scale energy storage" gives
a vector near neither.
This is what lets a topic be expressed as a region rather than a string. The
old system could not represent "sports" as a concept, only as the literal
characters s-p-o-r-t-s, which is why 18 of its 32 feedback clicks were its user
hand-rejecting baseball recaps that blocklist = ["sports"] never matched.
A generative LLM is used only for prose — the daily digest write-up and cluster headlines — at roughly 1–30 calls/day through LiteLLM, never for per-item scoring.
Two routing lanes
Notifications are decided on two independent axes, because "something I might like" and "something I need to know about" are different questions:
- interest — high similarity to the profile's interest vectors.
- importance — burst velocity, independent-source-bucket count, editorial prominence, and aggregator-native scores. Deliberately not raw corroboration count, which just measures how many wire-reprinting outlets you subscribed to.
Vetoes are absolute on both lanes. In the old system the bypass lane ignored relevance entirely, so a blocklisted item could still fire at priority 5.
Layout
news/ the pipeline: sources, store, embed, cluster, score, route, deliver
profile/ all configuration (see below)
tests/
deploy/
Configuration
Nothing about the reader is compiled in. news/*.py contains no interest, no
place name, no feed URL and no personal identifier; all of it is data in three
TOML files under profile/ (override the directory with NEWS_CONFIG_DIR):
| file | what it is | who edits it |
|---|---|---|
profile.toml |
who you are: interests, vetoes, demotes, importance, delivery | you, often |
sources.toml |
what to read, with independence buckets and trust weights | you, occasionally |
config.toml |
machinery: HTTP, embedding model, clustering and scoring constants | rarely |
To use this as someone else: replace profile.toml and sources.toml, and change
user_agent in config.toml to your own contact. That is the whole job.
tests/test_config.py exists to keep this true — it builds a configuration for a
different, invented reader and asserts that no part of the author's identity or
preferences reaches the running code.
Two constants deserve a warning. [clustering].threshold (0.82) and
[scoring].veto_absolute (0.60) were calibrated against a specific 3,885-item
corpus with a specific embedding model. Changing [embedding].model invalidates
both, because a different model puts cosine similarity on a different scale.
Re-run the calibration in profile/CALIBRATION.md before trusting them.
Commands
news poll one fetch -> embed -> cluster cycle
news sources sources and item counts
news clusters inspect clusters and their corroboration
news score score clusters on both axes
news simulate what would these thresholds actually have sent?
simulate is the tuning surface. The old system's replay harness hardcoded its
parameters and so could never sweep the thresholds it existed to tune.
Web
/ triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
/tune move the thresholds and budgets, see what would have been sent
/config what each block matches, plus a box to try a new filter and save it
/digest the day's briefing, mobile-first
/healthz
Everything except /api/feedback/* sits behind the identity gateway. That one
path is exempt because ntfy runs notification buttons from the phone and a
native app cannot complete an interactive login; it is HMAC-signed instead.
Status
Running in production. Ingest, clustering, two-axis scoring, routing, ntfy delivery with working feedback buttons, the digest page and the triage UI are all live.