connor
d57b9e092c
Connor's observation: two outlets running the same story can be caught by comparing the text, without anyone having to know who owns whom. Corroboration is now origin_count(), a union-find where two items share an origin if EITHER they are in the same bucket, OR their text is near-identical (normalised title match, or cosine >= 0.97). The rules are complementary: buckets are same-owner/different-text (NYT filing five distinct AfD pieces), text matching is different-owner/same-text (a local station reprinting wire). On real data it fires exactly where it should: the vote-by-mail cluster collapses 2 buckets -> 1 origin, catching the States Newsroom syndication automatically -- the case previously fixed by hand in config -- and a Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no static bucket can express since WCPO is genuinely independent on local stories. The eight Renoir-heist reports, each independently written, correctly stay at 8. Across 91 multi-item clusters: 174 buckets -> 163 true origins. Note this is the one place a lexical comparison is right. Clustering fails on lexical methods because paraphrases share no wording, which is why the SimHash stage was measured and deleted -- but syndicated copy is literally the same bytes. First formulation was wrong and the data caught it: summing syndication groups and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only ever reduce corroboration, which the union-find guarantees and a test asserts.
news
A personal news aggregator: many sources in, two to five well-chosen push notifications and one readable daily digest out.
This is a ground-up rewrite of news_triage, which is still
running. The rewrite exists because the old system had no representation of what
its user actually likes — its whole preference model was three keywords fed to
BM25 plus a two-term substring blocklist, and everything else was ranking
machinery with nothing meaningful to rank.
The core idea
Taste is a set of vectors, not a bag of keywords.
Every interest, veto and importance category in profile/profile.toml
is written as prose and embedded with a small CPU model (bge-small-en-v1.5,
~130 MB, ONNX, no torch, no GPU — so it never contends with the llama-swap stack).
An item's score is the max cosine similarity across a block's probes, never the
mean: averaging "Middle East geopolitics" with "grid-scale energy storage" gives
a vector near neither.
This is what lets a topic be expressed as a region rather than a string. The
old system could not represent "sports" as a concept, only as the literal
characters s-p-o-r-t-s, which is why 18 of its 32 feedback clicks were its user
hand-rejecting baseball recaps that blocklist = ["sports"] never matched.
A generative LLM is used only for prose — the daily digest write-up and cluster headlines — at roughly 1–30 calls/day through LiteLLM, never for per-item scoring.
Two routing lanes
Notifications are decided on two independent axes, because "something I might like" and "something I need to know about" are different questions:
- interest — high similarity to the profile's interest vectors.
- importance — burst velocity, independent-source-bucket count, editorial prominence, and aggregator-native scores. Deliberately not raw corroboration count, which just measures how many wire-reprinting outlets you subscribed to.
Vetoes are absolute on both lanes. In the old system the bypass lane ignored relevance entirely, so a blocklisted item could still fire at priority 5.
Layout
news/ the pipeline: sources, store, embed, cluster, score, route, deliver
profile/ all configuration (see below)
tests/
deploy/
Configuration
Nothing about the reader is compiled in. news/*.py contains no interest, no
place name, no feed URL and no personal identifier; all of it is data in three
TOML files under profile/ (override the directory with NEWS_CONFIG_DIR):
| file | what it is | who edits it |
|---|---|---|
profile.toml |
who you are: interests, vetoes, demotes, importance, delivery | you, often |
sources.toml |
what to read, with independence buckets and trust weights | you, occasionally |
config.toml |
machinery: HTTP, embedding model, clustering and scoring constants | rarely |
To use this as someone else: replace profile.toml and sources.toml, and change
user_agent in config.toml to your own contact. That is the whole job.
tests/test_config.py exists to keep this true — it builds a configuration for a
different, invented reader and asserts that no part of the author's identity or
preferences reaches the running code.
Two constants deserve a warning. [clustering].threshold (0.82) and
[scoring].veto_absolute (0.60) were calibrated against a specific 3,885-item
corpus with a specific embedding model. Changing [embedding].model invalidates
both, because a different model puts cosine similarity on a different scale.
Re-run the calibration in profile/CALIBRATION.md before trusting them.
Status
Early. profile/ is written; the pipeline is not yet built.