connor
23966a2d63
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
92 lines
4.2 KiB
Markdown
92 lines
4.2 KiB
Markdown
# news
|
||
|
||
A personal news aggregator: many sources in, two to five well-chosen push
|
||
notifications and one readable daily digest out.
|
||
|
||
This is a ground-up rewrite of [`news_triage`](../news_triage), which is still
|
||
running. The rewrite exists because the old system had no representation of what
|
||
its user actually likes — its whole preference model was three keywords fed to
|
||
BM25 plus a two-term substring blocklist, and everything else was ranking
|
||
machinery with nothing meaningful to rank.
|
||
|
||
## The core idea
|
||
|
||
Taste is a set of **vectors**, not a bag of keywords.
|
||
|
||
Every interest, veto and importance category in [`profile/profile.toml`](profile/profile.toml)
|
||
is written as prose and embedded with a small CPU model (`bge-small-en-v1.5`,
|
||
~130 MB, ONNX, no torch, no GPU — so it never contends with the llama-swap stack).
|
||
An item's score is the max cosine similarity across a block's probes, never the
|
||
mean: averaging "Middle East geopolitics" with "grid-scale energy storage" gives
|
||
a vector near neither.
|
||
|
||
This is what lets a topic be expressed as a *region* rather than a string. The
|
||
old system could not represent "sports" as a concept, only as the literal
|
||
characters `s-p-o-r-t-s`, which is why 18 of its 32 feedback clicks were its user
|
||
hand-rejecting baseball recaps that `blocklist = ["sports"]` never matched.
|
||
|
||
A generative LLM is used only for prose — the daily digest write-up and cluster
|
||
headlines — at roughly 1–30 calls/day through LiteLLM, never for per-item scoring.
|
||
|
||
## Two routing lanes
|
||
|
||
Notifications are decided on two independent axes, because "something I might
|
||
like" and "something I need to know about" are different questions:
|
||
|
||
- **interest** — high similarity to the profile's interest vectors.
|
||
- **importance** — burst velocity, independent-source-bucket count, editorial
|
||
prominence, and aggregator-native scores. Deliberately *not* raw corroboration
|
||
count, which just measures how many wire-reprinting outlets you subscribed to.
|
||
|
||
Vetoes are absolute on **both** lanes. In the old system the bypass lane ignored
|
||
relevance entirely, so a blocklisted item could still fire at priority 5.
|
||
|
||
## Layout
|
||
|
||
news/ the pipeline: sources, store, embed, cluster, score, route, deliver
|
||
profile/ all configuration (see below)
|
||
tests/
|
||
deploy/
|
||
|
||
## Configuration
|
||
|
||
Nothing about the reader is compiled in. `news/*.py` contains no interest, no
|
||
place name, no feed URL and no personal identifier; all of it is data in three
|
||
TOML files under `profile/` (override the directory with `NEWS_CONFIG_DIR`):
|
||
|
||
| file | what it is | who edits it |
|
||
|---|---|---|
|
||
| `profile.toml` | who you are: interests, vetoes, demotes, importance, delivery | you, often |
|
||
| `sources.toml` | what to read, with independence buckets and trust weights | you, occasionally |
|
||
| `config.toml` | machinery: HTTP, embedding model, clustering and scoring constants | rarely |
|
||
|
||
To use this as someone else: replace `profile.toml` and `sources.toml`, and change
|
||
`user_agent` in `config.toml` to your own contact. That is the whole job.
|
||
|
||
`tests/test_config.py` exists to keep this true — it builds a configuration for a
|
||
different, invented reader and asserts that no part of the author's identity or
|
||
preferences reaches the running code.
|
||
|
||
Two constants deserve a warning. `[clustering].threshold` (0.82) and
|
||
`[scoring].veto_absolute` (0.60) were calibrated against a specific 3,885-item
|
||
corpus with a specific embedding model. **Changing `[embedding].model` invalidates
|
||
both**, because a different model puts cosine similarity on a different scale.
|
||
Re-run the calibration in `profile/CALIBRATION.md` before trusting them.
|
||
|
||
## Commands
|
||
|
||
news poll one fetch -> embed -> cluster cycle
|
||
news sources sources and item counts
|
||
news clusters inspect clusters and their corroboration
|
||
news score score clusters on both axes
|
||
news simulate what would these thresholds actually have sent?
|
||
|
||
`simulate` is the tuning surface. The old system's replay harness hardcoded its
|
||
parameters and so could never sweep the thresholds it existed to tune.
|
||
|
||
## Status
|
||
|
||
Ingest, embedding, clustering, two-axis scoring and routing are built and
|
||
running against live feeds. Delivery (ntfy, the digest page) and the web UI are
|
||
not yet written.
|