# news

A personal news aggregator: many sources in, two to five well-chosen push
notifications and one readable daily digest out.

This is a ground-up rewrite of [`news_triage`](../news_triage), which is still
running. The rewrite exists because the old system had no representation of what
its user actually likes — its whole preference model was three keywords fed to
BM25 plus a two-term substring blocklist, and everything else was ranking
machinery with nothing meaningful to rank.

## The core idea

Taste is a set of **vectors**, not a bag of keywords.

Every interest, veto and importance category in [`profile/profile.toml`](profile/profile.toml)
is written as prose and embedded with a small CPU model (`bge-small-en-v1.5`,
~130 MB, ONNX, no torch, no GPU — so it never contends with the llama-swap stack).
An item's score is the max cosine similarity across a block's probes, never the
mean: averaging "Middle East geopolitics" with "grid-scale energy storage" gives
a vector near neither.

This is what lets a topic be expressed as a *region* rather than a string. The
old system could not represent "sports" as a concept, only as the literal
characters `s-p-o-r-t-s`, which is why 18 of its 32 feedback clicks were its user
hand-rejecting baseball recaps that `blocklist = ["sports"]` never matched.

A generative LLM is used only for prose — the daily digest write-up and cluster
headlines — at roughly 1–30 calls/day through LiteLLM, never for per-item scoring.

## Two routing lanes

Notifications are decided on two independent axes, because "something I might
like" and "something I need to know about" are different questions:

- **interest** — high similarity to the profile's interest vectors.
- **importance** — burst velocity, independent-source-bucket count, editorial
  prominence, and aggregator-native scores. Deliberately *not* raw corroboration
  count, which just measures how many wire-reprinting outlets you subscribed to.

Vetoes are absolute on **both** lanes. In the old system the bypass lane ignored
relevance entirely, so a blocklisted item could still fire at priority 5.

## Layout

    news/        the pipeline: sources, store, embed, cluster, score, route, deliver
    profile/     all configuration (see below)
    tests/
    deploy/

## Configuration

Nothing about the reader is compiled in. `news/*.py` contains no interest, no
place name, no feed URL and no personal identifier; all of it is data in three
TOML files under `profile/` (override the directory with `NEWS_CONFIG_DIR`):

| file | what it is | who edits it |
|---|---|---|
| `profile.toml` | who you are: interests, vetoes, demotes, importance, delivery | you, often |
| `sources.toml` | what to read, with independence buckets and trust weights | you, occasionally |
| `config.toml` | machinery: HTTP, embedding model, clustering and scoring constants | rarely |

To use this as someone else: replace `profile.toml` and `sources.toml`, and change
`user_agent` in `config.toml` to your own contact. That is the whole job.

`tests/test_config.py` exists to keep this true — it builds a configuration for a
different, invented reader and asserts that no part of the author's identity or
preferences reaches the running code.

Two constants deserve a warning. `[clustering].threshold` (0.82) and
`[scoring].veto_absolute` (0.60) were calibrated against a specific 3,885-item
corpus with a specific embedding model. **Changing `[embedding].model` invalidates
both**, because a different model puts cosine similarity on a different scale.
Re-run the calibration in `profile/CALIBRATION.md` before trusting them.

## Commands

    news poll        one fetch -> embed -> cluster cycle
    news sources     sources and item counts
    news clusters    inspect clusters and their corroboration
    news score       score clusters on both axes
    news simulate    what would these thresholds actually have sent?

`simulate` is the tuning surface. The old system's replay harness hardcoded its
parameters and so could never sweep the thresholds it existed to tune.

## Web

    /         triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
    /tune     move the thresholds and budgets, see what would have been sent
    /digest   the day's briefing, mobile-first
    /healthz

Everything except `/api/feedback/*` sits behind the identity gateway. That one
path is exempt because ntfy runs notification buttons from the phone and a
native app cannot complete an interactive login; it is HMAC-signed instead.

## Status

Running in production. Ingest, clustering, two-axis scoring, routing, ntfy
delivery with working feedback buttons, the digest page and the triage UI are
all live.