connor
5f4bc8d890
Connor pointed out he may use the site from a laptop away from home. The gated /ui/feedback/ path was worse than just inconvenient: fetch() follows redirects, so an expired Authelia session would go 302 -> login page -> HTTP 200, and the UI would mark the card labelled having recorded nothing. Silent data loss on the one signal the system learns from. Both surfaces now post to the same HMAC-signed public endpoint, with `redirect: 'error'` so a redirect fails loudly instead of looking like success. The gated route is gone -- one mechanism, not two. /config shows what every block actually matches right now, and a box to try a filter before saving it: describe what you want caught, see what it catches and what sits just below the line, move the threshold, then save it as a veto or a demote. That boundary view is the point -- a crypto filter at 0.55 catches 131 clusters including "Security updates for Monday", and at 0.62 catches 4, which you can only know by looking. Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries the reasoning behind every constant and a comment-stripping round-trip would quietly destroy the record of why each number is what it is. Verified: 97 top-level comment lines in, 97 out. The config mount had to stop being :ro, and ~/config/news2 is now owned 10001:1000 group-writable so the container can save while hand-editing still works. `:U` would have fixed the container and locked the user out.
104 lines
4.7 KiB
Markdown
104 lines
4.7 KiB
Markdown
# news
|
||
|
||
A personal news aggregator: many sources in, two to five well-chosen push
|
||
notifications and one readable daily digest out.
|
||
|
||
This is a ground-up rewrite of [`news_triage`](../news_triage), which is still
|
||
running. The rewrite exists because the old system had no representation of what
|
||
its user actually likes — its whole preference model was three keywords fed to
|
||
BM25 plus a two-term substring blocklist, and everything else was ranking
|
||
machinery with nothing meaningful to rank.
|
||
|
||
## The core idea
|
||
|
||
Taste is a set of **vectors**, not a bag of keywords.
|
||
|
||
Every interest, veto and importance category in [`profile/profile.toml`](profile/profile.toml)
|
||
is written as prose and embedded with a small CPU model (`bge-small-en-v1.5`,
|
||
~130 MB, ONNX, no torch, no GPU — so it never contends with the llama-swap stack).
|
||
An item's score is the max cosine similarity across a block's probes, never the
|
||
mean: averaging "Middle East geopolitics" with "grid-scale energy storage" gives
|
||
a vector near neither.
|
||
|
||
This is what lets a topic be expressed as a *region* rather than a string. The
|
||
old system could not represent "sports" as a concept, only as the literal
|
||
characters `s-p-o-r-t-s`, which is why 18 of its 32 feedback clicks were its user
|
||
hand-rejecting baseball recaps that `blocklist = ["sports"]` never matched.
|
||
|
||
A generative LLM is used only for prose — the daily digest write-up and cluster
|
||
headlines — at roughly 1–30 calls/day through LiteLLM, never for per-item scoring.
|
||
|
||
## Two routing lanes
|
||
|
||
Notifications are decided on two independent axes, because "something I might
|
||
like" and "something I need to know about" are different questions:
|
||
|
||
- **interest** — high similarity to the profile's interest vectors.
|
||
- **importance** — burst velocity, independent-source-bucket count, editorial
|
||
prominence, and aggregator-native scores. Deliberately *not* raw corroboration
|
||
count, which just measures how many wire-reprinting outlets you subscribed to.
|
||
|
||
Vetoes are absolute on **both** lanes. In the old system the bypass lane ignored
|
||
relevance entirely, so a blocklisted item could still fire at priority 5.
|
||
|
||
## Layout
|
||
|
||
news/ the pipeline: sources, store, embed, cluster, score, route, deliver
|
||
profile/ all configuration (see below)
|
||
tests/
|
||
deploy/
|
||
|
||
## Configuration
|
||
|
||
Nothing about the reader is compiled in. `news/*.py` contains no interest, no
|
||
place name, no feed URL and no personal identifier; all of it is data in three
|
||
TOML files under `profile/` (override the directory with `NEWS_CONFIG_DIR`):
|
||
|
||
| file | what it is | who edits it |
|
||
|---|---|---|
|
||
| `profile.toml` | who you are: interests, vetoes, demotes, importance, delivery | you, often |
|
||
| `sources.toml` | what to read, with independence buckets and trust weights | you, occasionally |
|
||
| `config.toml` | machinery: HTTP, embedding model, clustering and scoring constants | rarely |
|
||
|
||
To use this as someone else: replace `profile.toml` and `sources.toml`, and change
|
||
`user_agent` in `config.toml` to your own contact. That is the whole job.
|
||
|
||
`tests/test_config.py` exists to keep this true — it builds a configuration for a
|
||
different, invented reader and asserts that no part of the author's identity or
|
||
preferences reaches the running code.
|
||
|
||
Two constants deserve a warning. `[clustering].threshold` (0.82) and
|
||
`[scoring].veto_absolute` (0.60) were calibrated against a specific 3,885-item
|
||
corpus with a specific embedding model. **Changing `[embedding].model` invalidates
|
||
both**, because a different model puts cosine similarity on a different scale.
|
||
Re-run the calibration in `profile/CALIBRATION.md` before trusting them.
|
||
|
||
## Commands
|
||
|
||
news poll one fetch -> embed -> cluster cycle
|
||
news sources sources and item counts
|
||
news clusters inspect clusters and their corroboration
|
||
news score score clusters on both axes
|
||
news simulate what would these thresholds actually have sent?
|
||
|
||
`simulate` is the tuning surface. The old system's replay harness hardcoded its
|
||
parameters and so could never sweep the thresholds it existed to tune.
|
||
|
||
## Web
|
||
|
||
/ triage feed -- ranked, keyboard-driven (j/k move, i/n label, o open)
|
||
/tune move the thresholds and budgets, see what would have been sent
|
||
/config what each block matches, plus a box to try a new filter and save it
|
||
/digest the day's briefing, mobile-first
|
||
/healthz
|
||
|
||
Everything except `/api/feedback/*` sits behind the identity gateway. That one
|
||
path is exempt because ntfy runs notification buttons from the phone and a
|
||
native app cannot complete an interactive login; it is HMAC-signed instead.
|
||
|
||
## Status
|
||
|
||
Running in production. Ingest, clustering, two-axis scoring, routing, ntfy
|
||
delivery with working feedback buttons, the digest page and the triage UI are
|
||
all live.
|