The list was ordered by max(interest, importance) and nothing on the page said
so, which makes it impossible to tell whether an odd ranking is a bug or the
intended order.
Five orders, with the active one explained in a line beneath the control:
best highest of the two scores (unchanged default)
interest purely the taste profile, importance ignored entirely
important purely the importance axis -- velocity, corroboration, prominence
new most recently seen, regardless of score
covered most independent newsrooms, syndicated copy collapsed
"My interests" is the one that was actually missing. The two-axis design means a
quiet story that is very much the reader's can sit below a big one that is not,
and until now there was no way to see past that.
An unknown sort falls back to the default rather than erroring.
Cards showed a hard-capped 5 members with no way to see the rest. They now
render every report, with the first three inline and the remainder behind a
native <details> disclosure -- so it works without JavaScript and stays keyboard
accessible. Each entry carries its publication time and outlet, and the summary
names how many reports across how many outlets. `e` expands the selected card.
Loaded inline rather than through a lazy endpoint because the data is small:
measured on this corpus the largest cluster is 23 members and multi-item
clusters hold 249 between them.
Also fixes a bug this introduced. The sources-list comprehension was byte
identical in triage() and digest_page(), so a str.replace() intended for one hit
both, and digest_page started reading a published_at column its query does not
select -- four tests went red. The assert only checked that the anchor EXISTS,
not that it is UNIQUE. Edits now assert the occurrence count. The digest query
gained the column properly, since a timestamp is useful there too.
Three states rather than two: system (follow the OS), light, dark. A two-state
toggle cannot express "just do what my laptop does", which is the state most
people actually want.
Two details that are easy to get wrong and were both handled deliberately:
- The dark rules are guarded with :root:not([data-theme="light"]). Without that
guard, choosing light does nothing on a machine whose OS is set to dark,
because the media query keeps winning -- the toggle appears to work in one
direction only.
- The stored theme is applied by an inline script placed ahead of any rendered
content, so there is no flash of the wrong palette on load. A deferred script
would repaint after first paint, visibly.
localStorage access is wrapped in try/catch: it throws outright in some
private-browsing modes, and a theme preference is not worth a broken page.
The palette moved out of all four templates into _theme.html, which also removes
four near-identical copies of the nav CSS that had already started to drift.
Tests assert the toggle, the guard, the pre-paint ordering, and that no page
redeclares the palette locally.
Connor: too much world news, very little software/aerospace/urbanism/Cincinnati.
Measuring found three separate causes.
1. The digest ranked by max(interest, importance), and importance is largely a
"big world news" detector -- 22 of 28 slots were entered on the importance
axis. A personal reading list ordered by global importance is the wrong
objective. The bulk of the digest is now ranked by INTEREST, with a bounded
allowance (digest_importance_slots = 6) so a major event still makes the
record on a busy day.
2. No per-group cap, so conflict and diplomacy took 18 of 28 slots.
digest_max_per_group = 4, with unused capacity flowing back to the best
remaining items so a quiet day still fills.
3. HN maturation could never happen. Items came only from `search_by_date`,
which returns posts at 2-3 points; once one ages out of the newest window its
score is never refetched, so it can never clear the bar and HN's best content
was structurally invisible. 45 software clusters were sitting in
"aggregator-immature". Added the front page via the `search` endpoint, which
returns the crowd's verdict (measured: median 117 points, max 912) and whose
re-polling is what actually supplies maturation. hn_min_score 150 -> 100,
since a higher bar rejects most of the front page.
Digest went from 8 groups (conflict 10, diplomacy 8, software 0) to 13 groups
with a far flatter distribution, and the software slot is now
"Benchmarking Qwen3.8 27B quantizations" -- the local-inference material that
was asked for.
Process note: the first two attempts at this silently did nothing. I used
str.replace() without asserting the anchor matched, so a stale anchor wrote the
file back unchanged while reporting success, and I measured "no change" twice
before checking the file. Edits here now assert before writing.
news-cycle.timer runs poll then notify every 15 minutes; news-digest.timer runs
the briefing at 08:45 daily. Both enabled and verified: a manual cycle fetched
1047 items, embedded 95 new ones and clustered them in 74s.
Units are installed as COPIES rather than symlinks. The previous service's units
were symlinked into docs/projects/news/deploy/, and when that repo was renamed
to news_triage they became dangling and the timer failed every ten minutes,
unnoticed, until this rebuild found it. Copying costs a re-copy after editing;
symlinking cost days of silent failure.
Deliberate details: the poll ExecStart is `-` prefixed so a failed fetch does not
skip notify; RandomizedDelaySec spreads polling off the quarter hour rather than
hitting 56 sources at once; the digest is Persistent so a machine that was off
at 08:45 still sends rather than skipping the day.
The pages were only reachable by typing URLs, and the digest had no navigation
at all. A single _nav.html partial now appears on all four, marking the current
page, so Triage / Digest / Tune / Config are one click apart.
Parameterised test asserts every page links to the others and marks itself
active, so a page added later cannot ship stranded.
Connor pointed out he may use the site from a laptop away from home. The gated
/ui/feedback/ path was worse than just inconvenient: fetch() follows redirects,
so an expired Authelia session would go 302 -> login page -> HTTP 200, and the
UI would mark the card labelled having recorded nothing. Silent data loss on the
one signal the system learns from.
Both surfaces now post to the same HMAC-signed public endpoint, with
`redirect: 'error'` so a redirect fails loudly instead of looking like success.
The gated route is gone -- one mechanism, not two.
/config shows what every block actually matches right now, and a box to try a
filter before saving it: describe what you want caught, see what it catches and
what sits just below the line, move the threshold, then save it as a veto or a
demote. That boundary view is the point -- a crypto filter at 0.55 catches 131
clusters including "Security updates for Monday", and at 0.62 catches 4, which
you can only know by looking.
Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries
the reasoning behind every constant and a comment-stripping round-trip would
quietly destroy the record of why each number is what it is. Verified: 97
top-level comment lines in, 97 out.
The config mount had to stop being :ro, and ~/config/news2 is now owned
10001:1000 group-writable so the container can save while hand-editing still
works. `:U` would have fixed the container and locked the user out.
/ is the labelling machine: ranked clusters, one keystroke per judgement
(j/k move, i/n label, o open), advancing automatically so a run of judgements is
a run of single keys. Feedback volume is the whole point -- the old system
collected 32 labels because its buttons did not persist and its UI could only be
clicked.
Browser feedback posts to /ui/feedback/, deliberately NOT under /api/feedback/,
which Caddy routes around Authelia for the ntfy buttons. The browser path stays
behind the identity gateway and so needs no signature.
/tune drives the same route() the live service calls, so what it shows is what
would have been sent. The old replay harness hardcoded its parameters and could
never sweep the thresholds it existed to tune.
Building it corrected a claim I had already made twice. I said the budgets bind
before the thresholds, then wrote that the subject quota binds before the
budget. Measured on this corpus, neither is generally true: at the shipped
budgets (3 interest, 2 importance) the BUDGET binds and the quota is inert --
moving it 1 to 3 changes nothing. Raise budgets to 8 and the QUOTA takes over,
1/2/3 giving 10/11/13 alerts. The page now says exactly that, with the numbers,
rather than a rule of thumb that happened to be wrong.
Connor spotted a Missouri redistricting story labelled ohio-state-politics.
Measuring showed that was not an isolated slip: 66% of the winners of a
place-based block named no place at all. Glastonbury, Anthony Albanese and a
Tim Montgomerie story were all scoring as italy-mexico; a singing-apraxia story
was ohio-state-politics.
Embeddings model meaning well and named entities poorly -- "Italian politics"
and "Australian politics" are nearly the same vector. Place is identity, not
meaning, so it is matched lexically, which is the same division of labour that
makes lexical comparison right for syndication detection and wrong for
clustering. Blocks carry `requires_any`; a block cannot win for an item whose
text names none of its terms.
Word boundaries, not substrings: "rome" is inside "chrome" and "mali" inside
"normalise". That is the trap the old blocklist fell into with "nfl" inside
"conflict".
The first implementation was wrong in a way worth recording, because it looked
correct and measured almost unchanged (66% -> 30%). It wrote a sentinel into the
RAW score before normalisation -- but when most items are gated, that sentinel
becomes the block's own mean, so a gated item z-scored to roughly zero and could
still win. Normalisation erased the gate. Applied after normalisation instead:
66% -> 0%, and place-block winners fell from 140 to 10, so 130 items now land in
the block they actually belong to.
Found in production, not in tests. SQLite's default busy timeout is ZERO, so
when a poll held the write lock while storing ~1,500 embeddings, a concurrent
feedback POST failed instantly with "database is locked" rather than waiting.
Polls take minutes and run every fifteen, so this was a wide failure window on
the single endpoint that must never drop a write -- feedback is the only way the
taste model learns anything, and the previous system's entire failure was
collecting 32 labels.
Connections now set busy_timeout=15s. Regression test included, and its first
draft was wrong in an instructive way: the competing thread held the write lock
continuously, which no finite timeout can survive. It now writes in bursts with
gaps, like a real poll cycle, which is the contention actually being tested.
Also cleans up the static-site plumbing left over from the Rust service: the
caddy /srv/news bind mount, the built SPA under data/news/web, and the
news-update systemd units -- which turned out to be DANGLING symlinks into
docs/projects/news/deploy/, the path the old repo used before being renamed to
news_triage, and had been failing every ten minutes unnoticed ever since.
Applied to ~/compose.yml (snapshot: ~/compose.yml.pre-news-v2). The image is
tagged localhost/news-py:latest rather than reusing the registry tag, so the
Rust image stays on disk as the rollback path, and the old database at
./data/news/db is untouched -- rollback is a two-line edit.
Verified from the public internet, not the LAN:
GET / -> 403, Authelia gating browser paths
POST /api/feedback/../bogus -> {"error":"bad signature"} from the APP,
proving the carve-out bypasses Authelia
POST /api/feedback/../<valid HMAC> -> {"ok":true}, and the row persists
That last line is the loop that never fired once in v0.2, and it is now proven
working from off-LAN with a signature check in front of it.
Applied to the live Caddyfile (snapshot: ~/Caddyfile.pre-news-public) and
reloaded; ddns_update published the A record and Let's Encrypt issued a cert.
The block moved out of the "Internal only -- no public DNS record" section into
the Public section rather than just gaining a marker in place. Those section
headers are load-bearing documentation, and a public block sitting under an
internal header would make the file lie about its own security posture.
/api/feedback/* bypasses Authelia because ntfy runs action buttons from the
phone and a native app cannot complete an interactive login -- the exemption
ntfy, Immich and Jellyfin already have on this box. That path is HMAC-signed
per (cluster, kind), so a leaked URL casts one vote on one story.
Verified after reload: / and /healthz return 403 from Authelia while
/api/feedback/... returns 404, and a 404 from the still-running old backend is
what proves Caddy routed it through unauthenticated.
Worth recording, since it is irreversible: publishing the name also publishes it
to Certificate Transparency, which is append-only and world-readable. The
hostname is public forever regardless of any later change.
Two-stage build. The builder bakes the embedding model into the image (~65 MB),
so a cold container needs no outbound access to Hugging Face and starts scoring
in 0.4s -- verified with HF_HUB_OFFLINE=1.
Three things the smoke test caught that would each have failed in production:
- Debian's base image already ships a system account named "news", so useradd
exits 9 on the collision. The runtime user is "app".
- The image runs as uid 10001 rather than root, so a host-owned volume gives
SQLite "attempt to write a readonly database". Compose needs the `U` mount
flag.
- The baked model cache was root-owned, making fastembed log a permission
warning and discard its tree cache on every start.
Deployment targets the EXISTING service rather than a parallel one: same vhost,
and NTFY_TOPIC=news-triage because the old service defaulted to that rather than
setting it, so the phone subscription keeps working.
The Caddy change is the consequential one and is documented as such: the vhost
moves from LAN-only to public, with /api/feedback/* bypassing Authelia because
ntfy runs action buttons from the phone and a native app cannot complete an
interactive login -- the same exemption ntfy, Immich and Jellyfin already have
on this box. That path is HMAC-signed per (cluster, kind).
ntfy push is deliberately short: a title, one line, and a Click through to the
story. The digest is a WEB PAGE, not a push payload -- the old system sent the
whole report as one long markdown message and ntfy turned it into a .md
attachment, which is what made it miserable to read on a phone.
Feedback buttons finally work. The old ones never fired once: NEWS_API_BASE_URL
was unset so they were dropped behind a single boot WARN, and /api/feedback had
no auth anyway. ntfy runs http actions FROM THE PHONE, so the endpoint must be
public -- which is why each button now carries an HMAC signed over
(cluster_id, kind). A leaked URL authorises one vote on one story rather than
acting as a bearer token for the whole model. Missing config drops the buttons
loudly instead of silently.
Digest prose is one LLM call per day through LiteLLM, and that is the entire
generative-model surface of this system -- clustering, relevance, importance and
vetoes are all embeddings and arithmetic. Feed text is passed as JSON data,
never interpolated into prompt prose, so a headline cannot read as an
instruction. Any failure falls back to the plain list; the digest never depends
on a model being reachable. Verified against the live proxy: 20s, themed
paragraphs, no invented facts.
Digest page is mobile-first, dark-mode aware, and renders from notification_log
so it is a record of what was actually sent.
deploy/README.md documents the Caddy vhost and env the buttons require, and uses
a python healthcheck rather than curl -- the v0.2 image shipped no curl, so its
healthcheck failed forever while the service was fine.
62 tests.
Following Connor's point that corpus prevalence reflects how much an outlet
publishes rather than what deserves a notification: allocation should track
stated weights, not corpus composition in EITHER direction. Penalising a topic
for being prevalent is a bug; rewarding it would just amplify whoever publishes
most. This rules out the "reserve slots proportional to corpus share" model that
the previous commit's findings were pointing toward.
Two fixes:
1. Normalisation moves from per-block to per-GROUP. Taking the max across blocks
gives a subject one draw per block, so foreign affairs -- six of 23 blocks --
beat everything else by having more chances. It held 70% of the digest while
software-systems, a quarter of the corpus, held none. Collapsing to the group
before normalising prices that in. Digest conflict share 13 -> 9, space 2 ->
4, and us-national appears at all.
2. The per-group quota now keys on the INTEREST group for both lanes. Importance
blocks describe a kind of importance, not a subject, so two stories on one
subject could each take a slot via different importance blocks -- which is
what happened: both importance alerts were Middle East, one via
war-escalation and one via economic-break.
Interest alerts went from five-of-five Middle East to Ohio redistricting,
political theory and tech policy.
Also demoted prerelease/nightly build entries. The first alert software-systems
ever won was "Nvim development (prerelease) build", which matches the block
correctly and is worthless as a notification.
Documented that switching normalizer requires re-tuning [routing] thresholds:
they are in z-score units, and leaving them at 2.0 under "top-anchored"
silences the interest lane entirely.
Connor noticed 5 of 5 alerts were Middle East. Measuring showed that was not
proportionate: Middle East blocks are ~11% of the eligible corpus while
software-systems alone is 25%, yet Middle East took 35% of the top 20 and
software took none.
Two mechanisms, both measured:
1. Five conflict-* blocks could each claim their own alert slot. Interest blocks
now carry a `group`, and routing enforces max_alerts_per_group_per_day, so
related blocks share one quota. This alone moved the mix from 5/5 Middle East
to three distinct groups.
2. Z-scoring subtracts each block's MEAN, which silently encodes how much of the
corpus that block matches. software-systems has the HIGHEST mean raw
similarity of any block (0.537) and the LOWEST max z (2.51);
conflict-mideast-israel has a lower mean (0.460) and reaches 3.47. A topic is
penalised for being well covered in your own feed list.
Added `normalize = "top-anchored"` as an alternative that rescales each block by
its own upper tail instead of its mean, leaving `weight` to express preference
rather than corpus composition.
Also ruled out a plausible-sounding cause rather than assuming it: HN and
Lobsters items are title-only and much shorter than news articles, so terse
titles seemed likely to match prose probes poorly. Measured the opposite --
title-only HN items score 0.618 against the software block, ABOVE Phoronix/LWN
with summaries (0.610) and Ars Technica (0.540). Short text is not the problem;
prevalence is.
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
Connor's point: "several outlets carried this" covers situations that carry
different amounts of information, and origin_count() flattened them all to one
number. Splitting on same/different newsroom and same/different text gives four
cases:
same newsroom, same text -> pure duplicate (BBC World + BBC top). Zero
signal; the front-page placement it does carry is
already captured by sources.prominence.
same newsroom, diff text -> DEPTH. A newsroom filing four pieces on one story
is a real editorial judgement, previously
discarded entirely.
diff newsroom, same text -> PICKUPS. Syndicated copy; each carrier chose to
run it. Weak but non-zero.
diff newsroom, diff text -> ORIGINS. Independent judgement and independent
newsgathering. Strongest.
score = origins + 0.3*log1p(pickups) + 0.2*log1p(depth), weights in config.
Origins stay dominant; the others break ties among the many two- and
three-origin clusters, and log damping stops a high-volume feed inflating depth.
Measuring first changed the design. Depth outnumbers pickups 78 to 11 across 91
real clusters, because these sources are mostly national outlets writing their
own copy rather than local papers running wire -- so the more valuable half was
the one not originally proposed. UK settlements scores 10.55 from 10 origins
plus 15 same-newsroom pieces; AfD scores 5.55 from 5 origins plus 15.
Tests assert the four-case ordering directly, and that twenty pieces from one
newsroom still rank below two independent ones. Both needed a helper generating
vectors with exact pairwise cosine: the band between the 0.82 clustering
threshold and the 0.97 syndication threshold is narrow, and two earlier drafts
of these tests used ad-hoc vectors that fell outside it and asserted nothing.
Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.
Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).
On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.
Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.
First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
Reading real clustered output caught a config error the tests could not:
Ohio Capital Journal and Kentucky Lantern are both States Newsroom outlets
syndicating the same national desk, and carried 10 byte-identical headlines in a
single poll -- more than any other source pair. Configured as separate buckets
they would have doubled the corroboration signal on every syndicated story,
which is exactly the failure the bucket concept exists to prevent.
Found by checking identical headlines across all source pairs rather than
spot-fixing the one noticed by eye. That sweep also confirmed the buckets
already set deliberately are right: BBC Top/World, Ars Technica/Science and
NPR News/Politics each share a bucket and each showed shared headlines.
Regression test added.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Second elicitation round. The significant correction is Connor's own framing:
"my actual war interest is low, but unfortunately all wars are important." The
click history looked war-heavy only because world-news feeds are war-heavy, so
geopolitics-at-weight-1.0 was reading the corpus, not the reader. Statecraft is
now its own block and armed conflict is five weighted bands (anti-imperial,
US-involved, Mideast/Israel, undercovered, general) routed mainly by importance.
New: space-aerospace at 0.95, the highest-weighted block -- Connor is an
aerospace engineer. Plus science-discovery, environment-ecology,
ohio-state-politics, left-movements, anticolonial-sovereignty, far-right-watch,
political-theory. Linux/ricing/neovim folded into software-systems.
Introduces `demote` blocks, distinct from hard vetoes, for categories adjacent to
real interests: crypto/AI hype sits next to AI regulation, culture-war outrage
next to far-right-watch, true crime next to criminal-justice policy. Hard-vetoing
those would take real stories with them -- the lesson from the Europa League
story that both a substring blocklist and a cosine threshold wrongly killed.
Sources 39 -> 56, all fetched and verified: space (SpaceNews, Spaceflight Now,
NASASpaceflight, NASA, ESA, Phys.org), Linux (Phoronix, LWN, It's FOSS,
r/unixporn), left/anticolonial (Intercept, Drop Site, Tribune, African
Arguments). Rejected on inspection: nature.rss is journal metadata ("Author
Correction: ...") and sciencedaily is a press-release mill.
Embedding stays English-only, measured rather than assumed: the same story in
Spanish and English scored 0.758 and 0.600 cosine against a random-pair p99 of
0.634, so one pair was indistinguishable from an unrelated article. Italy and
Mexico are covered by English-language sources instead.
Also fixes clustering performance: centroids are held in memory across a batch
instead of one SELECT plus a numpy rebuild per item, which had taken ~15 minutes
for 1,258 items.
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a
cycle; `news sources` and `news clusters` inspect the result. Replay and the
live service share run_cycle so tuning and production cannot diverge -- the old
system's replay harness hardcoded default params and could never sweep the
thresholds it existed to tune.
Calibrated, not guessed:
- Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup
capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76
items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The
old system produced 515 clusters of which 514 were singletons.
- Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting
one wire story is one confirmation rather than ten.
Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of
separation on titles and -0.2 with summaries -- news duplicates are paraphrases,
which is the same lexical trap that sank the old Jaccard clustering.
Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every
feed URL times out from this host) and Tech Policy Press (serves HTML at every
documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss
works). Reddit moved to .rss because .json now 403s without OAuth, and its
intervals raised because it 429s after three quick requests. Deutsche Welle is
back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled
quick-xml parser could not.
Ground-up replacement for news_triage, whose signal quality failed because it
had no representation of what its user likes -- three keywords fed to BM25 plus
a two-term substring blocklist.
profile/ holds the taste model as prose and TOML: 11 interest blocks, 3 veto
blocks, 5 importance blocks, each a set of probe sentences to be embedded.
The [scoring] constants are fitted, not guessed. CALIBRATION.md records the
measurements against all 3,885 items in the v0.2 database and 40 real feedback
labels, including two findings that changed the design:
- compare blocks by z-score within block, never raw cosine; raw cosine let
geopolitics win 78% of items and buried known-liked items at median rank 1246
of 3885 (293 after normalizing)
- veto needs a margin over the interest signal, not a threshold; a plain 0.55
cutoff falsely killed "Belgian mayor blocks club from hosting Israeli opponent
in Europa League", which the old substring blocklist also killed
connor