A headline with a curly quote in the Title header made httpx raise
UnicodeEncodeError, which failed news-cycle and dropped every alert after
it in the run. Non-ASCII header values are now sent as UTF-8 encoded-words,
which ntfy decodes.
Adding article summaries to the digest prompt pushed the call past its 180s
timeout, because the model's reasoning grew with the input: 7,830 reasoning
tokens against a 5,722-character briefing.
GLM-5.3 and GLM-5.3-Flash use FORCED thinking and cannot have it disabled --
`thinking = {type = "disabled"}` returns HTTP 500, and the undocumented
`chat_template_kwargs.enable_thinking = false` only suppressed about 73% of it.
The supported control is `reasoning_effort`, taking low/high/max, and it must be
sent NESTED inside `extra_body`: LiteLLM validates top-level parameters against
a per-provider allowlist and rejects it outright with
"UnsupportedParamsError: zai does not support parameters: ['reasoning_effort']".
Measured on a real 28-story payload, with max as a control to prove the
parameter was genuinely routed rather than swallowed:
(none) 195s 7,830 reasoning 5,722 char briefing
low 20s 7 reasoning 3,648 char briefing, 7 paragraphs
high 117s 4,277 reasoning 4,730 char briefing, 8 paragraphs
`low` covers every theme in the digest -- world, US, Ohio, Cincinnati, Kentucky,
Australia -- and keeps the concrete detail the summaries exist to supply.
Separately, the reply is now plain prose. The JSON envelope carried exactly one
field and bought nothing but a failure mode: an unescaped quote inside the value
made json.loads throw and a complete briefing was lost to the templated list.
It was intermittent -- the same prompt at temperature 0 parsed on one run and
not the next -- which is worse than a reliable failure, because nobody goes
looking. `extract_briefing` still tolerates a code fence or a stray envelope,
including a malformed one, but neither is required. The INPUT stays JSON data;
that is what stops a headline reading as an instruction.
Today's digest regenerated: 28 stories, 3,651-character briefing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
Clustering runs at assignment time only, so tightening the rules changed what
happened to new items and left every stored cluster exactly as it was. Sixteen
blobs holding 2,379 items stayed frozen in the database after the fix shipped,
including the 110-item "Losing control of AI is the plan" cluster a triage card
was still reporting as 110 sources.
`news recluster` re-applies the current rules to stored items, and preserves
cluster IDs while doing it: each cluster keeps its row and its founding item,
and only the other members are detached and re-assigned. Rebuilding from
scratch would mint new ids and orphan all 246 rows in notification_log,
digest_items and feedback -- blanking every past digest page, which is the thing
that was just fixed. A test pins that property.
Run against production, against a backup at data/news2/news.db.pre-recluster:
largest cluster 665 -> 39, clusters of 50+ 16 -> 0, multi-item clusters 1,964 ->
2,032, every foreign reference still resolving (196/196, 46/46, 4/4), and no
item in two clusters. A following poll cycle formed no new blob.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
CLUSTERING. Merging on similarity to the running centroid alone let the centroid
walk. Every merge pulls the mean toward what has accumulated, so a cluster that
keeps growing converges on its SUBJECT rather than its event -- and a subject
centroid is within 0.82 of nearly everything on that subject. Measured on the
110-item "Losing control of AI is the plan" cluster: 10 members within 0.82 of
the founding story, 110 within 0.82 of the final centroid.
Two new bars. `anchor_threshold` requires similarity to the founding item, not
just the mean; it is clamped to `threshold` so lowering that in config is not
silently overridden. `discussion_threshold` holds a merge to a near-duplicate
bar when BOTH sides are discussion sources -- HN, Reddit, Lobsters, arXiv,
GitHub -- which publish independent posts about a subject rather than reports of
an event, and were being counted as corroboration for each other. Measured:
discussion merges that are the same link cross-posted sit at 0.990 similarity,
topical ones at 0.769; 0.90 keeps 89% of the former and rejects 98% of the
latter, and the latter were 2,745 of 2,876.
Re-clustering the full corpus through the real code: zero clusters of 50+, down
from 16 holding 2,269 items; largest is 33, down from 667. The arXiv blob goes
665 -> 1, the AI one 110 -> 15, and a 108-item Houthi blob splits into two
distinct events. Known-good events are intact (Space Force 33 -> 30, Turkey
LGBTQ+ raids 25 -> 23).
THE PLACE GATE was a union across a quota group where it should have been an
intersection. `local` holds three blocks with three disjoint gazetteers, so an
item had to name a term from all three to survive. "Flash flood relief:
Cincinnati City Council advances new stormwater plan" scored 0.697 against the
cincinnati probes -- the highest of any block on either axis, z 4.04 against
urbanism's 2.10 -- and was zeroed for not saying "Kentucky". That, not the
gazetteer, is why the block took 0 of 196 digest slots in a week. Cincinnati now
wins 12 clusters and Kentucky 5, both reaching the digest.
`local_sources` still earns its place for the second-order cause: only 16% of
clusters led by a Cincinnati newsroom name any gazetteer term, and widening the
list to the whole metro reached just 31%. Provenance settles what a headline
does not say.
DIGEST PROSE now receives each story's cleaned summary, not only its headline.
With headlines alone an opinion piece arrived as nothing but its own
provocation. Measured: 23 of 27 summaries carry 15+ words absent from the
headline; 8 of 27 needed a trailer stripped, 3 a byline block, 2 HTML entities.
SPACE. Three probes added for satellites, launch policy and agency contracts --
the block covered only civil and scientific spaceflight, so anything about
orbital operations had nothing to match and lost to a war block. Deliberately no
defence-space probe: that is the one that would win the Space Force story, and
whether military space belongs here is Connor's call. The `war-escalation` probe
that named "weapons of mass destruction" was matching the word "weapons"; it now
describes the act of escalating. That story's importance falls 5.22 -> 4.40.
Five space sources added, all verified live: European Spaceflight, Payload,
SpacePolicyOnline, The Planetary Society, NASA Science. The old three returned
17 items in a week against 255 from one Cincinnati station, so even a
well-scoring space story rarely had anything to score.
Also: a profile with no [[veto]] blocks raised UnboundLocalError on the first
item scored.
ALGORITHM.md records the whole pipeline and why each decision is what it is,
including the conclusions this project has reached and since corrected, and the
defects that are known and still open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
A cluster stays eligible as long as fresh reports keep joining it, which
refreshes last_seen_at. That is right -- an ongoing story is today's news -- but
it is not bounded by the window, so the digest drifts into being mostly re-runs.
Measured against the live 4,372-cluster corpus, replaying today's recorded
digest as a prior day: 21 of 28 slots would go to stories already delivered,
leaving 7 new.
So cap them at three rather than excluding them. A digest gives each story about
a sentence, so an update that does not move the headline is barely expressed
here anyway -- but a genuine escalation on a story being followed must not be
silently dropped, and the survivors are the strongest by max(interest,
importance), the same "best match" measure the triage feed's default sort uses.
Same corpus with the cap: 3 repeats, 25 new, digest still full at 28. The freed
slots had good material waiting below the line the whole time.
Applied BEFORE the admission passes, not inside them. The importance allowance
would otherwise smuggle repeats past the cap, which is exactly how an earlier
attempt at the group cap failed, and a test now pins that. Serendipity is
covered too: the candidate list is rebuilt after capping, so a capped repeat
cannot return through a reserved slot.
`day < today` in the lookback so re-running the digest does not read its own
rows as prior appearances and cap the whole thing to three.
NOT DEPLOYED -- image unbuilt, running container still on d78eec0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
Notifications, the digest and the triage queue now share one 24h window. The
app is built on a daily cadence -- the recent news of today -- and 48h meant
the digest could carry a story purely because the window was wider than the
rhythm the app runs on. Measured on a live day: 7 of 16 carried-over digest
stories age out.
The triage dropdown keeps 48h / 7 days / everything for catching up after a
few days away; only the default moves.
[clustering] window_hours stays at 48 deliberately. That governs how far back a
new item looks for a cluster to join, not what reaches the reader, and cutting
it would split a two-day story into two clusters instead of keeping it as one.
Note what this does NOT fix: staleness is measured on the cluster's
last_seen_at, refreshed whenever a new report joins, so a story reported on
again this morning stays inside the window however long it has been running.
That is correct -- an ongoing story is today's news -- but it means 9 of those
16 carry-overs survive, and repeat suppression is still a separate question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
Two separate faults, both reported as "I keep seeing the same stories".
The digest page read notification_log -- the ALERT ledger -- because nothing
ever recorded which stories the digest chose. The digest itself was fine: this
morning's run selected 28 stories and the briefing was written from all of
them, but the page could only find the single story that had been pushed that
day, so it rendered one item beside prose discussing twenty-eight.
Membership cannot be recomputed at read time. route() is stateful -- daily
budgets, per-lane gaps, the already-notified set -- so asking it in the evening
what the 08:45 digest contained returns a different set. So `news digest` now
snapshots its selection into digest_items, and the page reads that back,
unioned with the day's alerts: route() drops an already-notified cluster before
the digest lane sees it, so without the union the day's most notable stories
would be missing from the day's own page.
The triage queue had no recency window at all. Scoring covers every cluster
ever stored on purpose -- the z-score population should describe what news
looks like, not just the last two days -- but the page rendered that entire
archive ranked by score, minus only the handful of labelled items. Nothing aged
out. Measured on the live database: 24 of the first 60 cards were over a day
old and 11 were feed backfill published before the system existed. route() has
had a staleness rule since the beginning; the triage view simply never
consulted it.
triage_window_hours defaults to 48, matching max_age_hours -- nothing older can
be notified, so nothing older needs labelling -- with a dropdown on the page
for 24h / 48h / 7 days / everything.
Also: `news digest --dry-run` wrote the briefing to the database, so previewing
a digest overwrote the real one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015xPLgmcXCc9JjVwmQF63eM
Generated from the supplied 1254px artwork: 16, 32 and 180px PNGs plus a
multi-size .ico, 40K in total against the 571K source.
Two details that matter. The source carries wide transparent margins, so it is
cropped to its content bbox and re-squared before scaling -- scaling it directly
leaves the artwork a smudge in the middle of a 32px frame. And /favicon.ico is
served from the root as well as being linked, because browsers request that path
regardless of what the link tags say.
Original and the regeneration recipe are kept in news/static/README.md; Pillow
is not a runtime dependency, it arrives via fastembed and was used once here,
which is why the outputs are committed.
44rem, roughly 88 characters at this font size -- wider than a classic prose
measure but still bounded, and centred with margin:0 auto rather than pinned to
the left edge of the 54rem container.
Centring also fixes an incidental oddity: the accent rule now sits against the
text block instead of stranded against the page edge.
The four views had 38 / 52 / 54 / 54rem max-widths and three different padding
sets, so content visibly jumped when navigating between them and the digest sat
noticeably narrower than the rest.
One rule now lives in the shared _theme.html partial that every page already
includes, with a tighter mobile padding so the gutters match there too. Triage
keeps only a larger bottom padding, for its fixed help bar.
The digest was narrow deliberately -- prose reads badly much past ~70
characters -- so the briefing keeps its own 40rem measure inside the wider
container. That is the part worth preserving; the container width is not.
Test asserts all four pages resolve to a single container rule, so this cannot
drift again.
Reddit returns 403 or 429 depending on the moment -- measured across four
subreddits in one pass: two 403, two 429, while three of the four had ingested
44-55 items earlier, so the source works intermittently. Only the 429 was
handled softly, so the 403 raised and logged an error every fifteen minutes for
a source that was fine.
Neither status is actionable per cycle. Both now back off quietly and record the
status on the source row, where it is visible via `news sources`.
The NYT is unreadable without a subscription, but dropping it would throw away
real signal -- knowing the NYT covered a story is corroboration, and it feeds
importance. So sources carry a `paywalled` flag: they still cluster and
corroborate, but are never chosen as the link for a notification or a headline
when a readable report of the same story exists, and are marked in the UI.
Deliberately NOT doing paywall circumvention. The fix is to prefer a source you
can actually read, and to add outlets that do not have the problem.
Five free New York outlets added, all verified live: The City, Gothamist,
New York Focus, Documented NY and amNY -- independent and largely nonprofit,
covering the NYC politics the NYT was being relied on for. Hell Gate was probed
too and has no discoverable feed at any of the usual paths.
Includes an additive migration for the new column, since CREATE TABLE IF NOT
EXISTS will not add a column to a table that already exists.
Three bugs from the first day of real use.
1. Quiet hours were evaluated in UTC. The router is handed a UTC `now` and
compared its hour against "22:00-07:00" directly, so notifications were
silenced from 18:00 local -- prime reading time -- and the whole day's budget
was released at 03:00 local. The reported avalanche landed at 03:39 and 03:55.
Quiet hours are now converted to the reader's wall clock first.
2. The timezone never resolved anyway. TZ was "America/Louisville", which exists
on this Arch host but NOT in a Debian container, which ships only the
canonical "America/Kentucky/Louisville". The container fell back to UTC and
`date` printed "America" as the zone abbreviation. The zone is now read from
config rather than the environment, so a bad TZ cannot silently change
behaviour, and the compose entry uses the canonical name. NOTE: ~20 other
services in compose.yml carry the same bad string; they were left alone since
it only affects their log timestamps.
3. Budgets are not a rate. Six notifications fired in a single minute because
the daily allowance could all be spent in one cycle. Each lane now has a
minimum gap (interest 75min, importance 30min) and sends at most one per
cycle, which turns the budget into a rate spread across the day.
Separately, the morning digest had silently fallen back to the templated list.
The real 28-story payload takes 70.3s to generate against a 60s timeout. Raised
to 180s -- and write_prose now LOGS why it failed instead of returning None in
silence, which is what made a one-line timeout cost a debugging session.
The list was ordered by max(interest, importance) and nothing on the page said
so, which makes it impossible to tell whether an odd ranking is a bug or the
intended order.
Five orders, with the active one explained in a line beneath the control:
best highest of the two scores (unchanged default)
interest purely the taste profile, importance ignored entirely
important purely the importance axis -- velocity, corroboration, prominence
new most recently seen, regardless of score
covered most independent newsrooms, syndicated copy collapsed
"My interests" is the one that was actually missing. The two-axis design means a
quiet story that is very much the reader's can sit below a big one that is not,
and until now there was no way to see past that.
An unknown sort falls back to the default rather than erroring.
Cards showed a hard-capped 5 members with no way to see the rest. They now
render every report, with the first three inline and the remainder behind a
native <details> disclosure -- so it works without JavaScript and stays keyboard
accessible. Each entry carries its publication time and outlet, and the summary
names how many reports across how many outlets. `e` expands the selected card.
Loaded inline rather than through a lazy endpoint because the data is small:
measured on this corpus the largest cluster is 23 members and multi-item
clusters hold 249 between them.
Also fixes a bug this introduced. The sources-list comprehension was byte
identical in triage() and digest_page(), so a str.replace() intended for one hit
both, and digest_page started reading a published_at column its query does not
select -- four tests went red. The assert only checked that the anchor EXISTS,
not that it is UNIQUE. Edits now assert the occurrence count. The digest query
gained the column properly, since a timestamp is useful there too.
Three states rather than two: system (follow the OS), light, dark. A two-state
toggle cannot express "just do what my laptop does", which is the state most
people actually want.
Two details that are easy to get wrong and were both handled deliberately:
- The dark rules are guarded with :root:not([data-theme="light"]). Without that
guard, choosing light does nothing on a machine whose OS is set to dark,
because the media query keeps winning -- the toggle appears to work in one
direction only.
- The stored theme is applied by an inline script placed ahead of any rendered
content, so there is no flash of the wrong palette on load. A deferred script
would repaint after first paint, visibly.
localStorage access is wrapped in try/catch: it throws outright in some
private-browsing modes, and a theme preference is not worth a broken page.
The palette moved out of all four templates into _theme.html, which also removes
four near-identical copies of the nav CSS that had already started to drift.
Tests assert the toggle, the guard, the pre-paint ordering, and that no page
redeclares the palette locally.
The pages were only reachable by typing URLs, and the digest had no navigation
at all. A single _nav.html partial now appears on all four, marking the current
page, so Triage / Digest / Tune / Config are one click apart.
Parameterised test asserts every page links to the others and marks itself
active, so a page added later cannot ship stranded.
Connor pointed out he may use the site from a laptop away from home. The gated
/ui/feedback/ path was worse than just inconvenient: fetch() follows redirects,
so an expired Authelia session would go 302 -> login page -> HTTP 200, and the
UI would mark the card labelled having recorded nothing. Silent data loss on the
one signal the system learns from.
Both surfaces now post to the same HMAC-signed public endpoint, with
`redirect: 'error'` so a redirect fails loudly instead of looking like success.
The gated route is gone -- one mechanism, not two.
/config shows what every block actually matches right now, and a box to try a
filter before saving it: describe what you want caught, see what it catches and
what sits just below the line, move the threshold, then save it as a veto or a
demote. That boundary view is the point -- a crypto filter at 0.55 catches 131
clusters including "Security updates for Monday", and at 0.62 catches 4, which
you can only know by looking.
Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries
the reasoning behind every constant and a comment-stripping round-trip would
quietly destroy the record of why each number is what it is. Verified: 97
top-level comment lines in, 97 out.
The config mount had to stop being :ro, and ~/config/news2 is now owned
10001:1000 group-writable so the container can save while hand-editing still
works. `:U` would have fixed the container and locked the user out.
Connor spotted a Missouri redistricting story labelled ohio-state-politics.
Measuring showed that was not an isolated slip: 66% of the winners of a
place-based block named no place at all. Glastonbury, Anthony Albanese and a
Tim Montgomerie story were all scoring as italy-mexico; a singing-apraxia story
was ohio-state-politics.
Embeddings model meaning well and named entities poorly -- "Italian politics"
and "Australian politics" are nearly the same vector. Place is identity, not
meaning, so it is matched lexically, which is the same division of labour that
makes lexical comparison right for syndication detection and wrong for
clustering. Blocks carry `requires_any`; a block cannot win for an item whose
text names none of its terms.
Word boundaries, not substrings: "rome" is inside "chrome" and "mali" inside
"normalise". That is the trap the old blocklist fell into with "nfl" inside
"conflict".
The first implementation was wrong in a way worth recording, because it looked
correct and measured almost unchanged (66% -> 30%). It wrote a sentinel into the
RAW score before normalisation -- but when most items are gated, that sentinel
becomes the block's own mean, so a gated item z-scored to roughly zero and could
still win. Normalisation erased the gate. Applied after normalisation instead:
66% -> 0%, and place-block winners fell from 140 to 10, so 130 items now land in
the block they actually belong to.
Found in production, not in tests. SQLite's default busy timeout is ZERO, so
when a poll held the write lock while storing ~1,500 embeddings, a concurrent
feedback POST failed instantly with "database is locked" rather than waiting.
Polls take minutes and run every fifteen, so this was a wide failure window on
the single endpoint that must never drop a write -- feedback is the only way the
taste model learns anything, and the previous system's entire failure was
collecting 32 labels.
Connections now set busy_timeout=15s. Regression test included, and its first
draft was wrong in an instructive way: the competing thread held the write lock
continuously, which no finite timeout can survive. It now writes in bursts with
gaps, like a real poll cycle, which is the contention actually being tested.
Also cleans up the static-site plumbing left over from the Rust service: the
caddy /srv/news bind mount, the built SPA under data/news/web, and the
news-update systemd units -- which turned out to be DANGLING symlinks into
docs/projects/news/deploy/, the path the old repo used before being renamed to
news_triage, and had been failing every ten minutes unnoticed ever since.
ntfy push is deliberately short: a title, one line, and a Click through to the
story. The digest is a WEB PAGE, not a push payload -- the old system sent the
whole report as one long markdown message and ntfy turned it into a .md
attachment, which is what made it miserable to read on a phone.
Feedback buttons finally work. The old ones never fired once: NEWS_API_BASE_URL
was unset so they were dropped behind a single boot WARN, and /api/feedback had
no auth anyway. ntfy runs http actions FROM THE PHONE, so the endpoint must be
public -- which is why each button now carries an HMAC signed over
(cluster_id, kind). A leaked URL authorises one vote on one story rather than
acting as a bearer token for the whole model. Missing config drops the buttons
loudly instead of silently.
Digest prose is one LLM call per day through LiteLLM, and that is the entire
generative-model surface of this system -- clustering, relevance, importance and
vetoes are all embeddings and arithmetic. Feed text is passed as JSON data,
never interpolated into prompt prose, so a headline cannot read as an
instruction. Any failure falls back to the plain list; the digest never depends
on a model being reachable. Verified against the live proxy: 20s, themed
paragraphs, no invented facts.
Digest page is mobile-first, dark-mode aware, and renders from notification_log
so it is a record of what was actually sent.
deploy/README.md documents the Caddy vhost and env the buttons require, and uses
a python healthcheck rather than curl -- the v0.2 image shipped no curl, so its
healthcheck failed forever while the service was fine.
62 tests.
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
Connor's point: "several outlets carried this" covers situations that carry
different amounts of information, and origin_count() flattened them all to one
number. Splitting on same/different newsroom and same/different text gives four
cases:
same newsroom, same text -> pure duplicate (BBC World + BBC top). Zero
signal; the front-page placement it does carry is
already captured by sources.prominence.
same newsroom, diff text -> DEPTH. A newsroom filing four pieces on one story
is a real editorial judgement, previously
discarded entirely.
diff newsroom, same text -> PICKUPS. Syndicated copy; each carrier chose to
run it. Weak but non-zero.
diff newsroom, diff text -> ORIGINS. Independent judgement and independent
newsgathering. Strongest.
score = origins + 0.3*log1p(pickups) + 0.2*log1p(depth), weights in config.
Origins stay dominant; the others break ties among the many two- and
three-origin clusters, and log damping stops a high-volume feed inflating depth.
Measuring first changed the design. Depth outnumbers pickups 78 to 11 across 91
real clusters, because these sources are mostly national outlets writing their
own copy rather than local papers running wire -- so the more valuable half was
the one not originally proposed. UK settlements scores 10.55 from 10 origins
plus 15 same-newsroom pieces; AfD scores 5.55 from 5 origins plus 15.
Tests assert the four-case ordering directly, and that twenty pieces from one
newsroom still rank below two independent ones. Both needed a helper generating
vectors with exact pairwise cosine: the band between the 0.82 clustering
threshold and the 0.97 syndication threshold is narrow, and two earlier drafts
of these tests used ad-hoc vectors that fell outside it and asserted nothing.
Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.
Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).
On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.
Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.
First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
Reading real clustered output caught a config error the tests could not:
Ohio Capital Journal and Kentucky Lantern are both States Newsroom outlets
syndicating the same national desk, and carried 10 byte-identical headlines in a
single poll -- more than any other source pair. Configured as separate buckets
they would have doubled the corroboration signal on every syndicated story,
which is exactly the failure the bucket concept exists to prevent.
Found by checking identical headlines across all source pairs rather than
spot-fixing the one noticed by eye. That sweep also confirmed the buckets
already set deliberately are right: BBC Top/World, Ars Technica/Science and
NPR News/Politics each share a bucket and each showed shared headlines.
Regression test added.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a
cycle; `news sources` and `news clusters` inspect the result. Replay and the
live service share run_cycle so tuning and production cannot diverge -- the old
system's replay harness hardcoded default params and could never sweep the
thresholds it existed to tune.
Calibrated, not guessed:
- Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup
capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76
items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The
old system produced 515 clusters of which 514 were singletons.
- Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting
one wire story is one confirmation rather than ten.
Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of
separation on titles and -0.2 with summaries -- news duplicates are paraphrases,
which is the same lexical trap that sank the old Jaccard clustering.
Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every
feed URL times out from this host) and Tech Policy Press (serves HTML at every
documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss
works). Reddit moved to .rss because .json now 403s without OAuth, and its
intervals raised because it 429s after three quick requests. Deutsche Welle is
back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled
quick-xml parser could not.
connor
