Generated from the supplied 1254px artwork: 16, 32 and 180px PNGs plus a
multi-size .ico, 40K in total against the 571K source.
Two details that matter. The source carries wide transparent margins, so it is
cropped to its content bbox and re-squared before scaling -- scaling it directly
leaves the artwork a smudge in the middle of a 32px frame. And /favicon.ico is
served from the root as well as being linked, because browsers request that path
regardless of what the link tags say.
Original and the regeneration recipe are kept in news/static/README.md; Pillow
is not a runtime dependency, it arrives via fastembed and was used once here,
which is why the outputs are committed.
44rem, roughly 88 characters at this font size -- wider than a classic prose
measure but still bounded, and centred with margin:0 auto rather than pinned to
the left edge of the 54rem container.
Centring also fixes an incidental oddity: the accent rule now sits against the
text block instead of stranded against the page edge.
The four views had 38 / 52 / 54 / 54rem max-widths and three different padding
sets, so content visibly jumped when navigating between them and the digest sat
noticeably narrower than the rest.
One rule now lives in the shared _theme.html partial that every page already
includes, with a tighter mobile padding so the gutters match there too. Triage
keeps only a larger bottom padding, for its fixed help bar.
The digest was narrow deliberately -- prose reads badly much past ~70
characters -- so the briefing keeps its own 40rem measure inside the wider
container. That is the part worth preserving; the container width is not.
Test asserts all four pages resolve to a single container rule, so this cannot
drift again.
Reddit returns 403 or 429 depending on the moment -- measured across four
subreddits in one pass: two 403, two 429, while three of the four had ingested
44-55 items earlier, so the source works intermittently. Only the 429 was
handled softly, so the 403 raised and logged an error every fifteen minutes for
a source that was fine.
Neither status is actionable per cycle. Both now back off quietly and record the
status on the source row, where it is visible via `news sources`.
The NYT is unreadable without a subscription, but dropping it would throw away
real signal -- knowing the NYT covered a story is corroboration, and it feeds
importance. So sources carry a `paywalled` flag: they still cluster and
corroborate, but are never chosen as the link for a notification or a headline
when a readable report of the same story exists, and are marked in the UI.
Deliberately NOT doing paywall circumvention. The fix is to prefer a source you
can actually read, and to add outlets that do not have the problem.
Five free New York outlets added, all verified live: The City, Gothamist,
New York Focus, Documented NY and amNY -- independent and largely nonprofit,
covering the NYC politics the NYT was being relied on for. Hell Gate was probed
too and has no discoverable feed at any of the usual paths.
Includes an additive migration for the new column, since CREATE TABLE IF NOT
EXISTS will not add a column to a table that already exists.
Three bugs from the first day of real use.
1. Quiet hours were evaluated in UTC. The router is handed a UTC `now` and
compared its hour against "22:00-07:00" directly, so notifications were
silenced from 18:00 local -- prime reading time -- and the whole day's budget
was released at 03:00 local. The reported avalanche landed at 03:39 and 03:55.
Quiet hours are now converted to the reader's wall clock first.
2. The timezone never resolved anyway. TZ was "America/Louisville", which exists
on this Arch host but NOT in a Debian container, which ships only the
canonical "America/Kentucky/Louisville". The container fell back to UTC and
`date` printed "America" as the zone abbreviation. The zone is now read from
config rather than the environment, so a bad TZ cannot silently change
behaviour, and the compose entry uses the canonical name. NOTE: ~20 other
services in compose.yml carry the same bad string; they were left alone since
it only affects their log timestamps.
3. Budgets are not a rate. Six notifications fired in a single minute because
the daily allowance could all be spent in one cycle. Each lane now has a
minimum gap (interest 75min, importance 30min) and sends at most one per
cycle, which turns the budget into a rate spread across the day.
Separately, the morning digest had silently fallen back to the templated list.
The real 28-story payload takes 70.3s to generate against a 60s timeout. Raised
to 180s -- and write_prose now LOGS why it failed instead of returning None in
silence, which is what made a one-line timeout cost a debugging session.
The list was ordered by max(interest, importance) and nothing on the page said
so, which makes it impossible to tell whether an odd ranking is a bug or the
intended order.
Five orders, with the active one explained in a line beneath the control:
best highest of the two scores (unchanged default)
interest purely the taste profile, importance ignored entirely
important purely the importance axis -- velocity, corroboration, prominence
new most recently seen, regardless of score
covered most independent newsrooms, syndicated copy collapsed
"My interests" is the one that was actually missing. The two-axis design means a
quiet story that is very much the reader's can sit below a big one that is not,
and until now there was no way to see past that.
An unknown sort falls back to the default rather than erroring.
Cards showed a hard-capped 5 members with no way to see the rest. They now
render every report, with the first three inline and the remainder behind a
native <details> disclosure -- so it works without JavaScript and stays keyboard
accessible. Each entry carries its publication time and outlet, and the summary
names how many reports across how many outlets. `e` expands the selected card.
Loaded inline rather than through a lazy endpoint because the data is small:
measured on this corpus the largest cluster is 23 members and multi-item
clusters hold 249 between them.
Also fixes a bug this introduced. The sources-list comprehension was byte
identical in triage() and digest_page(), so a str.replace() intended for one hit
both, and digest_page started reading a published_at column its query does not
select -- four tests went red. The assert only checked that the anchor EXISTS,
not that it is UNIQUE. Edits now assert the occurrence count. The digest query
gained the column properly, since a timestamp is useful there too.
Three states rather than two: system (follow the OS), light, dark. A two-state
toggle cannot express "just do what my laptop does", which is the state most
people actually want.
Two details that are easy to get wrong and were both handled deliberately:
- The dark rules are guarded with :root:not([data-theme="light"]). Without that
guard, choosing light does nothing on a machine whose OS is set to dark,
because the media query keeps winning -- the toggle appears to work in one
direction only.
- The stored theme is applied by an inline script placed ahead of any rendered
content, so there is no flash of the wrong palette on load. A deferred script
would repaint after first paint, visibly.
localStorage access is wrapped in try/catch: it throws outright in some
private-browsing modes, and a theme preference is not worth a broken page.
The palette moved out of all four templates into _theme.html, which also removes
four near-identical copies of the nav CSS that had already started to drift.
Tests assert the toggle, the guard, the pre-paint ordering, and that no page
redeclares the palette locally.
The pages were only reachable by typing URLs, and the digest had no navigation
at all. A single _nav.html partial now appears on all four, marking the current
page, so Triage / Digest / Tune / Config are one click apart.
Parameterised test asserts every page links to the others and marks itself
active, so a page added later cannot ship stranded.
Connor pointed out he may use the site from a laptop away from home. The gated
/ui/feedback/ path was worse than just inconvenient: fetch() follows redirects,
so an expired Authelia session would go 302 -> login page -> HTTP 200, and the
UI would mark the card labelled having recorded nothing. Silent data loss on the
one signal the system learns from.
Both surfaces now post to the same HMAC-signed public endpoint, with
`redirect: 'error'` so a redirect fails loudly instead of looking like success.
The gated route is gone -- one mechanism, not two.
/config shows what every block actually matches right now, and a box to try a
filter before saving it: describe what you want caught, see what it catches and
what sits just below the line, move the threshold, then save it as a veto or a
demote. That boundary view is the point -- a crypto filter at 0.55 catches 131
clusters including "Security updates for Monday", and at 0.62 catches 4, which
you can only know by looking.
Saves use tomlkit rather than tomllib plus a dump, because profile.toml carries
the reasoning behind every constant and a comment-stripping round-trip would
quietly destroy the record of why each number is what it is. Verified: 97
top-level comment lines in, 97 out.
The config mount had to stop being :ro, and ~/config/news2 is now owned
10001:1000 group-writable so the container can save while hand-editing still
works. `:U` would have fixed the container and locked the user out.
Connor spotted a Missouri redistricting story labelled ohio-state-politics.
Measuring showed that was not an isolated slip: 66% of the winners of a
place-based block named no place at all. Glastonbury, Anthony Albanese and a
Tim Montgomerie story were all scoring as italy-mexico; a singing-apraxia story
was ohio-state-politics.
Embeddings model meaning well and named entities poorly -- "Italian politics"
and "Australian politics" are nearly the same vector. Place is identity, not
meaning, so it is matched lexically, which is the same division of labour that
makes lexical comparison right for syndication detection and wrong for
clustering. Blocks carry `requires_any`; a block cannot win for an item whose
text names none of its terms.
Word boundaries, not substrings: "rome" is inside "chrome" and "mali" inside
"normalise". That is the trap the old blocklist fell into with "nfl" inside
"conflict".
The first implementation was wrong in a way worth recording, because it looked
correct and measured almost unchanged (66% -> 30%). It wrote a sentinel into the
RAW score before normalisation -- but when most items are gated, that sentinel
becomes the block's own mean, so a gated item z-scored to roughly zero and could
still win. Normalisation erased the gate. Applied after normalisation instead:
66% -> 0%, and place-block winners fell from 140 to 10, so 130 items now land in
the block they actually belong to.
Found in production, not in tests. SQLite's default busy timeout is ZERO, so
when a poll held the write lock while storing ~1,500 embeddings, a concurrent
feedback POST failed instantly with "database is locked" rather than waiting.
Polls take minutes and run every fifteen, so this was a wide failure window on
the single endpoint that must never drop a write -- feedback is the only way the
taste model learns anything, and the previous system's entire failure was
collecting 32 labels.
Connections now set busy_timeout=15s. Regression test included, and its first
draft was wrong in an instructive way: the competing thread held the write lock
continuously, which no finite timeout can survive. It now writes in bursts with
gaps, like a real poll cycle, which is the contention actually being tested.
Also cleans up the static-site plumbing left over from the Rust service: the
caddy /srv/news bind mount, the built SPA under data/news/web, and the
news-update systemd units -- which turned out to be DANGLING symlinks into
docs/projects/news/deploy/, the path the old repo used before being renamed to
news_triage, and had been failing every ten minutes unnoticed ever since.
ntfy push is deliberately short: a title, one line, and a Click through to the
story. The digest is a WEB PAGE, not a push payload -- the old system sent the
whole report as one long markdown message and ntfy turned it into a .md
attachment, which is what made it miserable to read on a phone.
Feedback buttons finally work. The old ones never fired once: NEWS_API_BASE_URL
was unset so they were dropped behind a single boot WARN, and /api/feedback had
no auth anyway. ntfy runs http actions FROM THE PHONE, so the endpoint must be
public -- which is why each button now carries an HMAC signed over
(cluster_id, kind). A leaked URL authorises one vote on one story rather than
acting as a bearer token for the whole model. Missing config drops the buttons
loudly instead of silently.
Digest prose is one LLM call per day through LiteLLM, and that is the entire
generative-model surface of this system -- clustering, relevance, importance and
vetoes are all embeddings and arithmetic. Feed text is passed as JSON data,
never interpolated into prompt prose, so a headline cannot read as an
instruction. Any failure falls back to the plain list; the digest never depends
on a model being reachable. Verified against the live proxy: 20s, themed
paragraphs, no invented facts.
Digest page is mobile-first, dark-mode aware, and renders from notification_log
so it is a record of what was actually sent.
deploy/README.md documents the Caddy vhost and env the buttons require, and uses
a python healthcheck rather than curl -- the v0.2 image shipped no curl, so its
healthcheck failed forever while the service was fine.
62 tests.
Scoring (news/score.py): every profile block is embedded and compared by
z-score WITHIN block, then weighted. Veto fires on absolute similarity OR a
margin over the interest signal; demotes apply a penalty instead of excluding,
because those categories sit adjacent to real interests.
Importance is deliberately NOT corroboration count -- the old system's error,
which measured the subscription list rather than the news. It combines block
similarity, corroboration, burst velocity, editorial prominence and aggregator
score, all weighted from config.
Routing (news/route.py): two lanes with separate budgets. Interest never
overrides quiet hours; importance may, within a hard nightly cap. Vetoes are
absolute on BOTH lanes -- the v0.2 bypass lane never consulted relevance, so a
blocklisted item could still fire at priority 5, and a test now asserts it
cannot. Budgets are spent strongest-first rather than first-come.
Measured on 1,318 real clusters: 2.5 notifications/day and a 14-item digest,
inside the 2-5/day target.
Three defects the real data exposed, none of which unit tests would have caught:
- velocity was computed as origins/hours with a 0.5h floor, giving every
SINGLE-source cluster a spurious velocity of 2.0. Growth rate is undefined
for a cluster that has not grown; it is now zero below two origins.
- deferred items (quiet hours, budget exhausted, immature) settled straight
into the digest and bypassed its cap, producing 198 items/day. Everything
that misses an alert lane now competes for one ranked, capped digest.
- feeds return large archives on first fetch -- 88 Strong Towns items going back
five months, and a BBC entry titled "BBC News app" dated 2025. Those belong in
the scoring population but must never notify, so routing now has a recency
requirement.
Reading the output also found two profile gaps: recurring columns and link
round-ups ("Catching Our Eye News Roundup", "Astronomy Picture of the Day")
scored as top-ranked interest items. They are on-topic but publish on a
schedule, so they are demoted rather than vetoed and still reach the digest.
Connor's point: "several outlets carried this" covers situations that carry
different amounts of information, and origin_count() flattened them all to one
number. Splitting on same/different newsroom and same/different text gives four
cases:
same newsroom, same text -> pure duplicate (BBC World + BBC top). Zero
signal; the front-page placement it does carry is
already captured by sources.prominence.
same newsroom, diff text -> DEPTH. A newsroom filing four pieces on one story
is a real editorial judgement, previously
discarded entirely.
diff newsroom, same text -> PICKUPS. Syndicated copy; each carrier chose to
run it. Weak but non-zero.
diff newsroom, diff text -> ORIGINS. Independent judgement and independent
newsgathering. Strongest.
score = origins + 0.3*log1p(pickups) + 0.2*log1p(depth), weights in config.
Origins stay dominant; the others break ties among the many two- and
three-origin clusters, and log damping stops a high-volume feed inflating depth.
Measuring first changed the design. Depth outnumbers pickups 78 to 11 across 91
real clusters, because these sources are mostly national outlets writing their
own copy rather than local papers running wire -- so the more valuable half was
the one not originally proposed. UK settlements scores 10.55 from 10 origins
plus 15 same-newsroom pieces; AfD scores 5.55 from 5 origins plus 15.
Tests assert the four-case ordering directly, and that twenty pieces from one
newsroom still rank below two independent ones. Both needed a helper generating
vectors with exact pairwise cosine: the band between the 0.82 clustering
threshold and the 0.97 syndication threshold is narrow, and two earlier drafts
of these tests used ad-hoc vectors that fell outside it and asserted nothing.
Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.
Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).
On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.
Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.
First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
Reading real clustered output caught a config error the tests could not:
Ohio Capital Journal and Kentucky Lantern are both States Newsroom outlets
syndicating the same national desk, and carried 10 byte-identical headlines in a
single poll -- more than any other source pair. Configured as separate buckets
they would have doubled the corroboration signal on every syndicated story,
which is exactly the failure the bucket concept exists to prevent.
Found by checking identical headlines across all source pairs rather than
spot-fixing the one noticed by eye. That sweep also confirmed the buckets
already set deliberately are right: BBC Top/World, Ars Technica/Science and
NPR News/Politics each share a bucket and each showed shared headlines.
Regression test added.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
Pipeline: fetch -> normalise -> embed (CPU) -> cluster. `news poll` runs a
cycle; `news sources` and `news clusters` inspect the result. Replay and the
live service share run_cycle so tuning and production cannot diverge -- the old
system's replay harness hardcoded default params and could never sweep the
thresholds it existed to tune.
Calibrated, not guessed:
- Cluster threshold 0.82, from reading real output. 0.70 merged "Solheim Cup
capsules" with "FIBA Women's World Cup Glance"; 0.82 gives clean groups (76
items on the AfD Saxony-Anhalt election, 27 on the Nepal tunnel rescue). The
old system produced 515 clusters of which 514 were singletons.
- Corroboration counts DISTINCT buckets, not feeds, so ten outlets reprinting
one wire story is one confirmation rather than ten.
Removed after measurement: a SimHash near-duplicate stage. It gave +1.2 bits of
separation on titles and -0.2 with summaries -- news duplicates are paraphrases,
which is the same lexical trap that sank the old Jaccard clustering.
Sources: all 39 fetched against the live web, not assumed. Dropped CBC (every
feed URL times out from this host) and Tech Policy Press (serves HTML at every
documented feed path). Fixed WVXU (index.rss is an empty stub; local-news.rss
works). Reddit moved to .rss because .json now 403s without OAuth, and its
intervals raised because it 429s after three quick requests. Deutsche Welle is
back: it ships RSS 1.0/RDF, which feedparser reads and the old hand-rolled
quick-xml parser could not.
connor