Found in production, not in tests. SQLite's default busy timeout is ZERO, so
when a poll held the write lock while storing ~1,500 embeddings, a concurrent
feedback POST failed instantly with "database is locked" rather than waiting.
Polls take minutes and run every fifteen, so this was a wide failure window on
the single endpoint that must never drop a write -- feedback is the only way the
taste model learns anything, and the previous system's entire failure was
collecting 32 labels.
Connections now set busy_timeout=15s. Regression test included, and its first
draft was wrong in an instructive way: the competing thread held the write lock
continuously, which no finite timeout can survive. It now writes in bursts with
gaps, like a real poll cycle, which is the contention actually being tested.
Also cleans up the static-site plumbing left over from the Rust service: the
caddy /srv/news bind mount, the built SPA under data/news/web, and the
news-update systemd units -- which turned out to be DANGLING symlinks into
docs/projects/news/deploy/, the path the old repo used before being renamed to
news_triage, and had been failing every ten minutes unnoticed ever since.
Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.
Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).
On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.
Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.
First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
Audit found three things compiled in that shouldn't have been: the User-Agent
carried the author's domain, the cluster threshold and window were module
constants the pipeline never overrode, and the embedding model was pinned --
which blocked the one change most likely to be wanted, swapping in a
multilingual model for native-language sources.
Configuration is now three files with distinct roles: config.toml (machinery),
profile.toml (who the reader is), sources.toml (what to read). All are
relocatable via NEWS_CONFIG_DIR. news/*.py now contains no interest, place name,
feed URL, or personal identifier.
tests/test_config.py keeps it that way: it builds a configuration for an
invented reader and asserts none of the author's identity or preferences reaches
the running code. Writing it immediately found a real bug -- CONFIG_DIR was
resolved at import time, so NEWS_CONFIG_DIR silently did nothing if set after
the module was first imported. Now resolved per instantiation.
Documented the coupling that portability creates: [clustering].threshold and
[scoring].veto_absolute are cosine constants calibrated against one corpus with
one model, so changing [embedding].model invalidates both.
connor