Detect syndication from text, not just from configured ownership

Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.

Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).

On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.

Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.

First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
This commit is contained in:
connor committed 2026-09-08 15:49:51 -04:00
1 parent d047c58d8a
commit d57b9e092c
7 files changed
+160 -9

No files matched your search

+2 -1
View File
@@ -64,7 +64,8 @@ def test_machinery_values_are_read_from_config(cfgdir):
assert cfg.http["timeout_secs"] == 7
assert cfg.http["concurrency"] == 3
assert cfg.embedding == {"model": "intfloat/multilingual-e5-small", "dim": 512}
assert cfg.clustering == {"threshold": 0.5, "window_hours": 6}
assert cfg.clustering["threshold"] == 0.5
assert cfg.clustering["window_hours"] == 6
assert cfg.scoring["veto_absolute"] == 0.99