Detect syndication from text, not just from configured ownership

Connor's observation: two outlets running the same story can be caught by
comparing the text, without anyone having to know who owns whom.

Corroboration is now origin_count(), a union-find where two items share an
origin if EITHER they are in the same bucket, OR their text is near-identical
(normalised title match, or cosine >= 0.97). The rules are complementary:
buckets are same-owner/different-text (NYT filing five distinct AfD pieces),
text matching is different-owner/same-text (a local station reprinting wire).

On real data it fires exactly where it should: the vote-by-mail cluster
collapses 2 buckets -> 1 origin, catching the States Newsroom syndication
automatically -- the case previously fixed by hand in config -- and a
Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no
static bucket can express since WCPO is genuinely independent on local stories.
The eight Renoir-heist reports, each independently written, correctly stay at 8.
Across 91 multi-item clusters: 174 buckets -> 163 true origins.

Note this is the one place a lexical comparison is right. Clustering fails on
lexical methods because paraphrases share no wording, which is why the SimHash
stage was measured and deleted -- but syndicated copy is literally the same
bytes.

First formulation was wrong and the data caught it: summing syndication groups
and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only
ever reduce corroboration, which the union-find guarantees and a test asserts.
This commit is contained in:
connor committed 2026-09-08 15:49:51 -04:00
1 parent d047c58d8a
commit d57b9e092c
7 files changed
+160 -9

No files matched your search

+57
View File
@@ -96,3 +96,60 @@ def test_syndicated_partners_in_one_bucket_count_once(con):
assert out[0].cluster_id == out[1].cluster_id
assert cluster.bucket_count(con, out[0].cluster_id) == 1, (
"syndicated partners must not read as independent confirmation")
def _cluster_of(con, items_spec, vec):
"""Put every item in one cluster and return its id."""
rows = [add(con, iid, sid, when, chash) for iid, sid, when, chash in items_spec]
out = cluster.assign(con, rows, {c: vec for *_, c in items_spec})
return out[0].cluster_id
def test_origin_count_collapses_syndicated_copy_across_newsrooms(con):
"""Different owners, identical text: a wire reprint is not corroboration.
This is the case no static bucket can express -- a local station is
independent on local stories and a pure reprinter on national ones.
"""
v = unit(1, 0, 0)
con.execute("INSERT OR REPLACE INTO embeddings (content_hash,dim,vec) VALUES (?,?,?)",
("hw", 3, store.to_blob(v)))
rows = [add(con, "w1", "s1", "2026-01-01T00:00:00+00:00", "hw"),
add(con, "w2", "s2", "2026-01-01T00:20:00+00:00", "hw")]
# identical headline text across two different buckets
con.execute("UPDATE items SET title='Wire dispatch: minister resigns' WHERE id IN ('w1','w2')")
rows = con.execute("SELECT * FROM items WHERE id IN ('w1','w2') ORDER BY published_at").fetchall()
cid = cluster.assign(con, list(rows), {"hw": v})[0].cluster_id
assert cluster.bucket_count(con, cid) == 2
assert cluster.origin_count(con, cid) == 1
def test_origin_count_keeps_independently_written_reports_separate(con):
"""Same event, different newsrooms, different wording: real corroboration."""
# cosine ~0.90: above the 0.82 clustering threshold so they land in one
# cluster, below the 0.97 syndication threshold so they stay distinct
# origins. The first draft of this test used ~0.999 vectors and correctly
# merged them -- the test data was wrong, not the rule.
a, b = unit(1, 0, 0), unit(0.90, 0.436, 0)
for h, v in (("ha", a), ("hb", b)):
con.execute("INSERT OR REPLACE INTO embeddings (content_hash,dim,vec) VALUES (?,?,?)",
(h, 3, store.to_blob(v)))
add(con, "r1", "s1", "2026-01-01T00:00:00+00:00", "ha")
add(con, "r2", "s2", "2026-01-01T00:30:00+00:00", "hb")
con.execute("UPDATE items SET title='Thieves steal two paintings' WHERE id='r1'")
con.execute("UPDATE items SET title='Renoir works taken in museum heist' WHERE id='r2'")
rows = con.execute("SELECT * FROM items WHERE id IN ('r1','r2') ORDER BY published_at").fetchall()
cid = cluster.assign(con, list(rows), {"ha": a, "hb": b})[0].cluster_id
assert cluster.origin_count(con, cid) == 2
def test_origin_count_never_exceeds_bucket_count(con):
"""Syndication detection may only ever reduce corroboration."""
v = unit(1, 0, 0)
con.execute("INSERT OR REPLACE INTO embeddings (content_hash,dim,vec) VALUES (?,?,?)",
("hx", 3, store.to_blob(v)))
specs = [("x1", "s1", "2026-01-01T00:00:00+00:00", "hx"),
("x2", "s2", "2026-01-01T00:10:00+00:00", "hx"),
("x3", "s1", "2026-01-01T00:20:00+00:00", "hx")]
cid = _cluster_of(con, specs, v)
assert cluster.origin_count(con, cid) <= cluster.bucket_count(con, cid)