Detect syndication from text, not just from configured ownership
Connor's observation: two outlets running the same story can be caught by comparing the text, without anyone having to know who owns whom. Corroboration is now origin_count(), a union-find where two items share an origin if EITHER they are in the same bucket, OR their text is near-identical (normalised title match, or cosine >= 0.97). The rules are complementary: buckets are same-owner/different-text (NYT filing five distinct AfD pieces), text matching is different-owner/same-text (a local station reprinting wire). On real data it fires exactly where it should: the vote-by-mail cluster collapses 2 buckets -> 1 origin, catching the States Newsroom syndication automatically -- the case previously fixed by hand in config -- and a Smithsonian story collapses 3 -> 2, catching WCPO reprinting wire, which no static bucket can express since WCPO is genuinely independent on local stories. The eight Renoir-heist reports, each independently written, correctly stay at 8. Across 91 multi-item clusters: 174 buckets -> 163 true origins. Note this is the one place a lexical comparison is right. Clustering fails on lexical methods because paraphrases share no wording, which is why the SimHash stage was measured and deleted -- but syndicated copy is literally the same bytes. First formulation was wrong and the data caught it: summing syndication groups and solo buckets produced 3 origins for a 2-bucket cluster. Syndication may only ever reduce corroboration, which the union-find guarantees and a test asserts.
This commit is contained in:
1 parent
d047c58d8a
commit
d57b9e092c
7 files changed
+160
-9
No files matched your search
@@ -40,6 +40,12 @@ dim = 384
|
||||
threshold = 0.82
|
||||
window_hours = 48
|
||||
|
||||
# Above this cosine, two items from DIFFERENT newsrooms are treated as the same
|
||||
# syndicated copy rather than independent corroboration. Deliberately high:
|
||||
# eight outlets each writing their own Renoir-heist headline stayed separate,
|
||||
# while a local station reprinting a wire dispatch collapsed correctly.
|
||||
syndication_threshold = 0.97
|
||||
|
||||
[scoring]
|
||||
# Compare blocks by z-score WITHIN each block, never by raw cosine: cosine
|
||||
# magnitude is not comparable across blocks, and a plain max let one block win
|
||||
|
||||
Reference in new issue
Block a user