Handle paywalled sources; add free New York coverage

The NYT is unreadable without a subscription, but dropping it would throw away
real signal -- knowing the NYT covered a story is corroboration, and it feeds
importance. So sources carry a `paywalled` flag: they still cluster and
corroborate, but are never chosen as the link for a notification or a headline
when a readable report of the same story exists, and are marked in the UI.

Deliberately NOT doing paywall circumvention. The fix is to prefer a source you
can actually read, and to add outlets that do not have the problem.

Five free New York outlets added, all verified live: The City, Gothamist,
New York Focus, Documented NY and amNY -- independent and largely nonprofit,
covering the NYC politics the NYT was being relied on for. Hell Gate was probed
too and has no discoverable feed at any of the usual paths.

Includes an additive migration for the new column, since CREATE TABLE IF NOT
EXISTS will not add a column to a table that already exists.
This commit is contained in:
connor committed 2026-09-09 10:01:43 -04:00
1 parent 98bba62976
commit 9cba8db7bb
8 files changed
+127 -14

No files matched your search

+49
View File
@@ -6,6 +6,11 @@
# confirmations. Getting this wrong is what made the old system fire priority-5
# pushes on anything two feeds happened to carry.
#
# `paywalled = true` marks a source whose articles cannot be read without a
# subscription. Such a source still contributes to clustering and corroboration,
# but is never chosen as the link for a notification or a headline when a
# readable report of the same story exists.
#
# `weight` is editorial trust, carried over from Connor's old config where he
# had deliberately set Al Jazeera highest (1.2) and NYT lowest (0.8). It affects
# importance and tie-breaking, never whether an item is visible at all.
@@ -42,11 +47,16 @@ name = "Guardian World"
url = "https://www.theguardian.com/world/rss"
bucket = "guardian"
# Kept, but marked paywalled. Its articles cannot actually be read without a
# subscription, so it must never be the link you are handed -- but knowing the
# NYT covered a story is still real corroboration, and dropping it would throw
# that signal away. See `paywalled` above.
[[source]]
name = "NYT World"
url = "https://rss.nytimes.com/services/xml/rss/nyt/World.xml"
bucket = "nyt"
weight = 0.8
paywalled = true
[[source]]
name = "NPR News"
@@ -411,3 +421,42 @@ weight = 1.1
url = "https://www.dropsitenews.com/feed"
bucket = "drop-site"
poll_interval_secs = 7200
# ==================================================== New York / local left ====
# Added 2026-09-09. NYC politics matters to Connor and the NYT is unreadable
# without a subscription, so these are free, independent outlets covering the
# same ground. All verified live.
[[source]]
name = "The City"
url = "https://www.thecity.nyc/rss/"
bucket = "the-city-nyc"
weight = 1.1
poll_interval_secs = 3600
[[source]]
name = "Gothamist"
url = "https://gothamist.com/feed"
bucket = "gothamist"
poll_interval_secs = 3600
[[source]]
name = "New York Focus"
url = "https://www.nysfocus.com/feed"
bucket = "ny-focus"
weight = 1.1
poll_interval_secs = 7200
[[source]]
name = "Documented NY"
url = "https://documentedny.com/feed/"
bucket = "documented-ny"
weight = 1.1
poll_interval_secs = 7200
[[source]]
name = "amNY"
url = "https://www.amny.com/feed/"
bucket = "amny"
poll_interval_secs = 7200