Treat a 403 as blocked rather than an error

Reddit returns 403 or 429 depending on the moment -- measured across four
subreddits in one pass: two 403, two 429, while three of the four had ingested
44-55 items earlier, so the source works intermittently. Only the 429 was
handled softly, so the 403 raised and logged an error every fifteen minutes for
a source that was fine.

Neither status is actionable per cycle. Both now back off quietly and record the
status on the source row, where it is visible via `news sources`.
This commit is contained in:
connor committed 2026-09-09 10:04:38 -04:00
1 parent 9cba8db7bb
commit c03df8c1b5
3 files changed
+15 -2

No files matched your search

+6 -1
View File
@@ -64,7 +64,12 @@ async def fetch(client, source) -> tuple[list[dict], dict]:
# Rate limiting is an expected steady state, not a failure. Reddit in
# particular 429s readily on its .rss views. Surface it as a state so the
# caller can back off, rather than raising and logging an error every cycle.
if resp.status_code == 429:
# 429 and 403 are both "the server said no". Reddit returns either one
# depending on the moment -- measured across four subreddits in a single
# pass: two 403, two 429, while all four had ingested items earlier. Neither
# is actionable per cycle, and raising on the 403 logged an error every
# fifteen minutes for a source that was working intermittently.
if resp.status_code in (403, 429):
state["retry_after"] = resp.headers.get("retry-after")
state["rate_limited"] = True
return [], state
+1 -1
View File
@@ -149,7 +149,7 @@ bucket = "lobsters"
poll_interval_secs = 1800
# Reddit's .json endpoints return 403 without OAuth as of 2026-09; the .rss
# views work unauthenticated but 429 aggressively -- three requests in quick
# views work unauthenticated but 403/429 aggressively -- three requests in quick
# succession from one IP were enough to get throttled during validation, and the
# block persisted across serialized retries. Hence the long intervals here.
# The fetcher treats 429 as a state to back off from, not an error.
+8
View File
@@ -75,6 +75,14 @@ async def test_429_is_reported_as_rate_limited_not_raised():
assert state["retry_after"] == "120"
@pytest.mark.asyncio
async def test_403_is_also_treated_as_blocked_not_an_error():
"""Reddit returns 403 or 429 depending on the moment; neither is actionable
per cycle, and raising on one logged an error every fifteen minutes."""
items, state = await _fetch("", status=403)
assert items == [] and state["rate_limited"] is True
@pytest.mark.asyncio
async def test_server_error_still_raises():
with pytest.raises(httpx.HTTPStatusError):