Treat a 403 as blocked rather than an error
Reddit returns 403 or 429 depending on the moment -- measured across four subreddits in one pass: two 403, two 429, while three of the four had ingested 44-55 items earlier, so the source works intermittently. Only the 429 was handled softly, so the 403 raised and logged an error every fifteen minutes for a source that was fine. Neither status is actionable per cycle. Both now back off quietly and record the status on the source row, where it is visible via `news sources`.
This commit is contained in:
1 parent
9cba8db7bb
commit
c03df8c1b5
3 files changed
+15
-2
No files matched your search
+6
-1
@@ -64,7 +64,12 @@ async def fetch(client, source) -> tuple[list[dict], dict]:
|
||||
# Rate limiting is an expected steady state, not a failure. Reddit in
|
||||
# particular 429s readily on its .rss views. Surface it as a state so the
|
||||
# caller can back off, rather than raising and logging an error every cycle.
|
||||
if resp.status_code == 429:
|
||||
# 429 and 403 are both "the server said no". Reddit returns either one
|
||||
# depending on the moment -- measured across four subreddits in a single
|
||||
# pass: two 403, two 429, while all four had ingested items earlier. Neither
|
||||
# is actionable per cycle, and raising on the 403 logged an error every
|
||||
# fifteen minutes for a source that was working intermittently.
|
||||
if resp.status_code in (403, 429):
|
||||
state["retry_after"] = resp.headers.get("retry-after")
|
||||
state["rate_limited"] = True
|
||||
return [], state
|
||||
|
||||
@@ -149,7 +149,7 @@ bucket = "lobsters"
|
||||
poll_interval_secs = 1800
|
||||
|
||||
# Reddit's .json endpoints return 403 without OAuth as of 2026-09; the .rss
|
||||
# views work unauthenticated but 429 aggressively -- three requests in quick
|
||||
# views work unauthenticated but 403/429 aggressively -- three requests in quick
|
||||
# succession from one IP were enough to get throttled during validation, and the
|
||||
# block persisted across serialized retries. Hence the long intervals here.
|
||||
# The fetcher treats 429 as a state to back off from, not an error.
|
||||
|
||||
@@ -75,6 +75,14 @@ async def test_429_is_reported_as_rate_limited_not_raised():
|
||||
assert state["retry_after"] == "120"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_403_is_also_treated_as_blocked_not_an_error():
|
||||
"""Reddit returns 403 or 429 depending on the moment; neither is actionable
|
||||
per cycle, and raising on one logged an error every fifteen minutes."""
|
||||
items, state = await _fetch("", status=403)
|
||||
assert items == [] and state["rate_limited"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_server_error_still_raises():
|
||||
with pytest.raises(httpx.HTTPStatusError):
|
||||
|
||||
Reference in new issue
Block a user