fix: YouTube timeout bump to 90s + Reddit 429 fail-fast
YouTube: Add youtube_future timeout key (60/90/120s for quick/default/deep) separate from the shared future timeout. YouTube needs more time because it does search + parallel transcript fetching. Previously, 20 videos + 5 transcripts exceeded the 60s budget and all results were discarded. Reddit 429: Propagate rate-limit errors instead of swallowing them. Enrichment now uses 10s timeout / 1 retry (was 30s / 3 retries). On first 429, cancel remaining enrichment and skip Phase 2 Reddit. Total time wasted on 429 drops from ~75s to ~12s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -235,7 +235,7 @@ def search_subreddits(
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
data = http.get(full_url, headers=headers, timeout=15)
|
||||
data = http.get(full_url, headers=headers, timeout=15, retries=1)
|
||||
|
||||
# Reddit search returns {"data": {"children": [...]}}
|
||||
children = data.get("data", {}).get("children", [])
|
||||
@@ -267,6 +267,9 @@ def search_subreddits(
|
||||
|
||||
except http.HTTPError as e:
|
||||
_log_info(f"Subreddit search failed for r/{sub}: {e}")
|
||||
if e.status_code == 429:
|
||||
_log_info("Reddit rate-limited (429) — skipping remaining subreddits")
|
||||
break
|
||||
except Exception as e:
|
||||
_log_info(f"Subreddit search error for r/{sub}: {e}")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user