fix(reddit): restore free path via keyless RSS + shreddit scrape (.json is dead) (#457)
* test(reddit): add live RSS + shreddit comment fixtures Captured from reddit.com on 2026-05-29 (search.rss listing + the /svc/shreddit/comments partial), trimmed to a representative subset plus two synthetic edge cases (deleted author, negative score) for offline parser tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(http): add keyless get_text helper Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or network failure so tiered callers fall through cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): keyless RSS discovery (search.rss + listing feeds) Replaces the now-403 search.json with keyless Atom feeds, normalized to the existing reddit_public post shape. Scores are placeholder zeros, backfilled during shreddit enrichment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): keyless shreddit comment scraper Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id} (score/author/created/permalink + thingId-anchored body) into top comments, matching reddit_enrich output. Replaces the dead {thread}.json enrichment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): tiered keyless orchestrator Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery -> Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup still engages when every keyless tier is empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): route free path through keyless pipeline (.json is dead) search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py and other callers need no change. Removes the dead .json enrichment helpers; search/_parse_posts remain as the demoted Tier 0 attempt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): request sort=top so true top comments land on page 1 Guarantees the highest-scored comments are captured even on large threads, independent of Reddit's default comment sort. Local score re-sort remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): recover post upvote scores via keyless listing partials The shreddit community-more-posts partial server-renders each post's score and comment count (works for normal users, not IP-gated), unlike RSS or the comments endpoint. Use it as a scored discovery source and to backfill scores onto RSS-discovered posts (subreddits derived from results when not provided). Ranking now uses real upvote score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): listings backfill scores only on bare queries, not discovery Caught running the full pipeline on a bare topic: deriving subreddits from noisy RSS results and merging their top/hot listings flooded results with high-upvote off-topic posts. Now derived-subreddit listings are used only to backfill scores onto keyword-matched RSS posts; listing cards are merged as discovery only when the caller explicitly provides subreddits (on-topic). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
+16
-75
@@ -326,83 +326,24 @@ class TestMissingSubreddit:
|
||||
assert results == []
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests for comment enrichment (Unit 2)
|
||||
# search_reddit_public is now a thin shim over the keyless pipeline.
|
||||
# Full discovery + enrichment behavior is covered in test_reddit_keyless.py.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEnrichmentIntegration:
|
||||
"""search_reddit_public enriches top posts with comments."""
|
||||
class TestSearchRedditPublicDelegatesToKeyless:
|
||||
"""search_reddit_public delegates to reddit_keyless.search_and_enrich."""
|
||||
|
||||
@mock.patch("lib.reddit_public._enrich_post")
|
||||
@mock.patch("lib.reddit_public.urllib.request.urlopen")
|
||||
def test_search_enriches_top_5_by_default(self, mock_urlopen, mock_enrich):
|
||||
listing = _make_reddit_listing([
|
||||
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
|
||||
"score": 100 - i, "created_utc": 1711670400}
|
||||
for i in range(10)
|
||||
])
|
||||
mock_urlopen.return_value = _mock_urlopen_ok(listing)
|
||||
mock_enrich.side_effect = lambda item, timeout=10: item # pass-through
|
||||
def test_delegates_with_all_args(self):
|
||||
with mock.patch("lib.reddit_keyless.search_and_enrich") as mock_keyless:
|
||||
mock_keyless.return_value = [{"id": "R1", "title": "x"}]
|
||||
results = reddit_public.search_reddit_public(
|
||||
"test", "2024-03-01", "2024-03-31",
|
||||
depth="quick", subreddits=["ClaudeAI"],
|
||||
)
|
||||
|
||||
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
|
||||
|
||||
assert len(results) == 10
|
||||
# Default depth enriches top 5
|
||||
assert mock_enrich.call_count == 5
|
||||
|
||||
@mock.patch("lib.reddit_public._enrich_post")
|
||||
@mock.patch("lib.reddit_public.urllib.request.urlopen")
|
||||
def test_enrichment_timeout_keeps_posts(self, mock_urlopen, mock_enrich):
|
||||
listing = _make_reddit_listing([
|
||||
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
|
||||
"score": 100 - i, "created_utc": 1711670400}
|
||||
for i in range(10)
|
||||
])
|
||||
mock_urlopen.return_value = _mock_urlopen_ok(listing)
|
||||
|
||||
# Some enrichments raise, some succeed
|
||||
call_count = {"n": 0}
|
||||
def _side_effect(item, timeout=10):
|
||||
call_count["n"] += 1
|
||||
if call_count["n"] % 2 == 0:
|
||||
raise TimeoutError("enrichment timed out")
|
||||
return item
|
||||
mock_enrich.side_effect = _side_effect
|
||||
|
||||
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
|
||||
|
||||
# All 10 posts should still be returned
|
||||
assert len(results) == 10
|
||||
|
||||
@mock.patch("lib.reddit_public._enrich_post")
|
||||
@mock.patch("lib.reddit_public.urllib.request.urlopen")
|
||||
def test_all_enrichment_fails_all_posts_returned(self, mock_urlopen, mock_enrich):
|
||||
listing = _make_reddit_listing([
|
||||
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
|
||||
"score": 100 - i, "created_utc": 1711670400}
|
||||
for i in range(10)
|
||||
])
|
||||
mock_urlopen.return_value = _mock_urlopen_ok(listing)
|
||||
mock_enrich.side_effect = Exception("total failure")
|
||||
|
||||
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
|
||||
|
||||
# All posts returned despite enrichment failure
|
||||
assert len(results) == 10
|
||||
|
||||
@mock.patch("lib.reddit_public._enrich_post")
|
||||
@mock.patch("lib.reddit_public.urllib.request.urlopen")
|
||||
def test_quick_depth_enriches_top_3(self, mock_urlopen, mock_enrich):
|
||||
listing = _make_reddit_listing([
|
||||
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
|
||||
"score": 100 - i, "created_utc": 1711670400}
|
||||
for i in range(10)
|
||||
])
|
||||
mock_urlopen.return_value = _mock_urlopen_ok(listing)
|
||||
mock_enrich.side_effect = lambda item, timeout=10: item
|
||||
|
||||
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31", depth="quick")
|
||||
|
||||
assert len(results) == 10
|
||||
# Quick depth enriches only top 3
|
||||
assert mock_enrich.call_count == 3
|
||||
assert results == [{"id": "R1", "title": "x"}]
|
||||
mock_keyless.assert_called_once_with(
|
||||
"test", "2024-03-01", "2024-03-31",
|
||||
depth="quick", subreddits=["ClaudeAI"],
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user