sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(reddit): restore free path via keyless RSS + shreddit scrape (.json is dead) (#457)

Browse Source 
* test(reddit): add live RSS + shreddit comment fixtures

Captured from reddit.com on 2026-05-29 (search.rss listing + the
/svc/shreddit/comments partial), trimmed to a representative subset plus
two synthetic edge cases (deleted author, negative score) for offline
parser tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(http): add keyless get_text helper

Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or
network failure so tiered callers fall through cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless RSS discovery (search.rss + listing feeds)

Replaces the now-403 search.json with keyless Atom feeds, normalized to the
existing reddit_public post shape. Scores are placeholder zeros, backfilled
during shreddit enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless shreddit comment scraper

Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id}
(score/author/created/permalink + thingId-anchored body) into top comments,
matching reddit_enrich output. Replaces the dead {thread}.json enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): tiered keyless orchestrator

Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery ->
Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup
still engages when every keyless tier is empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): route free path through keyless pipeline (.json is dead)

search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py
and other callers need no change. Removes the dead .json enrichment helpers;
search/_parse_posts remain as the demoted Tier 0 attempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): request sort=top so true top comments land on page 1

Guarantees the highest-scored comments are captured even on large threads,
independent of Reddit's default comment sort. Local score re-sort remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): recover post upvote scores via keyless listing partials

The shreddit community-more-posts partial server-renders each post's score
and comment count (works for normal users, not IP-gated), unlike RSS or the
comments endpoint. Use it as a scored discovery source and to backfill scores
onto RSS-discovered posts (subreddits derived from results when not provided).
Ranking now uses real upvote score.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): listings backfill scores only on bare queries, not discovery

Caught running the full pipeline on a bare topic: deriving subreddits from
noisy RSS results and merging their top/hot listings flooded results with
high-upvote off-topic posts. Now derived-subreddit listings are used only to
backfill scores onto keyword-matched RSS posts; listing cards are merged as
discovery only when the caller explicitly provides subreddits (on-topic).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-05-29 14:43:56 -05:00
committed by GitHub
parent 1e03af19e0
commit 8d3a9e4368
14 changed files with 1389 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/last30days/scripts/lib/reddit_public.py
+25 -141
View File
@@ -1,9 +1,16 @@
"""Standalone Reddit public JSON search module.
"""Reddit public ``.json`` search module (demoted to keyless Tier 0).
Searches Reddit using the free public JSON endpoints (no API key required).
Promoted from last-resort fallback to robust primary free path.
Reddit's public ``.json`` endpoints now return HTTP 403 from most contexts
(shreddit anti-bot), so this is no longer the primary free path. The keyless
pipeline (see reddit_keyless.py) still calls ``search`` as a cheap one-shot
Tier 0 attempt — a residential machine may occasionally get a 200 — before
falling through to RSS discovery (reddit_rss.py) and shreddit comment
enrichment (reddit_shreddit.py).
Endpoints:
``search_reddit_public`` is retained as a compatibility shim that delegates to
the keyless pipeline, so existing callers (pipeline.py) need no change.
Endpoints (Tier 0):
- Global: https://www.reddit.com/search.json?q={query}&sort=relevance&t=month&limit={limit}
- Subreddit: https://www.reddit.com/r/{sub}/search.json?q={query}&restrict_sr=on&sort=relevance&t=month
@@ -18,7 +25,6 @@ import time
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from typing import Any, Dict, List, Optional
@@ -35,13 +41,6 @@ DEPTH_LIMITS = {
"deep": 50,
}
# How many top posts to enrich with comments, by depth
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
MAX_RETRIES = 3
BASE_BACKOFF = 2.0 # seconds
@@ -237,78 +236,6 @@ def search(
return unique[:limit]
def _enrich_post(item: Dict[str, Any], timeout: int = 10) -> Dict[str, Any]:
"""Enrich a single post with top comments. Never raises."""
try:
from . import reddit_enrich
thread_data = reddit_enrich.fetch_thread_data(item["url"], timeout=timeout)
if not thread_data:
return item
parsed = reddit_enrich.parse_thread_data(thread_data)
comments = parsed.get("comments", [])
top = reddit_enrich.get_top_comments(comments)
item["top_comments"] = [
{
"score": c.get("score", 0),
"excerpt": (c.get("body") or "")[:200],
"author": c.get("author", ""),
}
for c in top[:10]
]
except Exception:
# Never discard — keep post with empty metadata
pass
return item
def _enrich_posts(posts: List[Dict[str, Any]], depth: str = "default") -> List[Dict[str, Any]]:
"""Enrich top N posts with comment data using threads. Total budget 45s."""
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
to_enrich = posts[:limit]
rest = posts[limit:]
if not to_enrich:
return posts
enriched = []
try:
with ThreadPoolExecutor(max_workers=min(limit, 4)) as executor:
futures = {
executor.submit(_enrich_post, post, 10): i
for i, post in enumerate(to_enrich)
}
# Collect results with 45s total budget
import concurrent.futures
done, not_done = concurrent.futures.wait(futures, timeout=45)
# Build result list preserving order
result_map: Dict[int, Dict[str, Any]] = {}
for future in done:
idx = futures[future]
try:
result_map[idx] = future.result(timeout=0)
except Exception:
result_map[idx] = to_enrich[idx]
# Any not-done futures: keep original post
for future in not_done:
idx = futures[future]
result_map[idx] = to_enrich[idx]
future.cancel()
enriched = [result_map[i] for i in range(len(to_enrich))]
except Exception:
enriched = to_enrich
return enriched + rest
def _search_subreddit(sub: str, topic: str, depth: str, timeout: int = 15) -> List[Dict[str, Any]]:
"""Search a single subreddit. Never raises."""
try:
return search(topic, depth=depth, subreddit=sub, timeout=timeout)
except Exception as e:
_log(f"Subreddit search failed for r/{sub}: {e}")
return []
def search_reddit_public(
topic: str,
from_date: str,
@@ -316,12 +243,17 @@ def search_reddit_public(
depth: str = "default",
subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""High-level Reddit public search matching the openai_reddit interface.
"""High-level free Reddit search + enrichment (keyless).
When subreddits are provided (from agent planning), searches each targeted
sub first, then does global search, and deduplicates across both. This
mirrors the SC search_and_enrich() flow where pre-resolved subreddits get
priority.
Thin compatibility shim over the tiered keyless pipeline: the legacy
``.json`` search/enrichment endpoints now return HTTP 403, so this delegates
to ``reddit_keyless.search_and_enrich`` (Tier 0 one-shot ``.json`` →
Tier 1 RSS discovery → Tier 2 shreddit comment enrichment). The name and
signature are preserved so ``pipeline.py`` and other callers need no change
and the ScrapeCreators backup still engages when this returns empty.
The module-level ``search`` / ``_parse_posts`` helpers remain in use as the
keyless pipeline's demoted Tier 0 ``.json`` attempt.
Args:
topic: Search topic
@@ -332,57 +264,9 @@ def search_reddit_public(
Returns:
List of normalized item dicts matching ScrapeCreators output format.
Empty list on total failure (so SC backup can engage).
"""
all_posts: List[Dict[str, Any]] = []
# Phase 1: Search targeted subreddits in parallel (if provided)
if subreddits:
_log(f"Searching {len(subreddits)} targeted subreddits: {subreddits}")
workers = min(4, len(subreddits))
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(_search_subreddit, sub, topic, depth): sub
for sub in subreddits
}
for future in futures:
sub = futures[future]
try:
sub_posts = future.result(timeout=30)
_log(f" -> {len(sub_posts)} results from r/{sub}")
all_posts.extend(sub_posts)
except (Exception, FuturesTimeoutError) as e:
_log(f" -> r/{sub} failed: {e}")
# Phase 2: Global search
global_posts = search(topic, depth=depth)
all_posts.extend(global_posts)
# Deduplicate by URL (targeted results keep priority since they come first)
seen_urls: set = set()
results: List[Dict[str, Any]] = []
for post in all_posts:
if post["url"] not in seen_urls:
seen_urls.add(post["url"])
results.append(post)
# Date filter: keep posts in range or with unknown dates
filtered = []
for item in results:
d = item.get("date")
if d is None or (from_date <= d <= to_date):
filtered.append(item)
# Sort by engagement (score desc)
filtered.sort(
key=lambda x: x.get("engagement", {}).get("score", 0),
reverse=True,
from . import reddit_keyless
return reddit_keyless.search_and_enrich(
topic, from_date, to_date, depth=depth, subreddits=subreddits
)
# Enrich top posts with comments
filtered = _enrich_posts(filtered, depth=depth)
# Re-index IDs
for i, item in enumerate(filtered):
item["id"] = f"R{i + 1}"
return filtered