fix(reddit): restore free path via keyless RSS + shreddit scrape (.json is dead) (#457)
* test(reddit): add live RSS + shreddit comment fixtures Captured from reddit.com on 2026-05-29 (search.rss listing + the /svc/shreddit/comments partial), trimmed to a representative subset plus two synthetic edge cases (deleted author, negative score) for offline parser tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(http): add keyless get_text helper Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or network failure so tiered callers fall through cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): keyless RSS discovery (search.rss + listing feeds) Replaces the now-403 search.json with keyless Atom feeds, normalized to the existing reddit_public post shape. Scores are placeholder zeros, backfilled during shreddit enrichment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): keyless shreddit comment scraper Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id} (score/author/created/permalink + thingId-anchored body) into top comments, matching reddit_enrich output. Replaces the dead {thread}.json enrichment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): tiered keyless orchestrator Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery -> Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup still engages when every keyless tier is empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): route free path through keyless pipeline (.json is dead) search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py and other callers need no change. Removes the dead .json enrichment helpers; search/_parse_posts remain as the demoted Tier 0 attempt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): request sort=top so true top comments land on page 1 Guarantees the highest-scored comments are captured even on large threads, independent of Reddit's default comment sort. Local score re-sort remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reddit): recover post upvote scores via keyless listing partials The shreddit community-more-posts partial server-renders each post's score and comment count (works for normal users, not IP-gated), unlike RSS or the comments endpoint. Use it as a scored discovery source and to backfill scores onto RSS-discovered posts (subreddits derived from results when not provided). Ranking now uses real upvote score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(reddit): listings backfill scores only on bare queries, not discovery Caught running the full pipeline on a bare topic: deriving subreddits from noisy RSS results and merging their top/hot listings flooded results with high-upvote off-topic posts. Now derived-subreddit listings are used only to backfill scores onto keyword-matched RSS posts; listing cards are merged as discovery only when the caller explicitly provides subreddits (on-topic). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -223,6 +223,53 @@ def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, st
|
||||
return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)
|
||||
|
||||
|
||||
BROWSER_USER_AGENT = (
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/124.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
|
||||
def get_text(
|
||||
url: str,
|
||||
timeout: int = DEFAULT_TIMEOUT,
|
||||
retries: int = 2,
|
||||
accept: str = "*/*",
|
||||
headers: Optional[Dict[str, str]] = None,
|
||||
) -> Optional[str]:
|
||||
"""Fetch a URL and return decoded text, or None on any failure.
|
||||
|
||||
Keyless helper for Reddit RSS and shreddit HTML endpoints — the free path
|
||||
that replaced the now-403 ``.json`` endpoints. Sends a browser User-Agent
|
||||
and never raises: returns None on HTTP error, network failure, or timeout
|
||||
so tiered callers can fall through to the next source.
|
||||
|
||||
Args:
|
||||
url: Request URL
|
||||
timeout: HTTP timeout per attempt in seconds
|
||||
retries: Number of retries on failure (kept low — these tiers fail fast)
|
||||
accept: Accept header value (e.g. "application/atom+xml", "text/html")
|
||||
headers: Optional extra headers merged over the defaults
|
||||
|
||||
Returns:
|
||||
Decoded response body as text, or None on failure.
|
||||
"""
|
||||
merged = {
|
||||
"User-Agent": BROWSER_USER_AGENT,
|
||||
"Accept": accept,
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
if headers:
|
||||
merged.update(headers)
|
||||
try:
|
||||
return request(
|
||||
"GET", url, headers=merged, timeout=timeout, retries=retries, raw=True
|
||||
)
|
||||
except HTTPError as e:
|
||||
log(f"get_text failed ({e}): {url}")
|
||||
return None
|
||||
|
||||
|
||||
def scrapecreators_headers(token: str) -> Dict[str, str]:
|
||||
"""Build ScrapeCreators request headers (x-api-key + JSON content type)."""
|
||||
return {
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Keyless Reddit pipeline: tiered free search + comment enrichment.
|
||||
|
||||
Replaces the dead ``.json`` free path. Discovery tiers, cheapest/most-likely
|
||||
first; enrichment then runs on whatever was discovered:
|
||||
|
||||
Tier 0 one-shot legacy ``.json`` search — demoted. Datacenter IPs get 403,
|
||||
but a residential machine (where the skill usually runs) may still
|
||||
get 200, so it is worth one cheap try. Honors the "brute-force .json"
|
||||
intent without depending on it.
|
||||
Tier 1 RSS discovery (reddit_rss) — keyless, robust, the load-bearing path.
|
||||
Tier 2 shreddit comment + count enrichment (reddit_shreddit) for top posts.
|
||||
|
||||
Returns ``[]`` (never raises) so ``pipeline.py`` can fall through to the
|
||||
ScrapeCreators backup when every keyless tier comes up empty.
|
||||
"""
|
||||
|
||||
import concurrent.futures
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from . import reddit_rss, reddit_shreddit, reddit_listing
|
||||
|
||||
ENRICH_LIMITS = reddit_shreddit.ENRICH_LIMITS
|
||||
ENRICH_BUDGET = 45 # seconds total across all enrichment threads
|
||||
MAX_ENRICH_WORKERS = 4
|
||||
MAX_DERIVED_SUBS = 5 # subreddits derived from RSS results for score backfill
|
||||
|
||||
|
||||
def _log(msg: str) -> None:
|
||||
sys.stderr.write(f"[RedditKeyless] {msg}\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def _tier0_json(topic: str, depth: str) -> List[Dict[str, Any]]:
|
||||
"""One cheap global ``.json`` discovery attempt. Returns [] on the 403 wall."""
|
||||
try:
|
||||
from . import reddit_public
|
||||
return reddit_public.search(topic, depth=depth) or []
|
||||
except Exception as e: # never let the demoted tier sink the run
|
||||
_log(f"Tier 0 (.json) unavailable: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def _top_subreddits(posts: List[Dict[str, Any]], limit: int = MAX_DERIVED_SUBS) -> List[str]:
|
||||
"""Most frequent subreddits across discovered posts (for score backfill)."""
|
||||
counts = Counter(p.get("subreddit", "") for p in posts if p.get("subreddit"))
|
||||
return [sub for sub, _ in counts.most_common(limit)]
|
||||
|
||||
|
||||
def _apply_scores(post: Dict[str, Any], scored: Dict[str, int]) -> None:
|
||||
post["score"] = scored["score"]
|
||||
post["num_comments"] = scored["num_comments"]
|
||||
post.setdefault("engagement", {})["score"] = scored["score"]
|
||||
post["engagement"]["num_comments"] = scored["num_comments"]
|
||||
|
||||
|
||||
def _discover(topic: str, depth: str, subreddits: Optional[List[str]]) -> List[Dict[str, Any]]:
|
||||
# Tier 0: demoted one-shot .json (dead for normal users too, but free to try).
|
||||
posts = _tier0_json(topic, depth)
|
||||
if posts:
|
||||
_log(f"Tier 0 (.json) returned {len(posts)} posts")
|
||||
return posts
|
||||
|
||||
# Tier 1: keyless discovery. RSS gives breadth (incl. global keyword search);
|
||||
# the listing partials give real upvote scores.
|
||||
rss_posts = reddit_rss.search_rss(topic, depth=depth, subreddits=subreddits)
|
||||
|
||||
if subreddits:
|
||||
# Targeted run: the caller chose these subreddits, so their listing cards
|
||||
# are on-topic — include them as scored discovery AND as a score source.
|
||||
listing_posts = reddit_listing.fetch_listings(subreddits, depth=depth, query=topic)
|
||||
score_source = listing_posts
|
||||
else:
|
||||
# Bare global run: subreddits derived from noisy RSS results are NOT
|
||||
# reliably on-topic, so their listings are used ONLY to backfill scores
|
||||
# onto the keyword-matched RSS posts — never merged as discovery, which
|
||||
# would flood results with high-upvote but irrelevant posts.
|
||||
listing_posts = []
|
||||
derived = _top_subreddits(rss_posts)
|
||||
score_source = reddit_listing.fetch_listings(derived, depth=depth, query=topic)
|
||||
_log(
|
||||
f"Tier 1 (RSS) {len(rss_posts)} posts; "
|
||||
f"{'listing discovery ' + str(len(listing_posts)) if subreddits else 'score-only'}; "
|
||||
f"{len(score_source)} scored cards"
|
||||
)
|
||||
|
||||
# Score lookup by post id, from the scored listing cards.
|
||||
score_map: Dict[str, Dict[str, int]] = {}
|
||||
for p in score_source:
|
||||
pid = p.get("metadata", {}).get("post_id", "")
|
||||
if pid:
|
||||
score_map[pid] = {"score": p["score"], "num_comments": p["num_comments"]}
|
||||
|
||||
# Merge: scored listing posts first (targeted only), then RSS breadth,
|
||||
# backfilled with real scores where the post appears in a listing.
|
||||
merged: List[Dict[str, Any]] = []
|
||||
seen: set = set()
|
||||
for p in listing_posts:
|
||||
if p["url"] not in seen:
|
||||
seen.add(p["url"])
|
||||
merged.append(p)
|
||||
for p in rss_posts:
|
||||
if p["url"] in seen:
|
||||
continue
|
||||
pid = reddit_listing._post_id(p["url"])
|
||||
if pid in score_map:
|
||||
_apply_scores(p, score_map[pid])
|
||||
seen.add(p["url"])
|
||||
merged.append(p)
|
||||
return merged
|
||||
|
||||
|
||||
def _enrich_one(post: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Attach shreddit comments + real comment count. Never raises."""
|
||||
try:
|
||||
data = reddit_shreddit.fetch_comments(post.get("url", ""))
|
||||
if data.get("top_comments"):
|
||||
post["top_comments"] = data["top_comments"]
|
||||
if data.get("comment_insights"):
|
||||
post["comment_insights"] = data["comment_insights"]
|
||||
num = data.get("num_comments")
|
||||
if num is not None:
|
||||
post["num_comments"] = num
|
||||
post.setdefault("engagement", {})["num_comments"] = num
|
||||
except Exception:
|
||||
pass # keep the post with whatever discovery gave us
|
||||
return post
|
||||
|
||||
|
||||
def _enrich(posts: List[Dict[str, Any]], depth: str) -> List[Dict[str, Any]]:
|
||||
"""Enrich the top N posts with comments under a total time budget."""
|
||||
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
|
||||
to_enrich = posts[:limit]
|
||||
rest = posts[limit:]
|
||||
if not to_enrich:
|
||||
return posts
|
||||
|
||||
result_map: Dict[int, Dict[str, Any]] = {}
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=min(limit, MAX_ENRICH_WORKERS)) as executor:
|
||||
futures = {
|
||||
executor.submit(_enrich_one, post): i
|
||||
for i, post in enumerate(to_enrich)
|
||||
}
|
||||
done, not_done = concurrent.futures.wait(futures, timeout=ENRICH_BUDGET)
|
||||
for future in done:
|
||||
idx = futures[future]
|
||||
try:
|
||||
result_map[idx] = future.result(timeout=0)
|
||||
except Exception:
|
||||
result_map[idx] = to_enrich[idx]
|
||||
for future in not_done:
|
||||
idx = futures[future]
|
||||
result_map[idx] = to_enrich[idx]
|
||||
future.cancel()
|
||||
enriched = [result_map[i] for i in range(len(to_enrich))]
|
||||
except Exception:
|
||||
enriched = to_enrich
|
||||
|
||||
return enriched + rest
|
||||
|
||||
|
||||
def search_and_enrich(
|
||||
topic: str,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
depth: str = "default",
|
||||
subreddits: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Full keyless Reddit pipeline: discover (Tier 0/1) then enrich (Tier 2).
|
||||
|
||||
Args:
|
||||
topic: Search topic
|
||||
from_date: Start date (YYYY-MM-DD)
|
||||
to_date: End date (YYYY-MM-DD)
|
||||
depth: 'quick', 'default', or 'deep'
|
||||
subreddits: Optional pre-resolved subreddit names (without r/)
|
||||
|
||||
Returns:
|
||||
List of normalized item dicts matching the reddit_public output shape,
|
||||
with top_comments/comment_insights attached on enriched posts.
|
||||
Empty list when all keyless tiers fail (so SC backup can engage).
|
||||
"""
|
||||
posts = _discover(topic, depth, subreddits)
|
||||
if not posts:
|
||||
return []
|
||||
|
||||
# Date filter: keep posts in range or with unknown dates (mirrors reddit_public).
|
||||
posts = [
|
||||
p for p in posts
|
||||
if p.get("date") is None or (from_date <= p["date"] <= to_date)
|
||||
]
|
||||
|
||||
# Rank before enrichment by real upvote score (from listing cards / backfill),
|
||||
# then query relevance, then recency. Posts without a recovered score sort by
|
||||
# the latter two — same behavior as before scores were available.
|
||||
posts.sort(
|
||||
key=lambda p: (
|
||||
p.get("engagement", {}).get("score", 0) or 0,
|
||||
p.get("relevance", 0) or 0,
|
||||
p.get("date") or "",
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
posts = _enrich(posts, depth)
|
||||
|
||||
for i, post in enumerate(posts):
|
||||
post["id"] = f"R{i + 1}"
|
||||
|
||||
return posts
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Keyless Reddit listing scrape via shreddit /svc partials — with real scores.
|
||||
|
||||
The subreddit listing partial
|
||||
``/svc/shreddit/community-more-posts/{sort}/?name={sub}[&t={range}]`` serves
|
||||
HTTP 200 with no API key and **server-renders each post's upvote score**, which
|
||||
neither RSS nor the comments endpoint provides. Each post is a
|
||||
``<shreddit-post>`` element whose start-tag attributes carry ``score``,
|
||||
``comment-count``, ``post-title``, ``permalink``, ``author``, ``subreddit-name``
|
||||
and ``created-timestamp``.
|
||||
|
||||
This is the keyless source of post-level upvotes. It works for normal users on
|
||||
ordinary connections (verified), so reddit_keyless uses it both as a scored
|
||||
discovery source and to backfill scores onto RSS-discovered posts.
|
||||
"""
|
||||
|
||||
import html as _html
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from . import http
|
||||
from .relevance import token_overlap_relevance
|
||||
|
||||
# Listing sorts pulled per subreddit, by depth.
|
||||
LISTING_SORTS = {
|
||||
"quick": ["top"],
|
||||
"default": ["top", "hot"],
|
||||
"deep": ["top", "hot", "new"],
|
||||
}
|
||||
DEPTH_LIMITS = {"quick": 10, "default": 25, "deep": 50}
|
||||
TIMEFRAME = "month"
|
||||
MAX_WORKERS = 4
|
||||
LISTING_TIMEOUT = 15
|
||||
|
||||
_POST_CARD = re.compile(r"<shreddit-post(?=[\s>])[^>]*>")
|
||||
|
||||
|
||||
def _log(msg: str) -> None:
|
||||
sys.stderr.write(f"[RedditListing] {msg}\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def _attr(tag: str, name: str) -> Optional[str]:
|
||||
m = re.search(rf'\b{name}="([^"]*)"', tag)
|
||||
return _html.unescape(m.group(1)) if m else None
|
||||
|
||||
|
||||
def _to_date(value: Optional[str]) -> Optional[str]:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(value.strip()).date().isoformat()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _to_epoch(value: Optional[str]) -> Optional[float]:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip())
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.timestamp()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _post_id(permalink: str) -> str:
|
||||
m = re.search(r"/comments/([A-Za-z0-9]+)", permalink or "")
|
||||
return m.group(1) if m else ""
|
||||
|
||||
|
||||
def parse_cards(html_text: str, query: str = "") -> List[Dict[str, Any]]:
|
||||
"""Parse <shreddit-post> cards into normalized post dicts with real scores."""
|
||||
posts: List[Dict[str, Any]] = []
|
||||
for m in _POST_CARD.finditer(html_text or ""):
|
||||
tag = m.group(0)
|
||||
permalink = _attr(tag, "permalink") or ""
|
||||
if "/comments/" not in permalink:
|
||||
continue
|
||||
try:
|
||||
score = int(_attr(tag, "score") or 0)
|
||||
except ValueError:
|
||||
score = 0
|
||||
try:
|
||||
num_comments = int(_attr(tag, "comment-count") or 0)
|
||||
except ValueError:
|
||||
num_comments = 0
|
||||
title = _attr(tag, "post-title") or ""
|
||||
author = _attr(tag, "author") or "[deleted]"
|
||||
subreddit = _attr(tag, "subreddit-name") or ""
|
||||
created = _attr(tag, "created-timestamp")
|
||||
url = f"https://www.reddit.com{permalink}"
|
||||
|
||||
posts.append({
|
||||
"id": "",
|
||||
"title": title,
|
||||
"url": url,
|
||||
"score": score,
|
||||
"num_comments": num_comments,
|
||||
"subreddit": subreddit,
|
||||
"created_utc": _to_epoch(created),
|
||||
"author": author if author not in ("[deleted]", "[removed]") else "[deleted]",
|
||||
"selftext": "",
|
||||
"date": _to_date(created),
|
||||
"engagement": {
|
||||
"score": score,
|
||||
"num_comments": num_comments,
|
||||
"upvote_ratio": None,
|
||||
},
|
||||
"relevance": round(token_overlap_relevance(query, title), 3) if query else 0.0,
|
||||
"why_relevant": "Reddit listing",
|
||||
"metadata": {"post_id": _post_id(permalink)},
|
||||
})
|
||||
return posts
|
||||
|
||||
|
||||
def _listing_url(subreddit: str, sort: str) -> str:
|
||||
sub = subreddit.removeprefix("r/").strip()
|
||||
url = f"https://www.reddit.com/svc/shreddit/community-more-posts/{sort}/?name={sub}"
|
||||
if sort == "top":
|
||||
url += f"&t={TIMEFRAME}"
|
||||
return url
|
||||
|
||||
|
||||
def _fetch_one(subreddit: str, sort: str, query: str) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
text = http.get_text(_listing_url(subreddit, sort), timeout=LISTING_TIMEOUT,
|
||||
accept="text/html")
|
||||
return parse_cards(text, query) if text else []
|
||||
except Exception as e:
|
||||
_log(f"listing fetch failed r/{subreddit} {sort}: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def fetch_listings(
|
||||
subreddits: List[str],
|
||||
depth: str = "default",
|
||||
query: str = "",
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Fetch scored post cards across subreddits × depth-appropriate sorts.
|
||||
|
||||
Returns deduped normalized posts (with real scores), unranked/unsliced —
|
||||
the caller merges these with other sources, ranks, and slices.
|
||||
"""
|
||||
if not subreddits:
|
||||
return []
|
||||
sorts = LISTING_SORTS.get(depth, LISTING_SORTS["default"])
|
||||
jobs = [(sub, sort) for sub in subreddits for sort in sorts]
|
||||
all_posts: List[Dict[str, Any]] = []
|
||||
with ThreadPoolExecutor(max_workers=min(MAX_WORKERS, len(jobs)) or 1) as executor:
|
||||
futures = {executor.submit(_fetch_one, sub, sort, query): (sub, sort)
|
||||
for sub, sort in jobs}
|
||||
for future in futures:
|
||||
try:
|
||||
all_posts.extend(future.result(timeout=LISTING_TIMEOUT + 5))
|
||||
except (Exception, FuturesTimeoutError) as e:
|
||||
_log(f"listing future failed: {e}")
|
||||
|
||||
seen: set = set()
|
||||
unique: List[Dict[str, Any]] = []
|
||||
for p in all_posts:
|
||||
if p["url"] not in seen:
|
||||
seen.add(p["url"])
|
||||
unique.append(p)
|
||||
return unique
|
||||
|
||||
|
||||
def score_index(subreddits: List[str], depth: str = "default") -> Dict[str, Dict[str, int]]:
|
||||
"""Build a {post_id: {score, num_comments}} map from subreddit listings.
|
||||
|
||||
Used to backfill real scores onto posts discovered via RSS, which carries
|
||||
no engagement numbers.
|
||||
"""
|
||||
index: Dict[str, Dict[str, int]] = {}
|
||||
for p in fetch_listings(subreddits, depth=depth):
|
||||
pid = p.get("metadata", {}).get("post_id") or _post_id(p["url"])
|
||||
if pid:
|
||||
index[pid] = {"score": p["score"], "num_comments": p["num_comments"]}
|
||||
return index
|
||||
@@ -1,9 +1,16 @@
|
||||
"""Standalone Reddit public JSON search module.
|
||||
"""Reddit public ``.json`` search module (demoted to keyless Tier 0).
|
||||
|
||||
Searches Reddit using the free public JSON endpoints (no API key required).
|
||||
Promoted from last-resort fallback to robust primary free path.
|
||||
Reddit's public ``.json`` endpoints now return HTTP 403 from most contexts
|
||||
(shreddit anti-bot), so this is no longer the primary free path. The keyless
|
||||
pipeline (see reddit_keyless.py) still calls ``search`` as a cheap one-shot
|
||||
Tier 0 attempt — a residential machine may occasionally get a 200 — before
|
||||
falling through to RSS discovery (reddit_rss.py) and shreddit comment
|
||||
enrichment (reddit_shreddit.py).
|
||||
|
||||
Endpoints:
|
||||
``search_reddit_public`` is retained as a compatibility shim that delegates to
|
||||
the keyless pipeline, so existing callers (pipeline.py) need no change.
|
||||
|
||||
Endpoints (Tier 0):
|
||||
- Global: https://www.reddit.com/search.json?q={query}&sort=relevance&t=month&limit={limit}
|
||||
- Subreddit: https://www.reddit.com/r/{sub}/search.json?q={query}&restrict_sr=on&sort=relevance&t=month
|
||||
|
||||
@@ -18,7 +25,6 @@ import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
@@ -35,13 +41,6 @@ DEPTH_LIMITS = {
|
||||
"deep": 50,
|
||||
}
|
||||
|
||||
# How many top posts to enrich with comments, by depth
|
||||
ENRICH_LIMITS = {
|
||||
"quick": 3,
|
||||
"default": 5,
|
||||
"deep": 8,
|
||||
}
|
||||
|
||||
MAX_RETRIES = 3
|
||||
BASE_BACKOFF = 2.0 # seconds
|
||||
|
||||
@@ -237,78 +236,6 @@ def search(
|
||||
return unique[:limit]
|
||||
|
||||
|
||||
def _enrich_post(item: Dict[str, Any], timeout: int = 10) -> Dict[str, Any]:
|
||||
"""Enrich a single post with top comments. Never raises."""
|
||||
try:
|
||||
from . import reddit_enrich
|
||||
thread_data = reddit_enrich.fetch_thread_data(item["url"], timeout=timeout)
|
||||
if not thread_data:
|
||||
return item
|
||||
parsed = reddit_enrich.parse_thread_data(thread_data)
|
||||
comments = parsed.get("comments", [])
|
||||
top = reddit_enrich.get_top_comments(comments)
|
||||
item["top_comments"] = [
|
||||
{
|
||||
"score": c.get("score", 0),
|
||||
"excerpt": (c.get("body") or "")[:200],
|
||||
"author": c.get("author", ""),
|
||||
}
|
||||
for c in top[:10]
|
||||
]
|
||||
except Exception:
|
||||
# Never discard — keep post with empty metadata
|
||||
pass
|
||||
return item
|
||||
|
||||
|
||||
def _enrich_posts(posts: List[Dict[str, Any]], depth: str = "default") -> List[Dict[str, Any]]:
|
||||
"""Enrich top N posts with comment data using threads. Total budget 45s."""
|
||||
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
|
||||
to_enrich = posts[:limit]
|
||||
rest = posts[limit:]
|
||||
|
||||
if not to_enrich:
|
||||
return posts
|
||||
|
||||
enriched = []
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=min(limit, 4)) as executor:
|
||||
futures = {
|
||||
executor.submit(_enrich_post, post, 10): i
|
||||
for i, post in enumerate(to_enrich)
|
||||
}
|
||||
# Collect results with 45s total budget
|
||||
import concurrent.futures
|
||||
done, not_done = concurrent.futures.wait(futures, timeout=45)
|
||||
# Build result list preserving order
|
||||
result_map: Dict[int, Dict[str, Any]] = {}
|
||||
for future in done:
|
||||
idx = futures[future]
|
||||
try:
|
||||
result_map[idx] = future.result(timeout=0)
|
||||
except Exception:
|
||||
result_map[idx] = to_enrich[idx]
|
||||
# Any not-done futures: keep original post
|
||||
for future in not_done:
|
||||
idx = futures[future]
|
||||
result_map[idx] = to_enrich[idx]
|
||||
future.cancel()
|
||||
enriched = [result_map[i] for i in range(len(to_enrich))]
|
||||
except Exception:
|
||||
enriched = to_enrich
|
||||
|
||||
return enriched + rest
|
||||
|
||||
|
||||
def _search_subreddit(sub: str, topic: str, depth: str, timeout: int = 15) -> List[Dict[str, Any]]:
|
||||
"""Search a single subreddit. Never raises."""
|
||||
try:
|
||||
return search(topic, depth=depth, subreddit=sub, timeout=timeout)
|
||||
except Exception as e:
|
||||
_log(f"Subreddit search failed for r/{sub}: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def search_reddit_public(
|
||||
topic: str,
|
||||
from_date: str,
|
||||
@@ -316,12 +243,17 @@ def search_reddit_public(
|
||||
depth: str = "default",
|
||||
subreddits: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""High-level Reddit public search matching the openai_reddit interface.
|
||||
"""High-level free Reddit search + enrichment (keyless).
|
||||
|
||||
When subreddits are provided (from agent planning), searches each targeted
|
||||
sub first, then does global search, and deduplicates across both. This
|
||||
mirrors the SC search_and_enrich() flow where pre-resolved subreddits get
|
||||
priority.
|
||||
Thin compatibility shim over the tiered keyless pipeline: the legacy
|
||||
``.json`` search/enrichment endpoints now return HTTP 403, so this delegates
|
||||
to ``reddit_keyless.search_and_enrich`` (Tier 0 one-shot ``.json`` →
|
||||
Tier 1 RSS discovery → Tier 2 shreddit comment enrichment). The name and
|
||||
signature are preserved so ``pipeline.py`` and other callers need no change
|
||||
and the ScrapeCreators backup still engages when this returns empty.
|
||||
|
||||
The module-level ``search`` / ``_parse_posts`` helpers remain in use as the
|
||||
keyless pipeline's demoted Tier 0 ``.json`` attempt.
|
||||
|
||||
Args:
|
||||
topic: Search topic
|
||||
@@ -332,57 +264,9 @@ def search_reddit_public(
|
||||
|
||||
Returns:
|
||||
List of normalized item dicts matching ScrapeCreators output format.
|
||||
Empty list on total failure (so SC backup can engage).
|
||||
"""
|
||||
all_posts: List[Dict[str, Any]] = []
|
||||
|
||||
# Phase 1: Search targeted subreddits in parallel (if provided)
|
||||
if subreddits:
|
||||
_log(f"Searching {len(subreddits)} targeted subreddits: {subreddits}")
|
||||
workers = min(4, len(subreddits))
|
||||
with ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
futures = {
|
||||
executor.submit(_search_subreddit, sub, topic, depth): sub
|
||||
for sub in subreddits
|
||||
}
|
||||
for future in futures:
|
||||
sub = futures[future]
|
||||
try:
|
||||
sub_posts = future.result(timeout=30)
|
||||
_log(f" -> {len(sub_posts)} results from r/{sub}")
|
||||
all_posts.extend(sub_posts)
|
||||
except (Exception, FuturesTimeoutError) as e:
|
||||
_log(f" -> r/{sub} failed: {e}")
|
||||
|
||||
# Phase 2: Global search
|
||||
global_posts = search(topic, depth=depth)
|
||||
all_posts.extend(global_posts)
|
||||
|
||||
# Deduplicate by URL (targeted results keep priority since they come first)
|
||||
seen_urls: set = set()
|
||||
results: List[Dict[str, Any]] = []
|
||||
for post in all_posts:
|
||||
if post["url"] not in seen_urls:
|
||||
seen_urls.add(post["url"])
|
||||
results.append(post)
|
||||
|
||||
# Date filter: keep posts in range or with unknown dates
|
||||
filtered = []
|
||||
for item in results:
|
||||
d = item.get("date")
|
||||
if d is None or (from_date <= d <= to_date):
|
||||
filtered.append(item)
|
||||
|
||||
# Sort by engagement (score desc)
|
||||
filtered.sort(
|
||||
key=lambda x: x.get("engagement", {}).get("score", 0),
|
||||
reverse=True,
|
||||
from . import reddit_keyless
|
||||
return reddit_keyless.search_and_enrich(
|
||||
topic, from_date, to_date, depth=depth, subreddits=subreddits
|
||||
)
|
||||
|
||||
# Enrich top posts with comments
|
||||
filtered = _enrich_posts(filtered, depth=depth)
|
||||
|
||||
# Re-index IDs
|
||||
for i, item in enumerate(filtered):
|
||||
item["id"] = f"R{i + 1}"
|
||||
|
||||
return filtered
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
"""Keyless Reddit discovery via public RSS/Atom feeds.
|
||||
|
||||
Reddit's ``.json`` search endpoints now return HTTP 403 (shreddit anti-bot).
|
||||
RSS feeds still serve HTTP 200 with no API key, so this module uses them for
|
||||
post discovery, replacing ``reddit_public.search`` as the free search path.
|
||||
|
||||
Two feed families are combined and deduped:
|
||||
- search: /search.rss?q=... and /r/{sub}/search.rss?q=...&restrict_sr=on
|
||||
- listing: /r/{sub}/{top,hot}.rss?t=month
|
||||
|
||||
RSS entries carry no engagement score, so ``score``/``num_comments`` start at 0
|
||||
and are backfilled during shreddit enrichment (see reddit_shreddit.py). Output
|
||||
dicts match the normalized shape emitted by ``reddit_public._parse_posts`` so
|
||||
downstream code (pipeline, renderer) is unaffected.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import quote_plus
|
||||
|
||||
from . import http
|
||||
from .relevance import token_overlap_relevance
|
||||
|
||||
ATOM = "{http://www.w3.org/2005/Atom}"
|
||||
|
||||
# Mirror reddit_public depth-aware limits so the two free paths behave alike.
|
||||
DEPTH_LIMITS = {
|
||||
"quick": 10,
|
||||
"default": 25,
|
||||
"deep": 50,
|
||||
}
|
||||
|
||||
# Listing sorts pulled per subreddit (in addition to search), for volume.
|
||||
LISTING_SORTS = {
|
||||
"quick": ["top"],
|
||||
"default": ["top", "hot"],
|
||||
"deep": ["top", "hot", "new"],
|
||||
}
|
||||
|
||||
MAX_WORKERS = 4
|
||||
FEED_TIMEOUT = 15
|
||||
|
||||
|
||||
def _log(msg: str) -> None:
|
||||
sys.stderr.write(f"[RedditRSS] {msg}\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def _iso_to_date(value: Optional[str]) -> Optional[str]:
|
||||
"""Parse an ISO-8601 timestamp (e.g. 2026-05-20T18:48:31+00:00) to YYYY-MM-DD."""
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip())
|
||||
return dt.date().isoformat()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _iso_to_epoch(value: Optional[str]) -> Optional[float]:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(value.strip())
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt.timestamp()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _subreddit_from(category: str, url: str) -> str:
|
||||
"""Derive subreddit name from the entry category or, failing that, the URL."""
|
||||
if category:
|
||||
return category
|
||||
# URL form: https://www.reddit.com/r/{sub}/comments/{id}/...
|
||||
parts = url.split("/r/", 1)
|
||||
if len(parts) == 2:
|
||||
return parts[1].split("/", 1)[0]
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed(xml_text: str, query: str = "") -> List[Dict[str, Any]]:
|
||||
"""Parse an Atom feed string into normalized post dicts. Never raises."""
|
||||
if not xml_text:
|
||||
return []
|
||||
try:
|
||||
root = ET.fromstring(xml_text)
|
||||
except ET.ParseError as e:
|
||||
_log(f"feed parse error: {e}")
|
||||
return []
|
||||
|
||||
posts: List[Dict[str, Any]] = []
|
||||
for entry in root.iter(f"{ATOM}entry"):
|
||||
link_el = entry.find(f"{ATOM}link")
|
||||
url = link_el.get("href", "").strip() if link_el is not None else ""
|
||||
if not url or "/comments/" not in url:
|
||||
continue
|
||||
|
||||
title_el = entry.find(f"{ATOM}title")
|
||||
title = (title_el.text or "").strip() if title_el is not None else ""
|
||||
|
||||
author = ""
|
||||
author_el = entry.find(f"{ATOM}author/{ATOM}name")
|
||||
if author_el is not None and author_el.text:
|
||||
author = author_el.text.strip().removeprefix("/u/").removeprefix("u/")
|
||||
if author in ("[deleted]", "[removed]", ""):
|
||||
author = "[deleted]"
|
||||
|
||||
cat_el = entry.find(f"{ATOM}category")
|
||||
category = cat_el.get("term", "").strip() if cat_el is not None else ""
|
||||
subreddit = _subreddit_from(category, url)
|
||||
|
||||
updated_el = entry.find(f"{ATOM}updated")
|
||||
updated = (updated_el.text or "").strip() if updated_el is not None else ""
|
||||
|
||||
content_el = entry.find(f"{ATOM}content")
|
||||
selftext = ""
|
||||
if content_el is not None and content_el.text:
|
||||
# Strip the simplest HTML; renderer only needs an excerpt.
|
||||
import re as _re
|
||||
selftext = _re.sub(r"<[^>]+>", " ", content_el.text)
|
||||
selftext = _re.sub(r"\s+", " ", selftext).strip()[:500]
|
||||
|
||||
relevance = round(token_overlap_relevance(query, title), 3) if query else 0.0
|
||||
|
||||
posts.append({
|
||||
"id": "", # assigned after dedup
|
||||
"title": title,
|
||||
"url": url,
|
||||
"score": 0, # backfilled by shreddit enrichment
|
||||
"num_comments": 0, # backfilled by shreddit enrichment
|
||||
"subreddit": subreddit,
|
||||
"created_utc": _iso_to_epoch(updated),
|
||||
"author": author,
|
||||
"selftext": selftext,
|
||||
"date": _iso_to_date(updated),
|
||||
"engagement": {
|
||||
"score": 0,
|
||||
"num_comments": 0,
|
||||
"upvote_ratio": None,
|
||||
},
|
||||
"relevance": relevance,
|
||||
"why_relevant": "Reddit RSS",
|
||||
"metadata": {},
|
||||
})
|
||||
|
||||
return posts
|
||||
|
||||
|
||||
def _build_urls(query: str, depth: str, subreddits: Optional[List[str]]) -> List[str]:
|
||||
"""Build the keyless RSS feed URLs to fan out across."""
|
||||
q = quote_plus(query)
|
||||
urls: List[str] = [
|
||||
f"https://www.reddit.com/search.rss?q={q}&sort=relevance&t=month"
|
||||
]
|
||||
for raw_sub in (subreddits or []):
|
||||
sub = raw_sub.removeprefix("r/").strip()
|
||||
if not sub:
|
||||
continue
|
||||
urls.append(
|
||||
f"https://www.reddit.com/r/{sub}/search.rss"
|
||||
f"?q={q}&restrict_sr=on&sort=relevance&t=month"
|
||||
)
|
||||
for sort in LISTING_SORTS.get(depth, LISTING_SORTS["default"]):
|
||||
urls.append(f"https://www.reddit.com/r/{sub}/{sort}.rss?t=month")
|
||||
return urls
|
||||
|
||||
|
||||
def _fetch_feed(url: str, query: str) -> List[Dict[str, Any]]:
|
||||
"""Fetch and parse one feed. Never raises."""
|
||||
try:
|
||||
text = http.get_text(url, timeout=FEED_TIMEOUT, accept="application/atom+xml")
|
||||
return _parse_feed(text, query) if text else []
|
||||
except Exception as e: # defensive: a single bad feed must not sink the run
|
||||
_log(f"feed fetch failed for {url}: {e}")
|
||||
return []
|
||||
|
||||
|
||||
def search_rss(
|
||||
query: str,
|
||||
depth: str = "default",
|
||||
subreddits: Optional[List[str]] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Discover Reddit posts for a query via keyless RSS feeds.
|
||||
|
||||
Args:
|
||||
query: Search query string
|
||||
depth: 'quick', 'default', or 'deep' — controls result limit and feeds
|
||||
subreddits: Optional pre-resolved subreddit names (without r/) to target
|
||||
|
||||
Returns:
|
||||
List of normalized post dicts (deduped by URL, capped by depth),
|
||||
with placeholder scores to be backfilled during enrichment.
|
||||
Empty list on any failure.
|
||||
"""
|
||||
limit = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
|
||||
urls = _build_urls(query, depth, subreddits)
|
||||
|
||||
all_posts: List[Dict[str, Any]] = []
|
||||
workers = min(MAX_WORKERS, len(urls)) or 1
|
||||
with ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
futures = {executor.submit(_fetch_feed, url, query): url for url in urls}
|
||||
for future in futures:
|
||||
try:
|
||||
all_posts.extend(future.result(timeout=FEED_TIMEOUT + 5))
|
||||
except (Exception, FuturesTimeoutError) as e:
|
||||
_log(f"feed future failed: {e}")
|
||||
|
||||
# Dedupe by URL (first occurrence wins).
|
||||
seen: set = set()
|
||||
unique: List[Dict[str, Any]] = []
|
||||
for post in all_posts:
|
||||
if post["url"] not in seen:
|
||||
seen.add(post["url"])
|
||||
unique.append(post)
|
||||
|
||||
for i, post in enumerate(unique):
|
||||
post["id"] = f"R{i + 1}"
|
||||
|
||||
return unique[:limit]
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Keyless Reddit comment enrichment via shreddit /svc endpoints.
|
||||
|
||||
Reddit's ``{thread}.json`` endpoint now returns HTTP 403. The shreddit partial
|
||||
endpoint ``/svc/shreddit/comments/r/{sub}/t3_{id}`` still serves HTTP 200 HTML
|
||||
with no API key, embedding each comment as a ``<shreddit-comment>`` custom
|
||||
element whose start-tag attributes carry ``score`` / ``author`` / ``created`` /
|
||||
``permalink``, and whose body lives in a ``<div id="{thingId}-post-rtjson-content">``
|
||||
block. This module parses that markup into top comments, matching the
|
||||
``top_comments`` / ``comment_insights`` shape produced by ``reddit_enrich`` so
|
||||
the renderer is unaffected.
|
||||
|
||||
Limitation: the comments endpoint carries the real comment count
|
||||
(``total-comments``) but not the post's upvote score, so post-level ``score``
|
||||
cannot be recovered keylessly here (ScrapeCreators backup still provides it).
|
||||
"""
|
||||
|
||||
import html as _html
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from . import http
|
||||
from . import reddit_enrich
|
||||
|
||||
# Up to N posts enriched per run, by depth (mirrors reddit_public.ENRICH_LIMITS).
|
||||
ENRICH_LIMITS = {
|
||||
"quick": 3,
|
||||
"default": 5,
|
||||
"deep": 8,
|
||||
}
|
||||
|
||||
# Max comments returned per post (independent of how many posts get enriched).
|
||||
MAX_COMMENTS = 10
|
||||
|
||||
SVC_TIMEOUT = 12
|
||||
|
||||
# Match the exact <shreddit-comment> element start tag, not <shreddit-comment-tree>
|
||||
# or <shreddit-comment-tree-stats> (lookahead requires whitespace or '>').
|
||||
_COMMENT_START = re.compile(r"<shreddit-comment(?=[\s>])[^>]*>")
|
||||
_TOTAL_COMMENTS = re.compile(r'total-comments="(\d+)"')
|
||||
_PARA = re.compile(r"<p[^>]*>(.*?)</p>", re.S)
|
||||
_TAG = re.compile(r"<[^>]+>")
|
||||
_WS = re.compile(r"\s+")
|
||||
_NEXT_RTJSON = re.compile(r'id="t1_[A-Za-z0-9]+-(?:comment|post)-rtjson-content"')
|
||||
|
||||
|
||||
def _log(msg: str) -> None:
|
||||
sys.stderr.write(f"[RedditShreddit] {msg}\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def extract_post_ref(url: str) -> Optional[tuple]:
|
||||
"""Return (subreddit, post_id) from a Reddit thread URL, or None."""
|
||||
m = re.search(r"/r/([^/]+)/comments/([A-Za-z0-9]+)", url or "")
|
||||
if not m:
|
||||
return None
|
||||
return m.group(1), m.group(2)
|
||||
|
||||
|
||||
def _svc_url(subreddit: str, post_id: str) -> str:
|
||||
# sort=top guarantees Reddit front-loads the highest-scored comments on the
|
||||
# first page, so the true top comments are captured even on huge threads
|
||||
# (we still re-sort by score locally as a backstop).
|
||||
return (
|
||||
f"https://www.reddit.com/svc/shreddit/comments/r/{subreddit}/t3_{post_id}"
|
||||
f"?sort=top"
|
||||
)
|
||||
|
||||
|
||||
def _attr(tag: str, name: str) -> str:
|
||||
m = re.search(rf'\b{name}="([^"]*)"', tag)
|
||||
return _html.unescape(m.group(1)) if m else ""
|
||||
|
||||
|
||||
def _iso_to_date(value: str) -> Optional[str]:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(value.strip()).date().isoformat()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _body_for(html_text: str, thing_id: str) -> str:
|
||||
"""Extract a comment's text body, anchored on its unique thingId.
|
||||
|
||||
The body div id embeds the comment's thingId, so this assigns body→comment
|
||||
correctly even for nested replies. The slice is bounded by the next
|
||||
comment's rtjson anchor to avoid swallowing child-comment text.
|
||||
"""
|
||||
if not thing_id:
|
||||
return ""
|
||||
anchor = f'id="{thing_id}-post-rtjson-content"'
|
||||
idx = html_text.find(anchor)
|
||||
if idx == -1:
|
||||
return ""
|
||||
window = html_text[idx + len(anchor): idx + len(anchor) + 8000]
|
||||
nxt = _NEXT_RTJSON.search(window)
|
||||
if nxt:
|
||||
window = window[: nxt.start()]
|
||||
paras = _PARA.findall(window)
|
||||
if not paras:
|
||||
return ""
|
||||
text = " ".join(_TAG.sub("", p) for p in paras)
|
||||
return _WS.sub(" ", _html.unescape(text)).strip()
|
||||
|
||||
|
||||
def parse_comments(html_text: str, limit: int = MAX_COMMENTS) -> List[Dict[str, Any]]:
|
||||
"""Parse <shreddit-comment> elements into scored comment dicts (sorted desc)."""
|
||||
comments: List[Dict[str, Any]] = []
|
||||
for m in _COMMENT_START.finditer(html_text or ""):
|
||||
tag = m.group(0)
|
||||
author = _attr(tag, "author") or "[deleted]"
|
||||
if author in ("[deleted]", "[removed]"):
|
||||
continue
|
||||
thing_id = _attr(tag, "thingId")
|
||||
body = _body_for(html_text, thing_id)
|
||||
if not body or body in ("[deleted]", "[removed]"):
|
||||
continue
|
||||
try:
|
||||
score = int(_attr(tag, "score") or 0)
|
||||
except ValueError:
|
||||
score = 0
|
||||
permalink = _attr(tag, "permalink")
|
||||
comments.append({
|
||||
"score": score,
|
||||
"author": author,
|
||||
"body": body[:300],
|
||||
"excerpt": body[:200],
|
||||
"permalink": permalink,
|
||||
"date": _iso_to_date(_attr(tag, "created")),
|
||||
"url": f"https://reddit.com{permalink}" if permalink else "",
|
||||
})
|
||||
|
||||
comments.sort(key=lambda c: c.get("score", 0), reverse=True)
|
||||
return comments[:limit]
|
||||
|
||||
|
||||
def _total_comments(html_text: str) -> Optional[int]:
|
||||
m = _TOTAL_COMMENTS.search(html_text or "")
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def fetch_comments(
|
||||
post_url: str,
|
||||
timeout: int = SVC_TIMEOUT,
|
||||
) -> Dict[str, Any]:
|
||||
"""Fetch and parse top comments for a Reddit post via the shreddit endpoint.
|
||||
|
||||
Args:
|
||||
post_url: Reddit thread URL (…/r/{sub}/comments/{id}/…)
|
||||
timeout: HTTP timeout in seconds
|
||||
|
||||
Returns:
|
||||
Dict with 'top_comments' (list, reddit_enrich shape), 'comment_insights'
|
||||
(list[str]), and 'num_comments' (int or None). Empty/None on any
|
||||
failure — never raises, so the caller can fall through to SC backup.
|
||||
"""
|
||||
ref = extract_post_ref(post_url)
|
||||
if not ref:
|
||||
return {"top_comments": [], "comment_insights": [], "num_comments": None}
|
||||
sub, post_id = ref
|
||||
|
||||
html_text = http.get_text(_svc_url(sub, post_id), timeout=timeout, accept="text/html")
|
||||
if not html_text:
|
||||
return {"top_comments": [], "comment_insights": [], "num_comments": None}
|
||||
|
||||
comments = parse_comments(html_text, limit=MAX_COMMENTS)
|
||||
insights = reddit_enrich.extract_comment_insights(comments)
|
||||
return {
|
||||
"top_comments": [
|
||||
{
|
||||
"score": c["score"],
|
||||
"date": c["date"],
|
||||
"author": c["author"],
|
||||
"excerpt": c["excerpt"],
|
||||
"url": c["url"],
|
||||
}
|
||||
for c in comments
|
||||
],
|
||||
"comment_insights": insights,
|
||||
"num_comments": _total_comments(html_text),
|
||||
}
|
||||
Reference in New Issue
Block a user