* test(reddit): add live RSS + shreddit comment fixtures
Captured from reddit.com on 2026-05-29 (search.rss listing + the
/svc/shreddit/comments partial), trimmed to a representative subset plus
two synthetic edge cases (deleted author, negative score) for offline
parser tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(http): add keyless get_text helper
Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or
network failure so tiered callers fall through cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): keyless RSS discovery (search.rss + listing feeds)
Replaces the now-403 search.json with keyless Atom feeds, normalized to the
existing reddit_public post shape. Scores are placeholder zeros, backfilled
during shreddit enrichment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): keyless shreddit comment scraper
Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id}
(score/author/created/permalink + thingId-anchored body) into top comments,
matching reddit_enrich output. Replaces the dead {thread}.json enrichment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): tiered keyless orchestrator
Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery ->
Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup
still engages when every keyless tier is empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): route free path through keyless pipeline (.json is dead)
search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py
and other callers need no change. Removes the dead .json enrichment helpers;
search/_parse_posts remain as the demoted Tier 0 attempt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): request sort=top so true top comments land on page 1
Guarantees the highest-scored comments are captured even on large threads,
independent of Reddit's default comment sort. Local score re-sort remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): recover post upvote scores via keyless listing partials
The shreddit community-more-posts partial server-renders each post's score
and comment count (works for normal users, not IP-gated), unlike RSS or the
comments endpoint. Use it as a scored discovery source and to backfill scores
onto RSS-discovered posts (subreddits derived from results when not provided).
Ranking now uses real upvote score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): listings backfill scores only on bare queries, not discovery
Caught running the full pipeline on a bare topic: deriving subreddits from
noisy RSS results and merging their top/hot listings flooded results with
high-upvote off-topic posts. Now derived-subreddit listings are used only to
backfill scores onto keyword-matched RSS posts; listing cards are merged as
discovery only when the caller explicitly provides subreddits (on-topic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reddit's public JSON endpoint returns 403 to requests carrying the
generic User-Agent and minimal header set urllib defaults to, while
matching curl requests succeed. Switch to a current-Chrome User-Agent
and add Accept-Language / Accept-Encoding / Connection headers so the
fingerprint matches a normal browser. Reddit now serves gzip when
Accept-Encoding includes it, so decompress the body before JSON parse.
Update the user-agent assertion in tests/test_reddit_public.py to match
the new browser-like string. Closes#199.
Co-authored-by: Franco Carballar <francocarballar@gmail.com>
str.lstrip("r/") treats its argument as a character set, stripping
leading r and / repeatedly. Subreddits starting with 'r' (e.g. r/robotics,
r/ruby) were silently mangled to 'obotics' / 'uby'. Replace with
str.removeprefix("r/") at all four call sites. Python 3.9+ pattern is
safe here — project requires 3.12. Closes#288.
Co-authored-by: Alex Key <alexanderkey0508@gmail.com>
Matt Van Horn