* test(reddit): add live RSS + shreddit comment fixtures
Captured from reddit.com on 2026-05-29 (search.rss listing + the
/svc/shreddit/comments partial), trimmed to a representative subset plus
two synthetic edge cases (deleted author, negative score) for offline
parser tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(http): add keyless get_text helper
Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or
network failure so tiered callers fall through cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): keyless RSS discovery (search.rss + listing feeds)
Replaces the now-403 search.json with keyless Atom feeds, normalized to the
existing reddit_public post shape. Scores are placeholder zeros, backfilled
during shreddit enrichment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): keyless shreddit comment scraper
Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id}
(score/author/created/permalink + thingId-anchored body) into top comments,
matching reddit_enrich output. Replaces the dead {thread}.json enrichment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): tiered keyless orchestrator
Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery ->
Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup
still engages when every keyless tier is empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): route free path through keyless pipeline (.json is dead)
search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py
and other callers need no change. Removes the dead .json enrichment helpers;
search/_parse_posts remain as the demoted Tier 0 attempt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): request sort=top so true top comments land on page 1
Guarantees the highest-scored comments are captured even on large threads,
independent of Reddit's default comment sort. Local score re-sort remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(reddit): recover post upvote scores via keyless listing partials
The shreddit community-more-posts partial server-renders each post's score
and comment count (works for normal users, not IP-gated), unlike RSS or the
comments endpoint. Use it as a scored discovery source and to backfill scores
onto RSS-discovered posts (subreddits derived from results when not provided).
Ranking now uses real upvote score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(reddit): listings backfill scores only on bare queries, not discovery
Caught running the full pipeline on a bare topic: deriving subreddits from
noisy RSS results and merging their top/hot listings flooded results with
high-upvote off-topic posts. Now derived-subreddit listings are used only to
backfill scores onto keyword-matched RSS posts; listing cards are merged as
discovery only when the caller explicitly provides subreddits (on-topic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
evaluate_search_quality.py and e2e_comparison.py both reference
fixtures/eval_topics.json with hardcoded fallbacks. Supply the
actual fixture: 8 topics spanning all intent types, selected via
MMR dispersion across domains (tech, health, sports, finance,
consumer products).
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.
- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _compute_text_similarity() now checks outcome names with bidirectional
substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polymarket results now rank by text similarity, volume, liquidity, price
movement, and competitive score instead of API return position. Also fixes
pagination (DEPTH_CONFIG now controls page count, not a no-op limit param)
and caps results after re-ranking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Research topics across Reddit + X from the last 30 days using
OpenAI and xAI APIs. Features:
- Auto model selection (GPT-5.x, Grok-3)
- Popularity-aware scoring (relevance + recency + engagement)
- Reddit thread enrichment with real metrics
- Near-duplicate detection
- Multiple emit modes (compact, json, context, path)
- 24h caching with --refresh bypass
- NUX for API key setup
- 87 passing unit tests
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Matt Van Horn