From 30b973f62ece58e9607de2cb4ff323f5f2d85191 Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Thu, 5 Mar 2026 17:24:41 -0800 Subject: [PATCH] docs: add Reddit ScrapeCreators v2 improvements plan Three focused improvements based on 5 full-pipeline beta tests: 1. Elevate top Reddit comments in scoring and rendering 2. Improve subreddit discovery heuristic for ambiguous queries 3. Make ScrapeCreators the default recommended Reddit method Co-Authored-By: Claude Opus 4.6 --- ...dit-scrapecreators-v2-improvements-plan.md | 255 ++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 docs/plans/2026-03-05-feat-reddit-scrapecreators-v2-improvements-plan.md diff --git a/docs/plans/2026-03-05-feat-reddit-scrapecreators-v2-improvements-plan.md b/docs/plans/2026-03-05-feat-reddit-scrapecreators-v2-improvements-plan.md new file mode 100644 index 0000000..acb8568 --- /dev/null +++ b/docs/plans/2026-03-05-feat-reddit-scrapecreators-v2-improvements-plan.md @@ -0,0 +1,255 @@ +# feat: Reddit ScrapeCreators v2 — Improvements from Beta Testing + +**Date:** 2026-03-05 +**Type:** Enhancement +**Version:** v2.9 → v2.9.1-beta (or v3.0-beta if shipping to public) +**Branch:** `feat/reddit-scrapecreators` (continue existing branch) + +--- + +## Summary + +Three focused improvements to the Reddit ScrapeCreators integration based on 5 full-pipeline tests ("Claude Code skills", "Kanye West", "Anthropic odds", "best rap songs lately", "Nano Banana Pro prompting"): + +1. **Elevate top Reddit comments** — give weight to the wittiest/highest-voted comment in scoring and rendering +2. **Improve subreddit discovery** — tune heuristic so ambiguous queries find discussion subs, not utility subs +3. **Make ScrapeCreators the default recommended Reddit method** — update onboarding, SKILL.md metadata, and env.py messaging + +--- + +## Problem Statement + +### 1. Comments are undervalued +- ScrapeCreators returns real comment data with scores, but top comments only appear as `Insights:` text under each Reddit item +- The top comment (often the funniest/cleverest reply) gets no special treatment — it's just one of 3 comment excerpts +- Reddit's value IS the comments — upvoted replies are the distilled crowd wisdom +- Currently `comment_insights` are truncated at 150 chars and only 3 are shown per item in compact output +- No scoring bonus for posts that have high-quality comment threads + +### 2. Subreddit discovery picks wrong subs for ambiguous queries +- "best rap songs lately" discovered `r/NameThatSong` and `r/findthatsong` (utility subs for identifying songs) instead of discussion subs like `r/hiphopheads` or `r/rap` +- "Kanye West" picked `r/ConcertsIndia_` as second sub — tangential at best +- The current heuristic is pure frequency count on `subreddit` field from global results, with no relevance weighting +- Utility/meta subs often dominate because the same query matches many "help me find X" posts + +### 3. Onboarding still suggests OpenAI as the primary Reddit method +- SKILL.md metadata says `primaryEnv: OPENAI_API_KEY` and `requires.env: [OPENAI_API_KEY]` +- The web-only mode banner mentions "OPENAI_API_KEY or codex login → Reddit threads" +- `env.py` error messages direct users to OpenAI for Reddit access +- ScrapeCreators is cheaper ($0.012 vs $0.03-0.10), faster (17s vs 60-90s), returns real data, and shares a key with TikTok + Instagram +- New users should be told: "Get a SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, all three)" + +--- + +## Implementation Plan + +### Task 1: Elevate Top Comments in Scoring and Rendering + +**Goal:** Give Reddit posts a scoring bonus when they have highly-engaged comment threads, and render the #1 comment with special treatment. + +**Files to modify:** +- `scripts/lib/reddit.py` — enrich with `top_comment_score` metadata +- `scripts/lib/score.py` — add comment quality bonus to Reddit scoring +- `scripts/lib/render.py` — render top comment with special formatting +- `scripts/lib/schema.py` — add `top_comment_excerpt` field to RedditItem (optional, may just use existing `top_comments[0]`) + +#### 1a. Comment enrichment improvements (`scripts/lib/reddit.py`) + +- [ ] In `enrich_with_comments()`, after sorting comments by score, tag the item with: + - `top_comment_excerpt`: The highest-scored comment's body (up to 200 chars) + - `top_comment_score`: The upvote count of the #1 comment + - `top_comment_author`: Author of the #1 comment +- [ ] Increase comment excerpt length from 300 → 400 chars for top comment only (funny/clever comments need more room) +- [ ] Increase `comment_insights` limit from 7 → 10 (we have the data, show it) +- [ ] For posts with enriched comments, store the comment count ratio: `top_comment_score / post_score` — a high ratio means the comment outshines the post (Reddit gold) + +#### 1b. Scoring bonus for comment quality (`scripts/lib/score.py`) + +- [ ] In `compute_reddit_engagement_raw()`, add a comment quality signal: + - Current formula: `0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)` + - New formula: `0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)` + - This gives a ~10% weight to comment quality, slightly reducing post score and comment count weights + - Posts where the community engaged deeply (high top-comment score) rank higher +- [ ] Need to pass `top_comment_score` through the engagement data — either: + - Option A: Add `top_comment_score` to `schema.Engagement` (cleanest) + - Option B: Read from `item.top_comments[0].score` during scoring (no schema change) + - **Recommend Option B** to avoid schema bloat — scoring can peek at `top_comments` + +#### 1c. Render top comment prominently (`scripts/lib/render.py`) + +- [ ] In `render_compact()` Reddit section, after the `Insights:` block, add a "Top Comment:" line for items that have top_comments: + ``` + **R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt] + Claude Code creator: In the next version, introducing two new skills + https://www.reddit.com/r/ClaudeAI/comments/... + *Reddit global search* + 💬 Top comment (247 upvotes): "So are they /batch migrating to Rust? :)" + Insights: + - TL;DR generated automatically after 50 comments... + - He's /batch migrating code daily?.. + ``` +- [ ] Only show `💬 Top comment` for items where `top_comments[0].score >= 10` (skip low-engagement comments) +- [ ] Truncate at 200 chars with `...` if needed +- [ ] Also update `render_full_report()` to include the top comment prominently + +#### 1d. Update SKILL.md synthesis instructions + +- [ ] In the "Judge Agent: Synthesize All Sources" section, add guidance: + ``` + 5b. For Reddit: Pay special attention to top comments — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes, quote it directly in your synthesis. Reddit's value is in the comments. + ``` +- [ ] In the citation priority list, add: "When citing Reddit, prefer quoting top comments over just the thread title" + +--- + +### Task 2: Improve Subreddit Discovery Heuristic + +**Goal:** Find topical discussion subs rather than utility/meta subs. + +**Files to modify:** +- `scripts/lib/reddit.py` — improve `discover_subreddits()` logic + +#### 2a. Add relevance-weighted subreddit scoring + +- [ ] Replace pure frequency count with a weighted score: + ```python + def discover_subreddits(results, topic, max_subs=5): + core = _extract_core_subject(topic) + core_words = set(core.lower().split()) + + scores = Counter() + for post in results: + sub = post.get("subreddit", "") + if not sub: + continue + + # Base: frequency count + base = 1.0 + + # Bonus: subreddit name contains a core topic word + sub_lower = sub.lower() + if any(w in sub_lower for w in core_words if len(w) > 2): + base += 2.0 + + # Penalty: known utility/meta subreddits + if sub_lower in UTILITY_SUBS: + base *= 0.3 + + # Bonus: post engagement (high-engagement posts = better sub) + ups = post.get("ups") or post.get("score", 0) + if ups > 100: + base += 0.5 + + scores[sub] += base + + return [sub for sub, _ in scores.most_common(max_subs)] + ``` + +#### 2b. Define utility/meta subreddit blocklist + +- [ ] Add a small set of subs that are "find X for me" or "identify X" rather than discussion: + ```python + UTILITY_SUBS = frozenset({ + 'namethatsong', 'findthatsong', 'tipofmytongue', + 'whatisthissong', 'helpmefind', 'whatisthisthing', + 'whatsthissong', 'findareddit', 'subredditdrama', + }) + ``` +- [ ] Keep this small and focused — don't over-filter. Only penalty (0.3x), not ban. + +#### 2c. Try secondary query for subreddit discovery + +- [ ] If the first global search returns <3 unique subreddits above threshold, run a second global search with just `{core subject}` (stripped even further) to cast a wider net for subreddit frequencies +- [ ] This helps niche topics where the full query is too specific + +--- + +### Task 3: Make ScrapeCreators the Default Reddit Method + +**Goal:** New users should be guided to ScrapeCreators first, not OpenAI. + +**Files to modify:** +- `SKILL.md` — metadata section, onboarding banner, security section +- `scripts/lib/env.py` — error messages and missing key guidance +- `scripts/lib/render.py` — web-only mode banner + +#### 3a. Update SKILL.md metadata + +- [ ] Change `primaryEnv: OPENAI_API_KEY` → `primaryEnv: SCRAPECREATORS_API_KEY` +- [ ] Change `requires.env: [OPENAI_API_KEY]` → `requires.env: [SCRAPECREATORS_API_KEY]` +- [ ] Keep OPENAI_API_KEY mentioned but as optional/legacy + +#### 3b. Update web-only mode banner (`scripts/lib/render.py`) + +- [ ] Change the current banner: + ``` + - `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments + ``` + To: + ``` + - `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views + - `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost) + ``` + +#### 3c. Update env.py messaging + +- [ ] In `get_missing_keys()`, when Reddit is missing, suggest ScrapeCreators first: + - Current: returns `'reddit'` which triggers "Add OPENAI_API_KEY or run codex login" in SKILL.md + - Add a helper: `get_setup_hint(missing)` that returns: + - For 'reddit': `"Add SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, ~$0.002/search)"` + - For 'x': `"Add XAI_API_KEY for X posts"` + - For 'all': `"Add SCRAPECREATORS_API_KEY (Reddit+TikTok+Instagram) and XAI_API_KEY (X)"` + +#### 3d. Update Security & Permissions section in SKILL.md + +- [ ] Add ScrapeCreators Reddit to the security section: + ``` + - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit, TikTok, and Instagram search (requires SCRAPECREATORS_API_KEY) + ``` +- [ ] Move "Sends search queries to OpenAI's Responses API for Reddit discovery" to a "Legacy:" subsection +- [ ] Update "Reddit" description in `allowed-tools` or tags if needed + +#### 3e. Update render.py coverage note + +- [ ] In `render_compact()`, the coverage note for `reddit-only` currently says "Add an xAI key" +- [ ] When ScrapeCreators is the active Reddit source, no need to mention OpenAI at all + +--- + +## Acceptance Criteria + +- [ ] Top Reddit comment is rendered with `💬` prefix and upvote count for enriched posts +- [ ] Posts with high top-comment scores rank slightly higher (visible in score differences) +- [ ] "best rap songs lately" discovers at least one discussion sub (r/hiphopheads, r/rap, r/Music, etc.) instead of only utility subs +- [ ] SKILL.md `primaryEnv` is `SCRAPECREATORS_API_KEY` +- [ ] Web-only mode banner recommends ScrapeCreators first +- [ ] All 5 test topics still pass (run same tests as before) +- [ ] No regression in OpenAI fallback path + +--- + +## Files Changed (Summary) + +| File | Change | +|------|--------| +| `scripts/lib/reddit.py` | Improve `discover_subreddits()` with relevance weighting, add utility sub penalties, enhance `enrich_with_comments()` top comment metadata | +| `scripts/lib/score.py` | Add 10% comment quality weight to Reddit engagement formula | +| `scripts/lib/render.py` | Add `💬 Top comment` line to compact output, update web-only banner | +| `scripts/lib/env.py` | Add `get_setup_hint()`, update missing key messaging | +| `SKILL.md` | Change `primaryEnv`, update onboarding banner, add comment synthesis guidance, update security section | + +--- + +## Cost Impact + +No cost increase. Same number of API calls per search. The changes are all in local logic (scoring, rendering, discovery heuristic). + +--- + +## Testing Plan + +1. Re-run the same 5 test topics from beta testing +2. Verify top comments appear with `💬` in output +3. Verify "best rap songs lately" discovers at least one discussion subreddit +4. Verify `--diagnose` output recommends ScrapeCreators +5. Verify OpenAI fallback still works (unset SCRAPECREATORS_API_KEY, set OPENAI_API_KEY)