Compare commits
7 Commits
v2.8.0
...
temp-tweet
| Author | SHA1 | Date | |
|---|---|---|---|
 Matt Van Horn
|
69aa98f3f8 | ||
|
Matt Van Horn
|
6edb813bea | ||
|
Matt Van Horn
|
4d35b53eab | ||
|
Matt Van Horn
|
2247800003 | ||
|
Matt Van Horn
|
7048fe7b83 | ||
|
Matt Van Horn
|
30b973f62e | ||
|
Matt Van Horn
|
09b09946c0 |
@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [2.9.0] - 2026-03-05
|
||||||
|
|
||||||
|
### Highlights
|
||||||
|
|
||||||
|
ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
|
||||||
|
- Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
|
||||||
|
- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
|
||||||
|
- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
|
||||||
|
- Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
|
||||||
|
- Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
|
||||||
|
- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
|
||||||
|
- SKILL.md synthesis instructions updated to emphasize quoting top comments
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
|
||||||
|
- Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
|
||||||
|
|
||||||
## [2.8.0] - 2026-03-04
|
## [2.8.0] - 2026-03-04
|
||||||
|
|
||||||
### Highlights
|
### Highlights
|
||||||
@@ -88,6 +114,7 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
|
|||||||
|
|
||||||
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
|
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
|
||||||
|
|
||||||
|
[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
|
||||||
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
|
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
|
||||||
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
|
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
|
||||||
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
|
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
|
||||||
|
|||||||
@@ -1,10 +1,14 @@
|
|||||||
# /last30days v2.8
|
# /last30days v2.9
|
||||||
|
|
||||||
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
|
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
|
||||||
|
|
||||||
|
**New in v2.9 — ScrapeCreators Reddit + Top Comments + Smart Discovery:**
|
||||||
|
|
||||||
|
Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default — one `SCRAPECREATORS_API_KEY` covers Reddit, TikTok, and Instagram (3 sources, 1 key). Smart subreddit discovery finds the right communities automatically, and top comments are elevated with a 10% scoring weight and `💬` display with upvote counts. [Details below.](#whats-new-in-v29)
|
||||||
|
|
||||||
**New in v2.8 — Instagram Reels + ScrapeCreators:**
|
**New in v2.8 — Instagram Reels + ScrapeCreators:**
|
||||||
|
|
||||||
Instagram Reels is now the 8th signal source. TikTok and Instagram both run on [ScrapeCreators](https://scrapecreators.com) — one API key covers both. Search any topic and get trending Reels with views, likes, spoken-word transcripts, and hashtags. [Details below.](#whats-new-in-v28)
|
Instagram Reels is now the 8th signal source. TikTok and Instagram both run on ScrapeCreators — one API key covers both. [Details below.](#whats-new-in-v28)
|
||||||
|
|
||||||
**New in V2.5 - dramatically better results:**
|
**New in V2.5 - dramatically better results:**
|
||||||
|
|
||||||
@@ -31,9 +35,9 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
|
|||||||
# Add your API keys (optional if signed in to Codex)
|
# Add your API keys (optional if signed in to Codex)
|
||||||
mkdir -p ~/.config/last30days
|
mkdir -p ~/.config/last30days
|
||||||
cat > ~/.config/last30days/.env << 'EOF'
|
cat > ~/.config/last30days/.env << 'EOF'
|
||||||
OPENAI_API_KEY=sk-... # optional if using `codex login`
|
SCRAPECREATORS_API_KEY=... # Reddit + TikTok + Instagram (one key, all three) — scrapecreators.com
|
||||||
XAI_API_KEY=xai-... # optional - cookie auth is default for X search
|
OPENAI_API_KEY=sk-... # optional — legacy Reddit fallback if using `codex login`
|
||||||
SCRAPECREATORS_API_KEY=... # optional - for TikTok + Instagram (scrapecreators.com)
|
XAI_API_KEY=xai-... # optional — cookie auth is default for X search
|
||||||
EOF
|
EOF
|
||||||
chmod 600 ~/.config/last30days/.env
|
chmod 600 ~/.config/last30days/.env
|
||||||
```
|
```
|
||||||
@@ -937,6 +941,50 @@ If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1)
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## What's New in v2.9
|
||||||
|
|
||||||
|
### ScrapeCreators Reddit as default
|
||||||
|
|
||||||
|
Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
|
||||||
|
```
|
||||||
|
|
||||||
|
### Smart subreddit discovery
|
||||||
|
|
||||||
|
Subreddit discovery now uses relevance-weighted scoring instead of pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.).
|
||||||
|
|
||||||
|
| Topic | Before (v2.8) | After (v2.9) |
|
||||||
|
|-------|---------------|--------------|
|
||||||
|
| Claude Code skills | Generic programming subs | r/ClaudeAI, r/ClaudeCode, r/openclaw |
|
||||||
|
| Kanye West | r/AskReddit, r/OutOfTheLoop | r/hiphopheads, r/Kanye, r/NFCWestMemeWar |
|
||||||
|
| Nano Banana Pro | r/techsupport, r/whatisthisthing | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
|
||||||
|
|
||||||
|
### Top comments elevated
|
||||||
|
|
||||||
|
Top comments now carry a 10% weight in the engagement scoring formula and are displayed prominently with `💬` and upvote counts:
|
||||||
|
|
||||||
|
```
|
||||||
|
**R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
|
||||||
|
Claude Code creator: In the next version, introducing two new skills
|
||||||
|
💬 Top comment (245 pts): "This is going to change how everyone works with Claude"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Updated scoring formula:** `0.50 × log1p(score) + 0.35 × log1p(comments) + 0.05 × (ratio×10) + 0.10 × log1p(top_comment_score)` (was 0.55/0.40/0.05).
|
||||||
|
|
||||||
|
### Beta test results
|
||||||
|
|
||||||
|
| Topic | Time | Threads | Discovered Subreddits |
|
||||||
|
|-------|------|---------|----------------------|
|
||||||
|
| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
|
||||||
|
| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
|
||||||
|
| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
|
||||||
|
| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
|
||||||
|
| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## What's New in v2.8
|
## What's New in v2.8
|
||||||
|
|
||||||
### Instagram Reels as a source
|
### Instagram Reels as a source
|
||||||
@@ -1096,13 +1144,13 @@ Thanks to the contributors who helped shape V2:
|
|||||||
|
|
||||||
| Destination | Data Sent | API Key Required |
|
| Destination | Data Sent | API Key Required |
|
||||||
|------------|-----------|-----------------|
|
|------------|-----------|-----------------|
|
||||||
| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
|
| `api.scrapecreators.com` | Search query (Reddit + TikTok + Instagram) | SCRAPECREATORS_API_KEY |
|
||||||
|
| `api.openai.com` | Search query (legacy Reddit fallback) | OPENAI_API_KEY |
|
||||||
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
|
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
|
||||||
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
|
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
|
||||||
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
|
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
|
||||||
| `hn.algolia.com` | Search query | None (public API) |
|
| `hn.algolia.com` | Search query | None (public API) |
|
||||||
| `gamma-api.polymarket.com` | Search query | None (public API) |
|
| `gamma-api.polymarket.com` | Search query | None (public API) |
|
||||||
| `api.scrapecreators.com` | Search query (TikTok + Instagram) | SCRAPECREATORS_API_KEY |
|
|
||||||
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
|
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
|
||||||
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
|
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
|
||||||
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
|
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: last30days
|
name: last30days
|
||||||
version: "2.8"
|
version: "2.9"
|
||||||
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
|
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
|
||||||
argument-hint: 'last30 AI video tools, last30 best project management tools'
|
argument-hint: 'last30 AI video tools, last30 best project management tools'
|
||||||
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
||||||
@@ -11,11 +11,11 @@ metadata:
|
|||||||
emoji: "📰"
|
emoji: "📰"
|
||||||
requires:
|
requires:
|
||||||
env:
|
env:
|
||||||
- OPENAI_API_KEY
|
- SCRAPECREATORS_API_KEY
|
||||||
bins:
|
bins:
|
||||||
- node
|
- node
|
||||||
- python3
|
- python3
|
||||||
primaryEnv: OPENAI_API_KEY
|
primaryEnv: SCRAPECREATORS_API_KEY
|
||||||
files:
|
files:
|
||||||
- "scripts/*"
|
- "scripts/*"
|
||||||
homepage: https://github.com/mvanhorn/last30days-skill
|
homepage: https://github.com/mvanhorn/last30days-skill
|
||||||
@@ -30,7 +30,7 @@ metadata:
|
|||||||
- prompts
|
- prompts
|
||||||
---
|
---
|
||||||
|
|
||||||
# last30days v2.8: Research Any Topic from the Last 30 Days
|
# last30days v2.9: Research Any Topic from the Last 30 Days
|
||||||
|
|
||||||
Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
|
Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
|
||||||
|
|
||||||
@@ -239,9 +239,10 @@ The Judge Agent must:
|
|||||||
2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
|
2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
|
||||||
3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
|
3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
|
||||||
4. Weight WebSearch sources LOWER (no engagement data)
|
4. Weight WebSearch sources LOWER (no engagement data)
|
||||||
4. Identify patterns that appear across ALL sources (strongest signals)
|
5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes (shown as `💬 Top comment (N upvotes)`), quote it directly in your synthesis. Reddit's value is in the comments.
|
||||||
5. Note any contradictions between sources
|
6. Identify patterns that appear across ALL sources (strongest signals)
|
||||||
6. Extract the top 3-5 actionable insights
|
7. Note any contradictions between sources
|
||||||
|
8. Extract the top 3-5 actionable insights
|
||||||
|
|
||||||
7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.
|
7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.
|
||||||
|
|
||||||
@@ -345,7 +346,7 @@ CITATION RULE: Cite sources sparingly to prove research is real.
|
|||||||
|
|
||||||
CITATION PRIORITY (most to least preferred):
|
CITATION PRIORITY (most to least preferred):
|
||||||
1. @handles from X — "per @handle" (these prove the tool's unique value)
|
1. @handles from X — "per @handle" (these prove the tool's unique value)
|
||||||
2. r/subreddits from Reddit — "per r/subreddit"
|
2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, prefer quoting top comments over just the thread title)
|
||||||
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
|
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
|
||||||
4. TikTok creators — "per @creator on TikTok" (viral/trending signal)
|
4. TikTok creators — "per @creator on TikTok" (viral/trending signal)
|
||||||
5. Instagram creators — "per @creator on Instagram" (influencer/creator signal)
|
5. Instagram creators — "per @creator on Instagram" (influencer/creator signal)
|
||||||
@@ -596,12 +597,13 @@ Want another prompt? Just tell me what you're creating next.
|
|||||||
## Security & Permissions
|
## Security & Permissions
|
||||||
|
|
||||||
**What this skill does:**
|
**What this skill does:**
|
||||||
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
|
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit search, subreddit discovery, and comment enrichment (requires SCRAPECREATORS_API_KEY — same key as TikTok + Instagram)
|
||||||
|
- Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
|
||||||
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
|
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
|
||||||
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
|
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
|
||||||
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
|
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
|
||||||
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
||||||
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (requires SCRAPECREATORS_API_KEY, PAYG after 100 free credits)
|
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (same SCRAPECREATORS_API_KEY as Reddit, PAYG after 100 free credits)
|
||||||
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
||||||
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
||||||
- Stores research findings in local SQLite database (watchlist mode only)
|
- Stores research findings in local SQLite database (watchlist mode only)
|
||||||
|
|||||||
@@ -0,0 +1,255 @@
|
|||||||
|
# feat: Reddit ScrapeCreators v2 — Improvements from Beta Testing
|
||||||
|
|
||||||
|
**Date:** 2026-03-05
|
||||||
|
**Type:** Enhancement
|
||||||
|
**Version:** v2.9 → v2.9.1-beta (or v3.0-beta if shipping to public)
|
||||||
|
**Branch:** `feat/reddit-scrapecreators` (continue existing branch)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Three focused improvements to the Reddit ScrapeCreators integration based on 5 full-pipeline tests ("Claude Code skills", "Kanye West", "Anthropic odds", "best rap songs lately", "Nano Banana Pro prompting"):
|
||||||
|
|
||||||
|
1. **Elevate top Reddit comments** — give weight to the wittiest/highest-voted comment in scoring and rendering
|
||||||
|
2. **Improve subreddit discovery** — tune heuristic so ambiguous queries find discussion subs, not utility subs
|
||||||
|
3. **Make ScrapeCreators the default recommended Reddit method** — update onboarding, SKILL.md metadata, and env.py messaging
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
### 1. Comments are undervalued
|
||||||
|
- ScrapeCreators returns real comment data with scores, but top comments only appear as `Insights:` text under each Reddit item
|
||||||
|
- The top comment (often the funniest/cleverest reply) gets no special treatment — it's just one of 3 comment excerpts
|
||||||
|
- Reddit's value IS the comments — upvoted replies are the distilled crowd wisdom
|
||||||
|
- Currently `comment_insights` are truncated at 150 chars and only 3 are shown per item in compact output
|
||||||
|
- No scoring bonus for posts that have high-quality comment threads
|
||||||
|
|
||||||
|
### 2. Subreddit discovery picks wrong subs for ambiguous queries
|
||||||
|
- "best rap songs lately" discovered `r/NameThatSong` and `r/findthatsong` (utility subs for identifying songs) instead of discussion subs like `r/hiphopheads` or `r/rap`
|
||||||
|
- "Kanye West" picked `r/ConcertsIndia_` as second sub — tangential at best
|
||||||
|
- The current heuristic is pure frequency count on `subreddit` field from global results, with no relevance weighting
|
||||||
|
- Utility/meta subs often dominate because the same query matches many "help me find X" posts
|
||||||
|
|
||||||
|
### 3. Onboarding still suggests OpenAI as the primary Reddit method
|
||||||
|
- SKILL.md metadata says `primaryEnv: OPENAI_API_KEY` and `requires.env: [OPENAI_API_KEY]`
|
||||||
|
- The web-only mode banner mentions "OPENAI_API_KEY or codex login → Reddit threads"
|
||||||
|
- `env.py` error messages direct users to OpenAI for Reddit access
|
||||||
|
- ScrapeCreators is cheaper ($0.012 vs $0.03-0.10), faster (17s vs 60-90s), returns real data, and shares a key with TikTok + Instagram
|
||||||
|
- New users should be told: "Get a SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, all three)"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Plan
|
||||||
|
|
||||||
|
### Task 1: Elevate Top Comments in Scoring and Rendering
|
||||||
|
|
||||||
|
**Goal:** Give Reddit posts a scoring bonus when they have highly-engaged comment threads, and render the #1 comment with special treatment.
|
||||||
|
|
||||||
|
**Files to modify:**
|
||||||
|
- `scripts/lib/reddit.py` — enrich with `top_comment_score` metadata
|
||||||
|
- `scripts/lib/score.py` — add comment quality bonus to Reddit scoring
|
||||||
|
- `scripts/lib/render.py` — render top comment with special formatting
|
||||||
|
- `scripts/lib/schema.py` — add `top_comment_excerpt` field to RedditItem (optional, may just use existing `top_comments[0]`)
|
||||||
|
|
||||||
|
#### 1a. Comment enrichment improvements (`scripts/lib/reddit.py`)
|
||||||
|
|
||||||
|
- [x] In `enrich_with_comments()`, after sorting comments by score, tag the item with:
|
||||||
|
- `top_comment_excerpt`: The highest-scored comment's body (up to 200 chars)
|
||||||
|
- `top_comment_score`: The upvote count of the #1 comment
|
||||||
|
- `top_comment_author`: Author of the #1 comment
|
||||||
|
- [x] Increase comment excerpt length from 300 → 400 chars for top comment only (funny/clever comments need more room)
|
||||||
|
- [x] Increase `comment_insights` limit from 7 → 10 (we have the data, show it)
|
||||||
|
- [x] For posts with enriched comments, store the comment count ratio: `top_comment_score / post_score` — a high ratio means the comment outshines the post (Reddit gold)
|
||||||
|
|
||||||
|
#### 1b. Scoring bonus for comment quality (`scripts/lib/score.py`)
|
||||||
|
|
||||||
|
- [x] In `compute_reddit_engagement_raw()`, add a comment quality signal:
|
||||||
|
- Current formula: `0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)`
|
||||||
|
- New formula: `0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)`
|
||||||
|
- This gives a ~10% weight to comment quality, slightly reducing post score and comment count weights
|
||||||
|
- Posts where the community engaged deeply (high top-comment score) rank higher
|
||||||
|
- [x] Need to pass `top_comment_score` through the engagement data — either:
|
||||||
|
- Option A: Add `top_comment_score` to `schema.Engagement` (cleanest)
|
||||||
|
- Option B: Read from `item.top_comments[0].score` during scoring (no schema change)
|
||||||
|
- **Recommend Option B** to avoid schema bloat — scoring can peek at `top_comments`
|
||||||
|
|
||||||
|
#### 1c. Render top comment prominently (`scripts/lib/render.py`)
|
||||||
|
|
||||||
|
- [x] In `render_compact()` Reddit section, after the `Insights:` block, add a "Top Comment:" line for items that have top_comments:
|
||||||
|
```
|
||||||
|
**R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
|
||||||
|
Claude Code creator: In the next version, introducing two new skills
|
||||||
|
https://www.reddit.com/r/ClaudeAI/comments/...
|
||||||
|
*Reddit global search*
|
||||||
|
💬 Top comment (247 upvotes): "So are they /batch migrating to Rust? :)"
|
||||||
|
Insights:
|
||||||
|
- TL;DR generated automatically after 50 comments...
|
||||||
|
- He's /batch migrating code daily?..
|
||||||
|
```
|
||||||
|
- [x] Only show `💬 Top comment` for items where `top_comments[0].score >= 10` (skip low-engagement comments)
|
||||||
|
- [x] Truncate at 200 chars with `...` if needed
|
||||||
|
- [x] Also update `render_full_report()` to include the top comment prominently
|
||||||
|
|
||||||
|
#### 1d. Update SKILL.md synthesis instructions
|
||||||
|
|
||||||
|
- [x] In the "Judge Agent: Synthesize All Sources" section, add guidance:
|
||||||
|
```
|
||||||
|
5b. For Reddit: Pay special attention to top comments — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes, quote it directly in your synthesis. Reddit's value is in the comments.
|
||||||
|
```
|
||||||
|
- [x] In the citation priority list, add: "When citing Reddit, prefer quoting top comments over just the thread title"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Improve Subreddit Discovery Heuristic
|
||||||
|
|
||||||
|
**Goal:** Find topical discussion subs rather than utility/meta subs.
|
||||||
|
|
||||||
|
**Files to modify:**
|
||||||
|
- `scripts/lib/reddit.py` — improve `discover_subreddits()` logic
|
||||||
|
|
||||||
|
#### 2a. Add relevance-weighted subreddit scoring
|
||||||
|
|
||||||
|
- [x] Replace pure frequency count with a weighted score:
|
||||||
|
```python
|
||||||
|
def discover_subreddits(results, topic, max_subs=5):
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
core_words = set(core.lower().split())
|
||||||
|
|
||||||
|
scores = Counter()
|
||||||
|
for post in results:
|
||||||
|
sub = post.get("subreddit", "")
|
||||||
|
if not sub:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Base: frequency count
|
||||||
|
base = 1.0
|
||||||
|
|
||||||
|
# Bonus: subreddit name contains a core topic word
|
||||||
|
sub_lower = sub.lower()
|
||||||
|
if any(w in sub_lower for w in core_words if len(w) > 2):
|
||||||
|
base += 2.0
|
||||||
|
|
||||||
|
# Penalty: known utility/meta subreddits
|
||||||
|
if sub_lower in UTILITY_SUBS:
|
||||||
|
base *= 0.3
|
||||||
|
|
||||||
|
# Bonus: post engagement (high-engagement posts = better sub)
|
||||||
|
ups = post.get("ups") or post.get("score", 0)
|
||||||
|
if ups > 100:
|
||||||
|
base += 0.5
|
||||||
|
|
||||||
|
scores[sub] += base
|
||||||
|
|
||||||
|
return [sub for sub, _ in scores.most_common(max_subs)]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2b. Define utility/meta subreddit blocklist
|
||||||
|
|
||||||
|
- [x] Add a small set of subs that are "find X for me" or "identify X" rather than discussion:
|
||||||
|
```python
|
||||||
|
UTILITY_SUBS = frozenset({
|
||||||
|
'namethatsong', 'findthatsong', 'tipofmytongue',
|
||||||
|
'whatisthissong', 'helpmefind', 'whatisthisthing',
|
||||||
|
'whatsthissong', 'findareddit', 'subredditdrama',
|
||||||
|
})
|
||||||
|
```
|
||||||
|
- [x] Keep this small and focused — don't over-filter. Only penalty (0.3x), not ban.
|
||||||
|
|
||||||
|
#### 2c. Try secondary query for subreddit discovery
|
||||||
|
|
||||||
|
- [x] If the first global search returns <3 unique subreddits above threshold, run a second global search with just `{core subject}` (stripped even further) to cast a wider net for subreddit frequencies
|
||||||
|
- [x] This helps niche topics where the full query is too specific
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Make ScrapeCreators the Default Reddit Method
|
||||||
|
|
||||||
|
**Goal:** New users should be guided to ScrapeCreators first, not OpenAI.
|
||||||
|
|
||||||
|
**Files to modify:**
|
||||||
|
- `SKILL.md` — metadata section, onboarding banner, security section
|
||||||
|
- `scripts/lib/env.py` — error messages and missing key guidance
|
||||||
|
- `scripts/lib/render.py` — web-only mode banner
|
||||||
|
|
||||||
|
#### 3a. Update SKILL.md metadata
|
||||||
|
|
||||||
|
- [x] Change `primaryEnv: OPENAI_API_KEY` → `primaryEnv: SCRAPECREATORS_API_KEY`
|
||||||
|
- [x] Change `requires.env: [OPENAI_API_KEY]` → `requires.env: [SCRAPECREATORS_API_KEY]`
|
||||||
|
- [x] Keep OPENAI_API_KEY mentioned but as optional/legacy
|
||||||
|
|
||||||
|
#### 3b. Update web-only mode banner (`scripts/lib/render.py`)
|
||||||
|
|
||||||
|
- [x] Change the current banner:
|
||||||
|
```
|
||||||
|
- `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments
|
||||||
|
```
|
||||||
|
To:
|
||||||
|
```
|
||||||
|
- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views
|
||||||
|
- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3c. Update env.py messaging
|
||||||
|
|
||||||
|
- [x] In `get_missing_keys()`, when Reddit is missing, suggest ScrapeCreators first:
|
||||||
|
- Current: returns `'reddit'` which triggers "Add OPENAI_API_KEY or run codex login" in SKILL.md
|
||||||
|
- Add a helper: `get_setup_hint(missing)` that returns:
|
||||||
|
- For 'reddit': `"Add SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, ~$0.002/search)"`
|
||||||
|
- For 'x': `"Add XAI_API_KEY for X posts"`
|
||||||
|
- For 'all': `"Add SCRAPECREATORS_API_KEY (Reddit+TikTok+Instagram) and XAI_API_KEY (X)"`
|
||||||
|
|
||||||
|
#### 3d. Update Security & Permissions section in SKILL.md
|
||||||
|
|
||||||
|
- [x] Add ScrapeCreators Reddit to the security section:
|
||||||
|
```
|
||||||
|
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit, TikTok, and Instagram search (requires SCRAPECREATORS_API_KEY)
|
||||||
|
```
|
||||||
|
- [x] Move "Sends search queries to OpenAI's Responses API for Reddit discovery" to a "Legacy:" subsection
|
||||||
|
- [x] Update "Reddit" description in `allowed-tools` or tags if needed
|
||||||
|
|
||||||
|
#### 3e. Update render.py coverage note
|
||||||
|
|
||||||
|
- [x] In `render_compact()`, the coverage note for `reddit-only` currently says "Add an xAI key"
|
||||||
|
- [x] When ScrapeCreators is the active Reddit source, no need to mention OpenAI at all
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] Top Reddit comment is rendered with `💬` prefix and upvote count for enriched posts
|
||||||
|
- [x] Posts with high top-comment scores rank slightly higher (visible in score differences)
|
||||||
|
- [x] "best rap songs lately" discovers at least one discussion sub (r/hiphopheads, r/rap, r/Music, etc.) instead of only utility subs
|
||||||
|
- [x] SKILL.md `primaryEnv` is `SCRAPECREATORS_API_KEY`
|
||||||
|
- [x] Web-only mode banner recommends ScrapeCreators first
|
||||||
|
- [x] All 5 test topics still pass (run same tests as before)
|
||||||
|
- [x] No regression in OpenAI fallback path
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files Changed (Summary)
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `scripts/lib/reddit.py` | Improve `discover_subreddits()` with relevance weighting, add utility sub penalties, enhance `enrich_with_comments()` top comment metadata |
|
||||||
|
| `scripts/lib/score.py` | Add 10% comment quality weight to Reddit engagement formula |
|
||||||
|
| `scripts/lib/render.py` | Add `💬 Top comment` line to compact output, update web-only banner |
|
||||||
|
| `scripts/lib/env.py` | Add `get_setup_hint()`, update missing key messaging |
|
||||||
|
| `SKILL.md` | Change `primaryEnv`, update onboarding banner, add comment synthesis guidance, update security section |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cost Impact
|
||||||
|
|
||||||
|
No cost increase. Same number of API calls per search. The changes are all in local logic (scoring, rendering, discovery heuristic).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing Plan
|
||||||
|
|
||||||
|
1. Re-run the same 5 test topics from beta testing
|
||||||
|
2. Verify top comments appear with `💬` in output
|
||||||
|
3. Verify "best rap songs lately" discovers at least one discussion subreddit
|
||||||
|
4. Verify `--diagnose` output recommends ScrapeCreators
|
||||||
|
5. Verify OpenAI fallback still works (unset SCRAPECREATORS_API_KEY, set OPENAI_API_KEY)
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
# Partner Credits: Zero-Registration Free Tier for last30days Users
|
||||||
|
|
||||||
|
**Type:** Partnership proposal for ScrapeCreators
|
||||||
|
**Date:** 2026-03-05
|
||||||
|
**Status:** Draft proposal
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Pitch
|
||||||
|
|
||||||
|
Every last30days user gets 100 free ScrapeCreators credits — Reddit, TikTok, Instagram — without ever visiting scrapecreators.com or creating an account. When they run out, the skill tells them where to upgrade. ScrapeCreators gets a distribution channel. last30days gets a killer default experience.
|
||||||
|
|
||||||
|
## The Problem
|
||||||
|
|
||||||
|
Right now, new last30days users hit a wall:
|
||||||
|
|
||||||
|
1. Install the skill (30 seconds)
|
||||||
|
2. Try `/last30days AI video tools`
|
||||||
|
3. Get told they need a `SCRAPECREATORS_API_KEY`
|
||||||
|
4. Have to go to scrapecreators.com, register, get a key, paste it into `.env`
|
||||||
|
5. Many never come back
|
||||||
|
|
||||||
|
The best Reddit experience requires a key. The friction kills adoption.
|
||||||
|
|
||||||
|
## The Proposal: Machine-Bound Partner Tokens
|
||||||
|
|
||||||
|
### How It Works
|
||||||
|
|
||||||
|
**ScrapeCreators side:**
|
||||||
|
1. Issue last30days a **partner ID** (e.g., `partner_last30days`)
|
||||||
|
2. Accept a new header: `X-Partner-Device: <device_hash>`
|
||||||
|
3. On first request per device hash: allocate 100 credits, no registration needed
|
||||||
|
4. Track usage: `(partner_id, device_hash) → credits_remaining`
|
||||||
|
5. When credits hit 0: return `402` with upgrade URL in response body
|
||||||
|
|
||||||
|
**last30days side:**
|
||||||
|
1. On first run, generate a **device fingerprint** and cache it locally
|
||||||
|
2. If user has no `SCRAPECREATORS_API_KEY`, send requests with partner headers instead
|
||||||
|
3. When 402 comes back, show a friendly "upgrade" message
|
||||||
|
|
||||||
|
That's it. No accounts, no OAuth, no registration flow.
|
||||||
|
|
||||||
|
### The Device Fingerprint
|
||||||
|
|
||||||
|
```python
|
||||||
|
import hashlib, platform, uuid, os
|
||||||
|
|
||||||
|
def get_device_id():
|
||||||
|
"""Generate a stable, hard-to-forge device fingerprint."""
|
||||||
|
# Use the OS-level machine ID (persists across reinstalls on most systems)
|
||||||
|
machine_id = _get_machine_id()
|
||||||
|
|
||||||
|
# Salt with the partner ID so the hash is useless outside this context
|
||||||
|
raw = f"last30days:{machine_id}"
|
||||||
|
return hashlib.sha256(raw.encode()).hexdigest()
|
||||||
|
|
||||||
|
def _get_machine_id():
|
||||||
|
"""Get the OS hardware/machine ID."""
|
||||||
|
if platform.system() == "Darwin":
|
||||||
|
# macOS: IOPlatformUUID (hardware-bound, survives OS reinstall)
|
||||||
|
import subprocess
|
||||||
|
result = subprocess.run(
|
||||||
|
["ioreg", "-rd1", "-c", "IOPlatformExpertDevice"],
|
||||||
|
capture_output=True, text=True
|
||||||
|
)
|
||||||
|
for line in result.stdout.splitlines():
|
||||||
|
if "IOPlatformUUID" in line:
|
||||||
|
return line.split('"')[-2]
|
||||||
|
|
||||||
|
elif platform.system() == "Linux":
|
||||||
|
# Linux: /etc/machine-id (set at install time)
|
||||||
|
try:
|
||||||
|
return open("/etc/machine-id").read().strip()
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Fallback: MAC address + hostname (less stable but reasonable)
|
||||||
|
return f"{uuid.getnode()}:{platform.node()}"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why this works:**
|
||||||
|
- macOS `IOPlatformUUID` is hardware-bound — can't change it without a new motherboard
|
||||||
|
- Linux `/etc/machine-id` is set at OS install — persists across reboots
|
||||||
|
- Hashed with `last30days:` prefix so the raw ID is never sent to ScrapeCreators
|
||||||
|
- Cached locally in `~/.config/last30days/.device_id` after first generation
|
||||||
|
|
||||||
|
### API Request Format
|
||||||
|
|
||||||
|
```
|
||||||
|
# Without partner credits (existing flow — user has their own key)
|
||||||
|
GET /v1/reddit/search?query=AI+tools
|
||||||
|
x-api-key: sc_user_abc123
|
||||||
|
|
||||||
|
# With partner credits (new — no registration needed)
|
||||||
|
GET /v1/reddit/search?query=AI+tools
|
||||||
|
x-api-key: sc_partner_last30days
|
||||||
|
X-Partner-Device: a1b2c3d4e5f6... (sha256 hex)
|
||||||
|
```
|
||||||
|
|
||||||
|
ScrapeCreators treats `sc_partner_last30days` as a special key class:
|
||||||
|
- Requires `X-Partner-Device` header
|
||||||
|
- Credits tracked per device hash, not per API key
|
||||||
|
- Rate limited per device (e.g., 10 requests/minute)
|
||||||
|
- 100 credits per unique device, lifetime
|
||||||
|
|
||||||
|
### What Counts as a Credit
|
||||||
|
|
||||||
|
One API call = one credit. A typical `/last30days` run uses roughly:
|
||||||
|
- 2-4 Reddit searches (global + subreddit drilldowns)
|
||||||
|
- 1-2 TikTok searches + 2-3 transcript fetches
|
||||||
|
- 1-2 Instagram searches + 2-3 transcript fetches
|
||||||
|
|
||||||
|
So ~10-15 credits per run. 100 credits ≈ **7-10 full research runs** before upgrade.
|
||||||
|
|
||||||
|
That's enough to get hooked.
|
||||||
|
|
||||||
|
## Abuse Prevention
|
||||||
|
|
||||||
|
### What we're defending against
|
||||||
|
|
||||||
|
| Threat | Likelihood | Impact |
|
||||||
|
|--------|-----------|--------|
|
||||||
|
| User spoofs device ID to get infinite credits | Low | Medium |
|
||||||
|
| Script generates thousands of fake device IDs | Medium | High |
|
||||||
|
| User shares partner key for non-last30days use | Low | Low |
|
||||||
|
|
||||||
|
### Defenses (simplest first)
|
||||||
|
|
||||||
|
**1. Hardware-bound device ID (primary defense)**
|
||||||
|
- macOS IOPlatformUUID can't be changed without hardware swap
|
||||||
|
- Linux machine-id requires root to change and breaks other software
|
||||||
|
- Not a cookie or config file — it's the machine itself
|
||||||
|
|
||||||
|
**2. Rate limiting per device (ScrapeCreators side)**
|
||||||
|
- 10 requests/minute per device hash
|
||||||
|
- Prevents scripted rapid-fire abuse even with valid device IDs
|
||||||
|
- Normal usage never hits this — a full run takes 60-70 seconds with natural gaps
|
||||||
|
|
||||||
|
**3. IP rate limiting on new device registrations (ScrapeCreators side)**
|
||||||
|
- Max 3 new device hashes per IP per day
|
||||||
|
- Stops "generate 1000 device IDs from one server" attacks
|
||||||
|
- Legitimate users: one machine, one device ID, done
|
||||||
|
|
||||||
|
**4. Total partner pool cap (safety valve)**
|
||||||
|
- ScrapeCreators sets a monthly cap on total partner credits (e.g., 50,000/month)
|
||||||
|
- If last30days goes viral and blows the cap, both parties renegotiate
|
||||||
|
- Prevents runaway costs from unexpected growth
|
||||||
|
|
||||||
|
### What we're NOT doing (intentional simplicity)
|
||||||
|
|
||||||
|
- No CAPTCHAs
|
||||||
|
- No email verification
|
||||||
|
- No phone verification
|
||||||
|
- No browser fingerprinting
|
||||||
|
- No token signing or crypto
|
||||||
|
- No account creation whatsoever
|
||||||
|
|
||||||
|
The goal is zero friction. The device ID is "good enough" — it stops casual abuse and scripts. A determined attacker could maybe get 200-300 free credits by VM gymnastics, but that's not worth defending against when paid plans are cheap.
|
||||||
|
|
||||||
|
## User Experience
|
||||||
|
|
||||||
|
### First run (no key configured)
|
||||||
|
|
||||||
|
```
|
||||||
|
$ /last30days AI video tools
|
||||||
|
|
||||||
|
🔍 Searching Reddit, TikTok, Instagram...
|
||||||
|
ℹ️ Using 100 free partner credits from ScrapeCreators (93 remaining)
|
||||||
|
Get your own key for unlimited use: scrapecreators.com/last30days
|
||||||
|
|
||||||
|
[... normal results ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Credits running low
|
||||||
|
|
||||||
|
```
|
||||||
|
ℹ️ 12 partner credits remaining. Get unlimited access: scrapecreators.com/last30days
|
||||||
|
```
|
||||||
|
|
||||||
|
### Credits exhausted
|
||||||
|
|
||||||
|
```
|
||||||
|
⚠️ Free partner credits used up!
|
||||||
|
|
||||||
|
Reddit, TikTok, and Instagram require a ScrapeCreators API key.
|
||||||
|
Get one at: scrapecreators.com/last30days (100 free credits on signup, then pay-as-you-go)
|
||||||
|
|
||||||
|
Continuing with X, YouTube, Hacker News, Polymarket, and web search...
|
||||||
|
```
|
||||||
|
|
||||||
|
Key detail: the skill **doesn't stop working** — it gracefully falls back to the sources that don't need a key. The user still gets value, but they see what they're missing.
|
||||||
|
|
||||||
|
### After upgrade
|
||||||
|
|
||||||
|
```
|
||||||
|
$ echo 'SCRAPECREATORS_API_KEY=sc_abc123' >> ~/.config/last30days/.env
|
||||||
|
|
||||||
|
# Next run — partner headers no longer sent, user's own key used
|
||||||
|
```
|
||||||
|
|
||||||
|
## What ScrapeCreators Gets
|
||||||
|
|
||||||
|
1. **Distribution channel** — every last30days install is a potential paying customer
|
||||||
|
2. **Zero support burden** — no accounts to manage for free tier users
|
||||||
|
3. **Qualified leads** — users who exhaust 100 credits are proven power users
|
||||||
|
4. **Co-marketing** — "Powered by ScrapeCreators" in every skill run
|
||||||
|
5. **Usage data** — anonymous device-level usage patterns across topics
|
||||||
|
|
||||||
|
## What last30days Gets
|
||||||
|
|
||||||
|
1. **Zero-config Reddit** — install and go, no registration anywhere
|
||||||
|
2. **TikTok and Instagram included** — three sources work out of the box
|
||||||
|
3. **Lower barrier to adoption** — the #1 friction point eliminated
|
||||||
|
4. **Upgrade path built in** — natural conversion funnel
|
||||||
|
|
||||||
|
## Implementation Effort
|
||||||
|
|
||||||
|
### ScrapeCreators side (their work)
|
||||||
|
- [ ] Create partner key class with per-device credit tracking
|
||||||
|
- [ ] Accept `X-Partner-Device` header on partner keys
|
||||||
|
- [ ] Return `402` with upgrade URL when credits exhausted
|
||||||
|
- [ ] Rate limit: 10 req/min per device, 3 new devices/day per IP
|
||||||
|
- [ ] Dashboard for last30days to see aggregate partner usage
|
||||||
|
|
||||||
|
### last30days side (our work)
|
||||||
|
- [ ] `scripts/lib/device_id.py` — generate and cache device fingerprint (~30 lines)
|
||||||
|
- [ ] Update `scripts/lib/env.py` — fall back to partner auth when no user key
|
||||||
|
- [ ] Update `_sc_headers()` in reddit.py, tiktok.py, instagram.py — add partner headers
|
||||||
|
- [ ] Handle 402 response — show upgrade message, continue with other sources
|
||||||
|
- [ ] Show credits remaining in run output (from response header)
|
||||||
|
|
||||||
|
### Suggested response headers from ScrapeCreators
|
||||||
|
|
||||||
|
```
|
||||||
|
X-Partner-Credits-Remaining: 87
|
||||||
|
X-Partner-Credits-Total: 100
|
||||||
|
X-Partner-Upgrade-URL: https://scrapecreators.com/last30days
|
||||||
|
```
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Credit pool negotiation** — what monthly cap works for ScrapeCreators?
|
||||||
|
2. **Referral tracking** — should `scrapecreators.com/last30days` give a signup bonus or revenue share?
|
||||||
|
3. **Credit count per endpoint** — should transcript fetches cost the same as searches?
|
||||||
|
4. **Expiration** — do unused partner credits expire (e.g., 90 days)?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
One device ID. One partner key. One header. 100 free credits. Zero registration.
|
||||||
|
|
||||||
|
The entire abuse prevention is: your computer has a hardware ID that you can't easily change. That's it. Simple, clever, and good enough.
|
||||||
+25
-33
@@ -1,52 +1,44 @@
|
|||||||
The AI world reinvents itself every month. This skill keeps you current.
|
The AI world reinvents itself every month. This skill keeps you current.
|
||||||
|
|
||||||
`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago.
|
`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
|
||||||
|
|
||||||
## Three Headline Features
|
## Three Headline Features in v2.9
|
||||||
|
|
||||||
**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search.
|
**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.
|
||||||
|
|
||||||
**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
|
**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw — not generic programming subs.
|
||||||
|
|
||||||
**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
|
**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with `💬` and upvote counts. Reddit's value is in the comments — now the skill surfaces them.
|
||||||
|
|
||||||
Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies.
|
Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** — no external CLI needed.
|
||||||
|
|
||||||
## Real Results (verified Feb 15)
|
## Beta Test Results (v2.9)
|
||||||
|
|
||||||
| Topic | Reddit | X | YouTube | Web |
|
| Topic | Time | Threads | Discovered Subreddits |
|
||||||
|-------|--------|---|---------|-----|
|
|-------|------|---------|----------------------|
|
||||||
| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages |
|
| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
|
||||||
| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages |
|
| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
|
||||||
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages |
|
| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
|
||||||
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages |
|
| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
|
||||||
| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages |
|
| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
|
||||||
|
|
||||||
## What's New
|
## What's New
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- Open-class skill with watchlist, briefing, and history modes
|
- ScrapeCreators Reddit backend with keyword search and subreddit discovery
|
||||||
- YouTube search + transcript extraction via yt-dlp
|
- Smart subreddit discovery with relevance-weighted scoring
|
||||||
- OpenAI Codex CLI compatibility
|
- Utility subreddit blocklist (`UTILITY_SUBS`)
|
||||||
- Bundled Twitter/X search (vendored Bird GraphQL, MIT)
|
- Top comment scoring (10% engagement weight) and prominent rendering
|
||||||
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro)
|
- Comment excerpts increased to 400 chars, insights raised to 10
|
||||||
- `--diagnose` flag for source status checking
|
|
||||||
- `--store` flag for SQLite accumulation
|
|
||||||
- Conversational first-run experience (NUX)
|
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
- Two-phase search architecture (entity-aware drill-down)
|
- `primaryEnv` → `SCRAPECREATORS_API_KEY` (one key for Reddit, TikTok, Instagram)
|
||||||
- Reddit JSON enrichment for real engagement metrics
|
- Reddit engagement scoring: `0.55/0.40/0.05` → `0.50/0.35/0.05/0.10`
|
||||||
- Smarter query construction with auto-retry on 0 results
|
- SKILL.md synthesis instructions emphasize quoting top comments
|
||||||
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
|
|
||||||
- `--days=N` configurable lookback (thanks @jonthebeef)
|
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
- YouTube/Reddit timeout resilience
|
- Utility sub noise in subreddit discovery
|
||||||
- Reddit 429 rate limit fail-fast
|
- Reddit no longer requires `OPENAI_API_KEY`
|
||||||
- Eager import crash in Codex environments
|
|
||||||
- X search returning 0 results on popular topics
|
|
||||||
- Windows Unicode crash (thanks @JosephOIbrahim)
|
|
||||||
|
|
||||||
## New Contributors
|
## New Contributors
|
||||||
|
|
||||||
@@ -70,4 +62,4 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
|
|||||||
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
|
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
|
||||||
```
|
```
|
||||||
|
|
||||||
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
|
30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.
|
||||||
|
|||||||
+55
-13
@@ -140,6 +140,7 @@ from lib import (
|
|||||||
models,
|
models,
|
||||||
normalize,
|
normalize,
|
||||||
openai_reddit,
|
openai_reddit,
|
||||||
|
reddit,
|
||||||
reddit_enrich,
|
reddit_enrich,
|
||||||
render,
|
render,
|
||||||
schema,
|
schema,
|
||||||
@@ -171,19 +172,51 @@ def _search_reddit(
|
|||||||
depth: str,
|
depth: str,
|
||||||
mock: bool,
|
mock: bool,
|
||||||
) -> tuple:
|
) -> tuple:
|
||||||
"""Search Reddit via OpenAI (runs in thread).
|
"""Search Reddit (runs in thread).
|
||||||
|
|
||||||
|
Uses ScrapeCreators when SCRAPECREATORS_API_KEY is available (preferred).
|
||||||
|
Falls back to OpenAI Responses API otherwise.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Tuple of (reddit_items, raw_openai, error)
|
Tuple of (reddit_items, raw_response, error, used_scrapecreators)
|
||||||
"""
|
"""
|
||||||
raw_openai = None
|
raw_response = None
|
||||||
reddit_error = None
|
reddit_error = None
|
||||||
|
used_scrapecreators = False
|
||||||
|
|
||||||
|
sc_token = config.get("SCRAPECREATORS_API_KEY")
|
||||||
|
|
||||||
if mock:
|
if mock:
|
||||||
raw_openai = load_fixture("openai_sample.json")
|
raw_response = load_fixture("openai_sample.json")
|
||||||
else:
|
elif sc_token:
|
||||||
|
# === ScrapeCreators path (preferred) ===
|
||||||
|
used_scrapecreators = True
|
||||||
try:
|
try:
|
||||||
raw_openai = openai_reddit.search_reddit(
|
sys.stderr.write("[Reddit] Using ScrapeCreators API\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
result = reddit.search_and_enrich(
|
||||||
|
topic, from_date, to_date,
|
||||||
|
depth=depth, token=sc_token,
|
||||||
|
)
|
||||||
|
reddit_items = result.get("items", [])
|
||||||
|
if result.get("error"):
|
||||||
|
reddit_error = result["error"]
|
||||||
|
return reddit_items, result, reddit_error, used_scrapecreators
|
||||||
|
except Exception as e:
|
||||||
|
reddit_error = f"ScrapeCreators: {type(e).__name__}: {e}"
|
||||||
|
sys.stderr.write(f"[Reddit] ScrapeCreators failed: {e}\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
# Fall through to OpenAI if we have that key
|
||||||
|
if not config.get("OPENAI_API_KEY"):
|
||||||
|
return [], {"error": str(e)}, reddit_error, used_scrapecreators
|
||||||
|
used_scrapecreators = False
|
||||||
|
sys.stderr.write("[Reddit] Falling back to OpenAI\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
# === OpenAI path (fallback) ===
|
||||||
|
if not mock:
|
||||||
|
try:
|
||||||
|
raw_response = openai_reddit.search_reddit(
|
||||||
config["OPENAI_API_KEY"],
|
config["OPENAI_API_KEY"],
|
||||||
selected_models["openai"],
|
selected_models["openai"],
|
||||||
topic,
|
topic,
|
||||||
@@ -194,14 +227,14 @@ def _search_reddit(
|
|||||||
account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
|
account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
|
||||||
)
|
)
|
||||||
except http.HTTPError as e:
|
except http.HTTPError as e:
|
||||||
raw_openai = {"error": str(e)}
|
raw_response = {"error": str(e)}
|
||||||
reddit_error = f"API error: {e}"
|
reddit_error = f"API error: {e}"
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raw_openai = {"error": str(e)}
|
raw_response = {"error": str(e)}
|
||||||
reddit_error = f"{type(e).__name__}: {e}"
|
reddit_error = f"{type(e).__name__}: {e}"
|
||||||
|
|
||||||
# Parse response
|
# Parse response
|
||||||
reddit_items = openai_reddit.parse_reddit_response(raw_openai or {})
|
reddit_items = openai_reddit.parse_reddit_response(raw_response or {})
|
||||||
|
|
||||||
# Quick retry with simpler query if few results
|
# Quick retry with simpler query if few results
|
||||||
if len(reddit_items) < 5 and not mock and not reddit_error:
|
if len(reddit_items) < 5 and not mock and not reddit_error:
|
||||||
@@ -218,7 +251,6 @@ def _search_reddit(
|
|||||||
account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
|
account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
|
||||||
)
|
)
|
||||||
retry_items = openai_reddit.parse_reddit_response(retry_raw)
|
retry_items = openai_reddit.parse_reddit_response(retry_raw)
|
||||||
# Add items not already found (by URL)
|
|
||||||
existing_urls = {item.get("url") for item in reddit_items}
|
existing_urls = {item.get("url") for item in reddit_items}
|
||||||
for item in retry_items:
|
for item in retry_items:
|
||||||
if item.get("url") not in existing_urls:
|
if item.get("url") not in existing_urls:
|
||||||
@@ -245,7 +277,7 @@ def _search_reddit(
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
return reddit_items, raw_openai, reddit_error
|
return reddit_items, raw_response, reddit_error, used_scrapecreators
|
||||||
|
|
||||||
|
|
||||||
def _search_x(
|
def _search_x(
|
||||||
@@ -887,10 +919,11 @@ def run_research(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Collect results (with timeouts to prevent indefinite blocking)
|
# Collect results (with timeouts to prevent indefinite blocking)
|
||||||
|
reddit_used_sc = False # Track if ScrapeCreators was used for Reddit
|
||||||
if reddit_future:
|
if reddit_future:
|
||||||
reddit_timeout = timeouts.get("reddit_future", future_timeout)
|
reddit_timeout = timeouts.get("reddit_future", future_timeout)
|
||||||
try:
|
try:
|
||||||
reddit_items, raw_openai, reddit_error = reddit_future.result(timeout=reddit_timeout)
|
reddit_items, raw_openai, reddit_error, reddit_used_sc = reddit_future.result(timeout=reddit_timeout)
|
||||||
if reddit_error and progress:
|
if reddit_error and progress:
|
||||||
progress.show_error(f"Reddit error: {reddit_error}")
|
progress.show_error(f"Reddit error: {reddit_error}")
|
||||||
except TimeoutError:
|
except TimeoutError:
|
||||||
@@ -1022,11 +1055,19 @@ def run_research(
|
|||||||
sys.stderr.flush()
|
sys.stderr.flush()
|
||||||
|
|
||||||
# Enrich Reddit items with real data (parallel, capped)
|
# Enrich Reddit items with real data (parallel, capped)
|
||||||
|
# Skip enrichment if ScrapeCreators already provided comments + engagement
|
||||||
enrich_max = timeouts["enrich_max_items"]
|
enrich_max = timeouts["enrich_max_items"]
|
||||||
enrich_total_timeout = timeouts["enrich_total"]
|
enrich_total_timeout = timeouts["enrich_total"]
|
||||||
items_to_enrich = reddit_items[:enrich_max]
|
items_to_enrich = reddit_items[:enrich_max]
|
||||||
rate_limited = False # Set True if Reddit returns 429 during enrichment
|
rate_limited = False # Set True if Reddit returns 429 during enrichment
|
||||||
|
|
||||||
|
if reddit_used_sc and items_to_enrich:
|
||||||
|
# ScrapeCreators already enriched items with comments — just copy to raw list
|
||||||
|
sys.stderr.write(f"[Reddit] Skipping old enrichment — ScrapeCreators already provided comments\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
raw_reddit_enriched = list(reddit_items[:enrich_max])
|
||||||
|
items_to_enrich = [] # Skip the enrichment block below
|
||||||
|
|
||||||
if items_to_enrich:
|
if items_to_enrich:
|
||||||
if progress:
|
if progress:
|
||||||
progress.start_reddit_enrich(1, len(items_to_enrich))
|
progress.start_reddit_enrich(1, len(items_to_enrich))
|
||||||
@@ -1101,11 +1142,12 @@ def run_research(
|
|||||||
|
|
||||||
# Phase 2: Supplemental search based on entities from Phase 1
|
# Phase 2: Supplemental search based on entities from Phase 1
|
||||||
# Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting
|
# Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting
|
||||||
|
# Also skip Reddit supplemental when ScrapeCreators was used (subreddit drilling already done)
|
||||||
if depth != "quick" and not mock and (reddit_items or x_items):
|
if depth != "quick" and not mock and (reddit_items or x_items):
|
||||||
sup_reddit, sup_x = _run_supplemental(
|
sup_reddit, sup_x = _run_supplemental(
|
||||||
topic, reddit_items, x_items,
|
topic, reddit_items, x_items,
|
||||||
from_date, to_date, depth, x_source, progress,
|
from_date, to_date, depth, x_source, progress,
|
||||||
skip_reddit=rate_limited,
|
skip_reddit=(rate_limited or reddit_used_sc),
|
||||||
resolved_handle=resolved_handle,
|
resolved_handle=resolved_handle,
|
||||||
)
|
)
|
||||||
if sup_reddit:
|
if sup_reddit:
|
||||||
|
|||||||
+33
-9
@@ -220,18 +220,42 @@ def config_exists() -> bool:
|
|||||||
return CONFIG_FILE.exists()
|
return CONFIG_FILE.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def is_reddit_available(config: Dict[str, Any]) -> bool:
|
||||||
|
"""Check if Reddit search is available.
|
||||||
|
|
||||||
|
Reddit can use either ScrapeCreators (preferred) or OpenAI.
|
||||||
|
"""
|
||||||
|
has_sc = bool(config.get('SCRAPECREATORS_API_KEY'))
|
||||||
|
has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
|
||||||
|
return has_sc or has_openai
|
||||||
|
|
||||||
|
|
||||||
|
def get_reddit_source(config: Dict[str, Any]) -> Optional[str]:
|
||||||
|
"""Determine which Reddit backend to use.
|
||||||
|
|
||||||
|
Priority: ScrapeCreators (cheaper, faster) > OpenAI (legacy)
|
||||||
|
|
||||||
|
Returns: 'scrapecreators', 'openai', or None
|
||||||
|
"""
|
||||||
|
if config.get('SCRAPECREATORS_API_KEY'):
|
||||||
|
return 'scrapecreators'
|
||||||
|
if config.get('OPENAI_API_KEY') and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK:
|
||||||
|
return 'openai'
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def get_available_sources(config: Dict[str, Any]) -> str:
|
def get_available_sources(config: Dict[str, Any]) -> str:
|
||||||
"""Determine which sources are available based on API keys.
|
"""Determine which sources are available based on API keys.
|
||||||
|
|
||||||
Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
|
Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
|
||||||
"""
|
"""
|
||||||
has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
|
has_reddit = is_reddit_available(config)
|
||||||
has_xai = bool(config.get('XAI_API_KEY'))
|
has_xai = bool(config.get('XAI_API_KEY'))
|
||||||
has_web = has_web_search_keys(config)
|
has_web = has_web_search_keys(config)
|
||||||
|
|
||||||
if has_openai and has_xai:
|
if has_reddit and has_xai:
|
||||||
return 'all' if has_web else 'both'
|
return 'all' if has_web else 'both'
|
||||||
elif has_openai:
|
elif has_reddit:
|
||||||
return 'reddit-web' if has_web else 'reddit'
|
return 'reddit-web' if has_web else 'reddit'
|
||||||
elif has_xai:
|
elif has_xai:
|
||||||
return 'x-web' if has_web else 'x'
|
return 'x-web' if has_web else 'x'
|
||||||
@@ -263,11 +287,11 @@ def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
|
|||||||
|
|
||||||
|
|
||||||
def get_missing_keys(config: Dict[str, Any]) -> str:
|
def get_missing_keys(config: Dict[str, Any]) -> str:
|
||||||
"""Determine which sources are missing (accounting for Bird).
|
"""Determine which sources are missing (accounting for Bird and ScrapeCreators).
|
||||||
|
|
||||||
Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
|
Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
|
||||||
"""
|
"""
|
||||||
has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
|
has_reddit = is_reddit_available(config)
|
||||||
has_xai = bool(config.get('XAI_API_KEY'))
|
has_xai = bool(config.get('XAI_API_KEY'))
|
||||||
has_web = has_web_search_keys(config)
|
has_web = has_web_search_keys(config)
|
||||||
|
|
||||||
@@ -277,14 +301,14 @@ def get_missing_keys(config: Dict[str, Any]) -> str:
|
|||||||
|
|
||||||
has_x = has_xai or has_bird
|
has_x = has_xai or has_bird
|
||||||
|
|
||||||
if has_openai and has_x and has_web:
|
if has_reddit and has_x and has_web:
|
||||||
return 'none'
|
return 'none'
|
||||||
elif has_openai and has_x:
|
elif has_reddit and has_x:
|
||||||
return 'web' # Missing web search keys
|
return 'web' # Missing web search keys
|
||||||
elif has_openai:
|
elif has_reddit:
|
||||||
return 'x' # Missing X source (and possibly web)
|
return 'x' # Missing X source (and possibly web)
|
||||||
elif has_x:
|
elif has_x:
|
||||||
return 'reddit' # Missing OpenAI key (and possibly web)
|
return 'reddit' # Missing Reddit source (and possibly web)
|
||||||
else:
|
else:
|
||||||
return 'all' # Missing everything
|
return 'all' # Missing everything
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,603 @@
|
|||||||
|
"""Reddit search via ScrapeCreators API for /last30days.
|
||||||
|
|
||||||
|
Uses ScrapeCreators REST API to search Reddit globally, discover relevant
|
||||||
|
subreddits, run targeted subreddit searches, and fetch comment trees.
|
||||||
|
|
||||||
|
Replaces openai_reddit.py as the primary Reddit search backend.
|
||||||
|
Falls back to openai_reddit.py if SCRAPECREATORS_API_KEY is missing but
|
||||||
|
OPENAI_API_KEY is present.
|
||||||
|
|
||||||
|
Requires SCRAPECREATORS_API_KEY in config (same key as TikTok + Instagram).
|
||||||
|
API docs: https://scrapecreators.com/docs
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from collections import Counter
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Dict, List, Optional, Set
|
||||||
|
|
||||||
|
try:
|
||||||
|
import requests as _requests
|
||||||
|
except ImportError:
|
||||||
|
_requests = None
|
||||||
|
|
||||||
|
from . import http
|
||||||
|
|
||||||
|
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/reddit"
|
||||||
|
|
||||||
|
# Depth configurations: how many API calls per phase
|
||||||
|
DEPTH_CONFIG = {
|
||||||
|
"quick": {
|
||||||
|
"global_searches": 1,
|
||||||
|
"subreddit_searches": 2,
|
||||||
|
"comment_enrichments": 3,
|
||||||
|
"timeframe": "week",
|
||||||
|
},
|
||||||
|
"default": {
|
||||||
|
"global_searches": 2,
|
||||||
|
"subreddit_searches": 3,
|
||||||
|
"comment_enrichments": 5,
|
||||||
|
"timeframe": "month",
|
||||||
|
},
|
||||||
|
"deep": {
|
||||||
|
"global_searches": 3,
|
||||||
|
"subreddit_searches": 5,
|
||||||
|
"comment_enrichments": 8,
|
||||||
|
"timeframe": "month",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stopwords for query extraction
|
||||||
|
NOISE_WORDS = frozenset({
|
||||||
|
'best', 'top', 'good', 'great', 'awesome', 'killer',
|
||||||
|
'latest', 'new', 'news', 'update', 'updates',
|
||||||
|
'trending', 'hottest', 'popular',
|
||||||
|
'practices', 'features', 'tips',
|
||||||
|
'recommendations', 'advice',
|
||||||
|
'prompt', 'prompts', 'prompting',
|
||||||
|
'methods', 'strategies', 'approaches',
|
||||||
|
'how', 'to', 'the', 'a', 'an', 'for', 'with',
|
||||||
|
'of', 'in', 'on', 'is', 'are', 'what', 'which',
|
||||||
|
'guide', 'tutorial', 'using',
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def _log(msg: str):
|
||||||
|
"""Log to stderr."""
|
||||||
|
sys.stderr.write(f"[Reddit] {msg}\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def _sc_headers(token: str) -> Dict[str, str]:
|
||||||
|
"""Build ScrapeCreators request headers."""
|
||||||
|
return {
|
||||||
|
"x-api-key": token,
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_core_subject(topic: str) -> str:
|
||||||
|
"""Extract core subject from verbose query.
|
||||||
|
|
||||||
|
Strips meta/research words to keep only the core product/concept name.
|
||||||
|
"""
|
||||||
|
text = topic.lower().strip()
|
||||||
|
|
||||||
|
# Strip multi-word prefixes
|
||||||
|
prefixes = [
|
||||||
|
'what are the best', 'what is the best', 'what are the latest',
|
||||||
|
'what are people saying about', 'what do people think about',
|
||||||
|
'how do i use', 'how to use', 'how to',
|
||||||
|
'what are', 'what is', 'tips for', 'best practices for',
|
||||||
|
]
|
||||||
|
for p in prefixes:
|
||||||
|
if text.startswith(p + ' '):
|
||||||
|
text = text[len(p):].strip()
|
||||||
|
|
||||||
|
words = text.split()
|
||||||
|
filtered = [w for w in words if w not in NOISE_WORDS]
|
||||||
|
|
||||||
|
result = ' '.join(filtered) if filtered else text
|
||||||
|
return result.rstrip('?!.')
|
||||||
|
|
||||||
|
|
||||||
|
def expand_reddit_queries(topic: str, depth: str) -> List[str]:
|
||||||
|
"""Generate multiple Reddit search queries from a topic.
|
||||||
|
|
||||||
|
Uses local logic (no LLM call needed):
|
||||||
|
1. Extract core subject (strip noise words)
|
||||||
|
2. Include original topic if different from core
|
||||||
|
3. For default/deep: add casual/review variant
|
||||||
|
4. For deep: add problem/issues variant
|
||||||
|
|
||||||
|
Returns 1-4 query strings depending on depth.
|
||||||
|
"""
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
queries = [core]
|
||||||
|
|
||||||
|
# Broader variant: include more context from original topic
|
||||||
|
original_clean = topic.strip().rstrip('?!.')
|
||||||
|
if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
|
||||||
|
queries.append(original_clean)
|
||||||
|
|
||||||
|
if depth in ("default", "deep"):
|
||||||
|
queries.append(f"{core} worth it OR thoughts OR review")
|
||||||
|
|
||||||
|
if depth == "deep":
|
||||||
|
queries.append(f"{core} issues OR problems OR bug OR broken")
|
||||||
|
|
||||||
|
return queries
|
||||||
|
|
||||||
|
|
||||||
|
# Known utility/meta subreddits that match queries but aren't discussion subs.
|
||||||
|
# These get a 0.3x penalty (not banned) in subreddit discovery scoring.
|
||||||
|
UTILITY_SUBS = frozenset({
|
||||||
|
'namethatsong', 'findthatsong', 'tipofmytongue',
|
||||||
|
'whatisthissong', 'helpmefind', 'whatisthisthing',
|
||||||
|
'whatsthissong', 'findareddit', 'subredditdrama',
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def discover_subreddits(
|
||||||
|
results: List[Dict[str, Any]],
|
||||||
|
topic: str = "",
|
||||||
|
max_subs: int = 5,
|
||||||
|
) -> List[str]:
|
||||||
|
"""Extract top subreddits from global search results with relevance weighting.
|
||||||
|
|
||||||
|
Uses frequency + topic-word matching + utility-sub penalties + engagement
|
||||||
|
bonus to find discussion subs rather than utility/meta subs.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
results: List of post dicts from global search
|
||||||
|
topic: Original search topic (for relevance matching)
|
||||||
|
max_subs: Maximum subreddits to return
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Top subreddit names sorted by weighted score
|
||||||
|
"""
|
||||||
|
core = _extract_core_subject(topic) if topic else ""
|
||||||
|
core_words = set(core.lower().split()) if core else set()
|
||||||
|
|
||||||
|
scores = Counter()
|
||||||
|
for post in results:
|
||||||
|
sub = post.get("subreddit", "")
|
||||||
|
if not sub:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Base: frequency count
|
||||||
|
base = 1.0
|
||||||
|
|
||||||
|
# Bonus: subreddit name contains a core topic word
|
||||||
|
sub_lower = sub.lower()
|
||||||
|
if core_words and any(w in sub_lower for w in core_words if len(w) > 2):
|
||||||
|
base += 2.0
|
||||||
|
|
||||||
|
# Penalty: known utility/meta subreddits
|
||||||
|
if sub_lower in UTILITY_SUBS:
|
||||||
|
base *= 0.3
|
||||||
|
|
||||||
|
# Bonus: post engagement (high-engagement posts = better sub)
|
||||||
|
ups = post.get("ups") or post.get("score", 0)
|
||||||
|
if ups and ups > 100:
|
||||||
|
base += 0.5
|
||||||
|
|
||||||
|
scores[sub] += base
|
||||||
|
|
||||||
|
return [sub for sub, _ in scores.most_common(max_subs)]
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_date(created_utc) -> Optional[str]:
|
||||||
|
"""Convert Unix timestamp to YYYY-MM-DD."""
|
||||||
|
if not created_utc:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
dt = datetime.fromtimestamp(float(created_utc), tz=timezone.utc)
|
||||||
|
return dt.strftime("%Y-%m-%d")
|
||||||
|
except (ValueError, TypeError, OSError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global") -> Dict[str, Any]:
|
||||||
|
"""Normalize a ScrapeCreators Reddit post to our internal format."""
|
||||||
|
permalink = post.get("permalink", "")
|
||||||
|
url = f"https://www.reddit.com{permalink}" if permalink else post.get("url", "")
|
||||||
|
|
||||||
|
# Ensure URL looks like a Reddit thread
|
||||||
|
if url and "reddit.com" not in url:
|
||||||
|
url = ""
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": f"R{idx}",
|
||||||
|
"reddit_id": post.get("id", ""),
|
||||||
|
"title": str(post.get("title", "")).strip(),
|
||||||
|
"url": url,
|
||||||
|
"subreddit": str(post.get("subreddit", "")).strip(),
|
||||||
|
"date": _parse_date(post.get("created_utc")),
|
||||||
|
"engagement": {
|
||||||
|
"score": post.get("ups") or post.get("score", 0),
|
||||||
|
"num_comments": post.get("num_comments", 0),
|
||||||
|
"upvote_ratio": post.get("upvote_ratio"),
|
||||||
|
},
|
||||||
|
"relevance": 0.7,
|
||||||
|
"why_relevant": f"Reddit {source_label} search",
|
||||||
|
"selftext": str(post.get("selftext", ""))[:500],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _global_search(
|
||||||
|
query: str,
|
||||||
|
token: str,
|
||||||
|
sort: str = "relevance",
|
||||||
|
timeframe: str = "month",
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Search across all of Reddit via ScrapeCreators global search.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: Search query
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
sort: Sort order (relevance, hot, top, new)
|
||||||
|
timeframe: Time filter (hour, day, week, month, year, all)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of post dicts
|
||||||
|
"""
|
||||||
|
if not _requests:
|
||||||
|
_log("requests library not installed, falling back to urllib")
|
||||||
|
# Use stdlib http module as fallback
|
||||||
|
try:
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
params = urlencode({"query": query, "sort": sort, "timeframe": timeframe})
|
||||||
|
url = f"{SCRAPECREATORS_BASE}/search?{params}"
|
||||||
|
headers = _sc_headers(token)
|
||||||
|
headers["User-Agent"] = http.USER_AGENT
|
||||||
|
data = http.get(url, headers=headers, timeout=30, retries=2)
|
||||||
|
return data.get("posts", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Global search error (urllib): {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = _requests.get(
|
||||||
|
f"{SCRAPECREATORS_BASE}/search",
|
||||||
|
params={"query": query, "sort": sort, "timeframe": timeframe},
|
||||||
|
headers=_sc_headers(token),
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
return data.get("posts", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Global search error: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _subreddit_search(
|
||||||
|
subreddit: str,
|
||||||
|
query: str,
|
||||||
|
token: str,
|
||||||
|
sort: str = "relevance",
|
||||||
|
timeframe: str = "month",
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Search within a specific subreddit via ScrapeCreators.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
subreddit: Subreddit name (without r/)
|
||||||
|
query: Search query
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
sort: Sort order
|
||||||
|
timeframe: Time filter
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of post dicts
|
||||||
|
"""
|
||||||
|
if not _requests:
|
||||||
|
try:
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
params = urlencode({
|
||||||
|
"subreddit": subreddit, "query": query,
|
||||||
|
"sort": sort, "timeframe": timeframe,
|
||||||
|
})
|
||||||
|
url = f"{SCRAPECREATORS_BASE}/subreddit/search?{params}"
|
||||||
|
headers = _sc_headers(token)
|
||||||
|
headers["User-Agent"] = http.USER_AGENT
|
||||||
|
data = http.get(url, headers=headers, timeout=30, retries=2)
|
||||||
|
return data.get("posts", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Subreddit search error (urllib) for r/{subreddit}: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = _requests.get(
|
||||||
|
f"{SCRAPECREATORS_BASE}/subreddit/search",
|
||||||
|
params={
|
||||||
|
"subreddit": subreddit,
|
||||||
|
"query": query,
|
||||||
|
"sort": sort,
|
||||||
|
"timeframe": timeframe,
|
||||||
|
},
|
||||||
|
headers=_sc_headers(token),
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
return data.get("posts", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Subreddit search error for r/{subreddit}: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_post_comments(
|
||||||
|
url: str,
|
||||||
|
token: str,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Fetch comments for a Reddit post via ScrapeCreators.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url: Reddit post URL or permalink
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of comment dicts with score, author, body, etc.
|
||||||
|
"""
|
||||||
|
if not _requests:
|
||||||
|
try:
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
params = urlencode({"url": url})
|
||||||
|
api_url = f"{SCRAPECREATORS_BASE}/post/comments?{params}"
|
||||||
|
headers = _sc_headers(token)
|
||||||
|
headers["User-Agent"] = http.USER_AGENT
|
||||||
|
data = http.get(api_url, headers=headers, timeout=30, retries=2)
|
||||||
|
return data.get("comments", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Comment fetch error (urllib): {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = _requests.get(
|
||||||
|
f"{SCRAPECREATORS_BASE}/post/comments",
|
||||||
|
params={"url": url},
|
||||||
|
headers=_sc_headers(token),
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
return data.get("comments", data.get("data", []))
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Comment fetch error: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _dedupe_posts(posts: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||||
|
"""Deduplicate posts by reddit_id, keeping first occurrence."""
|
||||||
|
seen_ids = set()
|
||||||
|
seen_urls = set()
|
||||||
|
unique = []
|
||||||
|
for post in posts:
|
||||||
|
rid = post.get("reddit_id", "")
|
||||||
|
url = post.get("url", "")
|
||||||
|
if rid and rid in seen_ids:
|
||||||
|
continue
|
||||||
|
if url and url in seen_urls:
|
||||||
|
continue
|
||||||
|
if rid:
|
||||||
|
seen_ids.add(rid)
|
||||||
|
if url:
|
||||||
|
seen_urls.add(url)
|
||||||
|
unique.append(post)
|
||||||
|
return unique
|
||||||
|
|
||||||
|
|
||||||
|
def search_reddit(
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
depth: str = "default",
|
||||||
|
token: str = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Full Reddit search: multi-query global discovery + subreddit drill-down.
|
||||||
|
|
||||||
|
This is the main entry point. Replaces openai_reddit.search_reddit().
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic: Search topic
|
||||||
|
from_date: Start date (YYYY-MM-DD)
|
||||||
|
to_date: End date (YYYY-MM-DD)
|
||||||
|
depth: 'quick', 'default', or 'deep'
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with 'items' list and optional 'error'.
|
||||||
|
"""
|
||||||
|
if not token:
|
||||||
|
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
|
||||||
|
|
||||||
|
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||||
|
timeframe = config["timeframe"]
|
||||||
|
|
||||||
|
# === Phase 1: Query Expansion ===
|
||||||
|
queries = expand_reddit_queries(topic, depth)
|
||||||
|
_log(f"Expanded '{topic}' into {len(queries)} queries: {queries}")
|
||||||
|
|
||||||
|
# === Phase 2: Global Discovery ===
|
||||||
|
all_raw_posts = []
|
||||||
|
max_global = config["global_searches"]
|
||||||
|
|
||||||
|
for i, query in enumerate(queries[:max_global]):
|
||||||
|
sort = "relevance" if i == 0 else "top"
|
||||||
|
_log(f"Global search {i+1}/{max_global}: '{query}' (sort={sort})")
|
||||||
|
posts = _global_search(query, token, sort=sort, timeframe=timeframe)
|
||||||
|
_log(f" -> {len(posts)} results")
|
||||||
|
all_raw_posts.extend(posts)
|
||||||
|
|
||||||
|
# Normalize all posts
|
||||||
|
all_items = []
|
||||||
|
for i, post in enumerate(all_raw_posts):
|
||||||
|
item = _normalize_post(post, i + 1, "global")
|
||||||
|
all_items.append(item)
|
||||||
|
|
||||||
|
# === Phase 3: Subreddit Discovery + Targeted Search ===
|
||||||
|
discovered_subs = discover_subreddits(all_raw_posts, topic=topic, max_subs=config["subreddit_searches"])
|
||||||
|
_log(f"Discovered subreddits: {discovered_subs}")
|
||||||
|
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
for sub in discovered_subs[:config["subreddit_searches"]]:
|
||||||
|
_log(f"Subreddit search: r/{sub} for '{core}'")
|
||||||
|
sub_posts = _subreddit_search(sub, core, token, sort="relevance", timeframe=timeframe)
|
||||||
|
_log(f" -> {len(sub_posts)} results from r/{sub}")
|
||||||
|
for j, post in enumerate(sub_posts):
|
||||||
|
item = _normalize_post(post, len(all_items) + j + 1, f"r/{sub}")
|
||||||
|
all_items.append(item)
|
||||||
|
|
||||||
|
# === Phase 4: Deduplicate ===
|
||||||
|
all_items = _dedupe_posts(all_items)
|
||||||
|
_log(f"After dedup: {len(all_items)} unique posts")
|
||||||
|
|
||||||
|
# === Phase 5: Date filter ===
|
||||||
|
in_range = []
|
||||||
|
out_of_range = 0
|
||||||
|
for item in all_items:
|
||||||
|
if item["date"] and from_date <= item["date"] <= to_date:
|
||||||
|
in_range.append(item)
|
||||||
|
elif item["date"] is None:
|
||||||
|
in_range.append(item) # Keep unknown dates
|
||||||
|
else:
|
||||||
|
out_of_range += 1
|
||||||
|
|
||||||
|
if in_range:
|
||||||
|
all_items = in_range
|
||||||
|
if out_of_range:
|
||||||
|
_log(f"Filtered {out_of_range} posts outside date range")
|
||||||
|
else:
|
||||||
|
_log(f"No posts within date range, keeping all {len(all_items)}")
|
||||||
|
|
||||||
|
# === Phase 6: Sort by engagement ===
|
||||||
|
all_items.sort(
|
||||||
|
key=lambda x: (x.get("engagement", {}).get("score", 0) or 0),
|
||||||
|
reverse=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Re-index IDs
|
||||||
|
for i, item in enumerate(all_items):
|
||||||
|
item["id"] = f"R{i+1}"
|
||||||
|
|
||||||
|
_log(f"Final: {len(all_items)} Reddit posts")
|
||||||
|
return {"items": all_items}
|
||||||
|
|
||||||
|
|
||||||
|
def enrich_with_comments(
|
||||||
|
items: List[Dict[str, Any]],
|
||||||
|
token: str,
|
||||||
|
depth: str = "default",
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Enrich top items with comment data from ScrapeCreators.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
items: Reddit items from search_reddit()
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
depth: Depth for comment limit
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Items with top_comments and comment_insights added.
|
||||||
|
"""
|
||||||
|
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||||
|
max_comments = config["comment_enrichments"]
|
||||||
|
|
||||||
|
if not items or not token:
|
||||||
|
return items
|
||||||
|
|
||||||
|
top_items = items[:max_comments]
|
||||||
|
_log(f"Enriching comments for {len(top_items)} posts")
|
||||||
|
|
||||||
|
for item in top_items:
|
||||||
|
url = item.get("url", "")
|
||||||
|
if not url:
|
||||||
|
continue
|
||||||
|
|
||||||
|
raw_comments = fetch_post_comments(url, token)
|
||||||
|
if not raw_comments:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Parse comments into our format
|
||||||
|
top_comments = []
|
||||||
|
insights = []
|
||||||
|
|
||||||
|
for ci, c in enumerate(raw_comments[:10]): # Take top 10 comments
|
||||||
|
body = c.get("body", "")
|
||||||
|
if not body or body in ("[deleted]", "[removed]"):
|
||||||
|
continue
|
||||||
|
|
||||||
|
score = c.get("ups") or c.get("score", 0)
|
||||||
|
author = c.get("author", "[deleted]")
|
||||||
|
permalink = c.get("permalink", "")
|
||||||
|
comment_url = f"https://reddit.com{permalink}" if permalink else ""
|
||||||
|
|
||||||
|
# Top comment gets more room (400 chars) — funny/clever comments need it
|
||||||
|
max_excerpt = 400 if ci == 0 else 300
|
||||||
|
top_comments.append({
|
||||||
|
"score": score,
|
||||||
|
"date": _parse_date(c.get("created_utc")),
|
||||||
|
"author": author,
|
||||||
|
"excerpt": body[:max_excerpt],
|
||||||
|
"url": comment_url,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Extract insights from substantive comments
|
||||||
|
if len(body) >= 30 and author not in ("[deleted]", "[removed]", "AutoModerator"):
|
||||||
|
insight = body[:150]
|
||||||
|
if len(body) > 150:
|
||||||
|
for i, char in enumerate(insight):
|
||||||
|
if char in '.!?' and i > 50:
|
||||||
|
insight = insight[:i+1]
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
insight = insight.rstrip() + "..."
|
||||||
|
insights.append(insight)
|
||||||
|
|
||||||
|
# Sort comments by score
|
||||||
|
top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
|
||||||
|
|
||||||
|
item["top_comments"] = top_comments[:10]
|
||||||
|
item["comment_insights"] = insights[:10]
|
||||||
|
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def search_and_enrich(
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
depth: str = "default",
|
||||||
|
token: str = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Full Reddit pipeline: search + comment enrichment.
|
||||||
|
|
||||||
|
This is the convenience function that does everything.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic: Search topic
|
||||||
|
from_date: Start date (YYYY-MM-DD)
|
||||||
|
to_date: End date (YYYY-MM-DD)
|
||||||
|
depth: 'quick', 'default', or 'deep'
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with 'items' list. Items include top_comments and comment_insights.
|
||||||
|
"""
|
||||||
|
result = search_reddit(topic, from_date, to_date, depth, token)
|
||||||
|
items = result.get("items", [])
|
||||||
|
|
||||||
|
if items and token:
|
||||||
|
items = enrich_with_comments(items, token, depth)
|
||||||
|
result["items"] = items
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||||
|
"""Parse ScrapeCreators response to item list.
|
||||||
|
|
||||||
|
Compatibility shim matching openai_reddit.parse_reddit_response() signature.
|
||||||
|
"""
|
||||||
|
return response.get("items", [])
|
||||||
@@ -1,4 +1,9 @@
|
|||||||
"""Reddit thread enrichment with real engagement metrics."""
|
"""Reddit thread enrichment with real engagement metrics.
|
||||||
|
|
||||||
|
Supports two backends:
|
||||||
|
1. ScrapeCreators API (preferred) - no rate limits, 1 credit/call
|
||||||
|
2. reddit.com/.json (fallback) - free but 429-prone
|
||||||
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from typing import Any, Dict, List, Optional
|
from typing import Any, Dict, List, Optional
|
||||||
@@ -254,3 +259,67 @@ def enrich_reddit_item(
|
|||||||
item["comment_insights"] = extract_comment_insights(top_comments)
|
item["comment_insights"] = extract_comment_insights(top_comments)
|
||||||
|
|
||||||
return item
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def enrich_reddit_item_sc(
|
||||||
|
item: Dict[str, Any],
|
||||||
|
token: str,
|
||||||
|
timeout: int = 30,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Enrich a Reddit item using ScrapeCreators comment API.
|
||||||
|
|
||||||
|
No rate limit risk. Uses 1 credit per call.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
item: Reddit item dict (already has engagement from search)
|
||||||
|
token: ScrapeCreators API key
|
||||||
|
timeout: HTTP timeout
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Enriched item with top_comments and comment_insights
|
||||||
|
"""
|
||||||
|
from . import reddit as reddit_mod
|
||||||
|
|
||||||
|
url = item.get("url", "")
|
||||||
|
if not url:
|
||||||
|
return item
|
||||||
|
|
||||||
|
raw_comments = reddit_mod.fetch_post_comments(url, token)
|
||||||
|
if not raw_comments:
|
||||||
|
return item
|
||||||
|
|
||||||
|
top_comments = []
|
||||||
|
for c in raw_comments[:10]:
|
||||||
|
body = c.get("body", "")
|
||||||
|
if not body or body in ("[deleted]", "[removed]"):
|
||||||
|
continue
|
||||||
|
|
||||||
|
score = c.get("ups") or c.get("score", 0)
|
||||||
|
author = c.get("author", "[deleted]")
|
||||||
|
permalink = c.get("permalink", "")
|
||||||
|
comment_url = f"https://reddit.com{permalink}" if permalink else ""
|
||||||
|
|
||||||
|
top_comments.append({
|
||||||
|
"score": score,
|
||||||
|
"date": dates.timestamp_to_date(c.get("created_utc")) if c.get("created_utc") else None,
|
||||||
|
"author": author,
|
||||||
|
"body": body[:300],
|
||||||
|
"excerpt": body[:200],
|
||||||
|
"url": comment_url,
|
||||||
|
})
|
||||||
|
|
||||||
|
top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
|
||||||
|
|
||||||
|
item["top_comments"] = []
|
||||||
|
for c in top_comments:
|
||||||
|
item["top_comments"].append({
|
||||||
|
"score": c.get("score", 0),
|
||||||
|
"date": c.get("date"),
|
||||||
|
"author": c.get("author", ""),
|
||||||
|
"excerpt": c.get("excerpt", ""),
|
||||||
|
"url": c.get("url", ""),
|
||||||
|
})
|
||||||
|
|
||||||
|
item["comment_insights"] = extract_comment_insights(top_comments)
|
||||||
|
|
||||||
|
return item
|
||||||
|
|||||||
+22
-5
@@ -108,11 +108,11 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
|||||||
lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
|
lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("---")
|
lines.append("---")
|
||||||
lines.append("**⚡ Want better results?** Add API keys or sign in to Codex to unlock Reddit & X data:")
|
lines.append("**⚡ Want better results?** Add API keys to unlock Reddit, TikTok, Instagram & X data:")
|
||||||
lines.append("- `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments")
|
lines.append("- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views")
|
||||||
lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
|
lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
|
||||||
|
lines.append("- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)")
|
||||||
lines.append("- Edit `~/.config/last30days/.env` to add keys")
|
lines.append("- Edit `~/.config/last30days/.env` to add keys")
|
||||||
lines.append("- If already signed in but still seeing this, re-run `codex login`")
|
|
||||||
lines.append("---")
|
lines.append("---")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
@@ -137,7 +137,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
|||||||
lines.append("*💡 Tip: Add an xAI key (`XAI_API_KEY`) for X/Twitter data and better triangulation.*")
|
lines.append("*💡 Tip: Add an xAI key (`XAI_API_KEY`) for X/Twitter data and better triangulation.*")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
elif report.mode == "x-only" and missing_keys in ("reddit", "none"):
|
elif report.mode == "x-only" and missing_keys in ("reddit", "none"):
|
||||||
lines.append("*💡 Tip: Add OPENAI_API_KEY or run `codex login` for Reddit data and better triangulation. If already signed in, re-run `codex login`.*")
|
lines.append("*💡 Tip: Add `SCRAPECREATORS_API_KEY` for Reddit + TikTok + Instagram data (one key, all three) and better triangulation.*")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
# Reddit items
|
# Reddit items
|
||||||
@@ -174,7 +174,15 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
|||||||
lines.append(f" {item.url}")
|
lines.append(f" {item.url}")
|
||||||
lines.append(f" *{item.why_relevant}*")
|
lines.append(f" *{item.why_relevant}*")
|
||||||
|
|
||||||
# Top comment insights
|
# Top comment (elevated — Reddit's value IS the comments)
|
||||||
|
if item.top_comments and item.top_comments[0].score >= 10:
|
||||||
|
tc = item.top_comments[0]
|
||||||
|
excerpt = tc.excerpt[:200]
|
||||||
|
if len(tc.excerpt) > 200:
|
||||||
|
excerpt = excerpt.rstrip() + "..."
|
||||||
|
lines.append(f' \U0001f4ac Top comment ({tc.score} upvotes): "{excerpt}"')
|
||||||
|
|
||||||
|
# Comment insights
|
||||||
if item.comment_insights:
|
if item.comment_insights:
|
||||||
lines.append(" Insights:")
|
lines.append(" Insights:")
|
||||||
for insight in item.comment_insights[:3]:
|
for insight in item.comment_insights[:3]:
|
||||||
@@ -622,6 +630,15 @@ def render_full_report(report: schema.Report) -> str:
|
|||||||
eng = item.engagement
|
eng = item.engagement
|
||||||
lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
|
lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
|
||||||
|
|
||||||
|
if item.top_comments and item.top_comments[0].score >= 10:
|
||||||
|
tc = item.top_comments[0]
|
||||||
|
excerpt = tc.excerpt[:200]
|
||||||
|
if len(tc.excerpt) > 200:
|
||||||
|
excerpt = excerpt.rstrip() + "..."
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f'**\U0001f4ac Top Comment** ({tc.score} upvotes, u/{tc.author}):')
|
||||||
|
lines.append(f'> {excerpt}')
|
||||||
|
|
||||||
if item.comment_insights:
|
if item.comment_insights:
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("**Key Insights from Comments:**")
|
lines.append("**Key Insights from Comments:**")
|
||||||
|
|||||||
+17
-5
@@ -31,10 +31,16 @@ def log1p_safe(x: Optional[int]) -> float:
|
|||||||
return math.log1p(x)
|
return math.log1p(x)
|
||||||
|
|
||||||
|
|
||||||
def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
|
def compute_reddit_engagement_raw(
|
||||||
|
engagement: Optional[schema.Engagement],
|
||||||
|
top_comment_score: Optional[int] = None,
|
||||||
|
) -> Optional[float]:
|
||||||
"""Compute raw engagement score for Reddit item.
|
"""Compute raw engagement score for Reddit item.
|
||||||
|
|
||||||
Formula: 0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)
|
Formula: 0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)
|
||||||
|
|
||||||
|
The 10% comment quality weight rewards posts where the community engaged deeply
|
||||||
|
— a highly upvoted top comment means the thread sparked real discussion.
|
||||||
"""
|
"""
|
||||||
if engagement is None:
|
if engagement is None:
|
||||||
return None
|
return None
|
||||||
@@ -45,8 +51,9 @@ def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Op
|
|||||||
score = log1p_safe(engagement.score)
|
score = log1p_safe(engagement.score)
|
||||||
comments = log1p_safe(engagement.num_comments)
|
comments = log1p_safe(engagement.num_comments)
|
||||||
ratio = (engagement.upvote_ratio or 0.5) * 10
|
ratio = (engagement.upvote_ratio or 0.5) * 10
|
||||||
|
top_cmt = log1p_safe(top_comment_score)
|
||||||
|
|
||||||
return 0.55 * score + 0.40 * comments + 0.05 * ratio
|
return 0.50 * score + 0.35 * comments + 0.05 * ratio + 0.10 * top_cmt
|
||||||
|
|
||||||
|
|
||||||
def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
|
def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
|
||||||
@@ -113,8 +120,13 @@ def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem
|
|||||||
if not items:
|
if not items:
|
||||||
return items
|
return items
|
||||||
|
|
||||||
# Compute raw engagement scores
|
# Compute raw engagement scores (with top comment quality signal)
|
||||||
eng_raw = [compute_reddit_engagement_raw(item.engagement) for item in items]
|
eng_raw = []
|
||||||
|
for item in items:
|
||||||
|
top_cmt_score = None
|
||||||
|
if item.top_comments:
|
||||||
|
top_cmt_score = item.top_comments[0].score
|
||||||
|
eng_raw.append(compute_reddit_engagement_raw(item.engagement, top_cmt_score))
|
||||||
|
|
||||||
# Normalize engagement to 0-100
|
# Normalize engagement to 0-100
|
||||||
eng_normalized = normalize_to_100(eng_raw)
|
eng_normalized = normalize_to_100(eng_raw)
|
||||||
|
|||||||
@@ -0,0 +1,41 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<style>
|
||||||
|
body { font-family: -apple-system, sans-serif; max-width: 600px; margin: 40px auto; padding: 20px; background: #f5f5f5; }
|
||||||
|
.tweet { background: white; border-radius: 12px; padding: 24px; box-shadow: 0 1px 3px rgba(0,0,0,0.1); white-space: pre-wrap; font-size: 15px; line-height: 1.5; }
|
||||||
|
button { margin-top: 16px; padding: 10px 20px; background: #1d9bf0; color: white; border: none; border-radius: 20px; font-size: 14px; cursor: pointer; }
|
||||||
|
button:hover { background: #1a8cd8; }
|
||||||
|
.copied { background: #00ba7c; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="tweet" id="tweet">/last30days v2.9 is out. Reddit just got a massive upgrade.
|
||||||
|
|
||||||
|
Top comments are now first-class. The best comment on each thread gets 10% scoring weight and shows up like:
|
||||||
|
|
||||||
|
r/ClaudeAI — "Anthropic Released 32 Page Detailed Guide on Building Claude Skills" [1,470pts, 112 comments]
|
||||||
|
💬 Top comment (567 pts): "Can't wait to have Claude read this and explain it to me."
|
||||||
|
|
||||||
|
Reddit's value was always in the comments. Now the skill actually reads them.
|
||||||
|
|
||||||
|
The other big change: smart subreddit discovery. It scores every candidate by frequency × recency × topic-word match and blocklists utility subs that used to pollute results.
|
||||||
|
|
||||||
|
Before: search "Kanye West" → r/AskReddit, r/OutOfTheLoop
|
||||||
|
Now: r/hiphopheads, r/Kanye, r/NFCWestMemeWar
|
||||||
|
|
||||||
|
Shoutout to @ScrapeCreators — one API key now covers Reddit, TikTok, and Instagram. Three sources, one key. Should be cheaper than using an OpenAI key too.</div>
|
||||||
|
<button onclick="copyTweet()">Copy to clipboard</button>
|
||||||
|
<script>
|
||||||
|
function copyTweet() {
|
||||||
|
const text = document.getElementById('tweet').textContent;
|
||||||
|
navigator.clipboard.writeText(text).then(() => {
|
||||||
|
const btn = document.querySelector('button');
|
||||||
|
btn.textContent = 'Copied!';
|
||||||
|
btn.classList.add('copied');
|
||||||
|
setTimeout(() => { btn.textContent = 'Copy to clipboard'; btn.classList.remove('copied'); }, 2000);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Reference in New Issue
Block a user