From 4d35b53eabd2c3b4e4e24c1e3d0905944579b90e Mon Sep 17 00:00:00 2001 From: Matt Van Horn Date: Thu, 5 Mar 2026 18:03:40 -0800 Subject: [PATCH] =?UTF-8?q?docs:=20v2.9.0=20release=20=E2=80=94=20ScrapeCr?= =?UTF-8?q?eators=20Reddit=20default,=20top=20comments,=20smart=20discover?= =?UTF-8?q?y?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 --- CHANGELOG.md | 27 +++++++++++++++++++++ README.md | 62 ++++++++++++++++++++++++++++++++++++++++++------ SKILL.md | 12 +++++----- release-notes.md | 58 +++++++++++++++++++------------------------- 4 files changed, 113 insertions(+), 46 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cb40071..a9320f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [2.9.0] - 2026-03-05 + +### Highlights + +ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display. + +### Added + +- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) β€” keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com` +- Smart subreddit discovery with relevance-weighted scoring: frequency Γ— recency Γ— topic-word match, replacing pure frequency count +- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results +- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)` +- Top comment rendering: `πŸ’¬ Top comment` lines with upvote counts in compact and full report output +- Comment excerpt length increased from 300 β†’ 400 chars; `comment_insights` limit raised from 7 β†’ 10 + +### Changed + +- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` β€” one key now powers Reddit, TikTok, and Instagram +- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) β†’ `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment) +- SKILL.md synthesis instructions updated to emphasize quoting top comments + +### Fixed + +- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics) +- Reddit search no longer requires `OPENAI_API_KEY` β€” ScrapeCreators API handles search directly + ## [2.8.0] - 2026-03-04 ### Highlights @@ -88,6 +114,7 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a Initial public release. Reddit + X search via OpenAI Responses API and xAI API. +[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0 [2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0 [2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0 [1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0 diff --git a/README.md b/README.md index 8f15475..4194836 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,14 @@ -# /last30days v2.8 +# /last30days v2.9 **The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know. +**New in v2.9 β€” ScrapeCreators Reddit + Top Comments + Smart Discovery:** + +Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default β€” one `SCRAPECREATORS_API_KEY` covers Reddit, TikTok, and Instagram (3 sources, 1 key). Smart subreddit discovery finds the right communities automatically, and top comments are elevated with a 10% scoring weight and `πŸ’¬` display with upvote counts. [Details below.](#whats-new-in-v29) + **New in v2.8 β€” Instagram Reels + ScrapeCreators:** -Instagram Reels is now the 8th signal source. TikTok and Instagram both run on [ScrapeCreators](https://scrapecreators.com) β€” one API key covers both. Search any topic and get trending Reels with views, likes, spoken-word transcripts, and hashtags. [Details below.](#whats-new-in-v28) +Instagram Reels is now the 8th signal source. TikTok and Instagram both run on ScrapeCreators β€” one API key covers both. [Details below.](#whats-new-in-v28) **New in V2.5 - dramatically better results:** @@ -31,9 +35,9 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last # Add your API keys (optional if signed in to Codex) mkdir -p ~/.config/last30days cat > ~/.config/last30days/.env << 'EOF' -OPENAI_API_KEY=sk-... # optional if using `codex login` -XAI_API_KEY=xai-... # optional - cookie auth is default for X search -SCRAPECREATORS_API_KEY=... # optional - for TikTok + Instagram (scrapecreators.com) +SCRAPECREATORS_API_KEY=... # Reddit + TikTok + Instagram (one key, all three) β€” scrapecreators.com +OPENAI_API_KEY=sk-... # optional β€” legacy Reddit fallback if using `codex login` +XAI_API_KEY=xai-... # optional β€” cookie auth is default for X search EOF chmod 600 ~/.config/last30days/.env ``` @@ -937,6 +941,50 @@ If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1) --- +## What's New in v2.9 + +### ScrapeCreators Reddit as default + +Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram β€” three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. + +```bash +echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env +``` + +### Smart subreddit discovery + +Subreddit discovery now uses relevance-weighted scoring instead of pure frequency count. Each candidate subreddit is scored by `frequency Γ— recency Γ— topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.). + +| Topic | Before (v2.8) | After (v2.9) | +|-------|---------------|--------------| +| Claude Code skills | Generic programming subs | r/ClaudeAI, r/ClaudeCode, r/openclaw | +| Kanye West | r/AskReddit, r/OutOfTheLoop | r/hiphopheads, r/Kanye, r/NFCWestMemeWar | +| Nano Banana Pro | r/techsupport, r/whatisthisthing | r/GeminiAI, r/nanobanana2pro, r/macbookpro | + +### Top comments elevated + +Top comments now carry a 10% weight in the engagement scoring formula and are displayed prominently with `πŸ’¬` and upvote counts: + +``` +**R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt] + Claude Code creator: In the next version, introducing two new skills + πŸ’¬ Top comment (245 pts): "This is going to change how everyone works with Claude" +``` + +**Updated scoring formula:** `0.50 Γ— log1p(score) + 0.35 Γ— log1p(comments) + 0.05 Γ— (ratioΓ—10) + 0.10 Γ— log1p(top_comment_score)` (was 0.55/0.40/0.05). + +### Beta test results + +| Topic | Time | Threads | Discovered Subreddits | +|-------|------|---------|----------------------| +| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw | +| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye | +| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI | +| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans | +| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro | + +--- + ## What's New in v2.8 ### Instagram Reels as a source @@ -1096,13 +1144,13 @@ Thanks to the contributors who helped shape V2: | Destination | Data Sent | API Key Required | |------------|-----------|-----------------| -| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY | +| `api.scrapecreators.com` | Search query (Reddit + TikTok + Instagram) | SCRAPECREATORS_API_KEY | +| `api.openai.com` | Search query (legacy Reddit fallback) | OPENAI_API_KEY | | `reddit.com` | Thread URLs for enrichment | None (public JSON) | | Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY | | `youtube.com` (via yt-dlp) | Search query | None (public search) | | `hn.algolia.com` | Search query | None (public API) | | `gamma-api.polymarket.com` | Search query | None (public API) | -| `api.scrapecreators.com` | Search query (TikTok + Instagram) | SCRAPECREATORS_API_KEY | | `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY | | `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY | | `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY | diff --git a/SKILL.md b/SKILL.md index bee4f2c..8d41c8a 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,10 +1,10 @@ --- -name: last30daysbeta -version: "2.9-beta" -description: "BETA: Research a topic from the last 30 days with ScrapeCreators Reddit. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts." -argument-hint: 'last30daysbeta AI video tools, last30daysbeta best project management tools' +name: last30days +version: "2.9" +description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts." +argument-hint: 'last30 AI video tools, last30 best project management tools' allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch -homepage: https://github.com/mvanhorn/last30days-skill-private +homepage: https://github.com/mvanhorn/last30days-skill user-invocable: true metadata: clawdbot: @@ -30,7 +30,7 @@ metadata: - prompts --- -# last30days v2.8: Research Any Topic from the Last 30 Days +# last30days v2.9: Research Any Topic from the Last 30 Days Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now. diff --git a/release-notes.md b/release-notes.md index 1414a30..949d048 100644 --- a/release-notes.md +++ b/release-notes.md @@ -1,52 +1,44 @@ The AI world reinvents itself every month. This skill keeps you current. -`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago. +`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. -## Three Headline Features +## Three Headline Features in v2.9 -**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search. +**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram β€” three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure. -**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain. +**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency Γ— recency Γ— topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw β€” not generic programming subs. -**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`. +**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with `πŸ’¬` and upvote counts. Reddit's value is in the comments β€” now the skill surfaces them. -Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies. +Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** β€” no external CLI needed. -## Real Results (verified Feb 15) +## Beta Test Results (v2.9) -| Topic | Reddit | X | YouTube | Web | -|-------|--------|---|---------|-----| -| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages | -| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages | -| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages | -| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages | -| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages | +| Topic | Time | Threads | Discovered Subreddits | +|-------|------|---------|----------------------| +| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw | +| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye | +| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI | +| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans | +| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro | ## What's New ### Added -- Open-class skill with watchlist, briefing, and history modes -- YouTube search + transcript extraction via yt-dlp -- OpenAI Codex CLI compatibility -- Bundled Twitter/X search (vendored Bird GraphQL, MIT) -- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro) -- `--diagnose` flag for source status checking -- `--store` flag for SQLite accumulation -- Conversational first-run experience (NUX) +- ScrapeCreators Reddit backend with keyword search and subreddit discovery +- Smart subreddit discovery with relevance-weighted scoring +- Utility subreddit blocklist (`UTILITY_SUBS`) +- Top comment scoring (10% engagement weight) and prominent rendering +- Comment excerpts increased to 400 chars, insights raised to 10 ### Changed -- Two-phase search architecture (entity-aware drill-down) -- Reddit JSON enrichment for real engagement metrics -- Smarter query construction with auto-retry on 0 results -- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%) -- `--days=N` configurable lookback (thanks @jonthebeef) +- `primaryEnv` β†’ `SCRAPECREATORS_API_KEY` (one key for Reddit, TikTok, Instagram) +- Reddit engagement scoring: `0.55/0.40/0.05` β†’ `0.50/0.35/0.05/0.10` +- SKILL.md synthesis instructions emphasize quoting top comments ### Fixed -- YouTube/Reddit timeout resilience -- Reddit 429 rate limit fail-fast -- Eager import crash in Codex environments -- X search returning 0 results on popular topics -- Windows Unicode crash (thanks @JosephOIbrahim) +- Utility sub noise in subreddit discovery +- Reddit no longer requires `OPENAI_API_KEY` ## New Contributors @@ -70,4 +62,4 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days ``` -30 days of research. 30 seconds of work. Four sources. Zero stale prompts. +30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.