fix: append -raw suffix to saved research filenames

e.g. sam-altman-raw.md instead of sam-altman.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(release): v2.9.4 - move save into Python script, zero post-invitation noise
2026-03-06 16:56:03 -08:00 · 2026-03-06 16:27:53 -08:00 · 2026-03-06 15:50:17 -08:00 · 2026-03-06 15:41:55 -08:00 · 2026-03-06 14:59:06 -08:00 · 2026-03-06 10:34:22 -08:00
28 changed files with 3358 additions and 330 deletions
@@ -1,7 +1,7 @@
 {
  "name": "last30days",
  "description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
-  "version": "2.1.0",
+  "version": "2.9.1",
  "author": {
    "name": "mvanhorn"
  },
@@ -5,6 +5,102 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

+## [2.9.4] - 2026-03-06
+
+### Changed
+
+- Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
+- Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
+- No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research
+
+## [2.9.3] - 2026-03-06
+
+### Fixed
+
+- **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
+- Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)
+
+## [2.9.2] - 2026-03-06
+
+### Fixed
+
+- Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
+- Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
+- Add `📎` footer line for save path instead of verbose confirmation
+
+## [2.9.1] - 2026-03-05
+
+### Highlights
+
+Auto-save research briefings to `~/Documents/Last30Days/` as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
+
+### Added
+
+- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to `~/Documents/Last30Days/{topic-slug}.md` after every run
+- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
+- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
+- Agent mode (`--agent`) also saves research files
+- Brief confirmation after save: "Saved to ~/Documents/Last30Days/{slug}.md"
+
+### Credits
+
+- [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))
+
+## [2.9.0] - 2026-03-05
+
+### Highlights
+
+ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
+
+### Added
+
+- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
+- Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
+- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
+- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
+- Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
+- Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
+
+### Changed
+
+- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
+- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
+- SKILL.md synthesis instructions updated to emphasize quoting top comments
+
+### Fixed
+
+- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
+- Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
+
+## [2.8.0] - 2026-03-04
+
+### Highlights
+
+Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.
+
+### Added
+
+- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
+- `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
+- Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
+- URL-to-name extraction examples in SKILL.md for cleaner web source display
+- `--search=instagram` flag support
+
+### Changed
+
+- TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
+- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
+- SKILL.md version bumped to v2.8
+- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
+- Security section updated: Apify → ScrapeCreators references
+
+### Fixed
+
+- Web stats line showing full URLs instead of plain domain names
+- Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
+- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
+- `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding
+
 ## [2.1.0] - 2026-02-15

 ### Highlights
@@ -59,5 +155,8 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a

 Initial public release. Reddit + X search via OpenAI Responses API and xAI API.

+[2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
+[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
+[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
 [2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
 [1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
@@ -1,10 +1,16 @@
-# /last30days v2.7
+# /last30days v2.9.1

-**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
+**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.

-**New in V2.7 — TikTok as a source:**
+**New in v2.9.1 — Auto-save to ~/Documents/Last30Days/:** Every run now saves the complete briefing as a topic-named `.md` file to your Documents folder. Build a personal research library automatically. Inspired by [@devin_explores](https://x.com/devin_explores).

-TikTok is now the 7th signal source. Search any topic and get viral TikTok videos with views, likes, hashtags, and extracted captions — scored and ranked alongside Reddit, X, and YouTube. Powered by [Apify](https://apify.com) with a BYO API key ($5/month free credits, no credit card). [Details below.](#whats-new-in-v27)
+**New in v2.9 — ScrapeCreators Reddit + Top Comments + Smart Discovery:**
+
+Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default — one `SCRAPECREATORS_API_KEY` covers Reddit, TikTok, and Instagram (3 sources, 1 key). Smart subreddit discovery finds the right communities automatically, and top comments are elevated with a 10% scoring weight and `💬` display with upvote counts. [Details below.](#whats-new-in-v29)
+
+**New in v2.8 — Instagram Reels + ScrapeCreators:**
+
+Instagram Reels is now the 8th signal source. TikTok and Instagram both run on ScrapeCreators — one API key covers both. [Details below.](#whats-new-in-v28)

 **New in V2.5 - dramatically better results:**

@@ -31,9 +37,9 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
 # Add your API keys (optional if signed in to Codex)
 mkdir -p ~/.config/last30days
 cat > ~/.config/last30days/.env << 'EOF'
-OPENAI_API_KEY=sk-...      # optional if using `codex login`
-XAI_API_KEY=xai-...        # optional - cookie auth is default for X search
-APIFY_API_TOKEN=apify_...  # optional - for TikTok (free $5/mo at apify.com)
+SCRAPECREATORS_API_KEY=... # Reddit + TikTok + Instagram (one key, all three) — scrapecreators.com
+OPENAI_API_KEY=sk-...      # optional — legacy Reddit fallback if using `codex login`
+XAI_API_KEY=xai-...        # optional — cookie auth is default for X search
 EOF
 chmod 600 ~/.config/last30days/.env
 ```
@@ -129,7 +135,7 @@ Examples:

 ## What It Does

-1. **Researches** - Scans Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web for discussions from the last 30 days
+1. **Researches** - Scans Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web for discussions from the last 30 days
 2. **Synthesizes** - Identifies patterns, best practices, and what actually works
 3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer

@@ -937,31 +943,70 @@ If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1)

 ---

-## What's New in V2.7
+## What's New in v2.9

-### TikTok as a source
+### ScrapeCreators Reddit as default

-**See what's going viral on TikTok.** Search any topic and get the top TikTok videos with views, likes, hashtags, and extracted captions — scored and ranked alongside all other sources. Cross-source convergence detection catches when the same story trends on TikTok AND Reddit AND X.
-
-Search "Iran Israel" and you get:
- 🎵 TikTok: 12 videos │ 61,618,200 views │ 2,645,694 likes │ 5 with captions
- @suaradotcom: 20.1M views — Iranian missiles striking Tel Aviv
- @itvnews: 14.7M views — Missile getting through Iron Dome
- @bbcnews: 12.5M views — US and Israel struck Iran, killing Khamenei
-
-Search "Leah Halton" and TikTok is the primary signal:
- 🎵 TikTok: 15 videos │ 152.6M views │ 10.9M likes │ 5 with captions
- @looooooooch: 108.8M views — her "Recreation #Inverted" viral hit
- @allyouseeisai: 17.7M views — AI-generated content of her
-
-**Powered by [Apify](https://apify.com)** — sign up for free ($5/month credits, no credit card) and add your token:
+Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search.

 ```bash
-echo 'APIFY_API_TOKEN=apify_api_...' >> ~/.config/last30days/.env
+echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
 ```

-The shared Apify client wrapper is designed for future Facebook and Instagram sources using the same token.
+### Smart subreddit discovery

+Subreddit discovery now uses relevance-weighted scoring instead of pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.).
+
+| Topic | Before (v2.8) | After (v2.9) |
+|-------|---------------|--------------|
+| Claude Code skills | Generic programming subs | r/ClaudeAI, r/ClaudeCode, r/openclaw |
+| Kanye West | r/AskReddit, r/OutOfTheLoop | r/hiphopheads, r/Kanye, r/NFCWestMemeWar |
+| Nano Banana Pro | r/techsupport, r/whatisthisthing | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
+
+### Top comments elevated
+
+Top comments now carry a 10% weight in the engagement scoring formula and are displayed prominently with `💬` and upvote counts:
+
+```
+**R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
+  Claude Code creator: In the next version, introducing two new skills
+  💬 Top comment (245 pts): "This is going to change how everyone works with Claude"
+```
+
+**Updated scoring formula:** `0.50 × log1p(score) + 0.35 × log1p(comments) + 0.05 × (ratio×10) + 0.10 × log1p(top_comment_score)` (was 0.55/0.40/0.05).
+
+### Beta test results
+
+| Topic | Time | Threads | Discovered Subreddits |
+|-------|------|---------|----------------------|
+| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
+| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
+| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
+| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
+| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
+
+---
+
+## What's New in v2.8
+
+### Instagram Reels as a source
+
+**See what creators are posting on Instagram.** Search any topic and get trending Reels with views, likes, spoken-word transcripts, and hashtags — scored and ranked alongside all other sources.
+
+Search "AI tools" and you get:
+- 📸 Instagram: 5 reels │ 1.4M views │ 30K likes │ 3 with transcripts
+- @danmartell: 803K views — "AI tools from 2025 vs 2026"
+- @karimehta05: 112K views — "5 AI Tools I Swear By"
+
+### TikTok + Instagram on ScrapeCreators
+
+Both TikTok and Instagram are powered by [ScrapeCreators](https://scrapecreators.com) — one API key covers both sources. 100 free credits, then pay-as-you-go.
+
+```bash
+echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
+```
+
+**Migrating from Apify?** Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. The old key is no longer used.
 ---

 ## What's New in V2.5
@@ -1101,13 +1146,13 @@ Thanks to the contributors who helped shape V2:

 | Destination | Data Sent | API Key Required |
 |------------|-----------|-----------------|
-| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
+| `api.scrapecreators.com` | Search query (Reddit + TikTok + Instagram) | SCRAPECREATORS_API_KEY |
+| `api.openai.com` | Search query (legacy Reddit fallback) | OPENAI_API_KEY |
 | `reddit.com` | Thread URLs for enrichment | None (public JSON) |
 | Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
 | `youtube.com` (via yt-dlp) | Search query | None (public search) |
 | `hn.algolia.com` | Search query | None (public API) |
 | `gamma-api.polymarket.com` | Search query | None (public API) |
-| `api.apify.com` | Search query (TikTok) | APIFY_API_TOKEN |
 | `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
 | `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
 | `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
@@ -1126,6 +1171,6 @@ Each API key is transmitted only to its respective endpoint. Your OpenAI key is

 ---

-*30 days of research. 30 seconds of work. Seven sources. Zero stale prompts.*
+*30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.*

-*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. TikTok. Web.  - All synthesized into expert answers and copy-paste prompts.*
+*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. TikTok. Instagram. Web. — All synthesized into expert answers and copy-paste prompts.*
@@ -1,7 +1,7 @@
 ---
 name: last30days
-version: "2.7"
-description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
+version: "2.9.2"
+description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
 argument-hint: 'last30 AI video tools, last30 best project management tools'
 allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
 homepage: https://github.com/mvanhorn/last30days-skill
@@ -11,11 +11,11 @@ metadata:
    emoji: "📰"
    requires:
      env:
-        - OPENAI_API_KEY
+        - SCRAPECREATORS_API_KEY
      bins:
        - node
        - python3
-    primaryEnv: OPENAI_API_KEY
+    primaryEnv: SCRAPECREATORS_API_KEY
    files:
      - "scripts/*"
    homepage: https://github.com/mvanhorn/last30days-skill
@@ -30,7 +30,7 @@ metadata:
      - prompts
 ---

-# last30days v2.7: Research Any Topic from the Last 30 Days
+# last30days v2.9.4: Research Any Topic from the Last 30 Days

 Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.

@@ -123,6 +123,8 @@ If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):
 5. **Skip** the follow-up invitation ("I'm now an expert on X...")
 6. **Output** the complete research report and stop - do not wait for further input

+Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir` (handled by the script, no extra tool calls).
+
 Agent mode report format:

 ```
@@ -165,22 +167,24 @@ if [ -z "${SKILL_ROOT:-}" ]; then
  exit 1
 fi

-python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web  # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
+python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web --save-dir=~/Documents/Last30Days  # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
 ```

 Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically takes 1-3 minutes.

 The script will automatically:
 - Detect available API keys
- Run Reddit/X/YouTube/TikTok/Hacker News/Polymarket searches
- Output ALL results including YouTube transcripts, TikTok captions, HN comments, and prediction market odds
+- Run Reddit/X/YouTube/TikTok/Instagram/Hacker News/Polymarket searches
+- Output ALL results including YouTube transcripts, TikTok captions, Instagram captions, HN comments, and prediction market odds

-**Read the ENTIRE output.** It contains SEVEN data sections in this order: Reddit items, X items, YouTube items, TikTok items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
+**Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.

 **YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.

 **TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block.

+**Instagram Reels items in the output look like:** `**{IG_id}** (score:N) @{creator} (date) [N views, N likes]` followed by caption text, URL, and optional transcript. Count them and include them in your synthesis and stats block. Instagram provides unique creator/influencer perspective — weight it alongside TikTok.
+
 ---

 ## STEP 2: DO WEBSEARCH AFTER SCRIPT COMPLETES
@@ -237,9 +241,10 @@ The Judge Agent must:
 2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
 3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
 4. Weight WebSearch sources LOWER (no engagement data)
-4. Identify patterns that appear across ALL sources (strongest signals)
-5. Note any contradictions between sources
-6. Extract the top 3-5 actionable insights
+5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes (shown as `💬 Top comment (N upvotes)`), quote it directly in your synthesis. Reddit's value is in the comments.
+6. Identify patterns that appear across ALL sources (strongest signals)
+7. Note any contradictions between sources
+8. Extract the top 3-5 actionable insights

 7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.

@@ -343,21 +348,23 @@ CITATION RULE: Cite sources sparingly to prove research is real.

 CITATION PRIORITY (most to least preferred):
 1. @handles from X — "per @handle" (these prove the tool's unique value)
-2. r/subreddits from Reddit — "per r/subreddit"
+2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, prefer quoting top comments over just the thread title)
 3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
 4. TikTok creators — "per @creator on TikTok" (viral/trending signal)
-5. HN discussions — "per HN" or "per hn/username" (developer community signal)
-6. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
-7. Web sources — ONLY when Reddit/X/YouTube/TikTok/HN/Polymarket don't cover that specific fact
+5. Instagram creators — "per @creator on Instagram" (influencer/creator signal)
+6. HN discussions — "per HN" or "per hn/username" (developer community signal)
+7. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
+8. Web sources — ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact

 The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
 When both a web article and an X post cover the same fact, cite the X post.

-URL FORMATTING: NEVER paste raw URLs in the output.
+URL FORMATTING: NEVER paste raw URLs anywhere in the output — not in synthesis, not in stats, not in sources.
 - **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
 - **GOOD:** "per Rolling Stone"
- **GOOD:** "per Complex"
-Use the publication name, not the URL. The user doesn't need links — they need clean, readable text.
+- **BAD stats line:** `🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/...`
+- **GOOD stats line:** `🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee`
+Use the publication/site name, not the URL. The user doesn't need links — they need clean, readable text.

 **BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
 **GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per @honest30bgfan_"
@@ -399,6 +406,7 @@ KEY PATTERNS from the research:
 ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
 ├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
 ├─ 🎵 TikTok: {N} videos │ {N} views │ {N} likes │ {N} with captions
+├─ 📸 Instagram: {N} reels │ {N} views │ {N} likes │ {N} with captions
 ├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
 ├─ 📊 Polymarket: {N} markets │ {short summary of up to 5 most relevant market odds, e.g. "Championship: 12%, #1 Seed: 28%, Big 12: 64%, vs Kansas: 71%"}
 ├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name
@@ -406,7 +414,18 @@ KEY PATTERNS from the research:
 ---
 ```

-**WebSearch citation note:** The WebSearch tool requires source citation. This requirement is satisfied by naming the web sources on the 🌐 Web: line above (plain names, no URLs — URLs wrap badly in terminals). Do NOT append a separate "Sources:" section after the invitation.
+**🌐 Web: line — how to extract site names from URLs:**
+Strip the protocol, path, and `www.` — use the recognizable publication name:
+- `https://later.com/blog/instagram-reels-trends/` → **Later**
+- `https://socialbee.com/blog/instagram-trends/` → **SocialBee**
+- `https://buffer.com/resources/instagram-algorithms/` → **Buffer**
+- `https://www.cnn.com/2026/02/22/tech/...` → **CNN**
+- `https://medium.com/the-ai-studio/...` → **Medium**
+- `https://radicaldatascience.wordpress.com/...` → **Radical Data Science**
+List as comma-separated plain names: `Later, SocialBee, Buffer, CNN, Medium`
+
+**⚠️ WebSearch citation — ALREADY SATISFIED. DO NOT ADD A SOURCES SECTION.**
+The WebSearch tool mandates source citation. That requirement is FULLY satisfied by the source names on the 🌐 Web: line above. Do NOT append a separate "Sources:" section at the end of your response. Do NOT list URLs anywhere. The 🌐 Web: line IS your citation. Nothing more is needed.

 **CRITICAL: Omit any source line that returned 0 results.** Do NOT show "0 threads", "0 stories", "0 markets", or "(no results this cycle)". If a source found nothing, DELETE that line entirely - don't include it at all.
 NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
@@ -481,7 +500,7 @@ For `/last30days war in Iran` (NEWS):

 ## WAIT FOR USER'S RESPONSE

-After showing the stats summary with your invitation, **STOP and wait** for the user to respond.
+**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`.

 ---

@@ -570,7 +589,7 @@ After delivering a prompt, end with:
 ```
 ---
 📚 Expert in: {TOPIC} for {TARGET_TOOL}
-📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} TikTok videos ({sum} views) + {n} HN stories ({sum} points) + {n} web pages
+📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} TikTok videos ({sum} views) + {n} Instagram reels ({sum} views) + {n} HN stories ({sum} points) + {n} web pages

 Want another prompt? Just tell me what you're creating next.
 ```
@@ -580,15 +599,17 @@ Want another prompt? Just tell me what you're creating next.
 ## Security & Permissions

 **What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
+- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit search, subreddit discovery, and comment enrichment (requires SCRAPECREATORS_API_KEY — same key as TikTok + Instagram)
+- Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
 - Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
 - Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
 - Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
 - Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Sends search queries to Apify API (`api.apify.com`) for TikTok search and caption extraction (requires APIFY_API_TOKEN, free tier: $5/month credits)
+- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (same SCRAPECREATORS_API_KEY as Reddit, PAYG after 100 free credits)
 - Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
 - Fetches public Reddit thread data from `reddit.com` for engagement metrics
 - Stores research findings in local SQLite database (watchlist mode only)
+- Saves research briefings as .md files to ~/Documents/Last30Days/

 **What this skill does NOT do:**
 - Does not post, like, or modify content on any platform
@@ -597,7 +618,7 @@ Want another prompt? Just tell me what you're creating next.
 - Does not log, cache, or write API keys to output files
 - Does not send data to any endpoint not listed above
 - Hacker News and Polymarket sources are always available (no API key, no binary dependency)
- TikTok source requires APIFY_API_TOKEN (sign up at apify.com for free $5/month credits, no CC)
+- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (same key covers both; 100 free credits, then PAYG)
 - Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output

 **Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
@@ -0,0 +1,111 @@
+---
+title: Fix SKILL.md Argument Flag Forwarding
+type: fix
+status: completed
+date: 2026-03-03
+---
+
+# Fix SKILL.md Argument Flag Forwarding
+
+## Overview
+
+`"$ARGUMENTS"` in SKILL.md wraps the entire user input in double quotes, making argparse treat flags like `--store` as part of the topic string instead of CLI flags.
+
+## Problem
+
+SKILL.md line 168:
+```bash
+python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
+```
+
+`$ARGUMENTS` is a Claude Code template variable replaced via string substitution before bash runs. The double quotes cause word-joining:
+
+- User types: `/last30days AI video tools --store`
+- Claude Code expands to: `python3 script.py "AI video tools --store" --emit=compact`
+- argparse sees: `topic="AI video tools --store"`, `--store` never parsed
+
+## Proposed Solution
+
+Two coordinated changes:
+
+### 1. Remove quotes around `$ARGUMENTS` in SKILL.md
+
+**File:** `SKILL.md:168`
+
+```bash
+# Before:
+python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
+
+# After:
+python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --no-native-web
+```
+
+Now bash word-splits the expansion: `python3 script.py AI video tools --store --emit=compact`
+
+### 2. Change argparse `topic` from `nargs="?"` to `nargs="*"`
+
+**File:** `scripts/last30days.py:1040`
+
+```python
+# Before:
+parser.add_argument("topic", nargs="?", help="Topic to research")
+# args.topic = "AI video tools" (single string) or None
+
+# After:
+parser.add_argument("topic", nargs="*", help="Topic to research")
+# args.topic = ["AI", "video", "tools"] (list) or []
+```
+
+Then immediately after `parser.parse_args()` (line 1124), join the list back to a string:
+
+```python
+args = parser.parse_args()
+args.topic = " ".join(args.topic) if args.topic else None
+```
+
+**Why this works for both invocation styles:**
+
+| Invocation | argparse receives | topic result |
+|---|---|---|
+| `script.py AI video tools --store` (Claude Code) | `["AI", "video", "tools"]` + `--store` | `"AI video tools"` |
+| `script.py "AI video tools" --store` (direct CLI) | `["AI video tools"]` + `--store` | `"AI video tools"` |
+| `script.py --store` (no topic) | `[]` + `--store` | `None` |
+
+### 3. Document missing flags in SKILL.md Options section
+
+**File:** `SKILL.md:223-227`
+
+Add after the existing `--deep` line:
+
+```
+- `--store` -> Persist findings to SQLite database for later querying
+- `--search=SOURCES` -> Comma-separated source filter (e.g., `--search=reddit,hn`)
+- `--include-web` -> Include general web search alongside primary sources
+- `--diagnose` -> Show source availability diagnostics and exit
+- `--timeout=SECS` -> Global timeout in seconds (default: 180, quick: 90, deep: 300)
+```
+
+Note: `--sort-x` was listed in the issue but does not exist in the Python argparse. Skip it.
+
+## Acceptance Criteria
+
+- [x] `/last30days AI video tools --store` correctly passes `--store` to Python script
+- [x] `/last30days AI video tools` still works (multi-word topic without flags)
+- [x] Direct CLI: `python3 last30days.py "AI video tools" --store` still works
+- [x] `--diagnose`, `--search=reddit,hn`, `--timeout=120` all forward correctly
+- [x] All 5 missing flags documented in SKILL.md Options section
+- [x] Existing tests pass (`python3 -m pytest tests/`)
+
+## Files Changed
+
+| File | Change |
+|---|---|
+| `SKILL.md:168` | Remove quotes around `$ARGUMENTS` |
+| `scripts/last30days.py:1040` | `nargs="?"` -> `nargs="*"` |
+| `scripts/last30days.py:1124` | Add `args.topic = " ".join(args.topic) if args.topic else None` |
+| `SKILL.md:223-227` | Add 5 missing flags to Options section |
+
+## Sources
+
+- GitHub issue: https://github.com/mvanhorn/last30days-skill/issues/36
+- Reporter: @nicolefinateri
@@ -0,0 +1,156 @@
+# refactor: Replace Apify with pay-as-you-go TikTok API
+
+**Type:** refactor
+**Date:** 2026-03-03
+**Status:** Draft
+
+## Problem
+
+Apify requires a monthly subscription even for low-volume usage. We need a true pay-as-you-go TikTok data API that returns structured video data (views, likes, comments, shares, author, hashtags, captions) from keyword search.
+
+## Current Architecture
+
+Our TikTok integration makes **exactly 2 API calls per /last30days invocation**:
+
+1. **Search call** — keyword search, returns 10-40 videos with engagement metrics
+2. **Caption enrichment call** — fetches spoken-word subtitles for top 3-8 videos
+
+We consume these fields from the response:
+- `id`, `text` (description), `webVideoUrl`, `authorMeta.name`
+- `playCount`, `diggCount`, `commentCount`, `shareCount`
+- `hashtags[].name`, `videoMeta.duration`
+- `createTimeISO` or `createTime` (date)
+- `subtitleText` / `subtitles` (captions, secondary call)
+
+Files involved:
+- `scripts/lib/tiktok.py` — search, parse, relevance scoring, caption fetching
+- `scripts/lib/apify_client_wrapper.py` — shared Apify client (also designed for future FB/IG)
+- `scripts/lib/schema.py` — `TikTokItem` dataclass
+- `scripts/lib/normalize.py` — `normalize_tiktok()`
+- `scripts/last30days.py` — orchestrator (calls `tiktok.search_and_enrich()`)
+
+## Why Most of Those Alternatives Won't Work
+
+The services in the user's table (Spider, Zyte, Scrappey, Serper.dev) are **general web scrapers** — they return raw HTML, not structured TikTok data. We'd have to build our own HTML parser, handle anti-bot protections, and reverse-engineer TikTok's response format. That's a completely different (and fragile) approach.
+
+What we need is a **TikTok-specific API** that returns structured JSON with engagement metrics from keyword search.
+
+## Alternatives Evaluated
+
+### RECOMMENDED: ScrapeCreators — Best True Pay-As-You-Go
+
+| Attribute | Details |
+|-----------|---------|
+| **TikTok structured data** | Yes — 19 dedicated endpoints including keyword search |
+| **Pricing model** | True PAYG — buy credits, credits never expire |
+| **Cost at our volume** | ~$0.60/month ($10 buys 5,000 credits, lasts 16-33 months) |
+| **Free tier** | 100-10,000 free credits on signup (no credit card) |
+| **Python SDK** | No dedicated SDK — simple REST API (`requests.get()`) |
+| **Search endpoint** | "Search by Keyword" and "Top Search" |
+| **Risk** | Newer service, limited track record |
+
+**Why it's the best fit:** $10 literally lasts over a year at our volume. No subscription, no expiring credits. The lack of a Python SDK is irrelevant — it's a single `requests.get()` call.
+
+### Runner-up: EnsembleData — Best SDK, But Subscription
+
+| Attribute | Details |
+|-----------|---------|
+| **TikTok structured data** | Full — 15+ endpoints, all engagement metrics |
+| **Pricing model** | Monthly subscription ($100/mo after 7-day trial) |
+| **Cost at our volume** | $0 during trial (50 units/day), $100/mo after |
+| **Free tier** | 50 units/day for 7 days, no CC required |
+| **Python SDK** | Yes — `pip install ensembledata` |
+| **Search endpoint** | "Keyword Search" returns ~20 posts/call (1 unit) |
+| **Risk** | $100/mo is overkill for 5-10 searches/day |
+
+**Verdict:** Best data quality and SDK, but $100/mo is absurd for our ~150-300 requests/month. Same subscription problem as Apify.
+
+### Backup: tikwm.com — Free but Risky
+
+| Attribute | Details |
+|-----------|---------|
+| **TikTok structured data** | Likely yes (needs live testing) |
+| **Pricing model** | Completely free, no API key required |
+| **Cost at our volume** | $0 |
+| **Free tier** | 5,000 requests/day |
+| **Python SDK** | Community wrappers (damirTAG/TikTok-Module, kittenbark/tikwm) |
+| **Search endpoint** | `https://www.tikwm.com/api/feed/search?keywords=TERM&count=20` |
+| **Risk** | Unaffiliated third-party, could disappear anytime, no SLA |
+
+**Verdict:** Great for development/testing. Too risky as sole production backend. Could be a zero-cost fallback.
+
+### Not Recommended
+
+| Service | Why Not |
+|---------|---------|
+| **TikAPI** | $50-189/mo subscription — same problem as Apify |
+| **Bright Data** | $499/mo minimum — enterprise pricing |
+| **davidteather/TikTok-Api** | Video search is broken, requires Playwright, fragile |
+| **Spider, Zyte, Scrappey** | Generic scrapers — return raw HTML, no TikTok structure |
+| **Piloterr** | No TikTok endpoints, subscription only |
+
+## Recommended Approach
+
+### Option A: ScrapeCreators as primary (Recommended)
+
+- [ ] Sign up for ScrapeCreators, get free credits
+- [ ] Test the "Search by Keyword" endpoint to verify it returns all required fields
+- [ ] Refactor `tiktok.py` to use ScrapeCreators REST API instead of Apify actor
+- [ ] Keep `apify_client_wrapper.py` for future FB/IG (or refactor to generic wrapper)
+- [ ] Update `.env` config: `SCRAPECREATORS_API_KEY` replaces `APIFY_API_TOKEN` for TikTok
+- [ ] Update README, SKILL.md installation instructions
+- [ ] Buy $10 credits after confirming it works
+
+### Option B: tikwm.com as primary (Zero cost, higher risk)
+
+- [ ] Test tikwm.com search endpoint to verify response schema
+- [ ] If it returns engagement metrics, implement as primary backend
+- [ ] Add ScrapeCreators as paid fallback when tikwm fails
+- [ ] No API key required — simplest user setup
+
+### Option C: Keep Apify, document the subscription requirement
+
+- [ ] Update README to clarify Apify requires a paid plan
+- [ ] Add note about the free $5/mo credits tier (if it still works without subscription)
+- [ ] Ship as-is with clear billing expectations
+
+## Implementation Plan (Option A)
+
+### Phase 1: Validate ScrapeCreators API
+
+- [ ] Sign up and get API key
+- [ ] Test keyword search endpoint: `GET /tiktok/search?keyword={topic}&count=20`
+- [ ] Verify response contains: video ID, play count, likes, comments, shares, author, hashtags, date, description
+- [ ] Test caption/subtitle availability (or confirm description text is sufficient)
+
+### Phase 2: Swap the Backend
+
+- [ ] Create `scripts/lib/scrapecreators_client.py` (simple REST wrapper, ~40 lines)
+- [ ] Refactor `tiktok.py:search_tiktok()` to call ScrapeCreators instead of Apify
+- [ ] Map ScrapeCreators response fields to our existing item dict format
+- [ ] Refactor `tiktok.py:fetch_captions()` — check if ScrapeCreators provides subtitles, else use description text only
+- [ ] Update `env.py` to read `SCRAPECREATORS_API_KEY` (keep `APIFY_API_TOKEN` for backward compat)
+- [ ] Update `scripts/lib/ui.py` spinner messages if needed
+
+### Phase 3: Update Docs & Ship
+
+- [ ] Update README.md installation section (new API key)
+- [ ] Update SKILL.md
+- [ ] Update `~/.config/last30days/.env` locally
+- [ ] Run full test suite
+- [ ] Commit and push
+
+## Acceptance Criteria
+
+- [ ] TikTok search returns structured data with views, likes, comments, shares
+- [ ] No subscription required — pay-as-you-go only
+- [ ] Cost < $1/month at normal usage (5-10 searches/day)
+- [ ] Existing test suite passes with new backend
+- [ ] Graceful degradation when API key is missing (same behavior as today)
+
+## Open Questions
+
+1. Does ScrapeCreators' keyword search support date filtering, or do we filter post-API like we do with Apify?
+2. Does ScrapeCreators return subtitle/caption data, or just video descriptions?
+3. What's the response time? Apify actors took 30-120 seconds. REST APIs should be faster.
+4. Should we keep Apify as a fallback backend (user configures one or the other)?
@@ -0,0 +1,210 @@
+# feat: Add Instagram and Facebook Sources via ScrapeCreators API
+
+**Date:** 2026-03-04
+**Type:** Enhancement
+**Priority:** Instagram first, Facebook conditional ("if it's good")
+
+## Summary
+
+Add Instagram Reels and Facebook as new research sources in last30days, using the same ScrapeCreators REST API already powering TikTok. Instagram is the primary target; Facebook is a follow-on if the pattern works well.
+
+Both sources share the existing `SCRAPECREATORS_API_KEY` — no new API keys needed.
+
+## Approach
+
+Replicate the TikTok integration pattern exactly. Each source follows the same 8-step pipeline:
+
+```
+tiktok.py pattern → instagram.py (new) → facebook.py (new, conditional)
+```
+
+## ScrapeCreators API Endpoints
+
+### Instagram
+
+| Endpoint | Path | Params | Credits | Notes |
+|----------|------|--------|---------|-------|
+| **Search Reels** | `GET /v1/instagram/reels/search` | `keyword`, pagination | 1 per 10 reels, max 60/req | Keyword search via Google (IG search requires login). V2 also available. |
+| **Transcript** | `GET /v2/instagram/media/transcript` | `url` | 1 | Returns `{transcripts: [{id, shortcode, text}]}`. Videos <2min only. |
+| **Comments** | `GET /v2/instagram/post/comments` | `url`, `cursor` | 1 | Returns `{comments: [{id, text, created_at, user}]}`. 100-300 per call. |
+| **User Reels** | `GET /v1/instagram/user/reels` | `handle` or `user_id`, `max_id`, `trim` | 1 | All reels from a profile. Response: `{items: [...], paging_info}` |
+
+**Primary search strategy:** `/v1/instagram/reels/search` with keyword param for topic search. This is the analog to TikTok's `/search/keyword`.
+
+**Response fields per reel item:**
+- `pk` / `code` (shortcode) — reel ID
+- `taken_at` — unix timestamp
+- `play_count` / `ig_play_count` — views
+- `like_count` — likes
+- `comment_count` — comments
+- `video_duration` — seconds
+- `has_audio` — boolean
+- `user` object — username, full_name, is_verified, profile_pic_url
+- `caption` object — text content
+- Media URLs for thumbnails and video versions
+
+### Facebook
+
+| Endpoint | Path | Params | Credits | Notes |
+|----------|------|--------|---------|-------|
+| **Profile Posts** | `GET /v1/facebook/profile/posts` | `url` or `pageId`, `cursor` | 1 | Returns 3 posts at a time with engagement |
+| **Profile Reels** | `GET /v1/facebook/profile/reels` | similar | 1 | 10 reels at a time |
+| **Post** | `GET /v1/facebook/post` | `url` | 1 | Single post/reel by URL |
+| **Transcript** | `GET /v1/facebook/post/transcript` | `url` | 1 | Video transcript, <2min |
+| **Comments** | `GET /v1/facebook/post/comments` | `url`, `feedback_id` | 1 | Post/reel comments |
+
+**Facebook limitation:** No keyword search endpoint. Only profile-based scraping (3 posts at a time). This makes Facebook significantly less useful for topic-based research vs. Instagram's keyword search.
+
+**Response fields per post:**
+- `id` — post ID
+- `text` — post content
+- `url` / `permalink` — post URL
+- `author` — `{name, short_name, id}`
+- `reactionCount` — total reactions
+- `commentCount` — comments
+- `videoViewCount` — video views (if applicable)
+- `publishTime` — unix timestamp
+- `topComments` — array of `{id, text, publishTime, author}`
+
+## Implementation Plan
+
+### Phase 1: Instagram Source (Primary)
+
+#### 1.1 Create `scripts/lib/instagram.py`
+- [x] Copy structure from `scripts/lib/tiktok.py`
+- [x] Change `SCRAPECREATORS_BASE` to `"https://api.scrapecreators.com"`
+- [x] Implement `search_instagram()` → calls `/v1/instagram/reels/search`
+  - Params: `keyword=core_topic`
+  - Parse response `items` array
+  - Extract: `pk`/`code` as video_id, `taken_at` as date, `play_count`/`like_count`/`comment_count` as engagement, `user.username` as author, `caption.text` as text
+  - Build URL: `https://www.instagram.com/reel/{code}`
+  - Reuse `_extract_core_subject()`, `_compute_relevance()`, `_tokenize()` from tiktok.py (or factor into shared util)
+  - Apply date range filter, sort by views descending
+- [x] Implement `fetch_captions()` → calls `/v2/instagram/media/transcript`
+  - For top N items (per depth config), fetch transcript
+  - Response: `{transcripts: [{id, shortcode, text}]}`
+  - Fallback to caption text if transcript unavailable
+  - Truncate to 500 words
+- [x] Implement `search_and_enrich()` → combines search + captions
+- [x] Implement `parse_instagram_response()` → returns `response.get("items", [])`
+- [x] Reuse shared helpers: `_sc_headers()`, `_log()`, `_clean_webvtt()`, `DEPTH_CONFIG`, `STOPWORDS`, `SYNONYMS`
+
+**Key difference from TikTok:** Instagram response uses `play_count`/`like_count`/`comment_count` directly (no `statistics` wrapper), `user.username` (not `author.unique_id`), `caption.text` (not `desc`), `taken_at` (not `create_time`), `code` shortcode for URL construction.
+
+#### 1.2 Add `InstagramItem` to `scripts/lib/schema.py`
+- [x] Add dataclass mirroring `TikTokItem` structure:
+  ```python
+  @dataclass
+  class InstagramItem:
+      id: str              # "IG1", "IG2", ...
+      text: str            # caption text
+      url: str             # https://www.instagram.com/reel/{code}
+      author_name: str     # Instagram handle
+      date: Optional[str]  # YYYY-MM-DD from taken_at
+      date_confidence: str # "high"
+      engagement: Optional[Engagement]  # views, likes, num_comments
+      caption_snippet: str  # transcript or caption text
+      hashtags: List[str]   # extracted from caption
+      relevance: float
+      why_relevant: str
+      subs: SubScores
+      score: int
+      cross_refs: List[str]
+  ```
+
+#### 1.3 Add normalization to `scripts/lib/normalize.py`
+- [x] Add `normalize_instagram_items()` function
+  - Assign IDs as `IG1`, `IG2`, ...
+  - Map engagement: `views=play_count`, `likes=like_count`, `num_comments=comment_count`
+  - Set `date_confidence="high"` (unix timestamp)
+
+#### 1.4 Add scoring to `scripts/lib/score.py`
+- [x] Add `compute_instagram_engagement_raw()` — same formula as TikTok:
+  `0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)`
+  Views dominate on Instagram Reels just like TikTok.
+- [x] Add `score_instagram_items()` — same weights: 45% relevance, 25% recency, 30% engagement
+
+#### 1.5 Add dedup to `scripts/lib/dedupe.py`
+- [x] Add `dedupe_instagram()` — same as `dedupe_tiktok()`, calls `dedupe_items()` with 0.7 threshold
+- [x] Update `get_item_text()` to handle `InstagramItem`
+- [x] Update `_get_cross_source_text()` for cross-source linking
+- [x] Add `IG` prefix to cross-ref detection in `cross_source_link()`
+
+#### 1.6 Add rendering to `scripts/lib/render.py`
+- [x] Add Instagram section in `render_compact()` — same pattern as TikTok block (lines 251-285)
+  - Show: score, @author, date, views/likes, caption snippet, hashtags, why_relevant
+- [x] Update data freshness check to include `instagram_recent`
+- [x] Update stats footer to include Instagram count
+- [x] Add `'IG'` to cross-ref source name mapping
+
+#### 1.7 Add `Report.instagram` field to `scripts/lib/schema.py`
+- [x] Add `instagram: List[InstagramItem]` and `instagram_error: str` to `Report` dataclass
+
+#### 1.8 Integrate into `scripts/last30days.py` orchestrator
+- [x] Add `"instagram"` to `VALID_SEARCH_SOURCES`
+- [x] Add `import` for `instagram` module in `scripts/lib/`
+- [x] Add `is_instagram_available()` check in `env.py` — reuse `SCRAPECREATORS_API_KEY` (same key as TikTok)
+- [x] Add `get_instagram_token()` in `env.py` — same as `get_tiktok_token()`, returns `SCRAPECREATORS_API_KEY`
+- [x] Add `_search_instagram()` helper in orchestrator (mirrors `_search_tiktok()`)
+- [x] Add Instagram to the thread pool executor block
+- [x] Add Instagram timeout config (same as TikTok: 90/120/150s for quick/default/deep)
+- [x] Wire through pipeline: normalize → filter → score → sort → dedupe → cross-link → report
+- [x] Add Instagram to `progress.show_complete()` and UI spinner
+
+#### 1.9 Add to watchlist extraction in `scripts/watchlist.py`
+- [x] Add Instagram findings loop in `_run_topic()` (mirrors TikTok block at lines 204-213)
+
+#### 1.10 Update README.md
+- [x] Add Instagram to the sources list
+- [x] Note that `SCRAPECREATORS_API_KEY` covers both TikTok and Instagram
+
+### Phase 2: Facebook Source (Conditional)
+
+**Recommendation: SKIP Facebook for now.** Here's why:
+
+1. **No keyword search endpoint** — Facebook only offers profile-based scraping (`/profile/posts` returns 3 posts at a time). Can't search by topic.
+2. **Low relevance for topic research** — Without keyword search, we'd need to know specific Facebook pages to scrape, which defeats the purpose of automated topic discovery.
+3. **Poor ROI** — 3 posts per API call is very limited compared to Instagram's 60 reels per search.
+4. **Same API key** — If Facebook search is added later, it's trivial to add since it shares `SCRAPECREATORS_API_KEY`.
+
+If the user still wants Facebook, the implementation would follow the same pattern but would need a different discovery strategy (e.g., hardcoded page list per topic, or using the Ad Library search for commercial topics).
+
+## Files to Create/Modify
+
+| File | Action | Description |
+|------|--------|-------------|
+| `scripts/lib/instagram.py` | **CREATE** | Instagram search + transcript via ScrapeCreators |
+| `scripts/lib/schema.py` | MODIFY | Add `InstagramItem` dataclass, add `instagram` to `Report` |
+| `scripts/lib/normalize.py` | MODIFY | Add `normalize_instagram_items()` |
+| `scripts/lib/score.py` | MODIFY | Add `compute_instagram_engagement_raw()`, `score_instagram_items()` |
+| `scripts/lib/dedupe.py` | MODIFY | Add `dedupe_instagram()`, update text extractors |
+| `scripts/lib/render.py` | MODIFY | Add Instagram render section, update stats |
+| `scripts/lib/env.py` | MODIFY | Add `is_instagram_available()`, `get_instagram_token()` |
+| `scripts/last30days.py` | MODIFY | Add Instagram to orchestrator pipeline |
+| `scripts/watchlist.py` | MODIFY | Add Instagram findings extraction |
+| `README.md` | MODIFY | Add Instagram to sources list |
+
+## Shared Code Opportunity
+
+`_extract_core_subject()`, `_compute_relevance()`, `_tokenize()`, `STOPWORDS`, `SYNONYMS`, and `DEPTH_CONFIG` are duplicated between `tiktok.py` and the new `instagram.py`. Two options:
+
+1. **Copy-paste** (simpler, matches current pattern) — each source module is self-contained
+2. **Extract to shared module** (cleaner) — move to `scripts/lib/search_utils.py`
+
+**Recommendation:** Copy-paste for now to match existing pattern. Refactor later if a third ScrapeCreators source is added.
+
+## Testing Strategy
+
+- [x] Run `python3 scripts/lib/instagram.py` with test keyword (if standalone test added)
+- [x] Run `python3 scripts/last30days.py "instagram reels trends" --search=instagram` to test isolated
+- [x] Run full multi-source: `python3 scripts/last30days.py "AI tools" --search=reddit,instagram`
+- [x] Verify JSON output: `--emit=json` includes `instagram` key
+- [x] Verify watchlist extraction works with Instagram findings
+- [x] Check credit usage is reasonable (1 credit per 10 reels search + 1 per transcript)
+
+## Credits Budget
+
+Per research run with Instagram at `default` depth:
+- Search: 1 credit (per 10 reels, returns up to 20) ≈ 2 credits
+- Transcripts: 5 credits (max_captions=5 at default depth)
+- **Total: ~7 credits per topic** (vs TikTok ~6 credits)
@@ -0,0 +1,147 @@
+---
+title: "feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration"
+type: enhancement
+status: pending
+date: 2026-03-04
+---
+
+# feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration
+
+## Summary
+
+Ship everything from the last sprint as one combined GitHub release: TikTok's migration from Apify to ScrapeCreators (already committed) + Instagram Reels as the 8th source (uncommitted) + SKILL.md URL regression fixes. Version bump to v2.8.
+
+## What's Shipping
+
+### 1. Instagram Reels — 8th source (NEW)
+- Search Instagram Reels by keyword via ScrapeCreators `/v1/instagram/reels/search`
+- Spoken-word transcript extraction via `/v2/instagram/media/transcript`
+- Full pipeline: search → normalize → score → dedupe → cross-link → render
+- Shares `SCRAPECREATORS_API_KEY` with TikTok (no new API key needed)
+- ~7 credits per topic at default depth
+
+### 2. TikTok — Apify → ScrapeCreators migration (ALREADY COMMITTED)
+- Replaced Apify dependency with ScrapeCreators API
+- Same functionality, different backend
+- No more `APIFY_API_TOKEN` — uses `SCRAPECREATORS_API_KEY`
+
+### 3. SKILL.md quality fixes
+- Instagram added to stats template, citation priority, data sections, footer
+- URL regression fix: explicit URL-to-name extraction rules, stronger anti-Sources instruction
+- Security section updated: Apify → ScrapeCreators
+
+## Release Checklist
+
+### Pre-commit: Update docs
+
+- [ ] **README.md** — Update for v2.8:
+  - [ ] Change title from "v2.7" to "v2.8"
+  - [ ] Add "New in v2.8" banner: Instagram Reels + ScrapeCreators migration
+  - [ ] Update installation section: `APIFY_API_TOKEN` → `SCRAPECREATORS_API_KEY`
+  - [ ] Add Instagram to the "How it works" flow description (line 132)
+  - [ ] Update TikTok section: replace Apify references with ScrapeCreators
+  - [ ] Add Instagram section (after TikTok section, ~line 963)
+  - [ ] Update API endpoints table: `api.apify.com` → `api.scrapecreators.com`, add Instagram endpoints
+  - [ ] Update closing tagline to include Instagram
+  - [ ] Remove "The shared Apify client wrapper is designed for future Facebook and Instagram sources" (line 963) — Instagram is here now
+
+- [ ] **CHANGELOG.md** — Add v2.8.0 entry:
+  ```
+  ## [2.8.0] - 2026-03-04
+
+  ### Highlights
+
+  Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements.
+
+  ### Added
+
+  - Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction
+  - Instagram items in SKILL.md stats template, citation priority, and output footer
+  - URL-to-name extraction examples in SKILL.md for cleaner web source display
+
+  ### Changed
+
+  - TikTok backend migrated from Apify to ScrapeCreators API (same key covers TikTok + Instagram)
+  - `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
+  - SKILL.md version bumped to v2.8
+  - WebSearch citation instruction strengthened to prevent trailing Sources: blocks
+
+  ### Fixed
+
+  - Web stats line showing full URLs instead of plain domain names (regression from v2.7)
+  - Trailing "Sources:" block appearing after invitation (WebSearch tool mandate conflict)
+  - Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
+  ```
+
+- [ ] **SKILL.md** frontmatter — Bump version from "2.7" to "2.8"
+
+- [ ] **SKILL.md** description — Add Instagram to the description field
+
+### Commit & tag
+
+- [ ] Stage all changes: modified files + 4 new files (`scripts/lib/instagram.py`, 3 plan docs)
+- [ ] Commit with message:
+  ```
+  feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
+
+  - Add Instagram Reels as 8th research source via ScrapeCreators API
+  - Migrate TikTok from Apify to ScrapeCreators (same API key)
+  - Add SCRAPECREATORS_API_KEY config (replaces APIFY_API_TOKEN)
+  - Fix web stats URL regression and trailing Sources: block
+  - Fix Instagram/TikTok not running in --search=instagram web-only path
+  - Update SKILL.md with Instagram stats, citations, URL formatting rules
+  ```
+- [ ] Create tag: `git tag -a v2.8.0 -m "v2.8.0: Instagram Reels + ScrapeCreators"`
+- [ ] Push: `git push origin main --tags`
+
+### Post-push: GitHub release
+
+- [ ] Create GitHub release via `gh release create v2.8.0`:
+  ```
+  ## What's New in v2.8
+
+  **Instagram Reels** is now the 8th signal source. Search any topic and get trending Instagram Reels with views, likes, and spoken-word transcripts — scored and ranked alongside Reddit, X, YouTube, TikTok, HN, Polymarket, and the web.
+
+  **TikTok migrated to ScrapeCreators API.** Same functionality, new backend. Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. One key now covers both TikTok and Instagram.
+
+  ### Setup
+
+  Sign up at [scrapecreators.com](https://scrapecreators.com) (100 free credits, then PAYG) and add your key:
+
+  ```bash
+  echo 'SCRAPECREATORS_API_KEY=your_key' >> ~/.config/last30days/.env
+  ```
+
+  ### Breaking Change
+
+  - `APIFY_API_TOKEN` is no longer used. Replace with `SCRAPECREATORS_API_KEY`.
+
+  ### Bug Fixes
+
+  - Fixed web source URLs leaking into stats display
+  - Fixed Instagram/TikTok not running when used with `--search=` flag
+  ```
+
+### Post-release: Sync
+
+- [ ] Run `bash scripts/sync.sh` to deploy to all 4 skill destinations
+- [ ] Verify Instagram works in a fresh `/last30days` session
+
+## Files Modified (this release)
+
+| File | Status | Description |
+|------|--------|-------------|
+| `scripts/lib/instagram.py` | NEW | Instagram search + transcript via ScrapeCreators |
+| `scripts/lib/schema.py` | MODIFIED | InstagramItem dataclass, Report.instagram field |
+| `scripts/lib/normalize.py` | MODIFIED | normalize_instagram_items() |
+| `scripts/lib/score.py` | MODIFIED | score_instagram_items(), engagement formula |
+| `scripts/lib/dedupe.py` | MODIFIED | dedupe_instagram(), cross-source linking |
+| `scripts/lib/render.py` | MODIFIED | Instagram render section, stats |
+| `scripts/lib/env.py` | MODIFIED | is_instagram_available(), get_instagram_token() |
+| `scripts/lib/ui.py` | MODIFIED | Instagram spinner messages |
+| `scripts/last30days.py` | MODIFIED | Instagram in orchestrator pipeline |
+| `scripts/watchlist.py` | MODIFIED | Instagram findings extraction |
+| `SKILL.md` | MODIFIED | Instagram in stats/citations/footer, URL fixes |
+| `README.md` | TO UPDATE | Instagram section, ScrapeCreators migration |
+| `CHANGELOG.md` | TO UPDATE | v2.8.0 entry |
+| `docs/plans/*.md` | NEW (3) | Plan documents for this work |
@@ -0,0 +1,135 @@
+---
+title: "fix: Web sources showing full URLs instead of plain domain names"
+type: fix
+status: pending
+date: 2026-03-04
+---
+
+# fix: Web Sources Showing Full URLs Instead of Plain Domain Names
+
+## Problem
+
+Two regressions in the `/last30days` skill output:
+
+### Regression 1: Full URLs on the Web stats line
+
+The `🌐 Web:` stats line is showing full URLs instead of plain source names:
+
+**BAD (current — "Instagram Trends" run):**
+```
+├─ 🌐 Web: 10+ pages — https://later.com/blog/instagram-reels-trends/,
+https://socialbee.com/blog/instagram-trends/,
+https://buffer.com/resources/instagram-algorithms/,
+https://metricool.com/instagram-trends/,
+https://napoleoncat.com/blog/instagram-reels-trends/
+```
+
+**GOOD (expected):**
+```
+├─ 🌐 Web: 10+ pages — Later, SocialBee, Buffer, Metricool, NapoleonCat
+```
+
+### Regression 2: Trailing Sources: block with full URLs
+
+A separate `Sources:` section appears at the bottom of the response with full URLs:
+
+```
+Sources:
+- https://www.heyorca.com/blog/instagram-social-news
+- https://socialbee.com/blog/instagram-updates/
+- https://buffer.com/resources/instagram-algorithms/
+- https://www.cnn.com/2026/02/22/tech/social-media-addiction-trial-tobacco-moment
+```
+
+This was fixed in commit `82efa61` (2026-03-02) but is regressing intermittently.
+
+## Root Cause
+
+The SKILL.md instructions at lines 219-221, 404, and 409 already say the right thing:
+- Line 404: `├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name`
+- Line 409: `"plain names, no URLs — URLs wrap badly in terminals"`
+- Lines 219-221: "DO NOT output a separate Sources: block"
+
+But the model ignores these because:
+
+1. **The template `Source Name` is too abstract.** The model sees WebSearch results with full URLs and doesn't know how to extract a human-friendly name from `https://later.com/blog/instagram-reels-trends/`. It needs explicit examples showing the transformation.
+
+2. **The WebSearch system mandate still wins.** The WebSearch tool's built-in instruction (`"you MUST include a Sources: section"`) outcompetes the skill's instruction. The current countermeasure (line 409) works sometimes but not reliably — it needs to be stronger and repeated.
+
+3. **No explicit extraction rule.** The model needs a concrete rule for turning URLs into names: strip protocol, strip path, strip `www.`, capitalize.
+
+## Proposed Solution
+
+**SKILL.md edits only. No Python changes.**
+
+### Fix 1: Add explicit URL-to-name examples in the stats template (line 404 area)
+
+After the stats template block, add concrete examples showing the transformation:
+
+```
+**🌐 Web: line formatting:**
+- Extract the SITE NAME from each URL — strip protocol, path, and "www."
+- Use the publication's proper name when recognizable
+- Examples:
+  - https://later.com/blog/instagram-reels-trends/ → "Later"
+  - https://socialbee.com/blog/instagram-trends/ → "SocialBee"
+  - https://buffer.com/resources/instagram-algorithms/ → "Buffer"
+  - https://www.cnn.com/2026/02/22/tech/... → "CNN"
+  - https://medium.com/the-ai-studio/... → "Medium"
+  - https://radicaldatascience.wordpress.com/... → "Radical Data Science"
+- NEVER paste the URL itself. ONLY the site name as plain text.
+- Separate names with commas: "Later, SocialBee, Buffer, CNN, Medium"
+```
+
+### Fix 2: Strengthen the anti-Sources instruction (line 409 area)
+
+Replace the current single-paragraph note with a louder, more explicit instruction:
+
+```
+**⚠️ WebSearch citation requirement — ALREADY SATISFIED above.**
+The WebSearch tool mandates source citation. That requirement is FULLY satisfied
+by the source names on the 🌐 Web: line above. Do NOT append a separate
+"Sources:" section at the end of your response. Do NOT list URLs anywhere in
+your output. The 🌐 Web: line IS your citation. You're done.
+```
+
+### Fix 3: Add a negative example in the URL FORMATTING section (line 356 area)
+
+Extend the existing BAD/GOOD examples to cover the stats line specifically:
+
+```
+URL FORMATTING: NEVER paste raw URLs anywhere in the output.
+- BAD: "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
+- GOOD: "per Rolling Stone"
+- BAD stats line: "🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/..."
+- GOOD stats line: "🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee"
+```
+
+### Fix 4: Update Security section (line 588, 600)
+
+While we're in SKILL.md, update the stale Apify references to ScrapeCreators:
+- Line 588: Change Apify reference to ScrapeCreators for TikTok
+- Line 600: Update TikTok requirement note
+- Add Instagram source mention
+
+## Files to Modify
+
+| File | Action | Description |
+|------|--------|-------------|
+| `SKILL.md` | MODIFY | Strengthen URL formatting rules, add examples, fix Apify refs |
+
+## Implementation Steps
+
+- [x] Add URL-to-name extraction examples after stats template (after line 407)
+- [x] Strengthen anti-Sources instruction (replace line 409)
+- [x] Add BAD/GOOD stats line example to URL FORMATTING section (around line 356)
+- [x] Update Security section: Apify → ScrapeCreators, add Instagram (lines 588, 600)
+- [x] Run `bash scripts/sync.sh` to deploy to all destinations
+- [ ] Test with `/last30days Instagram Trends` — confirm plain names, no trailing Sources:
+
+## Acceptance Criteria
+
+- [ ] `🌐 Web:` line shows plain names only (e.g., "Later, SocialBee, Buffer")
+- [ ] No `Sources:` section appears at the bottom of the response
+- [ ] No raw URLs appear anywhere in the output (synthesis, stats, or footer)
+- [ ] Security section reflects current source stack (ScrapeCreators, not Apify)
@@ -0,0 +1,152 @@
+---
+title: "feat: Auto-save research results to ~/Documents/Last30Days/"
+type: feat
+status: completed
+date: 2026-03-05
+---
+
+# feat: Auto-save research results to ~/Documents/Last30Days/
+
+## Overview
+
+Every time the last30days skill completes a research run, automatically save the full briefing - inquiry, synthesis, stats, and follow-up suggestions - as a topic-named `.md` file in `~/Documents/Last30Days/`. Inspired by how users like @devin_explores are already manually saving results to build a personal research library (see screenshot - 17 topic files in a `Last30Days` Finder folder, each 9-34 KB).
+
+## Problem Statement / Motivation
+
+The skill's most valuable output - the assistant's synthesized "What I learned" briefing with stats and citations - only exists in the conversation. Once the session ends, it's gone. Users like @devin_explores work around this by manually copying output into .md files. Meanwhile, the Python script already writes raw data to `~/.local/share/last30days/out/`, but:
+
+1. It overwrites on every run (no history)
+2. It only contains pre-synthesis data (scored items), not the assistant's expert briefing
+3. It's in a hidden dot-directory users don't naturally browse
+
+The feature makes saving automatic and puts files where users expect them - the Documents folder, visible in Finder/file explorer.
+
+## Proposed Solution
+
+Add a **Write tool step in SKILL.md** after the synthesis/stats/invitation block that saves the complete briefing to `~/Documents/Last30Days/{topic-slug}.md`. This is a SKILL.md-only change (no Python script modifications needed) because the content to save is the assistant's synthesized output, which only exists in the SKILL.md flow.
+
+### Why SKILL.md, not the Python script
+
+The Python script (`last30days.py`) runs first and produces raw scored items. The assistant then synthesizes these into the "What I learned" briefing, stats block, and invitation. The synthesis is the valuable part - it's what @devin_explores is saving. The script can't produce this because it runs before synthesis happens.
+
+### File naming
+
+Convert the TOPIC variable to a kebab-case slug for the filename:
+- "Claude Code best practices" -> `claude-code-best-practices.md`
+- "best rap songs 2026" -> `best-rap-songs-2026.md`
+- "nano banana 2 prompting guide" -> `nano-banana-2-prompting-guide.md`
+
+This matches the screenshot pattern exactly (e.g., `anthropic-claude-code-best-practices.md`, `seedance-video-prompting-guide.md`).
+
+If a file with the same slug already exists, append a date suffix: `claude-code-best-practices-2026-03-05.md`. This handles re-researching the same topic without overwriting previous results.
+
+### File content
+
+The saved .md file should contain the complete research output in this order:
+
+```markdown
+# {TOPIC}
+
+> Researched {date} | Query type: {QUERY_TYPE} | Target tool: {TARGET_TOOL or "general"}
+
+## What I learned
+
+{The full synthesis section - topics, patterns, citations}
+
+## Stats
+
+{The full stats box with source counts and engagement}
+
+## Follow-up suggestions
+
+{The 2-3 specific suggestions from the invitation block}
+
+---
+*Generated by [last30days](https://github.com/mvanhorn/last30days-skill) v2.9*
+```
+
+### Implementation location in SKILL.md
+
+Insert a new section between the current "LAST - Invitation" display and the "WAIT FOR USER'S RESPONSE" section. The Write tool call happens silently - no user prompt, no opt-in. Just save and briefly confirm.
+
+## Technical Considerations
+
+- **Cross-platform paths**: `~/Documents/` exists on macOS and most Linux desktops. On systems where it doesn't exist, `mkdir -p` handles creation. Windows WSL users get it too.
+- **Permissions**: The Write tool in Claude Code can write to `~/Documents/` without issues. No sandbox concerns since this is the user's own Documents folder.
+- **Filename sanitization**: Strip special characters, collapse whitespace to hyphens, lowercase. Keep it simple - no need for a library, just basic string ops in the SKILL.md instructions.
+- **File size**: Based on the screenshot (9-34 KB files), the synthesis output is well within reasonable bounds.
+- **No opt-out flag needed initially**: This is the default behavior. If users complain, a `--no-save` flag can be added later. Start with always-on since the screenshot proves users want this.
+
+## Acceptance Criteria
+
+- [x] Running `/last30days {topic}` creates `~/Documents/Last30Days/{topic-slug}.md` automatically
+- [x] File contains: title, date, query metadata, full synthesis, stats block, follow-up suggestions
+- [x] Filename is kebab-case slug of the topic (e.g., `claude-code-skills-guide.md`)
+- [x] Duplicate topics get a date suffix instead of overwriting
+- [x] Directory `~/Documents/Last30Days/` is created automatically if it doesn't exist
+- [x] A brief confirmation line appears after the stats (e.g., "Saved to ~/Documents/Last30Days/claude-code-skills-guide.md")
+- [x] Agent mode (`--agent`) also saves the file
+- [x] No changes to the Python script - this is purely a SKILL.md addition
+
+## Implementation Steps
+
+### Step 1: Add save instructions to SKILL.md
+
+Insert a new section after the invitation block (after line ~496, before "WAIT FOR USER'S RESPONSE" at line ~499):
+
+**New section in `SKILL.md`:**
+
+```markdown
+## Save Research to Documents
+
+After displaying the invitation, save the complete research briefing:
+
+1. Generate the filename from TOPIC:
+   - Lowercase the topic
+   - Replace spaces and special characters with hyphens
+   - Remove consecutive hyphens
+   - Trim to 60 characters max
+   - Example: "Claude Code Best Practices" -> "claude-code-best-practices"
+
+2. Check if file already exists. If so, append today's date:
+   - "claude-code-best-practices.md" exists -> use "claude-code-best-practices-2026-03-05.md"
+
+3. Use the Write tool to save to ~/Documents/Last30Days/{slug}.md with this content:
+   - H1 title: the TOPIC
+   - Metadata line: date, QUERY_TYPE, TARGET_TOOL
+   - Full "What I learned" synthesis (everything you just displayed)
+   - Full stats block
+   - Follow-up suggestions from the invitation
+   - Footer with skill attribution
+
+4. Confirm briefly: "Saved to ~/Documents/Last30Days/{slug}.md"
+```
+
+### Step 2: Update agent mode section
+
+The `--agent` mode section (line ~116) skips interactive elements but should still save. Add a note that agent mode saves the file with the same logic.
+
+### Step 3: Update Security & Permissions section
+
+Add to the "What this skill does" list (line ~599):
+- "Saves research briefings as .md files to ~/Documents/Last30Days/"
+
+## Success Metrics
+
+- Users accumulate a browsable library of .md research files in their Documents folder
+- No more manual copy-paste workflow to save results
+- Files are immediately findable in Finder/file explorer search
+
+## Dependencies & Risks
+
+- **Low risk**: Write tool is already in the skill's `allowed-tools` list
+- **Low risk**: ~/Documents/ is a standard, user-owned directory
+- **Edge case**: If the skill is interrupted mid-run (before synthesis), no file is saved - this is correct behavior since there's nothing to save yet
+- **Edge case**: Very long topics could produce unwieldy filenames - the 60-char truncation handles this
+
+## Sources & References
+
+- Screenshot from @devin_explores showing manual .md file library in ~/Documents/Last30Days/
+- Current output pipeline: `scripts/lib/render.py:798` (`write_outputs()`) writes to `~/.local/share/last30days/out/`
+- SKILL.md synthesis flow: lines 275-496 (internalize research -> show summary -> invitation)
+- Existing `--emit` modes: `scripts/last30days.py:1700` (`output_result()`)
@@ -0,0 +1,255 @@
+# feat: Reddit ScrapeCreators v2 — Improvements from Beta Testing
+
+**Date:** 2026-03-05
+**Type:** Enhancement
+**Version:** v2.9 → v2.9.1-beta (or v3.0-beta if shipping to public)
+**Branch:** `feat/reddit-scrapecreators` (continue existing branch)
+
+---
+
+## Summary
+
+Three focused improvements to the Reddit ScrapeCreators integration based on 5 full-pipeline tests ("Claude Code skills", "Kanye West", "Anthropic odds", "best rap songs lately", "Nano Banana Pro prompting"):
+
+1. **Elevate top Reddit comments** — give weight to the wittiest/highest-voted comment in scoring and rendering
+2. **Improve subreddit discovery** — tune heuristic so ambiguous queries find discussion subs, not utility subs
+3. **Make ScrapeCreators the default recommended Reddit method** — update onboarding, SKILL.md metadata, and env.py messaging
+
+---
+
+## Problem Statement
+
+### 1. Comments are undervalued
+- ScrapeCreators returns real comment data with scores, but top comments only appear as `Insights:` text under each Reddit item
+- The top comment (often the funniest/cleverest reply) gets no special treatment — it's just one of 3 comment excerpts
+- Reddit's value IS the comments — upvoted replies are the distilled crowd wisdom
+- Currently `comment_insights` are truncated at 150 chars and only 3 are shown per item in compact output
+- No scoring bonus for posts that have high-quality comment threads
+
+### 2. Subreddit discovery picks wrong subs for ambiguous queries
+- "best rap songs lately" discovered `r/NameThatSong` and `r/findthatsong` (utility subs for identifying songs) instead of discussion subs like `r/hiphopheads` or `r/rap`
+- "Kanye West" picked `r/ConcertsIndia_` as second sub — tangential at best
+- The current heuristic is pure frequency count on `subreddit` field from global results, with no relevance weighting
+- Utility/meta subs often dominate because the same query matches many "help me find X" posts
+
+### 3. Onboarding still suggests OpenAI as the primary Reddit method
+- SKILL.md metadata says `primaryEnv: OPENAI_API_KEY` and `requires.env: [OPENAI_API_KEY]`
+- The web-only mode banner mentions "OPENAI_API_KEY or codex login → Reddit threads"
+- `env.py` error messages direct users to OpenAI for Reddit access
+- ScrapeCreators is cheaper ($0.012 vs $0.03-0.10), faster (17s vs 60-90s), returns real data, and shares a key with TikTok + Instagram
+- New users should be told: "Get a SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, all three)"
+
+---
+
+## Implementation Plan
+
+### Task 1: Elevate Top Comments in Scoring and Rendering
+
+**Goal:** Give Reddit posts a scoring bonus when they have highly-engaged comment threads, and render the #1 comment with special treatment.
+
+**Files to modify:**
+- `scripts/lib/reddit.py` — enrich with `top_comment_score` metadata
+- `scripts/lib/score.py` — add comment quality bonus to Reddit scoring
+- `scripts/lib/render.py` — render top comment with special formatting
+- `scripts/lib/schema.py` — add `top_comment_excerpt` field to RedditItem (optional, may just use existing `top_comments[0]`)
+
+#### 1a. Comment enrichment improvements (`scripts/lib/reddit.py`)
+
+- [x] In `enrich_with_comments()`, after sorting comments by score, tag the item with:
+  - `top_comment_excerpt`: The highest-scored comment's body (up to 200 chars)
+  - `top_comment_score`: The upvote count of the #1 comment
+  - `top_comment_author`: Author of the #1 comment
+- [x] Increase comment excerpt length from 300 → 400 chars for top comment only (funny/clever comments need more room)
+- [x] Increase `comment_insights` limit from 7 → 10 (we have the data, show it)
+- [x] For posts with enriched comments, store the comment count ratio: `top_comment_score / post_score` — a high ratio means the comment outshines the post (Reddit gold)
+
+#### 1b. Scoring bonus for comment quality (`scripts/lib/score.py`)
+
+- [x] In `compute_reddit_engagement_raw()`, add a comment quality signal:
+  - Current formula: `0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)`
+  - New formula: `0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)`
+  - This gives a ~10% weight to comment quality, slightly reducing post score and comment count weights
+  - Posts where the community engaged deeply (high top-comment score) rank higher
+- [x] Need to pass `top_comment_score` through the engagement data — either:
+  - Option A: Add `top_comment_score` to `schema.Engagement` (cleanest)
+  - Option B: Read from `item.top_comments[0].score` during scoring (no schema change)
+  - **Recommend Option B** to avoid schema bloat — scoring can peek at `top_comments`
+
+#### 1c. Render top comment prominently (`scripts/lib/render.py`)
+
+- [x] In `render_compact()` Reddit section, after the `Insights:` block, add a "Top Comment:" line for items that have top_comments:
+  ```
+  **R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
+    Claude Code creator: In the next version, introducing two new skills
+    https://www.reddit.com/r/ClaudeAI/comments/...
+    *Reddit global search*
+    💬 Top comment (247 upvotes): "So are they /batch migrating to Rust? :)"
+    Insights:
+      - TL;DR generated automatically after 50 comments...
+      - He's /batch migrating code daily?..
+  ```
+- [x] Only show `💬 Top comment` for items where `top_comments[0].score >= 10` (skip low-engagement comments)
+- [x] Truncate at 200 chars with `...` if needed
+- [x] Also update `render_full_report()` to include the top comment prominently
+
+#### 1d. Update SKILL.md synthesis instructions
+
+- [x] In the "Judge Agent: Synthesize All Sources" section, add guidance:
+  ```
+  5b. For Reddit: Pay special attention to top comments — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes, quote it directly in your synthesis. Reddit's value is in the comments.
+  ```
+- [x] In the citation priority list, add: "When citing Reddit, prefer quoting top comments over just the thread title"
+
+---
+
+### Task 2: Improve Subreddit Discovery Heuristic
+
+**Goal:** Find topical discussion subs rather than utility/meta subs.
+
+**Files to modify:**
+- `scripts/lib/reddit.py` — improve `discover_subreddits()` logic
+
+#### 2a. Add relevance-weighted subreddit scoring
+
+- [x] Replace pure frequency count with a weighted score:
+  ```python
+  def discover_subreddits(results, topic, max_subs=5):
+      core = _extract_core_subject(topic)
+      core_words = set(core.lower().split())
+
+      scores = Counter()
+      for post in results:
+          sub = post.get("subreddit", "")
+          if not sub:
+              continue
+
+          # Base: frequency count
+          base = 1.0
+
+          # Bonus: subreddit name contains a core topic word
+          sub_lower = sub.lower()
+          if any(w in sub_lower for w in core_words if len(w) > 2):
+              base += 2.0
+
+          # Penalty: known utility/meta subreddits
+          if sub_lower in UTILITY_SUBS:
+              base *= 0.3
+
+          # Bonus: post engagement (high-engagement posts = better sub)
+          ups = post.get("ups") or post.get("score", 0)
+          if ups > 100:
+              base += 0.5
+
+          scores[sub] += base
+
+      return [sub for sub, _ in scores.most_common(max_subs)]
+  ```
+
+#### 2b. Define utility/meta subreddit blocklist
+
+- [x] Add a small set of subs that are "find X for me" or "identify X" rather than discussion:
+  ```python
+  UTILITY_SUBS = frozenset({
+      'namethatsong', 'findthatsong', 'tipofmytongue',
+      'whatisthissong', 'helpmefind', 'whatisthisthing',
+      'whatsthissong', 'findareddit', 'subredditdrama',
+  })
+  ```
+- [x] Keep this small and focused — don't over-filter. Only penalty (0.3x), not ban.
+
+#### 2c. Try secondary query for subreddit discovery
+
+- [x] If the first global search returns <3 unique subreddits above threshold, run a second global search with just `{core subject}` (stripped even further) to cast a wider net for subreddit frequencies
+- [x] This helps niche topics where the full query is too specific
+
+---
+
+### Task 3: Make ScrapeCreators the Default Reddit Method
+
+**Goal:** New users should be guided to ScrapeCreators first, not OpenAI.
+
+**Files to modify:**
+- `SKILL.md` — metadata section, onboarding banner, security section
+- `scripts/lib/env.py` — error messages and missing key guidance
+- `scripts/lib/render.py` — web-only mode banner
+
+#### 3a. Update SKILL.md metadata
+
+- [x] Change `primaryEnv: OPENAI_API_KEY` → `primaryEnv: SCRAPECREATORS_API_KEY`
+- [x] Change `requires.env: [OPENAI_API_KEY]` → `requires.env: [SCRAPECREATORS_API_KEY]`
+- [x] Keep OPENAI_API_KEY mentioned but as optional/legacy
+
+#### 3b. Update web-only mode banner (`scripts/lib/render.py`)
+
+- [x] Change the current banner:
+  ```
+  - `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments
+  ```
+  To:
+  ```
+  - `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views
+  - `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)
+  ```
+
+#### 3c. Update env.py messaging
+
+- [x] In `get_missing_keys()`, when Reddit is missing, suggest ScrapeCreators first:
+  - Current: returns `'reddit'` which triggers "Add OPENAI_API_KEY or run codex login" in SKILL.md
+  - Add a helper: `get_setup_hint(missing)` that returns:
+    - For 'reddit': `"Add SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, ~$0.002/search)"`
+    - For 'x': `"Add XAI_API_KEY for X posts"`
+    - For 'all': `"Add SCRAPECREATORS_API_KEY (Reddit+TikTok+Instagram) and XAI_API_KEY (X)"`
+
+#### 3d. Update Security & Permissions section in SKILL.md
+
+- [x] Add ScrapeCreators Reddit to the security section:
+  ```
+  - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit, TikTok, and Instagram search (requires SCRAPECREATORS_API_KEY)
+  ```
+- [x] Move "Sends search queries to OpenAI's Responses API for Reddit discovery" to a "Legacy:" subsection
+- [x] Update "Reddit" description in `allowed-tools` or tags if needed
+
+#### 3e. Update render.py coverage note
+
+- [x] In `render_compact()`, the coverage note for `reddit-only` currently says "Add an xAI key"
+- [x] When ScrapeCreators is the active Reddit source, no need to mention OpenAI at all
+
+---
+
+## Acceptance Criteria
+
+- [x] Top Reddit comment is rendered with `💬` prefix and upvote count for enriched posts
+- [x] Posts with high top-comment scores rank slightly higher (visible in score differences)
+- [x] "best rap songs lately" discovers at least one discussion sub (r/hiphopheads, r/rap, r/Music, etc.) instead of only utility subs
+- [x] SKILL.md `primaryEnv` is `SCRAPECREATORS_API_KEY`
+- [x] Web-only mode banner recommends ScrapeCreators first
+- [x] All 5 test topics still pass (run same tests as before)
+- [x] No regression in OpenAI fallback path
+
+---
+
+## Files Changed (Summary)
+
+| File | Change |
+|------|--------|
+| `scripts/lib/reddit.py` | Improve `discover_subreddits()` with relevance weighting, add utility sub penalties, enhance `enrich_with_comments()` top comment metadata |
+| `scripts/lib/score.py` | Add 10% comment quality weight to Reddit engagement formula |
+| `scripts/lib/render.py` | Add `💬 Top comment` line to compact output, update web-only banner |
+| `scripts/lib/env.py` | Add `get_setup_hint()`, update missing key messaging |
+| `SKILL.md` | Change `primaryEnv`, update onboarding banner, add comment synthesis guidance, update security section |
+
+---
+
+## Cost Impact
+
+No cost increase. Same number of API calls per search. The changes are all in local logic (scoring, rendering, discovery heuristic).
+
+---
+
+## Testing Plan
+
+1. Re-run the same 5 test topics from beta testing
+2. Verify top comments appear with `💬` in output
+3. Verify "best rap songs lately" discovers at least one discussion subreddit
+4. Verify `--diagnose` output recommends ScrapeCreators
+5. Verify OpenAI fallback still works (unset SCRAPECREATORS_API_KEY, set OPENAI_API_KEY)
@@ -0,0 +1,83 @@
+---
+title: "fix: Eliminate save-related output noise after research"
+type: fix
+status: active
+date: 2026-03-06
+---
+
+# fix: Eliminate save-related output noise after research
+
+## Problem Statement
+
+After research completes, the auto-save feature adds 3-5 lines of unwanted output below the clean invitation block. Every approach tried so far has made things worse:
+
+| Version | Approach | Lines added | Side effects |
+|---------|----------|-------------|--------------|
+| v2.9.1 | `Write` tool | ~5 | Shows "Wrote 43 lines to..." |
+| v2.9.2 | `run_in_background: true` | ~5 | Background callback triggers 2-4 min cogitation, model hallucinates fake "Human:" messages and generates unsolicited multi-paragraph responses |
+| v2.9.3 | Foreground `cat >` heredoc | ~3 | Shows `(No output)`, still triggers 2-4 min cogitation |
+
+**Current v2.9.3 noise (3 lines):**
+```
+⏺ Bash(mkdir -p ~/Documents/Last30Days && cat > ...)
+  ⎿  (No output)
+✻ Churned for 2m 38s
+```
+
+The root cause is that ANY tool call after the invitation creates unavoidable Claude Code UI chrome, and the model cogitates for minutes regardless of foreground vs background.
+
+## Proposed Solutions
+
+### Option A: Remove auto-save entirely (Recommended)
+
+Remove the entire "Save Research to Documents" section from SKILL.md. The research lives in the conversation. Zero extra tool calls, zero extra lines, zero cogitation.
+
+**Changes to `SKILL.md`:**
+- Delete the "Save Research to Documents" section (~45 lines)
+- Update agent mode reference (line 126) - remove save mention
+- Update security section (line 612) - change "Saves research briefings" to past tense or conditional
+- Remove `📎` footer line from invitation format
+- Simplify "WAIT FOR USER'S RESPONSE" section
+
+**What users lose:** Auto-saved .md files in `~/Documents/Last30Days/`
+**What users gain:** Clean output ending exactly at the invitation block
+
+### Option B: Move save into Python script
+
+Add `--save-dir` flag to `last30days.py`. The script saves its raw output during its existing Bash call (which already runs). Zero extra tool calls.
+
+**Changes:**
+- `scripts/last30days.py` - add `--save-dir` argument, write raw output to file at end of execution
+- `SKILL.md` - add `--save-dir=~/Documents/Last30Days` to the script invocation, remove save section
+
+**What users lose:** Saved file contains raw research data, not Claude's synthesis
+**What users gain:** Zero extra lines, data is still preserved
+
+### Option C: Opt-in save via follow-up command
+
+Remove auto-save. Add a note to the invitation: "Say 'save' to save this research." When user says "save", run the Bash heredoc then.
+
+**Changes to `SKILL.md`:**
+- Delete auto-save section
+- Add "save" as a recognized intent in "WHEN USER RESPONDS" section
+- Save logic only runs when explicitly requested
+
+**What users lose:** Nothing - save is still available on demand
+**What users gain:** Clean output by default, save when they want it
+
+## Acceptance Criteria
+
+- [ ] Research output ends at the invitation block with zero tool calls after it
+- [ ] No `(No output)` line visible after invitation
+- [ ] No multi-minute cogitation after invitation
+- [ ] No hallucinated fake user messages
+- [ ] Version bumped to v2.9.4
+- [ ] CHANGELOG updated
+- [ ] Synced to ~/.claude, ~/.agents, ~/.codex via sync.sh
+
+## Context
+
+- Source: `/Users/mvanhorn/last30days-skill-private/SKILL.md`
+- Deployed to: `~/.claude/skills/last30days/SKILL.md`
+- Sync command: `bash scripts/sync.sh`
+- Current version: v2.9.3
@@ -1,52 +1,48 @@
 The AI world reinvents itself every month. This skill keeps you current.

-`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago.
+`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.

-## Three Headline Features
+## What's New in v2.9.1

-**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search.
+**Auto-save to ~/Documents/Last30Days/.** Every run now saves the complete research briefing - synthesis, stats, and follow-up suggestions - as a topic-named `.md` file to your Documents folder. Build a personal research library without lifting a finger. Inspired by [@devin_explores](https://x.com/devin_explores) who was already doing this manually.

-**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
+## Three Headline Features in v2.9

-**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
+**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram - three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.

-Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies.
+**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency x recency x topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw - not generic programming subs.

-## Real Results (verified Feb 15)
+**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with upvote counts. Reddit's value is in the comments - now the skill surfaces them.

-| Topic | Reddit | X | YouTube | Web |
-|-------|--------|---|---------|-----|
-| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages |
-| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages |
-| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages |
-| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages |
-| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages |
+Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** - no external CLI needed.
+
+## Beta Test Results (v2.9)
+
+| Topic | Time | Threads | Discovered Subreddits |
+|-------|------|---------|----------------------|
+| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
+| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
+| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
+| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
+| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |

 ## What's New

 ### Added
- Open-class skill with watchlist, briefing, and history modes
- YouTube search + transcript extraction via yt-dlp
- OpenAI Codex CLI compatibility
- Bundled Twitter/X search (vendored Bird GraphQL, MIT)
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro)
- `--diagnose` flag for source status checking
- `--store` flag for SQLite accumulation
- Conversational first-run experience (NUX)
+- ScrapeCreators Reddit backend with keyword search and subreddit discovery
+- Smart subreddit discovery with relevance-weighted scoring
+- Utility subreddit blocklist (`UTILITY_SUBS`)
+- Top comment scoring (10% engagement weight) and prominent rendering
+- Comment excerpts increased to 400 chars, insights raised to 10

 ### Changed
- Two-phase search architecture (entity-aware drill-down)
- Reddit JSON enrichment for real engagement metrics
- Smarter query construction with auto-retry on 0 results
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
- `--days=N` configurable lookback (thanks @jonthebeef)
+- `primaryEnv` → `SCRAPECREATORS_API_KEY` (one key for Reddit, TikTok, Instagram)
+- Reddit engagement scoring: `0.55/0.40/0.05` → `0.50/0.35/0.05/0.10`
+- SKILL.md synthesis instructions emphasize quoting top comments

 ### Fixed
- YouTube/Reddit timeout resilience
- Reddit 429 rate limit fail-fast
- Eager import crash in Codex environments
- X search returning 0 results on popular topics
- Windows Unicode crash (thanks @JosephOIbrahim)
+- Utility sub noise in subreddit discovery
+- Reddit no longer requires `OPENAI_API_KEY`

 ## New Contributors

@@ -70,4 +66,4 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
 git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
 ```

-30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
+30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.
@@ -38,13 +38,13 @@ _child_pids: set = set()
 _child_pids_lock = threading.Lock()

 TIMEOUT_PROFILES = {
-    "quick":   {"global": 90,  "future": 30, "reddit_future": 60,  "youtube_future": 60,  "tiktok_future": 90,   "hackernews_future": 30,  "polymarket_future": 15,  "http": 15, "enrich_per": 8,  "enrich_total": 30, "enrich_max_items": 10},
-    "default": {"global": 180, "future": 60, "reddit_future": 90,  "youtube_future": 90,  "tiktok_future": 120,  "hackernews_future": 60,  "polymarket_future": 30,  "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
-    "deep":    {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "tiktok_future": 150,  "hackernews_future": 90,  "polymarket_future": 45,  "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
+    "quick":   {"global": 90,  "future": 30, "reddit_future": 60,  "youtube_future": 60,  "tiktok_future": 90,   "instagram_future": 90,   "hackernews_future": 30,  "polymarket_future": 15,  "http": 15, "enrich_per": 8,  "enrich_total": 30, "enrich_max_items": 10},
+    "default": {"global": 180, "future": 60, "reddit_future": 90,  "youtube_future": 90,  "tiktok_future": 120,  "instagram_future": 120,  "hackernews_future": 60,  "polymarket_future": 30,  "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
+    "deep":    {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "tiktok_future": 150,  "instagram_future": 150,  "hackernews_future": 90,  "polymarket_future": 45,  "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
 }

 # Valid source names for the --search flag
-VALID_SEARCH_SOURCES = {"reddit", "x", "hn", "youtube", "tiktok", "polymarket", "web"}
+VALID_SEARCH_SOURCES = {"reddit", "x", "hn", "youtube", "tiktok", "instagram", "polymarket", "web"}


 def parse_search_flag(search_str: str) -> set:
@@ -140,12 +140,14 @@ from lib import (
    models,
    normalize,
    openai_reddit,
+    reddit,
    reddit_enrich,
    render,
    schema,
    score,
    ui,
    tiktok,
+    instagram,
    websearch,
    xai_x,
    youtube_yt,
@@ -170,19 +172,51 @@ def _search_reddit(
    depth: str,
    mock: bool,
 ) -> tuple:
-    """Search Reddit via OpenAI (runs in thread).
+    """Search Reddit (runs in thread).
+
+    Uses ScrapeCreators when SCRAPECREATORS_API_KEY is available (preferred).
+    Falls back to OpenAI Responses API otherwise.

    Returns:
-        Tuple of (reddit_items, raw_openai, error)
+        Tuple of (reddit_items, raw_response, error, used_scrapecreators)
    """
-    raw_openai = None
+    raw_response = None
    reddit_error = None
+    used_scrapecreators = False
+
+    sc_token = config.get("SCRAPECREATORS_API_KEY")

    if mock:
-        raw_openai = load_fixture("openai_sample.json")
-    else:
+        raw_response = load_fixture("openai_sample.json")
+    elif sc_token:
+        # === ScrapeCreators path (preferred) ===
+        used_scrapecreators = True
        try:
-            raw_openai = openai_reddit.search_reddit(
+            sys.stderr.write("[Reddit] Using ScrapeCreators API\n")
+            sys.stderr.flush()
+            result = reddit.search_and_enrich(
+                topic, from_date, to_date,
+                depth=depth, token=sc_token,
+            )
+            reddit_items = result.get("items", [])
+            if result.get("error"):
+                reddit_error = result["error"]
+            return reddit_items, result, reddit_error, used_scrapecreators
+        except Exception as e:
+            reddit_error = f"ScrapeCreators: {type(e).__name__}: {e}"
+            sys.stderr.write(f"[Reddit] ScrapeCreators failed: {e}\n")
+            sys.stderr.flush()
+            # Fall through to OpenAI if we have that key
+            if not config.get("OPENAI_API_KEY"):
+                return [], {"error": str(e)}, reddit_error, used_scrapecreators
+            used_scrapecreators = False
+            sys.stderr.write("[Reddit] Falling back to OpenAI\n")
+            sys.stderr.flush()
+
+    # === OpenAI path (fallback) ===
+    if not mock:
+        try:
+            raw_response = openai_reddit.search_reddit(
                config["OPENAI_API_KEY"],
                selected_models["openai"],
                topic,
@@ -193,14 +227,14 @@ def _search_reddit(
                account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
            )
        except http.HTTPError as e:
-            raw_openai = {"error": str(e)}
+            raw_response = {"error": str(e)}
            reddit_error = f"API error: {e}"
        except Exception as e:
-            raw_openai = {"error": str(e)}
+            raw_response = {"error": str(e)}
            reddit_error = f"{type(e).__name__}: {e}"

    # Parse response
-    reddit_items = openai_reddit.parse_reddit_response(raw_openai or {})
+    reddit_items = openai_reddit.parse_reddit_response(raw_response or {})

    # Quick retry with simpler query if few results
    if len(reddit_items) < 5 and not mock and not reddit_error:
@@ -217,7 +251,6 @@ def _search_reddit(
                    account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
                )
                retry_items = openai_reddit.parse_reddit_response(retry_raw)
-                # Add items not already found (by URL)
                existing_urls = {item.get("url") for item in reddit_items}
                for item in retry_items:
                    if item.get("url") not in existing_urls:
@@ -244,7 +277,7 @@ def _search_reddit(
        except Exception:
            pass

-    return reddit_items, raw_openai, reddit_error
+    return reddit_items, raw_response, reddit_error, used_scrapecreators


 def _search_x(
@@ -351,7 +384,7 @@ def _search_tiktok(
    depth: str,
    token: str,
 ) -> tuple:
-    """Search TikTok via Apify (runs in thread).
+    """Search TikTok via ScrapeCreators (runs in thread).

    Returns:
        Tuple of (tiktok_items, tiktok_error)
@@ -373,6 +406,35 @@ def _search_tiktok(
    return tiktok_items, tiktok_error


+def _search_instagram(
+    topic: str,
+    from_date: str,
+    to_date: str,
+    depth: str,
+    token: str,
+) -> tuple:
+    """Search Instagram via ScrapeCreators (runs in thread).
+
+    Returns:
+        Tuple of (instagram_items, instagram_error)
+    """
+    instagram_error = None
+
+    try:
+        response = instagram.search_and_enrich(
+            topic, from_date, to_date, depth=depth, token=token,
+        )
+    except Exception as e:
+        return [], f"{type(e).__name__}: {e}"
+
+    instagram_items = instagram.parse_instagram_response(response)
+
+    if response.get("error"):
+        instagram_error = response["error"]
+
+    return instagram_items, instagram_error
+
+
 def _search_hackernews(
    topic: str,
    from_date: str,
@@ -664,6 +726,7 @@ def run_research(
    x_source: str = "xai",
    run_youtube: bool = False,
    run_tiktok: bool = False,
+    run_instagram: bool = False,
    timeouts: dict = None,
    resolved_handle: str = None,
    do_hackernews: bool = True,
@@ -673,9 +736,11 @@ def run_research(
    """Run the research pipeline.

    Returns:
-        Tuple of (reddit_items, x_items, youtube_items, tiktok_items, web_items, web_needed,
+        Tuple of (reddit_items, x_items, youtube_items, tiktok_items, instagram_items,
+                  hackernews_items, polymarket_items, web_items, web_needed,
                  raw_openai, raw_xai, raw_reddit_enriched,
-                  reddit_error, x_error, youtube_error, tiktok_error, web_error)
+                  reddit_error, x_error, youtube_error, tiktok_error, instagram_error,
+                  hackernews_error, polymarket_error, web_error)

    Note: web_needed is True when web search should be performed by the assistant
    (i.e., no native web search API keys are configured). When native web search
@@ -689,6 +754,7 @@ def run_research(
    x_items = []
    youtube_items = []
    tiktok_items = []
+    instagram_items = []
    hackernews_items = []
    polymarket_items = []
    web_items = []
@@ -699,6 +765,7 @@ def run_research(
    x_error = None
    youtube_error = None
    tiktok_error = None
+    instagram_error = None
    hackernews_error = None
    polymarket_error = None
    web_error = None
@@ -729,7 +796,7 @@ def run_research(
            if progress:
                progress.start_web_only()
                progress.end_web_only()
-        # Still run YouTube in web-only mode if yt-dlp is available
+        # Still run YouTube/TikTok/Instagram in web-only mode if available
        if run_youtube:
            if progress:
                progress.start_youtube()
@@ -743,7 +810,34 @@ def run_research(
                    progress.show_error(f"YouTube error: {e}")
            if progress:
                progress.end_youtube(len(youtube_items))
-        return reddit_items, x_items, youtube_items, tiktok_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, hackernews_error, polymarket_error, web_error
+        if run_tiktok:
+            if progress:
+                progress.start_tiktok()
+            try:
+                tiktok_items, tiktok_error = _search_tiktok(topic, from_date, to_date, depth, env.get_tiktok_token(config))
+                if tiktok_error and progress:
+                    progress.show_error(f"TikTok error: {tiktok_error}")
+            except Exception as e:
+                tiktok_error = f"{type(e).__name__}: {e}"
+                if progress:
+                    progress.show_error(f"TikTok error: {e}")
+            if progress:
+                progress.end_tiktok(len(tiktok_items))
+        if run_instagram:
+            if progress:
+                progress.start_instagram()
+            try:
+                ig_timeout = timeouts.get("instagram_future", future_timeout)
+                instagram_items, instagram_error = _search_instagram(topic, from_date, to_date, depth, env.get_instagram_token(config))
+                if instagram_error and progress:
+                    progress.show_error(f"Instagram error: {instagram_error}")
+            except Exception as e:
+                instagram_error = f"{type(e).__name__}: {e}"
+                if progress:
+                    progress.show_error(f"Instagram error: {e}")
+            if progress:
+                progress.end_instagram(len(instagram_items))
+        return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error

    # Determine which searches to run
    do_reddit = sources in ("both", "reddit", "all", "reddit-web")
@@ -756,10 +850,11 @@ def run_research(
    x_future = None
    youtube_future = None
    tiktok_future = None
+    instagram_future = None
    hackernews_future = None
    polymarket_future = None
    web_future = None
-    max_workers = 2 + (1 if run_youtube else 0) + (1 if run_tiktok else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)
+    max_workers = 2 + (1 if run_youtube else 0) + (1 if run_tiktok else 0) + (1 if run_instagram else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Submit searches
@@ -791,7 +886,15 @@ def run_research(
                progress.start_tiktok()
            tiktok_future = executor.submit(
                _search_tiktok, topic, from_date, to_date, depth,
-                config.get('APIFY_API_TOKEN', ''),
+                env.get_tiktok_token(config),
+            )
+
+        if run_instagram:
+            if progress:
+                progress.start_instagram()
+            instagram_future = executor.submit(
+                _search_instagram, topic, from_date, to_date, depth,
+                env.get_instagram_token(config),
            )

        if do_hackernews:
@@ -816,10 +919,11 @@ def run_research(
            )

        # Collect results (with timeouts to prevent indefinite blocking)
+        reddit_used_sc = False  # Track if ScrapeCreators was used for Reddit
        if reddit_future:
            reddit_timeout = timeouts.get("reddit_future", future_timeout)
            try:
-                reddit_items, raw_openai, reddit_error = reddit_future.result(timeout=reddit_timeout)
+                reddit_items, raw_openai, reddit_error, reddit_used_sc = reddit_future.result(timeout=reddit_timeout)
                if reddit_error and progress:
                    progress.show_error(f"Reddit error: {reddit_error}")
            except TimeoutError:
@@ -883,6 +987,23 @@ def run_research(
            if progress:
                progress.end_tiktok(len(tiktok_items))

+        if instagram_future:
+            ig_timeout = timeouts.get("instagram_future", future_timeout)
+            try:
+                instagram_items, instagram_error = instagram_future.result(timeout=ig_timeout)
+                if instagram_error and progress:
+                    progress.show_error(f"Instagram error: {instagram_error}")
+            except TimeoutError:
+                instagram_error = f"Instagram search timed out after {ig_timeout}s"
+                if progress:
+                    progress.show_error(instagram_error)
+            except Exception as e:
+                instagram_error = f"{type(e).__name__}: {e}"
+                if progress:
+                    progress.show_error(f"Instagram error: {e}")
+            if progress:
+                progress.end_instagram(len(instagram_items))
+
        if hackernews_future:
            hn_timeout = timeouts.get("hackernews_future", future_timeout)
            try:
@@ -934,11 +1055,19 @@ def run_research(
            sys.stderr.flush()

    # Enrich Reddit items with real data (parallel, capped)
+    # Skip enrichment if ScrapeCreators already provided comments + engagement
    enrich_max = timeouts["enrich_max_items"]
    enrich_total_timeout = timeouts["enrich_total"]
    items_to_enrich = reddit_items[:enrich_max]
    rate_limited = False  # Set True if Reddit returns 429 during enrichment

+    if reddit_used_sc and items_to_enrich:
+        # ScrapeCreators already enriched items with comments — just copy to raw list
+        sys.stderr.write(f"[Reddit] Skipping old enrichment — ScrapeCreators already provided comments\n")
+        sys.stderr.flush()
+        raw_reddit_enriched = list(reddit_items[:enrich_max])
+        items_to_enrich = []  # Skip the enrichment block below
+
    if items_to_enrich:
        if progress:
            progress.start_reddit_enrich(1, len(items_to_enrich))
@@ -1013,11 +1142,12 @@ def run_research(

    # Phase 2: Supplemental search based on entities from Phase 1
    # Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting
+    # Also skip Reddit supplemental when ScrapeCreators was used (subreddit drilling already done)
    if depth != "quick" and not mock and (reddit_items or x_items):
        sup_reddit, sup_x = _run_supplemental(
            topic, reddit_items, x_items,
            from_date, to_date, depth, x_source, progress,
-            skip_reddit=rate_limited,
+            skip_reddit=(rate_limited or reddit_used_sc),
            resolved_handle=resolved_handle,
        )
        if sup_reddit:
@@ -1025,7 +1155,7 @@ def run_research(
        if sup_x:
            x_items.extend(sup_x)

-    return reddit_items, x_items, youtube_items, tiktok_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, hackernews_error, polymarket_error, web_error
+    return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error


 def main():
@@ -1037,7 +1167,7 @@ def main():
    parser = argparse.ArgumentParser(
        description="Research a topic from the last N days on Reddit + X"
    )
-    parser.add_argument("topic", nargs="?", help="Topic to research")
+    parser.add_argument("topic", nargs="*", help="Topic to research")
    parser.add_argument("--mock", action="store_true", help="Use fixtures")
    parser.add_argument(
        "--emit",
@@ -1120,8 +1250,16 @@ def main():
        default=False,
        help="Skip native web search backends (Parallel/Brave/OpenRouter). Use when the assistant has its own WebSearch tool.",
    )
+    parser.add_argument(
+        "--save-dir",
+        type=str,
+        default=None,
+        metavar="DIR",
+        help="Auto-save raw research output to DIR/{topic-slug}.md",
+    )

    args = parser.parse_args()
+    args.topic = " ".join(args.topic) if args.topic else None

    # Enable debug logging if requested
    if args.debug:
@@ -1159,8 +1297,11 @@ def main():
    # Auto-detect yt-dlp for YouTube search
    has_ytdlp = env.is_ytdlp_available()

-    # Auto-detect Apify for TikTok
-    has_apify = env.is_apify_available(config)
+    # Auto-detect ScrapeCreators/Apify for TikTok
+    has_tiktok = env.is_tiktok_available(config)
+
+    # Auto-detect ScrapeCreators for Instagram
+    has_instagram = env.is_instagram_available(config)

    # --diagnose: show source availability and exit
    if args.diagnose:
@@ -1173,7 +1314,8 @@ def main():
            "bird_authenticated": x_source_status["bird_authenticated"],
            "bird_username": x_source_status.get("bird_username"),
            "youtube": has_ytdlp,
-            "tiktok": has_apify,
+            "tiktok": has_tiktok,
+            "instagram": has_instagram,
            "hackernews": True,
            "polymarket": True,
            "web_search_backend": web_source,
@@ -1203,7 +1345,7 @@ def main():
        "bird_authenticated": x_source_status["bird_authenticated"],
        "bird_username": x_source_status.get("bird_username"),
        "youtube": has_ytdlp,
-        "tiktok": has_apify,
+        "tiktok": has_tiktok,
        "hackernews": True,
        "polymarket": True,
        "web_search_backend": "deferred to assistant" if args.no_native_web else web_source,
@@ -1288,7 +1430,8 @@ def main():
    search_do_hackernews = True
    search_do_polymarket = True
    search_run_youtube = has_ytdlp
-    search_run_tiktok = has_apify
+    search_run_tiktok = has_tiktok
+    search_run_instagram = has_instagram
    if args.search:
        search_sources = parse_search_flag(args.search)
        has_reddit = "reddit" in search_sources
@@ -1296,7 +1439,8 @@ def main():
        search_do_hackernews = "hn" in search_sources
        search_do_polymarket = "polymarket" in search_sources
        search_run_youtube = "youtube" in search_sources and has_ytdlp
-        search_run_tiktok = "tiktok" in search_sources and has_apify
+        search_run_tiktok = "tiktok" in search_sources and has_tiktok
+        search_run_instagram = "instagram" in search_sources and has_instagram
        include_search_web = "web" in search_sources
        # Map to existing sources string
        if has_reddit and has_x:
@@ -1310,7 +1454,7 @@ def main():
            sources = "web"  # hn/polymarket only; no Reddit/X

    # Run research
-    reddit_items, x_items, youtube_items, tiktok_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, hackernews_error, polymarket_error, web_error = run_research(
+    reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error = run_research(
        args.topic,
        sources,
        config,
@@ -1323,6 +1467,7 @@ def main():
        x_source=x_source or "xai",
        run_youtube=search_run_youtube,
        run_tiktok=search_run_tiktok,
+        run_instagram=search_run_instagram,
        timeouts=timeouts,
        resolved_handle=args.x_handle,
        do_hackernews=search_do_hackernews,
@@ -1338,6 +1483,7 @@ def main():
    normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
    normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else []
    normalized_tiktok = normalize.normalize_tiktok_items(tiktok_items, from_date, to_date) if tiktok_items else []
+    normalized_ig = normalize.normalize_instagram_items(instagram_items, from_date, to_date) if instagram_items else []
    normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
    normalized_pm = normalize.normalize_polymarket_items(polymarket_items, from_date, to_date) if polymarket_items else []
    normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
@@ -1352,6 +1498,8 @@ def main():
    filtered_youtube = normalized_youtube
    # TikTok: hard date filter (tiktok.py already pre-filters, but safety net)
    filtered_tiktok = normalize.filter_by_date_range(normalized_tiktok, from_date, to_date) if normalized_tiktok else []
+    # Instagram: hard date filter (instagram.py already pre-filters, but safety net)
+    filtered_ig = normalize.filter_by_date_range(normalized_ig, from_date, to_date) if normalized_ig else []
    filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
    # Polymarket: skip hard date filter - markets are active/traded, updatedAt is fine
    filtered_pm = normalized_pm
@@ -1362,6 +1510,7 @@ def main():
    scored_x = score.score_x_items(filtered_x)
    scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else []
    scored_tiktok = score.score_tiktok_items(filtered_tiktok) if filtered_tiktok else []
+    scored_ig = score.score_instagram_items(filtered_ig) if filtered_ig else []
    scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
    scored_pm = score.score_polymarket_items(filtered_pm) if filtered_pm else []
    scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
@@ -1371,6 +1520,7 @@ def main():
    sorted_x = score.sort_items(scored_x)
    sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
    sorted_tiktok = score.sort_items(scored_tiktok) if scored_tiktok else []
+    sorted_ig = score.sort_items(scored_ig) if scored_ig else []
    sorted_hn = score.sort_items(scored_hn) if scored_hn else []
    sorted_pm = score.sort_items(scored_pm) if scored_pm else []
    sorted_web = score.sort_items(scored_web) if scored_web else []
@@ -1380,6 +1530,7 @@ def main():
    deduped_x = dedupe.dedupe_x(sorted_x)
    deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else []
    deduped_tiktok = dedupe.dedupe_tiktok(sorted_tiktok) if sorted_tiktok else []
+    deduped_ig = dedupe.dedupe_instagram(sorted_ig) if sorted_ig else []
    deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
    deduped_pm = dedupe.dedupe_polymarket(sorted_pm) if sorted_pm else []
    deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
@@ -1393,7 +1544,7 @@ def main():

    # Cross-source linking: annotate items that discuss the same story
    dedupe.cross_source_link(
-        deduped_reddit, deduped_x, deduped_youtube, deduped_tiktok, deduped_hn, deduped_pm, deduped_web,
+        deduped_reddit, deduped_x, deduped_youtube, deduped_tiktok, deduped_ig, deduped_hn, deduped_pm, deduped_web,
    )

    progress.end_processing()
@@ -1411,6 +1562,7 @@ def main():
    report.x = deduped_x
    report.youtube = deduped_youtube
    report.tiktok = deduped_tiktok
+    report.instagram = deduped_ig
    report.hackernews = deduped_hn
    report.polymarket = deduped_pm
    report.web = deduped_web
@@ -1418,6 +1570,7 @@ def main():
    report.x_error = x_error
    report.youtube_error = youtube_error
    report.tiktok_error = tiktok_error
+    report.instagram_error = instagram_error
    report.hackernews_error = hackernews_error
    report.polymarket_error = polymarket_error
    report.web_error = web_error
@@ -1433,7 +1586,7 @@ def main():
    if sources == "web":
        progress.show_web_only_complete()
    else:
-        progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm), len(deduped_tiktok))
+        progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm), len(deduped_tiktok), len(deduped_ig))

    # Build source info for status footer
    source_info = {}
@@ -1448,14 +1601,30 @@ def main():
        source_info["youtube_skip_reason"] = "yt-dlp not installed — fix: brew install yt-dlp"
    elif has_ytdlp and not report.youtube:
        source_info["youtube_skip_reason"] = "0 results (query may be too specific)"
-    if not has_apify:
-        source_info["tiktok_skip_reason"] = "No APIFY_API_TOKEN — sign up free at apify.com"
+    if not has_tiktok:
+        source_info["tiktok_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
+    if not has_instagram:
+        source_info["instagram_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
    if not web_source:
        source_info["web_skip_reason"] = "assistant will use WebSearch (add BRAVE_API_KEY for native search)"

    # Output result
    output_result(report, args.emit, web_needed, args.topic, from_date, to_date, missing_keys, args.days, source_info)

+    # Auto-save raw research to file if --save-dir is set
+    if args.save_dir:
+        import re
+        save_dir = Path(args.save_dir).expanduser()
+        save_dir.mkdir(parents=True, exist_ok=True)
+        slug = re.sub(r'[^a-z0-9]+', '-', args.topic.lower()).strip('-')[:60]
+        save_path = save_dir / f"{slug}-raw.md"
+        if save_path.exists():
+            save_path = save_dir / f"{slug}-raw-{datetime.now().strftime('%Y-%m-%d')}.md"
+        content = render.render_compact(report, missing_keys=missing_keys)
+        content += "\n" + render.render_source_status(report, source_info)
+        save_path.write_text(content, encoding="utf-8")
+        print(f"📎 {save_path}", file=sys.stderr)
+
    # Persist findings to SQLite if requested
    if args.store:
        import store as store_mod
@@ -1515,6 +1684,16 @@ def main():
                "engagement_score": item.engagement.volume if item.engagement and item.engagement.volume else 0,
                "relevance_score": item.relevance,
            })
+        for item in deduped_ig:
+            findings.append({
+                "source": "instagram",
+                "url": item.url,
+                "title": item.text[:100],
+                "author": item.author_name,
+                "content": item.caption_snippet[:500] if item.caption_snippet else item.text,
+                "engagement_score": item.engagement.views if item.engagement and item.engagement.views else 0,
+                "relevance_score": item.relevance,
+            })
        for item in deduped_web:
            findings.append({
                "source": "web",
@@ -1,80 +0,0 @@
-"""Shared Apify client utilities for last30days sources.
-
-Provides a common wrapper around the apify-client SDK so that
-TikTok, Facebook, Instagram (future) all share the same client
-initialization, error handling, and cost-control patterns.
-
-One APIFY_API_TOKEN covers all Apify-backed sources.
-"""
-
-import sys
-from typing import Any, Dict, List, Optional
-
-try:
-    from apify_client import ApifyClient
-except ImportError:
-    ApifyClient = None
-
-
-def is_apify_available() -> bool:
-    """Check if the apify-client library is installed."""
-    return ApifyClient is not None
-
-
-def get_apify_client(token: str) -> "ApifyClient":
-    """Initialize Apify client with token.
-
-    Args:
-        token: Apify API token (from https://console.apify.com)
-
-    Returns:
-        Initialized ApifyClient instance
-
-    Raises:
-        ImportError: If apify-client is not installed
-    """
-    if ApifyClient is None:
-        raise ImportError(
-            "apify-client is not installed. Run: pip install apify-client"
-        )
-    return ApifyClient(token=token)
-
-
-def run_actor_sync(
-    client: "ApifyClient",
-    actor_id: str,
-    run_input: Dict[str, Any],
-    timeout_secs: int = 300,
-    max_items: Optional[int] = None,
-) -> List[Dict[str, Any]]:
-    """Run an Apify actor synchronously and return dataset items.
-
-    Args:
-        client: Initialized ApifyClient
-        actor_id: Actor identifier, e.g. "clockworks/tiktok-scraper"
-        run_input: Actor-specific input dict
-        timeout_secs: Max wait time (default 5 min)
-        max_items: Cap on returned items (cost control)
-
-    Returns:
-        List of result dicts from the actor's default dataset
-    """
-    run = client.actor(actor_id).call(
-        run_input=run_input,
-        timeout_secs=timeout_secs,
-        logger=None,  # Suppress verbose actor log streaming to stderr
-    )
-
-    dataset_id = run["defaultDatasetId"]
-    items = list(client.dataset(dataset_id).iterate_items())
-
-    if max_items and len(items) > max_items:
-        items = items[:max_items]
-    return items
-
-
-def _log(msg: str):
-    """Log to stderr (only in interactive terminals; spinner handles non-TTY)."""
-    if sys.stderr.isatty():
-        sys.stderr.write(f"[Apify] {msg}\n")
-        sys.stderr.flush()
@@ -46,7 +46,7 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:


 AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem, schema.TikTokItem,
-                schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem]
+                schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem]


 def get_item_text(item: AnyItem) -> str:
@@ -59,6 +59,8 @@ def get_item_text(item: AnyItem) -> str:
        return f"{item.title} {item.channel_name}"
    elif isinstance(item, schema.TikTokItem):
        return f"{item.text} {item.author_name}"
+    elif isinstance(item, schema.InstagramItem):
+        return f"{item.text} {item.author_name}"
    elif isinstance(item, schema.PolymarketItem):
        return f"{item.title} {item.question}"
    elif isinstance(item, schema.WebSearchItem):
@@ -78,6 +80,8 @@ def _get_cross_source_text(item: AnyItem) -> str:
        return item.text[:100]
    if isinstance(item, schema.TikTokItem):
        return item.text[:100]
+    if isinstance(item, schema.InstagramItem):
+        return item.text[:100]
    if isinstance(item, schema.HackerNewsItem):
        title = item.title
        if title.startswith("Show HN:"):
@@ -206,6 +210,14 @@ def dedupe_tiktok(
    return dedupe_items(items, threshold)


+def dedupe_instagram(
+    items: List[schema.InstagramItem],
+    threshold: float = 0.7,
+) -> List[schema.InstagramItem]:
+    """Dedupe Instagram items."""
+    return dedupe_items(items, threshold)
+
+
 def dedupe_hackernews(
    items: List[schema.HackerNewsItem],
    threshold: float = 0.7,
@@ -203,6 +203,7 @@ def get_config() -> Dict[str, Any]:
        ('OPENAI_MODEL_PIN', None),
        ('XAI_MODEL_POLICY', 'latest'),
        ('XAI_MODEL_PIN', None),
+        ('SCRAPECREATORS_API_KEY', None),
        ('APIFY_API_TOKEN', None),
        ('AUTH_TOKEN', None),
        ('CT0', None),
@@ -219,18 +220,42 @@ def config_exists() -> bool:
    return CONFIG_FILE.exists()


+def is_reddit_available(config: Dict[str, Any]) -> bool:
+    """Check if Reddit search is available.
+
+    Reddit can use either ScrapeCreators (preferred) or OpenAI.
+    """
+    has_sc = bool(config.get('SCRAPECREATORS_API_KEY'))
+    has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
+    return has_sc or has_openai
+
+
+def get_reddit_source(config: Dict[str, Any]) -> Optional[str]:
+    """Determine which Reddit backend to use.
+
+    Priority: ScrapeCreators (cheaper, faster) > OpenAI (legacy)
+
+    Returns: 'scrapecreators', 'openai', or None
+    """
+    if config.get('SCRAPECREATORS_API_KEY'):
+        return 'scrapecreators'
+    if config.get('OPENAI_API_KEY') and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK:
+        return 'openai'
+    return None
+
+
 def get_available_sources(config: Dict[str, Any]) -> str:
    """Determine which sources are available based on API keys.

    Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
    """
-    has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
+    has_reddit = is_reddit_available(config)
    has_xai = bool(config.get('XAI_API_KEY'))
    has_web = has_web_search_keys(config)

-    if has_openai and has_xai:
+    if has_reddit and has_xai:
        return 'all' if has_web else 'both'
-    elif has_openai:
+    elif has_reddit:
        return 'reddit-web' if has_web else 'reddit'
    elif has_xai:
        return 'x-web' if has_web else 'x'
@@ -262,11 +287,11 @@ def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:


 def get_missing_keys(config: Dict[str, Any]) -> str:
-    """Determine which sources are missing (accounting for Bird).
+    """Determine which sources are missing (accounting for Bird and ScrapeCreators).

    Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
    """
-    has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
+    has_reddit = is_reddit_available(config)
    has_xai = bool(config.get('XAI_API_KEY'))
    has_web = has_web_search_keys(config)

@@ -276,14 +301,14 @@ def get_missing_keys(config: Dict[str, Any]) -> str:

    has_x = has_xai or has_bird

-    if has_openai and has_x and has_web:
+    if has_reddit and has_x and has_web:
        return 'none'
-    elif has_openai and has_x:
+    elif has_reddit and has_x:
        return 'web'  # Missing web search keys
-    elif has_openai:
+    elif has_reddit:
        return 'x'  # Missing X source (and possibly web)
    elif has_x:
-        return 'reddit'  # Missing OpenAI key (and possibly web)
+        return 'reddit'  # Missing Reddit source (and possibly web)
    else:
        return 'all'  # Missing everything

@@ -407,13 +432,35 @@ def is_polymarket_available() -> bool:
    return True


-def is_apify_available(config: Dict[str, Any]) -> bool:
-    """Check if Apify token is configured for TikTok/social scraping.
+def is_tiktok_available(config: Dict[str, Any]) -> bool:
+    """Check if TikTok source is available (ScrapeCreators or legacy Apify).

-    Returns True if APIFY_API_TOKEN is set. One token covers
-    TikTok, Facebook, Instagram (all Apify-backed sources).
+    Returns True if SCRAPECREATORS_API_KEY or APIFY_API_TOKEN is set.
    """
-    return bool(config.get('APIFY_API_TOKEN'))
+    return bool(config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN'))
+
+
+def get_tiktok_token(config: Dict[str, Any]) -> str:
+    """Get TikTok API token, preferring ScrapeCreators over legacy Apify."""
+    return config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN') or ''
+
+
+def is_instagram_available(config: Dict[str, Any]) -> bool:
+    """Check if Instagram source is available (ScrapeCreators).
+
+    Returns True if SCRAPECREATORS_API_KEY is set.
+    Instagram uses the same key as TikTok.
+    """
+    return bool(config.get('SCRAPECREATORS_API_KEY'))
+
+
+def get_instagram_token(config: Dict[str, Any]) -> str:
+    """Get Instagram API token (same ScrapeCreators key as TikTok)."""
+    return config.get('SCRAPECREATORS_API_KEY') or ''
+
+
+# Backward compat alias
+is_apify_available = is_tiktok_available


 def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
@@ -0,0 +1,437 @@
+"""Instagram Reels search via ScrapeCreators API for /last30days.
+
+Uses ScrapeCreators REST API to search Instagram Reels by keyword, extract
+engagement metrics (views, likes, comments), and fetch video transcripts.
+
+Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
+API docs: https://scrapecreators.com/docs
+"""
+
+import re
+import sys
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional, Set
+
+try:
+    import requests as _requests
+except ImportError:
+    _requests = None
+
+SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
+
+# Depth configurations: how many results to fetch / captions to extract
+DEPTH_CONFIG = {
+    "quick":   {"results_per_page": 10, "max_captions": 3},
+    "default": {"results_per_page": 20, "max_captions": 5},
+    "deep":    {"results_per_page": 40, "max_captions": 8},
+}
+
+# Max words to keep from each caption
+CAPTION_MAX_WORDS = 500
+
+# Stopwords for relevance computation (shared with tiktok.py pattern)
+STOPWORDS = frozenset({
+    'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
+    'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
+    'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
+    'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
+    'all', 'just', 'get', 'has', 'have', 'was', 'will',
+})
+
+# Synonym groups for relevance scoring
+SYNONYMS = {
+    'hip': {'rap', 'hiphop'},
+    'hop': {'rap', 'hiphop'},
+    'rap': {'hip', 'hop', 'hiphop'},
+    'hiphop': {'rap', 'hip', 'hop'},
+    'js': {'javascript'},
+    'javascript': {'js'},
+    'ts': {'typescript'},
+    'typescript': {'ts'},
+    'ai': {'artificial', 'intelligence'},
+    'ml': {'machine', 'learning'},
+    'react': {'reactjs'},
+    'reactjs': {'react'},
+}
+
+
+def _tokenize(text: str) -> Set[str]:
+    """Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
+    words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
+    tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
+    expanded = set(tokens)
+    for t in tokens:
+        if t in SYNONYMS:
+            expanded.update(SYNONYMS[t])
+    return expanded
+
+
+def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> float:
+    """Compute relevance as ratio of query tokens found in text + hashtags.
+
+    Uses ratio overlap (intersection / query_length). Hashtags provide
+    an Instagram-specific relevance boost. Floors at 0.1.
+    """
+    q_tokens = _tokenize(query)
+
+    # Combine text and hashtags for matching
+    combined = text
+    if hashtags:
+        combined = f"{text} {' '.join(hashtags)}"
+    t_tokens = _tokenize(combined)
+
+    # Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
+    if hashtags:
+        for tag in hashtags:
+            tag_lower = tag.lower()
+            for qt in q_tokens:
+                if qt in tag_lower and qt != tag_lower:
+                    t_tokens.add(qt)
+
+    if not q_tokens:
+        return 0.5  # Neutral fallback
+
+    overlap = len(q_tokens & t_tokens)
+    ratio = overlap / len(q_tokens)
+    return max(0.1, min(1.0, ratio))
+
+
+def _extract_core_subject(topic: str) -> str:
+    """Extract core subject from verbose query for Instagram search.
+
+    Strips meta/research words to keep only the core product/concept name.
+    """
+    text = topic.lower().strip()
+
+    # Strip multi-word prefixes
+    prefixes = [
+        'what are the best', 'what is the best', 'what are the latest',
+        'what are people saying about', 'what do people think about',
+        'how do i use', 'how to use', 'how to',
+        'what are', 'what is', 'tips for', 'best practices for',
+    ]
+    for p in prefixes:
+        if text.startswith(p + ' '):
+            text = text[len(p):].strip()
+
+    # Strip individual noise words
+    noise = {
+        'best', 'top', 'good', 'great', 'awesome', 'killer',
+        'latest', 'new', 'news', 'update', 'updates',
+        'trending', 'hottest', 'popular', 'viral',
+        'practices', 'features',
+        'recommendations', 'advice',
+        'prompt', 'prompts', 'prompting',
+        'methods', 'strategies', 'approaches',
+    }
+    words = text.split()
+    filtered = [w for w in words if w not in noise]
+
+    result = ' '.join(filtered) if filtered else text
+    return result.rstrip('?!.')
+
+
+def _log(msg: str):
+    """Log to stderr (only in interactive terminals; spinner handles non-TTY)."""
+    if sys.stderr.isatty():
+        sys.stderr.write(f"[Instagram] {msg}\n")
+        sys.stderr.flush()
+
+
+def _sc_headers(token: str) -> Dict[str, str]:
+    """Build ScrapeCreators request headers."""
+    return {
+        "x-api-key": token,
+        "Content-Type": "application/json",
+    }
+
+
+def _parse_date(item: Dict[str, Any]) -> Optional[str]:
+    """Parse date from ScrapeCreators Instagram item to YYYY-MM-DD.
+
+    Handles taken_at as ISO string (e.g. "2026-02-26T16:00:00.000Z")
+    or unix timestamp.
+    """
+    ts = item.get("taken_at")
+    if not ts:
+        return None
+
+    # Try ISO string first (ScrapeCreators reels/search returns this)
+    if isinstance(ts, str):
+        try:
+            # Handle "2026-02-26T16:00:00.000Z" format
+            dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
+            return dt.strftime("%Y-%m-%d")
+        except (ValueError, TypeError):
+            pass
+        # Try just the date portion
+        if len(ts) >= 10:
+            return ts[:10]
+
+    # Fall back to unix timestamp
+    try:
+        dt = datetime.fromtimestamp(int(ts), tz=timezone.utc)
+        return dt.strftime("%Y-%m-%d")
+    except (ValueError, TypeError, OSError):
+        pass
+
+    return None
+
+
+def _extract_hashtags(caption_text: str) -> List[str]:
+    """Extract hashtags from Instagram caption text."""
+    if not caption_text:
+        return []
+    return re.findall(r'#(\w+)', caption_text)
+
+
+def search_instagram(
+    topic: str,
+    from_date: str,
+    to_date: str,
+    depth: str = "default",
+    token: str = None,
+) -> Dict[str, Any]:
+    """Search Instagram Reels via ScrapeCreators API.
+
+    Args:
+        topic: Search topic
+        from_date: Start date (YYYY-MM-DD)
+        to_date: End date (YYYY-MM-DD)
+        depth: 'quick', 'default', or 'deep'
+        token: ScrapeCreators API key
+
+    Returns:
+        Dict with 'items' list and optional 'error'.
+    """
+    if not token:
+        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
+
+    if not _requests:
+        return {"items": [], "error": "requests library not installed"}
+
+    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+    core_topic = _extract_core_subject(topic)
+
+    _log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
+
+    try:
+        resp = _requests.get(
+            f"{SCRAPECREATORS_BASE}/v1/instagram/reels/search",
+            params={"query": core_topic},
+            headers=_sc_headers(token),
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+    except Exception as e:
+        _log(f"ScrapeCreators error: {e}")
+        return {"items": [], "error": f"{type(e).__name__}: {e}"}
+
+    # Items are in the 'reels' array (ScrapeCreators v1 response)
+    raw_items = data.get("reels") or data.get("items") or data.get("data") or []
+
+    # Limit to configured count
+    raw_items = raw_items[:config["results_per_page"]]
+
+    # Parse items
+    items = []
+    for raw in raw_items:
+        if not isinstance(raw, dict):
+            continue
+
+        # Extract reel ID and shortcode
+        reel_pk = str(raw.get("id", raw.get("pk", "")))
+        shortcode = raw.get("shortcode", raw.get("code", ""))
+
+        # Caption text — can be a string or dict depending on endpoint
+        caption_obj = raw.get("caption", "")
+        if isinstance(caption_obj, dict):
+            text = caption_obj.get("text", "")
+        elif isinstance(caption_obj, str):
+            text = caption_obj
+        else:
+            text = raw.get("desc", raw.get("text", ""))
+
+        # Engagement metrics
+        play_count = raw.get("video_play_count") or raw.get("video_view_count") or raw.get("play_count") or 0
+        like_count = raw.get("like_count") or 0
+        comment_count = raw.get("comment_count") or 0
+
+        # Author info — 'owner' in reels/search, 'user' in user/reels
+        owner = raw.get("owner") or raw.get("user") or {}
+        author_name = owner.get("username", "")
+
+        # Duration
+        duration = raw.get("video_duration")
+
+        # Date
+        date_str = _parse_date(raw)
+
+        # Hashtags from caption text
+        hashtags = _extract_hashtags(text)
+
+        # Compute relevance with hashtag boost
+        relevance = _compute_relevance(core_topic, text, hashtags)
+
+        # Build URL — prefer API-provided url, fallback to shortcode
+        url = raw.get("url", "")
+        if not url and shortcode:
+            url = f"https://www.instagram.com/reel/{shortcode}"
+
+        items.append({
+            "video_id": reel_pk,
+            "text": text,
+            "url": url,
+            "author_name": author_name,
+            "date": date_str,
+            "engagement": {
+                "views": play_count,
+                "likes": like_count,
+                "comments": comment_count,
+            },
+            "hashtags": hashtags,
+            "duration": duration,
+            "relevance": relevance,
+            "why_relevant": f"Instagram: {text[:60]}" if text else f"Instagram: {core_topic}",
+            "caption_snippet": "",  # populated by fetch_captions
+        })
+
+    # Hard date filter
+    in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
+    out_of_range = len(items) - len(in_range)
+    if in_range:
+        items = in_range
+        if out_of_range:
+            _log(f"Filtered {out_of_range} reels outside date range")
+    else:
+        _log(f"No reels within date range, keeping all {len(items)}")
+
+    # Sort by views descending
+    items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
+
+    _log(f"Found {len(items)} Instagram reels")
+    return {"items": items}
+
+
+def fetch_captions(
+    video_items: List[Dict[str, Any]],
+    token: str,
+    depth: str = "default",
+) -> Dict[str, str]:
+    """Fetch transcripts for top N Instagram reels via ScrapeCreators.
+
+    Strategy:
+    1. Use the 'text' field (caption) as baseline
+    2. For top N, call /v2/instagram/media/transcript for spoken-word captions
+
+    Args:
+        video_items: Items from search_instagram()
+        token: ScrapeCreators API key
+        depth: Depth level for caption limit
+
+    Returns:
+        Dict mapping video_id -> caption text (truncated to 500 words)
+    """
+    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+    max_captions = config["max_captions"]
+
+    if not video_items or not token or not _requests:
+        return {}
+
+    top_items = video_items[:max_captions]
+    _log(f"Enriching captions for {len(top_items)} reels")
+
+    captions = {}
+
+    # First pass: use text field as caption (always available, free)
+    for item in top_items:
+        vid = item["video_id"]
+        text = item.get("text", "")
+        if text:
+            words = text.split()
+            if len(words) > CAPTION_MAX_WORDS:
+                text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
+            captions[vid] = text
+
+    # Second pass: try to get spoken-word transcripts (1 credit each)
+    for item in top_items:
+        vid = item["video_id"]
+        url = item.get("url", "")
+        if not url:
+            continue
+        try:
+            resp = _requests.get(
+                f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
+                params={"url": url},
+                headers=_sc_headers(token),
+                timeout=15,
+            )
+            if resp.status_code == 200:
+                data = resp.json()
+                transcripts = data.get("transcripts") or []
+                if transcripts and isinstance(transcripts, list):
+                    # Combine all transcript segments
+                    transcript_text = " ".join(
+                        t.get("text", "") for t in transcripts
+                        if isinstance(t, dict) and t.get("text")
+                    )
+                    if transcript_text:
+                        words = transcript_text.split()
+                        if len(words) > CAPTION_MAX_WORDS:
+                            transcript_text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
+                        captions[vid] = transcript_text
+        except Exception as e:
+            _log(f"Transcript fetch failed for {vid}: {e}")
+
+    got = sum(1 for v in captions.values() if v)
+    _log(f"Got captions for {got}/{len(top_items)} reels")
+    return captions
+
+
+def search_and_enrich(
+    topic: str,
+    from_date: str,
+    to_date: str,
+    depth: str = "default",
+    token: str = None,
+) -> Dict[str, Any]:
+    """Full Instagram search: find reels, then fetch captions for top results.
+
+    Args:
+        topic: Search topic
+        from_date: Start date (YYYY-MM-DD)
+        to_date: End date (YYYY-MM-DD)
+        depth: 'quick', 'default', or 'deep'
+        token: ScrapeCreators API key
+
+    Returns:
+        Dict with 'items' list. Each item has a 'caption_snippet' field.
+    """
+    # Step 1: Search
+    search_result = search_instagram(topic, from_date, to_date, depth, token)
+    items = search_result.get("items", [])
+
+    if not items:
+        return search_result
+
+    # Step 2: Fetch captions for top N
+    captions = fetch_captions(items, token, depth)
+
+    # Step 3: Attach captions to items
+    for item in items:
+        vid = item["video_id"]
+        caption = captions.get(vid)
+        if caption:
+            item["caption_snippet"] = caption
+
+    return {"items": items, "error": search_result.get("error")}
+
+
+def parse_instagram_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
+    """Parse Instagram search response to normalized format.
+
+    Returns:
+        List of item dicts ready for normalization.
+    """
+    return response.get("items", [])
@@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union

 from . import dates, schema

-T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.HackerNewsItem, schema.PolymarketItem)
+T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem)


 def filter_by_date_range(
@@ -247,6 +247,52 @@ def normalize_tiktok_items(
    return normalized


+def normalize_instagram_items(
+    items: List[Dict[str, Any]],
+    from_date: str,
+    to_date: str,
+) -> List[schema.InstagramItem]:
+    """Normalize raw Instagram items to schema.
+
+    Args:
+        items: Raw Instagram items from ScrapeCreators
+        from_date: Start of date range
+        to_date: End of date range
+
+    Returns:
+        List of InstagramItem objects
+    """
+    normalized = []
+
+    for i, item in enumerate(items):
+        # Parse engagement
+        eng_raw = item.get("engagement") or {}
+        engagement = schema.Engagement(
+            views=eng_raw.get("views"),
+            likes=eng_raw.get("likes"),
+            num_comments=eng_raw.get("comments"),
+        )
+
+        # Instagram dates are reliable (exact timestamps from ScrapeCreators)
+        date_str = item.get("date")
+
+        normalized.append(schema.InstagramItem(
+            id=f"IG{i+1}",
+            text=item.get("text", ""),
+            url=item.get("url", ""),
+            author_name=item.get("author_name", ""),
+            date=date_str,
+            date_confidence="high",
+            engagement=engagement,
+            caption_snippet=item.get("caption_snippet", ""),
+            hashtags=item.get("hashtags", []),
+            relevance=item.get("relevance", 0.7),
+            why_relevant=item.get("why_relevant", ""),
+        ))
+
+    return normalized
+
+
 def normalize_hackernews_items(
    items: List[Dict[str, Any]],
    from_date: str,
@@ -0,0 +1,603 @@
+"""Reddit search via ScrapeCreators API for /last30days.
+
+Uses ScrapeCreators REST API to search Reddit globally, discover relevant
+subreddits, run targeted subreddit searches, and fetch comment trees.
+
+Replaces openai_reddit.py as the primary Reddit search backend.
+Falls back to openai_reddit.py if SCRAPECREATORS_API_KEY is missing but
+OPENAI_API_KEY is present.
+
+Requires SCRAPECREATORS_API_KEY in config (same key as TikTok + Instagram).
+API docs: https://scrapecreators.com/docs
+"""
+
+import re
+import sys
+from collections import Counter
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional, Set
+
+try:
+    import requests as _requests
+except ImportError:
+    _requests = None
+
+from . import http
+
+SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/reddit"
+
+# Depth configurations: how many API calls per phase
+DEPTH_CONFIG = {
+    "quick": {
+        "global_searches": 1,
+        "subreddit_searches": 2,
+        "comment_enrichments": 3,
+        "timeframe": "week",
+    },
+    "default": {
+        "global_searches": 2,
+        "subreddit_searches": 3,
+        "comment_enrichments": 5,
+        "timeframe": "month",
+    },
+    "deep": {
+        "global_searches": 3,
+        "subreddit_searches": 5,
+        "comment_enrichments": 8,
+        "timeframe": "month",
+    },
+}
+
+# Stopwords for query extraction
+NOISE_WORDS = frozenset({
+    'best', 'top', 'good', 'great', 'awesome', 'killer',
+    'latest', 'new', 'news', 'update', 'updates',
+    'trending', 'hottest', 'popular',
+    'practices', 'features', 'tips',
+    'recommendations', 'advice',
+    'prompt', 'prompts', 'prompting',
+    'methods', 'strategies', 'approaches',
+    'how', 'to', 'the', 'a', 'an', 'for', 'with',
+    'of', 'in', 'on', 'is', 'are', 'what', 'which',
+    'guide', 'tutorial', 'using',
+})
+
+
+def _log(msg: str):
+    """Log to stderr."""
+    sys.stderr.write(f"[Reddit] {msg}\n")
+    sys.stderr.flush()
+
+
+def _sc_headers(token: str) -> Dict[str, str]:
+    """Build ScrapeCreators request headers."""
+    return {
+        "x-api-key": token,
+        "Content-Type": "application/json",
+    }
+
+
+def _extract_core_subject(topic: str) -> str:
+    """Extract core subject from verbose query.
+
+    Strips meta/research words to keep only the core product/concept name.
+    """
+    text = topic.lower().strip()
+
+    # Strip multi-word prefixes
+    prefixes = [
+        'what are the best', 'what is the best', 'what are the latest',
+        'what are people saying about', 'what do people think about',
+        'how do i use', 'how to use', 'how to',
+        'what are', 'what is', 'tips for', 'best practices for',
+    ]
+    for p in prefixes:
+        if text.startswith(p + ' '):
+            text = text[len(p):].strip()
+
+    words = text.split()
+    filtered = [w for w in words if w not in NOISE_WORDS]
+
+    result = ' '.join(filtered) if filtered else text
+    return result.rstrip('?!.')
+
+
+def expand_reddit_queries(topic: str, depth: str) -> List[str]:
+    """Generate multiple Reddit search queries from a topic.
+
+    Uses local logic (no LLM call needed):
+    1. Extract core subject (strip noise words)
+    2. Include original topic if different from core
+    3. For default/deep: add casual/review variant
+    4. For deep: add problem/issues variant
+
+    Returns 1-4 query strings depending on depth.
+    """
+    core = _extract_core_subject(topic)
+    queries = [core]
+
+    # Broader variant: include more context from original topic
+    original_clean = topic.strip().rstrip('?!.')
+    if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
+        queries.append(original_clean)
+
+    if depth in ("default", "deep"):
+        queries.append(f"{core} worth it OR thoughts OR review")
+
+    if depth == "deep":
+        queries.append(f"{core} issues OR problems OR bug OR broken")
+
+    return queries
+
+
+# Known utility/meta subreddits that match queries but aren't discussion subs.
+# These get a 0.3x penalty (not banned) in subreddit discovery scoring.
+UTILITY_SUBS = frozenset({
+    'namethatsong', 'findthatsong', 'tipofmytongue',
+    'whatisthissong', 'helpmefind', 'whatisthisthing',
+    'whatsthissong', 'findareddit', 'subredditdrama',
+})
+
+
+def discover_subreddits(
+    results: List[Dict[str, Any]],
+    topic: str = "",
+    max_subs: int = 5,
+) -> List[str]:
+    """Extract top subreddits from global search results with relevance weighting.
+
+    Uses frequency + topic-word matching + utility-sub penalties + engagement
+    bonus to find discussion subs rather than utility/meta subs.
+
+    Args:
+        results: List of post dicts from global search
+        topic: Original search topic (for relevance matching)
+        max_subs: Maximum subreddits to return
+
+    Returns:
+        Top subreddit names sorted by weighted score
+    """
+    core = _extract_core_subject(topic) if topic else ""
+    core_words = set(core.lower().split()) if core else set()
+
+    scores = Counter()
+    for post in results:
+        sub = post.get("subreddit", "")
+        if not sub:
+            continue
+
+        # Base: frequency count
+        base = 1.0
+
+        # Bonus: subreddit name contains a core topic word
+        sub_lower = sub.lower()
+        if core_words and any(w in sub_lower for w in core_words if len(w) > 2):
+            base += 2.0
+
+        # Penalty: known utility/meta subreddits
+        if sub_lower in UTILITY_SUBS:
+            base *= 0.3
+
+        # Bonus: post engagement (high-engagement posts = better sub)
+        ups = post.get("ups") or post.get("score", 0)
+        if ups and ups > 100:
+            base += 0.5
+
+        scores[sub] += base
+
+    return [sub for sub, _ in scores.most_common(max_subs)]
+
+
+def _parse_date(created_utc) -> Optional[str]:
+    """Convert Unix timestamp to YYYY-MM-DD."""
+    if not created_utc:
+        return None
+    try:
+        dt = datetime.fromtimestamp(float(created_utc), tz=timezone.utc)
+        return dt.strftime("%Y-%m-%d")
+    except (ValueError, TypeError, OSError):
+        return None
+
+
+def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global") -> Dict[str, Any]:
+    """Normalize a ScrapeCreators Reddit post to our internal format."""
+    permalink = post.get("permalink", "")
+    url = f"https://www.reddit.com{permalink}" if permalink else post.get("url", "")
+
+    # Ensure URL looks like a Reddit thread
+    if url and "reddit.com" not in url:
+        url = ""
+
+    return {
+        "id": f"R{idx}",
+        "reddit_id": post.get("id", ""),
+        "title": str(post.get("title", "")).strip(),
+        "url": url,
+        "subreddit": str(post.get("subreddit", "")).strip(),
+        "date": _parse_date(post.get("created_utc")),
+        "engagement": {
+            "score": post.get("ups") or post.get("score", 0),
+            "num_comments": post.get("num_comments", 0),
+            "upvote_ratio": post.get("upvote_ratio"),
+        },
+        "relevance": 0.7,
+        "why_relevant": f"Reddit {source_label} search",
+        "selftext": str(post.get("selftext", ""))[:500],
+    }
+
+
+def _global_search(
+    query: str,
+    token: str,
+    sort: str = "relevance",
+    timeframe: str = "month",
+) -> List[Dict[str, Any]]:
+    """Search across all of Reddit via ScrapeCreators global search.
+
+    Args:
+        query: Search query
+        token: ScrapeCreators API key
+        sort: Sort order (relevance, hot, top, new)
+        timeframe: Time filter (hour, day, week, month, year, all)
+
+    Returns:
+        List of post dicts
+    """
+    if not _requests:
+        _log("requests library not installed, falling back to urllib")
+        # Use stdlib http module as fallback
+        try:
+            from urllib.parse import urlencode
+            params = urlencode({"query": query, "sort": sort, "timeframe": timeframe})
+            url = f"{SCRAPECREATORS_BASE}/search?{params}"
+            headers = _sc_headers(token)
+            headers["User-Agent"] = http.USER_AGENT
+            data = http.get(url, headers=headers, timeout=30, retries=2)
+            return data.get("posts", data.get("data", []))
+        except Exception as e:
+            _log(f"Global search error (urllib): {e}")
+            return []
+
+    try:
+        resp = _requests.get(
+            f"{SCRAPECREATORS_BASE}/search",
+            params={"query": query, "sort": sort, "timeframe": timeframe},
+            headers=_sc_headers(token),
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        return data.get("posts", data.get("data", []))
+    except Exception as e:
+        _log(f"Global search error: {e}")
+        return []
+
+
+def _subreddit_search(
+    subreddit: str,
+    query: str,
+    token: str,
+    sort: str = "relevance",
+    timeframe: str = "month",
+) -> List[Dict[str, Any]]:
+    """Search within a specific subreddit via ScrapeCreators.
+
+    Args:
+        subreddit: Subreddit name (without r/)
+        query: Search query
+        token: ScrapeCreators API key
+        sort: Sort order
+        timeframe: Time filter
+
+    Returns:
+        List of post dicts
+    """
+    if not _requests:
+        try:
+            from urllib.parse import urlencode
+            params = urlencode({
+                "subreddit": subreddit, "query": query,
+                "sort": sort, "timeframe": timeframe,
+            })
+            url = f"{SCRAPECREATORS_BASE}/subreddit/search?{params}"
+            headers = _sc_headers(token)
+            headers["User-Agent"] = http.USER_AGENT
+            data = http.get(url, headers=headers, timeout=30, retries=2)
+            return data.get("posts", data.get("data", []))
+        except Exception as e:
+            _log(f"Subreddit search error (urllib) for r/{subreddit}: {e}")
+            return []
+
+    try:
+        resp = _requests.get(
+            f"{SCRAPECREATORS_BASE}/subreddit/search",
+            params={
+                "subreddit": subreddit,
+                "query": query,
+                "sort": sort,
+                "timeframe": timeframe,
+            },
+            headers=_sc_headers(token),
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        return data.get("posts", data.get("data", []))
+    except Exception as e:
+        _log(f"Subreddit search error for r/{subreddit}: {e}")
+        return []
+
+
+def fetch_post_comments(
+    url: str,
+    token: str,
+) -> List[Dict[str, Any]]:
+    """Fetch comments for a Reddit post via ScrapeCreators.
+
+    Args:
+        url: Reddit post URL or permalink
+        token: ScrapeCreators API key
+
+    Returns:
+        List of comment dicts with score, author, body, etc.
+    """
+    if not _requests:
+        try:
+            from urllib.parse import urlencode
+            params = urlencode({"url": url})
+            api_url = f"{SCRAPECREATORS_BASE}/post/comments?{params}"
+            headers = _sc_headers(token)
+            headers["User-Agent"] = http.USER_AGENT
+            data = http.get(api_url, headers=headers, timeout=30, retries=2)
+            return data.get("comments", data.get("data", []))
+        except Exception as e:
+            _log(f"Comment fetch error (urllib): {e}")
+            return []
+
+    try:
+        resp = _requests.get(
+            f"{SCRAPECREATORS_BASE}/post/comments",
+            params={"url": url},
+            headers=_sc_headers(token),
+            timeout=30,
+        )
+        resp.raise_for_status()
+        data = resp.json()
+        return data.get("comments", data.get("data", []))
+    except Exception as e:
+        _log(f"Comment fetch error: {e}")
+        return []
+
+
+def _dedupe_posts(posts: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    """Deduplicate posts by reddit_id, keeping first occurrence."""
+    seen_ids = set()
+    seen_urls = set()
+    unique = []
+    for post in posts:
+        rid = post.get("reddit_id", "")
+        url = post.get("url", "")
+        if rid and rid in seen_ids:
+            continue
+        if url and url in seen_urls:
+            continue
+        if rid:
+            seen_ids.add(rid)
+        if url:
+            seen_urls.add(url)
+        unique.append(post)
+    return unique
+
+
+def search_reddit(
+    topic: str,
+    from_date: str,
+    to_date: str,
+    depth: str = "default",
+    token: str = None,
+) -> Dict[str, Any]:
+    """Full Reddit search: multi-query global discovery + subreddit drill-down.
+
+    This is the main entry point. Replaces openai_reddit.search_reddit().
+
+    Args:
+        topic: Search topic
+        from_date: Start date (YYYY-MM-DD)
+        to_date: End date (YYYY-MM-DD)
+        depth: 'quick', 'default', or 'deep'
+        token: ScrapeCreators API key
+
+    Returns:
+        Dict with 'items' list and optional 'error'.
+    """
+    if not token:
+        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
+
+    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+    timeframe = config["timeframe"]
+
+    # === Phase 1: Query Expansion ===
+    queries = expand_reddit_queries(topic, depth)
+    _log(f"Expanded '{topic}' into {len(queries)} queries: {queries}")
+
+    # === Phase 2: Global Discovery ===
+    all_raw_posts = []
+    max_global = config["global_searches"]
+
+    for i, query in enumerate(queries[:max_global]):
+        sort = "relevance" if i == 0 else "top"
+        _log(f"Global search {i+1}/{max_global}: '{query}' (sort={sort})")
+        posts = _global_search(query, token, sort=sort, timeframe=timeframe)
+        _log(f"  -> {len(posts)} results")
+        all_raw_posts.extend(posts)
+
+    # Normalize all posts
+    all_items = []
+    for i, post in enumerate(all_raw_posts):
+        item = _normalize_post(post, i + 1, "global")
+        all_items.append(item)
+
+    # === Phase 3: Subreddit Discovery + Targeted Search ===
+    discovered_subs = discover_subreddits(all_raw_posts, topic=topic, max_subs=config["subreddit_searches"])
+    _log(f"Discovered subreddits: {discovered_subs}")
+
+    core = _extract_core_subject(topic)
+    for sub in discovered_subs[:config["subreddit_searches"]]:
+        _log(f"Subreddit search: r/{sub} for '{core}'")
+        sub_posts = _subreddit_search(sub, core, token, sort="relevance", timeframe=timeframe)
+        _log(f"  -> {len(sub_posts)} results from r/{sub}")
+        for j, post in enumerate(sub_posts):
+            item = _normalize_post(post, len(all_items) + j + 1, f"r/{sub}")
+            all_items.append(item)
+
+    # === Phase 4: Deduplicate ===
+    all_items = _dedupe_posts(all_items)
+    _log(f"After dedup: {len(all_items)} unique posts")
+
+    # === Phase 5: Date filter ===
+    in_range = []
+    out_of_range = 0
+    for item in all_items:
+        if item["date"] and from_date <= item["date"] <= to_date:
+            in_range.append(item)
+        elif item["date"] is None:
+            in_range.append(item)  # Keep unknown dates
+        else:
+            out_of_range += 1
+
+    if in_range:
+        all_items = in_range
+        if out_of_range:
+            _log(f"Filtered {out_of_range} posts outside date range")
+    else:
+        _log(f"No posts within date range, keeping all {len(all_items)}")
+
+    # === Phase 6: Sort by engagement ===
+    all_items.sort(
+        key=lambda x: (x.get("engagement", {}).get("score", 0) or 0),
+        reverse=True,
+    )
+
+    # Re-index IDs
+    for i, item in enumerate(all_items):
+        item["id"] = f"R{i+1}"
+
+    _log(f"Final: {len(all_items)} Reddit posts")
+    return {"items": all_items}
+
+
+def enrich_with_comments(
+    items: List[Dict[str, Any]],
+    token: str,
+    depth: str = "default",
+) -> List[Dict[str, Any]]:
+    """Enrich top items with comment data from ScrapeCreators.
+
+    Args:
+        items: Reddit items from search_reddit()
+        token: ScrapeCreators API key
+        depth: Depth for comment limit
+
+    Returns:
+        Items with top_comments and comment_insights added.
+    """
+    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+    max_comments = config["comment_enrichments"]
+
+    if not items or not token:
+        return items
+
+    top_items = items[:max_comments]
+    _log(f"Enriching comments for {len(top_items)} posts")
+
+    for item in top_items:
+        url = item.get("url", "")
+        if not url:
+            continue
+
+        raw_comments = fetch_post_comments(url, token)
+        if not raw_comments:
+            continue
+
+        # Parse comments into our format
+        top_comments = []
+        insights = []
+
+        for ci, c in enumerate(raw_comments[:10]):  # Take top 10 comments
+            body = c.get("body", "")
+            if not body or body in ("[deleted]", "[removed]"):
+                continue
+
+            score = c.get("ups") or c.get("score", 0)
+            author = c.get("author", "[deleted]")
+            permalink = c.get("permalink", "")
+            comment_url = f"https://reddit.com{permalink}" if permalink else ""
+
+            # Top comment gets more room (400 chars) — funny/clever comments need it
+            max_excerpt = 400 if ci == 0 else 300
+            top_comments.append({
+                "score": score,
+                "date": _parse_date(c.get("created_utc")),
+                "author": author,
+                "excerpt": body[:max_excerpt],
+                "url": comment_url,
+            })
+
+            # Extract insights from substantive comments
+            if len(body) >= 30 and author not in ("[deleted]", "[removed]", "AutoModerator"):
+                insight = body[:150]
+                if len(body) > 150:
+                    for i, char in enumerate(insight):
+                        if char in '.!?' and i > 50:
+                            insight = insight[:i+1]
+                            break
+                    else:
+                        insight = insight.rstrip() + "..."
+                insights.append(insight)
+
+        # Sort comments by score
+        top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
+
+        item["top_comments"] = top_comments[:10]
+        item["comment_insights"] = insights[:10]
+
+    return items
+
+
+def search_and_enrich(
+    topic: str,
+    from_date: str,
+    to_date: str,
+    depth: str = "default",
+    token: str = None,
+) -> Dict[str, Any]:
+    """Full Reddit pipeline: search + comment enrichment.
+
+    This is the convenience function that does everything.
+
+    Args:
+        topic: Search topic
+        from_date: Start date (YYYY-MM-DD)
+        to_date: End date (YYYY-MM-DD)
+        depth: 'quick', 'default', or 'deep'
+        token: ScrapeCreators API key
+
+    Returns:
+        Dict with 'items' list. Items include top_comments and comment_insights.
+    """
+    result = search_reddit(topic, from_date, to_date, depth, token)
+    items = result.get("items", [])
+
+    if items and token:
+        items = enrich_with_comments(items, token, depth)
+        result["items"] = items
+
+    return result
+
+
+def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
+    """Parse ScrapeCreators response to item list.
+
+    Compatibility shim matching openai_reddit.parse_reddit_response() signature.
+    """
+    return response.get("items", [])
@@ -1,4 +1,9 @@
-"""Reddit thread enrichment with real engagement metrics."""
+"""Reddit thread enrichment with real engagement metrics.
+
+Supports two backends:
+1. ScrapeCreators API (preferred) - no rate limits, 1 credit/call
+2. reddit.com/.json (fallback) - free but 429-prone
+"""

 import re
 from typing import Any, Dict, List, Optional
@@ -254,3 +259,67 @@ def enrich_reddit_item(
    item["comment_insights"] = extract_comment_insights(top_comments)

    return item
+
+
+def enrich_reddit_item_sc(
+    item: Dict[str, Any],
+    token: str,
+    timeout: int = 30,
+) -> Dict[str, Any]:
+    """Enrich a Reddit item using ScrapeCreators comment API.
+
+    No rate limit risk. Uses 1 credit per call.
+
+    Args:
+        item: Reddit item dict (already has engagement from search)
+        token: ScrapeCreators API key
+        timeout: HTTP timeout
+
+    Returns:
+        Enriched item with top_comments and comment_insights
+    """
+    from . import reddit as reddit_mod
+
+    url = item.get("url", "")
+    if not url:
+        return item
+
+    raw_comments = reddit_mod.fetch_post_comments(url, token)
+    if not raw_comments:
+        return item
+
+    top_comments = []
+    for c in raw_comments[:10]:
+        body = c.get("body", "")
+        if not body or body in ("[deleted]", "[removed]"):
+            continue
+
+        score = c.get("ups") or c.get("score", 0)
+        author = c.get("author", "[deleted]")
+        permalink = c.get("permalink", "")
+        comment_url = f"https://reddit.com{permalink}" if permalink else ""
+
+        top_comments.append({
+            "score": score,
+            "date": dates.timestamp_to_date(c.get("created_utc")) if c.get("created_utc") else None,
+            "author": author,
+            "body": body[:300],
+            "excerpt": body[:200],
+            "url": comment_url,
+        })
+
+    top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
+
+    item["top_comments"] = []
+    for c in top_comments:
+        item["top_comments"].append({
+            "score": c.get("score", 0),
+            "date": c.get("date"),
+            "author": c.get("author", ""),
+            "excerpt": c.get("excerpt", ""),
+            "url": c.get("url", ""),
+        })
+
+    item["comment_insights"] = extract_comment_insights(top_comments)
+
+    return item
@@ -26,6 +26,8 @@ def _xref_tag(item) -> str:
            source_names.add('YouTube')
        elif ref_id.startswith('TK'):
            source_names.add('TikTok')
+        elif ref_id.startswith('IG'):
+            source_names.add('Instagram')
        elif ref_id.startswith('HN'):
            source_names.add('HN')
        elif ref_id.startswith('PM'):
@@ -60,9 +62,10 @@ def _assess_data_freshness(report: schema.Report) -> dict:
    pm_recent = sum(1 for p in report.polymarket if p.date and p.date >= report.range_from)

    tiktok_recent = sum(1 for t in report.tiktok if t.date and t.date >= report.range_from)
+    ig_recent = sum(1 for ig in report.instagram if ig.date and ig.date >= report.range_from)

-    total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent + tiktok_recent
-    total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket) + len(report.tiktok)
+    total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent + tiktok_recent + ig_recent
+    total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket) + len(report.tiktok) + len(report.instagram)

    return {
        "reddit_recent": reddit_recent,
@@ -105,11 +108,11 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
        lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
        lines.append("")
        lines.append("---")
-        lines.append("**⚡ Want better results?** Add API keys or sign in to Codex to unlock Reddit & X data:")
-        lines.append("- `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments")
+        lines.append("**⚡ Want better results?** Add API keys to unlock Reddit, TikTok, Instagram & X data:")
+        lines.append("- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views")
        lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
+        lines.append("- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)")
        lines.append("- Edit `~/.config/last30days/.env` to add keys")
-        lines.append("- If already signed in but still seeing this, re-run `codex login`")
        lines.append("---")
        lines.append("")

@@ -134,7 +137,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
        lines.append("*💡 Tip: Add an xAI key (`XAI_API_KEY`) for X/Twitter data and better triangulation.*")
        lines.append("")
    elif report.mode == "x-only" and missing_keys in ("reddit", "none"):
-        lines.append("*💡 Tip: Add OPENAI_API_KEY or run `codex login` for Reddit data and better triangulation. If already signed in, re-run `codex login`.*")
+        lines.append("*💡 Tip: Add `SCRAPECREATORS_API_KEY` for Reddit + TikTok + Instagram data (one key, all three) and better triangulation.*")
        lines.append("")

    # Reddit items
@@ -171,7 +174,15 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
            lines.append(f"  {item.url}")
            lines.append(f"  *{item.why_relevant}*")

-            # Top comment insights
+            # Top comment (elevated — Reddit's value IS the comments)
+            if item.top_comments and item.top_comments[0].score >= 10:
+                tc = item.top_comments[0]
+                excerpt = tc.excerpt[:200]
+                if len(tc.excerpt) > 200:
+                    excerpt = excerpt.rstrip() + "..."
+                lines.append(f'  \U0001f4ac Top comment ({tc.score} upvotes): "{excerpt}"')
+
+            # Comment insights
            if item.comment_insights:
                lines.append("  Insights:")
                for insight in item.comment_insights[:3]:
@@ -284,6 +295,42 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")

+    # Instagram items
+    if report.instagram_error:
+        lines.append("### Instagram Reels")
+        lines.append("")
+        lines.append(f"**ERROR:** {report.instagram_error}")
+        lines.append("")
+    elif report.instagram:
+        lines.append("### Instagram Reels")
+        lines.append("")
+        for item in report.instagram[:limit]:
+            eng_str = ""
+            if item.engagement:
+                eng = item.engagement
+                parts = []
+                if eng.views is not None:
+                    parts.append(f"{eng.views:,} views")
+                if eng.likes is not None:
+                    parts.append(f"{eng.likes:,} likes")
+                if parts:
+                    eng_str = f" [{', '.join(parts)}]"
+
+            date_str = f" ({item.date})" if item.date else ""
+
+            lines.append(f"**{item.id}** (score:{item.score}) @{item.author_name}{date_str}{eng_str}{_xref_tag(item)}")
+            lines.append(f"  {item.text[:200]}")
+            lines.append(f"  {item.url}")
+            if item.caption_snippet and item.caption_snippet != item.text[:len(item.caption_snippet)]:
+                snippet = item.caption_snippet[:200]
+                if len(item.caption_snippet) > 200:
+                    snippet += "..."
+                lines.append(f"  Caption: {snippet}")
+            if item.hashtags:
+                lines.append(f"  Tags: {' '.join('#' + h for h in item.hashtags[:8])}")
+            lines.append(f"  *{item.why_relevant}*")
+            lines.append("")
+
    # Hacker News items
    if report.hackernews_error:
        lines.append("### Hacker News Stories")
@@ -455,6 +502,14 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
        lines.append(f"  ✅ TikTok: {len(report.tiktok)} videos ({with_captions} with captions)")
    # Hide when zero results

+    # Instagram
+    if report.instagram_error:
+        lines.append(f"  ❌ Instagram: error — {report.instagram_error}")
+    elif report.instagram:
+        with_captions = sum(1 for v in report.instagram if getattr(v, 'caption_snippet', None))
+        lines.append(f"  ✅ Instagram: {len(report.instagram)} reels ({with_captions} with captions)")
+    # Hide when zero results
+
    # Hacker News
    if report.hackernews_error:
        lines.append(f"  ❌ HN: error - {report.hackernews_error}")
@@ -508,6 +563,8 @@ def render_context_snippet(report: schema.Report) -> str:
        all_items.append((item.score, "X", item.text[:50] + "...", item.url))
    for item in report.tiktok[:5]:
        all_items.append((item.score, "TikTok", item.text[:50] + "...", item.url))
+    for item in report.instagram[:5]:
+        all_items.append((item.score, "Instagram", item.text[:50] + "...", item.url))
    for item in report.hackernews[:5]:
        all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
    for item in report.polymarket[:5]:
@@ -573,6 +630,15 @@ def render_full_report(report: schema.Report) -> str:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")

+            if item.top_comments and item.top_comments[0].score >= 10:
+                tc = item.top_comments[0]
+                excerpt = tc.excerpt[:200]
+                if len(tc.excerpt) > 200:
+                    excerpt = excerpt.rstrip() + "..."
+                lines.append("")
+                lines.append(f'**\U0001f4ac Top Comment** ({tc.score} upvotes, u/{tc.author}):')
+                lines.append(f'> {excerpt}')
+
            if item.comment_insights:
                lines.append("")
                lines.append("**Key Insights from Comments:**")
@@ -624,6 +690,29 @@ def render_full_report(report: schema.Report) -> str:
            lines.append(f"> {item.text[:300]}")
            lines.append("")

+    # Instagram section
+    if report.instagram:
+        lines.append("## Instagram Reels")
+        lines.append("")
+        for item in report.instagram:
+            lines.append(f"### {item.id}: @{item.author_name}")
+            lines.append("")
+            lines.append(f"- **URL:** {item.url}")
+            lines.append(f"- **Date:** {item.date or 'Unknown'}")
+            lines.append(f"- **Score:** {item.score}/100")
+            lines.append(f"- **Relevance:** {item.why_relevant}")
+
+            if item.engagement:
+                eng = item.engagement
+                lines.append(f"- **Engagement:** {eng.views or '?'} views, {eng.likes or '?'} likes, {eng.num_comments or '?'} comments")
+
+            if item.hashtags:
+                lines.append(f"- **Hashtags:** {' '.join('#' + h for h in item.hashtags[:10])}")
+
+            lines.append("")
+            lines.append(f"> {item.text[:300]}")
+            lines.append("")
+
    # HN section
    if report.hackernews:
        lines.append("## Hacker News Stories")
@@ -275,6 +275,45 @@ class TikTokItem:
        return d


+@dataclass
+class InstagramItem:
+    """Normalized Instagram item."""
+    id: str                    # "IG1", "IG2", ...
+    text: str                  # caption text
+    url: str                   # https://www.instagram.com/reel/{code}
+    author_name: str           # Instagram handle
+    date: Optional[str] = None
+    date_confidence: str = "high"  # ScrapeCreators provides exact timestamps
+    engagement: Optional[Engagement] = None  # views, likes, num_comments
+    caption_snippet: str = ""  # spoken-word caption (if available), else text
+    hashtags: List[str] = field(default_factory=list)
+    relevance: float = 0.7
+    why_relevant: str = ""
+    subs: SubScores = field(default_factory=SubScores)
+    score: int = 0
+    cross_refs: List[str] = field(default_factory=list)
+
+    def to_dict(self) -> Dict[str, Any]:
+        d = {
+            'id': self.id,
+            'text': self.text,
+            'url': self.url,
+            'author_name': self.author_name,
+            'date': self.date,
+            'date_confidence': self.date_confidence,
+            'engagement': self.engagement.to_dict() if self.engagement else None,
+            'caption_snippet': self.caption_snippet,
+            'hashtags': self.hashtags,
+            'relevance': self.relevance,
+            'why_relevant': self.why_relevant,
+            'subs': self.subs.to_dict(),
+            'score': self.score,
+        }
+        if self.cross_refs:
+            d['cross_refs'] = self.cross_refs
+        return d
+
+
@dataclass
 class HackerNewsItem:
    """Normalized Hacker News item."""
@@ -374,6 +413,7 @@ class Report:
    web: List[WebSearchItem] = field(default_factory=list)
    youtube: List[YouTubeItem] = field(default_factory=list)
    tiktok: List[TikTokItem] = field(default_factory=list)
+    instagram: List[InstagramItem] = field(default_factory=list)
    hackernews: List[HackerNewsItem] = field(default_factory=list)
    polymarket: List[PolymarketItem] = field(default_factory=list)
    best_practices: List[str] = field(default_factory=list)
@@ -385,6 +425,7 @@ class Report:
    web_error: Optional[str] = None
    youtube_error: Optional[str] = None
    tiktok_error: Optional[str] = None
+    instagram_error: Optional[str] = None
    hackernews_error: Optional[str] = None
    polymarket_error: Optional[str] = None
    # Handle resolution
@@ -409,6 +450,7 @@ class Report:
            'web': [w.to_dict() for w in self.web],
            'youtube': [y.to_dict() for y in self.youtube],
            'tiktok': [t.to_dict() for t in self.tiktok],
+            'instagram': [ig.to_dict() for ig in self.instagram],
            'hackernews': [h.to_dict() for h in self.hackernews],
            'polymarket': [p.to_dict() for p in self.polymarket],
            'best_practices': self.best_practices,
@@ -427,6 +469,8 @@ class Report:
            d['youtube_error'] = self.youtube_error
        if self.tiktok_error:
            d['tiktok_error'] = self.tiktok_error
+        if self.instagram_error:
+            d['instagram_error'] = self.instagram_error
        if self.hackernews_error:
            d['hackernews_error'] = self.hackernews_error
        if self.polymarket_error:
@@ -558,6 +602,30 @@ class Report:
                cross_refs=t.get('cross_refs', []),
            ))

+        # Reconstruct Instagram items
+        ig_items = []
+        for ig in data.get('instagram', []):
+            eng = None
+            if ig.get('engagement'):
+                eng = Engagement(**ig['engagement'])
+            subs = SubScores(**ig.get('subs', {})) if ig.get('subs') else SubScores()
+            ig_items.append(InstagramItem(
+                id=ig['id'],
+                text=ig.get('text', ''),
+                url=ig['url'],
+                author_name=ig.get('author_name', ''),
+                date=ig.get('date'),
+                date_confidence=ig.get('date_confidence', 'high'),
+                engagement=eng,
+                caption_snippet=ig.get('caption_snippet', ''),
+                hashtags=ig.get('hashtags', []),
+                relevance=ig.get('relevance', 0.7),
+                why_relevant=ig.get('why_relevant', ''),
+                subs=subs,
+                score=ig.get('score', 0),
+                cross_refs=ig.get('cross_refs', []),
+            ))
+
        # Reconstruct HackerNews items
        hn_items = []
        for h in data.get('hackernews', []):
@@ -623,6 +691,7 @@ class Report:
            web=web_items,
            youtube=youtube_items,
            tiktok=tiktok_items,
+            instagram=ig_items,
            hackernews=hn_items,
            polymarket=pm_items,
            best_practices=data.get('best_practices', []),
@@ -633,6 +702,7 @@ class Report:
            web_error=data.get('web_error'),
            youtube_error=data.get('youtube_error'),
            tiktok_error=data.get('tiktok_error'),
+            instagram_error=data.get('instagram_error'),
            hackernews_error=data.get('hackernews_error'),
            polymarket_error=data.get('polymarket_error'),
            resolved_x_handle=data.get('resolved_x_handle'),
@@ -31,10 +31,16 @@ def log1p_safe(x: Optional[int]) -> float:
    return math.log1p(x)


-def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
+def compute_reddit_engagement_raw(
+    engagement: Optional[schema.Engagement],
+    top_comment_score: Optional[int] = None,
+) -> Optional[float]:
    """Compute raw engagement score for Reddit item.

-    Formula: 0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)
+    Formula: 0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)
+
+    The 10% comment quality weight rewards posts where the community engaged deeply
+    — a highly upvoted top comment means the thread sparked real discussion.
    """
    if engagement is None:
        return None
@@ -45,8 +51,9 @@ def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Op
    score = log1p_safe(engagement.score)
    comments = log1p_safe(engagement.num_comments)
    ratio = (engagement.upvote_ratio or 0.5) * 10
+    top_cmt = log1p_safe(top_comment_score)

-    return 0.55 * score + 0.40 * comments + 0.05 * ratio
+    return 0.50 * score + 0.35 * comments + 0.05 * ratio + 0.10 * top_cmt


 def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
@@ -113,8 +120,13 @@ def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem
    if not items:
        return items

-    # Compute raw engagement scores
-    eng_raw = [compute_reddit_engagement_raw(item.engagement) for item in items]
+    # Compute raw engagement scores (with top comment quality signal)
+    eng_raw = []
+    for item in items:
+        top_cmt_score = None
+        if item.top_comments:
+            top_cmt_score = item.top_comments[0].score
+        eng_raw.append(compute_reddit_engagement_raw(item.engagement, top_cmt_score))

    # Normalize engagement to 0-100
    eng_normalized = normalize_to_100(eng_raw)
@@ -339,6 +351,65 @@ def score_tiktok_items(items: List[schema.TikTokItem]) -> List[schema.TikTokItem
    return items


+def compute_instagram_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
+    """Compute raw engagement score for Instagram item.
+
+    Formula: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
+    Views dominate on Instagram Reels — they're the primary discovery signal.
+    """
+    if engagement is None:
+        return None
+
+    if engagement.views is None and engagement.likes is None:
+        return None
+
+    views = log1p_safe(engagement.views)
+    likes = log1p_safe(engagement.likes)
+    comments = log1p_safe(engagement.num_comments)
+
+    return 0.50 * views + 0.30 * likes + 0.20 * comments
+
+
+def score_instagram_items(items: List[schema.InstagramItem]) -> List[schema.InstagramItem]:
+    """Compute scores for Instagram items.
+
+    Uses same weight structure as TikTok (relevance + recency + engagement).
+    """
+    if not items:
+        return items
+
+    eng_raw = [compute_instagram_engagement_raw(item.engagement) for item in items]
+    eng_normalized = normalize_to_100(eng_raw)
+
+    for i, item in enumerate(items):
+        rel_score = int(item.relevance * 100)
+        rec_score = dates.recency_score(item.date)
+
+        if eng_normalized[i] is not None:
+            eng_score = int(eng_normalized[i])
+        else:
+            eng_score = DEFAULT_ENGAGEMENT
+
+        item.subs = schema.SubScores(
+            relevance=rel_score,
+            recency=rec_score,
+            engagement=eng_score,
+        )
+
+        overall = (
+            WEIGHT_RELEVANCE * rel_score +
+            WEIGHT_RECENCY * rec_score +
+            WEIGHT_ENGAGEMENT * eng_score
+        )
+
+        if eng_raw[i] is None:
+            overall -= UNKNOWN_ENGAGEMENT_PENALTY
+
+        item.score = max(0, min(100, int(overall)))
+
+    return items
+
+
 def compute_hackernews_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for Hacker News item.

@@ -512,7 +583,7 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
    return items


-def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
+def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
    """Sort items by score (descending), then date, then source priority.

    Args:
@@ -538,12 +609,14 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
            source_priority = 2
        elif isinstance(item, schema.TikTokItem):
            source_priority = 3
-        elif isinstance(item, schema.HackerNewsItem):
+        elif isinstance(item, schema.InstagramItem):
            source_priority = 4
-        elif isinstance(item, schema.PolymarketItem):
+        elif isinstance(item, schema.HackerNewsItem):
            source_priority = 5
-        else:  # WebSearchItem
+        elif isinstance(item, schema.PolymarketItem):
            source_priority = 6
+        else:  # WebSearchItem
+            source_priority = 7

        # Quaternary: title/text for stability
        text = getattr(item, "title", "") or getattr(item, "text", "")
@@ -1,9 +1,10 @@
-"""TikTok search via Apify clockworks/tiktok-scraper for /last30days.
+"""TikTok search via ScrapeCreators API for /last30days.

-Uses the Apify platform to search TikTok by keyword, extract engagement
-metrics (views, likes, comments), and optionally pull video captions.
+Uses ScrapeCreators REST API to search TikTok by keyword, extract engagement
+metrics (views, likes, comments, shares), and fetch video transcripts.

-Requires APIFY_API_TOKEN in config. Free tier: $5/month credits.
+Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
+API docs: https://scrapecreators.com/docs
 """

 import re
@@ -11,9 +12,12 @@ import sys
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional, Set

-from . import apify_client_wrapper
+try:
+    import requests as _requests
+except ImportError:
+    _requests = None

-ACTOR_ID = "clockworks/tiktok-scraper"
+SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/tiktok"

 # Depth configurations: how many results to fetch / captions to extract
 DEPTH_CONFIG = {
@@ -76,7 +80,7 @@ def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> flo
        combined = f"{text} {' '.join(hashtags)}"
    t_tokens = _tokenize(combined)

-    # Split concatenated hashtags (e.g., "claudecode" → "claude", "code")
+    # Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
    if hashtags:
        for tag in hashtags:
            tag_lower = tag.lower()
@@ -134,20 +138,20 @@ def _log(msg: str):
        sys.stderr.flush()


+def _sc_headers(token: str) -> Dict[str, str]:
+    """Build ScrapeCreators request headers."""
+    return {
+        "x-api-key": token,
+        "Content-Type": "application/json",
+    }
+
+
 def _parse_date(item: Dict[str, Any]) -> Optional[str]:
-    """Parse date from Apify TikTok item to YYYY-MM-DD.
+    """Parse date from ScrapeCreators TikTok item to YYYY-MM-DD.

-    Handles both createTimeISO (ISO string) and createTime (unix timestamp).
+    Handles create_time (unix timestamp).
    """
-    iso = item.get("createTimeISO")
-    if iso:
-        try:
-            dt = datetime.fromisoformat(iso.replace("Z", "+00:00"))
-            return dt.strftime("%Y-%m-%d")
-        except (ValueError, TypeError):
-            pass
-
-    ts = item.get("createTime")
+    ts = item.get("create_time")
    if ts:
        try:
            dt = datetime.fromtimestamp(int(ts), tz=timezone.utc)
@@ -158,6 +162,26 @@ def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    return None


+def _clean_webvtt(text: str) -> str:
+    """Strip WebVTT timestamps and headers from transcript text."""
+    if not text:
+        return ""
+    lines = text.split('\n')
+    cleaned = []
+    for line in lines:
+        line = line.strip()
+        if not line:
+            continue
+        if line.startswith('WEBVTT'):
+            continue
+        if re.match(r'^\d{2}:\d{2}', line):
+            continue
+        if '-->' in line:
+            continue
+        cleaned.append(line)
+    return ' '.join(cleaned)
+
+
 def search_tiktok(
    topic: str,
    from_date: str,
@@ -165,23 +189,23 @@ def search_tiktok(
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
-    """Search TikTok via Apify.
+    """Search TikTok via ScrapeCreators API.

    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
-        token: Apify API token
+        token: ScrapeCreators API key

    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
-        return {"items": [], "error": "No APIFY_API_TOKEN configured"}
+        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}

-    if not apify_client_wrapper.is_apify_available():
-        return {"items": [], "error": "apify-client not installed (pip install apify-client)"}
+    if not _requests:
+        return {"items": [], "error": "requests library not installed"}

    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
@@ -189,48 +213,61 @@ def search_tiktok(
    _log(f"Searching TikTok for '{core_topic}' (depth={depth}, count={config['results_per_page']})")

    try:
-        client = apify_client_wrapper.get_apify_client(token)
-        run_input = {
-            "searchQueries": [core_topic],
-            "resultsPerPage": config["results_per_page"],
-            "shouldDownloadSubtitles": False,
-            "shouldDownloadVideos": False,
-            "shouldDownloadCovers": False,
-        }
-        raw_items = apify_client_wrapper.run_actor_sync(
-            client, ACTOR_ID, run_input,
-            timeout_secs=120,
-            max_items=config["results_per_page"],
+        resp = _requests.get(
+            f"{SCRAPECREATORS_BASE}/search/keyword",
+            params={"query": core_topic, "sort_by": "relevance"},
+            headers=_sc_headers(token),
+            timeout=30,
        )
+        resp.raise_for_status()
+        data = resp.json()
    except Exception as e:
-        _log(f"Apify error: {e}")
+        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}

+    # Items are nested under aweme_info
+    raw_entries = data.get("search_item_list") or data.get("data") or []
+    raw_items = []
+    for entry in raw_entries:
+        if isinstance(entry, dict):
+            info = entry.get("aweme_info", entry)
+            raw_items.append(info)
+
+    # Limit to configured count
+    raw_items = raw_items[:config["results_per_page"]]
+
    # Parse items
    items = []
    for raw in raw_items:
-        video_id = str(raw.get("id", ""))
-        text = raw.get("text", "")
-        play_count = raw.get("playCount") or 0
-        digg_count = raw.get("diggCount") or 0
-        comment_count = raw.get("commentCount") or 0
-        share_count = raw.get("shareCount") or 0
-        author_meta = raw.get("authorMeta") or {}
-        author_name = author_meta.get("name", "")
-        web_url = raw.get("webVideoUrl", "")
-        hashtags_raw = raw.get("hashtags") or []
-        hashtag_names = [h.get("name", "") for h in hashtags_raw if isinstance(h, dict)]
-        duration = (raw.get("videoMeta") or {}).get("duration")
+        video_id = str(raw.get("aweme_id", ""))
+        text = raw.get("desc", "")
+        stats = raw.get("statistics") or {}
+        play_count = stats.get("play_count") or 0
+        digg_count = stats.get("digg_count") or 0
+        comment_count = stats.get("comment_count") or 0
+        share_count = stats.get("share_count") or 0
+        author = raw.get("author") or {}
+        author_name = author.get("unique_id", "")
+        share_url = raw.get("share_url", "")
+        text_extra = raw.get("text_extra") or []
+        hashtag_names = [t.get("hashtag_name", "") for t in text_extra
+                         if isinstance(t, dict) and t.get("hashtag_name")]
+        duration = (raw.get("video") or {}).get("duration")

        date_str = _parse_date(raw)

        # Compute relevance with hashtag boost
        relevance = _compute_relevance(core_topic, text, hashtag_names)

+        # Build URL: prefer share_url, fallback to constructed URL
+        url = share_url.split("?")[0] if share_url else ""
+        if not url and author_name and video_id:
+            url = f"https://www.tiktok.com/@{author_name}/video/{video_id}"
+
        items.append({
            "video_id": video_id,
            "text": text,
-            "url": web_url or f"https://www.tiktok.com/@{author_name}/video/{video_id}",
+            "url": url,
            "author_name": author_name,
            "date": date_str,
            "engagement": {
@@ -268,24 +305,24 @@ def fetch_captions(
    token: str,
    depth: str = "default",
 ) -> Dict[str, str]:
-    """Fetch captions for top N TikTok videos.
+    """Fetch transcripts for top N TikTok videos via ScrapeCreators.

    Strategy:
-    1. Primary: Use the 'text' field (video description) — always free
-    2. For top N, re-run actor with shouldDownloadSubtitles for spoken-word
+    1. Use the 'text' field (video description) as baseline caption
+    2. For top N, call /video/transcript for spoken-word captions

    Args:
        video_items: Items from search_tiktok()
-        token: Apify API token
+        token: ScrapeCreators API key
        depth: Depth level for caption limit

    Returns:
-        Dict mapping video_id → caption text (truncated to 500 words)
+        Dict mapping video_id -> caption text (truncated to 500 words)
    """
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    max_captions = config["max_captions"]

-    if not video_items or not token:
+    if not video_items or not token or not _requests:
        return {}

    top_items = video_items[:max_captions]
@@ -303,35 +340,33 @@ def fetch_captions(
                text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
            captions[vid] = text

-    # Second pass: try to get spoken-word subtitles for top videos
-    try:
-        urls = [item["url"] for item in top_items if item.get("url")]
-        if urls:
-            client = apify_client_wrapper.get_apify_client(token)
-            run_input = {
-                "postURLs": urls,
-                "shouldDownloadSubtitles": True,
-                "shouldDownloadVideos": False,
-                "shouldDownloadCovers": False,
-            }
-            subtitle_items = apify_client_wrapper.run_actor_sync(
-                client, ACTOR_ID, run_input,
-                timeout_secs=60,
-                max_items=max_captions,
+    # Second pass: try to get spoken-word transcripts (1 credit each)
+    for item in top_items:
+        vid = item["video_id"]
+        url = item.get("url", "")
+        if not url:
+            continue
+        try:
+            resp = _requests.get(
+                f"{SCRAPECREATORS_BASE}/video/transcript",
+                params={"url": url},
+                headers=_sc_headers(token),
+                timeout=15,
            )
-            for raw in subtitle_items:
-                vid = str(raw.get("id", ""))
-                # Check for subtitle text in the response
-                subtitle_text = raw.get("subtitleText") or raw.get("subtitles") or ""
-                if isinstance(subtitle_text, list):
-                    subtitle_text = " ".join(str(s) for s in subtitle_text)
-                if subtitle_text and vid:
-                    words = subtitle_text.split()
-                    if len(words) > CAPTION_MAX_WORDS:
-                        subtitle_text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
-                    captions[vid] = subtitle_text  # Override text with spoken-word
-    except Exception as e:
-        _log(f"Subtitle enrichment failed (using text captions): {e}")
+            if resp.status_code == 200:
+                data = resp.json()
+                transcript = data.get("transcript")
+                if transcript:
+                    if isinstance(transcript, list):
+                        transcript = " ".join(str(s) for s in transcript)
+                    transcript = _clean_webvtt(transcript)
+                    if transcript:
+                        words = transcript.split()
+                        if len(words) > CAPTION_MAX_WORDS:
+                            transcript = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
+                        captions[vid] = transcript
+        except Exception as e:
+            _log(f"Transcript fetch failed for {vid}: {e}")

    got = sum(1 for v in captions.values() if v)
    _log(f"Got captions for {got}/{len(top_items)} videos")
@@ -352,7 +387,7 @@ def search_and_enrich(
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
-        token: Apify API token
+        token: ScrapeCreators API key

    Returns:
        Dict with 'items' list. Each item has a 'caption_snippet' field.
@@ -77,6 +77,12 @@ TIKTOK_MESSAGES = [
    "Scanning TikTok for relevant content...",
 ]

+INSTAGRAM_MESSAGES = [
+    "Searching Instagram Reels...",
+    "Finding what's trending on Instagram...",
+    "Scanning Instagram for relevant reels...",
+]
+
 HN_MESSAGES = [
    "Searching Hacker News...",
    "Scanning HN front page stories...",
@@ -286,6 +292,15 @@ class ProgressDisplay:
        if self.spinner:
            self.spinner.stop(f"{Colors.PURPLE}TikTok{Colors.RESET} Found {count} videos")

+    def start_instagram(self):
+        msg = random.choice(INSTAGRAM_MESSAGES)
+        self.spinner = Spinner(f"{Colors.PURPLE}Instagram{Colors.RESET} {msg}", Colors.PURPLE)
+        self.spinner.start()
+
+    def end_instagram(self, count: int):
+        if self.spinner:
+            self.spinner.stop(f"{Colors.PURPLE}Instagram{Colors.RESET} Found {count} reels")
+
    def start_hackernews(self):
        msg = random.choice(HN_MESSAGES)
        self.spinner = Spinner(f"{Colors.YELLOW}HN{Colors.RESET} {msg}", Colors.YELLOW, quiet=True)
@@ -313,7 +328,7 @@ class ProgressDisplay:
        if self.spinner:
            self.spinner.stop()

-    def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0, tiktok_count: int = 0):
+    def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0, tiktok_count: int = 0, ig_count: int = 0):
        elapsed = time.time() - self.start_time
        if IS_TTY:
            sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
@@ -324,6 +339,8 @@ class ProgressDisplay:
                sys.stderr.write(f"  {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
            if tiktok_count:
                sys.stderr.write(f"  {Colors.PURPLE}TikTok:{Colors.RESET} {tiktok_count} videos")
+            if ig_count:
+                sys.stderr.write(f"  {Colors.PURPLE}Instagram:{Colors.RESET} {ig_count} reels")
            if hn_count:
                sys.stderr.write(f"  {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories")
            if pm_count:
@@ -335,6 +352,8 @@ class ProgressDisplay:
                parts.append(f"YouTube: {youtube_count} videos")
            if tiktok_count:
                parts.append(f"TikTok: {tiktok_count} videos")
+            if ig_count:
+                parts.append(f"Instagram: {ig_count} reels")
            if hn_count:
                parts.append(f"HN: {hn_count} stories")
            if pm_count:
@@ -211,6 +211,16 @@ def _run_topic(topic: dict) -> dict:
                "engagement_score": (item.get("engagement") or {}).get("views", 0),
                "relevance_score": item.get("relevance", 0),
            })
+        for item in data.get("instagram", []):
+            findings.append({
+                "source": "instagram",
+                "url": item.get("url", ""),
+                "title": (item.get("caption_snippet", "") or "")[:120],
+                "author": item.get("author_name", ""),
+                "content": item.get("caption_snippet", ""),
+                "engagement_score": (item.get("engagement") or {}).get("views", 0),
+                "relevance_score": item.get("relevance", 0),
+            })

        # Store with dedup
        counts = store.store_findings(run_id, topic_id, findings)
@@ -58,14 +58,10 @@ class TestExtractCoreSubject(unittest.TestCase):


 class TestParseDate(unittest.TestCase):
-    """Test date parsing from Apify items."""
-
-    def test_iso_date(self):
-        item = {"createTimeISO": "2026-02-28T17:44:35.000Z"}
-        self.assertEqual(tiktok._parse_date(item), "2026-02-28")
+    """Test date parsing from ScrapeCreators items."""

    def test_unix_timestamp(self):
-        item = {"createTime": 1756403075}
+        item = {"create_time": 1756403075}
        result = tiktok._parse_date(item)
        self.assertIsNotNone(result)
        self.assertRegex(result, r"\d{4}-\d{2}-\d{2}")
@@ -75,6 +71,19 @@ class TestParseDate(unittest.TestCase):
        self.assertIsNone(tiktok._parse_date(item))


+class TestCleanWebVTT(unittest.TestCase):
+    """Test WebVTT transcript cleaning."""
+
+    def test_strips_timestamps(self):
+        raw = "WEBVTT\n\n00:00:00.000 --> 00:00:02.000\nHello world\n\n00:00:02.000 --> 00:00:04.000\nGoodbye"
+        result = tiktok._clean_webvtt(raw)
+        self.assertEqual(result, "Hello world Goodbye")
+
+    def test_empty_input(self):
+        self.assertEqual(tiktok._clean_webvtt(""), "")
+        self.assertEqual(tiktok._clean_webvtt(None), "")
+
+
 class TestNormalizeTikTokItems(unittest.TestCase):
    """Test TikTok normalization."""
Author	SHA1	Message	Date
Matt Van Horn	e690d61a12	fix: append -raw suffix to saved research filenames e.g. sam-altman-raw.md instead of sam-altman.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 16:56:03 -08:00
Matt Van Horn	6d5acb9121	feat(release): v2.9.4 - move save into Python script, zero post-invitation noise Add --save-dir flag to last30days.py that saves raw research output during the existing script run. Remove entire "Save Research to Documents" section from SKILL.md (~45 lines). No more extra tool calls, no (No output), no multi-minute cogitation after invitation. Tested: --mock confirms file creation and duplicate date suffixing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 16:27:53 -08:00
Matt Van Horn	28d223e43c	fix(release): v2.9.3 - foreground save, fix hallucinated user messages CRITICAL: run_in_background callbacks caused model to re-engage after save, hallucinate fake "Human:" messages, and generate unsolicited multi-paragraph responses. Switch to foreground cat > heredoc which executes sub-second with no callback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 15:50:17 -08:00
Matt Van Horn	471badd329	fix(release): v2.9.2 - silent save, no follow-up text after background save - Background Bash heredoc instead of Write tool - Suppress response text on save completion - 📎 footer line replaces verbose confirmation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 15:41:55 -08:00
Matt Van Horn	5b1636f94f	fix: merge upstream save fix (background Bash instead of Write tool) Resolves merge conflict, keeping upstream's approach: - Background heredoc save instead of Write tool - Adds 📎 footer line - No more "Wrote N lines..." cluttering output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 14:59:06 -08:00
Matt Van Horn	18f6273f7c	fix: save research silently via background Bash, not Write tool The Write tool displays "Wrote N lines..." after the invitation, ruining the end-of-run experience. Now saves via background Bash with a subtle 📎 footer line in the invitation text. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-06 10:34:22 -08:00
Matt Van Horn	cc774d5e69	feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/ Sync from public repo. Every run now saves the complete briefing as a topic-named .md file to ~/Documents/Last30Days/. Credit @devin_explores. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 19:50:43 -08:00
Matt Van Horn	8cbbe87c3e	docs: add v2.9.1 auto-save note to README Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 19:27:18 -08:00
Matt Van Horn	49d993b162	feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/ Bump version to 2.9.1, update changelog and release notes. Credit @devin_explores for inspiring the feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 19:15:42 -08:00
Matt Van Horn	6afc094bb1	Merge pull request #51 from mvanhorn/feat/auto-save-documents feat(skill): auto-save research results to ~/Documents/Last30Days/	2026-03-05 19:12:56 -08:00
Matt Van Horn	f6a1769e35	feat(skill): auto-save research results to ~/Documents/Last30Days/ Every run now automatically saves the complete briefing (synthesis, stats, follow-up suggestions) as a topic-named .md file in the user's Documents folder. Agent mode also saves. No Python script changes - this is purely a SKILL.md instruction addition. Inspired by @devin_explores manually saving results to build a personal research library. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 19:06:27 -08:00
Matt Van Horn	4d35b53eab	docs: v2.9.0 release — ScrapeCreators Reddit default, top comments, smart discovery Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 18:03:40 -08:00
Matt Van Horn	2247800003	chore: clean up Reddit log prefix, mark plan tasks complete	2026-03-05 18:01:24 -08:00
Matt Van Horn	7048fe7b83	feat(reddit): elevate top comments, improve subreddit discovery, default to ScrapeCreators Three improvements from beta testing: 1. Top comments: 10% scoring weight for comment quality, 💬 top comment rendered prominently in compact/full output, increased insight limits 2. Subreddit discovery: relevance-weighted scoring with topic word matching, utility sub penalties (UTILITY_SUBS blocklist), engagement bonus 3. Default method: SKILL.md primaryEnv → SCRAPECREATORS_API_KEY, web-only banner recommends SC first, security section updated Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 17:29:18 -08:00
Matt Van Horn	30b973f62e	docs: add Reddit ScrapeCreators v2 improvements plan Three focused improvements based on 5 full-pipeline beta tests: 1. Elevate top Reddit comments in scoring and rendering 2. Improve subreddit discovery heuristic for ambiguous queries 3. Make ScrapeCreators the default recommended Reddit method Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 17:24:41 -08:00
Matt Van Horn	09b09946c0	feat: replace OpenAI Reddit search with ScrapeCreators API - New scripts/lib/reddit.py: multi-query expansion, global search, subreddit discovery, targeted subreddit search, comment enrichment - 68 results in 17s vs ~15 results in 60-90s (OpenAI) - Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper) - Real engagement data (score, comments, dates) from API - No more 429 rate limits on comment enrichment - Falls back to OpenAI if SCRAPECREATORS_API_KEY missing - Registered as last30daysbeta for parallel local testing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 15:55:02 -08:00
Matt Van Horn	db75f9e341	feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration Add Instagram Reels as the 8th research source via ScrapeCreators API. One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram. - Add scripts/lib/instagram.py: keyword search, transcript extraction, relevance scoring, engagement metrics (views, likes, comments) - Add InstagramItem to schema, normalization, scoring, dedup, rendering - Add Instagram to orchestrator pipeline, watchlist, and UI spinners - Update SKILL.md: stats template, citation priority, item format, URL-to-name extraction rules, anti-Sources instruction - Update README and CHANGELOG for v2.8 - Fix: Instagram/TikTok not running in --search= web-only path - Fix: web stats line showing full URLs instead of domain names - Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 07:00:51 -08:00
Matt Van Horn	740dcc5789	docs: update README and SKILL.md for ScrapeCreators TikTok API Replace all Apify references with ScrapeCreators. Key points: - No subscription required (was $5/mo with Apify) - 100 free credits, pay-as-you-go after - SCRAPECREATORS_API_KEY replaces APIFY_API_TOKEN - Backwards compatible: APIFY_API_TOKEN still works as fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 05:17:30 -08:00
Matt Van Horn	e03046bd49	refactor(tiktok): replace Apify with ScrapeCreators API Root cause of empty TikTok results: Apify required monthly subscription. ScrapeCreators is PAYG with 100 free credits and no subscription. Key fix: ScrapeCreators nests items under aweme_info wrapper (search_item_list[].aweme_info.{fields}), which the previous implementation missed, causing all fields to be empty. Changes: - Rewrite tiktok.py to use ScrapeCreators REST API - Add aweme_info unwrapping for correct field extraction - Add transcript fetching via /video/transcript endpoint - Add SCRAPECREATORS_API_KEY to env.py config - Update last30days.py to use env.get_tiktok_token() - Delete apify_client_wrapper.py (no longer needed) - Update tests for new date field format (create_time) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 13:58:51 -08:00
Matt Van Horn	1d18bee1a2	fix(skill): forward CLI flags through $ARGUMENTS to Python script Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits the expansion, and change argparse topic from nargs="?" to nargs="*" so multi-word topics still work. Also document --store, --include-web, --diagnose, and --timeout flags in the Options section. Closes #36 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 13:52:08 -08:00