docs: v2.9.0 release — ScrapeCreators Reddit default, top comments, smart discovery

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
chore: clean up Reddit log prefix, mark plan tasks complete
2026-03-05 18:03:40 -08:00 · 2026-03-05 18:01:24 -08:00 · 2026-03-05 17:29:18 -08:00 · 2026-03-05 17:24:41 -08:00 · 2026-03-05 15:55:02 -08:00 · 2026-03-04 07:00:51 -08:00
40 changed files with 6165 additions and 169 deletions
@@ -1,4 +1,5 @@
 {
  "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
  "name": "last30days",
  "owner": {
    "name": "mvanhorn",
@@ -11,7 +12,7 @@
  "plugins": [
    {
      "name": "last30days",
-      "source": "."
+      "source": "./"
    }
  ]
 }
@@ -0,0 +1,16 @@
 # Private benchmark / evaluation artifacts — never push to upstream
 docs/comparison-results/
 scripts/evaluate-synthesis.py
 scripts/generate-synthesis-inputs.py
 fixtures/polymarket_sample.json
 docs/v2.1-tweets.md
 docs/30-day-anniversary-thread.md
 docs/30-day-anniversary-tweets.md
 variants/open/references/research.md
 # OS / tool files
 .DS_Store
 .claude/
 .entire/
 __pycache__/
 *.pyc
@@ -5,6 +5,61 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [2.9.0] - 2026-03-05
 ### Highlights
 ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
 ### Added
 - ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
 - Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
 - `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
 - Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
 - Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
 - Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
 ### Changed
 - `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
 - Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
 - SKILL.md synthesis instructions updated to emphasize quoting top comments
 ### Fixed
 - Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
 - Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
 ## [2.8.0] - 2026-03-04
 ### Highlights
 Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.
 ### Added
 - Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
 - `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
 - Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
 - URL-to-name extraction examples in SKILL.md for cleaner web source display
 - `--search=instagram` flag support
 ### Changed
 - TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
 - `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
 - SKILL.md version bumped to v2.8
 - WebSearch citation instruction strengthened to prevent trailing Sources: blocks
 - Security section updated: Apify → ScrapeCreators references
 ### Fixed
 - Web stats line showing full URLs instead of plain domain names
 - Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
 - Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
 - `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding
 ## [2.1.0] - 2026-02-15
 ### Highlights
@@ -59,5 +114,7 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
 Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
 [2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
 [2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
 [2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
 [1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
@@ -0,0 +1,21 @@
 MIT License
 Copyright (c) 2026 Matt Van Horn
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
@@ -1,6 +1,14 @@
-# /last30days v2.5
+# /last30days v2.9
-**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
+**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
 **New in v2.9 — ScrapeCreators Reddit + Top Comments + Smart Discovery:**
 Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default — one `SCRAPECREATORS_API_KEY` covers Reddit, TikTok, and Instagram (3 sources, 1 key). Smart subreddit discovery finds the right communities automatically, and top comments are elevated with a 10% scoring weight and `💬` display with upvote counts. [Details below.](#whats-new-in-v29)
 **New in v2.8 — Instagram Reels + ScrapeCreators:**
 Instagram Reels is now the 8th signal source. TikTok and Instagram both run on ScrapeCreators — one API key covers both. [Details below.](#whats-new-in-v28)
 **New in V2.5 - dramatically better results:**
@@ -24,15 +32,18 @@
 # Clone the repo
 git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
-# Add your API keys
+# Add your API keys (optional if signed in to Codex)
 mkdir -p ~/.config/last30days
 cat > ~/.config/last30days/.env << 'EOF'
-OPENAI_API_KEY=sk-...
+SCRAPECREATORS_API_KEY=... # Reddit + TikTok + Instagram (one key, all three) — scrapecreators.com
-XAI_API_KEY=xai-...       # optional  - cookie auth is default for X search
+OPENAI_API_KEY=sk-...      # optional — legacy Reddit fallback if using `codex login`
 XAI_API_KEY=xai-...        # optional — cookie auth is default for X search
 EOF
 chmod 600 ~/.config/last30days/.env
 ```
 If you're signed in to Codex (`codex login`), the skill will use your Codex credentials for the OpenAI Responses API and you can omit `OPENAI_API_KEY`. If you're not signed in, run `codex login` first.
 ### X Search Authentication
 X search reads your existing browser cookies  - no API keys or login commands needed.
@@ -122,7 +133,7 @@ Examples:
 ## What It Does
-1. **Researches** - Scans Reddit, X, YouTube, Hacker News, Polymarket, and the web for discussions from the last 30 days
+1. **Researches** - Scans Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web for discussions from the last 30 days
 2. **Synthesizes** - Identifies patterns, best practices, and what actually works
 3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
@@ -884,6 +895,22 @@ This example shows /last30days discovering **emerging developer workflows** - re
 At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH.
 ## Troubleshooting
 ### macOS: SSL Certificate Verify Failed
 If you see `[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate`, your Python installation is missing SSL root certificates. This only affects Python installed from python.org — **Homebrew users are not affected**.
 ```bash
 # Check which Python you have
 which python3
 # Homebrew: /opt/homebrew/bin/python3 or /usr/local/bin/python3
 # Python.org: /Library/Frameworks/Python.framework/...
 # Fix: run the certificate installer (adjust version as needed)
 sudo "/Applications/Python 3.12/Install Certificates.command"
 ```
 ## How It Works
 ### Two-Phase Search Architecture
@@ -914,6 +941,72 @@ If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1)
 ---
 ## What's New in v2.9
 ### ScrapeCreators Reddit as default
 Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search.
 ```bash
 echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
 ```
 ### Smart subreddit discovery
 Subreddit discovery now uses relevance-weighted scoring instead of pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.).
 | Topic | Before (v2.8) | After (v2.9) |
 |-------|---------------|--------------|
 | Claude Code skills | Generic programming subs | r/ClaudeAI, r/ClaudeCode, r/openclaw |
 | Kanye West | r/AskReddit, r/OutOfTheLoop | r/hiphopheads, r/Kanye, r/NFCWestMemeWar |
 | Nano Banana Pro | r/techsupport, r/whatisthisthing | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
 ### Top comments elevated
 Top comments now carry a 10% weight in the engagement scoring formula and are displayed prominently with `💬` and upvote counts:
 ```
 **R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
  Claude Code creator: In the next version, introducing two new skills
  💬 Top comment (245 pts): "This is going to change how everyone works with Claude"
 ```
 **Updated scoring formula:** `0.50 × log1p(score) + 0.35 × log1p(comments) + 0.05 × (ratio×10) + 0.10 × log1p(top_comment_score)` (was 0.55/0.40/0.05).
 ### Beta test results
 | Topic | Time | Threads | Discovered Subreddits |
 |-------|------|---------|----------------------|
 | Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
 | Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
 | Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
 | Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
 | Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
 ---
 ## What's New in v2.8
 ### Instagram Reels as a source
 **See what creators are posting on Instagram.** Search any topic and get trending Reels with views, likes, spoken-word transcripts, and hashtags — scored and ranked alongside all other sources.
 Search "AI tools" and you get:
 - 📸 Instagram: 5 reels │ 1.4M views │ 30K likes │ 3 with transcripts
 - @danmartell: 803K views — "AI tools from 2025 vs 2026"
 - @karimehta05: 112K views — "5 AI Tools I Swear By"
 ### TikTok + Instagram on ScrapeCreators
 Both TikTok and Instagram are powered by [ScrapeCreators](https://scrapecreators.com) — one API key covers both sources. 100 free credits, then pay-as-you-go.
 ```bash
 echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
 ```
 **Migrating from Apify?** Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. The old key is no longer used.
 ---
 ## What's New in V2.5
 ### Polymarket prediction markets and Hacker News
@@ -937,7 +1030,7 @@ No API keys required for either source. Inspired by community PRs from [@ARJ999]
 ### Multi-signal quality-ranked relevance scoring
-**Every result across all six sources runs through a composite scoring pipeline.** V2.5 doesn't just find more content - it ranks it with significantly higher precision.
+**Every result across all seven sources runs through a composite scoring pipeline.** V2.5 doesn't just find more content - it ranks it with significantly higher precision.
 **Text similarity engine** - Bidirectional substring matching with synonym expansion ("hip hop" matches "rap", "MacBook" matches "Mac", "AI video" matches "text to video") and token-level overlap scoring. A rap music mix titled "Lit Hip Hop Mix 2026" went from relevance 0.33 (almost filtered out) to 0.71. Title + transcript matching catches videos that discuss your topic without mentioning it in the title.
@@ -1051,7 +1144,8 @@ Thanks to the contributors who helped shape V2:
 | Destination | Data Sent | API Key Required |
 |------------|-----------|-----------------|
-| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
+| `api.scrapecreators.com` | Search query (Reddit + TikTok + Instagram) | SCRAPECREATORS_API_KEY |
 | `api.openai.com` | Search query (legacy Reddit fallback) | OPENAI_API_KEY |
 | `reddit.com` | Thread URLs for enrichment | None (public JSON) |
 | Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
 | `youtube.com` (via yt-dlp) | Search query | None (public search) |
@@ -1075,6 +1169,6 @@ Each API key is transmitted only to its respective endpoint. Your OpenAI key is
 ---
-*30 days of research. 30 seconds of work. Six sources. Zero stale prompts.*
+*30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.*
-*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. Web.  - All synthesized into expert answers and copy-paste prompts.*
+*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. TikTok. Instagram. Web. — All synthesized into expert answers and copy-paste prompts.*
@@ -1,7 +1,7 @@
 ---
 name: last30days
-version: "2.6"
+version: "2.9"
-description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
+description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
 argument-hint: 'last30 AI video tools, last30 best project management tools'
 allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
 homepage: https://github.com/mvanhorn/last30days-skill
@@ -11,11 +11,11 @@ metadata:
    emoji: "📰"
    requires:
      env:
-        - OPENAI_API_KEY
+        - SCRAPECREATORS_API_KEY
      bins:
        - node
        - python3
-    primaryEnv: OPENAI_API_KEY
+    primaryEnv: SCRAPECREATORS_API_KEY
    files:
      - "scripts/*"
    homepage: https://github.com/mvanhorn/last30days-skill
@@ -24,14 +24,15 @@ metadata:
      - reddit
      - x
      - youtube
      - tiktok
      - hackernews
      - trends
      - prompts
 ---
-# last30days v2.5: Research Any Topic from the Last 30 Days
+# last30days v2.9: Research Any Topic from the Last 30 Days
-Research ANY topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
+Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
 ## CRITICAL: Parse User Intent
@@ -64,7 +65,7 @@ Common patterns:
 **DISPLAY your parsing to the user.** Before running any tools, output:
 ```
-I'll research {TOPIC} across Reddit, X, and the web to find what's been discussed in the last 30 days.
+I'll research {TOPIC} across Reddit, X, TikTok, and the web to find what's been discussed in the last 30 days.
 Parsed intent:
 - TOPIC = {TOPIC}
@@ -126,7 +127,7 @@ Agent mode report format:
 ```
 ## Research Report: {TOPIC}
-Generated: {date} | Sources: Reddit, X, YouTube, HN, Polymarket, Web
+Generated: {date} | Sources: Reddit, X, YouTube, TikTok, HN, Polymarket, Web
 ### Key Findings
 [3-5 bullet points, highest-signal insights with citations]
@@ -146,6 +147,8 @@ Generated: {date} | Sources: Reddit, X, YouTube, HN, Polymarket, Web
 **CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.**
 **IMPORTANT: The script handles API key/Codex auth detection automatically.** Run it and check the output to determine mode.
 ```bash
 # Find skill root — works in repo checkout, Claude Code, or Codex install
 for dir in \
@@ -162,20 +165,24 @@ if [ -z "${SKILL_ROOT:-}" ]; then
  exit 1
 fi
-python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact  # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
+python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web  # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
 ```
 Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically takes 1-3 minutes.
 The script will automatically:
 - Detect available API keys
- Run Reddit/X/YouTube/Hacker News/Polymarket searches
+- Run Reddit/X/YouTube/TikTok/Instagram/Hacker News/Polymarket searches
- Output ALL results including YouTube transcripts, HN comments, and prediction market odds
+- Output ALL results including YouTube transcripts, TikTok captions, Instagram captions, HN comments, and prediction market odds
-**Read the ENTIRE output.** It contains SIX data sections in this order: Reddit items, X items, YouTube items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
+**Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
 **YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
 **TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block.
 **Instagram Reels items in the output look like:** `**{IG_id}** (score:N) @{creator} (date) [N views, N likes]` followed by caption text, URL, and optional transcript. Count them and include them in your synthesis and stats block. Instagram provides unique creator/influencer perspective — weight it alongside TikTok.
 ---
 ## STEP 2: DO WEBSEARCH AFTER SCRIPT COMPLETES
@@ -211,7 +218,9 @@ For ALL query types:
 - **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
 - EXCLUDE reddit.com, x.com, twitter.com (covered by script)
 - INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
+- **DO NOT output a separate "Sources:" block** — instead, include the top 3-5 web
  source names as inline links on the 🌐 Web: stats line (see stats format below).
  The WebSearch tool requires citation; satisfy it there, not as a trailing section.
 **Options** (passed through from user's command):
 - `--days=N` → Look back N days instead of 30 (e.g., `--days=7` for weekly roundup)
@@ -228,10 +237,12 @@ For ALL query types:
 The Judge Agent must:
 1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
 2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
-3. Weight WebSearch sources LOWER (no engagement data)
+3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
-4. Identify patterns that appear across ALL sources (strongest signals)
+4. Weight WebSearch sources LOWER (no engagement data)
-5. Note any contradictions between sources
+5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes (shown as `💬 Top comment (N upvotes)`), quote it directly in your synthesis. Reddit's value is in the comments.
-6. Extract the top 3-5 actionable insights
+6. Identify patterns that appear across ALL sources (strongest signals)
 7. Note any contradictions between sources
 8. Extract the top 3-5 actionable insights
 7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.
@@ -335,20 +346,23 @@ CITATION RULE: Cite sources sparingly to prove research is real.
 CITATION PRIORITY (most to least preferred):
 1. @handles from X — "per @handle" (these prove the tool's unique value)
-2. r/subreddits from Reddit — "per r/subreddit"
+2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, prefer quoting top comments over just the thread title)
 3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
-4. HN discussions — "per HN" or "per hn/username" (developer community signal)
+4. TikTok creators — "per @creator on TikTok" (viral/trending signal)
-5. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
+5. Instagram creators — "per @creator on Instagram" (influencer/creator signal)
-6. Web sources — ONLY when Reddit/X/YouTube/HN/Polymarket don't cover that specific fact
+6. HN discussions — "per HN" or "per hn/username" (developer community signal)
 7. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
 8. Web sources — ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact
 The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
 When both a web article and an X post cover the same fact, cite the X post.
-URL FORMATTING: NEVER paste raw URLs in the output.
+URL FORMATTING: NEVER paste raw URLs anywhere in the output — not in synthesis, not in stats, not in sources.
 - **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
 - **GOOD:** "per Rolling Stone"
- **GOOD:** "per Complex"
+- **BAD stats line:** `🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/...`
-Use the publication name, not the URL. The user doesn't need links — they need clean, readable text.
+- **GOOD stats line:** `🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee`
 Use the publication/site name, not the URL. The user doesn't need links — they need clean, readable text.
 **BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
 **GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per @honest30bgfan_"
@@ -389,13 +403,28 @@ KEY PATTERNS from the research:
 ├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
 ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
 ├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
 ├─ 🎵 TikTok: {N} videos │ {N} views │ {N} likes │ {N} with captions
 ├─ 📸 Instagram: {N} reels │ {N} views │ {N} likes │ {N} with captions
 ├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
 ├─ 📊 Polymarket: {N} markets │ {short summary of up to 5 most relevant market odds, e.g. "Championship: 12%, #1 Seed: 28%, Big 12: 64%, vs Kansas: 71%"}
-├─ 🌐 Web: {N} pages (supplementary)
+├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name
 └─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
 ---
 ```
 **🌐 Web: line — how to extract site names from URLs:**
 Strip the protocol, path, and `www.` — use the recognizable publication name:
 - `https://later.com/blog/instagram-reels-trends/` → **Later**
 - `https://socialbee.com/blog/instagram-trends/` → **SocialBee**
 - `https://buffer.com/resources/instagram-algorithms/` → **Buffer**
 - `https://www.cnn.com/2026/02/22/tech/...` → **CNN**
 - `https://medium.com/the-ai-studio/...` → **Medium**
 - `https://radicaldatascience.wordpress.com/...` → **Radical Data Science**
 List as comma-separated plain names: `Later, SocialBee, Buffer, CNN, Medium`
 **⚠️ WebSearch citation — ALREADY SATISFIED. DO NOT ADD A SOURCES SECTION.**
 The WebSearch tool mandates source citation. That requirement is FULLY satisfied by the source names on the 🌐 Web: line above. Do NOT append a separate "Sources:" section at the end of your response. Do NOT list URLs anywhere. The 🌐 Web: line IS your citation. Nothing more is needed.
 **CRITICAL: Omit any source line that returned 0 results.** Do NOT show "0 threads", "0 stories", "0 markets", or "(no results this cycle)". If a source found nothing, DELETE that line entirely - don't include it at all.
 NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
@@ -558,7 +587,7 @@ After delivering a prompt, end with:
 ```
 ---
 📚 Expert in: {TOPIC} for {TARGET_TOOL}
-📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} HN stories ({sum} points) + {n} web pages
+📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} TikTok videos ({sum} views) + {n} Instagram reels ({sum} views) + {n} HN stories ({sum} points) + {n} web pages
 Want another prompt? Just tell me what you're creating next.
 ```
@@ -568,11 +597,13 @@ Want another prompt? Just tell me what you're creating next.
 ## Security & Permissions
 **What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
+- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit search, subreddit discovery, and comment enrichment (requires SCRAPECREATORS_API_KEY — same key as TikTok + Instagram)
 - Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
 - Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
 - Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
 - Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
 - Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
 - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (same SCRAPECREATORS_API_KEY as Reddit, PAYG after 100 free credits)
 - Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
 - Fetches public Reddit thread data from `reddit.com` for engagement metrics
 - Stores research findings in local SQLite database (watchlist mode only)
@@ -584,6 +615,7 @@ Want another prompt? Just tell me what you're creating next.
 - Does not log, cache, or write API keys to output files
 - Does not send data to any endpoint not listed above
 - Hacker News and Polymarket sources are always available (no API key, no binary dependency)
 - TikTok and Instagram sources require SCRAPECREATORS_API_KEY (same key covers both; 100 free credits, then PAYG)
 - Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output
 **Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
@@ -2,7 +2,7 @@
 ## Overview
-`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet.
+`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet. OpenAI auth can come from `OPENAI_API_KEY` or Codex login credentials.
 The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
@@ -10,7 +10,7 @@ The skill operates in three modes depending on available API keys: **reddit-only
 The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
- **env.py**: Load and validate API keys from `~/.config/last30days/.env`
+- **env.py**: Load API keys from `~/.config/last30days/.env` and Codex auth from `~/.codex/auth.json`
 - **dates.py**: Date range calculation and confidence scoring
 - **cache.py**: 24-hour TTL caching keyed by topic + date range
 - **http.py**: stdlib-only HTTP client with retry logic
@@ -0,0 +1,250 @@
 ---
 title: "feat: close PR #37 - Codex auth finalization and clean close"
 type: feat
 status: active
 date: 2026-03-02
 ---
 # feat: Close PR #37 - Codex Auth Finalization
 ## Overview
 PR #37 (`iliaal:codex-auth-merged`) adds Codex auth so users with `codex login` can use the
 skill without an `OPENAI_API_KEY`. The good news: **all of its core changes are already on main**.
 PR #38 (which we merged earlier today) was branched directly from #37, so the Codex auth code
 rode in with that merge.
 The task is to:
 1. Verify the Codex auth integration is intact and compatible with v2.6 additions
 2. Fix the one known test isolation bug that PR #37 shipped with
 3. Close PR #37 with a clear explanation and a thank-you to the contributor
 ## What PR #37 Added (Now All on Main)
 ### Core Codex auth system (`scripts/lib/env.py`)
 - `CODEX_AUTH_FILE` path constant (`~/.codex/auth.json`)
 - `OpenAIAuth` dataclass: `token`, `source`, `status`, `account_id`, `codex_auth_file`
 - `_decode_jwt_payload()` - JWT base64 decode without verification
 - `_token_expired()` - checks JWT `exp` claim with 60s leeway
 - `extract_chatgpt_account_id()` - extracts `chatgpt_account_id` from JWT `https://api.openai.com/auth` claim
 - `load_codex_auth()` - reads `~/.codex/auth.json`
 - `get_codex_access_token()` - returns `(token, status)` tuple
 - `get_openai_auth()` - priority chain: `OPENAI_API_KEY` env var > `.env` file key > Codex token
 ### Codex endpoint routing (`scripts/lib/openai_reddit.py`)
 - `CODEX_RESPONSES_URL = "https://chatgpt.com/backend-api/codex/responses"`
 - `_parse_sse_chunk()` / `_parse_sse_stream()` / `_parse_codex_stream()` - SSE response parsing
 - Headers injected for Codex path: `chatgpt-account-id`, `OpenAI-Beta: responses=v1`, `originator: pi`
 - Codex payload: `store: false`, `stream: true`
 - `CODEX_FALLBACK_MODELS` retry chain: `gpt-5.1-codex-mini` → `gpt-5.2`
 ### Tests (`tests/test_codex_auth.py`)
 - 22 unit tests covering JWT decode, expiry, account ID extraction, auth resolution,
  SSE parsing, payload building, source availability
 - 21/22 pass; 1 has a test isolation bug (see below)
 ## What PR #37 Contains That Must NOT Go to Public Repo
 These files are in the #37 branch but are internal benchmarking artifacts. They should
 never land on the public `upstream` remote:
 | Path | Why private |
 |------|-------------|
 | `docs/comparison-results/` (55+ files) | Benchmark JSON/MD from synthesis quality testing |
 | `docs/plans/*.md` (9 internal plan docs) | Private planning documents |
 | `docs/v2.1-tweets.md` | Internal launch tweet drafts |
 | `variants/open/references/research.md` | Internal research notes |
 | `scripts/evaluate-synthesis.py` | Internal evaluation script |
 | `scripts/generate-synthesis-inputs.py` | Internal benchmark input generator |
 | `fixtures/polymarket_sample.json` | Used by internal eval scripts |
 ## What PR #37 Has That's OLDER Than Main
 These files in PR #37 are earlier versions than what main has - we keep our versions:
 - `SKILL.md` - PR #37 is v2.1; main is v2.6 (keep v2.6)
 - `README.md` - PR #37's is missing HN/Polymarket; main's is current (keep main)
 - `SPEC.md` - PR #37 has an older spec (keep main)
 - `scripts/lib/hackernews.py` - NOT in PR #37; main has the full HN integration
 - `scripts/lib/polymarket.py` - PR #37 has an older version without quality ranking
 - `scripts/sync.sh` - minor differences; main's version is correct
 ## Known Issue: Test Isolation Bug
 **File:** `tests/test_codex_auth.py`
 **Test:** `TestGetOpenaiAuth::test_api_key_takes_priority`
 ```python
 def test_api_key_takes_priority(self):
    """OPENAI_API_KEY in env file should be preferred over Codex."""
    file_env = {"OPENAI_API_KEY": "sk-test123"}
    auth = env.get_openai_auth(file_env)
    self.assertEqual(auth.token, "sk-test123")  # FAILS if OPENAI_API_KEY set in shell
 ```
 **Root cause:** `get_openai_auth()` checks `os.environ.get("OPENAI_API_KEY")` first (env var
 priority). The test sets `file_env` but does NOT patch `os.environ`, so the real
 `OPENAI_API_KEY` from the developer's shell wins.
 **Fix:**
 ```python
@patch.dict(os.environ, {}, clear=False)
 def test_api_key_takes_priority(self):
 ```
 But we also need to REMOVE `OPENAI_API_KEY` from the patched env:
 ```python
@patch.dict(os.environ, {"OPENAI_API_KEY": ""}, clear=False)
 def test_api_key_takes_priority(self):
 ```
 Actually the cleanest fix:
 ```python
 def test_api_key_takes_priority(self):
    """OPENAI_API_KEY in env file should be preferred over Codex."""
    with patch.dict(os.environ, {}, clear=True):
        # Restore non-OPENAI env vars to avoid side effects
        file_env = {"OPENAI_API_KEY": "sk-test123"}
        auth = env.get_openai_auth(file_env)
        self.assertEqual(auth.token, "sk-test123")
 ```
 Or the minimal fix (remove just OPENAI_API_KEY without nuking entire env):
 ```python
@patch.dict(os.environ, {"OPENAI_API_KEY": "sk-test123"})
 def test_api_key_takes_priority(self):
    """OPENAI_API_KEY in env var should be preferred over Codex."""
    auth = env.get_openai_auth({})
    self.assertEqual(auth.source, "api_key")
    self.assertEqual(auth.token, "sk-test123")
    self.assertIsNone(auth.account_id)
 ```
 This reframes the test as "env var takes priority over empty file_env" which is equally
 valid and sidesteps the isolation problem entirely.
 ## Acceptance Criteria
 - [ ] Verify `tests/test_codex_auth.py` runs 22/22 clean (no isolation failures)
 - [ ] Fix `test_api_key_takes_priority` with the minimal patch approach above
 - [ ] Run full test suite: confirm only the 5 pre-existing stale model failures remain
 - [ ] Verify `env.py` on main has `is_hackernews_available()` and `is_polymarket_available()`
  (they were REMOVED in PR #37 but should be on main since #38 preserved them)
 - [ ] Verify `scripts/sync.sh` deploys to `~/.claude/skills/last30daysCROSS` correctly
  (PR #37's sync.sh may be missing this; main's version should have it)
 - [ ] Close PR #37 with a comment explaining the code landed via #38
 - [ ] Add `docs/comparison-results/` to `.gitignore` in the private repo so benchmark
  files never accidentally get committed to the upstream public repo
 ## Implementation Steps
 ### Step 1: Fix the test
 In the **private source repo** (`/Users/mvanhorn/last30days-skill-private/`):
 Edit `tests/test_codex_auth.py` line 77-84. Replace the bare test with the `@patch.dict` version above. Run `python3 -m pytest tests/test_codex_auth.py -v` to confirm 22/22.
 ### Step 2: Run full test suite
 ```bash
 cd /Users/mvanhorn/last30days-skill-private
 python3 -m pytest tests/ -v 2>&1 | tail -20
 ```
 Expected: only `test_reddit_search_basic`, `test_default_model`, `test_model_pin`, and
 similar model-name tests fail (the 5 pre-existing stale model failures from before PR #37).
 Codex auth tests should all pass.
 ### Step 3: Verify the private docs protection
 Check that `.gitignore` (or the upstream push config) prevents `docs/comparison-results/`
 from leaking to the public GitHub remote.
 ```bash
 cat /Users/mvanhorn/last30days-skill-private/.gitignore | grep -E "comparison|evaluate|generate"
 ```
 If not present, add:
 ```
 docs/comparison-results/
 scripts/evaluate-synthesis.py
 scripts/generate-synthesis-inputs.py
 fixtures/polymarket_sample.json
 docs/v2.1-tweets.md
 variants/open/references/research.md
 ```
 ### Step 4: Commit and sync
 ```bash
 cd /Users/mvanhorn/last30days-skill-private
 git add tests/test_codex_auth.py
 git commit -m "fix(tests): patch OPENAI_API_KEY env isolation in test_api_key_takes_priority"
 bash scripts/sync.sh
 ```
 Then push to upstream (public):
 ```bash
 git push upstream main
 ```
 ### Step 5: Close PR #37
 Post a comment on PR #37 explaining what happened, then close it:
 ```
 Thanks @iliaal! 🙏 This was a great contribution.
 The Codex auth changes landed in main via PR #38, which was branched from your
 `codex-auth-merged` branch. So all the core auth code is already shipping:
 - JWT decoding + expiry checking in env.py ✅
 - Codex endpoint routing + SSE parsing in openai_reddit.py ✅
 - 22 unit tests in test_codex_auth.py ✅
 - CODEX_FALLBACK_MODELS retry chain ✅
 Since then we've also shipped v2.5 (HN + Polymarket sources) and v2.6 (agent-native
 invocation with --agent flag), so SKILL.md and README are already ahead of this branch.
 Closing as the changes are incorporated. Thanks again for the excellent work!
 ```
 ## Technical Considerations
 **Why can't we just merge #37 directly?**
 Three reasons:
 1. SKILL.md/README in #37 are v2.1 - they'd overwrite our v2.6 improvements
 2. The `docs/comparison-results/` directory (55+ benchmark files) would go to the public repo
 3. The test isolation bug would ship a flaky test to everyone who sets `OPENAI_API_KEY`
 **Cherry-pick vs close approach:**
 Since the Codex auth code is already on main, cherry-picking would be redundant. The cleanest
 path is to fix the test bug on main, then close #37 with an explanation.
 **Future private-docs hygiene:**
 The `docs/comparison-results/` files should be gitignored or moved to a separate
 private branch so this situation doesn't repeat. These are internal QA benchmarks -
 they belong in the private repo only, never in `upstream`.
 ## Dependencies & Risks
 **Low risk** - this is test cleanup + PR bookkeeping. The feature itself is already running.
 **Codex auth live test** - We can't easily verify the live Codex auth flow without a `codex login`
 session. If a user reports auth issues, the test suite gives good coverage of the logic;
 live testing would require a Codex-authenticated environment.
 ## Sources & References
 - PR #37: https://github.com/mvanhorn/last30days-skill/pull/37 (iliaal: codex-auth-merged)
 - PR #38 (merged): fixed Bird X auth, brought Codex auth to main as a side effect
 - `tests/test_codex_auth.py` - 22 unit tests for the Codex auth system
 - `scripts/lib/env.py` lines 26-175 - Codex auth core logic
 - `scripts/lib/openai_reddit.py` lines 45-310 - Codex endpoint routing
@@ -0,0 +1,102 @@
 ---
 title: "fix: suppress trailing Sources: block from WebSearch tool mandate"
 type: fix
 status: completed
 date: 2026-03-02
 ---
 # fix: Suppress Trailing Sources: Block from WebSearch Tool Mandate
 ## Problem
 After the skill completes, a `Sources:` block appears below the invitation text:
 ```
 I'm now an expert on the Dor Brothers. Some things I can help with:
 ...
 Sources:
  - Movie starring Logan Paul made exclusively with AI released - Newsweek
  - The Dor Brothers: Pioneers in AI Video Production
  - ...
 ```
 This happens because the `WebSearch` tool has a **system-level mandatory instruction**:
 > "After answering the user's question, you MUST include a 'Sources:' section at the end of your response"
 SKILL.md already says `DO NOT output "Sources:" list` (line 214) but this is too vague - it doesn't address the WebSearch tool mandate explicitly, so the tool's system instruction wins. The model dutifully appends Sources: after all skill output is done.
 ## Root Cause
 Two competing instructions:
 1. **WebSearch system mandate** (higher authority): "MUST include Sources: at end of response"
 2. **SKILL.md line 214** (lower authority): "DO NOT output Sources: list"
 The model follows #1 because it's framed as a critical system requirement.
 The fix: **satisfy the WebSearch citation requirement INSIDE the stats block**, then explicitly tell the model the requirement is already fulfilled and no trailing section is needed.
 ## Proposed Solution
 **Two-part SKILL.md edit only. No Python changes.**
 ### Part 1: Update the Step 2 instruction (line 214)
 **Current:**
 ```
 - **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
 ```
 **Replace with:**
 ```
 - **DO NOT output a separate "Sources:" block** — instead, include the top 3-5 web
  source names as inline links on the 🌐 Web: stats line (see stats format below).
  This satisfies the WebSearch tool's citation requirement inline without a trailing section.
 ```
 ### Part 2: Update the stats block format to include web source links
 **Current stats line:**
 ```
 ├─ 🌐 Web: {N} pages (supplementary)
 ```
 **Replace with:**
 ```
 ├─ 🌐 Web: {N} pages — [Source Name](url), [Source Name](url), [Source Name](url)
 ```
 And immediately after the closing `---` of the stats block, add:
 ```
 **WebSearch citation note:** Source links are included in the 🌐 Web: line above.
 The WebSearch tool citation requirement is satisfied. Do NOT append a separate
 "Sources:" section after the invitation.
 ```
 ## Acceptance Criteria
 - [x] The trailing `Sources:` block no longer appears after the invitation
 - [x] Web source links appear cleanly on the `🌐 Web:` stats line
 - [ ] Manual test: run `/last30days dor brothers` and confirm no trailing Sources: block
 - [x] Synced to all 4 destinations via `sync.sh`
 ## Implementation Steps
 1. Edit `SKILL.md` line 214 (Step 2 section) - replace weak "DO NOT" with redirect instruction
 2. Edit `SKILL.md` stats block format - add `— [Source](url), ...` to the 🌐 Web: line
 3. Add WebSearch citation note after the stats block closing `---`
 4. Run `bash scripts/sync.sh` to deploy
 5. Test with `/last30days [any topic]` and confirm no trailing Sources:
 ## Context
 - **Why not just fight the mandate?** The WebSearch system instruction is authoritative. We can't override it with a soft "don't do this." We need to redirect it.
 - **Why the stats block?** It's the natural place for source metadata and it appears before the invitation, so satisfying the citation there prevents the trailing append.
 - **SKILL.md is the only file that needs to change.** No Python script changes required.
 ## Sources & References
 - SKILL.md line 214: current weak instruction
 - SKILL.md stats block section: where web source links will live
 - Screenshot: user-reported Sources: trailing block (session context)
@@ -0,0 +1,43 @@
 # feat: Skip native web search when running in Claude Code
 **Type:** enhancement
 **Date:** 2026-03-03
 **Detail level:** MINIMAL
 ## Problem
 When `/last30days` runs in Claude Code, web search happens twice:
 1. The Python script uses Parallel AI / Brave / OpenRouter (costs API credits)
 2. SKILL.md tells Claude to run its built-in WebSearch tool (free, better quality)
 This is redundant. Claude's WebSearch is better and free. In OpenClaw, there's no WebSearch tool, so native backends are essential there.
 ## Solution
 Add a `--no-native-web` CLI flag to `last30days.py`. When set, the script skips native web search backends even if API keys are configured, and emits the `### WEBSEARCH REQUIRED ###` signal so the assistant handles it.
 Update SKILL.md invocation to include the flag.
 ## Changes
 ### 1. `scripts/last30days.py`
 - [ ] Add `--no-native-web` argument to argparse (store_true, default False)
 - [ ] When `args.no_native_web` is True, force `web_backend = None` regardless of API keys
 - [ ] This naturally triggers `web_needed = True` → emits `### WEBSEARCH REQUIRED ###` signal
 - [ ] Update diagnostic banner to show "Web: deferred to assistant" when flag is active
 ### 2. `SKILL.md`
 - [ ] Add `--no-native-web` to the invocation command on line 168:
  ```
  python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
  ```
 ### 3. OpenClaw / `--agent` mode
 - [ ] No changes needed — OpenClaw invocations don't read SKILL.md, they call the script directly without `--no-native-web`, so Parallel AI/Brave/OpenRouter still work
 ## Acceptance Criteria
 - [ ] Claude Code sessions: script skips Parallel AI, Claude uses WebSearch (no API credits spent)
 - [ ] OpenClaw sessions: script still uses Parallel AI / Brave / OpenRouter as before
 - [ ] `--no-native-web` flag can be combined with `--include-web` (flag wins, web deferred)
 - [ ] Diagnostic output clearly shows web is deferred to assistant
@@ -0,0 +1,417 @@
 ---
 title: "feat: Add TikTok as 7th source via Apify"
 type: feat
 date: 2026-03-03
 ---
 # feat: Add TikTok Signal via Apify
 ## Overview
 Add TikTok as the 7th research source alongside Reddit, X, YouTube, HN, Polymarket, and Web. Use the **Apify** platform (`clockworks/tiktok-scraper` actor) to search TikTok by keyword, extract engagement metrics (views, likes, comments), and optionally pull video captions for synthesis enrichment — mirroring the YouTube pattern.
 **Why this matters:** TikTok is where trends break first for many topics (products, music, culture, tech tips, news reactions). A viral TikTok with 2M views is a stronger signal than a tweet with 500 likes. The skill currently misses this entirely.
 **Why Apify:** BYO API key, $5/month free credits (no CC required), pay-per-result pricing, Python SDK (`apify-client`), and the same actor platform supports Facebook and Instagram scrapers — so this investment pays forward.
 ## Proposed Solution
 ### Architecture: Shared Apify Client + Per-Source Modules
 ```
 scripts/lib/
  apify_client_wrapper.py   ← NEW: shared Apify client init + helpers (reused by FB/IG later)
  tiktok.py                 ← NEW: TikTok search, captions, relevance
  # future:
  # facebook.py             ← uses same apify_client_wrapper.py
  # instagram.py            ← uses same apify_client_wrapper.py
 ```
 This design means adding Facebook or Instagram later is just a new `facebook.py` module — the Apify client setup, token validation, and error handling are already done.
 ### Data Flow
 ```
 User topic + date range
     ↓
 [apify_client_wrapper.py] init client with APIFY_API_TOKEN
     ↓
 [tiktok.py] search_tiktok()
  ├─ Call clockworks/tiktok-scraper actor (sync API, ≤5min)
  ├─ Input: searchQueries=[core_topic], resultsPerPage=N (depth-aware)
  ├─ Parse: id, text, playCount, diggCount, commentCount, createTimeISO, authorMeta, webVideoUrl, hashtags
  ├─ Sort by playCount (views) descending
  ├─ Compute relevance via token-overlap (reuse youtube_yt._compute_relevance pattern)
  └─ Return items
     ↓
 [tiktok.py] fetch_captions() (optional enrichment for top N)
  ├─ Re-call actor with shouldDownloadSubtitles=true for top videos
  ├─ OR use video text/description as lightweight "caption" alternative
  └─ Truncate to 500 words, attach as caption_snippet
     ↓
 [normalize.py] normalize_tiktok_items() → List[TikTokItem]
     ↓
 [score.py] score_tiktok_items()
  ├─ compute_tiktok_engagement_raw(): 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
  ├─ Weighted: 0.45*relevance + 0.25*recency + 0.30*engagement
  └─ Same formula as YouTube (views-dominant)
     ↓
 [dedupe.py] dedupe_tiktok() + cross_source_link()
     ↓
 [render.py] render TikTok section
     ↓
 [SKILL.md] stats line: 🎵 TikTok: N videos │ N views │ N with captions
 ```
 ## Technical Approach
 ### Phase 1: Apify Client Wrapper (`scripts/lib/apify_client_wrapper.py`)
 Shared module for all Apify-backed sources. Keeps TikTok, Facebook, Instagram from duplicating client setup.
 ```python
 """Shared Apify client utilities for last30days sources."""
 from apify_client import ApifyClient
 from typing import Optional, Dict, Any, List
 def get_apify_client(token: str) -> ApifyClient:
    """Initialize Apify client with token."""
    return ApifyClient(token=token)
 def run_actor_sync(
    client: ApifyClient,
    actor_id: str,
    run_input: Dict[str, Any],
    timeout_secs: int = 300,
    max_items: int = None,
 ) -> List[Dict[str, Any]]:
    """Run an Apify actor synchronously and return dataset items.
    Args:
        client: Initialized ApifyClient
        actor_id: e.g. "clockworks/tiktok-scraper"
        run_input: Actor-specific input dict
        timeout_secs: Max wait time (default 5 min)
        max_items: Cap on returned items (cost control)
    Returns:
        List of result dicts from the actor's default dataset
    """
    run = client.actor(actor_id).call(
        run_input=run_input,
        timeout_secs=timeout_secs,
    )
    items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
    if max_items:
        items = items[:max_items]
    return items
 ```
 **Key design decisions:**
 - Single `APIFY_API_TOKEN` env var for all Apify sources (TikTok, future FB, IG)
 - `run_actor_sync()` wraps the call+wait+fetch pattern used by every Apify actor
 - `max_items` param provides cost control (important with $5 free credits)
 ### Phase 2: TikTok Search Module (`scripts/lib/tiktok.py`)
 ```python
 """TikTok search via Apify clockworks/tiktok-scraper."""
 ACTOR_ID = "clockworks/tiktok-scraper"
 DEPTH_CONFIG = {
    "quick":   {"results_per_page": 10, "max_captions": 3},
    "default": {"results_per_page": 20, "max_captions": 5},
    "deep":    {"results_per_page": 40, "max_captions": 8},
 }
 def search_tiktok(topic, from_date, to_date, depth="default", token=None):
    """Search TikTok via Apify.
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    # 1. Init client via apify_client_wrapper
    # 2. Build input: searchQueries=[_extract_core_subject(topic)], resultsPerPage=N
    # 3. Call run_actor_sync(client, ACTOR_ID, input, timeout=120)
    # 4. Parse items: extract id, text, playCount, diggCount, commentCount,
    #    shareCount, createTimeISO, authorMeta.name, webVideoUrl, hashtags
    # 5. Filter by date range (from_date to to_date)
    # 6. Sort by playCount descending
    # 7. Compute relevance via _compute_relevance(topic, item_text)
    # 8. Return structured items
 def fetch_captions(video_items, token, depth="default"):
    """Fetch captions/subtitles for top N TikTok videos.
    Strategy: Re-run actor with shouldDownloadSubtitles=true for
    specific video URLs, OR fall back to video text/description
    as a lightweight alternative.
    Returns:
        Dict mapping video_id → caption_text (truncated to 500 words)
    """
 def search_and_enrich(topic, from_date, to_date, depth="default", token=None):
    """Search + caption enrichment orchestrator (mirrors youtube_yt.search_and_transcribe)."""
 def parse_tiktok_response(response):
    """Extract items list from search_and_enrich response."""
 ```
 **Apify actor input for keyword search:**
 ```json
 {
    "searchQueries": ["claude code tips"],
    "resultsPerPage": 20,
    "shouldDownloadSubtitles": false,
    "shouldDownloadVideos": false,
    "shouldDownloadCovers": false
 }
 ```
 **Apify actor output fields we use:**
 | Apify Field | Our Field | Notes |
 |---|---|---|
 | `id` | `id` | TikTok video ID |
 | `text` | `caption` | Video caption/description |
 | `playCount` | `engagement.views` | Primary engagement signal |
 | `diggCount` | `engagement.likes` | Secondary signal |
 | `commentCount` | `engagement.num_comments` | Tertiary signal |
 | `shareCount` | (stored but not scored) | Available for future use |
 | `createTimeISO` | `date` | Parse to YYYY-MM-DD |
 | `authorMeta.name` | `author_name` | Creator handle |
 | `authorMeta.fans` | (stored but not scored) | Follower count |
 | `webVideoUrl` | `url` | Direct TikTok link |
 | `hashtags[].name` | `hashtags` | For relevance boosting |
 | `videoMeta.duration` | `duration` | For filtering very short clips |
 **Relevance scoring:** Reuse the token-overlap algorithm from `youtube_yt._compute_relevance()`. Additionally boost relevance when topic tokens appear in hashtags (TikTok-specific signal).
 **Caption enrichment strategy:**
 1. **Primary:** Use the `text` field (video description/caption) — always available, free
 2. **Enhanced:** For top N videos, re-run actor with `shouldDownloadSubtitles: true` to get spoken-word captions
 3. **Fallback:** If subtitles unavailable, use `text` field alone (most TikTok videos have descriptive captions)
 This is cheaper than YouTube transcripts (no second yt-dlp call needed for the basic case).
 ### Phase 3: Schema + Normalization
 **`scripts/lib/schema.py` — add TikTokItem dataclass:**
 ```python
@dataclass
 class TikTokItem:
    """Normalized TikTok item."""
    id: str                    # video_id
    text: str                  # caption/description
    url: str                   # webVideoUrl
    author_name: str           # authorMeta.name
    date: Optional[str] = None
    date_confidence: str = "high"  # Apify provides exact timestamps
    engagement: Optional[Engagement] = None  # views, likes, num_comments
    caption_snippet: str = ""  # spoken-word caption (if available), else text
    hashtags: List[str] = field(default_factory=list)
    relevance: float = 0.7
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    cross_refs: List[str] = field(default_factory=list)
 ```
 **`scripts/lib/schema.py` — add to Engagement dataclass:**
 - `shares: Optional[int] = None` — TikTok shares (also useful for future Facebook)
 **`scripts/lib/schema.py` — add to Report dataclass:**
 - `tiktok: List[TikTokItem] = field(default_factory=list)`
 - `tiktok_error: Optional[str] = None`
 **`scripts/lib/normalize.py` — add `normalize_tiktok_items()`:**
 - Parse `createTimeISO` → YYYY-MM-DD
 - Create Engagement(views=playCount, likes=diggCount, num_comments=commentCount)
 - Create TikTokItem objects
 - Hard date filter (like Reddit/X, not soft like YouTube)
 ### Phase 4: Scoring
 **`scripts/lib/score.py` — add TikTok scoring:**
 ```python
 def compute_tiktok_engagement_raw(engagement):
    """TikTok engagement: views-dominant like YouTube.
    0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
    """
    views = getattr(engagement, 'views', 0) or 0
    likes = getattr(engagement, 'likes', 0) or 0
    comments = getattr(engagement, 'num_comments', 0) or 0
    return 0.50 * log1p(views) + 0.30 * log1p(likes) + 0.20 * log1p(comments)
 def score_tiktok_items(items):
    """Score TikTok items. Same weights as YouTube:
    0.45*relevance + 0.25*recency + 0.30*engagement"""
 ```
 ### Phase 5: Deduplication + Cross-Source Linking
 **`scripts/lib/dedupe.py`:**
 ```python
 def dedupe_tiktok(items, threshold=0.7):
    """Dedupe TikTok items via Jaccard similarity on text + author_name."""
    return dedupe_items(items, threshold)
 ```
 - Text extraction for similarity: `text + author_name` (mirrors YouTube's `title + channel_name`)
 - Add `tiktok` to `cross_source_link()` — compare TikTok items with all other sources
 - Cross-ref prefix: `"TK"` (e.g., `TK3` for TikTok item 3)
 ### Phase 6: Rendering
 **`scripts/lib/render.py` — add TikTok section:**
 ```markdown
 ### TikTok Videos
 **TK1** (score:87) @creator_name (2026-02-28) [2.1M views, 45K likes]
  Caption: "This Claude Code trick saved me hours... #claudecode #ai"
  https://www.tiktok.com/@creator/video/1234567890
  Spoken: "So I found this insane trick with Claude Code where you can..."
  *TikTok: This Claude Code trick saved me hours*
 ```
 **Stats line for SKILL.md:**
 ```
 ├─ 🎵 TikTok: {N} videos │ {N} views │ {N} with captions
 ```
 ### Phase 7: Environment + Config
 **`scripts/lib/env.py` — add Apify support:**
 ```python
 def is_apify_available(config: Dict[str, Any]) -> bool:
    """Check if Apify token is configured for TikTok/social scraping."""
    return bool(config.get('APIFY_API_TOKEN'))
 ```
 - New env var: `APIFY_API_TOKEN`
 - Add to `get_config()` key list
 - Add to `get_available_sources()` / `get_missing_keys()` logic
 - Single token covers TikTok + future Facebook + Instagram
 **User setup:**
 ```bash
 # Add to ~/.config/last30days/.env
 APIFY_API_TOKEN=apify_api_xxxxxxxxxxxxx
 ```
 Or get free token: Sign up at https://console.apify.com → Settings → Integrations → Personal API Token.
 ### Phase 8: Orchestrator Integration
 **`scripts/last30days.py` changes:**
 1. Add `"tiktok"` to `VALID_SEARCH_SOURCES` set (line 47)
 2. Add `tiktok_future` var + timeout to `TIMEOUT_PROFILES`:
   ```python
   "tiktok_future": 120  # Apify actors can be slow on first run
   ```
 3. Add `do_tiktok` bool + `run_tiktok` parameter to `run_research()`
 4. Submit `_search_tiktok()` to ThreadPoolExecutor (now max 7+1 workers)
 5. Collect TikTok results with timeout
 6. Add tiktok to return tuple + progress display
 7. Wire tiktok into normalize → score → dedupe → cross-link → render pipeline in main
 ### Phase 9: SKILL.md Updates
 1. Add TikTok to stats box template
 2. Add TikTok citation rule: `@creator on TikTok`
 3. Add TikTok to source weight guidance (rank between YouTube and HN)
 4. Document `APIFY_API_TOKEN` in setup section
 ### Phase 10: Dependency
 ```bash
 pip install apify-client
 ```
 - `apify-client` is the only new dependency
 - Requires Python 3.10+ (already required by the project)
 - No new binary dependencies (unlike yt-dlp for YouTube)
 ## Files to Create / Modify
 ### New Files
 | File | Purpose |
 |---|---|
 | `scripts/lib/apify_client_wrapper.py` | Shared Apify client init + `run_actor_sync()` helper |
 | `scripts/lib/tiktok.py` | TikTok search, caption extraction, relevance scoring |
 | `tests/test_tiktok.py` | Unit tests for TikTok module |
 | `fixtures/tiktok_search.json` | Mock Apify response for testing |
 ### Modified Files
 | File | Changes |
 |---|---|
 | `scripts/lib/schema.py` | Add `TikTokItem` dataclass, `shares` to Engagement, `tiktok`/`tiktok_error` to Report |
 | `scripts/lib/normalize.py` | Add `normalize_tiktok_items()` |
 | `scripts/lib/score.py` | Add `compute_tiktok_engagement_raw()`, `score_tiktok_items()` |
 | `scripts/lib/dedupe.py` | Add `dedupe_tiktok()`, add tiktok to `cross_source_link()` |
 | `scripts/lib/render.py` | Add TikTok rendering section, stats line |
 | `scripts/lib/env.py` | Add `APIFY_API_TOKEN` handling, `is_apify_available()` |
 | `scripts/last30days.py` | Add tiktok to orchestrator pipeline, `VALID_SEARCH_SOURCES`, `TIMEOUT_PROFILES` |
 | `SKILL.md` | Add TikTok stats line, citation rules, source weights |
 | `README.md` | Add TikTok to source list, Apify setup instructions |
 ## Future: Facebook + Instagram via Apify
 The `apify_client_wrapper.py` module is designed to be reused. Adding Facebook would look like:
 ```python
 # scripts/lib/facebook.py
 from . import apify_client_wrapper
 ACTOR_ID = "apify/facebook-posts-scraper"  # or "scraper_one/facebook-posts-search"
 def search_facebook(topic, from_date, to_date, depth="default", token=None):
    client = apify_client_wrapper.get_apify_client(token)
    run_input = {
        "searchType": "posts",
        "searchTerms": [topic],
        "maxPosts": DEPTH_CONFIG[depth]["max_posts"],
    }
    items = apify_client_wrapper.run_actor_sync(client, ACTOR_ID, run_input)
    # Parse: text, likes, comments, shares, time, user.name, url
    ...
 ```
 **Facebook fields available:** `text`, `likes`, `comments`, `shares`, `time`/`timestamp`, `user.name`, `url`, `reactions_count`
 **Instagram** would follow the same pattern with `apify/instagram-scraper` or similar.
 Same `APIFY_API_TOKEN` — no additional keys needed.
 ## Cost Analysis
 **Per research run (default depth, 20 results):**
 - Clockworks TikTok scraper: ~$0.10 per 20 results ($5/1000)
 - Free tier: ~50 research runs per month on $5 free credits
 - With captions (re-run for top 5): ~$0.15 total per run → ~33 runs/month free
 **Comparison:** YouTube costs $0 (yt-dlp is free). TikTok costs ~$0.10-0.15/run. This is acceptable given the signal value and tracks with the BYO key model.
 ## Acceptance Criteria
 - [ ] `APIFY_API_TOKEN` in `.env` enables TikTok source automatically
 - [ ] TikTok appears in parallel search alongside other 6 sources
 - [ ] Results include: video URL, caption, author, views, likes, comments, date
 - [ ] Caption enrichment works for top N videos (configurable by depth)
 - [ ] Relevance scoring filters off-topic viral videos
 - [ ] Cross-source linking detects when TikTok + Reddit/YouTube discuss same topic
 - [ ] Stats box shows: `🎵 TikTok: N videos │ N views │ N with captions`
 - [ ] `--search=tiktok` flag works for TikTok-only research
 - [ ] Graceful degradation: if no APIFY_API_TOKEN, TikTok silently skipped
 - [ ] Mock mode works with `fixtures/tiktok_search.json`
 - [ ] Tests pass for search, normalize, score, dedupe, render
 - [ ] `apify_client_wrapper.py` is generic enough for Facebook/Instagram reuse
@@ -0,0 +1,111 @@
 ---
 title: Fix SKILL.md Argument Flag Forwarding
 type: fix
 status: completed
 date: 2026-03-03
 ---
 # Fix SKILL.md Argument Flag Forwarding
 ## Overview
 `"$ARGUMENTS"` in SKILL.md wraps the entire user input in double quotes, making argparse treat flags like `--store` as part of the topic string instead of CLI flags.
 ## Problem
 SKILL.md line 168:
 ```bash
 python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
 ```
 `$ARGUMENTS` is a Claude Code template variable replaced via string substitution before bash runs. The double quotes cause word-joining:
 - User types: `/last30days AI video tools --store`
 - Claude Code expands to: `python3 script.py "AI video tools --store" --emit=compact`
 - argparse sees: `topic="AI video tools --store"`, `--store` never parsed
 ## Proposed Solution
 Two coordinated changes:
 ### 1. Remove quotes around `$ARGUMENTS` in SKILL.md
 **File:** `SKILL.md:168`
 ```bash
 # Before:
 python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
 # After:
 python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --no-native-web
 ```
 Now bash word-splits the expansion: `python3 script.py AI video tools --store --emit=compact`
 ### 2. Change argparse `topic` from `nargs="?"` to `nargs="*"`
 **File:** `scripts/last30days.py:1040`
 ```python
 # Before:
 parser.add_argument("topic", nargs="?", help="Topic to research")
 # args.topic = "AI video tools" (single string) or None
 # After:
 parser.add_argument("topic", nargs="*", help="Topic to research")
 # args.topic = ["AI", "video", "tools"] (list) or []
 ```
 Then immediately after `parser.parse_args()` (line 1124), join the list back to a string:
 ```python
 args = parser.parse_args()
 args.topic = " ".join(args.topic) if args.topic else None
 ```
 **Why this works for both invocation styles:**
 | Invocation | argparse receives | topic result |
 |---|---|---|
 | `script.py AI video tools --store` (Claude Code) | `["AI", "video", "tools"]` + `--store` | `"AI video tools"` |
 | `script.py "AI video tools" --store` (direct CLI) | `["AI video tools"]` + `--store` | `"AI video tools"` |
 | `script.py --store` (no topic) | `[]` + `--store` | `None` |
 ### 3. Document missing flags in SKILL.md Options section
 **File:** `SKILL.md:223-227`
 Add after the existing `--deep` line:
 ```
 - `--store` -> Persist findings to SQLite database for later querying
 - `--search=SOURCES` -> Comma-separated source filter (e.g., `--search=reddit,hn`)
 - `--include-web` -> Include general web search alongside primary sources
 - `--diagnose` -> Show source availability diagnostics and exit
 - `--timeout=SECS` -> Global timeout in seconds (default: 180, quick: 90, deep: 300)
 ```
 Note: `--sort-x` was listed in the issue but does not exist in the Python argparse. Skip it.
 ## Acceptance Criteria
 - [x] `/last30days AI video tools --store` correctly passes `--store` to Python script
 - [x] `/last30days AI video tools` still works (multi-word topic without flags)
 - [x] Direct CLI: `python3 last30days.py "AI video tools" --store` still works
 - [x] `--diagnose`, `--search=reddit,hn`, `--timeout=120` all forward correctly
 - [x] All 5 missing flags documented in SKILL.md Options section
 - [x] Existing tests pass (`python3 -m pytest tests/`)
 ## Files Changed
 | File | Change |
 |---|---|
 | `SKILL.md:168` | Remove quotes around `$ARGUMENTS` |
 | `scripts/last30days.py:1040` | `nargs="?"` -> `nargs="*"` |
 | `scripts/last30days.py:1124` | Add `args.topic = " ".join(args.topic) if args.topic else None` |
 | `SKILL.md:223-227` | Add 5 missing flags to Options section |
 ## Sources
 - GitHub issue: https://github.com/mvanhorn/last30days-skill/issues/36
 - Reporter: @nicolefinateri
@@ -0,0 +1,479 @@
 # Triage All Open GitHub Issues — Proposed Actions
 **Date:** 2026-03-03
 **Repo:** `mvanhorn/last30days-skill`
 **Open issues:** 16 (as of this triage)
 **Codebase version:** v2.1.0 (Bird vendored, 7 sources: Reddit, X, YouTube, HN, Polymarket, TikTok, Web)
 ---
 ## Legend
 | Action | Meaning |
 |--------|---------|
 | **CLOSE (spam)** | Spam / solicitation — close without comment |
 | **CLOSE (resolved)** | Already fixed in current version — comment & close |
 | **CLOSE (duplicate)** | Duplicate of another issue — link & close |
 | **COMMENT** | Respond with info, no code change needed |
 | **FIX** | Code or docs change needed — described below |
 | **REJECT** | Valid issue but won't address — explain why |
 ---
 ## Issue-by-Issue Triage
 ### #43 — Security Assessment Offer - Claude Code Skill with Social Media APIs
 **Author:** Neo-Assistent | **Created:** 2026-02-26
 > Unsolicited marketing from "SkillSec" offering a free security audit.
 **Proposed action:** **CLOSE (spam)**
 No comment needed. This is a cold sales pitch, not a bug or feature request. Close silently or with a brief "closing — not a bug report or feature request."
 ---
 ### #42 — License & Secondary Development Permission
 **Author:** AllenX95 | **Created:** 2026-02-26
 > Asks what license the project uses and whether modification for open-source models (Kimi K2.5) is allowed.
 **Proposed action:** **COMMENT + CLOSE (resolved)**
 Once #35/#34 are addressed (LICENSE file added), comment:
 > The project is MIT licensed (see LICENSE file in repo root, added in v2.1.x). You're free to modify, fork, and adapt it for any purpose including integration with other models. The MIT license places no restrictions on secondary development.
 **Depends on:** Adding the LICENSE file (see #35/#34 below).
 ---
 ### #41 — Can I use openrouter API to replace with openai API to do the reddit search?
 **Author:** iklynow-hue | **Created:** 2026-02-25
 > Asks about using OpenRouter instead of OpenAI for Reddit search. Has one off-topic spam comment from @gbessoni.
 **Proposed action:** **COMMENT + CLOSE**
 Comment:
 > OpenRouter is already partially supported — `scripts/lib/openrouter_search.py` provides a Perplexity-via-OpenRouter integration for web search. However, the Reddit search specifically uses OpenAI's Responses API (`web_search_preview` tool), which is an OpenAI-specific feature that OpenRouter doesn't proxy.
 >
 > If you want to avoid OpenAI entirely, you can still get results from X (via vendored Bird search or xAI), YouTube (yt-dlp), Hacker News (free API), and Polymarket (free API) — no OpenAI key needed for those. Reddit is the one source that requires `OPENAI_API_KEY`.
 >
 > For budget control, consider using `--quick` mode which reduces API calls.
 Also: **minimize/hide** the off-topic spam comment from @gbessoni ("I'm testing out a new idea for reddit called Redd").
 ---
 ### #40 — watchlist: search_queries field stored but never used by _run_topic()
 **Author:** tejasgadhia | **Created:** 2026-02-23
 > `add --queries "q1,q2,q3"` stores queries in DB but `_run_topic()` always passes `topic["name"]` as the search query. Long descriptive names produce poor results on X and web search.
 **Proposed action:** **FIX (accept)**
 This is a legitimate, well-documented bug. The fix is straightforward:
 **File:** `scripts/watchlist.py` — `_run_topic()` (~line 136)
 ```python
 # Current (broken):
 cmd = [sys.executable, str(SCRIPT_DIR / "last30days.py"), topic["name"], "--emit=json"]
 # Fix: prefer search_queries when available
 import json as _json
 queries = _json.loads(topic["search_queries"]) if topic.get("search_queries") else [topic["name"]]
 search_term = queries[0] if queries else topic["name"]
 cmd = [sys.executable, str(SCRIPT_DIR / "last30days.py"), search_term, "--emit=json"]
 ```
 **Comment to post:**
 > Good catch — you're right that `search_queries` is stored but never read. Will fix `_run_topic()` to prefer `search_queries[0]` over `topic["name"]` when available.
 ---
 ### #39 — watchlist.py: YouTube findings never stored + run-one silent output
 **Author:** tejasgadhia | **Created:** 2026-02-23
 > **Bug 1:** YouTube findings extracted by the research script but dropped in `_run_topic()` because there's no YouTube loop in the findings extraction.
 > **Bug 2:** `cmd_run_one()` doesn't print the result (cosmetic).
 **Proposed action:** **FIX (accept both)**
 Both are real bugs with clear fixes provided by the reporter.
 **Bug 1 fix** — `scripts/watchlist.py` findings extraction (~lines 170-193): add YouTube + TikTok loops:
 ```python
 for item in data.get("youtube", []):
    findings.append({
        "source": "youtube",
        "url": item.get("url", ""),
        "title": item.get("title", ""),
        "author": item.get("channel_name", item.get("channel", "")),
        "content": item.get("transcript_snippet", "") or item.get("title", ""),
        "engagement_score": (item.get("engagement") or {}).get("views", 0),
        "relevance_score": item.get("relevance", 0),
    })
 for item in data.get("tiktok", []):
    findings.append({
        "source": "tiktok",
        "url": item.get("url", ""),
        "title": item.get("caption_snippet", "")[:120],
        "author": item.get("author", ""),
        "content": item.get("caption_snippet", ""),
        "engagement_score": (item.get("engagement") or {}).get("views", 0),
        "relevance_score": item.get("relevance", 0),
    })
 ```
 **Bug 2 fix** — `scripts/watchlist.py` `cmd_run_one()`:
 ```python
 result = _run_topic(topic)
 print(json.dumps(result, default=str))
 ```
 **Comment to post:**
 > Both confirmed. Fixing YouTube (and adding TikTok while we're at it) findings extraction, plus `run-one` output. Thanks for the detailed report.
 ---
 ### #36 — SKILL.md doesn't forward --store, --include-web, --sort-x, --diagnose, --timeout to script
 **Author:** nicolefinateri | **Created:** 2026-02-21
 > `"$ARGUMENTS"` is passed as a single quoted string, so argparse treats the whole thing as the topic positional. Flags like `--store` get swallowed. Also, 5 flags are undocumented in SKILL.md.
 **Proposed action:** **COMMENT + CLOSE (resolved / won't fix)**
 This needs investigation. The `"$ARGUMENTS"` expansion in SKILL.md (line 168) relies on Claude Code's variable expansion behavior. If Claude Code expands `$ARGUMENTS` before the shell sees it, the quoting around it means all words become one argument. However, in practice, Claude Code injects the literal text into the bash script, so `"$ARGUMENTS"` expands to `"best AI tools --store"` which the shell passes as a single string to Python — and Python's argparse sees it as one positional arg.
 **However:** Looking at the actual SKILL.md line 168:
 ```bash
 python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
 ```
 This is a known Claude Code behavior — `$ARGUMENTS` is expanded by the template engine, not the shell. The expansion produces the raw text, so the double quotes cause everything to be one argument.
 **The real fix:** Remove the quotes around `$ARGUMENTS`:
 ```bash
 python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact
 ```
 This lets word-splitting happen so argparse sees separate positional + flag args.
 **Risk:** Topic names with spaces would break (e.g., `AI video tools` would become 3 separate args). Need to verify how Claude Code handles `$ARGUMENTS` expansion. If it's truly a template variable replaced before bash execution, then the fix is more nuanced — may need to parse flags out in bash first.
 **Comment to post:**
 > Thanks for the detailed report. The `$ARGUMENTS` quoting is tricky — it's a Claude Code template variable, not a shell variable. I'll investigate the exact expansion behavior and fix the forwarding. Re: undocumented flags — will add them to the Options section.
 **Priority:** Medium — flags like `--store` and `--diagnose` are useful but not critical path.
 ---
 ### #35 — Add LICENSE file to repository
 **Author:** HackJob7418 | **Created:** 2026-02-21
 > Points out that `plugin.json` declares MIT but no LICENSE file exists in repo root.
 **Proposed action:** **FIX (accept)**
 Trivial fix. Add `LICENSE` file with standard MIT text to repo root.
 **Comment to post:**
 > You're right — added MIT LICENSE file to repo root. Thanks for catching the mismatch.
 ---
 ### #34 — License missing
 **Author:** jdsika (Carlo van Driesten) | **Created:** 2026-02-21
 > Same request as #35 — add a LICENSE file.
 **Proposed action:** **CLOSE (duplicate of #35)**
 **Comment to post:**
 > Duplicate of #35 — adding MIT LICENSE file. Thanks for the nudge!
 ---
 ### #32 — Cannot install Claude Code plugin marketplace
 **Author:** rayshan (Ray Shan) | **Created:** 2026-02-19
 > `marketplace.json` schema validation fails: `plugins.0.source: Invalid input`
 **Proposed action:** **FIX (investigate + fix)**
 The current `marketplace.json` has:
 ```json
 "plugins": [{"name": "last30days", "source": "."}]
 ```
 The Anthropic marketplace schema may require `source` to be a URL or a path in a specific format, not bare `.`. Need to check the expected schema.
 **Possible fix:** Update `source` to match whatever the skills.sh/marketplace validator expects. Likely needs to be a relative path to the SKILL.md or plugin directory:
 ```json
 "plugins": [{"name": "last30days", "source": "./SKILL.md"}]
 ```
 Or the schema may have changed since the file was written. Check skills.sh docs.
 **Comment to post:**
 > Thanks for the report + screenshot. The marketplace.json schema likely changed — I'll update the `source` field format. Can you share what version of the `skills` CLI you're using? (`npx skills --version`)
 **Priority:** High — blocks installation for marketplace users.
 ---
 ### #31 — mvanhorn/last30days is being trashed on skills.sh
 **Author:** PiotrAleksander (Piotr Mrzygłosz) | **Created:** 2026-02-19
 > skills.sh security audit gives the skill bad scores. A mirror at `sickn33/antigravity-awesome-skills` has better scores.
 **Proposed action:** **COMMENT + investigate**
 This is a reputation/trust issue. The skills.sh audit likely flags things like:
 - No LICENSE file (fixed by #35)
 - Shell execution in SKILL.md (inherent to how the skill works)
 - External API calls (Reddit, X, YouTube — core functionality)
 - No pinned dependencies
 **Comment to post:**
 > Thanks for flagging this. A few things:
 >
 > 1. The missing LICENSE file (#35) is being added — that should help the audit score.
 > 2. The skill necessarily makes external API calls (that's its core purpose — researching across platforms). Any "security warning" about API calls is expected behavior, not a vulnerability.
 > 3. The mirror you found (`sickn33/antigravity-awesome-skills`) may be an older fork (v1) that doesn't include the more recent integrations. Simpler code = fewer audit flags, but also fewer features.
 > 4. I'll review the specific audit findings on skills.sh and address any legitimate concerns.
 >
 > If you can share the specific warnings you're seeing, I can address them directly.
 **Priority:** Medium — reputation matters but the core issue is likely the missing LICENSE + inherent design of making API calls.
 ---
 ### #30 — Bird cookie auth: source availability mapping misses reddit-web + Bird combination
 **Author:** volarian-vai | **Created:** 2026-02-19
 > When using Bird cookie auth without `XAI_API_KEY` but with a web search key (e.g., `BRAVE_API_KEY`), the source override logic misses the `reddit-web` → `all` mapping.
 **Proposed action:** **FIX (accept)**
 Legitimate edge case in source availability logic. The fix is a small addition to the Bird override block in `scripts/last30days.py` (~line 917):
 ```python
 if x_source == 'bird':
    if available == 'reddit':
        available = 'both'
    elif available == 'reddit-web':
        available = 'all'
    elif available == 'web':
        available = 'x-web'
 ```
 **Comment to post:**
 > Good catch on the missing `reddit-web` case. The fix is straightforward — adding the mapping. Also fixing `web` → `x-web` as you noted.
 **Priority:** Low-medium — affects a specific env var combination (Bird auth + no XAI key + Brave key).
 ---
 ### #29 — YouTube stats show 'yt-dlp not installed' when search returns 0 results
 **Author:** alexkrivov | **Created:** 2026-02-19
 > When yt-dlp runs successfully but returns 0 results, the stats footer says "yt-dlp not installed" because `report.youtube` is `[]` (falsy) and the fallback message is wrong.
 **Proposed action:** **FIX (accept)**
 Clear bug with a clear root cause. Two-part fix:
 **1.** In `scripts/last30days.py` (~line 1085), set a proper skip reason when YouTube returns 0:
 ```python
 if has_ytdlp and not yt_results:
    source_info["youtube_skip_reason"] = "0 results (query may be too specific)"
 ```
 **2.** In `scripts/lib/render.py` (~line 288), change the default fallback:
 ```python
 reason = source_info.get("youtube_skip_reason", "not available")
 ```
 **Comment to post:**
 > Confirmed — the `[]`-is-falsy bug plus a misleading default string. Will fix both the skip reason and the fallback. Thanks for the detailed trace.
 **Priority:** Low — cosmetic/UX but misleading for users trying to debug setup.
 ---
 ### #22 — Feature request: Surface more Bird CLI capabilities (engagement sorting, time windows, thread following)
 **Author:** nicolefinateri | **Created:** 2026-02-09
 > Requests: (1) `--sort-x likes` flag, (2) `--since 3h` granular time windows, (3) thread following via `bird thread <id>`.
 **Proposed action:** **COMMENT (partial accept, defer)**
 Good feature requests but some are already partially addressed:
 1. **Engagement sorting** — The `--sort-x=MODE` flag already exists in `last30days.py` argparse (line ~1080) but isn't documented in SKILL.md (related to #36). Sorting by `likes`, `engagement`, `recent`, or `score` is already implemented.
 2. **Granular time windows** — The `--days` flag exists (1-30), but hour-level granularity (`--since 3h`) would be new. Low priority since most research use cases are days-scale.
 3. **Thread following** — Would be a significant feature addition. The vendored Bird search in v2.1 may not expose thread fetching. Would need investigation.
 **Comment to post:**
 > Great ideas! Update on each:
 >
 > 1. **Engagement sorting:** `--sort-x=MODE` already exists in the script (likes, engagement, recent, score). It wasn't documented in SKILL.md — fixing that (#36). Should work if you call the Python script directly.
 > 2. **Granular time windows:** Interesting idea. Currently `--days=N` goes down to 1 day. Hour-level would need a new flag and Bird-specific query modification. Adding to backlog.
 > 3. **Thread following:** Love this in principle — high-engagement thread starters are gold. Would need to assess what the vendored Bird search supports. May be a v2.2+ feature.
 >
 > A PR for any of these would be welcome!
 **Priority:** Low — nice-to-haves; #1 is already done, #2/#3 are backlog.
 ---
 ### #19 — Bird is missing
 **Author:** brianjking (Brian J King) | **Created:** 2026-02-09
 > Bird GitHub repo is gone. Comments confirm npm package is deprecated, Homebrew install fails. Multiple users affected.
 **Proposed action:** **CLOSE (resolved in v2.1)**
 This was a major pain point that drove the v2.1 vendoring decision. Bird's GraphQL client is now bundled directly in the repo at `scripts/lib/vendor/bird-search/`. No external npm install needed.
 **Comment to post:**
 > **Resolved in v2.1.0** (released Feb 15, 2026).
 >
 > Bird's Twitter GraphQL search client is now vendored directly into the skill at `scripts/lib/vendor/bird-search/`. No npm install, no Homebrew, no external dependency. Just needs Node.js 22+ in your PATH.
 >
 > If you're on an older version, update to v2.1:
 > ```
 > npx skills add https://github.com/mvanhorn/last30days-skill
 > ```
 >
 > Authentication still works via Safari cookies (auto-detected) or `AUTH_TOKEN`/`CT0` environment variables. See README for setup.
 >
 > Thanks to everyone who reported this — it's what motivated bundling the search client directly.
 ---
 ### #4 — Add macOS Python SSL certificate prerequisite to README
 **Author:** joshdaws (Josh Daws) | **Created:** 2026-01-27
 > Python.org macOS installer doesn't include SSL certificates. Users get `CERTIFICATE_VERIFY_FAILED` errors on all API calls.
 **Proposed action:** **FIX (accept — docs change)**
 Add a troubleshooting section to README.md. The reporter's suggested text is good. This only affects python.org installs (not Homebrew), but it's a confusing error when hit.
 **Comment to post:**
 > Good call — adding a troubleshooting section to the README for this. The `certifi` fallback is interesting but adds a dependency; the docs fix is simpler and sufficient since Homebrew Python (the most common Claude Code setup) isn't affected.
 **Priority:** Low — affects a subset of macOS users, but easy to fix with docs.
 ---
 ### #2 — npx skills add fails for this package
 **Author:** mikecfisher (Mike Fisher) | **Created:** 2026-01-26
 > `npx skills add` can't find any skills in the repo. Owner commented that PR #1 restructures to standard plugin format.
 **Proposed action:** **CLOSE (resolved)**
 The repo was restructured to standard plugin format in PR #1 (SKILL.md at root, .claude-plugin/ directory). The `skills` CLI should now detect the skill correctly. If #32's marketplace.json schema issue is also fixed, this should be fully resolved.
 **Comment to post:**
 > This was fixed in the repo restructuring (PR #1, merged Jan 27). The skill now has a proper `SKILL.md` at root + `.claude-plugin/` directory.
 >
 > If you're still having trouble, it may be the marketplace.json schema issue tracked in #32. Please try again and reopen if the problem persists.
 ---
 ## Summary Table
 | # | Title | Action | Priority | Effort |
 |---|-------|--------|----------|--------|
 | **43** | Security Assessment Offer | **CLOSE (spam)** | — | None |
 | **42** | License & Secondary Dev Permission | **COMMENT + CLOSE** | Low | None (after #35) |
 | **41** | OpenRouter API replacement | **COMMENT + CLOSE** | Low | None |
 | **40** | watchlist: search_queries unused | **FIX** | Medium | Small (~10 lines) |
 | **39** | watchlist: YouTube not stored + silent run-one | **FIX** | Medium | Small (~25 lines) |
 | **36** | SKILL.md flag forwarding | **COMMENT + investigate** | Medium | Medium (SKILL.md rewrite) |
 | **35** | Add LICENSE file | **FIX** | High | Trivial (1 file) |
 | **34** | License missing | **CLOSE (dup of #35)** | — | None |
 | **32** | marketplace.json schema error | **FIX** | High | Small (schema update) |
 | **31** | skills.sh bad audit scores | **COMMENT** | Medium | None (after #35) |
 | **30** | Bird source mapping edge case | **FIX** | Low | Small (~3 lines) |
 | **29** | YouTube "not installed" false message | **FIX** | Low | Small (~5 lines) |
 | **22** | Bird feature requests | **COMMENT (partial accept)** | Low | None (backlog) |
 | **19** | Bird is missing | **CLOSE (resolved v2.1)** | — | None |
 | **4** | macOS SSL certificate docs | **FIX (docs)** | Low | Small (README section) |
 | **2** | npx skills add fails | **CLOSE (resolved)** | — | None |
 ---
 ## Recommended Execution Order
 ### Batch 1 — Quick wins (close/comment only, no code)
 1. Close #43 (spam)
 2. Close #34 (dup of #35)
 3. Close #19 (resolved in v2.1)
 4. Close #2 (resolved by repo restructure)
 5. Comment + close #41 (OpenRouter question)
 6. Comment #22 (Bird feature requests — partial accept, backlog)
 7. Comment #31 (skills.sh audit — will improve after LICENSE fix)
 ### Batch 2 — Trivial fixes
 8. **#35** — Add MIT LICENSE file to repo root
 9. Comment + close #42 (license question — now answered by LICENSE file)
 ### Batch 3 — Small code fixes
 10. **#29** — Fix YouTube "not installed" false message (`render.py` + `last30days.py`)
 11. **#30** — Fix Bird source mapping for `reddit-web` combo (`last30days.py`)
 12. **#40** — Fix watchlist `search_queries` usage (`watchlist.py`)
 13. **#39** — Fix watchlist YouTube/TikTok extraction + `run-one` output (`watchlist.py`)
 ### Batch 4 — Investigation needed
 14. **#32** — Fix marketplace.json schema (needs schema research)
 15. **#36** — Fix `$ARGUMENTS` flag forwarding (needs Claude Code expansion testing)
 16. **#4** — Add SSL troubleshooting to README
 ---
 ## Proposed Comment Templates
 ### For spam (#43):
 > Closing — this is not a bug report or feature request.
 ### For duplicates (#34):
 > Duplicate of #35. Adding MIT LICENSE file — thanks!
 ### For resolved issues (#19, #2):
 > Resolved in v2.1.0. [details specific to issue]. Closing — please reopen if you're still experiencing this on the latest version.
 ### For questions (#41, #42):
 > [Answer the question]. Closing as answered — feel free to reopen if you have follow-ups.
 ### For accepted bugs (#29, #30, #39, #40):
 > Confirmed — [brief acknowledgment]. Fix incoming. Thanks for the detailed report.
@@ -0,0 +1,156 @@
 # refactor: Replace Apify with pay-as-you-go TikTok API
 **Type:** refactor
 **Date:** 2026-03-03
 **Status:** Draft
 ## Problem
 Apify requires a monthly subscription even for low-volume usage. We need a true pay-as-you-go TikTok data API that returns structured video data (views, likes, comments, shares, author, hashtags, captions) from keyword search.
 ## Current Architecture
 Our TikTok integration makes **exactly 2 API calls per /last30days invocation**:
 1. **Search call** — keyword search, returns 10-40 videos with engagement metrics
 2. **Caption enrichment call** — fetches spoken-word subtitles for top 3-8 videos
 We consume these fields from the response:
 - `id`, `text` (description), `webVideoUrl`, `authorMeta.name`
 - `playCount`, `diggCount`, `commentCount`, `shareCount`
 - `hashtags[].name`, `videoMeta.duration`
 - `createTimeISO` or `createTime` (date)
 - `subtitleText` / `subtitles` (captions, secondary call)
 Files involved:
 - `scripts/lib/tiktok.py` — search, parse, relevance scoring, caption fetching
 - `scripts/lib/apify_client_wrapper.py` — shared Apify client (also designed for future FB/IG)
 - `scripts/lib/schema.py` — `TikTokItem` dataclass
 - `scripts/lib/normalize.py` — `normalize_tiktok()`
 - `scripts/last30days.py` — orchestrator (calls `tiktok.search_and_enrich()`)
 ## Why Most of Those Alternatives Won't Work
 The services in the user's table (Spider, Zyte, Scrappey, Serper.dev) are **general web scrapers** — they return raw HTML, not structured TikTok data. We'd have to build our own HTML parser, handle anti-bot protections, and reverse-engineer TikTok's response format. That's a completely different (and fragile) approach.
 What we need is a **TikTok-specific API** that returns structured JSON with engagement metrics from keyword search.
 ## Alternatives Evaluated
 ### RECOMMENDED: ScrapeCreators — Best True Pay-As-You-Go
 | Attribute | Details |
 |-----------|---------|
 | **TikTok structured data** | Yes — 19 dedicated endpoints including keyword search |
 | **Pricing model** | True PAYG — buy credits, credits never expire |
 | **Cost at our volume** | ~$0.60/month ($10 buys 5,000 credits, lasts 16-33 months) |
 | **Free tier** | 100-10,000 free credits on signup (no credit card) |
 | **Python SDK** | No dedicated SDK — simple REST API (`requests.get()`) |
 | **Search endpoint** | "Search by Keyword" and "Top Search" |
 | **Risk** | Newer service, limited track record |
 **Why it's the best fit:** $10 literally lasts over a year at our volume. No subscription, no expiring credits. The lack of a Python SDK is irrelevant — it's a single `requests.get()` call.
 ### Runner-up: EnsembleData — Best SDK, But Subscription
 | Attribute | Details |
 |-----------|---------|
 | **TikTok structured data** | Full — 15+ endpoints, all engagement metrics |
 | **Pricing model** | Monthly subscription ($100/mo after 7-day trial) |
 | **Cost at our volume** | $0 during trial (50 units/day), $100/mo after |
 | **Free tier** | 50 units/day for 7 days, no CC required |
 | **Python SDK** | Yes — `pip install ensembledata` |
 | **Search endpoint** | "Keyword Search" returns ~20 posts/call (1 unit) |
 | **Risk** | $100/mo is overkill for 5-10 searches/day |
 **Verdict:** Best data quality and SDK, but $100/mo is absurd for our ~150-300 requests/month. Same subscription problem as Apify.
 ### Backup: tikwm.com — Free but Risky
 | Attribute | Details |
 |-----------|---------|
 | **TikTok structured data** | Likely yes (needs live testing) |
 | **Pricing model** | Completely free, no API key required |
 | **Cost at our volume** | $0 |
 | **Free tier** | 5,000 requests/day |
 | **Python SDK** | Community wrappers (damirTAG/TikTok-Module, kittenbark/tikwm) |
 | **Search endpoint** | `https://www.tikwm.com/api/feed/search?keywords=TERM&count=20` |
 | **Risk** | Unaffiliated third-party, could disappear anytime, no SLA |
 **Verdict:** Great for development/testing. Too risky as sole production backend. Could be a zero-cost fallback.
 ### Not Recommended
 | Service | Why Not |
 |---------|---------|
 | **TikAPI** | $50-189/mo subscription — same problem as Apify |
 | **Bright Data** | $499/mo minimum — enterprise pricing |
 | **davidteather/TikTok-Api** | Video search is broken, requires Playwright, fragile |
 | **Spider, Zyte, Scrappey** | Generic scrapers — return raw HTML, no TikTok structure |
 | **Piloterr** | No TikTok endpoints, subscription only |
 ## Recommended Approach
 ### Option A: ScrapeCreators as primary (Recommended)
 - [ ] Sign up for ScrapeCreators, get free credits
 - [ ] Test the "Search by Keyword" endpoint to verify it returns all required fields
 - [ ] Refactor `tiktok.py` to use ScrapeCreators REST API instead of Apify actor
 - [ ] Keep `apify_client_wrapper.py` for future FB/IG (or refactor to generic wrapper)
 - [ ] Update `.env` config: `SCRAPECREATORS_API_KEY` replaces `APIFY_API_TOKEN` for TikTok
 - [ ] Update README, SKILL.md installation instructions
 - [ ] Buy $10 credits after confirming it works
 ### Option B: tikwm.com as primary (Zero cost, higher risk)
 - [ ] Test tikwm.com search endpoint to verify response schema
 - [ ] If it returns engagement metrics, implement as primary backend
 - [ ] Add ScrapeCreators as paid fallback when tikwm fails
 - [ ] No API key required — simplest user setup
 ### Option C: Keep Apify, document the subscription requirement
 - [ ] Update README to clarify Apify requires a paid plan
 - [ ] Add note about the free $5/mo credits tier (if it still works without subscription)
 - [ ] Ship as-is with clear billing expectations
 ## Implementation Plan (Option A)
 ### Phase 1: Validate ScrapeCreators API
 - [ ] Sign up and get API key
 - [ ] Test keyword search endpoint: `GET /tiktok/search?keyword={topic}&count=20`
 - [ ] Verify response contains: video ID, play count, likes, comments, shares, author, hashtags, date, description
 - [ ] Test caption/subtitle availability (or confirm description text is sufficient)
 ### Phase 2: Swap the Backend
 - [ ] Create `scripts/lib/scrapecreators_client.py` (simple REST wrapper, ~40 lines)
 - [ ] Refactor `tiktok.py:search_tiktok()` to call ScrapeCreators instead of Apify
 - [ ] Map ScrapeCreators response fields to our existing item dict format
 - [ ] Refactor `tiktok.py:fetch_captions()` — check if ScrapeCreators provides subtitles, else use description text only
 - [ ] Update `env.py` to read `SCRAPECREATORS_API_KEY` (keep `APIFY_API_TOKEN` for backward compat)
 - [ ] Update `scripts/lib/ui.py` spinner messages if needed
 ### Phase 3: Update Docs & Ship
 - [ ] Update README.md installation section (new API key)
 - [ ] Update SKILL.md
 - [ ] Update `~/.config/last30days/.env` locally
 - [ ] Run full test suite
 - [ ] Commit and push
 ## Acceptance Criteria
 - [ ] TikTok search returns structured data with views, likes, comments, shares
 - [ ] No subscription required — pay-as-you-go only
 - [ ] Cost < $1/month at normal usage (5-10 searches/day)
 - [ ] Existing test suite passes with new backend
 - [ ] Graceful degradation when API key is missing (same behavior as today)
 ## Open Questions
 1. Does ScrapeCreators' keyword search support date filtering, or do we filter post-API like we do with Apify?
 2. Does ScrapeCreators return subtitle/caption data, or just video descriptions?
 3. What's the response time? Apify actors took 30-120 seconds. REST APIs should be faster.
 4. Should we keep Apify as a fallback backend (user configures one or the other)?
@@ -0,0 +1,210 @@
 # feat: Add Instagram and Facebook Sources via ScrapeCreators API
 **Date:** 2026-03-04
 **Type:** Enhancement
 **Priority:** Instagram first, Facebook conditional ("if it's good")
 ## Summary
 Add Instagram Reels and Facebook as new research sources in last30days, using the same ScrapeCreators REST API already powering TikTok. Instagram is the primary target; Facebook is a follow-on if the pattern works well.
 Both sources share the existing `SCRAPECREATORS_API_KEY` — no new API keys needed.
 ## Approach
 Replicate the TikTok integration pattern exactly. Each source follows the same 8-step pipeline:
 ```
 tiktok.py pattern → instagram.py (new) → facebook.py (new, conditional)
 ```
 ## ScrapeCreators API Endpoints
 ### Instagram
 | Endpoint | Path | Params | Credits | Notes |
 |----------|------|--------|---------|-------|
 | **Search Reels** | `GET /v1/instagram/reels/search` | `keyword`, pagination | 1 per 10 reels, max 60/req | Keyword search via Google (IG search requires login). V2 also available. |
 | **Transcript** | `GET /v2/instagram/media/transcript` | `url` | 1 | Returns `{transcripts: [{id, shortcode, text}]}`. Videos <2min only. |
 | **Comments** | `GET /v2/instagram/post/comments` | `url`, `cursor` | 1 | Returns `{comments: [{id, text, created_at, user}]}`. 100-300 per call. |
 | **User Reels** | `GET /v1/instagram/user/reels` | `handle` or `user_id`, `max_id`, `trim` | 1 | All reels from a profile. Response: `{items: [...], paging_info}` |
 **Primary search strategy:** `/v1/instagram/reels/search` with keyword param for topic search. This is the analog to TikTok's `/search/keyword`.
 **Response fields per reel item:**
 - `pk` / `code` (shortcode) — reel ID
 - `taken_at` — unix timestamp
 - `play_count` / `ig_play_count` — views
 - `like_count` — likes
 - `comment_count` — comments
 - `video_duration` — seconds
 - `has_audio` — boolean
 - `user` object — username, full_name, is_verified, profile_pic_url
 - `caption` object — text content
 - Media URLs for thumbnails and video versions
 ### Facebook
 | Endpoint | Path | Params | Credits | Notes |
 |----------|------|--------|---------|-------|
 | **Profile Posts** | `GET /v1/facebook/profile/posts` | `url` or `pageId`, `cursor` | 1 | Returns 3 posts at a time with engagement |
 | **Profile Reels** | `GET /v1/facebook/profile/reels` | similar | 1 | 10 reels at a time |
 | **Post** | `GET /v1/facebook/post` | `url` | 1 | Single post/reel by URL |
 | **Transcript** | `GET /v1/facebook/post/transcript` | `url` | 1 | Video transcript, <2min |
 | **Comments** | `GET /v1/facebook/post/comments` | `url`, `feedback_id` | 1 | Post/reel comments |
 **Facebook limitation:** No keyword search endpoint. Only profile-based scraping (3 posts at a time). This makes Facebook significantly less useful for topic-based research vs. Instagram's keyword search.
 **Response fields per post:**
 - `id` — post ID
 - `text` — post content
 - `url` / `permalink` — post URL
 - `author` — `{name, short_name, id}`
 - `reactionCount` — total reactions
 - `commentCount` — comments
 - `videoViewCount` — video views (if applicable)
 - `publishTime` — unix timestamp
 - `topComments` — array of `{id, text, publishTime, author}`
 ## Implementation Plan
 ### Phase 1: Instagram Source (Primary)
 #### 1.1 Create `scripts/lib/instagram.py`
 - [x] Copy structure from `scripts/lib/tiktok.py`
 - [x] Change `SCRAPECREATORS_BASE` to `"https://api.scrapecreators.com"`
 - [x] Implement `search_instagram()` → calls `/v1/instagram/reels/search`
  - Params: `keyword=core_topic`
  - Parse response `items` array
  - Extract: `pk`/`code` as video_id, `taken_at` as date, `play_count`/`like_count`/`comment_count` as engagement, `user.username` as author, `caption.text` as text
  - Build URL: `https://www.instagram.com/reel/{code}`
  - Reuse `_extract_core_subject()`, `_compute_relevance()`, `_tokenize()` from tiktok.py (or factor into shared util)
  - Apply date range filter, sort by views descending
 - [x] Implement `fetch_captions()` → calls `/v2/instagram/media/transcript`
  - For top N items (per depth config), fetch transcript
  - Response: `{transcripts: [{id, shortcode, text}]}`
  - Fallback to caption text if transcript unavailable
  - Truncate to 500 words
 - [x] Implement `search_and_enrich()` → combines search + captions
 - [x] Implement `parse_instagram_response()` → returns `response.get("items", [])`
 - [x] Reuse shared helpers: `_sc_headers()`, `_log()`, `_clean_webvtt()`, `DEPTH_CONFIG`, `STOPWORDS`, `SYNONYMS`
 **Key difference from TikTok:** Instagram response uses `play_count`/`like_count`/`comment_count` directly (no `statistics` wrapper), `user.username` (not `author.unique_id`), `caption.text` (not `desc`), `taken_at` (not `create_time`), `code` shortcode for URL construction.
 #### 1.2 Add `InstagramItem` to `scripts/lib/schema.py`
 - [x] Add dataclass mirroring `TikTokItem` structure:
  ```python
  @dataclass
  class InstagramItem:
      id: str              # "IG1", "IG2", ...
      text: str            # caption text
      url: str             # https://www.instagram.com/reel/{code}
      author_name: str     # Instagram handle
      date: Optional[str]  # YYYY-MM-DD from taken_at
      date_confidence: str # "high"
      engagement: Optional[Engagement]  # views, likes, num_comments
      caption_snippet: str  # transcript or caption text
      hashtags: List[str]   # extracted from caption
      relevance: float
      why_relevant: str
      subs: SubScores
      score: int
      cross_refs: List[str]
  ```
 #### 1.3 Add normalization to `scripts/lib/normalize.py`
 - [x] Add `normalize_instagram_items()` function
  - Assign IDs as `IG1`, `IG2`, ...
  - Map engagement: `views=play_count`, `likes=like_count`, `num_comments=comment_count`
  - Set `date_confidence="high"` (unix timestamp)
 #### 1.4 Add scoring to `scripts/lib/score.py`
 - [x] Add `compute_instagram_engagement_raw()` — same formula as TikTok:
  `0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)`
  Views dominate on Instagram Reels just like TikTok.
 - [x] Add `score_instagram_items()` — same weights: 45% relevance, 25% recency, 30% engagement
 #### 1.5 Add dedup to `scripts/lib/dedupe.py`
 - [x] Add `dedupe_instagram()` — same as `dedupe_tiktok()`, calls `dedupe_items()` with 0.7 threshold
 - [x] Update `get_item_text()` to handle `InstagramItem`
 - [x] Update `_get_cross_source_text()` for cross-source linking
 - [x] Add `IG` prefix to cross-ref detection in `cross_source_link()`
 #### 1.6 Add rendering to `scripts/lib/render.py`
 - [x] Add Instagram section in `render_compact()` — same pattern as TikTok block (lines 251-285)
  - Show: score, @author, date, views/likes, caption snippet, hashtags, why_relevant
 - [x] Update data freshness check to include `instagram_recent`
 - [x] Update stats footer to include Instagram count
 - [x] Add `'IG'` to cross-ref source name mapping
 #### 1.7 Add `Report.instagram` field to `scripts/lib/schema.py`
 - [x] Add `instagram: List[InstagramItem]` and `instagram_error: str` to `Report` dataclass
 #### 1.8 Integrate into `scripts/last30days.py` orchestrator
 - [x] Add `"instagram"` to `VALID_SEARCH_SOURCES`
 - [x] Add `import` for `instagram` module in `scripts/lib/`
 - [x] Add `is_instagram_available()` check in `env.py` — reuse `SCRAPECREATORS_API_KEY` (same key as TikTok)
 - [x] Add `get_instagram_token()` in `env.py` — same as `get_tiktok_token()`, returns `SCRAPECREATORS_API_KEY`
 - [x] Add `_search_instagram()` helper in orchestrator (mirrors `_search_tiktok()`)
 - [x] Add Instagram to the thread pool executor block
 - [x] Add Instagram timeout config (same as TikTok: 90/120/150s for quick/default/deep)
 - [x] Wire through pipeline: normalize → filter → score → sort → dedupe → cross-link → report
 - [x] Add Instagram to `progress.show_complete()` and UI spinner
 #### 1.9 Add to watchlist extraction in `scripts/watchlist.py`
 - [x] Add Instagram findings loop in `_run_topic()` (mirrors TikTok block at lines 204-213)
 #### 1.10 Update README.md
 - [x] Add Instagram to the sources list
 - [x] Note that `SCRAPECREATORS_API_KEY` covers both TikTok and Instagram
 ### Phase 2: Facebook Source (Conditional)
 **Recommendation: SKIP Facebook for now.** Here's why:
 1. **No keyword search endpoint** — Facebook only offers profile-based scraping (`/profile/posts` returns 3 posts at a time). Can't search by topic.
 2. **Low relevance for topic research** — Without keyword search, we'd need to know specific Facebook pages to scrape, which defeats the purpose of automated topic discovery.
 3. **Poor ROI** — 3 posts per API call is very limited compared to Instagram's 60 reels per search.
 4. **Same API key** — If Facebook search is added later, it's trivial to add since it shares `SCRAPECREATORS_API_KEY`.
 If the user still wants Facebook, the implementation would follow the same pattern but would need a different discovery strategy (e.g., hardcoded page list per topic, or using the Ad Library search for commercial topics).
 ## Files to Create/Modify
 | File | Action | Description |
 |------|--------|-------------|
 | `scripts/lib/instagram.py` | **CREATE** | Instagram search + transcript via ScrapeCreators |
 | `scripts/lib/schema.py` | MODIFY | Add `InstagramItem` dataclass, add `instagram` to `Report` |
 | `scripts/lib/normalize.py` | MODIFY | Add `normalize_instagram_items()` |
 | `scripts/lib/score.py` | MODIFY | Add `compute_instagram_engagement_raw()`, `score_instagram_items()` |
 | `scripts/lib/dedupe.py` | MODIFY | Add `dedupe_instagram()`, update text extractors |
 | `scripts/lib/render.py` | MODIFY | Add Instagram render section, update stats |
 | `scripts/lib/env.py` | MODIFY | Add `is_instagram_available()`, `get_instagram_token()` |
 | `scripts/last30days.py` | MODIFY | Add Instagram to orchestrator pipeline |
 | `scripts/watchlist.py` | MODIFY | Add Instagram findings extraction |
 | `README.md` | MODIFY | Add Instagram to sources list |
 ## Shared Code Opportunity
 `_extract_core_subject()`, `_compute_relevance()`, `_tokenize()`, `STOPWORDS`, `SYNONYMS`, and `DEPTH_CONFIG` are duplicated between `tiktok.py` and the new `instagram.py`. Two options:
 1. **Copy-paste** (simpler, matches current pattern) — each source module is self-contained
 2. **Extract to shared module** (cleaner) — move to `scripts/lib/search_utils.py`
 **Recommendation:** Copy-paste for now to match existing pattern. Refactor later if a third ScrapeCreators source is added.
 ## Testing Strategy
 - [x] Run `python3 scripts/lib/instagram.py` with test keyword (if standalone test added)
 - [x] Run `python3 scripts/last30days.py "instagram reels trends" --search=instagram` to test isolated
 - [x] Run full multi-source: `python3 scripts/last30days.py "AI tools" --search=reddit,instagram`
 - [x] Verify JSON output: `--emit=json` includes `instagram` key
 - [x] Verify watchlist extraction works with Instagram findings
 - [x] Check credit usage is reasonable (1 credit per 10 reels search + 1 per transcript)
 ## Credits Budget
 Per research run with Instagram at `default` depth:
 - Search: 1 credit (per 10 reels, returns up to 20) ≈ 2 credits
 - Transcripts: 5 credits (max_captions=5 at default depth)
 - **Total: ~7 credits per topic** (vs TikTok ~6 credits)
@@ -0,0 +1,147 @@
 ---
 title: "feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration"
 type: enhancement
 status: pending
 date: 2026-03-04
 ---
 # feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration
 ## Summary
 Ship everything from the last sprint as one combined GitHub release: TikTok's migration from Apify to ScrapeCreators (already committed) + Instagram Reels as the 8th source (uncommitted) + SKILL.md URL regression fixes. Version bump to v2.8.
 ## What's Shipping
 ### 1. Instagram Reels — 8th source (NEW)
 - Search Instagram Reels by keyword via ScrapeCreators `/v1/instagram/reels/search`
 - Spoken-word transcript extraction via `/v2/instagram/media/transcript`
 - Full pipeline: search → normalize → score → dedupe → cross-link → render
 - Shares `SCRAPECREATORS_API_KEY` with TikTok (no new API key needed)
 - ~7 credits per topic at default depth
 ### 2. TikTok — Apify → ScrapeCreators migration (ALREADY COMMITTED)
 - Replaced Apify dependency with ScrapeCreators API
 - Same functionality, different backend
 - No more `APIFY_API_TOKEN` — uses `SCRAPECREATORS_API_KEY`
 ### 3. SKILL.md quality fixes
 - Instagram added to stats template, citation priority, data sections, footer
 - URL regression fix: explicit URL-to-name extraction rules, stronger anti-Sources instruction
 - Security section updated: Apify → ScrapeCreators
 ## Release Checklist
 ### Pre-commit: Update docs
 - [ ] **README.md** — Update for v2.8:
  - [ ] Change title from "v2.7" to "v2.8"
  - [ ] Add "New in v2.8" banner: Instagram Reels + ScrapeCreators migration
  - [ ] Update installation section: `APIFY_API_TOKEN` → `SCRAPECREATORS_API_KEY`
  - [ ] Add Instagram to the "How it works" flow description (line 132)
  - [ ] Update TikTok section: replace Apify references with ScrapeCreators
  - [ ] Add Instagram section (after TikTok section, ~line 963)
  - [ ] Update API endpoints table: `api.apify.com` → `api.scrapecreators.com`, add Instagram endpoints
  - [ ] Update closing tagline to include Instagram
  - [ ] Remove "The shared Apify client wrapper is designed for future Facebook and Instagram sources" (line 963) — Instagram is here now
 - [ ] **CHANGELOG.md** — Add v2.8.0 entry:
  ```
  ## [2.8.0] - 2026-03-04
  ### Highlights
  Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements.
  ### Added
  - Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction
  - Instagram items in SKILL.md stats template, citation priority, and output footer
  - URL-to-name extraction examples in SKILL.md for cleaner web source display
  ### Changed
  - TikTok backend migrated from Apify to ScrapeCreators API (same key covers TikTok + Instagram)
  - `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
  - SKILL.md version bumped to v2.8
  - WebSearch citation instruction strengthened to prevent trailing Sources: blocks
  ### Fixed
  - Web stats line showing full URLs instead of plain domain names (regression from v2.7)
  - Trailing "Sources:" block appearing after invitation (WebSearch tool mandate conflict)
  - Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
  ```
 - [ ] **SKILL.md** frontmatter — Bump version from "2.7" to "2.8"
 - [ ] **SKILL.md** description — Add Instagram to the description field
 ### Commit & tag
 - [ ] Stage all changes: modified files + 4 new files (`scripts/lib/instagram.py`, 3 plan docs)
 - [ ] Commit with message:
  ```
  feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
  - Add Instagram Reels as 8th research source via ScrapeCreators API
  - Migrate TikTok from Apify to ScrapeCreators (same API key)
  - Add SCRAPECREATORS_API_KEY config (replaces APIFY_API_TOKEN)
  - Fix web stats URL regression and trailing Sources: block
  - Fix Instagram/TikTok not running in --search=instagram web-only path
  - Update SKILL.md with Instagram stats, citations, URL formatting rules
  ```
 - [ ] Create tag: `git tag -a v2.8.0 -m "v2.8.0: Instagram Reels + ScrapeCreators"`
 - [ ] Push: `git push origin main --tags`
 ### Post-push: GitHub release
 - [ ] Create GitHub release via `gh release create v2.8.0`:
  ```
  ## What's New in v2.8
  **Instagram Reels** is now the 8th signal source. Search any topic and get trending Instagram Reels with views, likes, and spoken-word transcripts — scored and ranked alongside Reddit, X, YouTube, TikTok, HN, Polymarket, and the web.
  **TikTok migrated to ScrapeCreators API.** Same functionality, new backend. Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. One key now covers both TikTok and Instagram.
  ### Setup
  Sign up at [scrapecreators.com](https://scrapecreators.com) (100 free credits, then PAYG) and add your key:
  ```bash
  echo 'SCRAPECREATORS_API_KEY=your_key' >> ~/.config/last30days/.env
  ```
  ### Breaking Change
  - `APIFY_API_TOKEN` is no longer used. Replace with `SCRAPECREATORS_API_KEY`.
  ### Bug Fixes
  - Fixed web source URLs leaking into stats display
  - Fixed Instagram/TikTok not running when used with `--search=` flag
  ```
 ### Post-release: Sync
 - [ ] Run `bash scripts/sync.sh` to deploy to all 4 skill destinations
 - [ ] Verify Instagram works in a fresh `/last30days` session
 ## Files Modified (this release)
 | File | Status | Description |
 |------|--------|-------------|
 | `scripts/lib/instagram.py` | NEW | Instagram search + transcript via ScrapeCreators |
 | `scripts/lib/schema.py` | MODIFIED | InstagramItem dataclass, Report.instagram field |
 | `scripts/lib/normalize.py` | MODIFIED | normalize_instagram_items() |
 | `scripts/lib/score.py` | MODIFIED | score_instagram_items(), engagement formula |
 | `scripts/lib/dedupe.py` | MODIFIED | dedupe_instagram(), cross-source linking |
 | `scripts/lib/render.py` | MODIFIED | Instagram render section, stats |
 | `scripts/lib/env.py` | MODIFIED | is_instagram_available(), get_instagram_token() |
 | `scripts/lib/ui.py` | MODIFIED | Instagram spinner messages |
 | `scripts/last30days.py` | MODIFIED | Instagram in orchestrator pipeline |
 | `scripts/watchlist.py` | MODIFIED | Instagram findings extraction |
 | `SKILL.md` | MODIFIED | Instagram in stats/citations/footer, URL fixes |
 | `README.md` | TO UPDATE | Instagram section, ScrapeCreators migration |
 | `CHANGELOG.md` | TO UPDATE | v2.8.0 entry |
 | `docs/plans/*.md` | NEW (3) | Plan documents for this work |
@@ -0,0 +1,135 @@
 ---
 title: "fix: Web sources showing full URLs instead of plain domain names"
 type: fix
 status: pending
 date: 2026-03-04
 ---
 # fix: Web Sources Showing Full URLs Instead of Plain Domain Names
 ## Problem
 Two regressions in the `/last30days` skill output:
 ### Regression 1: Full URLs on the Web stats line
 The `🌐 Web:` stats line is showing full URLs instead of plain source names:
 **BAD (current — "Instagram Trends" run):**
 ```
 ├─ 🌐 Web: 10+ pages — https://later.com/blog/instagram-reels-trends/,
 https://socialbee.com/blog/instagram-trends/,
 https://buffer.com/resources/instagram-algorithms/,
 https://metricool.com/instagram-trends/,
 https://napoleoncat.com/blog/instagram-reels-trends/
 ```
 **GOOD (expected):**
 ```
 ├─ 🌐 Web: 10+ pages — Later, SocialBee, Buffer, Metricool, NapoleonCat
 ```
 ### Regression 2: Trailing Sources: block with full URLs
 A separate `Sources:` section appears at the bottom of the response with full URLs:
 ```
 Sources:
 - https://www.heyorca.com/blog/instagram-social-news
 - https://socialbee.com/blog/instagram-updates/
 - https://buffer.com/resources/instagram-algorithms/
 - https://www.cnn.com/2026/02/22/tech/social-media-addiction-trial-tobacco-moment
 ```
 This was fixed in commit `82efa61` (2026-03-02) but is regressing intermittently.
 ## Root Cause
 The SKILL.md instructions at lines 219-221, 404, and 409 already say the right thing:
 - Line 404: `├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name`
 - Line 409: `"plain names, no URLs — URLs wrap badly in terminals"`
 - Lines 219-221: "DO NOT output a separate Sources: block"
 But the model ignores these because:
 1. **The template `Source Name` is too abstract.** The model sees WebSearch results with full URLs and doesn't know how to extract a human-friendly name from `https://later.com/blog/instagram-reels-trends/`. It needs explicit examples showing the transformation.
 2. **The WebSearch system mandate still wins.** The WebSearch tool's built-in instruction (`"you MUST include a Sources: section"`) outcompetes the skill's instruction. The current countermeasure (line 409) works sometimes but not reliably — it needs to be stronger and repeated.
 3. **No explicit extraction rule.** The model needs a concrete rule for turning URLs into names: strip protocol, strip path, strip `www.`, capitalize.
 ## Proposed Solution
 **SKILL.md edits only. No Python changes.**
 ### Fix 1: Add explicit URL-to-name examples in the stats template (line 404 area)
 After the stats template block, add concrete examples showing the transformation:
 ```
 **🌐 Web: line formatting:**
 - Extract the SITE NAME from each URL — strip protocol, path, and "www."
 - Use the publication's proper name when recognizable
 - Examples:
  - https://later.com/blog/instagram-reels-trends/ → "Later"
  - https://socialbee.com/blog/instagram-trends/ → "SocialBee"
  - https://buffer.com/resources/instagram-algorithms/ → "Buffer"
  - https://www.cnn.com/2026/02/22/tech/... → "CNN"
  - https://medium.com/the-ai-studio/... → "Medium"
  - https://radicaldatascience.wordpress.com/... → "Radical Data Science"
 - NEVER paste the URL itself. ONLY the site name as plain text.
 - Separate names with commas: "Later, SocialBee, Buffer, CNN, Medium"
 ```
 ### Fix 2: Strengthen the anti-Sources instruction (line 409 area)
 Replace the current single-paragraph note with a louder, more explicit instruction:
 ```
 **⚠️ WebSearch citation requirement — ALREADY SATISFIED above.**
 The WebSearch tool mandates source citation. That requirement is FULLY satisfied
 by the source names on the 🌐 Web: line above. Do NOT append a separate
 "Sources:" section at the end of your response. Do NOT list URLs anywhere in
 your output. The 🌐 Web: line IS your citation. You're done.
 ```
 ### Fix 3: Add a negative example in the URL FORMATTING section (line 356 area)
 Extend the existing BAD/GOOD examples to cover the stats line specifically:
 ```
 URL FORMATTING: NEVER paste raw URLs anywhere in the output.
 - BAD: "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
 - GOOD: "per Rolling Stone"
 - BAD stats line: "🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/..."
 - GOOD stats line: "🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee"
 ```
 ### Fix 4: Update Security section (line 588, 600)
 While we're in SKILL.md, update the stale Apify references to ScrapeCreators:
 - Line 588: Change Apify reference to ScrapeCreators for TikTok
 - Line 600: Update TikTok requirement note
 - Add Instagram source mention
 ## Files to Modify
 | File | Action | Description |
 |------|--------|-------------|
 | `SKILL.md` | MODIFY | Strengthen URL formatting rules, add examples, fix Apify refs |
 ## Implementation Steps
 - [x] Add URL-to-name extraction examples after stats template (after line 407)
 - [x] Strengthen anti-Sources instruction (replace line 409)
 - [x] Add BAD/GOOD stats line example to URL FORMATTING section (around line 356)
 - [x] Update Security section: Apify → ScrapeCreators, add Instagram (lines 588, 600)
 - [x] Run `bash scripts/sync.sh` to deploy to all destinations
 - [ ] Test with `/last30days Instagram Trends` — confirm plain names, no trailing Sources:
 ## Acceptance Criteria
 - [ ] `🌐 Web:` line shows plain names only (e.g., "Later, SocialBee, Buffer")
 - [ ] No `Sources:` section appears at the bottom of the response
 - [ ] No raw URLs appear anywhere in the output (synthesis, stats, or footer)
 - [ ] Security section reflects current source stack (ScrapeCreators, not Apify)
@@ -0,0 +1,255 @@
 # feat: Reddit ScrapeCreators v2 — Improvements from Beta Testing
 **Date:** 2026-03-05
 **Type:** Enhancement
 **Version:** v2.9 → v2.9.1-beta (or v3.0-beta if shipping to public)
 **Branch:** `feat/reddit-scrapecreators` (continue existing branch)
 ---
 ## Summary
 Three focused improvements to the Reddit ScrapeCreators integration based on 5 full-pipeline tests ("Claude Code skills", "Kanye West", "Anthropic odds", "best rap songs lately", "Nano Banana Pro prompting"):
 1. **Elevate top Reddit comments** — give weight to the wittiest/highest-voted comment in scoring and rendering
 2. **Improve subreddit discovery** — tune heuristic so ambiguous queries find discussion subs, not utility subs
 3. **Make ScrapeCreators the default recommended Reddit method** — update onboarding, SKILL.md metadata, and env.py messaging
 ---
 ## Problem Statement
 ### 1. Comments are undervalued
 - ScrapeCreators returns real comment data with scores, but top comments only appear as `Insights:` text under each Reddit item
 - The top comment (often the funniest/cleverest reply) gets no special treatment — it's just one of 3 comment excerpts
 - Reddit's value IS the comments — upvoted replies are the distilled crowd wisdom
 - Currently `comment_insights` are truncated at 150 chars and only 3 are shown per item in compact output
 - No scoring bonus for posts that have high-quality comment threads
 ### 2. Subreddit discovery picks wrong subs for ambiguous queries
 - "best rap songs lately" discovered `r/NameThatSong` and `r/findthatsong` (utility subs for identifying songs) instead of discussion subs like `r/hiphopheads` or `r/rap`
 - "Kanye West" picked `r/ConcertsIndia_` as second sub — tangential at best
 - The current heuristic is pure frequency count on `subreddit` field from global results, with no relevance weighting
 - Utility/meta subs often dominate because the same query matches many "help me find X" posts
 ### 3. Onboarding still suggests OpenAI as the primary Reddit method
 - SKILL.md metadata says `primaryEnv: OPENAI_API_KEY` and `requires.env: [OPENAI_API_KEY]`
 - The web-only mode banner mentions "OPENAI_API_KEY or codex login → Reddit threads"
 - `env.py` error messages direct users to OpenAI for Reddit access
 - ScrapeCreators is cheaper ($0.012 vs $0.03-0.10), faster (17s vs 60-90s), returns real data, and shares a key with TikTok + Instagram
 - New users should be told: "Get a SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, all three)"
 ---
 ## Implementation Plan
 ### Task 1: Elevate Top Comments in Scoring and Rendering
 **Goal:** Give Reddit posts a scoring bonus when they have highly-engaged comment threads, and render the #1 comment with special treatment.
 **Files to modify:**
 - `scripts/lib/reddit.py` — enrich with `top_comment_score` metadata
 - `scripts/lib/score.py` — add comment quality bonus to Reddit scoring
 - `scripts/lib/render.py` — render top comment with special formatting
 - `scripts/lib/schema.py` — add `top_comment_excerpt` field to RedditItem (optional, may just use existing `top_comments[0]`)
 #### 1a. Comment enrichment improvements (`scripts/lib/reddit.py`)
 - [x] In `enrich_with_comments()`, after sorting comments by score, tag the item with:
  - `top_comment_excerpt`: The highest-scored comment's body (up to 200 chars)
  - `top_comment_score`: The upvote count of the #1 comment
  - `top_comment_author`: Author of the #1 comment
 - [x] Increase comment excerpt length from 300 → 400 chars for top comment only (funny/clever comments need more room)
 - [x] Increase `comment_insights` limit from 7 → 10 (we have the data, show it)
 - [x] For posts with enriched comments, store the comment count ratio: `top_comment_score / post_score` — a high ratio means the comment outshines the post (Reddit gold)
 #### 1b. Scoring bonus for comment quality (`scripts/lib/score.py`)
 - [x] In `compute_reddit_engagement_raw()`, add a comment quality signal:
  - Current formula: `0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)`
  - New formula: `0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)`
  - This gives a ~10% weight to comment quality, slightly reducing post score and comment count weights
  - Posts where the community engaged deeply (high top-comment score) rank higher
 - [x] Need to pass `top_comment_score` through the engagement data — either:
  - Option A: Add `top_comment_score` to `schema.Engagement` (cleanest)
  - Option B: Read from `item.top_comments[0].score` during scoring (no schema change)
  - **Recommend Option B** to avoid schema bloat — scoring can peek at `top_comments`
 #### 1c. Render top comment prominently (`scripts/lib/render.py`)
 - [x] In `render_compact()` Reddit section, after the `Insights:` block, add a "Top Comment:" line for items that have top_comments:
  ```
  **R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
    Claude Code creator: In the next version, introducing two new skills
    https://www.reddit.com/r/ClaudeAI/comments/...
    *Reddit global search*
    💬 Top comment (247 upvotes): "So are they /batch migrating to Rust? :)"
    Insights:
      - TL;DR generated automatically after 50 comments...
      - He's /batch migrating code daily?..
  ```
 - [x] Only show `💬 Top comment` for items where `top_comments[0].score >= 10` (skip low-engagement comments)
 - [x] Truncate at 200 chars with `...` if needed
 - [x] Also update `render_full_report()` to include the top comment prominently
 #### 1d. Update SKILL.md synthesis instructions
 - [x] In the "Judge Agent: Synthesize All Sources" section, add guidance:
  ```
  5b. For Reddit: Pay special attention to top comments — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes, quote it directly in your synthesis. Reddit's value is in the comments.
  ```
 - [x] In the citation priority list, add: "When citing Reddit, prefer quoting top comments over just the thread title"
 ---
 ### Task 2: Improve Subreddit Discovery Heuristic
 **Goal:** Find topical discussion subs rather than utility/meta subs.
 **Files to modify:**
 - `scripts/lib/reddit.py` — improve `discover_subreddits()` logic
 #### 2a. Add relevance-weighted subreddit scoring
 - [x] Replace pure frequency count with a weighted score:
  ```python
  def discover_subreddits(results, topic, max_subs=5):
      core = _extract_core_subject(topic)
      core_words = set(core.lower().split())
      scores = Counter()
      for post in results:
          sub = post.get("subreddit", "")
          if not sub:
              continue
          # Base: frequency count
          base = 1.0
          # Bonus: subreddit name contains a core topic word
          sub_lower = sub.lower()
          if any(w in sub_lower for w in core_words if len(w) > 2):
              base += 2.0
          # Penalty: known utility/meta subreddits
          if sub_lower in UTILITY_SUBS:
              base *= 0.3
          # Bonus: post engagement (high-engagement posts = better sub)
          ups = post.get("ups") or post.get("score", 0)
          if ups > 100:
              base += 0.5
          scores[sub] += base
      return [sub for sub, _ in scores.most_common(max_subs)]
  ```
 #### 2b. Define utility/meta subreddit blocklist
 - [x] Add a small set of subs that are "find X for me" or "identify X" rather than discussion:
  ```python
  UTILITY_SUBS = frozenset({
      'namethatsong', 'findthatsong', 'tipofmytongue',
      'whatisthissong', 'helpmefind', 'whatisthisthing',
      'whatsthissong', 'findareddit', 'subredditdrama',
  })
  ```
 - [x] Keep this small and focused — don't over-filter. Only penalty (0.3x), not ban.
 #### 2c. Try secondary query for subreddit discovery
 - [x] If the first global search returns <3 unique subreddits above threshold, run a second global search with just `{core subject}` (stripped even further) to cast a wider net for subreddit frequencies
 - [x] This helps niche topics where the full query is too specific
 ---
 ### Task 3: Make ScrapeCreators the Default Reddit Method
 **Goal:** New users should be guided to ScrapeCreators first, not OpenAI.
 **Files to modify:**
 - `SKILL.md` — metadata section, onboarding banner, security section
 - `scripts/lib/env.py` — error messages and missing key guidance
 - `scripts/lib/render.py` — web-only mode banner
 #### 3a. Update SKILL.md metadata
 - [x] Change `primaryEnv: OPENAI_API_KEY` → `primaryEnv: SCRAPECREATORS_API_KEY`
 - [x] Change `requires.env: [OPENAI_API_KEY]` → `requires.env: [SCRAPECREATORS_API_KEY]`
 - [x] Keep OPENAI_API_KEY mentioned but as optional/legacy
 #### 3b. Update web-only mode banner (`scripts/lib/render.py`)
 - [x] Change the current banner:
  ```
  - `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments
  ```
  To:
  ```
  - `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views
  - `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)
  ```
 #### 3c. Update env.py messaging
 - [x] In `get_missing_keys()`, when Reddit is missing, suggest ScrapeCreators first:
  - Current: returns `'reddit'` which triggers "Add OPENAI_API_KEY or run codex login" in SKILL.md
  - Add a helper: `get_setup_hint(missing)` that returns:
    - For 'reddit': `"Add SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, ~$0.002/search)"`
    - For 'x': `"Add XAI_API_KEY for X posts"`
    - For 'all': `"Add SCRAPECREATORS_API_KEY (Reddit+TikTok+Instagram) and XAI_API_KEY (X)"`
 #### 3d. Update Security & Permissions section in SKILL.md
 - [x] Add ScrapeCreators Reddit to the security section:
  ```
  - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit, TikTok, and Instagram search (requires SCRAPECREATORS_API_KEY)
  ```
 - [x] Move "Sends search queries to OpenAI's Responses API for Reddit discovery" to a "Legacy:" subsection
 - [x] Update "Reddit" description in `allowed-tools` or tags if needed
 #### 3e. Update render.py coverage note
 - [x] In `render_compact()`, the coverage note for `reddit-only` currently says "Add an xAI key"
 - [x] When ScrapeCreators is the active Reddit source, no need to mention OpenAI at all
 ---
 ## Acceptance Criteria
 - [x] Top Reddit comment is rendered with `💬` prefix and upvote count for enriched posts
 - [x] Posts with high top-comment scores rank slightly higher (visible in score differences)
 - [x] "best rap songs lately" discovers at least one discussion sub (r/hiphopheads, r/rap, r/Music, etc.) instead of only utility subs
 - [x] SKILL.md `primaryEnv` is `SCRAPECREATORS_API_KEY`
 - [x] Web-only mode banner recommends ScrapeCreators first
 - [x] All 5 test topics still pass (run same tests as before)
 - [x] No regression in OpenAI fallback path
 ---
 ## Files Changed (Summary)
 | File | Change |
 |------|--------|
 | `scripts/lib/reddit.py` | Improve `discover_subreddits()` with relevance weighting, add utility sub penalties, enhance `enrich_with_comments()` top comment metadata |
 | `scripts/lib/score.py` | Add 10% comment quality weight to Reddit engagement formula |
 | `scripts/lib/render.py` | Add `💬 Top comment` line to compact output, update web-only banner |
 | `scripts/lib/env.py` | Add `get_setup_hint()`, update missing key messaging |
 | `SKILL.md` | Change `primaryEnv`, update onboarding banner, add comment synthesis guidance, update security section |
 ---
 ## Cost Impact
 No cost increase. Same number of API calls per search. The changes are all in local logic (scoring, rendering, discovery heuristic).
 ---
 ## Testing Plan
 1. Re-run the same 5 test topics from beta testing
 2. Verify top comments appear with `💬` in output
 3. Verify "best rap songs lately" discovers at least one discussion subreddit
 4. Verify `--diagnose` output recommends ScrapeCreators
 5. Verify OpenAI fallback still works (unset SCRAPECREATORS_API_KEY, set OPENAI_API_KEY)
@@ -0,0 +1,58 @@
 {
  "items": [
    {
      "video_id": "7543693751290481942",
      "text": "This Claude Code trick saved me hours #claudecode #ai #coding",
      "url": "https://www.tiktok.com/@codemaster/video/7543693751290481942",
      "author_name": "codemaster",
      "date": "2026-02-28",
      "engagement": {
        "views": 2100000,
        "likes": 45000,
        "comments": 1200,
        "shares": 8400
      },
      "hashtags": ["claudecode", "ai", "coding"],
      "duration": 45,
      "relevance": 0.85,
      "why_relevant": "TikTok: This Claude Code trick saved me hours #claude",
      "caption_snippet": "So I found this insane trick with Claude Code where you can use slash commands to automate everything"
    },
    {
      "video_id": "7543100200112345678",
      "text": "AI coding tools comparison 2026 - Claude vs Copilot vs Cursor #ai #devtools",
      "url": "https://www.tiktok.com/@techreviewer/video/7543100200112345678",
      "author_name": "techreviewer",
      "date": "2026-02-25",
      "engagement": {
        "views": 850000,
        "likes": 22000,
        "comments": 890,
        "shares": 3200
      },
      "hashtags": ["ai", "devtools"],
      "duration": 60,
      "relevance": 0.7,
      "why_relevant": "TikTok: AI coding tools comparison 2026 - Claude vs Copi",
      "caption_snippet": ""
    },
    {
      "video_id": "7543200300223456789",
      "text": "You need to try Claude Code RIGHT NOW #programming #tips",
      "url": "https://www.tiktok.com/@devtips/video/7543200300223456789",
      "author_name": "devtips",
      "date": "2026-03-01",
      "engagement": {
        "views": 500000,
        "likes": 15000,
        "comments": 450,
        "shares": 2100
      },
      "hashtags": ["programming", "tips"],
      "duration": 30,
      "relevance": 0.6,
      "why_relevant": "TikTok: You need to try Claude Code RIGHT NOW #programm",
      "caption_snippet": "Let me show you why Claude Code is the best AI coding tool right now"
    }
  ]
 }
@@ -1,52 +1,44 @@
 The AI world reinvents itself every month. This skill keeps you current.
-`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago.
+`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
-## Three Headline Features
+## Three Headline Features in v2.9
-**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search.
+**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.
-**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
+**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw — not generic programming subs.
-**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
+**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with `💬` and upvote counts. Reddit's value is in the comments — now the skill surfaces them.
-Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies.
+Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** — no external CLI needed.
-## Real Results (verified Feb 15)
+## Beta Test Results (v2.9)
-| Topic | Reddit | X | YouTube | Web |
+| Topic | Time | Threads | Discovered Subreddits |
-|-------|--------|---|---------|-----|
+|-------|------|---------|----------------------|
-| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages |
+| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
-| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages |
+| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
-| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages |
+| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
-| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages |
+| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
-| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages |
+| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
 ## What's New
 ### Added
- Open-class skill with watchlist, briefing, and history modes
+- ScrapeCreators Reddit backend with keyword search and subreddit discovery
- YouTube search + transcript extraction via yt-dlp
+- Smart subreddit discovery with relevance-weighted scoring
- OpenAI Codex CLI compatibility
+- Utility subreddit blocklist (`UTILITY_SUBS`)
- Bundled Twitter/X search (vendored Bird GraphQL, MIT)
+- Top comment scoring (10% engagement weight) and prominent rendering
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro)
+- Comment excerpts increased to 400 chars, insights raised to 10
 - `--diagnose` flag for source status checking
 - `--store` flag for SQLite accumulation
 - Conversational first-run experience (NUX)
 ### Changed
- Two-phase search architecture (entity-aware drill-down)
+- `primaryEnv` → `SCRAPECREATORS_API_KEY` (one key for Reddit, TikTok, Instagram)
- Reddit JSON enrichment for real engagement metrics
+- Reddit engagement scoring: `0.55/0.40/0.05` → `0.50/0.35/0.05/0.10`
- Smarter query construction with auto-retry on 0 results
+- SKILL.md synthesis instructions emphasize quoting top comments
 - Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
 - `--days=N` configurable lookback (thanks @jonthebeef)
 ### Fixed
- YouTube/Reddit timeout resilience
+- Utility sub noise in subreddit discovery
- Reddit 429 rate limit fail-fast
+- Reddit no longer requires `OPENAI_API_KEY`
 - Eager import crash in Codex environments
 - X search returning 0 results on popular topics
 - Windows Unicode crash (thanks @JosephOIbrahim)
 ## New Contributors
@@ -70,4 +62,4 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
 git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
 ```
-30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
+30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.
@@ -38,11 +38,45 @@ _child_pids: set = set()
 _child_pids_lock = threading.Lock()
 TIMEOUT_PROFILES = {
-    "quick":   {"global": 90,  "future": 30, "reddit_future": 60,  "youtube_future": 60,  "hackernews_future": 30,  "polymarket_future": 15,  "http": 15, "enrich_per": 8,  "enrich_total": 30, "enrich_max_items": 10},
+    "quick":   {"global": 90,  "future": 30, "reddit_future": 60,  "youtube_future": 60,  "tiktok_future": 90,   "instagram_future": 90,   "hackernews_future": 30,  "polymarket_future": 15,  "http": 15, "enrich_per": 8,  "enrich_total": 30, "enrich_max_items": 10},
-    "default": {"global": 180, "future": 60, "reddit_future": 90,  "youtube_future": 90,  "hackernews_future": 60,  "polymarket_future": 30,  "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
+    "default": {"global": 180, "future": 60, "reddit_future": 90,  "youtube_future": 90,  "tiktok_future": 120,  "instagram_future": 120,  "hackernews_future": 60,  "polymarket_future": 30,  "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
-    "deep":    {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90,  "polymarket_future": 45,  "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
+    "deep":    {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "tiktok_future": 150,  "instagram_future": 150,  "hackernews_future": 90,  "polymarket_future": 45,  "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
 }
 # Valid source names for the --search flag
 VALID_SEARCH_SOURCES = {"reddit", "x", "hn", "youtube", "tiktok", "instagram", "polymarket", "web"}
 def parse_search_flag(search_str: str) -> set:
    """Parse and validate the --search flag value.
    Args:
        search_str: Comma-separated source names (e.g. "reddit,hn")
    Returns:
        Set of validated source names
    Raises:
        SystemExit: If invalid sources are specified
    """
    sources = set()
    for s in search_str.split(","):
        s = s.strip().lower()
        if not s:
            continue
        if s not in VALID_SEARCH_SOURCES:
            print(
                f"Error: Unknown search source '{s}'. "
                f"Valid: {', '.join(sorted(VALID_SEARCH_SOURCES))}",
                file=sys.stderr,
            )
            sys.exit(1)
        sources.add(s)
    if not sources:
        print("Error: --search requires at least one source.", file=sys.stderr)
        sys.exit(1)
    return sources
 def register_child_pid(pid: int):
    """Track a child process for cleanup."""
@@ -106,11 +140,14 @@ from lib import (
    models,
    normalize,
    openai_reddit,
    reddit,
    reddit_enrich,
    render,
    schema,
    score,
    ui,
    tiktok,
    instagram,
    websearch,
    xai_x,
    youtube_yt,
@@ -135,35 +172,69 @@ def _search_reddit(
    depth: str,
    mock: bool,
 ) -> tuple:
-    """Search Reddit via OpenAI (runs in thread).
+    """Search Reddit (runs in thread).
    Uses ScrapeCreators when SCRAPECREATORS_API_KEY is available (preferred).
    Falls back to OpenAI Responses API otherwise.
    Returns:
-        Tuple of (reddit_items, raw_openai, error)
+        Tuple of (reddit_items, raw_response, error, used_scrapecreators)
    """
-    raw_openai = None
+    raw_response = None
    reddit_error = None
    used_scrapecreators = False
    sc_token = config.get("SCRAPECREATORS_API_KEY")
    if mock:
-        raw_openai = load_fixture("openai_sample.json")
+        raw_response = load_fixture("openai_sample.json")
-    else:
+    elif sc_token:
        # === ScrapeCreators path (preferred) ===
        used_scrapecreators = True
        try:
-            raw_openai = openai_reddit.search_reddit(
+            sys.stderr.write("[Reddit] Using ScrapeCreators API\n")
            sys.stderr.flush()
            result = reddit.search_and_enrich(
                topic, from_date, to_date,
                depth=depth, token=sc_token,
            )
            reddit_items = result.get("items", [])
            if result.get("error"):
                reddit_error = result["error"]
            return reddit_items, result, reddit_error, used_scrapecreators
        except Exception as e:
            reddit_error = f"ScrapeCreators: {type(e).__name__}: {e}"
            sys.stderr.write(f"[Reddit] ScrapeCreators failed: {e}\n")
            sys.stderr.flush()
            # Fall through to OpenAI if we have that key
            if not config.get("OPENAI_API_KEY"):
                return [], {"error": str(e)}, reddit_error, used_scrapecreators
            used_scrapecreators = False
            sys.stderr.write("[Reddit] Falling back to OpenAI\n")
            sys.stderr.flush()
    # === OpenAI path (fallback) ===
    if not mock:
        try:
            raw_response = openai_reddit.search_reddit(
                config["OPENAI_API_KEY"],
                selected_models["openai"],
                topic,
                from_date,
                to_date,
                depth=depth,
                auth_source=config.get("OPENAI_AUTH_SOURCE", "api_key"),
                account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
            )
        except http.HTTPError as e:
-            raw_openai = {"error": str(e)}
+            raw_response = {"error": str(e)}
            reddit_error = f"API error: {e}"
        except Exception as e:
-            raw_openai = {"error": str(e)}
+            raw_response = {"error": str(e)}
            reddit_error = f"{type(e).__name__}: {e}"
    # Parse response
-    reddit_items = openai_reddit.parse_reddit_response(raw_openai or {})
+    reddit_items = openai_reddit.parse_reddit_response(raw_response or {})
    # Quick retry with simpler query if few results
    if len(reddit_items) < 5 and not mock and not reddit_error:
@@ -176,9 +247,10 @@ def _search_reddit(
                    core,
                    from_date, to_date,
                    depth=depth,
                    auth_source=config.get("OPENAI_AUTH_SOURCE", "api_key"),
                    account_id=config.get("OPENAI_CHATGPT_ACCOUNT_ID"),
                )
                retry_items = openai_reddit.parse_reddit_response(retry_raw)
                # Add items not already found (by URL)
                existing_urls = {item.get("url") for item in reddit_items}
                for item in retry_items:
                    if item.get("url") not in existing_urls:
@@ -205,7 +277,7 @@ def _search_reddit(
        except Exception:
            pass
-    return reddit_items, raw_openai, reddit_error
+    return reddit_items, raw_response, reddit_error, used_scrapecreators
 def _search_x(
@@ -305,6 +377,64 @@ def _search_youtube(
    return youtube_items, youtube_error
 def _search_tiktok(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str,
    token: str,
 ) -> tuple:
    """Search TikTok via ScrapeCreators (runs in thread).
    Returns:
        Tuple of (tiktok_items, tiktok_error)
    """
    tiktok_error = None
    try:
        response = tiktok.search_and_enrich(
            topic, from_date, to_date, depth=depth, token=token,
        )
    except Exception as e:
        return [], f"{type(e).__name__}: {e}"
    tiktok_items = tiktok.parse_tiktok_response(response)
    if response.get("error"):
        tiktok_error = response["error"]
    return tiktok_items, tiktok_error
 def _search_instagram(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str,
    token: str,
 ) -> tuple:
    """Search Instagram via ScrapeCreators (runs in thread).
    Returns:
        Tuple of (instagram_items, instagram_error)
    """
    instagram_error = None
    try:
        response = instagram.search_and_enrich(
            topic, from_date, to_date, depth=depth, token=token,
        )
    except Exception as e:
        return [], f"{type(e).__name__}: {e}"
    instagram_items = instagram.parse_instagram_response(response)
    if response.get("error"):
        instagram_error = response["error"]
    return instagram_items, instagram_error
 def _search_hackernews(
    topic: str,
    from_date: str,
@@ -595,15 +725,22 @@ def run_research(
    progress: ui.ProgressDisplay = None,
    x_source: str = "xai",
    run_youtube: bool = False,
    run_tiktok: bool = False,
    run_instagram: bool = False,
    timeouts: dict = None,
    resolved_handle: str = None,
    do_hackernews: bool = True,
    do_polymarket: bool = True,
    no_native_web: bool = False,
 ) -> tuple:
    """Run the research pipeline.
    Returns:
-        Tuple of (reddit_items, x_items, youtube_items, web_items, web_needed,
+        Tuple of (reddit_items, x_items, youtube_items, tiktok_items, instagram_items,
                  hackernews_items, polymarket_items, web_items, web_needed,
                  raw_openai, raw_xai, raw_reddit_enriched,
-                  reddit_error, x_error, youtube_error, web_error)
+                  reddit_error, x_error, youtube_error, tiktok_error, instagram_error,
                  hackernews_error, polymarket_error, web_error)
    Note: web_needed is True when web search should be performed by the assistant
    (i.e., no native web search API keys are configured). When native web search
@@ -616,6 +753,8 @@ def run_research(
    reddit_items = []
    x_items = []
    youtube_items = []
    tiktok_items = []
    instagram_items = []
    hackernews_items = []
    polymarket_items = []
    web_items = []
@@ -625,13 +764,15 @@ def run_research(
    reddit_error = None
    x_error = None
    youtube_error = None
    tiktok_error = None
    instagram_error = None
    hackernews_error = None
    polymarket_error = None
    web_error = None
    # Determine web search mode
    do_web = sources in ("all", "web", "reddit-web", "x-web")
-    web_backend = env.get_web_search_source(config) if do_web else None
+    web_backend = env.get_web_search_source(config) if (do_web and not no_native_web) else None
    web_needed = do_web and not web_backend
    # Web-only mode
@@ -655,7 +796,7 @@ def run_research(
            if progress:
                progress.start_web_only()
                progress.end_web_only()
-        # Still run YouTube in web-only mode if yt-dlp is available
+        # Still run YouTube/TikTok/Instagram in web-only mode if available
        if run_youtube:
            if progress:
                progress.start_youtube()
@@ -669,22 +810,51 @@ def run_research(
                    progress.show_error(f"YouTube error: {e}")
            if progress:
                progress.end_youtube(len(youtube_items))
-        return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error
+        if run_tiktok:
            if progress:
                progress.start_tiktok()
            try:
                tiktok_items, tiktok_error = _search_tiktok(topic, from_date, to_date, depth, env.get_tiktok_token(config))
                if tiktok_error and progress:
                    progress.show_error(f"TikTok error: {tiktok_error}")
            except Exception as e:
                tiktok_error = f"{type(e).__name__}: {e}"
                if progress:
                    progress.show_error(f"TikTok error: {e}")
            if progress:
                progress.end_tiktok(len(tiktok_items))
        if run_instagram:
            if progress:
                progress.start_instagram()
            try:
                ig_timeout = timeouts.get("instagram_future", future_timeout)
                instagram_items, instagram_error = _search_instagram(topic, from_date, to_date, depth, env.get_instagram_token(config))
                if instagram_error and progress:
                    progress.show_error(f"Instagram error: {instagram_error}")
            except Exception as e:
                instagram_error = f"{type(e).__name__}: {e}"
                if progress:
                    progress.show_error(f"Instagram error: {e}")
            if progress:
                progress.end_instagram(len(instagram_items))
        return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error
    # Determine which searches to run
    do_reddit = sources in ("both", "reddit", "all", "reddit-web")
    do_x = sources in ("both", "x", "all", "x-web")
-    do_hackernews = True  # HN is always available (no API key)
+    # do_hackernews / do_polymarket are always True by default, but can be
-    do_polymarket = True  # Polymarket is always available (no API key)
+    # restricted via the --search flag to run a focused source subset.
    # Run Reddit, X, YouTube, HN, Polymarket, and Web searches in parallel
    reddit_future = None
    x_future = None
    youtube_future = None
    tiktok_future = None
    instagram_future = None
    hackernews_future = None
    polymarket_future = None
    web_future = None
-    max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)
+    max_workers = 2 + (1 if run_youtube else 0) + (1 if run_tiktok else 0) + (1 if run_instagram else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Submit searches
@@ -711,6 +881,22 @@ def run_research(
                _search_youtube, topic, from_date, to_date, depth
            )
        if run_tiktok:
            if progress:
                progress.start_tiktok()
            tiktok_future = executor.submit(
                _search_tiktok, topic, from_date, to_date, depth,
                env.get_tiktok_token(config),
            )
        if run_instagram:
            if progress:
                progress.start_instagram()
            instagram_future = executor.submit(
                _search_instagram, topic, from_date, to_date, depth,
                env.get_instagram_token(config),
            )
        if do_hackernews:
            if progress:
                progress.start_hackernews()
@@ -733,10 +919,11 @@ def run_research(
            )
        # Collect results (with timeouts to prevent indefinite blocking)
        reddit_used_sc = False  # Track if ScrapeCreators was used for Reddit
        if reddit_future:
            reddit_timeout = timeouts.get("reddit_future", future_timeout)
            try:
-                reddit_items, raw_openai, reddit_error = reddit_future.result(timeout=reddit_timeout)
+                reddit_items, raw_openai, reddit_error, reddit_used_sc = reddit_future.result(timeout=reddit_timeout)
                if reddit_error and progress:
                    progress.show_error(f"Reddit error: {reddit_error}")
            except TimeoutError:
@@ -783,6 +970,40 @@ def run_research(
            if progress:
                progress.end_youtube(len(youtube_items))
        if tiktok_future:
            tk_timeout = timeouts.get("tiktok_future", future_timeout)
            try:
                tiktok_items, tiktok_error = tiktok_future.result(timeout=tk_timeout)
                if tiktok_error and progress:
                    progress.show_error(f"TikTok error: {tiktok_error}")
            except TimeoutError:
                tiktok_error = f"TikTok search timed out after {tk_timeout}s"
                if progress:
                    progress.show_error(tiktok_error)
            except Exception as e:
                tiktok_error = f"{type(e).__name__}: {e}"
                if progress:
                    progress.show_error(f"TikTok error: {e}")
            if progress:
                progress.end_tiktok(len(tiktok_items))
        if instagram_future:
            ig_timeout = timeouts.get("instagram_future", future_timeout)
            try:
                instagram_items, instagram_error = instagram_future.result(timeout=ig_timeout)
                if instagram_error and progress:
                    progress.show_error(f"Instagram error: {instagram_error}")
            except TimeoutError:
                instagram_error = f"Instagram search timed out after {ig_timeout}s"
                if progress:
                    progress.show_error(instagram_error)
            except Exception as e:
                instagram_error = f"{type(e).__name__}: {e}"
                if progress:
                    progress.show_error(f"Instagram error: {e}")
            if progress:
                progress.end_instagram(len(instagram_items))
        if hackernews_future:
            hn_timeout = timeouts.get("hackernews_future", future_timeout)
            try:
@@ -834,11 +1055,19 @@ def run_research(
            sys.stderr.flush()
    # Enrich Reddit items with real data (parallel, capped)
    # Skip enrichment if ScrapeCreators already provided comments + engagement
    enrich_max = timeouts["enrich_max_items"]
    enrich_total_timeout = timeouts["enrich_total"]
    items_to_enrich = reddit_items[:enrich_max]
    rate_limited = False  # Set True if Reddit returns 429 during enrichment
    if reddit_used_sc and items_to_enrich:
        # ScrapeCreators already enriched items with comments — just copy to raw list
        sys.stderr.write(f"[Reddit] Skipping old enrichment — ScrapeCreators already provided comments\n")
        sys.stderr.flush()
        raw_reddit_enriched = list(reddit_items[:enrich_max])
        items_to_enrich = []  # Skip the enrichment block below
    if items_to_enrich:
        if progress:
            progress.start_reddit_enrich(1, len(items_to_enrich))
@@ -913,11 +1142,12 @@ def run_research(
    # Phase 2: Supplemental search based on entities from Phase 1
    # Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting
    # Also skip Reddit supplemental when ScrapeCreators was used (subreddit drilling already done)
    if depth != "quick" and not mock and (reddit_items or x_items):
        sup_reddit, sup_x = _run_supplemental(
            topic, reddit_items, x_items,
            from_date, to_date, depth, x_source, progress,
-            skip_reddit=rate_limited,
+            skip_reddit=(rate_limited or reddit_used_sc),
            resolved_handle=resolved_handle,
        )
        if sup_reddit:
@@ -925,7 +1155,7 @@ def run_research(
        if sup_x:
            x_items.extend(sup_x)
-    return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error
+    return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error
 def main():
@@ -937,7 +1167,7 @@ def main():
    parser = argparse.ArgumentParser(
        description="Research a topic from the last N days on Reddit + X"
    )
-    parser.add_argument("topic", nargs="?", help="Topic to research")
+    parser.add_argument("topic", nargs="*", help="Topic to research")
    parser.add_argument("--mock", action="store_true", help="Use fixtures")
    parser.add_argument(
        "--emit",
@@ -1003,8 +1233,26 @@ def main():
        metavar="HANDLE",
        help="Resolved X handle for topic entity (without @). Searched unfiltered in Phase 2.",
    )
    parser.add_argument(
        "--search",
        type=str,
        default=None,
        metavar="SOURCES",
        help=(
            "Comma-separated list of sources to run. "
            f"Valid: {', '.join(sorted(VALID_SEARCH_SOURCES))}. "
            "Example: --search reddit,hn  (default: all configured sources)"
        ),
    )
    parser.add_argument(
        "--no-native-web",
        action="store_true",
        default=False,
        help="Skip native web search backends (Parallel/Brave/OpenRouter). Use when the assistant has its own WebSearch tool.",
    )
    args = parser.parse_args()
    args.topic = " ".join(args.topic) if args.topic else None
    # Enable debug logging if requested
    if args.debug:
@@ -1032,6 +1280,9 @@ def main():
    # Load config
    config = env.get_config()
    # Inject .env credentials into Bird module before auth check
    bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
    # Auto-detect Bird (no prompts - just use it if available)
    x_source_status = env.get_x_source_status(config)
    x_source = x_source_status["source"]  # 'bird', 'xai', or None
@@ -1039,6 +1290,12 @@ def main():
    # Auto-detect yt-dlp for YouTube search
    has_ytdlp = env.is_ytdlp_available()
    # Auto-detect ScrapeCreators/Apify for TikTok
    has_tiktok = env.is_tiktok_available(config)
    # Auto-detect ScrapeCreators for Instagram
    has_instagram = env.is_instagram_available(config)
    # --diagnose: show source availability and exit
    if args.diagnose:
        web_source = env.get_web_search_source(config)
@@ -1050,6 +1307,8 @@ def main():
            "bird_authenticated": x_source_status["bird_authenticated"],
            "bird_username": x_source_status.get("bird_username"),
            "youtube": has_ytdlp,
            "tiktok": has_tiktok,
            "instagram": has_instagram,
            "hackernews": True,
            "polymarket": True,
            "web_search_backend": web_source,
@@ -1079,9 +1338,10 @@ def main():
        "bird_authenticated": x_source_status["bird_authenticated"],
        "bird_username": x_source_status.get("bird_username"),
        "youtube": has_ytdlp,
        "tiktok": has_tiktok,
        "hackernews": True,
        "polymarket": True,
-        "web_search_backend": web_source,
+        "web_search_backend": "deferred to assistant" if args.no_native_web else web_source,
    }
    ui.show_diagnostic_banner(diag)
@@ -1092,8 +1352,10 @@ def main():
    if x_source == 'bird':
        if available == 'reddit':
            available = 'both'  # Now have both Reddit + X (via Bird)
        elif available == 'reddit-web':
            available = 'all'  # Reddit + X (via Bird) + Web
        elif available == 'web':
-            available = 'x'  # Now have X via Bird
+            available = 'x-web'  # X via Bird + Web
    # Mock mode can work without keys
    if args.mock:
@@ -1157,8 +1419,35 @@ def main():
    else:
        mode = sources
    # Apply --search flag: restrict sources to the specified subset
    search_do_hackernews = True
    search_do_polymarket = True
    search_run_youtube = has_ytdlp
    search_run_tiktok = has_tiktok
    search_run_instagram = has_instagram
    if args.search:
        search_sources = parse_search_flag(args.search)
        has_reddit = "reddit" in search_sources
        has_x = "x" in search_sources
        search_do_hackernews = "hn" in search_sources
        search_do_polymarket = "polymarket" in search_sources
        search_run_youtube = "youtube" in search_sources and has_ytdlp
        search_run_tiktok = "tiktok" in search_sources and has_tiktok
        search_run_instagram = "instagram" in search_sources and has_instagram
        include_search_web = "web" in search_sources
        # Map to existing sources string
        if has_reddit and has_x:
            sources = "both" + ("-web" if include_search_web else "")
            sources = "all" if include_search_web else "both"
        elif has_reddit:
            sources = "reddit-web" if include_search_web else "reddit"
        elif has_x:
            sources = "x-web" if include_search_web else "x"
        else:
            sources = "web"  # hn/polymarket only; no Reddit/X
    # Run research
-    reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error = run_research(
+    reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error = run_research(
        args.topic,
        sources,
        config,
@@ -1169,9 +1458,14 @@ def main():
        args.mock,
        progress,
        x_source=x_source or "xai",
-        run_youtube=has_ytdlp,
+        run_youtube=search_run_youtube,
        run_tiktok=search_run_tiktok,
        run_instagram=search_run_instagram,
        timeouts=timeouts,
        resolved_handle=args.x_handle,
        do_hackernews=search_do_hackernews,
        do_polymarket=search_do_polymarket,
        no_native_web=args.no_native_web,
    )
    # Processing phase
@@ -1181,6 +1475,8 @@ def main():
    normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
    normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
    normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else []
    normalized_tiktok = normalize.normalize_tiktok_items(tiktok_items, from_date, to_date) if tiktok_items else []
    normalized_ig = normalize.normalize_instagram_items(instagram_items, from_date, to_date) if instagram_items else []
    normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
    normalized_pm = normalize.normalize_polymarket_items(polymarket_items, from_date, to_date) if polymarket_items else []
    normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
@@ -1193,6 +1489,10 @@ def main():
    # that prefers recent videos but keeps older ones for evergreen topics.
    # YouTube content has a longer shelf life than tweets/posts.
    filtered_youtube = normalized_youtube
    # TikTok: hard date filter (tiktok.py already pre-filters, but safety net)
    filtered_tiktok = normalize.filter_by_date_range(normalized_tiktok, from_date, to_date) if normalized_tiktok else []
    # Instagram: hard date filter (instagram.py already pre-filters, but safety net)
    filtered_ig = normalize.filter_by_date_range(normalized_ig, from_date, to_date) if normalized_ig else []
    filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
    # Polymarket: skip hard date filter - markets are active/traded, updatedAt is fine
    filtered_pm = normalized_pm
@@ -1202,6 +1502,8 @@ def main():
    scored_reddit = score.score_reddit_items(filtered_reddit)
    scored_x = score.score_x_items(filtered_x)
    scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else []
    scored_tiktok = score.score_tiktok_items(filtered_tiktok) if filtered_tiktok else []
    scored_ig = score.score_instagram_items(filtered_ig) if filtered_ig else []
    scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
    scored_pm = score.score_polymarket_items(filtered_pm) if filtered_pm else []
    scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
@@ -1210,6 +1512,8 @@ def main():
    sorted_reddit = score.sort_items(scored_reddit)
    sorted_x = score.sort_items(scored_x)
    sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
    sorted_tiktok = score.sort_items(scored_tiktok) if scored_tiktok else []
    sorted_ig = score.sort_items(scored_ig) if scored_ig else []
    sorted_hn = score.sort_items(scored_hn) if scored_hn else []
    sorted_pm = score.sort_items(scored_pm) if scored_pm else []
    sorted_web = score.sort_items(scored_web) if scored_web else []
@@ -1218,6 +1522,8 @@ def main():
    deduped_reddit = dedupe.dedupe_reddit(sorted_reddit)
    deduped_x = dedupe.dedupe_x(sorted_x)
    deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else []
    deduped_tiktok = dedupe.dedupe_tiktok(sorted_tiktok) if sorted_tiktok else []
    deduped_ig = dedupe.dedupe_instagram(sorted_ig) if sorted_ig else []
    deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
    deduped_pm = dedupe.dedupe_polymarket(sorted_pm) if sorted_pm else []
    deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
@@ -1231,7 +1537,7 @@ def main():
    # Cross-source linking: annotate items that discuss the same story
    dedupe.cross_source_link(
-        deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_pm, deduped_web,
+        deduped_reddit, deduped_x, deduped_youtube, deduped_tiktok, deduped_ig, deduped_hn, deduped_pm, deduped_web,
    )
    progress.end_processing()
@@ -1248,12 +1554,16 @@ def main():
    report.reddit = deduped_reddit
    report.x = deduped_x
    report.youtube = deduped_youtube
    report.tiktok = deduped_tiktok
    report.instagram = deduped_ig
    report.hackernews = deduped_hn
    report.polymarket = deduped_pm
    report.web = deduped_web
    report.reddit_error = reddit_error
    report.x_error = x_error
    report.youtube_error = youtube_error
    report.tiktok_error = tiktok_error
    report.instagram_error = instagram_error
    report.hackernews_error = hackernews_error
    report.polymarket_error = polymarket_error
    report.web_error = web_error
@@ -1269,7 +1579,7 @@ def main():
    if sources == "web":
        progress.show_web_only_complete()
    else:
-        progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm))
+        progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm), len(deduped_tiktok), len(deduped_ig))
    # Build source info for status footer
    source_info = {}
@@ -1282,6 +1592,12 @@ def main():
            source_info["x_skip_reason"] = "No Bird CLI or XAI_API_KEY (Node.js 22+ needed for Bird)"
    if not has_ytdlp:
        source_info["youtube_skip_reason"] = "yt-dlp not installed — fix: brew install yt-dlp"
    elif has_ytdlp and not report.youtube:
        source_info["youtube_skip_reason"] = "0 results (query may be too specific)"
    if not has_tiktok:
        source_info["tiktok_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
    if not has_instagram:
        source_info["instagram_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
    if not web_source:
        source_info["web_skip_reason"] = "assistant will use WebSearch (add BRAVE_API_KEY for native search)"
@@ -1347,6 +1663,16 @@ def main():
                "engagement_score": item.engagement.volume if item.engagement and item.engagement.volume else 0,
                "relevance_score": item.relevance,
            })
        for item in deduped_ig:
            findings.append({
                "source": "instagram",
                "url": item.url,
                "title": item.text[:100],
                "author": item.author_name,
                "content": item.caption_snippet[:500] if item.caption_snippet else item.text,
                "engagement_score": item.engagement.views if item.engagement and item.engagement.views else 0,
                "relevance_score": item.relevance,
            })
        for item in deduped_web:
            findings.append({
                "source": "web",
@@ -24,6 +24,24 @@ DEPTH_CONFIG = {
    "deep": 60,
 }
 # Module-level credentials injected from .env config
 _credentials: Dict[str, str] = {}
 def set_credentials(auth_token: Optional[str], ct0: Optional[str]):
    """Inject AUTH_TOKEN/CT0 from .env config so Node subprocesses can use them."""
    if auth_token:
        _credentials['AUTH_TOKEN'] = auth_token
    if ct0:
        _credentials['CT0'] = ct0
 def _subprocess_env() -> Dict[str, str]:
    """Build env dict for Node subprocesses, merging injected credentials."""
    env = os.environ.copy()
    env.update(_credentials)
    return env
 def _log(msg: str):
    """Log to stderr."""
@@ -71,9 +89,11 @@ def _extract_core_subject(topic: str) -> str:
        # Research/meta descriptors
        'best', 'top', 'good', 'great', 'awesome', 'killer',
        'latest', 'new', 'news', 'update', 'updates',
        'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
        'practices', 'features', 'guide', 'tutorial',
        'recommendations', 'advice', 'review', 'reviews',
        'usecases', 'examples', 'comparison', 'versus', 'vs',
        'plugin', 'plugins', 'skill', 'skills', 'tool', 'tools',
        # Prompting meta words
        'prompt', 'prompts', 'prompting', 'techniques', 'tips',
        'tricks', 'methods', 'strategies', 'approaches',
@@ -112,6 +132,7 @@ def is_bird_authenticated() -> Optional[str]:
            capture_output=True,
            text=True,
            timeout=15,
            env=_subprocess_env(),
        )
        if result.returncode == 0 and result.stdout.strip():
            return result.stdout.strip().split('\n')[0]
@@ -187,6 +208,7 @@ def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
            stderr=subprocess.PIPE,
            text=True,
            preexec_fn=preexec,
            env=_subprocess_env(),
        )
        # Register for cleanup tracking (if available)
@@ -266,6 +288,21 @@ def search_x(
        _log(f"0 results for '{core_topic}', retrying with '{shorter}'")
        query = f"{shorter} since:{from_date}"
        response = _run_bird_search(query, count, timeout)
        items = parse_bird_response(response)
    # Last-chance retry: use strongest remaining token (often the product name)
    if not items and core_words:
        low_signal = {
            'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
            'best', 'top', 'latest', 'new', 'plugin', 'plugins',
            'skill', 'skills', 'tool', 'tools',
        }
        candidates = [w for w in core_words if w not in low_signal]
        if candidates:
            strongest = max(candidates, key=len)
            _log(f"0 results for '{core_topic}', retrying with strongest token '{strongest}'")
            query = f"{strongest} since:{from_date}"
            response = _run_bird_search(query, count, timeout)
    return response
@@ -45,8 +45,8 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
    return intersection / union if union > 0 else 0.0
-AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
+AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem, schema.TikTokItem,
-                schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem]
+                schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem]
 def get_item_text(item: AnyItem) -> str:
@@ -57,6 +57,10 @@ def get_item_text(item: AnyItem) -> str:
        return item.title
    elif isinstance(item, schema.YouTubeItem):
        return f"{item.title} {item.channel_name}"
    elif isinstance(item, schema.TikTokItem):
        return f"{item.text} {item.author_name}"
    elif isinstance(item, schema.InstagramItem):
        return f"{item.text} {item.author_name}"
    elif isinstance(item, schema.PolymarketItem):
        return f"{item.title} {item.question}"
    elif isinstance(item, schema.WebSearchItem):
@@ -74,6 +78,10 @@ def _get_cross_source_text(item: AnyItem) -> str:
    """
    if isinstance(item, schema.XItem):
        return item.text[:100]
    if isinstance(item, schema.TikTokItem):
        return item.text[:100]
    if isinstance(item, schema.InstagramItem):
        return item.text[:100]
    if isinstance(item, schema.HackerNewsItem):
        title = item.title
        if title.startswith("Show HN:"):
@@ -194,6 +202,22 @@ def dedupe_youtube(
    return dedupe_items(items, threshold)
 def dedupe_tiktok(
    items: List[schema.TikTokItem],
    threshold: float = 0.7,
 ) -> List[schema.TikTokItem]:
    """Dedupe TikTok items."""
    return dedupe_items(items, threshold)
 def dedupe_instagram(
    items: List[schema.InstagramItem],
    threshold: float = 0.7,
 ) -> List[schema.InstagramItem]:
    """Dedupe Instagram items."""
    return dedupe_items(items, threshold)
 def dedupe_hackernews(
    items: List[schema.HackerNewsItem],
    threshold: float = 0.7,
@@ -1,9 +1,12 @@
 """Environment and API key management for last30days skill."""
 import base64
 import json
 import os
 import time
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Optional, Dict, Any
+from typing import Optional, Dict, Any, Literal
 # Allow override via environment variable for testing
 # Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
@@ -20,6 +23,29 @@ else:
    CONFIG_DIR = Path.home() / ".config" / "last30days"
    CONFIG_FILE = CONFIG_DIR / ".env"
 CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))
 AuthSource = Literal["api_key", "codex", "none"]
 AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]
 AUTH_SOURCE_API_KEY: AuthSource = "api_key"
 AUTH_SOURCE_CODEX: AuthSource = "codex"
 AUTH_SOURCE_NONE: AuthSource = "none"
 AUTH_STATUS_OK: AuthStatus = "ok"
 AUTH_STATUS_MISSING: AuthStatus = "missing"
 AUTH_STATUS_EXPIRED: AuthStatus = "expired"
 AUTH_STATUS_MISSING_ACCOUNT_ID: AuthStatus = "missing_account_id"
@dataclass(frozen=True)
 class OpenAIAuth:
    token: Optional[str]
    source: AuthSource
    status: AuthStatus
    account_id: Optional[str]
    codex_auth_file: str
 def load_env_file(path: Path) -> Dict[str, str]:
    """Load environment variables from a file."""
@@ -44,14 +70,131 @@ def load_env_file(path: Path) -> Dict[str, str]:
    return env
 def _decode_jwt_payload(token: str) -> Optional[Dict[str, Any]]:
    """Decode JWT payload without verification."""
    try:
        parts = token.split(".")
        if len(parts) < 2:
            return None
        payload_b64 = parts[1]
        pad = "=" * (-len(payload_b64) % 4)
        decoded = base64.urlsafe_b64decode(payload_b64 + pad)
        return json.loads(decoded.decode("utf-8"))
    except Exception:
        return None
 def _token_expired(token: str, leeway_seconds: int = 60) -> bool:
    """Check if JWT token is expired."""
    payload = _decode_jwt_payload(token)
    if not payload:
        return False
    exp = payload.get("exp")
    if not exp:
        return False
    return exp <= (time.time() + leeway_seconds)
 def extract_chatgpt_account_id(access_token: str) -> Optional[str]:
    """Extract chatgpt_account_id from JWT token."""
    payload = _decode_jwt_payload(access_token)
    if not payload:
        return None
    auth_claim = payload.get("https://api.openai.com/auth", {})
    if isinstance(auth_claim, dict):
        return auth_claim.get("chatgpt_account_id")
    return None
 def load_codex_auth(path: Path = CODEX_AUTH_FILE) -> Dict[str, Any]:
    """Load Codex auth JSON."""
    if not path.exists():
        return {}
    try:
        with open(path, "r") as f:
            return json.load(f)
    except Exception:
        return {}
 def get_codex_access_token() -> tuple[Optional[str], str]:
    """Get Codex access token from auth.json.
    Returns:
        (token, status) where status is 'ok', 'missing', or 'expired'
    """
    auth = load_codex_auth()
    token = None
    if isinstance(auth, dict):
        tokens = auth.get("tokens") or {}
        if isinstance(tokens, dict):
            token = tokens.get("access_token")
        if not token:
            token = auth.get("access_token")
    if not token:
        return None, AUTH_STATUS_MISSING
    if _token_expired(token):
        return None, AUTH_STATUS_EXPIRED
    return token, AUTH_STATUS_OK
 def get_openai_auth(file_env: Dict[str, str]) -> OpenAIAuth:
    """Resolve OpenAI auth from API key or Codex login."""
    api_key = os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY')
    if api_key:
        return OpenAIAuth(
            token=api_key,
            source=AUTH_SOURCE_API_KEY,
            status=AUTH_STATUS_OK,
            account_id=None,
            codex_auth_file=str(CODEX_AUTH_FILE),
        )
    codex_token, codex_status = get_codex_access_token()
    if codex_token:
        account_id = extract_chatgpt_account_id(codex_token)
        if account_id:
            return OpenAIAuth(
                token=codex_token,
                source=AUTH_SOURCE_CODEX,
                status=AUTH_STATUS_OK,
                account_id=account_id,
                codex_auth_file=str(CODEX_AUTH_FILE),
            )
        return OpenAIAuth(
            token=None,
            source=AUTH_SOURCE_CODEX,
            status=AUTH_STATUS_MISSING_ACCOUNT_ID,
            account_id=None,
            codex_auth_file=str(CODEX_AUTH_FILE),
        )
    return OpenAIAuth(
        token=None,
        source=AUTH_SOURCE_NONE,
        status=codex_status,
        account_id=None,
        codex_auth_file=str(CODEX_AUTH_FILE),
    )
 def get_config() -> Dict[str, Any]:
    """Load configuration from ~/.config/last30days/.env and environment."""
    # Load from config file first (if configured)
    file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
-    # Build config: process.env > .env file
+    openai_auth = get_openai_auth(file_env)
    # Build config: Codex/OpenAI auth + process.env > .env file
    config = {
        'OPENAI_API_KEY': openai_auth.token,
        'OPENAI_AUTH_SOURCE': openai_auth.source,
        'OPENAI_AUTH_STATUS': openai_auth.status,
        'OPENAI_CHATGPT_ACCOUNT_ID': openai_auth.account_id,
        'CODEX_AUTH_FILE': openai_auth.codex_auth_file,
    }
    keys = [
        ('OPENAI_API_KEY', None),
        ('XAI_API_KEY', None),
        ('OPENROUTER_API_KEY', None),
        ('PARALLEL_API_KEY', None),
@@ -60,9 +203,12 @@ def get_config() -> Dict[str, Any]:
        ('OPENAI_MODEL_PIN', None),
        ('XAI_MODEL_POLICY', 'latest'),
        ('XAI_MODEL_PIN', None),
        ('SCRAPECREATORS_API_KEY', None),
        ('APIFY_API_TOKEN', None),
        ('AUTH_TOKEN', None),
        ('CT0', None),
    ]
    config = {}
    for key, default in keys:
        config[key] = os.environ.get(key) or file_env.get(key, default)
@@ -74,18 +220,42 @@ def config_exists() -> bool:
    return CONFIG_FILE.exists()
 def is_reddit_available(config: Dict[str, Any]) -> bool:
    """Check if Reddit search is available.
    Reddit can use either ScrapeCreators (preferred) or OpenAI.
    """
    has_sc = bool(config.get('SCRAPECREATORS_API_KEY'))
    has_openai = bool(config.get('OPENAI_API_KEY')) and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK
    return has_sc or has_openai
 def get_reddit_source(config: Dict[str, Any]) -> Optional[str]:
    """Determine which Reddit backend to use.
    Priority: ScrapeCreators (cheaper, faster) > OpenAI (legacy)
    Returns: 'scrapecreators', 'openai', or None
    """
    if config.get('SCRAPECREATORS_API_KEY'):
        return 'scrapecreators'
    if config.get('OPENAI_API_KEY') and config.get('OPENAI_AUTH_STATUS') == AUTH_STATUS_OK:
        return 'openai'
    return None
 def get_available_sources(config: Dict[str, Any]) -> str:
    """Determine which sources are available based on API keys.
    Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
    """
-    has_openai = bool(config.get('OPENAI_API_KEY'))
+    has_reddit = is_reddit_available(config)
    has_xai = bool(config.get('XAI_API_KEY'))
    has_web = has_web_search_keys(config)
-    if has_openai and has_xai:
+    if has_reddit and has_xai:
        return 'all' if has_web else 'both'
-    elif has_openai:
+    elif has_reddit:
        return 'reddit-web' if has_web else 'reddit'
    elif has_xai:
        return 'x-web' if has_web else 'x'
@@ -117,11 +287,11 @@ def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
 def get_missing_keys(config: Dict[str, Any]) -> str:
-    """Determine which sources are missing (accounting for Bird).
+    """Determine which sources are missing (accounting for Bird and ScrapeCreators).
    Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
    """
-    has_openai = bool(config.get('OPENAI_API_KEY'))
+    has_reddit = is_reddit_available(config)
    has_xai = bool(config.get('XAI_API_KEY'))
    has_web = has_web_search_keys(config)
@@ -131,14 +301,14 @@ def get_missing_keys(config: Dict[str, Any]) -> str:
    has_x = has_xai or has_bird
-    if has_openai and has_x and has_web:
+    if has_reddit and has_x and has_web:
        return 'none'
-    elif has_openai and has_x:
+    elif has_reddit and has_x:
        return 'web'  # Missing web search keys
-    elif has_openai:
+    elif has_reddit:
        return 'x'  # Missing X source (and possibly web)
    elif has_x:
-        return 'reddit'  # Missing OpenAI key (and possibly web)
+        return 'reddit'  # Missing Reddit source (and possibly web)
    else:
        return 'all'  # Missing everything
@@ -170,7 +340,7 @@ def validate_sources(requested: str, available: str, include_web: bool = False)
        elif requested == 'web':
            return 'web', None
        else:
-            return 'web', f"Only web search keys configured. Add OPENAI_API_KEY for Reddit, XAI_API_KEY for X."
+            return 'web', "Only web search keys configured. Add OPENAI_API_KEY (or run codex login) for Reddit, XAI_API_KEY for X."
    if requested == 'auto':
        # Add web to sources if include_web is set
@@ -262,6 +432,37 @@ def is_polymarket_available() -> bool:
    return True
 def is_tiktok_available(config: Dict[str, Any]) -> bool:
    """Check if TikTok source is available (ScrapeCreators or legacy Apify).
    Returns True if SCRAPECREATORS_API_KEY or APIFY_API_TOKEN is set.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN'))
 def get_tiktok_token(config: Dict[str, Any]) -> str:
    """Get TikTok API token, preferring ScrapeCreators over legacy Apify."""
    return config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN') or ''
 def is_instagram_available(config: Dict[str, Any]) -> bool:
    """Check if Instagram source is available (ScrapeCreators).
    Returns True if SCRAPECREATORS_API_KEY is set.
    Instagram uses the same key as TikTok.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def get_instagram_token(config: Dict[str, Any]) -> str:
    """Get Instagram API token (same ScrapeCreators key as TikTok)."""
    return config.get('SCRAPECREATORS_API_KEY') or ''
 # Backward compat alias
 is_apify_available = is_tiktok_available
 def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
    """Get detailed X source status for UI decisions.
@@ -38,6 +38,7 @@ def request(
    json_data: Optional[Dict[str, Any]] = None,
    timeout: int = DEFAULT_TIMEOUT,
    retries: int = MAX_RETRIES,
    raw: bool = False,
 ) -> Dict[str, Any]:
    """Make an HTTP request and return JSON response.
@@ -50,7 +51,7 @@ def request(
        retries: Number of retries on failure
    Returns:
-        Parsed JSON response
+        Parsed JSON response (or raw text if raw=True)
    Raises:
        HTTPError: On request failure
@@ -66,8 +67,6 @@ def request(
    req = urllib.request.Request(url, data=data, headers=headers, method=method)
    log(f"{method} {url}")
    if json_data:
        log(f"Payload keys: {list(json_data.keys())}")
    last_error = None
    for attempt in range(retries):
@@ -75,6 +74,8 @@ def request(
            with urllib.request.urlopen(req, timeout=timeout) as response:
                body = response.read().decode('utf-8')
                log(f"Response: {response.status} ({len(body)} bytes)")
                if raw:
                    return body
                return json.loads(body) if body else {}
        except urllib.error.HTTPError as e:
            body = None
@@ -84,7 +85,8 @@ def request(
                pass
            log(f"HTTP Error {e.code}: {e.reason}")
            if body:
-                log(f"Error body: {body[:500]}")
+                snippet = " ".join(body.split())
                log(f"Error body: {snippet[:200]}")
            last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)
            # Don't retry client errors (4xx) except rate limits
@@ -137,6 +139,11 @@ def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]]
    return request("POST", url, headers=headers, json_data=json_data, **kwargs)
 def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> str:
    """Make a POST request with JSON body and return raw text."""
    return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)
 def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
    """Fetch Reddit thread JSON.
@@ -0,0 +1,437 @@
 """Instagram Reels search via ScrapeCreators API for /last30days.
 Uses ScrapeCreators REST API to search Instagram Reels by keyword, extract
 engagement metrics (views, likes, comments), and fetch video transcripts.
 Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
 API docs: https://scrapecreators.com/docs
 """
 import re
 import sys
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional, Set
 try:
    import requests as _requests
 except ImportError:
    _requests = None
 SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
 # Depth configurations: how many results to fetch / captions to extract
 DEPTH_CONFIG = {
    "quick":   {"results_per_page": 10, "max_captions": 3},
    "default": {"results_per_page": 20, "max_captions": 5},
    "deep":    {"results_per_page": 40, "max_captions": 8},
 }
 # Max words to keep from each caption
 CAPTION_MAX_WORDS = 500
 # Stopwords for relevance computation (shared with tiktok.py pattern)
 STOPWORDS = frozenset({
    'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
    'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
    'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
    'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
    'all', 'just', 'get', 'has', 'have', 'was', 'will',
 })
 # Synonym groups for relevance scoring
 SYNONYMS = {
    'hip': {'rap', 'hiphop'},
    'hop': {'rap', 'hiphop'},
    'rap': {'hip', 'hop', 'hiphop'},
    'hiphop': {'rap', 'hip', 'hop'},
    'js': {'javascript'},
    'javascript': {'js'},
    'ts': {'typescript'},
    'typescript': {'ts'},
    'ai': {'artificial', 'intelligence'},
    'ml': {'machine', 'learning'},
    'react': {'reactjs'},
    'reactjs': {'react'},
 }
 def _tokenize(text: str) -> Set[str]:
    """Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
    words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
    tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
    expanded = set(tokens)
    for t in tokens:
        if t in SYNONYMS:
            expanded.update(SYNONYMS[t])
    return expanded
 def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> float:
    """Compute relevance as ratio of query tokens found in text + hashtags.
    Uses ratio overlap (intersection / query_length). Hashtags provide
    an Instagram-specific relevance boost. Floors at 0.1.
    """
    q_tokens = _tokenize(query)
    # Combine text and hashtags for matching
    combined = text
    if hashtags:
        combined = f"{text} {' '.join(hashtags)}"
    t_tokens = _tokenize(combined)
    # Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
    if hashtags:
        for tag in hashtags:
            tag_lower = tag.lower()
            for qt in q_tokens:
                if qt in tag_lower and qt != tag_lower:
                    t_tokens.add(qt)
    if not q_tokens:
        return 0.5  # Neutral fallback
    overlap = len(q_tokens & t_tokens)
    ratio = overlap / len(q_tokens)
    return max(0.1, min(1.0, ratio))
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for Instagram search.
    Strips meta/research words to keep only the core product/concept name.
    """
    text = topic.lower().strip()
    # Strip multi-word prefixes
    prefixes = [
        'what are the best', 'what is the best', 'what are the latest',
        'what are people saying about', 'what do people think about',
        'how do i use', 'how to use', 'how to',
        'what are', 'what is', 'tips for', 'best practices for',
    ]
    for p in prefixes:
        if text.startswith(p + ' '):
            text = text[len(p):].strip()
    # Strip individual noise words
    noise = {
        'best', 'top', 'good', 'great', 'awesome', 'killer',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features',
        'recommendations', 'advice',
        'prompt', 'prompts', 'prompting',
        'methods', 'strategies', 'approaches',
    }
    words = text.split()
    filtered = [w for w in words if w not in noise]
    result = ' '.join(filtered) if filtered else text
    return result.rstrip('?!.')
 def _log(msg: str):
    """Log to stderr (only in interactive terminals; spinner handles non-TTY)."""
    if sys.stderr.isatty():
        sys.stderr.write(f"[Instagram] {msg}\n")
        sys.stderr.flush()
 def _sc_headers(token: str) -> Dict[str, str]:
    """Build ScrapeCreators request headers."""
    return {
        "x-api-key": token,
        "Content-Type": "application/json",
    }
 def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    """Parse date from ScrapeCreators Instagram item to YYYY-MM-DD.
    Handles taken_at as ISO string (e.g. "2026-02-26T16:00:00.000Z")
    or unix timestamp.
    """
    ts = item.get("taken_at")
    if not ts:
        return None
    # Try ISO string first (ScrapeCreators reels/search returns this)
    if isinstance(ts, str):
        try:
            # Handle "2026-02-26T16:00:00.000Z" format
            dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
            return dt.strftime("%Y-%m-%d")
        except (ValueError, TypeError):
            pass
        # Try just the date portion
        if len(ts) >= 10:
            return ts[:10]
    # Fall back to unix timestamp
    try:
        dt = datetime.fromtimestamp(int(ts), tz=timezone.utc)
        return dt.strftime("%Y-%m-%d")
    except (ValueError, TypeError, OSError):
        pass
    return None
 def _extract_hashtags(caption_text: str) -> List[str]:
    """Extract hashtags from Instagram caption text."""
    if not caption_text:
        return []
    return re.findall(r'#(\w+)', caption_text)
 def search_instagram(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Search Instagram Reels via ScrapeCreators API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
    if not _requests:
        return {"items": [], "error": "requests library not installed"}
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
    _log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
    try:
        resp = _requests.get(
            f"{SCRAPECREATORS_BASE}/v1/instagram/reels/search",
            params={"query": core_topic},
            headers=_sc_headers(token),
            timeout=30,
        )
        resp.raise_for_status()
        data = resp.json()
    except Exception as e:
        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}
    # Items are in the 'reels' array (ScrapeCreators v1 response)
    raw_items = data.get("reels") or data.get("items") or data.get("data") or []
    # Limit to configured count
    raw_items = raw_items[:config["results_per_page"]]
    # Parse items
    items = []
    for raw in raw_items:
        if not isinstance(raw, dict):
            continue
        # Extract reel ID and shortcode
        reel_pk = str(raw.get("id", raw.get("pk", "")))
        shortcode = raw.get("shortcode", raw.get("code", ""))
        # Caption text — can be a string or dict depending on endpoint
        caption_obj = raw.get("caption", "")
        if isinstance(caption_obj, dict):
            text = caption_obj.get("text", "")
        elif isinstance(caption_obj, str):
            text = caption_obj
        else:
            text = raw.get("desc", raw.get("text", ""))
        # Engagement metrics
        play_count = raw.get("video_play_count") or raw.get("video_view_count") or raw.get("play_count") or 0
        like_count = raw.get("like_count") or 0
        comment_count = raw.get("comment_count") or 0
        # Author info — 'owner' in reels/search, 'user' in user/reels
        owner = raw.get("owner") or raw.get("user") or {}
        author_name = owner.get("username", "")
        # Duration
        duration = raw.get("video_duration")
        # Date
        date_str = _parse_date(raw)
        # Hashtags from caption text
        hashtags = _extract_hashtags(text)
        # Compute relevance with hashtag boost
        relevance = _compute_relevance(core_topic, text, hashtags)
        # Build URL — prefer API-provided url, fallback to shortcode
        url = raw.get("url", "")
        if not url and shortcode:
            url = f"https://www.instagram.com/reel/{shortcode}"
        items.append({
            "video_id": reel_pk,
            "text": text,
            "url": url,
            "author_name": author_name,
            "date": date_str,
            "engagement": {
                "views": play_count,
                "likes": like_count,
                "comments": comment_count,
            },
            "hashtags": hashtags,
            "duration": duration,
            "relevance": relevance,
            "why_relevant": f"Instagram: {text[:60]}" if text else f"Instagram: {core_topic}",
            "caption_snippet": "",  # populated by fetch_captions
        })
    # Hard date filter
    in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
    out_of_range = len(items) - len(in_range)
    if in_range:
        items = in_range
        if out_of_range:
            _log(f"Filtered {out_of_range} reels outside date range")
    else:
        _log(f"No reels within date range, keeping all {len(items)}")
    # Sort by views descending
    items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
    _log(f"Found {len(items)} Instagram reels")
    return {"items": items}
 def fetch_captions(
    video_items: List[Dict[str, Any]],
    token: str,
    depth: str = "default",
 ) -> Dict[str, str]:
    """Fetch transcripts for top N Instagram reels via ScrapeCreators.
    Strategy:
    1. Use the 'text' field (caption) as baseline
    2. For top N, call /v2/instagram/media/transcript for spoken-word captions
    Args:
        video_items: Items from search_instagram()
        token: ScrapeCreators API key
        depth: Depth level for caption limit
    Returns:
        Dict mapping video_id -> caption text (truncated to 500 words)
    """
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    max_captions = config["max_captions"]
    if not video_items or not token or not _requests:
        return {}
    top_items = video_items[:max_captions]
    _log(f"Enriching captions for {len(top_items)} reels")
    captions = {}
    # First pass: use text field as caption (always available, free)
    for item in top_items:
        vid = item["video_id"]
        text = item.get("text", "")
        if text:
            words = text.split()
            if len(words) > CAPTION_MAX_WORDS:
                text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
            captions[vid] = text
    # Second pass: try to get spoken-word transcripts (1 credit each)
    for item in top_items:
        vid = item["video_id"]
        url = item.get("url", "")
        if not url:
            continue
        try:
            resp = _requests.get(
                f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
                params={"url": url},
                headers=_sc_headers(token),
                timeout=15,
            )
            if resp.status_code == 200:
                data = resp.json()
                transcripts = data.get("transcripts") or []
                if transcripts and isinstance(transcripts, list):
                    # Combine all transcript segments
                    transcript_text = " ".join(
                        t.get("text", "") for t in transcripts
                        if isinstance(t, dict) and t.get("text")
                    )
                    if transcript_text:
                        words = transcript_text.split()
                        if len(words) > CAPTION_MAX_WORDS:
                            transcript_text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
                        captions[vid] = transcript_text
        except Exception as e:
            _log(f"Transcript fetch failed for {vid}: {e}")
    got = sum(1 for v in captions.values() if v)
    _log(f"Got captions for {got}/{len(top_items)} reels")
    return captions
 def search_and_enrich(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Full Instagram search: find reels, then fetch captions for top results.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list. Each item has a 'caption_snippet' field.
    """
    # Step 1: Search
    search_result = search_instagram(topic, from_date, to_date, depth, token)
    items = search_result.get("items", [])
    if not items:
        return search_result
    # Step 2: Fetch captions for top N
    captions = fetch_captions(items, token, depth)
    # Step 3: Attach captions to items
    for item in items:
        vid = item["video_id"]
        caption = captions.get(vid)
        if caption:
            item["caption_snippet"] = caption
    return {"items": items, "error": search_result.get("error")}
 def parse_instagram_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse Instagram search response to normalized format.
    Returns:
        List of item dicts ready for normalization.
    """
    return response.get("items", [])
@@ -3,11 +3,12 @@
 import re
 from typing import Dict, List, Optional, Tuple
-from . import cache, http
+from . import cache, http, env
 # OpenAI API
 OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
 OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4.1", "gpt-4o"]
 CODEX_FALLBACK_MODELS = ["gpt-5.1-codex-mini", "gpt-5.2"]
 # xAI API - Agent Tools API requires grok-4 family
 XAI_MODELS_URL = "https://api.x.ai/v1/models"
@@ -157,6 +158,15 @@ def get_models(
    result = {"openai": None, "xai": None}
    if config.get("OPENAI_API_KEY"):
        if config.get("OPENAI_AUTH_SOURCE") == env.AUTH_SOURCE_CODEX:
            # Codex auth doesn't use the OpenAI models list endpoint
            policy = config.get("OPENAI_MODEL_POLICY", "auto")
            pin = config.get("OPENAI_MODEL_PIN")
            if policy == "pinned" and pin:
                result["openai"] = pin
            else:
                result["openai"] = CODEX_FALLBACK_MODELS[0]
        else:
            result["openai"] = select_openai_model(
                config["OPENAI_API_KEY"],
                config.get("OPENAI_MODEL_POLICY", "auto"),
@@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union
 from . import dates, schema
-T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem)
+T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem)
 def filter_by_date_range(
@@ -200,6 +200,99 @@ def normalize_youtube_items(
    return normalized
 def normalize_tiktok_items(
    items: List[Dict[str, Any]],
    from_date: str,
    to_date: str,
 ) -> List[schema.TikTokItem]:
    """Normalize raw TikTok items to schema.
    Args:
        items: Raw TikTok items from Apify
        from_date: Start of date range
        to_date: End of date range
    Returns:
        List of TikTokItem objects
    """
    normalized = []
    for i, item in enumerate(items):
        # Parse engagement
        eng_raw = item.get("engagement") or {}
        engagement = schema.Engagement(
            views=eng_raw.get("views"),
            likes=eng_raw.get("likes"),
            num_comments=eng_raw.get("comments"),
            shares=eng_raw.get("shares"),
        )
        # TikTok dates are reliable (exact timestamps from Apify)
        date_str = item.get("date")
        normalized.append(schema.TikTokItem(
            id=f"TK{i+1}",
            text=item.get("text", ""),
            url=item.get("url", ""),
            author_name=item.get("author_name", ""),
            date=date_str,
            date_confidence="high",
            engagement=engagement,
            caption_snippet=item.get("caption_snippet", ""),
            hashtags=item.get("hashtags", []),
            relevance=item.get("relevance", 0.7),
            why_relevant=item.get("why_relevant", ""),
        ))
    return normalized
 def normalize_instagram_items(
    items: List[Dict[str, Any]],
    from_date: str,
    to_date: str,
 ) -> List[schema.InstagramItem]:
    """Normalize raw Instagram items to schema.
    Args:
        items: Raw Instagram items from ScrapeCreators
        from_date: Start of date range
        to_date: End of date range
    Returns:
        List of InstagramItem objects
    """
    normalized = []
    for i, item in enumerate(items):
        # Parse engagement
        eng_raw = item.get("engagement") or {}
        engagement = schema.Engagement(
            views=eng_raw.get("views"),
            likes=eng_raw.get("likes"),
            num_comments=eng_raw.get("comments"),
        )
        # Instagram dates are reliable (exact timestamps from ScrapeCreators)
        date_str = item.get("date")
        normalized.append(schema.InstagramItem(
            id=f"IG{i+1}",
            text=item.get("text", ""),
            url=item.get("url", ""),
            author_name=item.get("author_name", ""),
            date=date_str,
            date_confidence="high",
            engagement=engagement,
            caption_snippet=item.get("caption_snippet", ""),
            hashtags=item.get("hashtags", []),
            relevance=item.get("relevance", 0.7),
            why_relevant=item.get("why_relevant", ""),
        ))
    return normalized
 def normalize_hackernews_items(
    items: List[Dict[str, Any]],
    from_date: str,
@@ -5,7 +5,7 @@ import re
 import sys
 from typing import Any, Dict, List, Optional
-from . import http
+from . import http, env
 # Fallback models when the selected model isn't accessible (e.g., org not verified for GPT-5)
 # Note: gpt-4o-mini does NOT support web_search with filters param, so exclude it
@@ -42,6 +42,93 @@ def _is_model_access_error(error: http.HTTPError) -> bool:
 OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
 CODEX_RESPONSES_URL = "https://chatgpt.com/backend-api/codex/responses"
 CODEX_INSTRUCTIONS = (
    "You are a research assistant for a skill that summarizes what people are "
    "discussing in the last 30 days. Your goal is to find relevant Reddit threads "
    "about the topic and return ONLY the required JSON. Be inclusive (return more "
    "rather than fewer), but avoid irrelevant results. Prefer threads with discussion "
    "and comments. If you can infer a date, include it; otherwise use null. "
    "Do not include developers.reddit.com or business.reddit.com."
 )
 def _parse_sse_chunk(chunk: str) -> Optional[Dict[str, Any]]:
    """Parse a single SSE chunk into a JSON object."""
    lines = chunk.split("\n")
    data_lines = []
    for line in lines:
        if line.startswith("data:"):
            data_lines.append(line[5:].strip())
    if not data_lines:
        return None
    data = "\n".join(data_lines).strip()
    if not data or data == "[DONE]":
        return None
    try:
        return json.loads(data)
    except json.JSONDecodeError:
        return None
 def _parse_sse_stream_raw(raw: str) -> List[Dict[str, Any]]:
    """Parse SSE stream from raw text and return JSON events."""
    events: List[Dict[str, Any]] = []
    buffer = ""
    for chunk in raw.splitlines(keepends=True):
        buffer += chunk
        while "\n\n" in buffer:
            event_chunk, buffer = buffer.split("\n\n", 1)
            event = _parse_sse_chunk(event_chunk)
            if event is not None:
                events.append(event)
    if buffer.strip():
        event = _parse_sse_chunk(buffer)
        if event is not None:
            events.append(event)
    return events
 def _parse_codex_stream(raw: str) -> Dict[str, Any]:
    """Parse SSE stream from Codex responses into a response-like dict."""
    events = _parse_sse_stream_raw(raw)
    # Prefer explicit completed response payload if present
    for evt in reversed(events):
        if isinstance(evt, dict):
            if evt.get("type") == "response.completed" and isinstance(evt.get("response"), dict):
                return evt["response"]
            if isinstance(evt.get("response"), dict):
                return evt["response"]
    # Fallback: reconstruct output text from deltas
    output_text = ""
    for evt in events:
        if not isinstance(evt, dict):
            continue
        delta = evt.get("delta")
        if isinstance(delta, str):
            output_text += delta
            continue
        text = evt.get("text")
        if isinstance(text, str):
            output_text += text
    if output_text:
        return {
            "output": [
                {
                    "type": "message",
                    "content": [{"type": "output_text", "text": output_text}],
                }
            ]
        }
    return {}
 # Depth configurations: (min, max) threads to request
 # Request MORE than needed since many get filtered by date
@@ -116,6 +203,35 @@ def _build_subreddit_query(topic: str) -> str:
    return f"r/{sub_name} site:reddit.com"
 def _build_payload(model: str, instructions_text: str, input_text: str, auth_source: str) -> Dict[str, Any]:
    """Build responses payload for OpenAI or Codex endpoints."""
    payload = {
        "model": model,
        "store": False,
        "tools": [
            {
                "type": "web_search",
                "filters": {
                    "allowed_domains": ["reddit.com"]
                }
            }
        ],
        "include": ["web_search_call.action.sources"],
        "instructions": instructions_text,
        "input": input_text,
    }
    if auth_source == env.AUTH_SOURCE_CODEX:
        payload["input"] = [
            {
                "type": "message",
                "role": "user",
                "content": [{"type": "input_text", "text": input_text}],
            }
        ]
        payload["stream"] = True
    return payload
 def search_reddit(
    api_key: str,
    model: str,
@@ -123,6 +239,8 @@ def search_reddit(
    from_date: str,
    to_date: str,
    depth: str = "default",
    auth_source: str = "api_key",
    account_id: Optional[str] = None,
    mock_response: Optional[Dict] = None,
    _retry: bool = False,
 ) -> Dict[str, Any]:
@@ -145,10 +263,23 @@ def search_reddit(
    min_items, max_items = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    if auth_source == env.AUTH_SOURCE_CODEX:
        if not account_id:
            raise ValueError("Missing chatgpt_account_id for Codex auth")
        headers = {
            "Authorization": f"Bearer {api_key}",
            "chatgpt-account-id": account_id,
            "OpenAI-Beta": "responses=experimental",
            "originator": "pi",
            "Content-Type": "application/json",
        }
        url = CODEX_RESPONSES_URL
    else:
        headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        }
        url = OPENAI_RESPONSES_URL
    # Adjust timeout based on depth (generous for OpenAI web_search which can be slow)
    timeout = 90 if depth == "quick" else 120 if depth == "default" else 180
@@ -166,6 +297,28 @@ def search_reddit(
        max_items=max_items,
    )
    if auth_source == env.AUTH_SOURCE_CODEX:
        # Codex auth: try model with fallback chain
        from . import models as models_mod
        codex_models_to_try = [model] + [m for m in models_mod.CODEX_FALLBACK_MODELS if m != model]
        instructions_text = CODEX_INSTRUCTIONS + "\n\n" + input_text
        last_error = None
        for current_model in codex_models_to_try:
            try:
                payload = _build_payload(current_model, instructions_text, topic, auth_source)
                raw = http.post_raw(url, payload, headers=headers, timeout=timeout)
                return _parse_codex_stream(raw or "")
            except http.HTTPError as e:
                last_error = e
                if e.status_code == 400:
                    _log_info(f"Model {current_model} not supported on Codex, trying fallback...")
                    continue
                raise
        if last_error:
            raise last_error
        raise http.HTTPError("No Codex-compatible models available")
    # Standard API key auth: try model fallback chain
    last_error = None
    for current_model in models_to_try:
        payload = {
@@ -183,7 +336,7 @@ def search_reddit(
        }
        try:
-            return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
+            return http.post(url, payload, headers=headers, timeout=timeout)
        except http.HTTPError as e:
            last_error = e
            if _is_model_access_error(e):
@@ -0,0 +1,603 @@
 """Reddit search via ScrapeCreators API for /last30days.
 Uses ScrapeCreators REST API to search Reddit globally, discover relevant
 subreddits, run targeted subreddit searches, and fetch comment trees.
 Replaces openai_reddit.py as the primary Reddit search backend.
 Falls back to openai_reddit.py if SCRAPECREATORS_API_KEY is missing but
 OPENAI_API_KEY is present.
 Requires SCRAPECREATORS_API_KEY in config (same key as TikTok + Instagram).
 API docs: https://scrapecreators.com/docs
 """
 import re
 import sys
 from collections import Counter
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional, Set
 try:
    import requests as _requests
 except ImportError:
    _requests = None
 from . import http
 SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/reddit"
 # Depth configurations: how many API calls per phase
 DEPTH_CONFIG = {
    "quick": {
        "global_searches": 1,
        "subreddit_searches": 2,
        "comment_enrichments": 3,
        "timeframe": "week",
    },
    "default": {
        "global_searches": 2,
        "subreddit_searches": 3,
        "comment_enrichments": 5,
        "timeframe": "month",
    },
    "deep": {
        "global_searches": 3,
        "subreddit_searches": 5,
        "comment_enrichments": 8,
        "timeframe": "month",
    },
 }
 # Stopwords for query extraction
 NOISE_WORDS = frozenset({
    'best', 'top', 'good', 'great', 'awesome', 'killer',
    'latest', 'new', 'news', 'update', 'updates',
    'trending', 'hottest', 'popular',
    'practices', 'features', 'tips',
    'recommendations', 'advice',
    'prompt', 'prompts', 'prompting',
    'methods', 'strategies', 'approaches',
    'how', 'to', 'the', 'a', 'an', 'for', 'with',
    'of', 'in', 'on', 'is', 'are', 'what', 'which',
    'guide', 'tutorial', 'using',
 })
 def _log(msg: str):
    """Log to stderr."""
    sys.stderr.write(f"[Reddit] {msg}\n")
    sys.stderr.flush()
 def _sc_headers(token: str) -> Dict[str, str]:
    """Build ScrapeCreators request headers."""
    return {
        "x-api-key": token,
        "Content-Type": "application/json",
    }
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query.
    Strips meta/research words to keep only the core product/concept name.
    """
    text = topic.lower().strip()
    # Strip multi-word prefixes
    prefixes = [
        'what are the best', 'what is the best', 'what are the latest',
        'what are people saying about', 'what do people think about',
        'how do i use', 'how to use', 'how to',
        'what are', 'what is', 'tips for', 'best practices for',
    ]
    for p in prefixes:
        if text.startswith(p + ' '):
            text = text[len(p):].strip()
    words = text.split()
    filtered = [w for w in words if w not in NOISE_WORDS]
    result = ' '.join(filtered) if filtered else text
    return result.rstrip('?!.')
 def expand_reddit_queries(topic: str, depth: str) -> List[str]:
    """Generate multiple Reddit search queries from a topic.
    Uses local logic (no LLM call needed):
    1. Extract core subject (strip noise words)
    2. Include original topic if different from core
    3. For default/deep: add casual/review variant
    4. For deep: add problem/issues variant
    Returns 1-4 query strings depending on depth.
    """
    core = _extract_core_subject(topic)
    queries = [core]
    # Broader variant: include more context from original topic
    original_clean = topic.strip().rstrip('?!.')
    if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
        queries.append(original_clean)
    if depth in ("default", "deep"):
        queries.append(f"{core} worth it OR thoughts OR review")
    if depth == "deep":
        queries.append(f"{core} issues OR problems OR bug OR broken")
    return queries
 # Known utility/meta subreddits that match queries but aren't discussion subs.
 # These get a 0.3x penalty (not banned) in subreddit discovery scoring.
 UTILITY_SUBS = frozenset({
    'namethatsong', 'findthatsong', 'tipofmytongue',
    'whatisthissong', 'helpmefind', 'whatisthisthing',
    'whatsthissong', 'findareddit', 'subredditdrama',
 })
 def discover_subreddits(
    results: List[Dict[str, Any]],
    topic: str = "",
    max_subs: int = 5,
 ) -> List[str]:
    """Extract top subreddits from global search results with relevance weighting.
    Uses frequency + topic-word matching + utility-sub penalties + engagement
    bonus to find discussion subs rather than utility/meta subs.
    Args:
        results: List of post dicts from global search
        topic: Original search topic (for relevance matching)
        max_subs: Maximum subreddits to return
    Returns:
        Top subreddit names sorted by weighted score
    """
    core = _extract_core_subject(topic) if topic else ""
    core_words = set(core.lower().split()) if core else set()
    scores = Counter()
    for post in results:
        sub = post.get("subreddit", "")
        if not sub:
            continue
        # Base: frequency count
        base = 1.0
        # Bonus: subreddit name contains a core topic word
        sub_lower = sub.lower()
        if core_words and any(w in sub_lower for w in core_words if len(w) > 2):
            base += 2.0
        # Penalty: known utility/meta subreddits
        if sub_lower in UTILITY_SUBS:
            base *= 0.3
        # Bonus: post engagement (high-engagement posts = better sub)
        ups = post.get("ups") or post.get("score", 0)
        if ups and ups > 100:
            base += 0.5
        scores[sub] += base
    return [sub for sub, _ in scores.most_common(max_subs)]
 def _parse_date(created_utc) -> Optional[str]:
    """Convert Unix timestamp to YYYY-MM-DD."""
    if not created_utc:
        return None
    try:
        dt = datetime.fromtimestamp(float(created_utc), tz=timezone.utc)
        return dt.strftime("%Y-%m-%d")
    except (ValueError, TypeError, OSError):
        return None
 def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global") -> Dict[str, Any]:
    """Normalize a ScrapeCreators Reddit post to our internal format."""
    permalink = post.get("permalink", "")
    url = f"https://www.reddit.com{permalink}" if permalink else post.get("url", "")
    # Ensure URL looks like a Reddit thread
    if url and "reddit.com" not in url:
        url = ""
    return {
        "id": f"R{idx}",
        "reddit_id": post.get("id", ""),
        "title": str(post.get("title", "")).strip(),
        "url": url,
        "subreddit": str(post.get("subreddit", "")).strip(),
        "date": _parse_date(post.get("created_utc")),
        "engagement": {
            "score": post.get("ups") or post.get("score", 0),
            "num_comments": post.get("num_comments", 0),
            "upvote_ratio": post.get("upvote_ratio"),
        },
        "relevance": 0.7,
        "why_relevant": f"Reddit {source_label} search",
        "selftext": str(post.get("selftext", ""))[:500],
    }
 def _global_search(
    query: str,
    token: str,
    sort: str = "relevance",
    timeframe: str = "month",
 ) -> List[Dict[str, Any]]:
    """Search across all of Reddit via ScrapeCreators global search.
    Args:
        query: Search query
        token: ScrapeCreators API key
        sort: Sort order (relevance, hot, top, new)
        timeframe: Time filter (hour, day, week, month, year, all)
    Returns:
        List of post dicts
    """
    if not _requests:
        _log("requests library not installed, falling back to urllib")
        # Use stdlib http module as fallback
        try:
            from urllib.parse import urlencode
            params = urlencode({"query": query, "sort": sort, "timeframe": timeframe})
            url = f"{SCRAPECREATORS_BASE}/search?{params}"
            headers = _sc_headers(token)
            headers["User-Agent"] = http.USER_AGENT
            data = http.get(url, headers=headers, timeout=30, retries=2)
            return data.get("posts", data.get("data", []))
        except Exception as e:
            _log(f"Global search error (urllib): {e}")
            return []
    try:
        resp = _requests.get(
            f"{SCRAPECREATORS_BASE}/search",
            params={"query": query, "sort": sort, "timeframe": timeframe},
            headers=_sc_headers(token),
            timeout=30,
        )
        resp.raise_for_status()
        data = resp.json()
        return data.get("posts", data.get("data", []))
    except Exception as e:
        _log(f"Global search error: {e}")
        return []
 def _subreddit_search(
    subreddit: str,
    query: str,
    token: str,
    sort: str = "relevance",
    timeframe: str = "month",
 ) -> List[Dict[str, Any]]:
    """Search within a specific subreddit via ScrapeCreators.
    Args:
        subreddit: Subreddit name (without r/)
        query: Search query
        token: ScrapeCreators API key
        sort: Sort order
        timeframe: Time filter
    Returns:
        List of post dicts
    """
    if not _requests:
        try:
            from urllib.parse import urlencode
            params = urlencode({
                "subreddit": subreddit, "query": query,
                "sort": sort, "timeframe": timeframe,
            })
            url = f"{SCRAPECREATORS_BASE}/subreddit/search?{params}"
            headers = _sc_headers(token)
            headers["User-Agent"] = http.USER_AGENT
            data = http.get(url, headers=headers, timeout=30, retries=2)
            return data.get("posts", data.get("data", []))
        except Exception as e:
            _log(f"Subreddit search error (urllib) for r/{subreddit}: {e}")
            return []
    try:
        resp = _requests.get(
            f"{SCRAPECREATORS_BASE}/subreddit/search",
            params={
                "subreddit": subreddit,
                "query": query,
                "sort": sort,
                "timeframe": timeframe,
            },
            headers=_sc_headers(token),
            timeout=30,
        )
        resp.raise_for_status()
        data = resp.json()
        return data.get("posts", data.get("data", []))
    except Exception as e:
        _log(f"Subreddit search error for r/{subreddit}: {e}")
        return []
 def fetch_post_comments(
    url: str,
    token: str,
 ) -> List[Dict[str, Any]]:
    """Fetch comments for a Reddit post via ScrapeCreators.
    Args:
        url: Reddit post URL or permalink
        token: ScrapeCreators API key
    Returns:
        List of comment dicts with score, author, body, etc.
    """
    if not _requests:
        try:
            from urllib.parse import urlencode
            params = urlencode({"url": url})
            api_url = f"{SCRAPECREATORS_BASE}/post/comments?{params}"
            headers = _sc_headers(token)
            headers["User-Agent"] = http.USER_AGENT
            data = http.get(api_url, headers=headers, timeout=30, retries=2)
            return data.get("comments", data.get("data", []))
        except Exception as e:
            _log(f"Comment fetch error (urllib): {e}")
            return []
    try:
        resp = _requests.get(
            f"{SCRAPECREATORS_BASE}/post/comments",
            params={"url": url},
            headers=_sc_headers(token),
            timeout=30,
        )
        resp.raise_for_status()
        data = resp.json()
        return data.get("comments", data.get("data", []))
    except Exception as e:
        _log(f"Comment fetch error: {e}")
        return []
 def _dedupe_posts(posts: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
    """Deduplicate posts by reddit_id, keeping first occurrence."""
    seen_ids = set()
    seen_urls = set()
    unique = []
    for post in posts:
        rid = post.get("reddit_id", "")
        url = post.get("url", "")
        if rid and rid in seen_ids:
            continue
        if url and url in seen_urls:
            continue
        if rid:
            seen_ids.add(rid)
        if url:
            seen_urls.add(url)
        unique.append(post)
    return unique
 def search_reddit(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Full Reddit search: multi-query global discovery + subreddit drill-down.
    This is the main entry point. Replaces openai_reddit.search_reddit().
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    timeframe = config["timeframe"]
    # === Phase 1: Query Expansion ===
    queries = expand_reddit_queries(topic, depth)
    _log(f"Expanded '{topic}' into {len(queries)} queries: {queries}")
    # === Phase 2: Global Discovery ===
    all_raw_posts = []
    max_global = config["global_searches"]
    for i, query in enumerate(queries[:max_global]):
        sort = "relevance" if i == 0 else "top"
        _log(f"Global search {i+1}/{max_global}: '{query}' (sort={sort})")
        posts = _global_search(query, token, sort=sort, timeframe=timeframe)
        _log(f"  -> {len(posts)} results")
        all_raw_posts.extend(posts)
    # Normalize all posts
    all_items = []
    for i, post in enumerate(all_raw_posts):
        item = _normalize_post(post, i + 1, "global")
        all_items.append(item)
    # === Phase 3: Subreddit Discovery + Targeted Search ===
    discovered_subs = discover_subreddits(all_raw_posts, topic=topic, max_subs=config["subreddit_searches"])
    _log(f"Discovered subreddits: {discovered_subs}")
    core = _extract_core_subject(topic)
    for sub in discovered_subs[:config["subreddit_searches"]]:
        _log(f"Subreddit search: r/{sub} for '{core}'")
        sub_posts = _subreddit_search(sub, core, token, sort="relevance", timeframe=timeframe)
        _log(f"  -> {len(sub_posts)} results from r/{sub}")
        for j, post in enumerate(sub_posts):
            item = _normalize_post(post, len(all_items) + j + 1, f"r/{sub}")
            all_items.append(item)
    # === Phase 4: Deduplicate ===
    all_items = _dedupe_posts(all_items)
    _log(f"After dedup: {len(all_items)} unique posts")
    # === Phase 5: Date filter ===
    in_range = []
    out_of_range = 0
    for item in all_items:
        if item["date"] and from_date <= item["date"] <= to_date:
            in_range.append(item)
        elif item["date"] is None:
            in_range.append(item)  # Keep unknown dates
        else:
            out_of_range += 1
    if in_range:
        all_items = in_range
        if out_of_range:
            _log(f"Filtered {out_of_range} posts outside date range")
    else:
        _log(f"No posts within date range, keeping all {len(all_items)}")
    # === Phase 6: Sort by engagement ===
    all_items.sort(
        key=lambda x: (x.get("engagement", {}).get("score", 0) or 0),
        reverse=True,
    )
    # Re-index IDs
    for i, item in enumerate(all_items):
        item["id"] = f"R{i+1}"
    _log(f"Final: {len(all_items)} Reddit posts")
    return {"items": all_items}
 def enrich_with_comments(
    items: List[Dict[str, Any]],
    token: str,
    depth: str = "default",
 ) -> List[Dict[str, Any]]:
    """Enrich top items with comment data from ScrapeCreators.
    Args:
        items: Reddit items from search_reddit()
        token: ScrapeCreators API key
        depth: Depth for comment limit
    Returns:
        Items with top_comments and comment_insights added.
    """
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    max_comments = config["comment_enrichments"]
    if not items or not token:
        return items
    top_items = items[:max_comments]
    _log(f"Enriching comments for {len(top_items)} posts")
    for item in top_items:
        url = item.get("url", "")
        if not url:
            continue
        raw_comments = fetch_post_comments(url, token)
        if not raw_comments:
            continue
        # Parse comments into our format
        top_comments = []
        insights = []
        for ci, c in enumerate(raw_comments[:10]):  # Take top 10 comments
            body = c.get("body", "")
            if not body or body in ("[deleted]", "[removed]"):
                continue
            score = c.get("ups") or c.get("score", 0)
            author = c.get("author", "[deleted]")
            permalink = c.get("permalink", "")
            comment_url = f"https://reddit.com{permalink}" if permalink else ""
            # Top comment gets more room (400 chars) — funny/clever comments need it
            max_excerpt = 400 if ci == 0 else 300
            top_comments.append({
                "score": score,
                "date": _parse_date(c.get("created_utc")),
                "author": author,
                "excerpt": body[:max_excerpt],
                "url": comment_url,
            })
            # Extract insights from substantive comments
            if len(body) >= 30 and author not in ("[deleted]", "[removed]", "AutoModerator"):
                insight = body[:150]
                if len(body) > 150:
                    for i, char in enumerate(insight):
                        if char in '.!?' and i > 50:
                            insight = insight[:i+1]
                            break
                    else:
                        insight = insight.rstrip() + "..."
                insights.append(insight)
        # Sort comments by score
        top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
        item["top_comments"] = top_comments[:10]
        item["comment_insights"] = insights[:10]
    return items
 def search_and_enrich(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Full Reddit pipeline: search + comment enrichment.
    This is the convenience function that does everything.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list. Items include top_comments and comment_insights.
    """
    result = search_reddit(topic, from_date, to_date, depth, token)
    items = result.get("items", [])
    if items and token:
        items = enrich_with_comments(items, token, depth)
        result["items"] = items
    return result
 def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse ScrapeCreators response to item list.
    Compatibility shim matching openai_reddit.parse_reddit_response() signature.
    """
    return response.get("items", [])
@@ -1,4 +1,9 @@
-"""Reddit thread enrichment with real engagement metrics."""
+"""Reddit thread enrichment with real engagement metrics.
 Supports two backends:
 1. ScrapeCreators API (preferred) - no rate limits, 1 credit/call
 2. reddit.com/.json (fallback) - free but 429-prone
 """
 import re
 from typing import Any, Dict, List, Optional
@@ -254,3 +259,67 @@ def enrich_reddit_item(
    item["comment_insights"] = extract_comment_insights(top_comments)
    return item
 def enrich_reddit_item_sc(
    item: Dict[str, Any],
    token: str,
    timeout: int = 30,
 ) -> Dict[str, Any]:
    """Enrich a Reddit item using ScrapeCreators comment API.
    No rate limit risk. Uses 1 credit per call.
    Args:
        item: Reddit item dict (already has engagement from search)
        token: ScrapeCreators API key
        timeout: HTTP timeout
    Returns:
        Enriched item with top_comments and comment_insights
    """
    from . import reddit as reddit_mod
    url = item.get("url", "")
    if not url:
        return item
    raw_comments = reddit_mod.fetch_post_comments(url, token)
    if not raw_comments:
        return item
    top_comments = []
    for c in raw_comments[:10]:
        body = c.get("body", "")
        if not body or body in ("[deleted]", "[removed]"):
            continue
        score = c.get("ups") or c.get("score", 0)
        author = c.get("author", "[deleted]")
        permalink = c.get("permalink", "")
        comment_url = f"https://reddit.com{permalink}" if permalink else ""
        top_comments.append({
            "score": score,
            "date": dates.timestamp_to_date(c.get("created_utc")) if c.get("created_utc") else None,
            "author": author,
            "body": body[:300],
            "excerpt": body[:200],
            "url": comment_url,
        })
    top_comments.sort(key=lambda c: c.get("score", 0), reverse=True)
    item["top_comments"] = []
    for c in top_comments:
        item["top_comments"].append({
            "score": c.get("score", 0),
            "date": c.get("date"),
            "author": c.get("author", ""),
            "excerpt": c.get("excerpt", ""),
            "url": c.get("url", ""),
        })
    item["comment_insights"] = extract_comment_insights(top_comments)
    return item
@@ -4,7 +4,7 @@ import json
 import os
 import tempfile
 from pathlib import Path
-from typing import List, Optional
+from typing import Optional
 from . import schema
@@ -24,6 +24,10 @@ def _xref_tag(item) -> str:
            source_names.add('X')
        elif ref_id.startswith('YT'):
            source_names.add('YouTube')
        elif ref_id.startswith('TK'):
            source_names.add('TikTok')
        elif ref_id.startswith('IG'):
            source_names.add('Instagram')
        elif ref_id.startswith('HN'):
            source_names.add('HN')
        elif ref_id.startswith('PM'):
@@ -57,8 +61,11 @@ def _assess_data_freshness(report: schema.Report) -> dict:
    hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from)
    pm_recent = sum(1 for p in report.polymarket if p.date and p.date >= report.range_from)
-    total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent
+    tiktok_recent = sum(1 for t in report.tiktok if t.date and t.date >= report.range_from)
-    total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket)
+    ig_recent = sum(1 for ig in report.instagram if ig.date and ig.date >= report.range_from)
    total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent + tiktok_recent + ig_recent
    total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket) + len(report.tiktok) + len(report.instagram)
    return {
        "reddit_recent": reddit_recent,
@@ -101,9 +108,10 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
        lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
        lines.append("")
        lines.append("---")
-        lines.append("**⚡ Want better results?** Add API keys to unlock Reddit & X data:")
+        lines.append("**⚡ Want better results?** Add API keys to unlock Reddit, TikTok, Instagram & X data:")
-        lines.append("- `OPENAI_API_KEY` → Reddit threads with real upvotes & comments")
+        lines.append("- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views")
        lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
        lines.append("- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)")
        lines.append("- Edit `~/.config/last30days/.env` to add keys")
        lines.append("---")
        lines.append("")
@@ -125,11 +133,11 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
    lines.append("")
    # Coverage note for partial coverage
-    if report.mode == "reddit-only" and missing_keys == "x":
+    if report.mode == "reddit-only" and missing_keys in ("x", "none"):
-        lines.append("*💡 Tip: Add XAI_API_KEY for X/Twitter data and better triangulation.*")
+        lines.append("*💡 Tip: Add an xAI key (`XAI_API_KEY`) for X/Twitter data and better triangulation.*")
        lines.append("")
-    elif report.mode == "x-only" and missing_keys == "reddit":
+    elif report.mode == "x-only" and missing_keys in ("reddit", "none"):
-        lines.append("*💡 Tip: Add OPENAI_API_KEY for Reddit data and better triangulation.*")
+        lines.append("*💡 Tip: Add `SCRAPECREATORS_API_KEY` for Reddit + TikTok + Instagram data (one key, all three) and better triangulation.*")
        lines.append("")
    # Reddit items
@@ -166,9 +174,17 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
            lines.append(f"  {item.url}")
            lines.append(f"  *{item.why_relevant}*")
-            # Top comment insights
+            # Top comment (elevated — Reddit's value IS the comments)
            if item.top_comments and item.top_comments[0].score >= 10:
                tc = item.top_comments[0]
                excerpt = tc.excerpt[:200]
                if len(tc.excerpt) > 200:
                    excerpt = excerpt.rstrip() + "..."
                lines.append(f'  \U0001f4ac Top comment ({tc.score} upvotes): "{excerpt}"')
            # Comment insights
            if item.comment_insights:
-                lines.append(f"  Insights:")
+                lines.append("  Insights:")
                for insight in item.comment_insights[:3]:
                    lines.append(f"    - {insight}")
@@ -243,6 +259,78 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")
    # TikTok items
    if report.tiktok_error:
        lines.append("### TikTok Videos")
        lines.append("")
        lines.append(f"**ERROR:** {report.tiktok_error}")
        lines.append("")
    elif report.tiktok:
        lines.append("### TikTok Videos")
        lines.append("")
        for item in report.tiktok[:limit]:
            eng_str = ""
            if item.engagement:
                eng = item.engagement
                parts = []
                if eng.views is not None:
                    parts.append(f"{eng.views:,} views")
                if eng.likes is not None:
                    parts.append(f"{eng.likes:,} likes")
                if parts:
                    eng_str = f" [{', '.join(parts)}]"
            date_str = f" ({item.date})" if item.date else ""
            lines.append(f"**{item.id}** (score:{item.score}) @{item.author_name}{date_str}{eng_str}{_xref_tag(item)}")
            lines.append(f"  {item.text[:200]}")
            lines.append(f"  {item.url}")
            if item.caption_snippet and item.caption_snippet != item.text[:len(item.caption_snippet)]:
                snippet = item.caption_snippet[:200]
                if len(item.caption_snippet) > 200:
                    snippet += "..."
                lines.append(f"  Caption: {snippet}")
            if item.hashtags:
                lines.append(f"  Tags: {' '.join('#' + h for h in item.hashtags[:8])}")
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")
    # Instagram items
    if report.instagram_error:
        lines.append("### Instagram Reels")
        lines.append("")
        lines.append(f"**ERROR:** {report.instagram_error}")
        lines.append("")
    elif report.instagram:
        lines.append("### Instagram Reels")
        lines.append("")
        for item in report.instagram[:limit]:
            eng_str = ""
            if item.engagement:
                eng = item.engagement
                parts = []
                if eng.views is not None:
                    parts.append(f"{eng.views:,} views")
                if eng.likes is not None:
                    parts.append(f"{eng.likes:,} likes")
                if parts:
                    eng_str = f" [{', '.join(parts)}]"
            date_str = f" ({item.date})" if item.date else ""
            lines.append(f"**{item.id}** (score:{item.score}) @{item.author_name}{date_str}{eng_str}{_xref_tag(item)}")
            lines.append(f"  {item.text[:200]}")
            lines.append(f"  {item.url}")
            if item.caption_snippet and item.caption_snippet != item.text[:len(item.caption_snippet)]:
                snippet = item.caption_snippet[:200]
                if len(item.caption_snippet) > 200:
                    snippet += "..."
                lines.append(f"  Caption: {snippet}")
            if item.hashtags:
                lines.append(f"  Tags: {' '.join('#' + h for h in item.hashtags[:8])}")
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")
    # Hacker News items
    if report.hackernews_error:
        lines.append("### Hacker News Stories")
@@ -406,6 +494,22 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
        lines.append(f"  ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)")
    # Hide when zero results (no skip reason line needed)
    # TikTok
    if report.tiktok_error:
        lines.append(f"  ❌ TikTok: error — {report.tiktok_error}")
    elif report.tiktok:
        with_captions = sum(1 for v in report.tiktok if getattr(v, 'caption_snippet', None))
        lines.append(f"  ✅ TikTok: {len(report.tiktok)} videos ({with_captions} with captions)")
    # Hide when zero results
    # Instagram
    if report.instagram_error:
        lines.append(f"  ❌ Instagram: error — {report.instagram_error}")
    elif report.instagram:
        with_captions = sum(1 for v in report.instagram if getattr(v, 'caption_snippet', None))
        lines.append(f"  ✅ Instagram: {len(report.instagram)} reels ({with_captions} with captions)")
    # Hide when zero results
    # Hacker News
    if report.hackernews_error:
        lines.append(f"  ❌ HN: error - {report.hackernews_error}")
@@ -457,6 +561,10 @@ def render_context_snippet(report: schema.Report) -> str:
        all_items.append((item.score, "Reddit", item.title, item.url))
    for item in report.x[:5]:
        all_items.append((item.score, "X", item.text[:50] + "...", item.url))
    for item in report.tiktok[:5]:
        all_items.append((item.score, "TikTok", item.text[:50] + "...", item.url))
    for item in report.instagram[:5]:
        all_items.append((item.score, "Instagram", item.text[:50] + "...", item.url))
    for item in report.hackernews[:5]:
        all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
    for item in report.polymarket[:5]:
@@ -522,6 +630,15 @@ def render_full_report(report: schema.Report) -> str:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
            if item.top_comments and item.top_comments[0].score >= 10:
                tc = item.top_comments[0]
                excerpt = tc.excerpt[:200]
                if len(tc.excerpt) > 200:
                    excerpt = excerpt.rstrip() + "..."
                lines.append("")
                lines.append(f'**\U0001f4ac Top Comment** ({tc.score} upvotes, u/{tc.author}):')
                lines.append(f'> {excerpt}')
            if item.comment_insights:
                lines.append("")
                lines.append("**Key Insights from Comments:**")
@@ -550,6 +667,52 @@ def render_full_report(report: schema.Report) -> str:
            lines.append(f"> {item.text}")
            lines.append("")
    # TikTok section
    if report.tiktok:
        lines.append("## TikTok Videos")
        lines.append("")
        for item in report.tiktok:
            lines.append(f"### {item.id}: @{item.author_name}")
            lines.append("")
            lines.append(f"- **URL:** {item.url}")
            lines.append(f"- **Date:** {item.date or 'Unknown'}")
            lines.append(f"- **Score:** {item.score}/100")
            lines.append(f"- **Relevance:** {item.why_relevant}")
            if item.engagement:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.views or '?'} views, {eng.likes or '?'} likes, {eng.num_comments or '?'} comments")
            if item.hashtags:
                lines.append(f"- **Hashtags:** {' '.join('#' + h for h in item.hashtags[:10])}")
            lines.append("")
            lines.append(f"> {item.text[:300]}")
            lines.append("")
    # Instagram section
    if report.instagram:
        lines.append("## Instagram Reels")
        lines.append("")
        for item in report.instagram:
            lines.append(f"### {item.id}: @{item.author_name}")
            lines.append("")
            lines.append(f"- **URL:** {item.url}")
            lines.append(f"- **Date:** {item.date or 'Unknown'}")
            lines.append(f"- **Score:** {item.score}/100")
            lines.append(f"- **Relevance:** {item.why_relevant}")
            if item.engagement:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.views or '?'} views, {eng.likes or '?'} likes, {eng.num_comments or '?'} comments")
            if item.hashtags:
                lines.append(f"- **Hashtags:** {' '.join('#' + h for h in item.hashtags[:10])}")
            lines.append("")
            lines.append(f"> {item.text[:300]}")
            lines.append("")
    # HN section
    if report.hackernews:
        lines.append("## Hacker News Stories")
@@ -630,6 +793,8 @@ def render_full_report(report: schema.Report) -> str:
    return "\n".join(lines)
 def write_outputs(
    report: schema.Report,
    raw_openai: Optional[dict] = None,
@@ -22,6 +22,9 @@ class Engagement:
    # YouTube fields
    views: Optional[int] = None
    # TikTok / Facebook fields
    shares: Optional[int] = None
    # Polymarket fields
    volume: Optional[float] = None
    liquidity: Optional[float] = None
@@ -44,6 +47,8 @@ class Engagement:
            d['quotes'] = self.quotes
        if self.views is not None:
            d['views'] = self.views
        if self.shares is not None:
            d['shares'] = self.shares
        if self.volume is not None:
            d['volume'] = self.volume
        if self.liquidity is not None:
@@ -231,6 +236,84 @@ class YouTubeItem:
        return d
@dataclass
 class TikTokItem:
    """Normalized TikTok item."""
    id: str                    # video_id
    text: str                  # caption/description
    url: str                   # webVideoUrl
    author_name: str           # authorMeta.name
    date: Optional[str] = None
    date_confidence: str = "high"  # Apify provides exact timestamps
    engagement: Optional[Engagement] = None  # views, likes, num_comments, shares
    caption_snippet: str = ""  # spoken-word caption (if available), else text
    hashtags: List[str] = field(default_factory=list)
    relevance: float = 0.7
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    cross_refs: List[str] = field(default_factory=list)
    def to_dict(self) -> Dict[str, Any]:
        d = {
            'id': self.id,
            'text': self.text,
            'url': self.url,
            'author_name': self.author_name,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'engagement': self.engagement.to_dict() if self.engagement else None,
            'caption_snippet': self.caption_snippet,
            'hashtags': self.hashtags,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
        if self.cross_refs:
            d['cross_refs'] = self.cross_refs
        return d
@dataclass
 class InstagramItem:
    """Normalized Instagram item."""
    id: str                    # "IG1", "IG2", ...
    text: str                  # caption text
    url: str                   # https://www.instagram.com/reel/{code}
    author_name: str           # Instagram handle
    date: Optional[str] = None
    date_confidence: str = "high"  # ScrapeCreators provides exact timestamps
    engagement: Optional[Engagement] = None  # views, likes, num_comments
    caption_snippet: str = ""  # spoken-word caption (if available), else text
    hashtags: List[str] = field(default_factory=list)
    relevance: float = 0.7
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    cross_refs: List[str] = field(default_factory=list)
    def to_dict(self) -> Dict[str, Any]:
        d = {
            'id': self.id,
            'text': self.text,
            'url': self.url,
            'author_name': self.author_name,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'engagement': self.engagement.to_dict() if self.engagement else None,
            'caption_snippet': self.caption_snippet,
            'hashtags': self.hashtags,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
        if self.cross_refs:
            d['cross_refs'] = self.cross_refs
        return d
@dataclass
 class HackerNewsItem:
    """Normalized Hacker News item."""
@@ -329,6 +412,8 @@ class Report:
    x: List[XItem] = field(default_factory=list)
    web: List[WebSearchItem] = field(default_factory=list)
    youtube: List[YouTubeItem] = field(default_factory=list)
    tiktok: List[TikTokItem] = field(default_factory=list)
    instagram: List[InstagramItem] = field(default_factory=list)
    hackernews: List[HackerNewsItem] = field(default_factory=list)
    polymarket: List[PolymarketItem] = field(default_factory=list)
    best_practices: List[str] = field(default_factory=list)
@@ -339,6 +424,8 @@ class Report:
    x_error: Optional[str] = None
    web_error: Optional[str] = None
    youtube_error: Optional[str] = None
    tiktok_error: Optional[str] = None
    instagram_error: Optional[str] = None
    hackernews_error: Optional[str] = None
    polymarket_error: Optional[str] = None
    # Handle resolution
@@ -362,6 +449,8 @@ class Report:
            'x': [x.to_dict() for x in self.x],
            'web': [w.to_dict() for w in self.web],
            'youtube': [y.to_dict() for y in self.youtube],
            'tiktok': [t.to_dict() for t in self.tiktok],
            'instagram': [ig.to_dict() for ig in self.instagram],
            'hackernews': [h.to_dict() for h in self.hackernews],
            'polymarket': [p.to_dict() for p in self.polymarket],
            'best_practices': self.best_practices,
@@ -378,6 +467,10 @@ class Report:
            d['web_error'] = self.web_error
        if self.youtube_error:
            d['youtube_error'] = self.youtube_error
        if self.tiktok_error:
            d['tiktok_error'] = self.tiktok_error
        if self.instagram_error:
            d['instagram_error'] = self.instagram_error
        if self.hackernews_error:
            d['hackernews_error'] = self.hackernews_error
        if self.polymarket_error:
@@ -485,6 +578,54 @@ class Report:
                cross_refs=y.get('cross_refs', []),
            ))
        # Reconstruct TikTok items
        tiktok_items = []
        for t in data.get('tiktok', []):
            eng = None
            if t.get('engagement'):
                eng = Engagement(**t['engagement'])
            subs = SubScores(**t.get('subs', {})) if t.get('subs') else SubScores()
            tiktok_items.append(TikTokItem(
                id=t['id'],
                text=t.get('text', ''),
                url=t['url'],
                author_name=t.get('author_name', ''),
                date=t.get('date'),
                date_confidence=t.get('date_confidence', 'high'),
                engagement=eng,
                caption_snippet=t.get('caption_snippet', ''),
                hashtags=t.get('hashtags', []),
                relevance=t.get('relevance', 0.7),
                why_relevant=t.get('why_relevant', ''),
                subs=subs,
                score=t.get('score', 0),
                cross_refs=t.get('cross_refs', []),
            ))
        # Reconstruct Instagram items
        ig_items = []
        for ig in data.get('instagram', []):
            eng = None
            if ig.get('engagement'):
                eng = Engagement(**ig['engagement'])
            subs = SubScores(**ig.get('subs', {})) if ig.get('subs') else SubScores()
            ig_items.append(InstagramItem(
                id=ig['id'],
                text=ig.get('text', ''),
                url=ig['url'],
                author_name=ig.get('author_name', ''),
                date=ig.get('date'),
                date_confidence=ig.get('date_confidence', 'high'),
                engagement=eng,
                caption_snippet=ig.get('caption_snippet', ''),
                hashtags=ig.get('hashtags', []),
                relevance=ig.get('relevance', 0.7),
                why_relevant=ig.get('why_relevant', ''),
                subs=subs,
                score=ig.get('score', 0),
                cross_refs=ig.get('cross_refs', []),
            ))
        # Reconstruct HackerNews items
        hn_items = []
        for h in data.get('hackernews', []):
@@ -549,6 +690,8 @@ class Report:
            x=x_items,
            web=web_items,
            youtube=youtube_items,
            tiktok=tiktok_items,
            instagram=ig_items,
            hackernews=hn_items,
            polymarket=pm_items,
            best_practices=data.get('best_practices', []),
@@ -558,6 +701,8 @@ class Report:
            x_error=data.get('x_error'),
            web_error=data.get('web_error'),
            youtube_error=data.get('youtube_error'),
            tiktok_error=data.get('tiktok_error'),
            instagram_error=data.get('instagram_error'),
            hackernews_error=data.get('hackernews_error'),
            polymarket_error=data.get('polymarket_error'),
            resolved_x_handle=data.get('resolved_x_handle'),
@@ -31,10 +31,16 @@ def log1p_safe(x: Optional[int]) -> float:
    return math.log1p(x)
-def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
+def compute_reddit_engagement_raw(
    engagement: Optional[schema.Engagement],
    top_comment_score: Optional[int] = None,
 ) -> Optional[float]:
    """Compute raw engagement score for Reddit item.
-    Formula: 0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)
+    Formula: 0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)
    The 10% comment quality weight rewards posts where the community engaged deeply
    — a highly upvoted top comment means the thread sparked real discussion.
    """
    if engagement is None:
        return None
@@ -45,8 +51,9 @@ def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Op
    score = log1p_safe(engagement.score)
    comments = log1p_safe(engagement.num_comments)
    ratio = (engagement.upvote_ratio or 0.5) * 10
    top_cmt = log1p_safe(top_comment_score)
-    return 0.55 * score + 0.40 * comments + 0.05 * ratio
+    return 0.50 * score + 0.35 * comments + 0.05 * ratio + 0.10 * top_cmt
 def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
@@ -113,8 +120,13 @@ def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem
    if not items:
        return items
-    # Compute raw engagement scores
+    # Compute raw engagement scores (with top comment quality signal)
-    eng_raw = [compute_reddit_engagement_raw(item.engagement) for item in items]
+    eng_raw = []
    for item in items:
        top_cmt_score = None
        if item.top_comments:
            top_cmt_score = item.top_comments[0].score
        eng_raw.append(compute_reddit_engagement_raw(item.engagement, top_cmt_score))
    # Normalize engagement to 0-100
    eng_normalized = normalize_to_100(eng_raw)
@@ -280,6 +292,124 @@ def score_youtube_items(items: List[schema.YouTubeItem]) -> List[schema.YouTubeI
    return items
 def compute_tiktok_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for TikTok item.
    Formula: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
    Views dominate on TikTok — they're the primary discovery signal.
    """
    if engagement is None:
        return None
    if engagement.views is None and engagement.likes is None:
        return None
    views = log1p_safe(engagement.views)
    likes = log1p_safe(engagement.likes)
    comments = log1p_safe(engagement.num_comments)
    return 0.50 * views + 0.30 * likes + 0.20 * comments
 def score_tiktok_items(items: List[schema.TikTokItem]) -> List[schema.TikTokItem]:
    """Compute scores for TikTok items.
    Uses same weight structure as YouTube (relevance + recency + engagement).
    """
    if not items:
        return items
    eng_raw = [compute_tiktok_engagement_raw(item.engagement) for item in items]
    eng_normalized = normalize_to_100(eng_raw)
    for i, item in enumerate(items):
        rel_score = int(item.relevance * 100)
        rec_score = dates.recency_score(item.date)
        if eng_normalized[i] is not None:
            eng_score = int(eng_normalized[i])
        else:
            eng_score = DEFAULT_ENGAGEMENT
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=eng_score,
        )
        overall = (
            WEIGHT_RELEVANCE * rel_score +
            WEIGHT_RECENCY * rec_score +
            WEIGHT_ENGAGEMENT * eng_score
        )
        if eng_raw[i] is None:
            overall -= UNKNOWN_ENGAGEMENT_PENALTY
        item.score = max(0, min(100, int(overall)))
    return items
 def compute_instagram_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for Instagram item.
    Formula: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
    Views dominate on Instagram Reels — they're the primary discovery signal.
    """
    if engagement is None:
        return None
    if engagement.views is None and engagement.likes is None:
        return None
    views = log1p_safe(engagement.views)
    likes = log1p_safe(engagement.likes)
    comments = log1p_safe(engagement.num_comments)
    return 0.50 * views + 0.30 * likes + 0.20 * comments
 def score_instagram_items(items: List[schema.InstagramItem]) -> List[schema.InstagramItem]:
    """Compute scores for Instagram items.
    Uses same weight structure as TikTok (relevance + recency + engagement).
    """
    if not items:
        return items
    eng_raw = [compute_instagram_engagement_raw(item.engagement) for item in items]
    eng_normalized = normalize_to_100(eng_raw)
    for i, item in enumerate(items):
        rel_score = int(item.relevance * 100)
        rec_score = dates.recency_score(item.date)
        if eng_normalized[i] is not None:
            eng_score = int(eng_normalized[i])
        else:
            eng_score = DEFAULT_ENGAGEMENT
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=eng_score,
        )
        overall = (
            WEIGHT_RELEVANCE * rel_score +
            WEIGHT_RECENCY * rec_score +
            WEIGHT_ENGAGEMENT * eng_score
        )
        if eng_raw[i] is None:
            overall -= UNKNOWN_ENGAGEMENT_PENALTY
        item.score = max(0, min(100, int(overall)))
    return items
 def compute_hackernews_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for Hacker News item.
@@ -453,7 +583,7 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
    return items
-def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
+def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
    """Sort items by score (descending), then date, then source priority.
    Args:
@@ -470,19 +600,23 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
        date = item.date or "0000-00-00"
        date_key = -int(date.replace("-", ""))
-        # Tertiary: source priority (Reddit > X > YouTube > HN > Polymarket > WebSearch)
+        # Tertiary: source priority (Reddit > X > YouTube > TikTok > HN > Polymarket > WebSearch)
        if isinstance(item, schema.RedditItem):
            source_priority = 0
        elif isinstance(item, schema.XItem):
            source_priority = 1
        elif isinstance(item, schema.YouTubeItem):
            source_priority = 2
-        elif isinstance(item, schema.HackerNewsItem):
+        elif isinstance(item, schema.TikTokItem):
            source_priority = 3
-        elif isinstance(item, schema.PolymarketItem):
+        elif isinstance(item, schema.InstagramItem):
            source_priority = 4
-        else:  # WebSearchItem
+        elif isinstance(item, schema.HackerNewsItem):
            source_priority = 5
        elif isinstance(item, schema.PolymarketItem):
            source_priority = 6
        else:  # WebSearchItem
            source_priority = 7
        # Quaternary: title/text for stability
        text = getattr(item, "title", "") or getattr(item, "text", "")
@@ -0,0 +1,421 @@
 """TikTok search via ScrapeCreators API for /last30days.
 Uses ScrapeCreators REST API to search TikTok by keyword, extract engagement
 metrics (views, likes, comments, shares), and fetch video transcripts.
 Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
 API docs: https://scrapecreators.com/docs
 """
 import re
 import sys
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional, Set
 try:
    import requests as _requests
 except ImportError:
    _requests = None
 SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/tiktok"
 # Depth configurations: how many results to fetch / captions to extract
 DEPTH_CONFIG = {
    "quick":   {"results_per_page": 10, "max_captions": 3},
    "default": {"results_per_page": 20, "max_captions": 5},
    "deep":    {"results_per_page": 40, "max_captions": 8},
 }
 # Max words to keep from each caption
 CAPTION_MAX_WORDS = 500
 # Stopwords for relevance computation (shared with youtube_yt.py pattern)
 STOPWORDS = frozenset({
    'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
    'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
    'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
    'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
    'all', 'just', 'get', 'has', 'have', 'was', 'will',
 })
 # Synonym groups for relevance scoring
 SYNONYMS = {
    'hip': {'rap', 'hiphop'},
    'hop': {'rap', 'hiphop'},
    'rap': {'hip', 'hop', 'hiphop'},
    'hiphop': {'rap', 'hip', 'hop'},
    'js': {'javascript'},
    'javascript': {'js'},
    'ts': {'typescript'},
    'typescript': {'ts'},
    'ai': {'artificial', 'intelligence'},
    'ml': {'machine', 'learning'},
    'react': {'reactjs'},
    'reactjs': {'react'},
 }
 def _tokenize(text: str) -> Set[str]:
    """Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
    words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
    tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
    expanded = set(tokens)
    for t in tokens:
        if t in SYNONYMS:
            expanded.update(SYNONYMS[t])
    return expanded
 def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> float:
    """Compute relevance as ratio of query tokens found in text + hashtags.
    Uses ratio overlap (intersection / query_length). Hashtags provide
    a TikTok-specific relevance boost. Floors at 0.1.
    """
    q_tokens = _tokenize(query)
    # Combine text and hashtags for matching
    combined = text
    if hashtags:
        combined = f"{text} {' '.join(hashtags)}"
    t_tokens = _tokenize(combined)
    # Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
    if hashtags:
        for tag in hashtags:
            tag_lower = tag.lower()
            for qt in q_tokens:
                if qt in tag_lower and qt != tag_lower:
                    t_tokens.add(qt)
    if not q_tokens:
        return 0.5  # Neutral fallback
    overlap = len(q_tokens & t_tokens)
    ratio = overlap / len(q_tokens)
    return max(0.1, min(1.0, ratio))
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for TikTok search.
    Strips meta/research words to keep only the core product/concept name.
    """
    text = topic.lower().strip()
    # Strip multi-word prefixes
    prefixes = [
        'what are the best', 'what is the best', 'what are the latest',
        'what are people saying about', 'what do people think about',
        'how do i use', 'how to use', 'how to',
        'what are', 'what is', 'tips for', 'best practices for',
    ]
    for p in prefixes:
        if text.startswith(p + ' '):
            text = text[len(p):].strip()
    # Strip individual noise words
    noise = {
        'best', 'top', 'good', 'great', 'awesome', 'killer',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features',
        'recommendations', 'advice',
        'prompt', 'prompts', 'prompting',
        'methods', 'strategies', 'approaches',
    }
    words = text.split()
    filtered = [w for w in words if w not in noise]
    result = ' '.join(filtered) if filtered else text
    return result.rstrip('?!.')
 def _log(msg: str):
    """Log to stderr (only in interactive terminals; spinner handles non-TTY)."""
    if sys.stderr.isatty():
        sys.stderr.write(f"[TikTok] {msg}\n")
        sys.stderr.flush()
 def _sc_headers(token: str) -> Dict[str, str]:
    """Build ScrapeCreators request headers."""
    return {
        "x-api-key": token,
        "Content-Type": "application/json",
    }
 def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    """Parse date from ScrapeCreators TikTok item to YYYY-MM-DD.
    Handles create_time (unix timestamp).
    """
    ts = item.get("create_time")
    if ts:
        try:
            dt = datetime.fromtimestamp(int(ts), tz=timezone.utc)
            return dt.strftime("%Y-%m-%d")
        except (ValueError, TypeError, OSError):
            pass
    return None
 def _clean_webvtt(text: str) -> str:
    """Strip WebVTT timestamps and headers from transcript text."""
    if not text:
        return ""
    lines = text.split('\n')
    cleaned = []
    for line in lines:
        line = line.strip()
        if not line:
            continue
        if line.startswith('WEBVTT'):
            continue
        if re.match(r'^\d{2}:\d{2}', line):
            continue
        if '-->' in line:
            continue
        cleaned.append(line)
    return ' '.join(cleaned)
 def search_tiktok(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Search TikTok via ScrapeCreators API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
    if not _requests:
        return {"items": [], "error": "requests library not installed"}
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
    _log(f"Searching TikTok for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
    try:
        resp = _requests.get(
            f"{SCRAPECREATORS_BASE}/search/keyword",
            params={"query": core_topic, "sort_by": "relevance"},
            headers=_sc_headers(token),
            timeout=30,
        )
        resp.raise_for_status()
        data = resp.json()
    except Exception as e:
        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}
    # Items are nested under aweme_info
    raw_entries = data.get("search_item_list") or data.get("data") or []
    raw_items = []
    for entry in raw_entries:
        if isinstance(entry, dict):
            info = entry.get("aweme_info", entry)
            raw_items.append(info)
    # Limit to configured count
    raw_items = raw_items[:config["results_per_page"]]
    # Parse items
    items = []
    for raw in raw_items:
        video_id = str(raw.get("aweme_id", ""))
        text = raw.get("desc", "")
        stats = raw.get("statistics") or {}
        play_count = stats.get("play_count") or 0
        digg_count = stats.get("digg_count") or 0
        comment_count = stats.get("comment_count") or 0
        share_count = stats.get("share_count") or 0
        author = raw.get("author") or {}
        author_name = author.get("unique_id", "")
        share_url = raw.get("share_url", "")
        text_extra = raw.get("text_extra") or []
        hashtag_names = [t.get("hashtag_name", "") for t in text_extra
                         if isinstance(t, dict) and t.get("hashtag_name")]
        duration = (raw.get("video") or {}).get("duration")
        date_str = _parse_date(raw)
        # Compute relevance with hashtag boost
        relevance = _compute_relevance(core_topic, text, hashtag_names)
        # Build URL: prefer share_url, fallback to constructed URL
        url = share_url.split("?")[0] if share_url else ""
        if not url and author_name and video_id:
            url = f"https://www.tiktok.com/@{author_name}/video/{video_id}"
        items.append({
            "video_id": video_id,
            "text": text,
            "url": url,
            "author_name": author_name,
            "date": date_str,
            "engagement": {
                "views": play_count,
                "likes": digg_count,
                "comments": comment_count,
                "shares": share_count,
            },
            "hashtags": hashtag_names,
            "duration": duration,
            "relevance": relevance,
            "why_relevant": f"TikTok: {text[:60]}" if text else f"TikTok: {core_topic}",
            "caption_snippet": "",  # populated by fetch_captions
        })
    # Hard date filter
    in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
    out_of_range = len(items) - len(in_range)
    if in_range:
        items = in_range
        if out_of_range:
            _log(f"Filtered {out_of_range} videos outside date range")
    else:
        _log(f"No videos within date range, keeping all {len(items)}")
    # Sort by views descending
    items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
    _log(f"Found {len(items)} TikTok videos")
    return {"items": items}
 def fetch_captions(
    video_items: List[Dict[str, Any]],
    token: str,
    depth: str = "default",
 ) -> Dict[str, str]:
    """Fetch transcripts for top N TikTok videos via ScrapeCreators.
    Strategy:
    1. Use the 'text' field (video description) as baseline caption
    2. For top N, call /video/transcript for spoken-word captions
    Args:
        video_items: Items from search_tiktok()
        token: ScrapeCreators API key
        depth: Depth level for caption limit
    Returns:
        Dict mapping video_id -> caption text (truncated to 500 words)
    """
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    max_captions = config["max_captions"]
    if not video_items or not token or not _requests:
        return {}
    top_items = video_items[:max_captions]
    _log(f"Enriching captions for {len(top_items)} videos")
    captions = {}
    # First pass: use text field as caption (always available, free)
    for item in top_items:
        vid = item["video_id"]
        text = item.get("text", "")
        if text:
            words = text.split()
            if len(words) > CAPTION_MAX_WORDS:
                text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
            captions[vid] = text
    # Second pass: try to get spoken-word transcripts (1 credit each)
    for item in top_items:
        vid = item["video_id"]
        url = item.get("url", "")
        if not url:
            continue
        try:
            resp = _requests.get(
                f"{SCRAPECREATORS_BASE}/video/transcript",
                params={"url": url},
                headers=_sc_headers(token),
                timeout=15,
            )
            if resp.status_code == 200:
                data = resp.json()
                transcript = data.get("transcript")
                if transcript:
                    if isinstance(transcript, list):
                        transcript = " ".join(str(s) for s in transcript)
                    transcript = _clean_webvtt(transcript)
                    if transcript:
                        words = transcript.split()
                        if len(words) > CAPTION_MAX_WORDS:
                            transcript = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
                        captions[vid] = transcript
        except Exception as e:
            _log(f"Transcript fetch failed for {vid}: {e}")
    got = sum(1 for v in captions.values() if v)
    _log(f"Got captions for {got}/{len(top_items)} videos")
    return captions
 def search_and_enrich(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Full TikTok search: find videos, then fetch captions for top results.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list. Each item has a 'caption_snippet' field.
    """
    # Step 1: Search
    search_result = search_tiktok(topic, from_date, to_date, depth, token)
    items = search_result.get("items", [])
    if not items:
        return search_result
    # Step 2: Fetch captions for top N
    captions = fetch_captions(items, token, depth)
    # Step 3: Attach captions to items
    for item in items:
        vid = item["video_id"]
        caption = captions.get(vid)
        if caption:
            item["caption_snippet"] = caption
    return {"items": items, "error": search_result.get("error")}
 def parse_tiktok_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse TikTok search response to normalized format.
    Returns:
        List of item dicts ready for normalization.
    """
    return response.get("items", [])
@@ -1,6 +1,5 @@
 """Terminal UI utilities for last30days skill."""
 import os
 import sys
 import time
 import threading
@@ -72,6 +71,18 @@ YOUTUBE_MESSAGES = [
    "Fetching transcripts...",
 ]
 TIKTOK_MESSAGES = [
    "Searching TikTok for trending videos...",
    "Finding what's viral on TikTok...",
    "Scanning TikTok for relevant content...",
 ]
 INSTAGRAM_MESSAGES = [
    "Searching Instagram Reels...",
    "Finding what's trending on Instagram...",
    "Scanning Instagram for relevant reels...",
 ]
 HN_MESSAGES = [
    "Searching Hacker News...",
    "Scanning HN front page stories...",
@@ -117,7 +128,7 @@ I just researched that for you. Here's what I've got right now:
 {status_line}
-You can unlock more sources with API keys — just ask me how and I'll walk you through it. More sources means better research, but it works fine as-is.
+You can unlock more sources with API keys or by signing in to Codex — just ask me how and I'll walk you through it. More sources means better research, but it works fine as-is.
 Some examples of what you can do:
 - "last30 what are people saying about Figma"
@@ -131,7 +142,7 @@ Just start with "last30" and talk to me like normal.
 # Shorter promo for single missing key
 PROMO_SINGLE_KEY = {
-    "reddit": "\n💡 You can unlock Reddit with an OpenAI API key — just ask me how.\n",
+    "reddit": "\n💡 You can unlock Reddit with an OpenAI API key or by running `codex login` — just ask me how.\n",
    "x": "\n💡 You can unlock X with an xAI API key — just ask me how.\n",
 }
@@ -265,13 +276,31 @@ class ProgressDisplay:
    def start_youtube(self):
        msg = random.choice(YOUTUBE_MESSAGES)
-        self.spinner = Spinner(f"{Colors.RED}YouTube{Colors.RESET} {msg}", Colors.RED, quiet=True)
+        self.spinner = Spinner(f"{Colors.RED}YouTube{Colors.RESET} {msg}", Colors.RED)
        self.spinner.start()
    def end_youtube(self, count: int):
        if self.spinner:
            self.spinner.stop(f"{Colors.RED}YouTube{Colors.RESET} Found {count} videos")
    def start_tiktok(self):
        msg = random.choice(TIKTOK_MESSAGES)
        self.spinner = Spinner(f"{Colors.PURPLE}TikTok{Colors.RESET} {msg}", Colors.PURPLE)
        self.spinner.start()
    def end_tiktok(self, count: int):
        if self.spinner:
            self.spinner.stop(f"{Colors.PURPLE}TikTok{Colors.RESET} Found {count} videos")
    def start_instagram(self):
        msg = random.choice(INSTAGRAM_MESSAGES)
        self.spinner = Spinner(f"{Colors.PURPLE}Instagram{Colors.RESET} {msg}", Colors.PURPLE)
        self.spinner.start()
    def end_instagram(self, count: int):
        if self.spinner:
            self.spinner.stop(f"{Colors.PURPLE}Instagram{Colors.RESET} Found {count} reels")
    def start_hackernews(self):
        msg = random.choice(HN_MESSAGES)
        self.spinner = Spinner(f"{Colors.YELLOW}HN{Colors.RESET} {msg}", Colors.YELLOW, quiet=True)
@@ -299,7 +328,7 @@ class ProgressDisplay:
        if self.spinner:
            self.spinner.stop()
-    def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0):
+    def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0, tiktok_count: int = 0, ig_count: int = 0):
        elapsed = time.time() - self.start_time
        if IS_TTY:
            sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
@@ -308,6 +337,10 @@ class ProgressDisplay:
            sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts")
            if youtube_count:
                sys.stderr.write(f"  {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
            if tiktok_count:
                sys.stderr.write(f"  {Colors.PURPLE}TikTok:{Colors.RESET} {tiktok_count} videos")
            if ig_count:
                sys.stderr.write(f"  {Colors.PURPLE}Instagram:{Colors.RESET} {ig_count} reels")
            if hn_count:
                sys.stderr.write(f"  {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories")
            if pm_count:
@@ -317,6 +350,10 @@ class ProgressDisplay:
            parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"]
            if youtube_count:
                parts.append(f"YouTube: {youtube_count} videos")
            if tiktok_count:
                parts.append(f"TikTok: {tiktok_count} videos")
            if ig_count:
                parts.append(f"Instagram: {ig_count} reels")
            if hn_count:
                parts.append(f"HN: {hn_count} stories")
            if pm_count:
@@ -86,7 +86,8 @@ def cmd_run_one(args):
        print(json.dumps({"error": f'Topic not found: "{args.topic}"'}))
        sys.exit(1)
-    _run_topic(topic)
+    result = _run_topic(topic)
    print(json.dumps(result, default=str))
 def cmd_run_all(args):
@@ -132,11 +133,13 @@ def _run_topic(topic: dict) -> dict:
    run_id = store.record_run(topic_id, source_mode="both", status="running")
    try:
-        # Run the research script
+        # Prefer custom search_queries over topic name (#40)
        search_queries = json.loads(topic["search_queries"]) if topic.get("search_queries") else None
        search_term = search_queries[0] if search_queries else topic["name"]
        cmd = [
            sys.executable,
            str(SCRIPT_DIR / "last30days.py"),
-            topic["name"],
+            search_term,
            "--emit=json",
        ]
        result = subprocess.run(
@@ -188,6 +191,36 @@ def _run_topic(topic: dict) -> dict:
                "engagement_score": (item.get("engagement") or {}).get("likes", 0),
                "relevance_score": item.get("relevance", 0),
            })
        for item in data.get("youtube", []):
            findings.append({
                "source": "youtube",
                "url": item.get("url", ""),
                "title": item.get("title", ""),
                "author": item.get("channel_name", item.get("channel", "")),
                "content": item.get("transcript_snippet", "") or item.get("title", ""),
                "engagement_score": (item.get("engagement") or {}).get("views", 0),
                "relevance_score": item.get("relevance", 0),
            })
        for item in data.get("tiktok", []):
            findings.append({
                "source": "tiktok",
                "url": item.get("url", ""),
                "title": (item.get("caption_snippet", "") or "")[:120],
                "author": item.get("author", ""),
                "content": item.get("caption_snippet", ""),
                "engagement_score": (item.get("engagement") or {}).get("views", 0),
                "relevance_score": item.get("relevance", 0),
            })
        for item in data.get("instagram", []):
            findings.append({
                "source": "instagram",
                "url": item.get("url", ""),
                "title": (item.get("caption_snippet", "") or "")[:120],
                "author": item.get("author_name", ""),
                "content": item.get("caption_snippet", ""),
                "engagement_score": (item.get("engagement") or {}).get("views", 0),
                "relevance_score": item.get("relevance", 0),
            })
        # Store with dedup
        counts = store.store_findings(run_id, topic_id, findings)
@@ -0,0 +1,57 @@
 """Tests for bird_x module."""
 import sys
 import unittest
 from pathlib import Path
 from unittest import mock
 sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
 from lib import bird_x
 class TestExtractCoreSubject(unittest.TestCase):
    def test_strips_trending_noise(self):
        result = bird_x._extract_core_subject("trendiest Claude Code skills")
        self.assertNotIn("trendiest", result)
        self.assertIn("claude", result.lower())
    def test_strips_tool_noise(self):
        result = bird_x._extract_core_subject("best AI tools for coding")
        self.assertNotIn("tools", result)
        self.assertNotIn("best", result)
    def test_strips_skill_noise(self):
        result = bird_x._extract_core_subject("top claude code skills")
        self.assertNotIn("skills", result)
        self.assertNotIn("top", result)
 class TestBirdSearchRetries(unittest.TestCase):
    def test_last_chance_retry_uses_strongest_token(self):
        """When shorter retry also returns 0, uses longest non-noise token."""
        empty = {"items": []}
        with mock.patch.object(bird_x, "_extract_core_subject", return_value="best codex skill plugin"), \
             mock.patch.object(bird_x, "parse_bird_response", return_value=[]), \
             mock.patch.object(bird_x, "_run_bird_search", return_value=empty) as run_mock:
            bird_x.search_x("best codex skill plugin", "2026-01-01", "2026-01-31", depth="quick")
        # Should try: original, shorter (2-word), last-chance (strongest token)
        self.assertEqual(run_mock.call_count, 3)
        queries = [call.args[0] for call in run_mock.call_args_list]
        # Last call should use "codex" (longest non-noise word)
        self.assertIn("codex", queries[2])
    def test_no_retry_when_first_query_has_results(self):
        """No retry when first query succeeds."""
        result = {"items": [{"id": "1"}]}
        with mock.patch.object(bird_x, "_extract_core_subject", return_value="nano banana"), \
             mock.patch.object(bird_x, "parse_bird_response", return_value=[{"id": "1"}]), \
             mock.patch.object(bird_x, "_run_bird_search", return_value=result) as run_mock:
            bird_x.search_x("nano banana prompting", "2026-01-01", "2026-01-31")
        self.assertEqual(run_mock.call_count, 1)
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,200 @@
 """Tests for Codex auth integration (env.py + openai_reddit.py)."""
 import base64
 import json
 import os
 import sys
 import time
 import unittest
 from pathlib import Path
 from unittest.mock import patch
 # Add scripts directory to path
 sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
 from lib import env, openai_reddit
 def _make_jwt(payload: dict) -> str:
    """Build a fake JWT with the given payload (no signature verification)."""
    header = base64.urlsafe_b64encode(json.dumps({"alg": "none"}).encode()).rstrip(b"=")
    body = base64.urlsafe_b64encode(json.dumps(payload).encode()).rstrip(b"=")
    return f"{header.decode()}.{body.decode()}.fakesig"
 class TestDecodeJwtPayload(unittest.TestCase):
    def test_valid_jwt(self):
        token = _make_jwt({"sub": "user123", "exp": 9999999999})
        result = env._decode_jwt_payload(token)
        self.assertEqual(result["sub"], "user123")
    def test_invalid_jwt(self):
        self.assertIsNone(env._decode_jwt_payload("not-a-jwt"))
    def test_empty_string(self):
        self.assertIsNone(env._decode_jwt_payload(""))
 class TestTokenExpired(unittest.TestCase):
    def test_not_expired(self):
        token = _make_jwt({"exp": int(time.time()) + 3600})
        self.assertFalse(env._token_expired(token))
    def test_expired(self):
        token = _make_jwt({"exp": int(time.time()) - 100})
        self.assertTrue(env._token_expired(token))
    def test_no_exp_claim(self):
        token = _make_jwt({"sub": "user"})
        self.assertFalse(env._token_expired(token))
 class TestExtractChatgptAccountId(unittest.TestCase):
    def test_extracts_account_id(self):
        token = _make_jwt({
            "https://api.openai.com/auth": {
                "chatgpt_account_id": "acct_abc123"
            }
        })
        self.assertEqual(env.extract_chatgpt_account_id(token), "acct_abc123")
    def test_missing_auth_claim(self):
        token = _make_jwt({"sub": "user"})
        self.assertIsNone(env.extract_chatgpt_account_id(token))
    def test_missing_account_id_in_claim(self):
        token = _make_jwt({
            "https://api.openai.com/auth": {"other_field": "value"}
        })
        self.assertIsNone(env.extract_chatgpt_account_id(token))
 class TestGetOpenaiAuth(unittest.TestCase):
    @patch.dict(os.environ, {}, clear=True)
    def test_api_key_takes_priority(self):
        """OPENAI_API_KEY in env file is used when env var is not set."""
        file_env = {"OPENAI_API_KEY": "sk-test123"}
        auth = env.get_openai_auth(file_env)
        self.assertEqual(auth.source, "api_key")
        self.assertEqual(auth.status, "ok")
        self.assertEqual(auth.token, "sk-test123")
        self.assertIsNone(auth.account_id)
    @patch.dict(os.environ, {"OPENAI_API_KEY": "sk-from-env"}, clear=False)
    def test_env_var_takes_priority(self):
        """OPENAI_API_KEY env var should be preferred over file."""
        file_env = {}
        auth = env.get_openai_auth(file_env)
        self.assertEqual(auth.source, "api_key")
        self.assertEqual(auth.token, "sk-from-env")
    def test_no_keys_returns_none_source(self):
        """No API key and no Codex auth → source=none."""
        fake_path = Path("/tmp/nonexistent_codex_auth_test.json")
        with patch.object(env, 'CODEX_AUTH_FILE', fake_path):
            # Also patch get_codex_access_token to avoid reading real auth file
            with patch.object(env, 'get_codex_access_token', return_value=(None, "missing")):
                environ_copy = {k: v for k, v in os.environ.items() if k != "OPENAI_API_KEY"}
                with patch.dict(os.environ, environ_copy, clear=True):
                    auth = env.get_openai_auth({})
                    self.assertEqual(auth.source, "none")
                    self.assertIsNone(auth.token)
 class TestLoadCodexAuth(unittest.TestCase):
    def test_nonexistent_file(self):
        result = env.load_codex_auth(Path("/tmp/nonexistent_codex_auth.json"))
        self.assertEqual(result, {})
    def test_valid_json(self):
        import tempfile
        data = {"tokens": {"access_token": "tok123"}}
        with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f:
            json.dump(data, f)
            f.flush()
            result = env.load_codex_auth(Path(f.name))
        os.unlink(f.name)
        self.assertEqual(result["tokens"]["access_token"], "tok123")
 class TestGetAvailableSourcesWithAuth(unittest.TestCase):
    def test_codex_auth_ok_counts_as_openai(self):
        config = {
            "OPENAI_API_KEY": "codex-token",
            "OPENAI_AUTH_STATUS": "ok",
            "XAI_API_KEY": None,
        }
        result = env.get_available_sources(config)
        self.assertIn("reddit", result)
    def test_codex_auth_expired_not_counted(self):
        config = {
            "OPENAI_API_KEY": None,
            "OPENAI_AUTH_STATUS": "expired",
            "XAI_API_KEY": None,
        }
        result = env.get_available_sources(config)
        self.assertEqual(result, "web")
 class TestParseCodexStream(unittest.TestCase):
    def test_response_completed_event(self):
        """Should extract response from response.completed SSE event."""
        sse = (
            'data: {"type":"response.created","response":{"id":"r1"}}\n\n'
            'data: {"type":"response.completed","response":{"id":"r1","output":[{"type":"message","content":[{"type":"output_text","text":"hello"}]}]}}\n\n'
        )
        result = openai_reddit._parse_codex_stream(sse)
        self.assertIn("output", result)
    def test_delta_fallback(self):
        """Should reconstruct text from delta events."""
        sse = (
            'data: {"delta":"hel"}\n\n'
            'data: {"delta":"lo"}\n\n'
        )
        result = openai_reddit._parse_codex_stream(sse)
        self.assertIn("output", result)
        text = result["output"][0]["content"][0]["text"]
        self.assertEqual(text, "hello")
    def test_empty_stream(self):
        result = openai_reddit._parse_codex_stream("")
        self.assertEqual(result, {})
 class TestBuildPayload(unittest.TestCase):
    def test_api_key_payload(self):
        payload = openai_reddit._build_payload(
            "gpt-4o", "instructions", "input text", "api_key"
        )
        self.assertEqual(payload["model"], "gpt-4o")
        self.assertEqual(payload["input"], "input text")
        self.assertNotIn("stream", payload)
    def test_codex_payload_has_stream(self):
        payload = openai_reddit._build_payload(
            "gpt-4o", "instructions", "input text", env.AUTH_SOURCE_CODEX
        )
        self.assertTrue(payload["stream"])
        # Input should be structured message format for Codex
        self.assertIsInstance(payload["input"], list)
        self.assertEqual(payload["input"][0]["role"], "user")
    def test_codex_payload_has_store_false(self):
        payload = openai_reddit._build_payload(
            "gpt-4o", "inst", "text", env.AUTH_SOURCE_CODEX
        )
        self.assertFalse(payload["store"])
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,268 @@
 """Tests for TikTok module (search, normalize, score, dedupe, render)."""
 import json
 import sys
 import unittest
 from pathlib import Path
 # Add lib to path
 sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
 from lib import schema, score, normalize, dedupe, render
 from lib import tiktok
 class TestTikTokRelevance(unittest.TestCase):
    """Test relevance scoring for TikTok items."""
    def test_exact_match(self):
        rel = tiktok._compute_relevance("claude code", "Claude Code tricks and tips")
        self.assertGreaterEqual(rel, 0.8)
    def test_partial_match(self):
        rel = tiktok._compute_relevance("claude code tips", "Best AI tools for coding")
        self.assertLess(rel, 0.5)
    def test_hashtag_boost(self):
        """Hashtags should boost relevance."""
        rel_no_hash = tiktok._compute_relevance("claude code", "random video about stuff")
        rel_with_hash = tiktok._compute_relevance("claude code", "random video about stuff", ["claudecode", "ai"])
        self.assertGreater(rel_with_hash, rel_no_hash)
    def test_empty_query(self):
        rel = tiktok._compute_relevance("", "Some video title")
        self.assertEqual(rel, 0.5)
    def test_floor(self):
        rel = tiktok._compute_relevance("quantum physics", "cat dancing video")
        self.assertGreaterEqual(rel, 0.1)
 class TestExtractCoreSubject(unittest.TestCase):
    """Test core subject extraction for TikTok search."""
    def test_strips_prefix(self):
        result = tiktok._extract_core_subject("what are the best claude code tips")
        self.assertNotIn("what are the best", result)
        self.assertIn("claude", result)
    def test_strips_noise(self):
        result = tiktok._extract_core_subject("latest trending updates on React")
        self.assertNotIn("latest", result)
        self.assertNotIn("trending", result)
        self.assertIn("react", result.lower())
    def test_preserves_core(self):
        result = tiktok._extract_core_subject("Claude Code")
        self.assertEqual(result, "claude code")
 class TestParseDate(unittest.TestCase):
    """Test date parsing from ScrapeCreators items."""
    def test_unix_timestamp(self):
        item = {"create_time": 1756403075}
        result = tiktok._parse_date(item)
        self.assertIsNotNone(result)
        self.assertRegex(result, r"\d{4}-\d{2}-\d{2}")
    def test_no_date(self):
        item = {}
        self.assertIsNone(tiktok._parse_date(item))
 class TestCleanWebVTT(unittest.TestCase):
    """Test WebVTT transcript cleaning."""
    def test_strips_timestamps(self):
        raw = "WEBVTT\n\n00:00:00.000 --> 00:00:02.000\nHello world\n\n00:00:02.000 --> 00:00:04.000\nGoodbye"
        result = tiktok._clean_webvtt(raw)
        self.assertEqual(result, "Hello world Goodbye")
    def test_empty_input(self):
        self.assertEqual(tiktok._clean_webvtt(""), "")
        self.assertEqual(tiktok._clean_webvtt(None), "")
 class TestNormalizeTikTokItems(unittest.TestCase):
    """Test TikTok normalization."""
    def setUp(self):
        self.fixtures_dir = Path(__file__).parent.parent / "fixtures"
        with open(self.fixtures_dir / "tiktok_search.json") as f:
            data = json.load(f)
        self.raw_items = data["items"]
    def test_normalizes_items(self):
        items = normalize.normalize_tiktok_items(self.raw_items, "2026-02-01", "2026-03-03")
        self.assertEqual(len(items), 3)
        self.assertIsInstance(items[0], schema.TikTokItem)
    def test_ids_are_sequential(self):
        items = normalize.normalize_tiktok_items(self.raw_items, "2026-02-01", "2026-03-03")
        self.assertEqual(items[0].id, "TK1")
        self.assertEqual(items[1].id, "TK2")
        self.assertEqual(items[2].id, "TK3")
    def test_engagement_parsed(self):
        items = normalize.normalize_tiktok_items(self.raw_items, "2026-02-01", "2026-03-03")
        eng = items[0].engagement
        self.assertIsNotNone(eng)
        self.assertEqual(eng.views, 2100000)
        self.assertEqual(eng.likes, 45000)
        self.assertEqual(eng.shares, 8400)
    def test_hashtags_preserved(self):
        items = normalize.normalize_tiktok_items(self.raw_items, "2026-02-01", "2026-03-03")
        self.assertEqual(items[0].hashtags, ["claudecode", "ai", "coding"])
    def test_caption_snippet_preserved(self):
        items = normalize.normalize_tiktok_items(self.raw_items, "2026-02-01", "2026-03-03")
        self.assertIn("slash commands", items[0].caption_snippet)
 class TestScoreTikTokItems(unittest.TestCase):
    """Test TikTok scoring."""
    def test_engagement_scoring(self):
        eng = schema.Engagement(views=1000000, likes=50000, num_comments=2000)
        raw = score.compute_tiktok_engagement_raw(eng)
        self.assertIsNotNone(raw)
        self.assertGreater(raw, 0)
    def test_none_engagement(self):
        raw = score.compute_tiktok_engagement_raw(None)
        self.assertIsNone(raw)
    def test_empty_engagement(self):
        eng = schema.Engagement()
        raw = score.compute_tiktok_engagement_raw(eng)
        self.assertIsNone(raw)
    def test_scoring_pipeline(self):
        items = [
            schema.TikTokItem(
                id="TK1", text="High views video", url="https://tiktok.com/1",
                author_name="creator1", date="2026-03-01",
                engagement=schema.Engagement(views=2000000, likes=50000, num_comments=1000),
                relevance=0.9,
            ),
            schema.TikTokItem(
                id="TK2", text="Low views video", url="https://tiktok.com/2",
                author_name="creator2", date="2026-02-20",
                engagement=schema.Engagement(views=1000, likes=50, num_comments=5),
                relevance=0.5,
            ),
        ]
        scored = score.score_tiktok_items(items)
        self.assertEqual(len(scored), 2)
        self.assertGreater(scored[0].score, 0)
        self.assertGreater(scored[0].score, scored[1].score)
 class TestDedupeTikTok(unittest.TestCase):
    """Test TikTok deduplication."""
    def test_no_dupes(self):
        items = [
            schema.TikTokItem(id="TK1", text="Totally different video A",
                              url="https://tiktok.com/1", author_name="a", score=80),
            schema.TikTokItem(id="TK2", text="Completely unique video B",
                              url="https://tiktok.com/2", author_name="b", score=70),
        ]
        result = dedupe.dedupe_tiktok(items)
        self.assertEqual(len(result), 2)
    def test_removes_dupes(self):
        items = [
            schema.TikTokItem(id="TK1", text="Claude Code is amazing for AI coding",
                              url="https://tiktok.com/1", author_name="a", score=80),
            schema.TikTokItem(id="TK2", text="Claude Code is amazing for AI coding wow",
                              url="https://tiktok.com/2", author_name="a", score=60),
        ]
        result = dedupe.dedupe_tiktok(items)
        self.assertEqual(len(result), 1)
        self.assertEqual(result[0].id, "TK1")  # Higher score kept
 class TestRenderTikTok(unittest.TestCase):
    """Test TikTok rendering in reports."""
    def test_renders_tiktok_section(self):
        report = schema.Report(
            topic="test", range_from="2026-02-01", range_to="2026-03-03",
            generated_at="2026-03-03T00:00:00Z", mode="all",
            tiktok=[
                schema.TikTokItem(
                    id="TK1", text="Video caption here", url="https://tiktok.com/1",
                    author_name="creator", date="2026-03-01", score=85,
                    engagement=schema.Engagement(views=1000000, likes=50000),
                    hashtags=["ai", "coding"],
                    why_relevant="TikTok: Video caption here",
                ),
            ],
        )
        output = render.render_compact(report)
        self.assertIn("### TikTok Videos", output)
        self.assertIn("TK1", output)
        self.assertIn("@creator", output)
        self.assertIn("1,000,000 views", output)
    def test_renders_source_status(self):
        report = schema.Report(
            topic="test", range_from="2026-02-01", range_to="2026-03-03",
            generated_at="2026-03-03T00:00:00Z", mode="all",
            tiktok=[
                schema.TikTokItem(
                    id="TK1", text="test", url="https://tiktok.com/1",
                    author_name="creator", caption_snippet="some caption",
                ),
            ],
        )
        status = render.render_source_status(report)
        self.assertIn("TikTok", status)
        self.assertIn("1 videos", status)
    def test_xref_tag_tiktok(self):
        """Test that TK prefix is recognized in cross-ref tags."""
        item = schema.RedditItem(id="R1", title="test", url="test", subreddit="test",
                                  cross_refs=["TK1"])
        tag = render._xref_tag(item)
        self.assertIn("TikTok", tag)
 class TestSchemaRoundtrip(unittest.TestCase):
    """Test TikTokItem serialization round-trip via Report."""
    def test_to_dict_and_back(self):
        original = schema.TikTokItem(
            id="TK1", text="Test caption", url="https://tiktok.com/1",
            author_name="creator", date="2026-03-01",
            date_confidence="high",
            engagement=schema.Engagement(views=100, likes=10, num_comments=5, shares=3),
            caption_snippet="spoken words",
            hashtags=["test", "ai"],
            relevance=0.8, why_relevant="TikTok: Test",
            subs=schema.SubScores(relevance=80, recency=90, engagement=70),
            score=80, cross_refs=["R1"],
        )
        report = schema.Report(
            topic="test", range_from="2026-02-01", range_to="2026-03-03",
            generated_at="2026-03-03T00:00:00Z", mode="all",
            tiktok=[original],
        )
        d = report.to_dict()
        restored = schema.Report.from_dict(d)
        self.assertEqual(len(restored.tiktok), 1)
        tk = restored.tiktok[0]
        self.assertEqual(tk.id, "TK1")
        self.assertEqual(tk.author_name, "creator")
        self.assertEqual(tk.hashtags, ["test", "ai"])
        self.assertEqual(tk.engagement.views, 100)
        self.assertEqual(tk.engagement.shares, 3)
        self.assertEqual(tk.caption_snippet, "spoken words")
        self.assertEqual(tk.cross_refs, ["R1"])
 if __name__ == "__main__":
    unittest.main()
Author	SHA1	Message	Date
Matt Van Horn	4d35b53eab	docs: v2.9.0 release — ScrapeCreators Reddit default, top comments, smart discovery Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 18:03:40 -08:00
Matt Van Horn	2247800003	chore: clean up Reddit log prefix, mark plan tasks complete	2026-03-05 18:01:24 -08:00
Matt Van Horn	7048fe7b83	feat(reddit): elevate top comments, improve subreddit discovery, default to ScrapeCreators Three improvements from beta testing: 1. Top comments: 10% scoring weight for comment quality, 💬 top comment rendered prominently in compact/full output, increased insight limits 2. Subreddit discovery: relevance-weighted scoring with topic word matching, utility sub penalties (UTILITY_SUBS blocklist), engagement bonus 3. Default method: SKILL.md primaryEnv → SCRAPECREATORS_API_KEY, web-only banner recommends SC first, security section updated Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 17:29:18 -08:00
Matt Van Horn	30b973f62e	docs: add Reddit ScrapeCreators v2 improvements plan Three focused improvements based on 5 full-pipeline beta tests: 1. Elevate top Reddit comments in scoring and rendering 2. Improve subreddit discovery heuristic for ambiguous queries 3. Make ScrapeCreators the default recommended Reddit method Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 17:24:41 -08:00
Matt Van Horn	09b09946c0	feat: replace OpenAI Reddit search with ScrapeCreators API - New scripts/lib/reddit.py: multi-query expansion, global search, subreddit discovery, targeted subreddit search, comment enrichment - 68 results in 17s vs ~15 results in 60-90s (OpenAI) - Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper) - Real engagement data (score, comments, dates) from API - No more 429 rate limits on comment enrichment - Falls back to OpenAI if SCRAPECREATORS_API_KEY missing - Registered as last30daysbeta for parallel local testing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 15:55:02 -08:00
Matt Van Horn	db75f9e341	feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration Add Instagram Reels as the 8th research source via ScrapeCreators API. One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram. - Add scripts/lib/instagram.py: keyword search, transcript extraction, relevance scoring, engagement metrics (views, likes, comments) - Add InstagramItem to schema, normalization, scoring, dedup, rendering - Add Instagram to orchestrator pipeline, watchlist, and UI spinners - Update SKILL.md: stats template, citation priority, item format, URL-to-name extraction rules, anti-Sources instruction - Update README and CHANGELOG for v2.8 - Fix: Instagram/TikTok not running in --search= web-only path - Fix: web stats line showing full URLs instead of domain names - Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 07:00:51 -08:00
Matt Van Horn	740dcc5789	docs: update README and SKILL.md for ScrapeCreators TikTok API Replace all Apify references with ScrapeCreators. Key points: - No subscription required (was $5/mo with Apify) - 100 free credits, pay-as-you-go after - SCRAPECREATORS_API_KEY replaces APIFY_API_TOKEN - Backwards compatible: APIFY_API_TOKEN still works as fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-04 05:17:30 -08:00
Matt Van Horn	e03046bd49	refactor(tiktok): replace Apify with ScrapeCreators API Root cause of empty TikTok results: Apify required monthly subscription. ScrapeCreators is PAYG with 100 free credits and no subscription. Key fix: ScrapeCreators nests items under aweme_info wrapper (search_item_list[].aweme_info.{fields}), which the previous implementation missed, causing all fields to be empty. Changes: - Rewrite tiktok.py to use ScrapeCreators REST API - Add aweme_info unwrapping for correct field extraction - Add transcript fetching via /video/transcript endpoint - Add SCRAPECREATORS_API_KEY to env.py config - Update last30days.py to use env.get_tiktok_token() - Delete apify_client_wrapper.py (no longer needed) - Update tests for new date field format (create_time) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 13:58:51 -08:00
Matt Van Horn	1d18bee1a2	fix(skill): forward CLI flags through $ARGUMENTS to Python script Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits the expansion, and change argparse topic from nargs="?" to nargs="*" so multi-word topics still work. Also document --store, --include-web, --diagnose, and --timeout flags in the Options section. Closes #36 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 13:52:08 -08:00
Matt Van Horn	fb00856bff	docs: update README for v2.7 with TikTok examples and installation Add TikTok as 7th source throughout README: new V2.7 banner, real search examples (Iran Israel: 61.6M views, Leah Halton: 152.6M views), APIFY_API_TOKEN in installation, Apify in security table, fix stale "six sources" references to "seven sources". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:52:51 -08:00
Matt Van Horn	d7b354b2cf	fix(ui): suppress [TikTok] and [Apify] log lines in non-TTY mode Only print debug log lines when running in an interactive terminal. In Claude Code (non-TTY), the spinner system handles progress display, so these raw log lines just add noise. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:45:00 -08:00
Matt Van Horn	7c5763d048	fix(apify): suppress verbose actor log streaming to stderr Pass logger=None to Apify .call() to prevent the SDK from streaming raw actor run logs (status messages, crawler stats, warnings) that drown out the clean spinner UI in Claude Code. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:43:38 -08:00
Matt Van Horn	61729b9ae7	fix(ui): show YouTube and TikTok progress spinners in Claude Code Remove quiet=True from YouTube and TikTok spinners so they display the same colored emoji progress lines as Reddit and X in non-TTY mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:36:50 -08:00
Matt Van Horn	b990aed40e	feat: add --no-native-web flag to skip Parallel AI in Claude Code When running in Claude Code, the assistant has a built-in WebSearch tool that's free and higher quality than Parallel AI/Brave/OpenRouter. Adding --no-native-web to the SKILL.md invocation defers web search to the assistant, saving API credits. OpenClaw invocations don't pass this flag, so they continue using native web backends. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:34:03 -08:00
Matt Van Horn	d4ac57f041	fix(tiktok): restore missing websearch import in orchestrator The websearch module import was dropped when the tiktok import was added, causing the script to crash during the rendering phase after all data was successfully collected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:25:30 -08:00
Matt Van Horn	1db0b6054a	feat(tiktok): add TikTok as 7th signal source via Apify Add TikTok search, scoring, and rendering using the Apify platform (clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN ($5/month free credits, no CC required). The shared apify_client_wrapper module is designed for reuse by future Facebook/Instagram sources. - New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py - Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok - Pipeline: normalize → filter → score → sort → dedupe → cross-link → render - Scoring: 0.50log1p(views) + 0.30log1p(likes) + 0.20*log1p(comments) - SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs - 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:08:19 -08:00
Matt Van Horn	5e5d586f7d	fix: triage all 16 open GitHub issues — close 9, fix 6, comment 1 Batch 1 (closed): #43 spam, #34 dup, #19 resolved, #2 resolved, #41 answered Batch 2: Added MIT LICENSE file (#35), closed #42 (license question) Batch 3 code fixes: - #29: YouTube skip reason shows "0 results" instead of "not installed" - #30: Bird source mapping handles reddit-web + Bird combo - #39: watchlist.py extracts YouTube + TikTok findings, run-one prints output - #40: watchlist.py uses search_queries field when available Batch 4: - #32: marketplace.json source "." → "./" with $schema ref - #36: commented with investigation plan ($ARGUMENTS forwarding) - #4: Added SSL troubleshooting section to README Also commented on #22 (Bird features) and #31 (skills.sh audit). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 06:06:55 -08:00
Matt Van Horn	94b6b6eb7b	feat(search): add --search flag for source filtering Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped shape what we built in v2.5. Cherry-picks the source-filtering concept as a clean implementation against our existing architecture. --search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web Example: --search reddit,hn (run only Reddit + Hacker News) Also: - bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry - render: show xAI tip for reddit-only mode regardless of missing_keys value - tests: new test_bird_x.py (5 tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 00:43:33 -08:00
Matt Van Horn	6ae4b16791	feat(bird_x): add noise words + last-chance retry with strongest token Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool noise words to _extract_core_subject, and a last-chance retry that falls back to the longest non-noise token when 2-word retry also returns 0 results. cache.py and render.py env overrides were already on main. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-03 00:39:52 -08:00
Matt Van Horn	82efa6100b	fix(skill): use plain source names in Web stats line, not URLs URLs in markdown links wrap badly in terminals (discovered after first fix attempt). Change to plain names like "Newsweek, Sportskeeda, Medium" on the Web: stats line. Update citation note to explain the reason. Tested on Dor Brothers, Kanye West, Logan Paul - no trailing Sources: block appeared in any of the three test runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-02 23:54:06 -08:00
Matt Van Horn	a52ed30109	fix(skill): suppress trailing Sources: block from WebSearch tool mandate The WebSearch tool has a system-level mandate to append a Sources: section at the end of every response. SKILL.md's old "DO NOT output Sources: list" instruction was too weak to override it. Fix: redirect citations into the stats block's Web: line as inline links. The WebSearch citation requirement is satisfied there; an explicit note after the stats block tells the model not to append a separate trailing section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-02 23:44:55 -08:00
Matt Van Horn	78678e3919	chore: add .gitignore and PR #37 finalization plan - .gitignore: protect docs/comparison-results/ and other private benchmark artifacts from accidental upstream push - docs/plans: add plan for PR #37 Codex auth finalization Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-02 23:44:55 -08:00
Matt Van Horn	04bfb5381d	fix(tests): patch env isolation in test_api_key_takes_priority The test was picking up the real OPENAI_API_KEY from the shell environment, causing it to fail on any machine with that key set. Added @patch.dict(os.environ, {}, clear=True) so the test runs in a clean env and exercises the file_env path as intended. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-02 23:36:37 -08:00
Ilia Alshanetsky	d7bff81757	fix(bird_x): pass .env credentials to Node subprocesses for WSL2/headless auth * chore: fix YAML error in argument-hint * add codex auth support to responses API * Use gpt-5.1-codex-mini as default model for Codex auth Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for Codex endpoint which doesn't support standard OpenAI models like gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex search path. Also adds test_codex_auth.py with 22 unit tests covering JWT decoding, auth resolution, SSE parsing, and payload building. * Pass .env credentials to Bird Node subprocesses for X auth On platforms without browser cookie access (e.g. WSL2), Bird's vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox or Chrome cookie stores. The .env config file already supports these values, but they were only loaded into the Python config dict — never exported to the environment of Node subprocesses. - Add AUTH_TOKEN/CT0 to env.py config key loading - Add set_credentials()/\_subprocess_env() to bird_x.py to inject credentials into the env dict passed to subprocess.run/Popen - Call set_credentials() in main() before Bird auth detection --------- Co-authored-by: Justin Williams <jblwilliams@gmail.com>	2026-03-02 23:24:59 -08:00