sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources

Browse Source 
v3 rewrites the search engine from the ground up:

- Intelligent pre-research: resolves X handles, GitHub repos, subreddits,
  TikTok hashtags, and YouTube channels before searching
- GitHub person-mode: PR velocity, top repos by stars, release notes
- GitHub project-mode: live star counts, README, releases, top issues
- ELI5 mode: plain language synthesis, no jargon
- 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket,
  GitHub, Threads, Pinterest, Perplexity, Bluesky, Web
- Free Reddit comments via public JSON (no API key needed)
- Fun judge v2: humor scoring baked into narrative
- Cookie consent before browser scanning
- 10,000 free ScrapeCreators calls
- 1,012 tests

Thank you to the community contributors whose issues and PRs shaped v3:
@uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116),
@thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78),
@zl190 (#115), @hnshah (#84, #85, #86)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-04-08 10:52:23 -07:00
parent 61904b31e3
commit 0a9ff16dfc
397 changed files with 21427 additions and 53106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-02-03-bird-cli-implementation.md
-929
View File
@@ -1,929 +0,0 @@
# Bird CLI Integration Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add Bird CLI as a free, zero-config alternative to xAI for X/Twitter searches with interactive installation.
**Architecture:** New `bird_x.py` module handles Bird detection, installation prompts, and search. Modified `env.py` determines X source priority (Bird → xAI → WebSearch). Main script prompts for Bird install if not found.
**Tech Stack:** Python 3, subprocess for Bird CLI calls, existing lib modules for normalization/scoring.
---
### Task 1: Create bird_x.py - Detection Functions
**Files:**
- Create: `scripts/lib/bird_x.py`
**Step 1: Create the module with detection functions**
```python
"""Bird CLI client for X (Twitter) search."""
import json
import shutil
import subprocess
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[Bird] {msg}\n")
sys.stderr.flush()
def is_bird_installed() -> bool:
"""Check if Bird CLI is installed."""
return shutil.which("bird") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if Bird is authenticated by running 'bird whoami'.
Returns:
Username if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
try:
result = subprocess.run(
["bird", "whoami"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0 and result.stdout.strip():
# Output is typically the username
return result.stdout.strip().split('\n')[0]
return None
except (subprocess.TimeoutExpired, FileNotFoundError, Exception):
return None
def check_npm_available() -> bool:
"""Check if npm is available for installation."""
return shutil.which("npm") is not None
```
**Step 2: Verify the module loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add detection functions for Bird CLI"
```
---
### Task 2: Add Bird Installation Functions
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add installation function**
Add after `check_npm_available()`:
```python
def install_bird() -> Tuple[bool, str]:
"""Install Bird CLI via npm.
Returns:
Tuple of (success, message).
"""
if not check_npm_available():
return False, "npm not found. Install Node.js first, or install Bird manually: https://github.com/steipete/bird"
try:
_log("Installing Bird CLI...")
result = subprocess.run(
["npm", "install", "-g", "@steipete/bird"],
capture_output=True,
text=True,
timeout=120,
)
if result.returncode == 0:
return True, "Bird CLI installed successfully!"
else:
error = result.stderr.strip() or result.stdout.strip() or "Unknown error"
return False, f"Installation failed: {error}"
except subprocess.TimeoutExpired:
return False, "Installation timed out"
except Exception as e:
return False, f"Installation error: {e}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
username = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": username is not None,
"username": username,
"can_install": check_npm_available(),
}
```
**Step 2: Verify functions work**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.get_bird_status())"`
Expected: Dict with installed/authenticated status
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add installation and status functions"
```
---
### Task 3: Add Bird Search Function
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add depth config and search function**
Add after imports at top:
```python
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
```
Add after `get_bird_status()`:
```python
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
# Build command
cmd = [
"bird", "search",
topic,
"--since", from_date,
"-n", str(count),
"--json",
]
# Adjust timeout based on depth
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
)
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []}
# Parse JSON output
output = result.stdout.strip()
if not output:
return {"items": []}
return json.loads(output)
except subprocess.TimeoutExpired:
return {"error": "Search timed out", "items": []}
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
```
**Step 2: Verify search function signature**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; import inspect; print(inspect.signature(bird_x.search_x))"`
Expected: `(topic: str, from_date: str, to_date: str, depth: str = 'default') -> Dict[str, Any]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add search_x function"
```
---
### Task 4: Add Bird Response Parser
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add parse function**
Add at end of file:
```python
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
screen_name = tweet.get("user", {}).get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first
if "T" in created_at:
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info
user = tweet.get("user", {})
author_handle = user.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict
engagement = {
"likes": tweet.get("like_count") or tweet.get("favorite_count"),
"reposts": tweet.get("retweet_count"),
"replies": tweet.get("reply_count"),
"quotes": tweet.get("quote_count"),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
}
items.append(item)
return items
```
**Step 2: Verify parser handles empty input**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.parse_bird_response({}))"`
Expected: `[]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add response parser matching xai_x format"
```
---
### Task 5: Add UI Functions for Bird Prompts
**Files:**
- Modify: `scripts/lib/ui.py`
**Step 1: Add Bird-related messages and prompts**
Add after `PROMO_SINGLE_KEY_PLAIN` dict (around line 128):
```python
# Bird CLI prompts
BIRD_INSTALL_PROMPT = f"""
{Colors.CYAN}{Colors.BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━{Colors.RESET}
{Colors.CYAN}🐦 FREE X/TWITTER SEARCH AVAILABLE{Colors.RESET}
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_INSTALL_PROMPT_PLAIN = """
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🐦 FREE X/TWITTER SEARCH AVAILABLE
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_AUTH_HELP = f"""
{Colors.YELLOW}Bird authentication failed.{Colors.RESET}
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: {Colors.BOLD}bird check{Colors.RESET} to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
BIRD_AUTH_HELP_PLAIN = """
Bird authentication failed.
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: bird check to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
```
**Step 2: Add prompt functions to ProgressDisplay class**
Add these methods to the `ProgressDisplay` class (after `show_promo` method, around line 310):
```python
def prompt_bird_install(self) -> bool:
"""Prompt user to install Bird CLI.
Returns:
True if user wants to install, False otherwise.
"""
if IS_TTY:
sys.stderr.write(BIRD_INSTALL_PROMPT)
else:
sys.stderr.write(BIRD_INSTALL_PROMPT_PLAIN)
sys.stderr.flush()
try:
response = input("Install Bird CLI now? (y/n): ").strip().lower()
return response in ('y', 'yes')
except (EOFError, KeyboardInterrupt):
return False
def show_bird_install_success(self, username: str):
"""Show Bird installation success message."""
msg = f"{Colors.GREEN}✓ Bird installed and authenticated as @{username}{Colors.RESET}\n" if IS_TTY else f"✓ Bird installed and authenticated as @{username}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_install_failed(self, error: str):
"""Show Bird installation failure message."""
msg = f"{Colors.RED}✗ Bird installation failed: {error}{Colors.RESET}\n" if IS_TTY else f"✗ Bird installation failed: {error}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_auth_help(self):
"""Show Bird authentication help."""
if IS_TTY:
sys.stderr.write(BIRD_AUTH_HELP)
else:
sys.stderr.write(BIRD_AUTH_HELP_PLAIN)
sys.stderr.flush()
```
**Step 3: Verify new methods exist**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib.ui import ProgressDisplay; p = ProgressDisplay('test', show_banner=False); print(hasattr(p, 'prompt_bird_install'))"`
Expected: `True`
**Step 4: Commit**
```bash
git add scripts/lib/ui.py
git commit -m "feat(ui): add Bird CLI install prompts and auth help"
```
---
### Task 6: Update env.py with X Source Detection
**Files:**
- Modify: `scripts/lib/env.py`
**Step 1: Add get_x_source function**
Add at end of file:
```python
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) → xAI (paid API)
Args:
config: Configuration dict from get_config()
Returns:
'bird' if Bird is installed and authenticated,
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
from . import bird_x
# Check Bird first (free option)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return None
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"can_install_bird": bird_status["can_install"],
}
```
**Step 2: Verify function works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; print(env.get_x_source_status(env.get_config()))"`
Expected: Dict with source status
**Step 3: Commit**
```bash
git add scripts/lib/env.py
git commit -m "feat(env): add X source detection with Bird priority"
```
---
### Task 7: Update __init__.py to Export bird_x
**Files:**
- Modify: `scripts/lib/__init__.py`
**Step 1: Add bird_x to imports**
Replace file contents with:
```python
# last30days library modules
from . import bird_x
```
**Step 2: Verify import works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/__init__.py
git commit -m "feat(lib): export bird_x module"
```
---
### Task 8: Integrate Bird into Main Script - Part 1 (Setup Phase)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Add bird_x import**
Add `bird_x` to the imports from lib (around line 36):
```python
from lib import (
bird_x,
dates,
dedupe,
env,
http,
models,
normalize,
openai_reddit,
reddit_enrich,
render,
schema,
score,
ui,
websearch,
xai_x,
)
```
**Step 2: Add Bird setup function**
Add after the imports, before `load_fixture`:
```python
def setup_bird_if_needed(progress: ui.ProgressDisplay) -> Optional[str]:
"""Check Bird status and offer installation if needed.
Returns:
'bird' if Bird is ready to use,
'declined' if user declined install,
None if Bird not available and couldn't be installed.
"""
status = bird_x.get_bird_status()
# Already working
if status["authenticated"]:
return 'bird'
# Installed but not authenticated
if status["installed"]:
progress.show_bird_auth_help()
return None
# Not installed - offer to install if npm available
if status["can_install"]:
if progress.prompt_bird_install():
success, message = bird_x.install_bird()
if success:
# Check if auth works now
username = bird_x.is_bird_authenticated()
if username:
progress.show_bird_install_success(username)
return 'bird'
else:
progress.show_bird_auth_help()
return None
else:
progress.show_bird_install_failed(message)
return None
else:
return 'declined'
return None
```
**Step 3: Verify script still loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "import scripts.last30days; print('OK')"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): add Bird setup function"
```
---
### Task 9: Integrate Bird into Main Script - Part 2 (Search Dispatch)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Modify _search_x function to support Bird**
Replace the `_search_x` function (around line 119-159) with:
```python
def _search_x(
topic: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str,
mock: bool,
x_source: str = "xai",
) -> tuple:
"""Search X via Bird CLI or xAI (runs in thread).
Args:
x_source: 'bird' or 'xai' - which backend to use
Returns:
Tuple of (x_items, raw_response, error)
"""
raw_response = None
x_error = None
if mock:
raw_response = load_fixture("xai_sample.json")
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
# Use Bird if specified
if x_source == "bird":
try:
raw_response = bird_x.search_x(
topic,
from_date,
to_date,
depth=depth,
)
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = bird_x.parse_bird_response(raw_response or {})
# Check for error in response
if raw_response and raw_response.get("error") and not x_error:
x_error = raw_response["error"]
return x_items, raw_response, x_error
# Use xAI (original behavior)
try:
raw_response = xai_x.search_x(
config["XAI_API_KEY"],
selected_models["xai"],
topic,
from_date,
to_date,
depth=depth,
)
except http.HTTPError as e:
raw_response = {"error": str(e)}
x_error = f"API error: {e}"
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
```
**Step 2: Update run_research to accept x_source parameter**
Find the `run_research` function signature (around line 161) and add `x_source` parameter:
```python
def run_research(
topic: str,
sources: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str = "default",
mock: bool = False,
progress: ui.ProgressDisplay = None,
x_source: str = "xai",
) -> tuple:
```
Then update the `_search_x` call inside (around line 218-222) to pass `x_source`:
```python
x_future = executor.submit(
_search_x, topic, config, selected_models,
from_date, to_date, depth, mock, x_source
)
```
**Step 3: Verify script syntax is valid**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -m py_compile scripts/last30days.py && echo "OK"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): dispatch X search to Bird or xAI"
```
---
### Task 10: Integrate Bird into Main Script - Part 3 (Main Function)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Update main() to check Bird before research**
In the `main()` function, after loading config and before checking available sources (around line 345-355), add Bird setup:
Find this section:
```python
# Load config
config = env.get_config()
# Check available sources
available = env.get_available_sources(config)
```
Replace with:
```python
# Load config
config = env.get_config()
# Initialize progress display early for Bird prompts
progress = ui.ProgressDisplay(args.topic, show_banner=True)
# Check Bird availability and offer install if needed
x_source_status = env.get_x_source_status(config)
x_source = x_source_status["source"]
# If no X source and Bird can be installed, offer it
if x_source is None and x_source_status["can_install_bird"]:
bird_result = setup_bird_if_needed(progress)
if bird_result == 'bird':
x_source = 'bird'
# Refresh status
x_source_status = env.get_x_source_status(config)
# Check available sources (now accounting for Bird)
available = env.get_available_sources(config)
# Override available if Bird is ready
if x_source == 'bird':
if available == 'reddit':
available = 'both' # Now have both Reddit + X (via Bird)
elif available == 'web':
available = 'x' # Now have X via Bird
```
**Step 2: Remove duplicate progress initialization**
Find and remove the later `progress = ui.ProgressDisplay(...)` line (around line 371) since we now create it earlier.
**Step 3: Pass x_source to run_research**
Find the `run_research` call (around line 413) and add `x_source` parameter:
```python
reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error = run_research(
args.topic,
sources,
config,
selected_models,
from_date,
to_date,
depth,
args.mock,
progress,
x_source=x_source or "xai",
)
```
**Step 4: Verify script runs with --help**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py --help`
Expected: Help text displays without errors
**Step 5: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): integrate Bird setup into main flow"
```
---
### Task 11: Test End-to-End with Mock Mode
**Files:**
- None (testing only)
**Step 1: Test mock mode still works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py "Claude Code" --mock --emit=compact 2>&1 | head -20`
Expected: Output showing research results without errors
**Step 2: Test Bird detection (informational)**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; import json; print(json.dumps(env.get_x_source_status(env.get_config()), indent=2))"`
Expected: JSON showing current Bird/xAI status
**Step 3: Commit any fixes if needed, then final commit**
```bash
git add -A
git commit -m "feat(bird): complete Bird CLI integration
- Add bird_x.py module for Bird CLI detection, install, and search
- Add UI prompts for interactive Bird installation
- Update env.py with X source priority (Bird > xAI)
- Integrate Bird into main research flow
- Bird uses browser cookies (free, no API key needed)"
```
---
### Task 12: Push to Private Repo
**Files:**
- None (git only)
**Step 1: Push all changes**
Run: `cd /Users/mvanhorn/last30days-skill-private && git push origin main`
Expected: Changes pushed to private repo
**Step 2: Verify commit history**
Run: `cd /Users/mvanhorn/last30days-skill-private && git log --oneline -10`
Expected: Shows Bird integration commits
---
## Summary
After completing all tasks, the skill will:
1. Check if Bird CLI is installed on startup
2. If not installed but npm available, prompt user to install
3. If installed, verify authentication via `bird whoami`
4. If authenticated, use Bird for all X searches (free)
5. If not, fall back to xAI (if key exists) or WebSearch
6. Output format identical regardless of backend used
docs/plans/2026-02-03-bird-cli-integration-design.md
-102
View File
@@ -1,102 +0,0 @@
# Bird CLI Integration Design
**Date:** 2026-02-03
**Status:** Approved
## Overview
Add Bird CLI as an alternative X/Twitter search source for the last30days skill. Bird uses browser cookie authentication (free, no API key) and provides direct access to X's GraphQL API.
## Goals
- Provide free X search without requiring xAI API key
- Seamless fallback: Bird → xAI → WebSearch
- Interactive onboarding for users without Bird installed
- Output parity with existing xAI implementation
## Detection & Priority Flow
```
On startup:
1. Check: Is Bird installed? (`which bird`)
├─ No → Offer to install: "Bird CLI not found. Install for free X search? (y/n)"
│ ├─ Yes → Run `npm install -g @steipete/bird`
│ └─ No → Continue to step 2
│
└─ Yes → Check: Is Bird authenticated? (`bird whoami`)
├─ Success → Use Bird for X searches
└─ Fail → Show: "Bird auth failed. Run `bird check` to diagnose."
Continue to step 2
2. Fall back to xAI if XAI_API_KEY exists
3. Fall back to WebSearch if nothing else available
```
**Priority order:** Bird → xAI → WebSearch
## New Module: `scripts/lib/bird_x.py`
### Functions
- `is_bird_installed()` → checks `which bird`, returns bool
- `is_bird_authenticated()` → runs `bird whoami`, returns username or None
- `install_bird()` → runs `npm install -g @steipete/bird`, returns success bool
- `search_x(topic, from_date, to_date, depth)` → runs `bird search` with JSON output
- `parse_bird_response(json)` → converts to same format as `xai_x.parse_x_response()`
### Search Command
```bash
bird search "Claude Code skills" --since 2026-01-04 -n 30 --json
```
- `--since` filters to last 30 days
- `-n 30` controls result count (maps to depth: quick=12, default=30, deep=60)
- `--json` gives machine-readable output
### Output Mapping
| Bird field | Our field |
|------------|-----------|
| `text` | `text` |
| `permanent_url` | `url` |
| `user.screen_name` | `author_handle` |
| `created_at` | `date` (parse to YYYY-MM-DD) |
| `like_count` | `engagement.likes` |
| `retweet_count` | `engagement.reposts` |
| `reply_count` | `engagement.replies` |
| `quote_count` | `engagement.quotes` |
Relevance: Default to 0.7, let `score.py` re-rank based on engagement.
## Modified Files
| File | Change |
|------|--------|
| `env.py` | Add `get_x_source()` → returns `'bird'`, `'xai'`, or `None` |
| `last30days.py` | Check Bird availability with interactive install prompt before research |
| `last30days.py` | In `_search_x()`, dispatch to `bird_x` or `xai_x` based on source |
| `ui.py` | Add `prompt_bird_install()` and `show_bird_auth_help()` |
## Unchanged Files
- `normalize.py` - Bird output matches xAI format after parsing
- `score.py` - Same scoring logic applies
- `dedupe.py` - Same deduplication logic
- `render.py` - X results labeled as "X" regardless of backend
## Error Handling
| Scenario | Behavior |
|----------|----------|
| Bird installed but no browser cookies | Show `bird check` guidance, fall back to xAI |
| Bird search returns 0 results | Retry with simplified query (same as xAI logic) |
| Bird search times out | Fall back to xAI if available, else WebSearch |
| npm not installed (can't install Bird) | Skip Bird, continue with xAI/WebSearch |
| User declines Bird install | Remember for session, don't ask again |
**Timeout:** 30 seconds for Bird commands
## Output Labels
Results labeled as "X" regardless of whether Bird or xAI was used. Users care about the data, not the backend.
docs/plans/2026-02-06-feat-automated-v1-v2-test-harness-plan.md
-281
View File
@@ -1,281 +0,0 @@
---
title: "feat: Automated v1 vs v2 test harness using claude --print"
type: feat
date: 2026-02-06
---
# feat: Automated V1 vs V2 Test Harness
## Overview
Build a bash script that swaps SKILL.md between v1 (upstream) and v2 (current), runs `claude --print "/last30days [query]"` for all 17 test queries on each version, captures output to files, then generates a comparison doc with analysis.
## How It Works
```
┌─────────────────────────────────────────────────┐
│ test-v1-vs-v2.sh │
│ │
│ 1. Save current SKILL.md as .v2 backup │
│ 2. Install v1 SKILL.md from upstream │
│ 3. Loop 17 queries → claude --print → v1/*.txt │
│ 4. Restore v2 SKILL.md │
│ 5. Loop 17 queries → claude --print → v2/*.txt │
│ 6. Generate comparison doc │
└─────────────────────────────────────────────────┘
```
Each `claude --print` call:
- Invokes the /last30days skill exactly as a user would
- Runs the Python script (real API calls to OpenAI + xAI)
- Runs WebSearch
- Applies SKILL.md presentation instructions
- Returns the full formatted output
- Exits (no interactive session)
## Implementation
### File: `scripts/test-v1-vs-v2.sh`
```bash
#!/bin/bash
set -euo pipefail
# === Config ===
SKILL_DIR="$HOME/.claude/skills/last30days"
REPO_DIR="/Users/mvanhorn/last30days-skill-private"
OUT_DIR="$REPO_DIR/docs/test-results/v1-vs-v2-$(date +%Y%m%d-%H%M%S)"
V1_DIR="$OUT_DIR/v1"
V2_DIR="$OUT_DIR/v2"
mkdir -p "$V1_DIR" "$V2_DIR"
# All 17 test queries (from README + plans)
declare -a QUERIES=(
"prompting techniques for chatgpt for legal questions"
"best clawdbot use cases"
"how to best setup clawdbot"
"prompting tips for nano banana pro for ios designs"
"top claude code skills"
"using ChatGPT to make images of dogs"
"research best practices for beautiful remotion animation videos in claude code"
"photorealistic people in nano banana pro"
"What are the best rap songs lately"
"what are people saying about DeepSeek R1"
"best practices for cursor rules files for Cursor"
"prompt advice for using suno to make killer songs in simple mode"
"how do I use Codex with Claude Code on same app to make it better"
"kanye west"
"howie.ai"
"open claw"
"nano banana pro prompting"
)
declare -a TYPES=(
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"HOW-TO"
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"GENERAL"
"PROMPTING"
"PROMPTING"
"RECOMMENDATIONS"
"NEWS"
"PROMPTING"
"PROMPTING"
"HOW-TO"
"NEWS"
"GENERAL"
"GENERAL"
"PROMPTING"
)
slugify() {
echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | head -c 60
}
run_version() {
local version="$1"
local outdir="$2"
echo ""
echo "=========================================="
echo " Running $version — ${#QUERIES[@]} queries"
echo "=========================================="
for i in "${!QUERIES[@]}"; do
local query="${QUERIES[$i]}"
local type="${TYPES[$i]}"
local slug=$(slugify "$query")
local num=$((i + 1))
local outfile="$outdir/${num}-${slug}.txt"
echo ""
echo "[$version] ($num/${#QUERIES[@]}) $query [$type]"
echo " → $outfile"
# Run claude --print with the skill invocation
# --no-session-persistence: don't save to session history
# Timeout after 5 minutes per query (generous for slow API calls)
if timeout 300 claude --print \
"/last30days $query" \
> "$outfile" 2>"$outdir/${num}-${slug}.stderr.txt"; then
echo " ✅ Done ($(wc -l < "$outfile") lines)"
else
echo " ❌ Failed or timed out"
echo "FAILED: timeout or error" >> "$outfile"
fi
# Brief pause between queries to avoid rate limits
sleep 2
done
}
# === Phase 1: Test V1 ===
echo "📦 Backing up current SKILL.md..."
cp "$SKILL_DIR/SKILL.md" "$SKILL_DIR/SKILL.md.v2.bak"
echo "📥 Installing V1 SKILL.md from upstream..."
cd "$REPO_DIR"
git show upstream/main:SKILL.md > "$SKILL_DIR/SKILL.md"
# Also save a copy for reference
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v1-SKILL.md"
run_version "V1" "$V1_DIR"
# === Phase 2: Test V2 ===
echo ""
echo "📥 Restoring V2 SKILL.md..."
cp "$SKILL_DIR/SKILL.md.v2.bak" "$SKILL_DIR/SKILL.md"
# Also save a copy for reference
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v2-SKILL.md"
run_version "V2" "$V2_DIR"
# === Phase 3: Generate summary ===
echo ""
echo "=========================================="
echo " Generating comparison summary"
echo "=========================================="
SUMMARY="$OUT_DIR/comparison-summary.md"
cat > "$SUMMARY" << 'HEADER'
# V1 vs V2 Comparison Results
Generated: $(date)
## Output Files
| # | Query | Type | V1 Lines | V2 Lines |
|---|-------|------|----------|----------|
HEADER
# Replace the date placeholder
sed -i '' "s/\$(date)/$(date)/" "$SUMMARY"
for i in "${!QUERIES[@]}"; do
local query="${QUERIES[$i]}"
local type="${TYPES[$i]}"
local slug=$(slugify "$query")
local num=$((i + 1))
local v1file="$V1_DIR/${num}-${slug}.txt"
local v2file="$V2_DIR/${num}-${slug}.txt"
local v1lines=$(wc -l < "$v1file" 2>/dev/null || echo "0")
local v2lines=$(wc -l < "$v2file" 2>/dev/null || echo "0")
echo "| $num | \`$query\` | $type | $v1lines | $v2lines |" >> "$SUMMARY"
done
cat >> "$SUMMARY" << 'FOOTER'
## Scorecard Template
For each query, score both versions on:
| Dimension | V1 | V2 | Notes |
|-----------|----|----|-------|
| Query Parsing Display (1-5) | | | |
| Source Coverage (1-5) | | | |
| Citation Quality (1-5) | | | |
| Summary Structure (1-5) | | | |
| Stats Box Format (1-5) | | | |
| Research Grounding (1-5) | | | |
## Next Step
Read each pair of output files and score them using the test plan at:
`docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md`
FOOTER
echo ""
echo "✅ All done!"
echo "📁 Results: $OUT_DIR"
echo "📊 Summary: $SUMMARY"
echo ""
echo "V1 outputs: $V1_DIR/"
echo "V2 outputs: $V2_DIR/"
echo ""
echo "To review, run:"
echo " open $OUT_DIR"
# Cleanup backup
rm -f "$SKILL_DIR/SKILL.md.v2.bak"
```
## Acceptance Criteria
- [ ] Script runs all 17 queries on v1 SKILL.md
- [ ] Script runs all 17 queries on v2 SKILL.md
- [ ] Each query output saved to a separate .txt file
- [ ] Comparison summary generated with line counts
- [ ] SKILL.md restored to v2 after testing
- [ ] Both SKILL.md versions saved in output dir for reference
- [ ] Script handles timeouts gracefully (5 min per query)
- [ ] Brief pause between queries to avoid rate limits
## Cost Estimate
- 34 total `claude --print` invocations
- Each invocation: ~1 Python script run (OpenAI + xAI API) + 2-3 WebSearches + Claude response
- Estimated: ~$0.10-0.30 per invocation for API calls
- **Total estimate: $3-10 for the full run**
## Time Estimate
- Each query: ~1-3 minutes (script + WebSearch + synthesis)
- 17 queries × 2 versions = 34 runs
- **Total: ~45-90 minutes** (could run in background)
## How to Run
```bash
cd /Users/mvanhorn/last30days-skill-private
chmod +x scripts/test-v1-vs-v2.sh
./scripts/test-v1-vs-v2.sh
```
Or run in background:
```bash
nohup ./scripts/test-v1-vs-v2.sh > test-run.log 2>&1 &
tail -f test-run.log
```
## After the Run
Once all outputs are captured, Claude can read every file pair and generate the scored comparison doc with analysis — that's the part where I score each dimension 1-5 and write the final report.
## Files
| File | Purpose |
|------|---------|
| `scripts/test-v1-vs-v2.sh` | The test harness script |
| `docs/test-results/v1-vs-v2-*/` | Output directory (timestamped) |
| `docs/test-results/v1-vs-v2-*/v1/*.txt` | V1 outputs |
| `docs/test-results/v1-vs-v2-*/v2/*.txt` | V2 outputs |
| `docs/test-results/v1-vs-v2-*/comparison-summary.md` | Auto-generated summary |
docs/plans/2026-02-06-feat-free-reddit-mcp-integration-plan.md
-352
View File
@@ -1,352 +0,0 @@
---
title: "feat: Free Reddit via MCP — Remove OpenAI Key Requirement"
type: feat
date: 2026-02-06
---
# feat: Free Reddit via MCP — Remove OpenAI Key Requirement
## Overview
Replace the OpenAI Responses API (paid) for Reddit searching with a **free, zero-config Reddit MCP server**, eliminating the need for an `OPENAI_API_KEY` to get Reddit results in the last30days skill. This work happens in a **new forked repo** to keep the current release branch clean.
## Problem Statement
Today, the last30days skill requires an OpenAI API key (`OPENAI_API_KEY`) to search Reddit. This is the most expensive dependency in the stack — OpenAI charges per-call for the Responses API with web search. Users without an OpenAI key get zero Reddit results and fall back to WebSearch-only mode, which loses the engagement metrics (upvotes, comments) that make last30days uniquely valuable.
**Goal:** Make Reddit search completely free with no API key registration required.
## Research Findings
### Option Analysis
Five approaches were evaluated. Two stand out:
| Option | Free? | Search? | Engagement? | Setup | Notes |
|--------|-------|---------|-------------|-------|-------|
| **reddit-mcp-buddy** (Node.js) | Yes (anonymous mode) | Yes | Yes | One CLI command | 371 stars, 3-tier auth, most popular |
| **mcp-server-reddit** (Python) | Yes (redditwarp) | **No** | Yes | pip install | Recommended by ClaudeLog, but browse-only |
| **.json URL trick** (curl) | Yes | Yes | Yes | Zero | ~10 req/min, no dependencies |
| Reddit OAuth (PRAW) | Free but needs registration | Yes | Yes | App registration | 100 req/min, most reliable |
| RSS feeds | Yes | Basic | **No** | Zero | Least useful, no metrics |
### Top Contender: reddit-mcp-buddy
**GitHub:** [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy)
- 371 stars, actively maintained (last update: Jan 29, 2026)
- **Anonymous mode** — zero credentials, ~10 req/min
- **MCP tools:** `search_reddit`, `browse_subreddit`, `get_post_details`, `user_analysis`, `reddit_explain`
- **Install:** `claude mcp add --transport stdio reddit-mcp-buddy -s user -- npx -y reddit-mcp-buddy`
- Returns full engagement metrics (score, num_comments, upvote_ratio)
- TypeScript/Node.js (npx, no Python dependency)
### Strong Alternative: .json URL trick
- Append `.json` to any Reddit URL → full JSON response
- Search: `https://www.reddit.com/search.json?q=TOPIC&sort=relevance&t=month&limit=100`
- Zero dependencies, zero auth, zero setup
- Returns same data as official API (score, num_comments, created_utc, upvote_ratio)
- ~10 req/min rate limit (sufficient for skill use)
- Can be called via `curl` or Python `urllib`
### Also Considered
- **Hawstein/mcp-server-reddit** (134 stars, Python, no auth) — **No search tool**, only browse subreddits. Cannot replace OpenAI's keyword search capability.
- **Arindam200/reddit-mcp** (262 stars, PRAW) — Has search but **requires Reddit OAuth credentials**. Not truly zero-config.
- **adhikasp/mcp-reddit** (348 stars) — Hot threads only, no search. Last updated Dec 2024.
## Proposed Solution
### Approach A: MCP-First with .json Fallback (Recommended)
Add a new module `mcp_reddit.py` that:
1. **Detects** if a Reddit MCP server is configured in the user's Claude Code session
2. **If MCP available:** Calls MCP `search_reddit` tool via the skill's `allowed-tools` (the skill already allows tools — MCP tools become available when configured)
3. **If no MCP:** Falls back to Reddit's `.json` URL endpoints via `urllib` (stdlib, no dependencies)
4. **Normalize** MCP/JSON results to the existing `RedditItem` schema
5. **Enrich** with `reddit_enrich.py` (already works — fetches real thread data)
This gives users three tiers of Reddit access:
- **Tier 1 (best):** Reddit MCP installed → full search + engagement via MCP tools
- **Tier 2 (good):** No MCP, no keys → `.json` URL search + enrichment
- **Tier 3 (existing):** OpenAI key present → existing `openai_reddit.py` still works (backward compat)
### Approach B: .json-Only (Simpler)
Skip MCP entirely. Add a `reddit_json.py` module that uses `https://www.reddit.com/search.json` directly. Simpler but misses the MCP ecosystem integration.
### Approach C: MCP-Only (Cleaner)
Require MCP setup. Simpler code but adds a user setup step (`claude mcp add ...`).
**Recommendation: Approach A** — MCP-first with .json fallback gives zero-config Reddit search for everyone while rewarding users who set up MCP with a better experience.
## Technical Approach
### Architecture
```
User invokes /last30days "topic"
│
├─ env.py detects sources:
│ ├─ MCP reddit available? → mcp_reddit.py
│ ├─ No MCP, no key? → reddit_json.py (.json URL trick)
│ └─ OPENAI_API_KEY set? → openai_reddit.py (existing, backward compat)
│
├─ X search (Bird CLI / xAI — unchanged)
│
└─ Normalize → Score → Dedupe → Render (unchanged)
```
### New Files
#### `scripts/lib/reddit_json.py`
- Uses `urllib.request` (stdlib) to call Reddit's `.json` search endpoint
- URL: `https://www.reddit.com/search.json?q={topic}&sort=relevance&t=month&limit=100`
- Custom `User-Agent` header (required by Reddit)
- Parses response into the same format as `openai_reddit.py` output
- Handles pagination via `after` token if depth=deep (multiple requests)
- Rate limiting: 1 second delay between requests
#### `scripts/lib/mcp_reddit.py`
- Detects MCP availability (check if Reddit MCP tools exist in session)
- Formats MCP tool calls for `search_reddit` with topic + date filters
- Normalizes MCP response to match existing `RedditItem` schema
- Falls back to `reddit_json.py` if MCP unavailable
#### Modified Files
- **`scripts/lib/env.py`** — Add Reddit source detection: MCP → .json → OpenAI
- **`scripts/last30days.py`** — Route Reddit search through new priority chain
- **`scripts/lib/normalize.py`** — Add normalizer for `.json` endpoint response format
- **`SKILL.md`** — Document MCP setup as optional enhancement
### MCP Integration Design
The MCP approach has a subtle challenge: the last30days skill runs as a **Python subprocess** (`python3 last30days.py`), but MCP tools are available to **Claude's session**, not to subprocesses.
**Two paths to solve this:**
**Path 1: SKILL.md orchestration** — The SKILL.md workflow calls the MCP `search_reddit` tool directly (before or in parallel with the Python script), saves the MCP response to a temp file, and the Python script reads it:
```
SKILL.md workflow:
1. Call MCP search_reddit → save to /tmp/last30days_mcp_reddit.json
2. Call python3 last30days.py --reddit-from=/tmp/last30days_mcp_reddit.json --emit=compact
3. Python script reads pre-fetched Reddit data instead of calling OpenAI
```
**Path 2: .json only for subprocess** — The Python script uses `.json` URLs directly (no MCP needed in subprocess). MCP is a bonus for users who want Claude to also browse specific threads interactively.
**Recommendation: Path 2 for MVP, Path 1 as enhancement.** The `.json` approach is self-contained, testable, and doesn't require SKILL.md workflow changes. MCP can be layered on later.
### Source Priority Chain (updated env.py)
```python
def get_reddit_source(config: dict) -> str:
"""Returns: 'mcp', 'json', 'openai', or 'none'"""
# 1. Check for pre-fetched MCP data (from SKILL.md)
if os.path.exists(MCP_REDDIT_CACHE_PATH):
return 'mcp'
# 2. Always available — no key needed
# (reddit .json endpoints are free)
return 'json'
# 3. OpenAI key present → legacy path
# if config.get('OPENAI_API_KEY'):
# return 'openai'
```
For MVP, the `.json` path is **always available** so it becomes the default. OpenAI path remains as opt-in for users who want higher rate limits.
### Data Mapping: .json → RedditItem
Reddit `.json` search returns `data.children[].data` with:
```json
{
"title": "Post title",
"permalink": "/r/subreddit/comments/abc123/...",
"subreddit": "ClaudeAI",
"score": 42,
"num_comments": 15,
"upvote_ratio": 0.95,
"created_utc": 1738800000,
"selftext": "Post body...",
"url": "https://...",
"author": "username"
}
```
Maps cleanly to existing `RedditItem`:
| .json field | RedditItem field | Notes |
|-------------|------------------|-------|
| `title` | `title` | Direct |
| `permalink` | `url` | Prepend `https://www.reddit.com` |
| `subreddit` | `subreddit` | Direct |
| `score` | `engagement.score` | Direct |
| `num_comments` | `engagement.num_comments` | Direct |
| `upvote_ratio` | `engagement.upvote_ratio` | Direct |
| `created_utc` | `date` | Convert epoch → YYYY-MM-DD |
| `selftext` | (used for relevance) | AI relevance scoring needed |
| `author` | (not in current schema) | Ignore for now |
### Relevance Scoring Without AI
The current `openai_reddit.py` gets relevance scores **from OpenAI** (the AI judges how relevant each result is). With `.json` endpoints, we lose that AI relevance judgment.
**Options:**
1. **Keyword matching** — Score based on how many query terms appear in title + selftext. Simple but effective.
2. **TF-IDF-like** — Weight rarer query terms higher. More accurate but more code.
3. **Let Claude judge** — Pass results to Claude in SKILL.md and have Claude score relevance. Most accurate but changes the workflow.
4. **Skip relevance, rely on Reddit's sort** — Reddit's `sort=relevance` already ranks by relevance. Trust it and use position-based scoring (first result = 1.0, last = 0.5).
**Recommendation: Option 4 for MVP.** Reddit's search relevance ranking is already good. Use position-based relevance (1.0 → 0.5 linear decay over result set) combined with the existing engagement-based scoring. This requires zero external dependencies and no AI calls.
## Implementation Phases
### Phase 0: Fork Repository
- [ ] Create new repo `last30days-free-reddit` (or branch in private repo)
- [ ] `git clone /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit`
- [ ] Create feature branch: `feat/free-reddit`
- [ ] Verify all existing tests pass on the new branch
### Phase 1: reddit_json.py — Core .json Search
- [ ] Create `scripts/lib/reddit_json.py`
- `search_reddit_json(topic: str, depth: str, date_from: str, date_to: str) -> list[dict]`
- Build search URL with `q`, `sort=relevance`, `t=month`, `limit` (25/50/100 by depth)
- Custom `User-Agent: last30days-skill/2.0 (by /u/last30days-bot)`
- Parse `data.children[].data` → list of raw dicts
- Handle pagination for deep mode (follow `after` token, max 3 pages)
- Rate limit: `time.sleep(1.0)` between requests
- Error handling: 429 (rate limit), 403 (blocked), network errors → return empty + error msg
- [ ] Create `tests/test_reddit_json.py`
- Mock HTTP responses using fixture files
- Test: basic search parsing, pagination, rate limit handling, error cases
- [ ] Create `fixtures/reddit_json_search_sample.json`
- Real `.json` search response (sanitized)
### Phase 2: Normalize + Score .json Results
- [ ] Add `normalize_reddit_json()` to `scripts/lib/normalize.py`
- Convert `.json` response format → `RedditItem` schema
- `created_utc` (epoch) → `YYYY-MM-DD` string
- `permalink` → full URL
- Position-based relevance: `1.0 - (index / total * 0.5)` → range [1.0, 0.5]
- Date confidence: `"high"` (Reddit provides exact timestamps)
- [ ] Update `scripts/lib/score.py` if needed
- `.json` results already have real engagement metrics — existing scoring formula works as-is
- No penalty needed (unlike WebSearch which lacks engagement)
- [ ] Add tests for normalization + scoring of `.json` data
### Phase 3: Integration into Orchestrator
- [ ] Update `scripts/lib/env.py`
- Add `get_reddit_source(config) -> str` returning `'json'`, `'openai'`, or `'none'`
- Priority: `.json` always available (default), `'openai'` if key present and user prefers
- Add `REDDIT_SOURCE` config option: `auto` (default), `json`, `openai`
- Update `get_available_sources()` to always include Reddit (since `.json` is free)
- Update `get_missing_keys()` — Reddit no longer shows as "missing"
- [ ] Update `scripts/last30days.py`
- Add `_search_reddit_json()` function alongside existing `_search_reddit()`
- Route based on `get_reddit_source()`: json → `_search_reddit_json()`, openai → `_search_reddit()`
- Skip `reddit_enrich.py` for `.json` results (already have real engagement metrics!)
- Update progress/stats output to show source: "Reddit (free)" vs "Reddit (OpenAI)"
- [ ] Update SKILL.md promo messaging
- Remove "Add OPENAI_API_KEY for Reddit" messaging
- Instead: "Reddit search included free! Add OpenAI key for AI-enhanced relevance scoring."
- [ ] Integration tests: full pipeline with `.json` mock data
### Phase 4: Testing & Polish
- [ ] Run all existing tests — ensure backward compatibility
- [ ] Manual test: invoke `/last30days` with NO API keys → should get Reddit + WebSearch results
- [ ] Manual test: invoke with OPENAI_API_KEY → should still use OpenAI path (backward compat)
- [ ] Manual test: compare result quality — `.json` vs OpenAI for same topic
- [ ] Update README.md — document free Reddit access
- [ ] Update SPEC.md — document new source priority chain
### Phase 5 (Future): MCP Enhancement Layer
- [ ] Detect Reddit MCP in Claude's session
- [ ] SKILL.md pre-fetches via MCP `search_reddit` → saves to temp file
- [ ] Python script reads pre-fetched MCP data via `--reddit-from=` flag
- [ ] MCP results get AI-judged relevance (since Claude sees them)
- [ ] Better than `.json` position-based relevance
## Acceptance Criteria
### Functional
- [ ] `/last30days "any topic"` returns Reddit results with **zero API keys configured**
- [ ] Reddit results include real engagement metrics (score, comments, upvote_ratio)
- [ ] Results are properly scored, deduped, and rendered (same quality as OpenAI path)
- [ ] Existing OpenAI path still works when key is present
- [ ] Existing X search (Bird CLI / xAI) is unchanged
- [ ] Stats box shows correct source indicator ("Reddit (free)" or "Reddit (OpenAI)")
### Non-Functional
- [ ] No external Python packages (stdlib only — `urllib.request`, `json`, `time`)
- [ ] Respects Reddit rate limits (~10 req/min for unauthenticated)
- [ ] Graceful degradation if Reddit blocks requests (429/403 → empty results + error msg)
- [ ] All new code has unit tests with fixtures (no live API calls in tests)
### Quality Gates
- [ ] All existing tests pass (zero regressions)
- [ ] New tests cover: search parsing, normalization, scoring, error handling, pagination
- [ ] Manual smoke test passes with zero API keys
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Reddit blocks `.json` endpoints | Low | High | Fall back to OpenAI path; monitor for 429s |
| `.json` rate limit too restrictive | Medium | Medium | 1s delay between requests; cache results |
| Relevance quality lower without AI scoring | Medium | Medium | Reddit's `sort=relevance` is decent; can add keyword scoring later |
| Reddit changes `.json` response format | Low | Medium | Schema validation in normalize; fixture-based tests catch changes |
| Node.js MCP server dependency conflicts | N/A (Phase 5) | Low | MCP is optional enhancement, not required |
## Dependencies
- **None** — this feature uses only Python stdlib and Reddit's public `.json` endpoints
- Phase 5 (future) would add optional dependency on an MCP server
## Repository Setup
```bash
# Fork into new working repo
cp -r /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit
cd /Users/mvanhorn/last30days-free-reddit
# Create feature branch
git checkout -b feat/free-reddit
# Verify existing tests
python3 -m pytest tests/ -v
```
The new repo is disposable — once the feature is validated, changes merge back into `last30days-skill-private` via PR or cherry-pick.
## References
### Internal
- `scripts/lib/openai_reddit.py` — Current Reddit search (to be replaced/supplemented)
- `scripts/lib/env.py:get_available_sources()` — Source detection logic to update
- `scripts/lib/normalize.py` — Add `.json` normalizer alongside existing
- `scripts/lib/reddit_enrich.py` — May be skippable for `.json` results (already have engagement)
- `scripts/lib/schema.py:RedditItem` — Target schema (unchanged)
### External
- [Reddit .json search endpoint](https://www.reddit.com/search.json?q=test&sort=relevance&t=month&limit=25)
- [Simon Willison — Scraping Reddit via JSON API](https://til.simonwillison.net/reddit/scraping-reddit-json)
- [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy) — Best MCP option for Phase 5
- [Hawstein/mcp-server-reddit](https://github.com/Hawstein/mcp-server-reddit) — ClaudeLog-recommended MCP (no search though)
- [Reddit API Rate Limits Guide](https://painonsocial.com/blog/reddit-api-rate-limits-guide)
### Research Sources
- last30days skill output — community recommendations for Reddit search tools
- GitHub search — 10+ Reddit MCP repos evaluated
- npm/PyPI registries — package availability confirmed
- ClaudeLog — [Reddit MCP reference](https://claudelog.com/claude-code-mcps/reddit-mcp/)
docs/plans/2026-02-06-feat-last30days-bird-cli-release-plan.md
-391
View File
@@ -1,391 +0,0 @@
---
title: "feat: Release last30days v2 with Bird CLI to GitHub"
type: feat
date: 2026-02-06
---
# Release last30days v2 (Bird CLI) to GitHub
## Overview
Replace the current public `last30days-skill` on GitHub with the new Bird CLI-enhanced version from `last30days-skill-private`. The new version adds free X/Twitter search via Bird CLI while maintaining backward compatibility with xAI API keys.
**Goal:** Ship with confidence. No rollbacks.
## Current State
| | Old (Public) | New (Private) |
|---|---|---|
| **Repo** | `mvanhorn/last30days-skill` | `mvanhorn/last30days-skill-private` |
| **Local path** | `~/.claude/skills/last30days/` | `~/.claude/skills/last30daystest/` (symlink) |
| **Remote** | `origin` → public repo | `origin` → private, `upstream` → public |
| **Key addition** | -- | Bird CLI (`@steipete/bird`) for free X search |
| **X source chain** | xAI API only | Bird (free) → xAI (paid) → WebSearch |
| **Uses** | 136 | 18 |
| **Latest commit** | `cc892d7` | `4230fa2` |
**Why both show as `/last30days`:** Both `SKILL.md` files declare `name: last30days` in frontmatter. Claude Code discovers both from `~/.claude/skills/` and lists them separately.
---
## Phase 0: Clean Swap (Day 1)
Remove the old skill so only the new one is active. This eliminates ambiguity during testing.
### Steps
1. **Back up the old skill** (safety net):
```bash
mv ~/.claude/skills/last30days ~/.claude/skills/last30days.backup-v1
```
2. **Promote the new skill to primary**:
```bash
# Remove the test symlink
rm ~/.claude/skills/last30daystest
# Create new symlink with the primary name
ln -s /Users/mvanhorn/last30days-skill-private ~/.claude/skills/last30days
```
3. **Verify only one `/last30days` appears**:
- Open a new Claude Code session
- Type `/last` and confirm only ONE `/last30days` shows in autocomplete
- Confirm description mentions Bird CLI
4. **Rollback procedure** (if something goes wrong):
```bash
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
```
### Acceptance Criteria
- [ ] Only one `/last30days` appears in Claude Code autocomplete
- [ ] Old skill preserved at `~/.claude/skills/last30days.backup-v1`
- [ ] New skill responds to `/last30days` invocation
---
## Phase 1: Claude's Test Plan (Automated)
These are tests Claude can run autonomously to validate the new skill before the user touches it.
### 1.1 Script-Level Smoke Tests
Run the Python scripts directly to verify core functionality without invoking the full skill.
#### Bird CLI Detection
```bash
# Test: Bird is installed and authenticated
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
print('installed:', bird_x.is_bird_installed())
print('authenticated:', bird_x.is_bird_authenticated())
print('status:', bird_x.get_bird_status())
"
```
- [ ] `is_bird_installed()` returns True (or False with clear message)
- [ ] `is_bird_authenticated()` returns True if logged into X in browser
- [ ] `get_bird_status()` returns a dict with `installed`, `authenticated`, `available` keys
#### Environment & Source Detection
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source:', env.get_x_source(config))
print('has_openai:', bool(config.get('OPENAI_API_KEY')))
"
```
- [ ] `get_x_source()` returns `'bird'` if Bird available, `'xai'` if API key set, `None` otherwise
- [ ] Config loads from `~/.config/last30days/.env`
#### Bird Search (Direct)
```bash
python3 -c "
import sys, json; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
result = bird_x.search_x('Claude Code tips', '2026-01-07', '2026-02-06', 'quick')
print(json.dumps(result, indent=2, default=str)[:2000])
"
```
- [ ] Returns search results (list of dicts with `url`, `text`, `author_handle`)
- [ ] No Python tracebacks
- [ ] Results are from the expected date range
#### Full Research Pipeline (Compact Output)
```bash
cd /Users/mvanhorn/last30days-skill-private
python3 scripts/last30days.py "Claude Code tips" --emit=compact --quick 2>&1 | head -100
```
- [ ] Completes without error
- [ ] Output includes X results (via Bird or xAI)
- [ ] Output includes Reddit results (via OpenAI) if key configured
- [ ] Stats summary shows source counts
### 1.2 Source Fallback Tests
Verify graceful degradation when sources are unavailable.
#### Bird unavailable, xAI available
```bash
# Temporarily hide Bird
PATH_BACKUP="$PATH"
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v "$(dirname $(which bird 2>/dev/null))" | tr '\n' ':')
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source (no bird):', env.get_x_source(config))
"
export PATH="$PATH_BACKUP"
```
- [ ] Falls back to `'xai'` when Bird not in PATH
- [ ] No crash or unhandled exception
#### No X source at all
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = {} # empty config, no keys
print('x_source (nothing):', env.get_x_source(config))
"
```
- [ ] Returns `None`
- [ ] No crash
### 1.3 Response Parsing Tests
Validate that Bird responses are correctly normalized to the canonical schema.
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
# Test with sample Bird response format
sample = {
'tweets': [{
'permanentUrl': 'https://x.com/user/status/123',
'text': 'Test tweet about Claude Code',
'username': 'testuser',
'likeCount': 42,
'retweetCount': 10,
'replyCount': 5,
'timeParsed': '2026-02-01T12:00:00.000Z'
}]
}
parsed = bird_x.parse_bird_response(sample)
print('Parsed count:', len(parsed))
print('First item keys:', sorted(parsed[0].keys()) if parsed else 'EMPTY')
print('URL:', parsed[0].get('url'))
print('Author:', parsed[0].get('author_handle'))
"
```
- [ ] Parses correctly with expected keys
- [ ] Handles both camelCase and snake_case fields
- [ ] URL, text, author, engagement metrics all present
### 1.4 SKILL.md Validation
```bash
# Verify YAML frontmatter parses correctly
python3 -c "
import yaml
with open('/Users/mvanhorn/last30days-skill-private/SKILL.md') as f:
content = f.read()
# Extract YAML between --- markers
parts = content.split('---', 2)
meta = yaml.safe_load(parts[1])
print('name:', meta.get('name'))
print('context:', meta.get('context'))
print('agent:', meta.get('agent'))
print('allowed-tools:', meta.get('allowed-tools'))
"
```
- [ ] `name` is `last30days` (not `last30daystest`)
- [ ] `context` is `fork`
- [ ] `agent` is `Explore`
- [ ] `allowed-tools` includes `Bash`, `WebSearch`
### 1.5 Diff Audit (Old vs New)
```bash
# Verify the only meaningful addition is bird_x.py
diff -rq ~/.claude/skills/last30days.backup-v1/scripts/lib/ \
/Users/mvanhorn/last30days-skill-private/scripts/lib/ 2>/dev/null
```
- [ ] Only new file is `bird_x.py`
- [ ] Modified files: `env.py` (source detection), `__init__.py` (exports)
- [ ] No unexpected deletions or renames
---
## Phase 2: User's Test Plan (Manual)
These require human judgment - evaluating quality, UX, and real-world behavior.
### 2.1 Basic Invocation (5 min)
Open a fresh Claude Code session after Phase 0 is complete.
| # | Test | Command | Pass Criteria |
|---|------|---------|---------------|
| 1 | Simple topic | `/last30days AI music generation` | Returns results, shows source stats |
| 2 | Topic + tool | `/last30days Suno prompts for music production` | Returns results + generates a prompt |
| 3 | Quick mode | `/last30days --quick TypeScript tips` | Faster, fewer results, still valid |
| 4 | Empty input | `/last30days` | Prompts for topic (doesn't crash) |
### 2.2 Bird CLI Verification (5 min)
| # | Test | What to Check |
|---|------|---------------|
| 1 | Source indicator | Output shows Bird as X source (not xAI) |
| 2 | X results quality | X/Twitter results are real, recent, have engagement metrics |
| 3 | Bird promo | If Bird NOT installed, shows non-blocking info banner |
| 4 | Mixed sources | Both Reddit (OpenAI) and X (Bird) results appear |
### 2.3 Fallback Behavior (5 min)
| # | Test | Setup | Expected |
|---|------|-------|----------|
| 1 | No Bird | `npm uninstall -g @steipete/bird` temporarily | Falls back to xAI or WebSearch |
| 2 | No API keys | Rename `~/.config/last30days/.env` temporarily | WebSearch-only mode works |
| 3 | Restore | Reinstall bird + restore .env | Full mode returns |
### 2.4 Output Quality (10 min)
Run 3 real research queries you care about. For each, evaluate:
- [ ] Results are actually from the last 30 days (not stale)
- [ ] Engagement metrics (likes, upvotes) are present and reasonable
- [ ] No duplicate results
- [ ] Sources are properly cited with URLs
- [ ] Synthesis is grounded in actual results (not hallucinated)
- [ ] Generated prompts (if requested) are usable
### 2.5 Comparison Test (10 min)
Before removing the backup, run the SAME query on both versions:
```bash
# New version (active)
/last30days [your topic]
# Old version (temporarily restore)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# New Claude Code session
/last30days [same topic]
# Then swap back
```
- [ ] New version produces equal or better results
- [ ] No features regressed
- [ ] Bird results add value over xAI-only
---
## Phase 3: Release Plan (30-Day Timeline)
**Target release date:** March 1, 2026 (conservative buffer before March 8 deadline)
### Week 1: Feb 6-12 - Clean & Test
| Day | Task | Owner |
|-----|------|-------|
| Feb 6 | Phase 0: Clean swap (remove old, activate new) | User |
| Feb 6 | Phase 1: Claude runs automated tests | Claude |
| Feb 7-8 | Phase 2: User runs manual tests (2.1-2.4) | User |
| Feb 9 | Phase 2.5: Comparison test | User |
| Feb 10-12 | Fix any issues found during testing | Claude + User |
### Week 2: Feb 13-19 - Harden
| Day | Task | Owner |
|-----|------|-------|
| Feb 13 | Run edge cases: unicode topics, very long topics, special chars | Claude |
| Feb 14 | Test with Bird logged out (auth expiry scenario) | User |
| Feb 15 | Review all error messages for clarity | Claude |
| Feb 16-17 | Update README.md with Bird CLI setup instructions | Claude |
| Feb 18-19 | Buffer for fixes | Claude + User |
### Week 3: Feb 20-26 - Pre-Release
| Day | Task | Owner |
|-----|------|-------|
| Feb 20 | Final diff audit: private repo vs public repo | Claude |
| Feb 21 | Strip any private/test artifacts (test symlinks, debug prints) | Claude |
| Feb 22 | Update SKILL.md description if needed | Claude |
| Feb 23 | Dry-run: push to a branch on public repo (not main) | User |
| Feb 24 | Test installation from the branch (fresh `~/.claude/skills/`) | User |
| Feb 25-26 | Buffer for fixes | Claude + User |
### Week 4: Feb 27 - Mar 1 - Ship
| Day | Task | Owner |
|-----|------|-------|
| Feb 27 | Merge branch to main on public repo | User |
| Feb 28 | Create GitHub release with changelog | Claude + User |
| Mar 1 | Delete backup: `rm -rf ~/.claude/skills/last30days.backup-v1` | User |
| Mar 1 | Archive private repo (optional) | User |
### Release Checklist (Final Gate)
Before merging to `main` on the public repo:
- [ ] All Phase 1 automated tests pass
- [ ] All Phase 2 manual tests pass
- [ ] Comparison test shows new >= old quality
- [ ] SKILL.md frontmatter is correct (`name: last30days`, not `last30daystest`)
- [ ] README.md documents Bird CLI setup
- [ ] No debug/test artifacts in codebase
- [ ] No hardcoded paths (e.g., `/Users/mvanhorn/...`)
- [ ] `.env` files are gitignored
- [ ] Git history is clean (no "test" or "WIP" commits on main)
- [ ] Bird CLI failure doesn't break the skill (graceful fallback verified)
### Rollback Plan (Emergency)
If something goes wrong after release:
```bash
# Option 1: Revert to backup (if still exists)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# Option 2: Git revert on public repo
cd ~/.claude/skills/last30days
git log --oneline -5 # find the last good commit
git revert HEAD # revert the merge commit
git push origin main
# Option 3: Pin to old version
cd ~/.claude/skills/last30days
git checkout cc892d7 # last known good commit from old version
```
---
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Bird CLI breaks after X API changes | Medium | Low | Fallback to xAI/WebSearch still works |
| Bird auth expires silently | Medium | Low | `is_bird_authenticated()` check + user message |
| Old xAI workflows regress | Low | High | Comparison test in Phase 2.5 |
| Hardcoded paths in codebase | Low | Medium | Grep for `/Users/mvanhorn` before release |
| SKILL.md name still says `last30daystest` | Low | High | Already fixed in commit `4e972d0` |
## References
- Private repo: `https://github.com/mvanhorn/last30days-skill-private.git`
- Public repo: `https://github.com/mvanhorn/last30days-skill.git`
- Bird CLI: `https://github.com/steipete/bird`
- Bird implementation plan: `docs/plans/2026-02-03-bird-cli-implementation.md`
- Bird integration design: `docs/plans/2026-02-03-bird-cli-integration-design.md`
docs/plans/2026-02-06-feat-visible-query-parsing-display-plan.md
-91
View File
@@ -1,91 +0,0 @@
---
title: "feat: Add visible query parsing display before research starts"
type: feat
date: 2026-02-06
---
# feat: Add Visible Query Parsing Display
## Overview
The last30days skill parses user intent (TOPIC, QUERY_TYPE, TARGET_TOOL) internally but never shows the user what it understood. The agent jumps straight from the user's `/last30days kanye west` into running tools with a generic "I'll start the research script and web searches in parallel."
Users expect to see a reformulation of their query — confirming what the agent understood before it starts searching. This builds trust and lets users course-correct before waiting for results.
## Problem Statement
Current behavior:
```
User: /last30days kanye west
Agent: I'll start the research script and web searches in parallel.
[immediately runs bash + WebSearch]
```
Expected behavior:
```
User: /last30days kanye west
Agent: 🔍 **kanye west** · News
Searching Reddit, X, and the web for the latest on kanye west...
[then runs bash + WebSearch]
```
The "Parse User Intent" section in SKILL.md tells the agent to store variables internally but never instructs it to **display** them.
## Proposed Solution
Add an explicit "Display your parsing" instruction between the "Parse User Intent" section and "Research Execution" section in SKILL.md. One new block of text — no code changes, no script changes.
## Acceptance Criteria
- [ ] Agent displays parsed TOPIC and QUERY_TYPE before running any tools
- [ ] Display is concise (1-2 lines, not a verbose block)
- [ ] Agent still runs script + WebSearch in parallel after displaying
- [ ] No changes to Python scripts — SKILL.md only
## Implementation
### SKILL.md Change
**File:** `/Users/mvanhorn/last30days-skill-private/SKILL.md`
After the "Store these variables" block (line ~38) and before "Research Execution" (line ~42), add:
```markdown
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
```
### Sync
After editing SKILL.md:
```bash
cp /Users/mvanhorn/last30days-skill-private/SKILL.md ~/.claude/skills/last30days/SKILL.md
```
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should display: 🔍 **kanye west** · News
2. `/last30days best MCP servers` — should display: 🔍 **best MCP servers** · Recommendations
3. `/last30days nano banana pro prompting for ChatGPT` — should display with tool mention
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Add display instruction between Parse User Intent and Research Execution |
docs/plans/2026-02-06-fix-last30days-v2-formatting-reddit-citations-plan.md
-167
View File
@@ -1,167 +0,0 @@
---
title: "fix: last30days v2 formatting, Reddit results, and citation verbosity"
type: fix
date: 2026-02-06
---
# fix: last30days v2 Formatting, Reddit Results, and Citation Verbosity
## Overview
Four bugs found during v2 testing across 4 queries (kanye west, howie.ai, nano banana pro prompting, open claw). The skill IS executing (the agent:Explore removal worked) but output quality has regressed from v1.
## Problem Statement
| # | Bug | Severity | Where |
|---|-----|----------|-------|
| 1 | Stats emoji tree format ignored 3/4 times - agent renders plain text dashes instead | High | `SKILL.md` |
| 2 | Reddit returns 0 results for popular topics (kanye west, howie.ai) | High | `scripts/lib/openai_reddit.py` |
| 3 | Citations too verbose - every sentence has `(per @x, @y, @z; r/sub)` making summary unreadable | Medium | `SKILL.md` |
| 4 | Kanye summary is wall of text - no bold headers or paragraph breaks like nano banana pro got | Medium | `SKILL.md` |
## Proposed Fixes
### Fix 1: Stats Emoji Format Enforcement
**Root cause:** The agent ignores the emoji tree template even with BAD/GOOD examples. The template uses box-drawing characters (├─ └─) that the agent treats as decorative, not mandatory.
**Approach:** Instead of relying on the agent to copy box-drawing characters, provide the template as a **literal fill-in-the-blank** with placeholders that are impossible to misinterpret.
**File:** `SKILL.md` (stats section, currently around line 190)
**Change:** Replace the current template + BAD/GOOD examples with a single, strict fill-in format:
```
Copy this EXACTLY, replacing only the {placeholders}:
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
```
Remove the separate BAD/GOOD section (it adds length without helping).
### Fix 2: Reddit Returning 0 Results
**Root cause (from code analysis):**
1. `openai_reddit.py:53-93` - The `REDDIT_SEARCH_PROMPT` instructs the OpenAI model to strip noise words before searching. For "kanye west" this isn't the issue (no noise words), but for "howie.ai" it might strip "ai".
2. `openai_reddit.py:160-166` - The search is restricted to `allowed_domains: ["reddit.com"]` which depends on OpenAI's web_search indexing of Reddit.
3. `last30days.py:474-490` - Post-retrieval filtering: `normalize.filter_by_date_range()` + `score.score_reddit_items()` + `dedupe.dedupe_reddit()` can discard all results if date confidence is low.
4. `score.py:151-157` - Items with no engagement metrics get `-10` penalty, low date confidence gets `-10`. Combined that's `-20` which may push score below threshold.
**Approach (multi-layered):**
**A. Add subreddit-targeted search fallback** in `openai_reddit.py`:
- When the first search returns < 3 results, add a second search prompt that explicitly queries: `"r/{topic} site:reddit.com"` and `"{topic} subreddit site:reddit.com"`
- This catches cases where OpenAI's web_search doesn't find the obvious subreddit
**B. Soften post-retrieval scoring** in `score.py`:
- Change the no-engagement penalty from `-10` to `-3` (missing metrics ≠ irrelevant)
- Change low date confidence penalty from `-10` to `-5`
**C. Add minimum result guarantee** in `last30days.py`:
- If scoring filters out ALL results, keep the top 3 by raw relevance regardless of score
- Log a warning: "All Reddit results scored below threshold, keeping top 3 by relevance"
**Files to change:**
- `scripts/lib/openai_reddit.py` - Add subreddit fallback search (lines ~160-180)
- `scripts/lib/score.py` - Soften penalties (lines ~151-157)
- `scripts/last30days.py` - Add minimum result guarantee (lines ~474-490)
### Fix 3: Citations Too Verbose
**Root cause:** The SKILL.md instruction says "Every insight MUST cite at least one source" with a GOOD example showing `(per @XXX, 15 likes; r/kanye thread with 200 upvotes)` - this is too much detail per citation and the agent over-applies it.
**Approach:** Dial back to "cite 1-2 sources per KEY PATTERN, not per sentence. Use short format."
**File:** `SKILL.md` (citation section, currently around line 158)
**Change the citation rule to:**
```
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
BAD: "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
GOOD: "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
### Fix 4: Summary Formatting (Wall of Text vs Structured)
**Root cause:** The SKILL.md template for PROMPTING/NEWS/GENERAL shows:
```
What I learned:
[2-4 sentences synthesizing...]
```
This gives the agent permission to write a dense paragraph. The nano banana pro test got good formatting because PROMPTING queries naturally produce structured patterns. NEWS queries (kanye) produce narratives that become walls of text.
**Approach:** Add explicit structure to the NEWS/GENERAL format with bold topic headers.
**File:** `SKILL.md` (summary display section, around line 158)
**Change the PROMPTING/NEWS/GENERAL template to:**
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
The bold topic headers force structure. Each topic gets its own paragraph with a line break. No more wall-of-text narratives.
## Acceptance Criteria
- [ ] **Fix 1:** Stats box uses emoji tree format ├─ 🟠 🔵 🌐 └─ 🗣️ in 4/4 test queries
- [ ] **Fix 2:** "kanye west" returns >0 Reddit threads (r/kanye exists and is active)
- [ ] **Fix 3:** Summary citations are 1 per insight, short format, no engagement metrics inline
- [ ] **Fix 4:** NEWS/GENERAL summaries use bold topic headers with paragraph breaks, not wall of text
## Test Plan
Re-run the same 4 queries after fixes:
1. `/last30days kanye west` — NEWS: should get Reddit results, structured summary, emoji stats
2. `/last30days howie.ai` — GENERAL: should get Reddit if available, citations not verbose
3. `/last30days nano banana pro prompting` — PROMPTING: should maintain current good quality, reduce citation density
4. `/last30days open claw` — GENERAL: should cite @handles in summary, emoji stats
## Files to Modify
| File | Fix | Change |
|------|-----|--------|
| `SKILL.md` | 1, 3, 4 | Stats template, citation rules, summary structure |
| `scripts/lib/openai_reddit.py` | 2 | Add subreddit fallback search |
| `scripts/lib/score.py` | 2 | Soften scoring penalties |
| `scripts/last30days.py` | 2 | Add minimum result guarantee |
## References
- Current SKILL.md: `~/.claude/skills/last30days/SKILL.md`
- Private repo: `/Users/mvanhorn/last30days-skill-private/`
- Old working SKILL.md: `~/.claude/skills/last30days.backup-v1/SKILL.md`
- Reddit search module: `scripts/lib/openai_reddit.py:53-93` (prompt), `:160-166` (API call)
- Scoring module: `scripts/lib/score.py:151-157` (penalties)
- Main pipeline: `scripts/last30days.py:474-490` (filtering)
docs/plans/2026-02-06-fix-skill-execution-fork-mode-plan.md
-177
View File
@@ -1,177 +0,0 @@
---
title: "fix: Skill execution broken - fork mode subagent ignores bash and text instructions"
type: fix
date: 2026-02-06
---
# fix: Skill Execution Broken in Fork Mode
## Overview
The last30days v2 skill stopped running its Python script and stopped showing acknowledgment text. The agent jumps straight to WebSearch, ignoring all instructions to run bash first or output text. Five attempted fixes all failed.
## Root Cause (Confirmed via Research)
**The old v1 skill worked by accident.** GitHub Issue #17283 documented that `context: fork` and `agent: Explore` were **silently ignored** in older Claude Code versions. The skill ran **inline** in the main conversation — not in a forked subagent. That's why:
- The user saw acknowledgment text (output inline to conversation)
- The bash script ran (main model followed instructions inline)
- Progress was visible (tool calls shown normally)
**Claude Code 2.1+ fixed the bug** and now properly honors `context: fork`. The skill now truly runs in an isolated subagent where:
- The model decides tool ordering independently
- Text output instructions are deprioritized vs tool calls
- "RUN THIS FIRST" instructions are **suggestions**, not commands
- There is **no mechanism** to force tool ordering in a forked subagent
**This is why every SKILL.md rewrite failed** — the problem isn't the instructions, it's `context: fork` itself.
## Evidence
| Attempt | What we tried | Result |
|---------|--------------|--------|
| 1 | "YOUR FIRST ACTION: Run this command. EXECUTE." | Agent ran script sometimes, never showed ack text |
| 2 | "YOUR FIRST OUTPUT — before ANY tool calls" + progress block | Agent ignored text, jumped to WebSearch |
| 3 | Moved progress block to very first section | Agent ignored it entirely |
| 4 | "DO NOT skip this. DO NOT jump to tool calls first." | Agent still jumped to WebSearch |
| 5 | Embedded echo in bash block + "Do NOT start with WebSearch" | Agent still jumped to WebSearch, never ran bash |
## Proposed Fix
### Option A: Remove `context: fork` (Recommended)
**Remove `context: fork` from frontmatter.** The skill runs inline in the main conversation, exactly like the old v1 skill accidentally did.
**Why this works:**
- Inline execution follows instructions sequentially
- Text output appears directly to the user
- Bash commands run when instructed
- This is how the "working" v1 skill actually operated
**File:** `SKILL.md` frontmatter
**Change from:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
context: fork
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
**Change to:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
That's it. Remove the one line.
**Then restore the old v1 instruction flow:**
1. "Parse User Intent" section FIRST (generates acknowledgment text)
2. "Research Execution" with bash command
3. "Do WebSearch" while script runs
4. Synthesize and present
### Option B: Keep `context: fork` + Use `!`command`` Preprocessing
Use shell preprocessing syntax (`!`command``) to run the script **before** the model even sees the prompt:
```markdown
## Research data (auto-fetched)
!`python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1`
```
**Risk:** Not confirmed that `$ARGUMENTS` works in `!`command`` context. More complex. The user still won't see progress text during preprocessing.
### Recommendation: Option A
Remove `context: fork`. It's one line. The old skill worked inline. The v2 skill should too. Option B is a backup if inline mode causes context window issues.
## Implementation
### Step 1: Remove `context: fork` from frontmatter
Single line removal in `SKILL.md`.
### Step 2: Restore v1-style instruction flow
The SKILL.md opening should match the public v1 pattern:
```markdown
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
[... topic/tool/query type parsing ...]
Store these variables:
- TOPIC = ...
- TARGET_TOOL = ...
- QUERY_TYPE = ...
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
**Step 2: Do WebSearch** (while script runs)
[... websearch queries based on QUERY_TYPE ...]
**Step 3: Wait for script to complete**
[... synthesis instructions ...]
```
The key structural elements from v1 that need to return:
1. Descriptive intro paragraph
2. "Parse User Intent" BEFORE any tool calls
3. Script execution as a clearly labeled step
4. WebSearch as step 2 (not step 1)
### Step 3: Keep all v2 improvements
The v2-specific improvements (citation rules, stats template, Reddit fallback, scoring changes) stay. Only the frontmatter and instruction flow change.
### Step 4: Sync and test
Copy to `~/.claude/skills/last30days/SKILL.md`, test in new session.
## Acceptance Criteria
- [ ] `context: fork` removed from SKILL.md frontmatter
- [ ] Agent outputs acknowledgment text before running tools
- [ ] Python script actually executes (Reddit + X results appear)
- [ ] WebSearch supplements, doesn't replace, script results
- [ ] Stats emoji tree format renders correctly
- [ ] Citations are sparse (1 per insight, not 3-5)
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should see ack text, script runs, Reddit results
2. `/last30days open claw` — should see ack text, X results via Bird
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Remove `context: fork`, restore v1 instruction flow |
## References
- GitHub Issue #17283: `context: fork` was silently ignored (the bug that made v1 work)
- Claude Code Skills docs: `!`command`` preprocessing syntax
- Claude Code Subagents docs: `agent: Explore` uses Haiku, read-only tools
- Public v1 SKILL.md: `github.com/mvanhorn/last30days-skill`
docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md
-385
View File
@@ -1,385 +0,0 @@
---
title: "test: Compare v1 (public) vs v2 (private) last30days output quality"
type: test
date: 2026-02-06
---
# test: V1 vs V2 Comparison Test Plan
## Overview
Run the same queries through both the public v1 and private v2 of last30days, compare output quality across 7 dimensions, and determine if v2 is ready to ship as the new public version.
**This plan also includes a full feature audit** identifying everything v1 has that v2 is missing — some of those gaps need fixing before shipping.
---
## How to Run the Comparison
### Setup
**V1 (public upstream):** Check out upstream SKILL.md temporarily:
```bash
# Save current v2
cp ~/.claude/skills/last30days/SKILL.md ~/.claude/skills/last30days/SKILL.md.v2
# Install v1 from upstream
cd /Users/mvanhorn/last30days-skill-private
git show upstream/main:SKILL.md > ~/.claude/skills/last30days/SKILL.md
```
Run test queries in a NEW Claude Code session (one session per query to avoid context bleed). Save output.
**V2 (private current):** Restore v2:
```bash
cp ~/.claude/skills/last30days/SKILL.md.v2 ~/.claude/skills/last30days/SKILL.md
```
Run same queries in NEW sessions. Save output.
---
## ALL Test Queries
### From README Examples (13 documented use cases)
Every single example from the README, in order:
| # | Query | Type | README Section |
|---|-------|------|---------------|
| 1 | `prompting techniques for chatgpt for legal questions` | PROMPTING + TOOL | Example: Legal Prompting |
| 2 | `best clawdbot use cases` | RECOMMENDATIONS | Example: ClawdBot Use Cases |
| 3 | `how to best setup clawdbot` | HOW-TO | Example: ClawdBot Setup |
| 4 | `prompting tips for nano banana pro for ios designs` | PROMPTING + TOOL | Example: iOS App Mockup |
| 5 | `top claude code skills` | RECOMMENDATIONS | Example: Top Claude Code Skills |
| 6 | `using ChatGPT to make images of dogs` | GENERAL | Example: Dog as Human |
| 7 | `research best practices for beautiful remotion animation videos in claude code` | PROMPTING | Example: Remotion Launch Video |
| 8 | `photorealistic people in nano banana pro` | PROMPTING | Example: Photorealistic Portraits |
| 9 | `What are the best rap songs lately` | RECOMMENDATIONS | Example: Best Rap Songs |
| 10 | `what are people saying about DeepSeek R1` | NEWS | Example: DeepSeek R1 |
| 11 | `best practices for cursor rules files for Cursor` | PROMPTING | Example: Cursor Rules |
| 12 | `prompt advice for using suno to make killer songs in simple mode` | PROMPTING | Example: Suno AI Music |
| 13 | `how do I use Codex with Claude Code on same app to make it better` | HOW-TO | Example: Codex + Claude Code |
### From Plan Documents (4 additional battle-tested queries)
| # | Query | Type | Source |
|---|-------|------|--------|
| 14 | `kanye west` | NEWS | fix-v2-formatting plan, most-tested query |
| 15 | `howie.ai` | GENERAL | fix-v2-formatting plan, edge case (domain as topic) |
| 16 | `open claw` | GENERAL | fix-v2-formatting plan, X-heavy sources |
| 17 | `nano banana pro prompting` | PROMPTING | fix-v2-formatting plan |
### Follow-up Vision Tests (pick 4 from above, ask a follow-up)
These test the prompt-generation phase specifically:
| Base Query | Follow-up Vision |
|------------|-----------------|
| #4 (nano banana pro ios) | "make a mock-up of an app for moms who swim" |
| #6 (ChatGPT dog images) | "what would my dog look like as a human prompt" |
| #12 (suno music) | "Rap song about self aware AI that loves Claude Code" |
| #13 (codex + claude code) | "how do I build a review loop workflow" |
---
## FEATURE AUDIT: V1 vs V2
### Section-by-section comparison
I diffed the full v1 (upstream/main) SKILL.md against the current v2. Here's everything.
#### KEPT (in both versions) ✅
| Feature | V1 Location | V2 Location | Notes |
|---------|------------|------------|-------|
| Parse User Intent section | Lines 23-48 | Lines 12-38 | Same logic |
| QUERY_TYPE detection (4 types) | Lines 29-36 | Lines 18-22 | Same types |
| "Don't ask about tool before research" | Lines 49-51 | Lines 31-33 | Same rule |
| Store variables block | Lines 53-56 | Lines 35-38 | Same |
| Research script execution | Lines 81-86 | Lines 59-62 | Same command |
| WebSearch by QUERY_TYPE | Lines 99-127 | Lines 77-98 | Same queries |
| "Use user's exact terminology" | Lines 129-133 | Lines 100-101 | V2 shorter but same intent |
| Judge Agent synthesis | Lines 143-151 | Lines 113-124 | Same logic |
| Internalize research (ground in actual content) | Lines 159-165 | Lines 128-135 | V2 shorter |
| RECOMMENDATIONS: extract specific names | Lines 167-177 | Lines 137-145 | Same, v2 removes BAD/GOOD example |
| Prompt format matching | Lines 193-196 | Lines 149-153 | Same |
| Summary + Stats + Invitation flow | Lines 200-250 | Lines 157-236 | Same structure, different details |
| Wait for user's vision | Lines 254-258 | Lines 240-242 | Same |
| Write ONE perfect prompt | Lines 262-275 | Lines 246-266 | Same structure |
| Context memory | Lines 298-316 | Lines 278-288 | V2 shorter |
| Output summary footer | Lines 320-340 | Lines 292-302 | Different format |
| Depth options (quick/default/deep) | Lines 135-139 | Lines 106-109 | Same |
#### ADDED in V2 (improvements) ✨
| Feature | What it does | V2 Location |
|---------|-------------|------------|
| **Query parsing display** | Shows `🔍 **{TOPIC}** · {QUERY_TYPE}` before tools | Lines 40-53 |
| **Sparse citation rules** | BAD/GOOD examples, "1 per pattern, short format" | Lines 186-193 |
| **Bold topic headers** | `**{Topic 1}** — [1-2 sentences, per source]` format | Lines 195-208 |
| **Strict stats template** | "NEVER use plain text dashes", fill-in-blank | Lines 217-230 |
| **RECOMMENDATIONS source attribution** | Each item MUST have Sources: line with @handles | Lines 178-182 |
| **Reddit 0 results handling** | Explicit instruction for 0-thread line | Line 229 |
| **Bird CLI in stats** | "(via Bird/xAI)" notation | Line 223 |
#### ❌ MISSING FROM V2 — Features V1 Has That V2 Dropped
These are the regressions. Some are intentional simplifications, others are real gaps.
**1. Use Cases Block (intro section)**
- **V1 has:** 4 use case examples right after the intro: Prompting, Recommendations, News, General — with concrete examples
- **V2 has:** Nothing. Just the intro paragraph.
- **Impact:** LOW. The query type detection handles this. But it was nice onboarding.
- **Verdict:** Skip — not needed for execution quality.
**2. Setup Check Section (API key guidance)**
- **V1 has:** Full section explaining 3 modes (Full/Partial/Web-Only), first-time setup bash script, "API keys are OPTIONAL" messaging
- **V2 has:** Nothing. Script auto-detects.
- **Impact:** LOW for experienced users. HIGH for first-time users who don't have keys.
- **Verdict:** Skip for now — script handles auto-detection. Consider adding back for public release.
**3. Anti-Pattern Examples (synthesis quality guard)**
- **V1 has:** Explicit anti-pattern block: "If user asks about 'clawdbot skills' and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as 'Claude Code skills' just because both involve 'skills'." Plus BAD/GOOD synthesis examples for RECOMMENDATIONS.
- **V2 has:** Only "Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge" — no concrete examples.
- **Impact:** MEDIUM-HIGH. Without concrete anti-patterns, the agent may conflate similar-sounding things.
- **Verdict:** ⚠️ ADD BACK. At minimum, restore the BAD/GOOD RECOMMENDATIONS example and the "don't conflate" warning.
**4. Self-Check Instruction (pre-display validation)**
- **V1 has:** "SELF-CHECK before displaying: Re-read your 'What I learned' section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it."
- **V2 has:** Nothing.
- **Impact:** MEDIUM. The self-check forces the model to validate its own output.
- **Verdict:** ⚠️ ADD BACK. One line costs nothing and catches hallucination.
**5. Quality Checklist for Prompts ⭐**
- **V1 has:** Explicit checklist before delivering a prompt:
```
### Quality Checklist:
- [ ] FORMAT MATCHES RESEARCH - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
```
- **V2 has:** Only "If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT." — one line instead of 5 checks.
- **Impact:** HIGH. This is likely what the user noticed as missing — v1 prompts felt more polished because the agent ran a checklist before delivering.
- **Verdict:** ⚠️ ADD BACK. This is the "that's a great prompt" quality feel.
**6. Prompt Format Anti-Pattern**
- **V1 has:** "ANTI-PATTERN: Research says 'use JSON prompts with device specs' but you write plain prose. This defeats the entire purpose of the research."
- **V2 has:** Only the positive instruction (use the format research recommends).
- **Impact:** MEDIUM. Negative examples ("don't do this") are powerful for LLMs.
- **Verdict:** ⚠️ ADD BACK. One line.
**7. "IF USER ASKS FOR MORE OPTIONS" Section**
- **V1 has:** "Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested."
- **V2 has:** Nothing about handling multi-prompt requests.
- **Impact:** LOW-MEDIUM. Without it, agent might dump multiple prompts unprompted.
- **Verdict:** ⚠️ ADD BACK. Two lines.
**8. Web-Only Mode Stats Template + Promo**
- **V1 has:** Separate stats template for web-only mode with "💡 Want engagement metrics? Add API keys..." promo
- **V2 has:** Only the full-mode template. If running web-only, agent has no guidance.
- **Impact:** MEDIUM for users without API keys.
- **Verdict:** Consider adding back for public release. Lower priority for now.
**9. TARGET_TOOL Question Template**
- **V1 has:** Explicit AskUserQuestion block with 4 options: [Most relevant tool], Nano Banana Pro, ChatGPT/Claude, Other
- **V2 has:** "run research first, then ask AFTER showing results" — but no actual question template.
- **Impact:** LOW-MEDIUM. Agent will still ask, just less structured.
- **Verdict:** Skip — not critical.
**10. Context Memory: "Don't re-search" Instructions**
- **V1 has:** Explicit "DO NOT run new WebSearches — you already have the research. Answer from what you learned. Cite the Reddit threads, X posts, and web sources."
- **V2 has:** Only "Only do new research if the user explicitly asks about a DIFFERENT topic."
- **Impact:** MEDIUM. Without the explicit ban, agent may re-search on follow-ups, wasting time.
- **Verdict:** ⚠️ ADD BACK. Three lines.
**11. Output Summary Footer (emoji + engagement counts)**
- **V1 has:** `📚 Expert in: {TOPIC} for {TARGET_TOOL}` and `📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages`
- **V2 has:** `Expert in: {TOPIC} for {TARGET_TOOL}` and `Based on: {n} Reddit threads + {n} X posts + {n} web pages` — no emoji, no engagement counts.
- **Impact:** LOW but noticeable. The emoji + counts make the footer feel more substantial.
- **Verdict:** ⚠️ ADD BACK. Trivial fix.
---
## Priority Fix List (Before Shipping V2 as Public)
Based on the audit, these should be restored in V2 before it replaces V1:
### Must Fix (affects output quality)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 1 | **Quality Checklist for prompts** | The "that's a great prompt" feel. V1's 5-point checklist made prompts more polished. | Add 8 lines to SKILL.md |
| 2 | **Anti-pattern examples** | BAD/GOOD synthesis examples prevent agent from conflating research. | Add 5 lines |
| 3 | **Self-check instruction** | One-line pre-display validation catches hallucination. | Add 2 lines |
| 4 | **Context Memory: don't re-search** | Prevents wasting time re-searching on follow-ups. | Add 3 lines |
### Should Fix (polish)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 5 | **Prompt format anti-pattern** | Negative example reinforces "match the format". | Add 2 lines |
| 6 | **"IF USER ASKS FOR MORE OPTIONS"** | Prevents prompt dumping. | Add 2 lines |
| 7 | **Output footer emoji + engagement counts** | More polished footer. | Edit 3 lines |
### Skip for Now (nice-to-have for public release)
| # | Missing Feature | Why Skip |
|---|----------------|----------|
| 8 | Use cases block (intro) | Doesn't affect execution |
| 9 | Setup Check section | Script auto-detects; add back for public README |
| 10 | Web-only mode stats + promo | Lower priority, most users have keys |
| 11 | TARGET_TOOL question template | Agent handles this naturally |
---
## Scoring Dimensions (1-5 scale, 7 dimensions)
### 1. Query Parsing Display
Does the agent show what it understood before starting research?
| Score | Criteria |
|-------|----------|
| 1 | No acknowledgment, jumps straight to tools |
| 2 | Generic "I'll research this" with no specifics |
| 3 | Mentions the topic but not query type |
| 4 | Shows topic + query type clearly |
| 5 | Shows topic + query type + reformulated search terms |
### 2. Source Coverage
Did it actually use Reddit, X, AND web — or skip sources?
| Score | Criteria |
|-------|----------|
| 1 | WebSearch only, script didn't run |
| 2 | Script ran but returned 0 from one major source |
| 3 | 2 of 3 sources returned results |
| 4 | All 3 sources returned results |
| 5 | All 3 sources + good volume (10+ Reddit, 10+ X, 5+ web) |
### 3. Citation Quality
Are citations sparse and useful, or verbose and noisy?
| Score | Criteria |
|-------|----------|
| 1 | Every sentence has 3+ citations chained |
| 2 | Most sentences have multiple citations |
| 3 | 1-2 citations per insight, some over-citing |
| 4 | 1 citation per pattern, short format |
| 5 | Sparse citations that prove research is real without cluttering |
### 4. Summary Structure
Is the "What I learned" section scannable or a wall of text?
| Score | Criteria |
|-------|----------|
| 1 | Single paragraph wall of text |
| 2 | Multiple paragraphs but no structure |
| 3 | Some bold text but inconsistent |
| 4 | Bold topic headers with 1-2 sentence explanations |
| 5 | Clean topic headers + KEY PATTERNS list, easy to scan |
### 5. Stats Box Format
Does the emoji stats tree render correctly?
| Score | Criteria |
|-------|----------|
| 1 | No stats shown |
| 2 | Stats shown but plain text dashes, no emoji |
| 3 | Partial emoji format, some lines wrong |
| 4 | Correct ├─ └─ │ format with emoji, minor issues |
| 5 | Perfect emoji tree with accurate counts and top voices |
### 6. Research Grounding
Does the synthesis reflect the ACTUAL research, or generic pre-training knowledge?
| Score | Criteria |
|-------|----------|
| 1 | Entirely generic knowledge, no research content |
| 2 | Mentions some research but mostly generic |
| 3 | Mix of research and generic, some conflation |
| 4 | Clearly grounded in research, minor generic leakage |
| 5 | Every insight traceable to a specific source from the research |
### 7. Prompt Quality (follow-up tests only)
When user shares vision, is the generated prompt good?
| Score | Criteria |
|-------|----------|
| 1 | Generic prompt that ignores research |
| 2 | Mentions research topics but generic structure |
| 3 | Uses some research insights, decent prompt |
| 4 | Tailored to research, correct format for target tool |
| 5 | Uses research-recommended format, specific techniques, ready to paste, "that's a great prompt" feel |
---
## Comparison Scorecard Template
```
Query: [query text]
Version: V1 / V2
Date: YYYY-MM-DD
| Dimension | Score (1-5) | Notes |
|---------------------|-------------|-------|
| Query Parsing | | |
| Source Coverage | | |
| Citation Quality | | |
| Summary Structure | | |
| Stats Box Format | | |
| Research Grounding | | |
| Prompt Quality | | (follow-up tests only) |
| **TOTAL** | **/35** | |
Script output:
- Reddit: ___ threads / ___ upvotes / ___ comments
- X: ___ posts / ___ likes / ___ reposts
- Web: ___ pages
Observations:
[Free text notes]
```
---
## Execution Plan
### Phase 1: Fix the gaps first
Apply the 7 "Must Fix" + "Should Fix" items from the audit to V2 SKILL.md. This takes ~20 minutes since it's all small text additions.
### Phase 2: Smoke test (4 queries)
Run queries #14 (kanye west), #2 (best clawdbot use cases), #8 (photorealistic nano banana pro), #10 (DeepSeek R1) on V2 only. Verify the fixes work.
### Phase 3: Full comparison (all 17 queries)
Run all 17 queries on both V1 and V2. Fill scorecards.
### Phase 4: Follow-up vision tests (4 queries)
Run the 4 follow-up vision tests. Compare prompt quality — this is where the quality checklist fix matters most.
### Phase 5: Analysis
- Sum scores per version across all queries
- Identify any dimension where v1 consistently beats v2
- Decision: ship v2, or fix more gaps first
## Acceptance Criteria
- [x] Feature audit complete (this document)
- [x] Must-fix gaps restored in V2 SKILL.md
- [ ] All 17 queries run on V2
- [ ] At least 4 queries run on V1 for comparison
- [ ] 4 follow-up vision tests completed
- [ ] Scorecards filled for each
- [ ] Total score comparison documented
- [ ] Any V1 > V2 regressions identified with fix plan
- [ ] Go/no-go decision on shipping v2 as public
## Files
| File | Purpose |
|------|---------|
| `docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md` | This plan |
| `SKILL.md` | Apply Must Fix + Should Fix items |
| `docs/test-results/v1-vs-v2-comparison.md` | Results (to be created) |
docs/plans/2026-02-07-feat-bundle-bird-x-search-plan.md
-208
View File
@@ -1,208 +0,0 @@
---
title: "feat: Bundle Bird X search client to eliminate npm dependency"
type: feat
date: 2026-02-07
---
# feat: Bundle Bird X search client to eliminate npm dependency
## Overview
Replace the `subprocess.run(["bird", "search", ...])` dependency in `bird_x.py` with a vendored Node.js module that calls Twitter's GraphQL search API directly. This eliminates the need for users to `npm install -g @steipete/bird` and protects against the package being removed from npm.
Bird is MIT-licensed. We have the full compiled package archived at `vendor/steipete-bird-0.8.0.tgz` and forked to `github.com/mvanhorn/bird-cli-archive`.
## Problem Statement
@steipete deleted Bird's GitHub repo on 2026-02-07. The npm package still works today, but if he unpublishes it from npm:
- New users can't `npm install -g @steipete/bird`
- The `bird` binary disappears from PATH on fresh installs
- `bird_x.py` returns 0 X results for everyone without an xAI API key
- /last30days V2's headline feature ("free X search") stops working for new users
## Proposed Solution
**Vendor Bird's search-only subset as a Node.js module inside /last30days, called from Python via `subprocess.run(["node", ...])`.**
This is the minimal-change approach:
- Keep Python as the orchestrator (bird_x.py stays mostly the same)
- Replace `subprocess.run(["bird", "search", ...])` with `subprocess.run(["node", "vendor/bird-search.mjs", ...])`
- Extract only the search-related code from Bird (not posting, bookmarks, lists, etc.)
- Cookie auth stays the same (environment variables or browser extraction)
### Why not rewrite in pure Python?
Bird's search client uses Twitter's internal GraphQL API with:
- Rotating QueryIDs (hardcoded + runtime refresh from x.com)
- Specific request header construction (bearer token, csrf, client UUIDs)
- Cursor-based pagination with Twitter-specific response parsing
- The `@steipete/sweet-cookie` dependency for browser cookie extraction
Porting all of this to Python is ~1000 lines of fragile reverse-engineering. Vendoring the working JS code is faster, safer, and easier to maintain since the archive includes source maps for debugging.
## Technical Approach
### What we need from Bird
Only 8 files from `dist/lib/` (out of 30+):
1. `twitter-client-base.js` - HTTP client, auth headers, rate limiting
2. `twitter-client-search.js` - Search mixin (the core feature)
3. `twitter-client-utils.js` - Tweet parsing, cursor extraction
4. `twitter-client-constants.js` - API endpoints, QueryIDs
5. `twitter-client-types.js` - TypeScript type stubs
6. `cookies.js` - Cookie resolution (env vars, browser extraction)
7. `runtime-query-ids.js` - QueryID refresh from x.com
8. `paginate-cursor.js` - Cursor pagination helper
Plus:
- `features.json` - GraphQL feature flags
- `query-ids.json` - Hardcoded QueryID fallbacks
### What we DON'T need
Posting, bookmarks, lists, timelines, engagement, follow, media, news, user lookup, user tweets - all the non-search mixins. This cuts the vendored code roughly in half.
### Architecture
```
scripts/
lib/
bird_x.py # MODIFIED - calls node instead of bird binary
vendor/
bird-search/
bird-search.mjs # NEW - thin CLI wrapper, ~40 lines
lib/ # VENDORED - subset of Bird's dist/lib/
twitter-client-base.js
twitter-client-search.js
twitter-client-utils.js
twitter-client-constants.js
twitter-client-types.js
cookies.js
runtime-query-ids.js
paginate-cursor.js
features.json
query-ids.json
node_modules/ # VENDORED - sweet-cookie only
@steipete/
sweet-cookie/
package.json # Minimal, points to bird-search.mjs
LICENSE # Bird's MIT license (required by MIT terms)
```
### Implementation
#### 1. Create `bird-search.mjs` wrapper (~40 lines)
A minimal Node.js script that:
- Accepts: `node bird-search.mjs <query> --count <n> --json`
- Creates a TwitterClient with search mixin only
- Resolves cookies (env vars first, then browser extraction)
- Calls `client.search(query, count)`
- Outputs JSON to stdout
- Exits with code 0 on success, 1 on error
This replaces the full `bird` CLI binary. Same interface, fraction of the code.
#### 2. Modify `bird_x.py` - change subprocess target
```python
# BEFORE (current)
cmd = ["bird", "search", query, "-n", str(count), "--json"]
# AFTER (vendored)
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
cmd = ["node", bird_search, query, "--count", str(count), "--json"]
```
Same subprocess pattern. Same JSON output format. Minimal diff.
#### 3. Update auth check functions
```python
# BEFORE
def is_bird_installed() -> bool:
return shutil.which("bird") is not None
# AFTER
def is_bird_installed() -> bool:
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
return bird_search.exists() and shutil.which("node") is not None
```
`is_bird_authenticated()` changes from `bird whoami` to a quick Node.js cookie check or environment variable check.
`install_bird()` becomes a no-op (already vendored) or removes itself entirely.
#### 4. Vendor sweet-cookie
`@steipete/sweet-cookie` is the only runtime dependency. It handles browser cookie extraction on macOS/Linux. Options:
**Option A (recommended):** Vendor sweet-cookie into `vendor/bird-search/node_modules/`. It's small (one file). This makes the skill fully self-contained with zero npm installs.
**Option B:** Fall back to environment variables only (no browser cookie extraction). Users would need to manually set `AUTH_TOKEN` and `CT0` env vars. Simpler but worse UX.
Recommend Option A - vendor it.
#### 5. Update user-facing docs
- `README.md` - Remove "Install Bird CLI" section, replace with "Requires Node.js 22+"
- `SKILL.md` - Remove Bird CLI installation instructions
- Keep the fallback chain: vendored Bird search -> xAI API key -> web-only
## Acceptance Criteria
- [ ] `bird_x.py` calls vendored Node.js module instead of `bird` binary
- [ ] `search_x()` returns identical JSON format (no downstream changes needed)
- [ ] `search_handles()` works with vendored module
- [ ] Cookie auth works via environment variables (`AUTH_TOKEN`, `CT0`)
- [ ] Cookie auth works via browser extraction (sweet-cookie)
- [ ] `is_bird_installed()` checks for vendored module + Node.js
- [ ] `install_bird()` removed or returns success immediately
- [ ] No `npm install -g @steipete/bird` required anywhere
- [ ] Bird's MIT LICENSE included in vendor directory
- [ ] README updated to remove Bird CLI install steps
- [ ] SKILL.md updated to remove Bird CLI references
- [ ] Works on macOS (primary) and Linux
- [ ] Fallback to xAI API key still works if vendored search fails
## Files Changed
| File | Action | Description |
|------|--------|-------------|
| `scripts/lib/bird_x.py` | MODIFY | Replace `["bird", ...]` subprocess calls with `["node", "vendor/bird-search/bird-search.mjs", ...]` |
| `scripts/lib/vendor/bird-search/bird-search.mjs` | CREATE | Thin Node.js wrapper that imports Bird's search client and outputs JSON |
| `scripts/lib/vendor/bird-search/lib/*.js` | VENDOR | 8 files from Bird's dist/lib/ (search subset only) |
| `scripts/lib/vendor/bird-search/lib/features.json` | VENDOR | GraphQL feature flags |
| `scripts/lib/vendor/bird-search/lib/query-ids.json` | VENDOR | Hardcoded QueryID fallbacks |
| `scripts/lib/vendor/bird-search/node_modules/` | VENDOR | sweet-cookie package |
| `scripts/lib/vendor/bird-search/package.json` | CREATE | Minimal package.json for module resolution |
| `scripts/lib/vendor/bird-search/LICENSE` | COPY | Bird's MIT license |
| `README.md` | MODIFY | Remove Bird CLI install section, add Node.js 22+ requirement |
| `SKILL.md` | MODIFY | Remove Bird CLI references |
## Dependencies & Risks
**Node.js 22+ required** - Users who had Bird CLI already have Node.js. This is not a new dependency, just a version requirement. Claude Code environments typically have Node.js.
**Twitter API changes** - The GraphQL QueryIDs may rotate. Bird includes a runtime refresh mechanism (`runtime-query-ids.js`) that fetches new IDs from x.com. This is vendored and will continue working.
**sweet-cookie platform support** - Browser cookie extraction only works on macOS (Safari, Chrome, Firefox) and Linux (Chrome, Firefox). Windows users need manual env vars. This matches Bird CLI's existing behavior.
**Legal** - Bird is MIT licensed. MIT requires including the license notice in copies. We include `LICENSE` in the vendor directory. Using Twitter's internal API is the same legal gray area Bird always operated in - user accepted this when they used Bird.
## What This Does NOT Change
- Python remains the orchestrator - bird_x.py still does query construction, retry logic, response parsing
- The fallback chain stays: vendored search -> xAI API -> web-only
- Cookie auth mechanism is identical (env vars or browser extraction)
- JSON output format is identical - no changes needed in score.py or format.py
- xai_x.py is completely untouched
## References
- Bird CLI archive: `github.com/mvanhorn/bird-cli-archive`
- Local vendor tarball: `vendor/steipete-bird-0.8.0.tgz`
- Bird search implementation: `bird-cli-archive/dist/lib/twitter-client-search.js`
- Current bird_x.py: `scripts/lib/bird_x.py`
- Fallback chain: `scripts/lib/env.py:get_x_source()`
docs/plans/2026-02-07-feat-smart-supplemental-search-plan.md
-263
View File
@@ -1,263 +0,0 @@
---
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
type: feat
date: 2026-02-07
---
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
## Overview
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
## Problem Statement / Motivation
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
## Proposed Solution
### Architecture: Two-Phase Search
```
CURRENT (Phase 1 — unchanged):
Broad topic search → Reddit results + X results
↓
NEW (Phase 2 — supplemental):
Extract entities from Phase 1 results
↓ ↓
[SUBREDDITS] [@HANDLES + #HASHTAGS]
↓ ↓
Targeted Reddit Targeted X searches
searches per sub per handle/hashtag
↓ ↓
Merge + dedupe with Phase 1 results
```
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
### Feature 1: Entity Extraction Module (NEW FILE)
**File: `scripts/lib/entity_extract.py`**
A lightweight module that parses Phase 1 results and extracts:
**From X results:**
- `@handles` — from `author_handle` field + any @mentions in post text
- `#hashtags` — from post text
- Rank by frequency: handles that appear 2+ times are "key voices"
**From Reddit results:**
- `subreddit` names — from the `subreddit` field on each result
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
- Rank by frequency: subreddits with 2+ threads are "core communities"
**Output:**
```python
{
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
"x_hashtags": ["#openclaw", "#aitools"],
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
}
```
**Rules:**
- No hardcoded entities — everything discovered dynamically from Phase 1
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
- Skip the original topic's "obvious" subreddit if it was already searched
### Feature 2: Supplemental X Search (Bird)
**File: modify `scripts/lib/bird_x.py`**
Add a `search_handles()` function:
```python
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
"""Search top handles for topic-related content."""
results = []
for handle in handles[:5]:
# Uses Bird's support for X search operators
query = f"from:{handle} {topic} since:{from_date}"
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
# ... parse results, add to list
return results
```
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
```python
# xAI supports filtering to specific handles (max 10)
tools = [{
"type": "x_search",
"x_handles": {"allowed_x_handles": top_handles[:10]}
}]
```
### Feature 3: Supplemental Reddit Search
**File: modify `scripts/lib/openai_reddit.py`**
Add a `search_subreddits()` function:
```python
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
"""Search discovered subreddits for topic-related content."""
# Build multi-subreddit query for the OpenAI web_search prompt
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
# ... single OpenAI API call, same pattern as existing search
```
**Alternative approach — Reddit JSON API (free, no API key):**
```python
def search_subreddit_json(subreddit: str, topic: str) -> list:
"""Search a specific subreddit via Reddit's free JSON endpoint."""
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
# ... parse JSON response
```
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
### Feature 4: Orchestration Changes
**File: modify `scripts/last30days.py`**
After Phase 1 completes and enrichment is done, run Phase 2:
```python
# Phase 1 (existing — unchanged)
reddit_items, x_items = run_parallel_search(...)
# Phase 2 (new — supplemental)
if reddit_items or x_items:
entities = entity_extract.extract(reddit_items, x_items)
supplemental_reddit = []
supplemental_x = []
# Run supplemental searches in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
if entities["reddit_subreddits"]:
reddit_future = executor.submit(
openai_reddit.search_subreddits,
entities["reddit_subreddits"], topic, ...
)
if entities["x_handles"] and bird_available:
x_future = executor.submit(
bird_x.search_handles,
entities["x_handles"], topic, from_date, ...
)
# Merge with Phase 1
all_reddit = reddit_items + supplemental_reddit
all_x = x_items + supplemental_x
# Dedupe handles the rest
```
**Depth-dependent behavior:**
| Depth | Phase 2 behavior |
|---|---|
| `--quick` | Skip Phase 2 entirely (speed matters) |
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
**File: modify `scripts/lib/bird_x.py`**
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
```python
def expand_thread(tweet_id: str) -> list:
"""Fetch full thread for a high-engagement tweet."""
cmd = ["bird", "thread", tweet_id, "--json"]
# ... parse thread, extract key replies
```
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
## Technical Considerations
### Performance
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
### Cost
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
- X handle search via Bird: Free (uses your X login)
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
- Thread expansion: Free via Bird
### No New Dependencies
- Entity extraction is string parsing — no NLP libraries needed
- Reddit `.json` endpoint uses existing `http.py` transport
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
### Backward Compatibility
- Phase 2 is purely additive — all existing behavior unchanged
- If Phase 2 finds nothing, output is identical to current
- Deduplication handles any overlap between Phase 1 and Phase 2
## Acceptance Criteria
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
- [x] Supplemental X searches via Bird find additional content from key handles
- [x] Supplemental Reddit searches find content in discovered subreddits
- [x] Phase 2 results are properly merged and deduped with Phase 1
- [x] `--quick` mode skips Phase 2 entirely
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
- [x] No performance regression on `--quick` mode
- [ ] Default mode adds < 10 seconds of latency
- [x] Works with Bird-only, xAI-only, and both-available configurations
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
## Implementation Order
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
6. (Stretch) Thread expansion for high-engagement posts
## Research Sources
### Reddit Search Techniques
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
### X/Twitter Search Techniques
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
### Key Insight
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
## What We're NOT Doing
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
- **Not changing the output format** — Phase 2 results merge seamlessly
- **Not hardcoding any entities** — all discovery is dynamic from search results
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
- **Not replacing the current search** — Phase 2 supplements Phase 1
docs/plans/2026-02-07-fix-x-search-query-construction-plan.md
-147
View File
@@ -1,147 +0,0 @@
---
title: "fix: X search query too restrictive, returns 0 results on popular topics"
type: fix
date: 2026-02-07
---
# fix: X search query too restrictive, returns 0 results on popular topics
## Problem
`/last30days vibe motion best prompt techniques` returned **0 X posts** despite Vibe Motion being actively discussed on X (screenshots show posts from @Godid242, @KamilStanuch, @ColdStartTheory, @higgsfield_ai).
Root cause: `_extract_core_subject()` in `bird_x.py` produces overly specific queries. Bird/X search uses **literal keyword AND matching** — ALL words must appear in a tweet. The function kept 4 keywords (`vibe motion prompt techniques`) when only 2 (`vibe motion`) were needed.
## Three Bugs Found
### Bug 1: Multi-word noise phrases never match
```python
# Current code (bird_x.py:24-38)
noise = ['best', ..., 'what are', 'what is', 'how to', 'tips for', ...]
words = topic.lower().split() # splits into individual words
result = [w for w in words if w not in noise] # compares "what" against "what are" → no match!
```
`"what are people saying about DeepSeek R1"` → keeps `"what are people saying"` → **LOSES THE ENTIRE TOPIC**.
The multi-word entries (`"what are"`, `"how to"`, `"tips for"`, `"use cases"`) are dead code. They never match because `.split()` creates individual words but the noise list has multi-word strings.
### Bug 2: Missing meta/research words
The noise list has `"prompting"` but not `"prompt"`, `"prompts"`, `"techniques"`, `"tips"`, `"tricks"`, `"methods"`, etc.
- `"vibe motion best prompt techniques"` → `"vibe motion prompt techniques"` (4 words, should be 2)
- `"nano banana pro prompts for gemini"` → `"nano banana pro prompts"` (4 words, should be 3)
### Bug 3: No retry on 0 results
Reddit has multi-stage retry: full query → simplified core → subreddit fallback. X search runs once and accepts whatever comes back, even 0 results.
## Proposed Fix
All changes in `scripts/lib/bird_x.py`.
### Step 1: Fix `_extract_core_subject()` — strip phrases first, then words
```python
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search."""
text = topic.lower()
# Phase 1: Strip multi-word prefixes/suffixes (order matters - longest first)
prefixes = ['what are the best', 'what is the best', 'what are', 'what is',
'how to', 'how do i', 'tips for', 'best practices for']
for p in prefixes:
if text.startswith(p):
text = text[len(p):].strip()
break
suffixes = ['best practices', 'use cases', 'prompt techniques',
'prompting techniques']
for s in suffixes:
if text.endswith(s):
text = text[:-len(s)].strip()
break
# Phase 2: Split and filter individual noise words
noise = {'best', 'top', 'practices', 'features', 'killer', 'guide',
'tutorial', 'recommendations', 'advice', 'prompting', 'prompt',
'prompts', 'techniques', 'tips', 'tricks', 'methods',
'strategies', 'review', 'reviews', 'uses', 'usecases',
'examples', 'using', 'for', 'with', 'the', 'of', 'in', 'on',
'about', 'latest', 'new', 'news', 'update', 'updates',
'good', 'great', 'awesome', 'and', 'or', 'a', 'an', 'is',
'are', 'was', 'were', 'people', 'saying', 'think', 'said'}
words = text.split()
result = [w for w in words if w not in noise]
return ' '.join(result[:3]) or topic # Max 3 words (was 4)
```
**Expected results after fix:**
| Input | Before | After |
|-------|--------|-------|
| `vibe motion best prompt techniques` | `vibe motion prompt techniques` | `vibe motion` |
| `what are people saying about DeepSeek R1` | `what are people saying` | `deepseek r1` |
| `nano banana pro prompts for gemini` | `nano banana pro prompts` | `nano banana pro` |
| `open claw best uses` | `open claw uses` | `open claw` |
| `best claude code skills` | `claude code skills` | `claude code skills` |
| `kanye west` | `kanye west` | `kanye west` |
### Step 2: Add retry with simplified query on 0 results
In `search_x()`, after the initial search, if 0 items returned, retry with just the first 2 words of the core subject:
```python
def search_x(topic, from_date, to_date, depth="default"):
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
# ... existing Bird search code ...
items = parse_bird_response(response)
# Retry with fewer keywords if 0 results
if not items and len(core_topic.split()) > 2:
shorter = ' '.join(core_topic.split()[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
# ... retry Bird search ...
items = parse_bird_response(retry_response)
return response # or merged response
```
### Step 3 (optional): Cross-pollinate Reddit entities into X Phase 2
When X Phase 1 returns 0 results but Reddit found threads, extract brand/product names from Reddit thread titles and use them as X search fallback queries. This is lower priority — Steps 1-2 should fix most cases.
## Acceptance Criteria
- [x] `vibe motion best prompt techniques` returns >0 X posts (12 posts found)
- [x] `what are people saying about DeepSeek R1` produces query containing "deepseek r1" not "what are people saying"
- [x] No regressions on working queries (`kanye west`, `claude code skills`, `open claw`)
- [x] Retry fires when initial query returns 0, logged to stderr
- [x] `openai_reddit.py`'s `_extract_core_subject()` NOT changed (Reddit uses semantic search, not literal matching — the current function works fine there)
## Files to Change
- `scripts/lib/bird_x.py` — `_extract_core_subject()` rewrite + retry logic in `search_x()`
- `scripts/lib/bird_x.py` — `search_handles()` benefits automatically (calls `_extract_core_subject()`)
## Testing
```bash
# Mock mode (quick syntax check)
python3 scripts/last30days.py "vibe motion best prompt techniques" --mock --emit=compact 2>&1
# Live queries to verify X results
python3 scripts/last30days.py "vibe motion best prompt techniques" --quick --emit=compact 2>&1 | grep -E "X:|posts"
python3 scripts/last30days.py "what are people saying about DeepSeek R1" --quick --emit=compact 2>&1 | grep -E "X:|posts"
# Regression check
python3 scripts/last30days.py "kanye west" --quick --emit=compact 2>&1 | grep -E "X:|posts"
```
docs/plans/2026-02-07-release-v2-public-launch-plan.md
-67
View File
@@ -1,67 +0,0 @@
---
title: "release: Push V2 to public repo and launch"
type: release
date: 2026-02-07
---
# release: Push V2 to public repo and launch
## Overview
Push all V2 changes from the private development repo to the public GitHub repo, then sync the installed skill. 52 commits need to go from `origin/main` (private) to `upstream/main` (public).
## Current State
- **Private repo:** `https://github.com/mvanhorn/last30days-skill-private` - 52 commits ahead of public
- **Public repo:** `https://github.com/mvanhorn/last30days-skill` - last commit is V1 (`cc892d7`)
- **Upstream remote:** Already configured in private repo
- **Untracked files in private:** test logs, draft plans - these should NOT be pushed
## Steps
### Step 1: Clean up private repo
- [ ] Review untracked files - make sure nothing sensitive gets pushed
- [ ] Decide: commit the untracked docs/plans or leave them out of the push
### Step 2: Push to public
```bash
cd /Users/mvanhorn/last30days-skill-private
git push upstream main
```
This pushes all 52 commits from private `main` to public `main`. Since the private repo was forked from public, history is shared - this is a fast-forward push.
### Step 3: Sync installed skill
```bash
# Update the installed copy that Claude Code actually uses
cd ~/.claude/skills/last30days
git pull origin main
```
### Step 4: Verify
- [ ] Check `https://github.com/mvanhorn/last30days-skill` shows V2 README with new examples
- [ ] Check SKILL.md has the new argument-hint and timing disclaimer
- [ ] Check bird_x.py has the fixed `_extract_core_subject()` and retry logic
- [ ] Run a quick `/last30days` test to confirm installed skill works
## What Gets Published
Key files going public:
- `README.md` - V2 features, 3 new examples (Nano Banana Pro, Kanye, Vibe Motion), speed tradeoff note
- `SKILL.md` - New argument-hint, timing disclaimer, citation rules
- `scripts/lib/bird_x.py` - Fixed query construction + retry logic
- `scripts/lib/http.py` - USER_AGENT bumped to 2.0
- `.claude-plugin/plugin.json` - Marketplace support
- All Bird CLI integration code
- Phase 2 supplemental search code
- Model fallback chain
## Risks
- **Low risk:** This is a fast-forward push, no force push needed
- **Public API keys:** Already confirmed - no `.env` files or secrets in the repo
- **Untracked files:** Won't be pushed unless committed first
docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md
-243
View File
@@ -1,243 +0,0 @@
---
title: "feat: Add Codex CLI compatibility"
type: feat
date: 2026-02-14
---
# feat: Add Codex CLI Compatibility
## Overview
Make /last30days work as a Codex CLI skill alongside Claude Code. Both platforms use `SKILL.md` with YAML frontmatter — the gap is small but the details matter. Inspired by PR #24 (el-analista) and PR #5 (jblwilliams) on the public repo, applied to the v2.1 codebase.
## Research Findings
### How Codex Skills Work (from [official docs](https://developers.openai.com/codex/skills))
**Format:** Identical to Claude Code — `SKILL.md` with YAML frontmatter + Markdown body.
**Required frontmatter:** Only `name` and `description`. The official skill-creator guidance says "Do not include any other fields in YAML frontmatter." This is stricter than Claude Code which allows `version`, `allowed-tools`, `argument-hint`, etc.
**Discovery:** Codex uses "progressive disclosure" — it reads ONLY the `description` field to decide whether to invoke a skill. The body loads only after triggering. This means the description must be comprehensive about when to use/not use the skill.
**Invocation:** Users invoke with `$skill-name` or `/skills` menu. Codex can also implicitly match based on the description (configurable via `agents/openai.yaml`).
**Installation paths** (scanned in order):
| Scope | Path |
|-------|------|
| Folder | `$CWD/.agents/skills/` |
| Repo | `$REPO_ROOT/.agents/skills/` |
| User | `$HOME/.agents/skills/` |
| Admin | `/etc/codex/skills/` |
| System | Bundled |
Note: Some docs also mention `~/.codex/skills/` as an alias for `$HOME/.agents/skills/`. Both should be checked.
**`agents/openai.yaml`** (optional sidecar):
```yaml
interface:
display_name: "User-facing name"
short_description: "Brief description"
default_prompt: "Surrounding prompt template"
brand_color: "#hex"
policy:
allow_implicit_invocation: true
dependencies:
tools:
- type: "mcp"
value: "toolName"
```
**Size guidance:** Keep SKILL.md under 500 lines. Use `references/` directory for detailed docs that load on demand.
**Scripts:** Put executable code in `scripts/`. These can run without being loaded into context — good for our Python research engine.
### What Real Codex Skills Look Like (from [openai/skills catalog](https://github.com/openai/skills))
**openai-docs skill** — Uses MCP tools (`mcp__openaiDeveloperDocs__search_openai_docs`). Has a workflow section, fallback instructions if MCP isn't set up, and quality rules. Clean and focused.
**pdf skill** — Runs scripts (`pdftoppm`, `reportlab`), has file conventions (`tmp/pdfs/`, `output/pdf/`), specifies dependencies. Good example of a skill that shells out to tools like we do.
**skill-creator** — The meta-skill. Emphasizes "the context window is a public good" and treating the LLM as "already very smart — only add information it genuinely lacks." Has 6 creation steps, validation scripts, and naming conventions.
### Key Insight: Frontmatter Compatibility Problem
Claude Code SKILL.md uses:
```yaml
name: last30days
version: "2.1"
description: Research a topic...
argument-hint: 'nano banana pro prompts...'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
```
Codex wants only `name` and `description`. The question: does Codex error on unknown frontmatter fields, or ignore them?
**Safe answer:** Codex uses standard YAML parsing and likely ignores unknown keys. But the official guidance says "Do not include any other fields" — meaning it's untested territory and could break in future Codex updates.
**Our approach:** Keep one SKILL.md with Claude-specific fields. If Codex chokes, we add a thin wrapper. This is pragmatic — maintaining two SKILL.md files defeats the purpose of cross-platform compatibility.
### PR #24 Analysis (el-analista)
Good ideas to incorporate:
- Portable script path resolution (repo → Claude → Codex → agents)
- `agents/openai.yaml` for Codex discovery
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dir fallbacks with env var overrides
- Last-chance retry for Bird search (better query noise stripping)
Not applicable to v2.1:
- Based on v2.0 codebase — doesn't have YouTube, vendored Bird, or pipeline changes
- We'll cherry-pick the ideas, not the code
### PR #5 Analysis (jblwilliams)
Not needed:
- Codex JWT auth — our OpenAI API calls work natively in Codex already
- SSE response handling — we don't stream responses
- The 403 enrichment issues they hit are specific to Codex-hosted auth, not our use case
## Proposed Solution
Five changes, all additive — zero impact on existing Claude Code behavior:
### 1. Add `agents/openai.yaml` for Codex discovery
```yaml
interface:
display_name: "Last 30 Days"
short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
brand_color: "#FF6B35"
policy:
allow_implicit_invocation: true
```
### 2. Make SKILL.md script path portable
Replace the hardcoded Claude path with a lookup that checks multiple install locations:
```bash
# Find the skill root
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
if [ -z "${SKILL_ROOT:-}" ]; then
echo "ERROR: Could not find scripts/last30days.py" >&2
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
```
### 3. Platform-neutral Python output text
Replace "Claude" with "assistant" in LLM-facing output strings only. Human-facing docs (README, etc.) stay as-is.
Files:
- `scripts/last30days.py` — web search marker text (~3 lines)
- `scripts/lib/render.py` — docstrings + web-only banner (~4 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### 4. Sandbox-friendly cache/output dirs
Codex runs sandboxed. Add env var overrides + tempdir fallback (from PR #24):
**`scripts/lib/cache.py`:**
- Check `LAST30DAYS_CACHE_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/cache`
**`scripts/lib/render.py`:**
- Check `LAST30DAYS_OUTPUT_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/out`
### 5. README + installation docs
Add a "Codex Compatibility" section to README:
```markdown
## Codex Compatibility
This skill works in both Claude Code and OpenAI Codex CLI.
**Claude Code:** `git clone` into `~/.claude/skills/last30days`
**Codex CLI:** `git clone` into `~/.agents/skills/last30days`
Both use the same SKILL.md, same Python engine, same scripts.
The `agents/openai.yaml` provides Codex-specific discovery metadata.
```
## What We're NOT Doing
- **Separate SKILL.md for Codex** — One file, both platforms. Claude-specific frontmatter fields (`allowed-tools`, `version`, `argument-hint`) are likely ignored by Codex's YAML parser. If this breaks, we'll address it then.
- **Codex JWT auth (PR #5)** — Our OpenAI Responses API calls work natively in Codex. No special handling needed.
- **SSE streaming (PR #5)** — Not our use case.
- **Codex-specific tool names in SKILL.md** — Both LLMs understand "do a web search" and "run this bash command." The instructions work cross-platform as-is.
- **Publishing to openai/skills catalog** — Out of scope for now. Users install via git clone.
## Acceptance Criteria
- [x] `agents/openai.yaml` exists with proper `interface` and `policy` sections
- [x] SKILL.md uses portable path resolution (repo checkout, `~/.claude/skills/`, `~/.agents/skills/`, `~/.codex/skills/`)
- [x] Python scripts use "assistant" instead of "Claude" in LLM-facing output (~8 string replacements)
- [x] Cache dir falls back gracefully in sandboxed environments (`LAST30DAYS_CACHE_DIR` env var + `PermissionError` catch)
- [x] Output dir falls back gracefully in sandboxed environments (`LAST30DAYS_OUTPUT_DIR` env var + `PermissionError` catch)
- [x] Existing Claude Code behavior is unchanged (zero regressions)
- [x] README documents Codex installation path (`~/.agents/skills/last30days`)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
## Files to Create/Modify
### New Files
- `agents/openai.yaml` — Codex discovery metadata (~10 lines)
### Modified Files
- `SKILL.md` — Portable script path resolution (~15 lines changed)
- `README.md` — Add "Codex Compatibility" section (~15 lines)
- `scripts/last30days.py` — "Claude" → "assistant" in output strings (~3 lines)
- `scripts/lib/render.py` — "Claude" → "assistant" + output dir fallback (~15 lines)
- `scripts/lib/cache.py` — Cache dir env override + fallback (~12 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### Total scope: ~70 lines changed across 7 files. Small, additive, low risk.
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Codex rejects unknown YAML frontmatter (`allowed-tools`, etc.) | Low-Medium | Standard YAML parsers ignore unknown keys. If it breaks, strip Claude-specific fields and use `agents/openai.yaml` for metadata. |
| Codex sandbox blocks Node.js (vendored Bird) | Medium | Bird failure already falls back to xAI API. If no xAI key, X search skipped gracefully. |
| yt-dlp not in Codex sandbox PATH | Medium | YouTube already degrades gracefully — "yt-dlp not installed, skipping YouTube." |
| Codex sandbox blocks `~/.cache/` writes | Medium | Env var override + tempdir fallback handles this. (Proven approach from PR #24) |
| Codex changes skill discovery paths | Low | We check 5 paths. Easy to add more. |
| Codex description matching triggers on wrong queries | Low | Write description with clear "use when" / "do not use when" boundaries per official guidance. |
## References
### Community PRs
- [PR #24](https://github.com/mvanhorn/last30days-skill/pull/24) (el-analista) — Codex compatibility, portable paths, platform-neutral text
- [PR #5](https://github.com/mvanhorn/last30days-skill/pull/5) (jblwilliams) — Codex auth support
### Official Codex Docs
- [Agent Skills](https://developers.openai.com/codex/skills) — SKILL.md format, discovery, installation paths
- [AGENTS.md Guide](https://developers.openai.com/codex/guides/agents-md/) — Custom instructions, hierarchical loading
- [Codex CLI Features](https://developers.openai.com/codex/cli/features/) — Overview of CLI capabilities
- [Configuration Reference](https://developers.openai.com/codex/config-reference/) — config.toml, skill enable/disable
### Examples
- [openai/skills catalog](https://github.com/openai/skills) — Official curated skills
- [skill-creator](https://github.com/openai/skills/blob/main/skills/.system/skill-creator/SKILL.md) — Meta-skill for creating skills, best practices
- [pdf skill](https://github.com/openai/skills/blob/main/skills/.curated/pdf/SKILL.md) — Example of skill that runs external scripts
- [openai-docs skill](https://github.com/openai/skills/blob/main/skills/.curated/openai-docs/SKILL.md) — Example of MCP-backed skill
### Community Analysis
- [Skills in OpenAI Codex](https://blog.fsck.com/2025/12/19/codex-skills/) — Jesse Vincent's deep dive on skill internals
- [Simon Willison on skills adoption](https://simonw.substack.com/p/openai-are-quietly-adopting-skills) — Cross-platform skill format analysis
- [SkillsMP marketplace](https://skillsmp.com/) — Community marketplace supporting both Claude Code and Codex skills
docs/plans/2026-02-14-feat-merge-openclaw-variant-plan.md
-224
View File
@@ -1,224 +0,0 @@
---
title: "feat: Merge OpenClaw variant into main repo"
type: feat
date: 2026-02-14
---
# feat: Merge OpenClaw Variant into Main Repo
## Overview
Consolidate the `last30days-openclaw` project into `last30days-skill-private` so there's one unified Python engine powering both the main skill (Claude Code / Codex) and an "open" variant with watchlist, briefing, history, and built-in web search. The open variant also gets YouTube and Bird CLI — features the main project already has but openclaw was built before they existed.
## Problem Statement / Motivation
Right now there are two separate repos with diverging codebases:
- **`last30days-skill-private`** (main, Feb 14) — YouTube, vendored Bird, better scoring/normalization, Codex compat. But no built-in web search APIs and no persistence layer.
- **`last30days-openclaw`** (Feb 10) — SQLite store, watchlist, briefings, 3 web search backends (Parallel AI, Brave, OpenRouter). But frozen without YouTube or latest engine improvements.
They share ~80% of the same `scripts/lib/` files but are drifting apart. Maintaining two codebases is unsustainable.
**Goal:** One repo, one Python engine, two SKILL.md variants. Install once, works everywhere.
## Proposed Solution
Use `last30days-skill-private` as the base (it's 4 days newer with better code) and port the OpenClaw-exclusive features in:
### What gets ported from OpenClaw
| File | What it does | Destination |
|------|-------------|-------------|
| `scripts/store.py` | SQLite research accumulator (WAL, FTS5, dedup) | `scripts/store.py` |
| `scripts/watchlist.py` | Topic watchlist CLI (add/remove/list/run) | `scripts/watchlist.py` |
| `scripts/briefing.py` | Morning briefing generator (daily/weekly) | `scripts/briefing.py` |
| `scripts/lib/brave_search.py` | Brave Search API (free tier, 2K/mo) | `scripts/lib/brave_search.py` |
| `scripts/lib/parallel_search.py` | Parallel AI search (LLM-optimized) | `scripts/lib/parallel_search.py` |
| `scripts/lib/openrouter_search.py` | OpenRouter/Sonar Pro search | `scripts/lib/openrouter_search.py` |
| `references/research.md` | One-shot research instructions | `variants/open/references/research.md` |
| `references/watchlist.md` | Watchlist mode instructions | `variants/open/references/watchlist.md` |
| `references/briefing.md` | Briefing mode instructions | `variants/open/references/briefing.md` |
| `references/history.md` | History query instructions | `variants/open/references/history.md` |
### What gets upgraded in the ported code
- **`store.py`**: No changes needed — it's self-contained SQLite, works as-is
- **`watchlist.py`**: Remove OpenClaw cron-specific code, make cron setup generic (launchd on macOS, systemd on Linux, or manual cron)
- **`briefing.py`**: No changes needed
- **`scripts/lib/env.py`**: Merge OpenClaw's web search key support (`PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`) and `has_web_search_keys()` / `get_web_search_source()` functions into main's env.py. Drop the OpenClaw config loader (`~/.openclaw/openclaw.json`) — just use env vars and `~/.config/last30days/.env`
- **`scripts/last30days.py`**: Add OpenClaw's `_search_web()` function so the script can do web search natively when API keys are available (instead of always delegating to the assistant)
### What gets DROPPED from OpenClaw
| File | Why |
|------|-----|
| `scripts/cron_setup.py` | Too OpenClaw-platform-specific. Replace with generic scheduling docs. |
| OpenClaw config loader in `env.py` | `~/.openclaw/openclaw.json` path is platform-specific. Use env vars instead. |
| `.clawhubignore` | OpenClaw marketplace artifact, not needed in unified repo |
### New file: Open variant SKILL.md
Create `variants/open/SKILL.md` — the multi-mode skill with command routing:
```
variants/open/
├── SKILL.md # Router: watch, briefing, history, or one-shot
├── references/
│ ├── research.md # One-shot research instructions
│ ├── watchlist.md # Watchlist management instructions
│ ├── briefing.md # Briefing mode instructions
│ └── history.md # History query instructions
└── context.md # Agent memory (user preferences, source quality)
```
The open variant's SKILL.md points to `{baseDir}/scripts/last30days.py` (same engine) but adds the router and reference file system. It also adds the `--store` flag for persistence.
### How YouTube and Bird CLI get added to the open variant
They're already in `scripts/lib/youtube_yt.py` and `scripts/lib/vendor/bird/`. The open variant's SKILL.md just needs to mention YouTube in its description and the research.md reference file gets the YouTube stats line in the output format. No code changes needed — the Python engine already supports all four sources.
## Technical Considerations
### File Structure After Merge
```
last30days-skill-private/
├── SKILL.md # Main skill (Claude Code / Codex)
├── agents/openai.yaml # Codex discovery (existing)
├── variants/
│ └── open/
│ ├── SKILL.md # Open variant with routing
│ ├── references/
│ │ ├── research.md
│ │ ├── watchlist.md
│ │ ├── briefing.md
│ │ └── history.md
│ └── context.md
├── scripts/
│ ├── last30days.py # Unified engine (+ native web search)
│ ├── store.py # SQLite accumulator (from openclaw)
│ ├── watchlist.py # Watchlist CLI (from openclaw, genericized)
│ ├── briefing.py # Briefing generator (from openclaw)
│ └── lib/
│ ├── ... (existing files)
│ ├── brave_search.py # NEW from openclaw
│ ├── parallel_search.py # NEW from openclaw
│ ├── openrouter_search.py # NEW from openclaw
│ ├── youtube_yt.py # Existing
│ └── vendor/bird/ # Existing
└── README.md # Updated with open variant docs
```
### Installation for open variant users
```bash
# Claude Code (main skill — unchanged)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Open variant (with watchlist, briefings, history)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Then in Claude Code settings, point skill to variants/open/SKILL.md
# OR symlink:
ln -sf ~/.claude/skills/last30days/variants/open/SKILL.md ~/.claude/skills/last30days-open/SKILL.md
```
### env.py merge strategy
Main's `env.py` is the base. Add from OpenClaw:
- Three new key names: `PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`
- `has_web_search_keys()` function
- `get_web_search_source()` function — returns `'parallel'`, `'brave'`, or `'openrouter'`
- `get_available_sources()` update to include web-search-capable modes
### last30days.py merge strategy
Main's `last30days.py` is the base. Add from OpenClaw:
- `_search_web()` function that calls the appropriate web search backend
- `--store` CLI flag to persist findings to SQLite
- `--diagnose` CLI flag for source availability diagnostics
- Web results integration into the existing report pipeline (normalize → score → dedupe → render)
Keep main's:
- YouTube integration
- Phase 2 supplemental search
- 3-tier Reddit fallback
- Better error handling
- Minimum result guarantee
### Portable path resolution (already done)
The main SKILL.md already has portable path resolution (from Codex compat work):
```bash
for dir in "." "${CLAUDE_PLUGIN_ROOT:-}" "$HOME/.claude/skills/last30days" ...
```
The open variant's SKILL.md uses `{baseDir}` which resolves to the skill root. Both approaches work — we just need to make sure the open variant's references use `{baseDir}` consistently.
## Acceptance Criteria
- [x] `scripts/store.py` ported and working (SQLite creates on first use)
- [x] `scripts/watchlist.py` ported with generic scheduling (no OpenClaw cron dependency)
- [x] `scripts/briefing.py` ported and generates daily/weekly briefings
- [x] `scripts/lib/brave_search.py` ported and functional
- [x] `scripts/lib/parallel_search.py` ported and functional
- [x] `scripts/lib/openrouter_search.py` ported and functional
- [x] `scripts/lib/env.py` updated with web search key support
- [x] `scripts/last30days.py` has native `_search_web()` + `--store` + `--diagnose`
- [x] `variants/open/SKILL.md` exists with command routing (watch, briefing, history, research)
- [x] `variants/open/references/*.md` — all 4 reference files ported
- [x] Open variant mentions YouTube in description and research output format
- [x] Open variant uses same portable path resolution as main
- [x] Main SKILL.md behavior is unchanged (zero regressions)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
- [x] `python3 scripts/last30days.py "test topic" --diagnose` shows source availability
- [x] README documents open variant installation and usage
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| OpenClaw's store.py has import dependencies we don't have | Low | store.py uses only stdlib (sqlite3, json, datetime). Self-contained. |
| Web search backends need API keys to test | Medium | Each has a `--mock` or dry-run path. Test with real keys if available, mock otherwise. |
| watchlist.py depends on OpenClaw cron API | High | Known — strip cron_setup.py dependency, replace with generic docs for launchd/systemd/crontab. |
| Open variant SKILL.md is too long (>500 lines) | Medium | Use reference file pattern (already planned). Router SKILL.md stays under 100 lines. |
| env.py merge introduces regressions | Low | Main's env.py is well-tested. Additive changes only — new keys, new functions. |
| Two SKILL.md files = maintenance burden | Low | They serve different purposes. Main is simple one-shot. Open adds routing. Core engine is shared. |
## Files to Create/Modify
### New Files
- `variants/open/SKILL.md` — Open variant router (~100 lines)
- `variants/open/references/research.md` — One-shot research instructions (from openclaw, updated with YouTube)
- `variants/open/references/watchlist.md` — Watchlist management instructions (from openclaw)
- `variants/open/references/briefing.md` — Briefing mode instructions (from openclaw)
- `variants/open/references/history.md` — History query instructions (from openclaw)
- `variants/open/context.md` — Agent memory template
- `scripts/store.py` — SQLite accumulator (from openclaw, as-is)
- `scripts/watchlist.py` — Watchlist CLI (from openclaw, genericized)
- `scripts/briefing.py` — Briefing generator (from openclaw, as-is)
- `scripts/lib/brave_search.py` — Brave Search API (from openclaw)
- `scripts/lib/parallel_search.py` — Parallel AI search (from openclaw)
- `scripts/lib/openrouter_search.py` — OpenRouter/Sonar Pro search (from openclaw)
### Modified Files
- `scripts/lib/env.py` — Add web search key support (~30 lines added)
- `scripts/last30days.py` — Add `_search_web()`, `--store`, `--diagnose` (~80 lines added)
- `README.md` — Add open variant section (~20 lines)
### Total scope: ~12 new files (mostly copied), ~130 lines of new code in existing files.
## References
### Internal
- OpenClaw plan: `/Users/mvanhorn/last30days-openclaw/docs/plans/2026-02-10-feat-openclaw-last30days-skill-plan.md` (989 lines, comprehensive spec)
- Codex compat plan: `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md` (portable paths, platform-neutral text)
- OpenClaw source: `/Users/mvanhorn/last30days-openclaw/`
### Key files to port
- `store.py`: `/Users/mvanhorn/last30days-openclaw/scripts/store.py` (20KB, SQLite with FTS5)
- `watchlist.py`: `/Users/mvanhorn/last30days-openclaw/scripts/watchlist.py` (10KB)
- `briefing.py`: `/Users/mvanhorn/last30days-openclaw/scripts/briefing.py` (8KB)
- `brave_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/brave_search.py` (6KB)
- `parallel_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/parallel_search.py` (4KB)
- `openrouter_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/openrouter_search.py` (7KB)
- `env.py` (openclaw version): `/Users/mvanhorn/last30days-openclaw/scripts/lib/env.py` (9KB — has web search key functions)
docs/plans/2026-02-14-feat-youtube-transcript-search-plan.md
-315
View File
@@ -1,315 +0,0 @@
---
title: "feat: Add YouTube transcript search as 4th source"
type: feat
date: 2026-02-14
---
# feat: Add YouTube Transcript Search
## Overview
Add YouTube as a 4th research source alongside Reddit, X, and Web. Search for recent videos on the user's topic, fetch transcripts from the top results, and feed the transcript text into the synthesis — giving the Judge Agent access to what people are *saying* in video form, not just what they're posting on social media.
**Why this matters:** For many topics (tutorials, product reviews, drama breakdowns), the best content lives on YouTube, not Reddit or X. A 20-minute video review contains 10x the signal of a tweet. The skill currently misses all of it.
## Proposed Solution
Use **yt-dlp** (already installed via Homebrew) for both YouTube search and transcript extraction. No new API keys, no new dependencies. Follows the same "zero friction" philosophy as vendored Bird search.
### Two-step process per research run:
1. **Search**: `yt-dlp "ytsearch{N}:{topic}" --dateafter {30d_ago} --flat-playlist --print` → top videos by view count
2. **Transcripts**: For top 5 videos, extract auto-generated subtitles via `yt-dlp --write-auto-subs --skip-download`, clean VTT to plaintext in Python
### Why NOT use `summarize` CLI:
- Adds 146MB brew dependency (arm64-only binary)
- Calls OpenAI API per video ($0.01-0.03 each) — adds cost on top of existing API usage
- yt-dlp already extracts raw transcripts for free (covers ~95% of videos with auto-captions)
- Raw transcripts are better for synthesis anyway — the LLM doing synthesis (Claude) should interpret the content itself, not get a pre-summarized version
`summarize` is a great standalone tool, but for integration into a research pipeline where an LLM already synthesizes everything, raw transcripts are the right input.
## Technical Approach
### Architecture
New file: `scripts/lib/youtube_yt.py` (mirrors `bird_x.py` pattern)
```
yt-dlp search → metadata (title, views, channel, date)
↓
sort by views, take top N
↓
yt-dlp subtitle extraction → raw VTT files
↓
VTT cleanup → plaintext transcripts
↓
truncate to ~500 words per video
↓
normalize → YouTubeItem objects
↓
score, dedupe, render (same pipeline as Reddit/X)
```
### Implementation Phases
#### Phase 1: Search + Metadata (the fast part)
**New file: `scripts/lib/youtube_yt.py`**
Core search function:
```python
def search_youtube(topic: str, from_date: str, to_date: str, depth: str = "default") -> Dict[str, Any]:
"""Search YouTube via yt-dlp. No API key needed.
Returns:
Dict with 'items' list of video metadata dicts.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
date_filter = from_date.replace("-", "") # YYYYMMDD format
# yt-dlp search with metadata extraction
cmd = [
"yt-dlp",
f"ytsearch{count}:{topic}",
"--dateafter", date_filter,
"--flat-playlist",
"--print", "%(view_count)s\t%(id)s\t%(title)s\t%(channel)s\t%(upload_date)s\t%(like_count)s\t%(comment_count)s",
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
# Parse tab-separated output, sort by views, return top N
...
```
Depth config (matches existing pattern):
```python
DEPTH_CONFIG = {
"quick": 10, # search 10, transcript top 3
"default": 20, # search 20, transcript top 5
"deep": 40, # search 40, transcript top 8
}
TRANSCRIPT_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
```
**Key detail**: `yt-dlp --flat-playlist` returns exit code 0 with empty stdout when `--dateafter` filters out everything. Check for empty output, not error codes.
#### Phase 2: Transcript Extraction (the slow part)
For top N videos (by view count), fetch transcripts:
```python
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Returns:
Plaintext transcript string, or None if no captions available.
"""
cmd = [
"yt-dlp",
"--write-auto-subs",
"--sub-lang", "en",
"--sub-format", "vtt",
"--skip-download",
"-o", f"{temp_dir}/%(id)s",
f"https://www.youtube.com/watch?v={video_id}",
]
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
vtt_path = Path(temp_dir) / f"{video_id}.en.vtt"
if not vtt_path.exists():
return None
return _clean_vtt(vtt_path.read_text())
```
VTT cleanup (~10 lines of Python):
```python
def _clean_vtt(vtt_text: str) -> str:
"""Convert VTT subtitle format to clean plaintext."""
text = re.sub(r'^WEBVTT.*?\n\n', '', vtt_text, flags=re.DOTALL)
text = re.sub(r'\d{2}:\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}:\d{2}\.\d{3}.*\n', '', text)
text = re.sub(r'<[^>]+>', '', text)
lines = text.strip().split('\n')
seen = set()
unique = []
for line in lines:
stripped = line.strip()
if stripped and stripped not in seen:
seen.add(stripped)
unique.append(stripped)
return re.sub(r'\s+', ' ', ' '.join(unique)).strip()
```
**Parallelization**: Run transcript fetches in parallel using ThreadPoolExecutor (same pattern as Phase 2 supplemental searches for Reddit/X):
```python
def fetch_transcripts_parallel(video_ids: List[str], max_workers: int = 5) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel."""
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
for vid in video_ids
}
results = {}
for future in as_completed(futures):
vid = futures[future]
results[vid] = future.result()
return results
```
#### Phase 3: Integration into Pipeline
**Update `scripts/lib/schema.py`** — add YouTubeItem:
```python
@dataclass
class YouTubeItem:
id: str # video_id
title: str
url: str
channel_name: str
date: Optional[str]
date_confidence: str # always "high" for YouTube
engagement: Engagement # views, likes, comments
transcript_snippet: str # first ~500 words of transcript
relevance: float
why_relevant: str
subs: Optional[SubScores] = None
score: int = 0
```
Update `Report` to add:
```python
youtube: List[YouTubeItem] = field(default_factory=list)
youtube_error: Optional[str] = None
```
**Update `scripts/lib/score.py`** — YouTube-specific engagement weights:
```python
def compute_youtube_engagement_raw(views, likes, comments):
"""YouTube engagement: views dominate, likes secondary, comments tertiary."""
return (
0.50 * math.log1p(views or 0) +
0.35 * math.log1p(likes or 0) +
0.15 * math.log1p(comments or 0)
)
```
**Update `scripts/last30days.py`** — add YouTube to ThreadPoolExecutor:
```python
with ThreadPoolExecutor(max_workers=3) as executor: # was 2
if run_reddit:
reddit_future = executor.submit(_search_reddit, ...)
if run_x:
x_future = executor.submit(_search_x, ...)
if run_youtube:
youtube_future = executor.submit(_search_youtube, ...)
```
**Update `scripts/lib/render.py`** — YouTube section in compact output:
```
### YouTube Videos
**{id}** (score:{score}) {channel_name} ({date}) [{views} views, {likes} likes]
{title}
https://www.youtube.com/watch?v={id}
{transcript_snippet[:200]}...
*{why_relevant}*
```
**Update `scripts/lib/env.py`** — YouTube availability detection:
```python
def is_ytdlp_available() -> bool:
return shutil.which("yt-dlp") is not None
```
No API key needed. YouTube search is available whenever yt-dlp is in PATH.
#### Phase 4: SKILL.md Updates
Stats box adds YouTube line:
```
├─ 🎥 YouTube: {N} videos │ {N} views │ {N} transcripts
```
Citation priority updated:
```
1. @handles from X
2. YouTube creators — "per [Channel Name] on YouTube"
3. r/subreddits from Reddit
4. Web sources
```
Synthesis instructions updated to weight YouTube transcripts highly — a 20-minute video transcript with 500K views is a stronger signal than a tweet with 50 likes.
## Acceptance Criteria
- [x] `yt-dlp` search returns videos matching topic within date range
- [x] Transcripts extracted for top N videos (auto-generated captions)
- [x] Videos without captions gracefully skipped (no error)
- [x] YouTube results appear in compact output with engagement metrics
- [x] YouTube items scored and ranked alongside Reddit/X items
- [x] YouTube auto-activates when yt-dlp is available (no --sources flag needed)
- [x] SKILL.md stats box includes YouTube line
- [x] Transcript snippets (first ~500 words) included in output for LLM synthesis
- [ ] Total YouTube search + transcript extraction completes within 30 seconds
- [x] Works when yt-dlp is not installed (graceful degradation, no crash)
- [ ] Mock mode works for testing without network
## Dependencies & Risks
**Dependencies:**
- `yt-dlp` (Homebrew) — already installed, widely available via brew/pip/standalone
- No API keys needed
- No new Python packages (just subprocess + regex)
**Risks:**
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| yt-dlp search is slow (>10s) | Medium | Set 30s timeout, run in parallel with Reddit/X |
| YouTube blocks yt-dlp | Low | yt-dlp is actively maintained with anti-bot updates. Degrade gracefully. |
| Videos lack auto-captions | Medium (~5%) | Skip those videos, note in output. Transcript is enrichment, not required. |
| Transcript extraction adds latency | High | Only fetch top 3-5, run in parallel, use tempdir |
| yt-dlp not installed for some users | Medium | Auto-detect, skip YouTube with info message, don't error |
| Linux `--dateafter` date format differs | Low | Use Python to format date, not shell `date -v` |
## Files to Create/Modify
### New Files
- `scripts/lib/youtube_yt.py` — search, transcript extraction, parsing
- `tests/test_youtube_yt.py` — unit tests
- `fixtures/youtube_sample.json` — mock data for tests
### Modified Files
- `scripts/lib/schema.py` — add YouTubeItem, update Report
- `scripts/lib/normalize.py` — add normalize_youtube_items()
- `scripts/lib/score.py` — add YouTube engagement scoring
- `scripts/lib/dedupe.py` — add YouTube dedup (title + channel Jaccard)
- `scripts/lib/render.py` — add YouTube section to compact + full report
- `scripts/lib/env.py` — add yt-dlp availability check, update source detection
- `scripts/last30days.py` — add _search_youtube(), update run_research(), update arg parser
- `SKILL.md` — update stats box, citation rules, synthesis instructions
- `README.md` — document YouTube source, yt-dlp requirement
## Alternative Approaches Considered
**1. YouTube Data API v3** — Rejected. Requires API key + Google Cloud project. Adds friction, counter to "zero config" philosophy. 10K quota/day limit. yt-dlp has no limits.
**2. steipete/summarize for transcripts** — Rejected for MVP. Adds 146MB dependency, requires brew tap, calls OpenAI API per video (adds cost). Raw transcripts via yt-dlp are better input for our synthesis LLM anyway. Could revisit as optional enhancement for captionless videos.
**3. youtube-transcript-api Python package** — Considered. Lightweight, Python-native transcript fetcher. But adds a pip dependency to a project that currently has zero Python deps. yt-dlp is already a brew dependency we can auto-detect.
**4. Skip transcripts, just use metadata** — Rejected. Titles + view counts alone don't give the synthesis LLM enough to work with. Transcripts are what make YouTube a *research* source vs just a link list.
## Cost Impact
**Zero additional API cost.** yt-dlp scrapes YouTube directly. No API keys, no token usage. The only cost is the existing OpenAI/xAI calls for Reddit/X search, which are unchanged.
**Time impact:** Adds ~10-20 seconds to research (search + parallel transcript extraction), running in parallel with Reddit/X so effective wall-clock increase is minimal.
docs/plans/2026-02-15-feat-clawhub-scanner-compliance-plan.md
-194
View File
@@ -1,194 +0,0 @@
---
title: ClawHub Scanner Compliance for last30days-official
type: feat
date: 2026-02-15
---
# ClawHub Scanner Compliance for last30days-official
## Overview
Make the last30days skill pass ClawHub's security scanner (VirusTotal + Code Insight) so it can be published as `last30days-official`. The user's 8 other mvanhorn skills already pass - we replicate their exact pattern.
## Problem Statement
ClawHub requires skills to pass a multi-layer security scan before publication:
1. Metadata validation (frontmatter fields)
2. VirusTotal automated scanning
3. LLM-powered Code Insight analysis (checks if capabilities match documentation)
4. Credential handling review
The current last30days SKILL.md has basic `metadata.clawdbot` but is missing fields the scanner checks: `emoji`, `user-invocable`, `disable-model-invocation`, `files` declaration. There's no `## Security & Permissions` section (required pattern from passing skills). README has no security/privacy documentation.
## Proposed Solution
Follow the exact pattern from `clawdbot-skill-xai` and `clawdbot-skill-search-x` (both pass the scanner). Three files need changes.
### Fix 1: SKILL.md Frontmatter
Add missing scanner fields to the existing frontmatter:
```yaml
---
name: last30days
version: "2.1"
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web."
argument-hint: 'last30 AI video tools, last30 best project management tools'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill
user-invocable: true
disable-model-invocation: true
metadata:
clawdbot:
emoji: "📰"
requires:
env:
- OPENAI_API_KEY
bins:
- node
- python3
primaryEnv: OPENAI_API_KEY
files:
- "scripts/*"
homepage: https://github.com/mvanhorn/last30days-skill
tags:
- research
- reddit
- x
- youtube
- trends
- prompts
---
```
Key additions:
- `user-invocable: true` - human must trigger it
- `disable-model-invocation: true` - agent cannot self-trigger
- `emoji: "📰"` - required display field
- `files: ["scripts/*"]` - prevents false "instruction-only but has scripts" flag
### Fix 2: Security & Permissions Section in SKILL.md
Add to the bottom of SKILL.md (matches xai/search-x pattern exactly):
```markdown
## Security & Permissions
**What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
**What this skill does NOT do:**
- Does not post, like, or modify content on any platform
- Does not access your Reddit, X, or YouTube accounts
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
Review scripts before first use to verify behavior.
```
### Fix 3: Security & Privacy Section in README.md
Add after the "How It Works" section:
```markdown
## Security & Privacy
### Data that leaves your machine
| Destination | Data Sent | API Key Required |
|------------|-----------|-----------------|
| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
Your research topic is included in all outbound API requests. If you research sensitive topics, be aware that query strings are transmitted to the API providers listed above.
### Data stored locally
- API keys: `~/.config/last30days/.env` (chmod 600 recommended)
- Watchlist database: `~/.local/share/last30days/research.db` (SQLite)
- Briefings: `~/.local/share/last30days/briefs/`
### API key isolation
Each API key is transmitted only to its respective endpoint. Your OpenAI key is never sent to xAI, Brave, or any other provider. Browser cookies for X are read locally and used only for Twitter GraphQL requests.
```
### Fix 4: Update .claude-plugin Files
**plugin.json** - bump version, add youtube keyword:
```json
{
"name": "last30days",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0",
"author": {"name": "mvanhorn"},
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
"skills": ["./"]
}
```
**marketplace.json** - add version, update description:
```json
{
"name": "last30days",
"owner": {"name": "mvanhorn", "url": "https://github.com/mvanhorn"},
"metadata": {
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0"
},
"plugins": [{"name": "last30days", "source": "."}]
}
```
## What We DON'T Need
Based on the audit of your 8 passing skills:
- **Script-level security manifest headers** - your passing skills (xai, search-x, parallel) do NOT have these. The scanner relies on SKILL.md, not per-file headers.
- **Separate SECURITY.md file** - not needed; README section + SKILL.md section is sufficient.
- **Shell injection fixes** - already clean. All subprocess calls use list-form args, no `shell=True` anywhere.
- **Credential leak fixes** - already clean. All keys loaded from env vars, none hardcoded or logged.
## Acceptance Criteria
- [x] SKILL.md frontmatter has `user-invocable`, `disable-model-invocation`, `emoji`, `files`
- [x] SKILL.md has `## Security & Permissions` section with "does" and "does NOT do" lists
- [x] README.md has `## Security & Privacy` section with endpoint table and key isolation docs
- [x] plugin.json version bumped to 2.1.0, youtube keyword added
- [x] marketplace.json version and description updated
- [x] `python3 scripts/last30days.py --diagnose` still works after changes
- [x] Synced to all installed skill locations
- [ ] Published to ClawHub as `last30days-official` (when auth is fixed)
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Add frontmatter fields + Security & Permissions section |
| `README.md` | Add Security & Privacy section |
| `.claude-plugin/plugin.json` | Bump version, add youtube keyword |
| `.claude-plugin/marketplace.json` | Add version, update description |
## References
- [13-point ClawHub checklist](https://gist.github.com/adhishthite/0db995ecfe2f23e09d0b2d418491982c)
- [ClawHub docs](https://docs.openclaw.ai/tools/clawhub)
- [ClawHub Developer Guide 2026](https://www.digitalapplied.com/blog/clawhub-skills-marketplace-developer-guide-2026)
- Your passing skills: `clawdbot-skill-xai`, `clawdbot-skill-search-x` (exact pattern replicated)
docs/plans/2026-02-15-fix-watchlist-engagement-null-crash-plan.md
-76
View File
@@ -1,76 +0,0 @@
---
title: "fix: watchlist engagement null crash"
type: fix
date: 2026-02-15
---
# fix: Watchlist crashes on `engagement: null` from X posts
## Problem
When X posts return `engagement: null` (JSON null) instead of `engagement: {}` (empty object), the watchlist `_run_topic` findings parser crashes. This is because Python's `dict.get("engagement", {})` returns `None` when the key **exists** with value `None` — the default `{}` only applies when the key is **missing**.
```python
# CRASHES — .get() returns None, not {}
item.get("engagement", {}).get("likes", 0)
# AttributeError: 'NoneType' object has no attribute 'get'
```
**Reporter:** Soft launch tester, 2026-02-15
**Severity:** Medium — breaks watchlist `run-one` and `run-all` for any topic that pulls X posts with null engagement
**One-shot research unaffected** — the main `last30days.py` pipeline uses `normalize.py` which has `isinstance(eng_raw, dict)` guards
## Root Cause
Two locations use the vulnerable `dict.get("key", {})` pattern:
1. **`scripts/watchlist.py:188`** — THE REPORTED BUG
```python
"engagement_score": item.get("engagement", {}).get("likes", 0),
```
2. **`scripts/lib/normalize.py:177`** — YouTube normalizer (same pattern, latent)
```python
eng_raw = item.get("engagement", {})
```
Three other locations are already safe — they use `isinstance(eng_raw, dict)`:
- `scripts/lib/normalize.py:70` (Reddit)
- `scripts/lib/normalize.py:130` (X)
- `scripts/lib/xai_x.py:190`
## Fix
Apply the `or {}` idiom (as suggested by reporter):
### scripts/watchlist.py:188
```python
# Before
"engagement_score": item.get("engagement", {}).get("likes", 0),
# After
"engagement_score": (item.get("engagement") or {}).get("likes", 0),
```
### scripts/lib/normalize.py:177
```python
# Before
eng_raw = item.get("engagement", {})
# After
eng_raw = item.get("engagement") or {}
```
## Acceptance Criteria
- [ ] `watchlist.py run-one` handles X posts with `engagement: null` without crashing
- [ ] `watchlist.py run-one` handles X posts with `engagement: {}` (empty object)
- [ ] `watchlist.py run-one` handles X posts with no engagement key at all
- [ ] YouTube normalizer handles `engagement: null` without crashing
- [ ] Existing tests still pass
## Regarding the screenshot question
The tester asked "Did you use watchlist on open claw or Claude?" — watchlist is designed for the **open variant** (Open Claw). It works in Claude Code too since it's just Python + SQLite, but the SKILL.md routing for watchlist commands is only in `variants/open/SKILL.md`. The main `SKILL.md` is one-shot research only.
Answer to give tester: "Watchlist works in both — it's plain Python. But the skill routing that understands 'watch add topic' is in the open variant. In Claude Code you'd need to call the script directly or use the open variant SKILL.md."
docs/plans/2026-02-15-fix-youtube-display-and-search-quality-plan.md
-221
View File
@@ -1,221 +0,0 @@
---
title: Fix YouTube Display and Search Quality
type: fix
date: 2026-02-15
---
# Fix YouTube Display and Search Quality
## Overview
YouTube is the v2.1 headline feature but it's broken in two ways: results don't appear in Claude's synthesis (display bug), and search quality is worse than youtube.com (search bug). Both need fixing before launch.
## Problem Statement
**Display bug:** YouTube data exists in the script output but Claude never sees it. Reproduced on 4/5 recent test runs (Kanye, Seedance 2, Peter Steinberger, YouTube thumbnails). The skill worked once — the earlier "YouTube thumbnails" and "OpenClaw" runs showed YouTube stats — but subsequent runs silently dropped it.
**Search quality bug:** User searched "how to get on seedance 2" on youtube.com and got multiple recent results. yt-dlp returned 10 videos for the same query, but they included old irrelevant content because date filtering is broken.
## Root Cause Analysis
### Display Bug — Three compounding causes
1. **`2>&1` in SKILL.md bash command** (line 79) merges stderr progress messages into stdout. When Claude Code receives this mixed output, the YouTube section (which renders LAST after Reddit + X) can get lost in the noise or hit the 30K char Bash output limit.
2. **Background execution** (partially fixed). The old SKILL.md said "DO WEBSEARCH WHILE SCRIPT RUNS" which caused Claude to background the bash command. Backgrounded commands return truncated output via Task Output. *Already fixed in this session — SKILL.md now says FOREGROUND with 5-minute timeout.*
3. **No explicit model instruction to look for YouTube.** The SKILL.md tells Claude to synthesize but doesn't emphasize that YouTube data is in the script output and must be included.
### Search Quality Bug — `--flat-playlist` breaks date filtering
The yt-dlp command in `youtube_yt.py:110-116`:
```bash
yt-dlp ytsearch{count}:{query} --dateafter {YYYYMMDD} --flat-playlist --dump-json
```
**`--flat-playlist` causes three problems:**
1. `--dateafter` is silently ignored (no video-level metadata to filter on)
2. All items have `date: None` (upload_date not in flat-playlist JSON)
3. Old content leaks in (e.g., "the greatest youtube thumbnails of all time" returned for a 30-day query)
**`_extract_core_subject()` over-strips useful YouTube terms:**
- Strips "tips", "tutorial", "review" — but these ARE the content types people search for on YouTube
- "youtube thumbnail tips" → "youtube thumbnail" loses the intent signal
## Proposed Solution
### Phase 1: Fix Display (Critical — blocks launch)
#### 1a. Remove `2>&1` from SKILL.md bash command
**File:** `SKILL.md:79` (both `last30days-skill-private/SKILL.md` and `~/.claude/skills/last30days21/SKILL.md`)
```bash
# Before:
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
# After:
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
```
This removes ~1-5KB of progress spam from the model's input and ensures clean stdout-only output.
#### 1b. Add YouTube-specific synthesis instruction to SKILL.md
After the "Read the ENTIRE output" instruction, add:
```markdown
**The script output has THREE sections: Reddit items, X items, and YouTube items (in that order).
If you see YouTube items in the output, you MUST include them in your synthesis and stats block.
YouTube items look like: `**{video_id}** (score:N) {channel} [N views, N likes]`**
```
#### 1c. Verify fix with test run
Run `/last30days21 youtube thumbnail tips` and confirm YouTube appears in stats.
### Phase 2: Fix Search Quality (High — headline feature quality)
#### 2a. Remove `--flat-playlist` flag
**File:** `scripts/lib/youtube_yt.py:110-116`
```python
# Before:
cmd = [
"yt-dlp",
f"ytsearch{count}:{core_topic}",
"--dateafter", date_filter,
"--flat-playlist",
"--dump-json",
]
# After:
cmd = [
"yt-dlp",
f"ytsearch{count}:{core_topic}",
"--dateafter", date_filter,
"--dump-json",
"--no-warnings",
"--no-download",
]
```
**Impact:** Slower (yt-dlp resolves each video page for metadata) but:
- `--dateafter` actually works — filters old content
- `upload_date` populated — items get real dates
- Engagement metrics more accurate
**Risk:** Could increase search time from ~5s to ~30-60s for 20 videos. Mitigate by reducing default count or increasing timeout.
**Alternative if too slow:** Keep `--flat-playlist` but append year to search query:
```python
# Bias toward recent content since --dateafter doesn't work with flat-playlist
search_query = f"{core_topic} {from_date[:4]}" # e.g., "youtube thumbnail 2026"
```
#### 2b. Fix `_extract_core_subject()` for YouTube-relevant terms
**File:** `scripts/lib/youtube_yt.py:67-76`
Don't strip terms that are useful YouTube content type signals:
```python
# YouTube-specific: keep 'tips', 'tutorial', 'review' etc.
# These are stripped for Reddit/X search but are valuable for YouTube
noise = {
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features',
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
}
# NOTE: 'tips', 'tricks', 'tutorial', 'guide', 'review', 'reviews'
# are intentionally KEPT — they're YouTube content types
```
#### 2c. Clean question marks and trailing punctuation
**File:** `scripts/lib/youtube_yt.py:48-80`
```python
# At the end of _extract_core_subject():
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.') # Clean trailing punctuation
```
### Phase 3: Polish (Medium — nice to have before launch)
#### 3a. Increase subprocess timeout for non-flat-playlist mode
**File:** `scripts/lib/youtube_yt.py:119-121`
```python
# Before:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
# After — resolving video pages takes longer:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
```
#### 3b. Add year hint to search query for recency bias
Even with `--dateafter` working, YouTube's search algorithm ranks by relevance not recency. Adding the year helps:
```python
# After core_topic extraction:
import datetime
current_year = datetime.datetime.now().year
search_query = f"{core_topic} {current_year}"
```
#### 3c. Reduce compact render item count for YouTube
The compact render currently shows up to 15 YouTube items. With transcripts, this is too much output. Reduce to 10:
**File:** `scripts/lib/render.py` — in `render_compact()`, add YouTube-specific limit or reduce the default.
## Acceptance Criteria
- [ ] `/last30days21 youtube thumbnail tips` shows YouTube in stats block
- [ ] YouTube items have real dates (not `None`)
- [ ] Old content (> 30 days) filtered out by `--dateafter`
- [ ] "youtube thumbnail tips" search returns videos about thumbnail tips (not generic old content)
- [ ] "How to access Seedance 2" returns recent Seedance 2 tutorials
- [ ] Script completes within 5 minutes for default depth
- [ ] No `2>&1` in SKILL.md bash command
## Files to Modify
| File | Changes |
|------|---------|
| `SKILL.md` | Remove `2>&1`, add YouTube synthesis instruction |
| `scripts/lib/youtube_yt.py` | Remove `--flat-playlist`, fix noise words, add year hint, increase timeout |
| `scripts/lib/render.py` | Optional: reduce YouTube item limit in compact mode |
## Testing
```bash
# Quick smoke test — should show YouTube items with dates
cd ~/.claude/skills/last30days21
python3 scripts/last30days.py "youtube thumbnail tips" --quick --emit=compact 2>/dev/null | grep -c "youtube.com"
# Date test — should show YYYY-MM-DD dates, not None
python3 -c "
from scripts.lib import youtube_yt
r = youtube_yt.search_youtube('youtube thumbnail tips', '2026-01-16', '2026-02-15', depth='quick')
for v in r['items'][:3]: print(v['date'], v['title'][:50])
"
# Full integration — run the skill in Claude Code and verify YouTube in stats
```
## References
- SKILL.md bash command: `scripts/last30days.py` line 79
- YouTube search: `scripts/lib/youtube_yt.py` lines 83-174
- Core subject extraction: `scripts/lib/youtube_yt.py` lines 48-80
- Compact render: `scripts/lib/render.py` lines 48-238
- Prior YouTube plan: `docs/plans/2026-02-14-feat-youtube-transcript-search-plan.md`
docs/plans/2026-02-15-fix-youtube-timeout-and-reddit-resilience-plan.md
-102
View File
@@ -1,102 +0,0 @@
---
title: Fix YouTube Timeout and Reddit Resilience
type: fix
date: 2026-02-15
---
# Fix YouTube Timeout and Reddit Resilience
## Overview
YouTube search+transcript fetching exceeds the 60s future timeout on popular topics (20 videos + 5 transcripts), discarding all results. Reddit 429 rate-limiting burns through the entire time budget with aggressive retries, causing global timeouts. Both issues cause the script to return incomplete results and sometimes crash.
## Problem Statement
**YouTube timeout (blocks launch):** YouTube found 20 Seedance 2.0 videos and fetched 4/5 transcripts, but the 60s future timeout killed everything — `✗ Error: YouTube search timed out after 60s`. The data was there; the budget wasn't.
**Reddit 429 cascade (degrades reliability):** One enrichment item hitting 429 burns up to 93s (3 retries x 30s timeout + backoff) against a 45s budget. Phase 2 supplemental search then also 429s, burning another 30s. Total wasted: 75s on doomed requests, triggering the 180s global timeout.
Observed failures across 6 test runs:
- `seedance 2 access`: YouTube timed out (60s), Reddit 0 threads
- `kanye west bully`: Global timeout first run, needed --quick retry
- `Peter Steinberger`: Global timeout during enrichment
- `nano banana pro`: Reddit timed out, YouTube worked (5 videos in time)
- `kanye west bully` (retry): Reddit 0 threads, YouTube 4 videos
## Proposed Solution
### Fix 1: Bump YouTube future timeout (NOT YET DONE)
**File:** `scripts/last30days.py:40-44`
Give YouTube its own timeout, separate from the shared `future` timeout. YouTube inherently takes longer because it does search + parallel transcript fetching.
```python
# Option A: YouTube-specific timeout key
TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 30, "youtube_future": 60, ...},
"default": {"global": 180, "future": 60, "youtube_future": 90, ...},
"deep": {"global": 300, "future": 90, "youtube_future": 120, ...},
}
# Option B: Simpler — just bump default future to 90 for all sources
TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 45, ...},
"default": {"global": 180, "future": 90, ...},
"deep": {"global": 300, "future": 120, ...},
}
```
**Recommendation:** Option A (YouTube-specific key) — keeps Reddit/X futures tight while giving YouTube the breathing room it needs. Reddit/X finish in 20-40s; YouTube needs 60-90s for transcript fetching.
Then where YouTube future is collected (~line 646):
```python
youtube_timeout = timeouts.get("youtube_future", timeouts["future"])
youtube_items, youtube_error = youtube_future.result(timeout=youtube_timeout)
```
### Fix 2: Reddit 429 fail-fast (ALREADY DONE — needs commit)
**Status:** Implemented in working tree, uncommitted. Changes across 4 files:
| File | Change |
|------|--------|
| `scripts/lib/http.py` | `get_reddit_json()` accepts `timeout` and `retries` params (was hardcoded 30s/3) |
| `scripts/lib/reddit_enrich.py` | `RedditRateLimitError` exception; `fetch_thread_data()` propagates 429; `enrich_reddit_item()` defaults to 10s timeout / 1 retry |
| `scripts/lib/openai_reddit.py` | `search_subreddits()` reduced to 1 retry; breaks subreddit loop on first 429 |
| `scripts/last30days.py` | Enrichment loop catches `RedditRateLimitError`, cancels futures, bails; `rate_limited` flag skips Phase 2 Reddit |
**Impact:** 429 scenario drops from ~75s wasted to ~12s wasted.
## Acceptance Criteria
- [ ] YouTube `seedance 2 access` query completes with videos (was timing out at 60s)
- [ ] YouTube `kanye west bully` query returns 4+ videos with transcripts
- [ ] Reddit 429 detected within ~12s, remaining enrichment skipped
- [ ] Phase 2 Reddit skipped when rate-limited
- [ ] No global timeout on default depth for any of the 5 test queries
- [ ] All changes committed and synced to `~/.claude/skills/last30days21/` and `~/.claude/skills/last30days/`
## Files to Modify
| File | Status | Changes |
|------|--------|---------|
| `scripts/last30days.py` | Modify (partially done) | Add `youtube_future` to TIMEOUT_PROFILES; use it for YouTube future collection |
| `scripts/lib/http.py` | Done (uncommitted) | Parameterized `get_reddit_json()` timeout/retries |
| `scripts/lib/reddit_enrich.py` | Done (uncommitted) | `RedditRateLimitError`, fail-fast enrichment |
| `scripts/lib/openai_reddit.py` | Done (uncommitted) | 429 early-bail in `search_subreddits()` |
## Testing
```bash
# Quick smoke test — YouTube should complete within 90s
cd ~/.claude/skills/last30days21
python3 scripts/last30days.py "seedance 2 access" --emit=compact 2>&1 | grep -E "YouTube|timeout"
# Verify YouTube items in output
python3 scripts/last30days.py "kanye west bully" --quick --emit=compact 2>/dev/null | grep "youtube.com" | wc -l
# Full integration — run the skill in Claude Code
# /last30days21 seedance 2 access
# Verify YouTube appears in stats block
```
docs/plans/2026-02-17-feat-github-release-v2.1-plan.md
-255
View File
@@ -1,255 +0,0 @@
---
title: "feat: Create GitHub Release for v2.1"
type: feat
date: 2026-02-17
---
# Create GitHub Release for v2.1
## Overview
The last30days skill is at v2.1 with 2,747 stars and 317 forks, but has **zero git tags and zero GitHub Releases**. The code is already live on the public repo (`upstream/main`). All marketing copy is written. This plan creates a proper v2.1.0 GitHub Release from existing materials.
## Problem Statement
GitHub Releases provide:
- A landing page for each version with formatted release notes
- Discoverability (GitHub shows releases in the sidebar, feeds them to the Explore page)
- A stable reference point for users (`git checkout v2.1.0`)
- A "Full Changelog" diff link
- RSS feed for watchers
- New contributor callouts (community goodwill)
Currently users have no way to reference a specific version of the skill. The README says "v2.1" but there's nothing in git to anchor that.
## Proposed Solution
Create the first GitHub Release (v2.1.0) on the **public** repo using existing marketing copy. Use an annotated tag (not lightweight) for proper `git describe` support and fork propagation.
## Implementation Steps
### Phase 1: Tag and Release
#### 1. Create CHANGELOG.md
Create `CHANGELOG.md` following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format. Source material already exists:
- `README.md` "What's New in V2.1" and "What's New in V2" sections
- `docs/v2.1-tweets.md` (raw verified test results, feature descriptions)
- `docs/v2.1-launch-copy.md` (feature copy, social posts)
- `docs/pr-credits.md` (contributor credits)
- Git log (`git log --oneline` for commit references)
Structure:
```markdown
# Changelog
## [2.1.0] - 2026-02-15
### Highlights
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
Three headline features...
### Added
- Open-class skill with watchlists (SQLite-backed, FTS5)
- YouTube as 4th research source via yt-dlp (search + transcript extraction)
- OpenAI Codex CLI compatibility ($last30days invocation)
- Bundled X search (vendored Bird GraphQL client, no external CLI)
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar)
- Briefing and history modes (open variant)
- --diagnose flag for source status checking
- --store flag for SQLite accumulation
### Changed
- Smarter query construction (strips noise words, auto-retry)
- Two-phase search (Phase 2 entity-aware drill-down)
- Reddit JSON enrichment (real upvotes/comments from reddit.com/.json)
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
- Model auto-selection with 7-day cache
### Fixed
- YouTube timeout increased to 90s
- Reddit 429 rate limit fail-fast
- YouTube soft date filter (keeps evergreen content)
- Eager import crash in __init__.py (Codex compatibility)
### New Contributors
- @JosephOIbrahim - Windows Unicode fix
- @levineam - Model fallback for unverified orgs
- @jonthebeef - --days=N configurable lookback flag
### Credits
- @steipete - Bird CLI (vendored X search) and yt-dlp inspiration
- @galligan - Marketplace plugin inspiration
- @hutchins - Pushed for YouTube feature
## [1.0.0] - 2026-01-15
Initial public release. Reddit + X search via OpenAI and xAI APIs.
```
#### 2. Create annotated tag on the public repo
```bash
cd /Users/mvanhorn/last30days-skill-private
# Tag on the current HEAD (which matches upstream/main)
git tag -a v2.1.0 -m "Release v2.1.0: Watchlists, YouTube transcripts, Codex CLI, bundled X search"
# Push to public repo
git push upstream v2.1.0
```
Use annotated tag (not lightweight) because:
- Stores tagger metadata and date
- Works with `git describe`
- Propagates to forks (317 forks)
- Supports GPG signing if desired later
#### 3. Craft release notes
Assemble from existing copy. The release body should follow this structure:
```markdown
## Highlights
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across **Reddit, X, YouTube, and the web**
from the last 30 days, finds what the community is actually upvoting, sharing,
and saying on camera, and writes you a prompt that works today.
**Three headline features in v2.1:**
1. **Open-class skill with watchlists** - Track competitors, people, topics on a schedule.
Pair with Open Claw for automated briefings. SQLite-backed with FTS5 search.
2. **YouTube transcripts as a 4th source** - When yt-dlp is installed, searches YouTube,
grabs view counts, and reads the actual transcripts. A 20-minute review has 10x the
signal of one X post.
3. **Codex CLI compatibility** - Same skill, same engine, same four sources.
Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
Plus: **Bundled X search** - Vendored Bird GraphQL client. No external CLI needed.
Just Node.js 22+ and browser cookies.
## Real Results (verified 2/15)
| Topic | Reddit | X | YouTube | Web |
|-------|--------|---|---------|-----|
| Nano Banana Pro | - | 32 posts, 164 likes | 5 videos, 98K views | - |
| Seedance 2.0 | 21 threads | 33 posts | 20 videos | 5 pages |
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views | - |
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts | 18 videos, 6.15M views | - |
## What's New
### Added
- Open-class skill with watchlist, briefing, and history modes
- YouTube search + transcript extraction via yt-dlp
- OpenAI Codex CLI compatibility
- Bundled Twitter/X search (vendored Bird GraphQL)
- Native web search (Parallel AI, Brave, OpenRouter)
- `--diagnose` and `--store` flags
- Conversational first-run experience (NUX)
### Changed
- Two-phase search architecture (entity-aware drill-down)
- Reddit JSON enrichment for real engagement metrics
- Smarter query construction with auto-retry
- Engagement-weighted scoring algorithm
### Fixed
- YouTube/Reddit timeout resilience
- Reddit 429 rate limit fail-fast
- Eager import crash in Codex environments
## New Contributors
- @JosephOIbrahim - Windows Unicode fix
- @levineam - Model fallback for unverified orgs
- @jonthebeef - `--days=N` configurable lookback
## Credits
- @steipete - Bird CLI and yt-dlp/summarize inspiration
- @galligan - Marketplace plugin inspiration
- @hutchins - Pushed for YouTube feature
## Install
```bash
# Claude Code
claude install-skill https://github.com/mvanhorn/last30days-skill
# Manual
git clone https://github.com/mvanhorn/last30days-skill ~/.claude/skills/last30days
```
**Full Changelog**: https://github.com/mvanhorn/last30days-skill/commits/v2.1.0
```
#### 4. Create the GitHub Release
```bash
gh release create v2.1.0 \
--repo mvanhorn/last30days-skill \
--verify-tag \
--title "v2.1.0 - Watchlists, YouTube Transcripts, Codex CLI" \
-F release-notes.md
```
No binary assets needed - this is an interpreted skill, and GitHub auto-generates source archives.
### Phase 2: Future Automation (Optional)
#### 5. Add `.github/release.yml` for auto-generated notes on future releases
```yaml
changelog:
exclude:
labels:
- ignore-for-release
categories:
- title: "Breaking Changes"
labels: [breaking-change]
- title: "Features"
labels: [enhancement, feature]
- title: "Bug Fixes"
labels: [bug, fix]
- title: "Documentation"
labels: [documentation]
- title: "Other Changes"
labels: ["*"]
```
This enables `gh release create v2.2.0 --generate-notes` for future releases with automatic PR categorization.
## Acceptance Criteria
- [x] `CHANGELOG.md` exists in repo root following Keep a Changelog format
- [x] Annotated tag `v2.1.0` exists on the public repo
- [x] GitHub Release `v2.1.0` is published at `github.com/mvanhorn/last30days-skill/releases`
- [x] Release notes include: highlights, real results table, feature list, contributor credits, install instructions
- [x] Release appears in the repo sidebar on GitHub
## Key Decisions
1. **v2.1.0 only** - Don't retroactively create v1.0.0 or v2.0.0 tags. The git history doesn't have clean boundary commits for those versions, and retroactive tags add complexity without value.
2. **Release on the public repo** (`upstream`), not the private repo. Users see `mvanhorn/last30days-skill`.
3. **CHANGELOG.md as source of truth** - The release notes are derived from CHANGELOG.md, not the other way around. Future releases update CHANGELOG.md first, then create the release.
4. **Update star count** - The existing copy says "1.5K stars" but the repo now has 2,747. Update in release notes.
## References
- Existing copy: `docs/v2.1-launch-copy.md`, `docs/v2.1-tweets.md`
- Contributors: `docs/pr-credits.md`
- README features: `README.md` "What's New" sections
- GitHub Releases docs: https://docs.github.com/en/repositories/releasing-projects-on-github
- Keep a Changelog: https://keepachangelog.com/en/1.1.0/
docs/plans/2026-02-20-feat-last30days-com-trending-topics-plan.md
-274
View File
@@ -1,274 +0,0 @@
---
title: "feat: Last30Days.com - Automated Trending Topic Research"
type: feat
date: 2026-02-20
---
# Last30Days.com - Automated Trending Topic Research
## Overview
A website at Last30Days.com that automatically shows daily trending topics researched by the /last30days engine. The core challenge: the skill is query-driven (you give it a topic), but a trending page needs to *discover* what topics to research. This plan covers how to source trending topics, run them through the engine, and publish results.
## Problem Statement
The /last30days skill has 2,747 GitHub stars but no public-facing showcase. Users have to install the skill and run it themselves. A website that automatically shows trending topic results would:
1. **Drive installs** - people see the quality and want it for their own topics
2. **SEO surface area** - each topic page is indexable content
3. **Demonstrate capability** - "here's what /last30days found about X today" with real stats
4. **Content flywheel** - daily fresh content with zero manual curation
## Trending Topic Discovery: The Options
The skill has **no existing trending discovery mechanism** - it's entirely query-driven. Here are the available sources, ranked by practicality.
### Tier 1: Free, High Signal, Zero Friction
| Source | What It Returns | Auth | Cost | Best For |
|--------|----------------|------|------|----------|
| **Wikipedia Pageviews** | Top 100 most-viewed articles yesterday | None | Free | General public interest (news, culture, events) |
| **Hacker News** | Top 30 stories with scores | None | Free | Tech/startup topics |
| **Reddit r/all/hot + rising** | Hottest and rapidly rising posts | OAuth (free) | Free | Broad internet culture |
| **Google News RSS** | Top headlines, algorithmically curated | None | Free | Mainstream news |
| **YouTube mostPopular** | Top 50 trending videos by region | API key | Free (10K units/day) | Pop culture, entertainment |
### Tier 2: Very Cheap, High Value
| Source | What It Returns | Auth | Cost | Best For |
|--------|----------------|------|------|----------|
| **Perplexity Sonar API** | "What's trending today?" with sourced answers | API key | ~$1/month | Meta-aggregator that replaces multiple sources |
| **Google Trends (pytrends)** | Daily trending Google searches | None | Free but flaky | What people are actually searching |
| **Bird `getNews()`** | X Explore page trending topics | Browser cookies | Free | Real-time Twitter/X conversation |
### Tier 3: Paid, Skip for MVP
| Source | Cost | Why Skip |
|--------|------|----------|
| X/Twitter API | $200/month minimum | Too expensive; Bird `getNews()` is free |
| Exploding Topics | $249/month | Overkill for daily trends |
| TikTok | Gated, requires approval | Application process |
### Existing Codebase Hooks
Several pieces already exist in the codebase that could be leveraged:
1. **Bird `getNews()` is already on disk.** The vendored Bird library at `scripts/lib/vendor/bird-search/` includes `twitter-client-news.js` which fetches X's Explore page tabs (For You, Trending, News, Sports, Entertainment). Only needs a ~30-line `bird-news.mjs` wrapper to expose it. Zero API keys needed.
2. **`store.get_trending(days=7)`** at `scripts/store.py:559` ranks watchlist topics by recent finding activity. Could feed "what the skill has been researching" as a meta-signal.
3. **Scoring algorithm** at `scripts/lib/score.py` has engagement formulas for Reddit, X, and YouTube that could rank "buzz" by reweighting engagement >> relevance.
4. **Brave Trending API** exists but isn't implemented in `brave_search.py`. Could be added.
## Proposed Architecture
### Topic Discovery Pipeline (daily cron)
```
┌─────────────────────────────────────────────────┐
│ STEP 1: Fetch trending signals (parallel, free) │
├─────────────────────────────────────────────────┤
│ Wikipedia Pageviews ─┐ │
│ Hacker News Top 30 ─┤ │
│ Reddit r/all/hot ─┼─→ Raw topics + signals │
│ Google News RSS ─┤ │
│ YouTube mostPopular ─┤ │
│ Bird getNews() ─┘ │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ STEP 2: Cluster + rank │
├─────────────────────────────────────────────────┤
│ Extract topic keywords from titles │
│ Cluster by semantic similarity │
│ Rank by cross-source frequency │
│ ("Pope Leo XIV" on Wikipedia + Reddit + News │
│ = high confidence trend) │
│ Output: top 15-20 topics, ranked │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ STEP 3: Run /last30days on top topics │
├─────────────────────────────────────────────────┤
│ Top 3-5 topics: full research (--emit=json) │
│ → Full "What I learned" synthesis articles │
│ Remaining 10-15: quick research (--quick) │
│ → Stats teasers (32 X posts, 5 YouTube...) │
└─────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────┐
│ STEP 4: Publish to website │
├─────────────────────────────────────────────────┤
│ Generate static HTML/JSON │
│ Deploy to Last30Days.com │
│ RSS feed for subscribers │
└─────────────────────────────────────────────────┘
```
### Topic Clustering Strategy
The hardest part is deduplicating across sources. "Pope Francis Dies" (Google News), "Pope_Francis" (Wikipedia #1 viewed), and a r/worldnews post are all the same topic. Approaches:
**Option A: LLM clustering (recommended for MVP)**
Feed all raw titles to an LLM and ask it to cluster into distinct topics with a representative label. ~$0.01 per day via a fast model. Simple, accurate, handles edge cases.
**Option B: TF-IDF + cosine similarity**
Extract keywords, compute pairwise similarity, agglomerative clustering. No API cost but worse on paraphrased titles.
**Option C: Embedding similarity**
Embed all titles, cluster by cosine distance. Better than TF-IDF, costs a few cents per day.
### Website Architecture
**Option A: Static site (recommended for MVP)**
- Daily cron generates JSON + static HTML
- Host on GitHub Pages, Cloudflare Pages, or Vercel
- Zero server cost, zero maintenance
- Framework: plain HTML/CSS, or minimal Astro/11ty
**Option B: Next.js with ISR**
- Incremental Static Regeneration rebuilds pages daily
- More flexibility for future features (search, filtering, user accounts)
- Hosting: Vercel free tier
**Option C: Full web app**
- Database-backed, real-time updates, user accounts
- Overkill for MVP
### Cost Estimate (daily operation)
| Item | Cost |
|------|------|
| Trending source APIs | $0 (all free tier) |
| LLM clustering (fast model) | ~$0.01/day |
| Full research on 3-5 topics | ~$0.10-0.50/day (OpenAI API for Reddit search) |
| Quick research on 10-15 topics | ~$0.05-0.20/day |
| Static hosting | $0 (GitHub/Cloudflare Pages) |
| **Total** | **~$5-20/month** |
## Implementation Phases
### Phase 1: Topic Discovery Script (MVP)
Build `scripts/discover_trending.py` that:
- Fetches Wikipedia Pageviews, HN top stories, Reddit hot, Google News RSS in parallel
- Filters out evergreen/non-topical Wikipedia pages (e.g., "Main Page", "ChatGPT" permanent traffic)
- Uses a fast LLM to cluster raw titles into 15-20 distinct topics
- Outputs ranked JSON: `[{topic, sources, confidence, category}]`
**Acceptance criteria:**
- [ ] Fetches from at least 4 free trending sources in parallel
- [ ] Clusters raw titles into deduplicated topics via LLM
- [ ] Filters Wikipedia evergreen pages (maintain a blocklist)
- [ ] Outputs ranked JSON with topic name, source count, category
- [ ] Runs in < 60 seconds
- [ ] No paid API keys required for discovery (only LLM clustering)
### Phase 2: Research Automation
Wire discovered topics into the existing /last30days engine:
- Top 3-5 topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --deep`
- Remaining topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --quick`
- Store all results in `~/.local/share/last30days/trending/YYYY-MM-DD/`
**Acceptance criteria:**
- [ ] Orchestration script runs discovery then research sequentially
- [ ] Full research on top N topics, quick on the rest
- [ ] Results stored as dated JSON files
- [ ] Total daily runtime < 30 minutes
- [ ] Handles timeouts/failures gracefully (skip topic, continue)
### Phase 3: Website Generation
Build a static site generator that reads the daily JSON and produces Last30Days.com:
- Homepage: today's trending topics grid (title, category, key stat, source badges)
- Topic pages: full synthesis for showcase topics, stats teaser for others
- Archive: previous days accessible by date
- RSS feed
- CTA: "Want to research your own topic? Install the skill"
**Acceptance criteria:**
- [ ] Static HTML generated from daily JSON
- [ ] Homepage shows today's 15-20 trending topics
- [ ] 3-5 showcase topic pages with full synthesis
- [ ] Remaining topics show stats teasers + install CTA
- [ ] Deploys to Last30Days.com (Cloudflare Pages or similar)
- [ ] RSS feed for daily updates
- [ ] Mobile responsive
### Phase 4: Bird Trending Integration (bonus)
Wire up the already-vendored Bird `getNews()` for X trending:
- Create `scripts/lib/vendor/bird-search/bird-news.mjs` (~30 lines)
- Add X Explore trending data as a 5th discovery source
- X trends are the fastest-moving signal and fill the "what's happening right now" gap
**Acceptance criteria:**
- [ ] `bird-news.mjs` wrapper exposes X Explore trending topics
- [ ] Integrated into discovery pipeline as an additional source
- [ ] Falls back gracefully if no X session cookies available
## Alternative Approaches Considered
### Perplexity-only approach
Just ask Perplexity Sonar "what are the top 20 trending topics today?" daily. Simpler, but:
- Single point of failure
- Less transparent (can't show "sourced from Reddit, Wikipedia, HN")
- Model may hallucinate or miss niche topics
- **Verdict:** Good fallback, not primary.
### Curated topics (manual)
Manually pick topics each day. Defeats the purpose of automation. Could supplement for "editorial picks."
### Social listening tools (Brandwatch, Sprout Social, etc.)
Expensive ($500+/month), enterprise-focused, overkill.
## Technical Considerations
- **Rate limits:** Wikipedia, HN, and RSS have no practical limits for 1 daily call. Reddit's 100 QPM is generous. YouTube's 10K units/day allows ~3,000 `mostPopular` calls.
- **Wikipedia filtering:** The top Wikipedia pages are always "Main Page", "Special:Search", etc. Need a blocklist of ~50 evergreen pages plus heuristics (skip pages under 1,000 characters, skip disambiguation pages).
- **Cron timing:** Run discovery at ~2am UTC (after Wikipedia pageviews finalize for previous day). Run research at ~3am UTC. Deploy site by ~5am UTC.
- **Cost control:** The /last30days engine uses OpenAI API for Reddit search. Full research on 5 topics * $0.05-0.10 each = ~$0.25-0.50/day. Quick research is cheaper.
- **Domain:** Last30Days.com needs to be registered (check availability).
## Dependencies & Risks
| Risk | Mitigation |
|------|------------|
| pytrends breaks (Google changes) | pytrends is a bonus source, not required. Other 4+ sources sufficient. |
| Wikipedia pageviews delayed | Fall back to Perplexity Sonar for day's topics |
| OpenAI API cost spikes | Cap at 5 full + 15 quick topics per day; use --quick for most |
| Bird cookie auth stops working | X trending is a bonus source; discovery works without it |
| Domain not available | Check availability; alternatives: last30.day, last30days.app |
## Success Metrics
- Daily automated publishing with zero manual intervention
- 15-20 trending topics surfaced daily
- 3-5 full synthesis articles per day
- Site loads in < 2 seconds (static)
- Drives measurable GitHub star growth / skill installs
## References
### Trending APIs (free)
- Wikipedia Pageviews: `wikimedia.org/api/rest_v1/metrics/pageviews/top/{project}/{access}/{year}/{month}/{day}`
- Hacker News: `hacker-news.firebaseio.com/v0/topstories.json`
- Reddit: `oauth.reddit.com/r/all/hot` (or `reddit.com/r/all/hot.json` unauthenticated)
- Google News RSS: `news.google.com/rss`
- YouTube: `googleapis.com/youtube/v3/videos?chart=mostPopular`
### Existing codebase hooks
- Bird `getNews()`: `scripts/lib/vendor/bird-search/vendor/package/dist/lib/twitter-client-news.js`
- Store trending: `scripts/store.py:559` (`get_trending()`)
- Scoring: `scripts/lib/score.py` (engagement formulas)
- Brave Trending API: not implemented, available in Brave docs
### Inspiration
- [Keep a Changelog](https://keepachangelog.com/) - clean daily update format
- [Hacker News front page](https://news.ycombinator.com/) - minimal trending UI
- [Exploding Topics](https://explodingtopics.com/) - trending topic showcase (paid, $249/mo)
docs/plans/2026-02-24-feat-hacker-news-source-plan.md
-235
View File
@@ -1,235 +0,0 @@
---
title: "feat: Add Hacker News as a 5th research source"
type: feat
status: completed
date: 2026-02-24
---
# feat: Add Hacker News as a 5th Research Source
## Overview
Add Hacker News as a source to the last30days skill, using the free Algolia HN Search API (`hn.algolia.com/api/v1`). HN provides high-signal content from a technical audience — stories with high point counts and active comment threads are strong indicators of what the developer community actually cares about. No API key required.
## Problem Statement / Motivation
The skill currently covers Reddit, X, YouTube, and web. Hacker News is missing — and for technical topics, HN often surfaces discussions that don't appear on Reddit or X. HN's upvote system and comment culture produce high-quality signal: a 500-point story with 200 comments means the developer community is genuinely engaged. Community contributor @wkbaran proposed this in PR #26 alongside YouTube and Product Hunt. YouTube shipped in v2.1; now it's time for HN.
## Proposed Solution
Add `scripts/lib/hackernews.py` following the exact same pattern as `youtube_yt.py` (the simplest existing source — no API key, just HTTP calls). Use the Algolia HN Search API for discovery, then optionally fetch top comments from high-scoring stories for enrichment (like Reddit enrichment, but using the `/items/:id` endpoint instead of Reddit's JSON API).
### Two-phase approach (matches existing Reddit pattern):
1. **Phase 1 — Search**: Query Algolia for stories matching the topic within the date range. Get titles, URLs, points, comment counts.
2. **Phase 2 — Enrichment** (optional, top N stories): Fetch the `/items/:id` endpoint for the highest-scoring stories to get top-level comments. This gives "comment_insights" like Reddit enrichment does.
## Technical Approach
### Files to Create
#### `scripts/lib/hackernews.py`
The main source module. Pattern matches `youtube_yt.py` (simplest source).
```python
# scripts/lib/hackernews.py
"""Hacker News search via Algolia API (free, no auth required)."""
ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 10,
}
```
Key functions:
- `search_hackernews(topic, from_date, to_date, depth="default") -> Dict[str, Any]`
- Calls `hn.algolia.com/api/v1/search?query={topic}&tags=story&numericFilters=created_at_i>{from_ts},created_at_i<{to_ts}&hitsPerPage={count}`
- Uses `http.get()` — stdlib only, matches existing pattern
- Returns raw Algolia response
- `parse_hackernews_response(response: Dict) -> List[Dict]`
- Extracts hits, maps to raw dicts with fields: `id` (prefix "HN"), `title`, `url`, `hn_url`, `author`, `date`, `engagement` (points, num_comments), `why_relevant`, `relevance`
- `relevance` estimated from Algolia rank + engagement boost (same pattern as Bill's PR)
- `enrich_top_stories(items, depth="default") -> List[Dict]`
- Fetches `/items/{objectID}` for top N stories (by points)
- Extracts top-level comments (author, text, points)
- Adds `top_comments` and `comment_insights` fields (same structure as Reddit enrichment)
- Uses `ThreadPoolExecutor` for parallel fetching
- `_date_to_unix(date_str: str) -> int` — Helper, converts YYYY-MM-DD to Unix timestamp
#### `tests/test_hackernews.py`
Standard unittest pattern matching existing tests.
- Test `parse_hackernews_response` with sample Algolia response
- Test `_date_to_unix` conversion
- Test empty response handling
- Test enrichment parsing
- Test score integration with `score.py`
### Files to Modify
#### `scripts/lib/schema.py`
- [x] Add `HackerNewsItem` dataclass:
```python
@dataclass
class HackerNewsItem:
id: str # "HN1", "HN2", ...
title: str
url: str # Original article URL
hn_url: str # news.ycombinator.com/item?id=...
author: str # HN username
date: Optional[str]
date_confidence: str # Always "high" (Algolia provides exact timestamps)
engagement: Optional[Engagement] # points + num_comments
top_comments: List[Comment] # From enrichment
comment_insights: List[str] # From enrichment
relevance: float
why_relevant: str
subs: SubScores
score: int
```
- [x] Add `hackernews: List[HackerNewsItem] = field(default_factory=list)` to `Report`
- [x] Add `hackernews_error: Optional[str] = None` to `Report`
- [x] Update `Report.to_dict()` and `Report.from_dict()`
#### `scripts/lib/normalize.py`
- [x] Add `normalize_hackernews_items(items: List[Dict], from_date, to_date) -> List[schema.HackerNewsItem]`
- Maps raw dicts to `HackerNewsItem` dataclass instances
- Sets `date_confidence = "high"` (Algolia provides `created_at_i` exact timestamps)
- Converts `engagement` dict to `schema.Engagement(score=points, num_comments=num_comments)`
#### `scripts/lib/score.py`
- [x] Add `compute_hackernews_engagement_raw(engagement) -> float`
- Formula: `0.55 * log1p(points) + 0.45 * log1p(num_comments)`
- Points are the primary signal on HN; comments indicate depth of discussion
- [x] Add `score_hackernews_items(items) -> List[schema.HackerNewsItem]`
- Uses standard 45/25/30 weights (relevance/recency/engagement) — same as Reddit/X/YouTube
- [x] Update `sort_items()` to handle `HackerNewsItem`
- Source priority: Reddit > X > **HN** > YouTube > WebSearch
- HN slots between X and YouTube: higher signal than YouTube (curated upvotes vs raw views), but X has real-time pulse
#### `scripts/lib/dedupe.py`
- [x] Add `dedupe_hackernews(items, threshold=0.7) -> List[schema.HackerNewsItem]`
- [x] Update `get_item_text()` to handle `HackerNewsItem` (return `title`)
- [x] Consider cross-source dedup: HN stories often link to the same URLs that appear in web search results. Dedupe by URL match across `hackernews` and `websearch` items.
#### `scripts/lib/render.py`
- [x] Add HN section to `render_compact()`:
```
### Hacker News Stories
**HN1** (score:85) hn/username (2026-02-15) [350pts, 127cmt]
Story title here
https://news.ycombinator.com/item?id=12345
*Why relevant*
```
- [x] Add to `render_source_status()`:
```
✅ HN: {N} stories
```
- [x] Add to `render_full_report()` and `render_context_snippet()`
- [x] Update `_assess_data_freshness()` to include HN items
#### `scripts/last30days.py`
- [x] Import: `from lib import hackernews`
- [x] Add `_search_hackernews(topic, from_date, to_date, depth) -> (items, error)` wrapper function
- Calls `hackernews.search_hackernews()`, then `hackernews.parse_hackernews_response()`
- Returns `(items, None)` or `([], error_string)`
- [x] Add to `TIMEOUT_PROFILES`: `"hackernews_future": 60` (default), `30` (quick), `90` (deep)
- [x] Add HN to `ThreadPoolExecutor` in `run_research()`:
```python
if do_hackernews:
hn_future = executor.submit(_search_hackernews, topic, from_date, to_date, depth)
```
- Increment `max_workers` by 1 when HN is enabled
- [x] Collect results: `hn_items, hn_error = hn_future.result(timeout=hn_timeout)`
- [x] Add HN enrichment phase (after Reddit enrichment, before Phase 2):
```python
if hn_items:
hn_items = hackernews.enrich_top_stories(hn_items, depth=depth)
```
- [x] Add to processing pipeline: normalize -> filter_by_date_range -> score -> sort -> dedupe
- [x] Assign to `report.hackernews` and `report.hackernews_error`
- [x] Update status UI: add `⏳ 🟡 HN Searching Hacker News...` and `✓ 🟡 HN Found {N} stories`
#### `scripts/lib/env.py`
- [x] HN is always available (no API key, no binary dependency)
- [x] Update `get_available_sources()` to include HN in the source list
- [x] Update `validate_sources()` to accept `hn` as a valid source name
- [x] Add `hn` to the `--search` flag documentation
#### `SKILL.md`
- [x] Update description: "Sources: Reddit, X, YouTube, **Hacker News**, and web"
- [x] Add HN to stats block template:
```
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
```
- [x] Update `metadata.clawdbot.tags` to include `hackernews`
- [x] Update Security section: add `hn.algolia.com` to endpoints list
- [x] Update citation priority to include HN: Reddit > X > YouTube > HN > Web
## Acceptance Criteria
- [x] `python3 scripts/last30days.py "AI coding agents" --emit=compact` shows HN section with stories, points, and comment counts
- [x] HN stories include `hn_url` linking to the HN discussion page (not just the article URL)
- [x] Top stories are enriched with top comments (like Reddit enrichment)
- [x] HN runs in parallel with Reddit/X/YouTube (no serial bottleneck)
- [x] Scoring uses standard 45/25/30 weights with engagement formula tuned for HN metrics
- [x] Stats block shows: `├─ 🟡 HN: {N} stories │ {N} points │ {N} comments`
- [x] `--search=hn` works to run HN only; `--search=reddit,hn` works for combos
- [x] `--quick`, `--deep` flags adjust HN result count (15/30/60)
- [x] All existing tests still pass
- [x] New `tests/test_hackernews.py` with tests for parse, normalize, score, enrichment
- [x] No API key required — works out of the box
- [x] Cross-source URL dedup: HN stories linking to same URL as web results get deduped
- [x] SKILL.md updated with HN in stats block, security section, and citation priority
## Dependencies & Risks
**Low risk:**
- Algolia HN API is free, public, no auth, well-established (used since 2014)
- No new dependencies — uses existing `http.py` (stdlib urllib)
- Pattern is identical to YouTube source (simplest existing source)
**Medium risk:**
- Algolia has no officially documented rate limit, but aggressive use could get throttled
- Mitigation: Existing `http.py` exponential backoff handles 429s
- Default depth only requests 30 items (1 API call for search + N for enrichment)
- Comment enrichment adds N API calls (one per story) which could slow down quick mode
- Mitigation: Limit enrichment to top 3/5/10 stories by depth; use ThreadPoolExecutor
**Compatibility:**
- HN always available — doesn't break anything when other sources are missing
- Existing `--search` flag needs extension but is backward-compatible
## Sources & References
- PR #26 by @wkbaran: [HN implementation reference](https://github.com/mvanhorn/last30days-skill/pull/26)
- Algolia HN API docs: `https://hn.algolia.com/api`
- Existing patterns: `scripts/lib/youtube_yt.py` (simplest source), `scripts/lib/openai_reddit.py` (enrichment pattern)
- Scoring reference: `scripts/lib/score.py:compute_reddit_engagement_raw()`
- Schema reference: `scripts/lib/schema.py:RedditItem` (closest analog to HN)
docs/plans/2026-02-24-fix-hn-ordering-and-emoji-plan.md
-86
View File
@@ -1,86 +0,0 @@
---
title: "fix: HN ordering and stats block emoji formatting"
type: fix
status: completed
date: 2026-02-24
---
# fix: HN Ordering and Stats Block Emoji Formatting
## Overview
HN source appears before YouTube in several places (stats block, sort priority, source status) but should be last among the main sources (after YouTube, before Web). The test skill SKILL.md also has plain ASCII instead of box-drawing characters and colored circle emojis.
Desired order everywhere: Reddit > X > YouTube > HN > Web
## Problem Statement / Motivation
When testing `/last30daysHN`, the stats block output shows:
- No ✅ emoji, no 🟠 🔵 🟡 🔴 🌐 🗣️ colored circles
- Plain `|-` instead of `├─` and `|` instead of `│`
- HN appears before YouTube in the stats
The canonical format (established in commit `7c36866`) is:
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
├─ 🌐 Web: {N} pages (supplementary)
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
```
## Technical Approach
### Files to Modify
#### `scripts/lib/score.py` - sort_items()
- [x] Swap HN and YouTube priority in `sort_items()`:
- YouTube: priority 2 (was 3)
- HN: priority 3 (was 2)
#### `scripts/lib/render.py` - render_source_status()
- [x] Move HN section (lines ~318-324) to AFTER YouTube section (lines ~326-334)
- [x] Verify render_compact() already has correct order (YouTube before HN) - no change expected
#### `SKILL.md` (private repo root)
- [x] Move the 🟡 HN stats line AFTER the 🔴 YouTube stats line in the template
- [x] Move HN line after YouTube in the footer summary template too
#### `~/.claude/skills/last30daysHN/SKILL.md` (test skill)
- [x] Restore full emoji + box-drawing stats template with correct ordering
- [x] Use canonical format: ✅ ├─ 🟠 🔵 🔴 🟡 🌐 🗣️ └─ │
#### `scripts/lib/ui.py` - show_complete()
- [x] Move HN output after YouTube in both TTY and non-TTY code paths
### Files that are already correct (no change needed)
- `render_compact()` in render.py - already YouTube before HN
- `render_context_snippet()` - score-based ordering, no fixed order
- `last30days.py` pipeline - processing order doesn't affect display
## Acceptance Criteria
- [x] Stats block shows: Reddit > X > YouTube > HN > Web (in that order)
- [x] Stats block has all emojis: ✅ 🟠 🔵 🔴 🟡 🌐 🗣️
- [x] Stats block uses box-drawing chars: ├─ └─ │
- [x] sort_items() tiebreaker: YouTube before HN
- [x] render_source_status() shows YouTube before HN
- [x] ui.py show_complete() shows YouTube before HN
- [x] All existing tests still pass
- [x] Run sync.sh to deploy after fixes
## Sources & References
- Canonical emoji format established in commit `7c36866` ("Fix v2 output quality")
- YouTube emoji added in commit `c66ca7f` ("feat: Add YouTube as 4th research source")
- HN added in commit `38a7ea2` ("feat(hackernews): add Hacker News as 5th research source")
docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
-3348
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md
-212
View File
@@ -1,212 +0,0 @@
---
title: "feat: 15-Test Side-by-Side Comparison - Make CROSS the GOAT"
type: feat
status: active
date: 2026-02-25
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
---
# 15-Test Side-by-Side Comparison - Make CROSS the GOAT
## Overview
Run all 5 canonical topics through all 3 skill versions (base, HN, CROSS) for 15 total full last30days runs. Save every result. Analyze side-by-side. Judge which version is best. Then create an improvement plan to make CROSS the definitive next release.
## Problem Statement / Motivation
The previous analysis only ran 5 tests on the CROSS branch - never comparing the same topics across all 3 versions. Without true side-by-side data, we can't judge whether CROSS is actually better or if the new features (dynamic YouTube scoring, cross-refs) introduce regressions. The user wants to see all 15 results, know which version wins, and get a concrete plan to make CROSS the GOAT before shipping.
## The 3 Versions
| Version | Git State | Sources | YouTube Relevance | Cross-Refs |
|---------|-----------|---------|-------------------|------------|
| **Base** | commit `427a4e4` | Reddit, X, YouTube, Web | Hardcoded 0.7 | No |
| **HN** | `main` / `f60a435` | Reddit, X, YouTube, **HN**, Web | Hardcoded 0.7 | No |
| **CROSS** | `feat/youtube-relevance-cross-source` / `0591f55` | Reddit, X, YouTube, HN, Web | Dynamic 0.1-1.0 | Yes (Jaccard 0.5) |
## The 5 Topics
| # | Topic | Category | Why chosen |
|---|-------|----------|------------|
| 1 | "Claude Code skills and MCP servers" | Developer tools | Core user base topic |
| 2 | "Seedance AI video generation" | Creative AI | Trending topic, cross-platform buzz |
| 3 | "M4 MacBook Pro review" | Consumer tech | Product review, mainstream |
| 4 | "best rap songs 2026" | Pop culture | Non-tech stress test |
| 5 | "React vs Svelte 2026" | Framework debate | Dev community, opinion-heavy |
## Test Matrix (15 Runs)
Run in **topic-sequential** order (all 3 versions of topic 1 back-to-back, then topic 2, etc.) to minimize temporal confounds on YouTube/X search results.
| Run | Topic | Version | Output File |
|-----|-------|---------|-------------|
| 1 | Claude Code | Base | `base-1-claude-code.json` |
| 2 | Claude Code | HN | `hn-1-claude-code.json` |
| 3 | Claude Code | CROSS | `cross-1-claude-code.json` |
| 4 | Seedance | Base | `base-2-seedance.json` |
| 5 | Seedance | HN | `hn-2-seedance.json` |
| 6 | Seedance | CROSS | `cross-2-seedance.json` |
| 7 | MacBook | Base | `base-3-macbook.json` |
| 8 | MacBook | HN | `hn-3-macbook.json` |
| 9 | MacBook | CROSS | `cross-3-macbook.json` |
| 10 | Rap songs | Base | `base-4-rap.json` |
| 11 | Rap songs | HN | `hn-4-rap.json` |
| 12 | Rap songs | CROSS | `cross-4-rap.json` |
| 13 | React/Svelte | Base | `base-5-react-svelte.json` |
| 14 | React/Svelte | HN | `hn-5-react-svelte.json` |
| 15 | React/Svelte | CROSS | `cross-5-react-svelte.json` |
**Output directory:** `/tmp/last30days-comparison/full/`
## Execution Protocol
### Pre-flight (once)
- [x] Clear model cache: `rm -f ~/.cache/last30days/model_selection.json`
- [x] Run `--diagnose` on current branch, save as `diagnose-baseline.json`
- [x] Verify all API keys active: Reddit (OPENAI_API_KEY), X (Bird cookies or XAI_API_KEY), YouTube (yt-dlp), HN (no key needed), Web (parallel AI or Brave)
- [x] Create output dir: `mkdir -p /tmp/last30days-comparison/full`
- [x] Stash any uncommitted changes: `git stash` (none needed - no uncommitted changes)
### Per-topic loop (repeat 5 times)
For each topic, run all 3 versions back-to-back:
```bash
# CRITICAL: Clean __pycache__ between EVERY git checkout to prevent stale bytecode
cleanup() {
find scripts -name '__pycache__' -exec rm -rf {} + 2>/dev/null
find scripts -name '*.pyc' -delete 2>/dev/null
}
```
- [ ] **Step 1: Base version**
- `cleanup && git checkout 427a4e4`
- `python3 scripts/last30days.py --diagnose 2>/dev/null` (verify sources match baseline)
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/base-{N}-{slug}.json 2>/tmp/last30days-comparison/full/base-{N}-{slug}.log`
- [ ] **Step 2: HN version**
- `cleanup && git checkout main`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/hn-{N}-{slug}.json 2>/tmp/last30days-comparison/full/hn-{N}-{slug}.log`
- [ ] **Step 3: CROSS version**
- `cleanup && git checkout feat/youtube-relevance-cross-source`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/cross-{N}-{slug}.json 2>/tmp/last30days-comparison/full/cross-{N}-{slug}.log`
### Post-run
- [x] Return to feature branch: `git checkout feat/youtube-relevance-cross-source`
- [x] Verify all 15 JSON files exist and are non-empty
- [x] Check for `*_error` fields in any JSON output - flag but include (zero errors)
## Analysis Dimensions
### 1. Source Coverage Table
For each of 15 runs, count items per source:
| Topic | Version | Reddit | X | YouTube | HN | Web | Total |
|-------|---------|--------|---|---------|----|----|-------|
Expected: Base has 0 HN items. HN and CROSS should have identical source counts (same search code). Any differences indicate API non-determinism.
### 2. YouTube Relevance Comparison
**Matched-item analysis:** Match videos by `video_id` across Base/CROSS runs of the same topic. For each matched video:
- Base relevance: always 0.7
- CROSS relevance: dynamic score
- Delta and direction (did dynamic scoring promote or demote this video?)
**Aggregate analysis:** Distribution stats (min/avg/max/stddev) per version per topic.
### 3. Cross-Source Links (CROSS only)
- Count of items with cross_refs per topic
- Quality assessment: are the linked items actually about the same story?
- Which source pairs link most often? (Reddit-HN? YouTube-HN? X-Reddit?)
### 4. Score Distribution and Rankings
- Mean/median score of top 10 items per version per topic
- Rank position changes: do the same items appear in different orders?
- Does HN crowd out Reddit/X items in the top 10?
### 5. "Best Version" Judging Criteria
Define BEFORE analyzing to avoid bias:
| Metric | Weight | How measured |
|--------|--------|-------------|
| Source diversity | 25% | Shannon entropy across source types in top 15 items |
| Score quality | 25% | Mean score of top 10 items |
| Relevance accuracy | 25% | YouTube: do high-relevance videos actually match the query? Manual spot-check of top 3 + bottom 3 per topic |
| Bonus features | 25% | HN value-add (unique info not in other sources) + cross-ref utility (do xrefs add value to the reader?) |
### 6. HN Value-Add Assessment
For each topic where HN returns results:
- How many HN items appear in the overall top 10?
- Do HN items provide information not available from Reddit/X/YouTube?
- Are HN comment insights (top_comments) genuinely useful?
## Deliverables
### A. Raw Results Archive
All 15 JSON files plus logs saved in `/tmp/last30days-comparison/full/`, also copied to `docs/comparison-results/` for persistence.
### B. Comparison Summary Table
A single markdown table showing all 15 runs with key metrics per cell.
### C. Version Verdict
Clear judgment: which version is best overall, and best per topic category (tech, consumer, culture).
### D. CROSS Improvement Plan
Concrete changes to make CROSS the GOAT:
**Based on the previous analysis (see origin doc), likely improvements include:**
1. **Fix cross-source linking** - Switch from char-trigram Jaccard (0.5) to hybrid similarity (max of trigram + token Jaccard) at threshold 0.40. Previous modeling showed this goes from 1 link to 24 links across 5 tests.
2. **Cross-ref rendering** - Change from cryptic `[xref: X1, HN3]` IDs to human-readable `[also on: Reddit, HN]` labels.
3. **HN search broadening** - Investigate why React/Svelte returns 0 HN items.
4. **YouTube relevance** - Already working well, may need minor threshold tuning based on 15-test data.
5. **Score normalization** - Ensure HN items don't systematically crowd out other sources in the merged ranking.
**New improvements to discover from 15-test data:**
- Regressions introduced by CROSS changes
- Edge cases where Base or HN outperforms CROSS
- Topic-specific tuning opportunities
## Acceptance Criteria
- [x] All 15 JSON result files saved and non-empty
- [x] All 15 runs use identical source routing (verified via --diagnose)
- [x] __pycache__ cleaned between every git checkout
- [x] Comparison summary table covers all 15 runs
- [x] YouTube matched-item analysis for at least 3 topics (all 5 done)
- [x] Cross-source link quality spot-check for CROSS runs
- [x] Clear version verdict with supporting data
- [x] CROSS improvement plan with specific code changes
- [x] All results appended to the analysis document for the user to review
## Technical Considerations
- **__pycache__ poisoning**: Python caches bytecode. Switching git commits without clearing `__pycache__` means old code runs even after checkout. MUST clean between every checkout.
- **API rate limiting**: 15 runs hitting Reddit, X, YouTube, HN APIs. The `--quick` mode and natural ~30-90s per run provide spacing, but monitor for 429s in logs.
- **YouTube non-determinism**: yt-dlp search results shift over time. Topic-sequential ordering minimizes this by running all 3 versions of the same topic within ~3-5 minutes.
- **X search session**: Bird CLI depends on browser cookies. If session expires mid-run, X results degrade silently. Check X item counts across runs.
- **`--quick` limitation**: Results reflect Phase 1 only (8-12 items per source). Full pipeline with `--deep` might show different patterns. Document this caveat.
## Sources & References
- Previous analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md)
- Implementation plan: [docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md](docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md)
- Existing comparison harness: [scripts/test-v1-vs-v2.sh](scripts/test-v1-vs-v2.sh)
- Scoring weights: [scripts/lib/score.py](scripts/lib/score.py) - 45% relevance + 25% recency + 30% engagement
docs/plans/2026-02-25-feat-goat-synthesis-output-comparison-plan.md
-471
View File
@@ -1,471 +0,0 @@
---
title: "feat: GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement"
type: feat
status: completed
date: 2026-02-25
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
---
# GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement
## Overview
The synthesis output - the "What I learned" / "I'm now an expert on X" narrative - is the ONLY thing that matters. All data collection, scoring, deduplication, and rendering exist solely to produce the best possible synthesis. This plan runs 15 synthesis comparisons (5 topics x 3 versions), judges which version produces the best final output, and creates concrete changes to make CROSS the GOAT.
## Problem Statement / Motivation
The previous 15-test comparison analyzed the data layer (JSON files, item counts, YouTube scores, cross-ref counts). That analysis showed CROSS wins on data quality metrics. But data quality != output quality. The user's exact words: "that's all that matters... all DATA/methods/etc. should be about making it the best fucking results ever."
The paper.design example shows what great output looks like:
- "Paper Desktop + MCP is the big story" - a specific, grounded finding
- "per @stephenhaney" / "per r/UXDesign" - actual source citations
- "698 likes on X" - real engagement numbers woven into narrative
- "KEY PATTERNS from the research:" - structured patterns, not vague summaries
- "---I'm now an expert on paper.design" - confident, actionable invitation
We need to see what Base, HN, and CROSS actually produce as narratives, side-by-side, for the same topics. Then fix CROSS until its output is undeniably the best.
## Approach: Controlled Synthesis Comparison
### Why not re-run `claude --print "/last30days X"` 15 times?
Three problems:
1. **WebSearch non-determinism** - Claude's WebSearch tool returns different results per run, confounding version comparison
2. **QUERY_TYPE non-determinism** - Claude classifies "best rap songs 2026" as RECOMMENDATIONS one run and GENERAL the next, producing structurally different outputs
3. **Reddit API non-determinism** - The existing JSON data shows Base got 4 Reddit items for Claude Code while CROSS got 10, purely from API timing
### The controlled approach
Reuse the 15 JSON files already captured (same-day, topic-sequential, verified clean). For each:
1. Convert JSON to compact markdown using each version's `render_compact()` (preserving version-specific rendering like CROSS's `[xref:]` tags)
2. Feed the compact markdown to Claude with the synthesis instructions from each version's SKILL.md
3. Hardcode QUERY_TYPE per topic to eliminate classification variance
4. Exclude WebSearch to isolate Python pipeline differences
5. Save all 15 synthesis outputs for side-by-side comparison
This gives us 15 synthesis narratives produced from identical underlying data, with only the version's rendering and synthesis instructions varying. The fairest possible comparison.
## The 5 Topics and Their QUERY_TYPEs
| # | Topic | QUERY_TYPE | Rationale |
|---|-------|------------|-----------|
| 1 | "Claude Code skills and MCP servers" | GENERAL | Broad understanding, not a ranked list |
| 2 | "Seedance AI video generation" | NEWS | Recent product launch, what's happening |
| 3 | "M4 MacBook Pro review" | RECOMMENDATIONS | "review" implies evaluation/ranking |
| 4 | "best rap songs 2026" | RECOMMENDATIONS | "best" triggers ranked list |
| 5 | "React vs Svelte 2026" | GENERAL | Framework debate, not ranked |
## Technical Approach
### Phase 1: Setup and Data Preservation
#### 1a. Copy JSON data to repo (CRITICAL - `/tmp` is ephemeral)
```bash
mkdir -p docs/comparison-results/json
cp /tmp/last30days-comparison/full/*.json docs/comparison-results/json/
```
- [x] Copy all 15 JSON files to `docs/comparison-results/json/`
- [x] Verify all 15 files are present and non-empty
#### 1b. Build JSON-to-compact converter
The render pipeline differs per version. We need each version's `render_compact()` to produce the compact markdown that version would actually show Claude.
**Script: `scripts/generate-synthesis-inputs.py`**
```python
"""
For each of the 15 JSON result files, render compact markdown
using that version's render_compact() function.
The JSON files store raw data. Each version's render.py formats
it differently (CROSS adds [xref:] tags, relevance scores, etc.).
"""
import json
import sys
sys.path.insert(0, 'scripts')
from lib.render import render_compact
from lib.schema import Report
for json_file in sorted(glob('docs/comparison-results/json/*.json')):
data = json.load(open(json_file))
report = Report.from_dict(data)
compact = render_compact(report)
# Save as .md alongside the JSON
md_file = json_file.replace('/json/', '/compact/').replace('.json', '.md')
open(md_file, 'w').write(compact)
```
Problem: `render_compact()` differs across Base/HN/CROSS. Solution: run the converter 3 times, once per git checkout, only for that version's files.
- [x] Write `scripts/generate-synthesis-inputs.py`
- [x] Run on Base checkout for `base-*.json` files -> `docs/comparison-results/compact/base-*.md`
- [x] Run on HN checkout for `hn-*.json` files -> `docs/comparison-results/compact/hn-*.md`
- [x] Run on CROSS checkout for `cross-*.json` files -> `docs/comparison-results/compact/cross-*.md`
- [x] Clean `__pycache__` between every git checkout
#### 1c. Extract synthesis prompts from each version's SKILL.md
Each version's SKILL.md contains the instructions Claude uses to synthesize the compact data into the narrative. These differ across versions (CROSS has cross-ref instructions, HN has HN citation rules, etc.).
- [x] Check out Base (427a4e4), copy SKILL.md synthesis section (Judge Agent + Display phases) to `docs/comparison-results/prompts/base-synthesis-prompt.md`
- [x] Check out HN (main), copy to `docs/comparison-results/prompts/hn-synthesis-prompt.md`
- [x] Check out CROSS (feat/youtube-relevance-cross-source), copy to `docs/comparison-results/prompts/cross-synthesis-prompt.md`
- [x] For each, prepend the hardcoded QUERY_TYPE and topic name
### Phase 2: Generate 15 Synthesis Outputs
#### 2a. Write the synthesis runner
**Script: `scripts/run-synthesis-comparison.py`**
For each of the 15 (topic, version) pairs:
1. Read the compact markdown from `docs/comparison-results/compact/{version}-{n}-{slug}.md`
2. Read the synthesis prompt from `docs/comparison-results/prompts/{version}-synthesis-prompt.md`
3. Call the Anthropic API (Claude Sonnet 4.6 for cost/speed, or Opus 4.6 for max quality - configurable)
4. Template: "You are running /last30days. The user asked about '{topic}'. QUERY_TYPE = {query_type}. Here is the research data:\n\n{compact_markdown}\n\n{synthesis_prompt}"
5. Save response to `docs/comparison-results/synthesis/{version}-{n}-{slug}.md`
```python
"""
Generate synthesis narratives for all 15 test cases.
Uses Anthropic API directly to control the synthesis environment.
"""
import anthropic
import os
from pathlib import Path
client = anthropic.Anthropic()
TOPICS = [
(1, 'claude-code', 'Claude Code skills and MCP servers', 'GENERAL'),
(2, 'seedance', 'Seedance AI video generation', 'NEWS'),
(3, 'macbook', 'M4 MacBook Pro review', 'RECOMMENDATIONS'),
(4, 'rap', 'best rap songs 2026', 'RECOMMENDATIONS'),
(5, 'react-svelte', 'React vs Svelte 2026', 'GENERAL'),
]
VERSIONS = ['base', 'hn', 'cross']
for version in VERSIONS:
prompt_text = Path(f'docs/comparison-results/prompts/{version}-synthesis-prompt.md').read_text()
for num, slug, topic, qtype in TOPICS:
compact = Path(f'docs/comparison-results/compact/{version}-{num}-{slug}.md').read_text()
user_msg = f"""You are the /last30days skill. The user asked: "{topic}"
Parsed intent:
- TOPIC = {topic}
- TARGET_TOOL = unknown
- QUERY_TYPE = {qtype}
Here is the research output from the Python pipeline:
{compact}
Now synthesize this research into your expert narrative following these instructions:
{prompt_text}"""
response = client.messages.create(
model=os.environ.get('SYNTHESIS_MODEL', 'claude-sonnet-4-6-20250514'),
max_tokens=4096,
messages=[{"role": "user", "content": user_msg}]
)
output = response.content[0].text
out_path = Path(f'docs/comparison-results/synthesis/{version}-{num}-{slug}.md')
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(output)
print(f' {version}-{num}-{slug}: {len(output)} chars')
```
- [x] Write `scripts/run-synthesis-comparison.py`
- [x] Run it (15 API calls, ~2-3 min total)
- [x] Verify all 15 synthesis files exist and are non-empty
#### 2b. Quality check synthesis outputs
Spot-check 3 outputs (one per version) for:
- Did Claude actually produce a "What I learned" narrative?
- Are there real citations (per @handle, per r/subreddit)?
- Is the stats block present?
- Is there an invitation at the end?
If any output is broken (e.g., Claude refused or produced meta-commentary instead of synthesis), adjust the prompt and re-run that one.
- [x] Spot-check `cross-1-claude-code.md`, `hn-2-seedance.md`, `base-4-rap.md`
- [x] Fix any prompt issues and re-run failed outputs
### Phase 3: Evaluate - Which Version Produces the Best Output?
#### 3a. Define evaluation rubric (BEFORE reading any outputs)
Score each synthesis 1-5 on these dimensions:
| Dimension | Weight | 1 (Bad) | 3 (OK) | 5 (Great) |
|-----------|--------|---------|--------|-----------|
| **Groundedness** | 30% | Generic statements, no citations | Some citations but mixed with pre-trained knowledge | Every finding backed by specific source (per @handle, per r/sub, per channel) |
| **Specificity** | 25% | Vague ("AI video tools are improving") | Some specifics but also filler | Named entities, exact numbers, product versions ("Seedance 2.0 added lip sync per @aifilmmaker") |
| **Coverage** | 20% | Only mentions 1-2 sources | Mentions most sources but unevenly | Weaves findings from Reddit, X, YouTube, HN naturally into narrative |
| **Actionability** | 15% | "This is interesting" with no next step | Generic suggestions | Specific, research-derived suggestions ("I can show you Seedance 2.0's lip sync workflow") |
| **Format Compliance** | 10% | Missing stats block, no invitation | Partial stats, generic invitation | Perfect stats block with real counts, source-specific invitation |
Total: weighted average, 1.0 to 5.0
#### 3b. LLM-as-judge evaluation (blinded)
**Script: `scripts/evaluate-synthesis.py`**
For each topic (5 total), present all 3 versions to an evaluator LLM with version labels stripped. Ask it to score each on the rubric above.
```python
"""
Blinded LLM evaluation of synthesis outputs.
Strips version labels, presents as Version A/B/C in random order.
"""
import anthropic
import random
RUBRIC = """Score each version 1-5 on these dimensions:
1. GROUNDEDNESS (30%): Does the narrative cite specific sources?
Look for: "per @handle", "per r/subreddit", "per [channel] on YouTube"
1 = generic, no citations. 5 = every finding has a source.
2. SPECIFICITY (25%): Are findings specific or vague?
1 = "AI video is trending". 5 = "Seedance 2.0 added lip sync, 698 likes per @paper"
3. COVERAGE (20%): Does it represent findings from all available sources?
1 = only Reddit mentioned. 5 = Reddit, X, YouTube, HN woven naturally.
4. ACTIONABILITY (15%): Does the invitation give specific next steps based on research?
1 = "let me know if you want more". 5 = "I can walk you through Seedance 2.0's workflow"
5. FORMAT COMPLIANCE (10%): Stats block present with real counts? Citation format correct?
1 = missing stats. 5 = perfect stats block + source counts + top voices.
For each version, output:
- Groundedness: X/5
- Specificity: X/5
- Coverage: X/5
- Actionability: X/5
- Format: X/5
- Weighted Total: X.X/5.0
- One sentence on what makes this version better or worse than the others.
"""
# For each topic, randomly shuffle version order to prevent position bias
for topic in TOPICS:
versions = ['base', 'hn', 'cross']
random.shuffle(versions)
label_map = {v: chr(65+i) for i, v in enumerate(versions)} # A, B, C
# Present to evaluator
prompt = f"Topic: {topic}\n\n"
for v in versions:
text = read_synthesis(v, topic)
prompt += f"=== VERSION {label_map[v]} ===\n{text}\n\n"
prompt += RUBRIC
# Call evaluator (use Opus for best judgment)
response = evaluate(prompt)
# Map labels back to versions
save_evaluation(topic, response, label_map)
```
- [x] Write `scripts/evaluate-synthesis.py`
- [x] Run evaluation (5 API calls, one per topic)
- [x] Collect scores into summary table
#### 3c. Human spot-check
Read 3 synthesis outputs yourself (one per version, same topic) and verify the LLM evaluation makes sense. If the LLM scores don't match your gut, investigate.
- [x] Read all 3 versions for topic 1 (Claude Code) side-by-side
- [x] Read all 3 versions for topic 2 (Seedance) side-by-side
- [x] Verify LLM scores align with human judgment
#### 3d. Compile verdict
| Topic | Base | HN | CROSS | Winner |
|-------|------|-----|-------|--------|
| Claude Code | X.X | X.X | X.X | ? |
| Seedance | X.X | X.X | X.X | ? |
| MacBook | X.X | X.X | X.X | ? |
| Rap songs | X.X | X.X | X.X | ? |
| React/Svelte | X.X | X.X | X.X | ? |
| **Overall** | **X.X** | **X.X** | **X.X** | **?** |
Plus per-dimension breakdown:
- Which version is best at Groundedness?
- Which version is best at Specificity?
- Which version is best at Coverage?
- Which version is best at Actionability?
- Which version is best at Format Compliance?
- [x] Build summary table with scores
- [x] Identify per-dimension winners
- [x] Write verdict paragraph
### Phase 4: Make CROSS the GOAT
Based on the evaluation, identify the specific synthesis weaknesses and fix them. Changes fall into two buckets:
#### Bucket A: Data pipeline changes (score.py, dedupe.py, render.py, youtube_yt.py)
These affect what compact markdown Claude sees. From the previous JSON analysis, known issues:
1. **Cross-source linking nearly broken** (3/178 items linked at 0.5 threshold)
- Fix: hybrid similarity (token + trigram Jaccard) at 0.40 threshold
- File: `scripts/lib/dedupe.py`
- Expected impact: 26 items linked instead of 3
2. **Cross-ref rendering cryptic** (`[xref: HN5, HN4]` is meaningless)
- Fix: Show `[also on: HN, Reddit]` with source names
- File: `scripts/lib/render.py`
- Expected impact: Claude can naturally say "discussed on both Reddit and HN"
3. **YouTube synonym gap** ("hip hop" != "rap" in relevance scoring)
- Fix: SYNONYMS dict in youtube_yt.py
- File: `scripts/lib/youtube_yt.py`
- Expected impact: "Lit Hip Hop Mix 2026" gets 0.67+ instead of 0.33
4. **HN search too narrow** (React/Svelte returns 0 HN items)
- Fix: Split multi-keyword topics into OR queries for HN Algolia
- File: `scripts/lib/hackernews.py` (or wherever HN search lives)
- Expected impact: Framework debate topics get HN coverage
#### Bucket B: Synthesis instruction changes (SKILL.md)
These affect how Claude interprets the data. Specific changes to discover from the 15-output comparison:
5. **Cross-source narrative instruction** - If CROSS data has `[also on: HN, Reddit]` tags but Claude ignores them, add explicit instruction: "When items appear across multiple platforms, lead with that cross-platform signal - it's the strongest evidence of importance."
6. **YouTube transcript utilization** - CROSS provides transcript snippets. If Claude's synthesis doesn't use them, add: "YouTube transcripts contain direct quotes and technical details. Weave 1-2 transcript quotes into your synthesis."
7. **Source weighting for synthesis** - Current: Reddit/X > YouTube > Web. Should HN be weighted higher for tech topics? Should YouTube transcript content elevate YouTube's synthesis weight?
8. **Citation density tuning** - The paper.design example has ~1 citation per paragraph. If any version over-cites (every sentence) or under-cites (no sources), adjust the citation frequency instruction.
9. **Stats block accuracy** - Verify each version's stats block matches actual data. If counts are wrong, the render output may need to include pre-computed stats that Claude can copy directly.
10. **Invitation quality** - The best invitations reference specific things from the research ("I can walk you through Seedance 2.0's lip sync workflow"). If a version produces generic invitations ("let me know if you want to know more"), strengthen the instruction.
Changes 5-10 are discovered from Phase 3 analysis. They may or may not apply.
- [x] Implement Bucket A changes (data pipeline fixes from JSON analysis)
- [x] Analyze Phase 3 results to identify Bucket B changes needed
- [x] Implement Bucket B changes (SKILL.md synthesis instruction improvements)
- [x] Run `bash scripts/sync.sh` to deploy updated skill
### Phase 5: Validate
Re-run 5 synthesis outputs (one per topic, CROSS-after only) using the improved CROSS code, and compare against CROSS-before.
- [x] Generate 5 new compact markdowns from JSON data using improved render.py
- [x] Generate 5 new synthesis outputs using improved SKILL.md
- [x] Score with same rubric (LLM-as-judge, blinded against CROSS-before)
- [x] Verify improvement on every dimension, or iterate
## File Structure
```
docs/comparison-results/
json/ # 15 JSON files (already exist at /tmp, need copying)
base-1-claude-code.json
hn-1-claude-code.json
cross-1-claude-code.json
... (15 total)
compact/ # 15 compact markdown files (rendered per-version)
base-1-claude-code.md
hn-1-claude-code.md
cross-1-claude-code.md
... (15 total)
prompts/ # 3 synthesis prompts (extracted from each SKILL.md)
base-synthesis-prompt.md
hn-synthesis-prompt.md
cross-synthesis-prompt.md
synthesis/ # 15 synthesis narratives (the actual output!)
base-1-claude-code.md
hn-1-claude-code.md
cross-1-claude-code.md
... (15 total)
evaluation/ # 5 evaluation results (one per topic, blinded)
eval-1-claude-code.md
eval-2-seedance.md
eval-3-macbook.md
eval-4-rap.md
eval-5-react-svelte.md
summary.md # Final verdict with all scores and improvement plan
scripts/
generate-synthesis-inputs.py # JSON -> compact converter
run-synthesis-comparison.py # Compact + prompt -> synthesis via Claude API
evaluate-synthesis.py # Blinded LLM evaluation
```
## Acceptance Criteria
- [x] All 15 JSON files preserved in `docs/comparison-results/json/`
- [x] All 15 compact markdowns generated (version-specific rendering preserved)
- [x] All 3 synthesis prompts extracted from version-specific SKILL.md files
- [x] All 15 synthesis narratives generated and saved
- [x] All 15 syntheses scored on 5-dimension rubric (blinded, LLM-as-judge)
- [x] Summary table shows per-topic and overall winner
- [x] At least 3 synthesis outputs human-spot-checked against LLM scores
- [x] CROSS improvement plan includes both data pipeline (Bucket A) and synthesis instruction (Bucket B) changes
- [x] Bucket A changes implemented with tests
- [x] Bucket B changes implemented and deployed via sync.sh
- [x] 5 validation synthesis outputs show improvement over CROSS-before
- [x] Final summary.md documents everything: scores, verdict, changes, validation
## Dependencies & Risks
| Risk | Mitigation |
|------|------------|
| `/tmp` data loss before copying | Phase 1a is the first action - copy immediately |
| Claude API rate limits during synthesis | 15 calls is well under limits; add 2s sleep between calls |
| LLM evaluator bias toward longer outputs | Rubric weights specificity and groundedness, not length |
| QUERY_TYPE affecting output format | Hardcoded per topic in synthesis prompt |
| Reddit item-count variance confounding comparison | Note caveat; flag topics where counts differ >30% |
| render_compact() crash on JSON data | Test converter on 1 file before batch run |
## Technical Considerations
### Why Anthropic API instead of `claude --print`?
1. **Controlled environment** - Same model, same temperature, same max_tokens for all 15
2. **No WebSearch confound** - API calls don't have tool access unless we grant it
3. **Reproducible** - Can re-run with different models or prompts
4. **Faster** - API call takes ~10s vs ~3min for full `claude --print` with tools
### Why reuse JSON data instead of re-running the pipeline?
1. **Eliminates temporal non-determinism** - Same Reddit/X/YouTube data for all comparisons
2. **No rate limiting** - Zero API calls to source platforms
3. **Already validated** - 15 files verified clean, zero errors
4. **Still tests rendering differences** - Each version's render_compact() runs on its checkout
### What about the WebSearch step?
The full skill pipeline includes Claude doing WebSearch after the Python script. We deliberately exclude this because:
1. WebSearch results vary per run (different web results each time)
2. WebSearch is identical across all 3 versions (no version difference to test)
3. Including it would confound the comparison with noise
4. The Python pipeline data is where version differences live
## Sources & References
- Previous JSON analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md) - data-layer comparison with scoring verdict
- Previous test plan: [docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md](docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md) - execution protocol for 15 JSON test runs
- SKILL.md synthesis instructions: `SKILL.md` lines 165-320 (Judge Agent + Display phases)
- Scoring weights: `scripts/lib/score.py` - 45% relevance + 25% recency + 30% engagement
- Render pipeline: `scripts/lib/render.py:57` - `render_compact()` function
- Cross-source linking: `scripts/lib/dedupe.py:160` - `cross_source_link()` function
- YouTube relevance: `scripts/lib/youtube_yt.py` - `_compute_relevance()` token overlap
- Existing test harness: `scripts/test-v1-vs-v2.sh:104` - `claude --print` approach
- YouTube display bug: `docs/plans/2026-02-15-fix-youtube-display-and-search-quality-plan.md`
- Output formatting fixes: `docs/plans/2026-02-06-fix-last30days-v2-formatting-reddit-citations-plan.md`
docs/plans/2026-02-25-feat-polymarket-prediction-market-source-plan.md
-254
View File
@@ -1,254 +0,0 @@
---
title: "feat: Add Polymarket prediction market search as 6th source"
type: feat
status: completed
date: 2026-02-25
---
# feat: Add Polymarket Prediction Market Search
## Overview
Add Polymarket as a 6th research source to last30days. When researching any topic, search Polymarket's public API for relevant prediction markets - surfacing what people are putting real money on alongside what they're saying on Reddit/X/YouTube/HN/Web.
Example: "/last30days Arizona Basketball" would find markets on tournament seeding, Big 12 title odds, and March Madness outcomes. The signal is unique - betting odds reflect conviction, not just opinions.
## Proposed Solution
Hit Polymarket's Gamma API with **smart multi-query search** - not just the raw topic, but expanded related keywords to find markets a human researcher would think of. Show price movement context ("down from 23.7% peak") using the API's built-in price change fields. No API key needed.
Also: hide sources with zero results from the stats box (all sources, not just Polymarket).
## Technical Approach
### API Details
**Endpoint:** `GET https://gamma-api.polymarket.com/public-search?q={topic}&limit={N}`
- No authentication required (free, public, read-only)
- Rate limit: 350 req/10s (very generous)
- Returns `{"events": [...], "pagination": {...}}`
- Events contain nested `markets` arrays
- `outcomePrices` is a JSON-encoded string - must `json.loads()` it
- `volume` and `liquidity` are strings at market level, floats at event level
- **Price movement fields on every market:** `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange` - these are free, no extra API calls
### Intelligence Layer: Smart Query Expansion
A single keyword search is dumb. "Iran" should find markets about Iran strikes, nuclear program, sanctions, oil prices, Khamenei. "Arizona Basketball" should find NCAA tournament odds, Big 12 title, March Madness seeding.
**How it works:**
1. **Extract core subject** using `_extract_core_subject(topic)` (already exists for YouTube/X)
2. **Generate 2-4 expanded queries** from the topic:
- The raw topic itself: `"Arizona Basketball"`
- Broader context: `"Arizona NCAA"`, `"Arizona Big 12"`
- For people/entities: add key associated terms (e.g., "Iran" -> "Iran strikes", "Khamenei", "Iran nuclear")
3. **Run all queries in parallel** against `/public-search` (rate limit is 350/10s, we're using 3-4)
4. **Merge and dedupe** results across queries (same event ID = same market)
5. **Score by relevance to original topic** - markets found by the raw topic query get a relevance boost over expanded-query matches
**Query expansion strategy:**
```python
def _expand_polymarket_queries(topic: str) -> List[str]:
"""Generate 2-4 search queries to cast a wider net."""
core = _extract_core_subject(topic)
queries = [core] # Always include the core topic
# Split multi-word topics into component searches
words = core.split()
if len(words) >= 2:
# Try the first significant word alone (e.g., "Arizona" from "Arizona Basketball")
queries.append(words[0])
# Add the full topic if different from core
if topic.lower() != core.lower():
queries.append(topic)
return list(dict.fromkeys(queries))[:4] # Dedupe, cap at 4
```
This is the same approach as YouTube synonym expansion and X handle resolution - cast a wider net, then score by relevance.
### Price Movement Context
The API gives us price change data for free. Use it to make the output actually useful:
```
Will Arizona win the NCAA Tournament?
Yes: 12% (down 11.7% this month) | No: 88%
$342K vol24h | $2.1M liquidity
```
**Fields available:**
- `oneDayPriceChange` - "up 3% today"
- `oneWeekPriceChange` - "down 5% this week"
- `oneMonthPriceChange` - "down 11.7% this month"
Show the most significant movement (largest absolute change). Only show if change > 1% to avoid noise.
### Key Design Decisions
**Event-level granularity (not market-level).** Each event becomes one `PolymarketItem` showing its title and top 3 markets by volume. A "Trump" query returns 10 events, not 40+ individual markets. This keeps item counts manageable.
**Smart multi-query search with dedup.** Don't rely on one keyword search. Expand the topic into 2-4 related queries, run them all, merge by event ID. This catches markets that use different terminology than the user's query.
**Price movement context by default.** Show the most significant price change (day/week/month) inline with outcome prices. This makes prediction markets actually useful as a research signal - not just "Yes 65%" but "Yes 65%, up 12% this week."
**Date filtering uses `active=true` + `volume1mo > 0`.** Markets can be created months ago but still actively traded. Filter by recent trading activity, not creation date. Set `date` to `updatedAt` for recency scoring. Set `date_confidence` to "high" (API provides exact timestamps).
**Sort priority: below HN, above WebSearch.** Priority 4 (Reddit=0 > X=1 > YouTube=2 > HN=3 > Polymarket=4 > WebSearch=5). Prediction markets are supplementary signal.
**Multi-outcome markets: show top 3 outcomes by price.** Binary markets show "Yes: 65%, No: 35%". Multi-candidate markets truncate to top 3 with "and N more".
**Exclude closed/resolved markets for v1.** Only show `active=true, closed=false`.
**Link to event pages** (not individual market pages) - shows all related markets in context.
### Implementation Plan
#### Phase 1: Hide zero-result sources (separate commit)
- [x] `scripts/lib/render.py` - In `render_source_status()`, skip lines where count is 0
- [x] `SKILL.md` - Remove "(no results this cycle)" instructions, replace with "omit sources with zero results"
- [x] `variants/open/references/research.md` - Same update
- [x] `~/.claude/skills/last30daysCROSS/SKILL.md` - Same update (via sync.sh)
- [x] Test: run a query where HN returns 0, verify it's hidden
#### Phase 2: Polymarket source module
- [x] `scripts/lib/polymarket.py` - New file:
- `DEPTH_CONFIG = {"quick": 5, "default": 10, "deep": 20}` (event count per query)
- `_expand_queries(topic)` - generate 2-4 search queries from topic:
- Raw topic: `"Arizona Basketball"`
- Core subject extracted: `"Arizona"` (via `_extract_core_subject`)
- Component words for multi-word topics: first significant word alone
- Cap at 4 queries, dedupe
- `search_polymarket(topic, from_date, to_date, depth)` - run all expanded queries against `/public-search` in parallel (ThreadPoolExecutor), merge results by event ID, dedupe
- `parse_polymarket_response(response)` - extract events, flatten top markets per event
- `_format_price_movement(market)` - pick most significant price change from `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange`, format as "up/down X% this day/week/month", skip if < 1%
- URL-encode query params with `urllib.parse.urlencode`
- Filter: `active=true, closed=false, volume1mo > 0`
- Parse `outcomePrices` with `json.loads()`, handle malformed/missing gracefully
- Handle mixed types: `float(volume)` with try/except
- Relevance scoring: markets from raw topic query get 1.0 base, expanded queries get 0.7 base, then decay by position
- Return `{"items": [...]}` or `{"items": [], "error": "message"}`
#### Phase 3: Schema + normalization + scoring
- [x] `scripts/lib/schema.py` - Add `PolymarketItem` dataclass:
- `id`, `title` (event title), `question` (top market question), `url` (event URL)
- `outcome_prices: List[Tuple[str, float]]` (parsed, e.g. [("Yes", 0.65), ("No", 0.35)])
- `price_movement: Optional[str]` (formatted, e.g. "down 11.7% this month")
- `volume24hr: float`, `liquidity: float`, `end_date: Optional[str]`
- Standard fields: `date`, `date_confidence`, `engagement`, `relevance`, `why_relevant`, `subs`, `score`, `cross_refs`
- Add `volume: Optional[float]` and `liquidity: Optional[float]` to `Engagement` dataclass
- Add `polymarket: List[PolymarketItem]` and `polymarket_error: Optional[str]` to `Report`
- Update `Report.to_dict()` and `Report.from_dict()` (handle missing key for backward compat)
- [x] `scripts/lib/normalize.py` - Add `normalize_polymarket_items()`:
- Map raw API events to `PolymarketItem` instances
- Set `date` from `updatedAt`, `date_confidence = "high"`
- Build `Engagement(volume=volume24hr, liquidity=liquidity)`
- Update `TypeVar` to include `PolymarketItem`
- [x] `scripts/lib/score.py` - Add scoring:
- `compute_polymarket_engagement_raw()`: `0.60 * log1p(volume24hr) + 0.40 * log1p(liquidity)`
- `score_polymarket_items()`: standard 45/25/30 weights (relevance/recency/engagement)
- Update `sort_items()`: Polymarket priority = 4
#### Phase 4: Dedupe + render + UI
- [x] `scripts/lib/dedupe.py`:
- Add `PolymarketItem` to `AnyItem` union
- Add `dedupe_polymarket()` function
- Update `get_item_text()` for `PolymarketItem` (return `title + " " + question`)
- Update `_get_cross_source_text()` for cross-source linking
- [x] `scripts/lib/render.py`:
- Add `_xref_tag()` prefix: `PM` -> `Polymarket`
- Add Polymarket section to `render_compact()`:
```
### Prediction Markets (Polymarket)
**PM1** (score:72) [$342K vol24h, $2.1M liquidity]
Will Arizona win the NCAA Tournament?
Yes: 12% (down 11.7% this month) | No: 88%
https://polymarket.com/event/arizona-ncaa-tournament
**PM2** (score:65) [$89K vol24h, $450K liquidity]
Will Arizona win the Big 12 Tournament?
Yes: 28% (up 4% this week) | No: 72%
https://polymarket.com/event/arizona-big-12
```
- Add to `render_source_status()` (with zero-result hiding from Phase 1)
- Add to `render_full_report()` and `render_context_snippet()`
- Add to `_assess_data_freshness()`
- [x] `scripts/lib/ui.py`:
- Add `POLYMARKET_MESSAGES` list
- Add `start_polymarket()` / `end_polymarket()` methods
- Update `show_complete()` to accept `polymarket_count`
#### Phase 5: Wire into main pipeline
- [x] `scripts/last30days.py`:
- Import `from lib import polymarket`
- Add `_search_polymarket()` wrapper function
- Add `polymarket_future` to `TIMEOUT_PROFILES` (15s - API is fast)
- In `run_research()`: submit polymarket future to ThreadPoolExecutor, increment max_workers
- Collect results with timeout, add to processing pipeline (normalize -> filter -> score -> sort -> dedupe)
- Feed into `cross_source_link()` call
- Assign to `report.polymarket` / `report.polymarket_error`
- Update `--diagnose` output: `"polymarket": True` (always available)
- Update `--store` persistence loop for Polymarket items
- Update `show_complete()` call with `len(deduped_pm)`
- [x] `scripts/lib/env.py` - Add `is_polymarket_available()` -> always `True`
#### Phase 6: Documentation + deploy
- [x] `SKILL.md` - Update "6 sources", add Polymarket to stats box template, update Security section with `gamma-api.polymarket.com`
- [x] `variants/open/references/research.md` - Same source count + stats updates
- [x] `README.md` - Add Polymarket to feature list, update source count references
- [x] `SPEC.md` - Add `polymarket.py` to architecture list
- [x] Run `bash scripts/sync.sh` to deploy
#### Phase 7: Tests
- [x] `tests/test_polymarket.py` - New file:
- `TestParsePolymarketResponse` - binary markets, multi-outcome, malformed outcomePrices, missing fields
- `TestNormalizePolymarketItems` - schema mapping, date handling, type coercion
- `TestScorePolymarketItems` - engagement formula, zero volume, high volume
- [x] `fixtures/polymarket_sample.json` - Representative API response with edge cases
- [x] `tests/test_cross_source.py` - Add Polymarket-to-Reddit linking test case
## Acceptance Criteria
- [x] `/last30days "Arizona Basketball"` shows relevant Polymarket prediction markets with odds and volume
- [x] `/last30days "best rap songs 2026"` gracefully returns no Polymarket results (and hides the PM line from stats)
- [x] Sources with zero results are hidden from stats box (all sources, not just PM)
- [x] `--diagnose` shows `"polymarket": true`
- [x] Multi-outcome markets show top 3 outcomes
- [x] Polymarket items participate in cross-source linking
- [x] No API key required - works out of the box for all users
- [x] `--quick`, `--deep` flags affect Polymarket result count
- [x] Older cached reports without `polymarket` key load without errors
## Dependencies & Risks
**No blockers.** The Gamma API is free, unauthenticated, and has generous rate limits (350 req/10s). No new dependencies needed - just stdlib `urllib` (already used by HN module).
**Risk: API availability.** Polymarket could change or restrict their API. Mitigation: the source is optional and fails gracefully (returns empty list, hidden from stats).
**Risk: Relevance mismatch.** Many topics won't have prediction markets. The hide-zero-sources change ensures this doesn't pollute the output.
## Sources & References
- Polymarket Gamma API docs: https://docs.polymarket.com/developers/gamma-markets-api/overview
- Polymarket CLI: https://github.com/Polymarket/polymarket-cli
- HN source plan (pattern to follow): `docs/plans/2026-02-24-feat-hacker-news-source-plan.md`
- HN source implementation (most recent similar feature): `scripts/lib/hackernews.py`
- Schema: `scripts/lib/schema.py`
- Main pipeline: `scripts/last30days.py`
docs/plans/2026-02-25-feat-polymarket-quality-ranking-plan.md
-189
View File
@@ -1,189 +0,0 @@
---
title: "feat: Improve Polymarket result ranking with quality signals"
type: feat
status: completed
date: 2026-02-25
---
# feat: Improve Polymarket Result Ranking with Quality Signals
## Overview
When a topic like "OpenAI" returns 163+ Polymarket events, the current implementation ranks results almost entirely by API return position (75% weight on `i`), with only a tiny volume boost (0-15%). This means the scoring doesn't reflect actual market quality - a $1M/month market and a $5K/month market get nearly identical relevance scores if they're adjacent in the API response.
Fix the ranking so the most actively traded, fastest-moving, most contested markets bubble to the top.
## Problem Statement
Current relevance formula in `parse_polymarket_response()` (line 328-330):
```python
rank_score = max(0.3, 1.0 - (i * 0.03)) # 75% weight on position
engagement_boost = min(0.15, math.log1p(volume24hr) / 60)
relevance = min(1.0, rank_score * 0.75 + engagement_boost + 0.1)
```
Issues:
1. **Position dominance**: A market at position 2 with $0 volume scores higher than a market at position 8 with $1M volume
2. **`limit` parameter is a no-op**: The Gamma API always returns exactly 5 events per page regardless of `limit`. Our `DEPTH_CONFIG` values (5, 10, 20) do nothing
3. **Rich quality signals are ignored**: Event-level `volume1mo`, `volume1wk`, `competitive`, `commentCount` fields are available but unused
4. **Price movement is displayed but not scored**: Markets with dramatic price swings get no ranking boost
5. **No text-similarity scoring**: A tangential market that happens to mention "OpenAI" ranks the same as one directly about OpenAI
## API Findings (Verified)
**Pagination**: `?page=N` works as 1-indexed offset. Each page returns exactly 5 events. `hasMore: true` indicates more pages exist. `totalResults` gives total count.
**Event-level quality fields** (confirmed via live API):
| Field | Level | Example | Currently Used |
|-------|-------|---------|----------------|
| `volume24hr` | Event + Market | $13,334 | Market only (for engagement) |
| `volume1wk` | Event + Market | $1,051,626 | No |
| `volume1mo` | Event + Market | $1,133,684 | No |
| `liquidity` | Event + Market | $16,285 | Market only (for filtering) |
| `competitive` | Event + Market | 0.995 | No |
| `commentCount` | Event only | 2 | No |
| `oneDayPriceChange` | Market only | -0.02 | Display only, not scored |
| `oneWeekPriceChange` | Market only | -0.05 | Display only, not scored |
| `oneMonthPriceChange` | Market only | -0.117 | Display only, not scored |
**API naturally sorts well**: Page 1 has active high-volume markets ($1M+ monthly volume), page 3 is all dead historical markets ($0 volume). So the API's own ranking is decent - the problem is our scoring doesn't preserve this quality signal.
## Proposed Solution
### 1. Replace position-based relevance with quality-signal relevance
New formula in `parse_polymarket_response()`:
```python
# Text similarity: does the event title contain the search topic?
core = _extract_core_subject(topic).lower()
title_lower = title.lower()
if core and core in title_lower:
text_score = 1.0
else:
# Token overlap fallback
topic_tokens = set(core.lower().split())
title_tokens = set(title_lower.split())
overlap = len(topic_tokens & title_tokens)
text_score = overlap / max(len(topic_tokens), 1)
# Volume signal: log-scaled monthly volume (most stable signal)
vol_score = min(1.0, math.log1p(event_volume1mo) / 16) # ~$9M = 1.0
# Liquidity signal
liq_score = min(1.0, math.log1p(event_liquidity) / 14) # ~$1.2M = 1.0
# Price movement: largest absolute change, capped
max_change = max(
abs(oneDayPriceChange or 0) * 3, # Daily weighted 3x
abs(oneWeekPriceChange or 0) * 2, # Weekly weighted 2x
abs(oneMonthPriceChange or 0) * 1, # Monthly weighted 1x
)
movement_score = min(1.0, max_change * 5) # 20% change = 1.0
# Competitive bonus: markets near 50/50 are more interesting
competitive_score = event_competitive or 0
# Final relevance
relevance = (
0.30 * text_score +
0.30 * vol_score +
0.15 * liq_score +
0.15 * movement_score +
0.10 * competitive_score
)
```
### 2. Fix DEPTH_CONFIG to use pagination
```python
# Pages to fetch per query (API returns 5 events per page)
DEPTH_CONFIG = {
"quick": 1, # 5 events/query, ~5-10 unique after dedup
"default": 2, # 10 events/query, ~10-15 unique after dedup
"deep": 3, # 15 events/query, ~15-25 unique after dedup
}
```
### 3. Use event-level volume fields
Extract `volume1mo`, `volume1wk`, `liquidity`, and `competitive` from the event object (not just the top market). These are more stable signals than market-level `volume24hr`.
### 4. Cap results after re-ranking
After pagination, merge, dedup, and re-ranking, cap at a reasonable number before sending to the scoring pipeline:
```python
RESULT_CAP = {
"quick": 5,
"default": 10,
"deep": 20,
}
```
## Technical Approach
### Implementation Plan
#### Phase 1: Fix pagination and DEPTH_CONFIG
- [x] `scripts/lib/polymarket.py` - Change `DEPTH_CONFIG` to page counts: `{"quick": 1, "default": 2, "deep": 3}`
- [x] `scripts/lib/polymarket.py` - Add `RESULT_CAP` dict: `{"quick": 5, "default": 10, "deep": 20}`
- [x] `scripts/lib/polymarket.py` - Update `_search_single_query()` to accept a `page` parameter
- [x] `scripts/lib/polymarket.py` - Update `search_polymarket()` to fetch multiple pages per query in parallel (fire all `(query, page)` combinations into ThreadPoolExecutor at once)
- [x] `scripts/lib/polymarket.py` - Apply `RESULT_CAP` after merge + dedup, before returning events
- [x] `tests/test_polymarket.py` - Update `TestDepthConfig` tests for new page-count values
#### Phase 2: Extract event-level quality signals
- [x] `scripts/lib/polymarket.py` - In `parse_polymarket_response()`, extract event-level fields: `volume1mo`, `volume1wk`, `liquidity`, `competitive`, `commentCount`
- [x] `scripts/lib/polymarket.py` - Pass `topic` to `parse_polymarket_response()` (already has the parameter, just need to use it)
- [x] `fixtures/polymarket_sample.json` - Add event-level fields: `volume1mo`, `volume1wk`, `competitive`, `commentCount`, `volume24hr`, `liquidity`
#### Phase 3: Replace relevance formula
- [x] `scripts/lib/polymarket.py` - Replace position-based relevance formula with quality-signal formula (text similarity + volume + liquidity + price movement + competitive)
- [x] `scripts/lib/polymarket.py` - Add `_compute_text_similarity(topic, title)` helper
- [x] `tests/test_polymarket.py` - Add `TestTextSimilarity` test class
- [x] `tests/test_polymarket.py` - Add `TestQualityRanking` test: given events with varying volume/liquidity/text-match, verify high-volume title-matching events rank above low-volume tangential ones
#### Phase 4: Update engagement scoring
- [x] `scripts/lib/schema.py` - No changes needed (Engagement already has `volume` and `liquidity`)
- [x] `scripts/lib/polymarket.py` - Use event-level `volume1mo` instead of market-level `volume24hr` for the `volume24hr` field passed to normalization (or add a new field)
- [x] `scripts/lib/normalize.py` - Update `normalize_polymarket_items()` to use `volume1mo` for engagement volume if available, fallback to `volume24hr`
#### Phase 5: Tests and verification
- [x] Run full test suite
- [ ] Manual test: `/last30days "OpenAI" --emit=compact` - verify top markets are the most actively traded
- [ ] Manual test: `/last30days "Anthropic" --emit=compact` - verify quality ranking
- [ ] Manual test: `/last30days "best rap songs 2026" --emit=compact` - verify graceful zero results
- [x] Run `bash scripts/sync.sh` to deploy
## Acceptance Criteria
- [ ] "OpenAI" search surfaces IPO market cap, product announcements, and GPT benchmark markets (high volume) before niche/dead markets
- [x] Markets with $0 monthly volume are filtered out (already handled by liquidity filter, but verify)
- [x] `DEPTH_CONFIG` actually affects result count (quick=~5, default=~10, deep=~20)
- [x] Price movement is factored into ranking (markets with large swings rank higher)
- [x] Text-matching markets rank above tangential keyword matches
- [x] All existing tests pass (71 polymarket + full suite: 218 passed, 5 pre-existing failures)
- [ ] No performance regression - pagination adds latency but stays within timeout budgets
## Dependencies & Risks
**No blockers.** This is a scoring/ranking improvement within the existing Polymarket module. No new API keys, no new dependencies.
**Risk: Over-tuning the formula.** The weights (0.30/0.30/0.15/0.15/0.10) are educated guesses. May need iteration after testing with real queries. Mitigation: the formula is in one place (`parse_polymarket_response`) and easy to adjust.
**Risk: Pagination latency.** Deep mode with 3 pages x 4 queries = 12 API calls. All run in parallel via ThreadPoolExecutor. Gamma API is fast (~200-500ms per call), so worst case ~1-2s total. Well within the 45s deep timeout.
## Sources & References
- Polymarket Gamma API: `GET https://gamma-api.polymarket.com/public-search?q={topic}&page={N}`
- Current implementation: `scripts/lib/polymarket.py`
- Scoring pipeline: `scripts/lib/score.py`
- Original Polymarket plan: `docs/plans/2026-02-25-feat-polymarket-prediction-market-source-plan.md`
docs/plans/2026-02-25-feat-x-handle-resolution-plan.md
-291
View File
@@ -1,291 +0,0 @@
---
title: "feat: Resolve X handles for topic entities"
type: feat
status: completed
date: 2026-02-25
---
# feat: Resolve X handles for topic entities
## Overview
When a user searches for a person, brand, or creator (e.g., "Dor Brothers", "Jason Calacanis"), the skill should automatically resolve their X handle and search their posts directly. Currently, Phase 1 only finds posts that *mention* the topic keywords, and Phase 2 only drills into handles that appeared in Phase 1 results. This misses the entity's own posts entirely when they don't literally include the topic string.
## Problem Statement
**The Dor Brothers example:** Searching "Dor Brothers" found 30 X posts with 149 likes - all posts *about* them. But the Dor Brothers' own account posts constantly about their AI films, tools, and collabs. Those posts wouldn't appear in Phase 1 because the Dor Brothers don't write "Dor Brothers" in every tweet. Phase 2 can't help because entity_extract only finds handles already @mentioned in Phase 1 results.
**The Jason Calacanis example:** His handle is @jason - completely unguessable from his name. No amount of keyword searching or @mention extraction would discover it. You need an actual lookup.
**What's missing:** A handle resolution step that maps `"topic name" -> @handle`, then searches that handle's recent posts directly (without requiring topic keywords in the tweet text).
## User Base Constraints
- **80% Claude Code** - agent is Claude, has WebSearch tool
- **20% OpenClaw** - agent has web search capability
- **Most users have:** OPENAI_API_KEY
- **Some users have:** XAI_API_KEY, Bird (browser cookies)
- **Few users have:** BRAVE_API_KEY, PARALLEL_API_KEY, OPENROUTER_API_KEY
- **Everyone has:** Claude (it's the runtime)
**Critical constraint:** The Python script's web search backends (`brave_search.py`, `parallel_search.py`, `openrouter_search.py`) ALL filter out x.com URLs via `EXCLUDED_DOMAINS`. Building handle resolution inside the Python script would require bypassing this filter, and most users don't have the API keys for those backends anyway.
## Proposed Solution
Move handle resolution to the **SKILL.md agent layer**. The agent (Claude or OpenClaw) already has `WebSearch` as an allowed tool. It does a single WebSearch before running the Python script, extracts the handle, and passes it as a CLI argument.
### Architecture
```
SKILL.md Agent Flow (revised):
1. Parse intent (existing)
2. NEW: If topic looks like a person/brand, WebSearch("{topic} X twitter handle")
3. NEW: Extract @handle from results (agent intelligence, not regex)
4. Run script: python3 last30days.py "Dor Brothers" --x-handle=TheDorBrothers --emit=compact
5. Script uses --x-handle in Phase 2: from:TheDorBrothers (no topic filter)
6. WebSearch supplementary (existing Step 2)
7. Synthesize (existing)
```
### Why agent-level, not Python-level
| Approach | Works for | Problem |
|----------|-----------|---------|
| Brave API in Python | Users with BRAVE_API_KEY (~5%) | Almost nobody has the key |
| OpenAI Responses API in Python | Users with OPENAI_API_KEY (~90%) | Costs money, adds complexity, x.com URLs may be filtered |
| Agent WebSearch (SKILL.md) | **100% of users** | Adds ~5-8 sec before script |
| xAI API in Python | Users with XAI_API_KEY (~30%) | Not universal |
The agent approach wins because:
1. **100% availability** - Claude/OpenClaw always has WebSearch
2. **Zero API key requirements** - WebSearch is built into the agent runtime
3. **Smarter parsing** - Claude is better at "find the X handle for Dor Brothers" than any regex
4. **Simpler implementation** - SKILL.md instruction + `--x-handle` CLI arg, no new Python module
5. **No EXCLUDED_DOMAINS problem** - agent WebSearch is independent of the Python web search backends
The ~5-8 second latency is negligible against the 2-3 minute script runtime.
### SKILL.md Changes (Claude Code variant)
Add between intent parsing and Step 1:
```markdown
## Step 0.5: Resolve X Handle (if topic is a person/brand)
If TOPIC looks like a person, creator, brand, or specific account (1-3 words, proper noun),
do ONE WebSearch to find their X handle:
WebSearch("{TOPIC} X twitter handle")
From the results, extract the X/Twitter handle. Look for:
- Profile URLs: x.com/{handle} or twitter.com/{handle}
- Mentions like "@handle" in bios, articles, or social profiles
- "Follow @handle on X" patterns
If you find a clear, unambiguous handle, pass it to the script:
--x-handle={handle}
If ambiguous or not found, omit the flag. The script works fine without it.
Skip this step if:
- TOPIC is clearly not an entity (e.g., "best rap songs 2026", "how to use Docker")
- TOPIC already contains @ (e.g., "@elonmusk")
- Using --quick depth
```
### OpenClaw Variant Changes
Same instruction added to `variants/open/references/research.md` (or inline in the open SKILL.md routing). OpenClaw agents also have WebSearch available.
### Python Script Changes
**`last30days.py` - Add `--x-handle` argument:**
```python
parser.add_argument('--x-handle', type=str, default=None,
help='Resolved X handle for topic entity (without @)')
```
Pass `resolved_handle` to `_run_supplemental()`.
**`_run_supplemental()` - Accept and use resolved handle:**
```python
def _run_supplemental(
topic, reddit_items, x_items, from_date, to_date,
depth, x_source, progress=None, skip_reddit=False,
resolved_handle=None, # NEW
):
# Extract entities from Phase 1 (existing)
entities = entity_extract.extract_entities(...)
# Add resolved handle if not already in entity list
if resolved_handle and resolved_handle.lower() not in {h.lower() for h in entities["x_handles"]}:
# Search resolved handle separately - unfiltered (no topic keywords)
# This is the key difference from entity-extracted handles
resolved_future = executor.submit(
bird_x.search_handles,
[resolved_handle],
None, # topic=None means unfiltered search
from_date,
count_per=10,
)
```
**`bird_x.search_handles()` - Optional topic parameter:**
```python
def search_handles(handles, topic, from_date, count_per=5):
# topic is now Optional[str]
for handle in handles:
handle = handle.lstrip("@")
if topic:
core_topic = _extract_core_subject(topic)
query = f"from:{handle} {core_topic} since:{from_date}"
else:
# Unfiltered: get all recent posts from this handle
query = f"from:{handle} since:{from_date}"
```
### Why no topic filter for resolved handles
This is the key insight. When you resolve that @DorBrothers IS the Dor Brothers, you want ALL their recent posts - not just ones that literally contain "Dor Brothers." Their post about the Logan Paul collab says "our new AI film with @LoganPaul" - no mention of "Dor Brothers" anywhere. With topic filtering, you'd miss it. Without it, you get their full recent activity, which is exactly what the user wants.
## Technical Considerations
### Handle resolution requires Bird or xAI for Phase 2
The `--x-handle` is only useful if the script can search `from:{handle}`. Currently:
- **Bird:** supports `from:handle` via Twitter GraphQL (free)
- **xAI:** does NOT support `from:handle` (Grok semantic search only)
If the user only has xAI (no Bird), the resolved handle can't be searched in Phase 2. Options:
1. Skip Phase 2 handle search when `x_source == "xai"` (current behavior for entity handles too)
2. Future: Add handle search to xai_x.py using `allowed_x_handles` filter
For v1, accept this limitation. Bird is free and most X-enabled users have it.
### Relevance scoring for unfiltered handle posts
Bird-parsed items default to `relevance: 0.7`. Unfiltered resolved-handle posts have no topic-keyword signal. Set `relevance: 0.5` for these so engagement and recency drive ranking, preventing off-topic viral posts from the entity from outranking genuinely relevant Phase 1 results.
### Stats block display
When `--x-handle` is used and produces results, show it in the stats:
```
├─ 🔵 X: 38 posts │ 782+ likes │ 36+ reposts │ via @TheDorBrothers + keyword search
```
Add `resolved_x_handle` field to `Report` schema for this.
### Skip conditions for the agent
The SKILL.md instruction tells the agent to skip handle resolution when:
- Topic is clearly not an entity (multi-word generic phrases)
- Topic already contains @ (user provided the handle)
- Using `--quick` depth
- Agent judges it would be wasted effort
The agent's judgment here is a feature, not a bug. Claude is good at deciding "Dor Brothers = probably has an X account" vs "best rap songs 2026 = definitely not an entity."
## Acceptance Criteria
- [x] SKILL.md updated with Step 0.5 handle resolution instructions
- [x] OpenClaw variant updated with same instructions
- [x] `last30days.py` accepts `--x-handle` argument
- [x] `_run_supplemental()` accepts `resolved_handle` parameter
- [x] `bird_x.search_handles()` accepts `topic=None` for unfiltered search
- [x] Resolved handle searched with `from:{handle}` (no topic filter) when Bird is available
- [x] `Report` schema includes `resolved_x_handle` field
- [x] Stats block shows resolved handle when used
- [x] Graceful when `--x-handle` is provided but Bird is not available (skips silently)
- [ ] "Dor Brothers" search resolves handle and finds their direct posts (requires live test)
- [ ] "Jason Calacanis" search resolves @jason (requires live test)
- [ ] "best rap songs 2026" does NOT trigger handle resolution (agent judgment, not code)
## Test Plan
### Manual test cases
| # | Query | Expected | What it tests |
|---|-------|----------|---------------|
| 1 | "Dor Brothers" | Agent resolves handle, passes --x-handle, script finds their posts | Full happy path |
| 2 | "Jason Calacanis" | Agent resolves @jason (non-obvious handle) | Handle != name |
| 3 | "best rap songs 2026" | Agent skips handle resolution entirely | Non-entity detection |
| 4 | "OpenAI" | Agent may or may not resolve (judgment call) | Generic entity edge case |
| 5 | "Linus Ekenstam" | Agent resolves @LinusEkenstam | Person with matching handle |
| 6 | "Dor Brothers" with `--quick` | No handle resolution | Skip condition |
| 7 | "Dor Brothers" with xAI only (no Bird) | Handle resolved but Phase 2 skips it | Graceful degradation |
### Automated tests (`tests/`)
```python
# test_bird_x.py - new tests
def test_search_handles_unfiltered_mode():
"""bird_x.search_handles(topic=None) omits topic keywords from query."""
def test_search_handles_with_topic():
"""bird_x.search_handles(topic="AI films") includes topic in query (existing behavior)."""
# test_last30days.py - integration
def test_x_handle_arg_parsed():
"""--x-handle=TheDorBrothers is parsed and passed to _run_supplemental."""
def test_resolved_handle_dedup_with_entity_extract():
"""Resolved handle already in entity list is not double-searched."""
def test_resolved_handle_skipped_when_xai_only():
"""When x_source='xai', resolved handle is not searched (no from: support)."""
def test_resolved_handle_relevance_set_lower():
"""Items from resolved handle search get relevance 0.5, not 0.7."""
```
### E2E validation
```bash
# Run with explicit handle to test Python-side changes
python3 scripts/last30days.py "Dor Brothers" --x-handle=TheDorBrothers --emit=compact 2>&1 | grep -E "\[Phase|handle"
# Expected:
# [Phase 2] Drilling into @TheDorBrothers (resolved) + @handle1, @handle2 (extracted)
# [Phase 2] +0 Reddit, +8 X (5 from resolved handle)
```
```bash
# Full agent test (runs SKILL.md flow including handle resolution WebSearch)
claude --print "/last30days Dor Brothers" 2>&1 | grep -i "handle\|x-handle\|resolved"
```
## Files Modified
| File | Change |
|---|---|
| `SKILL.md` | Add Step 0.5: handle resolution via WebSearch |
| `variants/open/references/research.md` | Same handle resolution instructions for OpenClaw |
| `scripts/last30days.py` | Add `--x-handle` CLI arg, pass to `_run_supplemental()` |
| `scripts/lib/bird_x.py` | `search_handles()` gets `topic: Optional[str]` param |
| `scripts/lib/schema.py` | Add `resolved_x_handle: Optional[str]` to `Report` |
| `scripts/lib/render.py` | Show resolved handle in stats block |
| `tests/test_bird_x.py` | Tests for unfiltered search_handles mode |
| `tests/test_last30days.py` | Tests for --x-handle arg handling |
**NOT modified:** No new `handle_resolve.py` module. Resolution is agent intelligence, not Python code.
## Dependencies & Risks
- **Bird availability:** Resolved handle can only be searched when Bird is the X source. xAI doesn't support `from:handle` queries. ~70% of X-enabled users have Bird (free, browser cookies).
- **Agent judgment:** The agent decides whether a topic is an entity worth resolving. Claude is good at this, but it's non-deterministic. Some edge cases will be missed. This is acceptable - the feature is additive (no regression when it doesn't fire).
- **WebSearch latency:** Adds ~5-8 seconds before the script starts. Negligible against 2-3 minute script runtime.
- **Handle accuracy:** The agent could resolve to the wrong handle. Mitigated by the agent's ability to evaluate results (unlike a regex, Claude can tell if a result actually belongs to the queried entity).
## Sources
- Existing entity extraction: `scripts/lib/entity_extract.py`
- Phase 2 supplemental search: `scripts/last30days.py:387-513`
- Bird search handles: `scripts/lib/bird_x.py:273-346`
- SKILL.md agent flow: `SKILL.md:82-120`
- OpenClaw variant: `variants/open/SKILL.md`
- Smart supplemental search plan: `docs/plans/2026-02-07-feat-smart-supplemental-search-plan.md`
docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md
-226
View File
@@ -1,226 +0,0 @@
---
title: "feat: YouTube relevance scoring and cross-source linking"
type: feat
status: active
date: 2026-02-25
---
# feat: YouTube Relevance Scoring and Cross-Source Linking
## Overview
Two output quality improvements for the last30days monthiversary:
1. **YouTube relevance scoring** - Replace the hardcoded `relevance: 0.7` with real token-overlap scoring so relevant niche videos beat viral off-topic ones.
2. **Cross-source linking** - When the same story appears on Reddit + HN + X, annotate items with `[xref: R3, HN5]` so Claude can synthesize cross-platform discussion.
## Problem Statement / Motivation
**YouTube scoring is broken.** Every YouTube video starts with `relevance: 0.7` (70/100 subscore). Since the scoring formula is `45% relevance + 25% recency + 30% engagement`, all YouTube items share the same 31.5pt relevance floor. Ranking is purely engagement-driven - a viral off-topic video beats a niche relevant one.
**Cross-source coverage is invisible.** The same story often appears across Reddit, HN, and X (e.g., a product launch). Currently each source is deduped independently, but there's no signal telling Claude "these items are about the same thing." Claude has to manually notice the overlap, and often doesn't.
## Technical Approach
### Feature 1: YouTube Relevance Scoring
**Where:** `youtube_yt.py` (compute), `normalize.py` (pass through - already works)
**Algorithm:** Token ratio overlap between `core_topic` and video title.
```python
# youtube_yt.py - new function
STOPWORDS = {'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can'}
def _compute_relevance(query: str, title: str) -> float:
"""Compute relevance as ratio of query tokens found in title.
Uses ratio overlap (intersection / query_length) so short queries
score higher when fully represented in the title. Floors at 0.1.
"""
q_tokens = {w for w in query.lower().split() if w not in STOPWORDS and len(w) > 1}
t_tokens = {w for w in re.sub(r'[^\w\s]', ' ', title.lower()).split()
if w not in STOPWORDS and len(w) > 1}
if not q_tokens:
return 0.5 # Neutral fallback
overlap = len(q_tokens & t_tokens)
ratio = overlap / len(q_tokens)
return max(0.1, min(1.0, ratio))
```
**Key decisions:**
- **Use `core_topic`** (already stripped of noise words by `_extract_core_subject()`) - not the raw verbose query
- **Ratio overlap** (intersection/query_length), not strict Jaccard - so "Claude Code" (2 tokens) vs "Claude Code Tutorial" (3 tokens) = 2/2 = 1.0, not 2/3 = 0.67
- **Stopword removal** on both sides - prevents "How To Use Claude Code For Beginners" from diluting the match
- **Floor at 0.1** - even zero-match videos get `rel_score=10` not 0, consistent with other sources' defaults
- **No LLM call** - this is pure string matching, zero latency/cost
**Files to modify:**
#### `scripts/lib/youtube_yt.py`
- [x] Add `STOPWORDS` set and `_compute_relevance(query, title)` function
- [x] In `search_youtube()`, replace `"relevance": 0.7` (line 186) with `"relevance": _compute_relevance(core_topic, video.get("title", ""))`
- [x] Update `"why_relevant"` to be more descriptive: `f"YouTube: {title_excerpt}"`
#### `scripts/lib/normalize.py`
- No changes needed - already passes through `item.get("relevance", 0.7)` at line 196
#### `scripts/lib/score.py`
- No changes needed - `score_youtube_items()` already reads `item.relevance` correctly
### Feature 2: Cross-Source Linking
**Where:** `dedupe.py` (new function), `schema.py` (new field), `render.py` (display), `last30days.py` (call site)
**Algorithm:** Reuse existing Jaccard char-trigram similarity from `dedupe.py`. Compare items across sources at threshold 0.5 (lower than within-source 0.7 since titles differ more across platforms). Bidirectional - both items get the cross-ref.
**Key decisions:**
- **Link, don't merge** - keep items separate with `cross_refs: ["R3", "HN5"]`. Let Claude handle the narrative.
- **Bidirectional** - if R3 links to HN5, HN5 also links to R3. Claude reads source-by-source and needs to see the link from either direction.
- **Threshold 0.5** for cross-source (vs 0.7 within-source). Lower because the same story has different titles across platforms. Still high enough to avoid false positives.
- **Truncate X text to 100 chars** for comparison - full tweets dilute Jaccard against short Reddit/HN titles
- **Just IDs in cross_refs** - no similarity scores (noise for Claude's synthesis)
- **Include WebSearchItem** - add to `get_item_text()` using `item.title`
- **Don't persist to SQLite** - cross_refs are cheap to recompute, not worth a schema migration
**Performance:** With 45 items total (typical) and 10 source pairs, ~1000 char-trigram comparisons. Sub-millisecond. Even --deep mode (~150 items) is <12K comparisons - trivial.
**Files to modify:**
#### `scripts/lib/schema.py`
- [x] Add `cross_refs: List[str] = field(default_factory=list)` to all 5 item types (RedditItem, XItem, YouTubeItem, HackerNewsItem, WebSearchItem)
- [x] Update `to_dict()` on each item type to include `cross_refs` (only when non-empty)
- [x] Update `Report.from_dict()` to deserialize `cross_refs` for each item type
#### `scripts/lib/dedupe.py`
- [x] Update `get_item_text()` type hints to include `WebSearchItem` and handle it (use `item.title`)
- [x] Add `get_cross_source_text()` function - same as `get_item_text()` but truncates X text to 100 chars
- [x] Add `cross_source_link()` function:
```python
def cross_source_link(
*source_lists: List[Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
schema.HackerNewsItem, schema.WebSearchItem]],
threshold: float = 0.5,
) -> None:
"""Annotate items with cross-source references.
Compares items across different source types. When similarity exceeds
threshold, adds bidirectional cross_refs with the related item's ID.
Modifies items in-place.
"""
all_items = []
for source_list in source_lists:
all_items.extend(source_list)
if len(all_items) <= 1:
return
# Pre-compute trigrams using cross-source text extraction
ngrams = [get_ngrams(get_cross_source_text(item)) for item in all_items]
for i in range(len(all_items)):
for j in range(i + 1, len(all_items)):
# Skip same-source comparisons (already handled by per-source dedupe)
if type(all_items[i]) == type(all_items[j]):
continue
similarity = jaccard_similarity(ngrams[i], ngrams[j])
if similarity >= threshold:
# Bidirectional cross-reference
if all_items[j].id not in all_items[i].cross_refs:
all_items[i].cross_refs.append(all_items[j].id)
if all_items[i].id not in all_items[j].cross_refs:
all_items[j].cross_refs.append(all_items[i].id)
```
#### `scripts/last30days.py`
- [x] After the dedupe step (after line ~1107), call `dedupe.cross_source_link()`:
```python
# Cross-source linking
dedupe.cross_source_link(
deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_web,
)
```
#### `scripts/lib/render.py`
- [x] In `render_compact()`, append `[xref: ...]` to items that have cross_refs:
```python
# After the existing item line
if hasattr(item, 'cross_refs') and item.cross_refs:
xref_str = ', '.join(item.cross_refs)
line += f" [xref: {xref_str}]"
```
#### `SKILL.md`
- [x] Add a note in the synthesis instructions about cross-refs:
"Items tagged `[xref: ...]` reference the same story on another platform. Use these to triangulate: 'This was widely discussed - Reddit thread (R3) with 142 comments, HN discussion (HN5) with 89 points, and several X posts (X12).'"
### Tests
#### `tests/test_youtube_relevance.py` (new)
- [x] Test exact match: query "Claude Code" vs title "Claude Code" = 1.0
- [x] Test partial match: query "Claude Code" vs title "Claude Code Tutorial" = 1.0 (ratio)
- [x] Test low match: query "Claude Code" vs title "Python Web Scraping" = 0.1 (floor)
- [x] Test empty query: returns 0.5
- [x] Test empty title: returns 0.1
- [x] Test stopword handling: query "how to use Claude" vs title "Using Claude" = high match
- [x] Test integration: search_youtube returns varied relevance scores (mock yt-dlp)
#### `tests/test_cross_source.py` (new)
- [x] Test no cross-refs: unrelated items across sources
- [x] Test bidirectional: matching Reddit + HN items both get cross_refs
- [x] Test multi-source: same story on 3+ sources
- [x] Test same-source skip: items from same source type are not cross-linked
- [x] Test X text truncation: long tweet vs short Reddit title
- [x] Test empty lists: no crash on empty source lists
- [x] Test schema round-trip: cross_refs survive to_dict() / from_dict()
## Acceptance Criteria
- [x] YouTube items have varied relevance scores (not all 0.7)
- [x] A video with title matching the query scores higher than one without
- [x] Cross-source items about the same story have `cross_refs` pointing to each other
- [x] Cross-refs are bidirectional
- [x] Compact output shows `[xref: ...]` tags on linked items
- [x] SKILL.md tells Claude how to use cross-refs in synthesis
- [x] All existing tests still pass
- [x] New tests cover YouTube relevance edge cases
- [x] New tests cover cross-source linking edge cases
- [x] Run sync.sh to deploy after changes
## Dependencies & Risks
**Low risk:**
- Both features add fields/functions without changing existing logic
- YouTube relevance is a drop-in replacement for a hardcoded value
- Cross-source linking only adds metadata (cross_refs) without modifying item order or scores
- All existing tests should pass unchanged
**Edge cases:**
- CJK/non-English titles: token splitting works but relevance may be less accurate. Acceptable for v1.
- Very short queries (1 token): ratio overlap = 0 or 1. The 0.1 floor handles the 0 case.
- Clickbait YouTube titles with no query overlap: these correctly get low relevance now (improvement over blindly getting 0.7)
## Sources & References
- YouTube relevance: Currently hardcoded at `youtube_yt.py:186`
- Dedupe infrastructure: `dedupe.py` - Jaccard similarity on char trigrams
- Score weights: `score.py:8-10` - 45% relevance + 25% recency + 30% engagement
- Existing HN plan as template: `docs/plans/2026-02-24-fix-hn-ordering-and-emoji-plan.md`
docs/plans/2026-02-26-feat-polymarket-smarter-synthesis-plan.md
-185
View File
@@ -1,185 +0,0 @@
---
title: "feat: Smarter Polymarket synthesis - surface the most interesting markets"
type: feat
status: completed
date: 2026-02-26
---
# feat: Smarter Polymarket Synthesis
## Overview
Polymarket data is one of the most powerful signals when relevant - real money on outcomes cuts through opinion. But the current system buries the most interesting markets and gives the LLM zero guidance on how to synthesize prediction market data.
For "Arizona Basketball", the skill found 2 markets but only highlighted "Big 12 title: 68%" in the stats and synthesis. The user cares MORE about:
- NCAA Tournament championship odds (12%, up 3%)
- #1 seed odds (85%)
- Next game: Arizona vs Kansas (71% to win)
For "Iran War", the skill found 9 markets with $559M volume but only highlighted "strikes by Feb 28: 10% (down from 65%)". The user wanted regime change / Khamenei odds - the "bigger picture" structural question.
The problem has two layers: (1) the Python scoring penalizes multi-outcome markets where the topic is an outcome, and (2) the SKILL.md gives the LLM zero instructions for interpreting or highlighting prediction market data.
## Problem Statement
### Layer 1: Scoring penalizes the most interesting markets
`_compute_text_similarity()` (polymarket.py:236) only compares the search topic against the **event title**. It never checks outcome names.
Example: Market titled "Who will be the #1 overall seed in the 2026 NCAA Tournament?" with outcomes ["Arizona", "Duke", "Houston", "Auburn"]:
- Topic: "Arizona Basketball"
- text_score: **0.0** (neither "arizona" nor "basketball" in event title)
- 30% relevance penalty from this alone
This means the most contextually interesting markets (seeding, championship, matchup odds) get pushed below less interesting but title-matching markets (Big 12 regular season).
### Layer 2: SKILL.md has zero Polymarket synthesis guidance
The SKILL.md Judge Agent section tells the LLM how to weight Reddit (higher), YouTube (high), WebSearch (lower), but says **nothing** about:
- How to interpret prediction market probabilities
- Which markets are "most interesting" (championship > regular season)
- When to lead with prediction market odds vs other sources
- How to connect a specific outcome in a multi-outcome market to the user's topic
- How to use odds as a signal alongside social media sentiment
### Layer 3: Stats box loses information
The Polymarket stats line only has room for 1-2 market highlights. When there are 5+ relevant markets, the user misses the most interesting ones.
## Proposed Solution
### 1. Outcome-aware text similarity scoring
Update `_compute_text_similarity()` to check if the topic appears in any outcome name, with **bidirectional** substring matching. The check must work in both directions since the topic ("Arizona Basketball") is longer than the outcome name ("Arizona").
Collect outcome names from ALL active markets in the event (not just top market), since Gamma API can structure multi-outcome events as separate binary sub-markets.
Only match outcomes with probability > 1% to avoid noise from near-zero outcomes.
```python
def _compute_text_similarity(topic: str, title: str, outcomes: list = None) -> float:
core = _extract_core_subject(topic).lower()
title_lower = title.lower()
if not core:
return 0.5
# Full substring match in title
if core in title_lower:
return 1.0
# Check if topic appears in any outcome name (bidirectional)
if outcomes:
core_tokens = set(core.split()) # Hoist outside loop
best_outcome_score = 0.0
for outcome_name in outcomes:
outcome_lower = outcome_name.lower()
# Bidirectional substring: "arizona" in "arizona basketball" OR "arizona basketball" in "arizona wildcats game"
if core in outcome_lower or outcome_lower in core:
best_outcome_score = max(best_outcome_score, 0.85)
elif core_tokens & set(outcome_lower.split()):
best_outcome_score = max(best_outcome_score, 0.7)
if best_outcome_score > 0:
return best_outcome_score
# Token overlap fallback against title
topic_tokens = set(core.split())
title_tokens = set(title_lower.split())
if not topic_tokens:
return 0.5
overlap = len(topic_tokens & title_tokens)
return overlap / len(topic_tokens)
```
### 2. Surface the topic-matching outcome in display
When the topic matches an outcome name, reorder `outcome_prices` to put the matching outcome first before truncating to top 3. This ensures the LLM sees the user-relevant odds.
### 3. Add SKILL.md synthesis instructions for Polymarket
Add a dedicated section telling the LLM:
- **Prediction markets are high-signal when relevant.** Real money on outcomes > opinions.
- **Prefer markets that answer structural/long-term questions** (championships, regime changes, major milestones) over near-term deadline markets (weekly matchups, short-term event deadlines). When in doubt, the bigger question is more interesting.
- **When the topic is an outcome in a multi-outcome market, call out that specific outcome's odds and movement.** Don't just say "Polymarket has a #1 seed market" - say "Arizona has 85% chance of a #1 seed, up from 72%."
- **Weave odds into the "What I learned" narrative as supporting evidence.** "Final Four buzz is building - Polymarket gives Arizona a 12% chance to win the championship (up 3% this week), and 85% to earn a #1 seed."
- **Citation format:** "Polymarket has Arizona at 85% for a #1 seed (up from 72%)" - include the specific odds and movement, not just "per Polymarket."
- **Stats box:** Show up to 5 most relevant markets with odds. If more exist, show count.
Domain examples:
- Sports: championship/tournament odds > regular season title > weekly matchup
- Geopolitics: regime change > near-term strike deadline > sanctions
- Tech: major milestones (IPO, product launch) > incremental updates
- Elections: presidency > primary > individual state
### 4. Improve stats box template
Show up to 5 markets with odds, capped for readability:
```
├─ 📊 Polymarket: 5 markets (Championship: 12%, #1 Seed: 85%, Big 12: 68%, vs Kansas: 71%, NCAA: 12%)
```
### 5. Fix render.py volume label
The render module labels volume as "vol24h" even when `volume1mo` is the actual data source. Fix to "vol/mo" when monthly volume is used.
## Technical Approach
### Implementation Plan
#### Phase 1: Fix text similarity to check outcomes
- [x] `scripts/lib/polymarket.py` - Update `_compute_text_similarity()` to accept optional `outcomes` parameter with bidirectional substring matching and token overlap
- [x] `scripts/lib/polymarket.py` - In `parse_polymarket_response()`, collect outcome names from ALL active markets (not just top market), filter to outcomes with price > 1%, and pass to `_compute_text_similarity()`
- [x] `scripts/lib/polymarket.py` - Reorder `outcome_prices` to surface the topic-matching outcome first before truncating to top 3
- [x] `fixtures/polymarket_sample.json` - Add fixture event: "Who will be the #1 overall seed in the 2026 NCAA Tournament?" with outcomes ["Arizona", "Duke", "Houston", "Auburn"] where "Arizona" is NOT in the title
- [x] `tests/test_polymarket.py` - Add tests for outcome-aware text similarity: bidirectional substring, token overlap, no-match, low-probability filtering
- [x] `tests/test_polymarket.py` - Add test: multi-outcome market where topic is an outcome should rank higher than tangential title-match markets
- [x] `tests/test_polymarket.py` - Add test: topic-matching outcome is surfaced to front of outcome_prices display
#### Phase 2: Add SKILL.md Polymarket synthesis instructions
- [x] `SKILL.md` - Add "Prediction Markets" subsection to the Judge Agent section with:
- General heuristic: prefer structural/long-term markets over near-term deadlines
- Domain examples (sports, geopolitics, tech, elections)
- Citation format with specific odds and movement
- Instruction to weave odds into "What I learned" narrative
- [x] `SKILL.md` - Add Polymarket to citation priority list (between HN and Web) with format guidance
- [x] `SKILL.md` - Update stats box template: show up to 5 markets with odds
- [x] `variants/open/references/research.md` - Add condensed Polymarket synthesis guidance matching the open variant's style
- [x] `scripts/lib/render.py` - Fix "vol24h" label to "vol/mo" when volume1mo is the data source (changed to "volume")
#### Phase 3: Tests and verification
- [x] Run full test suite (229 passed, 5 pre-existing failures unrelated to this change)
- [ ] Manual test: `/last30days "Arizona Basketball"` - verify championship, seed, and matchup odds appear in synthesis
- [ ] Manual test: `/last30days "Iran War"` - verify regime change and structural outcome markets appear
- [x] Run `bash scripts/sync.sh` to deploy
## Acceptance Criteria
- [x] Multi-outcome markets where the topic is an outcome (e.g. "Arizona" in a seeding market) get text_score >= 0.7, not 0.0
- [x] Bidirectional matching works: "Arizona" as outcome matches topic "Arizona Basketball" (outcome_name in core)
- [x] Topic-matching outcome is surfaced to front of outcome_prices display (not hidden in "and N more")
- [x] Low-probability outcomes (< 1%) don't trigger outcome matching
- [x] SKILL.md instructs the LLM to highlight structural/long-term markets over near-term ones
- [x] SKILL.md provides citation format: "Polymarket has X at Y% (up/down Z%)"
- [x] Stats box shows up to 5 markets with odds
- [x] HN zero-result line is hidden (already fixed - verify on next run)
- [x] All existing tests pass + new outcome-aware similarity tests pass
- [x] render.py volume label is accurate
## Dependencies & Risks
**No blockers.** This is scoring improvements + SKILL.md instruction changes within the existing Polymarket module.
**Risk: Outcome name matching false positives.** A market "Will Arizona pass AI regulation?" would match "Arizona Basketball" on the word "Arizona" even though it's about the state, not the team. Mitigation: outcome match gets 0.7-0.85 (not 1.0), and volume/liquidity/movement signals still differentiate. A false positive at 0.85 text_score won't outrank a true title match at 1.0.
**Risk: Common-word false positives.** Words like "war," "AI," "US" could match generic outcomes. Mitigation: at 0.7 text_score (30% weight = 0.21 relevance), this is a small boost that won't override strong volume/liquidity signals from actually relevant markets. Monitor in testing.
**Risk: LLM still ignores synthesis instructions.** Mitigation: use CRITICAL formatting, specific do/don't examples, and concrete citation format.
## Sources & References
- Current text similarity: `scripts/lib/polymarket.py:236`
- Render format: `scripts/lib/render.py:282`
- SKILL.md synthesis: `SKILL.md:196` (Judge Agent section)
- Previous quality ranking plan: `docs/plans/2026-02-25-feat-polymarket-quality-ranking-plan.md`
docs/plans/2026-02-26-fix-polymarket-query-expansion-data-availability-plan.md
-235
View File
@@ -1,235 +0,0 @@
---
title: "fix: Polymarket query expansion - find markets where topic is an outcome, not just a title"
type: fix
status: active
date: 2026-02-26
---
# fix: Polymarket Query Expansion & Data Availability
## Overview
The Polymarket module misses the most interesting markets when the search topic is an **outcome** in a broader market rather than appearing in the event title. For "Arizona Basketball", the NCAA Tournament Winner (30 open markets) and #1 Seed (20 open markets) are invisible because "Arizona" only appears as an outcome, never in the event title. The Gamma API only searches titles/slugs.
The outcome-aware scoring (from the prior plan) works perfectly on fixture data - it correctly ranks markets where Arizona is an outcome. The problem is upstream: those markets never reach the scoring layer because the Gamma API never returns them.
## Problem Statement
### Root Cause: Gamma API is title/slug search only
`gamma-api.polymarket.com/public-search?q=X` only matches against event titles and slugs. It does NOT search outcome names, market descriptions, or tags.
**Live API verification** (2026-02-26):
| Query | Events returned | Arizona-relevant open markets |
|-------|----------------|-------------------------------|
| "Arizona Basketball" | 5 | 2 (Big 12 Champion, NAU vs Idaho) |
| "Arizona" | 5 | 0 (all political/closed) |
| "NCAA Tournament" | 5 | 2 (Tournament Winner: 30 open, #1 Seed: 20 open) |
| "Basketball" | 5 | 5 (conference champions, no NCAA) |
| "college basketball" | 5 | 1 (#1 seed) |
The championship and seeding markets that the user wants (`NCAA Tournament Winner`, `#1 Seed`) are only findable by searching "NCAA Tournament" or "NCAA" - terms that don't appear in "Arizona Basketball".
### Contributing Factor 1: Query expansion too narrow
`_expand_queries("Arizona Basketball")` generates only `["Arizona Basketball", "Arizona"]`.
It only tries the **first word** as a standalone query. The second word "Basketball" is never searched independently. This means conference-adjacent and tournament markets are invisible.
### Contributing Factor 2: No domain bridging
Even searching "Basketball" (all individual words) only returns conference champions. The leap from "Basketball" to "NCAA Tournament" requires discovering the domain context from initial results. Currently there is no second-pass expansion.
### Contributing Factor 3: Shallow default depth
`DEPTH_CONFIG["default"] = 2` pages (10 events per query). With 3 queries that's 30 raw events, but heavy dedup and closed-event filtering reduces this to 2-5 usable results.
### What works (don't break it)
- "Iran War" returned 9 perfect markets because "Iran" and "War" appear directly in event titles
- Outcome-aware scoring correctly ranks Arizona-outcome markets when they reach the scoring layer
- Topic-matching outcome reordering surfaces the right outcome first
## Proposed Solution
Three changes, all generic (no hardcoded domain knowledge):
### 1. Search ALL individual words, not just the first
Currently `_expand_queries()` only adds `words[0]` as a standalone query. Change to add **every word** as a standalone query, then dedupe.
For "Arizona Basketball": `["Arizona Basketball", "Arizona", "Basketball"]`
For "Iran War": `["Iran War", "Iran", "War"]` (same as before since both are short)
For "AI video generation tools": `["AI video generation tools", "AI", "video", "generation", "tools"]` (capped at 6)
Raise query cap from 4 to 6 to accommodate.
```python
def _expand_queries(topic: str) -> List[str]:
core = _extract_core_subject(topic)
queries = [core]
words = core.split()
if len(words) >= 2:
# Add ALL individual words (not just first)
for word in words:
if len(word) > 2: # skip very short words ("AI", "vs")
queries.append(word)
if topic.lower().strip() != core.lower():
queries.append(topic.strip())
# Dedupe, cap at 6
seen = set()
unique = []
for q in queries:
q_lower = q.lower().strip()
if q_lower and q_lower not in seen:
seen.add(q_lower)
unique.append(q.strip())
return unique[:6]
```
Note: "AI" is only 2 chars but is meaningful. Lower the threshold to `len(word) > 1` to catch it. Single-char words (rare) get filtered.
### 2. Second-pass context expansion from first-pass results
After the first-pass search, extract domain-indicator terms from event titles and run a focused second-pass search.
Algorithm:
1. Collect ALL event titles from first-pass results (including closed events)
2. Tokenize titles into bigrams (two-word sequences)
3. Count bigrams across events, filter out bigrams containing topic words
4. Take the top 1-2 most frequent non-topic bigrams as "domain indicators"
5. Search each domain indicator (1 page each)
6. Merge with first-pass results, dedupe, re-rank
**Example for "Arizona Basketball":**
First-pass titles include:
- "Big 12 Men's College Basketball 2025-2026 Regular Season Champion"
- "SEC Men's College Basketball 2025-2026 Regular Season Champion"
- "ACC Men's College Basketball 2025-2026 Regular Season Champion"
- "Big East Men's College Basketball 2025-2026 Regular Season Champion"
Frequent bigrams (excluding topic words): "college basketball" (4x), "regular season" (4x), "season champion" (4x)
Top domain indicator: **"college basketball"**
Searching "college basketball" returns: **"#1 seed in NCAA Tournament"** (20 open markets with Arizona as outcome!)
**Example for "Iran War":** First-pass already finds everything via title matches. Second-pass bigrams would be things like "iran strikes", "khamenei out" - searching these finds the same events (deduped). No regression.
```python
def _extract_domain_queries(topic: str, events: List[Dict]) -> List[str]:
"""Extract domain-indicator search terms from first-pass event titles."""
topic_words = set(_extract_core_subject(topic).lower().split())
# Collect bigrams from all event titles
bigram_counts = {}
for event in events:
title = event.get("title", "").lower()
words = re.findall(r'[a-z]+', title)
for i in range(len(words) - 1):
bigram = f"{words[i]} {words[i+1]}"
# Skip if both words are in topic (we already search the topic)
if words[i] in topic_words and words[i+1] in topic_words:
continue
# Skip very common filler
if any(w in ("the", "of", "in", "to", "a", "and", "vs", "will", "be") for w in (words[i], words[i+1])):
continue
bigram_counts[bigram] = bigram_counts.get(bigram, 0) + 1
# Return bigrams appearing in 2+ event titles
domain_queries = [bg for bg, count in sorted(bigram_counts.items(), key=lambda x: -x[1]) if count >= 2]
return domain_queries[:2]
```
### 3. Increase depth and result caps
```python
DEPTH_CONFIG = {
"quick": 1,
"default": 3, # was 2 (50% more raw results)
"deep": 4, # was 3
}
RESULT_CAP = {
"quick": 5,
"default": 15, # was 10 (more room for cross-domain markets)
"deep": 25, # was 20
}
```
This gives default searches 3 queries x 3 pages = 45 raw events (up from 2 queries x 2 pages = 20), plus 2 domain-indicator queries x 1 page = 10 more.
## Technical Approach
### Implementation Plan
#### Phase 1: Expand query generation
- [x] `scripts/lib/polymarket.py` - Update `_expand_queries()` to add ALL individual words (len > 1), raise cap from 4 to 6
- [x] `tests/test_polymarket.py` - Test: `_expand_queries("Arizona Basketball")` returns `["Arizona Basketball", "Arizona", "Basketball"]`
- [x] `tests/test_polymarket.py` - Test: `_expand_queries("Iran War")` returns `["Iran War", "Iran", "War"]`
- [x] `tests/test_polymarket.py` - Test: short words excluded, cap at 6
#### Phase 2: Second-pass context expansion (evolved: tags instead of bigrams)
- [x] `scripts/lib/polymarket.py` - Add `_extract_domain_queries()` using event TAGS (not bigrams - tags are more reliable, contain "NCAA CBB" etc.)
- [x] `scripts/lib/polymarket.py` - Update `search_polymarket()` with `_run_queries_parallel()` helper for two-pass search
- [x] `scripts/lib/polymarket.py` - Add `_shorten_question()` to extract team names from neg-risk binary market questions
- [x] `scripts/lib/polymarket.py` - Synthesize outcome_prices from binary sub-market questions (detects Yes/No pattern, not just negRisk flag)
- [x] `scripts/lib/polymarket.py` - Updated outcome reordering to use token-based matching for long question strings
- [x] `scripts/lib/polymarket.py` - Also pass market questions to `_compute_text_similarity()` for neg-risk events
- [x] `tests/test_polymarket.py` - Tests for tag-based domain extraction: frequent tags, generic tag filtering, topic word filtering, min frequency, cap, empty events
#### Phase 3: Increase depth and caps
- [x] `scripts/lib/polymarket.py` - Update `DEPTH_CONFIG`: default 2 -> 3, deep 3 -> 4
- [x] `scripts/lib/polymarket.py` - Update `RESULT_CAP`: default 10 -> 15, deep 20 -> 25
#### Phase 4: Tests and verification
- [x] Run full test suite (238 passed, 5 pre-existing failures unrelated)
- [x] `bash scripts/sync.sh` to deploy to CROSS
- [x] Live API test: "Arizona Basketball" - finds NCAA Tournament Winner (Arizona: 12%), #1 Seed (Arizona: 88%), Big 12 (Arizona: 69%)
- [x] Live API test: "Iran War" - 15 markets, no regression (domain expansion found "Geopolitics", "Middle East")
- [ ] Manual test: `/last30daysCROSS "Arizona Basketball"` - end-to-end with LLM synthesis
- [ ] Manual test: `/last30daysCROSS "Duke Basketball"` - verify NCAA Tournament markets appear for another team
## Acceptance Criteria
- [x] `_expand_queries()` searches ALL individual words, not just the first
- [x] Query cap raised from 4 to 6
- [x] Second-pass context expansion discovers domain-indicator terms from first-pass event tags
- [x] "Arizona Basketball" search finds NCAA Tournament Winner AND #1 Seed markets (via "NCAA" tag domain expansion)
- [x] "Iran War" search still returns 9+ markets (15 - no regression)
- [x] All existing tests pass + new query expansion tests pass (91 polymarket tests)
- [x] No hardcoded domain knowledge (uses event tags, not hardcoded terms)
- [x] Default depth increased to 3 pages per query
- [x] Default result cap increased to 15
- [x] Neg-risk binary markets show team names instead of Yes/No (via question extraction)
- [x] Topic-matching team surfaced first in outcome display
## Dependencies & Risks
**No blockers.** Changes are internal to `polymarket.py` and don't affect other modules.
**Risk: Second-pass adds latency.** 2 extra queries x 1 page each, at ~200ms per call = ~400ms additional latency. The queries run after first-pass completes (serial), so total Polymarket time goes from ~1.5s to ~2s. Acceptable since other sources (YouTube, X) take 10-30s.
**Risk: Bigram extraction produces noise.** Common title patterns like "regular season" or "2025-2026" could become domain queries. Mitigation: filter filler words, require 2+ title appearances, and cap at 2 queries. A noisy domain query just returns irrelevant events that get scored low by the existing relevance ranker.
**Risk: Individual word queries return unrelated events.** "Basketball" returns European basketball, "War" returns non-Iran conflicts. Mitigation: the outcome-aware scoring already handles this - events without topic-matching outcomes get low text_score (token overlap only) and sort to the bottom.
**Risk: Rate limiting.** Adding 4-6 extra API calls per search. Gamma API allows 350 req/10s, so even aggressive searching stays well under limits.
## Sources & References
- Query expansion: `scripts/lib/polymarket.py:60` (`_expand_queries`)
- Search orchestration: `scripts/lib/polymarket.py:109` (`search_polymarket`)
- Event filtering: `scripts/lib/polymarket.py:302-306`
- Depth config: `scripts/lib/polymarket.py:20-31`
- Previous outcome-aware scoring plan: `docs/plans/2026-02-26-feat-polymarket-smarter-synthesis-plan.md`
- Live Gamma API test results from 2026-02-26 (documented above)
docs/plans/2026-03-02-chore-open-pr-triage-plan.md
-201
View File
@@ -1,201 +0,0 @@
---
title: "chore: open PR triage - build or close decision for all 5 open PRs"
type: chore
status: active
date: 2026-03-02
---
# Open PR Triage
Decision record for each of the 5 open PRs on `mvanhorn/last30days-skill`.
For each: build/merge, cherry-pick, or close with explanation.
---
## PR #26 - Add HN, YouTube, and Product Hunt sources + --search flag
**Author:** wkbaran | **Size:** +3402/-49 | **Date:** 2026-02-15
### What it adds
- Hacker News source (Algolia API, no auth)
- YouTube source (YouTube Data API v3, optional key)
- Product Hunt source (GraphQL API, optional key)
- `--search=SOURCES` flag (e.g. `--search=reddit,hn,yt`)
- 58 new tests
### What's already on main
- **HN:** ✅ already merged (`scripts/lib/hackernews.py`)
- **YouTube:** ✅ already merged (`scripts/lib/youtube_yt.py`)
- **Product Hunt:** ❌ NOT on main
- **`--search` flag:** ❌ NOT on main
### Decision: CHERRY-PICK (`--search` flag only, skip Product Hunt)
The HN and YouTube modules in this PR are earlier versions than what's on
main. We should NOT take those. Product Hunt is not a priority source.
The one missing piece worth landing:
**`--search=SOURCES` flag** - lets callers (including agent mode) specify
which sources to run. Useful for `--search=hn,polymarket` or
`--search=reddit,x` focus runs.
### Plan
- Cherry-pick just `tests/test_search_flag.py` and the `--search` argument
wiring in `scripts/last30days.py`
- Skip `scripts/lib/producthunt.py` and all PH-related code entirely
- Do NOT take their `hackernews.py`, `youtube.py`, `render.py`, `normalize.py`,
`schema.py` - our versions are newer
- Update SKILL.md to document `--search` flag
- Comment on PR thanking wkbaran - his HN/YouTube work was an inspiration
for the sources we ended up building. Acknowledge that specifically.
### Acceptance Criteria
- [ ] `python3 scripts/last30days.py "AI coding tools" --search=hn,reddit` runs only HN + Reddit
- [ ] `python3 scripts/last30days.py "test" --search=x,web` skips Reddit
- [ ] `test_search_flag.py` passes (adapt to our test patterns, no PH references)
- [ ] SKILL.md updated with `--search` flag docs
- [ ] Sync via `bash scripts/sync.sh`
---
## PR #24 - Adding Codex compatibility
**Author:** el-analista | **Size:** +228/-29 | **Date:** 2026-02-11
### What it adds
- `agents/openai.yaml` Codex discovery metadata
- Portable script path resolution in SKILL.md (multi-install-path loop)
- Platform-neutral UI text (`assistant` instead of Claude-specific wording)
- `LAST30DAYS_CACHE_DIR` env override in `scripts/lib/cache.py`
- `LAST30DAYS_OUTPUT_DIR` env override in `scripts/lib/render.py`
- X/Bird query noise stripping + last-chance retry in `scripts/lib/bird_x.py`
- New tests: `test_bird_x.py`, `test_cache.py`, `test_render.py`
### What's already on main
- **Codex auth:** ✅ already merged (via PR #38)
- **Multi-path script resolution:** ✅ already in SKILL.md (the `for dir in` loop)
- **Platform-neutral wording:** ❌ NOT systematically applied
- **Cache/output dir overrides:** ❌ NOT on main
- **Bird X improvements:** ✅ partially (we have other bird_x fixes but may be missing noise stripping + retry)
### Decision: REVIEW AND PARTIAL CHERRY-PICK
The env var overrides (`LAST30DAYS_CACHE_DIR`, `LAST30DAYS_OUTPUT_DIR`) are
genuinely useful for sandboxed/containerized Codex environments. The bird_x
noise stripping and last-chance retry may improve X search quality and are
low-risk additions.
The platform-neutral wording change ("assistant" instead of Claude references)
should be skipped — SKILL.md is Claude-specific by design.
### Plan
- Read the diff carefully against our current `cache.py`, `render.py`, `bird_x.py`
- Cherry-pick the `LAST30DAYS_CACHE_DIR` and `LAST30DAYS_OUTPUT_DIR` env overrides
- Cherry-pick the bird_x noise stripping + retry logic if it doesn't conflict
with our existing bird_x changes
- Skip `agents/openai.yaml` — we have our own multi-path resolution
- Skip platform-neutral wording changes
- Comment on PR thanking contributor and explaining what landed
### Acceptance Criteria
- [ ] `LAST30DAYS_CACHE_DIR=/tmp/test python3 scripts/last30days.py "test" --mock` writes cache to /tmp/test
- [ ] `LAST30DAYS_OUTPUT_DIR=/tmp/out python3 scripts/last30days.py "test" --mock` writes output to /tmp/out
- [ ] Existing tests pass after cherry-pick
- [ ] `test_cache.py` and `test_render.py` adapted and passing
---
## PR #14 - Simplify to WebSearch-first, make API keys optional
**Author:** thangman1 | **Size:** +97/-168 | **Date:** 2026-02-01
### What it does
Removes the Reddit/X Python search engine as the primary data source.
Repositions Claude Code's built-in WebSearch as the "default" mode.
Strips engagement metrics (upvotes, likes, repost counts) from output.
Removes mode detection logic (Full Mode / Partial Mode / Web-Only Mode).
### Decision: CLOSE - do not merge
This PR inverts the core value proposition of last30days. The skill's
differentiation is **real engagement data** from Reddit threads and X posts —
upvotes, likes, reposts — that WebSearch cannot provide. Stripping that out
produces a worse tool than just asking Claude to search the web, which anyone
can already do.
The author's intent (lower barrier to entry, no API key required) is valid,
but the right solution is making HN + Polymarket work without any API key
(they already do), and making OPENAI_API_KEY easier to obtain — not removing
the Reddit/X engine.
### Action
- Comment on PR explaining why we're closing it
- Acknowledge the valid friction point (API key setup) and point to HN +
Polymarket as the zero-config sources
- Close PR
---
## PR #10 - OpenRouter API integration
**Author:** thetechreviewer | **Size:** +1029/-16 | **Date:** 2026-01-28
### What it adds
OpenRouter as an alternative to OpenAI for the Reddit discovery search.
Allows using any model available on OpenRouter instead of just OpenAI models.
### What's already on main
- **`scripts/lib/openrouter_search.py`:** ✅ ALREADY ON MAIN
- **Wired into `scripts/last30days.py`:** ✅ ALREADY ON MAIN (the `backend == "openrouter"` path)
### Decision: CLOSE - already merged
The OpenRouter integration that this PR introduced is already on main. It
arrived via internal work that post-dated this PR. The PR is stale.
### Action
- Comment on PR: "Thanks for this — OpenRouter support is already on main
(landed via internal work). Closing as incorporated."
- Close PR
---
## PR #5 - Add support for Codex auth with OpenAI Responses API
**Author:** jblwilliams | **Size:** +358/-66 | **Date:** 2026-01-27
### What it adds
- JWT-based Codex auth with `chatgpt_account_id`
- Codex endpoint routing (`https://chatgpt.com/backend-api/codex/responses`)
- SSE handling for streaming Codex responses
- Typed auth status/source dataclass
- Codex fallback model chain
### What's already on main
**All of this is already on main.** The Codex auth system landed via PR #37
(iliaal:codex-auth-merged), which in turn came in with PR #38. This PR (#5)
predates that work and covers the same ground.
### Decision: CLOSE - already incorporated
### Action
- Comment on PR: "Thanks for this early work on Codex auth! The same feature
landed on main via PR #37 (from a separate contributor who built on similar
ideas). The JWT decode, Codex endpoint routing, SSE parsing, and typed auth
dataclass are all live. Closing as incorporated."
- Close PR
---
## Summary Table
| PR | Author | Decision | Reason |
|----|--------|----------|--------|
| #26 | wkbaran | **Cherry-pick** | `--search` flag not on main; HN/YouTube already there; skip Product Hunt |
| #24 | el-analista | **Cherry-pick** | Cache dir overrides + bird_x improvements worth landing; Codex auth already there |
| #14 | thangman1 | **Close** | Removes engagement data, inverts core value proposition |
| #10 | thetechreviewer | **Close** | OpenRouter already on main |
| #5 | jblwilliams | **Close** | Codex auth already on main via PR #37/38 |
## Implementation Order (if proceeding)
1. Close PR #5, #10, #14 with comments (no code changes needed)
2. Cherry-pick PR #24 pieces (bird_x + cache/render env overrides)
3. Cherry-pick PR #26 pieces (Product Hunt + `--search` flag)
4. Sync and test
5. Push to upstream
docs/plans/2026-03-02-feat-close-pr37-codex-auth-finalization-plan.md
-250
View File
@@ -1,250 +0,0 @@
---
title: "feat: close PR #37 - Codex auth finalization and clean close"
type: feat
status: active
date: 2026-03-02
---
# feat: Close PR #37 - Codex Auth Finalization
## Overview
PR #37 (`iliaal:codex-auth-merged`) adds Codex auth so users with `codex login` can use the
skill without an `OPENAI_API_KEY`. The good news: **all of its core changes are already on main**.
PR #38 (which we merged earlier today) was branched directly from #37, so the Codex auth code
rode in with that merge.
The task is to:
1. Verify the Codex auth integration is intact and compatible with v2.6 additions
2. Fix the one known test isolation bug that PR #37 shipped with
3. Close PR #37 with a clear explanation and a thank-you to the contributor
## What PR #37 Added (Now All on Main)
### Core Codex auth system (`scripts/lib/env.py`)
- `CODEX_AUTH_FILE` path constant (`~/.codex/auth.json`)
- `OpenAIAuth` dataclass: `token`, `source`, `status`, `account_id`, `codex_auth_file`
- `_decode_jwt_payload()` - JWT base64 decode without verification
- `_token_expired()` - checks JWT `exp` claim with 60s leeway
- `extract_chatgpt_account_id()` - extracts `chatgpt_account_id` from JWT `https://api.openai.com/auth` claim
- `load_codex_auth()` - reads `~/.codex/auth.json`
- `get_codex_access_token()` - returns `(token, status)` tuple
- `get_openai_auth()` - priority chain: `OPENAI_API_KEY` env var > `.env` file key > Codex token
### Codex endpoint routing (`scripts/lib/openai_reddit.py`)
- `CODEX_RESPONSES_URL = "https://chatgpt.com/backend-api/codex/responses"`
- `_parse_sse_chunk()` / `_parse_sse_stream()` / `_parse_codex_stream()` - SSE response parsing
- Headers injected for Codex path: `chatgpt-account-id`, `OpenAI-Beta: responses=v1`, `originator: pi`
- Codex payload: `store: false`, `stream: true`
- `CODEX_FALLBACK_MODELS` retry chain: `gpt-5.1-codex-mini` → `gpt-5.2`
### Tests (`tests/test_codex_auth.py`)
- 22 unit tests covering JWT decode, expiry, account ID extraction, auth resolution,
SSE parsing, payload building, source availability
- 21/22 pass; 1 has a test isolation bug (see below)
## What PR #37 Contains That Must NOT Go to Public Repo
These files are in the #37 branch but are internal benchmarking artifacts. They should
never land on the public `upstream` remote:
| Path | Why private |
|------|-------------|
| `docs/comparison-results/` (55+ files) | Benchmark JSON/MD from synthesis quality testing |
| `docs/plans/*.md` (9 internal plan docs) | Private planning documents |
| `docs/v2.1-tweets.md` | Internal launch tweet drafts |
| `variants/open/references/research.md` | Internal research notes |
| `scripts/evaluate-synthesis.py` | Internal evaluation script |
| `scripts/generate-synthesis-inputs.py` | Internal benchmark input generator |
| `fixtures/polymarket_sample.json` | Used by internal eval scripts |
## What PR #37 Has That's OLDER Than Main
These files in PR #37 are earlier versions than what main has - we keep our versions:
- `SKILL.md` - PR #37 is v2.1; main is v2.6 (keep v2.6)
- `README.md` - PR #37's is missing HN/Polymarket; main's is current (keep main)
- `SPEC.md` - PR #37 has an older spec (keep main)
- `scripts/lib/hackernews.py` - NOT in PR #37; main has the full HN integration
- `scripts/lib/polymarket.py` - PR #37 has an older version without quality ranking
- `scripts/sync.sh` - minor differences; main's version is correct
## Known Issue: Test Isolation Bug
**File:** `tests/test_codex_auth.py`
**Test:** `TestGetOpenaiAuth::test_api_key_takes_priority`
```python
def test_api_key_takes_priority(self):
"""OPENAI_API_KEY in env file should be preferred over Codex."""
file_env = {"OPENAI_API_KEY": "sk-test123"}
auth = env.get_openai_auth(file_env)
self.assertEqual(auth.token, "sk-test123") # FAILS if OPENAI_API_KEY set in shell
```
**Root cause:** `get_openai_auth()` checks `os.environ.get("OPENAI_API_KEY")` first (env var
priority). The test sets `file_env` but does NOT patch `os.environ`, so the real
`OPENAI_API_KEY` from the developer's shell wins.
**Fix:**
```python
@patch.dict(os.environ, {}, clear=False)
def test_api_key_takes_priority(self):
```
But we also need to REMOVE `OPENAI_API_KEY` from the patched env:
```python
@patch.dict(os.environ, {"OPENAI_API_KEY": ""}, clear=False)
def test_api_key_takes_priority(self):
```
Actually the cleanest fix:
```python
def test_api_key_takes_priority(self):
"""OPENAI_API_KEY in env file should be preferred over Codex."""
with patch.dict(os.environ, {}, clear=True):
# Restore non-OPENAI env vars to avoid side effects
file_env = {"OPENAI_API_KEY": "sk-test123"}
auth = env.get_openai_auth(file_env)
self.assertEqual(auth.token, "sk-test123")
```
Or the minimal fix (remove just OPENAI_API_KEY without nuking entire env):
```python
@patch.dict(os.environ, {"OPENAI_API_KEY": "sk-test123"})
def test_api_key_takes_priority(self):
"""OPENAI_API_KEY in env var should be preferred over Codex."""
auth = env.get_openai_auth({})
self.assertEqual(auth.source, "api_key")
self.assertEqual(auth.token, "sk-test123")
self.assertIsNone(auth.account_id)
```
This reframes the test as "env var takes priority over empty file_env" which is equally
valid and sidesteps the isolation problem entirely.
## Acceptance Criteria
- [ ] Verify `tests/test_codex_auth.py` runs 22/22 clean (no isolation failures)
- [ ] Fix `test_api_key_takes_priority` with the minimal patch approach above
- [ ] Run full test suite: confirm only the 5 pre-existing stale model failures remain
- [ ] Verify `env.py` on main has `is_hackernews_available()` and `is_polymarket_available()`
(they were REMOVED in PR #37 but should be on main since #38 preserved them)
- [ ] Verify `scripts/sync.sh` deploys to `~/.claude/skills/last30daysCROSS` correctly
(PR #37's sync.sh may be missing this; main's version should have it)
- [ ] Close PR #37 with a comment explaining the code landed via #38
- [ ] Add `docs/comparison-results/` to `.gitignore` in the private repo so benchmark
files never accidentally get committed to the upstream public repo
## Implementation Steps
### Step 1: Fix the test
In the **private source repo** (`/Users/mvanhorn/last30days-skill-private/`):
Edit `tests/test_codex_auth.py` line 77-84. Replace the bare test with the `@patch.dict` version above. Run `python3 -m pytest tests/test_codex_auth.py -v` to confirm 22/22.
### Step 2: Run full test suite
```bash
cd /Users/mvanhorn/last30days-skill-private
python3 -m pytest tests/ -v 2>&1 | tail -20
```
Expected: only `test_reddit_search_basic`, `test_default_model`, `test_model_pin`, and
similar model-name tests fail (the 5 pre-existing stale model failures from before PR #37).
Codex auth tests should all pass.
### Step 3: Verify the private docs protection
Check that `.gitignore` (or the upstream push config) prevents `docs/comparison-results/`
from leaking to the public GitHub remote.
```bash
cat /Users/mvanhorn/last30days-skill-private/.gitignore | grep -E "comparison|evaluate|generate"
```
If not present, add:
```
docs/comparison-results/
scripts/evaluate-synthesis.py
scripts/generate-synthesis-inputs.py
fixtures/polymarket_sample.json
docs/v2.1-tweets.md
variants/open/references/research.md
```
### Step 4: Commit and sync
```bash
cd /Users/mvanhorn/last30days-skill-private
git add tests/test_codex_auth.py
git commit -m "fix(tests): patch OPENAI_API_KEY env isolation in test_api_key_takes_priority"
bash scripts/sync.sh
```
Then push to upstream (public):
```bash
git push upstream main
```
### Step 5: Close PR #37
Post a comment on PR #37 explaining what happened, then close it:
```
Thanks @iliaal! 🙏 This was a great contribution.
The Codex auth changes landed in main via PR #38, which was branched from your
`codex-auth-merged` branch. So all the core auth code is already shipping:
- JWT decoding + expiry checking in env.py ✅
- Codex endpoint routing + SSE parsing in openai_reddit.py ✅
- 22 unit tests in test_codex_auth.py ✅
- CODEX_FALLBACK_MODELS retry chain ✅
Since then we've also shipped v2.5 (HN + Polymarket sources) and v2.6 (agent-native
invocation with --agent flag), so SKILL.md and README are already ahead of this branch.
Closing as the changes are incorporated. Thanks again for the excellent work!
```
## Technical Considerations
**Why can't we just merge #37 directly?**
Three reasons:
1. SKILL.md/README in #37 are v2.1 - they'd overwrite our v2.6 improvements
2. The `docs/comparison-results/` directory (55+ benchmark files) would go to the public repo
3. The test isolation bug would ship a flaky test to everyone who sets `OPENAI_API_KEY`
**Cherry-pick vs close approach:**
Since the Codex auth code is already on main, cherry-picking would be redundant. The cleanest
path is to fix the test bug on main, then close #37 with an explanation.
**Future private-docs hygiene:**
The `docs/comparison-results/` files should be gitignored or moved to a separate
private branch so this situation doesn't repeat. These are internal QA benchmarks -
they belong in the private repo only, never in `upstream`.
## Dependencies & Risks
**Low risk** - this is test cleanup + PR bookkeeping. The feature itself is already running.
**Codex auth live test** - We can't easily verify the live Codex auth flow without a `codex login`
session. If a user reports auth issues, the test suite gives good coverage of the logic;
live testing would require a Codex-authenticated environment.
## Sources & References
- PR #37: https://github.com/mvanhorn/last30days-skill/pull/37 (iliaal: codex-auth-merged)
- PR #38 (merged): fixed Bird X auth, brought Codex auth to main as a side effect
- `tests/test_codex_auth.py` - 22 unit tests for the Codex auth system
- `scripts/lib/env.py` lines 26-175 - Codex auth core logic
- `scripts/lib/openai_reddit.py` lines 45-310 - Codex endpoint routing
docs/plans/2026-03-02-feat-last30days-v2.6-agent-native-invocation-plan.md
-154
View File
@@ -1,154 +0,0 @@
---
title: "feat: last30days v2.6 - agent-native invocation"
type: feat
status: completed
date: 2026-03-02
---
# feat: last30days v2.6 - Agent-Native Invocation
## Overview
v2.5 shipped as a human-only interactive skill. Agents calling it today would hang forever at
"WAIT FOR USER RESPONSE" and never receive research output. v2.6 makes the skill fully
invocable by other agents - delivering a complete research report instead of an interactive
conversation.
The `disable-model-invocation: true` flag was removed in a hotfix today (2026-03-02). v2.6
formalizes that fix, addresses the deeper interactive-flow problem, and updates all
documentation to match reality.
## Problem Statement
### 1. The flag was removed but the behavior is still broken
Removing `disable-model-invocation: true` lets agents call the skill. But the skill still:
- Calls `AskUserQuestion` mid-flow (blocks, waits for a human to click something)
- Ends with "WAIT FOR USER RESPONSE" (the calling agent gets nothing)
- Assumes a human is reading progress text in real-time
When an agent calls `/last30days plaud granola`, it needs a **completed research report
returned to it**, not a half-executed interactive session.
### 2. The Security section has a false statement
Line 558 of SKILL.md still reads:
```
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
```
This is now wrong. It actively misleads users and agents reading the skill docs.
### 3. No documented path for agent callers
Users who want to call `last30days` from another skill (e.g., "research this topic and then
build a plan") have no guidance on how to do it or what format to expect back.
## Proposed Solution
### Agent Mode: `--agent` flag
Add an `--agent` execution mode that skips all interactive elements and returns a structured
research report. Calling agents pass the flag explicitly:
```
/last30days plaud granola --agent
```
### Agent Mode Behavior
In agent mode, the skill:
1. **Skips** the intro display block ("I'll research X across Reddit...")
2. **Skips** `AskUserQuestion` for target tool clarification
3. **Skips** the "WAIT FOR USER RESPONSE" pause
4. **Skips** the follow-up invitation ("I'm now an expert on X...")
5. **Outputs** a complete, structured report with all findings, then terminates cleanly
The report format for agent mode:
```
## Research Report: {TOPIC}
Generated: {date} | Sources: Reddit, X, YouTube, HN, Polymarket, Web
### Key Findings
[3-5 bullet points, highest-signal insights with citations]
### {Source} Results
[Compact per-source sections with top items]
### Stats
{The existing stats block}
```
### Security Section Rewrite
Remove the false statement. Replace with:
```
- Can be invoked autonomously by agents via the Skill tool (inline mode, not forked)
- Pass `--agent` flag for non-interactive report output
```
### Version Bump
`version: "2.5"` → `version: "2.6"` in SKILL.md frontmatter.
## Technical Considerations
### Implementation: SKILL.md instructions only
Add a section to SKILL.md that reads:
```
## Agent Mode (--agent flag)
If --agent is in ARGUMENTS: skip intro display, skip AskUserQuestion, skip invitation,
skip WAIT block. Output the full research report and stop.
```
No Python script changes needed. The script already outputs everything via `--emit=compact` -
the LLM just skips the interactive wrapper. If LLM instruction-following proves unreliable in
testing, a v2.6.1 patch adds `--agent` natively to `last30days.py` (Python flag approach).
## Acceptance Criteria
- [x] `disable-model-invocation: true` is gone from SKILL.md (done in hotfix - verify)
- [x] Security section no longer claims the skill cannot be invoked autonomously
- [x] SKILL.md documents the `--agent` flag and its behavior
- [x] In `--agent` mode: no AskUserQuestion calls, no WAIT block, full report output
- [x] Version is `2.6` in frontmatter
- [x] Synced to all 4 destinations via `sync.sh`
- [ ] Manual test: `Skill { skill: "last30days", args: "plaud granola --agent" }` returns
a complete research report without hanging
## Implementation Steps
1. **Verify hotfix** - confirm `disable-model-invocation` is gone in private source
2. **Edit SKILL.md** - update Security section, add Agent Mode section, bump version
3. **Run sync.sh** - deploy to `~/.claude`, `~/.agents`, `~/.codex`
4. **Test** - invoke via Skill tool and verify report returns cleanly
All changes are in `SKILL.md` only. No Python changes needed for v2.6.
## Dependencies & Risks
**No script changes** - this is a pure SKILL.md update. Low risk.
**LLM instruction-following** - agent mode relies on the LLM reading and following the
`--agent` section. If testing shows it's unreliable, escalate to Option B (Python flag)
as a v2.6.1 patch.
**Skill list refresh** - after sync, Claude Code needs to restart or reload skills to see
the updated frontmatter. Users on older Claude Code versions may still see the old flag.
## Distribution Note
For users who installed from the public repo, they need to pull the update. The sync script
handles local installations. For `last30daysCROSS` variant, the same changes apply.
## Sources & References
- [fix-skill-execution-fork-mode-plan.md](./2026-02-06-fix-skill-execution-fork-mode-plan.md) - original fork-mode diagnosis
- SKILL.md line 558: false security claim to remove
- Hotfix applied 2026-03-02: removed `disable-model-invocation: true`
docs/plans/2026-03-02-fix-websearch-sources-trailing-block-plan.md
-102
View File
@@ -1,102 +0,0 @@
---
title: "fix: suppress trailing Sources: block from WebSearch tool mandate"
type: fix
status: completed
date: 2026-03-02
---
# fix: Suppress Trailing Sources: Block from WebSearch Tool Mandate
## Problem
After the skill completes, a `Sources:` block appears below the invitation text:
```
I'm now an expert on the Dor Brothers. Some things I can help with:
...
Sources:
- Movie starring Logan Paul made exclusively with AI released - Newsweek
- The Dor Brothers: Pioneers in AI Video Production
- ...
```
This happens because the `WebSearch` tool has a **system-level mandatory instruction**:
> "After answering the user's question, you MUST include a 'Sources:' section at the end of your response"
SKILL.md already says `DO NOT output "Sources:" list` (line 214) but this is too vague - it doesn't address the WebSearch tool mandate explicitly, so the tool's system instruction wins. The model dutifully appends Sources: after all skill output is done.
## Root Cause
Two competing instructions:
1. **WebSearch system mandate** (higher authority): "MUST include Sources: at end of response"
2. **SKILL.md line 214** (lower authority): "DO NOT output Sources: list"
The model follows #1 because it's framed as a critical system requirement.
The fix: **satisfy the WebSearch citation requirement INSIDE the stats block**, then explicitly tell the model the requirement is already fulfilled and no trailing section is needed.
## Proposed Solution
**Two-part SKILL.md edit only. No Python changes.**
### Part 1: Update the Step 2 instruction (line 214)
**Current:**
```
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
```
**Replace with:**
```
- **DO NOT output a separate "Sources:" block** — instead, include the top 3-5 web
source names as inline links on the 🌐 Web: stats line (see stats format below).
This satisfies the WebSearch tool's citation requirement inline without a trailing section.
```
### Part 2: Update the stats block format to include web source links
**Current stats line:**
```
├─ 🌐 Web: {N} pages (supplementary)
```
**Replace with:**
```
├─ 🌐 Web: {N} pages — [Source Name](url), [Source Name](url), [Source Name](url)
```
And immediately after the closing `---` of the stats block, add:
```
**WebSearch citation note:** Source links are included in the 🌐 Web: line above.
The WebSearch tool citation requirement is satisfied. Do NOT append a separate
"Sources:" section after the invitation.
```
## Acceptance Criteria
- [x] The trailing `Sources:` block no longer appears after the invitation
- [x] Web source links appear cleanly on the `🌐 Web:` stats line
- [ ] Manual test: run `/last30days dor brothers` and confirm no trailing Sources: block
- [x] Synced to all 4 destinations via `sync.sh`
## Implementation Steps
1. Edit `SKILL.md` line 214 (Step 2 section) - replace weak "DO NOT" with redirect instruction
2. Edit `SKILL.md` stats block format - add `— [Source](url), ...` to the 🌐 Web: line
3. Add WebSearch citation note after the stats block closing `---`
4. Run `bash scripts/sync.sh` to deploy
5. Test with `/last30days [any topic]` and confirm no trailing Sources:
## Context
- **Why not just fight the mandate?** The WebSearch system instruction is authoritative. We can't override it with a soft "don't do this." We need to redirect it.
- **Why the stats block?** It's the natural place for source metadata and it appears before the invitation, so satisfying the citation there prevents the trailing append.
- **SKILL.md is the only file that needs to change.** No Python script changes required.
## Sources & References
- SKILL.md line 214: current weak instruction
- SKILL.md stats block section: where web source links will live
- Screenshot: user-reported Sources: trailing block (session context)
docs/plans/2026-03-03-feat-skip-native-web-in-claude-code-plan.md
-43
View File
@@ -1,43 +0,0 @@
# feat: Skip native web search when running in Claude Code
**Type:** enhancement
**Date:** 2026-03-03
**Detail level:** MINIMAL
## Problem
When `/last30days` runs in Claude Code, web search happens twice:
1. The Python script uses Parallel AI / Brave / OpenRouter (costs API credits)
2. SKILL.md tells Claude to run its built-in WebSearch tool (free, better quality)
This is redundant. Claude's WebSearch is better and free. In OpenClaw, there's no WebSearch tool, so native backends are essential there.
## Solution
Add a `--no-native-web` CLI flag to `last30days.py`. When set, the script skips native web search backends even if API keys are configured, and emits the `### WEBSEARCH REQUIRED ###` signal so the assistant handles it.
Update SKILL.md invocation to include the flag.
## Changes
### 1. `scripts/last30days.py`
- [ ] Add `--no-native-web` argument to argparse (store_true, default False)
- [ ] When `args.no_native_web` is True, force `web_backend = None` regardless of API keys
- [ ] This naturally triggers `web_needed = True` → emits `### WEBSEARCH REQUIRED ###` signal
- [ ] Update diagnostic banner to show "Web: deferred to assistant" when flag is active
### 2. `SKILL.md`
- [ ] Add `--no-native-web` to the invocation command on line 168:
```
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
```
### 3. OpenClaw / `--agent` mode
- [ ] No changes needed — OpenClaw invocations don't read SKILL.md, they call the script directly without `--no-native-web`, so Parallel AI/Brave/OpenRouter still work
## Acceptance Criteria
- [ ] Claude Code sessions: script skips Parallel AI, Claude uses WebSearch (no API credits spent)
- [ ] OpenClaw sessions: script still uses Parallel AI / Brave / OpenRouter as before
- [ ] `--no-native-web` flag can be combined with `--include-web` (flag wins, web deferred)
- [ ] Diagnostic output clearly shows web is deferred to assistant
docs/plans/2026-03-03-feat-tiktok-apify-source-plan.md
-417
View File
@@ -1,417 +0,0 @@
---
title: "feat: Add TikTok as 7th source via Apify"
type: feat
date: 2026-03-03
---
# feat: Add TikTok Signal via Apify
## Overview
Add TikTok as the 7th research source alongside Reddit, X, YouTube, HN, Polymarket, and Web. Use the **Apify** platform (`clockworks/tiktok-scraper` actor) to search TikTok by keyword, extract engagement metrics (views, likes, comments), and optionally pull video captions for synthesis enrichment — mirroring the YouTube pattern.
**Why this matters:** TikTok is where trends break first for many topics (products, music, culture, tech tips, news reactions). A viral TikTok with 2M views is a stronger signal than a tweet with 500 likes. The skill currently misses this entirely.
**Why Apify:** BYO API key, $5/month free credits (no CC required), pay-per-result pricing, Python SDK (`apify-client`), and the same actor platform supports Facebook and Instagram scrapers — so this investment pays forward.
## Proposed Solution
### Architecture: Shared Apify Client + Per-Source Modules
```
scripts/lib/
apify_client_wrapper.py ← NEW: shared Apify client init + helpers (reused by FB/IG later)
tiktok.py ← NEW: TikTok search, captions, relevance
# future:
# facebook.py ← uses same apify_client_wrapper.py
# instagram.py ← uses same apify_client_wrapper.py
```
This design means adding Facebook or Instagram later is just a new `facebook.py` module — the Apify client setup, token validation, and error handling are already done.
### Data Flow
```
User topic + date range
↓
[apify_client_wrapper.py] init client with APIFY_API_TOKEN
↓
[tiktok.py] search_tiktok()
├─ Call clockworks/tiktok-scraper actor (sync API, ≤5min)
├─ Input: searchQueries=[core_topic], resultsPerPage=N (depth-aware)
├─ Parse: id, text, playCount, diggCount, commentCount, createTimeISO, authorMeta, webVideoUrl, hashtags
├─ Sort by playCount (views) descending
├─ Compute relevance via token-overlap (reuse youtube_yt._compute_relevance pattern)
└─ Return items
↓
[tiktok.py] fetch_captions() (optional enrichment for top N)
├─ Re-call actor with shouldDownloadSubtitles=true for top videos
├─ OR use video text/description as lightweight "caption" alternative
└─ Truncate to 500 words, attach as caption_snippet
↓
[normalize.py] normalize_tiktok_items() → List[TikTokItem]
↓
[score.py] score_tiktok_items()
├─ compute_tiktok_engagement_raw(): 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
├─ Weighted: 0.45*relevance + 0.25*recency + 0.30*engagement
└─ Same formula as YouTube (views-dominant)
↓
[dedupe.py] dedupe_tiktok() + cross_source_link()
↓
[render.py] render TikTok section
↓
[SKILL.md] stats line: 🎵 TikTok: N videos │ N views │ N with captions
```
## Technical Approach
### Phase 1: Apify Client Wrapper (`scripts/lib/apify_client_wrapper.py`)
Shared module for all Apify-backed sources. Keeps TikTok, Facebook, Instagram from duplicating client setup.
```python
"""Shared Apify client utilities for last30days sources."""
from apify_client import ApifyClient
from typing import Optional, Dict, Any, List
def get_apify_client(token: str) -> ApifyClient:
"""Initialize Apify client with token."""
return ApifyClient(token=token)
def run_actor_sync(
client: ApifyClient,
actor_id: str,
run_input: Dict[str, Any],
timeout_secs: int = 300,
max_items: int = None,
) -> List[Dict[str, Any]]:
"""Run an Apify actor synchronously and return dataset items.
Args:
client: Initialized ApifyClient
actor_id: e.g. "clockworks/tiktok-scraper"
run_input: Actor-specific input dict
timeout_secs: Max wait time (default 5 min)
max_items: Cap on returned items (cost control)
Returns:
List of result dicts from the actor's default dataset
"""
run = client.actor(actor_id).call(
run_input=run_input,
timeout_secs=timeout_secs,
)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
if max_items:
items = items[:max_items]
return items
```
**Key design decisions:**
- Single `APIFY_API_TOKEN` env var for all Apify sources (TikTok, future FB, IG)
- `run_actor_sync()` wraps the call+wait+fetch pattern used by every Apify actor
- `max_items` param provides cost control (important with $5 free credits)
### Phase 2: TikTok Search Module (`scripts/lib/tiktok.py`)
```python
"""TikTok search via Apify clockworks/tiktok-scraper."""
ACTOR_ID = "clockworks/tiktok-scraper"
DEPTH_CONFIG = {
"quick": {"results_per_page": 10, "max_captions": 3},
"default": {"results_per_page": 20, "max_captions": 5},
"deep": {"results_per_page": 40, "max_captions": 8},
}
def search_tiktok(topic, from_date, to_date, depth="default", token=None):
"""Search TikTok via Apify.
Returns:
Dict with 'items' list and optional 'error'.
"""
# 1. Init client via apify_client_wrapper
# 2. Build input: searchQueries=[_extract_core_subject(topic)], resultsPerPage=N
# 3. Call run_actor_sync(client, ACTOR_ID, input, timeout=120)
# 4. Parse items: extract id, text, playCount, diggCount, commentCount,
# shareCount, createTimeISO, authorMeta.name, webVideoUrl, hashtags
# 5. Filter by date range (from_date to to_date)
# 6. Sort by playCount descending
# 7. Compute relevance via _compute_relevance(topic, item_text)
# 8. Return structured items
def fetch_captions(video_items, token, depth="default"):
"""Fetch captions/subtitles for top N TikTok videos.
Strategy: Re-run actor with shouldDownloadSubtitles=true for
specific video URLs, OR fall back to video text/description
as a lightweight alternative.
Returns:
Dict mapping video_id → caption_text (truncated to 500 words)
"""
def search_and_enrich(topic, from_date, to_date, depth="default", token=None):
"""Search + caption enrichment orchestrator (mirrors youtube_yt.search_and_transcribe)."""
def parse_tiktok_response(response):
"""Extract items list from search_and_enrich response."""
```
**Apify actor input for keyword search:**
```json
{
"searchQueries": ["claude code tips"],
"resultsPerPage": 20,
"shouldDownloadSubtitles": false,
"shouldDownloadVideos": false,
"shouldDownloadCovers": false
}
```
**Apify actor output fields we use:**
| Apify Field | Our Field | Notes |
|---|---|---|
| `id` | `id` | TikTok video ID |
| `text` | `caption` | Video caption/description |
| `playCount` | `engagement.views` | Primary engagement signal |
| `diggCount` | `engagement.likes` | Secondary signal |
| `commentCount` | `engagement.num_comments` | Tertiary signal |
| `shareCount` | (stored but not scored) | Available for future use |
| `createTimeISO` | `date` | Parse to YYYY-MM-DD |
| `authorMeta.name` | `author_name` | Creator handle |
| `authorMeta.fans` | (stored but not scored) | Follower count |
| `webVideoUrl` | `url` | Direct TikTok link |
| `hashtags[].name` | `hashtags` | For relevance boosting |
| `videoMeta.duration` | `duration` | For filtering very short clips |
**Relevance scoring:** Reuse the token-overlap algorithm from `youtube_yt._compute_relevance()`. Additionally boost relevance when topic tokens appear in hashtags (TikTok-specific signal).
**Caption enrichment strategy:**
1. **Primary:** Use the `text` field (video description/caption) — always available, free
2. **Enhanced:** For top N videos, re-run actor with `shouldDownloadSubtitles: true` to get spoken-word captions
3. **Fallback:** If subtitles unavailable, use `text` field alone (most TikTok videos have descriptive captions)
This is cheaper than YouTube transcripts (no second yt-dlp call needed for the basic case).
### Phase 3: Schema + Normalization
**`scripts/lib/schema.py` — add TikTokItem dataclass:**
```python
@dataclass
class TikTokItem:
"""Normalized TikTok item."""
id: str # video_id
text: str # caption/description
url: str # webVideoUrl
author_name: str # authorMeta.name
date: Optional[str] = None
date_confidence: str = "high" # Apify provides exact timestamps
engagement: Optional[Engagement] = None # views, likes, num_comments
caption_snippet: str = "" # spoken-word caption (if available), else text
hashtags: List[str] = field(default_factory=list)
relevance: float = 0.7
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
cross_refs: List[str] = field(default_factory=list)
```
**`scripts/lib/schema.py` — add to Engagement dataclass:**
- `shares: Optional[int] = None` — TikTok shares (also useful for future Facebook)
**`scripts/lib/schema.py` — add to Report dataclass:**
- `tiktok: List[TikTokItem] = field(default_factory=list)`
- `tiktok_error: Optional[str] = None`
**`scripts/lib/normalize.py` — add `normalize_tiktok_items()`:**
- Parse `createTimeISO` → YYYY-MM-DD
- Create Engagement(views=playCount, likes=diggCount, num_comments=commentCount)
- Create TikTokItem objects
- Hard date filter (like Reddit/X, not soft like YouTube)
### Phase 4: Scoring
**`scripts/lib/score.py` — add TikTok scoring:**
```python
def compute_tiktok_engagement_raw(engagement):
"""TikTok engagement: views-dominant like YouTube.
0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
"""
views = getattr(engagement, 'views', 0) or 0
likes = getattr(engagement, 'likes', 0) or 0
comments = getattr(engagement, 'num_comments', 0) or 0
return 0.50 * log1p(views) + 0.30 * log1p(likes) + 0.20 * log1p(comments)
def score_tiktok_items(items):
"""Score TikTok items. Same weights as YouTube:
0.45*relevance + 0.25*recency + 0.30*engagement"""
```
### Phase 5: Deduplication + Cross-Source Linking
**`scripts/lib/dedupe.py`:**
```python
def dedupe_tiktok(items, threshold=0.7):
"""Dedupe TikTok items via Jaccard similarity on text + author_name."""
return dedupe_items(items, threshold)
```
- Text extraction for similarity: `text + author_name` (mirrors YouTube's `title + channel_name`)
- Add `tiktok` to `cross_source_link()` — compare TikTok items with all other sources
- Cross-ref prefix: `"TK"` (e.g., `TK3` for TikTok item 3)
### Phase 6: Rendering
**`scripts/lib/render.py` — add TikTok section:**
```markdown
### TikTok Videos
**TK1** (score:87) @creator_name (2026-02-28) [2.1M views, 45K likes]
Caption: "This Claude Code trick saved me hours... #claudecode #ai"
https://www.tiktok.com/@creator/video/1234567890
Spoken: "So I found this insane trick with Claude Code where you can..."
*TikTok: This Claude Code trick saved me hours*
```
**Stats line for SKILL.md:**
```
├─ 🎵 TikTok: {N} videos │ {N} views │ {N} with captions
```
### Phase 7: Environment + Config
**`scripts/lib/env.py` — add Apify support:**
```python
def is_apify_available(config: Dict[str, Any]) -> bool:
"""Check if Apify token is configured for TikTok/social scraping."""
return bool(config.get('APIFY_API_TOKEN'))
```
- New env var: `APIFY_API_TOKEN`
- Add to `get_config()` key list
- Add to `get_available_sources()` / `get_missing_keys()` logic
- Single token covers TikTok + future Facebook + Instagram
**User setup:**
```bash
# Add to ~/.config/last30days/.env
APIFY_API_TOKEN=apify_api_xxxxxxxxxxxxx
```
Or get free token: Sign up at https://console.apify.com → Settings → Integrations → Personal API Token.
### Phase 8: Orchestrator Integration
**`scripts/last30days.py` changes:**
1. Add `"tiktok"` to `VALID_SEARCH_SOURCES` set (line 47)
2. Add `tiktok_future` var + timeout to `TIMEOUT_PROFILES`:
```python
"tiktok_future": 120 # Apify actors can be slow on first run
```
3. Add `do_tiktok` bool + `run_tiktok` parameter to `run_research()`
4. Submit `_search_tiktok()` to ThreadPoolExecutor (now max 7+1 workers)
5. Collect TikTok results with timeout
6. Add tiktok to return tuple + progress display
7. Wire tiktok into normalize → score → dedupe → cross-link → render pipeline in main
### Phase 9: SKILL.md Updates
1. Add TikTok to stats box template
2. Add TikTok citation rule: `@creator on TikTok`
3. Add TikTok to source weight guidance (rank between YouTube and HN)
4. Document `APIFY_API_TOKEN` in setup section
### Phase 10: Dependency
```bash
pip install apify-client
```
- `apify-client` is the only new dependency
- Requires Python 3.10+ (already required by the project)
- No new binary dependencies (unlike yt-dlp for YouTube)
## Files to Create / Modify
### New Files
| File | Purpose |
|---|---|
| `scripts/lib/apify_client_wrapper.py` | Shared Apify client init + `run_actor_sync()` helper |
| `scripts/lib/tiktok.py` | TikTok search, caption extraction, relevance scoring |
| `tests/test_tiktok.py` | Unit tests for TikTok module |
| `fixtures/tiktok_search.json` | Mock Apify response for testing |
### Modified Files
| File | Changes |
|---|---|
| `scripts/lib/schema.py` | Add `TikTokItem` dataclass, `shares` to Engagement, `tiktok`/`tiktok_error` to Report |
| `scripts/lib/normalize.py` | Add `normalize_tiktok_items()` |
| `scripts/lib/score.py` | Add `compute_tiktok_engagement_raw()`, `score_tiktok_items()` |
| `scripts/lib/dedupe.py` | Add `dedupe_tiktok()`, add tiktok to `cross_source_link()` |
| `scripts/lib/render.py` | Add TikTok rendering section, stats line |
| `scripts/lib/env.py` | Add `APIFY_API_TOKEN` handling, `is_apify_available()` |
| `scripts/last30days.py` | Add tiktok to orchestrator pipeline, `VALID_SEARCH_SOURCES`, `TIMEOUT_PROFILES` |
| `SKILL.md` | Add TikTok stats line, citation rules, source weights |
| `README.md` | Add TikTok to source list, Apify setup instructions |
## Future: Facebook + Instagram via Apify
The `apify_client_wrapper.py` module is designed to be reused. Adding Facebook would look like:
```python
# scripts/lib/facebook.py
from . import apify_client_wrapper
ACTOR_ID = "apify/facebook-posts-scraper" # or "scraper_one/facebook-posts-search"
def search_facebook(topic, from_date, to_date, depth="default", token=None):
client = apify_client_wrapper.get_apify_client(token)
run_input = {
"searchType": "posts",
"searchTerms": [topic],
"maxPosts": DEPTH_CONFIG[depth]["max_posts"],
}
items = apify_client_wrapper.run_actor_sync(client, ACTOR_ID, run_input)
# Parse: text, likes, comments, shares, time, user.name, url
...
```
**Facebook fields available:** `text`, `likes`, `comments`, `shares`, `time`/`timestamp`, `user.name`, `url`, `reactions_count`
**Instagram** would follow the same pattern with `apify/instagram-scraper` or similar.
Same `APIFY_API_TOKEN` — no additional keys needed.
## Cost Analysis
**Per research run (default depth, 20 results):**
- Clockworks TikTok scraper: ~$0.10 per 20 results ($5/1000)
- Free tier: ~50 research runs per month on $5 free credits
- With captions (re-run for top 5): ~$0.15 total per run → ~33 runs/month free
**Comparison:** YouTube costs $0 (yt-dlp is free). TikTok costs ~$0.10-0.15/run. This is acceptable given the signal value and tracks with the BYO key model.
## Acceptance Criteria
- [ ] `APIFY_API_TOKEN` in `.env` enables TikTok source automatically
- [ ] TikTok appears in parallel search alongside other 6 sources
- [ ] Results include: video URL, caption, author, views, likes, comments, date
- [ ] Caption enrichment works for top N videos (configurable by depth)
- [ ] Relevance scoring filters off-topic viral videos
- [ ] Cross-source linking detects when TikTok + Reddit/YouTube discuss same topic
- [ ] Stats box shows: `🎵 TikTok: N videos │ N views │ N with captions`
- [ ] `--search=tiktok` flag works for TikTok-only research
- [ ] Graceful degradation: if no APIFY_API_TOKEN, TikTok silently skipped
- [ ] Mock mode works with `fixtures/tiktok_search.json`
- [ ] Tests pass for search, normalize, score, dedupe, render
- [ ] `apify_client_wrapper.py` is generic enough for Facebook/Instagram reuse
docs/plans/2026-03-03-fix-close-commented-github-issues-plan.md
-49
View File
@@ -1,49 +0,0 @@
---
title: Close Commented GitHub Issues and PRs
type: fix
status: active
date: 2026-03-03
origin: docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md
---
# Close Commented GitHub Issues and PRs
All 9 open issues and 2 open PRs on `mvanhorn/last30days-skill` were commented on (2026-03-03) with fix confirmations or responses, but none were actually closed on GitHub.
## Acceptance Criteria
- [ ] Close 6 issues where fixes were confirmed (#40, #39, #32, #30, #29, #4)
- [ ] Close PR #26 (superseded - features landed on main)
- [ ] Verify 3 issues remain open (#36, #31, #22) - acknowledged but unresolved
- [ ] Verify PR #24 remains open - explicitly left open per comment
## Actions
### Close as fixed (6 issues)
```bash
gh issue close 40 --repo mvanhorn/last30days-skill --reason completed
gh issue close 39 --repo mvanhorn/last30days-skill --reason completed
gh issue close 32 --repo mvanhorn/last30days-skill --reason completed
gh issue close 30 --repo mvanhorn/last30days-skill --reason completed
gh issue close 29 --repo mvanhorn/last30days-skill --reason completed
gh issue close 4 --repo mvanhorn/last30days-skill --reason completed
```
### Close superseded PR (#26)
```bash
gh pr close 26 --repo mvanhorn/last30days-skill
```
### Keep open (no action needed)
- #36 - SKILL.md flag forwarding (investigating)
- #31 - skills.sh audit scores (needs review)
- #22 - Bird feature requests (backlog)
- PR #24 - Codex compatibility (intentionally left open)
## Sources
- **Origin plan:** [docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md](2026-03-03-fix-triage-all-open-github-issues-plan.md)
- Commit: `5e5d586` - fix: triage all 16 open GitHub issues
docs/plans/2026-03-03-fix-skill-md-argument-flag-forwarding-plan.md
-111
View File
@@ -1,111 +0,0 @@
---
title: Fix SKILL.md Argument Flag Forwarding
type: fix
status: completed
date: 2026-03-03
---
# Fix SKILL.md Argument Flag Forwarding
## Overview
`"$ARGUMENTS"` in SKILL.md wraps the entire user input in double quotes, making argparse treat flags like `--store` as part of the topic string instead of CLI flags.
## Problem
SKILL.md line 168:
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
```
`$ARGUMENTS` is a Claude Code template variable replaced via string substitution before bash runs. The double quotes cause word-joining:
- User types: `/last30days AI video tools --store`
- Claude Code expands to: `python3 script.py "AI video tools --store" --emit=compact`
- argparse sees: `topic="AI video tools --store"`, `--store` never parsed
## Proposed Solution
Two coordinated changes:
### 1. Remove quotes around `$ARGUMENTS` in SKILL.md
**File:** `SKILL.md:168`
```bash
# Before:
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
# After:
python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --no-native-web
```
Now bash word-splits the expansion: `python3 script.py AI video tools --store --emit=compact`
### 2. Change argparse `topic` from `nargs="?"` to `nargs="*"`
**File:** `scripts/last30days.py:1040`
```python
# Before:
parser.add_argument("topic", nargs="?", help="Topic to research")
# args.topic = "AI video tools" (single string) or None
# After:
parser.add_argument("topic", nargs="*", help="Topic to research")
# args.topic = ["AI", "video", "tools"] (list) or []
```
Then immediately after `parser.parse_args()` (line 1124), join the list back to a string:
```python
args = parser.parse_args()
args.topic = " ".join(args.topic) if args.topic else None
```
**Why this works for both invocation styles:**
| Invocation | argparse receives | topic result |
|---|---|---|
| `script.py AI video tools --store` (Claude Code) | `["AI", "video", "tools"]` + `--store` | `"AI video tools"` |
| `script.py "AI video tools" --store` (direct CLI) | `["AI video tools"]` + `--store` | `"AI video tools"` |
| `script.py --store` (no topic) | `[]` + `--store` | `None` |
### 3. Document missing flags in SKILL.md Options section
**File:** `SKILL.md:223-227`
Add after the existing `--deep` line:
```
- `--store` -> Persist findings to SQLite database for later querying
- `--search=SOURCES` -> Comma-separated source filter (e.g., `--search=reddit,hn`)
- `--include-web` -> Include general web search alongside primary sources
- `--diagnose` -> Show source availability diagnostics and exit
- `--timeout=SECS` -> Global timeout in seconds (default: 180, quick: 90, deep: 300)
```
Note: `--sort-x` was listed in the issue but does not exist in the Python argparse. Skip it.
## Acceptance Criteria
- [x] `/last30days AI video tools --store` correctly passes `--store` to Python script
- [x] `/last30days AI video tools` still works (multi-word topic without flags)
- [x] Direct CLI: `python3 last30days.py "AI video tools" --store` still works
- [x] `--diagnose`, `--search=reddit,hn`, `--timeout=120` all forward correctly
- [x] All 5 missing flags documented in SKILL.md Options section
- [x] Existing tests pass (`python3 -m pytest tests/`)
## Files Changed
| File | Change |
|---|---|
| `SKILL.md:168` | Remove quotes around `$ARGUMENTS` |
| `scripts/last30days.py:1040` | `nargs="?"` -> `nargs="*"` |
| `scripts/last30days.py:1124` | Add `args.topic = " ".join(args.topic) if args.topic else None` |
| `SKILL.md:223-227` | Add 5 missing flags to Options section |
## Sources
- GitHub issue: https://github.com/mvanhorn/last30days-skill/issues/36
- Reporter: @nicolefinateri
docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md
-479
View File
@@ -1,479 +0,0 @@
# Triage All Open GitHub Issues — Proposed Actions
**Date:** 2026-03-03
**Repo:** `mvanhorn/last30days-skill`
**Open issues:** 16 (as of this triage)
**Codebase version:** v2.1.0 (Bird vendored, 7 sources: Reddit, X, YouTube, HN, Polymarket, TikTok, Web)
---
## Legend
| Action | Meaning |
|--------|---------|
| **CLOSE (spam)** | Spam / solicitation — close without comment |
| **CLOSE (resolved)** | Already fixed in current version — comment & close |
| **CLOSE (duplicate)** | Duplicate of another issue — link & close |
| **COMMENT** | Respond with info, no code change needed |
| **FIX** | Code or docs change needed — described below |
| **REJECT** | Valid issue but won't address — explain why |
---
## Issue-by-Issue Triage
### #43 — Security Assessment Offer - Claude Code Skill with Social Media APIs
**Author:** Neo-Assistent | **Created:** 2026-02-26
> Unsolicited marketing from "SkillSec" offering a free security audit.
**Proposed action:** **CLOSE (spam)**
No comment needed. This is a cold sales pitch, not a bug or feature request. Close silently or with a brief "closing — not a bug report or feature request."
---
### #42 — License & Secondary Development Permission
**Author:** AllenX95 | **Created:** 2026-02-26
> Asks what license the project uses and whether modification for open-source models (Kimi K2.5) is allowed.
**Proposed action:** **COMMENT + CLOSE (resolved)**
Once #35/#34 are addressed (LICENSE file added), comment:
> The project is MIT licensed (see LICENSE file in repo root, added in v2.1.x). You're free to modify, fork, and adapt it for any purpose including integration with other models. The MIT license places no restrictions on secondary development.
**Depends on:** Adding the LICENSE file (see #35/#34 below).
---
### #41 — Can I use openrouter API to replace with openai API to do the reddit search?
**Author:** iklynow-hue | **Created:** 2026-02-25
> Asks about using OpenRouter instead of OpenAI for Reddit search. Has one off-topic spam comment from @gbessoni.
**Proposed action:** **COMMENT + CLOSE**
Comment:
> OpenRouter is already partially supported — `scripts/lib/openrouter_search.py` provides a Perplexity-via-OpenRouter integration for web search. However, the Reddit search specifically uses OpenAI's Responses API (`web_search_preview` tool), which is an OpenAI-specific feature that OpenRouter doesn't proxy.
>
> If you want to avoid OpenAI entirely, you can still get results from X (via vendored Bird search or xAI), YouTube (yt-dlp), Hacker News (free API), and Polymarket (free API) — no OpenAI key needed for those. Reddit is the one source that requires `OPENAI_API_KEY`.
>
> For budget control, consider using `--quick` mode which reduces API calls.
Also: **minimize/hide** the off-topic spam comment from @gbessoni ("I'm testing out a new idea for reddit called Redd").
---
### #40 — watchlist: search_queries field stored but never used by _run_topic()
**Author:** tejasgadhia | **Created:** 2026-02-23
> `add --queries "q1,q2,q3"` stores queries in DB but `_run_topic()` always passes `topic["name"]` as the search query. Long descriptive names produce poor results on X and web search.
**Proposed action:** **FIX (accept)**
This is a legitimate, well-documented bug. The fix is straightforward:
**File:** `scripts/watchlist.py` — `_run_topic()` (~line 136)
```python
# Current (broken):
cmd = [sys.executable, str(SCRIPT_DIR / "last30days.py"), topic["name"], "--emit=json"]
# Fix: prefer search_queries when available
import json as _json
queries = _json.loads(topic["search_queries"]) if topic.get("search_queries") else [topic["name"]]
search_term = queries[0] if queries else topic["name"]
cmd = [sys.executable, str(SCRIPT_DIR / "last30days.py"), search_term, "--emit=json"]
```
**Comment to post:**
> Good catch — you're right that `search_queries` is stored but never read. Will fix `_run_topic()` to prefer `search_queries[0]` over `topic["name"]` when available.
---
### #39 — watchlist.py: YouTube findings never stored + run-one silent output
**Author:** tejasgadhia | **Created:** 2026-02-23
> **Bug 1:** YouTube findings extracted by the research script but dropped in `_run_topic()` because there's no YouTube loop in the findings extraction.
> **Bug 2:** `cmd_run_one()` doesn't print the result (cosmetic).
**Proposed action:** **FIX (accept both)**
Both are real bugs with clear fixes provided by the reporter.
**Bug 1 fix** — `scripts/watchlist.py` findings extraction (~lines 170-193): add YouTube + TikTok loops:
```python
for item in data.get("youtube", []):
findings.append({
"source": "youtube",
"url": item.get("url", ""),
"title": item.get("title", ""),
"author": item.get("channel_name", item.get("channel", "")),
"content": item.get("transcript_snippet", "") or item.get("title", ""),
"engagement_score": (item.get("engagement") or {}).get("views", 0),
"relevance_score": item.get("relevance", 0),
})
for item in data.get("tiktok", []):
findings.append({
"source": "tiktok",
"url": item.get("url", ""),
"title": item.get("caption_snippet", "")[:120],
"author": item.get("author", ""),
"content": item.get("caption_snippet", ""),
"engagement_score": (item.get("engagement") or {}).get("views", 0),
"relevance_score": item.get("relevance", 0),
})
```
**Bug 2 fix** — `scripts/watchlist.py` `cmd_run_one()`:
```python
result = _run_topic(topic)
print(json.dumps(result, default=str))
```
**Comment to post:**
> Both confirmed. Fixing YouTube (and adding TikTok while we're at it) findings extraction, plus `run-one` output. Thanks for the detailed report.
---
### #36 — SKILL.md doesn't forward --store, --include-web, --sort-x, --diagnose, --timeout to script
**Author:** nicolefinateri | **Created:** 2026-02-21
> `"$ARGUMENTS"` is passed as a single quoted string, so argparse treats the whole thing as the topic positional. Flags like `--store` get swallowed. Also, 5 flags are undocumented in SKILL.md.
**Proposed action:** **COMMENT + CLOSE (resolved / won't fix)**
This needs investigation. The `"$ARGUMENTS"` expansion in SKILL.md (line 168) relies on Claude Code's variable expansion behavior. If Claude Code expands `$ARGUMENTS` before the shell sees it, the quoting around it means all words become one argument. However, in practice, Claude Code injects the literal text into the bash script, so `"$ARGUMENTS"` expands to `"best AI tools --store"` which the shell passes as a single string to Python — and Python's argparse sees it as one positional arg.
**However:** Looking at the actual SKILL.md line 168:
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
```
This is a known Claude Code behavior — `$ARGUMENTS` is expanded by the template engine, not the shell. The expansion produces the raw text, so the double quotes cause everything to be one argument.
**The real fix:** Remove the quotes around `$ARGUMENTS`:
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact
```
This lets word-splitting happen so argparse sees separate positional + flag args.
**Risk:** Topic names with spaces would break (e.g., `AI video tools` would become 3 separate args). Need to verify how Claude Code handles `$ARGUMENTS` expansion. If it's truly a template variable replaced before bash execution, then the fix is more nuanced — may need to parse flags out in bash first.
**Comment to post:**
> Thanks for the detailed report. The `$ARGUMENTS` quoting is tricky — it's a Claude Code template variable, not a shell variable. I'll investigate the exact expansion behavior and fix the forwarding. Re: undocumented flags — will add them to the Options section.
**Priority:** Medium — flags like `--store` and `--diagnose` are useful but not critical path.
---
### #35 — Add LICENSE file to repository
**Author:** HackJob7418 | **Created:** 2026-02-21
> Points out that `plugin.json` declares MIT but no LICENSE file exists in repo root.
**Proposed action:** **FIX (accept)**
Trivial fix. Add `LICENSE` file with standard MIT text to repo root.
**Comment to post:**
> You're right — added MIT LICENSE file to repo root. Thanks for catching the mismatch.
---
### #34 — License missing
**Author:** jdsika (Carlo van Driesten) | **Created:** 2026-02-21
> Same request as #35 — add a LICENSE file.
**Proposed action:** **CLOSE (duplicate of #35)**
**Comment to post:**
> Duplicate of #35 — adding MIT LICENSE file. Thanks for the nudge!
---
### #32 — Cannot install Claude Code plugin marketplace
**Author:** rayshan (Ray Shan) | **Created:** 2026-02-19
> `marketplace.json` schema validation fails: `plugins.0.source: Invalid input`
**Proposed action:** **FIX (investigate + fix)**
The current `marketplace.json` has:
```json
"plugins": [{"name": "last30days", "source": "."}]
```
The Anthropic marketplace schema may require `source` to be a URL or a path in a specific format, not bare `.`. Need to check the expected schema.
**Possible fix:** Update `source` to match whatever the skills.sh/marketplace validator expects. Likely needs to be a relative path to the SKILL.md or plugin directory:
```json
"plugins": [{"name": "last30days", "source": "./SKILL.md"}]
```
Or the schema may have changed since the file was written. Check skills.sh docs.
**Comment to post:**
> Thanks for the report + screenshot. The marketplace.json schema likely changed — I'll update the `source` field format. Can you share what version of the `skills` CLI you're using? (`npx skills --version`)
**Priority:** High — blocks installation for marketplace users.
---
### #31 — mvanhorn/last30days is being trashed on skills.sh
**Author:** PiotrAleksander (Piotr Mrzygłosz) | **Created:** 2026-02-19
> skills.sh security audit gives the skill bad scores. A mirror at `sickn33/antigravity-awesome-skills` has better scores.
**Proposed action:** **COMMENT + investigate**
This is a reputation/trust issue. The skills.sh audit likely flags things like:
- No LICENSE file (fixed by #35)
- Shell execution in SKILL.md (inherent to how the skill works)
- External API calls (Reddit, X, YouTube — core functionality)
- No pinned dependencies
**Comment to post:**
> Thanks for flagging this. A few things:
>
> 1. The missing LICENSE file (#35) is being added — that should help the audit score.
> 2. The skill necessarily makes external API calls (that's its core purpose — researching across platforms). Any "security warning" about API calls is expected behavior, not a vulnerability.
> 3. The mirror you found (`sickn33/antigravity-awesome-skills`) may be an older fork (v1) that doesn't include the more recent integrations. Simpler code = fewer audit flags, but also fewer features.
> 4. I'll review the specific audit findings on skills.sh and address any legitimate concerns.
>
> If you can share the specific warnings you're seeing, I can address them directly.
**Priority:** Medium — reputation matters but the core issue is likely the missing LICENSE + inherent design of making API calls.
---
### #30 — Bird cookie auth: source availability mapping misses reddit-web + Bird combination
**Author:** volarian-vai | **Created:** 2026-02-19
> When using Bird cookie auth without `XAI_API_KEY` but with a web search key (e.g., `BRAVE_API_KEY`), the source override logic misses the `reddit-web` → `all` mapping.
**Proposed action:** **FIX (accept)**
Legitimate edge case in source availability logic. The fix is a small addition to the Bird override block in `scripts/last30days.py` (~line 917):
```python
if x_source == 'bird':
if available == 'reddit':
available = 'both'
elif available == 'reddit-web':
available = 'all'
elif available == 'web':
available = 'x-web'
```
**Comment to post:**
> Good catch on the missing `reddit-web` case. The fix is straightforward — adding the mapping. Also fixing `web` → `x-web` as you noted.
**Priority:** Low-medium — affects a specific env var combination (Bird auth + no XAI key + Brave key).
---
### #29 — YouTube stats show 'yt-dlp not installed' when search returns 0 results
**Author:** alexkrivov | **Created:** 2026-02-19
> When yt-dlp runs successfully but returns 0 results, the stats footer says "yt-dlp not installed" because `report.youtube` is `[]` (falsy) and the fallback message is wrong.
**Proposed action:** **FIX (accept)**
Clear bug with a clear root cause. Two-part fix:
**1.** In `scripts/last30days.py` (~line 1085), set a proper skip reason when YouTube returns 0:
```python
if has_ytdlp and not yt_results:
source_info["youtube_skip_reason"] = "0 results (query may be too specific)"
```
**2.** In `scripts/lib/render.py` (~line 288), change the default fallback:
```python
reason = source_info.get("youtube_skip_reason", "not available")
```
**Comment to post:**
> Confirmed — the `[]`-is-falsy bug plus a misleading default string. Will fix both the skip reason and the fallback. Thanks for the detailed trace.
**Priority:** Low — cosmetic/UX but misleading for users trying to debug setup.
---
### #22 — Feature request: Surface more Bird CLI capabilities (engagement sorting, time windows, thread following)
**Author:** nicolefinateri | **Created:** 2026-02-09
> Requests: (1) `--sort-x likes` flag, (2) `--since 3h` granular time windows, (3) thread following via `bird thread <id>`.
**Proposed action:** **COMMENT (partial accept, defer)**
Good feature requests but some are already partially addressed:
1. **Engagement sorting** — The `--sort-x=MODE` flag already exists in `last30days.py` argparse (line ~1080) but isn't documented in SKILL.md (related to #36). Sorting by `likes`, `engagement`, `recent`, or `score` is already implemented.
2. **Granular time windows** — The `--days` flag exists (1-30), but hour-level granularity (`--since 3h`) would be new. Low priority since most research use cases are days-scale.
3. **Thread following** — Would be a significant feature addition. The vendored Bird search in v2.1 may not expose thread fetching. Would need investigation.
**Comment to post:**
> Great ideas! Update on each:
>
> 1. **Engagement sorting:** `--sort-x=MODE` already exists in the script (likes, engagement, recent, score). It wasn't documented in SKILL.md — fixing that (#36). Should work if you call the Python script directly.
> 2. **Granular time windows:** Interesting idea. Currently `--days=N` goes down to 1 day. Hour-level would need a new flag and Bird-specific query modification. Adding to backlog.
> 3. **Thread following:** Love this in principle — high-engagement thread starters are gold. Would need to assess what the vendored Bird search supports. May be a v2.2+ feature.
>
> A PR for any of these would be welcome!
**Priority:** Low — nice-to-haves; #1 is already done, #2/#3 are backlog.
---
### #19 — Bird is missing
**Author:** brianjking (Brian J King) | **Created:** 2026-02-09
> Bird GitHub repo is gone. Comments confirm npm package is deprecated, Homebrew install fails. Multiple users affected.
**Proposed action:** **CLOSE (resolved in v2.1)**
This was a major pain point that drove the v2.1 vendoring decision. Bird's GraphQL client is now bundled directly in the repo at `scripts/lib/vendor/bird-search/`. No external npm install needed.
**Comment to post:**
> **Resolved in v2.1.0** (released Feb 15, 2026).
>
> Bird's Twitter GraphQL search client is now vendored directly into the skill at `scripts/lib/vendor/bird-search/`. No npm install, no Homebrew, no external dependency. Just needs Node.js 22+ in your PATH.
>
> If you're on an older version, update to v2.1:
> ```
> npx skills add https://github.com/mvanhorn/last30days-skill
> ```
>
> Authentication still works via Safari cookies (auto-detected) or `AUTH_TOKEN`/`CT0` environment variables. See README for setup.
>
> Thanks to everyone who reported this — it's what motivated bundling the search client directly.
---
### #4 — Add macOS Python SSL certificate prerequisite to README
**Author:** joshdaws (Josh Daws) | **Created:** 2026-01-27
> Python.org macOS installer doesn't include SSL certificates. Users get `CERTIFICATE_VERIFY_FAILED` errors on all API calls.
**Proposed action:** **FIX (accept — docs change)**
Add a troubleshooting section to README.md. The reporter's suggested text is good. This only affects python.org installs (not Homebrew), but it's a confusing error when hit.
**Comment to post:**
> Good call — adding a troubleshooting section to the README for this. The `certifi` fallback is interesting but adds a dependency; the docs fix is simpler and sufficient since Homebrew Python (the most common Claude Code setup) isn't affected.
**Priority:** Low — affects a subset of macOS users, but easy to fix with docs.
---
### #2 — npx skills add fails for this package
**Author:** mikecfisher (Mike Fisher) | **Created:** 2026-01-26
> `npx skills add` can't find any skills in the repo. Owner commented that PR #1 restructures to standard plugin format.
**Proposed action:** **CLOSE (resolved)**
The repo was restructured to standard plugin format in PR #1 (SKILL.md at root, .claude-plugin/ directory). The `skills` CLI should now detect the skill correctly. If #32's marketplace.json schema issue is also fixed, this should be fully resolved.
**Comment to post:**
> This was fixed in the repo restructuring (PR #1, merged Jan 27). The skill now has a proper `SKILL.md` at root + `.claude-plugin/` directory.
>
> If you're still having trouble, it may be the marketplace.json schema issue tracked in #32. Please try again and reopen if the problem persists.
---
## Summary Table
| # | Title | Action | Priority | Effort |
|---|-------|--------|----------|--------|
| **43** | Security Assessment Offer | **CLOSE (spam)** | — | None |
| **42** | License & Secondary Dev Permission | **COMMENT + CLOSE** | Low | None (after #35) |
| **41** | OpenRouter API replacement | **COMMENT + CLOSE** | Low | None |
| **40** | watchlist: search_queries unused | **FIX** | Medium | Small (~10 lines) |
| **39** | watchlist: YouTube not stored + silent run-one | **FIX** | Medium | Small (~25 lines) |
| **36** | SKILL.md flag forwarding | **COMMENT + investigate** | Medium | Medium (SKILL.md rewrite) |
| **35** | Add LICENSE file | **FIX** | High | Trivial (1 file) |
| **34** | License missing | **CLOSE (dup of #35)** | — | None |
| **32** | marketplace.json schema error | **FIX** | High | Small (schema update) |
| **31** | skills.sh bad audit scores | **COMMENT** | Medium | None (after #35) |
| **30** | Bird source mapping edge case | **FIX** | Low | Small (~3 lines) |
| **29** | YouTube "not installed" false message | **FIX** | Low | Small (~5 lines) |
| **22** | Bird feature requests | **COMMENT (partial accept)** | Low | None (backlog) |
| **19** | Bird is missing | **CLOSE (resolved v2.1)** | — | None |
| **4** | macOS SSL certificate docs | **FIX (docs)** | Low | Small (README section) |
| **2** | npx skills add fails | **CLOSE (resolved)** | — | None |
---
## Recommended Execution Order
### Batch 1 — Quick wins (close/comment only, no code)
1. Close #43 (spam)
2. Close #34 (dup of #35)
3. Close #19 (resolved in v2.1)
4. Close #2 (resolved by repo restructure)
5. Comment + close #41 (OpenRouter question)
6. Comment #22 (Bird feature requests — partial accept, backlog)
7. Comment #31 (skills.sh audit — will improve after LICENSE fix)
### Batch 2 — Trivial fixes
8. **#35** — Add MIT LICENSE file to repo root
9. Comment + close #42 (license question — now answered by LICENSE file)
### Batch 3 — Small code fixes
10. **#29** — Fix YouTube "not installed" false message (`render.py` + `last30days.py`)
11. **#30** — Fix Bird source mapping for `reddit-web` combo (`last30days.py`)
12. **#40** — Fix watchlist `search_queries` usage (`watchlist.py`)
13. **#39** — Fix watchlist YouTube/TikTok extraction + `run-one` output (`watchlist.py`)
### Batch 4 — Investigation needed
14. **#32** — Fix marketplace.json schema (needs schema research)
15. **#36** — Fix `$ARGUMENTS` flag forwarding (needs Claude Code expansion testing)
16. **#4** — Add SSL troubleshooting to README
---
## Proposed Comment Templates
### For spam (#43):
> Closing — this is not a bug report or feature request.
### For duplicates (#34):
> Duplicate of #35. Adding MIT LICENSE file — thanks!
### For resolved issues (#19, #2):
> Resolved in v2.1.0. [details specific to issue]. Closing — please reopen if you're still experiencing this on the latest version.
### For questions (#41, #42):
> [Answer the question]. Closing as answered — feel free to reopen if you have follow-ups.
### For accepted bugs (#29, #30, #39, #40):
> Confirmed — [brief acknowledgment]. Fix incoming. Thanks for the detailed report.
docs/plans/2026-03-03-refactor-tiktok-apify-alternative-plan.md
-156
View File
@@ -1,156 +0,0 @@
# refactor: Replace Apify with pay-as-you-go TikTok API
**Type:** refactor
**Date:** 2026-03-03
**Status:** Draft
## Problem
Apify requires a monthly subscription even for low-volume usage. We need a true pay-as-you-go TikTok data API that returns structured video data (views, likes, comments, shares, author, hashtags, captions) from keyword search.
## Current Architecture
Our TikTok integration makes **exactly 2 API calls per /last30days invocation**:
1. **Search call** — keyword search, returns 10-40 videos with engagement metrics
2. **Caption enrichment call** — fetches spoken-word subtitles for top 3-8 videos
We consume these fields from the response:
- `id`, `text` (description), `webVideoUrl`, `authorMeta.name`
- `playCount`, `diggCount`, `commentCount`, `shareCount`
- `hashtags[].name`, `videoMeta.duration`
- `createTimeISO` or `createTime` (date)
- `subtitleText` / `subtitles` (captions, secondary call)
Files involved:
- `scripts/lib/tiktok.py` — search, parse, relevance scoring, caption fetching
- `scripts/lib/apify_client_wrapper.py` — shared Apify client (also designed for future FB/IG)
- `scripts/lib/schema.py` — `TikTokItem` dataclass
- `scripts/lib/normalize.py` — `normalize_tiktok()`
- `scripts/last30days.py` — orchestrator (calls `tiktok.search_and_enrich()`)
## Why Most of Those Alternatives Won't Work
The services in the user's table (Spider, Zyte, Scrappey, Serper.dev) are **general web scrapers** — they return raw HTML, not structured TikTok data. We'd have to build our own HTML parser, handle anti-bot protections, and reverse-engineer TikTok's response format. That's a completely different (and fragile) approach.
What we need is a **TikTok-specific API** that returns structured JSON with engagement metrics from keyword search.
## Alternatives Evaluated
### RECOMMENDED: ScrapeCreators — Best True Pay-As-You-Go
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Yes — 19 dedicated endpoints including keyword search |
| **Pricing model** | True PAYG — buy credits, credits never expire |
| **Cost at our volume** | ~$0.60/month ($10 buys 5,000 credits, lasts 16-33 months) |
| **Free tier** | 100-10,000 free credits on signup (no credit card) |
| **Python SDK** | No dedicated SDK — simple REST API (`requests.get()`) |
| **Search endpoint** | "Search by Keyword" and "Top Search" |
| **Risk** | Newer service, limited track record |
**Why it's the best fit:** $10 literally lasts over a year at our volume. No subscription, no expiring credits. The lack of a Python SDK is irrelevant — it's a single `requests.get()` call.
### Runner-up: EnsembleData — Best SDK, But Subscription
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Full — 15+ endpoints, all engagement metrics |
| **Pricing model** | Monthly subscription ($100/mo after 7-day trial) |
| **Cost at our volume** | $0 during trial (50 units/day), $100/mo after |
| **Free tier** | 50 units/day for 7 days, no CC required |
| **Python SDK** | Yes — `pip install ensembledata` |
| **Search endpoint** | "Keyword Search" returns ~20 posts/call (1 unit) |
| **Risk** | $100/mo is overkill for 5-10 searches/day |
**Verdict:** Best data quality and SDK, but $100/mo is absurd for our ~150-300 requests/month. Same subscription problem as Apify.
### Backup: tikwm.com — Free but Risky
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Likely yes (needs live testing) |
| **Pricing model** | Completely free, no API key required |
| **Cost at our volume** | $0 |
| **Free tier** | 5,000 requests/day |
| **Python SDK** | Community wrappers (damirTAG/TikTok-Module, kittenbark/tikwm) |
| **Search endpoint** | `https://www.tikwm.com/api/feed/search?keywords=TERM&count=20` |
| **Risk** | Unaffiliated third-party, could disappear anytime, no SLA |
**Verdict:** Great for development/testing. Too risky as sole production backend. Could be a zero-cost fallback.
### Not Recommended
| Service | Why Not |
|---------|---------|
| **TikAPI** | $50-189/mo subscription — same problem as Apify |
| **Bright Data** | $499/mo minimum — enterprise pricing |
| **davidteather/TikTok-Api** | Video search is broken, requires Playwright, fragile |
| **Spider, Zyte, Scrappey** | Generic scrapers — return raw HTML, no TikTok structure |
| **Piloterr** | No TikTok endpoints, subscription only |
## Recommended Approach
### Option A: ScrapeCreators as primary (Recommended)
- [ ] Sign up for ScrapeCreators, get free credits
- [ ] Test the "Search by Keyword" endpoint to verify it returns all required fields
- [ ] Refactor `tiktok.py` to use ScrapeCreators REST API instead of Apify actor
- [ ] Keep `apify_client_wrapper.py` for future FB/IG (or refactor to generic wrapper)
- [ ] Update `.env` config: `SCRAPECREATORS_API_KEY` replaces `APIFY_API_TOKEN` for TikTok
- [ ] Update README, SKILL.md installation instructions
- [ ] Buy $10 credits after confirming it works
### Option B: tikwm.com as primary (Zero cost, higher risk)
- [ ] Test tikwm.com search endpoint to verify response schema
- [ ] If it returns engagement metrics, implement as primary backend
- [ ] Add ScrapeCreators as paid fallback when tikwm fails
- [ ] No API key required — simplest user setup
### Option C: Keep Apify, document the subscription requirement
- [ ] Update README to clarify Apify requires a paid plan
- [ ] Add note about the free $5/mo credits tier (if it still works without subscription)
- [ ] Ship as-is with clear billing expectations
## Implementation Plan (Option A)
### Phase 1: Validate ScrapeCreators API
- [ ] Sign up and get API key
- [ ] Test keyword search endpoint: `GET /tiktok/search?keyword={topic}&count=20`
- [ ] Verify response contains: video ID, play count, likes, comments, shares, author, hashtags, date, description
- [ ] Test caption/subtitle availability (or confirm description text is sufficient)
### Phase 2: Swap the Backend
- [ ] Create `scripts/lib/scrapecreators_client.py` (simple REST wrapper, ~40 lines)
- [ ] Refactor `tiktok.py:search_tiktok()` to call ScrapeCreators instead of Apify
- [ ] Map ScrapeCreators response fields to our existing item dict format
- [ ] Refactor `tiktok.py:fetch_captions()` — check if ScrapeCreators provides subtitles, else use description text only
- [ ] Update `env.py` to read `SCRAPECREATORS_API_KEY` (keep `APIFY_API_TOKEN` for backward compat)
- [ ] Update `scripts/lib/ui.py` spinner messages if needed
### Phase 3: Update Docs & Ship
- [ ] Update README.md installation section (new API key)
- [ ] Update SKILL.md
- [ ] Update `~/.config/last30days/.env` locally
- [ ] Run full test suite
- [ ] Commit and push
## Acceptance Criteria
- [ ] TikTok search returns structured data with views, likes, comments, shares
- [ ] No subscription required — pay-as-you-go only
- [ ] Cost < $1/month at normal usage (5-10 searches/day)
- [ ] Existing test suite passes with new backend
- [ ] Graceful degradation when API key is missing (same behavior as today)
## Open Questions
1. Does ScrapeCreators' keyword search support date filtering, or do we filter post-API like we do with Apify?
2. Does ScrapeCreators return subtitle/caption data, or just video descriptions?
3. What's the response time? Apify actors took 30-120 seconds. REST APIs should be faster.
4. Should we keep Apify as a fallback backend (user configures one or the other)?
docs/plans/2026-03-04-feat-instagram-facebook-source-plan.md
-210
View File
@@ -1,210 +0,0 @@
# feat: Add Instagram and Facebook Sources via ScrapeCreators API
**Date:** 2026-03-04
**Type:** Enhancement
**Priority:** Instagram first, Facebook conditional ("if it's good")
## Summary
Add Instagram Reels and Facebook as new research sources in last30days, using the same ScrapeCreators REST API already powering TikTok. Instagram is the primary target; Facebook is a follow-on if the pattern works well.
Both sources share the existing `SCRAPECREATORS_API_KEY` — no new API keys needed.
## Approach
Replicate the TikTok integration pattern exactly. Each source follows the same 8-step pipeline:
```
tiktok.py pattern → instagram.py (new) → facebook.py (new, conditional)
```
## ScrapeCreators API Endpoints
### Instagram
| Endpoint | Path | Params | Credits | Notes |
|----------|------|--------|---------|-------|
| **Search Reels** | `GET /v1/instagram/reels/search` | `keyword`, pagination | 1 per 10 reels, max 60/req | Keyword search via Google (IG search requires login). V2 also available. |
| **Transcript** | `GET /v2/instagram/media/transcript` | `url` | 1 | Returns `{transcripts: [{id, shortcode, text}]}`. Videos <2min only. |
| **Comments** | `GET /v2/instagram/post/comments` | `url`, `cursor` | 1 | Returns `{comments: [{id, text, created_at, user}]}`. 100-300 per call. |
| **User Reels** | `GET /v1/instagram/user/reels` | `handle` or `user_id`, `max_id`, `trim` | 1 | All reels from a profile. Response: `{items: [...], paging_info}` |
**Primary search strategy:** `/v1/instagram/reels/search` with keyword param for topic search. This is the analog to TikTok's `/search/keyword`.
**Response fields per reel item:**
- `pk` / `code` (shortcode) — reel ID
- `taken_at` — unix timestamp
- `play_count` / `ig_play_count` — views
- `like_count` — likes
- `comment_count` — comments
- `video_duration` — seconds
- `has_audio` — boolean
- `user` object — username, full_name, is_verified, profile_pic_url
- `caption` object — text content
- Media URLs for thumbnails and video versions
### Facebook
| Endpoint | Path | Params | Credits | Notes |
|----------|------|--------|---------|-------|
| **Profile Posts** | `GET /v1/facebook/profile/posts` | `url` or `pageId`, `cursor` | 1 | Returns 3 posts at a time with engagement |
| **Profile Reels** | `GET /v1/facebook/profile/reels` | similar | 1 | 10 reels at a time |
| **Post** | `GET /v1/facebook/post` | `url` | 1 | Single post/reel by URL |
| **Transcript** | `GET /v1/facebook/post/transcript` | `url` | 1 | Video transcript, <2min |
| **Comments** | `GET /v1/facebook/post/comments` | `url`, `feedback_id` | 1 | Post/reel comments |
**Facebook limitation:** No keyword search endpoint. Only profile-based scraping (3 posts at a time). This makes Facebook significantly less useful for topic-based research vs. Instagram's keyword search.
**Response fields per post:**
- `id` — post ID
- `text` — post content
- `url` / `permalink` — post URL
- `author` — `{name, short_name, id}`
- `reactionCount` — total reactions
- `commentCount` — comments
- `videoViewCount` — video views (if applicable)
- `publishTime` — unix timestamp
- `topComments` — array of `{id, text, publishTime, author}`
## Implementation Plan
### Phase 1: Instagram Source (Primary)
#### 1.1 Create `scripts/lib/instagram.py`
- [x] Copy structure from `scripts/lib/tiktok.py`
- [x] Change `SCRAPECREATORS_BASE` to `"https://api.scrapecreators.com"`
- [x] Implement `search_instagram()` → calls `/v1/instagram/reels/search`
- Params: `keyword=core_topic`
- Parse response `items` array
- Extract: `pk`/`code` as video_id, `taken_at` as date, `play_count`/`like_count`/`comment_count` as engagement, `user.username` as author, `caption.text` as text
- Build URL: `https://www.instagram.com/reel/{code}`
- Reuse `_extract_core_subject()`, `_compute_relevance()`, `_tokenize()` from tiktok.py (or factor into shared util)
- Apply date range filter, sort by views descending
- [x] Implement `fetch_captions()` → calls `/v2/instagram/media/transcript`
- For top N items (per depth config), fetch transcript
- Response: `{transcripts: [{id, shortcode, text}]}`
- Fallback to caption text if transcript unavailable
- Truncate to 500 words
- [x] Implement `search_and_enrich()` → combines search + captions
- [x] Implement `parse_instagram_response()` → returns `response.get("items", [])`
- [x] Reuse shared helpers: `_sc_headers()`, `_log()`, `_clean_webvtt()`, `DEPTH_CONFIG`, `STOPWORDS`, `SYNONYMS`
**Key difference from TikTok:** Instagram response uses `play_count`/`like_count`/`comment_count` directly (no `statistics` wrapper), `user.username` (not `author.unique_id`), `caption.text` (not `desc`), `taken_at` (not `create_time`), `code` shortcode for URL construction.
#### 1.2 Add `InstagramItem` to `scripts/lib/schema.py`
- [x] Add dataclass mirroring `TikTokItem` structure:
```python
@dataclass
class InstagramItem:
id: str # "IG1", "IG2", ...
text: str # caption text
url: str # https://www.instagram.com/reel/{code}
author_name: str # Instagram handle
date: Optional[str] # YYYY-MM-DD from taken_at
date_confidence: str # "high"
engagement: Optional[Engagement] # views, likes, num_comments
caption_snippet: str # transcript or caption text
hashtags: List[str] # extracted from caption
relevance: float
why_relevant: str
subs: SubScores
score: int
cross_refs: List[str]
```
#### 1.3 Add normalization to `scripts/lib/normalize.py`
- [x] Add `normalize_instagram_items()` function
- Assign IDs as `IG1`, `IG2`, ...
- Map engagement: `views=play_count`, `likes=like_count`, `num_comments=comment_count`
- Set `date_confidence="high"` (unix timestamp)
#### 1.4 Add scoring to `scripts/lib/score.py`
- [x] Add `compute_instagram_engagement_raw()` — same formula as TikTok:
`0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)`
Views dominate on Instagram Reels just like TikTok.
- [x] Add `score_instagram_items()` — same weights: 45% relevance, 25% recency, 30% engagement
#### 1.5 Add dedup to `scripts/lib/dedupe.py`
- [x] Add `dedupe_instagram()` — same as `dedupe_tiktok()`, calls `dedupe_items()` with 0.7 threshold
- [x] Update `get_item_text()` to handle `InstagramItem`
- [x] Update `_get_cross_source_text()` for cross-source linking
- [x] Add `IG` prefix to cross-ref detection in `cross_source_link()`
#### 1.6 Add rendering to `scripts/lib/render.py`
- [x] Add Instagram section in `render_compact()` — same pattern as TikTok block (lines 251-285)
- Show: score, @author, date, views/likes, caption snippet, hashtags, why_relevant
- [x] Update data freshness check to include `instagram_recent`
- [x] Update stats footer to include Instagram count
- [x] Add `'IG'` to cross-ref source name mapping
#### 1.7 Add `Report.instagram` field to `scripts/lib/schema.py`
- [x] Add `instagram: List[InstagramItem]` and `instagram_error: str` to `Report` dataclass
#### 1.8 Integrate into `scripts/last30days.py` orchestrator
- [x] Add `"instagram"` to `VALID_SEARCH_SOURCES`
- [x] Add `import` for `instagram` module in `scripts/lib/`
- [x] Add `is_instagram_available()` check in `env.py` — reuse `SCRAPECREATORS_API_KEY` (same key as TikTok)
- [x] Add `get_instagram_token()` in `env.py` — same as `get_tiktok_token()`, returns `SCRAPECREATORS_API_KEY`
- [x] Add `_search_instagram()` helper in orchestrator (mirrors `_search_tiktok()`)
- [x] Add Instagram to the thread pool executor block
- [x] Add Instagram timeout config (same as TikTok: 90/120/150s for quick/default/deep)
- [x] Wire through pipeline: normalize → filter → score → sort → dedupe → cross-link → report
- [x] Add Instagram to `progress.show_complete()` and UI spinner
#### 1.9 Add to watchlist extraction in `scripts/watchlist.py`
- [x] Add Instagram findings loop in `_run_topic()` (mirrors TikTok block at lines 204-213)
#### 1.10 Update README.md
- [x] Add Instagram to the sources list
- [x] Note that `SCRAPECREATORS_API_KEY` covers both TikTok and Instagram
### Phase 2: Facebook Source (Conditional)
**Recommendation: SKIP Facebook for now.** Here's why:
1. **No keyword search endpoint** — Facebook only offers profile-based scraping (`/profile/posts` returns 3 posts at a time). Can't search by topic.
2. **Low relevance for topic research** — Without keyword search, we'd need to know specific Facebook pages to scrape, which defeats the purpose of automated topic discovery.
3. **Poor ROI** — 3 posts per API call is very limited compared to Instagram's 60 reels per search.
4. **Same API key** — If Facebook search is added later, it's trivial to add since it shares `SCRAPECREATORS_API_KEY`.
If the user still wants Facebook, the implementation would follow the same pattern but would need a different discovery strategy (e.g., hardcoded page list per topic, or using the Ad Library search for commercial topics).
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `scripts/lib/instagram.py` | **CREATE** | Instagram search + transcript via ScrapeCreators |
| `scripts/lib/schema.py` | MODIFY | Add `InstagramItem` dataclass, add `instagram` to `Report` |
| `scripts/lib/normalize.py` | MODIFY | Add `normalize_instagram_items()` |
| `scripts/lib/score.py` | MODIFY | Add `compute_instagram_engagement_raw()`, `score_instagram_items()` |
| `scripts/lib/dedupe.py` | MODIFY | Add `dedupe_instagram()`, update text extractors |
| `scripts/lib/render.py` | MODIFY | Add Instagram render section, update stats |
| `scripts/lib/env.py` | MODIFY | Add `is_instagram_available()`, `get_instagram_token()` |
| `scripts/last30days.py` | MODIFY | Add Instagram to orchestrator pipeline |
| `scripts/watchlist.py` | MODIFY | Add Instagram findings extraction |
| `README.md` | MODIFY | Add Instagram to sources list |
## Shared Code Opportunity
`_extract_core_subject()`, `_compute_relevance()`, `_tokenize()`, `STOPWORDS`, `SYNONYMS`, and `DEPTH_CONFIG` are duplicated between `tiktok.py` and the new `instagram.py`. Two options:
1. **Copy-paste** (simpler, matches current pattern) — each source module is self-contained
2. **Extract to shared module** (cleaner) — move to `scripts/lib/search_utils.py`
**Recommendation:** Copy-paste for now to match existing pattern. Refactor later if a third ScrapeCreators source is added.
## Testing Strategy
- [x] Run `python3 scripts/lib/instagram.py` with test keyword (if standalone test added)
- [x] Run `python3 scripts/last30days.py "instagram reels trends" --search=instagram` to test isolated
- [x] Run full multi-source: `python3 scripts/last30days.py "AI tools" --search=reddit,instagram`
- [x] Verify JSON output: `--emit=json` includes `instagram` key
- [x] Verify watchlist extraction works with Instagram findings
- [x] Check credit usage is reasonable (1 credit per 10 reels search + 1 per transcript)
## Credits Budget
Per research run with Instagram at `default` depth:
- Search: 1 credit (per 10 reels, returns up to 20) ≈ 2 credits
- Transcripts: 5 credits (max_captions=5 at default depth)
- **Total: ~7 credits per topic** (vs TikTok ~6 credits)
docs/plans/2026-03-04-feat-v2.8-tiktok-instagram-release-plan.md
-147
View File
@@ -1,147 +0,0 @@
---
title: "feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration"
type: enhancement
status: pending
date: 2026-03-04
---
# feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration
## Summary
Ship everything from the last sprint as one combined GitHub release: TikTok's migration from Apify to ScrapeCreators (already committed) + Instagram Reels as the 8th source (uncommitted) + SKILL.md URL regression fixes. Version bump to v2.8.
## What's Shipping
### 1. Instagram Reels — 8th source (NEW)
- Search Instagram Reels by keyword via ScrapeCreators `/v1/instagram/reels/search`
- Spoken-word transcript extraction via `/v2/instagram/media/transcript`
- Full pipeline: search → normalize → score → dedupe → cross-link → render
- Shares `SCRAPECREATORS_API_KEY` with TikTok (no new API key needed)
- ~7 credits per topic at default depth
### 2. TikTok — Apify → ScrapeCreators migration (ALREADY COMMITTED)
- Replaced Apify dependency with ScrapeCreators API
- Same functionality, different backend
- No more `APIFY_API_TOKEN` — uses `SCRAPECREATORS_API_KEY`
### 3. SKILL.md quality fixes
- Instagram added to stats template, citation priority, data sections, footer
- URL regression fix: explicit URL-to-name extraction rules, stronger anti-Sources instruction
- Security section updated: Apify → ScrapeCreators
## Release Checklist
### Pre-commit: Update docs
- [ ] **README.md** — Update for v2.8:
- [ ] Change title from "v2.7" to "v2.8"
- [ ] Add "New in v2.8" banner: Instagram Reels + ScrapeCreators migration
- [ ] Update installation section: `APIFY_API_TOKEN` → `SCRAPECREATORS_API_KEY`
- [ ] Add Instagram to the "How it works" flow description (line 132)
- [ ] Update TikTok section: replace Apify references with ScrapeCreators
- [ ] Add Instagram section (after TikTok section, ~line 963)
- [ ] Update API endpoints table: `api.apify.com` → `api.scrapecreators.com`, add Instagram endpoints
- [ ] Update closing tagline to include Instagram
- [ ] Remove "The shared Apify client wrapper is designed for future Facebook and Instagram sources" (line 963) — Instagram is here now
- [ ] **CHANGELOG.md** — Add v2.8.0 entry:
```
## [2.8.0] - 2026-03-04
### Highlights
Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements.
### Added
- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction
- Instagram items in SKILL.md stats template, citation priority, and output footer
- URL-to-name extraction examples in SKILL.md for cleaner web source display
### Changed
- TikTok backend migrated from Apify to ScrapeCreators API (same key covers TikTok + Instagram)
- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
- SKILL.md version bumped to v2.8
- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
### Fixed
- Web stats line showing full URLs instead of plain domain names (regression from v2.7)
- Trailing "Sources:" block appearing after invitation (WebSearch tool mandate conflict)
- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
```
- [ ] **SKILL.md** frontmatter — Bump version from "2.7" to "2.8"
- [ ] **SKILL.md** description — Add Instagram to the description field
### Commit & tag
- [ ] Stage all changes: modified files + 4 new files (`scripts/lib/instagram.py`, 3 plan docs)
- [ ] Commit with message:
```
feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
- Add Instagram Reels as 8th research source via ScrapeCreators API
- Migrate TikTok from Apify to ScrapeCreators (same API key)
- Add SCRAPECREATORS_API_KEY config (replaces APIFY_API_TOKEN)
- Fix web stats URL regression and trailing Sources: block
- Fix Instagram/TikTok not running in --search=instagram web-only path
- Update SKILL.md with Instagram stats, citations, URL formatting rules
```
- [ ] Create tag: `git tag -a v2.8.0 -m "v2.8.0: Instagram Reels + ScrapeCreators"`
- [ ] Push: `git push origin main --tags`
### Post-push: GitHub release
- [ ] Create GitHub release via `gh release create v2.8.0`:
```
## What's New in v2.8
**Instagram Reels** is now the 8th signal source. Search any topic and get trending Instagram Reels with views, likes, and spoken-word transcripts — scored and ranked alongside Reddit, X, YouTube, TikTok, HN, Polymarket, and the web.
**TikTok migrated to ScrapeCreators API.** Same functionality, new backend. Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. One key now covers both TikTok and Instagram.
### Setup
Sign up at [scrapecreators.com](https://scrapecreators.com) (100 free credits, then PAYG) and add your key:
```bash
echo 'SCRAPECREATORS_API_KEY=your_key' >> ~/.config/last30days/.env
```
### Breaking Change
- `APIFY_API_TOKEN` is no longer used. Replace with `SCRAPECREATORS_API_KEY`.
### Bug Fixes
- Fixed web source URLs leaking into stats display
- Fixed Instagram/TikTok not running when used with `--search=` flag
```
### Post-release: Sync
- [ ] Run `bash scripts/sync.sh` to deploy to all 4 skill destinations
- [ ] Verify Instagram works in a fresh `/last30days` session
## Files Modified (this release)
| File | Status | Description |
|------|--------|-------------|
| `scripts/lib/instagram.py` | NEW | Instagram search + transcript via ScrapeCreators |
| `scripts/lib/schema.py` | MODIFIED | InstagramItem dataclass, Report.instagram field |
| `scripts/lib/normalize.py` | MODIFIED | normalize_instagram_items() |
| `scripts/lib/score.py` | MODIFIED | score_instagram_items(), engagement formula |
| `scripts/lib/dedupe.py` | MODIFIED | dedupe_instagram(), cross-source linking |
| `scripts/lib/render.py` | MODIFIED | Instagram render section, stats |
| `scripts/lib/env.py` | MODIFIED | is_instagram_available(), get_instagram_token() |
| `scripts/lib/ui.py` | MODIFIED | Instagram spinner messages |
| `scripts/last30days.py` | MODIFIED | Instagram in orchestrator pipeline |
| `scripts/watchlist.py` | MODIFIED | Instagram findings extraction |
| `SKILL.md` | MODIFIED | Instagram in stats/citations/footer, URL fixes |
| `README.md` | TO UPDATE | Instagram section, ScrapeCreators migration |
| `CHANGELOG.md` | TO UPDATE | v2.8.0 entry |
| `docs/plans/*.md` | NEW (3) | Plan documents for this work |
docs/plans/2026-03-04-fix-web-sources-url-regression-plan.md
-135
View File
@@ -1,135 +0,0 @@
---
title: "fix: Web sources showing full URLs instead of plain domain names"
type: fix
status: pending
date: 2026-03-04
---
# fix: Web Sources Showing Full URLs Instead of Plain Domain Names
## Problem
Two regressions in the `/last30days` skill output:
### Regression 1: Full URLs on the Web stats line
The `🌐 Web:` stats line is showing full URLs instead of plain source names:
**BAD (current — "Instagram Trends" run):**
```
├─ 🌐 Web: 10+ pages — https://later.com/blog/instagram-reels-trends/,
https://socialbee.com/blog/instagram-trends/,
https://buffer.com/resources/instagram-algorithms/,
https://metricool.com/instagram-trends/,
https://napoleoncat.com/blog/instagram-reels-trends/
```
**GOOD (expected):**
```
├─ 🌐 Web: 10+ pages — Later, SocialBee, Buffer, Metricool, NapoleonCat
```
### Regression 2: Trailing Sources: block with full URLs
A separate `Sources:` section appears at the bottom of the response with full URLs:
```
Sources:
- https://www.heyorca.com/blog/instagram-social-news
- https://socialbee.com/blog/instagram-updates/
- https://buffer.com/resources/instagram-algorithms/
- https://www.cnn.com/2026/02/22/tech/social-media-addiction-trial-tobacco-moment
```
This was fixed in commit `82efa61` (2026-03-02) but is regressing intermittently.
## Root Cause
The SKILL.md instructions at lines 219-221, 404, and 409 already say the right thing:
- Line 404: `├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name`
- Line 409: `"plain names, no URLs — URLs wrap badly in terminals"`
- Lines 219-221: "DO NOT output a separate Sources: block"
But the model ignores these because:
1. **The template `Source Name` is too abstract.** The model sees WebSearch results with full URLs and doesn't know how to extract a human-friendly name from `https://later.com/blog/instagram-reels-trends/`. It needs explicit examples showing the transformation.
2. **The WebSearch system mandate still wins.** The WebSearch tool's built-in instruction (`"you MUST include a Sources: section"`) outcompetes the skill's instruction. The current countermeasure (line 409) works sometimes but not reliably — it needs to be stronger and repeated.
3. **No explicit extraction rule.** The model needs a concrete rule for turning URLs into names: strip protocol, strip path, strip `www.`, capitalize.
## Proposed Solution
**SKILL.md edits only. No Python changes.**
### Fix 1: Add explicit URL-to-name examples in the stats template (line 404 area)
After the stats template block, add concrete examples showing the transformation:
```
**🌐 Web: line formatting:**
- Extract the SITE NAME from each URL — strip protocol, path, and "www."
- Use the publication's proper name when recognizable
- Examples:
- https://later.com/blog/instagram-reels-trends/ → "Later"
- https://socialbee.com/blog/instagram-trends/ → "SocialBee"
- https://buffer.com/resources/instagram-algorithms/ → "Buffer"
- https://www.cnn.com/2026/02/22/tech/... → "CNN"
- https://medium.com/the-ai-studio/... → "Medium"
- https://radicaldatascience.wordpress.com/... → "Radical Data Science"
- NEVER paste the URL itself. ONLY the site name as plain text.
- Separate names with commas: "Later, SocialBee, Buffer, CNN, Medium"
```
### Fix 2: Strengthen the anti-Sources instruction (line 409 area)
Replace the current single-paragraph note with a louder, more explicit instruction:
```
**⚠️ WebSearch citation requirement — ALREADY SATISFIED above.**
The WebSearch tool mandates source citation. That requirement is FULLY satisfied
by the source names on the 🌐 Web: line above. Do NOT append a separate
"Sources:" section at the end of your response. Do NOT list URLs anywhere in
your output. The 🌐 Web: line IS your citation. You're done.
```
### Fix 3: Add a negative example in the URL FORMATTING section (line 356 area)
Extend the existing BAD/GOOD examples to cover the stats line specifically:
```
URL FORMATTING: NEVER paste raw URLs anywhere in the output.
- BAD: "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- GOOD: "per Rolling Stone"
- BAD stats line: "🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/..."
- GOOD stats line: "🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee"
```
### Fix 4: Update Security section (line 588, 600)
While we're in SKILL.md, update the stale Apify references to ScrapeCreators:
- Line 588: Change Apify reference to ScrapeCreators for TikTok
- Line 600: Update TikTok requirement note
- Add Instagram source mention
## Files to Modify
| File | Action | Description |
|------|--------|-------------|
| `SKILL.md` | MODIFY | Strengthen URL formatting rules, add examples, fix Apify refs |
## Implementation Steps
- [x] Add URL-to-name extraction examples after stats template (after line 407)
- [x] Strengthen anti-Sources instruction (replace line 409)
- [x] Add BAD/GOOD stats line example to URL FORMATTING section (around line 356)
- [x] Update Security section: Apify → ScrapeCreators, add Instagram (lines 588, 600)
- [x] Run `bash scripts/sync.sh` to deploy to all destinations
- [ ] Test with `/last30days Instagram Trends` — confirm plain names, no trailing Sources:
## Acceptance Criteria
- [ ] `🌐 Web:` line shows plain names only (e.g., "Later, SocialBee, Buffer")
- [ ] No `Sources:` section appears at the bottom of the response
- [ ] No raw URLs appear anywhere in the output (synthesis, stats, or footer)
- [ ] Security section reflects current source stack (ScrapeCreators, not Apify)
docs/plans/2026-03-05-feat-auto-save-results-to-documents-plan.md
-152
View File
@@ -1,152 +0,0 @@
---
title: "feat: Auto-save research results to ~/Documents/Last30Days/"
type: feat
status: completed
date: 2026-03-05
---
# feat: Auto-save research results to ~/Documents/Last30Days/
## Overview
Every time the last30days skill completes a research run, automatically save the full briefing - inquiry, synthesis, stats, and follow-up suggestions - as a topic-named `.md` file in `~/Documents/Last30Days/`. Inspired by how users like @devin_explores are already manually saving results to build a personal research library (see screenshot - 17 topic files in a `Last30Days` Finder folder, each 9-34 KB).
## Problem Statement / Motivation
The skill's most valuable output - the assistant's synthesized "What I learned" briefing with stats and citations - only exists in the conversation. Once the session ends, it's gone. Users like @devin_explores work around this by manually copying output into .md files. Meanwhile, the Python script already writes raw data to `~/.local/share/last30days/out/`, but:
1. It overwrites on every run (no history)
2. It only contains pre-synthesis data (scored items), not the assistant's expert briefing
3. It's in a hidden dot-directory users don't naturally browse
The feature makes saving automatic and puts files where users expect them - the Documents folder, visible in Finder/file explorer.
## Proposed Solution
Add a **Write tool step in SKILL.md** after the synthesis/stats/invitation block that saves the complete briefing to `~/Documents/Last30Days/{topic-slug}.md`. This is a SKILL.md-only change (no Python script modifications needed) because the content to save is the assistant's synthesized output, which only exists in the SKILL.md flow.
### Why SKILL.md, not the Python script
The Python script (`last30days.py`) runs first and produces raw scored items. The assistant then synthesizes these into the "What I learned" briefing, stats block, and invitation. The synthesis is the valuable part - it's what @devin_explores is saving. The script can't produce this because it runs before synthesis happens.
### File naming
Convert the TOPIC variable to a kebab-case slug for the filename:
- "Claude Code best practices" -> `claude-code-best-practices.md`
- "best rap songs 2026" -> `best-rap-songs-2026.md`
- "nano banana 2 prompting guide" -> `nano-banana-2-prompting-guide.md`
This matches the screenshot pattern exactly (e.g., `anthropic-claude-code-best-practices.md`, `seedance-video-prompting-guide.md`).
If a file with the same slug already exists, append a date suffix: `claude-code-best-practices-2026-03-05.md`. This handles re-researching the same topic without overwriting previous results.
### File content
The saved .md file should contain the complete research output in this order:
```markdown
# {TOPIC}
> Researched {date} | Query type: {QUERY_TYPE} | Target tool: {TARGET_TOOL or "general"}
## What I learned
{The full synthesis section - topics, patterns, citations}
## Stats
{The full stats box with source counts and engagement}
## Follow-up suggestions
{The 2-3 specific suggestions from the invitation block}
---
*Generated by [last30days](https://github.com/mvanhorn/last30days-skill) v2.9*
```
### Implementation location in SKILL.md
Insert a new section between the current "LAST - Invitation" display and the "WAIT FOR USER'S RESPONSE" section. The Write tool call happens silently - no user prompt, no opt-in. Just save and briefly confirm.
## Technical Considerations
- **Cross-platform paths**: `~/Documents/` exists on macOS and most Linux desktops. On systems where it doesn't exist, `mkdir -p` handles creation. Windows WSL users get it too.
- **Permissions**: The Write tool in Claude Code can write to `~/Documents/` without issues. No sandbox concerns since this is the user's own Documents folder.
- **Filename sanitization**: Strip special characters, collapse whitespace to hyphens, lowercase. Keep it simple - no need for a library, just basic string ops in the SKILL.md instructions.
- **File size**: Based on the screenshot (9-34 KB files), the synthesis output is well within reasonable bounds.
- **No opt-out flag needed initially**: This is the default behavior. If users complain, a `--no-save` flag can be added later. Start with always-on since the screenshot proves users want this.
## Acceptance Criteria
- [x] Running `/last30days {topic}` creates `~/Documents/Last30Days/{topic-slug}.md` automatically
- [x] File contains: title, date, query metadata, full synthesis, stats block, follow-up suggestions
- [x] Filename is kebab-case slug of the topic (e.g., `claude-code-skills-guide.md`)
- [x] Duplicate topics get a date suffix instead of overwriting
- [x] Directory `~/Documents/Last30Days/` is created automatically if it doesn't exist
- [x] A brief confirmation line appears after the stats (e.g., "Saved to ~/Documents/Last30Days/claude-code-skills-guide.md")
- [x] Agent mode (`--agent`) also saves the file
- [x] No changes to the Python script - this is purely a SKILL.md addition
## Implementation Steps
### Step 1: Add save instructions to SKILL.md
Insert a new section after the invitation block (after line ~496, before "WAIT FOR USER'S RESPONSE" at line ~499):
**New section in `SKILL.md`:**
```markdown
## Save Research to Documents
After displaying the invitation, save the complete research briefing:
1. Generate the filename from TOPIC:
- Lowercase the topic
- Replace spaces and special characters with hyphens
- Remove consecutive hyphens
- Trim to 60 characters max
- Example: "Claude Code Best Practices" -> "claude-code-best-practices"
2. Check if file already exists. If so, append today's date:
- "claude-code-best-practices.md" exists -> use "claude-code-best-practices-2026-03-05.md"
3. Use the Write tool to save to ~/Documents/Last30Days/{slug}.md with this content:
- H1 title: the TOPIC
- Metadata line: date, QUERY_TYPE, TARGET_TOOL
- Full "What I learned" synthesis (everything you just displayed)
- Full stats block
- Follow-up suggestions from the invitation
- Footer with skill attribution
4. Confirm briefly: "Saved to ~/Documents/Last30Days/{slug}.md"
```
### Step 2: Update agent mode section
The `--agent` mode section (line ~116) skips interactive elements but should still save. Add a note that agent mode saves the file with the same logic.
### Step 3: Update Security & Permissions section
Add to the "What this skill does" list (line ~599):
- "Saves research briefings as .md files to ~/Documents/Last30Days/"
## Success Metrics
- Users accumulate a browsable library of .md research files in their Documents folder
- No more manual copy-paste workflow to save results
- Files are immediately findable in Finder/file explorer search
## Dependencies & Risks
- **Low risk**: Write tool is already in the skill's `allowed-tools` list
- **Low risk**: ~/Documents/ is a standard, user-owned directory
- **Edge case**: If the skill is interrupted mid-run (before synthesis), no file is saved - this is correct behavior since there's nothing to save yet
- **Edge case**: Very long topics could produce unwieldy filenames - the 60-char truncation handles this
## Sources & References
- Screenshot from @devin_explores showing manual .md file library in ~/Documents/Last30Days/
- Current output pipeline: `scripts/lib/render.py:798` (`write_outputs()`) writes to `~/.local/share/last30days/out/`
- SKILL.md synthesis flow: lines 275-496 (internalize research -> show summary -> invitation)
- Existing `--emit` modes: `scripts/last30days.py:1700` (`output_result()`)
docs/plans/2026-03-05-feat-reddit-scrapecreators-v2-improvements-plan.md
-255
View File
@@ -1,255 +0,0 @@
# feat: Reddit ScrapeCreators v2 — Improvements from Beta Testing
**Date:** 2026-03-05
**Type:** Enhancement
**Version:** v2.9 → v2.9.1-beta (or v3.0-beta if shipping to public)
**Branch:** `feat/reddit-scrapecreators` (continue existing branch)
---
## Summary
Three focused improvements to the Reddit ScrapeCreators integration based on 5 full-pipeline tests ("Claude Code skills", "Kanye West", "Anthropic odds", "best rap songs lately", "Nano Banana Pro prompting"):
1. **Elevate top Reddit comments** — give weight to the wittiest/highest-voted comment in scoring and rendering
2. **Improve subreddit discovery** — tune heuristic so ambiguous queries find discussion subs, not utility subs
3. **Make ScrapeCreators the default recommended Reddit method** — update onboarding, SKILL.md metadata, and env.py messaging
---
## Problem Statement
### 1. Comments are undervalued
- ScrapeCreators returns real comment data with scores, but top comments only appear as `Insights:` text under each Reddit item
- The top comment (often the funniest/cleverest reply) gets no special treatment — it's just one of 3 comment excerpts
- Reddit's value IS the comments — upvoted replies are the distilled crowd wisdom
- Currently `comment_insights` are truncated at 150 chars and only 3 are shown per item in compact output
- No scoring bonus for posts that have high-quality comment threads
### 2. Subreddit discovery picks wrong subs for ambiguous queries
- "best rap songs lately" discovered `r/NameThatSong` and `r/findthatsong` (utility subs for identifying songs) instead of discussion subs like `r/hiphopheads` or `r/rap`
- "Kanye West" picked `r/ConcertsIndia_` as second sub — tangential at best
- The current heuristic is pure frequency count on `subreddit` field from global results, with no relevance weighting
- Utility/meta subs often dominate because the same query matches many "help me find X" posts
### 3. Onboarding still suggests OpenAI as the primary Reddit method
- SKILL.md metadata says `primaryEnv: OPENAI_API_KEY` and `requires.env: [OPENAI_API_KEY]`
- The web-only mode banner mentions "OPENAI_API_KEY or codex login → Reddit threads"
- `env.py` error messages direct users to OpenAI for Reddit access
- ScrapeCreators is cheaper ($0.012 vs $0.03-0.10), faster (17s vs 60-90s), returns real data, and shares a key with TikTok + Instagram
- New users should be told: "Get a SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, all three)"
---
## Implementation Plan
### Task 1: Elevate Top Comments in Scoring and Rendering
**Goal:** Give Reddit posts a scoring bonus when they have highly-engaged comment threads, and render the #1 comment with special treatment.
**Files to modify:**
- `scripts/lib/reddit.py` — enrich with `top_comment_score` metadata
- `scripts/lib/score.py` — add comment quality bonus to Reddit scoring
- `scripts/lib/render.py` — render top comment with special formatting
- `scripts/lib/schema.py` — add `top_comment_excerpt` field to RedditItem (optional, may just use existing `top_comments[0]`)
#### 1a. Comment enrichment improvements (`scripts/lib/reddit.py`)
- [x] In `enrich_with_comments()`, after sorting comments by score, tag the item with:
- `top_comment_excerpt`: The highest-scored comment's body (up to 200 chars)
- `top_comment_score`: The upvote count of the #1 comment
- `top_comment_author`: Author of the #1 comment
- [x] Increase comment excerpt length from 300 → 400 chars for top comment only (funny/clever comments need more room)
- [x] Increase `comment_insights` limit from 7 → 10 (we have the data, show it)
- [x] For posts with enriched comments, store the comment count ratio: `top_comment_score / post_score` — a high ratio means the comment outshines the post (Reddit gold)
#### 1b. Scoring bonus for comment quality (`scripts/lib/score.py`)
- [x] In `compute_reddit_engagement_raw()`, add a comment quality signal:
- Current formula: `0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)`
- New formula: `0.50*log1p(score) + 0.35*log1p(num_comments) + 0.05*(upvote_ratio*10) + 0.10*log1p(top_comment_score)`
- This gives a ~10% weight to comment quality, slightly reducing post score and comment count weights
- Posts where the community engaged deeply (high top-comment score) rank higher
- [x] Need to pass `top_comment_score` through the engagement data — either:
- Option A: Add `top_comment_score` to `schema.Engagement` (cleanest)
- Option B: Read from `item.top_comments[0].score` during scoring (no schema change)
- **Recommend Option B** to avoid schema bloat — scoring can peek at `top_comments`
#### 1c. Render top comment prominently (`scripts/lib/render.py`)
- [x] In `render_compact()` Reddit section, after the `Insights:` block, add a "Top Comment:" line for items that have top_comments:
```
**R1** (score:80) r/ClaudeAI (2026-02-28) [666pts, 63cmt]
Claude Code creator: In the next version, introducing two new skills
https://www.reddit.com/r/ClaudeAI/comments/...
*Reddit global search*
💬 Top comment (247 upvotes): "So are they /batch migrating to Rust? :)"
Insights:
- TL;DR generated automatically after 50 comments...
- He's /batch migrating code daily?..
```
- [x] Only show `💬 Top comment` for items where `top_comments[0].score >= 10` (skip low-engagement comments)
- [x] Truncate at 200 chars with `...` if needed
- [x] Also update `render_full_report()` to include the top comment prominently
#### 1d. Update SKILL.md synthesis instructions
- [x] In the "Judge Agent: Synthesize All Sources" section, add guidance:
```
5b. For Reddit: Pay special attention to top comments — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes, quote it directly in your synthesis. Reddit's value is in the comments.
```
- [x] In the citation priority list, add: "When citing Reddit, prefer quoting top comments over just the thread title"
---
### Task 2: Improve Subreddit Discovery Heuristic
**Goal:** Find topical discussion subs rather than utility/meta subs.
**Files to modify:**
- `scripts/lib/reddit.py` — improve `discover_subreddits()` logic
#### 2a. Add relevance-weighted subreddit scoring
- [x] Replace pure frequency count with a weighted score:
```python
def discover_subreddits(results, topic, max_subs=5):
core = _extract_core_subject(topic)
core_words = set(core.lower().split())
scores = Counter()
for post in results:
sub = post.get("subreddit", "")
if not sub:
continue
# Base: frequency count
base = 1.0
# Bonus: subreddit name contains a core topic word
sub_lower = sub.lower()
if any(w in sub_lower for w in core_words if len(w) > 2):
base += 2.0
# Penalty: known utility/meta subreddits
if sub_lower in UTILITY_SUBS:
base *= 0.3
# Bonus: post engagement (high-engagement posts = better sub)
ups = post.get("ups") or post.get("score", 0)
if ups > 100:
base += 0.5
scores[sub] += base
return [sub for sub, _ in scores.most_common(max_subs)]
```
#### 2b. Define utility/meta subreddit blocklist
- [x] Add a small set of subs that are "find X for me" or "identify X" rather than discussion:
```python
UTILITY_SUBS = frozenset({
'namethatsong', 'findthatsong', 'tipofmytongue',
'whatisthissong', 'helpmefind', 'whatisthisthing',
'whatsthissong', 'findareddit', 'subredditdrama',
})
```
- [x] Keep this small and focused — don't over-filter. Only penalty (0.3x), not ban.
#### 2c. Try secondary query for subreddit discovery
- [x] If the first global search returns <3 unique subreddits above threshold, run a second global search with just `{core subject}` (stripped even further) to cast a wider net for subreddit frequencies
- [x] This helps niche topics where the full query is too specific
---
### Task 3: Make ScrapeCreators the Default Reddit Method
**Goal:** New users should be guided to ScrapeCreators first, not OpenAI.
**Files to modify:**
- `SKILL.md` — metadata section, onboarding banner, security section
- `scripts/lib/env.py` — error messages and missing key guidance
- `scripts/lib/render.py` — web-only mode banner
#### 3a. Update SKILL.md metadata
- [x] Change `primaryEnv: OPENAI_API_KEY` → `primaryEnv: SCRAPECREATORS_API_KEY`
- [x] Change `requires.env: [OPENAI_API_KEY]` → `requires.env: [SCRAPECREATORS_API_KEY]`
- [x] Keep OPENAI_API_KEY mentioned but as optional/legacy
#### 3b. Update web-only mode banner (`scripts/lib/render.py`)
- [x] Change the current banner:
```
- `OPENAI_API_KEY` or `codex login` → Reddit threads with real upvotes & comments
```
To:
```
- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views
- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)
```
#### 3c. Update env.py messaging
- [x] In `get_missing_keys()`, when Reddit is missing, suggest ScrapeCreators first:
- Current: returns `'reddit'` which triggers "Add OPENAI_API_KEY or run codex login" in SKILL.md
- Add a helper: `get_setup_hint(missing)` that returns:
- For 'reddit': `"Add SCRAPECREATORS_API_KEY for Reddit + TikTok + Instagram (one key, ~$0.002/search)"`
- For 'x': `"Add XAI_API_KEY for X posts"`
- For 'all': `"Add SCRAPECREATORS_API_KEY (Reddit+TikTok+Instagram) and XAI_API_KEY (X)"`
#### 3d. Update Security & Permissions section in SKILL.md
- [x] Add ScrapeCreators Reddit to the security section:
```
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit, TikTok, and Instagram search (requires SCRAPECREATORS_API_KEY)
```
- [x] Move "Sends search queries to OpenAI's Responses API for Reddit discovery" to a "Legacy:" subsection
- [x] Update "Reddit" description in `allowed-tools` or tags if needed
#### 3e. Update render.py coverage note
- [x] In `render_compact()`, the coverage note for `reddit-only` currently says "Add an xAI key"
- [x] When ScrapeCreators is the active Reddit source, no need to mention OpenAI at all
---
## Acceptance Criteria
- [x] Top Reddit comment is rendered with `💬` prefix and upvote count for enriched posts
- [x] Posts with high top-comment scores rank slightly higher (visible in score differences)
- [x] "best rap songs lately" discovers at least one discussion sub (r/hiphopheads, r/rap, r/Music, etc.) instead of only utility subs
- [x] SKILL.md `primaryEnv` is `SCRAPECREATORS_API_KEY`
- [x] Web-only mode banner recommends ScrapeCreators first
- [x] All 5 test topics still pass (run same tests as before)
- [x] No regression in OpenAI fallback path
---
## Files Changed (Summary)
| File | Change |
|------|--------|
| `scripts/lib/reddit.py` | Improve `discover_subreddits()` with relevance weighting, add utility sub penalties, enhance `enrich_with_comments()` top comment metadata |
| `scripts/lib/score.py` | Add 10% comment quality weight to Reddit engagement formula |
| `scripts/lib/render.py` | Add `💬 Top comment` line to compact output, update web-only banner |
| `scripts/lib/env.py` | Add `get_setup_hint()`, update missing key messaging |
| `SKILL.md` | Change `primaryEnv`, update onboarding banner, add comment synthesis guidance, update security section |
---
## Cost Impact
No cost increase. Same number of API calls per search. The changes are all in local logic (scoring, rendering, discovery heuristic).
---
## Testing Plan
1. Re-run the same 5 test topics from beta testing
2. Verify top comments appear with `💬` in output
3. Verify "best rap songs lately" discovers at least one discussion subreddit
4. Verify `--diagnose` output recommends ScrapeCreators
5. Verify OpenAI fallback still works (unset SCRAPECREATORS_API_KEY, set OPENAI_API_KEY)
docs/plans/2026-03-06-fix-save-output-noise-plan.md
-83
View File
@@ -1,83 +0,0 @@
---
title: "fix: Eliminate save-related output noise after research"
type: fix
status: active
date: 2026-03-06
---
# fix: Eliminate save-related output noise after research
## Problem Statement
After research completes, the auto-save feature adds 3-5 lines of unwanted output below the clean invitation block. Every approach tried so far has made things worse:
| Version | Approach | Lines added | Side effects |
|---------|----------|-------------|--------------|
| v2.9.1 | `Write` tool | ~5 | Shows "Wrote 43 lines to..." |
| v2.9.2 | `run_in_background: true` | ~5 | Background callback triggers 2-4 min cogitation, model hallucinates fake "Human:" messages and generates unsolicited multi-paragraph responses |
| v2.9.3 | Foreground `cat >` heredoc | ~3 | Shows `(No output)`, still triggers 2-4 min cogitation |
**Current v2.9.3 noise (3 lines):**
```
⏺ Bash(mkdir -p ~/Documents/Last30Days && cat > ...)
⎿ (No output)
✻ Churned for 2m 38s
```
The root cause is that ANY tool call after the invitation creates unavoidable Claude Code UI chrome, and the model cogitates for minutes regardless of foreground vs background.
## Proposed Solutions
### Option A: Remove auto-save entirely (Recommended)
Remove the entire "Save Research to Documents" section from SKILL.md. The research lives in the conversation. Zero extra tool calls, zero extra lines, zero cogitation.
**Changes to `SKILL.md`:**
- Delete the "Save Research to Documents" section (~45 lines)
- Update agent mode reference (line 126) - remove save mention
- Update security section (line 612) - change "Saves research briefings" to past tense or conditional
- Remove `📎` footer line from invitation format
- Simplify "WAIT FOR USER'S RESPONSE" section
**What users lose:** Auto-saved .md files in `~/Documents/Last30Days/`
**What users gain:** Clean output ending exactly at the invitation block
### Option B: Move save into Python script
Add `--save-dir` flag to `last30days.py`. The script saves its raw output during its existing Bash call (which already runs). Zero extra tool calls.
**Changes:**
- `scripts/last30days.py` - add `--save-dir` argument, write raw output to file at end of execution
- `SKILL.md` - add `--save-dir=~/Documents/Last30Days` to the script invocation, remove save section
**What users lose:** Saved file contains raw research data, not Claude's synthesis
**What users gain:** Zero extra lines, data is still preserved
### Option C: Opt-in save via follow-up command
Remove auto-save. Add a note to the invitation: "Say 'save' to save this research." When user says "save", run the Bash heredoc then.
**Changes to `SKILL.md`:**
- Delete auto-save section
- Add "save" as a recognized intent in "WHEN USER RESPONDS" section
- Save logic only runs when explicitly requested
**What users lose:** Nothing - save is still available on demand
**What users gain:** Clean output by default, save when they want it
## Acceptance Criteria
- [ ] Research output ends at the invitation block with zero tool calls after it
- [ ] No `(No output)` line visible after invitation
- [ ] No multi-minute cogitation after invitation
- [ ] No hallucinated fake user messages
- [ ] Version bumped to v2.9.4
- [ ] CHANGELOG updated
- [ ] Synced to ~/.claude, ~/.agents, ~/.codex via sync.sh
## Context
- Source: `/Users/mvanhorn/last30days-skill-private/SKILL.md`
- Deployed to: `~/.claude/skills/last30days/SKILL.md`
- Sync command: `bash scripts/sync.sh`
- Current version: v2.9.3
docs/plans/2026-03-07-feat-paperclip-marketing-automation-plan.md
-375
View File
@@ -1,375 +0,0 @@
---
title: "feat: Paperclip Marketing Automation for last30days"
type: feat
status: active
date: 2026-03-07
---
# Paperclip Marketing Automation for last30days
## Overview
Set up a Paperclip "company" that auto-runs marketing for the last30days open-source skill (3,800+ stars). Four agent roles handle daily demo showcases, release announcements, community engagement, and analytics - all using a draft-then-approve workflow through Paperclip's built-in approval gates.
The killer angle: **last30days markets itself by running itself.** The Content Creator agent runs `/last30days [trending topic] --agent` on hot topics daily, then drafts X threads showing the results. Every post is a live demo.
## Problem Statement / Motivation
- All marketing is currently manual - ~60KB of pre-drafted X threads sit in `docs/` unposted
- No automated community monitoring (GitHub issues, contributor shoutouts)
- No metrics tracking (star growth, fork trends, social engagement)
- Solo entrepreneur can't sustain daily content + community management + development
- The tool's best ad is itself running on interesting topics, but that requires daily effort
## Proposed Solution
A Paperclip company called **"last30days Marketing"** with 4 agent roles, all draft-then-approve.
### Company Structure
```
last30days Marketing (Company)
Mission: "Grow last30days to 10K GitHub stars through daily demo content,
release marketing, and community engagement"
Marketing Director (Claude Code agent)
- Sets daily topic priorities
- Reviews draft quality before surfacing to human
- Coordinates cross-agent work
Content Creator (Python script agent)
- Runs last30days on trending topics daily
- Drafts X showcase threads from the results
- Heartbeat: daily at 8 AM PT
Release Manager (Bash + Python agent)
- Watches for new git tags on upstream
- Drafts release announcement threads
- Heartbeat: every 6 hours (tag check is cheap)
Community Manager (Python script agent)
- Monitors GitHub issues/PRs via gh CLI
- Drafts welcome messages for new contributors
- Surfaces popular feature requests
- Heartbeat: every 4 hours
Analytics Analyst (Python script agent)
- Tracks GitHub stars, forks, traffic
- Tracks X engagement (@slashlast30days)
- Generates weekly digest
- Heartbeat: daily at 11 PM PT (collect), weekly Monday 9 AM (digest)
```
### Draft-Then-Approve Flow
```
Agent creates draft
-> Saved to ~/Documents/Last30Days/drafts/{agent}/{date}-{slug}.md
-> Paperclip approval gate triggers
-> Matt reviews in Paperclip UI (approve / reject / edit)
-> On approve: Python script posts to X API via tweepy
-> Audit log records everything
```
## Technical Approach
### Phase 1: Infrastructure (Day 1-2)
Set up Paperclip and external API access.
**Files to create:**
- `marketing/paperclip-config.yaml` - Company definition, org chart, budgets
- `marketing/scripts/post_to_x.py` - X API v2 posting via tweepy (draft queue -> X)
- `marketing/scripts/github_monitor.py` - GitHub event monitoring via gh CLI
- `marketing/scripts/metrics_collector.py` - Star/fork/engagement tracking
- `marketing/.env.example` - Required API keys template
**Setup steps:**
1. Install Paperclip locally:
```bash
git clone https://github.com/paperclipai/paperclip
cd paperclip && pnpm install && pnpm dev
```
2. Get X API v2 credentials (developer.x.com) for @slashlast30days
- Need: API key, API secret, Access token, Access token secret
- Permissions: Read + Write (posting)
3. GitHub token for monitoring (gh auth already configured)
4. Create the company in Paperclip UI at localhost:3100:
- Company name: "last30days Marketing"
- Mission: "Grow last30days to 10K GitHub stars"
- Monthly budget cap: $50 (mostly API token costs)
### Phase 2: Content Creator Pipeline (Day 3-5)
The core marketing engine. Uses last30days to market itself.
**How it works:**
1. **Topic Discovery** - `marketing/scripts/discover_topics.py`
- Scrapes trending topics from: Wikipedia pageviews API, HN front page, Reddit r/all
- Filters for topics that would make compelling demos (tech, culture, sports, geopolitics)
- Outputs ranked topic list to `~/Documents/Last30Days/topics-queue.json`
2. **Research & Draft** - `marketing/scripts/create_showcase.py`
- Picks top topic from queue
- Runs: `python3 scripts/last30days.py "{topic}" --agent --emit=compact --save-dir=~/Documents/Last30Days`
- Reads the saved research output
- Drafts a 1-2 tweet thread in the established style (see Style Guide below)
- Saves draft to `~/Documents/Last30Days/drafts/content/{date}-{slug}.md`
3. **Approval Gate** - Paperclip surfaces draft for review
- Matt approves/edits in Paperclip UI
- On approve: triggers `post_to_x.py` with the draft content
**Style Guide (extracted from existing launch threads):**
```
Format: Stats-first hook + key finding + tool credit
Example (from v2.5 launch):
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views).
5 HN stories. 9 Polymarket markets.
[Key finding in 2-3 sentences]
Polymarket: [relevant odds with specific numbers]
[One-liner showing the tool's value]
github.com/mvanhorn/last30days-skill
```
Rules:
- Always lead with the /last30days command that was run
- Always include the stats line (thread/post/video counts)
- Always end with the GitHub link
- Pick topics people care about RIGHT NOW
- Never use em dashes - use hyphens instead
- Keep threads to 1-2 tweets max for daily showcases
- Save longer threads (3-6 tweets) for releases
### Phase 3: Release Manager Pipeline (Day 5-6)
**How it works:**
1. **Tag Watcher** - `marketing/scripts/watch_releases.py`
- Runs `git -C /Users/mvanhorn/last30days-skill-private fetch upstream --tags` every 6 hours
- Compares local tags vs upstream tags
- On new tag: reads CHANGELOG.md diff since last tag
2. **Thread Drafter** - `marketing/scripts/draft_release_thread.py`
- Reads changelog diff + release-notes.md
- Drafts a 3-6 tweet thread following the v2.5 launch thread style
- Includes: version number, headline features, demo queries, contributor shoutouts
- Saves to `~/Documents/Last30Days/drafts/releases/{tag}.md`
3. **Approval Gate** - Same flow as content pipeline
**Template (from existing docs/v2.5-launch-tweets.md):**
```
Tweet 1: Announcement + 3 headline features + GitHub link
Tweet 2-4: One demo per tweet (command + stats + finding)
Tweet 5: Contributor shoutouts
Tweet 6: Install instructions
```
### Phase 4: Community Manager Pipeline (Day 6-7)
**How it works:**
1. **GitHub Monitor** - `marketing/scripts/github_monitor.py`
- Runs `gh issue list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt,labels`
- Runs `gh pr list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt`
- Compares against `~/Documents/Last30Days/community/seen.json` to detect new items
- Categorizes: bug report, feature request, question, PR
2. **Response Drafter** - `marketing/scripts/draft_community_response.py`
- For new issues: drafts a welcome + triage response
- For new PRs: drafts a thank-you + initial review comment
- For merged PRs: drafts a contributor shoutout tweet
- Saves to `~/Documents/Last30Days/drafts/community/{type}-{number}.md`
3. **Approval Gate** - GitHub responses go through same Paperclip approval
- Approved responses posted via `gh issue comment` or `gh pr comment`
- Shoutout tweets posted via `post_to_x.py`
**Response templates:**
```markdown
# New Issue (bug)
Thanks for the report! I'll look into this. Can you share:
- Your OS and Python version
- The exact command you ran
- Whether you have SCRAPECREATORS_API_KEY set
# New Issue (feature request)
Interesting idea! [1-2 sentences acknowledging the value].
Adding this to the backlog for consideration.
# New PR
Thanks for the contribution, @{author}! I'll review this shortly.
[If first-time contributor: Welcome to the project!]
# Merged PR (tweet)
Shoutout to @{author} for [what they did] in last30days v{version}!
[Brief description of the change and why it matters]
github.com/mvanhorn/last30days-skill/pull/{number}
```
### Phase 5: Analytics Pipeline (Day 7-8)
**How it works:**
1. **Metrics Collector** - `marketing/scripts/metrics_collector.py`
- Daily: GitHub stars, forks, open issues, open PRs (via `gh api`)
- Daily: X followers, tweet impressions for @slashlast30days (via X API v2)
- Stores in SQLite at `~/Documents/Last30Days/analytics.db`
2. **Weekly Digest** - `marketing/scripts/weekly_digest.py`
- Runs Monday 9 AM PT
- Generates markdown report with week-over-week changes:
- Star growth (absolute + rate)
- New forks
- Issues opened/closed
- PRs merged
- Top-performing tweets
- Notable community interactions
- Saves to `~/Documents/Last30Days/digests/{date}-weekly.md`
- Optionally sends via email (SendGrid) or Slack webhook
**Schema for analytics.db:**
```sql
CREATE TABLE daily_metrics (
date TEXT PRIMARY KEY,
github_stars INTEGER,
github_forks INTEGER,
github_open_issues INTEGER,
github_open_prs INTEGER,
x_followers INTEGER,
x_impressions INTEGER,
x_engagement_rate REAL
);
CREATE TABLE tweet_performance (
tweet_id TEXT PRIMARY KEY,
posted_at TEXT,
type TEXT, -- 'showcase', 'release', 'shoutout'
topic TEXT,
impressions INTEGER,
likes INTEGER,
retweets INTEGER,
replies INTEGER,
link_clicks INTEGER
);
```
## Technical Considerations
### API Costs
| Service | Usage | Est. Monthly Cost |
|---------|-------|-------------------|
| Paperclip | Self-hosted | $0 |
| X API v2 | Free tier (posting) | $0 |
| ScrapeCreators | ~30 daily research runs | ~$15 |
| GitHub API | gh CLI, already authed | $0 |
| Claude API | Marketing Director agent | ~$20 |
| **Total** | | **~$35/month** |
### Security
- API keys stored in `marketing/.env` (gitignored, never committed)
- X API tokens scoped to @slashlast30days only (not personal account)
- Paperclip budget cap prevents runaway spend
- All posts go through human approval gate - no autonomous posting
- GitHub token uses existing `gh auth` session
### Failure Modes
- **last30days script fails** - Content Creator skips that day, logs error, tries again tomorrow with new topic
- **X API rate limit** - Queue drafts and retry on next heartbeat cycle
- **Paperclip goes down** - Drafts accumulate in filesystem, nothing posts (safe failure)
- **Bad topic selection** - Marketing Director agent filters topics before research (no politics, no NSFW)
- **Stale approval queue** - If drafts pile up >3 days unapproved, send a nudge notification
### Topic Selection Criteria
The discover_topics.py script should filter for topics that:
1. Are trending NOW (Wikipedia pageview spike, HN front page, Reddit r/all)
2. Would produce interesting multi-source results (not too niche, not too broad)
3. Are safe for a developer tool brand (tech, sports, culture, science - avoid divisive politics)
4. Haven't been covered in the last 7 days (dedup against previous showcases)
5. Span different categories to show the tool's versatility (not all tech, not all sports)
Good examples (from existing launch threads): "Anthropic Pete Hegseth", "Seedance prompting", "Arizona basketball", "Iran war"
## Acceptance Criteria
- [ ] Paperclip running locally with "last30days Marketing" company created
- [ ] Content Creator agent runs last30days daily on a trending topic and produces a draft
- [ ] Release Manager agent detects new git tags and drafts announcement threads
- [ ] Community Manager agent detects new GitHub issues/PRs and drafts responses
- [ ] Analytics agent collects daily metrics and generates weekly digest
- [ ] All drafts go through Paperclip approval gate before posting
- [ ] X API integration posts approved drafts to @slashlast30days
- [ ] GitHub responses posted via gh CLI after approval
- [ ] Monthly budget stays under $50
- [ ] Style of generated tweets matches existing launch thread tone
## Success Metrics
- **Content velocity**: 5-7 showcase tweets/week (up from ~0 currently)
- **Star growth**: Track week-over-week acceleration after content starts
- **Time savings**: <5 min/day reviewing drafts vs 30-60 min/day manual marketing
- **Content quality**: Drafts require minimal editing before approval (>80% approved as-is within 2 weeks)
## Dependencies & Risks
| Dependency | Risk | Mitigation |
|-----------|------|------------|
| Paperclip stability | Early-stage project, may have bugs | Pin to specific commit, keep simple config |
| X API free tier | May get rate-limited or deprecated | Queue-based posting, daily limits |
| Topic discovery quality | Bad topics = bad demos | Human review in approval gate, topic blocklist |
| Claude API for Marketing Director | Cost could spike | Budget cap in Paperclip, simple prompts |
| ScrapeCreators API | PAYG costs scale with usage | Cap at 1 research run/day for showcases |
## File Structure
```
last30days-skill-private/
marketing/
README.md # Setup instructions
.env.example # Required API keys
paperclip-config.yaml # Company definition
scripts/
discover_topics.py # Trending topic discovery
create_showcase.py # Research + draft showcase tweet
draft_release_thread.py # Release announcement drafter
github_monitor.py # GitHub issue/PR monitor
draft_community_response.py # Community response drafter
post_to_x.py # X API posting (after approval)
metrics_collector.py # Daily metrics collection
weekly_digest.py # Weekly analytics digest
templates/
showcase.md # Tweet template for daily demos
release.md # Thread template for releases
community_responses.md # Response templates by type
```
## Sources & References
- Paperclip GitHub: github.com/paperclipai/paperclip
- Paperclip docs: paperclip.ing
- Existing launch threads: `docs/v2.5-launch-tweets.md`, `docs/v2.1-tweets.md`
- Existing launch copy: `docs/v2.1-launch-copy.md`
- Planned Last30Days.com: `docs/plans/2026-02-20-feat-last30days-com-trending-topics-plan.md`
- last30days `--agent` mode: SKILL.md line 115-142 (non-interactive output for automation)
docs/plans/2026-03-07-review-open-issues-and-prs-triage-plan.md
-265
View File
@@ -1,265 +0,0 @@
---
title: Triage Open Issues and PRs
type: refactor
status: active
date: 2026-03-07
---
# Triage Open Issues and PRs - last30days-skill
Consolidated review and action plan for all 6 open items (2 issues, 4 PRs) as of 2026-03-07.
---
## 1. PR #52 - Fix missing metadata files in skill upload bundle
**Author:** 04cb | **Size:** +0/-1 | **Fixes:** #46
**File changed:** `.clawhubignore` (removes `*.json` glob)
### What it does
The `*.json` pattern in `.clawhubignore` was excluding `.claude-plugin/marketplace.json` and `.claude-plugin/plugin.json` - the metadata files required for ClawHub skill upload. This caused the "Zip file contains path with invalid characters" error reported in #46.
### Risk analysis
- Removing `*.json` also includes test fixtures (`fixtures/*.json`) and vendored `package.json` in the bundle. None are sensitive or harmful.
- No `package-lock.json`, `skills-lock.json`, or credential files exist as `.json` in the repo.
- One-line change with zero production code impact.
### Recommendation: MERGE IMMEDIATELY
- [x] Merge PR #52
- [x] Close issue #46 (auto-closes via "Fixes #46")
- [ ] Verify upload works post-merge
---
## 2. Issue #46 - "Claude will not upload as it says thread is invalid"
**Author:** johnuppard | **Error:** "Zip file contains path with invalid characters"
### Recommendation: CLOSES WITH PR #52
No standalone action needed. PR #52 is the fix. After merge, comment on #46 confirming resolution and ask reporter to verify.
---
## 3. PR #50 - test: add tests for entity_extract module
**Author:** mark-c4r | **Size:** +167/-0 (tests only)
### What it does
Adds `tests/test_entity_extract.py` with 23 test cases covering all 4 public/private functions in `scripts/lib/entity_extract.py`:
| Function | Cases | Coverage |
|----------|-------|----------|
| `_extract_x_handles` | 8 | author handles, @mentions, generic filtering, case normalization, frequency ranking, edge cases |
| `_extract_x_hashtags` | 5 | basic extraction, multiple tags, frequency ranking, short tag filtering, empty input |
| `_extract_subreddits` | 6 | field extraction, cross-refs in comments, frequency ranking, r/ stripping |
| `extract_entities` | 4 | integration, max limits, empty inputs, return key validation |
### Code quality assessment
- All function signatures match the actual module implementation exactly
- All 23 tests validated against the real module - they pass
- Import pattern matches existing tests (`sys.path.insert` + `from lib import`)
- Uses `unittest.TestCase` consistent with all other test files
- No external dependencies, no API calls, no mocking needed (pure functions)
- Test data uses inline dicts matching real Reddit/X response structure
### Recommendation: MERGE AFTER LOCAL VERIFICATION
- [x] Pull branch and run `python3 -m pytest tests/test_entity_extract.py -v` (23/23 passed)
- [x] Run full suite `python3 -m pytest tests/` (317/320 passed - 3 pre-existing failures in test_models.py, unrelated)
- [x] Merge
This is a clean, well-structured community contribution that adds coverage to a previously untested module.
---
## 4. PR #48 - feat: add Xiaohongshu source + Reddit public fallback
**Author:** YJLi-new | **Size:** +523/-75 | **Files:** 5 modified
### What it does
Two features bundled in one PR:
**A. Xiaohongshu (Little Red Book) source:**
- New `scripts/lib/xiaohongshu_api.py` module for searching via xiaohongshu-mcp HTTP API
- Source alias: `--search xiaohongshu` or `--search xhs`
- Health/login availability checks in `env.py`
- Handles Chinese numeric suffixes (wan/yi) in engagement parsing
- Default API base: `http://host.docker.internal:18060` (Docker-hosted)
**B. Reddit public JSON fallback:**
- `search_reddit_public()` added to `openai_reddit.py`
- Uses `reddit.com/search/.json` endpoint (no API key required)
- Multiple query strategy (topic, core subject, quoted core)
- Engagement-based relevance heuristic (60% score, 40% comments)
- Supplemental retries correctly gated - only fire when OpenAI auth is present
### Architecture compliance
| Pattern | Status |
|---------|--------|
| Three-function module pattern (search/parse/enrich) | Follows existing patterns |
| `DEPTH_CONFIG` with quick/default/deep | Present |
| `_log()` gated on stderr.isatty() | Present |
| env.py availability check | Present with retry logic |
| ThreadPoolExecutor dispatch | Correctly wired with +1 worker |
| Render/UI integration | Consistent with existing sources |
### Issues found
1. **Health check duplication** - `is_xiaohongshu_available()` in env.py AND `search_feeds()` both probe login status. Redundant but harmless.
2. **`from_date`/`to_date` accepted but unused** for Xiaohongshu - relies on API-side time bucketing. Acceptable given API limitation.
3. **Over-defensive `setdefault()`** in normalization (id and source_domain already set by search_feeds). Harmless.
4. **Error message format inconsistency** between OpenAI and public Reddit paths. Minor.
5. **`get_available_sources()` docstring** not updated to reflect Reddit always being available now.
6. **No tests included** for new Xiaohongshu module or Reddit public fallback.
### Security
- No hardcoded credentials
- xsec_token comes from API response (not user input)
- Docker `host.docker.internal` default is safe for containerized environments
- All URLs use `https://` for public Xiaohongshu
### Recommendation: MERGE WITH CONDITIONS
The Reddit public fallback alone makes this worth merging - it makes Reddit work with zero API keys. Xiaohongshu adds value for Chinese-market research.
**Before merge:**
- [x] Run full test suite to confirm no regressions (292/297 pass, 5 pre-existing)
- [x] Verify Xiaohongshu gracefully skips when API is unavailable (returns False, no crash)
- [x] Resolved merge conflicts with ScrapeCreators Reddit (main). Priority: ScrapeCreators -> OpenAI -> public fallback
- [x] Merged to main and pushed
- [ ] Test Reddit public fallback locally: unset OPENAI_API_KEY, run `python3 scripts/last30days.py "test topic" --search reddit --emit=compact`
- [ ] Update `get_available_sources()` docstring to reflect Reddit always-available change
**After merge (follow-up):**
- [ ] Add `tests/test_xiaohongshu.py` with fixture-based tests
- [ ] Add `tests/test_reddit_public.py` for the fallback path
- [ ] Consider extracting health check duplication
---
## 5. PR #47 - feat: add Apify as unified API provider
**Author:** lapolazzati | **Size:** +1484/-73 | **Files:** 6 new + orchestrator changes
### What it does
Adds Apify as a single-token alternative (`APIFY_API_TOKEN`) covering Reddit, X, TikTok, and Instagram. Existing per-source keys take priority; Apify is a transparent fallback.
**New modules:**
- `apify_client.py` (163 lines) - shared HTTP client for Apify run-sync API
- `apify_reddit.py` (195 lines) - Reddit via `trudax/reddit-scraper` actor
- `apify_x.py` (221 lines) - X via `apidojo/tweet-scraper` actor
- `apify_tiktok.py` (284 lines) - TikTok via `clockworks/tiktok-scraper` actor
- `apify_instagram.py` (288 lines) - Instagram via `apify/instagram-reel-scraper` actor
**Source routing priority:**
```
Reddit: OpenAI -> Apify -> None
X: Bird -> xAI -> Apify -> None
TikTok: ScrapeCreators -> Apify -> None
Instagram: ScrapeCreators -> Apify -> None
```
### Critical issues
1. **Actor ID mismatch (BLOCKER):** Code uses different Apify actors than documented in README and plan.md.
| Source | In Code | In Docs |
|--------|---------|---------|
| Reddit | `trudax/reddit-scraper` | `automation-lab/reddit-scraper` |
| X | `apidojo/tweet-scraper` | `scraper_one/x-posts-search` |
| TikTok | `clockworks/tiktok-scraper` | `epctex/tiktok-search-scraper` |
| Instagram | `apify/instagram-reel-scraper` | matches |
Users following README instructions will hit wrong actors.
2. **No tests (BLOCKER):** 1484 lines of new code with zero test coverage. Each Apify module has its own date parsing, relevance scoring, and response normalization - all untested.
3. **Significant code duplication:** `apify_tiktok.py` and `apify_instagram.py` share nearly identical:
- `_tokenize()` (7 lines)
- `_compute_relevance()` (14 lines)
- `STOPWORDS` set
- `SYNONYMS` dict
- `_extract_core_subject()` (~20 lines)
4. **Version mismatch:** README says v2.9 but git history shows v2.9.4 on main. Version should be higher.
5. **plan.md included in commit** - implementation planning doc shouldn't be in the final merge.
### What's good
- Source routing logic in env.py is clean and backward-compatible
- Orchestrator dispatch uses consistent `_source` parameter pattern
- Existing source paths are completely untouched
- Timeout scaling is appropriate (90s quick, 150s default, 240s deep)
- Token handling follows security best practices (parameter passing, Bearer auth)
### Recommendation: REQUEST CHANGES
This PR is too large and has too many issues for a clean merge. Request the contributor to:
**Must fix before any merge:**
- [ ] Resolve actor ID mismatches (verify which actors actually work, update code OR docs)
- [ ] Add unit tests for all 5 new modules (at minimum: response normalization, date parsing)
- [ ] Fix version number
- [ ] Remove plan.md from commit
**Should fix:**
- [ ] Extract shared code from apify_tiktok/apify_instagram into `apify_common.py`
- [ ] Consider splitting into 2 PRs:
- PR A: `apify_client.py` + env.py routing (foundation)
- PR B: Individual source modules + tests (features)
**Status:** [x] Review posted requesting changes (2026-03-07)
**Comment template for PR:**
> Thanks for this contribution - the single-token approach is a great idea for simplifying setup. A few things need fixing before we can merge:
>
> 1. The Apify actor IDs in the code don't match the README/plan docs. Can you verify which actors are correct and align code + docs?
> 2. We need test coverage for the new modules - at minimum normalization tests with sample actor responses.
> 3. There's significant duplication between apify_tiktok.py and apify_instagram.py (_tokenize, _compute_relevance, STOPWORDS, SYNONYMS, _extract_core_subject). Could you extract shared code to an apify_common.py?
> 4. Version in README should be higher than v2.9.4 (current main).
> 5. Please remove plan.md from the commit.
---
## 6. Issue #45 - Add support for Gemini CLI
**Author:** alexferrari88 | **Request:** "Would it be possible to add support for Gemini CLI?"
### Assessment
This is a feature request to support Gemini CLI as a runtime alongside Claude Code and Codex. Key considerations:
- **Scope:** The skill is currently packaged for Claude Code (SKILL.md format) and Codex (~/.codex/skills/). Gemini CLI uses a different skill/extension format.
- **Effort:** Would require understanding Gemini CLI's plugin system, creating a compatible manifest, and potentially adapting the Python execution model.
- **Priority:** Low - Claude Code and Codex are the primary targets and where the user base is.
- **Community:** If the requester wants to contribute, they'd be best positioned to understand Gemini CLI's requirements.
### Recommendation: ACKNOWLEDGE AND BACKLOG
- [x] Respond with comment acknowledging and inviting contribution
- [x] Add a `help wanted` label
- [x] Keep open as a backlog item
---
## Priority Summary
| Priority | Item | Action | Risk |
|----------|------|--------|------|
| 1 | PR #52 (metadata fix) | ~~Merge now~~ DONE | None |
| 2 | Issue #46 (upload error) | ~~Auto-closes with PR #52~~ DONE | None |
| 3 | PR #50 (entity tests) | ~~Run tests, merge~~ DONE | None |
| 4 | PR #48 (Xiaohongshu + Reddit fallback) | ~~Test locally, merge with conditions~~ DONE | Low |
| 5 | Issue #45 (Gemini CLI) | ~~Acknowledge, backlog, label~~ DONE | None |
| 6 | PR #47 (Apify unified) | ~~Request changes~~ DONE (awaiting contributor) | Medium-High |
docs/plans/2026-03-08-feat-claude-code-plugin-marketplace-plan.md
-197
View File
@@ -1,197 +0,0 @@
---
title: "feat: Publish last30days as Claude Code marketplace plugin"
type: feat
status: completed
date: 2026-03-08
---
# feat: Publish last30days as Claude Code Marketplace Plugin
## Overview
Publish the last30days skill as a Claude Code plugin so users can install it with:
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
This gives users auto-updates, versioned installs, and the standard plugin management UX instead of manual git clone.
## Current State
The repo already has most of what's needed:
| Component | Status | Notes |
|-----------|--------|-------|
| `SKILL.md` | Exists | Source of truth at repo root |
| `skills/last30days/SKILL.md` | Exists (symlink) | Points to root SKILL.md, added for Gemini CLI |
| `scripts/` | Exists | Python research engine |
| `gemini-extension.json` | Exists | Gemini CLI support |
| `.claude-plugin/marketplace.json` | **Missing** | Needed to make repo a marketplace |
| `.claude-plugin/plugin.json` | **Missing** | Needed (in plugin subdir) to define the plugin |
| `plugins/last30days/` | **Missing** | Plugin directory structure |
## How It Works (compound-engineering as reference)
The repo serves as BOTH a **marketplace** (distribution channel) and contains **plugins**:
```
repo-root/ # = marketplace
├── .claude-plugin/
│ └── marketplace.json # Makes repo discoverable as marketplace
└── plugins/
└── plugin-name/ # = individual plugin
├── .claude-plugin/
│ └── plugin.json # Plugin manifest
└── skills/
└── skill-name/
└── SKILL.md
```
## Implementation Plan
### 1. Create marketplace manifest
#### `.claude-plugin/marketplace.json`
```json
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "last30days-skill",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"owner": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"plugins": [
{
"name": "last30days",
"description": "Research any topic from the last 30 days. Become an expert and write copy-paste-ready prompts.",
"version": "2.9.5",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"source": "./plugins/last30days",
"category": "productivity",
"homepage": "https://github.com/mvanhorn/last30days-skill"
}
]
}
```
### 2. Create plugin directory
#### `plugins/last30days/.claude-plugin/plugin.json`
```json
{
"name": "last30days",
"version": "2.9.5",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"author": {
"name": "Matt Van Horn",
"email": "mvanhorn@gmail.com",
"url": "https://github.com/mvanhorn"
},
"homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": [
"research",
"reddit",
"twitter",
"youtube",
"tiktok",
"trends",
"prompts"
]
}
```
### 3. Symlink skill and scripts into plugin directory
```bash
mkdir -p plugins/last30days/.claude-plugin
mkdir -p plugins/last30days/skills/last30days
# Symlink SKILL.md (single source of truth)
ln -s ../../../../SKILL.md plugins/last30days/skills/last30days/SKILL.md
# Symlink scripts directory (the Python engine)
ln -s ../../scripts plugins/last30days/scripts
# Symlink fixtures if needed
ln -s ../../fixtures plugins/last30days/fixtures
```
### 4. Update README with plugin install instructions
Add before the manual install section:
```markdown
### Plugin Install (recommended)
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
Then configure your API keys:
```bash
mkdir -p ~/.config/last30days
# Add SCRAPECREATORS_API_KEY to your env
```
```
### 5. Version bump workflow
When releasing new versions, update version in ALL manifests:
- `SKILL.md` frontmatter `version:`
- `gemini-extension.json` `version`
- `plugins/last30days/.claude-plugin/plugin.json` `version`
- `.claude-plugin/marketplace.json` plugins[0].version
Consider: a `scripts/bump-version.sh` that updates all four in one command.
## Open Question: Symlinks in Plugin Cache
**Risk:** When Claude Code installs a plugin, it clones the repo into `~/.claude/plugins/cache/`. Git preserves symlinks on macOS/Linux, so `plugins/last30days/skills/last30days/SKILL.md -> ../../../../SKILL.md` should resolve correctly because the full repo is cloned.
**Mitigation if symlinks fail:** Replace symlinks with a small build step in sync.sh that copies SKILL.md into the plugin directory. Or just copy the file and accept the duplication in the plugin directory (since the plugin cache is a read-only clone anyway - drift only matters in the source repo, not the installed copy).
**Test:** After implementing, verify with `git clone` into a temp dir and check symlink resolution.
## Acceptance Criteria
- [ ] `.claude-plugin/marketplace.json` exists with correct schema
- [ ] `plugins/last30days/.claude-plugin/plugin.json` exists
- [ ] `plugins/last30days/skills/last30days/SKILL.md` resolves to root SKILL.md
- [ ] `plugins/last30days/scripts` resolves to root scripts/
- [ ] README has plugin install instructions
- [ ] `/plugin marketplace add mvanhorn/last30days-skill` works
- [ ] `/plugin install last30days@last30days-skill` works
- [ ] `/last30days test topic` works after plugin install
- [ ] Standalone install (git clone to ~/.claude/skills/) still works
- [ ] Gemini CLI install still works
## User Experience After Publishing
```
# One-time setup
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
# Use it
/last30days AI video tools for business
# Auto-updates happen at startup when version bumps
```
## Sources
- Claude Code plugin docs: https://code.claude.com/docs/en/plugins
- Plugin reference: https://code.claude.com/docs/en/plugins-reference
- Marketplace docs: https://code.claude.com/docs/en/plugin-marketplaces
- compound-engineering reference: https://github.com/EveryInc/compound-engineering-plugin
- Your installed plugins: superpowers (4.3.1), swift-lsp (1.0.0), compound-engineering (2.38.1)
docs/plans/2026-03-08-feat-gemini-cli-support-implementation-plan.md
-180
View File
@@ -1,180 +0,0 @@
---
title: "feat: Add Gemini CLI extension support"
type: feat
status: completed
date: 2026-03-08
origin: docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md
---
# feat: Add Gemini CLI Extension Support
## Overview
Add Gemini CLI compatibility to last30days by cherry-picking the good parts from PR #53 (@alexferrari88) and fixing the implementation problems ourselves. Close PR #53 with a thank-you comment crediting the contributor.
Based on the review at `docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md`, we accept the concept but fix the approach to match our "one SKILL.md for all platforms" convention established during Codex compatibility work.
## What We Take from PR #53
| Component | Source | Changes Needed |
|-----------|--------|----------------|
| `gemini-extension.json` | PR #53 | Fix settings format (array, not object), bump version to 2.9.5 |
| Path resolution (for loop) | PR #53 | Take as-is, add to both SKILL.md and variants/open/SKILL.md |
| README.md install section | PR #53 | Take as-is |
## What We Skip from PR #53
| Component | Reason |
|-----------|--------|
| `skills/last30days/SKILL.md` (693-line copy) | Duplicate maintenance nightmare - use symlink instead |
| "or" tool name scattering in SKILL.md | LLMs translate tool intent natively - clutters prompt |
| Gemini tool names in `allowed-tools` | Gemini ignores `allowed-tools`; risks breaking Claude Code |
## Implementation Steps
### 1. Create `gemini-extension.json` (new file)
Based on PR #53 but with correct settings format (array per Gemini CLI docs) and v2.9.5 version.
```json
{
"name": "last30days-skill",
"version": "2.9.5",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
"name": "Extension Directory",
"description": "Extension installation directory (auto-set by Gemini CLI)",
"envVar": "GEMINI_EXTENSION_DIR",
"sensitive": false
},
{
"name": "ScrapeCreators API Key",
"description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
"envVar": "SCRAPECREATORS_API_KEY",
"sensitive": true
},
{
"name": "OpenAI API Key",
"description": "OpenAI API Key - optional fallback for Reddit discovery",
"envVar": "OPENAI_API_KEY",
"sensitive": true
},
{
"name": "xAI API Key",
"description": "xAI API Key for X/Twitter search (optional)",
"envVar": "XAI_API_KEY",
"sensitive": true
},
{
"name": "OpenRouter API Key",
"description": "OpenRouter API Key (optional)",
"envVar": "OPENROUTER_API_KEY",
"sensitive": true
},
{
"name": "Parallel AI API Key",
"description": "Parallel AI API Key (optional)",
"envVar": "PARALLEL_API_KEY",
"sensitive": true
},
{
"name": "Brave Search API Key",
"description": "Brave Search API Key (optional)",
"envVar": "BRAVE_API_KEY",
"sensitive": true
},
{
"name": "Apify API Token",
"description": "Apify API Token (optional legacy)",
"envVar": "APIFY_API_TOKEN",
"sensitive": true
},
{
"name": "Twitter AUTH_TOKEN",
"description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
"envVar": "AUTH_TOKEN",
"sensitive": true
},
{
"name": "Twitter CT0",
"description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
"envVar": "CT0",
"sensitive": true
}
]
}
```
### 2. Create `skills/last30days/SKILL.md` as symlink
```bash
mkdir -p skills/last30days
ln -s ../../SKILL.md skills/last30days/SKILL.md
```
Gemini CLI discovers skills from `skills/<name>/SKILL.md`. Symlink ensures single source of truth - no drift, no duplicate maintenance. Git tracks symlinks natively on macOS/Linux (Gemini CLI's target platforms).
### 3. Add Gemini paths to SKILL.md for-loop (line ~172)
Add these 3 entries after `"${CLAUDE_PLUGIN_ROOT:-}"`:
```bash
"${GEMINI_EXTENSION_DIR:-}" \
"$HOME/.gemini/extensions/last30days-skill" \
"$HOME/.gemini/extensions/last30days" \
```
### 4. Add same Gemini paths to `variants/open/SKILL.md` for-loop
Same 3 entries, same position.
### 5. Update `README.md` install section
Add Gemini CLI install before existing Claude Code section:
```markdown
### Gemini CLI
\`\`\`bash
gemini extensions install https://github.com/mvanhorn/last30days-skill.git
\`\`\`
### Claude Code / Codex
```
### 6. Close PR #53 with credit
Post comment thanking @alexferrari88, explaining we incorporated their work with modifications, and close the PR. Credit them in the commit message.
## Acceptance Criteria
- [ ] `gemini-extension.json` exists with correct array-format settings
- [ ] `skills/last30days/SKILL.md` is a symlink to `../../SKILL.md`
- [ ] Gemini path entries in SKILL.md for-loop (3 new entries)
- [ ] Gemini path entries in variants/open/SKILL.md for-loop (3 new entries)
- [ ] README.md has Gemini CLI install instructions
- [ ] No changes to `allowed-tools` in any SKILL.md
- [ ] No "or" tool name alternatives in any SKILL.md body
- [ ] Claude Code still works (`/last30days` runs normally)
- [ ] PR #53 closed with credit to @alexferrari88
- [ ] Commit message credits @alexferrari88 as co-author
## Confidence Assessment
**High confidence.** All changes are additive:
- New file (gemini-extension.json) - zero conflict risk
- Symlink (skills/last30days/SKILL.md) - zero conflict risk
- 3 lines added to a for-loop - trivially safe, short-circuits on first match
- README addition - simple text
- No changes to SKILL.md content, allowed-tools, or Python scripts
- No changes that could break existing Claude Code behavior
The only untested aspect is whether Gemini CLI correctly follows the symlink for skill discovery. If symlinks cause issues, fallback is a 2-line SKILL.md stub that imports the root (but this is unlikely - Gemini CLI runs on macOS/Linux where symlinks are native).
## Sources
- PR #53: https://github.com/mvanhorn/last30days-skill/pull/53
- Issue #45: https://github.com/mvanhorn/last30days-skill/issues/45
- Gemini CLI extension reference (settings array format confirmed): https://geminicli.com/docs/extensions/reference/
- Gemini CLI skills docs: https://geminicli.com/docs/cli/creating-skills/
- Review plan: `docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md`
docs/plans/2026-03-08-feat-perpetual-monitoring-mode-plan.md
-238
View File
@@ -1,238 +0,0 @@
---
title: "feat: Perpetual Monitoring Mode"
type: feat
status: active
date: 2026-03-08
---
# Perpetual Monitoring Mode
## Overview
Add a "perpetual monitoring" mode to last30days that lets users track topics over time with cumulative intelligence, delta reporting, and effortless re-runs. Designed for Claude Code's session model - not server cron.
**User signal:** @JTDaly asked "Can your skill be used to do this perpetually?" in response to a Perplexity AI S&P 500 earnings dashboard that auto-refreshes quarterly.
## Problem Statement
last30days is one-shot: ask, get a briefing, done. But the most valuable research is longitudinal - how conversations evolve, when new voices enter, when engagement spikes. Users want "set it and watch" without re-prompting from scratch every time.
## The Scheduling Reality in Claude Code
| Mechanism | Persistence | Max Duration | Fires When |
|-----------|------------|-------------|------------|
| **CronCreate / `/loop`** | Session-only (RAM) | 3 days, auto-expires | REPL idle |
| **System cron/launchd** | Permanent | Forever | Always |
| **SQLite watchlist** | Permanent (disk) | Forever | On demand |
**Key insight:** Don't build a scheduler. Build a **stateful watchlist** that composes with Claude Code's existing `/loop` for in-session automation, and survives across sessions via SQLite for manual re-runs. The `/loop` skill already exists and does scheduling perfectly - just make last30days a good citizen of it.
## What Already Exists
| Component | File | Status |
|-----------|------|--------|
| Topic CRUD + schedule fields | `scripts/watchlist.py` | Working |
| SQLite persistence with URL dedup, sighting counts | `scripts/store.py` | Working |
| Daily/weekly briefing generation | `scripts/briefing.py` | Working |
| Budget tracking (daily cap) | `scripts/store.py` | Working |
| FTS5 full-text search | `scripts/store.py` | Working |
| WAL mode for concurrent access | `scripts/store.py` | Working |
| `delivery_channel` setting in DB | `scripts/store.py` | Schema only |
| SKILL.md watchlist commands | `SKILL.md` | Missing |
## Proposed Solution
### The Composable Pattern
Instead of building scheduling into last30days, make last30days composable with Claude Code's existing tools:
```
# One-shot: research a topic and store findings
/last30 "S&P 500 earnings" --watch
# See what's new since last run (delta briefing)
/last30 briefing
# Automate with /loop (Claude Code native, session-scoped, 3-day max)
/loop 4h /last30 briefing
# Next session? Watchlist persists. Just re-loop or run manually.
/last30 briefing
```
The user's watchlist lives in SQLite forever. The scheduling is ephemeral by design - you opt into it each session. This matches how people actually use Claude Code: sessions, not servers.
### Phase 1: Watchlist + Briefing via SKILL.md (MVP)
Wire up the existing `watchlist.py` and `briefing.py` infrastructure through the skill interface.
**Tasks:**
- [ ] **SKILL.md additions** - New command branches:
- `/last30 watch add "topic"` - Add topic to watchlist, run initial research with `--store`
- `/last30 watch list` - Show watched topics with last-run timestamps and finding counts
- `/last30 watch remove "topic"` - Remove from watchlist
- `/last30 briefing` - Generate delta briefing across all watched topics (what's new)
- `/last30 briefing --weekly` - Weekly digest with trend analysis
- `/last30 "topic" --watch` - One-shot research that also adds to watchlist
- [ ] **`watchlist.py` updates:**
- `run-all` auto-adds `--store` flag so results persist
- `run-all` returns structured JSON (exit code + summary) for `/loop` consumption
- `run-one` returns per-topic summary for composability
- Add `--quick` default for watched topics (save API cost on recurring runs)
- [ ] **`briefing.py` updates:**
- `generate` outputs compact format suitable for Claude synthesis (like main script's `--emit=compact`)
- Include "last run" timestamp per topic so user knows freshness
- Flag stale topics (not updated in 48+ hours)
**Acceptance criteria:**
- [ ] `/last30 watch add "AI earnings"` persists topic and runs initial research
- [ ] `/last30 briefing` shows accumulated findings with "new since last briefing" markers
- [ ] `/loop 4h /last30 briefing` works out of the box (composability)
- [ ] Watchlist survives across Claude Code sessions (SQLite)
- [ ] Budget cap enforced (default $5/day)
### Phase 2: Delta Intelligence
Make briefings show what *changed*, not just what exists.
**Tasks:**
- [ ] **`scripts/lib/delta.py`** (~150 lines):
- New findings since last briefing (URLs with `first_seen > last_briefing_time`)
- Engagement spikes (>2x increase in score since last sighting)
- New voices (new @handles appearing for first time in a topic)
- Gone quiet (topics with no new findings in 2+ runs)
- [ ] **`briefing.py` delta integration:**
- Section: "Breaking" - high engagement + first seen this run
- Section: "Trending" - engagement increasing across runs
- Section: "New voices" - handles not seen before
- Section: "Gone quiet" - previously active, now silent
- [ ] **Smart re-run logic in `watchlist.py`:**
- Skip topics that were updated < 4 hours ago (avoid redundant API calls)
- Prioritize topics with most engagement change potential
- `--force` flag to override skip logic
**Acceptance criteria:**
- [ ] Briefings clearly distinguish new vs. previously-seen findings
- [ ] Engagement spikes flagged with specific metric ("upvotes 2.3x since yesterday")
- [ ] Redundant API calls avoided via smart skip logic
### Phase 3: Delivery + `/loop` Integration
Make the monitoring truly hands-off during a session.
**Tasks:**
- [ ] **Slack delivery** (`scripts/lib/deliver.py` ~100 lines):
- Webhook URL config: `watchlist.py config delivery slack --webhook "https://..."`
- Format briefing as Slack Block Kit (topic sections, trend indicators)
- Auto-deliver after `run-all` if webhook configured
- [ ] **`/last30 monitor` convenience command:**
- Shorthand that does: `run-all` + `briefing` + starts `/loop` automatically
- Prints: "Monitoring 5 topics every 4h. Auto-expires in 3 days. Run `/last30 monitor` again next session."
- Uses CronCreate directly (no `/loop` dependency) for tighter control
- [ ] **Session resume hint:**
- On `/last30 briefing`, if watchlist has topics but no `/loop` active, suggest:
"You have 5 watched topics. Run `/loop 4h /last30 briefing` to auto-refresh, or `/last30 monitor` for hands-off mode."
**Acceptance criteria:**
- [ ] Slack webhook delivery works end-to-end
- [ ] `/last30 monitor` starts automated loop with one command
- [ ] Clear messaging about 3-day session limit and how to resume
## Architecture: Why This Is Better Than Server Cron
```
Traditional approach (rejected):
System Cron -> watchlist.py run-all -> SQLite -> ??? deliver somehow
Claude Code-native approach:
/last30 watch add "topic" --> SQLite (persists forever)
/loop 4h /last30 briefing --> CronCreate (session, 3-day max)
|
v
Claude reads briefing.py output
Claude synthesizes with LLM judgment
Claude delivers via Slack webhook
Claude answers follow-up questions in context
```
The Claude Code-native approach is better because:
1. **LLM synthesis on every run** - not just raw data, but judgment ("this is unusual because...")
2. **Conversational** - user can ask follow-ups ("tell me more about the META earnings spike")
3. **Zero infrastructure** - no plist, no crontab, no daemon management
4. **Portable** - works on any OS where Claude Code runs
5. **Composable** - `/loop` is a general-purpose tool, not custom scheduling code
## Alternative Approaches Considered
### 1. Build custom scheduler.py with system cron/launchd
**Rejected.** Doesn't work in Claude Code's model. Platform-specific. Requires root/sudo for some configs. Users of a Claude Code skill shouldn't need to manage system daemons.
### 2. OpenClaw's persistent cron service
**Not applicable.** OpenClaw has its own cron system with database-backed scheduling, but last30days is an open-source skill that should work without OpenClaw. Could be an optional integration later.
### 3. Long-running Python daemon
**Rejected.** Fragile, wastes resources, doesn't benefit from LLM synthesis on each run.
### 4. Web dashboard with its own backend
**Deferred to Phase 4.** Dramatically increases scope. The briefing-to-Slack pattern delivers 80% of the value with 10% of the effort.
## Technical Considerations
**No new services.** Pure Python scripts + SQLite + SKILL.md instructions. Zero infrastructure.
**Cost control.** Each watched topic costs ~$0.05-0.30/run at `--quick` depth. 10 topics x 6 runs/day = $3-18/day. Budget cap in store.py already enforces limits.
**Composability contract.** `briefing.py generate` must output clean, parseable text that Claude can synthesize. No interactive prompts, no side effects beyond SQLite writes.
**Backwards compatibility.** All new. Existing `/last30 topic` unchanged. `--watch` flag is opt-in.
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Users expect "perpetual" to mean forever | High | Clear messaging: "3-day auto-expire per session, watchlist persists, re-run next session" |
| `/loop` changes or breaks | Low | Composability means we don't depend on `/loop` internals - just CronCreate |
| Budget overrun with many topics | Low | Budget cap already implemented in store.py |
| SQLite grows large over months | Low | Add 90-day retention policy |
| Slack webhook stops working | Low | Log failures, don't block pipeline, alert in next briefing |
## Success Metrics
- User can go from zero to monitoring in one command: `/last30 "S&P 500 earnings" --watch`
- `/last30 briefing` surfaces genuinely new information with delta markers
- The system composes cleanly with `/loop` - no special integration needed
- Watchlist persists across sessions - user picks up where they left off
- Clear, honest UX about session limits vs. persistent state
## Scope Boundaries
**In scope:**
- Watchlist CRUD via SKILL.md
- Delta-aware briefings
- `/loop` composability (not custom scheduling)
- Slack webhook delivery
- `/last30 monitor` convenience command
**Out of scope:**
- System cron/launchd integration
- Web dashboard
- Email delivery (Slack webhook is simpler, covers most users)
- Multi-user / team features
- Custom NLP beyond what exists
## Implementation Estimate
- Phase 1 (Watchlist + Briefing): SKILL.md additions, minor `watchlist.py` and `briefing.py` updates
- Phase 2 (Delta Intelligence): New `delta.py` module, `briefing.py` integration
- Phase 3 (Delivery + Monitor): New `deliver.py`, SKILL.md `/last30 monitor` command
Each phase ships independently. Phase 1 alone answers @JTDaly's question.
docs/plans/2026-03-08-fix-skill-md-save-section-regression-plan.md
-81
View File
@@ -1,81 +0,0 @@
---
title: "fix: Remove re-introduced Save Research to Documents section from SKILL.md"
type: fix
status: completed
date: 2026-03-08
---
# fix: Remove re-introduced Save Research to Documents section from SKILL.md
## Overview
PR merges on March 7 regressed SKILL.md by re-introducing the "Save Research to Documents" section that v2.9.4 (`6d5acb9`) intentionally removed. The save logic was moved into the Python script via `--save-dir` flag, but the merged branches (PR #48 Xiaohongshu, upstream merge) were forked before v2.9.4 and brought the old SKILL.md content back via merge resolution.
**Result:** The skill now saves research AFTER the "I'm now an expert" invitation via a separate Bash tool call, which causes extra cogitation time, "(No output)" noise, and sometimes hallucinated follow-up messages - the exact UX problems v2.9.4 fixed.
## Root Cause
1. Commit `6d5acb9` (v2.9.4, March 6) removed the ~45-line "Save Research to Documents" section and added `--save-dir=~/Documents/Last30Days` to the bash command
2. PR #48 (Xiaohongshu) branch contained commit `9950d01` with the OLD save-via-Bash version
3. Merging PR #48 on March 7 re-introduced the save section
4. Upstream merge (`28dff6e`) compounded it
## Three changes needed
All in `SKILL.md`:
### 1. Fix agent mode line (line 142)
**Current (broken):**
```
Agent mode still saves the research briefing to `~/Documents/Last30Days/` using the same logic as interactive mode (see "Save Research to Documents" section).
```
**Should be:**
```
Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir` (handled by the script, no extra tool calls).
```
### 2. Add `--save-dir` back to bash command (line 186)
**Current (broken):**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
```
**Should be:**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web --save-dir=~/Documents/Last30Days
```
### 3. Delete the entire "Save Research to Documents" section (lines 517-563)
Remove from `## Save Research to Documents` through the `---` separator before `## WAIT FOR USER'S RESPONSE`. This is ~47 lines.
### 4. Update "STOP and wait" line (line 567)
**Current:**
```
**STOP and wait** for the user to respond.
```
**Should be (matching v2.9.4):**
```
**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`.
```
## Post-edit steps
1. Run `bash /Users/mvanhorn/last30days-skill-private/scripts/sync.sh` to deploy to `~/.claude/skills/`
2. Push to `origin/main` and `upstream/main`
3. Verify with `diff` that installed skill matches repo
## Acceptance Criteria
- [ ] "Save Research to Documents" section is gone from SKILL.md
- [ ] `--save-dir=~/Documents/Last30Days` is in the bash command
- [ ] Agent mode line references `--save-dir`, not the deleted section
- [ ] "STOP and wait" line includes the "do NOT call any tools" reinforcement
- [ ] `sync.sh` deployed successfully
- [ ] `diff` between repo and `~/.claude/skills/last30days/SKILL.md` shows no differences
- [ ] Pushed to origin and upstream
docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md
-157
View File
@@ -1,157 +0,0 @@
---
title: "review: PR #53 - Gemini CLI Support"
type: review
status: active
date: 2026-03-08
---
# Review: PR #53 - Gemini CLI Support (alexferrari88)
## Verdict: MODIFY - Accept concept, reject implementation approach
PR #53 by @alexferrari88 adds Gemini CLI extension support. The intent is good and closes issue #45, but the implementation has structural problems that would create the exact maintenance nightmare we just fixed in v2.9.5.
## PR Summary
| File | Changes | Assessment |
|------|---------|------------|
| `gemini-extension.json` | +67 new file | **Accept with fixes** |
| `skills/last30days/SKILL.md` | +693 new file (full copy) | **Reject** - duplicate SKILL.md |
| `SKILL.md` | +9/-6 (tool name scattering) | **Reject** - wrong approach |
| `variants/open/SKILL.md` | +4/-1 | **Partially accept** (path resolution yes, tool names no) |
| `README.md` | +6 install instructions | **Accept** |
## Critical Issues
### 1. Duplicated SKILL.md (BLOCKER)
The PR creates `skills/last30days/SKILL.md` as a **full 693-line copy** of the root `SKILL.md`. This is the exact problem we just spent hours debugging - PR merges on March 7 regressed v2.9.4's save-section removal because branches had stale copies. A second SKILL.md guarantees this happens again.
The Codex compatibility work (see `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md`) explicitly chose **one SKILL.md for all platforms** to avoid this. Gemini CLI should follow the same pattern.
**Fix:** Delete `skills/last30days/SKILL.md`. Gemini CLI discovers skills from the extension's `skills/` directory, but we can either:
- (a) Symlink: `skills/last30days/SKILL.md -> ../../SKILL.md`
- (b) Use the root SKILL.md directly and configure `contextFileName` in gemini-extension.json to point to it
- (c) Have `skills/last30days/SKILL.md` be a thin wrapper that says "See root SKILL.md" (least ideal)
### 2. Based on v2.9.1, not v2.9.5 (BLOCKER)
The PR's copy of SKILL.md is based on v2.9.1 and includes:
- The "Save Research to Documents" section (removed in v2.9.4, re-removed in v2.9.5)
- Old agent mode line referencing deleted section
- Missing `--save-dir=~/Documents/Last30Days` flag
- Old version number
**Fix:** Rebase on current main (v2.9.5).
### 3. "Or" tool name scattering (REJECT)
The PR adds `WebSearch or google_web_search(...)` and similar patterns throughout SKILL.md. This is the wrong approach because:
- **LLMs already translate intent to tools.** When Gemini reads "do a WebSearch for X", it knows to use `google_web_search`. When Claude reads it, it uses `WebSearch`. The model handles this mapping natively.
- **Clutters the prompt.** SKILL.md is a 640-line prompt. Adding "or alternative_name" to every tool reference makes it harder for the model to parse.
- **Maintenance burden.** Every new platform means adding more "or" alternatives.
**Fix:** Remove all "or" alternatives. Keep Claude Code tool names in the SKILL.md body (they work as intent descriptions). If Gemini needs explicit tool mapping, that belongs in `GEMINI.md` (Gemini CLI's context file), not scattered through the skill instructions.
### 4. `allowed-tools` pollution (RISKY)
Adding `run_shell_command, read_file, write_file, ask_user, google_web_search` to `allowed-tools`:
```
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch, run_shell_command, read_file, write_file, ask_user, google_web_search
```
**Risk:** If Claude Code's parser is strict and rejects unknown tool names, this breaks the skill for all Claude Code users. If it silently ignores unknown names, it's harmless but noisy.
**Finding:** Gemini CLI only recognizes `name` and `description` in SKILL.md frontmatter. It **ignores** `allowed-tools` entirely. So adding Gemini tool names to `allowed-tools` provides zero benefit to Gemini users while potentially breaking Claude Code users.
**Fix:** Remove Gemini tool names from `allowed-tools`. They serve no purpose on either platform.
## What to Accept
### 1. gemini-extension.json (with fixes)
The manifest file is the right approach. However:
- [ ] **Verify settings format.** The PR uses object-key format (`"SCRAPECREATORS_API_KEY": { ... }`). Gemini CLI docs show array format (`[{ "name": "...", ... }]`). Need to confirm which is correct for the current Gemini CLI version. The researcher found array format in the docs.
- [ ] **Update version** from `2.9.1` to `2.9.5`
- [ ] **Consider adding `contextFileName`** to point to root SKILL.md instead of duplicating
### 2. Path resolution additions
Adding these to the bash `for` loop is correct and low-risk:
```bash
"${GEMINI_EXTENSION_DIR:-}" \
"$HOME/.gemini/extensions/last30days-skill" \
"$HOME/.gemini/extensions/last30days" \
```
This should be in both `SKILL.md` and `variants/open/SKILL.md`.
### 3. README.md install section
Clean and appropriate. Adding Gemini CLI install command before Claude Code section.
## Proposed Changes to Request from Contributor
### Must-fix (before merge)
1. **Delete `skills/last30days/SKILL.md`** - no duplicate. Either symlink or use `contextFileName` in manifest.
2. **Rebase on main** (v2.9.5) - the PR is based on stale code.
3. **Remove all "or" tool name alternatives** from SKILL.md and variants/open/SKILL.md body text.
4. **Remove Gemini tool names from `allowed-tools`** in all SKILL.md files.
5. **Verify `gemini-extension.json` settings format** against current Gemini CLI docs (array vs object).
### Nice-to-have
6. **Add a `GEMINI.md` context file** (optional) - can include a short note like "When this skill references 'WebSearch', use `google_web_search`. When it references 'Bash', use `run_shell_command`." This is the clean way to handle tool name translation.
7. **Update sync.sh** to optionally deploy to `~/.gemini/extensions/last30days/` (debatable - Gemini users may prefer `gemini extensions install` instead).
8. **Add `.gemini/` to the path check in sync.sh** import verification.
## Testing Plan
Before merging, verify:
- [ ] `gemini extensions install` works from the repo (or `gemini extensions link .` for local dev)
- [ ] Skill activates in Gemini CLI and the model can find `scripts/last30days.py`
- [ ] `GEMINI_EXTENSION_DIR` env var resolves correctly in the bash for-loop
- [ ] Claude Code still works with no regressions (run `/last30days test topic --mock` or similar)
- [ ] `allowed-tools` with only Claude Code tool names doesn't break Gemini CLI skill loading
## Comment Template for PR
```
Thanks for the contribution! Gemini CLI support is great to have, and the `gemini-extension.json` manifest and path resolution additions are solid.
A few things need changing before we can merge:
**Must-fix:**
1. **Remove `skills/last30days/SKILL.md`** - We maintain one SKILL.md to avoid sync drift (we literally just fixed a regression from this exact problem yesterday). Either symlink it or use `contextFileName` in the manifest to point to the root SKILL.md.
2. **Rebase on `main`** - The PR is based on v2.9.1 but we're now at v2.9.5. The "Save Research to Documents" section in your copy was removed, `--save-dir` was added to the bash command, and the version was bumped.
3. **Remove "or" tool name alternatives** from SKILL.md body (e.g., `WebSearch or google_web_search`). LLMs handle tool name translation natively - Gemini knows to use `google_web_search` when the skill says "search the web". Scattering alternatives clutters the prompt.
4. **Remove Gemini tool names from `allowed-tools`** - Gemini CLI ignores `allowed-tools` (it only reads `name` and `description` from SKILL.md frontmatter), so these provide no benefit. And they risk breaking Claude Code if its parser rejects unknown tool names.
5. **Verify `gemini-extension.json` settings format** - The Gemini CLI docs I found show settings as an array (`[{ "name": "...", ... }]`), not object keys (`{ "KEY": { ... } }`). Can you confirm which format your Gemini CLI version expects?
**Optional but recommended:**
6. Consider adding a `GEMINI.md` context file with a short tool-name translation note (e.g., "When this skill says 'WebSearch', use `google_web_search`"). This is the clean way to bridge tool names.
Happy to help work through any of these! The core approach (manifest + path resolution) is right.
```
## Sources
- PR #53: https://github.com/mvanhorn/last30days-skill/pull/53
- Issue #45: https://github.com/mvanhorn/last30days-skill/issues/45
- Gemini CLI extension docs: https://geminicli.com/docs/extensions/writing-extensions/
- Gemini CLI extension reference: https://geminicli.com/docs/extensions/reference/
- Gemini CLI skills docs: https://geminicli.com/docs/cli/creating-skills/
- Codex compatibility plan: `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md`
- v2.9.5 regression fix: commit `8f7fb5a` (today)
docs/plans/2026-03-09-docs-v295-readme-and-plugin-fix-plan.md
-75
View File
@@ -1,75 +0,0 @@
---
title: "docs: v2.9.5 README update, plugin.json fix, Claude Code install above fold"
type: docs
status: completed
date: 2026-03-09
---
# docs: v2.9.5 README update, plugin.json fix, Claude Code install above fold
Three things in one commit:
## 1. Fix plugin.json hooks field (Issue #62)
**Bug:** `"hooks": ["./hooks/"]` should be `"hooks": {}`. Array format is invalid per Claude Code plugin manifest schema. No hooks directory exists anyway.
**File:** `.claude-plugin/plugin.json:25`
**Fix:** Change `"hooks": ["./hooks/"]` to `"hooks": {}`
## 2. Add Claude Code install above ClawHub
Currently the README opens with ClawHub badge + install. User wants Claude Code (the default/best way) to be equally prominent ABOVE ClawHub.
**Add before the ClawHub badge:**
```
### Claude Code (recommended)
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
This matches the Installation section further down but puts it above the fold where people actually see it.
## 3. Update README to v2.9.5
**Header:** Change `v2.9.1` to `v2.9.5`
**Source list:** Add "Bluesky" everywhere sources are listed (currently says "Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web")
**New in v2.9.5 block** (replace the v2.9.1 block):
Features shipped since v2.9.1:
- **ScrapeCreators X backend** (`4b7087e`) - X/Twitter search now uses ScrapeCreators as a backend option alongside Bird cookies
- **Bluesky/AT Protocol** (`9a1059e`, `adb5a67`) - New social source with full pipeline (search, parse, normalize, score, dedupe, render). Opt-in via `BSKY_HANDLE` + `BSKY_APP_PASSWORD` app password
- **Comparative mode** (`ecf90c0`) - "X vs Y" queries run 3 parallel research passes and output side-by-side comparison
- **Per-project .env config** (PR #59) - `.last30days.env` in project root for per-project API keys
- **SessionStart config check** (PR #58) - Validates config on session start
- **Expanded test coverage** (PRs #56, #57) - Unit tests for untested modules + smoke tests + edge cases
- **Claude Code marketplace plugin** (`627947f`) - Install via `/plugin install`
- **Gemini CLI extension** (`2f16ff1`) - `gemini extensions install`
**Env vars section:** Add `BSKY_HANDLE` and `BSKY_APP_PASSWORD` to the optional keys table
## Files to Change
| File | Change |
|------|--------|
| `README.md` | Version bump, Claude Code above fold, new features, Bluesky in source lists, env vars |
| `.claude-plugin/plugin.json` | Fix hooks field |
## Acceptance Criteria
- [x] `"hooks": {}` in plugin.json (fixes #62)
- [x] Claude Code plugin install appears ABOVE ClawHub badge
- [x] README says v2.9.5
- [x] "New in v2.9.5" block lists all features since v2.9.1
- [x] Bluesky appears in source list references
- [x] BSKY_HANDLE/BSKY_APP_PASSWORD documented in env vars
- [ ] Plugin installs successfully after fix
## Sources
- Issue #62: https://github.com/mvanhorn/last30days-skill/issues/62
- Issue #63: https://github.com/mvanhorn/last30days-skill/issues/63 (skipped - score 4/10, niche)
- Recent commits: `4b7087e`, `9a1059e`, `ecf90c0`, `adb5a67`
- PRs #56-#59 merged since v2.9.1
docs/plans/2026-03-09-feat-truth-social-source-plan.md
-118
View File
@@ -1,118 +0,0 @@
---
title: "feat: Add Truth Social as opt-in source"
type: feat
status: completed
date: 2026-03-09
---
# feat: Add Truth Social as opt-in source
Add Truth Social (Mastodon fork) as an opt-in social source. When `TRUTHSOCIAL_TOKEN` is set, posts from Truth Social appear alongside other sources in research results. When not configured, completely silent.
## Problem / Motivation
Issue #63 requested Truth Social support. Truth Social is a Mastodon fork with ~7M monthly active users. For users who care about that community's perspective on a topic, it's a valuable signal source. Follows the same opt-in pattern as Bluesky.
## Approach
Use Truth Social's Mastodon-compatible API directly with `urllib3` (no external dependencies). One env var: `TRUTHSOCIAL_TOKEN` (bearer token). Follow the Bluesky source pattern exactly across all 10 pipeline files.
**Why bearer token (not username/password):** Truth Social's OAuth uses a non-standard `/oauth/v2/token` endpoint with hardcoded `client_id`/`client_secret` extracted from their JS bundle. These values break when Truth Social updates their frontend. A bearer token is more stable - user extracts it once from browser dev tools (Application > Local Storage > truthsocial.com > `access_token`) or via `truthbrush` CLI.
**API endpoint:**
```
GET https://truthsocial.com/api/v2/search
Authorization: Bearer {token}
Params: q={topic}&type=statuses&limit=40
```
**Response format:** Standard Mastodon status objects with `content` (HTML), `created_at`, `url`, `account`, `favourites_count`, `reblogs_count`, `replies_count`.
## Files to Change
| # | File | Change |
|---|------|--------|
| 1 | `scripts/lib/truthsocial.py` | **New file.** API client: `search_truthsocial(topic, from_date, to_date, depth, config)` + `parse_truthsocial_response(response)`. Strip HTML tags from `content`. Handle 401/403/429 gracefully. |
| 2 | `scripts/lib/env.py` | Add `('TRUTHSOCIAL_TOKEN', None)` to `get_config()`. Add `is_truthsocial_available(config)`. |
| 3 | `scripts/lib/schema.py` | Add `TruthSocialItem` dataclass (id prefix `"TS"`). Add `truthsocial`/`truthsocial_error` fields to `Report`. Update `to_dict()`/`from_dict()`. |
| 4 | `scripts/lib/normalize.py` | Add `normalize_truthsocial_items()`. Map Mastodon fields: `favourites_count` -> `likes`, `reblogs_count` -> `reposts`, `replies_count` -> `replies`. |
| 5 | `scripts/lib/score.py` | Add `compute_truthsocial_engagement_raw()` + `score_truthsocial_items()`. Same weighted log formula as Bluesky. |
| 6 | `scripts/lib/dedupe.py` | Add `dedupe_truthsocial()` (one-liner wrapping `dedupe_items`). |
| 7 | `scripts/lib/render.py` | Add Truth Social sections to `_xref_tag()`, `_assess_data_freshness()`, `render_compact()`, `render_source_status()`, `render_context_snippet()`, `render_full_report()`. |
| 8 | `scripts/last30days.py` | ~20 touchpoints: TIMEOUT_PROFILES, VALID_SEARCH_SOURCES, import, `_search_truthsocial()`, `run_research()` param + dispatch + collect, `main()` availability + diag + flag + normalize + score + sort + dedupe + report. |
| 9 | `SKILL.md` | Add Truth Social to source lists, optionalEnv (`TRUTHSOCIAL_TOKEN`), stats template, security/privacy section. |
| 10 | `tests/test_truthsocial.py` | **New file.** Tests: HTML stripping, date parsing, response parsing, empty response, missing fields, depth config, auth error handling, successful search with mocked HTTP. |
## Key Implementation Details
### HTML stripping (`truthsocial.py`)
Truth Social returns HTML content (`<p>Post text</p>`). Strip tags to plain text:
```python
import re
def _strip_html(html: str) -> str:
text = re.sub(r'<br\s*/?>', '\n', html)
text = re.sub(r'<[^>]+>', '', text)
return text.strip()
```
### Date filtering
Mastodon `created_at` is ISO 8601 (`2026-03-09T12:00:00.000Z`). Use `[:10]` slice for `YYYY-MM-DD` comparison against `from_date`/`to_date`.
### Engagement mapping
| Mastodon field | Internal field | Display |
|---------------|---------------|---------|
| `favourites_count` | `likes` | `{N}lk` |
| `reblogs_count` | `reposts` | `{N}rp` |
| `replies_count` | `replies` | `{N}re` |
### Error handling
| HTTP Status | Behavior |
|------------|----------|
| 200 | Parse and return results |
| 401 | Return `{"statuses": [], "error": "Truth Social token expired"}` |
| 403 | Return `{"statuses": [], "error": "Truth Social access denied (Cloudflare)"}` |
| 429 | Return `{"statuses": [], "error": "Truth Social rate limited"}` |
| Other | Return `{"statuses": [], "error": "Truth Social search failed: {status}"}` |
All errors return empty results gracefully - never crash the research run.
### DEPTH_CONFIG
| Depth | Limit |
|-------|-------|
| quick | 15 |
| default | 30 |
| deep | 60 |
## What NOT to Change
- `lib/__init__.py` - must stay bare (no eager imports)
- `cross_source_link.py` - works generically on items with `cross_refs` field
- `filter.py` - date filtering is generic
- No new pip dependencies
## Acceptance Criteria
- [x] `is_truthsocial_available()` returns False when `TRUTHSOCIAL_TOKEN` not set
- [x] No Truth Social stats line, no error, no mention when unconfigured
- [x] With valid `TRUTHSOCIAL_TOKEN`, posts are returned and rendered
- [x] HTML tags stripped from post content
- [x] Token expiry (401) returns empty results gracefully
- [x] Cloudflare block (403) returns empty results gracefully
- [x] `--diagnose` shows Truth Social availability status
- [x] SKILL.md documents `TRUTHSOCIAL_TOKEN` env var
- [x] All existing tests still pass
- [x] New tests cover: HTML stripping, parsing, auth errors, successful search
- [x] `bash scripts/sync.sh` deploys successfully
## Sources
- Issue #63: https://github.com/mvanhorn/last30days-skill/issues/63
- Truth Social API: Mastodon-compatible at `truthsocial.com/api/v2/search`
- Auth: Bearer token via browser dev tools or `truthbrush` CLI
- Pattern reference: `scripts/lib/bluesky.py` (most recent source addition)
- Bluesky auth plan: `docs/plans/2026-03-09-fix-bluesky-auth-opt-in-plan.md`
docs/plans/2026-03-09-fix-bluesky-auth-opt-in-plan.md
-105
View File
@@ -1,105 +0,0 @@
---
title: "fix: Make Bluesky opt-in with auth (searchPosts now requires authentication)"
type: fix
status: completed
date: 2026-03-09
---
# fix: Make Bluesky opt-in with auth (searchPosts now requires authentication)
## Problem
Bluesky's `app.bsky.feed.searchPosts` endpoint now returns **403 Forbidden** for unauthenticated requests. This broke today (2026-03-09), likely related to the CEO transition. The `searchActors` endpoint still works without auth, but `searchPosts` does not.
We built Bluesky as an always-on source (like HN and Polymarket) because the AT Protocol public API was documented as free. That's no longer true for post search.
**Current behavior:** Bluesky silently returns 0 results (403 caught, empty list returned). No stats line appears, no error shown. Users don't know it exists.
**Desired behavior:** Bluesky is opt-in via env vars. When not configured, completely invisible (no error, no stats line, no mention). When configured with auth, it works as before.
## Approach
Follow the same pattern as X/Twitter auth: env vars for credentials, availability check gates dispatch.
AT Protocol auth flow:
1. User creates an App Password at `bsky.app/settings/app-passwords`
2. User sets `BSKY_HANDLE` and `BSKY_APP_PASSWORD` env vars
3. `bluesky.py` calls `com.atproto.server.createSession` to get a bearer token
4. Subsequent `searchPosts` calls include `Authorization: Bearer {token}`
## Files to Change
| File | Change | Lines (est.) |
|------|--------|-------------|
| `scripts/lib/bluesky.py` | Add `_create_session()` auth, pass bearer token to search | ~25 |
| `scripts/lib/env.py` | Update `is_bluesky_available()` to check env vars; add env var loading | ~10 |
| `scripts/last30days.py` | Gate `do_bluesky` on `is_bluesky_available(config)`; update diag dicts | ~8 |
| `tests/test_bluesky.py` | Add auth session tests, update existing tests | ~15 |
| `SKILL.md` | Document `BSKY_HANDLE` / `BSKY_APP_PASSWORD` env vars in setup section | ~5 |
## Implementation Steps
### 1. `scripts/lib/env.py`
- Add `BSKY_HANDLE` and `BSKY_APP_PASSWORD` to the env var loading in `get_config()`
- Update `is_bluesky_available()`:
```python
def is_bluesky_available(config: Dict[str, Any]) -> bool:
return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))
```
### 2. `scripts/lib/bluesky.py`
- Add `_create_session(handle, app_password)` that POSTs to `https://bsky.social/xrpc/com.atproto.server.createSession`
- Cache the access token for the lifetime of the search (module-level or passed through)
- Update `search_bluesky()` to accept config dict, extract creds, create session, add `Authorization: Bearer {token}` header
- On auth failure, return `{"posts": [], "error": "Bluesky auth failed"}` (don't raise)
### 3. `scripts/last30days.py`
- Change `do_bluesky` default from `True` to checking `is_bluesky_available(config)`:
```python
has_bluesky = env.is_bluesky_available(config)
```
- Gate the `bluesky_future` submit on `has_bluesky` (already gated on `do_bluesky`, just wire it)
- Update both diagnostic dicts: `"bluesky": True` -> `"bluesky": has_bluesky`
- Pass `config` to `_search_bluesky()` so it can extract auth creds
### 4. `tests/test_bluesky.py`
- Test `_create_session` with mocked HTTP response
- Test `search_bluesky` with auth header injection
- Existing parse/normalize tests don't change (they don't hit the network)
### 5. `SKILL.md`
- Add Bluesky to the optional env vars section (near AUTH_TOKEN/CT0):
```
BSKY_HANDLE=your-handle.bsky.social
BSKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx
```
- Keep Bluesky in source lists but note "(requires app password)" in setup
## What NOT to Change
- `render.py` - already handles empty bluesky gracefully (hides stats line when 0 items)
- `schema.py`, `normalize.py`, `score.py`, `dedupe.py` - no changes needed, they work on items already
- No error display when unconfigured - completely silent, same as how TikTok/Instagram behave when SCRAPECREATORS_API_KEY is missing
## Acceptance Criteria
- [x] `is_bluesky_available()` returns False when env vars not set
- [x] No Bluesky stats line, no error, no mention when unconfigured
- [x] With valid `BSKY_HANDLE` + `BSKY_APP_PASSWORD`, posts are returned
- [x] Auth failure returns empty results gracefully (no crash)
- [x] SKILL.md documents the env vars
- [x] All existing Bluesky tests still pass
- [x] New auth tests added
- [x] `bash scripts/sync.sh` deploys successfully
## Sources
- AT Protocol auth: `POST https://bsky.social/xrpc/com.atproto.server.createSession` with `{identifier, password}`
- App passwords: `bsky.app/settings/app-passwords`
- Existing pattern: X auth with AUTH_TOKEN/CT0 in `env.py:256-257`
- Existing pattern: TikTok/Instagram opt-in via `is_tiktok_available()` in `env.py:499`