YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.
Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reddit and X are the star of the show. In Claude Code (non-TTY),
suppress ⏳ start messages for HN and YouTube so Reddit/X are the
first visible lines. HN/YouTube still show ✓ completion messages.
Also suppress [HN] debug logs in non-TTY to reduce output clutter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HN was appearing before YouTube in the stats block, sort tiebreaker,
and source status. Now consistently: Reddit > X > YouTube > HN > Web.
Also restored emoji + box-drawing chars in test skill SKILL.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Reddit search uses OpenAI's Responses API with web_search, which
frequently returns 429 rate limit errors. The previous retry logic used
linear backoff (1s, 2s, 3s) which is too aggressive for OpenAI's rate
limiter (often needs 10-60s waits).
Changes:
- Increase max retries from 3 to 5
- Switch from linear to exponential backoff (2s, 5s, 9s, 17s, 33s)
- Parse and respect Retry-After header from OpenAI 429 responses
- Fall back to cheaper models (gpt-4.1 → gpt-4o) on 429s, not just
on 400/403 access errors
- Remove gpt-4o-mini from fallback chain — it doesn't support
web_search with the filters parameter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add CHANGELOG.md (Keep a Changelog format) and release-notes.md
for the v2.1.0 GitHub Release. Also includes queued copy updates:
plugin.json version bump to 2.1.0, YouTube added to descriptions,
SKILL.md security/permissions section and ClawdBot metadata,
updated watchlist examples, and v2.1 launch post refinements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When X posts return engagement: null, dict.get("engagement", {})
returns None (key exists with null value), causing AttributeError.
Use `or {}` idiom to coalesce None to empty dict.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OpenAI Responses API web_search takes 60-90s but the generic
future timeout was killing it at 30s (quick) / 60s (default).
Added reddit_future key to TIMEOUT_PROFILES (60/90/120s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube: Add youtube_future timeout key (60/90/120s for quick/default/deep)
separate from the shared future timeout. YouTube needs more time because
it does search + parallel transcript fetching. Previously, 20 videos +
5 transcripts exceeded the 60s budget and all results were discarded.
Reddit 429: Propagate rate-limit errors instead of swallowing them.
Enrichment now uses 10s timeout / 1 retry (was 30s / 3 retries).
On first 429, cancel remaining enrichment and skip Phase 2 Reddit.
Total time wasted on 429 drops from ~75s to ~12s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hard date filter in the main script was double-filtering YouTube
results. youtube_yt.py already applies a soft date filter that prefers
recent videos but keeps older ones when < 3 are within range (for
evergreen topics like "youtube thumbnails"). The hard filter then
removed all of them, resulting in 0 YouTube items.
YouTube content has a longer shelf life than tweets/posts, so the
soft filter's fallback behavior is correct.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The `from . import bird_x` in __init__.py caused ImportError for all
other module imports (youtube_yt, render, etc.) when bird_x had issues,
crashing the entire script in Codex. Making it a bare package marker
lets Python resolve `from lib import X` normally.
Added scripts/sync.sh to deploy to all three skill locations
(~/.claude, ~/.agents, ~/.codex) with import verification.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace CLI-style promo banners with casual, conversational NUX message
that teaches users to say "last30" + natural language. Remove stale Bird
CLI npm install references and dead prompt_bird_install methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:
- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override
Also fixes the UX gap where missing sources were silently skipped:
- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove 2>&1 from SKILL.md so stderr doesn't pollute model input
- Run script in foreground (not background) with 5min timeout
- Add explicit YouTube synthesis instruction for Claude
- Remove --flat-playlist which broke date filtering (all dates were None)
- Move date filtering to Python with soft fallback for evergreen topics
- Keep 'tips', 'tutorial', 'review', 'guide' in YouTube search queries
- Increase yt-dlp timeout from 60s to 120s for full metadata fetch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _search_web() dispatches to Parallel AI / Brave / OpenRouter based on config
- Web results flow through full pipeline: normalize → score → dedupe
- --diagnose shows all source availability (API keys, Bird, YouTube, web backends)
- --store persists findings to SQLite via store.py for watchlist/briefing system
- run_research() now returns web_items alongside reddit/x/youtube
- web_needed flag only set when no native web backend is available
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add agents/openai.yaml for Codex skill discovery
- Make SKILL.md script path portable (repo, Claude, Codex, agents dirs)
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dirs with env var overrides and tempdir fallback
- Add Codex installation docs to README
Inspired by PR #24 (el-analista) and PR #5 (jblwilliams).
Zero impact on existing Claude Code behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube search and transcript extraction runs automatically when yt-dlp
is installed. Searches for topic videos from the last N days, fetches
auto-generated transcripts for top results, and feeds them through the
same scoring pipeline (relevance + recency + engagement) as Reddit/X.
New files:
- youtube_yt.py: search, transcript extraction, VTT cleanup
Modified files:
- schema.py: YouTubeItem dataclass, updated Report
- normalize.py: normalize_youtube_items()
- score.py: YouTube engagement scoring (views-dominated)
- dedupe.py: YouTube deduplication
- render.py: YouTube section in compact output
- env.py: is_ytdlp_available() check
- ui.py: YouTube progress messages
- last30days.py: _search_youtube(), parallel execution with Reddit/X
- SKILL.md: YouTube in stats box, citation priority
- README.md: YouTube docs, yt-dlp requirement, Peter shoutout
Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vendor Bird's Twitter GraphQL search client directly into /last30days,
eliminating the dependency on `npm install -g @steipete/bird`. X search
now works out of the box with just Node.js 22+ and browser cookies.
- Add vendored bird-search.mjs wrapper (search-only subset of Bird v0.8.0)
- Vendor @steipete/sweet-cookie for browser cookie extraction
- Update bird_x.py to call vendored Node.js module instead of `bird` binary
- Update README.md and SKILL.md for v2.1 (remove Bird CLI install steps)
- Include Bird's MIT LICENSE in vendor directory
The fallback chain is: vendored search -> xAI API key -> web-only mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in _extract_core_subject():
1. Multi-word noise phrases ("what are", "how to") never matched
because code compared individual words against multi-word strings.
"what are people saying about DeepSeek R1" became "what are people
saying" — losing the entire topic.
2. Missing meta words — "prompt", "techniques", "tips" weren't
filtered (only "prompting" was). "vibe motion best prompt
techniques" kept 4 keywords instead of 2.
3. No retry on 0 results — Reddit retries with simplified queries
but X accepted 0 and moved on.
Fix: Two-phase extraction (strip multi-word prefixes/suffixes first,
then individual noise words), expanded noise set, max 3 words (was 4),
and automatic retry with first 2 words when Bird returns 0 results.
Before → After:
- "vibe motion best prompt techniques" → "vibe motion" (was 4 words, 0 results)
- "what are people saying about DeepSeek R1" → "deepseek r1" (was "what are people saying")
- "nano banana pro prompts for gemini" → "nano banana pro" (was 4 words)
Tested: vibe motion (12 X posts, was 0), DeepSeek R1 (12 posts),
kanye west (12 posts, no regression).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add V2 to title
- Document Bird CLI installation (free X search)
- Add --days=N to options table
- Update Requirements to show Bird CLI as recommended free option
- Replace How It Works with two-phase search architecture
- Add What's New in V2 section with all new features
- Credit community contributors (@JosephOIbrahim, @levineam, @jonthebeef)
- Bump http.py USER_AGENT from 1.0 to 2.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the initial broad search (Phase 1), extract key entities from results
and run targeted secondary searches to surface content the broad pass missed:
- New entity_extract.py: parses @handles, #hashtags, subreddits from results
- bird_x.py: search_handles() does targeted from:handle searches via Bird CLI
- openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint
- last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes
Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X).
Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stats: replace BAD/GOOD examples with strict fill-in-the-blank template
- Reddit: add subreddit-targeted fallback search, soften scoring penalties
(engagement -10→-3, date confidence -10→-5), add minimum result guarantee
- Citations: limit to 1 per insight, short format, no engagement metrics
- Summary: add bold topic headers template for structured paragraphs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Add _extract_core_subject() to strip noise words before X search
- X search is literal keyword matching, not semantic
- "best open claw usecases" → "open claw" (27 results vs 0)
2. Fix ISO date detection bug
- Was: if "T" in created_at (matched "Tue" in "Tue Feb 03...")
- Now: if created_at[10] == "T" (proper ISO separator check)
- Dates now parse correctly: (2026-02-03) instead of (date unknown)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. last30days.py: Check isinstance(dict) before .get() - Bird returns
list on success, dict on error
2. bird_x.py: Update field mappings for Bird's actual response format:
- author.username not user.screen_name
- createdAt not created_at (camelCase)
- likeCount, retweetCount, etc. (camelCase)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Bird CLI doesn't support --since as a CLI flag.
Use X's search syntax: 'topic since:YYYY-MM-DD'
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove interactive "Install Bird CLI?" prompt
- Add Bird as FREE option in promo messages
- Auto-detect Bird silently (use if authenticated)
- Keep original flow: show promo → continue with available sources
Bird now appears in the promo alongside API keys:
🔵 X (Twitter)
└─ FREE: npm install -g @steipete/bird (uses browser session)
└─ Or: Add XAI_API_KEY (paid API)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move Bird CLI check to run before topic validation so users see
the install prompt even when running without a topic argument.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add LAST30DAYS_CONFIG_DIR env var to override config location
- Set LAST30DAYS_CONFIG_DIR="" in test skill for clean mode
- Test skill now ignores existing API keys to show Bird install prompt
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move progress display initialization early to support Bird prompts
- Check Bird availability and offer install if no X source available
- Override available sources when Bird is ready
- Pass x_source parameter to run_research function
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add bird_x import and setup_bird_if_needed function to main script.
The function checks Bird status and offers installation if needed,
returning 'bird' if ready, 'declined' if user declined, or None if
unavailable.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- is_bird_installed() checks if bird command is in PATH
- is_bird_authenticated() runs bird whoami to verify auth
- check_npm_available() checks if npm can install bird
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix invalid YAML in SKILL.md argument-hint (closes#8)
Wrapped value in single quotes to properly escape double quotes
- Add automatic model fallback for GPT-5 access errors (closes#9)
When OpenAI returns 400 for unverified orgs, retry with gpt-4o
- Add tests for model fallback logic
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Refactor run_research() to use ThreadPoolExecutor for parallel execution
- Reddit timeout/crash no longer blocks X search from running
- Add catch for ConnectionResetError/OSError in http.py
- Per-item error handling in Reddit enrichment (one failure doesn't crash all)
- Increase API timeouts from 60/90/120 to 90/120/180 seconds
- Add ClawdBot setup example to README
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds ⚠️ LIMITED RECENT DATA warning when:
- Fewer than 5 items are confirmed from the date range
- Tells Claude to be transparent with user about data freshness
Example output for obscure topic (June Oven):
"Only 4 item(s) confirmed from 2025-12-26 to 2026-01-25.
Results below may include older/evergreen content."
Popular topics (clawdbot, nano banana) don't show the warning.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
If initial search returns <5 threads, extract core subject and retry:
- "best nano banana prompting practices" → retry with "nano banana"
- Combines results from both searches, deduped by URL
Note: OpenAI's web_search still tends to find old content. This retry
helps cast a wider net but doesn't fully solve the recency issue.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
OpenAI's web_search doesn't reliably find recent Reddit content.
Strategy change: request MORE threads and let server-side date
filtering handle it.
Changes:
- Increased depth config (20-30 → 30-50 default)
- Simplified prompt to search broadly and include all matches
- Let server-side date filter remove old content
- Removed aggressive pre-filtering that was limiting results
Tradeoff: Some topics have more X coverage than Reddit due to
web_search limitations. This is acceptable.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The model was finding old threads (Nov-Dec 2025) that got filtered out,
leaving 0 results even for popular topics like "nano banana prompts".
Changes:
- Add explicit date filters: "after:{from_date}" and "2026"/"January 2026"
- Emphasize RECENT content is critical, old threads are filtered anyway
- Tell model to verify dates before including threads
Before: 0-2 Reddit threads for "nano banana prompting practices"
After: 16 Reddit threads from January 2026
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add EXTRACT CORE KEYWORDS step to search for main subject, not full phrase
("killer features of clawdbot" → search "clawdbot")
- Remove hardcoded subreddit list that biased results toward design/dev subs
- Simplify search strategies and URL validation rules
- Make prompt more concise - GPT-5.2 responds better to clearer instructions
Before: 0 threads found for "killer features of clawdbot"
After: 7 threads found including r/LocalLLaMA (43pts), r/selfhosted (26pts)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Skill now works without any API keys using WebSearch fallback
- Shows promo banner marketing Reddit/X data when keys are missing
- Partial mode (one key) shows shorter tip for the missing source
- Updated SKILL.md to document three modes: Full, Partial, Web-Only
- Added get_missing_keys() to env.py for promo logic
- Added show_promo(), start_web_only(), show_web_only_complete() to ui.py
- Updated render_compact() to include inline promo for web-only mode
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously Reddit was returning ~60% old content (some from 2022).
This commit adds multiple layers of date enforcement:
- Reddit prompt: Explicit from_date/to_date with "fewer results > older results"
- Hard filter: filter_by_date_range() in normalize.py excludes old content
- WebSearch Date Detective: Extracts dates from URLs (/2026/01/24/) and
snippets ("January 24, 2026", "3 days ago")
- WebSearch scoring: +10 bonus for verified dates, -20 penalty for unknown
The skill now guarantees only content from the last 30 days.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Claude's built-in WebSearch tool as a third research source for
/last30days. This enables the skill to work out of the box with zero
API keys while preserving Reddit/X as the primary sources.
Key changes:
- Add WebSearchItem schema for web results (no engagement metrics)
- Add score_websearch_items() with 55/45 relevance/recency weighting
- Apply -15pt source penalty so WebSearch ranks below Reddit/X
- Add --include-web CLI flag to opt-in to WebSearch
- Return 'web' mode when no API keys configured (zero-config)
- Update render.py with [WEB] source label formatting
When WebSearch is enabled, the script outputs instructions for Claude
to use its built-in WebSearch tool, then synthesize results together.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Matt Van Horn