Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.
- Add scripts/lib/instagram.py: keyword search, transcript extraction,
relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.
- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.
--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn (run only Reddit + Hacker News)
Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: fix YAML error in argument-hint
* add codex auth support to responses API
* Use gpt-5.1-codex-mini as default model for Codex auth
Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.
* Pass .env credentials to Bird Node subprocesses for X auth
On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.
- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection
---------
Co-authored-by: Justin Williams <jblwilliams@gmail.com>
- _compute_text_similarity() now checks outcome names with bidirectional
substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.
- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics
(4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further:
- dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold,
cross-source links went from 3 to 13 items across 5 topics
- render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags
- youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score)
- SKILL.md: instruction #7 tells Claude to lead with cross-platform signals
Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5
topics. Biggest gains in specificity (+0.8) and format compliance (+1.0).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.
Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HN was appearing before YouTube in the stats block, sort tiebreaker,
and source status. Now consistently: Reddit > X > YouTube > HN > Web.
Also restored emoji + box-drawing chars in test skill SKILL.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:
- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override
Also fixes the UX gap where missing sources were silently skipped:
- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add agents/openai.yaml for Codex skill discovery
- Make SKILL.md script path portable (repo, Claude, Codex, agents dirs)
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dirs with env var overrides and tempdir fallback
- Add Codex installation docs to README
Inspired by PR #24 (el-analista) and PR #5 (jblwilliams).
Zero impact on existing Claude Code behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube search and transcript extraction runs automatically when yt-dlp
is installed. Searches for topic videos from the last N days, fetches
auto-generated transcripts for top results, and feeds them through the
same scoring pipeline (relevance + recency + engagement) as Reddit/X.
New files:
- youtube_yt.py: search, transcript extraction, VTT cleanup
Modified files:
- schema.py: YouTubeItem dataclass, updated Report
- normalize.py: normalize_youtube_items()
- score.py: YouTube engagement scoring (views-dominated)
- dedupe.py: YouTube deduplication
- render.py: YouTube section in compact output
- env.py: is_ytdlp_available() check
- ui.py: YouTube progress messages
- last30days.py: _search_youtube(), parallel execution with Reddit/X
- SKILL.md: YouTube in stats box, citation priority
- README.md: YouTube docs, yt-dlp requirement, Peter shoutout
Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ⚠️ LIMITED RECENT DATA warning when:
- Fewer than 5 items are confirmed from the date range
- Tells Claude to be transparent with user about data freshness
Example output for obscure topic (June Oven):
"Only 4 item(s) confirmed from 2025-12-26 to 2026-01-25.
Results below may include older/evergreen content."
Popular topics (clawdbot, nano banana) don't show the warning.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Skill now works without any API keys using WebSearch fallback
- Shows promo banner marketing Reddit/X data when keys are missing
- Partial mode (one key) shows shorter tip for the missing source
- Updated SKILL.md to document three modes: Full, Partial, Web-Only
- Added get_missing_keys() to env.py for promo logic
- Added show_promo(), start_web_only(), show_web_only_complete() to ui.py
- Updated render_compact() to include inline promo for web-only mode
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Claude's built-in WebSearch tool as a third research source for
/last30days. This enables the skill to work out of the box with zero
API keys while preserving Reddit/X as the primary sources.
Key changes:
- Add WebSearchItem schema for web results (no engagement metrics)
- Add score_websearch_items() with 55/45 relevance/recency weighting
- Apply -15pt source penalty so WebSearch ranks below Reddit/X
- Add --include-web CLI flag to opt-in to WebSearch
- Return 'web' mode when no API keys configured (zero-config)
- Update render.py with [WEB] source label formatting
When WebSearch is enabled, the script outputs instructions for Claude
to use its built-in WebSearch tool, then synthesize results together.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Show "⚡ CACHED RESULTS (Xh old)" in compact output header
- Add "use --refresh for fresh data" hint
- Track from_cache and cache_age_hours in Report schema
- Update UI to show cache age in stderr message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- SKILL.md: Move "What I learned" BEFORE "Research Complete" stats
- Add error tracking to Report schema (reddit_error, x_error fields)
- Wrap OpenAI API calls in try/catch with clear error messages
- Show explicit error or "no results" messages in compact output
- Fix false positive error detection for null error fields
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Research topics across Reddit + X from the last 30 days using
OpenAI and xAI APIs. Features:
- Auto model selection (GPT-5.x, Grok-3)
- Popularity-aware scoring (relevance + recency + engagement)
- Reddit thread enrichment with real metrics
- Near-duplicate detection
- Multiple emit modes (compact, json, context, path)
- 24h caching with --refresh bypass
- NUX for API key setup
- 87 passing unit tests
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Matt Van Horn