Compare commits

...

108 Commits

Author SHA1 Message Date
Matt Van Horn a4d09e470e docs: add CHANGELOG.md, release notes, and v2.1 polish
Add CHANGELOG.md (Keep a Changelog format) and release-notes.md
for the v2.1.0 GitHub Release. Also includes queued copy updates:
plugin.json version bump to 2.1.0, YouTube added to descriptions,
SKILL.md security/permissions section and ClawdBot metadata,
updated watchlist examples, and v2.1 launch post refinements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 12:16:01 -08:00
Matt Van Horn 4547a095bf fix: correct OpenClaw GitHub URL
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:34:57 -08:00
Matt Van Horn 6ca037f387 docs: clarify watchlist is designed for Open Claw / always-on bots
Watchlist schedules are metadata only — nothing triggers runs
automatically. Updated README to be honest that cron/launchd or an
always-on bot like Open Claw is required for automated scheduling.
Local Claude Code users can still use run-one and run-all manually.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:33:43 -08:00
Matt Van Horn 7d6bac7b0e fix: handle engagement: null in watchlist and YouTube normalizer
When X posts return engagement: null, dict.get("engagement", {})
returns None (key exists with null value), causing AttributeError.
Use `or {}` idiom to coalesce None to empty dict.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:29:27 -08:00
Matt Van Horn 87bb469ab7 docs: remove em dashes, update terminology, replace Ralph Wiggum
- Replace all em dashes with hyphens
- "Claude Code skill" -> "skill" in intro
- Ralph Wiggum -> Seedance 2.0 / Nano Banana Pro
- tweet/Tweet -> post/Post throughout
- "signal of a tweet" -> "signal of a single post"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 01:12:22 -08:00
Matt Van Horn 3bf20ef072 docs: reframe v2.1 — watchlist #1, YouTube #2, Codex #3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 01:08:35 -08:00
Matt Van Horn 2d6855483d fix: Reddit future timeout — same bug as YouTube
OpenAI Responses API web_search takes 60-90s but the generic
future timeout was killing it at 30s (quick) / 60s (default).
Added reddit_future key to TIMEOUT_PROFILES (60/90/120s).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:57:32 -08:00
Matt Van Horn 20a859ecec fix: YouTube timeout bump to 90s + Reddit 429 fail-fast
YouTube: Add youtube_future timeout key (60/90/120s for quick/default/deep)
separate from the shared future timeout. YouTube needs more time because
it does search + parallel transcript fetching. Previously, 20 videos +
5 transcripts exceeded the 60s budget and all results were discarded.

Reddit 429: Propagate rate-limit errors instead of swallowing them.
Enrichment now uses 10s timeout / 1 retry (was 30s / 3 retries).
On first 429, cancel remaining enrichment and skip Phase 2 Reddit.
Total time wasted on 429 drops from ~75s to ~12s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:48:26 -08:00
Matt Van Horn c3640931ed fix: skip hard date filter for YouTube — keep evergreen content
The hard date filter in the main script was double-filtering YouTube
results. youtube_yt.py already applies a soft date filter that prefers
recent videos but keeps older ones when < 3 are within range (for
evergreen topics like "youtube thumbnails"). The hard filter then
removed all of them, resulting in 0 YouTube items.

YouTube content has a longer shelf life than tweets/posts, so the
soft filter's fallback behavior is correct.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:32:36 -08:00
Matt Van Horn 1acb5c6fff fix: remove eager import from __init__.py and add sync.sh
The `from . import bird_x` in __init__.py caused ImportError for all
other module imports (youtube_yt, render, etc.) when bird_x had issues,
crashing the entire script in Codex. Making it a bare package marker
lets Python resolve `from lib import X` normally.

Added scripts/sync.sh to deploy to all three skill locations
(~/.claude, ~/.agents, ~/.codex) with import verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:31:50 -08:00
Matt Van Horn 14b6ac06e1 feat(nux): conversational NUX with "last30" trigger and dynamic source status
Replace CLI-style promo banners with casual, conversational NUX message
that teaches users to say "last30" + natural language. Remove stale Bird
CLI npm install references and dead prompt_bird_install methods.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:19:38 -08:00
Matt Van Horn 06f74a4d0c fix: add timeouts, process cleanup, and source diagnostics
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:

- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override

Also fixes the UX gap where missing sources were silently skipped:

- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:05:36 -08:00
Matt Van Horn 7162eb6b36 fix: YouTube display and search quality
- Remove 2>&1 from SKILL.md so stderr doesn't pollute model input
- Run script in foreground (not background) with 5min timeout
- Add explicit YouTube synthesis instruction for Claude
- Remove --flat-playlist which broke date filtering (all dates were None)
- Move date filtering to Python with soft fallback for evergreen topics
- Keep 'tips', 'tutorial', 'review', 'guide' in YouTube search queries
- Increase yt-dlp timeout from 60s to 120s for full metadata fetch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:02:16 -08:00
Matt Van Horn e520db31d3 docs: mark all plan acceptance criteria as complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:48:15 -08:00
Matt Van Horn 23b7906d63 docs: add open variant section and new CLI flags to README
- Document open variant installation (watchlist, briefings, history)
- Add --store, --diagnose, --include-web to options table
- Note optional web search API keys (Parallel AI, Brave, OpenRouter)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:47:54 -08:00
Matt Van Horn 9ff00ce38b feat(open): create open variant SKILL.md and reference files
Router SKILL.md dispatches to mode-specific references:
- research.md: one-shot research with --store for persistence
- watchlist.md: add/remove/list/run topics
- briefing.md: daily/weekly briefing generation
- history.md: query/search accumulated findings
- context.md: agent memory template

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:46:35 -08:00
Matt Van Horn 08e2010554 feat(engine): add native web search, --store, and --diagnose to main engine
- _search_web() dispatches to Parallel AI / Brave / OpenRouter based on config
- Web results flow through full pipeline: normalize → score → dedupe
- --diagnose shows all source availability (API keys, Bird, YouTube, web backends)
- --store persists findings to SQLite via store.py for watchlist/briefing system
- run_research() now returns web_items alongside reddit/x/youtube
- web_needed flag only set when no native web backend is available

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:43:34 -08:00
Matt Van Horn ba330e9a0c feat(open): Port web search backends, persistence layer, and env merge from openclaw 2026-02-14 23:35:44 -08:00
Matt Van Horn a09413608d feat: Add Codex CLI compatibility
- Add agents/openai.yaml for Codex skill discovery
- Make SKILL.md script path portable (repo, Claude, Codex, agents dirs)
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dirs with env var overrides and tempdir fallback
- Add Codex installation docs to README

Inspired by PR #24 (el-analista) and PR #5 (jblwilliams).
Zero impact on existing Claude Code behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:18:53 -08:00
Matt Van Horn 9397fcc937 docs: Polish README for v2.1 launch + add launch copy with social posts
- Split V2.1 blurb into two clear bullets (YouTube + X search)
- Better X auth section with browser-specific notes and verify command
- Updated intro/tagline to mention YouTube as 4th source
- Added social post drafts and updated issue #19 response with YouTube

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 22:50:47 -08:00
Matt Van Horn c66ca7f43d feat: Add YouTube as 4th research source via yt-dlp
YouTube search and transcript extraction runs automatically when yt-dlp
is installed. Searches for topic videos from the last N days, fetches
auto-generated transcripts for top results, and feeds them through the
same scoring pipeline (relevance + recency + engagement) as Reddit/X.

New files:
- youtube_yt.py: search, transcript extraction, VTT cleanup

Modified files:
- schema.py: YouTubeItem dataclass, updated Report
- normalize.py: normalize_youtube_items()
- score.py: YouTube engagement scoring (views-dominated)
- dedupe.py: YouTube deduplication
- render.py: YouTube section in compact output
- env.py: is_ytdlp_available() check
- ui.py: YouTube progress messages
- last30days.py: _search_youtube(), parallel execution with Reddit/X
- SKILL.md: YouTube in stats box, citation priority
- README.md: YouTube docs, yt-dlp requirement, Peter shoutout

Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 21:38:04 -08:00
Matt Van Horn 31313c69ac v2.1: Bundle Bird X search - no external CLI needed
Vendor Bird's Twitter GraphQL search client directly into /last30days,
eliminating the dependency on `npm install -g @steipete/bird`. X search
now works out of the box with just Node.js 22+ and browser cookies.

- Add vendored bird-search.mjs wrapper (search-only subset of Bird v0.8.0)
- Vendor @steipete/sweet-cookie for browser cookie extraction
- Update bird_x.py to call vendored Node.js module instead of `bird` binary
- Update README.md and SKILL.md for v2.1 (remove Bird CLI install steps)
- Include Bird's MIT LICENSE in vendor directory

The fallback chain is: vendored search -> xAI API key -> web-only mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 17:16:04 -08:00
Matt Van Horn 0ec338c630 Add speed tradeoff note to README intro
Honest about 2-8 min research time vs V1's speed. Points users
to @mvanhorn / @slashlast30days for quick mode feedback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 11:01:54 -08:00
Matt Van Horn 205f8fa7ae Add timing disclaimer to parsed intent output
"Research typically takes 2-8 minutes (niche topics take longer)."
Shown right after parsed intent, before research starts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 11:00:06 -08:00
Matt Van Horn fdfe3c7e9e Add Vibe Motion as third README example showing brand-new tool research
Demonstrates /last30days researching a tool that launched 2 days ago —
before blog roundups existed. Real-time X/Reddit reactions from
@Hartdrawss, @rezkhere, @Totinhiiio plus honest Chase Jarvis review.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:57:35 -08:00
Matt Van Horn ff5cc1445c Add Kanye West as second README example showing current events mode
Real V2 output: 33 X posts (9.3K likes), @dekreationz viral post,
WSJ apology arc, BULLY album, Bianca Vanity Fair — demonstrates
/last30days as a news/culture research tool, not just prompting.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:56:25 -08:00
Matt Van Horn 04b532582c Add Nano Banana Pro as first README example, remove old duplicate
New example uses fresh V2 research output showing JSON prompting
dominance, @handle citations, and the full research-to-prompt flow.
Removed the old iOS App Mockup example which used the same SwimMom
prompt/image but with stale V1 data.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:54:01 -08:00
Matt Van Horn 45702efd63 Update argument-hint with real example queries
Replace generic "[topic]" and --days=7 with concrete examples that
show the range: prompting, news, recommendations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:49:52 -08:00
Matt Van Horn 232a948576 Reframe README V2 around better search results as primary feature
Lead with dramatically better X/Reddit search quality (smarter query
construction + Phase 2 supplemental), Bird CLI as secondary headline,
community contributions in their own section.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:46:08 -08:00
Matt Van Horn 41779b81c0 Fix X search returning 0 results on popular topics
Three bugs in _extract_core_subject():

1. Multi-word noise phrases ("what are", "how to") never matched
   because code compared individual words against multi-word strings.
   "what are people saying about DeepSeek R1" became "what are people
   saying" — losing the entire topic.

2. Missing meta words — "prompt", "techniques", "tips" weren't
   filtered (only "prompting" was). "vibe motion best prompt
   techniques" kept 4 keywords instead of 2.

3. No retry on 0 results — Reddit retries with simplified queries
   but X accepted 0 and moved on.

Fix: Two-phase extraction (strip multi-word prefixes/suffixes first,
then individual noise words), expanded noise set, max 3 words (was 4),
and automatic retry with first 2 words when Bird returns 0 results.

Before → After:
- "vibe motion best prompt techniques" → "vibe motion" (was 4 words, 0 results)
- "what are people saying about DeepSeek R1" → "deepseek r1" (was "what are people saying")
- "nano banana pro prompts for gemini" → "nano banana pro" (was 4 words)

Tested: vibe motion (12 X posts, was 0), DeepSeek R1 (12 posts),
kanye west (12 posts, no regression).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:43:45 -08:00
Matt Van Horn 83f8d7ece5 Add V2 highlights to README intro, credit @galligan for plugin inspiration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:28:54 -08:00
Matt Van Horn 52456e5820 Update README for V2 and bump USER_AGENT to 2.0
- Add V2 to title
- Document Bird CLI installation (free X search)
- Add --days=N to options table
- Update Requirements to show Bird CLI as recommended free option
- Replace How It Works with two-phase search architecture
- Add What's New in V2 section with all new features
- Credit community contributors (@JosephOIbrahim, @levineam, @jonthebeef)
- Bump http.py USER_AGENT from 1.0 to 2.0

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:26:53 -08:00
Matt Van Horn 6c620cbbf4 Smart invitations with research-specific example suggestions
Every invitation now includes 2-3 specific examples based on what the
research actually found. PROMPTING mode explicitly offers to write prompts
with creative examples. NEWS suggests follow-up questions about the
biggest stories. GENERAL suggests specific deep-dives and applications.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:11:30 -08:00
Matt Van Horn 88bca6e55c Fix invitation wording: add 'I'm now an expert' to all modes, remove prompt language from GENERAL
GENERAL mode said "write a prompt for it" which sounds weird for people/topics
("write a prompt for Kanye West"). Changed to "describe what you're working on
and I'll help." Added "I'm now an expert on {TOPIC}" prefix to all 4 modes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:04:10 -08:00
Matt Van Horn 1ae7a16c75 feat: Smart supplemental search — Phase 2 entity-aware drill-down
After the initial broad search (Phase 1), extract key entities from results
and run targeted secondary searches to surface content the broad pass missed:

- New entity_extract.py: parses @handles, #hashtags, subreddits from results
- bird_x.py: search_handles() does targeted from:handle searches via Bird CLI
- openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint
- last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes

Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X).
Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:00:28 -08:00
Matt Van Horn 650aa1100b Context-aware invitation, smart follow-ups, no raw URLs
- Invitation adapts to QUERY_TYPE: NEWS gets "ask me anything",
  PROMPTING gets "share your vision", RECOMMENDATIONS gets "compare options"
- Follow-up handler reads intent: questions get answers, not forced prompts
- Restored V1's "if they ask a question, answer it" context memory line
- URL formatting rule: "per Rolling Stone" not raw URLs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:34:30 -08:00
Matt Van Horn 6da4b2f5c7 Remove citation priority plan file
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:22:42 -08:00
Matt Van Horn 9da95019e6 Elevate Reddit/X citations over web sources in output
- Add citation priority rule: @handles > r/subreddits > web sources
- Fix examples: "per @honest30bgfan_" instead of "per Rolling Stone"
- Add "lead with people, not publications" instruction
- Stats box: web now shows "(supplementary)" instead of domain list
- No changes to actual research — same quality, better attribution framing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:22:07 -08:00
Matt Van Horn a953d57730 Add PR credits tracking for V2 launch
Track community PRs to thank contributors when V2 goes public.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:16:23 -08:00
Matt Van Horn 6d80af3807 Add Claude Code marketplace plugin support
- Add .claude-plugin/marketplace.json and plugin.json manifests
- Use ${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/skills/last30days} for script path
- Plugin install: uses CLAUDE_PLUGIN_ROOT when set
- Manual install: falls back to ~/.claude/skills/last30days (no breakage)
- SKILL.md stays at root, plugin.json points skills to "./"

Users can now install via:
  claude plugin marketplace add mvanhorn/last30days-skill
  claude plugin install last30days@last30days

Inspired by PR #1 (galligan) — lighter approach without repo restructure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:11:28 -08:00
Matt Van Horn 60f06ea86e Merge community PRs: Windows Unicode fix, 403 fallback, --days flag
- PR #17 (JosephOIbrahim): Fix UnicodeEncodeError on Windows cp1252
- PR #16 (levineam): Handle HTTP 403 model access errors, add gpt-4.1 fallback
- PR #18 (jonthebeef): Add --days=N flag for configurable lookback (1-30)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:05:46 -08:00
Matt Van Horn c8ddb60397 Revert query parsing display to V1 format
User preferred V1's "I'll research X across Reddit, X, and the web..."
with explicit Parsed intent block over V2's emoji format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 08:18:37 -08:00
Matt Van Horn 10f61c503d Restore 7 missing v1 features + add query parsing display + v1-vs-v2 test plan
Feature audit found 11 things v2 dropped from v1. Restored the 7 that affect
output quality: quality checklist for prompts, anti-pattern examples, self-check
instruction, "don't re-search" context memory, prompt format anti-pattern,
multi-prompt guidance, and emoji footer with engagement counts.

Also adds visible query parsing display (🔍 **{TOPIC}** · {QUERY_TYPE}) and
the full v1-vs-v2 comparison test plan with all 17 test queries from README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 15:27:46 -08:00
Matt Van Horn 4b0098b754 Remove context: fork — root cause of broken skill execution
The old v1 skill worked because context: fork was SILENTLY IGNORED
due to Claude Code bug #17283. The skill ran inline in the main
conversation. Claude Code 2.1+ fixed the bug and now properly
honors fork mode, creating an isolated subagent that ignores all
instruction ordering (text output, bash-first, etc.).

Fix: remove context: fork so the skill runs inline again, matching
the behavior the user knows works. Also restored v1-style
instruction flow: parse intent first, then run script, then WebSearch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:28:58 -08:00
Matt Van Horn df29bbb53d Revert to bash-first execution with echo progress display
The agent in fork mode ignores text output instructions and jumps
straight to WebSearch. Reverted to the original "YOUR FIRST ACTION:
Run this command. EXECUTE." pattern that actually worked. Embedded
progress display as echo statements inside the bash command so they
can't be skipped. The agent was not running the Python script at all
when the first instruction was about text output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:18:53 -08:00
Matt Van Horn a6dd0a8b0e Move progress block to very first instruction in SKILL.md
The agent was skipping the progress text because it was buried after
parsing logic at line 41. Restructured so the FIRST section is
"YOUR FIRST OUTPUT" with the progress block, BEFORE any tool call
instructions. Added explicit "DO NOT skip this" enforcement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:16:49 -08:00
Matt Van Horn c3120d29a3 Show visible progress block before research starts
The ui.py spinner/color output goes to stderr which Claude Code
buffers invisibly. Instead, have the agent output a visible progress
block with emoji showing Reddit/X/Web agents deploying.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:13:59 -08:00
Matt Van Horn 11f9ab8519 Restore acknowledgment step before research execution
Add back the brief topic acknowledgment ("Researching X — searching
Reddit for upvotes...") and descriptive intro that the v1 skill had.
The v2 version was jumping straight into script execution with no
user-facing feedback about what was being researched.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:10:49 -08:00
Matt Van Horn 7c36866524 Fix v2 output quality: stats format, Reddit results, citations, summary structure
- Stats: replace BAD/GOOD examples with strict fill-in-the-blank template
- Reddit: add subreddit-targeted fallback search, soften scoring penalties
  (engagement -10→-3, date confidence -10→-5), add minimum result guarantee
- Citations: limit to 1 per insight, short format, no engagement metrics
- Summary: add bold topic headers template for structured paragraphs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:04:46 -08:00
Matt Van Horn 38451d44c9 Restore original description, enforce emoji stats format, require source attribution
- Reverted description to original public repo copy
- Restored H1 heading, moved imperative instruction to bold paragraph
- Added BAD/GOOD examples enforcing emoji tree stats format
- Required @handle and r/subreddit citation in ALL summary types
- Removed agent:Explore and disable-model-invocation from frontmatter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 09:48:21 -08:00
Matt Van Horn 63de326df0 Fix skill not executing: remove agent:Explore, front-load run command
The Explore agent was treating SKILL.md as documentation to summarize
instead of instructions to follow. Removed agent:Explore from frontmatter
and restructured opening to immediately command script execution.

Also adds release plan for Bird CLI v2.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 09:38:32 -08:00
Matt Van Horn 4230fa2b69 Fix YAML parsing error in argument-hint field
Wrap value in single quotes to properly escape nested double quotes.
This fixes skills CLI installation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 07:37:17 -08:00
Matt Van Horn 4e972d009c Rename skill from last30daystest to last30days
- Update skill name and description (remove TEST markers)
- Update script path to ~/.claude/skills/last30days/
- Clean up header, keep Bird CLI features prominent

Ready for local testing as the main skill.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 16:11:21 -08:00
Matt Van Horn 17485ec797 Make RECOMMENDATIONS format flexible (table or stacked)
- Remove over-prescriptive format requirements
- Let Claude choose table vs stacked naturally based on content
- Keep critical requirement: Sources line with @handles
- Claude Code's renderer handles the responsive wrapping

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:55:41 -08:00
Matt Van Horn 56f9eceb9c Switch RECOMMENDATIONS to stacked card format (responsive)
- Replace ASCII table with stacked card format
- Each card: Tool/Mentions/Use Case/Sources on separate lines
- Cards separated by horizontal rule (────)
- Doesn't break when terminal window is narrow
- Still includes @handles in Sources line

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:12:56 -08:00
Matt Van Horn c534c3b9a5 Update RECOMMENDATIONS format to use ASCII tables with @handles
- Changed from numbered list to ASCII box-drawing table format
- Added "Key Sources" column that MUST include @handles from X posts
- Added explicit instructions to parse and display highest-engagement handles
- Matches the output quality of the original xAI version

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:04:41 -08:00
Matt Van Horn 3afc1213b8 Add explicit instructions to calculate engagement totals
Claude was summarizing qualitatively ('mostly casual conversation')
instead of summing actual engagement numbers from the output.

Added CRITICAL note to:
- Parse [Xlikes, Yrt] from each X post
- Sum totals for the stats display
- Show top voices with engagement (e.g., @handle (15K likes))

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:47:41 -08:00
Matt Van Horn e7610ddcac Fix Bird search: strip noise words + fix date parsing
1. Add _extract_core_subject() to strip noise words before X search
   - X search is literal keyword matching, not semantic
   - "best open claw usecases" → "open claw" (27 results vs 0)

2. Fix ISO date detection bug
   - Was: if "T" in created_at (matched "Tue" in "Tue Feb 03...")
   - Now: if created_at[10] == "T" (proper ISO separator check)
   - Dates now parse correctly: (2026-02-03) instead of (date unknown)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:38:38 -08:00
Matt Van Horn f800b5eb1b Fix Bird CLI response parsing
1. last30days.py: Check isinstance(dict) before .get() - Bird returns
   list on success, dict on error

2. bird_x.py: Update field mappings for Bird's actual response format:
   - author.username not user.screen_name
   - createdAt not created_at (camelCase)
   - likeCount, retweetCount, etc. (camelCase)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:00:40 -08:00
Matt Van Horn 2c398ef6cb Enable API keys: remove clean mode
Now reads from ~/.config/last30days/.env like the regular skill.
Bird CLI + API keys both work together.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:47:10 -08:00
Matt Van Horn bcf5c2cc3f Fix: embed date filter in query instead of --since flag
Bird CLI doesn't support --since as a CLI flag.
Use X's search syntax: 'topic since:YYYY-MM-DD'

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:26:51 -08:00
Matt Van Horn 89bdbae0fe Fix: restore pretty emojis and box chars in stats summary
Original format:
 All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2}

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:09:14 -08:00
Matt Van Horn 06061884f1 Update SKILL.md: remove interactive prompt references
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 12:39:16 -08:00
Matt Van Horn 976a6e2867 Match original NUX: non-blocking Bird promotion
- Remove interactive "Install Bird CLI?" prompt
- Add Bird as FREE option in promo messages
- Auto-detect Bird silently (use if authenticated)
- Keep original flow: show promo → continue with available sources

Bird now appears in the promo alongside API keys:
  🔵 X (Twitter)
     └─ FREE: npm install -g @steipete/bird (uses browser session)
     └─ Or: Add XAI_API_KEY (paid API)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 12:38:54 -08:00
Matt Van Horn 73b42c5c89 fix: show Bird install prompt before topic validation
Move Bird CLI check to run before topic validation so users see
the install prompt even when running without a topic argument.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:53:04 -08:00
Matt Van Horn fe9ebb9944 feat: add clean mode for testing Bird CLI flow
- Add LAST30DAYS_CONFIG_DIR env var to override config location
- Set LAST30DAYS_CONFIG_DIR="" in test skill for clean mode
- Test skill now ignores existing API keys to show Bird install prompt

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:48:40 -08:00
Matt Van Horn d122210e1e rename: use last30daystest as main skill name for testing 2026-02-03 11:32:47 -08:00
Matt Van Horn 80b9fa74dd feat: add test skill (last30daystest) for Bird CLI testing
Creates SKILL-TEST.md with name "last30daystest" so the Bird CLI
integration can be tested without affecting the original /last30days
skill installed from the public repo.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:06:43 -08:00
Matt Van Horn ecdee52d04 feat(main): integrate Bird setup into main flow
- Move progress display initialization early to support Bird prompts
- Check Bird availability and offer install if no X source available
- Override available sources when Bird is ready
- Pass x_source parameter to run_research function

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:04:56 -08:00
Matt Van Horn e57b82f1ad feat(main): dispatch X search to Bird or xAI 2026-02-03 11:03:29 -08:00
Matt Van Horn 7a7e35bcf5 feat(main): add Bird setup function
Add bird_x import and setup_bird_if_needed function to main script.
The function checks Bird status and offers installation if needed,
returning 'bird' if ready, 'declined' if user declined, or None if
unavailable.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:02:28 -08:00
Matt Van Horn 25d58a4494 feat(lib): export bird_x module
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:01:46 -08:00
Matt Van Horn df849e7278 feat(env): add X source detection with Bird priority 2026-02-03 11:01:23 -08:00
Matt Van Horn 57e238cb9c feat(ui): add Bird CLI install prompts and auth help 2026-02-03 11:00:42 -08:00
Matt Van Horn 34723c70ac feat(bird): add response parser matching xai_x format 2026-02-03 10:59:39 -08:00
Matt Van Horn 2910e83552 feat(bird): add search_x function 2026-02-03 10:58:44 -08:00
Matt Van Horn 444bd6e185 feat(bird): add installation and status functions 2026-02-03 10:57:57 -08:00
Matt Van Horn a7fb951257 feat(bird): add detection functions for Bird CLI
- is_bird_installed() checks if bird command is in PATH
- is_bird_authenticated() runs bird whoami to verify auth
- check_npm_available() checks if npm can install bird

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:57:26 -08:00
Matt Van Horn 8ae00ea214 docs: Add Bird CLI implementation plan
12-task implementation plan for integrating Bird CLI as free X/Twitter
search alternative. Includes detection, interactive install, search
dispatch, and main flow integration.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:53:03 -08:00
Matt Van Horn 950810be92 docs: Add Bird CLI integration design
Adds design doc for integrating Bird CLI as an alternative X/Twitter
search source. Bird uses browser cookies (free, no API key) and
provides direct GraphQL access to X.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:50:59 -08:00
Matt Van Horn cc892d736b revert: restore original argument-hint format (testing) 2026-01-28 13:43:10 -08:00
Matt Van Horn 6cc7dad86e fix: YAML frontmatter and GPT-5 model fallback
- Fix invalid YAML in SKILL.md argument-hint (closes #8)
  Wrapped value in single quotes to properly escape double quotes

- Add automatic model fallback for GPT-5 access errors (closes #9)
  When OpenAI returns 400 for unverified orgs, retry with gpt-4o

- Add tests for model fallback logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 11:07:33 -08:00
Matt Van Horn 3a4a727f4b Run Reddit and X searches in parallel, fix timeout handling
- Refactor run_research() to use ThreadPoolExecutor for parallel execution
- Reddit timeout/crash no longer blocks X search from running
- Add catch for ConnectionResetError/OSError in http.py
- Per-item error handling in Reddit enrichment (one failure doesn't crash all)
- Increase API timeouts from 60/90/120 to 90/120/180 seconds
- Add ClawdBot setup example to README

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 13:33:41 -08:00
Matt Van Horn 86422a74af feat: Add honesty warning when results aren't from last 30 days
Adds ⚠️ LIMITED RECENT DATA warning when:
- Fewer than 5 items are confirmed from the date range
- Tells Claude to be transparent with user about data freshness

Example output for obscure topic (June Oven):
"Only 4 item(s) confirmed from 2025-12-26 to 2026-01-25.
Results below may include older/evergreen content."

Popular topics (clawdbot, nano banana) don't show the warning.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 13:00:53 -08:00
Matt Van Horn e1d3570667 feat: Add automatic retry with simpler query for sparse Reddit results
If initial search returns <5 threads, extract core subject and retry:
- "best nano banana prompting practices" → retry with "nano banana"
- Combines results from both searches, deduped by URL

Note: OpenAI's web_search still tends to find old content. This retry
helps cast a wider net but doesn't fully solve the recency issue.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:44:29 -08:00
Matt Van Horn 2d43571875 fix: Simplify Reddit prompt and increase search volume
OpenAI's web_search doesn't reliably find recent Reddit content.
Strategy change: request MORE threads and let server-side date
filtering handle it.

Changes:
- Increased depth config (20-30 → 30-50 default)
- Simplified prompt to search broadly and include all matches
- Let server-side date filter remove old content
- Removed aggressive pre-filtering that was limiting results

Tradeoff: Some topics have more X coverage than Reddit due to
web_search limitations. This is acceptable.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:35:14 -08:00
Matt Van Horn acd7a533c0 fix: Emphasize recency in Reddit search to find recent threads
The model was finding old threads (Nov-Dec 2025) that got filtered out,
leaving 0 results even for popular topics like "nano banana prompts".

Changes:
- Add explicit date filters: "after:{from_date}" and "2026"/"January 2026"
- Emphasize RECENT content is critical, old threads are filtered anyway
- Tell model to verify dates before including threads

Before: 0-2 Reddit threads for "nano banana prompting practices"
After: 16 Reddit threads from January 2026

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:15:43 -08:00
Matt Van Horn b77d821f8d docs: Add Top Claude Code Skills example to README
Shows /last30days as a discovery tool for finding popular resources
in rapidly evolving ecosystems - ranked by community engagement.

Includes Remotion (17.3K likes), SkillsMP marketplace, Superpowers,
and Trail of Bits security skills.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:50:40 -08:00
Matt Van Horn 66d7b87049 docs: Add ClawdBot use cases example to README
Shows /last30days as a product research tool - discovering real-world
use cases with engagement metrics from r/LocalLLaMA, r/selfhosted, and
X posts from @steipete, @danpeguine, @dreetje.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:48:35 -08:00
Matt Van Horn 7dc3e7b716 fix: Improve Reddit search prompt to find actual discussions
- Add EXTRACT CORE KEYWORDS step to search for main subject, not full phrase
  ("killer features of clawdbot" → search "clawdbot")
- Remove hardcoded subreddit list that biased results toward design/dev subs
- Simplify search strategies and URL validation rules
- Make prompt more concise - GPT-5.2 responds better to clearer instructions

Before: 0 threads found for "killer features of clawdbot"
After: 7 threads found including r/LocalLLaMA (43pts), r/selfhosted (26pts)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:42:31 -08:00
Matt Van Horn 6fbfbb9ccc feat: Add web-only fallback mode with API key promo
- Skill now works without any API keys using WebSearch fallback
- Shows promo banner marketing Reddit/X data when keys are missing
- Partial mode (one key) shows shorter tip for the missing source
- Updated SKILL.md to document three modes: Full, Partial, Web-Only
- Added get_missing_keys() to env.py for promo logic
- Added show_promo(), start_web_only(), show_web_only_complete() to ui.py
- Updated render_compact() to include inline promo for web-only mode

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:32:24 -08:00
Matt Van Horn 6831e624b8 docs: Add photorealistic aging portrait example
Shows multi-panel consistency for photorealistic people:
- JSON structure with face preservation
- Same person at ages 10, 20, 40, 80
- Heterochromia + freckles maintained across panels
- Real generated result

Also added to SKILL.md intro examples.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:28:15 -08:00
Matt Van Horn 7f1ab7a2e9 fix: Remove duplicate stats display from Judge section
Stats were showing twice - once in Judge section, once at end.
Now Judge section just synthesizes internally, stats only appear
at the end right before invitation.

Order: What I learned -> Stats -> Share your vision

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:22:55 -08:00
Matt Van Horn 2192743c74 docs: Add iOS app mockup example (SwimMom for Nano Banana Pro)
Shows JSON-structured prompting for UI mockups:
- Layer-based structure (frame -> layout -> content)
- iOS 18 design system references
- 8pt grid and element limits
- Real generated mockup result

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:14:56 -08:00
Matt Van Horn 60c663697e fix: Reorder output - stats right before invitation
New order:
1. What I learned / KEY PATTERNS (or Most mentioned)
2. Stats (All agents reported back!)
3. Share your vision (last)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:09:52 -08:00
Matt Van Horn bcb1c7d71b feat: Reorder use cases - Prompting first, rename HOW-TO to PROMPTING
Prompting is the primary use case, so it should be first:
1. Prompting - "Midjourney prompts", "ChatGPT image generation"
2. Recommendations - "best Claude Code skills", "top AI tools"
3. News - "what's happening with OpenAI"
4. General - any topic

Also renamed HOW-TO query type to PROMPTING for clarity.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:07:35 -08:00
Matt Van Horn c8504d3b30 fix: Use user's exact terminology, don't add outdated tech names
Claude was "helpfully" adding DALL-E, GPT-4o to searches when user
asked about "ChatGPT image prompting". This uses outdated knowledge.

Now explicitly instructs: use the user's exact terminology, don't
substitute or add tech names. Trust the user knows what they're
asking about.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:05:45 -08:00
Matt Van Horn 6577da0b14 feat: More emojis and bring back Judge Agent in display
Updated research banner:
- Added Judge Agent: "Standing by to synthesize..."
- Better emoji descriptions: "Scanning for gold", "Catching latest takes"

Updated completion display:
- "All agents reported back!"
- "Judge Agent: Weighing the evidence..."
- Emojis on the Judge's evaluation criteria

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:03:41 -08:00
Matt Van Horn 6a7c29010d fix: Answer follow-up questions from research, don't re-search
After research is complete, Claude is an expert. Follow-up questions
should be answered from the research findings, not trigger new
WebSearches. Only do new research for a different topic.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:01:35 -08:00
Matt Van Horn b6cdbc9058 fix: Display research banner before background script starts
Since script runs in background, users don't see its live output.
Now Claude displays the fancy emoji banner immediately before
starting the background script.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:00:30 -08:00
Matt Van Horn 5f8623f129 feat: Smart query detection for better research results
Added QUERY_TYPE detection:
- RECOMMENDATIONS ("best X") → searches for lists, extracts specific names
- NEWS → searches for current events
- HOW-TO → searches for tutorials
- GENERAL → broad topic research

For RECOMMENDATIONS queries, synthesis now extracts specific entity names
with mention counts instead of generic patterns.

Example: "best Claude Code skills" now returns:
  "Most mentioned: /commit (5x), remotion (4x), git-worktree (3x)"
Instead of:
  "Skills are good. Keep them under 500 lines."

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 13:23:31 -08:00
Matt Van Horn 68420ca1e2 fix: Run Reddit/X script in background for parallel execution
Start the script first with run_in_background:true, then do WebSearch.
Both run in parallel so users see the script's progress display while
WebSearch also fetches results.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:48:07 -08:00
Matt Van Horn 0e4df9f38a fix: Clean up final display - no Sources, end with invitation
- Suppress "Sources:" output from WebSearch (redundant noise)
- Reorder: Research Complete stats → What I learned → Share your vision
- Simplified KEY PATTERNS to 3 items
- Clean ending users see while waiting for their input

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:38:51 -08:00
Matt Van Horn 85e7b6445c fix: Run WebSearch before script for better working state
Users now see the script's progress display as the "working state"
while waiting, rather than the WebSearch tool call.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:32:49 -08:00
Matt Van Horn e926a5aabd fix: Remove redundant Web Agent from banner display
WebSearch shows as visible tool calls, so announcing it in the
banner was redundant. Now only Reddit and X agents are shown
in the initial display.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:29:36 -08:00
Matt Van Horn 18696e7b05 fix: Enforce strict 30-day date filtering
Previously Reddit was returning ~60% old content (some from 2022).
This commit adds multiple layers of date enforcement:

- Reddit prompt: Explicit from_date/to_date with "fewer results > older results"
- Hard filter: filter_by_date_range() in normalize.py excludes old content
- WebSearch Date Detective: Extracts dates from URLs (/2026/01/24/) and
  snippets ("January 24, 2026", "3 days ago")
- WebSearch scoring: +10 bonus for verified dates, -20 penalty for unknown

The skill now guarantees only content from the last 30 days.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:23:34 -08:00
Matt Van Horn 354e8bad03 docs: Update README with marketing copy
Add new tagline emphasizing AI moves fast and this skill keeps you current.
Replace em-dashes with regular dashes/colons throughout.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:57:52 -08:00
Matt Van Horn 407801f8be feat: Add parallel research agents with Judge synthesis
- Deploy 3 agents in parallel: Reddit+X, WebSearch, Backup WebSearch
- Add visual progress indicators (🟠🔵🌐)
- Add Judge Agent to synthesize all sources
- Include web authors in top voices (e.g., "John on Medium")
- Weight Reddit/X higher than WebSearch (engagement signals)

Makes the research feel magical with parallel execution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:51:07 -08:00
201 changed files with 15977 additions and 362 deletions
+17
View File
@@ -0,0 +1,17 @@
{
"name": "last30days",
"owner": {
"name": "mvanhorn",
"url": "https://github.com/mvanhorn"
},
"metadata": {
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0"
},
"plugins": [
{
"name": "last30days",
"source": "."
}
]
}
+12
View File
@@ -0,0 +1,12 @@
{
"name": "last30days",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0",
"author": {
"name": "mvanhorn"
},
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
"skills": ["./"]
}
+63
View File
@@ -0,0 +1,63 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [2.1.0] - 2026-02-15
### Highlights
Three headline features: watchlists for always-on bots, YouTube transcripts as a 4th source, and Codex CLI compatibility. Plus bundled X search with no external CLI needed.
### Added
- Open-class skill with watchlists, briefings, and history modes (SQLite-backed, FTS5 full-text search, WAL mode) (`feat(open)`)
- YouTube as a 4th research source via yt-dlp -- search, view counts, and auto-generated transcript extraction (`feat: Add YouTube`)
- OpenAI Codex CLI compatibility -- install to `~/.agents/skills/last30days`, invoke with `$last30days` (`feat: Add Codex CLI`)
- Bundled X search -- vendored subset of Bird's Twitter GraphQL client (MIT, originally by @steipete), no external CLI needed (`v2.1: Bundle Bird X search`)
- Native web search backends: Parallel AI, Brave Search, OpenRouter/Perplexity Sonar Pro (`feat(engine)`)
- `--diagnose` flag for checking available sources and authentication status
- `--store` flag for SQLite accumulation (open variant)
- Conversational first-run experience (NUX) with dynamic source status (`feat(nux)`)
### Changed
- Smarter query construction -- strips noise words, auto-retries with shorter queries when X returns 0 results
- Two-phase search architecture -- Phase 1 discovers entities (@handles, r/subreddits), Phase 2 drills into them
- Reddit JSON enrichment -- real upvotes, comments, and upvote ratio from reddit.com/.json endpoint
- Engagement-weighted scoring: relevance 45%, recency 25%, engagement 30% (log1p dampening)
- Model auto-selection with 7-day cache and fallback chain (gpt-4.1 -> gpt-4o -> gpt-4o-mini)
- `--days=N` configurable lookback flag (thanks @jonthebeef, [#18](https://github.com/mvanhorn/last30days-skill/pull/18))
- Model fallback for unverified orgs (thanks @levineam, [#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- Marketplace plugin support via `.claude-plugin/plugin.json` (inspired by @galligan, [#1](https://github.com/mvanhorn/last30days-skill/pull/1))
### Fixed
- YouTube timeout increased to 90s, Reddit 429 rate limit fail-fast
- YouTube soft date filter -- keeps evergreen content instead of filtering to 0 results
- Eager import crash in `__init__.py` that broke Codex environments
- Reddit future timeout (same pattern as YouTube timeout bug)
- Process cleanup on timeout/kill -- tracks child PIDs for clean shutdown
- Windows Unicode fix for cp1252 emoji crash (thanks @JosephOIbrahim, [#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- X search returning 0 results on popular topics due to over-specific queries
### New Contributors
- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
### Credits
- @steipete -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
- @galligan -- Marketplace plugin inspiration
- @hutchins -- Pushed for YouTube feature
## [1.0.0] - 2026-01-15
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
+530 -24
View File
@@ -1,10 +1,20 @@
# /last30days # /last30days v2.1
A Claude Code skill that researches any topic across Reddit and X from the last 30 days, synthesizes the insights, and delivers expert-level answers. **The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, and the web from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago. Whether it's Seedance 2.0 access, Suno music prompts, or the latest Nano Banana Pro techniques, you'll prompt like someone who's been paying attention.
**Best for prompt research** — discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Figma AI, etc.) by learning from real community discussions and best practices. **New in V2.1 - three headline features:**
**But also great for anything trending** — music, culture, news, product recommendations, viral trends, or any question where "what are people saying right now?" matters. 1. **Open-class skill with watchlists.** Add any topic to a watchlist - your competitors, specific people, emerging technologies - and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw) where a bot can run research on a schedule and accumulate findings over time.
2. **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post - now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
3. **Works in OpenAI Codex CLI.** Same skill, same engine. Install to `~/.agents/skills/last30days` and invoke with `$last30days`. Claude Code and Codex users get the same research.
**New in V2:** Dramatically better search results. Smarter query construction finds posts that V1 missed entirely, and a new two-phase search automatically discovers key @handles and subreddits from initial results, then drills deeper. Free X search (no xAI key needed), `--days=N` for flexible lookback, and automatic model fallback. [Full changelog below.](#whats-new-in-v2)
**The tradeoff:** V2 finds way more content but takes longer - typically 2-8 minutes depending on how niche your topic is. The old V1 was faster but regularly missed results (like returning 0 X posts on trending topics). We think the depth is worth the wait, but if you'd use a faster "quick mode" that trades some depth for speed, let us know: [@mvanhorn](https://x.com/mvanhorn) / [@slashlast30days](https://x.com/slashlast30days).
**Best for prompt research**: discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Figma AI, etc.) by learning from real community discussions and best practices.
**But also great for anything trending**: music, culture, news, product recommendations, viral trends, or any question where "what are people saying right now?" matters.
## Installation ## Installation
@@ -16,11 +26,85 @@ git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last
mkdir -p ~/.config/last30days mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'EOF' cat > ~/.config/last30days/.env << 'EOF'
OPENAI_API_KEY=sk-... OPENAI_API_KEY=sk-...
XAI_API_KEY=xai-... XAI_API_KEY=xai-... # optional - cookie auth is default for X search
EOF EOF
chmod 600 ~/.config/last30days/.env chmod 600 ~/.config/last30days/.env
``` ```
### X Search Authentication
X search reads your existing browser cookies - no API keys or login commands needed.
**Safari (recommended on Mac):** Just be logged into x.com. No setup needed.
**Chrome:** Works, but macOS will prompt you to allow Keychain access the first time. Click "Allow" (or "Always Allow" to stop future prompts).
**Firefox:** Just be logged into x.com. No setup needed.
**Manual fallback:** If cookie auto-detection doesn't work, set these env vars (grab them from your browser's dev tools → Application → Cookies → x.com):
```bash
export AUTH_TOKEN=your_auth_token
export CT0=your_ct0_token
```
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
```
**Requirements:** Node.js 22+ (for the vendored Twitter GraphQL client).
### Codex CLI
This skill also works in OpenAI Codex CLI. Install to the Codex skills directory instead:
```bash
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
```
Same SKILL.md, same Python engine, same scripts. The `agents/openai.yaml` provides Codex-specific discovery metadata. Invoke with `$last30days` or through the `/skills` menu.
### Open Variant (Watchlist + Briefings) - For Always-On Bots
**Designed for [Open Claw](https://github.com/openclaw/openclaw) and similar always-on AI environments.** Add your competitors, specific people, or any topic to a watchlist. When paired with a cron job or always-on bot, /last30days re-researches them on a schedule and accumulates findings in a local SQLite database. Ask for a briefing anytime.
**Important:** The watchlist stores schedules as metadata, but nothing triggers runs automatically. You need an external scheduler (cron, launchd, or an always-on bot like Open Claw) to call `watchlist.py run-all` on a timer. In plain Claude Code, you can run `watch run-one` and `watch run-all` manually, but there's no background scheduling.
```bash
# Enable the open variant
cp variants/open/SKILL.md ~/.claude/skills/last30days/SKILL.md
# Add topics to your watchlist
last30 watch my biggest competitor every week
last30 watch Peter Steinberger every 30 days
last30 watch AI video tools monthly
last30 Y Combinator hot companies end of April and end of September
# Run research manually (or let your bot's cron handle it)
last30 run all my watched topics
# Search accumulated knowledge
last30 what have you found about AI video?
```
The open variant adds four modes on top of one-shot research:
- **Watchlist** - Track topics with `watch add "topic"`, run manually or via cron
- **Briefings** - Daily/weekly digests synthesized from accumulated findings
- **History** - Query and search your research database with full-text search
- **Native web search** - Built-in web search backends (Parallel AI, Brave, OpenRouter) run alongside Reddit/X/YouTube
Both variants use the same Python engine and scripts directory. The open variant adds command routing (`watch`, `briefing`, `history`) and references mode-specific instruction files.
**Optional web search API keys** (add to `~/.config/last30days/.env`):
```bash
PARALLEL_API_KEY=... # Parallel AI (preferred - LLM-optimized results)
BRAVE_API_KEY=... # Brave Search (free tier: 2,000 queries/month)
OPENROUTER_API_KEY=... # OpenRouter/Perplexity Sonar Pro
```
Check source availability: `python3 scripts/last30days.py --diagnose`
## Usage ## Usage
``` ```
@@ -36,16 +120,147 @@ Examples:
## What It Does ## What It Does
1. **Researches** - Scans Reddit and X for discussions from the last 30 days 1. **Researches** - Scans Reddit, X, and YouTube for discussions from the last 30 days
2. **Synthesizes** - Identifies patterns, best practices, and what actually works 2. **Synthesizes** - Identifies patterns, best practices, and what actually works
3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer 3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
### Use it for: ### Use it for:
- **Prompt research** "What prompting techniques work for legal questions in ChatGPT?" - **Prompt research** - "What prompting techniques work for legal questions in ChatGPT?"
- **Tool best practices** "How are people using Remotion with Claude Code?" - **Tool best practices** - "How are people using Remotion with Claude Code?"
- **Trend discovery** "What are the best rap songs right now?" - **Trend discovery** - "What are the best rap songs right now?"
- **Product research** "What do people think of the new M4 MacBook?" - **Product research** - "What do people think of the new M4 MacBook?"
- **Viral content** "What's the dog-as-human trend on ChatGPT?" - **Viral content** - "What's the dog-as-human trend on ChatGPT?"
---
## Example: Nano Banana Pro Prompting (Image Generation)
**Query:** `/last30days nano banana pro prompting`
**Research Output:**
> JSON prompting is the dominant technique - The X community overwhelmingly uses structured JSON prompts rather than plain text. Creators like @Xmira_belle, @Cicily_aura, @RubenSalvo_, and @mahivisuals all share prompts in JSON format with nested objects for metadata, subject descriptions, lighting, camera specs, and scene composition. @realdigitaldao made a static ad "in less than 5 min" using a simple JSON prompt.
>
> Prompt adherence is Nano Banana Pro's killer advantage - JSON prompts hit 92% precision for color accuracy, lighting ratios, and composition vs ~68% for natural language prompts, per fofr.ai. @artingent directly compared it to GPT Image 1.5 and noted Nano Banana Pro has higher prompt adherence - "GPT image 1.5 ignored the Top Down instruction."
>
> Stop writing "tag soup" - act like a Creative Director - The biggest shift is away from comma-separated keyword lists toward natural language descriptions with clear structure. The ICS Framework (Image type, Content, Style) and the 5-element formula (subject, composition, action, setting, style) are the two dominant approaches, per r/nanobanana2pro.
**Key patterns discovered:**
1. JSON > plain text - Structure prompts as JSON objects with keys for subject, composition, lighting, camera, style - per @Xmira_belle, @Cicily_aura
2. The 5-element formula - Subject + Composition + Action + Setting + Style - per r/nanobanana2pro
3. Separate subjects into distinct objects - Multi-character scenes need each person in their own JSON block to prevent "concept bleeding" - per fofr.ai
4. Use camera/lens metadata - Aperture, focal length, ISO, film stock as JSON keys for hidden photorealism control
5. 14 reference images max - Feed up to 14 reference images for brand/character consistency - per Google DeepMind
**Research Stats:** 8 Reddit threads (455 upvotes) + 31 X posts (183 likes, 68 reposts) from r/nanobanana2pro, r/promptingmagic, @Xmira_belle (106 likes), @Cicily_aura (45 likes)
**Then asked:** "make a mock-up of an app for moms who swim"
**Generated Prompt:**
```json
{
"image_type": "UI mockup",
"device": {
"frame": "iPhone 16 Pro",
"display": "realistic screen with subtle reflection",
"orientation": "portrait"
},
"app_concept": "SwimMom - fitness and community app for mothers who swim",
"screen": "home dashboard",
"design_system": {
"style": "iOS 18 native",
"corners": "rounded, 16px radius",
"shadows": "soft drop shadows, subtle depth",
"spacing": "8pt grid, generous whitespace"
},
"color_palette": {
"primary": "calming aqua blue (#4ECDC4)",
"secondary": "warm coral (#FF6B6B)",
"background": "clean white with soft blue gradient header"
},
"layout": {
"header": "greeting 'Good morning, Sarah' with wave icon, profile avatar top right",
"hero_card": "today's swim stats - laps, calories, time - with circular progress ring",
"section_2": "horizontal scroll of upcoming mom swim meetups with location pills",
"section_3": "two cards - 'My Goals' with streak counter, 'Find a Pool' with map pin",
"bottom_nav": "5 icons - Home (active), Workouts, Community, Schedule, Profile"
},
"visual_details": {
"illustrations": "friendly, minimal line art of swimming figure",
"typography": "SF Pro Display, clean hierarchy",
"mood": "energizing yet calm, supportive community feel"
}
}
```
**The Result:**
<p align="center">
<img src="assets/swimmom-mockup.jpeg" width="400" alt="SwimMom iOS app mockup"/>
</p>
This example shows /last30days learning that **JSON prompts dominate** the Nano Banana Pro community (not plain text), then applying that format to generate an iOS mockup in one shot. The research came from real X creators and Reddit threads - techniques you wouldn't find in official docs.
---
## Example: Kanye West (Current Events)
**Query:** `/last30days kanye west`
**Research Output:**
> **The Apology Arc** - On January 26, Ye published a full-page letter in The Wall Street Journal titled "To Those I've Hurt," apologizing for antisemitic comments and attributing it to a four-month manic episode, an undiagnosed brain injury from his 2002 car crash, and bipolar type-1 disorder. The ADL called it "long overdue" but said it "doesn't automatically undo his long history of antisemitism." Then on February 9, he posted "I love Hitler now what" - blowing up the goodwill almost immediately, per The Mirror. Fans on X are split on whether the apology was genuine or just pre-album PR, per @dekreationz's viral post (8.6K likes).
>
> **BULLY Album** - Twelfth studio album set for March 20 via YZY and Gamma. 13 tracks with Peso Pluma, Playboi Carti, and Ty Dolla Sign features. Sonically it leans toward 808s/MBDTF territory - more singing than rapping. Ye has reportedly removed all deepfaked/AI-generated vocals from the final version, per Rolling Stone.
>
> **Bianca Censori Breaks Silence** - In a Vanity Fair tell-all, Bianca spoke publicly for the first time about their marriage, saying "This year was a lot like doing CPR for months." She confirmed her outfits are her own choice and revealed their relationship started when she was head of architecture at Yeezy, per TMZ.
>
> **Kim/Instagram Drama** - A screenshot of Ye allegedly commenting on Kim's Complex magazine post went mega-viral on X. Multiple Grok fact-checks confirmed it's likely fabricated, but @dekreationz's post racked up nearly 9K likes regardless - showing the Ye/Kim narrative still drives massive engagement.
**Key patterns from the research:**
1. Apology-then-relapse cycle - The WSJ letter was immediately undercut by new inflammatory posts, making sincerity the dominant debate, per Washington Post
2. Album as PR vehicle - Skeptics across X tie the apology's timing directly to BULLY's March 20 release, per @trustGod_007
3. Bianca as her own story - Transitioning from "Kanye's wife" to individual public figure through Vanity Fair
4. Fake screenshot virality - Fabricated Ye content still goes massively viral before debunking, per @dekreationz
**Research Stats:** 3 Reddit threads (193 upvotes, 192 comments) + 33 X posts (9,295 likes, 279 reposts) from r/Kanye, r/hiphopheads, @dekreationz (8,684 likes), @trustGod_007
**Follow-up suggestions offered:**
> I'm now an expert on Kanye West. Some things I can help with:
> - What's the real story behind the apology letter - genuine accountability or pre-album PR?
> - Deep dive on BULLY: tracklist, Gamma deal, and removal of AI vocals
> - The Bianca Censori Vanity Fair interview - what she actually said vs. the tabloid headlines
This example shows /last30days as a **current events research tool** - no prompting, no target tool, just "what's happening with Kanye?" and you get a synthesized expert briefing pulled from 33 X posts and 3 Reddit threads with real engagement data.
---
## Example: Vibe Motion (Brand New AI Tool)
**Query:** `/last30days higgsfield motion vibe motion prompting`
**Research Output:**
> **Vibe Motion just launched (Feb 5-6, 2026) and it's blowing up** - Higgsfield dropped a Claude-powered motion design tool that generates editable motion graphics from plain-English prompts. It's the first AI video tool with actual reasoning, per @Hartdrawss, and @rezkhere called it out as something "no other AI can do." Linus Ekenstam said it "broke his brain" - work that took hours in After Effects now takes minutes.
>
> **It generates code, not pixels** - Unlike other AI video tools, Vibe Motion produces actual animation code (via Remotion under the hood), not hallucinated video. Text never breaks, edits stay consistent, and you get a controllable, editable asset, per @Totinhiiio.
>
> **Honest reviews: promising but not polished yet** - Chase Jarvis found results "okay" but noted 5+ minute render times, credit burn on iteration (8-60 credits per gen, $9 plan = ~150 credits), and that basic results are achievable faster with Canva. His verdict: "not quite ready for prime time" but the underlying tech shows significant potential.
**Key patterns discovered:**
1. Describe structure, not effects - Focus on timing, hierarchy, typography, and flow rather than expressive visual storytelling, per Higgsfield's official guide
2. Upload your actual assets first - Brand logos, product images, PDFs give the AI context to build around YOUR files, not generic placeholders
3. Use presets as starting points - Select a format (Infographics, Text Animation, Posters) before writing your prompt
4. Keep prompts conversational and direct - Short commands > long descriptions. "Create a kinetic typography intro" beats a paragraph of specs, per Segmind
5. Budget for iteration - Each generation burns credits, so get your prompt right before hitting generate, per Chase Jarvis
**Research Stats:** 10 Reddit threads + 30 X posts from @rezkhere, @Hartdrawss, @Totinhiiio + 14 web pages (Higgsfield blog, Chase Jarvis, SiliconANGLE, Segmind)
**Follow-up suggestions offered:**
> I'm now an expert on Higgsfield Vibe Motion prompting. What do you want to make?
> - A branded infographic animation with your company metrics animating in
> - A kinetic typography intro/bumper for a YouTube series
> - A logo reveal animation with particle effects and custom brand colors
This example shows /last30days researching **a tool that launched 2 days ago**. No blog roundups existed yet - the research came from X creators and Reddit threads reacting in real-time. You get prompting advice from the people actually using it, not SEO articles written after the fact.
--- ---
@@ -105,6 +320,125 @@ This example shows /last30days learning **domain-specific prompting techniques**
--- ---
## Example: ClawdBot Use Cases (Community Research)
**Query:** `/last30days best clawdbot use cases`
**Research Output:**
> ClawdBot is a self-hosted AI assistant that runs on your devices and connects via messaging apps (Telegram, WhatsApp, Discord, Slack, iMessage). Unlike ChatGPT/Claude, it has persistent memory and can proactively message you. The killer feature is its 24/7 autonomous operation - it handles multi-day tasks, monitors for replies, and initiates contact.
**Most mentioned use cases:**
🏆 Top 5 by engagement:
1. **Email/Calendar automation** - mentioned 8x+ (r/selfhosted, @dreetje, @danpeguine, MacStories)
- Morning briefings, spam filtering, auto-replies, recruiter declines
2. **Task management via chat** - mentioned 6x (r/LocalLLaMA, @danpeguine, multiple X posts)
- Todoist integration, importance scoring, time-blocking tasks
3. **Overnight coding agent** - mentioned 5x (@bffmike 225 likes, r/LocalLLaMA)
- "Drove my coding agents from 12:30-7am while I snoozed"
- Pulls repos, runs tests, commits if clean
4. **Smart home + life admin** - mentioned 5x (r/selfhosted, VelvetShark)
- WHOOP health data, Philips Hue, HomeAssistant, SSH to homelab
5. **Browser automation** - mentioned 4x (@steipete 243 likes, MacStories)
- Car negotiation via email/PDF, restaurant reservations via ElevenLabs calls
**Notable single mentions:**
- Custom meditation generation with TTS + ambient audio
- Built a full website (Stumblereads.com) from phone while putting baby to sleep
- Ray-Ban Meta glasses integration
- WhatsApp as control interface for everything
**Research Stats:** 9 Reddit threads (161 upvotes, 95 comments) + 19 X posts (2,018 likes, 96 reposts) from r/LocalLLaMA, r/selfhosted, @steipete, @danpeguine, @dreetje
This example shows /last30days as a **product research tool** - discovering real-world use cases and community sentiment for any tool or product, complete with engagement metrics that tell you what people actually care about.
---
## Example: ClawdBot Setup (Installation Guide)
**Query:** `/last30days how to best setup clawdbot`
**Research Output:**
> ClawdBot setup varies by deployment style - Docker is the most recommended path, but users are also deploying via bare metal and cloud VMs. The community has converged on key best practices: use the setup wizard for initial config, start with conservative hardware (you can scale up), and configure authentication before exposing any endpoints.
**🛠️ Setup patterns by engagement:**
| Pattern | Mentions | Key Sources |
|---------|----------|-------------|
| Docker Compose deployment | 6x | r/selfhosted, r/LocalLLaMA |
| Setup wizard (interactive config) | 5x | @dreetje, documentation |
| API key management | 4x | r/selfhosted, security threads |
| Hardware requirements | 4x | r/LocalLLaMA (GPU discussions) |
| Reverse proxy (Caddy/nginx) | 3x | r/selfhosted |
**Key setup insights from community:**
1. **Use the setup wizard** - Don't manually edit config files first; the wizard handles 90% of initial setup
2. **Start small on hardware** - 8GB RAM minimum, but 16GB+ recommended if running local models
3. **Docker > bare metal** - Easier updates, better isolation, community-tested compose files
4. **Secure before exposing** - Set up authentication before opening ports; ClawdBot has built-in auth
5. **Cost management** - Set API rate limits early; one user reported $200 overnight bill from runaway tasks
**Quick-start commands (community-verified):**
```bash
# Clone and setup
git clone https://github.com/clawdbot/clawdbot.git
cd clawdbot
# Run setup wizard (recommended)
./setup.sh
# Or Docker Compose (after config)
docker compose up -d
```
**Common gotchas mentioned:**
- Don't forget to set `CLAWDBOT_API_KEY` before first run
- Telegram bot token needs BotFather setup first
- If using local models, ensure CUDA drivers are installed
**Research Stats:** 8 Reddit threads (128 upvotes) + 22 X posts (24,000+ likes) from r/selfhosted, r/LocalLLaMA, @dreetje, @steipete
This example shows /last30days as a **setup guide aggregator** - pulling together scattered installation advice, gotchas, and best practices from real users who've already solved the problems you're about to encounter.
---
## Example: Top Claude Code Skills (Recommendations)
**Query:** `/last30days top claude code skills`
**Research Output:**
> The Claude Code skills ecosystem has exploded with marketplaces, curated lists, and viral skill announcements. The Remotion video skill got 17.3K likes on X. SkillsMP emerged as a marketplace with 60-87K+ skills. Multiple GitHub repos (awesome-claude-skills, Superpowers) are actively curated.
**🏆 Most mentioned skills/resources:**
| Rank | Skill/Resource | Mentions | Sources | Engagement |
|------|----------------|----------|---------|------------|
| 1 | Remotion skill | 4x | X (@Remotion, @joshua_xu_), web | 17.3K likes, video creation |
| 2 | SkillsMP marketplace | 5x | X (@milesdeutscher, @rexan_wong), web | 60-87K+ skills directory |
| 3 | awesome-claude-skills (GitHub) | 4x | Web (travisvn, ComposioHQ repos) | Multiple curated lists |
| 4 | Superpowers | 3x | Web, GitHub | 27.9K stars |
| 5 | HeyGen avatar skill | 2x | X (@joshua_xu_), web | 736 likes, AI avatars |
| 6 | Trail of Bits Security Skills | 2x | Web | CodeQL/Semgrep auditing |
| 7 | Claude Command Suite | 2x | GitHub, web | 148+ commands, 54 agents |
| 8 | MCP Builder | 2x | Web | Build MCP servers |
| 9 | Test-Driven Development | 2x | Web, skill guides | Pre-implementation testing |
| 10 | Systematic Debugging | 2x | Web | Root cause analysis |
**Notable single mentions:** UI/UX Pro Max (16.9K stars), SuperClaude framework, Compound Engineering Plugin, docx/pdf/pptx document skills, Nano-Banana, Connect (1000+ app integrations)
**Marketer skill stack** (from @vibecode_edu): Slack, Email Knowledge, Nano-Banana, Notion, Skill Creator, Twitter post, Typefully
**Key insight from Reddit:** Keep CLAUDE.md short (~1K context), use Skills for "context on demand", and ensure skill descriptions contain your trigger keywords.
**Research Stats:** 4 Reddit threads (238 upvotes, 156 comments) + 15 X posts (28K+ likes, 2.8K reposts) from r/ClaudeCode, r/ClaudeAI, @Remotion, @milesdeutscher
This example shows /last30days as a **discovery tool** - finding the most popular resources in a rapidly evolving ecosystem by tracking what the community is actually sharing and upvoting.
---
## Example: Discover Viral Trends (Dog as Human) ## Example: Discover Viral Trends (Dog as Human)
**Query:** `/last30days using ChatGPT to make images of dogs` **Query:** `/last30days using ChatGPT to make images of dogs`
@@ -152,7 +486,7 @@ This example shows how /last30days discovers **viral trends you didn't know exis
**Query:** `/last30days research best practices for beautiful remotion animation videos in claude code` **Query:** `/last30days research best practices for beautiful remotion animation videos in claude code`
**Research Output:** **Research Output:**
> The Remotion + Claude Code combination has emerged as a powerful workflow. Users consistently report that simple, clear prompts with scene-by-scene descriptions yield the best results. Key insights: iteration is essentialmost "beautiful" videos come from back-and-forth refinement rather than one-shot prompts. > The Remotion + Claude Code combination has emerged as a powerful workflow. Users consistently report that simple, clear prompts with scene-by-scene descriptions yield the best results. Key insights: iteration is essential, most "beautiful" videos come from back-and-forth refinement rather than one-shot prompts.
**Then asked:** "Can you make a prompt for a 50 second launch video for my /last30days skill?" **Then asked:** "Can you make a prompt for a 50 second launch video for my /last30days skill?"
@@ -208,6 +542,69 @@ This prompt was generated based on research from 10 Reddit threads (294 upvotes)
--- ---
## Example: Photorealistic Portraits (Aging Grid)
**Query:** `/last30days photorealistic people in nano banana pro`
**Research Output:**
> JSON prompts are the dominant format for photorealistic people in Nano Banana Pro. Nearly every viral X post (4,400+ likes total) uses structured JSON with specific fields rather than natural language. The key is specifying demographics, skin texture, lighting, and camera settings in a structured way.
>
> Recent caveat: Reddit reports Nano Banana Pro was "nerfed" in early January 2026 with reduced resolution and stricter moderation on faces. Starting a new chat can help if quality drops mid-session.
**Key patterns discovered:**
1. JSON structure with prompt_type, subject, lighting, camera_settings fields
2. Skin texture keywords: "natural pores", "realistic skin texture", "8K texture quality"
3. Face preservation: "preserve_original": true when using reference photos
4. Camera realism: specify lens (85mm f/1.4), depth of field, lighting setup
5. Wet skin close-ups are trending for maximum realism
6. Negative prompts: "no cartoon effect, no distortion, correct anatomy"
**Then asked:** "2x2 grid of the same person at age 10, 20, 40 and 80 - a woman with brown hair and a lot of freckles and cat eyes that are a different color"
**Generated Prompt:**
```json
{
"prompt_type": "Ultra-Photorealistic Multi-Panel Portrait",
"layout": "2x2 grid, seamless white dividers",
"aspect_ratio": "1:1",
"consistency": {
"same_person": true,
"preserve_features": ["bone structure", "freckle pattern", "eye shape", "heterochromia"]
},
"subject": {
"identity": "Same woman shown at 4 life stages",
"distinctive_features": {
"eyes": "Almond-shaped cat eyes with heterochromia - one green, one amber",
"skin": "Fair complexion densely covered in natural freckles across nose, cheeks, and forehead",
"hair": "Brown hair, natural texture"
}
},
"panels": [
{"position": "top-left", "age": 10, "expression": "curious childhood smile", "hair": "long brown hair with bangs"},
{"position": "top-right", "age": 20, "expression": "confident young adult", "hair": "shoulder-length brown hair"},
{"position": "bottom-left", "age": 40, "expression": "warm maternal gaze", "hair": "brown hair with subtle highlights"},
{"position": "bottom-right", "age": 80, "expression": "wise gentle smile", "hair": "silver-gray, formerly brown"}
],
"lighting": "Soft natural window light, consistent across all panels",
"camera": "85mm portrait lens, shallow depth of field, neutral background",
"texture_quality": "8K, natural skin texture with visible pores, realistic freckle detail",
"negative": "no cartoon effect, no airbrushed skin, no symmetrical freckles, maintain heterochromia in all panels"
}
```
**The Result:**
<p align="center">
<img src="assets/aging-portrait.jpeg" width="500" alt="Aging portrait grid - same woman at 10, 20, 40, and 80"/>
</p>
Same freckles. Same heterochromia. Same bone structure. Four life stages in one coherent image.
This example shows /last30days learning **advanced image generation patterns** (multi-panel consistency, face preservation across variations) and applying them to create a complex composite image.
---
## Example: Best Rap Songs Right Now ## Example: Best Rap Songs Right Now
**Query:** `/last30days What are the best rap songs lately` **Query:** `/last30days What are the best rap songs lately`
@@ -442,29 +839,138 @@ This example shows /last30days discovering **emerging developer workflows** - re
| Flag | Description | | Flag | Description |
|------|-------------| |------|-------------|
| `--quick` | Faster research, fewer sources (8-12 each) | | `--days=N` | Look back N days instead of 30 (e.g., `--days=7` for weekly roundup) |
| `--deep` | Comprehensive research (50-70 Reddit, 40-60 X) | | `--quick` | Faster research, fewer sources (8-12 each), skips supplemental search. YouTube: 10 videos, 3 transcripts |
| `--deep` | Comprehensive research (50-70 Reddit, 40-60 X) with extended supplemental. YouTube: 40 videos, 8 transcripts |
| `--debug` | Verbose logging for troubleshooting | | `--debug` | Verbose logging for troubleshooting |
| `--sources=reddit` | Reddit only | | `--sources=reddit` | Reddit only |
| `--sources=x` | X only | | `--sources=x` | X only |
| `--include-web` | Add native web search alongside Reddit/X (requires web search API key) |
| `--store` | Persist findings to SQLite database for watchlist/briefing integration |
| `--diagnose` | Show source availability diagnostics (API keys, Bird, YouTube, web backends) and exit |
## Requirements ## Requirements
- **OpenAI API key** - For Reddit research (uses web search) - **OpenAI API key** - For Reddit research (uses web search via Responses API)
- **xAI API key** - For X research (optional but recommended) - **Node.js 22+** - For X search (bundled Twitter GraphQL client)
- **X session** - Be logged into x.com in your browser, or set `AUTH_TOKEN`/`CT0` env vars
- **xAI API key** (optional fallback) - If the bundled search can't authenticate, falls back to xAI's Grok API
- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as a 4th source.
At least one key is required. At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH.
## How It Works ## How It Works
The skill uses: ### Two-Phase Search Architecture
- OpenAI's Responses API with web search to find Reddit discussions
- xAI's API with live X search to find posts **Phase 1: Broad discovery**
- Real Reddit thread enrichment for engagement metrics - OpenAI Responses API with `web_search` tool scoped to reddit.com
- Scoring algorithm that weighs recency, relevance, and engagement - Vendored Twitter GraphQL search (or xAI API fallback) for X search
- YouTube search + transcript extraction via yt-dlp (when installed)
- WebSearch for blogs, news, docs, tutorials
- Reddit JSON enrichment for real engagement metrics (upvotes, comments)
- Scoring algorithm weighing recency, relevance, and engagement
**Phase 2: Smart supplemental search** (new in V2)
- Extracts entities from Phase 1 results: @handles from X posts, subreddit names from Reddit
- Runs targeted follow-up searches: `from:@handle topic` on X, subreddit-scoped searches on Reddit
- Uses Reddit's free `.json` search endpoint (no API key needed for supplemental)
- Merges and deduplicates with Phase 1 results
- Skipped on `--quick` for speed; extended on `--deep`
### Model Fallback Chain
Reddit search (via OpenAI) automatically falls back through available models:
gpt-4.1 -> gpt-4o -> gpt-4o-mini
If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1), it tries the next one.
--- ---
*30 days of research. 30 seconds of work.* ## What's New in V2
*Prompt research. Trend discovery. Expert answers.* ### Way better X and Reddit results
V2 finds significantly more content than V1. Two major improvements:
**Smarter query construction** - V1 sent overly specific queries to X search (literal keyword AND matching), causing 0 results on topics that were actively trending. V2 aggressively strips research/meta words ("best", "prompt", "techniques", "tips") and question prefixes ("what are people saying about") to extract just the core topic. Example: `"vibe motion best prompt techniques"` now searches for `"vibe motion"` instead of `"vibe motion prompt techniques"` - going from 0 posts to 12+. Automatically retries with fewer keywords if the first attempt returns nothing.
**Smart supplemental search (Phase 2)** - After the initial broad search, extracts key @handles and subreddits from the results, then runs targeted follow-up searches to find content that keyword search alone misses. Example: researching "Open Claw" automatically discovers @openclaw, @steipete and drills into their posts. For Reddit, it hits the free `.json` search endpoint scoped to discovered subreddits - no extra API keys needed.
**Reddit JSON enrichment** - Fetches real upvote and comment counts from Reddit's free API for every thread, giving you actual engagement signals instead of estimates.
### Open-class skill with watchlists (v2.1)
**The biggest feature in v2.1 isn't a new source - it's what happens when you pair /last30days with an always-on bot.** The open variant adds a watchlist, briefings, and history. Add `"Competitor X"` to your watchlist, set it to weekly, and when your bot's cron job fires every Monday, you get a research briefing - what they shipped, what people said about it, what Reddit and X are discussing. The research accumulates in a local SQLite database, and you can query it anytime with natural language.
**Designed for [Open Claw](https://github.com/openclaw/openclaw) and similar always-on environments.** The watchlist stores schedules as metadata - you need cron, launchd, or a persistent bot to actually trigger runs. In Claude Code you can still use `run-one` and `run-all` manually.
### YouTube search with transcripts (v2.1)
**YouTube is now a 4th research source.** When yt-dlp is installed (`brew install yt-dlp`), /last30days automatically searches YouTube for your topic, fetches view counts and engagement data, and extracts auto-generated transcripts from the top videos. Transcripts give the synthesis engine actual content to work with - not just titles.
YouTube items go through the same scoring pipeline (relevance + recency + engagement) and are deduped, scored, and rendered alongside Reddit and X results. Views dominate YouTube's engagement formula since they're the primary discovery signal.
Inspired by [Peter Steinberger](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain. Peter's approach of combining yt-dlp for search/metadata with transcript extraction for content analysis was the direct inspiration for this feature.
### Works in OpenAI Codex CLI (v2.1)
**Same skill, different host.** Install to `~/.agents/skills/last30days` and invoke with `$last30days` inside Codex. The `agents/openai.yaml` provides Codex-specific discovery metadata. Same SKILL.md, same Python engine, same four sources.
### Bundled X search (v2.1)
**X search is fully self-contained** - No external `bird` CLI or xAI API key needed. /last30days bundles a vendored subset of Bird's Twitter GraphQL client (MIT licensed, by Peter Steinberger). Just be logged into x.com in your browser and it auto-detects your session. Falls back to xAI API if bundled search can't authenticate.
### Everything else
**`--days=N` flag** - Configurable lookback window. `/last30days topic --days=7` for a weekly roundup, `--days=14` for two weeks.
**Model fallback chain** - If your OpenAI org can't access gpt-4.1, automatically falls back to gpt-4o, then gpt-4o-mini. No config needed.
**Context-aware invitations** - After research, the skill generates specific follow-up suggestions based on what it actually learned (not generic templates). For example, after researching Nano Banana Pro it might suggest "Photorealistic product shots with natural lighting" rather than a generic "describe what you want."
**Citation priority** - Cites @handles from X and r/subreddits over web sources, because the skill's value is surfacing what *people* are saying, not what journalists wrote.
**Marketplace plugin support** - Ships with `.claude-plugin/plugin.json` for Claude Code marketplace compatibility. (Inspired by [@galligan](https://github.com/galligan)'s PR)
### Community contributions
Thanks to the contributors who helped shape V2:
- [@JosephOIbrahim](https://github.com/JosephOIbrahim) - Windows Unicode fix
- [@levineam](https://github.com/levineam) - Model fallback for unverified orgs
- [@jonthebeef](https://github.com/jonthebeef) - `--days=N` configurable lookback
---
## Security & Privacy
### Data that leaves your machine
| Destination | Data Sent | API Key Required |
|------------|-----------|-----------------|
| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
Your research topic is included in all outbound API requests. If you research sensitive topics, be aware that query strings are transmitted to the API providers listed above.
### Data stored locally
- API keys: `~/.config/last30days/.env` (chmod 600 recommended)
- Watchlist database: `~/.local/share/last30days/research.db` (SQLite)
- Briefings: `~/.local/share/last30days/briefs/`
### API key isolation
Each API key is transmitted only to its respective endpoint. Your OpenAI key is never sent to xAI, Brave, or any other provider. Browser cookies for X are read locally and used only for Twitter GraphQL requests.
---
*30 days of research. 30 seconds of work. Four sources. Zero stale prompts.*
*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. Web. - All synthesized into expert answers and copy-paste prompts.*
+391
View File
@@ -0,0 +1,391 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
context: fork
agent: Explore
disable-model-invocation: true
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
+354 -122
View File
@@ -1,16 +1,37 @@
--- ---
name: last30days name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool. version: "2.1"
argument-hint: "[topic] for [tool]" or "[topic]" description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web. Become an expert and write copy-paste-ready prompts."
context: fork argument-hint: 'last30 AI video tools, last30 best project management tools'
agent: Explore
disable-model-invocation: true
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill
user-invocable: true
disable-model-invocation: true
metadata:
clawdbot:
emoji: "📰"
requires:
env:
- OPENAI_API_KEY
bins:
- node
- python3
primaryEnv: OPENAI_API_KEY
files:
- "scripts/*"
homepage: https://github.com/mvanhorn/last30days-skill
tags:
- research
- reddit
- x
- youtube
- trends
- prompts
--- ---
# last30days: Become Expert → Write Prompts # last30days v2.1: Research Any Topic from the Last 30 Days
Research a topic across Reddit and X, internalize the best practices, then write **copy-paste-ready prompts** the user can immediately use with their target tool. Research ANY topic across Reddit, X, YouTube, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent ## CRITICAL: Parse User Intent
@@ -18,95 +39,141 @@ Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation") 1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney") 2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns: Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED - `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED - `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK - Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.** **IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it - If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results - If tool is NOT specified, run research first, then ask AFTER showing results
**Store the TOPIC** - you'll extract or ask about TARGET_TOOL later: **Store these variables:**
- `TOPIC = [extracted topic]` - `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]` - `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
--- **DISPLAY your parsing to the user.** Before running any tools, output:
## Setup Check ```
I'll research {TOPIC} across Reddit, X, and the web to find what's been discussed in the last 30 days.
Verify API key configuration exists: Parsed intent:
- TOPIC = {TOPIC}
- TARGET_TOOL = {TARGET_TOOL or "unknown"}
- QUERY_TYPE = {QUERY_TYPE}
```bash Research typically takes 2-8 minutes (niche topics take longer). Starting now.
if [ ! -f ~/.config/last30days/.env ]; then
echo "SETUP_NEEDED"
else
echo "CONFIGURED"
fi
``` ```
### If SETUP_NEEDED If TARGET_TOOL is known, mention it in the intro: "...to find {QUERY_TYPE}-style content for use in {TARGET_TOOL}."
Run NUX flow to configure API keys. Use AskUserQuestion to collect: This text MUST appear before you call any tools. It confirms to the user that you understood their request.
1. **OpenAI API Key** (optional but recommended for Reddit research)
2. **xAI API Key** (optional but recommended for X research)
Then create the config:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# At least one key is required
OPENAI_API_KEY=
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Please edit it to add your API keys, then run the skill again."
```
**STOP HERE if setup was needed.**
--- ---
## Research Execution ## Research Execution
Run the research orchestrator with the TOPIC. **Step 1: Run the research script (FOREGROUND — do NOT background this)**
**Depth options** (passed through from user's command): **CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.**
```bash
# Find skill root — works in repo checkout, Claude Code, or Codex install
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
if [ -z "${SKILL_ROOT:-}" ]; then
echo "ERROR: Could not find scripts/last30days.py" >&2
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
```
Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically takes 1-3 minutes.
The script will automatically:
- Detect available API keys
- Run Reddit/X/YouTube searches
- Output ALL results including YouTube transcripts
**Read the ENTIRE output.** It contains THREE data sections in this order: Reddit items, X items, and YouTube items. If you miss the YouTube section, you will produce incomplete stats.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
---
## STEP 2: DO WEBSEARCH AFTER SCRIPT COMPLETES
After the script finishes, do WebSearch to supplement with blogs, tutorials, and news.
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Options** (passed through from user's command):
- `--days=N` → Look back N days instead of 30 (e.g., `--days=7` for weekly roundup)
- `--quick` → Faster, fewer sources (8-12 each) - `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each) - (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X) - `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --include-web --emit=compact 2>&1
```
--- ---
## WebSearch Execution ## Judge Agent: Synthesize All Sources
**CRITICAL**: After the Python script completes, if you see `### WEBSEARCH REQUIRED ###` in the output, you MUST use your WebSearch tool to find additional sources. **After all searches complete, internally synthesize (don't display stats yet):**
**WebSearch query**: Use the TOPIC to search for recent content (last 30 days). The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
3. Weight WebSearch sources LOWER (no engagement data)
4. Identify patterns that appear across ALL sources (strongest signals)
5. Note any contradictions between sources
6. Extract the top 3-5 actionable insights
**What to search for**: **Do NOT display stats here - they come at the end, right before the invitation.**
- Blog posts, tutorials, documentation about {TOPIC}
- News articles, announcements
- Technical guides, best practices
**What to EXCLUDE** (already covered by Reddit/X):
- reddit.com URLs
- x.com or twitter.com URLs
**How many**: Find 8-15 high-quality, relevant web pages.
**After searching**: Include the WebSearch results in your synthesis. WebSearch results supplement Reddit/X but should be weighted LOWER (they lack engagement metrics like upvotes/likes that indicate community validation).
--- ---
@@ -121,93 +188,232 @@ Read the research output carefully. Pay attention to:
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says. **ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT: Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL. - **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords?
- The top 3-5 patterns/techniques that appeared across multiple sources - The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES - Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES - Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
--- ---
## THEN: Show Summary + Invite Vision ## THEN: Show Summary + Invite Vision
**CRITICAL ORDER**: Display sections in this EXACT sequence: **Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned with sources:
```
🏆 Most mentioned:
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle3, r/sub2, Complex
Notable mentions: [other specific things with 1-2 mentions]
```
**CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson)
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety)
- Parse @handles from research output and include the highest-engagement ones
- Format naturally - tables work well for wide terminals, stacked cards for narrow
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
CITATION PRIORITY (most to least preferred):
1. @handles from X — "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit — "per r/subreddit"
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
4. Web sources — ONLY when Reddit/X/YouTube don't cover that specific fact
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post.
URL FORMATTING: NEVER paste raw URLs in the output.
- **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- **GOOD:** "per Rolling Stone"
- **GOOD:** "per Complex"
Use the publication name, not the URL. The user doesn't need links — they need clean, readable text.
**BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
**GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per @honest30bgfan_"
**GOOD:** "Ye's apology got massive traction on r/hiphopheads"
**OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per Billboard"
**Lead with people, not publications.** Start each topic with what Reddit/X
users are saying/feeling, then add web context only if needed. The user came
here for the conversation, not the press release.
``` ```
---
What I learned: What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT. Quote or paraphrase what the sources said. If sources mention a specific product (ClawdBot, Cursor, etc.), use that name - don't substitute your own knowledge. The synthesis should be traceable back to the research results above.] **{Topic 1}** — [1-2 sentences about what people are saying, per @handle or r/sub]
--- **{Topic 2}** — [1-2 sentences, per @handle or r/sub]
TARGET TOOL: {tool from research or user input}
PROMPT FORMAT: [JSON / structured / natural language / keywords - whatever research recommends] **{Topic 3}** — [1-2 sentences, per @handle or r/sub]
KEY PATTERNS I'll use: KEY PATTERNS from the research:
1. [Pattern from research] 1. [Pattern] — per @handle
2. [Pattern from research] 2. [Pattern] — per r/sub
3. [Pattern from research] 3. [Pattern] — per @handle
4. [Pattern from research]
5. [Pattern from research]
---
📊 Research Complete
Analyzed {total_sources} sources from the last 30 days
├─ Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ X: {n} posts │ {sum} likes │ {sum} reposts
├─ Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2}, @{handle1}, @{handle2}
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
``` ```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice. **THEN - Stats (right before invitation):**
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it. **CRITICAL: Calculate actual totals from the research output.**
- Count posts/threads from each section
- Sum engagement: parse `[Xlikes, Yrt]` from each X post, `[Xpts, Ycmt]` from Reddit
- Identify top voices: highest-engagement @handles from X, most active subreddits
**Copy this EXACTLY, replacing only the {placeholders}:**
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
``` ```
What tool will you use these prompts with? ---
✅ All agents reported back!
Options: ├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those] ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
2. Nano Banana Pro (image generation) ├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
3. ChatGPT / Claude (text/code) ├─ 🌐 Web: {N} pages (supplementary)
4. Other (tell me) └─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
``` ```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts. If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely.
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**LAST - Invitation (adapt to QUERY_TYPE):**
**CRITICAL: Every invitation MUST include 2-3 specific example suggestions based on what you ACTUALLY learned from the research.** Don't be generic — show the user you absorbed the content by referencing real things from the results.
**If QUERY_TYPE = PROMPTING:**
```
---
I'm now an expert on {TOPIC} for {TARGET_TOOL}. What do you want to make? For example:
- [specific idea based on popular technique from research]
- [specific idea based on trending style/approach from research]
- [specific idea riffing on what people are actually creating]
Just describe your vision and I'll write a prompt you can paste straight into {TARGET_TOOL}.
```
**If QUERY_TYPE = RECOMMENDATIONS:**
```
---
I'm now an expert on {TOPIC}. Want me to go deeper? For example:
- [Compare specific item A vs item B from the results]
- [Explain why item C is trending right now]
- [Help you get started with item D]
```
**If QUERY_TYPE = NEWS:**
```
---
I'm now an expert on {TOPIC}. Some things you could ask:
- [Specific follow-up question about the biggest story]
- [Question about implications of a key development]
- [Question about what might happen next based on current trajectory]
```
**If QUERY_TYPE = GENERAL:**
```
---
I'm now an expert on {TOPIC}. Some things I can help with:
- [Specific question based on the most discussed aspect]
- [Specific creative/practical application of what you learned]
- [Deeper dive into a pattern or debate from the research]
```
**Example invitations (to show the quality bar):**
For `/last30days nano banana pro prompts for Gemini`:
> I'm now an expert on Nano Banana Pro for Gemini. What do you want to make? For example:
> - Photorealistic product shots with natural lighting (the most requested style right now)
> - Logo designs with embedded text (Gemini's new strength per the research)
> - Multi-reference style transfer from a mood board
>
> Just describe your vision and I'll write a prompt you can paste straight into Gemini.
For `/last30days kanye west` (GENERAL):
> I'm now an expert on Kanye West. Some things I can help with:
> - What's the real story behind the apology letter — genuine or PR move?
> - Break down the BULLY tracklist reactions and what fans are expecting
> - Compare how Reddit vs X are reacting to the Bianca narrative
For `/last30days war in Iran` (NEWS):
> I'm now an expert on the Iran situation. Some things you could ask:
> - What are the realistic escalation scenarios from here?
> - How is this playing differently in US vs international media?
> - What's the economic impact on oil markets so far?
--- ---
## WAIT FOR USER'S VISION ## WAIT FOR USER'S RESPONSE
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create. After showing the stats summary with your invitation, **STOP and wait** for the user to respond.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
--- ---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt ## WHEN USER RESPONDS
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise. **Read their response and match the intent:**
- If they ask a **QUESTION** about the topic → Answer from your research (no new searches, no prompt)
- If they ask to **GO DEEPER** on a subtopic → Elaborate using your research findings
- If they describe something they want to **CREATE** → Write ONE perfect prompt (see below)
- If they ask for a **PROMPT** explicitly → Write ONE perfect prompt (see below)
**Only write a prompt when the user wants one.** Don't force a prompt on someone who asked "what could happen next with Iran."
### Writing a Prompt
When the user wants a prompt, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends ### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:** **If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT.**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research. **ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Quality Checklist (run before delivering):
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
### Output Format: ### Output Format:
``` ```
@@ -215,20 +421,13 @@ Here's your prompt for {TARGET_TOOL}:
--- ---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.] [The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS]
--- ---
This uses [brief 1-line explanation of what research insight you applied]. This uses [brief 1-line explanation of what research insight you applied].
``` ```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
--- ---
## IF USER ASKS FOR MORE OPTIONS ## IF USER ASKS FOR MORE OPTIONS
@@ -251,8 +450,17 @@ For the rest of this conversation, remember:
- **TOPIC**: {topic} - **TOPIC**: {topic}
- **TARGET_TOOL**: {tool} - **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned} - **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
When the user asks for another prompt later, you don't need to re-research. Apply what you learned. **CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask a question** - answer it from your research findings
- **If they ask for a prompt** - write one using your expertise
Only do new research if the user explicitly asks about a DIFFERENT topic.
--- ---
@@ -263,7 +471,31 @@ After delivering a prompt, end with:
``` ```
--- ---
📚 Expert in: {TOPIC} for {TARGET_TOOL} 📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages 📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} web pages
Want another prompt? Just tell me what you're creating next. Want another prompt? Just tell me what you're creating next.
``` ```
---
## Security & Permissions
**What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
**What this skill does NOT do:**
- Does not post, like, or modify content on any platform
- Does not access your Reddit, X, or YouTube accounts
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
Review scripts before first use to verify behavior.
+8
View File
@@ -0,0 +1,8 @@
interface:
display_name: "Last 30 Days"
short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
brand_color: "#FF6B35"
policy:
allow_implicit_invocation: true
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 MiB

@@ -0,0 +1,929 @@
# Bird CLI Integration Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add Bird CLI as a free, zero-config alternative to xAI for X/Twitter searches with interactive installation.
**Architecture:** New `bird_x.py` module handles Bird detection, installation prompts, and search. Modified `env.py` determines X source priority (Bird → xAI → WebSearch). Main script prompts for Bird install if not found.
**Tech Stack:** Python 3, subprocess for Bird CLI calls, existing lib modules for normalization/scoring.
---
### Task 1: Create bird_x.py - Detection Functions
**Files:**
- Create: `scripts/lib/bird_x.py`
**Step 1: Create the module with detection functions**
```python
"""Bird CLI client for X (Twitter) search."""
import json
import shutil
import subprocess
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[Bird] {msg}\n")
sys.stderr.flush()
def is_bird_installed() -> bool:
"""Check if Bird CLI is installed."""
return shutil.which("bird") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if Bird is authenticated by running 'bird whoami'.
Returns:
Username if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
try:
result = subprocess.run(
["bird", "whoami"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0 and result.stdout.strip():
# Output is typically the username
return result.stdout.strip().split('\n')[0]
return None
except (subprocess.TimeoutExpired, FileNotFoundError, Exception):
return None
def check_npm_available() -> bool:
"""Check if npm is available for installation."""
return shutil.which("npm") is not None
```
**Step 2: Verify the module loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add detection functions for Bird CLI"
```
---
### Task 2: Add Bird Installation Functions
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add installation function**
Add after `check_npm_available()`:
```python
def install_bird() -> Tuple[bool, str]:
"""Install Bird CLI via npm.
Returns:
Tuple of (success, message).
"""
if not check_npm_available():
return False, "npm not found. Install Node.js first, or install Bird manually: https://github.com/steipete/bird"
try:
_log("Installing Bird CLI...")
result = subprocess.run(
["npm", "install", "-g", "@steipete/bird"],
capture_output=True,
text=True,
timeout=120,
)
if result.returncode == 0:
return True, "Bird CLI installed successfully!"
else:
error = result.stderr.strip() or result.stdout.strip() or "Unknown error"
return False, f"Installation failed: {error}"
except subprocess.TimeoutExpired:
return False, "Installation timed out"
except Exception as e:
return False, f"Installation error: {e}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
username = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": username is not None,
"username": username,
"can_install": check_npm_available(),
}
```
**Step 2: Verify functions work**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.get_bird_status())"`
Expected: Dict with installed/authenticated status
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add installation and status functions"
```
---
### Task 3: Add Bird Search Function
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add depth config and search function**
Add after imports at top:
```python
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
```
Add after `get_bird_status()`:
```python
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
# Build command
cmd = [
"bird", "search",
topic,
"--since", from_date,
"-n", str(count),
"--json",
]
# Adjust timeout based on depth
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
)
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []}
# Parse JSON output
output = result.stdout.strip()
if not output:
return {"items": []}
return json.loads(output)
except subprocess.TimeoutExpired:
return {"error": "Search timed out", "items": []}
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
```
**Step 2: Verify search function signature**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; import inspect; print(inspect.signature(bird_x.search_x))"`
Expected: `(topic: str, from_date: str, to_date: str, depth: str = 'default') -> Dict[str, Any]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add search_x function"
```
---
### Task 4: Add Bird Response Parser
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add parse function**
Add at end of file:
```python
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
screen_name = tweet.get("user", {}).get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first
if "T" in created_at:
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info
user = tweet.get("user", {})
author_handle = user.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict
engagement = {
"likes": tweet.get("like_count") or tweet.get("favorite_count"),
"reposts": tweet.get("retweet_count"),
"replies": tweet.get("reply_count"),
"quotes": tweet.get("quote_count"),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
}
items.append(item)
return items
```
**Step 2: Verify parser handles empty input**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.parse_bird_response({}))"`
Expected: `[]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add response parser matching xai_x format"
```
---
### Task 5: Add UI Functions for Bird Prompts
**Files:**
- Modify: `scripts/lib/ui.py`
**Step 1: Add Bird-related messages and prompts**
Add after `PROMO_SINGLE_KEY_PLAIN` dict (around line 128):
```python
# Bird CLI prompts
BIRD_INSTALL_PROMPT = f"""
{Colors.CYAN}{Colors.BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━{Colors.RESET}
{Colors.CYAN}🐦 FREE X/TWITTER SEARCH AVAILABLE{Colors.RESET}
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_INSTALL_PROMPT_PLAIN = """
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🐦 FREE X/TWITTER SEARCH AVAILABLE
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_AUTH_HELP = f"""
{Colors.YELLOW}Bird authentication failed.{Colors.RESET}
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: {Colors.BOLD}bird check{Colors.RESET} to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
BIRD_AUTH_HELP_PLAIN = """
Bird authentication failed.
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: bird check to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
```
**Step 2: Add prompt functions to ProgressDisplay class**
Add these methods to the `ProgressDisplay` class (after `show_promo` method, around line 310):
```python
def prompt_bird_install(self) -> bool:
"""Prompt user to install Bird CLI.
Returns:
True if user wants to install, False otherwise.
"""
if IS_TTY:
sys.stderr.write(BIRD_INSTALL_PROMPT)
else:
sys.stderr.write(BIRD_INSTALL_PROMPT_PLAIN)
sys.stderr.flush()
try:
response = input("Install Bird CLI now? (y/n): ").strip().lower()
return response in ('y', 'yes')
except (EOFError, KeyboardInterrupt):
return False
def show_bird_install_success(self, username: str):
"""Show Bird installation success message."""
msg = f"{Colors.GREEN}✓ Bird installed and authenticated as @{username}{Colors.RESET}\n" if IS_TTY else f"✓ Bird installed and authenticated as @{username}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_install_failed(self, error: str):
"""Show Bird installation failure message."""
msg = f"{Colors.RED}✗ Bird installation failed: {error}{Colors.RESET}\n" if IS_TTY else f"✗ Bird installation failed: {error}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_auth_help(self):
"""Show Bird authentication help."""
if IS_TTY:
sys.stderr.write(BIRD_AUTH_HELP)
else:
sys.stderr.write(BIRD_AUTH_HELP_PLAIN)
sys.stderr.flush()
```
**Step 3: Verify new methods exist**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib.ui import ProgressDisplay; p = ProgressDisplay('test', show_banner=False); print(hasattr(p, 'prompt_bird_install'))"`
Expected: `True`
**Step 4: Commit**
```bash
git add scripts/lib/ui.py
git commit -m "feat(ui): add Bird CLI install prompts and auth help"
```
---
### Task 6: Update env.py with X Source Detection
**Files:**
- Modify: `scripts/lib/env.py`
**Step 1: Add get_x_source function**
Add at end of file:
```python
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) → xAI (paid API)
Args:
config: Configuration dict from get_config()
Returns:
'bird' if Bird is installed and authenticated,
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
from . import bird_x
# Check Bird first (free option)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return None
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"can_install_bird": bird_status["can_install"],
}
```
**Step 2: Verify function works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; print(env.get_x_source_status(env.get_config()))"`
Expected: Dict with source status
**Step 3: Commit**
```bash
git add scripts/lib/env.py
git commit -m "feat(env): add X source detection with Bird priority"
```
---
### Task 7: Update __init__.py to Export bird_x
**Files:**
- Modify: `scripts/lib/__init__.py`
**Step 1: Add bird_x to imports**
Replace file contents with:
```python
# last30days library modules
from . import bird_x
```
**Step 2: Verify import works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/__init__.py
git commit -m "feat(lib): export bird_x module"
```
---
### Task 8: Integrate Bird into Main Script - Part 1 (Setup Phase)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Add bird_x import**
Add `bird_x` to the imports from lib (around line 36):
```python
from lib import (
bird_x,
dates,
dedupe,
env,
http,
models,
normalize,
openai_reddit,
reddit_enrich,
render,
schema,
score,
ui,
websearch,
xai_x,
)
```
**Step 2: Add Bird setup function**
Add after the imports, before `load_fixture`:
```python
def setup_bird_if_needed(progress: ui.ProgressDisplay) -> Optional[str]:
"""Check Bird status and offer installation if needed.
Returns:
'bird' if Bird is ready to use,
'declined' if user declined install,
None if Bird not available and couldn't be installed.
"""
status = bird_x.get_bird_status()
# Already working
if status["authenticated"]:
return 'bird'
# Installed but not authenticated
if status["installed"]:
progress.show_bird_auth_help()
return None
# Not installed - offer to install if npm available
if status["can_install"]:
if progress.prompt_bird_install():
success, message = bird_x.install_bird()
if success:
# Check if auth works now
username = bird_x.is_bird_authenticated()
if username:
progress.show_bird_install_success(username)
return 'bird'
else:
progress.show_bird_auth_help()
return None
else:
progress.show_bird_install_failed(message)
return None
else:
return 'declined'
return None
```
**Step 3: Verify script still loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "import scripts.last30days; print('OK')"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): add Bird setup function"
```
---
### Task 9: Integrate Bird into Main Script - Part 2 (Search Dispatch)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Modify _search_x function to support Bird**
Replace the `_search_x` function (around line 119-159) with:
```python
def _search_x(
topic: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str,
mock: bool,
x_source: str = "xai",
) -> tuple:
"""Search X via Bird CLI or xAI (runs in thread).
Args:
x_source: 'bird' or 'xai' - which backend to use
Returns:
Tuple of (x_items, raw_response, error)
"""
raw_response = None
x_error = None
if mock:
raw_response = load_fixture("xai_sample.json")
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
# Use Bird if specified
if x_source == "bird":
try:
raw_response = bird_x.search_x(
topic,
from_date,
to_date,
depth=depth,
)
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = bird_x.parse_bird_response(raw_response or {})
# Check for error in response
if raw_response and raw_response.get("error") and not x_error:
x_error = raw_response["error"]
return x_items, raw_response, x_error
# Use xAI (original behavior)
try:
raw_response = xai_x.search_x(
config["XAI_API_KEY"],
selected_models["xai"],
topic,
from_date,
to_date,
depth=depth,
)
except http.HTTPError as e:
raw_response = {"error": str(e)}
x_error = f"API error: {e}"
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
```
**Step 2: Update run_research to accept x_source parameter**
Find the `run_research` function signature (around line 161) and add `x_source` parameter:
```python
def run_research(
topic: str,
sources: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str = "default",
mock: bool = False,
progress: ui.ProgressDisplay = None,
x_source: str = "xai",
) -> tuple:
```
Then update the `_search_x` call inside (around line 218-222) to pass `x_source`:
```python
x_future = executor.submit(
_search_x, topic, config, selected_models,
from_date, to_date, depth, mock, x_source
)
```
**Step 3: Verify script syntax is valid**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -m py_compile scripts/last30days.py && echo "OK"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): dispatch X search to Bird or xAI"
```
---
### Task 10: Integrate Bird into Main Script - Part 3 (Main Function)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Update main() to check Bird before research**
In the `main()` function, after loading config and before checking available sources (around line 345-355), add Bird setup:
Find this section:
```python
# Load config
config = env.get_config()
# Check available sources
available = env.get_available_sources(config)
```
Replace with:
```python
# Load config
config = env.get_config()
# Initialize progress display early for Bird prompts
progress = ui.ProgressDisplay(args.topic, show_banner=True)
# Check Bird availability and offer install if needed
x_source_status = env.get_x_source_status(config)
x_source = x_source_status["source"]
# If no X source and Bird can be installed, offer it
if x_source is None and x_source_status["can_install_bird"]:
bird_result = setup_bird_if_needed(progress)
if bird_result == 'bird':
x_source = 'bird'
# Refresh status
x_source_status = env.get_x_source_status(config)
# Check available sources (now accounting for Bird)
available = env.get_available_sources(config)
# Override available if Bird is ready
if x_source == 'bird':
if available == 'reddit':
available = 'both' # Now have both Reddit + X (via Bird)
elif available == 'web':
available = 'x' # Now have X via Bird
```
**Step 2: Remove duplicate progress initialization**
Find and remove the later `progress = ui.ProgressDisplay(...)` line (around line 371) since we now create it earlier.
**Step 3: Pass x_source to run_research**
Find the `run_research` call (around line 413) and add `x_source` parameter:
```python
reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error = run_research(
args.topic,
sources,
config,
selected_models,
from_date,
to_date,
depth,
args.mock,
progress,
x_source=x_source or "xai",
)
```
**Step 4: Verify script runs with --help**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py --help`
Expected: Help text displays without errors
**Step 5: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): integrate Bird setup into main flow"
```
---
### Task 11: Test End-to-End with Mock Mode
**Files:**
- None (testing only)
**Step 1: Test mock mode still works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py "Claude Code" --mock --emit=compact 2>&1 | head -20`
Expected: Output showing research results without errors
**Step 2: Test Bird detection (informational)**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; import json; print(json.dumps(env.get_x_source_status(env.get_config()), indent=2))"`
Expected: JSON showing current Bird/xAI status
**Step 3: Commit any fixes if needed, then final commit**
```bash
git add -A
git commit -m "feat(bird): complete Bird CLI integration
- Add bird_x.py module for Bird CLI detection, install, and search
- Add UI prompts for interactive Bird installation
- Update env.py with X source priority (Bird > xAI)
- Integrate Bird into main research flow
- Bird uses browser cookies (free, no API key needed)"
```
---
### Task 12: Push to Private Repo
**Files:**
- None (git only)
**Step 1: Push all changes**
Run: `cd /Users/mvanhorn/last30days-skill-private && git push origin main`
Expected: Changes pushed to private repo
**Step 2: Verify commit history**
Run: `cd /Users/mvanhorn/last30days-skill-private && git log --oneline -10`
Expected: Shows Bird integration commits
---
## Summary
After completing all tasks, the skill will:
1. Check if Bird CLI is installed on startup
2. If not installed but npm available, prompt user to install
3. If installed, verify authentication via `bird whoami`
4. If authenticated, use Bird for all X searches (free)
5. If not, fall back to xAI (if key exists) or WebSearch
6. Output format identical regardless of backend used
@@ -0,0 +1,102 @@
# Bird CLI Integration Design
**Date:** 2026-02-03
**Status:** Approved
## Overview
Add Bird CLI as an alternative X/Twitter search source for the last30days skill. Bird uses browser cookie authentication (free, no API key) and provides direct access to X's GraphQL API.
## Goals
- Provide free X search without requiring xAI API key
- Seamless fallback: Bird → xAI → WebSearch
- Interactive onboarding for users without Bird installed
- Output parity with existing xAI implementation
## Detection & Priority Flow
```
On startup:
1. Check: Is Bird installed? (`which bird`)
├─ No → Offer to install: "Bird CLI not found. Install for free X search? (y/n)"
│ ├─ Yes → Run `npm install -g @steipete/bird`
│ └─ No → Continue to step 2
└─ Yes → Check: Is Bird authenticated? (`bird whoami`)
├─ Success → Use Bird for X searches
└─ Fail → Show: "Bird auth failed. Run `bird check` to diagnose."
Continue to step 2
2. Fall back to xAI if XAI_API_KEY exists
3. Fall back to WebSearch if nothing else available
```
**Priority order:** Bird → xAI → WebSearch
## New Module: `scripts/lib/bird_x.py`
### Functions
- `is_bird_installed()` → checks `which bird`, returns bool
- `is_bird_authenticated()` → runs `bird whoami`, returns username or None
- `install_bird()` → runs `npm install -g @steipete/bird`, returns success bool
- `search_x(topic, from_date, to_date, depth)` → runs `bird search` with JSON output
- `parse_bird_response(json)` → converts to same format as `xai_x.parse_x_response()`
### Search Command
```bash
bird search "Claude Code skills" --since 2026-01-04 -n 30 --json
```
- `--since` filters to last 30 days
- `-n 30` controls result count (maps to depth: quick=12, default=30, deep=60)
- `--json` gives machine-readable output
### Output Mapping
| Bird field | Our field |
|------------|-----------|
| `text` | `text` |
| `permanent_url` | `url` |
| `user.screen_name` | `author_handle` |
| `created_at` | `date` (parse to YYYY-MM-DD) |
| `like_count` | `engagement.likes` |
| `retweet_count` | `engagement.reposts` |
| `reply_count` | `engagement.replies` |
| `quote_count` | `engagement.quotes` |
Relevance: Default to 0.7, let `score.py` re-rank based on engagement.
## Modified Files
| File | Change |
|------|--------|
| `env.py` | Add `get_x_source()` → returns `'bird'`, `'xai'`, or `None` |
| `last30days.py` | Check Bird availability with interactive install prompt before research |
| `last30days.py` | In `_search_x()`, dispatch to `bird_x` or `xai_x` based on source |
| `ui.py` | Add `prompt_bird_install()` and `show_bird_auth_help()` |
## Unchanged Files
- `normalize.py` - Bird output matches xAI format after parsing
- `score.py` - Same scoring logic applies
- `dedupe.py` - Same deduplication logic
- `render.py` - X results labeled as "X" regardless of backend
## Error Handling
| Scenario | Behavior |
|----------|----------|
| Bird installed but no browser cookies | Show `bird check` guidance, fall back to xAI |
| Bird search returns 0 results | Retry with simplified query (same as xAI logic) |
| Bird search times out | Fall back to xAI if available, else WebSearch |
| npm not installed (can't install Bird) | Skip Bird, continue with xAI/WebSearch |
| User declines Bird install | Remember for session, don't ask again |
**Timeout:** 30 seconds for Bird commands
## Output Labels
Results labeled as "X" regardless of whether Bird or xAI was used. Users care about the data, not the backend.
@@ -0,0 +1,391 @@
---
title: "feat: Release last30days v2 with Bird CLI to GitHub"
type: feat
date: 2026-02-06
---
# Release last30days v2 (Bird CLI) to GitHub
## Overview
Replace the current public `last30days-skill` on GitHub with the new Bird CLI-enhanced version from `last30days-skill-private`. The new version adds free X/Twitter search via Bird CLI while maintaining backward compatibility with xAI API keys.
**Goal:** Ship with confidence. No rollbacks.
## Current State
| | Old (Public) | New (Private) |
|---|---|---|
| **Repo** | `mvanhorn/last30days-skill` | `mvanhorn/last30days-skill-private` |
| **Local path** | `~/.claude/skills/last30days/` | `~/.claude/skills/last30daystest/` (symlink) |
| **Remote** | `origin` → public repo | `origin` → private, `upstream` → public |
| **Key addition** | -- | Bird CLI (`@steipete/bird`) for free X search |
| **X source chain** | xAI API only | Bird (free) → xAI (paid) → WebSearch |
| **Uses** | 136 | 18 |
| **Latest commit** | `cc892d7` | `4230fa2` |
**Why both show as `/last30days`:** Both `SKILL.md` files declare `name: last30days` in frontmatter. Claude Code discovers both from `~/.claude/skills/` and lists them separately.
---
## Phase 0: Clean Swap (Day 1)
Remove the old skill so only the new one is active. This eliminates ambiguity during testing.
### Steps
1. **Back up the old skill** (safety net):
```bash
mv ~/.claude/skills/last30days ~/.claude/skills/last30days.backup-v1
```
2. **Promote the new skill to primary**:
```bash
# Remove the test symlink
rm ~/.claude/skills/last30daystest
# Create new symlink with the primary name
ln -s /Users/mvanhorn/last30days-skill-private ~/.claude/skills/last30days
```
3. **Verify only one `/last30days` appears**:
- Open a new Claude Code session
- Type `/last` and confirm only ONE `/last30days` shows in autocomplete
- Confirm description mentions Bird CLI
4. **Rollback procedure** (if something goes wrong):
```bash
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
```
### Acceptance Criteria
- [ ] Only one `/last30days` appears in Claude Code autocomplete
- [ ] Old skill preserved at `~/.claude/skills/last30days.backup-v1`
- [ ] New skill responds to `/last30days` invocation
---
## Phase 1: Claude's Test Plan (Automated)
These are tests Claude can run autonomously to validate the new skill before the user touches it.
### 1.1 Script-Level Smoke Tests
Run the Python scripts directly to verify core functionality without invoking the full skill.
#### Bird CLI Detection
```bash
# Test: Bird is installed and authenticated
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
print('installed:', bird_x.is_bird_installed())
print('authenticated:', bird_x.is_bird_authenticated())
print('status:', bird_x.get_bird_status())
"
```
- [ ] `is_bird_installed()` returns True (or False with clear message)
- [ ] `is_bird_authenticated()` returns True if logged into X in browser
- [ ] `get_bird_status()` returns a dict with `installed`, `authenticated`, `available` keys
#### Environment & Source Detection
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source:', env.get_x_source(config))
print('has_openai:', bool(config.get('OPENAI_API_KEY')))
"
```
- [ ] `get_x_source()` returns `'bird'` if Bird available, `'xai'` if API key set, `None` otherwise
- [ ] Config loads from `~/.config/last30days/.env`
#### Bird Search (Direct)
```bash
python3 -c "
import sys, json; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
result = bird_x.search_x('Claude Code tips', '2026-01-07', '2026-02-06', 'quick')
print(json.dumps(result, indent=2, default=str)[:2000])
"
```
- [ ] Returns search results (list of dicts with `url`, `text`, `author_handle`)
- [ ] No Python tracebacks
- [ ] Results are from the expected date range
#### Full Research Pipeline (Compact Output)
```bash
cd /Users/mvanhorn/last30days-skill-private
python3 scripts/last30days.py "Claude Code tips" --emit=compact --quick 2>&1 | head -100
```
- [ ] Completes without error
- [ ] Output includes X results (via Bird or xAI)
- [ ] Output includes Reddit results (via OpenAI) if key configured
- [ ] Stats summary shows source counts
### 1.2 Source Fallback Tests
Verify graceful degradation when sources are unavailable.
#### Bird unavailable, xAI available
```bash
# Temporarily hide Bird
PATH_BACKUP="$PATH"
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v "$(dirname $(which bird 2>/dev/null))" | tr '\n' ':')
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source (no bird):', env.get_x_source(config))
"
export PATH="$PATH_BACKUP"
```
- [ ] Falls back to `'xai'` when Bird not in PATH
- [ ] No crash or unhandled exception
#### No X source at all
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = {} # empty config, no keys
print('x_source (nothing):', env.get_x_source(config))
"
```
- [ ] Returns `None`
- [ ] No crash
### 1.3 Response Parsing Tests
Validate that Bird responses are correctly normalized to the canonical schema.
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
# Test with sample Bird response format
sample = {
'tweets': [{
'permanentUrl': 'https://x.com/user/status/123',
'text': 'Test tweet about Claude Code',
'username': 'testuser',
'likeCount': 42,
'retweetCount': 10,
'replyCount': 5,
'timeParsed': '2026-02-01T12:00:00.000Z'
}]
}
parsed = bird_x.parse_bird_response(sample)
print('Parsed count:', len(parsed))
print('First item keys:', sorted(parsed[0].keys()) if parsed else 'EMPTY')
print('URL:', parsed[0].get('url'))
print('Author:', parsed[0].get('author_handle'))
"
```
- [ ] Parses correctly with expected keys
- [ ] Handles both camelCase and snake_case fields
- [ ] URL, text, author, engagement metrics all present
### 1.4 SKILL.md Validation
```bash
# Verify YAML frontmatter parses correctly
python3 -c "
import yaml
with open('/Users/mvanhorn/last30days-skill-private/SKILL.md') as f:
content = f.read()
# Extract YAML between --- markers
parts = content.split('---', 2)
meta = yaml.safe_load(parts[1])
print('name:', meta.get('name'))
print('context:', meta.get('context'))
print('agent:', meta.get('agent'))
print('allowed-tools:', meta.get('allowed-tools'))
"
```
- [ ] `name` is `last30days` (not `last30daystest`)
- [ ] `context` is `fork`
- [ ] `agent` is `Explore`
- [ ] `allowed-tools` includes `Bash`, `WebSearch`
### 1.5 Diff Audit (Old vs New)
```bash
# Verify the only meaningful addition is bird_x.py
diff -rq ~/.claude/skills/last30days.backup-v1/scripts/lib/ \
/Users/mvanhorn/last30days-skill-private/scripts/lib/ 2>/dev/null
```
- [ ] Only new file is `bird_x.py`
- [ ] Modified files: `env.py` (source detection), `__init__.py` (exports)
- [ ] No unexpected deletions or renames
---
## Phase 2: User's Test Plan (Manual)
These require human judgment - evaluating quality, UX, and real-world behavior.
### 2.1 Basic Invocation (5 min)
Open a fresh Claude Code session after Phase 0 is complete.
| # | Test | Command | Pass Criteria |
|---|------|---------|---------------|
| 1 | Simple topic | `/last30days AI music generation` | Returns results, shows source stats |
| 2 | Topic + tool | `/last30days Suno prompts for music production` | Returns results + generates a prompt |
| 3 | Quick mode | `/last30days --quick TypeScript tips` | Faster, fewer results, still valid |
| 4 | Empty input | `/last30days` | Prompts for topic (doesn't crash) |
### 2.2 Bird CLI Verification (5 min)
| # | Test | What to Check |
|---|------|---------------|
| 1 | Source indicator | Output shows Bird as X source (not xAI) |
| 2 | X results quality | X/Twitter results are real, recent, have engagement metrics |
| 3 | Bird promo | If Bird NOT installed, shows non-blocking info banner |
| 4 | Mixed sources | Both Reddit (OpenAI) and X (Bird) results appear |
### 2.3 Fallback Behavior (5 min)
| # | Test | Setup | Expected |
|---|------|-------|----------|
| 1 | No Bird | `npm uninstall -g @steipete/bird` temporarily | Falls back to xAI or WebSearch |
| 2 | No API keys | Rename `~/.config/last30days/.env` temporarily | WebSearch-only mode works |
| 3 | Restore | Reinstall bird + restore .env | Full mode returns |
### 2.4 Output Quality (10 min)
Run 3 real research queries you care about. For each, evaluate:
- [ ] Results are actually from the last 30 days (not stale)
- [ ] Engagement metrics (likes, upvotes) are present and reasonable
- [ ] No duplicate results
- [ ] Sources are properly cited with URLs
- [ ] Synthesis is grounded in actual results (not hallucinated)
- [ ] Generated prompts (if requested) are usable
### 2.5 Comparison Test (10 min)
Before removing the backup, run the SAME query on both versions:
```bash
# New version (active)
/last30days [your topic]
# Old version (temporarily restore)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# New Claude Code session
/last30days [same topic]
# Then swap back
```
- [ ] New version produces equal or better results
- [ ] No features regressed
- [ ] Bird results add value over xAI-only
---
## Phase 3: Release Plan (30-Day Timeline)
**Target release date:** March 1, 2026 (conservative buffer before March 8 deadline)
### Week 1: Feb 6-12 - Clean & Test
| Day | Task | Owner |
|-----|------|-------|
| Feb 6 | Phase 0: Clean swap (remove old, activate new) | User |
| Feb 6 | Phase 1: Claude runs automated tests | Claude |
| Feb 7-8 | Phase 2: User runs manual tests (2.1-2.4) | User |
| Feb 9 | Phase 2.5: Comparison test | User |
| Feb 10-12 | Fix any issues found during testing | Claude + User |
### Week 2: Feb 13-19 - Harden
| Day | Task | Owner |
|-----|------|-------|
| Feb 13 | Run edge cases: unicode topics, very long topics, special chars | Claude |
| Feb 14 | Test with Bird logged out (auth expiry scenario) | User |
| Feb 15 | Review all error messages for clarity | Claude |
| Feb 16-17 | Update README.md with Bird CLI setup instructions | Claude |
| Feb 18-19 | Buffer for fixes | Claude + User |
### Week 3: Feb 20-26 - Pre-Release
| Day | Task | Owner |
|-----|------|-------|
| Feb 20 | Final diff audit: private repo vs public repo | Claude |
| Feb 21 | Strip any private/test artifacts (test symlinks, debug prints) | Claude |
| Feb 22 | Update SKILL.md description if needed | Claude |
| Feb 23 | Dry-run: push to a branch on public repo (not main) | User |
| Feb 24 | Test installation from the branch (fresh `~/.claude/skills/`) | User |
| Feb 25-26 | Buffer for fixes | Claude + User |
### Week 4: Feb 27 - Mar 1 - Ship
| Day | Task | Owner |
|-----|------|-------|
| Feb 27 | Merge branch to main on public repo | User |
| Feb 28 | Create GitHub release with changelog | Claude + User |
| Mar 1 | Delete backup: `rm -rf ~/.claude/skills/last30days.backup-v1` | User |
| Mar 1 | Archive private repo (optional) | User |
### Release Checklist (Final Gate)
Before merging to `main` on the public repo:
- [ ] All Phase 1 automated tests pass
- [ ] All Phase 2 manual tests pass
- [ ] Comparison test shows new >= old quality
- [ ] SKILL.md frontmatter is correct (`name: last30days`, not `last30daystest`)
- [ ] README.md documents Bird CLI setup
- [ ] No debug/test artifacts in codebase
- [ ] No hardcoded paths (e.g., `/Users/mvanhorn/...`)
- [ ] `.env` files are gitignored
- [ ] Git history is clean (no "test" or "WIP" commits on main)
- [ ] Bird CLI failure doesn't break the skill (graceful fallback verified)
### Rollback Plan (Emergency)
If something goes wrong after release:
```bash
# Option 1: Revert to backup (if still exists)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# Option 2: Git revert on public repo
cd ~/.claude/skills/last30days
git log --oneline -5 # find the last good commit
git revert HEAD # revert the merge commit
git push origin main
# Option 3: Pin to old version
cd ~/.claude/skills/last30days
git checkout cc892d7 # last known good commit from old version
```
---
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Bird CLI breaks after X API changes | Medium | Low | Fallback to xAI/WebSearch still works |
| Bird auth expires silently | Medium | Low | `is_bird_authenticated()` check + user message |
| Old xAI workflows regress | Low | High | Comparison test in Phase 2.5 |
| Hardcoded paths in codebase | Low | Medium | Grep for `/Users/mvanhorn` before release |
| SKILL.md name still says `last30daystest` | Low | High | Already fixed in commit `4e972d0` |
## References
- Private repo: `https://github.com/mvanhorn/last30days-skill-private.git`
- Public repo: `https://github.com/mvanhorn/last30days-skill.git`
- Bird CLI: `https://github.com/steipete/bird`
- Bird implementation plan: `docs/plans/2026-02-03-bird-cli-implementation.md`
- Bird integration design: `docs/plans/2026-02-03-bird-cli-integration-design.md`
@@ -0,0 +1,91 @@
---
title: "feat: Add visible query parsing display before research starts"
type: feat
date: 2026-02-06
---
# feat: Add Visible Query Parsing Display
## Overview
The last30days skill parses user intent (TOPIC, QUERY_TYPE, TARGET_TOOL) internally but never shows the user what it understood. The agent jumps straight from the user's `/last30days kanye west` into running tools with a generic "I'll start the research script and web searches in parallel."
Users expect to see a reformulation of their query — confirming what the agent understood before it starts searching. This builds trust and lets users course-correct before waiting for results.
## Problem Statement
Current behavior:
```
User: /last30days kanye west
Agent: I'll start the research script and web searches in parallel.
[immediately runs bash + WebSearch]
```
Expected behavior:
```
User: /last30days kanye west
Agent: 🔍 **kanye west** · News
Searching Reddit, X, and the web for the latest on kanye west...
[then runs bash + WebSearch]
```
The "Parse User Intent" section in SKILL.md tells the agent to store variables internally but never instructs it to **display** them.
## Proposed Solution
Add an explicit "Display your parsing" instruction between the "Parse User Intent" section and "Research Execution" section in SKILL.md. One new block of text — no code changes, no script changes.
## Acceptance Criteria
- [ ] Agent displays parsed TOPIC and QUERY_TYPE before running any tools
- [ ] Display is concise (1-2 lines, not a verbose block)
- [ ] Agent still runs script + WebSearch in parallel after displaying
- [ ] No changes to Python scripts — SKILL.md only
## Implementation
### SKILL.md Change
**File:** `/Users/mvanhorn/last30days-skill-private/SKILL.md`
After the "Store these variables" block (line ~38) and before "Research Execution" (line ~42), add:
```markdown
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
```
### Sync
After editing SKILL.md:
```bash
cp /Users/mvanhorn/last30days-skill-private/SKILL.md ~/.claude/skills/last30days/SKILL.md
```
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should display: 🔍 **kanye west** · News
2. `/last30days best MCP servers` — should display: 🔍 **best MCP servers** · Recommendations
3. `/last30days nano banana pro prompting for ChatGPT` — should display with tool mention
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Add display instruction between Parse User Intent and Research Execution |
@@ -0,0 +1,167 @@
---
title: "fix: last30days v2 formatting, Reddit results, and citation verbosity"
type: fix
date: 2026-02-06
---
# fix: last30days v2 Formatting, Reddit Results, and Citation Verbosity
## Overview
Four bugs found during v2 testing across 4 queries (kanye west, howie.ai, nano banana pro prompting, open claw). The skill IS executing (the agent:Explore removal worked) but output quality has regressed from v1.
## Problem Statement
| # | Bug | Severity | Where |
|---|-----|----------|-------|
| 1 | Stats emoji tree format ignored 3/4 times - agent renders plain text dashes instead | High | `SKILL.md` |
| 2 | Reddit returns 0 results for popular topics (kanye west, howie.ai) | High | `scripts/lib/openai_reddit.py` |
| 3 | Citations too verbose - every sentence has `(per @x, @y, @z; r/sub)` making summary unreadable | Medium | `SKILL.md` |
| 4 | Kanye summary is wall of text - no bold headers or paragraph breaks like nano banana pro got | Medium | `SKILL.md` |
## Proposed Fixes
### Fix 1: Stats Emoji Format Enforcement
**Root cause:** The agent ignores the emoji tree template even with BAD/GOOD examples. The template uses box-drawing characters (├─ └─) that the agent treats as decorative, not mandatory.
**Approach:** Instead of relying on the agent to copy box-drawing characters, provide the template as a **literal fill-in-the-blank** with placeholders that are impossible to misinterpret.
**File:** `SKILL.md` (stats section, currently around line 190)
**Change:** Replace the current template + BAD/GOOD examples with a single, strict fill-in format:
```
Copy this EXACTLY, replacing only the {placeholders}:
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
```
Remove the separate BAD/GOOD section (it adds length without helping).
### Fix 2: Reddit Returning 0 Results
**Root cause (from code analysis):**
1. `openai_reddit.py:53-93` - The `REDDIT_SEARCH_PROMPT` instructs the OpenAI model to strip noise words before searching. For "kanye west" this isn't the issue (no noise words), but for "howie.ai" it might strip "ai".
2. `openai_reddit.py:160-166` - The search is restricted to `allowed_domains: ["reddit.com"]` which depends on OpenAI's web_search indexing of Reddit.
3. `last30days.py:474-490` - Post-retrieval filtering: `normalize.filter_by_date_range()` + `score.score_reddit_items()` + `dedupe.dedupe_reddit()` can discard all results if date confidence is low.
4. `score.py:151-157` - Items with no engagement metrics get `-10` penalty, low date confidence gets `-10`. Combined that's `-20` which may push score below threshold.
**Approach (multi-layered):**
**A. Add subreddit-targeted search fallback** in `openai_reddit.py`:
- When the first search returns < 3 results, add a second search prompt that explicitly queries: `"r/{topic} site:reddit.com"` and `"{topic} subreddit site:reddit.com"`
- This catches cases where OpenAI's web_search doesn't find the obvious subreddit
**B. Soften post-retrieval scoring** in `score.py`:
- Change the no-engagement penalty from `-10` to `-3` (missing metrics ≠ irrelevant)
- Change low date confidence penalty from `-10` to `-5`
**C. Add minimum result guarantee** in `last30days.py`:
- If scoring filters out ALL results, keep the top 3 by raw relevance regardless of score
- Log a warning: "All Reddit results scored below threshold, keeping top 3 by relevance"
**Files to change:**
- `scripts/lib/openai_reddit.py` - Add subreddit fallback search (lines ~160-180)
- `scripts/lib/score.py` - Soften penalties (lines ~151-157)
- `scripts/last30days.py` - Add minimum result guarantee (lines ~474-490)
### Fix 3: Citations Too Verbose
**Root cause:** The SKILL.md instruction says "Every insight MUST cite at least one source" with a GOOD example showing `(per @XXX, 15 likes; r/kanye thread with 200 upvotes)` - this is too much detail per citation and the agent over-applies it.
**Approach:** Dial back to "cite 1-2 sources per KEY PATTERN, not per sentence. Use short format."
**File:** `SKILL.md` (citation section, currently around line 158)
**Change the citation rule to:**
```
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
BAD: "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
GOOD: "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
### Fix 4: Summary Formatting (Wall of Text vs Structured)
**Root cause:** The SKILL.md template for PROMPTING/NEWS/GENERAL shows:
```
What I learned:
[2-4 sentences synthesizing...]
```
This gives the agent permission to write a dense paragraph. The nano banana pro test got good formatting because PROMPTING queries naturally produce structured patterns. NEWS queries (kanye) produce narratives that become walls of text.
**Approach:** Add explicit structure to the NEWS/GENERAL format with bold topic headers.
**File:** `SKILL.md` (summary display section, around line 158)
**Change the PROMPTING/NEWS/GENERAL template to:**
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
The bold topic headers force structure. Each topic gets its own paragraph with a line break. No more wall-of-text narratives.
## Acceptance Criteria
- [ ] **Fix 1:** Stats box uses emoji tree format ├─ 🟠 🔵 🌐 └─ 🗣️ in 4/4 test queries
- [ ] **Fix 2:** "kanye west" returns >0 Reddit threads (r/kanye exists and is active)
- [ ] **Fix 3:** Summary citations are 1 per insight, short format, no engagement metrics inline
- [ ] **Fix 4:** NEWS/GENERAL summaries use bold topic headers with paragraph breaks, not wall of text
## Test Plan
Re-run the same 4 queries after fixes:
1. `/last30days kanye west` — NEWS: should get Reddit results, structured summary, emoji stats
2. `/last30days howie.ai` — GENERAL: should get Reddit if available, citations not verbose
3. `/last30days nano banana pro prompting` — PROMPTING: should maintain current good quality, reduce citation density
4. `/last30days open claw` — GENERAL: should cite @handles in summary, emoji stats
## Files to Modify
| File | Fix | Change |
|------|-----|--------|
| `SKILL.md` | 1, 3, 4 | Stats template, citation rules, summary structure |
| `scripts/lib/openai_reddit.py` | 2 | Add subreddit fallback search |
| `scripts/lib/score.py` | 2 | Soften scoring penalties |
| `scripts/last30days.py` | 2 | Add minimum result guarantee |
## References
- Current SKILL.md: `~/.claude/skills/last30days/SKILL.md`
- Private repo: `/Users/mvanhorn/last30days-skill-private/`
- Old working SKILL.md: `~/.claude/skills/last30days.backup-v1/SKILL.md`
- Reddit search module: `scripts/lib/openai_reddit.py:53-93` (prompt), `:160-166` (API call)
- Scoring module: `scripts/lib/score.py:151-157` (penalties)
- Main pipeline: `scripts/last30days.py:474-490` (filtering)
@@ -0,0 +1,177 @@
---
title: "fix: Skill execution broken - fork mode subagent ignores bash and text instructions"
type: fix
date: 2026-02-06
---
# fix: Skill Execution Broken in Fork Mode
## Overview
The last30days v2 skill stopped running its Python script and stopped showing acknowledgment text. The agent jumps straight to WebSearch, ignoring all instructions to run bash first or output text. Five attempted fixes all failed.
## Root Cause (Confirmed via Research)
**The old v1 skill worked by accident.** GitHub Issue #17283 documented that `context: fork` and `agent: Explore` were **silently ignored** in older Claude Code versions. The skill ran **inline** in the main conversation — not in a forked subagent. That's why:
- The user saw acknowledgment text (output inline to conversation)
- The bash script ran (main model followed instructions inline)
- Progress was visible (tool calls shown normally)
**Claude Code 2.1+ fixed the bug** and now properly honors `context: fork`. The skill now truly runs in an isolated subagent where:
- The model decides tool ordering independently
- Text output instructions are deprioritized vs tool calls
- "RUN THIS FIRST" instructions are **suggestions**, not commands
- There is **no mechanism** to force tool ordering in a forked subagent
**This is why every SKILL.md rewrite failed** — the problem isn't the instructions, it's `context: fork` itself.
## Evidence
| Attempt | What we tried | Result |
|---------|--------------|--------|
| 1 | "YOUR FIRST ACTION: Run this command. EXECUTE." | Agent ran script sometimes, never showed ack text |
| 2 | "YOUR FIRST OUTPUT — before ANY tool calls" + progress block | Agent ignored text, jumped to WebSearch |
| 3 | Moved progress block to very first section | Agent ignored it entirely |
| 4 | "DO NOT skip this. DO NOT jump to tool calls first." | Agent still jumped to WebSearch |
| 5 | Embedded echo in bash block + "Do NOT start with WebSearch" | Agent still jumped to WebSearch, never ran bash |
## Proposed Fix
### Option A: Remove `context: fork` (Recommended)
**Remove `context: fork` from frontmatter.** The skill runs inline in the main conversation, exactly like the old v1 skill accidentally did.
**Why this works:**
- Inline execution follows instructions sequentially
- Text output appears directly to the user
- Bash commands run when instructed
- This is how the "working" v1 skill actually operated
**File:** `SKILL.md` frontmatter
**Change from:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
context: fork
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
**Change to:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
That's it. Remove the one line.
**Then restore the old v1 instruction flow:**
1. "Parse User Intent" section FIRST (generates acknowledgment text)
2. "Research Execution" with bash command
3. "Do WebSearch" while script runs
4. Synthesize and present
### Option B: Keep `context: fork` + Use `!`command`` Preprocessing
Use shell preprocessing syntax (`!`command``) to run the script **before** the model even sees the prompt:
```markdown
## Research data (auto-fetched)
!`python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1`
```
**Risk:** Not confirmed that `$ARGUMENTS` works in `!`command`` context. More complex. The user still won't see progress text during preprocessing.
### Recommendation: Option A
Remove `context: fork`. It's one line. The old skill worked inline. The v2 skill should too. Option B is a backup if inline mode causes context window issues.
## Implementation
### Step 1: Remove `context: fork` from frontmatter
Single line removal in `SKILL.md`.
### Step 2: Restore v1-style instruction flow
The SKILL.md opening should match the public v1 pattern:
```markdown
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
[... topic/tool/query type parsing ...]
Store these variables:
- TOPIC = ...
- TARGET_TOOL = ...
- QUERY_TYPE = ...
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
**Step 2: Do WebSearch** (while script runs)
[... websearch queries based on QUERY_TYPE ...]
**Step 3: Wait for script to complete**
[... synthesis instructions ...]
```
The key structural elements from v1 that need to return:
1. Descriptive intro paragraph
2. "Parse User Intent" BEFORE any tool calls
3. Script execution as a clearly labeled step
4. WebSearch as step 2 (not step 1)
### Step 3: Keep all v2 improvements
The v2-specific improvements (citation rules, stats template, Reddit fallback, scoring changes) stay. Only the frontmatter and instruction flow change.
### Step 4: Sync and test
Copy to `~/.claude/skills/last30days/SKILL.md`, test in new session.
## Acceptance Criteria
- [ ] `context: fork` removed from SKILL.md frontmatter
- [ ] Agent outputs acknowledgment text before running tools
- [ ] Python script actually executes (Reddit + X results appear)
- [ ] WebSearch supplements, doesn't replace, script results
- [ ] Stats emoji tree format renders correctly
- [ ] Citations are sparse (1 per insight, not 3-5)
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should see ack text, script runs, Reddit results
2. `/last30days open claw` — should see ack text, X results via Bird
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Remove `context: fork`, restore v1 instruction flow |
## References
- GitHub Issue #17283: `context: fork` was silently ignored (the bug that made v1 work)
- Claude Code Skills docs: `!`command`` preprocessing syntax
- Claude Code Subagents docs: `agent: Explore` uses Haiku, read-only tools
- Public v1 SKILL.md: `github.com/mvanhorn/last30days-skill`
@@ -0,0 +1,385 @@
---
title: "test: Compare v1 (public) vs v2 (private) last30days output quality"
type: test
date: 2026-02-06
---
# test: V1 vs V2 Comparison Test Plan
## Overview
Run the same queries through both the public v1 and private v2 of last30days, compare output quality across 7 dimensions, and determine if v2 is ready to ship as the new public version.
**This plan also includes a full feature audit** identifying everything v1 has that v2 is missing — some of those gaps need fixing before shipping.
---
## How to Run the Comparison
### Setup
**V1 (public upstream):** Check out upstream SKILL.md temporarily:
```bash
# Save current v2
cp ~/.claude/skills/last30days/SKILL.md ~/.claude/skills/last30days/SKILL.md.v2
# Install v1 from upstream
cd /Users/mvanhorn/last30days-skill-private
git show upstream/main:SKILL.md > ~/.claude/skills/last30days/SKILL.md
```
Run test queries in a NEW Claude Code session (one session per query to avoid context bleed). Save output.
**V2 (private current):** Restore v2:
```bash
cp ~/.claude/skills/last30days/SKILL.md.v2 ~/.claude/skills/last30days/SKILL.md
```
Run same queries in NEW sessions. Save output.
---
## ALL Test Queries
### From README Examples (13 documented use cases)
Every single example from the README, in order:
| # | Query | Type | README Section |
|---|-------|------|---------------|
| 1 | `prompting techniques for chatgpt for legal questions` | PROMPTING + TOOL | Example: Legal Prompting |
| 2 | `best clawdbot use cases` | RECOMMENDATIONS | Example: ClawdBot Use Cases |
| 3 | `how to best setup clawdbot` | HOW-TO | Example: ClawdBot Setup |
| 4 | `prompting tips for nano banana pro for ios designs` | PROMPTING + TOOL | Example: iOS App Mockup |
| 5 | `top claude code skills` | RECOMMENDATIONS | Example: Top Claude Code Skills |
| 6 | `using ChatGPT to make images of dogs` | GENERAL | Example: Dog as Human |
| 7 | `research best practices for beautiful remotion animation videos in claude code` | PROMPTING | Example: Remotion Launch Video |
| 8 | `photorealistic people in nano banana pro` | PROMPTING | Example: Photorealistic Portraits |
| 9 | `What are the best rap songs lately` | RECOMMENDATIONS | Example: Best Rap Songs |
| 10 | `what are people saying about DeepSeek R1` | NEWS | Example: DeepSeek R1 |
| 11 | `best practices for cursor rules files for Cursor` | PROMPTING | Example: Cursor Rules |
| 12 | `prompt advice for using suno to make killer songs in simple mode` | PROMPTING | Example: Suno AI Music |
| 13 | `how do I use Codex with Claude Code on same app to make it better` | HOW-TO | Example: Codex + Claude Code |
### From Plan Documents (4 additional battle-tested queries)
| # | Query | Type | Source |
|---|-------|------|--------|
| 14 | `kanye west` | NEWS | fix-v2-formatting plan, most-tested query |
| 15 | `howie.ai` | GENERAL | fix-v2-formatting plan, edge case (domain as topic) |
| 16 | `open claw` | GENERAL | fix-v2-formatting plan, X-heavy sources |
| 17 | `nano banana pro prompting` | PROMPTING | fix-v2-formatting plan |
### Follow-up Vision Tests (pick 4 from above, ask a follow-up)
These test the prompt-generation phase specifically:
| Base Query | Follow-up Vision |
|------------|-----------------|
| #4 (nano banana pro ios) | "make a mock-up of an app for moms who swim" |
| #6 (ChatGPT dog images) | "what would my dog look like as a human prompt" |
| #12 (suno music) | "Rap song about self aware AI that loves Claude Code" |
| #13 (codex + claude code) | "how do I build a review loop workflow" |
---
## FEATURE AUDIT: V1 vs V2
### Section-by-section comparison
I diffed the full v1 (upstream/main) SKILL.md against the current v2. Here's everything.
#### KEPT (in both versions) ✅
| Feature | V1 Location | V2 Location | Notes |
|---------|------------|------------|-------|
| Parse User Intent section | Lines 23-48 | Lines 12-38 | Same logic |
| QUERY_TYPE detection (4 types) | Lines 29-36 | Lines 18-22 | Same types |
| "Don't ask about tool before research" | Lines 49-51 | Lines 31-33 | Same rule |
| Store variables block | Lines 53-56 | Lines 35-38 | Same |
| Research script execution | Lines 81-86 | Lines 59-62 | Same command |
| WebSearch by QUERY_TYPE | Lines 99-127 | Lines 77-98 | Same queries |
| "Use user's exact terminology" | Lines 129-133 | Lines 100-101 | V2 shorter but same intent |
| Judge Agent synthesis | Lines 143-151 | Lines 113-124 | Same logic |
| Internalize research (ground in actual content) | Lines 159-165 | Lines 128-135 | V2 shorter |
| RECOMMENDATIONS: extract specific names | Lines 167-177 | Lines 137-145 | Same, v2 removes BAD/GOOD example |
| Prompt format matching | Lines 193-196 | Lines 149-153 | Same |
| Summary + Stats + Invitation flow | Lines 200-250 | Lines 157-236 | Same structure, different details |
| Wait for user's vision | Lines 254-258 | Lines 240-242 | Same |
| Write ONE perfect prompt | Lines 262-275 | Lines 246-266 | Same structure |
| Context memory | Lines 298-316 | Lines 278-288 | V2 shorter |
| Output summary footer | Lines 320-340 | Lines 292-302 | Different format |
| Depth options (quick/default/deep) | Lines 135-139 | Lines 106-109 | Same |
#### ADDED in V2 (improvements) ✨
| Feature | What it does | V2 Location |
|---------|-------------|------------|
| **Query parsing display** | Shows `🔍 **{TOPIC}** · {QUERY_TYPE}` before tools | Lines 40-53 |
| **Sparse citation rules** | BAD/GOOD examples, "1 per pattern, short format" | Lines 186-193 |
| **Bold topic headers** | `**{Topic 1}** — [1-2 sentences, per source]` format | Lines 195-208 |
| **Strict stats template** | "NEVER use plain text dashes", fill-in-blank | Lines 217-230 |
| **RECOMMENDATIONS source attribution** | Each item MUST have Sources: line with @handles | Lines 178-182 |
| **Reddit 0 results handling** | Explicit instruction for 0-thread line | Line 229 |
| **Bird CLI in stats** | "(via Bird/xAI)" notation | Line 223 |
#### ❌ MISSING FROM V2 — Features V1 Has That V2 Dropped
These are the regressions. Some are intentional simplifications, others are real gaps.
**1. Use Cases Block (intro section)**
- **V1 has:** 4 use case examples right after the intro: Prompting, Recommendations, News, General — with concrete examples
- **V2 has:** Nothing. Just the intro paragraph.
- **Impact:** LOW. The query type detection handles this. But it was nice onboarding.
- **Verdict:** Skip — not needed for execution quality.
**2. Setup Check Section (API key guidance)**
- **V1 has:** Full section explaining 3 modes (Full/Partial/Web-Only), first-time setup bash script, "API keys are OPTIONAL" messaging
- **V2 has:** Nothing. Script auto-detects.
- **Impact:** LOW for experienced users. HIGH for first-time users who don't have keys.
- **Verdict:** Skip for now — script handles auto-detection. Consider adding back for public release.
**3. Anti-Pattern Examples (synthesis quality guard)**
- **V1 has:** Explicit anti-pattern block: "If user asks about 'clawdbot skills' and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as 'Claude Code skills' just because both involve 'skills'." Plus BAD/GOOD synthesis examples for RECOMMENDATIONS.
- **V2 has:** Only "Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge" — no concrete examples.
- **Impact:** MEDIUM-HIGH. Without concrete anti-patterns, the agent may conflate similar-sounding things.
- **Verdict:** ⚠️ ADD BACK. At minimum, restore the BAD/GOOD RECOMMENDATIONS example and the "don't conflate" warning.
**4. Self-Check Instruction (pre-display validation)**
- **V1 has:** "SELF-CHECK before displaying: Re-read your 'What I learned' section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it."
- **V2 has:** Nothing.
- **Impact:** MEDIUM. The self-check forces the model to validate its own output.
- **Verdict:** ⚠️ ADD BACK. One line costs nothing and catches hallucination.
**5. Quality Checklist for Prompts ⭐**
- **V1 has:** Explicit checklist before delivering a prompt:
```
### Quality Checklist:
- [ ] FORMAT MATCHES RESEARCH - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
```
- **V2 has:** Only "If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT." — one line instead of 5 checks.
- **Impact:** HIGH. This is likely what the user noticed as missing — v1 prompts felt more polished because the agent ran a checklist before delivering.
- **Verdict:** ⚠️ ADD BACK. This is the "that's a great prompt" quality feel.
**6. Prompt Format Anti-Pattern**
- **V1 has:** "ANTI-PATTERN: Research says 'use JSON prompts with device specs' but you write plain prose. This defeats the entire purpose of the research."
- **V2 has:** Only the positive instruction (use the format research recommends).
- **Impact:** MEDIUM. Negative examples ("don't do this") are powerful for LLMs.
- **Verdict:** ⚠️ ADD BACK. One line.
**7. "IF USER ASKS FOR MORE OPTIONS" Section**
- **V1 has:** "Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested."
- **V2 has:** Nothing about handling multi-prompt requests.
- **Impact:** LOW-MEDIUM. Without it, agent might dump multiple prompts unprompted.
- **Verdict:** ⚠️ ADD BACK. Two lines.
**8. Web-Only Mode Stats Template + Promo**
- **V1 has:** Separate stats template for web-only mode with "💡 Want engagement metrics? Add API keys..." promo
- **V2 has:** Only the full-mode template. If running web-only, agent has no guidance.
- **Impact:** MEDIUM for users without API keys.
- **Verdict:** Consider adding back for public release. Lower priority for now.
**9. TARGET_TOOL Question Template**
- **V1 has:** Explicit AskUserQuestion block with 4 options: [Most relevant tool], Nano Banana Pro, ChatGPT/Claude, Other
- **V2 has:** "run research first, then ask AFTER showing results" — but no actual question template.
- **Impact:** LOW-MEDIUM. Agent will still ask, just less structured.
- **Verdict:** Skip — not critical.
**10. Context Memory: "Don't re-search" Instructions**
- **V1 has:** Explicit "DO NOT run new WebSearches — you already have the research. Answer from what you learned. Cite the Reddit threads, X posts, and web sources."
- **V2 has:** Only "Only do new research if the user explicitly asks about a DIFFERENT topic."
- **Impact:** MEDIUM. Without the explicit ban, agent may re-search on follow-ups, wasting time.
- **Verdict:** ⚠️ ADD BACK. Three lines.
**11. Output Summary Footer (emoji + engagement counts)**
- **V1 has:** `📚 Expert in: {TOPIC} for {TARGET_TOOL}` and `📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages`
- **V2 has:** `Expert in: {TOPIC} for {TARGET_TOOL}` and `Based on: {n} Reddit threads + {n} X posts + {n} web pages` — no emoji, no engagement counts.
- **Impact:** LOW but noticeable. The emoji + counts make the footer feel more substantial.
- **Verdict:** ⚠️ ADD BACK. Trivial fix.
---
## Priority Fix List (Before Shipping V2 as Public)
Based on the audit, these should be restored in V2 before it replaces V1:
### Must Fix (affects output quality)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 1 | **Quality Checklist for prompts** | The "that's a great prompt" feel. V1's 5-point checklist made prompts more polished. | Add 8 lines to SKILL.md |
| 2 | **Anti-pattern examples** | BAD/GOOD synthesis examples prevent agent from conflating research. | Add 5 lines |
| 3 | **Self-check instruction** | One-line pre-display validation catches hallucination. | Add 2 lines |
| 4 | **Context Memory: don't re-search** | Prevents wasting time re-searching on follow-ups. | Add 3 lines |
### Should Fix (polish)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 5 | **Prompt format anti-pattern** | Negative example reinforces "match the format". | Add 2 lines |
| 6 | **"IF USER ASKS FOR MORE OPTIONS"** | Prevents prompt dumping. | Add 2 lines |
| 7 | **Output footer emoji + engagement counts** | More polished footer. | Edit 3 lines |
### Skip for Now (nice-to-have for public release)
| # | Missing Feature | Why Skip |
|---|----------------|----------|
| 8 | Use cases block (intro) | Doesn't affect execution |
| 9 | Setup Check section | Script auto-detects; add back for public README |
| 10 | Web-only mode stats + promo | Lower priority, most users have keys |
| 11 | TARGET_TOOL question template | Agent handles this naturally |
---
## Scoring Dimensions (1-5 scale, 7 dimensions)
### 1. Query Parsing Display
Does the agent show what it understood before starting research?
| Score | Criteria |
|-------|----------|
| 1 | No acknowledgment, jumps straight to tools |
| 2 | Generic "I'll research this" with no specifics |
| 3 | Mentions the topic but not query type |
| 4 | Shows topic + query type clearly |
| 5 | Shows topic + query type + reformulated search terms |
### 2. Source Coverage
Did it actually use Reddit, X, AND web — or skip sources?
| Score | Criteria |
|-------|----------|
| 1 | WebSearch only, script didn't run |
| 2 | Script ran but returned 0 from one major source |
| 3 | 2 of 3 sources returned results |
| 4 | All 3 sources returned results |
| 5 | All 3 sources + good volume (10+ Reddit, 10+ X, 5+ web) |
### 3. Citation Quality
Are citations sparse and useful, or verbose and noisy?
| Score | Criteria |
|-------|----------|
| 1 | Every sentence has 3+ citations chained |
| 2 | Most sentences have multiple citations |
| 3 | 1-2 citations per insight, some over-citing |
| 4 | 1 citation per pattern, short format |
| 5 | Sparse citations that prove research is real without cluttering |
### 4. Summary Structure
Is the "What I learned" section scannable or a wall of text?
| Score | Criteria |
|-------|----------|
| 1 | Single paragraph wall of text |
| 2 | Multiple paragraphs but no structure |
| 3 | Some bold text but inconsistent |
| 4 | Bold topic headers with 1-2 sentence explanations |
| 5 | Clean topic headers + KEY PATTERNS list, easy to scan |
### 5. Stats Box Format
Does the emoji stats tree render correctly?
| Score | Criteria |
|-------|----------|
| 1 | No stats shown |
| 2 | Stats shown but plain text dashes, no emoji |
| 3 | Partial emoji format, some lines wrong |
| 4 | Correct ├─ └─ │ format with emoji, minor issues |
| 5 | Perfect emoji tree with accurate counts and top voices |
### 6. Research Grounding
Does the synthesis reflect the ACTUAL research, or generic pre-training knowledge?
| Score | Criteria |
|-------|----------|
| 1 | Entirely generic knowledge, no research content |
| 2 | Mentions some research but mostly generic |
| 3 | Mix of research and generic, some conflation |
| 4 | Clearly grounded in research, minor generic leakage |
| 5 | Every insight traceable to a specific source from the research |
### 7. Prompt Quality (follow-up tests only)
When user shares vision, is the generated prompt good?
| Score | Criteria |
|-------|----------|
| 1 | Generic prompt that ignores research |
| 2 | Mentions research topics but generic structure |
| 3 | Uses some research insights, decent prompt |
| 4 | Tailored to research, correct format for target tool |
| 5 | Uses research-recommended format, specific techniques, ready to paste, "that's a great prompt" feel |
---
## Comparison Scorecard Template
```
Query: [query text]
Version: V1 / V2
Date: YYYY-MM-DD
| Dimension | Score (1-5) | Notes |
|---------------------|-------------|-------|
| Query Parsing | | |
| Source Coverage | | |
| Citation Quality | | |
| Summary Structure | | |
| Stats Box Format | | |
| Research Grounding | | |
| Prompt Quality | | (follow-up tests only) |
| **TOTAL** | **/35** | |
Script output:
- Reddit: ___ threads / ___ upvotes / ___ comments
- X: ___ posts / ___ likes / ___ reposts
- Web: ___ pages
Observations:
[Free text notes]
```
---
## Execution Plan
### Phase 1: Fix the gaps first
Apply the 7 "Must Fix" + "Should Fix" items from the audit to V2 SKILL.md. This takes ~20 minutes since it's all small text additions.
### Phase 2: Smoke test (4 queries)
Run queries #14 (kanye west), #2 (best clawdbot use cases), #8 (photorealistic nano banana pro), #10 (DeepSeek R1) on V2 only. Verify the fixes work.
### Phase 3: Full comparison (all 17 queries)
Run all 17 queries on both V1 and V2. Fill scorecards.
### Phase 4: Follow-up vision tests (4 queries)
Run the 4 follow-up vision tests. Compare prompt quality — this is where the quality checklist fix matters most.
### Phase 5: Analysis
- Sum scores per version across all queries
- Identify any dimension where v1 consistently beats v2
- Decision: ship v2, or fix more gaps first
## Acceptance Criteria
- [x] Feature audit complete (this document)
- [x] Must-fix gaps restored in V2 SKILL.md
- [ ] All 17 queries run on V2
- [ ] At least 4 queries run on V1 for comparison
- [ ] 4 follow-up vision tests completed
- [ ] Scorecards filled for each
- [ ] Total score comparison documented
- [ ] Any V1 > V2 regressions identified with fix plan
- [ ] Go/no-go decision on shipping v2 as public
## Files
| File | Purpose |
|------|---------|
| `docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md` | This plan |
| `SKILL.md` | Apply Must Fix + Should Fix items |
| `docs/test-results/v1-vs-v2-comparison.md` | Results (to be created) |
@@ -0,0 +1,208 @@
---
title: "feat: Bundle Bird X search client to eliminate npm dependency"
type: feat
date: 2026-02-07
---
# feat: Bundle Bird X search client to eliminate npm dependency
## Overview
Replace the `subprocess.run(["bird", "search", ...])` dependency in `bird_x.py` with a vendored Node.js module that calls Twitter's GraphQL search API directly. This eliminates the need for users to `npm install -g @steipete/bird` and protects against the package being removed from npm.
Bird is MIT-licensed. We have the full compiled package archived at `vendor/steipete-bird-0.8.0.tgz` and forked to `github.com/mvanhorn/bird-cli-archive`.
## Problem Statement
@steipete deleted Bird's GitHub repo on 2026-02-07. The npm package still works today, but if he unpublishes it from npm:
- New users can't `npm install -g @steipete/bird`
- The `bird` binary disappears from PATH on fresh installs
- `bird_x.py` returns 0 X results for everyone without an xAI API key
- /last30days V2's headline feature ("free X search") stops working for new users
## Proposed Solution
**Vendor Bird's search-only subset as a Node.js module inside /last30days, called from Python via `subprocess.run(["node", ...])`.**
This is the minimal-change approach:
- Keep Python as the orchestrator (bird_x.py stays mostly the same)
- Replace `subprocess.run(["bird", "search", ...])` with `subprocess.run(["node", "vendor/bird-search.mjs", ...])`
- Extract only the search-related code from Bird (not posting, bookmarks, lists, etc.)
- Cookie auth stays the same (environment variables or browser extraction)
### Why not rewrite in pure Python?
Bird's search client uses Twitter's internal GraphQL API with:
- Rotating QueryIDs (hardcoded + runtime refresh from x.com)
- Specific request header construction (bearer token, csrf, client UUIDs)
- Cursor-based pagination with Twitter-specific response parsing
- The `@steipete/sweet-cookie` dependency for browser cookie extraction
Porting all of this to Python is ~1000 lines of fragile reverse-engineering. Vendoring the working JS code is faster, safer, and easier to maintain since the archive includes source maps for debugging.
## Technical Approach
### What we need from Bird
Only 8 files from `dist/lib/` (out of 30+):
1. `twitter-client-base.js` - HTTP client, auth headers, rate limiting
2. `twitter-client-search.js` - Search mixin (the core feature)
3. `twitter-client-utils.js` - Tweet parsing, cursor extraction
4. `twitter-client-constants.js` - API endpoints, QueryIDs
5. `twitter-client-types.js` - TypeScript type stubs
6. `cookies.js` - Cookie resolution (env vars, browser extraction)
7. `runtime-query-ids.js` - QueryID refresh from x.com
8. `paginate-cursor.js` - Cursor pagination helper
Plus:
- `features.json` - GraphQL feature flags
- `query-ids.json` - Hardcoded QueryID fallbacks
### What we DON'T need
Posting, bookmarks, lists, timelines, engagement, follow, media, news, user lookup, user tweets - all the non-search mixins. This cuts the vendored code roughly in half.
### Architecture
```
scripts/
lib/
bird_x.py # MODIFIED - calls node instead of bird binary
vendor/
bird-search/
bird-search.mjs # NEW - thin CLI wrapper, ~40 lines
lib/ # VENDORED - subset of Bird's dist/lib/
twitter-client-base.js
twitter-client-search.js
twitter-client-utils.js
twitter-client-constants.js
twitter-client-types.js
cookies.js
runtime-query-ids.js
paginate-cursor.js
features.json
query-ids.json
node_modules/ # VENDORED - sweet-cookie only
@steipete/
sweet-cookie/
package.json # Minimal, points to bird-search.mjs
LICENSE # Bird's MIT license (required by MIT terms)
```
### Implementation
#### 1. Create `bird-search.mjs` wrapper (~40 lines)
A minimal Node.js script that:
- Accepts: `node bird-search.mjs <query> --count <n> --json`
- Creates a TwitterClient with search mixin only
- Resolves cookies (env vars first, then browser extraction)
- Calls `client.search(query, count)`
- Outputs JSON to stdout
- Exits with code 0 on success, 1 on error
This replaces the full `bird` CLI binary. Same interface, fraction of the code.
#### 2. Modify `bird_x.py` - change subprocess target
```python
# BEFORE (current)
cmd = ["bird", "search", query, "-n", str(count), "--json"]
# AFTER (vendored)
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
cmd = ["node", bird_search, query, "--count", str(count), "--json"]
```
Same subprocess pattern. Same JSON output format. Minimal diff.
#### 3. Update auth check functions
```python
# BEFORE
def is_bird_installed() -> bool:
return shutil.which("bird") is not None
# AFTER
def is_bird_installed() -> bool:
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
return bird_search.exists() and shutil.which("node") is not None
```
`is_bird_authenticated()` changes from `bird whoami` to a quick Node.js cookie check or environment variable check.
`install_bird()` becomes a no-op (already vendored) or removes itself entirely.
#### 4. Vendor sweet-cookie
`@steipete/sweet-cookie` is the only runtime dependency. It handles browser cookie extraction on macOS/Linux. Options:
**Option A (recommended):** Vendor sweet-cookie into `vendor/bird-search/node_modules/`. It's small (one file). This makes the skill fully self-contained with zero npm installs.
**Option B:** Fall back to environment variables only (no browser cookie extraction). Users would need to manually set `AUTH_TOKEN` and `CT0` env vars. Simpler but worse UX.
Recommend Option A - vendor it.
#### 5. Update user-facing docs
- `README.md` - Remove "Install Bird CLI" section, replace with "Requires Node.js 22+"
- `SKILL.md` - Remove Bird CLI installation instructions
- Keep the fallback chain: vendored Bird search -> xAI API key -> web-only
## Acceptance Criteria
- [ ] `bird_x.py` calls vendored Node.js module instead of `bird` binary
- [ ] `search_x()` returns identical JSON format (no downstream changes needed)
- [ ] `search_handles()` works with vendored module
- [ ] Cookie auth works via environment variables (`AUTH_TOKEN`, `CT0`)
- [ ] Cookie auth works via browser extraction (sweet-cookie)
- [ ] `is_bird_installed()` checks for vendored module + Node.js
- [ ] `install_bird()` removed or returns success immediately
- [ ] No `npm install -g @steipete/bird` required anywhere
- [ ] Bird's MIT LICENSE included in vendor directory
- [ ] README updated to remove Bird CLI install steps
- [ ] SKILL.md updated to remove Bird CLI references
- [ ] Works on macOS (primary) and Linux
- [ ] Fallback to xAI API key still works if vendored search fails
## Files Changed
| File | Action | Description |
|------|--------|-------------|
| `scripts/lib/bird_x.py` | MODIFY | Replace `["bird", ...]` subprocess calls with `["node", "vendor/bird-search/bird-search.mjs", ...]` |
| `scripts/lib/vendor/bird-search/bird-search.mjs` | CREATE | Thin Node.js wrapper that imports Bird's search client and outputs JSON |
| `scripts/lib/vendor/bird-search/lib/*.js` | VENDOR | 8 files from Bird's dist/lib/ (search subset only) |
| `scripts/lib/vendor/bird-search/lib/features.json` | VENDOR | GraphQL feature flags |
| `scripts/lib/vendor/bird-search/lib/query-ids.json` | VENDOR | Hardcoded QueryID fallbacks |
| `scripts/lib/vendor/bird-search/node_modules/` | VENDOR | sweet-cookie package |
| `scripts/lib/vendor/bird-search/package.json` | CREATE | Minimal package.json for module resolution |
| `scripts/lib/vendor/bird-search/LICENSE` | COPY | Bird's MIT license |
| `README.md` | MODIFY | Remove Bird CLI install section, add Node.js 22+ requirement |
| `SKILL.md` | MODIFY | Remove Bird CLI references |
## Dependencies & Risks
**Node.js 22+ required** - Users who had Bird CLI already have Node.js. This is not a new dependency, just a version requirement. Claude Code environments typically have Node.js.
**Twitter API changes** - The GraphQL QueryIDs may rotate. Bird includes a runtime refresh mechanism (`runtime-query-ids.js`) that fetches new IDs from x.com. This is vendored and will continue working.
**sweet-cookie platform support** - Browser cookie extraction only works on macOS (Safari, Chrome, Firefox) and Linux (Chrome, Firefox). Windows users need manual env vars. This matches Bird CLI's existing behavior.
**Legal** - Bird is MIT licensed. MIT requires including the license notice in copies. We include `LICENSE` in the vendor directory. Using Twitter's internal API is the same legal gray area Bird always operated in - user accepted this when they used Bird.
## What This Does NOT Change
- Python remains the orchestrator - bird_x.py still does query construction, retry logic, response parsing
- The fallback chain stays: vendored search -> xAI API -> web-only
- Cookie auth mechanism is identical (env vars or browser extraction)
- JSON output format is identical - no changes needed in score.py or format.py
- xai_x.py is completely untouched
## References
- Bird CLI archive: `github.com/mvanhorn/bird-cli-archive`
- Local vendor tarball: `vendor/steipete-bird-0.8.0.tgz`
- Bird search implementation: `bird-cli-archive/dist/lib/twitter-client-search.js`
- Current bird_x.py: `scripts/lib/bird_x.py`
- Fallback chain: `scripts/lib/env.py:get_x_source()`
@@ -0,0 +1,263 @@
---
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
type: feat
date: 2026-02-07
---
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
## Overview
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
## Problem Statement / Motivation
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
## Proposed Solution
### Architecture: Two-Phase Search
```
CURRENT (Phase 1 — unchanged):
Broad topic search → Reddit results + X results
NEW (Phase 2 — supplemental):
Extract entities from Phase 1 results
↓ ↓
[SUBREDDITS] [@HANDLES + #HASHTAGS]
↓ ↓
Targeted Reddit Targeted X searches
searches per sub per handle/hashtag
↓ ↓
Merge + dedupe with Phase 1 results
```
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
### Feature 1: Entity Extraction Module (NEW FILE)
**File: `scripts/lib/entity_extract.py`**
A lightweight module that parses Phase 1 results and extracts:
**From X results:**
- `@handles` — from `author_handle` field + any @mentions in post text
- `#hashtags` — from post text
- Rank by frequency: handles that appear 2+ times are "key voices"
**From Reddit results:**
- `subreddit` names — from the `subreddit` field on each result
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
- Rank by frequency: subreddits with 2+ threads are "core communities"
**Output:**
```python
{
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
"x_hashtags": ["#openclaw", "#aitools"],
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
}
```
**Rules:**
- No hardcoded entities — everything discovered dynamically from Phase 1
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
- Skip the original topic's "obvious" subreddit if it was already searched
### Feature 2: Supplemental X Search (Bird)
**File: modify `scripts/lib/bird_x.py`**
Add a `search_handles()` function:
```python
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
"""Search top handles for topic-related content."""
results = []
for handle in handles[:5]:
# Uses Bird's support for X search operators
query = f"from:{handle} {topic} since:{from_date}"
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
# ... parse results, add to list
return results
```
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
```python
# xAI supports filtering to specific handles (max 10)
tools = [{
"type": "x_search",
"x_handles": {"allowed_x_handles": top_handles[:10]}
}]
```
### Feature 3: Supplemental Reddit Search
**File: modify `scripts/lib/openai_reddit.py`**
Add a `search_subreddits()` function:
```python
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
"""Search discovered subreddits for topic-related content."""
# Build multi-subreddit query for the OpenAI web_search prompt
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
# ... single OpenAI API call, same pattern as existing search
```
**Alternative approach — Reddit JSON API (free, no API key):**
```python
def search_subreddit_json(subreddit: str, topic: str) -> list:
"""Search a specific subreddit via Reddit's free JSON endpoint."""
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
# ... parse JSON response
```
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
### Feature 4: Orchestration Changes
**File: modify `scripts/last30days.py`**
After Phase 1 completes and enrichment is done, run Phase 2:
```python
# Phase 1 (existing — unchanged)
reddit_items, x_items = run_parallel_search(...)
# Phase 2 (new — supplemental)
if reddit_items or x_items:
entities = entity_extract.extract(reddit_items, x_items)
supplemental_reddit = []
supplemental_x = []
# Run supplemental searches in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
if entities["reddit_subreddits"]:
reddit_future = executor.submit(
openai_reddit.search_subreddits,
entities["reddit_subreddits"], topic, ...
)
if entities["x_handles"] and bird_available:
x_future = executor.submit(
bird_x.search_handles,
entities["x_handles"], topic, from_date, ...
)
# Merge with Phase 1
all_reddit = reddit_items + supplemental_reddit
all_x = x_items + supplemental_x
# Dedupe handles the rest
```
**Depth-dependent behavior:**
| Depth | Phase 2 behavior |
|---|---|
| `--quick` | Skip Phase 2 entirely (speed matters) |
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
**File: modify `scripts/lib/bird_x.py`**
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
```python
def expand_thread(tweet_id: str) -> list:
"""Fetch full thread for a high-engagement tweet."""
cmd = ["bird", "thread", tweet_id, "--json"]
# ... parse thread, extract key replies
```
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
## Technical Considerations
### Performance
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
### Cost
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
- X handle search via Bird: Free (uses your X login)
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
- Thread expansion: Free via Bird
### No New Dependencies
- Entity extraction is string parsing — no NLP libraries needed
- Reddit `.json` endpoint uses existing `http.py` transport
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
### Backward Compatibility
- Phase 2 is purely additive — all existing behavior unchanged
- If Phase 2 finds nothing, output is identical to current
- Deduplication handles any overlap between Phase 1 and Phase 2
## Acceptance Criteria
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
- [x] Supplemental X searches via Bird find additional content from key handles
- [x] Supplemental Reddit searches find content in discovered subreddits
- [x] Phase 2 results are properly merged and deduped with Phase 1
- [x] `--quick` mode skips Phase 2 entirely
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
- [x] No performance regression on `--quick` mode
- [ ] Default mode adds < 10 seconds of latency
- [x] Works with Bird-only, xAI-only, and both-available configurations
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
## Implementation Order
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
6. (Stretch) Thread expansion for high-engagement posts
## Research Sources
### Reddit Search Techniques
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
### X/Twitter Search Techniques
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
### Key Insight
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
## What We're NOT Doing
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
- **Not changing the output format** — Phase 2 results merge seamlessly
- **Not hardcoding any entities** — all discovery is dynamic from search results
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
- **Not replacing the current search** — Phase 2 supplements Phase 1
@@ -0,0 +1,147 @@
---
title: "fix: X search query too restrictive, returns 0 results on popular topics"
type: fix
date: 2026-02-07
---
# fix: X search query too restrictive, returns 0 results on popular topics
## Problem
`/last30days vibe motion best prompt techniques` returned **0 X posts** despite Vibe Motion being actively discussed on X (screenshots show posts from @Godid242, @KamilStanuch, @ColdStartTheory, @higgsfield_ai).
Root cause: `_extract_core_subject()` in `bird_x.py` produces overly specific queries. Bird/X search uses **literal keyword AND matching** — ALL words must appear in a tweet. The function kept 4 keywords (`vibe motion prompt techniques`) when only 2 (`vibe motion`) were needed.
## Three Bugs Found
### Bug 1: Multi-word noise phrases never match
```python
# Current code (bird_x.py:24-38)
noise = ['best', ..., 'what are', 'what is', 'how to', 'tips for', ...]
words = topic.lower().split() # splits into individual words
result = [w for w in words if w not in noise] # compares "what" against "what are" → no match!
```
`"what are people saying about DeepSeek R1"` → keeps `"what are people saying"`**LOSES THE ENTIRE TOPIC**.
The multi-word entries (`"what are"`, `"how to"`, `"tips for"`, `"use cases"`) are dead code. They never match because `.split()` creates individual words but the noise list has multi-word strings.
### Bug 2: Missing meta/research words
The noise list has `"prompting"` but not `"prompt"`, `"prompts"`, `"techniques"`, `"tips"`, `"tricks"`, `"methods"`, etc.
- `"vibe motion best prompt techniques"``"vibe motion prompt techniques"` (4 words, should be 2)
- `"nano banana pro prompts for gemini"``"nano banana pro prompts"` (4 words, should be 3)
### Bug 3: No retry on 0 results
Reddit has multi-stage retry: full query → simplified core → subreddit fallback. X search runs once and accepts whatever comes back, even 0 results.
## Proposed Fix
All changes in `scripts/lib/bird_x.py`.
### Step 1: Fix `_extract_core_subject()` — strip phrases first, then words
```python
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search."""
text = topic.lower()
# Phase 1: Strip multi-word prefixes/suffixes (order matters - longest first)
prefixes = ['what are the best', 'what is the best', 'what are', 'what is',
'how to', 'how do i', 'tips for', 'best practices for']
for p in prefixes:
if text.startswith(p):
text = text[len(p):].strip()
break
suffixes = ['best practices', 'use cases', 'prompt techniques',
'prompting techniques']
for s in suffixes:
if text.endswith(s):
text = text[:-len(s)].strip()
break
# Phase 2: Split and filter individual noise words
noise = {'best', 'top', 'practices', 'features', 'killer', 'guide',
'tutorial', 'recommendations', 'advice', 'prompting', 'prompt',
'prompts', 'techniques', 'tips', 'tricks', 'methods',
'strategies', 'review', 'reviews', 'uses', 'usecases',
'examples', 'using', 'for', 'with', 'the', 'of', 'in', 'on',
'about', 'latest', 'new', 'news', 'update', 'updates',
'good', 'great', 'awesome', 'and', 'or', 'a', 'an', 'is',
'are', 'was', 'were', 'people', 'saying', 'think', 'said'}
words = text.split()
result = [w for w in words if w not in noise]
return ' '.join(result[:3]) or topic # Max 3 words (was 4)
```
**Expected results after fix:**
| Input | Before | After |
|-------|--------|-------|
| `vibe motion best prompt techniques` | `vibe motion prompt techniques` | `vibe motion` |
| `what are people saying about DeepSeek R1` | `what are people saying` | `deepseek r1` |
| `nano banana pro prompts for gemini` | `nano banana pro prompts` | `nano banana pro` |
| `open claw best uses` | `open claw uses` | `open claw` |
| `best claude code skills` | `claude code skills` | `claude code skills` |
| `kanye west` | `kanye west` | `kanye west` |
### Step 2: Add retry with simplified query on 0 results
In `search_x()`, after the initial search, if 0 items returned, retry with just the first 2 words of the core subject:
```python
def search_x(topic, from_date, to_date, depth="default"):
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
# ... existing Bird search code ...
items = parse_bird_response(response)
# Retry with fewer keywords if 0 results
if not items and len(core_topic.split()) > 2:
shorter = ' '.join(core_topic.split()[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
# ... retry Bird search ...
items = parse_bird_response(retry_response)
return response # or merged response
```
### Step 3 (optional): Cross-pollinate Reddit entities into X Phase 2
When X Phase 1 returns 0 results but Reddit found threads, extract brand/product names from Reddit thread titles and use them as X search fallback queries. This is lower priority — Steps 1-2 should fix most cases.
## Acceptance Criteria
- [x] `vibe motion best prompt techniques` returns >0 X posts (12 posts found)
- [x] `what are people saying about DeepSeek R1` produces query containing "deepseek r1" not "what are people saying"
- [x] No regressions on working queries (`kanye west`, `claude code skills`, `open claw`)
- [x] Retry fires when initial query returns 0, logged to stderr
- [x] `openai_reddit.py`'s `_extract_core_subject()` NOT changed (Reddit uses semantic search, not literal matching — the current function works fine there)
## Files to Change
- `scripts/lib/bird_x.py``_extract_core_subject()` rewrite + retry logic in `search_x()`
- `scripts/lib/bird_x.py``search_handles()` benefits automatically (calls `_extract_core_subject()`)
## Testing
```bash
# Mock mode (quick syntax check)
python3 scripts/last30days.py "vibe motion best prompt techniques" --mock --emit=compact 2>&1
# Live queries to verify X results
python3 scripts/last30days.py "vibe motion best prompt techniques" --quick --emit=compact 2>&1 | grep -E "X:|posts"
python3 scripts/last30days.py "what are people saying about DeepSeek R1" --quick --emit=compact 2>&1 | grep -E "X:|posts"
# Regression check
python3 scripts/last30days.py "kanye west" --quick --emit=compact 2>&1 | grep -E "X:|posts"
```
@@ -0,0 +1,243 @@
---
title: "feat: Add Codex CLI compatibility"
type: feat
date: 2026-02-14
---
# feat: Add Codex CLI Compatibility
## Overview
Make /last30days work as a Codex CLI skill alongside Claude Code. Both platforms use `SKILL.md` with YAML frontmatter — the gap is small but the details matter. Inspired by PR #24 (el-analista) and PR #5 (jblwilliams) on the public repo, applied to the v2.1 codebase.
## Research Findings
### How Codex Skills Work (from [official docs](https://developers.openai.com/codex/skills))
**Format:** Identical to Claude Code — `SKILL.md` with YAML frontmatter + Markdown body.
**Required frontmatter:** Only `name` and `description`. The official skill-creator guidance says "Do not include any other fields in YAML frontmatter." This is stricter than Claude Code which allows `version`, `allowed-tools`, `argument-hint`, etc.
**Discovery:** Codex uses "progressive disclosure" — it reads ONLY the `description` field to decide whether to invoke a skill. The body loads only after triggering. This means the description must be comprehensive about when to use/not use the skill.
**Invocation:** Users invoke with `$skill-name` or `/skills` menu. Codex can also implicitly match based on the description (configurable via `agents/openai.yaml`).
**Installation paths** (scanned in order):
| Scope | Path |
|-------|------|
| Folder | `$CWD/.agents/skills/` |
| Repo | `$REPO_ROOT/.agents/skills/` |
| User | `$HOME/.agents/skills/` |
| Admin | `/etc/codex/skills/` |
| System | Bundled |
Note: Some docs also mention `~/.codex/skills/` as an alias for `$HOME/.agents/skills/`. Both should be checked.
**`agents/openai.yaml`** (optional sidecar):
```yaml
interface:
display_name: "User-facing name"
short_description: "Brief description"
default_prompt: "Surrounding prompt template"
brand_color: "#hex"
policy:
allow_implicit_invocation: true
dependencies:
tools:
- type: "mcp"
value: "toolName"
```
**Size guidance:** Keep SKILL.md under 500 lines. Use `references/` directory for detailed docs that load on demand.
**Scripts:** Put executable code in `scripts/`. These can run without being loaded into context — good for our Python research engine.
### What Real Codex Skills Look Like (from [openai/skills catalog](https://github.com/openai/skills))
**openai-docs skill** — Uses MCP tools (`mcp__openaiDeveloperDocs__search_openai_docs`). Has a workflow section, fallback instructions if MCP isn't set up, and quality rules. Clean and focused.
**pdf skill** — Runs scripts (`pdftoppm`, `reportlab`), has file conventions (`tmp/pdfs/`, `output/pdf/`), specifies dependencies. Good example of a skill that shells out to tools like we do.
**skill-creator** — The meta-skill. Emphasizes "the context window is a public good" and treating the LLM as "already very smart — only add information it genuinely lacks." Has 6 creation steps, validation scripts, and naming conventions.
### Key Insight: Frontmatter Compatibility Problem
Claude Code SKILL.md uses:
```yaml
name: last30days
version: "2.1"
description: Research a topic...
argument-hint: 'nano banana pro prompts...'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
```
Codex wants only `name` and `description`. The question: does Codex error on unknown frontmatter fields, or ignore them?
**Safe answer:** Codex uses standard YAML parsing and likely ignores unknown keys. But the official guidance says "Do not include any other fields" — meaning it's untested territory and could break in future Codex updates.
**Our approach:** Keep one SKILL.md with Claude-specific fields. If Codex chokes, we add a thin wrapper. This is pragmatic — maintaining two SKILL.md files defeats the purpose of cross-platform compatibility.
### PR #24 Analysis (el-analista)
Good ideas to incorporate:
- Portable script path resolution (repo → Claude → Codex → agents)
- `agents/openai.yaml` for Codex discovery
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dir fallbacks with env var overrides
- Last-chance retry for Bird search (better query noise stripping)
Not applicable to v2.1:
- Based on v2.0 codebase — doesn't have YouTube, vendored Bird, or pipeline changes
- We'll cherry-pick the ideas, not the code
### PR #5 Analysis (jblwilliams)
Not needed:
- Codex JWT auth — our OpenAI API calls work natively in Codex already
- SSE response handling — we don't stream responses
- The 403 enrichment issues they hit are specific to Codex-hosted auth, not our use case
## Proposed Solution
Five changes, all additive — zero impact on existing Claude Code behavior:
### 1. Add `agents/openai.yaml` for Codex discovery
```yaml
interface:
display_name: "Last 30 Days"
short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
brand_color: "#FF6B35"
policy:
allow_implicit_invocation: true
```
### 2. Make SKILL.md script path portable
Replace the hardcoded Claude path with a lookup that checks multiple install locations:
```bash
# Find the skill root
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
if [ -z "${SKILL_ROOT:-}" ]; then
echo "ERROR: Could not find scripts/last30days.py" >&2
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
```
### 3. Platform-neutral Python output text
Replace "Claude" with "assistant" in LLM-facing output strings only. Human-facing docs (README, etc.) stay as-is.
Files:
- `scripts/last30days.py` — web search marker text (~3 lines)
- `scripts/lib/render.py` — docstrings + web-only banner (~4 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### 4. Sandbox-friendly cache/output dirs
Codex runs sandboxed. Add env var overrides + tempdir fallback (from PR #24):
**`scripts/lib/cache.py`:**
- Check `LAST30DAYS_CACHE_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/cache`
**`scripts/lib/render.py`:**
- Check `LAST30DAYS_OUTPUT_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/out`
### 5. README + installation docs
Add a "Codex Compatibility" section to README:
```markdown
## Codex Compatibility
This skill works in both Claude Code and OpenAI Codex CLI.
**Claude Code:** `git clone` into `~/.claude/skills/last30days`
**Codex CLI:** `git clone` into `~/.agents/skills/last30days`
Both use the same SKILL.md, same Python engine, same scripts.
The `agents/openai.yaml` provides Codex-specific discovery metadata.
```
## What We're NOT Doing
- **Separate SKILL.md for Codex** — One file, both platforms. Claude-specific frontmatter fields (`allowed-tools`, `version`, `argument-hint`) are likely ignored by Codex's YAML parser. If this breaks, we'll address it then.
- **Codex JWT auth (PR #5)** — Our OpenAI Responses API calls work natively in Codex. No special handling needed.
- **SSE streaming (PR #5)** — Not our use case.
- **Codex-specific tool names in SKILL.md** — Both LLMs understand "do a web search" and "run this bash command." The instructions work cross-platform as-is.
- **Publishing to openai/skills catalog** — Out of scope for now. Users install via git clone.
## Acceptance Criteria
- [x] `agents/openai.yaml` exists with proper `interface` and `policy` sections
- [x] SKILL.md uses portable path resolution (repo checkout, `~/.claude/skills/`, `~/.agents/skills/`, `~/.codex/skills/`)
- [x] Python scripts use "assistant" instead of "Claude" in LLM-facing output (~8 string replacements)
- [x] Cache dir falls back gracefully in sandboxed environments (`LAST30DAYS_CACHE_DIR` env var + `PermissionError` catch)
- [x] Output dir falls back gracefully in sandboxed environments (`LAST30DAYS_OUTPUT_DIR` env var + `PermissionError` catch)
- [x] Existing Claude Code behavior is unchanged (zero regressions)
- [x] README documents Codex installation path (`~/.agents/skills/last30days`)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
## Files to Create/Modify
### New Files
- `agents/openai.yaml` — Codex discovery metadata (~10 lines)
### Modified Files
- `SKILL.md` — Portable script path resolution (~15 lines changed)
- `README.md` — Add "Codex Compatibility" section (~15 lines)
- `scripts/last30days.py` — "Claude" → "assistant" in output strings (~3 lines)
- `scripts/lib/render.py` — "Claude" → "assistant" + output dir fallback (~15 lines)
- `scripts/lib/cache.py` — Cache dir env override + fallback (~12 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### Total scope: ~70 lines changed across 7 files. Small, additive, low risk.
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Codex rejects unknown YAML frontmatter (`allowed-tools`, etc.) | Low-Medium | Standard YAML parsers ignore unknown keys. If it breaks, strip Claude-specific fields and use `agents/openai.yaml` for metadata. |
| Codex sandbox blocks Node.js (vendored Bird) | Medium | Bird failure already falls back to xAI API. If no xAI key, X search skipped gracefully. |
| yt-dlp not in Codex sandbox PATH | Medium | YouTube already degrades gracefully — "yt-dlp not installed, skipping YouTube." |
| Codex sandbox blocks `~/.cache/` writes | Medium | Env var override + tempdir fallback handles this. (Proven approach from PR #24) |
| Codex changes skill discovery paths | Low | We check 5 paths. Easy to add more. |
| Codex description matching triggers on wrong queries | Low | Write description with clear "use when" / "do not use when" boundaries per official guidance. |
## References
### Community PRs
- [PR #24](https://github.com/mvanhorn/last30days-skill/pull/24) (el-analista) — Codex compatibility, portable paths, platform-neutral text
- [PR #5](https://github.com/mvanhorn/last30days-skill/pull/5) (jblwilliams) — Codex auth support
### Official Codex Docs
- [Agent Skills](https://developers.openai.com/codex/skills) — SKILL.md format, discovery, installation paths
- [AGENTS.md Guide](https://developers.openai.com/codex/guides/agents-md/) — Custom instructions, hierarchical loading
- [Codex CLI Features](https://developers.openai.com/codex/cli/features/) — Overview of CLI capabilities
- [Configuration Reference](https://developers.openai.com/codex/config-reference/) — config.toml, skill enable/disable
### Examples
- [openai/skills catalog](https://github.com/openai/skills) — Official curated skills
- [skill-creator](https://github.com/openai/skills/blob/main/skills/.system/skill-creator/SKILL.md) — Meta-skill for creating skills, best practices
- [pdf skill](https://github.com/openai/skills/blob/main/skills/.curated/pdf/SKILL.md) — Example of skill that runs external scripts
- [openai-docs skill](https://github.com/openai/skills/blob/main/skills/.curated/openai-docs/SKILL.md) — Example of MCP-backed skill
### Community Analysis
- [Skills in OpenAI Codex](https://blog.fsck.com/2025/12/19/codex-skills/) — Jesse Vincent's deep dive on skill internals
- [Simon Willison on skills adoption](https://simonw.substack.com/p/openai-are-quietly-adopting-skills) — Cross-platform skill format analysis
- [SkillsMP marketplace](https://skillsmp.com/) — Community marketplace supporting both Claude Code and Codex skills
@@ -0,0 +1,224 @@
---
title: "feat: Merge OpenClaw variant into main repo"
type: feat
date: 2026-02-14
---
# feat: Merge OpenClaw Variant into Main Repo
## Overview
Consolidate the `last30days-openclaw` project into `last30days-skill-private` so there's one unified Python engine powering both the main skill (Claude Code / Codex) and an "open" variant with watchlist, briefing, history, and built-in web search. The open variant also gets YouTube and Bird CLI — features the main project already has but openclaw was built before they existed.
## Problem Statement / Motivation
Right now there are two separate repos with diverging codebases:
- **`last30days-skill-private`** (main, Feb 14) — YouTube, vendored Bird, better scoring/normalization, Codex compat. But no built-in web search APIs and no persistence layer.
- **`last30days-openclaw`** (Feb 10) — SQLite store, watchlist, briefings, 3 web search backends (Parallel AI, Brave, OpenRouter). But frozen without YouTube or latest engine improvements.
They share ~80% of the same `scripts/lib/` files but are drifting apart. Maintaining two codebases is unsustainable.
**Goal:** One repo, one Python engine, two SKILL.md variants. Install once, works everywhere.
## Proposed Solution
Use `last30days-skill-private` as the base (it's 4 days newer with better code) and port the OpenClaw-exclusive features in:
### What gets ported from OpenClaw
| File | What it does | Destination |
|------|-------------|-------------|
| `scripts/store.py` | SQLite research accumulator (WAL, FTS5, dedup) | `scripts/store.py` |
| `scripts/watchlist.py` | Topic watchlist CLI (add/remove/list/run) | `scripts/watchlist.py` |
| `scripts/briefing.py` | Morning briefing generator (daily/weekly) | `scripts/briefing.py` |
| `scripts/lib/brave_search.py` | Brave Search API (free tier, 2K/mo) | `scripts/lib/brave_search.py` |
| `scripts/lib/parallel_search.py` | Parallel AI search (LLM-optimized) | `scripts/lib/parallel_search.py` |
| `scripts/lib/openrouter_search.py` | OpenRouter/Sonar Pro search | `scripts/lib/openrouter_search.py` |
| `references/research.md` | One-shot research instructions | `variants/open/references/research.md` |
| `references/watchlist.md` | Watchlist mode instructions | `variants/open/references/watchlist.md` |
| `references/briefing.md` | Briefing mode instructions | `variants/open/references/briefing.md` |
| `references/history.md` | History query instructions | `variants/open/references/history.md` |
### What gets upgraded in the ported code
- **`store.py`**: No changes needed — it's self-contained SQLite, works as-is
- **`watchlist.py`**: Remove OpenClaw cron-specific code, make cron setup generic (launchd on macOS, systemd on Linux, or manual cron)
- **`briefing.py`**: No changes needed
- **`scripts/lib/env.py`**: Merge OpenClaw's web search key support (`PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`) and `has_web_search_keys()` / `get_web_search_source()` functions into main's env.py. Drop the OpenClaw config loader (`~/.openclaw/openclaw.json`) — just use env vars and `~/.config/last30days/.env`
- **`scripts/last30days.py`**: Add OpenClaw's `_search_web()` function so the script can do web search natively when API keys are available (instead of always delegating to the assistant)
### What gets DROPPED from OpenClaw
| File | Why |
|------|-----|
| `scripts/cron_setup.py` | Too OpenClaw-platform-specific. Replace with generic scheduling docs. |
| OpenClaw config loader in `env.py` | `~/.openclaw/openclaw.json` path is platform-specific. Use env vars instead. |
| `.clawhubignore` | OpenClaw marketplace artifact, not needed in unified repo |
### New file: Open variant SKILL.md
Create `variants/open/SKILL.md` — the multi-mode skill with command routing:
```
variants/open/
├── SKILL.md # Router: watch, briefing, history, or one-shot
├── references/
│ ├── research.md # One-shot research instructions
│ ├── watchlist.md # Watchlist management instructions
│ ├── briefing.md # Briefing mode instructions
│ └── history.md # History query instructions
└── context.md # Agent memory (user preferences, source quality)
```
The open variant's SKILL.md points to `{baseDir}/scripts/last30days.py` (same engine) but adds the router and reference file system. It also adds the `--store` flag for persistence.
### How YouTube and Bird CLI get added to the open variant
They're already in `scripts/lib/youtube_yt.py` and `scripts/lib/vendor/bird/`. The open variant's SKILL.md just needs to mention YouTube in its description and the research.md reference file gets the YouTube stats line in the output format. No code changes needed — the Python engine already supports all four sources.
## Technical Considerations
### File Structure After Merge
```
last30days-skill-private/
├── SKILL.md # Main skill (Claude Code / Codex)
├── agents/openai.yaml # Codex discovery (existing)
├── variants/
│ └── open/
│ ├── SKILL.md # Open variant with routing
│ ├── references/
│ │ ├── research.md
│ │ ├── watchlist.md
│ │ ├── briefing.md
│ │ └── history.md
│ └── context.md
├── scripts/
│ ├── last30days.py # Unified engine (+ native web search)
│ ├── store.py # SQLite accumulator (from openclaw)
│ ├── watchlist.py # Watchlist CLI (from openclaw, genericized)
│ ├── briefing.py # Briefing generator (from openclaw)
│ └── lib/
│ ├── ... (existing files)
│ ├── brave_search.py # NEW from openclaw
│ ├── parallel_search.py # NEW from openclaw
│ ├── openrouter_search.py # NEW from openclaw
│ ├── youtube_yt.py # Existing
│ └── vendor/bird/ # Existing
└── README.md # Updated with open variant docs
```
### Installation for open variant users
```bash
# Claude Code (main skill — unchanged)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Open variant (with watchlist, briefings, history)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Then in Claude Code settings, point skill to variants/open/SKILL.md
# OR symlink:
ln -sf ~/.claude/skills/last30days/variants/open/SKILL.md ~/.claude/skills/last30days-open/SKILL.md
```
### env.py merge strategy
Main's `env.py` is the base. Add from OpenClaw:
- Three new key names: `PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`
- `has_web_search_keys()` function
- `get_web_search_source()` function — returns `'parallel'`, `'brave'`, or `'openrouter'`
- `get_available_sources()` update to include web-search-capable modes
### last30days.py merge strategy
Main's `last30days.py` is the base. Add from OpenClaw:
- `_search_web()` function that calls the appropriate web search backend
- `--store` CLI flag to persist findings to SQLite
- `--diagnose` CLI flag for source availability diagnostics
- Web results integration into the existing report pipeline (normalize → score → dedupe → render)
Keep main's:
- YouTube integration
- Phase 2 supplemental search
- 3-tier Reddit fallback
- Better error handling
- Minimum result guarantee
### Portable path resolution (already done)
The main SKILL.md already has portable path resolution (from Codex compat work):
```bash
for dir in "." "${CLAUDE_PLUGIN_ROOT:-}" "$HOME/.claude/skills/last30days" ...
```
The open variant's SKILL.md uses `{baseDir}` which resolves to the skill root. Both approaches work — we just need to make sure the open variant's references use `{baseDir}` consistently.
## Acceptance Criteria
- [x] `scripts/store.py` ported and working (SQLite creates on first use)
- [x] `scripts/watchlist.py` ported with generic scheduling (no OpenClaw cron dependency)
- [x] `scripts/briefing.py` ported and generates daily/weekly briefings
- [x] `scripts/lib/brave_search.py` ported and functional
- [x] `scripts/lib/parallel_search.py` ported and functional
- [x] `scripts/lib/openrouter_search.py` ported and functional
- [x] `scripts/lib/env.py` updated with web search key support
- [x] `scripts/last30days.py` has native `_search_web()` + `--store` + `--diagnose`
- [x] `variants/open/SKILL.md` exists with command routing (watch, briefing, history, research)
- [x] `variants/open/references/*.md` — all 4 reference files ported
- [x] Open variant mentions YouTube in description and research output format
- [x] Open variant uses same portable path resolution as main
- [x] Main SKILL.md behavior is unchanged (zero regressions)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
- [x] `python3 scripts/last30days.py "test topic" --diagnose` shows source availability
- [x] README documents open variant installation and usage
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| OpenClaw's store.py has import dependencies we don't have | Low | store.py uses only stdlib (sqlite3, json, datetime). Self-contained. |
| Web search backends need API keys to test | Medium | Each has a `--mock` or dry-run path. Test with real keys if available, mock otherwise. |
| watchlist.py depends on OpenClaw cron API | High | Known — strip cron_setup.py dependency, replace with generic docs for launchd/systemd/crontab. |
| Open variant SKILL.md is too long (>500 lines) | Medium | Use reference file pattern (already planned). Router SKILL.md stays under 100 lines. |
| env.py merge introduces regressions | Low | Main's env.py is well-tested. Additive changes only — new keys, new functions. |
| Two SKILL.md files = maintenance burden | Low | They serve different purposes. Main is simple one-shot. Open adds routing. Core engine is shared. |
## Files to Create/Modify
### New Files
- `variants/open/SKILL.md` — Open variant router (~100 lines)
- `variants/open/references/research.md` — One-shot research instructions (from openclaw, updated with YouTube)
- `variants/open/references/watchlist.md` — Watchlist management instructions (from openclaw)
- `variants/open/references/briefing.md` — Briefing mode instructions (from openclaw)
- `variants/open/references/history.md` — History query instructions (from openclaw)
- `variants/open/context.md` — Agent memory template
- `scripts/store.py` — SQLite accumulator (from openclaw, as-is)
- `scripts/watchlist.py` — Watchlist CLI (from openclaw, genericized)
- `scripts/briefing.py` — Briefing generator (from openclaw, as-is)
- `scripts/lib/brave_search.py` — Brave Search API (from openclaw)
- `scripts/lib/parallel_search.py` — Parallel AI search (from openclaw)
- `scripts/lib/openrouter_search.py` — OpenRouter/Sonar Pro search (from openclaw)
### Modified Files
- `scripts/lib/env.py` — Add web search key support (~30 lines added)
- `scripts/last30days.py` — Add `_search_web()`, `--store`, `--diagnose` (~80 lines added)
- `README.md` — Add open variant section (~20 lines)
### Total scope: ~12 new files (mostly copied), ~130 lines of new code in existing files.
## References
### Internal
- OpenClaw plan: `/Users/mvanhorn/last30days-openclaw/docs/plans/2026-02-10-feat-openclaw-last30days-skill-plan.md` (989 lines, comprehensive spec)
- Codex compat plan: `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md` (portable paths, platform-neutral text)
- OpenClaw source: `/Users/mvanhorn/last30days-openclaw/`
### Key files to port
- `store.py`: `/Users/mvanhorn/last30days-openclaw/scripts/store.py` (20KB, SQLite with FTS5)
- `watchlist.py`: `/Users/mvanhorn/last30days-openclaw/scripts/watchlist.py` (10KB)
- `briefing.py`: `/Users/mvanhorn/last30days-openclaw/scripts/briefing.py` (8KB)
- `brave_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/brave_search.py` (6KB)
- `parallel_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/parallel_search.py` (4KB)
- `openrouter_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/openrouter_search.py` (7KB)
- `env.py` (openclaw version): `/Users/mvanhorn/last30days-openclaw/scripts/lib/env.py` (9KB — has web search key functions)
@@ -0,0 +1,315 @@
---
title: "feat: Add YouTube transcript search as 4th source"
type: feat
date: 2026-02-14
---
# feat: Add YouTube Transcript Search
## Overview
Add YouTube as a 4th research source alongside Reddit, X, and Web. Search for recent videos on the user's topic, fetch transcripts from the top results, and feed the transcript text into the synthesis — giving the Judge Agent access to what people are *saying* in video form, not just what they're posting on social media.
**Why this matters:** For many topics (tutorials, product reviews, drama breakdowns), the best content lives on YouTube, not Reddit or X. A 20-minute video review contains 10x the signal of a tweet. The skill currently misses all of it.
## Proposed Solution
Use **yt-dlp** (already installed via Homebrew) for both YouTube search and transcript extraction. No new API keys, no new dependencies. Follows the same "zero friction" philosophy as vendored Bird search.
### Two-step process per research run:
1. **Search**: `yt-dlp "ytsearch{N}:{topic}" --dateafter {30d_ago} --flat-playlist --print` → top videos by view count
2. **Transcripts**: For top 5 videos, extract auto-generated subtitles via `yt-dlp --write-auto-subs --skip-download`, clean VTT to plaintext in Python
### Why NOT use `summarize` CLI:
- Adds 146MB brew dependency (arm64-only binary)
- Calls OpenAI API per video ($0.01-0.03 each) — adds cost on top of existing API usage
- yt-dlp already extracts raw transcripts for free (covers ~95% of videos with auto-captions)
- Raw transcripts are better for synthesis anyway — the LLM doing synthesis (Claude) should interpret the content itself, not get a pre-summarized version
`summarize` is a great standalone tool, but for integration into a research pipeline where an LLM already synthesizes everything, raw transcripts are the right input.
## Technical Approach
### Architecture
New file: `scripts/lib/youtube_yt.py` (mirrors `bird_x.py` pattern)
```
yt-dlp search → metadata (title, views, channel, date)
sort by views, take top N
yt-dlp subtitle extraction → raw VTT files
VTT cleanup → plaintext transcripts
truncate to ~500 words per video
normalize → YouTubeItem objects
score, dedupe, render (same pipeline as Reddit/X)
```
### Implementation Phases
#### Phase 1: Search + Metadata (the fast part)
**New file: `scripts/lib/youtube_yt.py`**
Core search function:
```python
def search_youtube(topic: str, from_date: str, to_date: str, depth: str = "default") -> Dict[str, Any]:
"""Search YouTube via yt-dlp. No API key needed.
Returns:
Dict with 'items' list of video metadata dicts.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
date_filter = from_date.replace("-", "") # YYYYMMDD format
# yt-dlp search with metadata extraction
cmd = [
"yt-dlp",
f"ytsearch{count}:{topic}",
"--dateafter", date_filter,
"--flat-playlist",
"--print", "%(view_count)s\t%(id)s\t%(title)s\t%(channel)s\t%(upload_date)s\t%(like_count)s\t%(comment_count)s",
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
# Parse tab-separated output, sort by views, return top N
...
```
Depth config (matches existing pattern):
```python
DEPTH_CONFIG = {
"quick": 10, # search 10, transcript top 3
"default": 20, # search 20, transcript top 5
"deep": 40, # search 40, transcript top 8
}
TRANSCRIPT_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
```
**Key detail**: `yt-dlp --flat-playlist` returns exit code 0 with empty stdout when `--dateafter` filters out everything. Check for empty output, not error codes.
#### Phase 2: Transcript Extraction (the slow part)
For top N videos (by view count), fetch transcripts:
```python
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Returns:
Plaintext transcript string, or None if no captions available.
"""
cmd = [
"yt-dlp",
"--write-auto-subs",
"--sub-lang", "en",
"--sub-format", "vtt",
"--skip-download",
"-o", f"{temp_dir}/%(id)s",
f"https://www.youtube.com/watch?v={video_id}",
]
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
vtt_path = Path(temp_dir) / f"{video_id}.en.vtt"
if not vtt_path.exists():
return None
return _clean_vtt(vtt_path.read_text())
```
VTT cleanup (~10 lines of Python):
```python
def _clean_vtt(vtt_text: str) -> str:
"""Convert VTT subtitle format to clean plaintext."""
text = re.sub(r'^WEBVTT.*?\n\n', '', vtt_text, flags=re.DOTALL)
text = re.sub(r'\d{2}:\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}:\d{2}\.\d{3}.*\n', '', text)
text = re.sub(r'<[^>]+>', '', text)
lines = text.strip().split('\n')
seen = set()
unique = []
for line in lines:
stripped = line.strip()
if stripped and stripped not in seen:
seen.add(stripped)
unique.append(stripped)
return re.sub(r'\s+', ' ', ' '.join(unique)).strip()
```
**Parallelization**: Run transcript fetches in parallel using ThreadPoolExecutor (same pattern as Phase 2 supplemental searches for Reddit/X):
```python
def fetch_transcripts_parallel(video_ids: List[str], max_workers: int = 5) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel."""
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
for vid in video_ids
}
results = {}
for future in as_completed(futures):
vid = futures[future]
results[vid] = future.result()
return results
```
#### Phase 3: Integration into Pipeline
**Update `scripts/lib/schema.py`** — add YouTubeItem:
```python
@dataclass
class YouTubeItem:
id: str # video_id
title: str
url: str
channel_name: str
date: Optional[str]
date_confidence: str # always "high" for YouTube
engagement: Engagement # views, likes, comments
transcript_snippet: str # first ~500 words of transcript
relevance: float
why_relevant: str
subs: Optional[SubScores] = None
score: int = 0
```
Update `Report` to add:
```python
youtube: List[YouTubeItem] = field(default_factory=list)
youtube_error: Optional[str] = None
```
**Update `scripts/lib/score.py`** — YouTube-specific engagement weights:
```python
def compute_youtube_engagement_raw(views, likes, comments):
"""YouTube engagement: views dominate, likes secondary, comments tertiary."""
return (
0.50 * math.log1p(views or 0) +
0.35 * math.log1p(likes or 0) +
0.15 * math.log1p(comments or 0)
)
```
**Update `scripts/last30days.py`** — add YouTube to ThreadPoolExecutor:
```python
with ThreadPoolExecutor(max_workers=3) as executor: # was 2
if run_reddit:
reddit_future = executor.submit(_search_reddit, ...)
if run_x:
x_future = executor.submit(_search_x, ...)
if run_youtube:
youtube_future = executor.submit(_search_youtube, ...)
```
**Update `scripts/lib/render.py`** — YouTube section in compact output:
```
### YouTube Videos
**{id}** (score:{score}) {channel_name} ({date}) [{views} views, {likes} likes]
{title}
https://www.youtube.com/watch?v={id}
{transcript_snippet[:200]}...
*{why_relevant}*
```
**Update `scripts/lib/env.py`** — YouTube availability detection:
```python
def is_ytdlp_available() -> bool:
return shutil.which("yt-dlp") is not None
```
No API key needed. YouTube search is available whenever yt-dlp is in PATH.
#### Phase 4: SKILL.md Updates
Stats box adds YouTube line:
```
├─ 🎥 YouTube: {N} videos │ {N} views │ {N} transcripts
```
Citation priority updated:
```
1. @handles from X
2. YouTube creators — "per [Channel Name] on YouTube"
3. r/subreddits from Reddit
4. Web sources
```
Synthesis instructions updated to weight YouTube transcripts highly — a 20-minute video transcript with 500K views is a stronger signal than a tweet with 50 likes.
## Acceptance Criteria
- [x] `yt-dlp` search returns videos matching topic within date range
- [x] Transcripts extracted for top N videos (auto-generated captions)
- [x] Videos without captions gracefully skipped (no error)
- [x] YouTube results appear in compact output with engagement metrics
- [x] YouTube items scored and ranked alongside Reddit/X items
- [x] YouTube auto-activates when yt-dlp is available (no --sources flag needed)
- [x] SKILL.md stats box includes YouTube line
- [x] Transcript snippets (first ~500 words) included in output for LLM synthesis
- [ ] Total YouTube search + transcript extraction completes within 30 seconds
- [x] Works when yt-dlp is not installed (graceful degradation, no crash)
- [ ] Mock mode works for testing without network
## Dependencies & Risks
**Dependencies:**
- `yt-dlp` (Homebrew) — already installed, widely available via brew/pip/standalone
- No API keys needed
- No new Python packages (just subprocess + regex)
**Risks:**
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| yt-dlp search is slow (>10s) | Medium | Set 30s timeout, run in parallel with Reddit/X |
| YouTube blocks yt-dlp | Low | yt-dlp is actively maintained with anti-bot updates. Degrade gracefully. |
| Videos lack auto-captions | Medium (~5%) | Skip those videos, note in output. Transcript is enrichment, not required. |
| Transcript extraction adds latency | High | Only fetch top 3-5, run in parallel, use tempdir |
| yt-dlp not installed for some users | Medium | Auto-detect, skip YouTube with info message, don't error |
| Linux `--dateafter` date format differs | Low | Use Python to format date, not shell `date -v` |
## Files to Create/Modify
### New Files
- `scripts/lib/youtube_yt.py` — search, transcript extraction, parsing
- `tests/test_youtube_yt.py` — unit tests
- `fixtures/youtube_sample.json` — mock data for tests
### Modified Files
- `scripts/lib/schema.py` — add YouTubeItem, update Report
- `scripts/lib/normalize.py` — add normalize_youtube_items()
- `scripts/lib/score.py` — add YouTube engagement scoring
- `scripts/lib/dedupe.py` — add YouTube dedup (title + channel Jaccard)
- `scripts/lib/render.py` — add YouTube section to compact + full report
- `scripts/lib/env.py` — add yt-dlp availability check, update source detection
- `scripts/last30days.py` — add _search_youtube(), update run_research(), update arg parser
- `SKILL.md` — update stats box, citation rules, synthesis instructions
- `README.md` — document YouTube source, yt-dlp requirement
## Alternative Approaches Considered
**1. YouTube Data API v3** — Rejected. Requires API key + Google Cloud project. Adds friction, counter to "zero config" philosophy. 10K quota/day limit. yt-dlp has no limits.
**2. steipete/summarize for transcripts** — Rejected for MVP. Adds 146MB dependency, requires brew tap, calls OpenAI API per video (adds cost). Raw transcripts via yt-dlp are better input for our synthesis LLM anyway. Could revisit as optional enhancement for captionless videos.
**3. youtube-transcript-api Python package** — Considered. Lightweight, Python-native transcript fetcher. But adds a pip dependency to a project that currently has zero Python deps. yt-dlp is already a brew dependency we can auto-detect.
**4. Skip transcripts, just use metadata** — Rejected. Titles + view counts alone don't give the synthesis LLM enough to work with. Transcripts are what make YouTube a *research* source vs just a link list.
## Cost Impact
**Zero additional API cost.** yt-dlp scrapes YouTube directly. No API keys, no token usage. The only cost is the existing OpenAI/xAI calls for Reddit/X search, which are unchanged.
**Time impact:** Adds ~10-20 seconds to research (search + parallel transcript extraction), running in parallel with Reddit/X so effective wall-clock increase is minimal.
+33
View File
@@ -0,0 +1,33 @@
# PR Credits — Thank After V2 Goes Live
When V2 is pushed to the public repo, comment on each PR to thank the contributor and let them know their work was integrated.
## Integrated (cherry-picked into V2)
| PR | Author | What | Status |
|---|---|---|---|
| [#17](https://github.com/mvanhorn/last30days-skill/pull/17) | **@JosephOIbrahim** | Windows Unicode fix (cp1252 emoji crash) | Merge or close with thanks |
| [#16](https://github.com/mvanhorn/last30days-skill/pull/16) | **@levineam** | Handle 403 model access errors + gpt-4.1 fallback | Merge or close with thanks |
| [#18](https://github.com/mvanhorn/last30days-skill/pull/18) | **@jonthebeef** | `--days=N` configurable lookback flag | Merge or close with thanks |
| [#1](https://github.com/mvanhorn/last30days-skill/pull/1) | **@galligan** (Matt Galligan) | Marketplace plugin conversion — we took a lighter approach inspired by his PR | Close with thanks, explain lighter approach |
## Already Fixed in V2 (close with thanks)
| PR | Author | What |
|---|---|---|
| [#15](https://github.com/mvanhorn/last30days-skill/pull/15) | **@rszrszrsz** | YAML argument-hint fix — already fixed in V2 |
| [#11](https://github.com/mvanhorn/last30days-skill/pull/11) | **@nerveband** | Same YAML fix (earlier) — already fixed in V2 |
## Not Integrated (close with explanation)
| PR | Author | What | Why |
|---|---|---|---|
| [#5](https://github.com/mvanhorn/last30days-skill/pull/5) | **@jblwilliams** | Codex auth with OpenAI Responses API | Good idea, too complex for now (358 lines SSE/JWT). May revisit. |
| [#14](https://github.com/mvanhorn/last30days-skill/pull/14) | **@thangman1** | WebSearch-first, API keys optional | Philosophical shift — V2 already does WebSearch in parallel |
| [#10](https://github.com/mvanhorn/last30days-skill/pull/10) | **@thetechreviewer** | OpenRouter API integration | Too large (1029 lines), adds MCP server |
## Suggested Comment Template
> Thanks for this PR! We integrated your [fix/feature] into V2 (commit XXXXX). Really appreciate the contribution. 🙏
>
> Closing this PR since the changes are now in main via a different commit, but full credit to you for the idea and implementation.
+122
View File
@@ -0,0 +1,122 @@
# v2.1 Launch Copy (DRAFT — do not publish)
## Context
Bird CLI (@steipete/bird) has been deprecated on npm and the GitHub repo deleted. steipete was asked to take it down (likely by X). Nobody has forked and republished it. Our v2.1 vendors a search-only subset of Bird v0.8.0 (MIT licensed) so users don't need any external tools.
YouTube transcripts are the second headline feature. Inspired by Peter Steinberger's yt-dlp + summarize toolchain. We use yt-dlp directly (no summarize dependency) — search YouTube, grab transcripts, feed them into the synthesis. Zero API keys, zero cost.
---
## README: "New in V2.1" blurb
**New in V2.1 — two headline features:**
- **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a tweet — now the skill reads it. Inspired by @steipete's yt-dlp + summarize toolchain.
- **X search is fully bundled.** No external `bird` CLI or xAI API key needed. Just Node.js 22+ and your browser cookies. Uses a vendored subset of Bird's Twitter GraphQL client (MIT licensed, originally by @steipete).
---
## README: X Search Authentication section
### X Search Authentication
X search reads your existing browser cookies — no API keys or login commands needed.
**Safari (recommended on Mac):** Just be logged into x.com. No setup needed.
**Chrome:** Works, but macOS will prompt you to allow Keychain access the first time. Click "Allow" (or "Always Allow" to stop future prompts).
**Firefox:** Just be logged into x.com. No setup needed.
**Manual fallback:** If cookie auto-detection doesn't work, set these env vars (grab them from your browser's dev tools → Application → Cookies → x.com):
```bash
export AUTH_TOKEN=your_auth_token
export CT0=your_ct0_token
```
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
```
---
## README: Install block env line
```
XAI_API_KEY=xai-... # optional — cookie auth is default for X search
```
---
## SKILL.md: Stats line
```
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
```
---
## GitHub issue #19 response (post AFTER publishing)
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted — steipete was asked to take it down.
>
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly — no external CLI, no `npm install`, no brew. Just Node.js 22+ and your browser cookies.
>
> It also adds **YouTube as a 4th source** — when yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
>
> If you're on a Mac, Safari is the easiest path for X — just be logged into x.com. Chrome works too but macOS will prompt for Keychain access the first time.
>
> If cookie auto-detection doesn't work, you can set `AUTH_TOKEN` and `CT0` env vars manually (grab from browser dev tools → Application → Cookies → x.com).
>
> The xAI API (`XAI_API_KEY`) also still works as a fallback.
---
## X/Social launch post
### Short (280 chars)
/last30days v2.1 is out 🚀
Two new features:
→ YouTube transcripts as a 4th source (yt-dlp)
→ X search fully bundled (no bird CLI needed)
Research any topic across Reddit, X, YouTube & web in one command.
h/t @steipete for the inspiration on both.
github.com/mvanhorn/last30days-skill
### Thread version (post 1)
/last30days v2.1 just shipped — two headline features:
1️⃣ YouTube transcripts as a 4th source
When yt-dlp is installed, the skill searches YouTube, grabs view counts, and extracts auto-generated transcripts from top videos. A 20-min review has 10x the signal of a tweet — now the synthesis reads it.
### Thread version (post 2)
2️⃣ X search is fully bundled
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Just be logged into x.com in your browser. No npm install, no API keys.
Both features inspired by @steipete's tooling.
### Thread version (post 3)
YouTube goes through the same scoring pipeline as Reddit and X — relevance, recency, engagement. Transcripts get truncated to ~500 words per video and fed into the synthesis engine alongside social posts.
Zero API keys for YouTube. Zero cost. Just `brew install yt-dlp`.
### Thread version (post 4)
Try it:
```
/last30days [any topic]
```
Reddit + X + YouTube + Web. Four sources, one command, copy-paste prompts.
github.com/mvanhorn/last30days-skill
+250
View File
@@ -0,0 +1,250 @@
# v2.1 Launch Posts - WORKING DRAFT
## Post 1 (Hook)
V2.1 of @slashlast30days launches today. Three headline features:
1. @openclaw + watchlists - automated research on your competitors, people, and topics
2. YouTube transcripts as a 4th source
3. Works in OpenAI Codex
## Post 2 (Watchlist + Open Claw - THE KILLER FEATURE)
@openclaw + WATCHLISTS.
Pair /last30days with @openclaw and it re-researches topics on a schedule across Reddit, X, YouTube, and the web.
"last30 watch my biggest competitor every week"
"last30 watch Peter Steinberger every 30 days"
"last30 watch AI video tools monthly"
Research that runs while you sleep. Designed for @openclaw and always-on bots.
## Post 3 (YouTube)
YOUTUBE IS NOW A 4TH SOURCE.
The skill searches YouTube, grabs view counts, and reads the actual transcripts. A 20-minute review has 10x the signal of one X post - now the skill reads it.
## Post 4 (Codex)
WORKS IN OPENAI CODEX.
Same skill, same engine, same four sources. Install to ~/.agents/skills/last30days and invoke with $last30days. Claude Code and Codex users get the same research.
## Post 5 (Example: Seedance 2.0 access)
Asked it how to access Seedance 2.0.
3 Reddit threads. 31 X posts. 20 YouTube videos (685K views, 4 transcripts read). 10 web pages. All four sources hit.
It found the real answer buried in Chinese YouTube tutorials: Little Skylark (Xiao Yun Que) - zero cost, no queue, no VPN. Just select Seedance 2.0 from the model dropdown. Also surfaced: Disney sent ByteDance a cease-and-desist over uncensored IP generation.
## Post 6 (Example: AI Generated Ads)
Asked it about AI generated ads.
12 Reddit threads. 29 X posts. 3 YouTube videos (83K views, 3 transcripts read). 30 web pages.
The finding that stuck: Svedka ran the first "primarily AI-generated" Super Bowl spot. Brand match: 7%. Industry norm: 63%. Meanwhile 86% of ad buyers are planning to use AI for video ads anyway. Cost is winning over quality.
## Post 7 (Example: Peter Steinberger)
Asked it about @steipete.
30 X posts. 5 YouTube videos (112K views). Found the Lex Fridman interview from 3 days ago.
Key reveal: OpenAI and Meta both made acquisition offers for OpenClaw. He said no. He's losing $10-20K/month maintaining it. "A fun project became a world project."
## Post 8 (Install)
Tell your @openclaw bot:
"Install the last30days skill from github.com/mvanhorn/last30days-skill"
That's it. One message.
## Post 9 (Close)
Thank you to @hutchins for pushing me to add YouTube and to @steipete whose summarize tool showed me how yt-dlp could power transcript extraction.
Try it: last30 [any topic]
github.com/mvanhorn/last30days-skill
PS: @steipete ClawHub login is broken right now so we can't publish the official skill there yet. Hoping for a fix soon.
---
## Standalone Posts
### Greg Isenberg best tips
Prompt: `last30 greg isenberg best tips`
1 Reddit thread. 6 X posts. 4 YouTube videos (all transcribed). 8 web pages.
His core playbook is ACP: Audience, Community, Product. Not the other way around.
Big thesis: "2026 is the GREATEST time to build a startup in 30 years." Boring industries, AI agents, rapid dev tools collapsing build time.
Most-watched: "Clawdbot Clearly Explained" (273K views), "Claude Code Built My $450K Marketing Campaign" (40K views), daily workflows with Kitze (83K views).
@gregisenberg @slashlast30days
### Lenny Rachitsky best learnings
Prompt: `last30 lenny rachitsky top learnings`
5 Reddit threads. 29 X posts. 20 YouTube videos (1.1M+ views, 5 transcribed). 30 web pages.
"Execution is no longer the bottleneck. Clarity is." That's the thesis running through everything @lennysan has been publishing lately.
He open-sourced 320 episode transcripts and the community went wild. 87 skills got built from them. Someone on Reddit distilled 86 discrete product skills from 100+ episodes.
Recent guest highlights: Sherwin Wu (OpenAI) says 95% of their engineers use Codex daily. Marc Andreessen: "This is as normal as it's going to be. It's going to be much weirder very soon." Dalton Caldwell (YC): "just don't die" and avoid tar pit ideas.
@lennysan @slashlast30days
---
## NOTES
- Thread leads with Open Claw + watchlist as the killer feature - pair with an always-on bot for automated research
- YouTube is the #2 hero - the stats ("685K views, 4 transcripts read") prove it works
- Codex compatibility is #3 - brief but shows cross-platform reach
- Example posts prove quality with verified results from real runs
- Consider screenshots of the actual output for each example post
- @steipete credit in close - he inspired YouTube (yt-dlp toolchain) and X search (Bird MIT code)
- @slashlast30days vs /last30days - which handle? Used /last30days above since it's the actual command
---
## RAW RESULTS (for reference / pulling quotes)
### Nano Banana Pro prompting (PROMPTING) - VERIFIED 2/15
Stats: 0 Reddit (timed out) | 32 X posts | 164 likes | 22 reposts | 5 YouTube videos | 98,539 views | 5 with transcripts | 10 web pages
Top voices: @KusoPhoto (106 likes), @TzqQaiser (35 likes) | Jake Dawson (15K views), AI Master (37K views)
Key findings:
- Structured JSON prompts are the meta - nested fields for character, scene, lighting beat plain prose. @TzqQaiser's viral post shows the format.
- Design brief > keyword stuffing - "The second I started writing prompts like a real design brief, everything changed" - Jake Dawson on YouTube
- 6-factor formula: Subject, Composition, Action, Setting, Style, then refine with camera/lighting - per Google's official blog
- ICS framework for infographics: Image type + Content + Style - leverages Nano Banana Pro's unique legible text rendering
- Scale logic for cinematic compositions - define size relationships and camera distance explicitly, per @Strength04_X
- Nano Banana Pro → video pipeline trending (generate image, animate with Kling 3.0 or Veo 3.1) - per @KusoPhoto
### Peter Steinberger / OpenClaw creator (GENERAL) - VERIFIED 2/15
Stats: 31 X posts | 0 Reddit (quiet) | YouTube timed out on 2 | 4 web pages
Top voices: @steipete | Lex Fridman podcast
Key findings:
- Lex Fridman podcast (Feb 12) went viral - "One of the most honest discussions I've seen"
- OpenAI and Meta made acquisition offers (conditional on keeping project open) - he declined
- Losing $10-20K/month maintaining OpenClaw, rejected crypto tokenization for funding
- 180K+ GitHub stars, 6,600 commits in 1 month - "A fun project became a world project"
- Also built: gogcli (Google Workspace CLI), summarize (URL/YouTube summarizer), bird (X/Twitter CLI)
- Pragmatic Engineer: "I ship code I don't read"
- Prediction: AI agents could dominate >60% of software economy by 2030
### Seedance 2.0 Prompting (PROMPTING) - VERIFIED 2/15
Stats: 21 Reddit threads | 33 X posts | 20 YouTube videos | 5 web pages
Top voices: @charliebcurran (61K+ likes) | r/AI_Agents, r/ChatGPT, r/PromptEngineering | AI Search (127K views), Theoretically Media (157K views), Dan Dingle (126K views)
Key findings:
- "Slow and continuous" is the #1 prompting secret - rough state transitions = worse outcomes, per r/AI_Agents
- Include timings in prompts (e.g., "0-3s: character walks, 3-6s: turns head") - per r/ChatGPT
- Image-to-video for consistency - start with a reference image, not text-only
- English works just as well as Chinese - per r/AI_India
- CapCut integration coming = "every 12 year old in America will have this superpower"
- Cost: ~$0.55/10s clips (~$3.30/min), Seedance 3.0 rumored at 1/8th price
- Prompt resources: GitHub repo of curated prompts, Prompt Director Pro (440 settings system)
- Top YouTube tutorials: "Seedance 2.0 crushes everything" (127K), "Claims the AI Video Throne" (157K), "ABUSING China's Crazy New Video AI" (126K)
### OpenClaw best use cases for business (RECOMMENDATIONS)
Stats: 35 Reddit threads | ~1,130 upvotes | ~566 comments | 23 X posts | ~24 likes | 20 YouTube videos | ~1,572,000 views | 5 with transcripts | 10 web pages
Top voices: @gio__aa (8 likes), @ericosiu, @artyomx | r/openclaw, r/clawdbot, r/LocalLLaMA
Key findings:
- Email & Inbox Automation - 8+ mentions. One user cleared 4,000+ emails in two days. 10-15 hours/week saved.
- Business Dashboards & Real-Time Reporting - 6+ mentions. @gio__aa: "Business dashboards are going to become one of the most popular use cases."
- Morning Briefings - 5+ mentions. Pulls from calendars, weather, emails, RSS, GitHub, Hacker News on a schedule.
- Content & SEO Pipelines - 5+ mentions. @ericosiu claims "$45k of pSEO work in 20 minutes."
- Full CRM & Business Operations - 4+ mentions. @artyomx runs a daycare business, legal cases, and family comms through it with 5 AI agents.
- Client Onboarding & Support - 4+ mentions. "70% of tickets handled autonomously."
- Competitive Monitoring & Scraping - 4+ mentions.
- Wrapper/Hosting SaaS - 3+ mentions. Building commercial wrappers around OpenClaw as a business.
Cautions: malware in a top-downloaded skill (236 upvotes on r/LocalLLaMA), $25-50/day token burn risk, hours of config for marginal savings.
### YouTube thumbnail tips (GENERAL)
Stats: 7 Reddit threads | 654 upvotes | 176 comments | 32 X posts | 110 likes | 53 reposts | 18 YouTube videos | 6,150,368 views | 5 with transcripts | 30 web pages
Top voices: @TeamYouTube, @thewindwolf64 | r/NewTubers, r/YouTubeThumbnailHub | Think Media (1.17M views), whirow (1.46M views)
Key findings:
- Simplicity is #1 - r/NewTubers post (654 upvotes) from someone who designed 346 thumbnails: one subject, one message, one second to understand. 3+ elements = ~23% lower CTR.
- Less text = more clicks - Under 4 words gets ~30% higher CTR. Mobile thumbnails shrink to 168x94px - text becomes unreadable.
- Faces still win but subtlety is trending - Faces boost CTR 20-30%, but exaggerated shock face is giving way to authentic expressions in 2026.
- "UnThumbnails" are a counter-trend - Nate Black (71K views): deliberately raw, less-designed thumbnails that stand out.
- AI tools changing the game - Nick Nimmin (90K views) showed free AI tools democratizing thumbnail creation.
- A/B test everything - YouTube's built-in thumbnail testing lets you test up to 3 versions per video.
### AI SaaS crash (NEWS)
Stats: 9 Reddit threads | 31 upvotes | 52 comments | 32 X posts | 39 likes | 2 reposts | 20 YouTube videos | 929,648 views | 5 with transcripts | 30 web pages
Top voices: @jasonlk (15 likes), @WarrenInTheBuff (11 likes), @xankriegor_ | r/SaaS, r/aiwars
Key findings:
- "SaaSpocalypse" - $285B wiped in a single day (Feb 3, 2026) after Anthropic launched Claude Cowork. Total losses exceeded $1T. Salesforce down 27% YTD, Oracle halved.
- @jasonlk: "The real inflection point wasn't January 2026. It was June 2024 - when Claude 3.5 Sonnet shipped." Public SaaS growth rates declined every quarter since 2021 peak.
- Seat-based pricing is the casualty - 10 AI agents replace 100 sales reps = no need for 100 Salesforce seats. $470B+ hyperscaler AI spend coming from enterprise software budgets.
- Not everyone buying the doom - Jensen Huang called it "the most illogical thing in the world." BofA called selloff irrational.
- Indian IT hit especially hard - biggest sell-off since 2020.
### Seedance 2.0 access (GENERAL) - VERIFIED 2/15, ALL 4 SOURCES
Stats: 3 Reddit threads | 114 upvotes | 183 comments | 31 X posts | 191 likes | 13 reposts | 20 YouTube videos | 685,297 views | 4 with transcripts | 10 web pages
Top voices: @markgadala (116 likes), @OrctonAI, @nemovideoai | Theoretically Media (158K views) | r/AIHubSpace
Key findings:
- Little Skylark (Xiao Yun Que) = best free method - zero cost, no queue, manually select Seedance 2.0 from model dropdown, per YouTube tutorials
- Jimeng (Dreamina) - 1 RMB trial (~$0.14), ~260 daily free credits, but severe congestion with hours-long waits for free users
- Doubao App - 10 free video gens/day, requires joining Feishu/Lark group and submitting UID (1-2 day wait)
- Feb 24 = global unlock - Dreamina + CapCut + API access through BytePlus
- IP controversy exploding - @markgadala's "fully uncensored Seedance 2" post (116 likes) went viral, Disney sent C&D to ByteDance, SAG-AFTRA slammed "blatant infringement" over AI Tom Cruise/Brad Pitt fight videos
- Third-party race - NemoVideoAI, ChatCut, RecCloud, Morph Studio all competing to be the English-language access point
- Quality consensus: "crushes everything" (AI Search, 128K views), "claims the AI video throne" (Theoretically Media, 158K views)
### AI Generated Ads (GENERAL) - VERIFIED 2/15, ALL 4 SOURCES
Stats: 12 Reddit threads | 5 upvotes | 15 comments | 29 X posts | 101 likes | 3 reposts | 3 YouTube videos | 82,896 views | 3 with transcripts | 30 web pages
Top voices: @CaptainMcKlide (77 likes), @ugcbykaytelynn | r/editors, r/AI_UGC_Marketing, BERNTH (39K views)
Key findings:
- Super Bowl 2026 was the watershed - 23% of ads (15/66) featured AI, reception "sharply negative," nearly 50% of social mentions critical
- Svedka ran the first "primarily AI-generated" national Super Bowl spot - brand match of just 7% vs 63% alcohol industry norm
- Massive perception gap - 82% of ad execs think Gen Z feels positive about AI ads, but only 45% of consumers do (IAB). Gen Z most hostile at 39% negative.
- AI UGC booming in e-commerce - r/AI_UGC_Marketing active hub, tools: Creatify, MakeUGC, ArcAds targeting dropshippers
- Quality still low - r/dropshipping: "the hand flip and rubbing on the face looks fake"
- @ugcbykaytelynn warns "AI generated ads RUIN your brand's image"
- Cost winning over quality - 86% of ad buyers using or planning gen AI for video ads, cost efficiency #1 driver (64%), per IAB
- BERNTH on YouTube bought AI-generated guitar product ads, documented absurdity - four-fingered hands, instruments don't match listings (39K views)
- Trust erosion spreading - people now question whether ANY media is real, even billboards, per @N0rbertas
### last30days skill (META/GENERAL) - VERIFIED 2/15
Stats: 0 Reddit | 30 X posts | 1,371 likes | 107 reposts | 5 YouTube videos | 112,082 views | 3 with transcripts | 10 web pages
Top voices: @gregisenberg (1,290 likes), @mvanhorn (34 likes) | Alejandro AO (39K views), Greg Isenberg (28K views)
Key findings:
- @gregisenberg's post (1,290 likes, 106 RT) + YouTube video (28K views) "The Claude Code Skill My Smartest Friends Use" was the breakout moment
- v2 feedback loop active - @trevin flagged OpenAI web_search not finding niche Reddit posts, suggested Brave API. @jonthebeef submitted PR for --days flag.
- People building on top - @tjarkoleifer created "re-skill" meta skill, @rajachirravuri recommends it as part of a PM stack
- Coverage: Alejandro AO crash course (39K views, 1,049 likes), Jason Calacanis on This Week in Startups (24K views)
- 1.5K GitHub stars, listed on skills.sh and Smithery
- Grok itself correctly attributed the skill when asked about ithah
+328
View File
@@ -0,0 +1,328 @@
# fix: Enforce Strict 30-Day Date Filtering
## Overview
The `/last30days` skill is returning content older than 30 days, violating its core promise. Analysis shows:
- **Reddit**: Only 40% of results within 30 days (9/15 were older, some from 2022!)
- **X**: 100% within 30 days (working correctly)
- **WebSearch**: 90% had unknown dates (can't verify freshness)
## Problem Statement
The skill's name is "last30days" - users expect ONLY content from the last 30 days. Currently:
1. **Reddit search prompt** says "prefer recent threads, but include older relevant ones if recent ones are scarce" - this is too permissive
2. **X search prompt** explicitly includes `from_date` and `to_date` - this is why it works
3. **WebSearch** returns pages without publication dates - we can't verify they're recent
4. **Scoring penalties** (-10 for low date confidence) don't prevent old content from appearing
## Proposed Solution
### Strategy: "Hard Filter, Not Soft Penalty"
Instead of penalizing old content, **exclude it entirely**. If it's not from the last 30 days, it shouldn't appear.
| Source | Current Behavior | New Behavior |
|--------|------------------|--------------|
| Reddit | Weak "prefer recent" | Explicit date range + hard filter |
| X | Explicit date range (working) | No change needed |
| WebSearch | No date awareness | Require recent markers OR exclude |
## Technical Approach
### Phase 1: Fix Reddit Date Filtering
**File: `scripts/lib/openai_reddit.py`**
Current prompt (line 33):
```
Find {min_items}-{max_items} relevant Reddit discussion threads.
Prefer recent threads, but include older relevant ones if recent ones are scarce.
```
New prompt:
```
Find {min_items}-{max_items} relevant Reddit discussion threads from {from_date} to {to_date}.
CRITICAL: Only include threads posted within the last 30 days (after {from_date}).
Do NOT include threads older than {from_date}, even if they seem relevant.
If you cannot find enough recent threads, return fewer results rather than older ones.
```
**Changes needed:**
1. Add `from_date` and `to_date` parameters to `search_reddit()` function
2. Inject dates into `REDDIT_SEARCH_PROMPT` like X does
3. Update caller in `last30days.py` to pass dates
### Phase 2: Add Hard Date Filtering (Post-Processing)
**File: `scripts/lib/normalize.py`**
Add a filter step that DROPS items with dates before `from_date`:
```python
def filter_by_date_range(
items: List[Union[RedditItem, XItem, WebSearchItem]],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List:
"""Hard filter: Remove items outside the date range.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
if item.date > to_date:
continue # DROP - future date (likely parsing error)
result.append(item)
return result
```
### Phase 3: WebSearch Date Intelligence
WebSearch CAN find recent content - Medium posts have dates, GitHub has commit timestamps, news sites have publication dates. We should **extract and prioritize** these signals.
**Strategy: "Date Detective"**
1. **Extract dates from URLs**: Many sites embed dates in URLs
- Medium: `medium.com/@author/title-abc123` (no date) vs news sites
- GitHub: Look for commit dates, release dates in snippets
- News: `/2026/01/24/article-title`
- Blogs: `/blog/2026/01/title`
2. **Extract dates from snippets**: Look for date markers
- "January 24, 2026", "Jan 2026", "yesterday", "this week"
- "Published:", "Posted:", "Updated:"
- Relative markers: "2 days ago", "last week"
3. **Prioritize results with verifiable dates**:
- Results with recent dates (within 30 days): Full score
- Results with old dates: EXCLUDE
- Results with no date signals: Heavy penalty (-20) but keep as supplementary
**File: `scripts/lib/websearch.py`**
Add date extraction functions:
```python
import re
from datetime import datetime, timedelta
# Patterns for date extraction
URL_DATE_PATTERNS = [
r'/(\d{4})/(\d{2})/(\d{2})/', # /2026/01/24/
r'/(\d{4})-(\d{2})-(\d{2})/', # /2026-01-24/
r'/(\d{4})(\d{2})(\d{2})/', # /20260124/
]
SNIPPET_DATE_PATTERNS = [
r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{1,2}),? (\d{4})',
r'(\d{1,2}) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{4})',
r'(\d{4})-(\d{2})-(\d{2})',
r'Published:?\s*(\d{4}-\d{2}-\d{2})',
r'(\d{1,2}) (days?|hours?|minutes?) ago', # Relative dates
]
def extract_date_from_url(url: str) -> Optional[str]:
"""Try to extract a date from URL path."""
for pattern in URL_DATE_PATTERNS:
match = re.search(pattern, url)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_from_snippet(snippet: str) -> Optional[str]:
"""Try to extract a date from text snippet."""
for pattern in SNIPPET_DATE_PATTERNS:
match = re.search(pattern, snippet, re.IGNORECASE)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_signals(url: str, snippet: str, title: str) -> tuple[Optional[str], str]:
"""Extract date from any available signal.
Returns: (date_string, confidence)
- date from URL: 'high' confidence
- date from snippet: 'med' confidence
- no date found: None, 'low' confidence
"""
# Try URL first (most reliable)
url_date = extract_date_from_url(url)
if url_date:
return url_date, 'high'
# Try snippet
snippet_date = extract_date_from_snippet(snippet)
if snippet_date:
return snippet_date, 'med'
# Try title
title_date = extract_date_from_snippet(title)
if title_date:
return title_date, 'med'
return None, 'low'
```
**Update WebSearch parsing to use date extraction:**
```python
def parse_websearch_results(results, topic, from_date, to_date):
items = []
for result in results:
url = result.get('url', '')
snippet = result.get('snippet', '')
title = result.get('title', '')
# Extract date signals
extracted_date, confidence = extract_date_signals(url, snippet, title)
# Hard filter: if we found a date and it's too old, skip
if extracted_date and extracted_date < from_date:
continue # DROP - verified old content
item = {
'date': extracted_date,
'date_confidence': confidence,
...
}
items.append(item)
return items
```
**File: `scripts/lib/score.py`**
Update WebSearch scoring to reward date-verified results:
```python
# WebSearch date confidence adjustments
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date (was 10)
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date
def score_websearch_items(items):
for item in items:
...
# Date confidence adjustments
if item.date_confidence == 'high':
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified dates
elif item.date_confidence == 'low':
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
...
```
**Result**: WebSearch results with verifiable recent dates rank well. Results with no dates are heavily penalized but still appear as supplementary context. Old verified content is excluded entirely.
### Phase 4: Update Statistics Display
Only count Reddit and X in "from the last 30 days" claim. WebSearch should be clearly labeled as supplementary.
## Acceptance Criteria
### Functional Requirements
- [x] Reddit search prompt includes explicit `from_date` and `to_date`
- [x] Items with dates before `from_date` are EXCLUDED, not just penalized
- [x] X search continues working (no regression)
- [x] WebSearch extracts dates from URLs (e.g., `/2026/01/24/`)
- [x] WebSearch extracts dates from snippets (e.g., "January 24, 2026")
- [x] WebSearch with verified recent dates gets +10 bonus
- [x] WebSearch with no date signals gets -20 penalty (but still appears)
- [x] WebSearch with verified OLD dates is EXCLUDED
### Non-Functional Requirements
- [ ] No increase in API latency
- [ ] Graceful handling when few recent results exist (return fewer, not older)
- [ ] Clear user messaging when results are limited due to strict filtering
### Quality Gates
- [ ] Test: Reddit search returns 0% results older than 30 days
- [ ] Test: X search continues to return 100% recent results
- [ ] Test: WebSearch is clearly differentiated in output
- [ ] Test: Edge case - topic with no recent content shows helpful message
## Implementation Order
1. **Phase 1**: Fix Reddit prompt (highest impact, simple change)
2. **Phase 2**: Add hard date filter in normalize.py (safety net)
3. **Phase 3**: Add WebSearch date extraction (URL + snippet parsing)
4. **Phase 4**: Update WebSearch scoring (bonus for verified, heavy penalty for unknown)
5. **Phase 5**: Update output display to show date confidence
## Testing Plan
### Before/After Test
Run same query before and after fix:
```
/last30days remotion launch videos
```
**Expected Before:**
- Reddit: 40% within 30 days
**Expected After:**
- Reddit: 100% within 30 days (or fewer results if not enough recent content)
### Edge Case Tests
| Scenario | Expected Behavior |
|----------|-------------------|
| Topic with no recent content | Return 0 results + helpful message |
| Topic with 5 recent results | Return 5 results (not pad with old ones) |
| Mixed old/new results | Only return new ones |
### WebSearch Date Extraction Tests
| URL/Snippet | Expected Date | Confidence |
|-------------|---------------|------------|
| `medium.com/blog/2026/01/15/title` | 2026-01-15 | high |
| `github.com/repo` + "Released Jan 20, 2026" | 2026-01-20 | med |
| `docs.example.com/guide` (no date signals) | None | low |
| `news.site.com/2024/05/old-article` | 2024-05-XX | EXCLUDE (too old) |
| Snippet: "Updated 3 days ago" | calculated | med |
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Fewer results for niche topics | High | Medium | Explain why in output |
| User confusion about reduced results | Medium | Low | Clear messaging |
| Date parsing errors exclude valid content | Low | Medium | Keep items with unknown dates, just label clearly |
## References
### Internal References
- Reddit search: `scripts/lib/openai_reddit.py:25-63`
- X search (working example): `scripts/lib/xai_x.py:26-55`
- Date confidence: `scripts/lib/dates.py:62-90`
- Scoring penalties: `scripts/lib/score.py:149-153`
- Normalization: `scripts/lib/normalize.py:49,99`
### External References
- OpenAI Responses API lacks native date filtering
- Must rely on prompt engineering + post-processing
+73
View File
@@ -0,0 +1,73 @@
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago.
## Three Headline Features
**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search.
**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies.
## Real Results (verified Feb 15)
| Topic | Reddit | X | YouTube | Web |
|-------|--------|---|---------|-----|
| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages |
| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages |
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages |
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages |
| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages |
## What's New
### Added
- Open-class skill with watchlist, briefing, and history modes
- YouTube search + transcript extraction via yt-dlp
- OpenAI Codex CLI compatibility
- Bundled Twitter/X search (vendored Bird GraphQL, MIT)
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro)
- `--diagnose` flag for source status checking
- `--store` flag for SQLite accumulation
- Conversational first-run experience (NUX)
### Changed
- Two-phase search architecture (entity-aware drill-down)
- Reddit JSON enrichment for real engagement metrics
- Smarter query construction with auto-retry on 0 results
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
- `--days=N` configurable lookback (thanks @jonthebeef)
### Fixed
- YouTube/Reddit timeout resilience
- Reddit 429 rate limit fail-fast
- Eager import crash in Codex environments
- X search returning 0 results on popular topics
- Windows Unicode crash (thanks @JosephOIbrahim)
## New Contributors
- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
## Credits
- [@steipete](https://github.com/steipete) -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
- [@galligan](https://github.com/galligan) -- Marketplace plugin inspiration
- [@hutchins](https://x.com/hutchins) -- Pushed for YouTube feature
## Install
```bash
# Claude Code
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Codex CLI
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
```
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
+260
View File
@@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Morning briefing generator for last30days.
Synthesizes accumulated findings into formatted briefings.
The Python script collects the data; the agent (via SKILL.md) does the
beautiful synthesis. This script provides the structured data.
Usage:
python3 briefing.py generate # Daily briefing data
python3 briefing.py generate --weekly # Weekly digest data
python3 briefing.py show [--date DATE] # Show saved briefing
"""
import argparse
import json
import sys
from datetime import datetime, timedelta
from pathlib import Path
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
import store
BRIEFS_DIR = Path.home() / ".local" / "share" / "last30days" / "briefs"
def generate_daily(since: str = None) -> dict:
"""Generate daily briefing data.
Returns structured data for the agent to synthesize into a beautiful briefing.
"""
store.init_db()
topics = store.list_topics()
if not topics:
return {
"status": "no_topics",
"message": "No watchlist topics yet. Add one with: last30days watch add \"your topic\"",
}
enabled = [t for t in topics if t["enabled"]]
if not enabled:
return {
"status": "no_enabled",
"message": "All topics are paused. Enable a topic to generate briefings.",
}
# Default: findings since yesterday
if not since:
since = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
briefing_topics = []
total_new = 0
for topic in enabled:
findings = store.get_new_findings(topic["id"], since)
last_run = topic.get("last_run")
last_status = topic.get("last_status", "unknown")
# Calculate staleness
stale = False
hours_ago = None
if last_run:
try:
run_dt = datetime.fromisoformat(last_run.replace("Z", "+00:00"))
hours_ago = (datetime.now() - run_dt.replace(tzinfo=None)).total_seconds() / 3600
stale = hours_ago > 36 # Stale if > 36 hours
except (ValueError, TypeError):
stale = True
topic_data = {
"name": topic["name"],
"findings": findings,
"new_count": len(findings),
"last_run": last_run,
"last_status": last_status,
"stale": stale,
"hours_ago": round(hours_ago, 1) if hours_ago else None,
}
# Extract top finding by engagement
if findings:
top = max(findings, key=lambda f: f.get("engagement_score", 0))
topic_data["top_finding"] = {
"title": top.get("source_title", ""),
"source": top.get("source", ""),
"author": top.get("author", ""),
"engagement": top.get("engagement_score", 0),
"content": top.get("content", "")[:300],
}
briefing_topics.append(topic_data)
total_new += len(findings)
# Cost info
daily_cost = store.get_daily_cost()
budget = float(store.get_setting("daily_budget", "5.00"))
# Find the single top finding across all topics (for TL;DR)
all_findings = []
for t in briefing_topics:
for f in t["findings"]:
f["_topic"] = t["name"]
all_findings.append(f)
top_overall = None
if all_findings:
top_overall = max(all_findings, key=lambda f: f.get("engagement_score", 0))
result = {
"status": "ok",
"date": datetime.now().strftime("%Y-%m-%d"),
"since": since,
"topics": briefing_topics,
"total_new": total_new,
"total_topics": len(briefing_topics),
"top_finding": {
"title": top_overall.get("source_title", ""),
"topic": top_overall.get("_topic", ""),
"engagement": top_overall.get("engagement_score", 0),
} if top_overall else None,
"cost": {
"daily": daily_cost,
"budget": budget,
},
"failed_topics": [
t["name"] for t in briefing_topics if t["last_status"] == "failed"
],
}
# Save briefing data
_save_briefing(result)
return result
def generate_weekly() -> dict:
"""Generate weekly digest data with trend analysis."""
store.init_db()
week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
two_weeks_ago = (datetime.now() - timedelta(days=14)).strftime("%Y-%m-%d")
topics = store.list_topics()
if not topics:
return {"status": "no_topics", "message": "No watchlist topics."}
weekly_topics = []
for topic in topics:
if not topic["enabled"]:
continue
# This week's findings
this_week = store.get_new_findings(topic["id"], week_ago)
# Last week's findings (for comparison)
conn = store._connect()
try:
last_week_rows = conn.execute(
"""SELECT * FROM findings
WHERE topic_id = ? AND first_seen >= ? AND first_seen < ? AND dismissed = 0
ORDER BY engagement_score DESC""",
(topic["id"], two_weeks_ago, week_ago),
).fetchall()
last_week = [dict(r) for r in last_week_rows]
finally:
conn.close()
this_engagement = sum(f.get("engagement_score", 0) for f in this_week)
last_engagement = sum(f.get("engagement_score", 0) for f in last_week)
# Trend calculation
if last_engagement > 0:
engagement_change = ((this_engagement - last_engagement) / last_engagement) * 100
else:
engagement_change = 100 if this_engagement > 0 else 0
weekly_topics.append({
"name": topic["name"],
"this_week_count": len(this_week),
"last_week_count": len(last_week),
"this_week_engagement": this_engagement,
"last_week_engagement": last_engagement,
"engagement_change_pct": round(engagement_change, 1),
"top_findings": this_week[:5], # Top 5 by engagement (already sorted)
})
result = {
"status": "ok",
"type": "weekly",
"week_of": week_ago,
"topics": weekly_topics,
}
_save_briefing(result, suffix="-weekly")
return result
def show_briefing(date: str = None) -> dict:
"""Load a saved briefing by date."""
if not date:
date = datetime.now().strftime("%Y-%m-%d")
path = BRIEFS_DIR / f"{date}.json"
if not path.exists():
# Try weekly
path = BRIEFS_DIR / f"{date}-weekly.json"
if not path.exists():
return {"status": "not_found", "message": f"No briefing found for {date}."}
with open(path) as f:
return json.load(f)
def _save_briefing(data: dict, suffix: str = ""):
"""Save briefing data to local archive."""
BRIEFS_DIR.mkdir(parents=True, exist_ok=True)
date = datetime.now().strftime("%Y-%m-%d")
path = BRIEFS_DIR / f"{date}{suffix}.json"
with open(path, "w") as f:
json.dump(data, f, indent=2, default=str)
def main():
parser = argparse.ArgumentParser(description="Generate last30days briefings")
sub = parser.add_subparsers(dest="command")
# generate
g = sub.add_parser("generate", help="Generate a briefing")
g.add_argument("--weekly", action="store_true", help="Weekly digest")
g.add_argument("--since", help="Findings since date (YYYY-MM-DD)")
# show
s = sub.add_parser("show", help="Show a saved briefing")
s.add_argument("--date", help="Date (YYYY-MM-DD, default: today)")
args = parser.parse_args()
if args.command == "generate":
if args.weekly:
result = generate_weekly()
else:
result = generate_daily(since=args.since)
print(json.dumps(result, indent=2, default=str))
elif args.command == "show":
result = show_briefing(date=args.date)
print(json.dumps(result, indent=2, default=str))
else:
parser.print_help()
sys.exit(1)
if __name__ == "__main__":
main()
+883 -85
View File
File diff suppressed because it is too large Load Diff
+436
View File
@@ -0,0 +1,436 @@
"""Bird X search client - vendored Twitter GraphQL search for /last30days v2.1.
Uses a vendored subset of @steipete/bird v0.8.0 (MIT License) to search X
via Twitter's GraphQL API. No external `bird` CLI binary needed - just Node.js 22+.
"""
import json
import os
import signal
import shutil
import subprocess
import sys
from pathlib import Path
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
# Path to the vendored bird-search wrapper
_BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[Bird] {msg}\n")
sys.stderr.flush()
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search.
X search is literal keyword AND matching — all words must appear.
Aggressively strip question/meta/research words to keep only the
core product/concept name (2-3 words max).
"""
text = topic.lower().strip()
# Phase 1: Strip multi-word prefixes (longest first)
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
break
# Phase 2: Strip multi-word suffixes
suffixes = [
'best practices', 'use cases', 'prompt techniques',
'prompting techniques', 'prompting tips',
]
for s in suffixes:
if text.endswith(' ' + s):
text = text[:-len(s)].strip()
break
# Phase 3: Filter individual noise words
_noise = {
# Question/filler words
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'and', 'or',
'of', 'in', 'on', 'for', 'with', 'about', 'to',
'people', 'saying', 'think', 'said', 'lately',
# Research/meta descriptors
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'practices', 'features', 'guide', 'tutorial',
'recommendations', 'advice', 'review', 'reviews',
'usecases', 'examples', 'comparison', 'versus', 'vs',
# Prompting meta words
'prompt', 'prompts', 'prompting', 'techniques', 'tips',
'tricks', 'methods', 'strategies', 'approaches',
# Action words
'using', 'uses', 'use',
}
words = text.split()
result = [w for w in words if w not in _noise]
return ' '.join(result[:3]) or topic.lower().strip() # Max 3 words
def is_bird_installed() -> bool:
"""Check if vendored Bird search module is available.
Returns:
True if bird-search.mjs exists and Node.js 22+ is in PATH.
"""
if not _BIRD_SEARCH_MJS.exists():
return False
return shutil.which("node") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if X credentials are available (env vars or browser cookies).
Returns:
Auth source string if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
try:
result = subprocess.run(
["node", str(_BIRD_SEARCH_MJS), "--whoami"],
capture_output=True,
text=True,
timeout=15,
)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip().split('\n')[0]
return None
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
return None
def check_npm_available() -> bool:
"""Check if npm is available (kept for API compatibility).
Returns:
True if 'npm' command is available in PATH, False otherwise.
"""
return shutil.which("npm") is not None
def install_bird() -> Tuple[bool, str]:
"""No-op - Bird search is vendored in v2.1, no installation needed.
Returns:
Tuple of (success, message).
"""
if is_bird_installed():
return True, "Bird search is bundled with /last30days v2.1 - no installation needed."
if not shutil.which("node"):
return False, "Node.js 22+ is required for X search. Install Node.js first."
return False, f"Vendored bird-search.mjs not found at {_BIRD_SEARCH_MJS}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird search status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
auth_source = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": auth_source is not None,
"username": auth_source, # Now returns auth source (e.g., "Safari", "env AUTH_TOKEN")
"can_install": True, # Always vendored in v2.1
}
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds
Returns:
Raw Bird JSON response or error dict.
"""
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count),
"--json",
]
# Use process groups for clean cleanup on timeout/kill
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
preexec_fn=preexec,
)
# Register for cleanup tracking (if available)
try:
from last30days import register_child_pid, unregister_child_pid
register_child_pid(proc.pid)
except ImportError:
pass
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
# Kill the entire process group
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
return {"error": f"Search timed out after {timeout}s", "items": []}
finally:
try:
from last30days import unregister_child_pid
unregister_child_pid(proc.pid)
except (ImportError, Exception):
pass
if proc.returncode != 0:
error = stderr.strip() if stderr else "Bird search failed"
return {"error": error, "items": []}
output = stdout.strip() if stdout else ""
if not output:
return {"items": []}
return json.loads(output)
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI with automatic retry on 0 results.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD) - unused but kept for API compatibility
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
# Extract core subject - X search is literal, not semantic
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
_log(f"Searching: {query}")
response = _run_bird_search(query, count, timeout)
# Check if we got results
items = parse_bird_response(response)
# Retry with fewer keywords if 0 results and query has 3+ words
core_words = core_topic.split()
if not items and len(core_words) > 2:
shorter = ' '.join(core_words[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
response = _run_bird_search(query, count, timeout)
return response
def search_handles(
handles: List[str],
topic: str,
from_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific X handles for topic-related content.
Runs targeted Bird searches using `from:handle topic` syntax.
Used in Phase 2 supplemental search after entity extraction.
Args:
handles: List of X handles to search (without @)
topic: Search topic (core subject, not full verbose query)
from_date: Start date (YYYY-MM-DD)
count_per: Results to request per handle
Returns:
List of raw item dicts (same format as parse_bird_response output).
"""
all_items = []
core_topic = _extract_core_subject(topic)
for handle in handles:
handle = handle.lstrip("@")
query = f"from:{handle} {core_topic} since:{from_date}"
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count_per),
"--json",
]
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
preexec_fn=preexec,
)
try:
stdout, stderr = proc.communicate(timeout=15)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
_log(f"Handle search timed out for @{handle}")
continue
if proc.returncode != 0:
_log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
continue
output = (stdout or "").strip()
if not output:
continue
response = json.loads(output)
items = parse_bird_response(response)
all_items.extend(items)
except json.JSONDecodeError:
_log(f"Invalid JSON from handle search for @{handle}")
except Exception as e:
_log(f"Handle search error for @{handle}: {e}")
return all_items
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
# Try different field structures Bird might use
author = tweet.get("author", {}) or tweet.get("user", {})
screen_name = author.get("username") or author.get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at/createdAt (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("createdAt") or tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first (e.g., "2026-02-03T22:33:32Z")
# Check for ISO date separator, not just "T" (which appears in "Tue")
if len(created_at) > 10 and created_at[10] == "T":
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info (Bird uses author.username, older format uses user.screen_name)
author = tweet.get("author", {}) or tweet.get("user", {})
author_handle = author.get("username") or author.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict (Bird uses camelCase: likeCount, retweetCount, etc.)
engagement = {
"likes": tweet.get("likeCount") or tweet.get("like_count") or tweet.get("favorite_count"),
"reposts": tweet.get("retweetCount") or tweet.get("retweet_count"),
"replies": tweet.get("replyCount") or tweet.get("reply_count"),
"quotes": tweet.get("quoteCount") or tweet.get("quote_count"),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
}
items.append(item)
return items
+213
View File
@@ -0,0 +1,213 @@
"""Brave Search web search for last30days skill.
Uses the Brave Search API as a fallback web search backend.
Simple, cheap (free tier: 2,000 queries/month), widely available.
API docs: https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
"""
import html
import re
import sys
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
from urllib.parse import urlencode, urlparse
from . import http
ENDPOINT = "https://api.search.brave.com/res/v1/web/search"
# Freshness codes: pd=24h, pw=7d, pm=31d
FRESHNESS_MAP = {1: "pd", 7: "pw", 31: "pm"}
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Brave Search API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: Brave Search API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
count = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
# Calculate days for freshness filter
days = _days_between(from_date, to_date)
freshness = _brave_freshness(days)
params = {
"q": topic,
"result_filter": "web,news",
"count": count,
"safesearch": "strict",
"text_decorations": 0,
"spellcheck": 0,
}
if freshness:
params["freshness"] = freshness
url = f"{ENDPOINT}?{urlencode(params)}"
sys.stderr.write(f"[Web] Searching Brave for: {topic}\n")
sys.stderr.flush()
response = http.request(
"GET",
url,
headers={"X-Subscription-Token": api_key},
timeout=15,
)
return _normalize_results(response, from_date, to_date)
def _days_between(from_date: str, to_date: str) -> int:
"""Calculate days between two YYYY-MM-DD dates."""
try:
d1 = datetime.strptime(from_date, "%Y-%m-%d")
d2 = datetime.strptime(to_date, "%Y-%m-%d")
return max(1, (d2 - d1).days)
except (ValueError, TypeError):
return 30
def _brave_freshness(days: Optional[int]) -> Optional[str]:
"""Convert days to Brave freshness parameter.
Uses canned codes for <=31d, explicit date range for longer periods.
"""
if days is None:
return None
code = next((v for d, v in sorted(FRESHNESS_MAP.items()) if days <= d), None)
if code:
return code
start = (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d")
end = datetime.now(timezone.utc).strftime("%Y-%m-%d")
return f"{start}to{end}"
def _normalize_results(
response: Dict[str, Any],
from_date: str,
to_date: str,
) -> List[Dict[str, Any]]:
"""Convert Brave Search response to websearch item schema.
Merges news + web results, cleans HTML entities, filters excluded domains.
"""
items = []
# Merge news results (tend to be more recent) with web results
raw_results = (
response.get("news", {}).get("results", []) +
response.get("web", {}).get("results", [])
)
for i, result in enumerate(raw_results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = _clean_html(str(result.get("title", "")).strip())
snippet = _clean_html(str(result.get("description", "")).strip())
if not title and not snippet:
continue
# Parse date from Brave's 'age' field or 'page_age'
date = _parse_brave_date(result.get("age"), result.get("page_age"))
date_confidence = "med" if date else "low"
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": snippet[:500],
"date": date,
"date_confidence": date_confidence,
"relevance": 0.6, # Brave doesn't provide relevance scores
"why_relevant": "",
})
sys.stderr.write(f"[Web] Brave: {len(items)} results\n")
sys.stderr.flush()
return items
def _clean_html(text: str) -> str:
"""Remove HTML tags and decode entities."""
text = re.sub(r"<[^>]*>", "", text)
text = html.unescape(text)
return text
def _parse_brave_date(age: Optional[str], page_age: Optional[str]) -> Optional[str]:
"""Parse Brave's age/page_age fields to YYYY-MM-DD.
Brave returns dates like "3 hours ago", "2 days ago", "January 24, 2026".
"""
text = age or page_age
if not text:
return None
text_lower = text.lower().strip()
now = datetime.now()
# "X hours ago" -> today
if re.search(r'\d+\s*hours?\s*ago', text_lower):
return now.strftime("%Y-%m-%d")
# "X days ago"
match = re.search(r'(\d+)\s*days?\s*ago', text_lower)
if match:
days = int(match.group(1))
if days <= 60:
return (now - timedelta(days=days)).strftime("%Y-%m-%d")
# "X weeks ago"
match = re.search(r'(\d+)\s*weeks?\s*ago', text_lower)
if match:
weeks = int(match.group(1))
return (now - timedelta(weeks=weeks)).strftime("%Y-%m-%d")
# ISO format: 2026-01-24T...
match = re.search(r'(\d{4}-\d{2}-\d{2})', text)
if match:
return match.group(1)
return None
+16 -3
View File
@@ -3,6 +3,7 @@
import hashlib import hashlib
import json import json
import os import os
import tempfile
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
from typing import Any, Optional from typing import Any, Optional
@@ -10,10 +11,22 @@ from typing import Any, Optional
CACHE_DIR = Path.home() / ".cache" / "last30days" CACHE_DIR = Path.home() / ".cache" / "last30days"
DEFAULT_TTL_HOURS = 24 DEFAULT_TTL_HOURS = 24
MODEL_CACHE_TTL_DAYS = 7 MODEL_CACHE_TTL_DAYS = 7
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
def ensure_cache_dir(): def ensure_cache_dir():
"""Ensure cache directory exists.""" """Ensure cache directory exists. Supports env override and sandbox fallback."""
global CACHE_DIR, MODEL_CACHE_FILE
env_dir = os.environ.get("LAST30DAYS_CACHE_DIR")
if env_dir:
CACHE_DIR = Path(env_dir)
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
try:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
except PermissionError:
CACHE_DIR = Path(tempfile.gettempdir()) / "last30days" / "cache"
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
CACHE_DIR.mkdir(parents=True, exist_ok=True) CACHE_DIR.mkdir(parents=True, exist_ok=True)
@@ -112,8 +125,8 @@ def clear_cache():
pass pass
# Model selection cache (longer TTL) # Model selection cache (longer TTL) — MODEL_CACHE_FILE is set at module level
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json" # and updated by ensure_cache_dir() if env override or fallback is needed.
def load_model_cache() -> dict: def load_model_cache() -> dict:
+11 -1
View File
@@ -36,10 +36,12 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
return intersection / union if union > 0 else 0.0 return intersection / union if union > 0 else 0.0
def get_item_text(item: Union[schema.RedditItem, schema.XItem]) -> str: def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem]) -> str:
"""Get comparable text from an item.""" """Get comparable text from an item."""
if isinstance(item, schema.RedditItem): if isinstance(item, schema.RedditItem):
return item.title return item.title
elif isinstance(item, schema.YouTubeItem):
return f"{item.title} {item.channel_name}"
else: else:
return item.text return item.text
@@ -118,3 +120,11 @@ def dedupe_x(
) -> List[schema.XItem]: ) -> List[schema.XItem]:
"""Dedupe X items.""" """Dedupe X items."""
return dedupe_items(items, threshold) return dedupe_items(items, threshold)
def dedupe_youtube(
items: List[schema.YouTubeItem],
threshold: float = 0.7,
) -> List[schema.YouTubeItem]:
"""Dedupe YouTube items."""
return dedupe_items(items, threshold)
+127
View File
@@ -0,0 +1,127 @@
"""Entity extraction from Phase 1 search results for supplemental searches."""
import re
from collections import Counter
from typing import Any, Dict, List
# Handles that appear too frequently to be useful for targeted search.
# These are generic/platform accounts, not topic-specific voices.
GENERIC_HANDLES = {
"elonmusk", "openai", "google", "microsoft", "apple", "meta",
"github", "youtube", "x", "twitter", "reddit", "wikipedia",
"nytimes", "washingtonpost", "cnn", "bbc", "reuters",
"verified", "jack", "sundarpichai",
}
def extract_entities(
reddit_items: List[Dict[str, Any]],
x_items: List[Dict[str, Any]],
max_handles: int = 5,
max_hashtags: int = 3,
max_subreddits: int = 5,
) -> Dict[str, List[str]]:
"""Extract key entities from Phase 1 results for supplemental searches.
Parses X results for @handles and #hashtags, Reddit results for subreddit
names and cross-referenced communities.
Args:
reddit_items: Raw Reddit item dicts from Phase 1
x_items: Raw X item dicts from Phase 1
max_handles: Maximum handles to return
max_hashtags: Maximum hashtags to return
max_subreddits: Maximum subreddits to return
Returns:
Dict with keys: x_handles, x_hashtags, reddit_subreddits
"""
handles = _extract_x_handles(x_items)
hashtags = _extract_x_hashtags(x_items)
subreddits = _extract_subreddits(reddit_items)
return {
"x_handles": handles[:max_handles],
"x_hashtags": hashtags[:max_hashtags],
"reddit_subreddits": subreddits[:max_subreddits],
}
def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank @handles from X results.
Sources handles from:
1. author_handle field (who posted)
2. @mentions in post text (who they're talking about/to)
Returns handles ranked by frequency, filtered for generic accounts.
"""
handle_counts = Counter()
for item in x_items:
# Author handle
author = item.get("author_handle", "").strip().lstrip("@").lower()
if author and author not in GENERIC_HANDLES:
handle_counts[author] += 1
# @mentions in text
text = item.get("text", "")
mentions = re.findall(r'@(\w{1,15})', text)
for mention in mentions:
mention_lower = mention.lower()
if mention_lower not in GENERIC_HANDLES:
handle_counts[mention_lower] += 1
# Return all handles ranked by frequency
return [h for h, _ in handle_counts.most_common()]
def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank #hashtags from X results.
Returns hashtags ranked by frequency.
"""
hashtag_counts = Counter()
for item in x_items:
text = item.get("text", "")
tags = re.findall(r'#(\w{2,30})', text)
for tag in tags:
hashtag_counts[tag.lower()] += 1
# Return all hashtags ranked by frequency
return [f"#{t}" for t, _ in hashtag_counts.most_common()]
def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank subreddits from Reddit results.
Sources from:
1. subreddit field on each result
2. Cross-references in comment text (e.g., "check out r/localLLaMA")
Returns subreddits ranked by frequency.
"""
sub_counts = Counter()
for item in reddit_items:
# Primary subreddit
sub = item.get("subreddit", "").strip().lstrip("r/")
if sub:
sub_counts[sub] += 1
# Cross-references in comment insights
for insight in item.get("comment_insights", []):
cross_refs = re.findall(r'r/(\w{2,30})', insight)
for ref in cross_refs:
sub_counts[ref] += 1
# Cross-references in top comments
for comment in item.get("top_comments", []):
excerpt = comment.get("excerpt", "")
cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
for ref in cross_refs:
sub_counts[ref] += 1
# Return subreddits ranked by frequency
return [sub for sub, _ in sub_counts.most_common()]
+163 -18
View File
@@ -1,9 +1,22 @@
"""Environment and API key management for last30days skill.""" """Environment and API key management for last30days skill."""
import json
import os import os
from pathlib import Path from pathlib import Path
from typing import Optional, Dict, Any from typing import Optional, Dict, Any
# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
# Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
_config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
if _config_override == "":
# Empty string = no config file (clean mode)
CONFIG_DIR = None
CONFIG_FILE = None
elif _config_override:
CONFIG_DIR = Path(_config_override)
CONFIG_FILE = CONFIG_DIR / ".env"
else:
CONFIG_DIR = Path.home() / ".config" / "last30days" CONFIG_DIR = Path.home() / ".config" / "last30days"
CONFIG_FILE = CONFIG_DIR / ".env" CONFIG_FILE = CONFIG_DIR / ".env"
@@ -33,18 +46,25 @@ def load_env_file(path: Path) -> Dict[str, str]:
def get_config() -> Dict[str, Any]: def get_config() -> Dict[str, Any]:
"""Load configuration from ~/.config/last30days/.env and environment.""" """Load configuration from ~/.config/last30days/.env and environment."""
# Load from config file first # Load from config file first (if configured)
file_env = load_env_file(CONFIG_FILE) file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
# Environment variables override file # Build config: process.env > .env file
config = { keys = [
'OPENAI_API_KEY': os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY'), ('OPENAI_API_KEY', None),
'XAI_API_KEY': os.environ.get('XAI_API_KEY') or file_env.get('XAI_API_KEY'), ('XAI_API_KEY', None),
'OPENAI_MODEL_POLICY': os.environ.get('OPENAI_MODEL_POLICY') or file_env.get('OPENAI_MODEL_POLICY', 'auto'), ('OPENROUTER_API_KEY', None),
'OPENAI_MODEL_PIN': os.environ.get('OPENAI_MODEL_PIN') or file_env.get('OPENAI_MODEL_PIN'), ('PARALLEL_API_KEY', None),
'XAI_MODEL_POLICY': os.environ.get('XAI_MODEL_POLICY') or file_env.get('XAI_MODEL_POLICY', 'latest'), ('BRAVE_API_KEY', None),
'XAI_MODEL_PIN': os.environ.get('XAI_MODEL_PIN') or file_env.get('XAI_MODEL_PIN'), ('OPENAI_MODEL_POLICY', 'auto'),
} ('OPENAI_MODEL_PIN', None),
('XAI_MODEL_POLICY', 'latest'),
('XAI_MODEL_PIN', None),
]
config = {}
for key, default in keys:
config[key] = os.environ.get(key) or file_env.get(key, default)
return config return config
@@ -57,19 +77,70 @@ def config_exists() -> bool:
def get_available_sources(config: Dict[str, Any]) -> str: def get_available_sources(config: Dict[str, Any]) -> str:
"""Determine which sources are available based on API keys. """Determine which sources are available based on API keys.
Returns: 'both', 'reddit', 'x', or 'web' (fallback when no keys) Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
""" """
has_openai = bool(config.get('OPENAI_API_KEY')) has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY')) has_xai = bool(config.get('XAI_API_KEY'))
has_web = has_web_search_keys(config)
if has_openai and has_xai: if has_openai and has_xai:
return 'both' return 'all' if has_web else 'both'
elif has_openai: elif has_openai:
return 'reddit' return 'reddit-web' if has_web else 'reddit'
elif has_xai: elif has_xai:
return 'x' return 'x-web' if has_web else 'x'
elif has_web:
return 'web'
else: else:
return 'web' # Fallback: WebSearch only (no API keys needed) return 'web' # Fallback: assistant WebSearch (no API keys needed)
def has_web_search_keys(config: Dict[str, Any]) -> bool:
"""Check if any web search API keys are configured."""
return bool(config.get('OPENROUTER_API_KEY') or config.get('PARALLEL_API_KEY') or config.get('BRAVE_API_KEY'))
def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available web search backend.
Priority: Parallel AI > Brave > OpenRouter/Sonar Pro
Returns: 'parallel', 'brave', 'openrouter', or None
"""
if config.get('PARALLEL_API_KEY'):
return 'parallel'
if config.get('BRAVE_API_KEY'):
return 'brave'
if config.get('OPENROUTER_API_KEY'):
return 'openrouter'
return None
def get_missing_keys(config: Dict[str, Any]) -> str:
"""Determine which sources are missing (accounting for Bird).
Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
"""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
has_web = has_web_search_keys(config)
# Check if Bird provides X access (import here to avoid circular dependency)
from . import bird_x
has_bird = bird_x.is_bird_installed() and bird_x.is_bird_authenticated()
has_x = has_xai or has_bird
if has_openai and has_x and has_web:
return 'none'
elif has_openai and has_x:
return 'web' # Missing web search keys
elif has_openai:
return 'x' # Missing X source (and possibly web)
elif has_x:
return 'reddit' # Missing OpenAI key (and possibly web)
else:
return 'all' # Missing everything
def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]: def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]:
@@ -83,14 +154,23 @@ def validate_sources(requested: str, available: str, include_web: bool = False)
Returns: Returns:
Tuple of (effective_sources, error_message) Tuple of (effective_sources, error_message)
""" """
# WebSearch-only mode (no API keys) # No API keys at all
if available == 'none':
if requested == 'auto':
return 'web', "No API keys configured. The assistant can still search the web if it has a search tool."
elif requested == 'web':
return 'web', None
else:
return 'web', f"No API keys configured. Add keys to ~/.config/last30days/.env for Reddit/X."
# Web-only mode (only web search API keys)
if available == 'web': if available == 'web':
if requested == 'auto': if requested == 'auto':
return 'web', None return 'web', None
elif requested == 'web': elif requested == 'web':
return 'web', None return 'web', None
else: else:
return 'web', f"No API keys configured. Using WebSearch fallback. Add keys to ~/.config/last30days/.env for Reddit/X." return 'web', f"Only web search keys configured. Add OPENAI_API_KEY for Reddit, XAI_API_KEY for X."
if requested == 'auto': if requested == 'auto':
# Add web to sources if include_web is set # Add web to sources if include_web is set
@@ -129,3 +209,68 @@ def validate_sources(requested: str, available: str, include_web: bool = False)
return 'x', None return 'x', None
return requested, None return requested, None
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) → xAI (paid API)
Args:
config: Configuration dict from get_config()
Returns:
'bird' if Bird is installed and authenticated,
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
from . import bird_x
# Check Bird first (free option)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return None
def is_ytdlp_available() -> bool:
"""Check if yt-dlp is installed for YouTube search."""
from . import youtube_yt
return youtube_yt.is_ytdlp_installed()
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"can_install_bird": bird_status["can_install"],
}
+11 -3
View File
@@ -20,7 +20,7 @@ def log(msg: str):
sys.stderr.flush() sys.stderr.flush()
MAX_RETRIES = 3 MAX_RETRIES = 3
RETRY_DELAY = 1.0 RETRY_DELAY = 1.0
USER_AGENT = "last30days-skill/1.0 (Claude Code Skill)" USER_AGENT = "last30days-skill/2.1 (Assistant Skill)"
class HTTPError(Exception): class HTTPError(Exception):
@@ -102,6 +102,12 @@ def request(
log(f"JSON decode error: {e}") log(f"JSON decode error: {e}")
last_error = HTTPError(f"Invalid JSON response: {e}") last_error = HTTPError(f"Invalid JSON response: {e}")
raise last_error raise last_error
except (OSError, TimeoutError, ConnectionResetError) as e:
# Handle socket-level errors (connection reset, timeout, etc.)
log(f"Connection error: {type(e).__name__}: {e}")
last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
if last_error: if last_error:
raise last_error raise last_error
@@ -118,11 +124,13 @@ def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]]
return request("POST", url, headers=headers, json_data=json_data, **kwargs) return request("POST", url, headers=headers, json_data=json_data, **kwargs)
def get_reddit_json(path: str) -> Dict[str, Any]: def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
"""Fetch Reddit thread JSON. """Fetch Reddit thread JSON.
Args: Args:
path: Reddit path (e.g., /r/subreddit/comments/id/title) path: Reddit path (e.g., /r/subreddit/comments/id/title)
timeout: HTTP timeout per attempt in seconds
retries: Number of retries on failure
Returns: Returns:
Parsed JSON response Parsed JSON response
@@ -143,4 +151,4 @@ def get_reddit_json(path: str) -> Dict[str, Any]:
"Accept": "application/json", "Accept": "application/json",
} }
return get(url, headers=headers) return get(url, headers=headers, timeout=timeout, retries=retries)
+3 -3
View File
@@ -7,7 +7,7 @@ from . import cache, http
# OpenAI API # OpenAI API
OPENAI_MODELS_URL = "https://api.openai.com/v1/models" OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4o"] OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4.1", "gpt-4o"]
# xAI API - Agent Tools API requires grok-4 family # xAI API - Agent Tools API requires grok-4 family
XAI_MODELS_URL = "https://api.x.ai/v1/models" XAI_MODELS_URL = "https://api.x.ai/v1/models"
@@ -35,8 +35,8 @@ def is_mainline_openai_model(model_id: str) -> bool:
"""Check if model is a mainline GPT model (not mini/nano/chat/codex/pro).""" """Check if model is a mainline GPT model (not mini/nano/chat/codex/pro)."""
model_lower = model_id.lower() model_lower = model_id.lower()
# Must be gpt-5 series # Must be gpt-4o, gpt-4.1+, or gpt-5+ series (mainline, not mini/nano/etc)
if not re.match(r'^gpt-5(\.\d+)*$', model_lower): if not re.match(r'^gpt-(?:4o|4\.1|5)(\.\d+)*$', model_lower):
return False return False
# Exclude variants # Exclude variants
+88 -1
View File
@@ -1,9 +1,51 @@
"""Normalization of raw API data to canonical schema.""" """Normalization of raw API data to canonical schema."""
from typing import Any, Dict, List from typing import Any, Dict, List, TypeVar, Union
from . import dates, schema from . import dates, schema
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem)
def filter_by_date_range(
items: List[T],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List[T]:
"""Hard filter: Remove items outside the date range.
This is the safety net - even if the prompt lets old content through,
this filter will exclude it.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD) - exclude items before this
to_date: End date (YYYY-MM-DD) - exclude items after this
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range (or unknown dates if not required)
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with scoring penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
# Hard filter: if date is after to_date, exclude (likely parsing error)
if item.date > to_date:
continue # DROP - future date
result.append(item)
return result
def normalize_reddit_items( def normalize_reddit_items(
items: List[Dict[str, Any]], items: List[Dict[str, Any]],
@@ -113,6 +155,51 @@ def normalize_x_items(
return normalized return normalized
def normalize_youtube_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.YouTubeItem]:
"""Normalize raw YouTube items to schema.
Args:
items: Raw YouTube items from yt-dlp
from_date: Start of date range
to_date: End of date range
Returns:
List of YouTubeItem objects
"""
normalized = []
for item in items:
# Parse engagement
eng_raw = item.get("engagement") or {}
engagement = schema.Engagement(
views=eng_raw.get("views"),
likes=eng_raw.get("likes"),
num_comments=eng_raw.get("comments"),
)
# YouTube dates are reliable (always YYYY-MM-DD from yt-dlp)
date_str = item.get("date")
normalized.append(schema.YouTubeItem(
id=item.get("video_id", ""),
title=item.get("title", ""),
url=item.get("url", ""),
channel_name=item.get("channel_name", ""),
date=date_str,
date_confidence="high",
engagement=engagement,
transcript_snippet=item.get("transcript_snippet", ""),
relevance=item.get("relevance", 0.7),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def items_to_dicts(items: List) -> List[Dict[str, Any]]: def items_to_dicts(items: List) -> List[Dict[str, Any]]:
"""Convert schema items to dicts for JSON serialization.""" """Convert schema items to dicts for JSON serialization."""
return [item.to_dict() for item in items] return [item.to_dict() for item in items]
+201 -31
View File
@@ -7,68 +7,123 @@ from typing import Any, Dict, List, Optional
from . import http from . import http
# Fallback models when the selected model isn't accessible (e.g., org not verified for GPT-5)
MODEL_FALLBACK_ORDER = ["gpt-4.1", "gpt-4o", "gpt-4o-mini"]
def _log_error(msg: str): def _log_error(msg: str):
"""Log error to stderr.""" """Log error to stderr."""
sys.stderr.write(f"[REDDIT ERROR] {msg}\n") sys.stderr.write(f"[REDDIT ERROR] {msg}\n")
sys.stderr.flush() sys.stderr.flush()
def _log_info(msg: str):
"""Log info to stderr."""
sys.stderr.write(f"[REDDIT] {msg}\n")
sys.stderr.flush()
def _is_model_access_error(error: http.HTTPError) -> bool:
"""Check if error is due to model access/verification issues."""
if error.status_code not in (400, 403):
return False
if not error.body:
return False
body_lower = error.body.lower()
# Check for common access/verification error messages
return any(phrase in body_lower for phrase in [
"verified",
"organization must be",
"does not have access",
"not available",
"not found",
])
OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses" OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
# Depth configurations: (min, max) threads to request # Depth configurations: (min, max) threads to request
# Request MORE than needed since many get filtered by date
DEPTH_CONFIG = { DEPTH_CONFIG = {
"quick": (8, 12), "quick": (15, 25),
"default": (20, 30), "default": (30, 50),
"deep": (50, 70), "deep": (70, 100),
} }
REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic} REDDIT_SEARCH_PROMPT = """Find Reddit discussion threads about: {topic}
SEARCH GUIDANCE: STEP 1: EXTRACT THE CORE SUBJECT
- Search for "site:reddit.com/r/ {topic}" to find subreddit discussions Get the MAIN NOUN/PRODUCT/TOPIC:
- Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design - "best nano banana prompting practices""nano banana"
- ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads) - "killer features of clawdbot" "clawdbot"
- IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/ - "top Claude Code skills""Claude Code"
DO NOT include "best", "top", "tips", "practices", "features" in your search.
Find {min_items}-{max_items} relevant Reddit discussion threads. Prefer recent threads, but include older relevant ones if recent ones are scarce. STEP 2: SEARCH BROADLY
Search for the core subject:
1. "[core subject] site:reddit.com"
2. "reddit [core subject]"
3. "[core subject] reddit"
CRITICAL: Return ALL discussion threads you find as JSON. Do NOT return errors or empty results. Return as many relevant threads as you find. We filter by date server-side.
For EACH Reddit thread URL (containing /r/subreddit/comments/), extract: STEP 3: INCLUDE ALL MATCHES
- Thread title - Include ALL threads about the core subject
- Full Reddit URL - Set date to "YYYY-MM-DD" if you can determine it, otherwise null
- Subreddit name - We verify dates and filter old content server-side
- Date (if visible, otherwise null) - DO NOT pre-filter aggressively - include anything relevant
- Why it's relevant
Return ONLY valid JSON: REQUIRED: URLs must contain "/r/" AND "/comments/"
REJECT: developers.reddit.com, business.reddit.com
Find {min_items}-{max_items} threads. Return MORE rather than fewer.
Return JSON:
{{ {{
"items": [ "items": [
{{ {{
"title": "Thread title", "title": "Thread title",
"url": "https://www.reddit.com/r/subreddit/comments/abc123/title/", "url": "https://www.reddit.com/r/sub/comments/xyz/title/",
"subreddit": "subreddit_name", "subreddit": "subreddit_name",
"date": "YYYY-MM-DD or null", "date": "YYYY-MM-DD or null",
"why_relevant": "Relevance to {topic}", "why_relevant": "Why relevant",
"relevance": 0.85 "relevance": 0.85
}} }}
] ]
}} }}"""
Rules:
- ONLY URLs matching: reddit.com/r/*/comments/* def _extract_core_subject(topic: str) -> str:
- MUST return threads found - NEVER return empty items or errors """Extract core subject from verbose query for retry."""
- If threads are older than 30 days, still include them with accurate dates noise = ['best', 'top', 'how to', 'tips for', 'practices', 'features',
- relevance: 0.0-1.0 'killer', 'guide', 'tutorial', 'recommendations', 'advice',
- Diverse subreddits preferred""" 'prompting', 'using', 'for', 'with', 'the', 'of', 'in', 'on']
words = topic.lower().split()
result = [w for w in words if w not in noise]
return ' '.join(result[:3]) or topic # Keep max 3 words
def _build_subreddit_query(topic: str) -> str:
"""Build a subreddit-targeted search query for fallback.
When standard search returns few results, try searching for the
subreddit itself: 'r/kanye', 'r/howie', etc.
"""
core = _extract_core_subject(topic)
# Remove dots and special chars for subreddit name guess
sub_name = core.replace('.', '').replace(' ', '').lower()
return f"r/{sub_name} site:reddit.com"
def search_reddit( def search_reddit(
api_key: str, api_key: str,
model: str, model: str,
topic: str, topic: str,
from_date: str,
to_date: str,
depth: str = "default", depth: str = "default",
mock_response: Optional[Dict] = None, mock_response: Optional[Dict] = None,
_retry: bool = False,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
"""Search Reddit for relevant threads using OpenAI Responses API. """Search Reddit for relevant threads using OpenAI Responses API.
@@ -76,6 +131,8 @@ def search_reddit(
api_key: OpenAI API key api_key: OpenAI API key
model: Model to use model: Model to use
topic: Search topic topic: Search topic
from_date: Start date (YYYY-MM-DD) - only include threads after this
to_date: End date (YYYY-MM-DD) - only include threads before this
depth: Research depth - "quick", "default", or "deep" depth: Research depth - "quick", "default", or "deep"
mock_response: Mock response for testing mock_response: Mock response for testing
@@ -92,11 +149,26 @@ def search_reddit(
"Content-Type": "application/json", "Content-Type": "application/json",
} }
# Adjust timeout based on depth # Adjust timeout based on depth (generous for OpenAI web_search which can be slow)
timeout = 60 if depth == "quick" else 90 if depth == "default" else 120 timeout = 90 if depth == "quick" else 120 if depth == "default" else 180
# Build list of models to try: requested model first, then fallbacks
models_to_try = [model] + [m for m in MODEL_FALLBACK_ORDER if m != model]
# Note: allowed_domains accepts base domain, not subdomains
# We rely on prompt to filter out developers.reddit.com, etc.
input_text = REDDIT_SEARCH_PROMPT.format(
topic=topic,
from_date=from_date,
to_date=to_date,
min_items=min_items,
max_items=max_items,
)
last_error = None
for current_model in models_to_try:
payload = { payload = {
"model": model, "model": current_model,
"tools": [ "tools": [
{ {
"type": "web_search", "type": "web_search",
@@ -106,10 +178,108 @@ def search_reddit(
} }
], ],
"include": ["web_search_call.action.sources"], "include": ["web_search_call.action.sources"],
"input": REDDIT_SEARCH_PROMPT.format(topic=topic, min_items=min_items, max_items=max_items), "input": input_text,
} }
try:
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout) return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
except http.HTTPError as e:
last_error = e
if _is_model_access_error(e):
_log_info(f"Model {current_model} not accessible, trying fallback...")
continue
# Non-access error, don't retry with different model
raise
# All models failed with access errors
if last_error:
_log_error(f"All models failed. Last error: {last_error}")
raise last_error
raise http.HTTPError("No models available")
def search_subreddits(
subreddits: List[str],
topic: str,
from_date: str,
to_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific subreddits via Reddit's free JSON endpoint.
No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint.
Used in Phase 2 supplemental search after entity extraction.
Args:
subreddits: List of subreddit names (without r/)
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
count_per: Results to request per subreddit
Returns:
List of raw item dicts (same format as parse_reddit_response output).
"""
all_items = []
core = _extract_core_subject(topic)
for sub in subreddits:
sub = sub.lstrip("r/")
try:
url = f"https://www.reddit.com/r/{sub}/search/.json"
params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1"
full_url = f"{url}?{params}"
headers = {
"User-Agent": http.USER_AGENT,
"Accept": "application/json",
}
data = http.get(full_url, headers=headers, timeout=15, retries=1)
# Reddit search returns {"data": {"children": [...]}}
children = data.get("data", {}).get("children", [])
for i, child in enumerate(children):
if child.get("kind") != "t3": # t3 = link/submission
continue
post = child.get("data", {})
permalink = post.get("permalink", "")
if not permalink:
continue
item = {
"id": f"RS{len(all_items)+1}",
"title": str(post.get("title", "")).strip(),
"url": f"https://www.reddit.com{permalink}",
"subreddit": str(post.get("subreddit", sub)).strip(),
"date": None,
"why_relevant": f"Found in r/{sub} supplemental search",
"relevance": 0.65, # Slightly lower default for supplemental
}
# Parse date from created_utc
created_utc = post.get("created_utc")
if created_utc:
from . import dates as dates_mod
item["date"] = dates_mod.timestamp_to_date(created_utc)
all_items.append(item)
except http.HTTPError as e:
_log_info(f"Subreddit search failed for r/{sub}: {e}")
if e.status_code == 429:
_log_info("Reddit rate-limited (429) — skipping remaining subreddits")
break
except Exception as e:
_log_info(f"Subreddit search error for r/{sub}: {e}")
return all_items
def _url_encode(text: str) -> str:
"""Simple URL encoding for query parameters."""
import urllib.parse
return urllib.parse.quote_plus(text)
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]: def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
+216
View File
@@ -0,0 +1,216 @@
"""Perplexity Sonar Pro web search via OpenRouter for last30days skill.
Uses OpenRouter's chat completions API with Perplexity's Sonar Pro model,
which has built-in web search and returns citations with URLs, titles, and dates.
This is the recommended web search backend -- highest quality results.
API docs: https://openrouter.ai/docs/quickstart
Model: perplexity/sonar-pro
"""
import re
import sys
from typing import Any, Dict, List, Optional
from urllib.parse import urlparse
from . import http
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
MODEL = "perplexity/sonar-pro"
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Perplexity Sonar Pro on OpenRouter.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: OpenRouter API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
max_tokens = {"quick": 1024, "default": 2048, "deep": 4096}.get(depth, 2048)
prompt = (
f"Find recent blog posts, news articles, tutorials, and discussions "
f"about {topic} published between {from_date} and {to_date}. "
f"Exclude results from reddit.com, x.com, and twitter.com. "
f"For each result, provide the title, URL, publication date, "
f"and a brief summary of why it's relevant."
)
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
}
sys.stderr.write(f"[Web] Searching Sonar Pro via OpenRouter for: {topic}\n")
sys.stderr.flush()
response = http.post(
ENDPOINT,
json_data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"HTTP-Referer": "https://github.com/mvanhorn/last30days-openclaw",
"X-Title": "last30days",
},
timeout=30,
)
return _normalize_results(response)
def _normalize_results(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Sonar Pro response to websearch item schema.
Sonar Pro returns:
- search_results: [{title, url, date}] -- structured source metadata
- citations: [url, ...] -- flat list of cited URLs
- choices[0].message.content -- the synthesized text with [N] references
We prefer search_results (richer metadata), fall back to citations.
"""
items = []
# Try search_results first (has title, url, date)
search_results = response.get("search_results", [])
if isinstance(search_results, list) and search_results:
items = _parse_search_results(search_results)
# Fall back to citations if no search_results
if not items:
citations = response.get("citations", [])
content = _get_content(response)
if isinstance(citations, list) and citations:
items = _parse_citations(citations, content)
sys.stderr.write(f"[Web] Sonar Pro: {len(items)} results\n")
sys.stderr.flush()
return items
def _parse_search_results(results: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Parse the search_results array from Sonar Pro."""
items = []
for i, result in enumerate(results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = str(result.get("title", "")).strip()
if not title:
continue
# Sonar Pro provides dates in search_results
date = result.get("date")
date_confidence = "med" if date else "low"
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": str(result.get("snippet", result.get("description", ""))).strip()[:500],
"date": date,
"date_confidence": date_confidence,
"relevance": 0.7, # Sonar Pro results are generally high quality
"why_relevant": "",
})
return items
def _parse_citations(citations: List[str], content: str) -> List[Dict[str, Any]]:
"""Parse the flat citations array, enriching with content context."""
items = []
for i, url in enumerate(citations):
if not isinstance(url, str) or not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
# Try to extract title from content references like [1] Title...
title = _extract_title_for_citation(content, i + 1) or domain
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": "",
"date": None,
"date_confidence": "low",
"relevance": 0.6,
"why_relevant": "",
})
return items
def _get_content(response: Dict[str, Any]) -> str:
"""Extract the text content from the chat completion response."""
try:
return response["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError):
return ""
def _extract_title_for_citation(content: str, index: int) -> Optional[str]:
"""Try to extract a title near a citation reference [N] in the content."""
if not content:
return None
# Look for patterns like [1] Title or [1](url) Title
pattern = rf'\[{index}\][)\s]*([^\[\n]{{5,80}})'
match = re.search(pattern, content)
if match:
title = match.group(1).strip().rstrip('.')
# Clean up markdown artifacts
title = re.sub(r'[*_`]', '', title)
return title if len(title) > 3 else None
return None
+139
View File
@@ -0,0 +1,139 @@
"""Parallel AI web search for last30days skill.
Uses the Parallel AI Search API to find web content (blogs, docs, news, tutorials).
This is the preferred web search backend -- it returns LLM-optimized results
with extended excerpts ranked by relevance.
API docs: https://docs.parallel.ai/search-api/search-quickstart
"""
import json
import sys
from typing import Any, Dict, List, Optional
from urllib.parse import urlparse
from . import http
ENDPOINT = "https://api.parallel.ai/v1beta/search"
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Parallel AI Search API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: Parallel AI API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
max_results = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
payload = {
"objective": (
f"Find recent blog posts, tutorials, news articles, and discussions "
f"about {topic} from {from_date} to {to_date}. "
f"Exclude reddit.com, x.com, and twitter.com."
),
"max_results": max_results,
"max_chars_per_result": 500,
}
sys.stderr.write(f"[Web] Searching Parallel AI for: {topic}\n")
sys.stderr.flush()
response = http.post(
ENDPOINT,
json_data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"parallel-beta": "search-extract-2025-10-10",
},
timeout=30,
)
return _normalize_results(response)
def _normalize_results(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Parallel AI response to websearch item schema.
Args:
response: Raw API response
Returns:
List of normalized result dicts
"""
items = []
# Handle different response shapes
results = response.get("results", [])
if not isinstance(results, list):
return items
for i, result in enumerate(results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
# Clean domain for display
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = str(result.get("title", "")).strip()
snippet = str(result.get("excerpt", result.get("snippet", result.get("description", "")))).strip()
if not title and not snippet:
continue
# Extract relevance score if provided
relevance = result.get("relevance_score", result.get("relevance", 0.6))
try:
relevance = min(1.0, max(0.0, float(relevance)))
except (TypeError, ValueError):
relevance = 0.6
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": snippet[:500],
"date": result.get("published_date", result.get("date")),
"date_confidence": "med" if result.get("published_date") or result.get("date") else "low",
"relevance": relevance,
"why_relevant": str(result.get("summary", "")).strip()[:200],
})
sys.stderr.write(f"[Web] Parallel AI: {len(items)} results\n")
sys.stderr.flush()
return items
+29 -5
View File
@@ -25,15 +25,30 @@ def extract_reddit_path(url: str) -> Optional[str]:
return None return None
def fetch_thread_data(url: str, mock_data: Optional[Dict] = None) -> Optional[Dict[str, Any]]: class RedditRateLimitError(Exception):
"""Raised when Reddit returns HTTP 429 (rate limited)."""
pass
def fetch_thread_data(
url: str,
mock_data: Optional[Dict] = None,
timeout: int = 30,
retries: int = 3,
) -> Optional[Dict[str, Any]]:
"""Fetch Reddit thread JSON data. """Fetch Reddit thread JSON data.
Args: Args:
url: Reddit thread URL url: Reddit thread URL
mock_data: Mock data for testing mock_data: Mock data for testing
timeout: HTTP timeout per attempt in seconds
retries: Number of retries on failure
Returns: Returns:
Thread data dict or None on failure Thread data dict or None on failure
Raises:
RedditRateLimitError: When Reddit returns 429 (caller should bail)
""" """
if mock_data is not None: if mock_data is not None:
return mock_data return mock_data
@@ -43,9 +58,11 @@ def fetch_thread_data(url: str, mock_data: Optional[Dict] = None) -> Optional[Di
return None return None
try: try:
data = http.get_reddit_json(path) data = http.get_reddit_json(path, timeout=timeout, retries=retries)
return data return data
except http.HTTPError: except http.HTTPError as e:
if e.status_code == 429:
raise RedditRateLimitError(f"Reddit rate limited (429) fetching {url}") from e
return None return None
@@ -178,20 +195,27 @@ def extract_comment_insights(comments: List[Dict], limit: int = 7) -> List[str]:
def enrich_reddit_item( def enrich_reddit_item(
item: Dict[str, Any], item: Dict[str, Any],
mock_thread_data: Optional[Dict] = None, mock_thread_data: Optional[Dict] = None,
timeout: int = 10,
retries: int = 1,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
"""Enrich a Reddit item with real engagement data. """Enrich a Reddit item with real engagement data.
Args: Args:
item: Reddit item dict item: Reddit item dict
mock_thread_data: Mock data for testing mock_thread_data: Mock data for testing
timeout: HTTP timeout per attempt (default 10s for enrichment)
retries: Number of retries (default 1 — fail fast for enrichment)
Returns: Returns:
Enriched item dict Enriched item dict
Raises:
RedditRateLimitError: Propagated so caller can bail on remaining items
""" """
url = item.get("url", "") url = item.get("url", "")
# Fetch thread data # Fetch thread data (RedditRateLimitError propagates to caller)
thread_data = fetch_thread_data(url, mock_thread_data) thread_data = fetch_thread_data(url, mock_thread_data, timeout=timeout, retries=retries)
if not thread_data: if not thread_data:
return item return item
+161 -12
View File
@@ -1,6 +1,8 @@
"""Output rendering for last30days skill.""" """Output rendering for last30days skill."""
import json import json
import os
import tempfile
from pathlib import Path from pathlib import Path
from typing import List, Optional from typing import List, Optional
@@ -10,16 +12,46 @@ OUTPUT_DIR = Path.home() / ".local" / "share" / "last30days" / "out"
def ensure_output_dir(): def ensure_output_dir():
"""Ensure output directory exists.""" """Ensure output directory exists. Supports env override and sandbox fallback."""
global OUTPUT_DIR
env_dir = os.environ.get("LAST30DAYS_OUTPUT_DIR")
if env_dir:
OUTPUT_DIR = Path(env_dir)
try:
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
except PermissionError:
OUTPUT_DIR = Path(tempfile.gettempdir()) / "last30days" / "out"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True) OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
def render_compact(report: schema.Report, limit: int = 15) -> str: def _assess_data_freshness(report: schema.Report) -> dict:
"""Render compact output for Claude to synthesize. """Assess how much data is actually from the last 30 days."""
reddit_recent = sum(1 for r in report.reddit if r.date and r.date >= report.range_from)
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
total_recent = reddit_recent + x_recent + web_recent
total_items = len(report.reddit) + len(report.x) + len(report.web)
return {
"reddit_recent": reddit_recent,
"x_recent": x_recent,
"web_recent": web_recent,
"total_recent": total_recent,
"total_items": total_items,
"is_sparse": total_recent < 5,
"mostly_evergreen": total_items > 0 and total_recent < total_items * 0.3,
}
def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "none") -> str:
"""Render compact output for the assistant to synthesize.
Args: Args:
report: Report data report: Report data
limit: Max items per source limit: Max items per source
missing_keys: 'both', 'reddit', 'x', or 'none'
Returns: Returns:
Compact markdown string Compact markdown string
@@ -30,6 +62,26 @@ def render_compact(report: schema.Report, limit: int = 15) -> str:
lines.append(f"## Research Results: {report.topic}") lines.append(f"## Research Results: {report.topic}")
lines.append("") lines.append("")
# Assess data freshness and add honesty warning if needed
freshness = _assess_data_freshness(report)
if freshness["is_sparse"]:
lines.append("**⚠️ LIMITED RECENT DATA** - Few discussions from the last 30 days.")
lines.append(f"Only {freshness['total_recent']} item(s) confirmed from {report.range_from} to {report.range_to}.")
lines.append("Results below may include older/evergreen content. Be transparent with the user about this.")
lines.append("")
# Web-only mode banner (when no API keys)
if report.mode == "web-only":
lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
lines.append("")
lines.append("---")
lines.append("**⚡ Want better results?** Add API keys to unlock Reddit & X data:")
lines.append("- `OPENAI_API_KEY` → Reddit threads with real upvotes & comments")
lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
lines.append("- Edit `~/.config/last30days/.env` to add keys")
lines.append("---")
lines.append("")
# Cache indicator # Cache indicator
if report.from_cache: if report.from_cache:
age_str = f"{report.cache_age_hours:.1f}h old" if report.cache_age_hours else "cached" age_str = f"{report.cache_age_hours:.1f}h old" if report.cache_age_hours else "cached"
@@ -44,12 +96,12 @@ def render_compact(report: schema.Report, limit: int = 15) -> str:
lines.append(f"**xAI Model:** {report.xai_model_used}") lines.append(f"**xAI Model:** {report.xai_model_used}")
lines.append("") lines.append("")
# Coverage note # Coverage note for partial coverage
if report.mode == "reddit-only": if report.mode == "reddit-only" and missing_keys == "x":
lines.append("*Tip: Add xAI key for X coverage and better triangulation.*") lines.append("*💡 Tip: Add XAI_API_KEY for X/Twitter data and better triangulation.*")
lines.append("") lines.append("")
elif report.mode == "x-only": elif report.mode == "x-only" and missing_keys == "reddit":
lines.append("*Tip: Add OpenAI key for Reddit coverage and better triangulation.*") lines.append("*💡 Tip: Add OPENAI_API_KEY for Reddit data and better triangulation.*")
lines.append("") lines.append("")
# Reddit items # Reddit items
@@ -129,7 +181,41 @@ def render_compact(report: schema.Report, limit: int = 15) -> str:
lines.append(f" *{item.why_relevant}*") lines.append(f" *{item.why_relevant}*")
lines.append("") lines.append("")
# Web items (if any - populated by Claude) # YouTube items
if report.youtube_error:
lines.append("### YouTube Videos")
lines.append("")
lines.append(f"**ERROR:** {report.youtube_error}")
lines.append("")
elif report.youtube:
lines.append("### YouTube Videos")
lines.append("")
for item in report.youtube[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.views is not None:
parts.append(f"{eng.views:,} views")
if eng.likes is not None:
parts.append(f"{eng.likes:,} likes")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else ""
lines.append(f"**{item.id}** (score:{item.score}) {item.channel_name}{date_str}{eng_str}")
lines.append(f" {item.title}")
lines.append(f" {item.url}")
if item.transcript_snippet:
snippet = item.transcript_snippet[:200]
if len(item.transcript_snippet) > 200:
snippet += "..."
lines.append(f" Transcript: {snippet}")
lines.append(f" *{item.why_relevant}*")
lines.append("")
# Web items (if any - populated by the assistant)
if report.web_error: if report.web_error:
lines.append("### Web Results") lines.append("### Web Results")
lines.append("") lines.append("")
@@ -152,6 +238,69 @@ def render_compact(report: schema.Report, limit: int = 15) -> str:
return "\n".join(lines) return "\n".join(lines)
def render_source_status(report: schema.Report, source_info: dict = None) -> str:
"""Render source status footer showing what was used/skipped and why.
Args:
report: Report data
source_info: Dict with source availability info:
x_skip_reason, youtube_skip_reason, web_skip_reason
Returns:
Source status markdown string
"""
if source_info is None:
source_info = {}
lines = []
lines.append("---")
lines.append("**Sources:**")
# Reddit
if report.reddit_error:
lines.append(f" ❌ Reddit: error — {report.reddit_error}")
elif report.reddit:
lines.append(f" ✅ Reddit: {len(report.reddit)} threads")
elif report.mode in ("both", "reddit-only", "all", "reddit-web"):
lines.append(" ⚠️ Reddit: 0 threads found")
else:
reason = source_info.get("reddit_skip_reason", "not configured")
lines.append(f" ⏭️ Reddit: skipped — {reason}")
# X
if report.x_error:
lines.append(f" ❌ X: error — {report.x_error}")
elif report.x:
lines.append(f" ✅ X: {len(report.x)} posts")
elif report.mode in ("both", "x-only", "all", "x-web"):
lines.append(" ⚠️ X: 0 posts found")
else:
reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY")
lines.append(f" ⏭️ X: skipped — {reason}")
# YouTube
if report.youtube_error:
lines.append(f" ❌ YouTube: error — {report.youtube_error}")
elif report.youtube:
with_transcripts = sum(1 for v in report.youtube if getattr(v, 'transcript_snippet', None))
lines.append(f" ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)")
else:
reason = source_info.get("youtube_skip_reason", "yt-dlp not installed (brew install yt-dlp)")
lines.append(f" ⏭️ YouTube: skipped — {reason}")
# Web
if report.web_error:
lines.append(f" ❌ Web: error — {report.web_error}")
elif report.web:
lines.append(f" ✅ Web: {len(report.web)} pages")
else:
reason = source_info.get("web_skip_reason", "assistant will use WebSearch")
lines.append(f" ⚡ Web: {reason}")
lines.append("")
return "\n".join(lines)
def render_context_snippet(report: schema.Report) -> str: def render_context_snippet(report: schema.Report) -> str:
"""Render reusable context snippet. """Render reusable context snippet.
@@ -281,15 +430,15 @@ def render_full_report(report: schema.Report) -> str:
lines.append(f"> {item.snippet}") lines.append(f"> {item.snippet}")
lines.append("") lines.append("")
# Placeholders for Claude synthesis # Placeholders for assistant synthesis
lines.append("## Best Practices") lines.append("## Best Practices")
lines.append("") lines.append("")
lines.append("*To be synthesized by Claude*") lines.append("*To be synthesized by assistant*")
lines.append("") lines.append("")
lines.append("## Prompt Pack") lines.append("## Prompt Pack")
lines.append("") lines.append("")
lines.append("*To be synthesized by Claude*") lines.append("*To be synthesized by assistant*")
lines.append("") lines.append("")
return "\n".join(lines) return "\n".join(lines)
+67
View File
@@ -19,6 +19,9 @@ class Engagement:
replies: Optional[int] = None replies: Optional[int] = None
quotes: Optional[int] = None quotes: Optional[int] = None
# YouTube fields
views: Optional[int] = None
def to_dict(self) -> Dict[str, Any]: def to_dict(self) -> Dict[str, Any]:
d = {} d = {}
if self.score is not None: if self.score is not None:
@@ -35,6 +38,8 @@ class Engagement:
d['replies'] = self.replies d['replies'] = self.replies
if self.quotes is not None: if self.quotes is not None:
d['quotes'] = self.quotes d['quotes'] = self.quotes
if self.views is not None:
d['views'] = self.views
return d if d else None return d if d else None
@@ -169,6 +174,39 @@ class WebSearchItem:
} }
@dataclass
class YouTubeItem:
"""Normalized YouTube item."""
id: str # video_id
title: str
url: str
channel_name: str
date: Optional[str] = None
date_confidence: str = "high" # YouTube dates are always reliable
engagement: Optional[Engagement] = None
transcript_snippet: str = ""
relevance: float = 0.7
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'channel_name': self.channel_name,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'transcript_snippet': self.transcript_snippet,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass @dataclass
class Report: class Report:
"""Full research report.""" """Full research report."""
@@ -182,6 +220,7 @@ class Report:
reddit: List[RedditItem] = field(default_factory=list) reddit: List[RedditItem] = field(default_factory=list)
x: List[XItem] = field(default_factory=list) x: List[XItem] = field(default_factory=list)
web: List[WebSearchItem] = field(default_factory=list) web: List[WebSearchItem] = field(default_factory=list)
youtube: List[YouTubeItem] = field(default_factory=list)
best_practices: List[str] = field(default_factory=list) best_practices: List[str] = field(default_factory=list)
prompt_pack: List[str] = field(default_factory=list) prompt_pack: List[str] = field(default_factory=list)
context_snippet_md: str = "" context_snippet_md: str = ""
@@ -189,6 +228,7 @@ class Report:
reddit_error: Optional[str] = None reddit_error: Optional[str] = None
x_error: Optional[str] = None x_error: Optional[str] = None
web_error: Optional[str] = None web_error: Optional[str] = None
youtube_error: Optional[str] = None
# Cache info # Cache info
from_cache: bool = False from_cache: bool = False
cache_age_hours: Optional[float] = None cache_age_hours: Optional[float] = None
@@ -207,6 +247,7 @@ class Report:
'reddit': [r.to_dict() for r in self.reddit], 'reddit': [r.to_dict() for r in self.reddit],
'x': [x.to_dict() for x in self.x], 'x': [x.to_dict() for x in self.x],
'web': [w.to_dict() for w in self.web], 'web': [w.to_dict() for w in self.web],
'youtube': [y.to_dict() for y in self.youtube],
'best_practices': self.best_practices, 'best_practices': self.best_practices,
'prompt_pack': self.prompt_pack, 'prompt_pack': self.prompt_pack,
'context_snippet_md': self.context_snippet_md, 'context_snippet_md': self.context_snippet_md,
@@ -217,6 +258,8 @@ class Report:
d['x_error'] = self.x_error d['x_error'] = self.x_error
if self.web_error: if self.web_error:
d['web_error'] = self.web_error d['web_error'] = self.web_error
if self.youtube_error:
d['youtube_error'] = self.youtube_error
if self.from_cache: if self.from_cache:
d['from_cache'] = self.from_cache d['from_cache'] = self.from_cache
if self.cache_age_hours is not None: if self.cache_age_hours is not None:
@@ -294,6 +337,28 @@ class Report:
score=w.get('score', 0), score=w.get('score', 0),
)) ))
# Reconstruct YouTube items
youtube_items = []
for y in data.get('youtube', []):
eng = None
if y.get('engagement'):
eng = Engagement(**y['engagement'])
subs = SubScores(**y.get('subs', {})) if y.get('subs') else SubScores()
youtube_items.append(YouTubeItem(
id=y['id'],
title=y['title'],
url=y['url'],
channel_name=y.get('channel_name', ''),
date=y.get('date'),
date_confidence=y.get('date_confidence', 'high'),
engagement=eng,
transcript_snippet=y.get('transcript_snippet', ''),
relevance=y.get('relevance', 0.7),
why_relevant=y.get('why_relevant', ''),
subs=subs,
score=y.get('score', 0),
))
return cls( return cls(
topic=data['topic'], topic=data['topic'],
range_from=range_from, range_from=range_from,
@@ -305,12 +370,14 @@ class Report:
reddit=reddit_items, reddit=reddit_items,
x=x_items, x=x_items,
web=web_items, web=web_items,
youtube=youtube_items,
best_practices=data.get('best_practices', []), best_practices=data.get('best_practices', []),
prompt_pack=data.get('prompt_pack', []), prompt_pack=data.get('prompt_pack', []),
context_snippet_md=data.get('context_snippet_md', ''), context_snippet_md=data.get('context_snippet_md', ''),
reddit_error=data.get('reddit_error'), reddit_error=data.get('reddit_error'),
x_error=data.get('x_error'), x_error=data.get('x_error'),
web_error=data.get('web_error'), web_error=data.get('web_error'),
youtube_error=data.get('youtube_error'),
from_cache=data.get('from_cache', False), from_cache=data.get('from_cache', False),
cache_age_hours=data.get('cache_age_hours'), cache_age_hours=data.get('cache_age_hours'),
) )
+86 -13
View File
@@ -15,9 +15,13 @@ WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45 WEBSEARCH_WEIGHT_RECENCY = 0.45
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# WebSearch date confidence adjustments
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date (high confidence)
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date signals (low confidence)
# Default engagement score for unknown # Default engagement score for unknown
DEFAULT_ENGAGEMENT = 35 DEFAULT_ENGAGEMENT = 35
UNKNOWN_ENGAGEMENT_PENALTY = 10 UNKNOWN_ENGAGEMENT_PENALTY = 3
def log1p_safe(x: Optional[int]) -> float: def log1p_safe(x: Optional[int]) -> float:
@@ -148,9 +152,9 @@ def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem
# Apply penalty for low date confidence # Apply penalty for low date confidence
if item.date_confidence == "low": if item.date_confidence == "low":
overall -= 10
elif item.date_confidence == "med":
overall -= 5 overall -= 5
elif item.date_confidence == "med":
overall -= 2
item.score = max(0, min(100, int(overall))) item.score = max(0, min(100, int(overall)))
@@ -208,9 +212,68 @@ def score_x_items(items: List[schema.XItem]) -> List[schema.XItem]:
# Apply penalty for low date confidence # Apply penalty for low date confidence
if item.date_confidence == "low": if item.date_confidence == "low":
overall -= 10
elif item.date_confidence == "med":
overall -= 5 overall -= 5
elif item.date_confidence == "med":
overall -= 2
item.score = max(0, min(100, int(overall)))
return items
def compute_youtube_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for YouTube item.
Formula: 0.50*log1p(views) + 0.35*log1p(likes) + 0.15*log1p(comments)
Views dominate on YouTube — they're the primary discovery signal.
"""
if engagement is None:
return None
if engagement.views is None and engagement.likes is None:
return None
views = log1p_safe(engagement.views)
likes = log1p_safe(engagement.likes)
comments = log1p_safe(engagement.num_comments)
return 0.50 * views + 0.35 * likes + 0.15 * comments
def score_youtube_items(items: List[schema.YouTubeItem]) -> List[schema.YouTubeItem]:
"""Compute scores for YouTube items.
Uses same weight structure as Reddit/X (relevance + recency + engagement).
"""
if not items:
return items
eng_raw = [compute_youtube_engagement_raw(item.engagement) for item in items]
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
item.score = max(0, min(100, int(overall))) item.score = max(0, min(100, int(overall)))
@@ -223,6 +286,11 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty. Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
This ensures WebSearch items rank below comparable Reddit/X items. This ensures WebSearch items rank below comparable Reddit/X items.
Date confidence adjustments:
- High confidence (URL-verified date): +10 bonus
- Med confidence (snippet-extracted date): no change
- Low confidence (no date signals): -20 penalty
Args: Args:
items: List of WebSearch items items: List of WebSearch items
@@ -255,18 +323,21 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency) # Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
overall -= WEBSEARCH_SOURCE_PENALTY overall -= WEBSEARCH_SOURCE_PENALTY
# Apply penalty for low date confidence # Apply date confidence adjustments
if item.date_confidence == "low": # High confidence (URL-verified): reward with bonus
overall -= 10 # Med confidence (snippet-extracted): neutral
elif item.date_confidence == "med": # Low confidence (no date signals): heavy penalty
overall -= 5 if item.date_confidence == "high":
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified recent dates
elif item.date_confidence == "low":
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
item.score = max(0, min(100, int(overall))) item.score = max(0, min(100, int(overall)))
return items return items
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem]]) -> List: def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem]]) -> List:
"""Sort items by score (descending), then date, then source priority. """Sort items by score (descending), then date, then source priority.
Args: Args:
@@ -283,13 +354,15 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
date = item.date or "0000-00-00" date = item.date or "0000-00-00"
date_key = -int(date.replace("-", "")) date_key = -int(date.replace("-", ""))
# Tertiary: source priority (Reddit > X > WebSearch) # Tertiary: source priority (Reddit > X > YouTube > WebSearch)
if isinstance(item, schema.RedditItem): if isinstance(item, schema.RedditItem):
source_priority = 0 source_priority = 0
elif isinstance(item, schema.XItem): elif isinstance(item, schema.XItem):
source_priority = 1 source_priority = 1
else: # WebSearchItem elif isinstance(item, schema.YouTubeItem):
source_priority = 2 source_priority = 2
else: # WebSearchItem
source_priority = 3
# Quaternary: title/text for stability # Quaternary: title/text for stability
text = getattr(item, "title", "") or getattr(item, "text", "") text = getattr(item, "title", "") or getattr(item, "text", "")
+229 -3
View File
@@ -64,6 +64,14 @@ ENRICHING_MESSAGES = [
"Analyzing discussions...", "Analyzing discussions...",
] ]
YOUTUBE_MESSAGES = [
"Searching YouTube for videos...",
"Finding relevant video content...",
"Scanning YouTube channels...",
"Discovering video discussions...",
"Fetching transcripts...",
]
PROCESSING_MESSAGES = [ PROCESSING_MESSAGES = [
"Crunching the data...", "Crunching the data...",
"Scoring and ranking...", "Scoring and ranking...",
@@ -72,6 +80,64 @@ PROCESSING_MESSAGES = [
"Organizing findings...", "Organizing findings...",
] ]
WEB_ONLY_MESSAGES = [
"Searching the web...",
"Finding blogs and docs...",
"Crawling news sites...",
"Discovering tutorials...",
]
def _build_nux_message(diag: dict = None) -> str:
"""Build conversational NUX message with dynamic source status."""
if diag:
reddit = "" if diag.get("openai") else ""
x = "" if diag.get("x_source") else ""
youtube = "" if diag.get("youtube") else ""
web = "" if diag.get("web_search_backend") else ""
status_line = f"Reddit {reddit}, X {x}, YouTube {youtube}, Web {web}"
else:
status_line = "YouTube ✓, Web ✓, Reddit ✗, X ✗"
return f"""
I just researched that for you. Here's what I've got right now:
{status_line}
You can unlock more sources with API keys just ask me how and I'll walk you through it. More sources means better research, but it works fine as-is.
Some examples of what you can do:
- "last30 what are people saying about Figma"
- "last30 watch my biggest competitor every week"
- "last30 watch Peter Steinberger every 30 days"
- "last30 watch AI video tools monthly"
- "last30 what have you found about AI video?"
Just start with "last30" and talk to me like normal.
"""
# Shorter promo for single missing key
PROMO_SINGLE_KEY = {
"reddit": "\n💡 You can unlock Reddit with an OpenAI API key — just ask me how.\n",
"x": "\n💡 You can unlock X with an xAI API key — just ask me how.\n",
}
# Bird auth help (for local users with vendored Bird CLI)
BIRD_AUTH_HELP = f"""
{Colors.YELLOW}Bird authentication failed.{Colors.RESET}
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Try again Bird reads your browser cookies automatically.
"""
BIRD_AUTH_HELP_PLAIN = """
Bird authentication failed.
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Try again Bird reads your browser cookies automatically.
"""
# Spinner frames # Spinner frames
SPINNER_FRAMES = ['', '', '', '', '', '', '', '', '', ''] SPINNER_FRAMES = ['', '', '', '', '', '', '', '', '', '']
DOTS_FRAMES = [' ', '. ', '.. ', '...'] DOTS_FRAMES = [' ', '. ', '.. ', '...']
@@ -182,6 +248,15 @@ class ProgressDisplay:
if self.spinner: if self.spinner:
self.spinner.stop(f"{Colors.CYAN}X{Colors.RESET} Found {count} posts") self.spinner.stop(f"{Colors.CYAN}X{Colors.RESET} Found {count} posts")
def start_youtube(self):
msg = random.choice(YOUTUBE_MESSAGES)
self.spinner = Spinner(f"{Colors.RED}YouTube{Colors.RESET} {msg}", Colors.RED)
self.spinner.start()
def end_youtube(self, count: int):
if self.spinner:
self.spinner.stop(f"{Colors.RED}YouTube{Colors.RESET} Found {count} videos")
def start_processing(self): def start_processing(self):
msg = random.choice(PROCESSING_MESSAGES) msg = random.choice(PROCESSING_MESSAGES)
self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE) self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE)
@@ -191,15 +266,21 @@ class ProgressDisplay:
if self.spinner: if self.spinner:
self.spinner.stop() self.spinner.stop()
def show_complete(self, reddit_count: int, x_count: int): def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0):
elapsed = time.time() - self.start_time elapsed = time.time() - self.start_time
if IS_TTY: if IS_TTY:
sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ") sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n") sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n")
sys.stderr.write(f" {Colors.YELLOW}Reddit:{Colors.RESET} {reddit_count} threads ") sys.stderr.write(f" {Colors.YELLOW}Reddit:{Colors.RESET} {reddit_count} threads ")
sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts\n\n") sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts")
if youtube_count:
sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
sys.stderr.write("\n\n")
else: else:
sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - Reddit: {reddit_count} threads, X: {x_count} posts\n") parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"]
if youtube_count:
parts.append(f"YouTube: {youtube_count} videos")
sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n")
sys.stderr.flush() sys.stderr.flush()
def show_cached(self, age_hours: float = None): def show_cached(self, age_hours: float = None):
@@ -214,6 +295,151 @@ class ProgressDisplay:
sys.stderr.write(f"{Colors.RED}✗ Error:{Colors.RESET} {message}\n") sys.stderr.write(f"{Colors.RED}✗ Error:{Colors.RESET} {message}\n")
sys.stderr.flush() sys.stderr.flush()
def start_web_only(self):
"""Show web-only mode indicator."""
msg = random.choice(WEB_ONLY_MESSAGES)
self.spinner = Spinner(f"{Colors.GREEN}Web{Colors.RESET} {msg}", Colors.GREEN)
self.spinner.start()
def end_web_only(self):
"""End web-only spinner."""
if self.spinner:
self.spinner.stop(f"{Colors.GREEN}Web{Colors.RESET} assistant will search the web")
def show_web_only_complete(self):
"""Show completion for web-only mode."""
elapsed = time.time() - self.start_time
if IS_TTY:
sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Ready for web search{Colors.RESET} ")
sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n")
sys.stderr.write(f" {Colors.GREEN}Web:{Colors.RESET} assistant will search blogs, docs & news\n\n")
else:
sys.stderr.write(f"✓ Ready for web search ({elapsed:.1f}s)\n")
sys.stderr.flush()
def show_promo(self, missing: str = "both", diag: dict = None):
"""Show NUX / promotional message for missing API keys.
Args:
missing: 'both', 'all', 'reddit', or 'x' - which keys are missing
diag: Optional diagnostics dict for dynamic source status
"""
if missing in ("both", "all"):
sys.stderr.write(_build_nux_message(diag))
elif missing in PROMO_SINGLE_KEY:
sys.stderr.write(PROMO_SINGLE_KEY[missing])
sys.stderr.flush()
def show_bird_auth_help(self):
"""Show Bird authentication help."""
if IS_TTY:
sys.stderr.write(BIRD_AUTH_HELP)
else:
sys.stderr.write(BIRD_AUTH_HELP_PLAIN)
sys.stderr.flush()
def show_diagnostic_banner(diag: dict):
"""Show pre-flight source status banner when sources are missing.
Args:
diag: Dict from env diagnostics with keys:
openai, xai, x_source, bird_installed, bird_authenticated,
bird_username, youtube, web_search_backend
"""
has_openai = diag.get("openai", False)
has_x = diag.get("x_source") is not None
has_youtube = diag.get("youtube", False)
has_web = diag.get("web_search_backend") is not None
# If everything is available, no banner needed
if has_openai and has_x and has_youtube and has_web:
return
lines = []
if IS_TTY:
lines.append(f"{Colors.DIM}┌─────────────────────────────────────────────────────┐{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}/last30days v2.1 — Source Status{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
# Reddit
if has_openai:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ Reddit{Colors.RESET} — OPENAI_API_KEY found {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ Reddit{Colors.RESET} — No OPENAI_API_KEY {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Add to ~/.config/last30days/.env {Colors.DIM}{Colors.RESET}")
# X/Twitter
if has_x:
source = diag.get("x_source", "")
username = diag.get("bird_username", "")
label = f"Bird ({username})" if source == "bird" and username else source.upper()
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ X/Twitter{Colors.RESET}{label} {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ X/Twitter{Colors.RESET} — No Bird CLI or XAI_API_KEY {Colors.DIM}{Colors.RESET}")
if diag.get("bird_installed"):
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Bird installed but not authenticated {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Log into x.com in your browser, then retry {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Needs Node.js 22+ (Bird is bundled) {Colors.DIM}{Colors.RESET}")
# YouTube
if has_youtube:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ YouTube{Colors.RESET} — yt-dlp found {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ YouTube{Colors.RESET} — yt-dlp not installed {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Fix: brew install yt-dlp (free) {Colors.DIM}{Colors.RESET}")
# Web
if has_web:
backend = diag.get("web_search_backend", "")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ Web{Colors.RESET}{backend} API {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.YELLOW}⚡ Web{Colors.RESET} — Using assistant's search tool {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} Config: {Colors.BOLD}~/.config/last30days/.env{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}└─────────────────────────────────────────────────────┘{Colors.RESET}")
else:
# Plain text for non-TTY (Claude Code / Codex)
lines.append("┌─────────────────────────────────────────────────────┐")
lines.append("│ /last30days v2.1 — Source Status │")
lines.append("│ │")
if has_openai:
lines.append("│ ✅ Reddit — OPENAI_API_KEY found │")
else:
lines.append("│ ❌ Reddit — No OPENAI_API_KEY │")
lines.append("│ └─ Add to ~/.config/last30days/.env │")
if has_x:
lines.append("│ ✅ X/Twitter — available │")
else:
lines.append("│ ❌ X/Twitter — No Bird CLI or XAI_API_KEY │")
if diag.get("bird_installed"):
lines.append("│ └─ Log into x.com in your browser, then retry │")
else:
lines.append("│ └─ Needs Node.js 22+ (Bird is bundled) │")
if has_youtube:
lines.append("│ ✅ YouTube — yt-dlp found │")
else:
lines.append("│ ❌ YouTube — yt-dlp not installed │")
lines.append("│ └─ Fix: brew install yt-dlp (free) │")
if has_web:
lines.append("│ ✅ Web — API search available │")
else:
lines.append("│ ⚡ Web — Using assistant's search tool │")
lines.append("│ │")
lines.append("│ Config: ~/.config/last30days/.env │")
lines.append("└─────────────────────────────────────────────────────┘")
sys.stderr.write("\n".join(lines) + "\n\n")
sys.stderr.flush()
def print_phase(phase: str, message: str): def print_phase(phase: str, message: str):
"""Print a phase message.""" """Print a phase message."""
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Peter Steinberger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+134
View File
@@ -0,0 +1,134 @@
#!/usr/bin/env node
/**
* bird-search.mjs - Vendored Bird CLI search wrapper for /last30days.
* Subset of @steipete/bird v0.8.0 (MIT License, Peter Steinberger).
*
* Usage:
* node bird-search.mjs <query> [--count N] [--json]
* node bird-search.mjs --whoami
* node bird-search.mjs --check
*/
import { resolveCredentials } from './lib/cookies.js';
import { TwitterClientBase } from './lib/twitter-client-base.js';
import { withSearch } from './lib/twitter-client-search.js';
// Build a search-only client (no posting, bookmarks, etc.)
const SearchClient = withSearch(TwitterClientBase);
const args = process.argv.slice(2);
// --check: verify that credentials can be resolved
if (args.includes('--check')) {
try {
const { cookies, warnings } = await resolveCredentials({});
if (cookies.authToken && cookies.ct0) {
process.stdout.write(JSON.stringify({ authenticated: true, source: cookies.source }));
process.exit(0);
} else {
process.stdout.write(JSON.stringify({ authenticated: false, warnings }));
process.exit(1);
}
} catch (err) {
process.stdout.write(JSON.stringify({ authenticated: false, error: err.message }));
process.exit(1);
}
}
// --whoami: check auth and output source
if (args.includes('--whoami')) {
try {
const { cookies } = await resolveCredentials({});
if (cookies.authToken && cookies.ct0) {
process.stdout.write(cookies.source || 'authenticated');
process.exit(0);
} else {
process.stderr.write('Not authenticated\n');
process.exit(1);
}
} catch (err) {
process.stderr.write(`Auth check failed: ${err.message}\n`);
process.exit(1);
}
}
// Parse search args
let query = null;
let count = 20;
let jsonOutput = false;
for (let i = 0; i < args.length; i++) {
if (args[i] === '--count' && args[i + 1]) {
count = parseInt(args[i + 1], 10);
i++;
} else if (args[i] === '-n' && args[i + 1]) {
count = parseInt(args[i + 1], 10);
i++;
} else if (args[i] === '--json') {
jsonOutput = true;
} else if (!args[i].startsWith('-')) {
query = args[i];
}
}
if (!query) {
process.stderr.write('Usage: node bird-search.mjs <query> [--count N] [--json]\n');
process.exit(1);
}
try {
// Resolve credentials (env vars, then browser cookies)
const { cookies, warnings } = await resolveCredentials({});
if (!cookies.authToken || !cookies.ct0) {
const msg = warnings.length > 0 ? warnings.join('; ') : 'No Twitter credentials found';
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: msg, items: [] }));
} else {
process.stderr.write(`Error: ${msg}\n`);
}
process.exit(1);
}
// Create search client
const client = new SearchClient({
cookies: {
authToken: cookies.authToken,
ct0: cookies.ct0,
cookieHeader: cookies.cookieHeader,
},
timeoutMs: 30000,
});
// Run search
const result = await client.search(query, count);
if (!result.success) {
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: result.error, items: [] }));
} else {
process.stderr.write(`Search failed: ${result.error}\n`);
}
process.exit(1);
}
// Output results
const tweets = result.tweets || [];
if (jsonOutput) {
process.stdout.write(JSON.stringify(tweets));
} else {
for (const tweet of tweets) {
const author = tweet.author?.username || 'unknown';
process.stdout.write(`@${author}: ${tweet.text?.slice(0, 200)}\n\n`);
}
}
process.exit(0);
} catch (err) {
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: err.message, items: [] }));
} else {
process.stderr.write(`Error: ${err.message}\n`);
}
process.exit(1);
}
+173
View File
@@ -0,0 +1,173 @@
/**
* Browser cookie extraction for Twitter authentication.
* Delegates to @steipete/sweet-cookie for Safari/Chrome/Firefox reads.
*/
import { getCookies } from '@steipete/sweet-cookie';
const TWITTER_COOKIE_NAMES = ['auth_token', 'ct0'];
const TWITTER_URL = 'https://x.com/';
const TWITTER_ORIGINS = ['https://x.com/', 'https://twitter.com/'];
const DEFAULT_COOKIE_TIMEOUT_MS = 30_000;
function normalizeValue(value) {
if (typeof value !== 'string') {
return null;
}
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : null;
}
function cookieHeader(authToken, ct0) {
return `auth_token=${authToken}; ct0=${ct0}`;
}
function buildEmpty() {
return { authToken: null, ct0: null, cookieHeader: null, source: null };
}
function readEnvCookie(cookies, keys, field) {
if (cookies[field]) {
return;
}
for (const key of keys) {
const value = normalizeValue(process.env[key]);
if (!value) {
continue;
}
cookies[field] = value;
if (!cookies.source) {
cookies.source = `env ${key}`;
}
break;
}
}
function resolveSources(cookieSource) {
if (Array.isArray(cookieSource)) {
return cookieSource;
}
if (cookieSource) {
return [cookieSource];
}
return ['safari', 'chrome', 'firefox'];
}
function labelForSource(source, profile) {
if (source === 'safari') {
return 'Safari';
}
if (source === 'chrome') {
return profile ? `Chrome profile "${profile}"` : 'Chrome default profile';
}
return profile ? `Firefox profile "${profile}"` : 'Firefox default profile';
}
function pickCookieValue(cookies, name) {
const matches = cookies.filter((c) => c?.name === name && typeof c.value === 'string');
if (matches.length === 0) {
return null;
}
const preferred = matches.find((c) => (c.domain ?? '').endsWith('x.com'));
if (preferred?.value) {
return preferred.value;
}
const twitter = matches.find((c) => (c.domain ?? '').endsWith('twitter.com'));
if (twitter?.value) {
return twitter.value;
}
return matches[0]?.value ?? null;
}
async function readTwitterCookiesFromBrowser(options) {
const warnings = [];
const out = buildEmpty();
const { cookies, warnings: providerWarnings } = await getCookies({
url: TWITTER_URL,
origins: TWITTER_ORIGINS,
names: [...TWITTER_COOKIE_NAMES],
browsers: [options.source],
mode: 'merge',
chromeProfile: options.chromeProfile,
firefoxProfile: options.firefoxProfile,
timeoutMs: options.cookieTimeoutMs,
});
warnings.push(...providerWarnings);
const authToken = pickCookieValue(cookies, 'auth_token');
const ct0 = pickCookieValue(cookies, 'ct0');
if (authToken) {
out.authToken = authToken;
}
if (ct0) {
out.ct0 = ct0;
}
if (out.authToken && out.ct0) {
out.cookieHeader = cookieHeader(out.authToken, out.ct0);
out.source = labelForSource(options.source, options.source === 'chrome' ? options.chromeProfile : options.firefoxProfile);
return { cookies: out, warnings };
}
if (options.source === 'safari') {
warnings.push('No Twitter cookies found in Safari. Make sure you are logged into x.com in Safari.');
}
else if (options.source === 'chrome') {
warnings.push('No Twitter cookies found in Chrome. Make sure you are logged into x.com in Chrome.');
}
else {
warnings.push('No Twitter cookies found in Firefox. Make sure you are logged into x.com in Firefox and the profile exists.');
}
return { cookies: out, warnings };
}
export async function extractCookiesFromSafari() {
return readTwitterCookiesFromBrowser({ source: 'safari' });
}
export async function extractCookiesFromChrome(profile) {
return readTwitterCookiesFromBrowser({ source: 'chrome', chromeProfile: profile });
}
export async function extractCookiesFromFirefox(profile) {
return readTwitterCookiesFromBrowser({ source: 'firefox', firefoxProfile: profile });
}
/**
* Resolve Twitter credentials from multiple sources.
* Priority: CLI args > environment variables > browsers (ordered).
*/
export async function resolveCredentials(options) {
const warnings = [];
const cookies = buildEmpty();
const cookieTimeoutMs = typeof options.cookieTimeoutMs === 'number' &&
Number.isFinite(options.cookieTimeoutMs) &&
options.cookieTimeoutMs > 0
? options.cookieTimeoutMs
: process.platform === 'darwin'
? DEFAULT_COOKIE_TIMEOUT_MS
: undefined;
if (options.authToken) {
cookies.authToken = options.authToken;
cookies.source = 'CLI argument';
}
if (options.ct0) {
cookies.ct0 = options.ct0;
if (!cookies.source) {
cookies.source = 'CLI argument';
}
}
readEnvCookie(cookies, ['AUTH_TOKEN', 'TWITTER_AUTH_TOKEN'], 'authToken');
readEnvCookie(cookies, ['CT0', 'TWITTER_CT0'], 'ct0');
if (cookies.authToken && cookies.ct0) {
cookies.cookieHeader = cookieHeader(cookies.authToken, cookies.ct0);
return { cookies, warnings };
}
const sourcesToTry = resolveSources(options.cookieSource);
for (const source of sourcesToTry) {
const res = await readTwitterCookiesFromBrowser({
source,
chromeProfile: options.chromeProfile,
firefoxProfile: options.firefoxProfile,
cookieTimeoutMs,
});
warnings.push(...res.warnings);
if (res.cookies.authToken && res.cookies.ct0) {
return { cookies: res.cookies, warnings };
}
}
if (!cookies.authToken) {
warnings.push('Missing auth_token - provide via --auth-token, AUTH_TOKEN env var, or login to x.com in Safari/Chrome/Firefox');
}
if (!cookies.ct0) {
warnings.push('Missing ct0 - provide via --ct0, CT0 env var, or login to x.com in Safari/Chrome/Firefox');
}
if (cookies.authToken && cookies.ct0) {
cookies.cookieHeader = cookieHeader(cookies.authToken, cookies.ct0);
}
return { cookies, warnings };
}
//# sourceMappingURL=cookies.js.map
+17
View File
@@ -0,0 +1,17 @@
{
"global": {
"responsive_web_grok_annotations_enabled": false,
"post_ctas_fetch_enabled": true,
"responsive_web_graphql_exclude_directive_enabled": true
},
"sets": {
"lists": {
"blue_business_profile_image_shape_enabled": true,
"tweetypie_unmention_optimization_enabled": true,
"responsive_web_text_conversations_enabled": false,
"interactive_text_enabled": true,
"vibe_api_enabled": true,
"responsive_web_twitter_blue_verified_badge_is_enabled": true
}
}
}
+37
View File
@@ -0,0 +1,37 @@
export async function paginateCursor(opts) {
const { maxPages, pageDelayMs = 1000 } = opts;
const seen = new Set();
const items = [];
let cursor = opts.cursor;
let pagesFetched = 0;
while (true) {
if (pagesFetched > 0 && pageDelayMs > 0) {
await opts.sleep(pageDelayMs);
}
const page = await opts.fetchPage(cursor);
if (!page.success) {
if (items.length > 0) {
return { success: false, error: page.error, items, nextCursor: cursor };
}
return page;
}
pagesFetched += 1;
for (const item of page.items) {
const key = opts.getKey(item);
if (seen.has(key)) {
continue;
}
seen.add(key);
items.push(item);
}
const pageCursor = page.cursor;
if (!pageCursor || pageCursor === cursor) {
return { success: true, items, nextCursor: undefined };
}
if (maxPages !== undefined && pagesFetched >= maxPages) {
return { success: true, items, nextCursor: pageCursor };
}
cursor = pageCursor;
}
}
//# sourceMappingURL=paginate-cursor.js.map
+20
View File
@@ -0,0 +1,20 @@
{
"CreateTweet": "nmdAQXJDxw6-0KKF2on7eA",
"CreateRetweet": "LFho5rIi4xcKO90p9jwG7A",
"CreateFriendship": "8h9JVdV8dlSyqyRDJEPCsA",
"DestroyFriendship": "ppXWuagMNXgvzx6WoXBW0Q",
"FavoriteTweet": "lI07N6Otwv1PhnEgXILM7A",
"DeleteBookmark": "Wlmlj2-xzyS1GN3a6cj-mQ",
"TweetDetail": "_NvJCnIjOW__EP5-RF197A",
"SearchTimeline": "6AAys3t42mosm_yTI_QENg",
"Bookmarks": "RV1g3b8n_SGOHwkqKYSCFw",
"BookmarkFolderTimeline": "KJIQpsvxrTfRIlbaRIySHQ",
"Following": "mWYeougg_ocJS2Vr1Vt28w",
"Followers": "SFYY3WsgwjlXSLlfnEUE4A",
"Likes": "ETJflBunfqNa1uE1mBPCaw",
"ExploreSidebar": "lpSN4M6qpimkF4nRFPE3nQ",
"ExplorePage": "kheAINB_4pzRDqkzG3K-ng",
"GenericTimelineById": "uGSr7alSjR9v6QJAIaqSKQ",
"TrendHistory": "Sj4T-jSB9pr0Mxtsc1UKZQ",
"AboutAccountQuery": "zs_jFPFT78rBpXv9Z3U2YQ"
}
+151
View File
@@ -0,0 +1,151 @@
import { existsSync, readFileSync } from 'node:fs';
import { mkdir, writeFile } from 'node:fs/promises';
import { homedir } from 'node:os';
import path from 'node:path';
// biome-ignore lint/correctness/useImportExtensions: JSON module import doesn't use .js extension.
import defaultOverrides from './features.json' with { type: 'json' };
const DEFAULT_CACHE_FILENAME = 'features.json';
let cachedOverrides = null;
function normalizeFeatureMap(value) {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
return {};
}
const result = {};
for (const [key, entry] of Object.entries(value)) {
if (typeof entry === 'boolean') {
result[key] = entry;
}
}
return result;
}
function normalizeOverrides(value) {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
return { global: {}, sets: {} };
}
const record = value;
const global = normalizeFeatureMap(record.global);
const sets = {};
const rawSets = record.sets && typeof record.sets === 'object' && !Array.isArray(record.sets)
? record.sets
: {};
for (const [setName, setValue] of Object.entries(rawSets)) {
const normalized = normalizeFeatureMap(setValue);
if (Object.keys(normalized).length > 0) {
sets[setName] = normalized;
}
}
return { global, sets };
}
function mergeOverrides(base, next) {
const sets = { ...base.sets };
for (const [setName, overrides] of Object.entries(next.sets)) {
const existing = sets[setName];
sets[setName] = existing ? { ...existing, ...overrides } : { ...overrides };
}
return {
global: { ...base.global, ...next.global },
sets,
};
}
function toFeatureOverrides(overrides) {
const result = {};
if (Object.keys(overrides.global).length > 0) {
result.global = overrides.global;
}
const setEntries = Object.entries(overrides.sets).filter(([, value]) => Object.keys(value).length > 0);
if (setEntries.length > 0) {
result.sets = Object.fromEntries(setEntries);
}
return result;
}
function resolveFeaturesCachePath() {
const override = process.env.BIRD_FEATURES_CACHE ?? process.env.BIRD_FEATURES_PATH;
if (override && override.trim().length > 0) {
return path.resolve(override.trim());
}
return path.join(homedir(), '.config', 'bird', DEFAULT_CACHE_FILENAME);
}
function readOverridesFromFile(cachePath) {
if (!existsSync(cachePath)) {
return null;
}
try {
const raw = readFileSync(cachePath, 'utf8');
return normalizeOverrides(JSON.parse(raw));
}
catch {
return null;
}
}
function readOverridesFromEnv() {
const raw = process.env.BIRD_FEATURES_JSON;
if (!raw || raw.trim().length === 0) {
return null;
}
try {
return normalizeOverrides(JSON.parse(raw));
}
catch {
return null;
}
}
function writeOverridesToDisk(cachePath, overrides) {
const payload = toFeatureOverrides(overrides);
return mkdir(path.dirname(cachePath), { recursive: true }).then(() => writeFile(cachePath, `${JSON.stringify(payload, null, 2)}\n`, 'utf8'));
}
export function loadFeatureOverrides() {
if (cachedOverrides) {
return cachedOverrides;
}
const base = normalizeOverrides(defaultOverrides);
const fromFile = readOverridesFromFile(resolveFeaturesCachePath());
const fromEnv = readOverridesFromEnv();
let merged = base;
if (fromFile) {
merged = mergeOverrides(merged, fromFile);
}
if (fromEnv) {
merged = mergeOverrides(merged, fromEnv);
}
cachedOverrides = merged;
return merged;
}
export function getFeatureOverridesSnapshot() {
const overrides = toFeatureOverrides(loadFeatureOverrides());
return {
cachePath: resolveFeaturesCachePath(),
overrides,
};
}
export function applyFeatureOverrides(setName, base) {
const overrides = loadFeatureOverrides();
const globalOverrides = overrides.global;
const setOverrides = overrides.sets[setName];
if (Object.keys(globalOverrides).length === 0 && (!setOverrides || Object.keys(setOverrides).length === 0)) {
return base;
}
if (setOverrides) {
return {
...base,
...globalOverrides,
...setOverrides,
};
}
return {
...base,
...globalOverrides,
};
}
export async function refreshFeatureOverridesCache() {
const cachePath = resolveFeaturesCachePath();
const base = normalizeOverrides(defaultOverrides);
const fromFile = readOverridesFromFile(cachePath);
const merged = mergeOverrides(base, fromFile ?? { global: {}, sets: {} });
await writeOverridesToDisk(cachePath, merged);
cachedOverrides = null;
return { cachePath, overrides: toFeatureOverrides(merged) };
}
export function clearFeatureOverridesCache() {
cachedOverrides = null;
}
//# sourceMappingURL=runtime-features.js.map
+264
View File
@@ -0,0 +1,264 @@
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { homedir } from 'node:os';
import path from 'node:path';
const DEFAULT_CACHE_FILENAME = 'query-ids-cache.json';
const DEFAULT_TTL_MS = 24 * 60 * 60 * 1000;
const DISCOVERY_PAGES = [
'https://x.com/?lang=en',
'https://x.com/explore',
'https://x.com/notifications',
'https://x.com/settings/profile',
];
const BUNDLE_URL_REGEX = /https:\/\/abs\.twimg\.com\/responsive-web\/client-web(?:-legacy)?\/[A-Za-z0-9.-]+\.js/g;
const QUERY_ID_REGEX = /^[a-zA-Z0-9_-]+$/;
const OPERATION_PATTERNS = [
{
regex: /e\.exports=\{queryId\s*:\s*["']([^"']+)["']\s*,\s*operationName\s*:\s*["']([^"']+)["']/gs,
operationGroup: 2,
queryIdGroup: 1,
},
{
regex: /e\.exports=\{operationName\s*:\s*["']([^"']+)["']\s*,\s*queryId\s*:\s*["']([^"']+)["']/gs,
operationGroup: 1,
queryIdGroup: 2,
},
{
regex: /operationName\s*[:=]\s*["']([^"']+)["'](.{0,4000}?)queryId\s*[:=]\s*["']([^"']+)["']/gs,
operationGroup: 1,
queryIdGroup: 3,
},
{
regex: /queryId\s*[:=]\s*["']([^"']+)["'](.{0,4000}?)operationName\s*[:=]\s*["']([^"']+)["']/gs,
operationGroup: 3,
queryIdGroup: 1,
},
];
const HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',
Accept: 'text/html,application/json;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
};
async function fetchText(fetchImpl, url) {
const response = await fetchImpl(url, { headers: HEADERS });
if (!response.ok) {
const body = await response.text().catch(() => '');
throw new Error(`HTTP ${response.status} for ${url}: ${body.slice(0, 120)}`);
}
return response.text();
}
function resolveDefaultCachePath() {
const override = process.env.BIRD_QUERY_IDS_CACHE;
if (override && override.trim().length > 0) {
return path.resolve(override.trim());
}
return path.join(homedir(), '.config', 'bird', DEFAULT_CACHE_FILENAME);
}
function parseSnapshot(raw) {
if (!raw || typeof raw !== 'object') {
return null;
}
const record = raw;
const fetchedAt = typeof record.fetchedAt === 'string' ? record.fetchedAt : null;
const ttlMs = typeof record.ttlMs === 'number' && Number.isFinite(record.ttlMs) ? record.ttlMs : null;
const ids = record.ids && typeof record.ids === 'object' ? record.ids : null;
const discovery = record.discovery && typeof record.discovery === 'object' ? record.discovery : null;
if (!fetchedAt || !ttlMs || !ids || !discovery) {
return null;
}
const pages = Array.isArray(discovery.pages) ? discovery.pages : null;
const bundles = Array.isArray(discovery.bundles) ? discovery.bundles : null;
if (!pages || !bundles) {
return null;
}
const normalizedIds = {};
for (const [key, value] of Object.entries(ids)) {
if (typeof value === 'string' && value.trim().length > 0) {
normalizedIds[key] = value.trim();
}
}
return {
fetchedAt,
ttlMs,
ids: normalizedIds,
discovery: {
pages: pages.filter((p) => typeof p === 'string'),
bundles: bundles.filter((b) => typeof b === 'string'),
},
};
}
async function readSnapshotFromDisk(cachePath) {
try {
const raw = await readFile(cachePath, 'utf8');
return parseSnapshot(JSON.parse(raw));
}
catch {
return null;
}
}
async function writeSnapshotToDisk(cachePath, snapshot) {
await mkdir(path.dirname(cachePath), { recursive: true });
await writeFile(cachePath, `${JSON.stringify(snapshot, null, 2)}\n`, 'utf8');
}
async function discoverBundles(fetchImpl) {
const bundles = new Set();
for (const page of DISCOVERY_PAGES) {
try {
const html = await fetchText(fetchImpl, page);
for (const match of html.matchAll(BUNDLE_URL_REGEX)) {
bundles.add(match[0]);
}
}
catch {
// ignore discovery page failures; other pages often work
}
}
const discovered = [...bundles];
if (discovered.length === 0) {
throw new Error('No client bundles discovered; x.com layout may have changed.');
}
return discovered;
}
function extractOperations(bundleContents, bundleLabel, targets, discovered) {
for (const pattern of OPERATION_PATTERNS) {
pattern.regex.lastIndex = 0;
while (true) {
const match = pattern.regex.exec(bundleContents);
if (match === null) {
break;
}
const operationName = match[pattern.operationGroup];
const queryId = match[pattern.queryIdGroup];
if (!operationName || !queryId) {
continue;
}
if (!targets.has(operationName)) {
continue;
}
if (!QUERY_ID_REGEX.test(queryId)) {
continue;
}
if (discovered.has(operationName)) {
continue;
}
discovered.set(operationName, { queryId, bundle: bundleLabel });
if (discovered.size === targets.size) {
return;
}
}
}
}
async function fetchAndExtract(fetchImpl, bundleUrls, targets) {
const discovered = new Map();
const CONCURRENCY = 6;
for (let i = 0; i < bundleUrls.length; i += CONCURRENCY) {
const chunk = bundleUrls.slice(i, i + CONCURRENCY);
await Promise.all(chunk.map(async (url) => {
if (discovered.size === targets.size) {
return;
}
const label = url.split('/').at(-1) ?? url;
try {
const js = await fetchText(fetchImpl, url);
extractOperations(js, label, targets, discovered);
}
catch {
// ignore failed bundles
}
}));
if (discovered.size === targets.size) {
break;
}
}
return discovered;
}
export function createRuntimeQueryIdStore(options = {}) {
const fetchImpl = options.fetchImpl ?? fetch;
const ttlMs = options.ttlMs ?? DEFAULT_TTL_MS;
const cachePath = options.cachePath ? path.resolve(options.cachePath) : resolveDefaultCachePath();
let memorySnapshot = null;
let loadOnce = null;
let refreshInFlight = null;
const loadSnapshot = async () => {
if (memorySnapshot) {
return memorySnapshot;
}
if (!loadOnce) {
loadOnce = (async () => {
const fromDisk = await readSnapshotFromDisk(cachePath);
memorySnapshot = fromDisk;
return fromDisk;
})();
}
return loadOnce;
};
const getSnapshotInfo = async () => {
const snapshot = await loadSnapshot();
if (!snapshot) {
return null;
}
const fetchedAtMs = new Date(snapshot.fetchedAt).getTime();
const ageMs = Number.isFinite(fetchedAtMs) ? Math.max(0, Date.now() - fetchedAtMs) : Number.POSITIVE_INFINITY;
const effectiveTtl = Number.isFinite(snapshot.ttlMs) ? snapshot.ttlMs : ttlMs;
const isFresh = ageMs <= effectiveTtl;
return { snapshot, cachePath, ageMs, isFresh };
};
const getQueryId = async (operationName) => {
const info = await getSnapshotInfo();
if (!info) {
return null;
}
return info.snapshot.ids[operationName] ?? null;
};
const refresh = async (operationNames, opts = {}) => {
if (refreshInFlight) {
return refreshInFlight;
}
refreshInFlight = (async () => {
const current = await getSnapshotInfo();
if (!opts.force && current?.isFresh) {
return current;
}
const targets = new Set(operationNames);
const bundleUrls = await discoverBundles(fetchImpl);
const discovered = await fetchAndExtract(fetchImpl, bundleUrls, targets);
if (discovered.size === 0) {
return current ?? null;
}
const ids = {};
for (const name of operationNames) {
const entry = discovered.get(name);
if (entry?.queryId) {
ids[name] = entry.queryId;
}
}
const snapshot = {
fetchedAt: new Date().toISOString(),
ttlMs,
ids,
discovery: {
pages: [...DISCOVERY_PAGES],
bundles: bundleUrls.map((url) => url.split('/').at(-1) ?? url),
},
};
await writeSnapshotToDisk(cachePath, snapshot);
memorySnapshot = snapshot;
return getSnapshotInfo();
})().finally(() => {
refreshInFlight = null;
});
return refreshInFlight;
};
return {
cachePath,
ttlMs,
getSnapshotInfo,
getQueryId,
refresh,
clearMemory() {
memorySnapshot = null;
loadOnce = null;
},
};
}
export const runtimeQueryIds = createRuntimeQueryIdStore();
//# sourceMappingURL=runtime-query-ids.js.map
@@ -0,0 +1,129 @@
import { randomBytes, randomUUID } from 'node:crypto';
import { runtimeQueryIds } from './runtime-query-ids.js';
import { QUERY_IDS, TARGET_QUERY_ID_OPERATIONS } from './twitter-client-constants.js';
import { normalizeQuoteDepth } from './twitter-client-utils.js';
export class TwitterClientBase {
authToken;
ct0;
cookieHeader;
userAgent;
timeoutMs;
quoteDepth;
clientUuid;
clientDeviceId;
clientUserId;
constructor(options) {
if (!options.cookies.authToken || !options.cookies.ct0) {
throw new Error('Both authToken and ct0 cookies are required');
}
this.authToken = options.cookies.authToken;
this.ct0 = options.cookies.ct0;
this.cookieHeader = options.cookies.cookieHeader || `auth_token=${this.authToken}; ct0=${this.ct0}`;
this.userAgent =
options.userAgent ||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36';
this.timeoutMs = options.timeoutMs;
this.quoteDepth = normalizeQuoteDepth(options.quoteDepth);
this.clientUuid = randomUUID();
this.clientDeviceId = randomUUID();
}
async sleep(ms) {
await new Promise((resolve) => setTimeout(resolve, ms));
}
async getQueryId(operationName) {
const cached = await runtimeQueryIds.getQueryId(operationName);
return cached ?? QUERY_IDS[operationName];
}
async refreshQueryIds() {
if (process.env.NODE_ENV === 'test') {
return;
}
try {
await runtimeQueryIds.refresh(TARGET_QUERY_ID_OPERATIONS, { force: true });
}
catch {
// ignore refresh failures; callers will fall back to baked-in IDs
}
}
async withRefreshedQueryIdsOn404(attempt) {
const firstAttempt = await attempt();
if (firstAttempt.success || !firstAttempt.had404) {
return { result: firstAttempt, refreshed: false };
}
await this.refreshQueryIds();
const secondAttempt = await attempt();
return { result: secondAttempt, refreshed: true };
}
async getTweetDetailQueryIds() {
const primary = await this.getQueryId('TweetDetail');
return Array.from(new Set([primary, '97JF30KziU00483E_8elBA', 'aFvUsJm2c-oDkJV75blV6g']));
}
async getSearchTimelineQueryIds() {
const primary = await this.getQueryId('SearchTimeline');
return Array.from(new Set([primary, 'M1jEez78PEfVfbQLvlWMvQ', '5h0kNbk3ii97rmfY6CdgAA', 'Tp1sewRU1AsZpBWhqCZicQ']));
}
async fetchWithTimeout(url, init) {
if (!this.timeoutMs || this.timeoutMs <= 0) {
return fetch(url, init);
}
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), this.timeoutMs);
try {
return await fetch(url, { ...init, signal: controller.signal });
}
finally {
clearTimeout(timeoutId);
}
}
getHeaders() {
return this.getJsonHeaders();
}
createTransactionId() {
return randomBytes(16).toString('hex');
}
getBaseHeaders() {
const headers = {
accept: '*/*',
'accept-language': 'en-US,en;q=0.9',
authorization: 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA',
'x-csrf-token': this.ct0,
'x-twitter-auth-type': 'OAuth2Session',
'x-twitter-active-user': 'yes',
'x-twitter-client-language': 'en',
'x-client-uuid': this.clientUuid,
'x-twitter-client-deviceid': this.clientDeviceId,
'x-client-transaction-id': this.createTransactionId(),
cookie: this.cookieHeader,
'user-agent': this.userAgent,
origin: 'https://x.com',
referer: 'https://x.com/',
};
if (this.clientUserId) {
headers['x-twitter-client-user-id'] = this.clientUserId;
}
return headers;
}
getJsonHeaders() {
return {
...this.getBaseHeaders(),
'content-type': 'application/json',
};
}
getUploadHeaders() {
// Note: do not set content-type; URLSearchParams/FormData need to set it (incl boundary) themselves.
return this.getBaseHeaders();
}
async ensureClientUserId() {
if (process.env.NODE_ENV === 'test') {
return;
}
if (this.clientUserId) {
return;
}
const result = await this.getCurrentUser();
if (result.success && result.user?.id) {
this.clientUserId = result.user.id;
}
}
}
//# sourceMappingURL=twitter-client-base.js.map
@@ -0,0 +1,50 @@
// biome-ignore lint/correctness/useImportExtensions: JSON module import doesn't use .js extension.
import queryIds from './query-ids.json' with { type: 'json' };
export const TWITTER_API_BASE = 'https://x.com/i/api/graphql';
export const TWITTER_GRAPHQL_POST_URL = 'https://x.com/i/api/graphql';
export const TWITTER_UPLOAD_URL = 'https://upload.twitter.com/i/media/upload.json';
export const TWITTER_MEDIA_METADATA_URL = 'https://x.com/i/api/1.1/media/metadata/create.json';
export const TWITTER_STATUS_UPDATE_URL = 'https://x.com/i/api/1.1/statuses/update.json';
export const SETTINGS_SCREEN_NAME_REGEX = /"screen_name":"([^"]+)"/;
export const SETTINGS_USER_ID_REGEX = /"user_id"\s*:\s*"(\d+)"/;
export const SETTINGS_NAME_REGEX = /"name":"([^"\\]*(?:\\.[^"\\]*)*)"/;
// Query IDs rotate frequently; the values in query-ids.json are refreshed by
// scripts/update-query-ids.ts. The fallback values keep the client usable if
// the file is missing or incomplete.
export const FALLBACK_QUERY_IDS = {
CreateTweet: 'TAJw1rBsjAtdNgTdlo2oeg',
CreateRetweet: 'ojPdsZsimiJrUGLR1sjUtA',
DeleteRetweet: 'iQtK4dl5hBmXewYZuEOKVw',
CreateFriendship: '8h9JVdV8dlSyqyRDJEPCsA',
DestroyFriendship: 'ppXWuagMNXgvzx6WoXBW0Q',
FavoriteTweet: 'lI07N6Otwv1PhnEgXILM7A',
UnfavoriteTweet: 'ZYKSe-w7KEslx3JhSIk5LA',
CreateBookmark: 'aoDbu3RHznuiSkQ9aNM67Q',
DeleteBookmark: 'Wlmlj2-xzyS1GN3a6cj-mQ',
TweetDetail: '97JF30KziU00483E_8elBA',
SearchTimeline: 'M1jEez78PEfVfbQLvlWMvQ',
UserArticlesTweets: '8zBy9h4L90aDL02RsBcCFg',
UserTweets: 'Wms1GvIiHXAPBaCr9KblaA',
Bookmarks: 'RV1g3b8n_SGOHwkqKYSCFw',
Following: 'BEkNpEt5pNETESoqMsTEGA',
Followers: 'kuFUYP9eV1FPoEy4N-pi7w',
Likes: 'JR2gceKucIKcVNB_9JkhsA',
BookmarkFolderTimeline: 'KJIQpsvxrTfRIlbaRIySHQ',
ListOwnerships: 'wQcOSjSQ8NtgxIwvYl1lMg',
ListMemberships: 'BlEXXdARdSeL_0KyKHHvvg',
ListLatestTweetsTimeline: '2TemLyqrMpTeAmysdbnVqw',
ListByRestId: 'wXzyA5vM_aVkBL9G8Vp3kw',
HomeTimeline: 'edseUwk9sP5Phz__9TIRnA',
HomeLatestTimeline: 'iOEZpOdfekFsxSlPQCQtPg',
ExploreSidebar: 'lpSN4M6qpimkF4nRFPE3nQ',
ExplorePage: 'kheAINB_4pzRDqkzG3K-ng',
GenericTimelineById: 'uGSr7alSjR9v6QJAIaqSKQ',
TrendHistory: 'Sj4T-jSB9pr0Mxtsc1UKZQ',
AboutAccountQuery: 'zs_jFPFT78rBpXv9Z3U2YQ',
};
export const QUERY_IDS = {
...FALLBACK_QUERY_IDS,
...queryIds,
};
export const TARGET_QUERY_ID_OPERATIONS = Object.keys(FALLBACK_QUERY_IDS);
//# sourceMappingURL=twitter-client-constants.js.map
@@ -0,0 +1,347 @@
import { applyFeatureOverrides } from './runtime-features.js';
export function buildArticleFeatures() {
return applyFeatureOverrides('article', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: false,
responsive_web_grok_annotations_enabled: false,
responsive_web_jetfuel_frame: true,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: true,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_enhance_cards_enabled: false,
});
}
export function buildTweetDetailFeatures() {
return applyFeatureOverrides('tweetDetail', {
...buildArticleFeatures(),
responsive_web_graphql_exclude_directive_enabled: true,
communities_web_enable_tweet_community_results_fetch: true,
responsive_web_twitter_article_plain_text_enabled: true,
responsive_web_twitter_article_seed_tweet_detail_enabled: true,
responsive_web_twitter_article_seed_tweet_summary_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
tweet_awards_web_tipping_enabled: false,
creator_subscriptions_quote_tweet_preview_enabled: false,
verified_phone_label_enabled: false,
});
}
export function buildArticleFieldToggles() {
return {
withPayments: false,
withAuxiliaryUserLabels: false,
withArticleRichContentState: true,
withArticlePlainText: true,
withGrokAnalyze: false,
withDisallowedReplyControls: false,
};
}
export function buildSearchFeatures() {
return applyFeatureOverrides('search', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: false,
responsive_web_grok_annotations_enabled: false,
responsive_web_jetfuel_frame: true,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
rweb_video_timestamps_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
articles_preview_enabled: true,
responsive_web_enhance_cards_enabled: false,
});
}
export function buildTweetCreateFeatures() {
return applyFeatureOverrides('tweetCreate', {
rweb_video_screen_enabled: true,
creator_subscriptions_tweet_preview_api_enabled: true,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: false,
responsive_web_grok_annotations_enabled: false,
responsive_web_jetfuel_frame: true,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: false,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
articles_preview_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_enhance_cards_enabled: false,
});
}
export function buildTimelineFeatures() {
return applyFeatureOverrides('timeline', {
...buildSearchFeatures(),
blue_business_profile_image_shape_enabled: true,
responsive_web_text_conversations_enabled: false,
tweetypie_unmention_optimization_enabled: true,
vibe_api_enabled: true,
responsive_web_twitter_blue_verified_badge_is_enabled: true,
interactive_text_enabled: true,
longform_notetweets_richtext_consumption_enabled: true,
responsive_web_media_download_video_enabled: false,
});
}
export function buildBookmarksFeatures() {
return applyFeatureOverrides('bookmarks', {
...buildTimelineFeatures(),
graphql_timeline_v2_bookmark_timeline: true,
});
}
export function buildLikesFeatures() {
return applyFeatureOverrides('likes', buildTimelineFeatures());
}
export function buildListsFeatures() {
return applyFeatureOverrides('lists', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: false,
responsive_web_grok_annotations_enabled: false,
responsive_web_jetfuel_frame: true,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: true,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_enhance_cards_enabled: false,
blue_business_profile_image_shape_enabled: false,
responsive_web_text_conversations_enabled: false,
tweetypie_unmention_optimization_enabled: true,
vibe_api_enabled: false,
interactive_text_enabled: false,
});
}
export function buildHomeTimelineFeatures() {
return applyFeatureOverrides('homeTimeline', {
...buildTimelineFeatures(),
});
}
export function buildUserTweetsFeatures() {
return applyFeatureOverrides('userTweets', {
rweb_video_screen_enabled: false,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: false,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: true,
responsive_web_jetfuel_frame: true,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: true,
responsive_web_grok_annotations_enabled: false,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: true,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_enhance_cards_enabled: false,
});
}
export function buildFollowingFeatures() {
return applyFeatureOverrides('following', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: false,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: true,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: false,
responsive_web_grok_analyze_post_followups_enabled: false,
responsive_web_grok_annotations_enabled: false,
responsive_web_jetfuel_frame: false,
post_ctas_fetch_enabled: true,
responsive_web_grok_share_attachment_enabled: false,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: true,
responsive_web_grok_show_grok_translated_post: false,
responsive_web_grok_analysis_button_from_backend: false,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: false,
responsive_web_grok_imagine_annotation_enabled: false,
responsive_web_grok_community_note_auto_translation_is_enabled: false,
responsive_web_enhance_cards_enabled: false,
});
}
export function buildExploreFeatures() {
return applyFeatureOverrides('explore', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: true,
responsive_web_grok_analyze_post_followups_enabled: true,
responsive_web_grok_annotations_enabled: true,
responsive_web_jetfuel_frame: true,
responsive_web_grok_share_attachment_enabled: true,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: true,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: true,
responsive_web_enhance_cards_enabled: false,
// Additional features required for ExploreSidebar
post_ctas_fetch_enabled: true,
rweb_video_timestamps_enabled: true,
});
}
//# sourceMappingURL=twitter-client-features.js.map
@@ -0,0 +1,157 @@
import { TWITTER_API_BASE } from './twitter-client-constants.js';
import { buildSearchFeatures } from './twitter-client-features.js';
import { extractCursorFromInstructions, parseTweetsFromInstructions } from './twitter-client-utils.js';
const RAW_QUERY_MISSING_REGEX = /must be defined/i;
function isQueryIdMismatch(payload) {
try {
const parsed = JSON.parse(payload);
return (parsed.errors?.some((error) => {
if (error?.extensions?.code === 'GRAPHQL_VALIDATION_FAILED') {
return true;
}
if (error?.path?.includes('rawQuery') && RAW_QUERY_MISSING_REGEX.test(error.message ?? '')) {
return true;
}
return false;
}) ?? false);
}
catch {
return false;
}
}
export function withSearch(Base) {
class TwitterClientSearch extends Base {
// biome-ignore lint/complexity/noUselessConstructor lint/suspicious/noExplicitAny: TS mixin constructor requirement.
constructor(...args) {
super(...args);
}
/**
* Search for tweets matching a query
*/
async search(query, count = 20, options = {}) {
return this.searchPaged(query, count, options);
}
/**
* Get all search results (paged)
*/
async getAllSearchResults(query, options) {
return this.searchPaged(query, Number.POSITIVE_INFINITY, options);
}
async searchPaged(query, limit, options = {}) {
const features = buildSearchFeatures();
const pageSize = 20;
const seen = new Set();
const tweets = [];
let cursor = options.cursor;
let nextCursor;
let pagesFetched = 0;
const { includeRaw = false, maxPages } = options;
const fetchPage = async (pageCount, pageCursor) => {
let lastError;
let had404 = false;
const queryIds = await this.getSearchTimelineQueryIds();
for (const queryId of queryIds) {
const variables = {
rawQuery: query,
count: pageCount,
querySource: 'typed_query',
product: 'Latest',
...(pageCursor ? { cursor: pageCursor } : {}),
};
const params = new URLSearchParams({
variables: JSON.stringify(variables),
});
const url = `${TWITTER_API_BASE}/${queryId}/SearchTimeline?${params.toString()}`;
try {
const response = await this.fetchWithTimeout(url, {
method: 'POST',
headers: this.getHeaders(),
body: JSON.stringify({ features, queryId }),
});
if (response.status === 404) {
had404 = true;
lastError = `HTTP ${response.status}`;
continue;
}
if (!response.ok) {
const text = await response.text();
const shouldRefreshQueryIds = (response.status === 400 || response.status === 422) && isQueryIdMismatch(text);
return {
success: false,
error: `HTTP ${response.status}: ${text.slice(0, 200)}`,
had404: had404 || shouldRefreshQueryIds,
};
}
const data = (await response.json());
if (data.errors && data.errors.length > 0) {
const shouldRefreshQueryIds = data.errors.some((error) => error?.extensions?.code === 'GRAPHQL_VALIDATION_FAILED');
return {
success: false,
error: data.errors.map((e) => e.message).join(', '),
had404: had404 || shouldRefreshQueryIds,
};
}
const instructions = data.data?.search_by_raw_query?.search_timeline?.timeline?.instructions;
const pageTweets = parseTweetsFromInstructions(instructions, { quoteDepth: this.quoteDepth, includeRaw });
const nextCursor = extractCursorFromInstructions(instructions);
return { success: true, tweets: pageTweets, cursor: nextCursor, had404 };
}
catch (error) {
lastError = error instanceof Error ? error.message : String(error);
}
}
return { success: false, error: lastError ?? 'Unknown error fetching search results', had404 };
};
const fetchWithRefresh = async (pageCount, pageCursor) => {
const firstAttempt = await fetchPage(pageCount, pageCursor);
if (firstAttempt.success) {
return firstAttempt;
}
if (firstAttempt.had404) {
await this.refreshQueryIds();
const secondAttempt = await fetchPage(pageCount, pageCursor);
if (secondAttempt.success) {
return secondAttempt;
}
return { success: false, error: secondAttempt.error };
}
return { success: false, error: firstAttempt.error };
};
const unlimited = limit === Number.POSITIVE_INFINITY;
while (unlimited || tweets.length < limit) {
const pageCount = unlimited ? pageSize : Math.min(pageSize, limit - tweets.length);
const page = await fetchWithRefresh(pageCount, cursor);
if (!page.success) {
return { success: false, error: page.error };
}
pagesFetched += 1;
let added = 0;
for (const tweet of page.tweets) {
if (seen.has(tweet.id)) {
continue;
}
seen.add(tweet.id);
tweets.push(tweet);
added += 1;
if (!unlimited && tweets.length >= limit) {
break;
}
}
const pageCursor = page.cursor;
if (!pageCursor || pageCursor === cursor || page.tweets.length === 0 || added === 0) {
nextCursor = undefined;
break;
}
if (maxPages && pagesFetched >= maxPages) {
nextCursor = pageCursor;
break;
}
cursor = pageCursor;
nextCursor = pageCursor;
}
return { success: true, tweets, nextCursor };
}
}
return TwitterClientSearch;
}
//# sourceMappingURL=twitter-client-search.js.map
@@ -0,0 +1,2 @@
export {};
//# sourceMappingURL=twitter-client-types.js.map
@@ -0,0 +1,511 @@
export function normalizeQuoteDepth(value) {
if (value === undefined || value === null) {
return 1;
}
if (!Number.isFinite(value)) {
return 1;
}
return Math.max(0, Math.floor(value));
}
export function firstText(...values) {
for (const value of values) {
if (typeof value === 'string') {
const trimmed = value.trim();
if (trimmed) {
return trimmed;
}
}
}
return undefined;
}
export function collectTextFields(value, keys, output) {
if (!value) {
return;
}
if (typeof value === 'string') {
return;
}
if (Array.isArray(value)) {
for (const item of value) {
collectTextFields(item, keys, output);
}
return;
}
if (typeof value === 'object') {
for (const [key, nested] of Object.entries(value)) {
if (keys.has(key)) {
if (typeof nested === 'string') {
const trimmed = nested.trim();
if (trimmed) {
output.push(trimmed);
}
continue;
}
}
collectTextFields(nested, keys, output);
}
}
}
export function uniqueOrdered(values) {
const seen = new Set();
const result = [];
for (const value of values) {
if (seen.has(value)) {
continue;
}
seen.add(value);
result.push(value);
}
return result;
}
/**
* Renders a Draft.js content_state into readable markdown/text format.
* Handles blocks (paragraphs, headers, lists) and entities (code blocks, links, tweets, dividers).
*/
export function renderContentState(contentState) {
if (!contentState?.blocks || contentState.blocks.length === 0) {
return undefined;
}
// Build entity lookup map from array/object formats
const entityMap = new Map();
const rawEntityMap = contentState.entityMap ?? [];
if (Array.isArray(rawEntityMap)) {
for (const entry of rawEntityMap) {
const key = Number.parseInt(entry.key, 10);
if (!Number.isNaN(key)) {
entityMap.set(key, entry.value);
}
}
}
else {
for (const [key, value] of Object.entries(rawEntityMap)) {
const keyNumber = Number.parseInt(key, 10);
if (!Number.isNaN(keyNumber)) {
entityMap.set(keyNumber, value);
}
}
}
const outputLines = [];
let orderedListCounter = 0;
let previousBlockType;
for (const block of contentState.blocks) {
// Reset ordered list counter when leaving ordered list context
if (block.type !== 'ordered-list-item' && previousBlockType === 'ordered-list-item') {
orderedListCounter = 0;
}
switch (block.type) {
case 'unstyled': {
// Plain paragraph - just output text with any inline formatting
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(text);
}
break;
}
case 'header-one': {
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`# ${text}`);
}
break;
}
case 'header-two': {
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`## ${text}`);
}
break;
}
case 'header-three': {
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`### ${text}`);
}
break;
}
case 'unordered-list-item': {
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`- ${text}`);
}
break;
}
case 'ordered-list-item': {
orderedListCounter++;
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`${orderedListCounter}. ${text}`);
}
break;
}
case 'blockquote': {
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(`> ${text}`);
}
break;
}
case 'atomic': {
// Atomic blocks are placeholders for embedded entities
const entityContent = renderAtomicBlock(block, entityMap);
if (entityContent) {
outputLines.push(entityContent);
}
break;
}
default: {
// Fallback: just output the text
const text = renderBlockText(block, entityMap);
if (text) {
outputLines.push(text);
}
}
}
previousBlockType = block.type;
}
const result = outputLines.join('\n\n');
return result.trim() || undefined;
}
/**
* Renders text content of a block, applying inline link entities.
*/
function renderBlockText(block, entityMap) {
let text = block.text;
// Handle LINK entities by appending URL in markdown format
// Process in reverse order to not mess up offsets
const linkRanges = (block.entityRanges ?? [])
.filter((range) => {
const entity = entityMap.get(range.key);
return entity?.type === 'LINK' && entity.data.url;
})
.sort((a, b) => b.offset - a.offset);
for (const range of linkRanges) {
const entity = entityMap.get(range.key);
if (entity?.data.url) {
const linkText = text.slice(range.offset, range.offset + range.length);
const markdownLink = `[${linkText}](${entity.data.url})`;
text = text.slice(0, range.offset) + markdownLink + text.slice(range.offset + range.length);
}
}
return text.trim();
}
/**
* Renders an atomic block by looking up its entity and returning appropriate content.
*/
function renderAtomicBlock(block, entityMap) {
const entityRanges = block.entityRanges ?? [];
if (entityRanges.length === 0) {
return undefined;
}
const entityKey = entityRanges[0].key;
const entity = entityMap.get(entityKey);
if (!entity) {
return undefined;
}
switch (entity.type) {
case 'MARKDOWN':
// Code blocks and other markdown content - output as-is
return entity.data.markdown?.trim();
case 'DIVIDER':
return '---';
case 'TWEET':
if (entity.data.tweetId) {
return `[Embedded Tweet: https://x.com/i/status/${entity.data.tweetId}]`;
}
return undefined;
case 'LINK':
if (entity.data.url) {
return `[Link: ${entity.data.url}]`;
}
return undefined;
case 'IMAGE':
// Images in atomic blocks - could extract URL if available
return '[Image]';
default:
return undefined;
}
}
export function extractArticleText(result) {
const article = result?.article;
if (!article) {
return undefined;
}
const articleResult = article.article_results?.result ?? article;
if (process.env.BIRD_DEBUG_ARTICLE === '1') {
console.error('[bird][debug][article] payload:', JSON.stringify({
rest_id: result?.rest_id,
article: articleResult,
note_tweet: result?.note_tweet?.note_tweet_results?.result ?? null,
}, null, 2));
}
const title = firstText(articleResult.title, article.title);
// Try to render from rich content_state first (Draft.js format with blocks + entityMap)
// This preserves code blocks, embedded tweets, markdown, etc.
const contentState = article.article_results?.result?.content_state;
const richBody = renderContentState(contentState);
if (richBody) {
// Rich content found - prepend title if not already included
if (title) {
const normalizedTitle = title.trim();
const trimmedBody = richBody.trimStart();
const headingMatches = [`# ${normalizedTitle}`, `## ${normalizedTitle}`, `### ${normalizedTitle}`];
const hasTitle = trimmedBody === normalizedTitle ||
trimmedBody.startsWith(`${normalizedTitle}\n`) ||
headingMatches.some((heading) => trimmedBody.startsWith(heading));
if (!hasTitle) {
return `${title}\n\n${richBody}`;
}
}
return richBody;
}
// Fallback to plain text extraction for articles without rich content_state
let body = firstText(articleResult.plain_text, article.plain_text, articleResult.body?.text, articleResult.body?.richtext?.text, articleResult.body?.rich_text?.text, articleResult.content?.text, articleResult.content?.richtext?.text, articleResult.content?.rich_text?.text, articleResult.text, articleResult.richtext?.text, articleResult.rich_text?.text, article.body?.text, article.body?.richtext?.text, article.body?.rich_text?.text, article.content?.text, article.content?.richtext?.text, article.content?.rich_text?.text, article.text, article.richtext?.text, article.rich_text?.text);
if (body && title && body.trim() === title.trim()) {
body = undefined;
}
if (!body) {
const collected = [];
collectTextFields(articleResult, new Set(['text', 'title']), collected);
collectTextFields(article, new Set(['text', 'title']), collected);
const unique = uniqueOrdered(collected);
const filtered = title ? unique.filter((value) => value !== title) : unique;
if (filtered.length > 0) {
body = filtered.join('\n\n');
}
}
if (title && body && !body.startsWith(title)) {
return `${title}\n\n${body}`;
}
return body ?? title;
}
export function extractNoteTweetText(result) {
const note = result?.note_tweet?.note_tweet_results?.result;
if (!note) {
return undefined;
}
return firstText(note.text, note.richtext?.text, note.rich_text?.text, note.content?.text, note.content?.richtext?.text, note.content?.rich_text?.text);
}
export function extractTweetText(result) {
return extractArticleText(result) ?? extractNoteTweetText(result) ?? firstText(result?.legacy?.full_text);
}
export function extractArticleMetadata(result) {
const article = result?.article;
if (!article) {
return undefined;
}
const articleResult = article.article_results?.result ?? article;
const title = firstText(articleResult.title, article.title);
if (!title) {
return undefined;
}
// preview_text is available in home timeline responses
const previewText = firstText(articleResult.preview_text, article.preview_text);
return { title, previewText };
}
export function extractMedia(result) {
// Prefer extended_entities (has video info), fall back to entities
const rawMedia = result?.legacy?.extended_entities?.media ?? result?.legacy?.entities?.media;
if (!rawMedia || rawMedia.length === 0) {
return undefined;
}
const media = [];
for (const item of rawMedia) {
if (!item.type || !item.media_url_https) {
continue;
}
const mediaItem = {
type: item.type,
url: item.media_url_https,
};
// Get dimensions from largest available size
const sizes = item.sizes;
if (sizes?.large) {
mediaItem.width = sizes.large.w;
mediaItem.height = sizes.large.h;
}
else if (sizes?.medium) {
mediaItem.width = sizes.medium.w;
mediaItem.height = sizes.medium.h;
}
// For thumbnails/previews
if (sizes?.small) {
mediaItem.previewUrl = `${item.media_url_https}:small`;
}
// Extract video URL for video/animated_gif
if ((item.type === 'video' || item.type === 'animated_gif') && item.video_info?.variants) {
// Prefer highest bitrate MP4, fall back to first MP4 when bitrate is missing.
const mp4Variants = item.video_info.variants.filter((v) => v.content_type === 'video/mp4' && typeof v.url === 'string');
const mp4WithBitrate = mp4Variants
.filter((v) => typeof v.bitrate === 'number')
.sort((a, b) => b.bitrate - a.bitrate);
const selectedVariant = mp4WithBitrate[0] ?? mp4Variants[0];
if (selectedVariant) {
mediaItem.videoUrl = selectedVariant.url;
}
if (typeof item.video_info.duration_millis === 'number') {
mediaItem.durationMs = item.video_info.duration_millis;
}
}
media.push(mediaItem);
}
return media.length > 0 ? media : undefined;
}
export function unwrapTweetResult(result) {
if (!result) {
return undefined;
}
if (result.tweet) {
return result.tweet;
}
return result;
}
export function mapTweetResult(result, quoteDepthOrOptions) {
const options = typeof quoteDepthOrOptions === 'number' ? { quoteDepth: quoteDepthOrOptions } : quoteDepthOrOptions;
const { quoteDepth, includeRaw = false } = options;
const userResult = result?.core?.user_results?.result;
const userLegacy = userResult?.legacy;
const userCore = userResult?.core;
const username = userLegacy?.screen_name ?? userCore?.screen_name;
const name = userLegacy?.name ?? userCore?.name ?? username;
const userId = userResult?.rest_id;
if (!result?.rest_id || !username) {
return undefined;
}
const text = extractTweetText(result);
if (!text) {
return undefined;
}
let quotedTweet;
if (quoteDepth > 0) {
const quotedResult = unwrapTweetResult(result.quoted_status_result?.result);
if (quotedResult) {
quotedTweet = mapTweetResult(quotedResult, { quoteDepth: quoteDepth - 1, includeRaw });
}
}
const media = extractMedia(result);
const article = extractArticleMetadata(result);
const tweetData = {
id: result.rest_id,
text,
createdAt: result.legacy?.created_at,
replyCount: result.legacy?.reply_count,
retweetCount: result.legacy?.retweet_count,
likeCount: result.legacy?.favorite_count,
conversationId: result.legacy?.conversation_id_str,
inReplyToStatusId: result.legacy?.in_reply_to_status_id_str ?? undefined,
author: {
username,
name: name || username,
},
authorId: userId,
quotedTweet,
media,
article,
};
if (includeRaw) {
tweetData._raw = result;
}
return tweetData;
}
export function findTweetInInstructions(instructions, tweetId) {
if (!instructions) {
return undefined;
}
for (const instruction of instructions) {
for (const entry of instruction.entries || []) {
const result = entry.content?.itemContent?.tweet_results?.result;
if (result?.rest_id === tweetId) {
return result;
}
}
}
return undefined;
}
export function collectTweetResultsFromEntry(entry) {
const results = [];
const pushResult = (result) => {
if (result?.rest_id) {
results.push(result);
}
};
const content = entry.content;
pushResult(content?.itemContent?.tweet_results?.result);
pushResult(content?.item?.itemContent?.tweet_results?.result);
for (const item of content?.items ?? []) {
pushResult(item?.item?.itemContent?.tweet_results?.result);
pushResult(item?.itemContent?.tweet_results?.result);
pushResult(item?.content?.itemContent?.tweet_results?.result);
}
return results;
}
export function parseTweetsFromInstructions(instructions, quoteDepthOrOptions) {
const options = typeof quoteDepthOrOptions === 'number' ? { quoteDepth: quoteDepthOrOptions } : quoteDepthOrOptions;
const { quoteDepth, includeRaw = false } = options;
const tweets = [];
const seen = new Set();
for (const instruction of instructions ?? []) {
for (const entry of instruction.entries ?? []) {
const results = collectTweetResultsFromEntry(entry);
for (const result of results) {
const mapped = mapTweetResult(result, { quoteDepth, includeRaw });
if (!mapped || seen.has(mapped.id)) {
continue;
}
seen.add(mapped.id);
tweets.push(mapped);
}
}
}
return tweets;
}
export function extractCursorFromInstructions(instructions, cursorType = 'Bottom') {
for (const instruction of instructions ?? []) {
for (const entry of instruction.entries ?? []) {
const content = entry.content;
if (content?.cursorType === cursorType && typeof content.value === 'string' && content.value.length > 0) {
return content.value;
}
}
}
return undefined;
}
export function parseUsersFromInstructions(instructions) {
if (!instructions) {
return [];
}
const users = [];
for (const instruction of instructions) {
if (!instruction.entries) {
continue;
}
for (const entry of instruction.entries) {
const content = entry?.content;
const rawUserResult = content?.itemContent?.user_results?.result;
const userResult = rawUserResult?.__typename === 'UserWithVisibilityResults' && rawUserResult.user
? rawUserResult.user
: rawUserResult;
if (!userResult || userResult.__typename !== 'User') {
continue;
}
const legacy = userResult.legacy;
const core = userResult.core;
const username = legacy?.screen_name ?? core?.screen_name;
if (!userResult.rest_id || !username) {
continue;
}
users.push({
id: userResult.rest_id,
username,
name: legacy?.name ?? core?.name ?? username,
description: legacy?.description,
followersCount: legacy?.followers_count,
followingCount: legacy?.friends_count,
isBlueVerified: userResult.is_blue_verified,
profileImageUrl: legacy?.profile_image_url_https ?? userResult.avatar?.image_url,
createdAt: legacy?.created_at ?? core?.created_at,
});
}
}
return users;
}
//# sourceMappingURL=twitter-client-utils.js.map
@@ -0,0 +1,22 @@
MIT License
Copyright (c) 2025 Peter Steinberger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
@@ -0,0 +1,29 @@
# @steipete/sweet-cookie
Inline-first browser cookie extraction for local tooling (no native addons).
Supports:
- Inline payloads (JSON / base64 / file) — most reliable path.
- Local browser reads (best effort): Chrome, Edge, Firefox, Safari (macOS).
Install:
```bash
npm i @steipete/sweet-cookie
```
Usage:
```ts
import { getCookies, toCookieHeader } from '@steipete/sweet-cookie';
const { cookies, warnings } = await getCookies({
url: 'https://example.com/',
names: ['session', 'csrf'],
browsers: ['chrome', 'edge', 'firefox', 'safari'],
});
for (const w of warnings) console.warn(w);
const cookieHeader = toCookieHeader(cookies, { dedupeByName: true });
```
Docs + extension exporter: see the repo root README.
@@ -0,0 +1,3 @@
export { getCookies, toCookieHeader } from './public.js';
export type { BrowserName, Cookie, CookieHeaderOptions, CookieMode, CookieSameSite, GetCookiesOptions, GetCookiesResult, } from './types.js';
//# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AACzD,YAAY,EACX,WAAW,EACX,MAAM,EACN,mBAAmB,EACnB,UAAU,EACV,cAAc,EACd,iBAAiB,EACjB,gBAAgB,GAChB,MAAM,YAAY,CAAC"}
@@ -0,0 +1,2 @@
export { getCookies, toCookieHeader } from './public.js';
//# sourceMappingURL=index.js.map
@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC"}
@@ -0,0 +1,8 @@
import type { GetCookiesResult } from '../types.js';
export declare function getCookiesFromChrome(options: {
profile?: string;
timeoutMs?: number;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null): Promise<GetCookiesResult>;
//# sourceMappingURL=chrome.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"chrome.d.ts","sourceRoot":"","sources":["../../src/providers/chrome.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAU,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAK5D,wBAAsB,oBAAoB,CACzC,OAAO,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EAC5F,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,GAChC,OAAO,CAAC,gBAAgB,CAAC,CA0B3B"}
@@ -0,0 +1,27 @@
import { getCookiesFromChromeSqliteLinux } from './chromeSqliteLinux.js';
import { getCookiesFromChromeSqliteMac } from './chromeSqliteMac.js';
import { getCookiesFromChromeSqliteWindows } from './chromeSqliteWindows.js';
export async function getCookiesFromChrome(options, origins, allowlistNames) {
const warnings = [];
// Platform dispatch only. All real logic lives in the per-OS providers.
if (process.platform === 'darwin') {
const r = await getCookiesFromChromeSqliteMac(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
if (process.platform === 'linux') {
const r = await getCookiesFromChromeSqliteLinux(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
if (process.platform === 'win32') {
const r = await getCookiesFromChromeSqliteWindows(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
return { cookies: [], warnings };
}
//# sourceMappingURL=chrome.js.map
@@ -0,0 +1 @@
{"version":3,"file":"chrome.js","sourceRoot":"","sources":["../../src/providers/chrome.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,+BAA+B,EAAE,MAAM,wBAAwB,CAAC;AACzE,OAAO,EAAE,6BAA6B,EAAE,MAAM,sBAAsB,CAAC;AACrE,OAAO,EAAE,iCAAiC,EAAE,MAAM,0BAA0B,CAAC;AAE7E,MAAM,CAAC,KAAK,UAAU,oBAAoB,CACzC,OAA4F,EAC5F,OAAiB,EACjB,cAAkC;IAElC,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,wEAAwE;IACxE,IAAI,OAAO,CAAC,QAAQ,KAAK,QAAQ,EAAE,CAAC;QACnC,MAAM,CAAC,GAAG,MAAM,6BAA6B,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QAChF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QAClC,MAAM,CAAC,GAAG,MAAM,+BAA+B,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QAClF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QAClC,MAAM,CAAC,GAAG,MAAM,iCAAiC,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QACpF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;AAClC,CAAC"}
@@ -0,0 +1,11 @@
export declare function deriveAes128CbcKeyFromPassword(password: string, options: {
iterations: number;
}): Buffer;
export declare function decryptChromiumAes128CbcCookieValue(encryptedValue: Uint8Array, keyCandidates: readonly Buffer[], options: {
stripHashPrefix: boolean;
treatUnknownPrefixAsPlaintext?: boolean;
}): string | null;
export declare function decryptChromiumAes256GcmCookieValue(encryptedValue: Uint8Array, key: Buffer, options: {
stripHashPrefix: boolean;
}): string | null;
//# sourceMappingURL=crypto.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"crypto.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/crypto.ts"],"names":[],"mappings":"AAIA,wBAAgB,8BAA8B,CAC7C,QAAQ,EAAE,MAAM,EAChB,OAAO,EAAE;IAAE,UAAU,EAAE,MAAM,CAAA;CAAE,GAC7B,MAAM,CAIR;AAED,wBAAgB,mCAAmC,CAClD,cAAc,EAAE,UAAU,EAC1B,aAAa,EAAE,SAAS,MAAM,EAAE,EAChC,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAC;IAAC,6BAA6B,CAAC,EAAE,OAAO,CAAA;CAAE,GAC5E,MAAM,GAAG,IAAI,CA2Bf;AAED,wBAAgB,mCAAmC,CAClD,cAAc,EAAE,UAAU,EAC1B,GAAG,EAAE,MAAM,EACX,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAA;CAAE,GACnC,MAAM,GAAG,IAAI,CAyBf"}
@@ -0,0 +1,100 @@
import { createDecipheriv, pbkdf2Sync } from 'node:crypto';
const UTF8_DECODER = new TextDecoder('utf-8', { fatal: true });
export function deriveAes128CbcKeyFromPassword(password, options) {
// Chromium derives the AES-128-CBC key from "Chrome Safe Storage" using PBKDF2.
// The salt/length/digest are fixed by Chromium ("saltysalt", 16 bytes, sha1).
return pbkdf2Sync(password, 'saltysalt', options.iterations, 16, 'sha1');
}
export function decryptChromiumAes128CbcCookieValue(encryptedValue, keyCandidates, options) {
const buf = Buffer.from(encryptedValue);
if (buf.length < 3)
return null;
// Chromium prefixes encrypted cookies with `v10`, `v11`, ... (three bytes).
const prefix = buf.subarray(0, 3).toString('utf8');
const hasVersionPrefix = /^v\d\d$/.test(prefix);
if (!hasVersionPrefix) {
// Some platforms (notably macOS) can store plaintext values in `encrypted_value`.
// Callers decide whether unknown prefixes should be treated as plaintext.
if (options.treatUnknownPrefixAsPlaintext === false)
return null;
return decodeCookieValueBytes(buf, false);
}
const ciphertext = buf.subarray(3);
if (!ciphertext.length)
return '';
for (const key of keyCandidates) {
// Try multiple candidates because Linux may fall back to empty passwords depending on keyring state.
const decrypted = tryDecryptAes128Cbc(ciphertext, key);
if (!decrypted)
continue;
const decoded = decodeCookieValueBytes(decrypted, options.stripHashPrefix);
if (decoded !== null)
return decoded;
}
return null;
}
export function decryptChromiumAes256GcmCookieValue(encryptedValue, key, options) {
const buf = Buffer.from(encryptedValue);
if (buf.length < 3)
return null;
const prefix = buf.subarray(0, 3).toString('utf8');
if (!/^v\d\d$/.test(prefix))
return null;
// AES-256-GCM layout:
// - 12-byte nonce
// - ciphertext
// - 16-byte authentication tag
const payload = buf.subarray(3);
if (payload.length < 12 + 16)
return null;
const nonce = payload.subarray(0, 12);
const authenticationTag = payload.subarray(payload.length - 16);
const ciphertext = payload.subarray(12, payload.length - 16);
try {
const decipher = createDecipheriv('aes-256-gcm', key, nonce);
decipher.setAuthTag(authenticationTag);
const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
return decodeCookieValueBytes(plaintext, options.stripHashPrefix);
}
catch {
return null;
}
}
function tryDecryptAes128Cbc(ciphertext, key) {
try {
// Chromium's legacy AES-128-CBC uses an IV of 16 spaces.
const iv = Buffer.alloc(16, 0x20);
const decipher = createDecipheriv('aes-128-cbc', key, iv);
decipher.setAutoPadding(false);
const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
return removePkcs7Padding(plaintext);
}
catch {
return null;
}
}
function removePkcs7Padding(value) {
if (!value.length)
return value;
const padding = value[value.length - 1];
if (!padding || padding > 16)
return value;
return value.subarray(0, value.length - padding);
}
function decodeCookieValueBytes(value, stripHashPrefix) {
// Chromium >= 24 prepends a 32-byte hash to cookie values.
const bytes = stripHashPrefix && value.length >= 32 ? value.subarray(32) : value;
try {
return stripLeadingControlChars(UTF8_DECODER.decode(bytes));
}
catch {
return null;
}
}
function stripLeadingControlChars(value) {
let i = 0;
while (i < value.length && value.charCodeAt(i) < 0x20)
i += 1;
return value.slice(i);
}
//# sourceMappingURL=crypto.js.map
@@ -0,0 +1 @@
{"version":3,"file":"crypto.js","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/crypto.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAE3D,MAAM,YAAY,GAAG,IAAI,WAAW,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;AAE/D,MAAM,UAAU,8BAA8B,CAC7C,QAAgB,EAChB,OAA+B;IAE/B,gFAAgF;IAChF,8EAA8E;IAC9E,OAAO,UAAU,CAAC,QAAQ,EAAE,WAAW,EAAE,OAAO,CAAC,UAAU,EAAE,EAAE,EAAE,MAAM,CAAC,CAAC;AAC1E,CAAC;AAED,MAAM,UAAU,mCAAmC,CAClD,cAA0B,EAC1B,aAAgC,EAChC,OAA8E;IAE9E,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;IACxC,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IAEhC,4EAA4E;IAC5E,MAAM,MAAM,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IACnD,MAAM,gBAAgB,GAAG,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAEhD,IAAI,CAAC,gBAAgB,EAAE,CAAC;QACvB,kFAAkF;QAClF,0EAA0E;QAC1E,IAAI,OAAO,CAAC,6BAA6B,KAAK,KAAK;YAAE,OAAO,IAAI,CAAC;QACjE,OAAO,sBAAsB,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;IAC3C,CAAC;IAED,MAAM,UAAU,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;IACnC,IAAI,CAAC,UAAU,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAElC,KAAK,MAAM,GAAG,IAAI,aAAa,EAAE,CAAC;QACjC,qGAAqG;QACrG,MAAM,SAAS,GAAG,mBAAmB,CAAC,UAAU,EAAE,GAAG,CAAC,CAAC;QACvD,IAAI,CAAC,SAAS;YAAE,SAAS;QACzB,MAAM,OAAO,GAAG,sBAAsB,CAAC,SAAS,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;QAC3E,IAAI,OAAO,KAAK,IAAI;YAAE,OAAO,OAAO,CAAC;IACtC,CAAC;IAED,OAAO,IAAI,CAAC;AACb,CAAC;AAED,MAAM,UAAU,mCAAmC,CAClD,cAA0B,EAC1B,GAAW,EACX,OAAqC;IAErC,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;IACxC,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IAChC,MAAM,MAAM,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IACnD,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC;QAAE,OAAO,IAAI,CAAC;IAEzC,sBAAsB;IACtB,kBAAkB;IAClB,eAAe;IACf,+BAA+B;IAC/B,MAAM,OAAO,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;IAChC,IAAI,OAAO,CAAC,MAAM,GAAG,EAAE,GAAG,EAAE;QAAE,OAAO,IAAI,CAAC;IAE1C,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IACtC,MAAM,iBAAiB,GAAG,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC;IAChE,MAAM,UAAU,GAAG,OAAO,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC;IAE7D,IAAI,CAAC;QACJ,MAAM,QAAQ,GAAG,gBAAgB,CAAC,aAAa,EAAE,GAAG,EAAE,KAAK,CAAC,CAAC;QAC7D,QAAQ,CAAC,UAAU,CAAC,iBAAiB,CAAC,CAAC;QACvC,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,EAAE,QAAQ,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QACjF,OAAO,sBAAsB,CAAC,SAAS,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;IACnE,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,mBAAmB,CAAC,UAAkB,EAAE,GAAW;IAC3D,IAAI,CAAC;QACJ,yDAAyD;QACzD,MAAM,EAAE,GAAG,MAAM,CAAC,KAAK,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;QAClC,MAAM,QAAQ,GAAG,gBAAgB,CAAC,aAAa,EAAE,GAAG,EAAE,EAAE,CAAC,CAAC;QAC1D,QAAQ,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QAC/B,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,EAAE,QAAQ,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QACjF,OAAO,kBAAkB,CAAC,SAAS,CAAC,CAAC;IACtC,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,kBAAkB,CAAC,KAAa;IACxC,IAAI,CAAC,KAAK,CAAC,MAAM;QAAE,OAAO,KAAK,CAAC;IAChC,MAAM,OAAO,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACxC,IAAI,CAAC,OAAO,IAAI,OAAO,GAAG,EAAE;QAAE,OAAO,KAAK,CAAC;IAC3C,OAAO,KAAK,CAAC,QAAQ,CAAC,CAAC,EAAE,KAAK,CAAC,MAAM,GAAG,OAAO,CAAC,CAAC;AAClD,CAAC;AAED,SAAS,sBAAsB,CAAC,KAAa,EAAE,eAAwB;IACtE,2DAA2D;IAC3D,MAAM,KAAK,GAAG,eAAe,IAAI,KAAK,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC;IACjF,IAAI,CAAC;QACJ,OAAO,wBAAwB,CAAC,YAAY,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;IAC7D,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,wBAAwB,CAAC,KAAa;IAC9C,IAAI,CAAC,GAAG,CAAC,CAAC;IACV,OAAO,CAAC,GAAG,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,GAAG,IAAI;QAAE,CAAC,IAAI,CAAC,CAAC;IAC9D,OAAO,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AACvB,CAAC"}
@@ -0,0 +1,25 @@
export type LinuxKeyringBackend = 'gnome' | 'kwallet' | 'basic';
/**
* Read the "Safe Storage" password from a Linux keyring.
*
* Chromium browsers typically store their cookie encryption password under:
* - service: "<Browser> Safe Storage"
* - account: "<Browser>"
*
* We keep this logic in JS (no native deps) and return an empty password on failure
* (Chromium may still have v10 cookies, and callers can use inline/export escape hatches).
*/
export declare function getLinuxChromiumSafeStoragePassword(options: {
backend?: LinuxKeyringBackend;
app: 'chrome' | 'edge';
}): Promise<{
password: string;
warnings: string[];
}>;
export declare function getLinuxChromeSafeStoragePassword(options?: {
backend?: LinuxKeyringBackend;
}): Promise<{
password: string;
warnings: string[];
}>;
//# sourceMappingURL=linuxKeyring.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"linuxKeyring.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/linuxKeyring.ts"],"names":[],"mappings":"AAEA,MAAM,MAAM,mBAAmB,GAAG,OAAO,GAAG,SAAS,GAAG,OAAO,CAAC;AAEhE;;;;;;;;;GASG;AACH,wBAAsB,mCAAmC,CAAC,OAAO,EAAE;IAClE,OAAO,CAAC,EAAE,mBAAmB,CAAC;IAC9B,GAAG,EAAE,QAAQ,GAAG,MAAM,CAAC;CACvB,GAAG,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,EAAE,CAAA;CAAE,CAAC,CA8DpD;AAED,wBAAsB,iCAAiC,CACtD,OAAO,GAAE;IAAE,OAAO,CAAC,EAAE,mBAAmB,CAAA;CAAO,GAC7C,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,EAAE,CAAA;CAAE,CAAC,CAInD"}
@@ -0,0 +1,104 @@
import { execCapture } from '../../util/exec.js';
/**
* Read the "Safe Storage" password from a Linux keyring.
*
* Chromium browsers typically store their cookie encryption password under:
* - service: "<Browser> Safe Storage"
* - account: "<Browser>"
*
* We keep this logic in JS (no native deps) and return an empty password on failure
* (Chromium may still have v10 cookies, and callers can use inline/export escape hatches).
*/
export async function getLinuxChromiumSafeStoragePassword(options) {
const warnings = [];
// Escape hatch: if callers already know the password (or want deterministic CI behavior),
// they can bypass keyring probing entirely.
const overrideKey = options.app === 'edge'
? 'SWEET_COOKIE_EDGE_SAFE_STORAGE_PASSWORD'
: 'SWEET_COOKIE_CHROME_SAFE_STORAGE_PASSWORD';
const override = readEnv(overrideKey);
if (override !== undefined)
return { password: override, warnings };
const backend = options.backend ?? parseLinuxKeyringBackend() ?? chooseLinuxKeyringBackend();
// `basic` means "don't try keyrings" (Chrome will fall back to older/less-secure schemes on some setups).
if (backend === 'basic')
return { password: '', warnings };
const service = options.app === 'edge' ? 'Microsoft Edge Safe Storage' : 'Chrome Safe Storage';
const account = options.app === 'edge' ? 'Microsoft Edge' : 'Chrome';
const folder = `${account} Keys`;
if (backend === 'gnome') {
// GNOME keyring: `secret-tool` is the simplest way to read libsecret entries.
const res = await execCapture('secret-tool', ['lookup', 'service', service, 'account', account], { timeoutMs: 3_000 });
if (res.code === 0)
return { password: res.stdout.trim(), warnings };
warnings.push('Failed to read Linux keyring via secret-tool; v11 cookies may be unavailable.');
return { password: '', warnings };
}
// KDE keyring: query KWallet via `kwallet-query`, but the wallet name differs across KDE versions.
const kdeVersion = (readEnv('KDE_SESSION_VERSION') ?? '').trim();
const serviceName = kdeVersion === '6'
? 'org.kde.kwalletd6'
: kdeVersion === '5'
? 'org.kde.kwalletd5'
: 'org.kde.kwalletd';
const walletPath = kdeVersion === '6'
? '/modules/kwalletd6'
: kdeVersion === '5'
? '/modules/kwalletd5'
: '/modules/kwalletd';
const wallet = await getKWalletNetworkWallet(serviceName, walletPath);
const passwordRes = await execCapture('kwallet-query', ['--read-password', service, '--folder', folder, wallet], { timeoutMs: 3_000 });
if (passwordRes.code !== 0) {
warnings.push('Failed to read Linux keyring via kwallet-query; v11 cookies may be unavailable.');
return { password: '', warnings };
}
if (passwordRes.stdout.toLowerCase().startsWith('failed to read'))
return { password: '', warnings };
return { password: passwordRes.stdout.trim(), warnings };
}
export async function getLinuxChromeSafeStoragePassword(options = {}) {
const args = { app: 'chrome' };
if (options.backend !== undefined)
args.backend = options.backend;
return await getLinuxChromiumSafeStoragePassword(args);
}
function parseLinuxKeyringBackend() {
const raw = readEnv('SWEET_COOKIE_LINUX_KEYRING');
if (!raw)
return undefined;
const normalized = raw.toLowerCase();
if (normalized === 'gnome')
return 'gnome';
if (normalized === 'kwallet')
return 'kwallet';
if (normalized === 'basic')
return 'basic';
return undefined;
}
function chooseLinuxKeyringBackend() {
const xdg = readEnv('XDG_CURRENT_DESKTOP') ?? '';
const isKde = xdg.split(':').some((p) => p.trim().toLowerCase() === 'kde') || !!readEnv('KDE_FULL_SESSION');
return isKde ? 'kwallet' : 'gnome';
}
async function getKWalletNetworkWallet(serviceName, walletPath) {
const res = await execCapture('dbus-send', [
'--session',
'--print-reply=literal',
`--dest=${serviceName}`,
walletPath,
'org.kde.KWallet.networkWallet',
], { timeoutMs: 3_000 });
const fallback = 'kdewallet';
if (res.code !== 0)
return fallback;
const raw = res.stdout.trim();
if (!raw)
return fallback;
return raw.replaceAll('"', '').trim() || fallback;
}
function readEnv(key) {
const value = process.env[key];
const trimmed = typeof value === 'string' ? value.trim() : '';
return trimmed.length ? trimmed : undefined;
}
//# sourceMappingURL=linuxKeyring.js.map
@@ -0,0 +1 @@
{"version":3,"file":"linuxKeyring.js","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/linuxKeyring.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AAIjD;;;;;;;;;GASG;AACH,MAAM,CAAC,KAAK,UAAU,mCAAmC,CAAC,OAGzD;IACA,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,0FAA0F;IAC1F,4CAA4C;IAC5C,MAAM,WAAW,GAChB,OAAO,CAAC,GAAG,KAAK,MAAM;QACrB,CAAC,CAAC,yCAAyC;QAC3C,CAAC,CAAC,2CAA2C,CAAC;IAChD,MAAM,QAAQ,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IACtC,IAAI,QAAQ,KAAK,SAAS;QAAE,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,QAAQ,EAAE,CAAC;IAEpE,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,wBAAwB,EAAE,IAAI,yBAAyB,EAAE,CAAC;IAC7F,0GAA0G;IAC1G,IAAI,OAAO,KAAK,OAAO;QAAE,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IAE3D,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,KAAK,MAAM,CAAC,CAAC,CAAC,6BAA6B,CAAC,CAAC,CAAC,qBAAqB,CAAC;IAC/F,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,KAAK,MAAM,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,QAAQ,CAAC;IACrE,MAAM,MAAM,GAAG,GAAG,OAAO,OAAO,CAAC;IAEjC,IAAI,OAAO,KAAK,OAAO,EAAE,CAAC;QACzB,8EAA8E;QAC9E,MAAM,GAAG,GAAG,MAAM,WAAW,CAC5B,aAAa,EACb,CAAC,QAAQ,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAAE,OAAO,CAAC,EAClD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;QACF,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC;YAAE,OAAO,EAAE,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC;QACrE,QAAQ,CAAC,IAAI,CAAC,+EAA+E,CAAC,CAAC;QAC/F,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,CAAC;IAED,mGAAmG;IACnG,MAAM,UAAU,GAAG,CAAC,OAAO,CAAC,qBAAqB,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACjE,MAAM,WAAW,GAChB,UAAU,KAAK,GAAG;QACjB,CAAC,CAAC,mBAAmB;QACrB,CAAC,CAAC,UAAU,KAAK,GAAG;YACnB,CAAC,CAAC,mBAAmB;YACrB,CAAC,CAAC,kBAAkB,CAAC;IACxB,MAAM,UAAU,GACf,UAAU,KAAK,GAAG;QACjB,CAAC,CAAC,oBAAoB;QACtB,CAAC,CAAC,UAAU,KAAK,GAAG;YACnB,CAAC,CAAC,oBAAoB;YACtB,CAAC,CAAC,mBAAmB,CAAC;IAEzB,MAAM,MAAM,GAAG,MAAM,uBAAuB,CAAC,WAAW,EAAE,UAAU,CAAC,CAAC;IACtE,MAAM,WAAW,GAAG,MAAM,WAAW,CACpC,eAAe,EACf,CAAC,iBAAiB,EAAE,OAAO,EAAE,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,EACxD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;IACF,IAAI,WAAW,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QAC5B,QAAQ,CAAC,IAAI,CACZ,iFAAiF,CACjF,CAAC;QACF,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,CAAC;IACD,IAAI,WAAW,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,UAAU,CAAC,gBAAgB,CAAC;QAChE,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,OAAO,EAAE,QAAQ,EAAE,WAAW,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC;AAC1D,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,iCAAiC,CACtD,UAA6C,EAAE;IAE/C,MAAM,IAAI,GAAqD,EAAE,GAAG,EAAE,QAAQ,EAAE,CAAC;IACjF,IAAI,OAAO,CAAC,OAAO,KAAK,SAAS;QAAE,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IAClE,OAAO,MAAM,mCAAmC,CAAC,IAAI,CAAC,CAAC;AACxD,CAAC;AAED,SAAS,wBAAwB;IAChC,MAAM,GAAG,GAAG,OAAO,CAAC,4BAA4B,CAAC,CAAC;IAClD,IAAI,CAAC,GAAG;QAAE,OAAO,SAAS,CAAC;IAC3B,MAAM,UAAU,GAAG,GAAG,CAAC,WAAW,EAAE,CAAC;IACrC,IAAI,UAAU,KAAK,OAAO;QAAE,OAAO,OAAO,CAAC;IAC3C,IAAI,UAAU,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IAC/C,IAAI,UAAU,KAAK,OAAO;QAAE,OAAO,OAAO,CAAC;IAC3C,OAAO,SAAS,CAAC;AAClB,CAAC;AAED,SAAS,yBAAyB;IACjC,MAAM,GAAG,GAAG,OAAO,CAAC,qBAAqB,CAAC,IAAI,EAAE,CAAC;IACjD,MAAM,KAAK,GACV,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,KAAK,KAAK,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,kBAAkB,CAAC,CAAC;IAC/F,OAAO,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC;AACpC,CAAC;AAED,KAAK,UAAU,uBAAuB,CAAC,WAAmB,EAAE,UAAkB;IAC7E,MAAM,GAAG,GAAG,MAAM,WAAW,CAC5B,WAAW,EACX;QACC,WAAW;QACX,uBAAuB;QACvB,UAAU,WAAW,EAAE;QACvB,UAAU;QACV,+BAA+B;KAC/B,EACD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;IACF,MAAM,QAAQ,GAAG,WAAW,CAAC;IAC7B,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC;QAAE,OAAO,QAAQ,CAAC;IACpC,MAAM,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;IAC9B,IAAI,CAAC,GAAG;QAAE,OAAO,QAAQ,CAAC;IAC1B,OAAO,GAAG,CAAC,UAAU,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,IAAI,QAAQ,CAAC;AACnD,CAAC;AAED,SAAS,OAAO,CAAC,GAAW;IAC3B,MAAM,KAAK,GAAG,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAC/B,MAAM,OAAO,GAAG,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;IAC9D,OAAO,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;AAC7C,CAAC"}
@@ -0,0 +1,10 @@
import type { GetCookiesResult } from '../../types.js';
export declare function getCookiesFromChromeSqliteDb(options: {
dbPath: string;
profile?: string;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null, decrypt: (encryptedValue: Uint8Array, options: {
stripHashPrefix: boolean;
}) => string | null): Promise<GetCookiesResult>;
//# sourceMappingURL=shared.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"shared.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/shared.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAA0B,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AAkB/E,wBAAsB,4BAA4B,CACjD,OAAO,EAAE;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EACxF,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,EAClC,OAAO,EAAE,CAAC,cAAc,EAAE,UAAU,EAAE,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAA;CAAE,KAAK,MAAM,GAAG,IAAI,GAC3F,OAAO,CAAC,gBAAgB,CAAC,CAsD3B"}
@@ -0,0 +1,293 @@
import { copyFileSync, existsSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { normalizeExpiration } from '../../util/expire.js';
import { hostMatchesCookieDomain } from '../../util/hostMatch.js';
import { importNodeSqlite, supportsReadBigInts } from '../../util/nodeSqlite.js';
import { isBunRuntime } from '../../util/runtime.js';
export async function getCookiesFromChromeSqliteDb(options, origins, allowlistNames, decrypt) {
const warnings = [];
// Chrome can keep its cookie DB locked and/or rely on WAL sidecars.
// Copying to a temp dir gives us a stable snapshot that both node:sqlite and bun:sqlite can open.
const tempDir = mkdtempSync(path.join(tmpdir(), 'sweet-cookie-chrome-'));
const tempDbPath = path.join(tempDir, 'Cookies');
try {
copyFileSync(options.dbPath, tempDbPath);
// If WAL is enabled, the latest writes might live in `Cookies-wal`/`Cookies-shm`.
// Copy them too when present so our snapshot reflects the current browser state.
copySidecar(options.dbPath, `${tempDbPath}-wal`, '-wal');
copySidecar(options.dbPath, `${tempDbPath}-shm`, '-shm');
}
catch (error) {
rmSync(tempDir, { recursive: true, force: true });
warnings.push(`Failed to copy Chrome cookie DB: ${error instanceof Error ? error.message : String(error)}`);
return { cookies: [], warnings };
}
try {
const hosts = origins.map((o) => new URL(o).hostname);
const where = buildHostWhereClause(hosts, 'host_key');
const metaVersion = await readChromiumMetaVersion(tempDbPath);
// Chromium >= 24 stores a 32-byte hash prefix in decrypted cookie values.
// We detect this via the `meta` table version and strip it when present.
const stripHashPrefix = metaVersion >= 24;
const rowsResult = await readChromeRows(tempDbPath, where);
if (!rowsResult.ok) {
warnings.push(rowsResult.error);
return { cookies: [], warnings };
}
const collectOptions = {};
if (options.profile)
collectOptions.profile = options.profile;
if (options.includeExpired !== undefined)
collectOptions.includeExpired = options.includeExpired;
const cookies = collectChromeCookiesFromRows(rowsResult.rows, collectOptions, hosts, allowlistNames, (encryptedValue) => decrypt(encryptedValue, { stripHashPrefix }), warnings);
return { cookies: dedupeCookies(cookies), warnings };
}
finally {
rmSync(tempDir, { recursive: true, force: true });
}
}
function collectChromeCookiesFromRows(rows, options, hosts, allowlistNames, decrypt, warnings) {
const cookies = [];
const now = Math.floor(Date.now() / 1000);
let warnedEncryptedType = false;
for (const row of rows) {
const name = typeof row.name === 'string' ? row.name : null;
if (!name)
continue;
if (allowlistNames && allowlistNames.size > 0 && !allowlistNames.has(name))
continue;
const hostKey = typeof row.host_key === 'string' ? row.host_key : null;
if (!hostKey)
continue;
if (!hostMatchesAny(hosts, hostKey))
continue;
const rowPath = typeof row.path === 'string' ? row.path : '';
const valueString = typeof row.value === 'string' ? row.value : null;
let value = valueString;
if (value === null || value.length === 0) {
// Many modern Chromium cookies keep `value` empty and only store `encrypted_value`.
// We decrypt on demand and drop rows we can't interpret.
const encryptedBytes = getEncryptedBytes(row);
if (!encryptedBytes) {
if (!warnedEncryptedType && row.encrypted_value !== undefined) {
warnings.push('Chrome cookie encrypted_value is in an unsupported type.');
warnedEncryptedType = true;
}
continue;
}
value = decrypt(encryptedBytes);
}
if (value === null)
continue;
const expiresRaw = typeof row.expires_utc === 'number' || typeof row.expires_utc === 'bigint'
? row.expires_utc
: tryParseInt(row.expires_utc);
const expires = normalizeExpiration(expiresRaw ?? undefined);
if (!options.includeExpired) {
if (expires && expires < now)
continue;
}
const secure = row.is_secure === 1 ||
row.is_secure === 1n ||
row.is_secure === '1' ||
row.is_secure === true;
const httpOnly = row.is_httponly === 1 ||
row.is_httponly === 1n ||
row.is_httponly === '1' ||
row.is_httponly === true;
const sameSite = normalizeChromiumSameSite(row.samesite);
const source = { browser: 'chrome' };
if (options.profile)
source.profile = options.profile;
const cookie = {
name,
value,
domain: hostKey.startsWith('.') ? hostKey.slice(1) : hostKey,
path: rowPath || '/',
secure,
httpOnly,
source,
};
if (expires !== undefined)
cookie.expires = expires;
if (sameSite !== undefined)
cookie.sameSite = sameSite;
cookies.push(cookie);
}
return cookies;
}
function tryParseInt(value) {
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? parsed : null;
}
if (typeof value !== 'string')
return null;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : null;
}
function normalizeChromiumSameSite(value) {
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? normalizeChromiumSameSite(parsed) : undefined;
}
if (typeof value === 'number') {
if (value === 2)
return 'Strict';
if (value === 1)
return 'Lax';
if (value === 0)
return 'None';
return undefined;
}
if (typeof value === 'string') {
const parsed = Number.parseInt(value, 10);
if (Number.isFinite(parsed))
return normalizeChromiumSameSite(parsed);
const normalized = value.toLowerCase();
if (normalized === 'strict')
return 'Strict';
if (normalized === 'lax')
return 'Lax';
if (normalized === 'none' || normalized === 'no_restriction')
return 'None';
}
return undefined;
}
function getEncryptedBytes(row) {
const raw = row.encrypted_value;
if (raw instanceof Uint8Array)
return raw;
return null;
}
async function readChromiumMetaVersion(dbPath) {
const sql = `SELECT value FROM meta WHERE key = 'version'`;
const result = isBunRuntime()
? await queryNodeOrBun({ kind: 'bun', dbPath, sql })
: await queryNodeOrBun({ kind: 'node', dbPath, sql });
if (!result.ok)
return 0;
const first = result.rows[0];
const value = first?.value;
if (typeof value === 'number')
return Math.floor(value);
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? Math.floor(parsed) : 0;
}
if (typeof value === 'string') {
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : 0;
}
return 0;
}
async function readChromeRows(dbPath, where) {
const sqliteKind = isBunRuntime() ? 'bun' : 'node';
const sqliteLabel = sqliteKind === 'bun' ? 'bun:sqlite' : 'node:sqlite';
const sql = `SELECT name, value, host_key, path, expires_utc, samesite, encrypted_value, ` +
`is_secure AS is_secure, is_httponly AS is_httponly ` +
`FROM cookies WHERE (${where}) ORDER BY expires_utc DESC;`;
const result = await queryNodeOrBun({ kind: sqliteKind, dbPath, sql });
if (result.ok)
return { ok: true, rows: result.rows };
// Intentionally strict: only support modern Chromium cookie DB schemas.
// If this fails, assume the local Chrome/Chromium is too old or uses a non-standard schema.
return {
ok: false,
error: `${sqliteLabel} failed reading Chrome cookies (requires modern Chromium, e.g. Chrome >= 100): ${result.error}`,
};
}
async function queryNodeOrBun(options) {
try {
if (options.kind === 'node') {
// Node's `node:sqlite` is synchronous and returns plain JS values. Keep it boxed in a
// small scope so callers don't need to care about runtime differences.
const { DatabaseSync } = await importNodeSqlite();
const dbOptions = { readOnly: true };
if (supportsReadBigInts()) {
dbOptions.readBigInts = true;
}
const db = new DatabaseSync(options.dbPath, dbOptions);
try {
const rows = db.prepare(options.sql).all();
return { ok: true, rows };
}
finally {
db.close();
}
}
// Bun's sqlite API has a different surface (`Database` + `.query().all()`).
const { Database } = await import('bun:sqlite');
const db = new Database(options.dbPath, { readonly: true });
try {
const rows = db.query(options.sql).all();
return { ok: true, rows };
}
finally {
db.close();
}
}
catch (error) {
return { ok: false, error: error instanceof Error ? error.message : String(error) };
}
}
function copySidecar(sourceDbPath, target, suffix) {
const sidecar = `${sourceDbPath}${suffix}`;
if (!existsSync(sidecar))
return;
try {
copyFileSync(sidecar, target);
}
catch {
// ignore
}
}
function buildHostWhereClause(hosts, column) {
const clauses = [];
for (const host of hosts) {
// Chrome cookies often live on parent domains (e.g. .google.com for gemini.google.com).
// Include parent domains so the SQL filter doesn't drop valid session cookies.
for (const candidate of expandHostCandidates(host)) {
const escaped = sqlLiteral(candidate);
const escapedDot = sqlLiteral(`.${candidate}`);
const escapedLike = sqlLiteral(`%.${candidate}`);
clauses.push(`${column} = ${escaped}`);
clauses.push(`${column} = ${escapedDot}`);
clauses.push(`${column} LIKE ${escapedLike}`);
}
}
return clauses.length ? clauses.join(' OR ') : '1=0';
}
function sqlLiteral(value) {
const escaped = value.replaceAll("'", "''");
return `'${escaped}'`;
}
function expandHostCandidates(host) {
const parts = host.split('.').filter(Boolean);
if (parts.length <= 1)
return [host];
const candidates = new Set();
candidates.add(host);
// Include parent domains down to two labels (avoid TLD-only fragments).
for (let i = 1; i <= parts.length - 2; i += 1) {
const candidate = parts.slice(i).join('.');
if (candidate)
candidates.add(candidate);
}
return Array.from(candidates);
}
function hostMatchesAny(hosts, cookieHost) {
const cookieDomain = cookieHost.startsWith('.') ? cookieHost.slice(1) : cookieHost;
return hosts.some((host) => hostMatchesCookieDomain(host, cookieDomain));
}
function dedupeCookies(cookies) {
const merged = new Map();
for (const cookie of cookies) {
const key = `${cookie.name}|${cookie.domain ?? ''}|${cookie.path ?? ''}`;
if (!merged.has(key))
merged.set(key, cookie);
}
return Array.from(merged.values());
}
//# sourceMappingURL=shared.js.map
File diff suppressed because one or more lines are too long
@@ -0,0 +1,10 @@
export declare function dpapiUnprotect(data: Buffer, options?: {
timeoutMs?: number;
}): Promise<{
ok: true;
value: Buffer;
} | {
ok: false;
error: string;
}>;
//# sourceMappingURL=windowsDpapi.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"windowsDpapi.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/windowsDpapi.ts"],"names":[],"mappings":"AAEA,wBAAsB,cAAc,CACnC,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAO,GAClC,OAAO,CAAC;IAAE,EAAE,EAAE,IAAI,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC,CA+BrE"}
@@ -0,0 +1,26 @@
import { execCapture } from '../../util/exec.js';
export async function dpapiUnprotect(data, options = {}) {
const timeoutMs = options.timeoutMs ?? 5_000;
// There is no cross-platform JS API for Windows DPAPI, and we explicitly avoid native addons.
// PowerShell can call ProtectedData.Unprotect for the current user, which matches Chrome's behavior.
const inputB64 = data.toString('base64');
const prelude = 'try { Add-Type -AssemblyName System.Security.Cryptography.ProtectedData -ErrorAction Stop } catch { try { Add-Type -AssemblyName System.Security -ErrorAction Stop } catch {} };';
const script = prelude +
`$in=[Convert]::FromBase64String('${inputB64}');` +
`$out=[System.Security.Cryptography.ProtectedData]::Unprotect($in,$null,[System.Security.Cryptography.DataProtectionScope]::CurrentUser);` +
`[Convert]::ToBase64String($out)`;
const res = await execCapture('powershell', ['-NoProfile', '-NonInteractive', '-Command', script], {
timeoutMs,
});
if (res.code !== 0) {
return { ok: false, error: res.stderr.trim() || `powershell exit ${res.code}` };
}
try {
const out = Buffer.from(res.stdout.trim(), 'base64');
return { ok: true, value: out };
}
catch (error) {
return { ok: false, error: error instanceof Error ? error.message : String(error) };
}
}
//# sourceMappingURL=windowsDpapi.js.map
@@ -0,0 +1 @@
{"version":3,"file":"windowsDpapi.js","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/windowsDpapi.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AAEjD,MAAM,CAAC,KAAK,UAAU,cAAc,CACnC,IAAY,EACZ,UAAkC,EAAE;IAEpC,MAAM,SAAS,GAAG,OAAO,CAAC,SAAS,IAAI,KAAK,CAAC;IAE7C,8FAA8F;IAC9F,qGAAqG;IACrG,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;IACzC,MAAM,OAAO,GACZ,kLAAkL,CAAC;IACpL,MAAM,MAAM,GACX,OAAO;QACP,oCAAoC,QAAQ,KAAK;QACjD,0IAA0I;QAC1I,iCAAiC,CAAC;IAEnC,MAAM,GAAG,GAAG,MAAM,WAAW,CAC5B,YAAY,EACZ,CAAC,YAAY,EAAE,iBAAiB,EAAE,UAAU,EAAE,MAAM,CAAC,EACrD;QACC,SAAS;KACT,CACD,CAAC;IACF,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QACpB,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,IAAI,mBAAmB,GAAG,CAAC,IAAI,EAAE,EAAE,CAAC;IACjF,CAAC;IAED,IAAI,CAAC;QACJ,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,QAAQ,CAAC,CAAC;QACrD,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC;IACjC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QAChB,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC;IACrF,CAAC;AACF,CAAC"}
@@ -0,0 +1,7 @@
import type { GetCookiesResult } from '../types.js';
export declare function getCookiesFromChromeSqliteLinux(options: {
profile?: string;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null): Promise<GetCookiesResult>;
//# sourceMappingURL=chromeSqliteLinux.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteLinux.d.ts","sourceRoot":"","sources":["../../src/providers/chromeSqliteLinux.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AASpD,wBAAsB,+BAA+B,CACpD,OAAO,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EACxE,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,GAChC,OAAO,CAAC,gBAAgB,CAAC,CAkD3B"}
@@ -0,0 +1,51 @@
import { decryptChromiumAes128CbcCookieValue, deriveAes128CbcKeyFromPassword, } from './chromeSqlite/crypto.js';
import { getLinuxChromeSafeStoragePassword } from './chromeSqlite/linuxKeyring.js';
import { getCookiesFromChromeSqliteDb } from './chromeSqlite/shared.js';
import { resolveChromiumCookiesDbLinux } from './chromium/linuxPaths.js';
export async function getCookiesFromChromeSqliteLinux(options, origins, allowlistNames) {
const args = {
configDirName: 'google-chrome',
};
if (options.profile !== undefined)
args.profile = options.profile;
const dbPath = resolveChromiumCookiesDbLinux(args);
if (!dbPath) {
return { cookies: [], warnings: ['Chrome cookies database not found.'] };
}
const { password, warnings: keyringWarnings } = await getLinuxChromeSafeStoragePassword();
// Linux uses multiple schemes depending on distro/keyring availability.
// - v10 often uses the hard-coded "peanuts" password
// - v11 uses "Chrome Safe Storage" from the keyring (may be empty/unavailable)
const v10Key = deriveAes128CbcKeyFromPassword('peanuts', { iterations: 1 });
const emptyKey = deriveAes128CbcKeyFromPassword('', { iterations: 1 });
const v11Key = deriveAes128CbcKeyFromPassword(password, { iterations: 1 });
const decrypt = (encryptedValue, opts) => {
const prefix = Buffer.from(encryptedValue).subarray(0, 3).toString('utf8');
if (prefix === 'v10') {
return decryptChromiumAes128CbcCookieValue(encryptedValue, [v10Key, emptyKey], {
stripHashPrefix: opts.stripHashPrefix,
treatUnknownPrefixAsPlaintext: false,
});
}
if (prefix === 'v11') {
return decryptChromiumAes128CbcCookieValue(encryptedValue, [v11Key, emptyKey], {
stripHashPrefix: opts.stripHashPrefix,
treatUnknownPrefixAsPlaintext: false,
});
}
return null;
};
const dbOptions = {
dbPath,
};
if (options.profile)
dbOptions.profile = options.profile;
if (options.includeExpired !== undefined)
dbOptions.includeExpired = options.includeExpired;
if (options.debug !== undefined)
dbOptions.debug = options.debug;
const result = await getCookiesFromChromeSqliteDb(dbOptions, origins, allowlistNames, decrypt);
result.warnings.unshift(...keyringWarnings);
return result;
}
//# sourceMappingURL=chromeSqliteLinux.js.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteLinux.js","sourceRoot":"","sources":["../../src/providers/chromeSqliteLinux.ts"],"names":[],"mappings":"AACA,OAAO,EACN,mCAAmC,EACnC,8BAA8B,GAC9B,MAAM,0BAA0B,CAAC;AAClC,OAAO,EAAE,iCAAiC,EAAE,MAAM,gCAAgC,CAAC;AACnF,OAAO,EAAE,4BAA4B,EAAE,MAAM,0BAA0B,CAAC;AACxE,OAAO,EAAE,6BAA6B,EAAE,MAAM,0BAA0B,CAAC;AAEzE,MAAM,CAAC,KAAK,UAAU,+BAA+B,CACpD,OAAwE,EACxE,OAAiB,EACjB,cAAkC;IAElC,MAAM,IAAI,GAAwD;QACjE,aAAa,EAAE,eAAe;KAC9B,CAAC;IACF,IAAI,OAAO,CAAC,OAAO,KAAK,SAAS;QAAE,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IAClE,MAAM,MAAM,GAAG,6BAA6B,CAAC,IAAI,CAAC,CAAC;IACnD,IAAI,CAAC,MAAM,EAAE,CAAC;QACb,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC,oCAAoC,CAAC,EAAE,CAAC;IAC1E,CAAC;IAED,MAAM,EAAE,QAAQ,EAAE,QAAQ,EAAE,eAAe,EAAE,GAAG,MAAM,iCAAiC,EAAE,CAAC;IAE1F,wEAAwE;IACxE,qDAAqD;IACrD,+EAA+E;IAC/E,MAAM,MAAM,GAAG,8BAA8B,CAAC,SAAS,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAC,CAAC;IAC5E,MAAM,QAAQ,GAAG,8BAA8B,CAAC,EAAE,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAC,CAAC;IACvE,MAAM,MAAM,GAAG,8BAA8B,CAAC,QAAQ,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAC,CAAC;IAE3E,MAAM,OAAO,GAAG,CACf,cAA0B,EAC1B,IAAkC,EAClB,EAAE;QAClB,MAAM,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;QAC3E,IAAI,MAAM,KAAK,KAAK,EAAE,CAAC;YACtB,OAAO,mCAAmC,CAAC,cAAc,EAAE,CAAC,MAAM,EAAE,QAAQ,CAAC,EAAE;gBAC9E,eAAe,EAAE,IAAI,CAAC,eAAe;gBACrC,6BAA6B,EAAE,KAAK;aACpC,CAAC,CAAC;QACJ,CAAC;QACD,IAAI,MAAM,KAAK,KAAK,EAAE,CAAC;YACtB,OAAO,mCAAmC,CAAC,cAAc,EAAE,CAAC,MAAM,EAAE,QAAQ,CAAC,EAAE;gBAC9E,eAAe,EAAE,IAAI,CAAC,eAAe;gBACrC,6BAA6B,EAAE,KAAK;aACpC,CAAC,CAAC;QACJ,CAAC;QACD,OAAO,IAAI,CAAC;IACb,CAAC,CAAC;IAEF,MAAM,SAAS,GACd;QACC,MAAM;KACN,CAAC;IACH,IAAI,OAAO,CAAC,OAAO;QAAE,SAAS,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IACzD,IAAI,OAAO,CAAC,cAAc,KAAK,SAAS;QAAE,SAAS,CAAC,cAAc,GAAG,OAAO,CAAC,cAAc,CAAC;IAC5F,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS;QAAE,SAAS,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC;IAEjE,MAAM,MAAM,GAAG,MAAM,4BAA4B,CAAC,SAAS,EAAE,OAAO,EAAE,cAAc,EAAE,OAAO,CAAC,CAAC;IAC/F,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,eAAe,CAAC,CAAC;IAC5C,OAAO,MAAM,CAAC;AACf,CAAC"}
@@ -0,0 +1,7 @@
import type { GetCookiesResult } from '../types.js';
export declare function getCookiesFromChromeSqliteMac(options: {
profile?: string;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null): Promise<GetCookiesResult>;
//# sourceMappingURL=chromeSqliteMac.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteMac.d.ts","sourceRoot":"","sources":["../../src/providers/chromeSqliteMac.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AASpD,wBAAsB,6BAA6B,CAClD,OAAO,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EACxE,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,GAChC,OAAO,CAAC,gBAAgB,CAAC,CA6C3B"}
@@ -0,0 +1,60 @@
import { homedir } from 'node:os';
import path from 'node:path';
import { decryptChromiumAes128CbcCookieValue, deriveAes128CbcKeyFromPassword, } from './chromeSqlite/crypto.js';
import { getCookiesFromChromeSqliteDb } from './chromeSqlite/shared.js';
import { readKeychainGenericPasswordFirst } from './chromium/macosKeychain.js';
import { resolveCookiesDbFromProfileOrRoots } from './chromium/paths.js';
export async function getCookiesFromChromeSqliteMac(options, origins, allowlistNames) {
const dbPath = resolveChromeCookiesDb(options.profile);
if (!dbPath) {
return { cookies: [], warnings: ['Chrome cookies database not found.'] };
}
const warnings = [];
// On macOS, Chrome stores its "Safe Storage" secret in Keychain.
// `security find-generic-password` is stable and avoids any native Node keychain modules.
const passwordResult = await readKeychainGenericPasswordFirst({
account: 'Chrome',
services: ['Chrome Safe Storage'],
timeoutMs: 3_000,
label: 'Chrome Safe Storage',
});
if (!passwordResult.ok) {
warnings.push(passwordResult.error);
return { cookies: [], warnings };
}
const chromePassword = passwordResult.password.trim();
if (!chromePassword) {
warnings.push('macOS Keychain returned an empty Chrome Safe Storage password.');
return { cookies: [], warnings };
}
// Chromium uses PBKDF2(password, "saltysalt", 1003, 16, sha1) for AES-128-CBC cookie values on macOS.
const key = deriveAes128CbcKeyFromPassword(chromePassword, { iterations: 1003 });
const decrypt = (encryptedValue, opts) => decryptChromiumAes128CbcCookieValue(encryptedValue, [key], {
stripHashPrefix: opts.stripHashPrefix,
treatUnknownPrefixAsPlaintext: true,
});
const dbOptions = {
dbPath,
};
if (options.profile)
dbOptions.profile = options.profile;
if (options.includeExpired !== undefined)
dbOptions.includeExpired = options.includeExpired;
if (options.debug !== undefined)
dbOptions.debug = options.debug;
const result = await getCookiesFromChromeSqliteDb(dbOptions, origins, allowlistNames, decrypt);
result.warnings.unshift(...warnings);
return result;
}
function resolveChromeCookiesDb(profile) {
const home = homedir();
/* c8 ignore next */
const roots = process.platform === 'darwin'
? [path.join(home, 'Library', 'Application Support', 'Google', 'Chrome')]
: [];
const args = { roots };
if (profile !== undefined)
args.profile = profile;
return resolveCookiesDbFromProfileOrRoots(args);
}
//# sourceMappingURL=chromeSqliteMac.js.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteMac.js","sourceRoot":"","sources":["../../src/providers/chromeSqliteMac.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,SAAS,CAAC;AAClC,OAAO,IAAI,MAAM,WAAW,CAAC;AAG7B,OAAO,EACN,mCAAmC,EACnC,8BAA8B,GAC9B,MAAM,0BAA0B,CAAC;AAClC,OAAO,EAAE,4BAA4B,EAAE,MAAM,0BAA0B,CAAC;AACxE,OAAO,EAAE,gCAAgC,EAAE,MAAM,6BAA6B,CAAC;AAC/E,OAAO,EAAE,kCAAkC,EAAE,MAAM,qBAAqB,CAAC;AAEzE,MAAM,CAAC,KAAK,UAAU,6BAA6B,CAClD,OAAwE,EACxE,OAAiB,EACjB,cAAkC;IAElC,MAAM,MAAM,GAAG,sBAAsB,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IACvD,IAAI,CAAC,MAAM,EAAE,CAAC;QACb,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC,oCAAoC,CAAC,EAAE,CAAC;IAC1E,CAAC;IAED,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,iEAAiE;IACjE,0FAA0F;IAC1F,MAAM,cAAc,GAAG,MAAM,gCAAgC,CAAC;QAC7D,OAAO,EAAE,QAAQ;QACjB,QAAQ,EAAE,CAAC,qBAAqB,CAAC;QACjC,SAAS,EAAE,KAAK;QAChB,KAAK,EAAE,qBAAqB;KAC5B,CAAC,CAAC;IACH,IAAI,CAAC,cAAc,CAAC,EAAE,EAAE,CAAC;QACxB,QAAQ,CAAC,IAAI,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IAClC,CAAC;IAED,MAAM,cAAc,GAAG,cAAc,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;IACtD,IAAI,CAAC,cAAc,EAAE,CAAC;QACrB,QAAQ,CAAC,IAAI,CAAC,gEAAgE,CAAC,CAAC;QAChF,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IAClC,CAAC;IAED,sGAAsG;IACtG,MAAM,GAAG,GAAG,8BAA8B,CAAC,cAAc,EAAE,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC,CAAC;IACjF,MAAM,OAAO,GAAG,CAAC,cAA0B,EAAE,IAAkC,EAAiB,EAAE,CACjG,mCAAmC,CAAC,cAAc,EAAE,CAAC,GAAG,CAAC,EAAE;QAC1D,eAAe,EAAE,IAAI,CAAC,eAAe;QACrC,6BAA6B,EAAE,IAAI;KACnC,CAAC,CAAC;IAEJ,MAAM,SAAS,GACd;QACC,MAAM;KACN,CAAC;IACH,IAAI,OAAO,CAAC,OAAO;QAAE,SAAS,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IACzD,IAAI,OAAO,CAAC,cAAc,KAAK,SAAS;QAAE,SAAS,CAAC,cAAc,GAAG,OAAO,CAAC,cAAc,CAAC;IAC5F,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS;QAAE,SAAS,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC;IAEjE,MAAM,MAAM,GAAG,MAAM,4BAA4B,CAAC,SAAS,EAAE,OAAO,EAAE,cAAc,EAAE,OAAO,CAAC,CAAC;IAC/F,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,GAAG,QAAQ,CAAC,CAAC;IACrC,OAAO,MAAM,CAAC;AACf,CAAC;AAED,SAAS,sBAAsB,CAAC,OAAgB;IAC/C,MAAM,IAAI,GAAG,OAAO,EAAE,CAAC;IACvB,oBAAoB;IACpB,MAAM,KAAK,GACV,OAAO,CAAC,QAAQ,KAAK,QAAQ;QAC5B,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,SAAS,EAAE,qBAAqB,EAAE,QAAQ,EAAE,QAAQ,CAAC,CAAC;QACzE,CAAC,CAAC,EAAE,CAAC;IACP,MAAM,IAAI,GAA6D,EAAE,KAAK,EAAE,CAAC;IACjF,IAAI,OAAO,KAAK,SAAS;QAAE,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;IAClD,OAAO,kCAAkC,CAAC,IAAI,CAAC,CAAC;AACjD,CAAC"}
@@ -0,0 +1,7 @@
import type { GetCookiesResult } from '../types.js';
export declare function getCookiesFromChromeSqliteWindows(options: {
profile?: string;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null): Promise<GetCookiesResult>;
//# sourceMappingURL=chromeSqliteWindows.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteWindows.d.ts","sourceRoot":"","sources":["../../src/providers/chromeSqliteWindows.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAMpD,wBAAsB,iCAAiC,CACtD,OAAO,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EACxE,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,GAChC,OAAO,CAAC,gBAAgB,CAAC,CAmC3B"}
@@ -0,0 +1,38 @@
import path from 'node:path';
import { decryptChromiumAes256GcmCookieValue } from './chromeSqlite/crypto.js';
import { getCookiesFromChromeSqliteDb } from './chromeSqlite/shared.js';
import { getWindowsChromiumMasterKey } from './chromium/windowsMasterKey.js';
import { resolveChromiumPathsWindows } from './chromium/windowsPaths.js';
export async function getCookiesFromChromeSqliteWindows(options, origins, allowlistNames) {
const resolveArgs = {
localAppDataVendorPath: path.join('Google', 'Chrome', 'User Data'),
};
if (options.profile !== undefined)
resolveArgs.profile = options.profile;
const { dbPath, userDataDir } = resolveChromiumPathsWindows(resolveArgs);
if (!dbPath || !userDataDir) {
return { cookies: [], warnings: ['Chrome cookies database not found.'] };
}
// On Windows, Chrome stores an AES key in `Local State` encrypted with DPAPI (CurrentUser).
// That master key is then used for AES-256-GCM cookie values (`v10`/`v11`/`v20` prefixes).
const masterKey = await getWindowsChromiumMasterKey(userDataDir, 'Chrome');
if (!masterKey.ok) {
return { cookies: [], warnings: [masterKey.error] };
}
const decrypt = (encryptedValue, opts) => {
return decryptChromiumAes256GcmCookieValue(encryptedValue, masterKey.value, {
stripHashPrefix: opts.stripHashPrefix,
});
};
const dbOptions = {
dbPath,
};
if (options.profile)
dbOptions.profile = options.profile;
if (options.includeExpired !== undefined)
dbOptions.includeExpired = options.includeExpired;
if (options.debug !== undefined)
dbOptions.debug = options.debug;
return await getCookiesFromChromeSqliteDb(dbOptions, origins, allowlistNames, decrypt);
}
//# sourceMappingURL=chromeSqliteWindows.js.map
@@ -0,0 +1 @@
{"version":3,"file":"chromeSqliteWindows.js","sourceRoot":"","sources":["../../src/providers/chromeSqliteWindows.ts"],"names":[],"mappings":"AAAA,OAAO,IAAI,MAAM,WAAW,CAAC;AAG7B,OAAO,EAAE,mCAAmC,EAAE,MAAM,0BAA0B,CAAC;AAC/E,OAAO,EAAE,4BAA4B,EAAE,MAAM,0BAA0B,CAAC;AACxE,OAAO,EAAE,2BAA2B,EAAE,MAAM,gCAAgC,CAAC;AAC7E,OAAO,EAAE,2BAA2B,EAAE,MAAM,4BAA4B,CAAC;AAEzE,MAAM,CAAC,KAAK,UAAU,iCAAiC,CACtD,OAAwE,EACxE,OAAiB,EACjB,cAAkC;IAElC,MAAM,WAAW,GAAsD;QACtE,sBAAsB,EAAE,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,QAAQ,EAAE,WAAW,CAAC;KAClE,CAAC;IACF,IAAI,OAAO,CAAC,OAAO,KAAK,SAAS;QAAE,WAAW,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IACzE,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,2BAA2B,CAAC,WAAW,CAAC,CAAC;IACzE,IAAI,CAAC,MAAM,IAAI,CAAC,WAAW,EAAE,CAAC;QAC7B,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC,oCAAoC,CAAC,EAAE,CAAC;IAC1E,CAAC;IAED,4FAA4F;IAC5F,2FAA2F;IAC3F,MAAM,SAAS,GAAG,MAAM,2BAA2B,CAAC,WAAW,EAAE,QAAQ,CAAC,CAAC;IAC3E,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,CAAC;QACnB,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,CAAC;IACrD,CAAC;IAED,MAAM,OAAO,GAAG,CACf,cAA0B,EAC1B,IAAkC,EAClB,EAAE;QAClB,OAAO,mCAAmC,CAAC,cAAc,EAAE,SAAS,CAAC,KAAK,EAAE;YAC3E,eAAe,EAAE,IAAI,CAAC,eAAe;SACrC,CAAC,CAAC;IACJ,CAAC,CAAC;IAEF,MAAM,SAAS,GACd;QACC,MAAM;KACN,CAAC;IACH,IAAI,OAAO,CAAC,OAAO;QAAE,SAAS,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IACzD,IAAI,OAAO,CAAC,cAAc,KAAK,SAAS;QAAE,SAAS,CAAC,cAAc,GAAG,OAAO,CAAC,cAAc,CAAC;IAC5F,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS;QAAE,SAAS,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC;IAEjE,OAAO,MAAM,4BAA4B,CAAC,SAAS,EAAE,OAAO,EAAE,cAAc,EAAE,OAAO,CAAC,CAAC;AACxF,CAAC"}
@@ -0,0 +1,5 @@
export declare function resolveChromiumCookiesDbLinux(options: {
configDirName: string;
profile?: string;
}): string | null;
//# sourceMappingURL=linuxPaths.d.ts.map
@@ -0,0 +1 @@
{"version":3,"file":"linuxPaths.d.ts","sourceRoot":"","sources":["../../../src/providers/chromium/linuxPaths.ts"],"names":[],"mappings":"AAMA,wBAAgB,6BAA6B,CAAC,OAAO,EAAE;IACtD,aAAa,EAAE,MAAM,CAAC;IACtB,OAAO,CAAC,EAAE,MAAM,CAAC;CACjB,GAAG,MAAM,GAAG,IAAI,CA0BhB"}

Some files were not shown because too many files have changed in this diff Show More