Compare commits

..

82 Commits

Author SHA1 Message Date
Matt Van Horn 3803f3df4b feat: v2.9.6 — free-first NUX, cookie extraction, quality scoring
Setup wizard with consent-first cookie extraction (Chrome/Firefox/Safari),
yt-dlp auto-install, ScrapeCreators push, quality scoring (5 core sources),
status banner redesign, honest Reddit labeling, inline YouTube transcripts,
Exa free web search, Reddit public fallback, and post-research quality nudge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 14:32:52 -07:00
Matt Van Horn 4d6224f79a feat(youtube): extract transcript highlights like Reddit comment gems
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.

SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:49:38 -07:00
Matt Van Horn 499074b564 fix(youtube): pass full transcripts to LLM instead of truncating to 200 chars
TRANSCRIPT_MAX_WORDS raised from 500 to 5000 so the LLM gets the full
content of most videos (up to ~25 minutes). Removed the second 200-char
truncation in render.py that was reducing transcripts to a single sentence
before the judge agent ever saw them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:28:01 -07:00
Matt Van Horn 6a5a0013c0 Merge pull request #70 from mvanhorn/fix/bluesky-cloudflare-error-messages
fix(bluesky): surface real error instead of misleading 'auth failed'
2026-03-15 22:56:00 -07:00
Matt Van Horn 0c6358ab98 fix(bluesky): surface real error instead of misleading "auth failed"
When Cloudflare blocks requests to bsky.social or public.api.bsky.app
with a 403, the error was swallowed by a generic except clause and
reported as "Bluesky auth failed" - misleading users into thinking
their credentials were wrong.

Now _create_session() preserves the specific error in _session_error,
and search_bluesky() surfaces it. Cloudflare 403s get a clear message
about network-level blocks. Actual 401s say "Invalid credentials".

Closes #69

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 22:13:28 -07:00
Matt Van Horn 7ffa508bb4 feat(last30days): improve ClawHub discoverability - description, tags
Updated frontmatter description to be more search-friendly for ClawHub.
Added 12 new tags: deep-research, twitter, bluesky, recency, news,
citations, multi-source, social-media, analysis, web-search, ai-skill,
clawhub. Also added 11 GitHub repo topics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 17:27:53 -07:00
Matt Van Horn c2e18dbbeb fix(skill): use full /last30days name in comparison follow-up suggestions
The short alias /last30 only works on some platforms. Claude Code requires
the full /last30days name, so the follow-up suggestions after comparison
research were producing "Unknown skill: last30" errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 07:55:17 -07:00
Matt Van Horn 52a22f5cb1 Merge pull request #65 from j-sperling/feat/search-quality-consolidation
Consolidate query/relevance modules and improve search quality
2026-03-14 07:31:24 -07:00
Matt Van Horn 3830274e11 Merge pull request #67 from j-sperling/feat/query-type-source-tiering
Add query-type-aware source tiering and scoring
2026-03-14 07:30:06 -07:00
Matt Van Horn 569745c0ed Merge pull request #66 from j-sperling/fix/endpoint-model-updates
Fix stale API endpoints and model identifiers
2026-03-14 07:29:56 -07:00
Jeffrey Sperling 058c4e1899 Classify prompt and animation queries earlier
Map prompt-oriented product searches and animation-oriented build searches away from the breaking-news default so source tiering and tiebreakers align with the benchmark topics.

Validation: uv run python -m unittest tests.test_query_type
2026-03-14 00:38:59 -07:00
Jeffrey Sperling c711e443fe Reduce Reddit and Polymarket false positives
Weight Reddit relevance toward titles, stop Polymarket from expanding low-signal standalone terms, and prevent short binary outcomes from matching unrelated queries.

Validation: uv run python -m unittest tests.test_reddit_sc tests.test_polymarket
2026-03-14 00:38:52 -07:00
Jeffrey Sperling 8c1dce95e8 Harden local search evaluation harness
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.

Validation: uv run python -m unittest tests.test_evaluate_search_quality
2026-03-14 00:38:43 -07:00
Jeffrey Sperling 3a0f3d8b19 Accept GOOGLE_API_KEY for local Gemini eval
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.

Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
2026-03-13 19:25:29 -07:00
Jeffrey Sperling 8eda5fad5c Add local search quality evaluation harness
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.

The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.

Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
2026-03-13 19:21:33 -07:00
Jeffrey Sperling 946af84f9a Tighten relevance scoring and Polymarket ranking
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.

Also apply the relevance filter to Polymarket and update the affected cross-source tests.

Validation: uv run python -m unittest
2026-03-13 19:21:25 -07:00
Jeffrey Sperling 0e46c7cb33 Pass X auth through handle drilldowns
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.

Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
2026-03-13 01:09:03 -07:00
Jeffrey Sperling b489663450 Loosen source tiering for usage queries
Classify prompting and animation queries as how_to so the stack does not treat them as generic breaking news. Also keep X available for how_to and preserve YouTube/HN coverage for breaking-news and prediction queries.

Validated with uv run python -m unittest tests.test_query_type and the five-query local comparison run used for PR #65 review.
2026-03-13 01:08:28 -07:00
Jeffrey Sperling 3aaf31b08d Document env-based X auth flow
Update README, launch copy, and UI guidance to prefer popup-free AUTH_TOKEN/CT0 configuration, and keep X backend selection on the verified Bird or xAI paths.

Validation: uv run python -m unittest tests.test_env_project
2026-03-12 21:07:09 -07:00
Jeffrey Sperling dd9a3f1482 Disable browser cookie fallback for local X auth
Prefer injected AUTH_TOKEN/CT0 for bundled Bird, disable browser-cookie probing in repo-invoked subprocesses, and keep repo-invoked yt-dlp from inheriting browser-cookie settings.

Validation: uv run python -m unittest tests.test_bird_x tests.test_youtube_yt
2026-03-12 21:07:04 -07:00
Jeffrey Sperling cbee987f65 Extract relevance_filter, add Bluesky/TruthSocial type hint + test coverage
- Extract _relevance_filter from last30days.py closure to score.relevance_filter()
  for testability
- Add BlueskyItem/TruthSocialItem to sort_items() type hint (was missing despite
  being in _ITEM_SOURCE_MAP)
- Add tests: Bluesky/TruthSocial engagement scoring, sort_items mixed sources,
  relevance_filter behavior (threshold, minimum-result guarantee, missing attr),
  select_openai_model HTTP 401/403 error paths
2026-03-11 19:09:06 -07:00
Jeffrey Sperling d8d2b97716 Gitignore mise.toml instead of removing it
Dev environment tool config is useful locally but shouldn't be tracked.
2026-03-11 19:01:33 -07:00
Jeffrey Sperling 036bcd2ae3 Address review feedback: deduplicate query_type, clean unused imports, fix defaults
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
  canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
  tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
  should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
2026-03-11 18:40:07 -07:00
Jeffrey Sperling 6c402f66b7 Update plan to reflect single upstream PR strategy 2026-03-11 18:32:45 -07:00
Jeffrey Sperling 046795c4ae Add post-retrieval relevance filtering across all sources
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.

This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 1002f1f020 Add platform-specific query optimizations
- hackernews: use extract_core_subject instead of raw topic, add
  points>5 filter and restrictSearchableAttributes=title to reduce
  noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
  filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
  queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
  to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
2026-03-11 18:32:45 -07:00
Jeffrey Sperling c5be117701 Replace hardcoded 0.7 relevance with computed token-overlap scores
- bird_x: parse_bird_response now accepts query param and computes
  token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement

This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 38caae3288 Add mise.toml for Python version pinning and implementation plan
Pin Python 3.12 via mise for consistent local development.
Add plan document for the query/relevance consolidation work.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 96948cc7c0 Deduplicate relevance code across youtube/tiktok/instagram/scrapecreators_x
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.

Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling dc88c215be Integrate shared query.py into per-source modules
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.

Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set

Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling fa42a5d031 Add urllib fallback for TikTok/Instagram when requests unavailable
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00
Matt Van Horn b38703e53d feat(truthsocial): Add Truth Social as opt-in source
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.

27 new tests, 440 total passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 00:14:39 -07:00
Matt Van Horn b6fd5ff406 docs: v2.9.5 README update, fix plugin.json hooks (#62)
- Fix plugin.json hooks field from invalid array to empty object
- Add Claude Code plugin install above ClawHub badge
- Version bump to v2.9.5 with new features block (Bluesky, comparative
  mode, ScrapeCreators X, per-project env, expanded tests)
- Add Bluesky to all source list references
- Document BSKY_HANDLE/BSKY_APP_PASSWORD env vars in install + optional sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:40:07 -07:00
Matt Van Horn adb5a672d9 fix(bluesky): make Bluesky opt-in with app password auth
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:27:39 -07:00
Matt Van Horn ecf90c0281 feat(skill): add comparative mode and Bluesky references to SKILL.md
Add COMPARISON query type for "X vs Y" research with 3 parallel passes.
Add Bluesky stats line and update all source list references.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:58:21 -07:00
Matt Van Horn 9a1059ee9d feat(bluesky): add Bluesky/AT Protocol as social source
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).

- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:54:20 -07:00
Matt Van Horn 4b7087e136 feat(x): add ScrapeCreators as X/Twitter search backend
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).

New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.

Fixes #55.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:34:39 -07:00
Matt Van Horn 82a006280e Merge pull request #58 from phjlljp/feat/session-start-config-check
feat: add SessionStart hook for config check
2026-03-09 21:55:57 -07:00
Matt Van Horn a7398c50cc Merge pull request #59 from phjlljp/feat/per-project-env-config
feat: add per-project .env config support
2026-03-09 21:55:51 -07:00
Matt Van Horn b1a0e2bcfc Merge pull request #57 from phjlljp/feat/smoke-tests-edge-cases
test: add smoke tests and edge case coverage
2026-03-09 21:55:44 -07:00
Matt Van Horn 75e4b8e2cd Merge pull request #56 from phjlljp/feat/unit-tests-untested-modules
test: add unit tests for untested modules
2026-03-09 21:55:42 -07:00
Matt Van Horn 25e27bdade fix(skill): unquote $ARGUMENTS and add marketplace plugin path
- Remove double quotes around $ARGUMENTS so argparse can parse flags
  like --deep, --store separately instead of as part of the topic string.
  Fixes #61.

- Add ~/.claude/plugins/marketplaces/last30days-skill to the path
  discovery loop so marketplace installs can find scripts/.
  Fixes #54.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:55:24 -07:00
P 3e3615e2c8 feat: add per-project .env config support
Add per-project configuration via .claude/last30days.env, discovered by
walking up from cwd. Uses the same .env format as the existing global
config — no new parsers or formats.

Priority (highest wins):
  1. Environment variables
  2. .claude/last30days.env (per-project)
  3. ~/.config/last30days/.env (global)

Also adds file permission checking — warns to stderr if config files
are readable by other users (should be chmod 600).

Includes tests for discovery, precedence, source tracking, and
permission warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:48:23 -04:00
P 9a58fe6481 feat: add SessionStart hook for config check
Add a lightweight hook that runs on session start to check if any
API keys are configured. Warns users if no config is found and
checks file permissions on existing config files.

Checks (in order): .claude/last30days.env, ~/.config/last30days/.env,
OPENAI_API_KEY env var, SCRAPECREATORS_API_KEY env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:47:36 -04:00
P 4756c20ec0 refactor: match upstream unittest convention
Convert all new tests from bare pytest style to unittest.TestCase
with sys.path.insert, matching the convention used by all existing
tests. Remove pyproject.toml and conftest.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:43:04 -04:00
P 8a9f734d14 test: add smoke tests and edge case coverage
Add end-to-end smoke tests and expand coverage for score and render modules:

- test_smoke.py (new): subprocess tests for --diagnose, --help, --mock,
  and missing topic. Validates exit codes, JSON structure, and source
  detection (HN/Polymarket always available).
- test_score.py: add TestCommentQualityWeight (top_comment_score boost),
  TestInstagramEngagement (basic scoring, views vs likes weighting)
- test_render.py: add TestEnsureOutputDir, TestXrefTag, TestRenderEmptyReport

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:41:58 -04:00
P 0979f506db test: add unit tests for untested modules
Add pytest infrastructure (pyproject.toml, conftest.py) and unit tests
for modules that previously had zero test coverage:

- test_schema_roundtrip.py: to_dict() serialization for all data classes
- test_reddit_enrich.py: URL parsing, thread data parsing, comment filtering
- test_reddit_sc.py: ScrapeCreators Reddit search (query expansion, subreddit discovery)
- test_instagram_sc.py: Instagram relevance scoring, tokenization, depth config

Includes fixtures/reddit_thread_sample.json for reddit_enrich tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:36:52 -04:00
Matt Van Horn 627947fc2c feat(plugin): publish as Claude Code marketplace plugin
Update .claude-plugin/marketplace.json and plugin.json to v2.9.5 with
full metadata. Add plugin install instructions to README as the
recommended install method. The repo root serves as both the marketplace
and the plugin - skills/last30days/SKILL.md (symlink) is discovered
automatically.

Users can now install with:
  /plugin marketplace add mvanhorn/last30days-skill
  /plugin install last30days@last30days-skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 13:09:33 -07:00
Matt Van Horn 2f16ff1ee8 feat(gemini): add Gemini CLI extension support
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.

Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.

Closes #45

Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:07:55 -07:00
Matt Van Horn 8f7fb5a7fe fix(release): v2.9.5 - remove re-introduced Save Research section
PR merges on March 7 (PR #48 Xiaohongshu, upstream merge) regressed
SKILL.md by re-introducing the "Save Research to Documents" section
that v2.9.4 removed. Those branches were forked before v2.9.4 and
brought the old content back via merge resolution.

Fixes: remove save section, restore --save-dir flag on bash command,
update agent mode line, add tool-call guard to STOP instruction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 11:55:35 -07:00
Matt Van Horn 4503da7920 docs: add perpetual monitoring mode plan (not building yet)
Explored adding scheduled re-runs and cumulative intelligence to
last30days. Concluded that Claude Code's session-scoped scheduling
(CronCreate/loop) can't support true perpetual monitoring since
jobs die when the terminal closes. Plan documents the architecture,
what exists, and why we're waiting for persistent background agents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 09:37:18 -07:00
Matt Van Horn 28dff6e7b2 Merge remote-tracking branch 'upstream/main' 2026-03-07 16:36:15 -08:00
Matt Van Horn 32992834ee Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:11:35 -08:00
Matt Van Horn 7dd8379c61 Merge origin/main into feat/xiaohongshu-reddit-public-fallback
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:08:20 -08:00
Matt Van Horn 408ac148ec Merge pull request #50 from mark-c4r/add-entity-extract-tests
test: add tests for entity_extract module
2026-03-07 15:59:32 -08:00
Matt Van Horn c9559252cd Merge pull request #52 from 04cb/fix/missing-metadata-files
Fix missing metadata files in skill upload bundle
2026-03-07 15:59:11 -08:00
04cb f70370a6f4 Fix missing metadata files in skill upload bundle 2026-03-07 18:10:44 +08:00
Matt Van Horn fad26d41fd fix: improve ClawHub security scan result
- Remove prompt-injection false positive ("you are now" → "treat yourself as")
- Declare AUTH_TOKEN and CT0 in frontmatter optionalEnv
- Clarify X token access language (no browser session access)
- Add permissions overview block near top of file

Zero functionality changes — metadata and prose only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:35:38 -08:00
Matt Van Horn ef1f380cda feat: publish to ClawHub as last30days-official
- Add ClawHub badge and install command to README
- Update SKILL.md: metadata.openclaw canonical key, license/author/repository fields, optionalEnv vars, added instagram/polymarket tags
- Add .clawhubignore to exclude binary assets and dev files from bundle

Published: https://clawhub.ai/skills/last30days-official

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:08:14 -08:00
Matt Van Horn e690d61a12 fix: append -raw suffix to saved research filenames
e.g. sam-altman-raw.md instead of sam-altman.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:56:03 -08:00
Matt Van Horn 6d5acb9121 feat(release): v2.9.4 - move save into Python script, zero post-invitation noise
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.

Tested: --mock confirms file creation and duplicate date suffixing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:27:53 -08:00
Matt Van Horn 28d223e43c fix(release): v2.9.3 - foreground save, fix hallucinated user messages
CRITICAL: run_in_background callbacks caused model to re-engage after
save, hallucinate fake "Human:" messages, and generate unsolicited
multi-paragraph responses. Switch to foreground cat > heredoc which
executes sub-second with no callback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:50:17 -08:00
Matt Van Horn 471badd329 fix(release): v2.9.2 - silent save, no follow-up text after background save
- Background Bash heredoc instead of Write tool
- Suppress response text on save completion
- 📎 footer line replaces verbose confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:41:55 -08:00
Matt Van Horn 5b1636f94f fix: merge upstream save fix (background Bash instead of Write tool)
Resolves merge conflict, keeping upstream's approach:
- Background heredoc save instead of Write tool
- Adds 📎 footer line
- No more "Wrote N lines..." cluttering output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 14:59:06 -08:00
Matt Van Horn 18f6273f7c fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:22 -08:00
Matt Van Horn 9950d01ab4 fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:11 -08:00
Matt Van Horn cc774d5e69 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Sync from public repo. Every run now saves the complete briefing as a
topic-named .md file to ~/Documents/Last30Days/. Credit @devin_explores.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:50:43 -08:00
Matt Van Horn 8cbbe87c3e docs: add v2.9.1 auto-save note to README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:27:18 -08:00
Matt Van Horn 49d993b162 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Bump version to 2.9.1, update changelog and release notes.
Credit @devin_explores for inspiring the feature.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:15:42 -08:00
Matt Van Horn 6afc094bb1 Merge pull request #51 from mvanhorn/feat/auto-save-documents
feat(skill): auto-save research results to ~/Documents/Last30Days/
2026-03-05 19:12:56 -08:00
Matt Van Horn f6a1769e35 feat(skill): auto-save research results to ~/Documents/Last30Days/
Every run now automatically saves the complete briefing (synthesis,
stats, follow-up suggestions) as a topic-named .md file in the user's
Documents folder. Agent mode also saves. No Python script changes -
this is purely a SKILL.md instruction addition.

Inspired by @devin_explores manually saving results to build a
personal research library.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:06:27 -08:00
mark-c4r d4328b5598 test: add tests for entity_extract module
Covers _extract_x_handles (8 cases), _extract_x_hashtags (5 cases),
_extract_subreddits (6 cases), and extract_entities integration (4 cases).
Follows existing test patterns from test_dedupe.py.
2026-03-05 16:12:33 -06:00
YJLi-new 788514ce8e feat: add Xiaohongshu source and Reddit public fallback
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
2026-03-05 20:54:33 +08:00
345 changed files with 32431 additions and 1216 deletions
+12 -7
View File
@@ -1,18 +1,23 @@
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "last30days",
"name": "last30days-skill",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"owner": {
"name": "mvanhorn",
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"metadata": {
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0"
},
"plugins": [
{
"name": "last30days",
"source": "./"
"description": "Research any topic from the last 30 days. Become an expert and write copy-paste-ready prompts.",
"version": "2.9.6",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"source": "./",
"category": "productivity",
"homepage": "https://github.com/mvanhorn/last30days-skill"
}
]
}
+9 -5
View File
@@ -1,12 +1,16 @@
{
"name": "last30days",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0",
"version": "2.9.6",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"author": {
"name": "mvanhorn"
"name": "Matt Van Horn",
"email": "mvanhorn@gmail.com",
"url": "https://github.com/mvanhorn"
},
"homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
"skills": ["./"]
"keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket"],
"skills": ["./"],
"hooks": {}
}
+18
View File
@@ -0,0 +1,18 @@
# Exclude binary assets and dev/test artifacts from ClawHub bundle
assets/
docs/
fixtures/
tests/
plans/
agents/
variants/
release-notes.md
SPEC.md
TASKS.md
SKILL-original.md
*.jsonl
*.mp3
*.jpeg
*.jpg
*.png
*.gif
+1
View File
@@ -14,3 +14,4 @@ variants/open/references/research.md
.entire/
__pycache__/
*.pyc
mise.toml
+42
View File
@@ -5,6 +5,47 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [2.9.4] - 2026-03-06
### Changed
- Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
- Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
- No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research
## [2.9.3] - 2026-03-06
### Fixed
- **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
- Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)
## [2.9.2] - 2026-03-06
### Fixed
- Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
- Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
- Add `📎` footer line for save path instead of verbose confirmation
## [2.9.1] - 2026-03-05
### Highlights
Auto-save research briefings to `~/Documents/Last30Days/` as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
### Added
- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to `~/Documents/Last30Days/{topic-slug}.md` after every run
- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
- Agent mode (`--agent`) also saves research files
- Brief confirmation after save: "Saved to ~/Documents/Last30Days/{slug}.md"
### Credits
- [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))
## [2.9.0] - 2026-03-05
### Highlights
@@ -114,6 +155,7 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
[2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
+21
View File
@@ -0,0 +1,21 @@
# last30days Skill
Claude Code skill for researching any topic across Reddit, X, YouTube, and web.
Python scripts with multi-source search aggregation.
## Structure
- `scripts/last30days.py` — main research engine
- `scripts/lib/` — search, enrichment, rendering modules
- `scripts/lib/vendor/bird-search/` — vendored X search client
- `SKILL.md` — skill definition (deployed to ~/.claude/skills/last30days/)
## Commands
```bash
python3 scripts/last30days.py "test query" --emit=compact # Run research
bash scripts/sync.sh # Deploy to ~/.claude, ~/.agents, ~/.codex
```
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- After edits: run `bash scripts/sync.sh` to deploy
- Git remotes: origin=private, upstream=public
+134 -53
View File
@@ -1,6 +1,28 @@
# /last30days v2.9
# /last30days v2.9.5
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
### Claude Code (recommended)
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
[![ClawHub](https://img.shields.io/badge/ClawHub-last30days--official-blue)](https://clawhub.ai/skills/last30days-official)
```bash
clawhub install last30days-official
```
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, and other sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
**New in v2.9.5 — Bluesky, Comparative Mode, and Config Improvements:**
- **Bluesky/AT Protocol** is now a social source. Opt-in via `BSKY_HANDLE` + `BSKY_APP_PASSWORD` (create at bsky.app/settings/app-passwords). Full pipeline: search, score, dedupe, render.
- **Comparative mode** - ask "X vs Y" (e.g., `/last30 Claude Code vs Codex`) and get 3 parallel research passes with a side-by-side comparison: strengths, weaknesses, head-to-head table, and a data-driven verdict.
- **Per-project .env config** - drop a `.claude/last30days.env` in your project root for per-project API keys.
- **SessionStart config check** - validates your config automatically when a Claude Code session starts.
- **Expanded test coverage** - 455+ tests across all modules.
**New in v2.9.1 — Auto-save to ~/Documents/Last30Days/:** Every run now saves the complete briefing as a topic-named `.md` file to your Documents folder. Build a personal research library automatically. Inspired by [@devin_explores](https://x.com/devin_explores).
**New in v2.9 — ScrapeCreators Reddit + Top Comments + Smart Discovery:**
@@ -20,7 +42,7 @@ Instagram Reels is now the 8th signal source. TikTok and Instagram both run on S
**New in V2:** Smarter query construction, two-phase supplemental search, free X search via bundled Bird client, `--days=N` flag, automatic model fallback. [Full changelog below.](#whats-new-in-v2)
**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Six sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness.
**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Up to 10 sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness.
**Best for prompt research**: discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Paper, etc.) by learning from real community discussions and best practices.
@@ -28,44 +50,114 @@ Instagram Reels is now the 8th signal source. TikTok and Instagram both run on S
## Installation
### Claude Code Plugin (recommended)
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
### Gemini CLI
```bash
gemini extensions install https://github.com/mvanhorn/last30days-skill.git
```
### Manual Install (Claude Code / Codex)
```bash
# Clone the repo
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
```
# Add your API keys (optional if signed in to Codex)
That's it. Reddit, Hacker News, and Polymarket work immediately with zero configuration. Run `/last30days` to unlock more sources.
---
## Setup: Progressive Source Unlocking
Start using /last30days immediately. Add sources when you want better results.
### 1. Zero Config (3 sources) — Just install
Reddit (public JSON), Hacker News, and Polymarket work out of the box. No API keys, no configuration.
### 2. Run the setup wizard (5+ sources)
```
/last30days setup
```
The setup wizard automatically extracts X/Twitter login cookies from your browsers (Chrome, Firefox, Safari) and checks for yt-dlp. Takes about 30 seconds. Your cookies stay in memory and are never saved to disk.
### 3. Add Exa (FREE — semantic web search)
Register at [exa.ai](https://exa.ai) for 1,000 free searches/month, no credit card required.
```bash
# Add to ~/.config/last30days/.env
EXA_API_KEY=...
```
### 4. Add ScrapeCreators (RECOMMENDED — Reddit comments + TikTok + Instagram)
**This is the single most impactful upgrade.** Reddit comments are often the highest-value research content — top-voted replies with real insights. ScrapeCreators unlocks comment enrichment plus TikTok and Instagram. Register at [scrapecreators.com](https://scrapecreators.com) for 100 free API calls (no credit card required). After that, pay-as-you-go. last30days receives no money from any API provider — no referrals, no kickbacks.
```bash
# Add to ~/.config/last30days/.env
SCRAPECREATORS_API_KEY=...
```
### 5. Add Bluesky (FREE — app password)
Create an app password at [bsky.app/settings/app-passwords](https://bsky.app/settings/app-passwords).
```bash
# Add to ~/.config/last30days/.env
BSKY_HANDLE=you.bsky.social
BSKY_APP_PASSWORD=xxxx-xxxx-xxxx
```
### 6. Optional paid web search backends
```bash
# Add to ~/.config/last30days/.env
PARALLEL_API_KEY=... # Parallel AI (preferred — LLM-optimized results)
BRAVE_API_KEY=... # Brave Search (free tier: 2,000 queries/month)
OPENROUTER_API_KEY=... # OpenRouter/Perplexity Sonar Pro
```
---
### Do I need API keys?
| Source | Free Method | API Key | Do you need the API key? |
|--------|------------|---------|--------------------------|
| Reddit | Public JSON (always works) | ScrapeCreators | **Yes, strongly recommended.** Unlocks top comments — often the most valuable content. |
| X/Twitter | Browser cookies (auto-extracted) | xAI API key (`XAI_API_KEY`) | **No.** Cookies give identical quality. The setup wizard handles this. |
| YouTube | yt-dlp (`brew install yt-dlp`) | N/A | **No API key exists.** Install yt-dlp for search; transcripts work without it. |
| Hacker News | Always free | N/A | **No.** Always works, no config needed. |
| Polymarket | Always free | N/A | **No.** Always works, no config needed. |
| Web search | N/A | Exa (`EXA_API_KEY`) | **Optional.** 1,000 free searches/month at exa.ai. |
| Bluesky | Free app password | N/A | **Optional.** Free app password at bsky.app. |
| TikTok | N/A | ScrapeCreators | **Optional.** Included with ScrapeCreators key. |
| Instagram | N/A | ScrapeCreators | **Optional.** Included with ScrapeCreators key. |
| Truth Social | Browser cookies | N/A | **Optional.** Auto-extracted if logged in. |
*last30days receives no money from any API provider — no referrals, no kickbacks.*
---
### Config file locations
For project-specific overrides, create `.claude/last30days.env` in the repo root. It overrides the global `~/.config/last30days/.env`.
```bash
# Global config
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'EOF'
SCRAPECREATORS_API_KEY=... # Reddit + TikTok + Instagram (one key, all three) — scrapecreators.com
OPENAI_API_KEY=sk-... # optional — legacy Reddit fallback if using `codex login`
XAI_API_KEY=xai-... # optional — cookie auth is default for X search
EOF
chmod 600 ~/.config/last30days/.env
# Project-specific config (optional)
# .claude/last30days.env
```
If you're signed in to Codex (`codex login`), the skill will use your Codex credentials for the OpenAI Responses API and you can omit `OPENAI_API_KEY`. If you're not signed in, run `codex login` first.
### X Search Authentication
X search reads your existing browser cookies - no API keys or login commands needed.
**Safari (recommended on Mac):** Just be logged into x.com. No setup needed.
**Chrome:** Works, but macOS will prompt you to allow Keychain access the first time. Click "Allow" (or "Always Allow" to stop future prompts).
**Firefox:** Just be logged into x.com. No setup needed.
**Manual fallback:** If cookie auto-detection doesn't work, set these env vars (grab them from your browser's dev tools → Application → Cookies → x.com):
```bash
export AUTH_TOKEN=your_auth_token
export CT0=your_ct0_token
```
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
```
**Requirements:** Node.js 22+ (for the vendored Twitter GraphQL client).
Check source availability: `python3 scripts/last30days.py --diagnose`
### Codex CLI
@@ -109,15 +201,6 @@ The open variant adds four modes on top of one-shot research:
Both variants use the same Python engine and scripts directory. The open variant adds command routing (`watch`, `briefing`, `history`) and references mode-specific instruction files.
**Optional web search API keys** (add to `~/.config/last30days/.env`):
```bash
PARALLEL_API_KEY=... # Parallel AI (preferred - LLM-optimized results)
BRAVE_API_KEY=... # Brave Search (free tier: 2,000 queries/month)
OPENROUTER_API_KEY=... # OpenRouter/Perplexity Sonar Pro
```
Check source availability: `python3 scripts/last30days.py --diagnose`
## Usage
```
@@ -133,7 +216,7 @@ Examples:
## What It Does
1. **Researches** - Scans Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web for discussions from the last 30 days
1. **Researches** - Scans Reddit, X, YouTube, and other sources for discussions from the last 30 days
2. **Synthesizes** - Identifies patterns, best practices, and what actually works
3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
@@ -887,13 +970,11 @@ This example shows /last30days discovering **emerging developer workflows** - re
## Requirements
- **OpenAI API key** - For Reddit research (uses web search via Responses API)
- **Python 3** - For the research engine
- **Node.js 22+** - For X search (bundled Twitter GraphQL client)
- **X session** - Be logged into x.com in your browser, or set `AUTH_TOKEN`/`CT0` env vars
- **xAI API key** (optional fallback) - If the bundled search can't authenticate, falls back to xAI's Grok API
- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as an additional source.
- **yt-dlp** (recommended) - For YouTube search. Install via `brew install yt-dlp` or `pip install yt-dlp`. Transcripts work without it.
At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH.
No API keys are required to start. Reddit, Hacker News, and Polymarket work out of the box. Run `/last30days setup` to unlock X/Twitter via browser cookies and configure additional sources. See [Setup: Progressive Source Unlocking](#setup-progressive-source-unlocking) for the full progression.
## Troubleshooting
@@ -945,7 +1026,7 @@ If your OpenAI org doesn't have access to a model (e.g., unverified for gpt-4.1)
### ScrapeCreators Reddit as default
Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search.
Reddit now runs on [ScrapeCreators](https://scrapecreators.com) by default. One `SCRAPECREATORS_API_KEY` powers Reddit, TikTok, and Instagram — three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. 100 free API calls, no credit card required — just register at [scrapecreators.com](https://scrapecreators.com), then pay-as-you-go. last30days receives no money from any API provider — no referrals, no kickbacks.
```bash
echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
@@ -998,7 +1079,7 @@ Search "AI tools" and you get:
### TikTok + Instagram on ScrapeCreators
Both TikTok and Instagram are powered by [ScrapeCreators](https://scrapecreators.com) — one API key covers both sources. 100 free credits, then pay-as-you-go.
Both TikTok and Instagram are powered by [ScrapeCreators](https://scrapecreators.com) — one API key covers both sources. Register at [scrapecreators.com](https://scrapecreators.com) for 100 free API calls (no credit card required). After that, pay-as-you-go. last30days receives no money from any API provider — no referrals, no kickbacks.
```bash
echo 'SCRAPECREATORS_API_KEY=your_key_here' >> ~/.config/last30days/.env
@@ -1100,7 +1181,7 @@ Inspired by [Peter Steinberger](https://x.com/steipete)'s yt-dlp + [summarize](h
### Bundled X search (v2.1)
**X search is fully self-contained** - No external `bird` CLI or xAI API key needed. /last30days bundles a vendored subset of Bird's Twitter GraphQL client (MIT licensed, by Peter Steinberger). Just be logged into x.com in your browser and it auto-detects your session. Falls back to xAI API if bundled search can't authenticate.
**X search is fully self-contained** - No external `bird` CLI install needed. /last30days bundles a vendored subset of Bird's Twitter GraphQL client (MIT licensed, by Peter Steinberger). With Node.js 22+ plus `AUTH_TOKEN` and `CT0`, it runs locally without browser-cookie prompts. Falls back to xAI API if bundled auth is not configured.
### Everything else (v2.1)
@@ -1147,7 +1228,7 @@ Thanks to the contributors who helped shape V2:
| `api.scrapecreators.com` | Search query (Reddit + TikTok + Instagram) | SCRAPECREATORS_API_KEY |
| `api.openai.com` | Search query (legacy Reddit fallback) | OPENAI_API_KEY |
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
| Twitter GraphQL / `api.x.ai` | Search query | AUTH_TOKEN/CT0 or XAI_API_KEY |
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
| `hn.algolia.com` | Search query | None (public API) |
| `gamma-api.polymarket.com` | Search query | None (public API) |
+267 -19
View File
@@ -1,17 +1,32 @@
---
name: last30days
version: "2.9"
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
version: "2.9.6"
description: "Deep research engine covering the last 30 days across 10+ sources - Reddit, X/Twitter, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. AI synthesizes findings into grounded, cited reports."
argument-hint: 'last30 AI video tools, last30 best project management tools'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill
repository: https://github.com/mvanhorn/last30days-skill
author: mvanhorn
license: MIT
user-invocable: true
metadata:
clawdbot:
openclaw:
emoji: "📰"
requires:
env:
- SCRAPECREATORS_API_KEY
optionalEnv:
- OPENAI_API_KEY
- XAI_API_KEY
- OPENROUTER_API_KEY
- PARALLEL_API_KEY
- BRAVE_API_KEY
- APIFY_API_TOKEN
- AUTH_TOKEN
- CT0
- BSKY_HANDLE
- BSKY_APP_PASSWORD
- TRUTHSOCIAL_TOKEN
bins:
- node
- python3
@@ -21,18 +36,154 @@ metadata:
homepage: https://github.com/mvanhorn/last30days-skill
tags:
- research
- deep-research
- reddit
- x
- twitter
- youtube
- tiktok
- instagram
- hackernews
- polymarket
- bluesky
- truthsocial
- trends
- prompts
- recency
- news
- citations
- multi-source
- social-media
- analysis
- web-search
- ai-skill
- clawhub
---
# last30days v2.9: Research Any Topic from the Last 30 Days
# last30days v2.9.5: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `~/Documents/Last30Days/`. X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now.
## Step 0: First-Run Setup Wizard
**CRITICAL: ALWAYS execute Step 0 BEFORE Step 1, even if the user provided a topic.** If the user typed `/last30days Mercer Island`, you MUST check for FIRST_RUN and present the wizard BEFORE running research. The topic "Mercer Island" is preserved — research runs immediately after the wizard completes. Do NOT skip the wizard because a topic was provided. The wizard takes 10 seconds and only runs once ever.
To detect first run: check if `~/.config/last30days/.env` exists. If it does NOT exist, this is a first run. **Do NOT run any Bash commands or show any command output to detect this — just check the file existence silently.** If the file exists and contains `SETUP_COMPLETE=true`, skip this section and proceed to Step 1.
**When first run is detected, you MUST follow these steps IN ORDER. Do NOT skip ahead to the topic picker or research. The sequence is: (1) welcome text → (2) setup modal → (3) run setup if chosen → (4) ScrapeCreators modal → (5) topic picker. You MUST start at step 1.**
**Step 1: Display the following welcome text ONCE as a normal message (not blockquoted). Then IMMEDIATELY call AskUserQuestion — do NOT repeat any of the welcome text inside the AskUserQuestion call.**
👋 Welcome to /last30days!
I research any topic across Reddit, X, YouTube, and other sources — synthesizing what people are actually saying right now.
To get the best results, I can:
🔍 **Scan your browser** for X/Twitter cookies (free X search — reads x.com only, never saved)
📺 **Install yt-dlp** for YouTube search + transcripts (free, open source, 190K+ GitHub stars)
**ScrapeCreators API key** unlocks Reddit comments + TikTok + Instagram (100 free to start — scrapecreators.com)
We recommend all 3 before your first run — it's what makes the magic. More community sources available later. We get no kickbacks from any of these.
**Then call AskUserQuestion with ONLY this question and these options — no additional text:**
Question: "How would you like to set up?"
Options:
- "Auto setup (~30 seconds) — scans browser for X cookies (free X search) and installs yt-dlp (free YouTube transcripts)"
- "Manual setup — show me what to configure"
- "Skip for now — Reddit (threads only), HN, Polymarket, Web"
**If the user picks 1 (Auto setup):**
Run the setup subcommand:
```bash
cd {SKILL_DIR} && python3 scripts/last30days.py setup
```
Show the user the results (what cookies were found, whether yt-dlp was installed).
**Then show the ScrapeCreators push (plain text, then modal):**
⭐ One more thing — Reddit comments are some of the best content on the internet. The top-voted replies often have sharper insights than the posts themselves. ScrapeCreators unlocks these (plus TikTok + Instagram) — 100 free to start, no credit card.
**Call AskUserQuestion:**
Question: "Want to add Reddit comments to your research?"
Options:
- "Open scrapecreators.com to get my free key" — run `open https://scrapecreators.com` via Bash to open in the user's browser. Then ask them to paste the API key they get. When they paste it, write SCRAPECREATORS_API_KEY={key} to ~/.config/last30days/.env
- "I have a key — let me paste it" — accept the key, write to .env
- "Skip for now — start researching" — proceed without ScrapeCreators
**After ScrapeCreators modal (or skip), show the first research topic modal:**
**Call AskUserQuestion:**
Question: "What do you want to research first?"
Options:
- "Claude Code vs Codex" — tech comparison
- "Sam Altman" — person in the news
- "Warriors Basketball" — sports
- "AI Legal Prompting Techniques" — niche/professional
- "Type my own topic"
If user picks an example, run research with that topic. If they pick "Type my own", ask them what they want to research. If the user originally provided a topic with the command (e.g., `/last30days Mercer Island`), skip this modal and use their topic directly.
**END OF FIRST-RUN WIZARD. Everything above in Step 0 ONLY runs on first run. If SETUP_COMPLETE=true exists in .env, skip ALL of Step 0 — no welcome, no setup, no ScrapeCreators modal, no topic picker. Go directly to Step 1 (Parse User Intent). The topic picker is ONLY for first-time users who haven't run /last30days before.**
**If the user picks 2 (Manual setup):**
Show them this guide (present as plain text, not blockquoted):
**The magic of /last30days is Reddit comments + X posts together.** Here's how to unlock each source.
Add these to `~/.config/last30days/.env`:
**🔍 X/Twitter** (pick one — this is the most important):
- `FROM_BROWSER=auto` — easiest, free. Scans your browser cookies for x.com login.
- `AUTH_TOKEN=xxx` + `CT0=xxx` — paste your X cookies manually (x.com → F12 → Application → Cookies)
- `XAI_API_KEY=xxx` — use an xAI/Grok API key instead (api.x.ai)
**⭐ Reddit Comments + TikTok + Instagram + YouTube backup** (one key, 5 platforms):
- `SCRAPECREATORS_API_KEY=xxx` — 100 free to start at scrapecreators.com. This is the single most impactful key — Reddit comments are where the sharpest insights live.
**💡 Reddit discovery fallback** (optional):
- `OPENAI_API_KEY=xxx` — ChatGPT web search for Reddit threads. Only used if ScrapeCreators isn't configured — SC is better.
**📺 YouTube** (primary engine):
- Run `brew install yt-dlp` — free, open source. ScrapeCreators covers YouTube as backup if you skip this.
**Bonus sources** (add anytime):
- `EXA_API_KEY=xxx` — semantic web search, 1K free/month (exa.ai)
- `BSKY_HANDLE=you.bsky.social` + `BSKY_APP_PASSWORD=xxx` — Bluesky (free app password)
- `BRAVE_API_KEY=xxx` — Brave web search
- `OPENROUTER_API_KEY=xxx` — Perplexity Sonar Pro search
Always add this last line: `SETUP_COMPLETE=true`
You can paste your API keys right here and I'll write the file for you. Some people say giving LLMs your keys is risky, others live in YOLO mode. Your call.
**If the user pastes keys**, write them to `~/.config/last30days/.env` (create the file and parent dirs if needed, append without overwriting existing keys, always include `SETUP_COMPLETE=true`). Then proceed with research.
**If the user says they'll do it themselves**, tell them: "Save the file, then run `/last30days <topic>` to research anything." Then proceed with research using whatever sources are currently available.
**If the user picks Skip:**
Proceed with research immediately using the user's original topic. Note: without setup, sources are limited to Reddit (threads only, no comments), HN, and Polymarket. X/Twitter and YouTube require setup.
---
## Do I Need API Keys?
When users ask about API keys, setup, or how to unlock more sources, reference this:
**You do NOT need API keys to use last30days.** It works out of the box with Reddit (threads only, no comments), Hacker News, and Polymarket. Browser cookies for X/Twitter are equivalent to an API key — just log into x.com in any browser and last30days will find your session automatically.
**Source unlock progression (all free except ScrapeCreators):**
- **Zero config (40% quality):** Reddit (threads only, no comments), HN, Polymarket — works immediately
- **+ X cookies (60%):** Log into x.com in any browser. last30days scans your cookies automatically. This is the same as having an API key — no signup required.
- **+ yt-dlp (80%):** `brew install yt-dlp` — open source, 190K+ GitHub stars. Enables YouTube search and transcripts.
- **+ ScrapeCreators (100%):** The only paid option worth mentioning. Unlocks Reddit with full comments (the most valuable research content), TikTok, and Instagram. 100 free API calls, no credit card — scrapecreators.com
**Key comparison:** X browser cookies = same access as an API key (free, no signup). ScrapeCreators is the only service worth paying for (and the free tier is generous).
last30days has no affiliation with any API provider — no referrals, no kickbacks.
---
## CRITICAL: Parse User Intent
@@ -44,6 +195,7 @@ Before doing anything, parse the user's input for:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **COMPARISON** - "X vs Y", "X versus Y", "compare X and Y", "X or Y which is better" → User wants a side-by-side comparison
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
@@ -52,6 +204,7 @@ Common patterns:
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "X vs Y" or "X versus Y" → QUERY_TYPE = COMPARISON, TOPIC_A = X, TOPIC_B = Y (split on ` vs ` or ` versus ` with spaces)
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
@@ -60,12 +213,14 @@ Common patterns:
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | COMPARISON | GENERAL]`
- `TOPIC_A = [first item]` (only if COMPARISON)
- `TOPIC_B = [second item]` (only if COMPARISON)
**DISPLAY your parsing to the user.** Before running any tools, output:
```
I'll research {TOPIC} across Reddit, X, TikTok, and the web to find what's been discussed in the last 30 days.
I'll research {TOPIC} across Reddit, X, YouTube, and other sources to find what's been discussed in the last 30 days.
Parsed intent:
- TOPIC = {TOPIC}
@@ -123,11 +278,13 @@ If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):
5. **Skip** the follow-up invitation ("I'm now an expert on X...")
6. **Output** the complete research report and stop - do not wait for further input
Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir` (handled by the script, no extra tool calls).
Agent mode report format:
```
## Research Report: {TOPIC}
Generated: {date} | Sources: Reddit, X, YouTube, TikTok, HN, Polymarket, Web
Generated: {date} | Sources: Reddit, X, Bluesky, YouTube, TikTok, HN, Polymarket, Web
### Key Findings
[3-5 bullet points, highest-signal insights with citations]
@@ -141,6 +298,28 @@ Generated: {date} | Sources: Reddit, X, YouTube, TikTok, HN, Polymarket, Web
---
## If QUERY_TYPE = COMPARISON
When the user asks "X vs Y", run THREE research passes in parallel:
**Pass 1 + 2 (parallel Bash calls):**
```bash
# Run BOTH of these as parallel Bash tool calls in a single message:
python3 "${SKILL_ROOT}/scripts/last30days.py" {TOPIC_A} --emit=compact --no-native-web --save-dir=~/Documents/Last30Days
python3 "${SKILL_ROOT}/scripts/last30days.py" {TOPIC_B} --emit=compact --no-native-web --save-dir=~/Documents/Last30Days
```
**Pass 3 (after passes 1+2 complete):**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B}" --emit=compact --no-native-web --save-dir=~/Documents/Last30Days
```
Then do WebSearch for: `{TOPIC_A} vs {TOPIC_B} comparison 2026` and `{TOPIC_A} vs {TOPIC_B} which is better`.
**Skip the normal Step 1 below** - go directly to the comparison synthesis format (see "If QUERY_TYPE = COMPARISON" in the synthesis section).
---
## Research Execution
**Step 1: Run the research script (FOREGROUND — do NOT background this)**
@@ -154,6 +333,10 @@ Generated: {date} | Sources: Reddit, X, YouTube, TikTok, HN, Polymarket, Web
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"${GEMINI_EXTENSION_DIR:-}" \
"$HOME/.claude/plugins/marketplaces/last30days-skill" \
"$HOME/.gemini/extensions/last30days-skill" \
"$HOME/.gemini/extensions/last30days" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
@@ -165,7 +348,7 @@ if [ -z "${SKILL_ROOT:-}" ]; then
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --no-native-web --save-dir=~/Documents/Last30Days # Add --x-handle=HANDLE if RESOLVED_HANDLE is set
```
Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically takes 1-3 minutes.
@@ -177,7 +360,7 @@ The script will automatically:
**Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, **transcript highlights** (pre-extracted quotable excerpts from the video), and an optional full transcript in a collapsible section. **Quote the highlights directly in your synthesis** - they are the YouTube equivalent of Reddit top comments. Attribute quotes to the channel name. Count them and include them in your synthesis and stats block.
**TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block.
@@ -240,9 +423,10 @@ The Judge Agent must:
3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
4. Weight WebSearch sources LOWER (no engagement data)
5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. When a top comment has high upvotes (shown as `💬 Top comment (N upvotes)`), quote it directly in your synthesis. Reddit's value is in the comments.
6. Identify patterns that appear across ALL sources (strongest signals)
7. Note any contradictions between sources
8. Extract the top 3-5 actionable insights
6. **For YouTube: Quote transcript highlights directly in your synthesis.** These are pre-extracted key moments from the video - treat them like Reddit top comments. Attribute to the channel name and include the actual quote. YouTube's value is in what creators SAY, not just their view counts.
7. Identify patterns that appear across ALL sources (strongest signals)
8. Note any contradictions between sources
9. Extract the top 3-5 actionable insights
7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.
@@ -299,6 +483,53 @@ When user asks "best X" or "top X", they want a LIST of specific things:
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### If QUERY_TYPE = COMPARISON
Structure the output as a side-by-side comparison using data from all three research passes:
```
# {TOPIC_A} vs {TOPIC_B}: What the Community Says (Last 30 Days)
## Quick Verdict
[1-2 sentence data-driven summary: which one the community prefers and why, with source counts]
## {TOPIC_A}
**Community Sentiment:** [Positive/Mixed/Negative] ({N} mentions across {sources})
**Strengths (what people love)**
- [Point 1 with source attribution]
- [Point 2]
**Weaknesses (common complaints)**
- [Point 1 with source attribution]
- [Point 2]
## {TOPIC_B}
**Community Sentiment:** [Positive/Mixed/Negative] ({N} mentions across {sources})
**Strengths (what people love)**
- [Point 1 with source attribution]
- [Point 2]
**Weaknesses (common complaints)**
- [Point 1 with source attribution]
- [Point 2]
## Head-to-Head
[Synthesis from the "A vs B" combined search - what people say when directly comparing]
| Dimension | {TOPIC_A} | {TOPIC_B} |
|-----------|-----------|-----------|
| [Key dimension 1] | [A's position] | [B's position] |
| [Key dimension 2] | [A's position] | [B's position] |
| [Key dimension 3] | [A's position] | [B's position] |
## The Bottom Line
Choose {TOPIC_A} if... Choose {TOPIC_B} if... (based on actual community data, not assumptions)
```
Then show combined stats from all three passes and the standard invitation section.
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
@@ -388,6 +619,10 @@ KEY PATTERNS from the research:
3. [Pattern] — per @handle
```
**THEN - Quality Nudge (if present in the output):**
If the research output contains a `**🔍 Research Coverage:**` block, render it verbatim right before the stats block. This tells the user which core sources are missing and how to unlock them. Do NOT render this block if it is absent from the output (100% coverage = no nudge).
**THEN - Stats (right before invitation):**
**CRITICAL: Calculate actual totals from the research output.**
@@ -406,6 +641,8 @@ KEY PATTERNS from the research:
├─ 🎵 TikTok: {N} videos │ {N} views │ {N} likes │ {N} with captions
├─ 📸 Instagram: {N} reels │ {N} views │ {N} likes │ {N} with captions
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
├─ 🦋 Bluesky: {N} posts │ {N} likes │ {N} reposts
├─ 🇺🇸 Truth Social: {N} posts │ {N} likes │ {N} reposts
├─ 📊 Polymarket: {N} markets │ {short summary of up to 5 most relevant market odds, e.g. "Championship: 12%, #1 Seed: 28%, Big 12: 64%, vs Kansas: 71%"}
├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
@@ -463,6 +700,16 @@ I'm now an expert on {TOPIC}. Some things you could ask:
- [Question about what might happen next based on current trajectory]
```
**If QUERY_TYPE = COMPARISON:**
```
---
I've compared {TOPIC_A} vs {TOPIC_B} using the latest community data. Some things you could ask:
- [Deep dive into {TOPIC_A} alone with /last30days {TOPIC_A}]
- [Deep dive into {TOPIC_B} alone with /last30days {TOPIC_B}]
- [Focus on a specific dimension from the comparison table]
- [Look at a different time period with --days=7 or --days=90]
```
**If QUERY_TYPE = GENERAL:**
```
---
@@ -498,7 +745,7 @@ For `/last30days war in Iran` (NEWS):
## WAIT FOR USER'S RESPONSE
After showing the stats summary with your invitation, **STOP and wait** for the user to respond.
**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`.
---
@@ -568,7 +815,7 @@ For the rest of this conversation, remember:
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
**CRITICAL: After research is complete, treat yourself as an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
@@ -599,14 +846,15 @@ Want another prompt? Just tell me what you're creating next.
**What this skill does:**
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for Reddit search, subreddit discovery, and comment enrichment (requires SCRAPECREATORS_API_KEY — same key as TikTok + Instagram)
- Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Sends search queries to Twitter's GraphQL API (via optional user-provided AUTH_TOKEN/CT0 env vars — no browser session access) or xAI's API (`api.x.ai`) for X search
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (same SCRAPECREATORS_API_KEY as Reddit, PAYG after 100 free credits)
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (same SCRAPECREATORS_API_KEY as Reddit, PAYG after 100 free API calls)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
- Saves research briefings as .md files to ~/Documents/Last30Days/
**What this skill does NOT do:**
- Does not post, like, or modify content on any platform
@@ -615,7 +863,7 @@ Want another prompt? Just tell me what you're creating next.
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Hacker News and Polymarket sources are always available (no API key, no binary dependency)
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (same key covers both; 100 free credits, then PAYG)
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (same key covers both; 100 free API calls, then PAYG)
- Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
+195
View File
@@ -0,0 +1,195 @@
# How Reddit & X Search Work in last30days
## Architecture Overview
```
User: /last30days "kanye west"
┌─────┴─────┐
↓ ↓ (concurrent via ThreadPoolExecutor)
[REDDIT] [X/TWITTER]
↓ ↓
OpenAI Bundled Bird or
API xAI API
↓ ↓
Parse Parse
↓ ↓
Enrich ───┘
(fetch ↓
actual [MERGE]
upvotes) ↓
↓ [NORMALIZE → FILTER → SCORE → DEDUPE]
└───────────↓
[OUTPUT to SKILL.md agent]
```
Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`.
---
## Reddit Search
### How it works
Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only.
**API Call:**
```
POST https://api.openai.com/v1/responses
Authorization: Bearer {OPENAI_API_KEY}
```
**Payload:**
```json
{
"model": "gpt-5.2",
"tools": [{
"type": "web_search",
"filters": { "allowed_domains": ["reddit.com"] }
}],
"input": "Search Reddit for threads about {topic}..."
}
```
The prompt asks the model to:
1. Extract core subject (strip noise words like "best", "tips", "top")
2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"`
3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance`
4. URLs must contain `/r/` AND `/comments/` (real threads only)
**Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini`
Triggers on HTTP 400/403 with access error keywords.
### Enrichment (the secret sauce)
After search, each thread gets **enriched** by hitting Reddit's free JSON API:
```
GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json
```
No API key needed. This returns the actual thread data:
| Data Point | Source |
|---|---|
| Upvotes (score) | Reddit JSON API |
| Comment count | Reddit JSON API |
| Upvote ratio | Reddit JSON API |
| Top 10 comments (text + score) | Reddit JSON API |
| 7 key comment insights | Extracted via heuristics |
| Actual post date | `created_utc` timestamp |
**This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates.
### Depth settings
| Depth | Threads requested | Timeout |
|---|---|---|
| `--quick` | 15-25 | 90s |
| default | 30-50 | 120s |
| `--deep` | 70-100 | 180s |
---
## X/Twitter Search
X search has **two backends** — the skill auto-detects which to use.
### Priority: Bundled Bird (env auth) → xAI API (paid)
```python
if node_available and AUTH_TOKEN and CT0:
use bundled Bird # Free, popup-free, env-authenticated
elif XAI_API_KEY:
use xAI API # Paid, uses grok-4-1-fast
else:
skip X entirely # No X results
```
### Backend 1: xAI API
**API Call:**
```
POST https://api.x.ai/v1/responses
Authorization: Bearer {XAI_API_KEY}
```
**Payload:**
```json
{
"model": "grok-4-1-fast",
"tools": [{ "type": "x_search" }],
"input": "Search X for posts about {topic} from {from_date} to {to_date}..."
}
```
The prompt asks grok to return JSON with:
- `text`, `url`, `author_handle`, `date`
- `engagement`: `{ likes, reposts, replies, quotes }`
- `why_relevant`, `relevance` score
**Engagement data comes from grok's x_search tool** - it has direct access to X's data.
### Backend 2: Bundled Bird client (free alternative)
The repo vendors a search-only subset of Bird's Twitter GraphQL client and shells out to it with Node.js. No global `bird` install is required. The Python wrapper passes `AUTH_TOKEN` and `CT0` via env, which keeps normal local runs headless and avoids browser-cookie prompts.
**Bundled Bird returns raw X API data** - likes, reposts, replies are real engagement metrics from X's API, not estimates.
| Metric | Bundled Bird | xAI API |
|---|---|---|
| Post text | Real | Real |
| Likes/reposts | Real (X API) | Real (x_search tool) |
| Replies/quotes | Real | Real |
| Author handle | Real | Real |
| Relevance score | Default 0.7 (re-ranked by score.py) | AI-assessed 0.0-1.0 |
### Depth settings
| Depth | xAI posts | Bundled Bird results | xAI timeout | Bird timeout |
|---|---|---|---|---|
| `--quick` | 8-12 | 12 | 90s | 30s |
| default | 20-30 | 30 | 120s | 45s |
| `--deep` | 40-60 | 60 | 180s | 60s |
---
## Post-Processing (both sources)
After both searches complete:
1. **Normalize** — consistent formatting, timezone handling
2. **Date filter** — hard filter to requested date range
3. **Score** — relevance scoring (engagement-weighted)
4. **Sort** — highest scores first
5. **Deduplicate** — remove duplicate URLs
6. **Fallback** — if all items filtered out, keep top 3 by relevance
---
## Error Handling
| Layer | Strategy |
|---|---|
| HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) |
| Model access errors | Automatic fallback to next model in chain |
| Reddit enrichment | Per-item try/catch; keeps unenriched item on failure |
| X source detection | Silent fallback from Bird → xAI → skip |
| Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user |
---
## Key Files
| File | Purpose |
|---|---|
| `scripts/last30days.py` | Main orchestrator, concurrent execution |
| `scripts/lib/openai_reddit.py` | Reddit search via OpenAI Responses API |
| `scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `scripts/lib/xai_x.py` | X search via xAI API |
| `scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `scripts/lib/models.py` | Auto-select best available model |
| `scripts/lib/env.py` | API key loading, source detection |
| `scripts/lib/http.py` | HTTP transport with retries |
| `scripts/lib/score.py` | Relevance scoring |
| `scripts/lib/dedupe.py` | URL-based deduplication |
@@ -0,0 +1,281 @@
---
title: "feat: Automated v1 vs v2 test harness using claude --print"
type: feat
date: 2026-02-06
---
# feat: Automated V1 vs V2 Test Harness
## Overview
Build a bash script that swaps SKILL.md between v1 (upstream) and v2 (current), runs `claude --print "/last30days [query]"` for all 17 test queries on each version, captures output to files, then generates a comparison doc with analysis.
## How It Works
```
┌─────────────────────────────────────────────────┐
│ test-v1-vs-v2.sh │
│ │
│ 1. Save current SKILL.md as .v2 backup │
│ 2. Install v1 SKILL.md from upstream │
│ 3. Loop 17 queries → claude --print → v1/*.txt │
│ 4. Restore v2 SKILL.md │
│ 5. Loop 17 queries → claude --print → v2/*.txt │
│ 6. Generate comparison doc │
└─────────────────────────────────────────────────┘
```
Each `claude --print` call:
- Invokes the /last30days skill exactly as a user would
- Runs the Python script (real API calls to OpenAI + xAI)
- Runs WebSearch
- Applies SKILL.md presentation instructions
- Returns the full formatted output
- Exits (no interactive session)
## Implementation
### File: `scripts/test-v1-vs-v2.sh`
```bash
#!/bin/bash
set -euo pipefail
# === Config ===
SKILL_DIR="$HOME/.claude/skills/last30days"
REPO_DIR="/Users/mvanhorn/last30days-skill-private"
OUT_DIR="$REPO_DIR/docs/test-results/v1-vs-v2-$(date +%Y%m%d-%H%M%S)"
V1_DIR="$OUT_DIR/v1"
V2_DIR="$OUT_DIR/v2"
mkdir -p "$V1_DIR" "$V2_DIR"
# All 17 test queries (from README + plans)
declare -a QUERIES=(
"prompting techniques for chatgpt for legal questions"
"best clawdbot use cases"
"how to best setup clawdbot"
"prompting tips for nano banana pro for ios designs"
"top claude code skills"
"using ChatGPT to make images of dogs"
"research best practices for beautiful remotion animation videos in claude code"
"photorealistic people in nano banana pro"
"What are the best rap songs lately"
"what are people saying about DeepSeek R1"
"best practices for cursor rules files for Cursor"
"prompt advice for using suno to make killer songs in simple mode"
"how do I use Codex with Claude Code on same app to make it better"
"kanye west"
"howie.ai"
"open claw"
"nano banana pro prompting"
)
declare -a TYPES=(
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"HOW-TO"
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"GENERAL"
"PROMPTING"
"PROMPTING"
"RECOMMENDATIONS"
"NEWS"
"PROMPTING"
"PROMPTING"
"HOW-TO"
"NEWS"
"GENERAL"
"GENERAL"
"PROMPTING"
)
slugify() {
echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | head -c 60
}
run_version() {
local version="$1"
local outdir="$2"
echo ""
echo "=========================================="
echo " Running $version${#QUERIES[@]} queries"
echo "=========================================="
for i in "${!QUERIES[@]}"; do
local query="${QUERIES[$i]}"
local type="${TYPES[$i]}"
local slug=$(slugify "$query")
local num=$((i + 1))
local outfile="$outdir/${num}-${slug}.txt"
echo ""
echo "[$version] ($num/${#QUERIES[@]}) $query [$type]"
echo "$outfile"
# Run claude --print with the skill invocation
# --no-session-persistence: don't save to session history
# Timeout after 5 minutes per query (generous for slow API calls)
if timeout 300 claude --print \
"/last30days $query" \
> "$outfile" 2>"$outdir/${num}-${slug}.stderr.txt"; then
echo " ✅ Done ($(wc -l < "$outfile") lines)"
else
echo " ❌ Failed or timed out"
echo "FAILED: timeout or error" >> "$outfile"
fi
# Brief pause between queries to avoid rate limits
sleep 2
done
}
# === Phase 1: Test V1 ===
echo "📦 Backing up current SKILL.md..."
cp "$SKILL_DIR/SKILL.md" "$SKILL_DIR/SKILL.md.v2.bak"
echo "📥 Installing V1 SKILL.md from upstream..."
cd "$REPO_DIR"
git show upstream/main:SKILL.md > "$SKILL_DIR/SKILL.md"
# Also save a copy for reference
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v1-SKILL.md"
run_version "V1" "$V1_DIR"
# === Phase 2: Test V2 ===
echo ""
echo "📥 Restoring V2 SKILL.md..."
cp "$SKILL_DIR/SKILL.md.v2.bak" "$SKILL_DIR/SKILL.md"
# Also save a copy for reference
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v2-SKILL.md"
run_version "V2" "$V2_DIR"
# === Phase 3: Generate summary ===
echo ""
echo "=========================================="
echo " Generating comparison summary"
echo "=========================================="
SUMMARY="$OUT_DIR/comparison-summary.md"
cat > "$SUMMARY" << 'HEADER'
# V1 vs V2 Comparison Results
Generated: $(date)
## Output Files
| # | Query | Type | V1 Lines | V2 Lines |
|---|-------|------|----------|----------|
HEADER
# Replace the date placeholder
sed -i '' "s/\$(date)/$(date)/" "$SUMMARY"
for i in "${!QUERIES[@]}"; do
local query="${QUERIES[$i]}"
local type="${TYPES[$i]}"
local slug=$(slugify "$query")
local num=$((i + 1))
local v1file="$V1_DIR/${num}-${slug}.txt"
local v2file="$V2_DIR/${num}-${slug}.txt"
local v1lines=$(wc -l < "$v1file" 2>/dev/null || echo "0")
local v2lines=$(wc -l < "$v2file" 2>/dev/null || echo "0")
echo "| $num | \`$query\` | $type | $v1lines | $v2lines |" >> "$SUMMARY"
done
cat >> "$SUMMARY" << 'FOOTER'
## Scorecard Template
For each query, score both versions on:
| Dimension | V1 | V2 | Notes |
|-----------|----|----|-------|
| Query Parsing Display (1-5) | | | |
| Source Coverage (1-5) | | | |
| Citation Quality (1-5) | | | |
| Summary Structure (1-5) | | | |
| Stats Box Format (1-5) | | | |
| Research Grounding (1-5) | | | |
## Next Step
Read each pair of output files and score them using the test plan at:
`docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md`
FOOTER
echo ""
echo "✅ All done!"
echo "📁 Results: $OUT_DIR"
echo "📊 Summary: $SUMMARY"
echo ""
echo "V1 outputs: $V1_DIR/"
echo "V2 outputs: $V2_DIR/"
echo ""
echo "To review, run:"
echo " open $OUT_DIR"
# Cleanup backup
rm -f "$SKILL_DIR/SKILL.md.v2.bak"
```
## Acceptance Criteria
- [ ] Script runs all 17 queries on v1 SKILL.md
- [ ] Script runs all 17 queries on v2 SKILL.md
- [ ] Each query output saved to a separate .txt file
- [ ] Comparison summary generated with line counts
- [ ] SKILL.md restored to v2 after testing
- [ ] Both SKILL.md versions saved in output dir for reference
- [ ] Script handles timeouts gracefully (5 min per query)
- [ ] Brief pause between queries to avoid rate limits
## Cost Estimate
- 34 total `claude --print` invocations
- Each invocation: ~1 Python script run (OpenAI + xAI API) + 2-3 WebSearches + Claude response
- Estimated: ~$0.10-0.30 per invocation for API calls
- **Total estimate: $3-10 for the full run**
## Time Estimate
- Each query: ~1-3 minutes (script + WebSearch + synthesis)
- 17 queries × 2 versions = 34 runs
- **Total: ~45-90 minutes** (could run in background)
## How to Run
```bash
cd /Users/mvanhorn/last30days-skill-private
chmod +x scripts/test-v1-vs-v2.sh
./scripts/test-v1-vs-v2.sh
```
Or run in background:
```bash
nohup ./scripts/test-v1-vs-v2.sh > test-run.log 2>&1 &
tail -f test-run.log
```
## After the Run
Once all outputs are captured, Claude can read every file pair and generate the scored comparison doc with analysis — that's the part where I score each dimension 1-5 and write the final report.
## Files
| File | Purpose |
|------|---------|
| `scripts/test-v1-vs-v2.sh` | The test harness script |
| `docs/test-results/v1-vs-v2-*/` | Output directory (timestamped) |
| `docs/test-results/v1-vs-v2-*/v1/*.txt` | V1 outputs |
| `docs/test-results/v1-vs-v2-*/v2/*.txt` | V2 outputs |
| `docs/test-results/v1-vs-v2-*/comparison-summary.md` | Auto-generated summary |
@@ -0,0 +1,352 @@
---
title: "feat: Free Reddit via MCP — Remove OpenAI Key Requirement"
type: feat
date: 2026-02-06
---
# feat: Free Reddit via MCP — Remove OpenAI Key Requirement
## Overview
Replace the OpenAI Responses API (paid) for Reddit searching with a **free, zero-config Reddit MCP server**, eliminating the need for an `OPENAI_API_KEY` to get Reddit results in the last30days skill. This work happens in a **new forked repo** to keep the current release branch clean.
## Problem Statement
Today, the last30days skill requires an OpenAI API key (`OPENAI_API_KEY`) to search Reddit. This is the most expensive dependency in the stack — OpenAI charges per-call for the Responses API with web search. Users without an OpenAI key get zero Reddit results and fall back to WebSearch-only mode, which loses the engagement metrics (upvotes, comments) that make last30days uniquely valuable.
**Goal:** Make Reddit search completely free with no API key registration required.
## Research Findings
### Option Analysis
Five approaches were evaluated. Two stand out:
| Option | Free? | Search? | Engagement? | Setup | Notes |
|--------|-------|---------|-------------|-------|-------|
| **reddit-mcp-buddy** (Node.js) | Yes (anonymous mode) | Yes | Yes | One CLI command | 371 stars, 3-tier auth, most popular |
| **mcp-server-reddit** (Python) | Yes (redditwarp) | **No** | Yes | pip install | Recommended by ClaudeLog, but browse-only |
| **.json URL trick** (curl) | Yes | Yes | Yes | Zero | ~10 req/min, no dependencies |
| Reddit OAuth (PRAW) | Free but needs registration | Yes | Yes | App registration | 100 req/min, most reliable |
| RSS feeds | Yes | Basic | **No** | Zero | Least useful, no metrics |
### Top Contender: reddit-mcp-buddy
**GitHub:** [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy)
- 371 stars, actively maintained (last update: Jan 29, 2026)
- **Anonymous mode** — zero credentials, ~10 req/min
- **MCP tools:** `search_reddit`, `browse_subreddit`, `get_post_details`, `user_analysis`, `reddit_explain`
- **Install:** `claude mcp add --transport stdio reddit-mcp-buddy -s user -- npx -y reddit-mcp-buddy`
- Returns full engagement metrics (score, num_comments, upvote_ratio)
- TypeScript/Node.js (npx, no Python dependency)
### Strong Alternative: .json URL trick
- Append `.json` to any Reddit URL → full JSON response
- Search: `https://www.reddit.com/search.json?q=TOPIC&sort=relevance&t=month&limit=100`
- Zero dependencies, zero auth, zero setup
- Returns same data as official API (score, num_comments, created_utc, upvote_ratio)
- ~10 req/min rate limit (sufficient for skill use)
- Can be called via `curl` or Python `urllib`
### Also Considered
- **Hawstein/mcp-server-reddit** (134 stars, Python, no auth) — **No search tool**, only browse subreddits. Cannot replace OpenAI's keyword search capability.
- **Arindam200/reddit-mcp** (262 stars, PRAW) — Has search but **requires Reddit OAuth credentials**. Not truly zero-config.
- **adhikasp/mcp-reddit** (348 stars) — Hot threads only, no search. Last updated Dec 2024.
## Proposed Solution
### Approach A: MCP-First with .json Fallback (Recommended)
Add a new module `mcp_reddit.py` that:
1. **Detects** if a Reddit MCP server is configured in the user's Claude Code session
2. **If MCP available:** Calls MCP `search_reddit` tool via the skill's `allowed-tools` (the skill already allows tools — MCP tools become available when configured)
3. **If no MCP:** Falls back to Reddit's `.json` URL endpoints via `urllib` (stdlib, no dependencies)
4. **Normalize** MCP/JSON results to the existing `RedditItem` schema
5. **Enrich** with `reddit_enrich.py` (already works — fetches real thread data)
This gives users three tiers of Reddit access:
- **Tier 1 (best):** Reddit MCP installed → full search + engagement via MCP tools
- **Tier 2 (good):** No MCP, no keys → `.json` URL search + enrichment
- **Tier 3 (existing):** OpenAI key present → existing `openai_reddit.py` still works (backward compat)
### Approach B: .json-Only (Simpler)
Skip MCP entirely. Add a `reddit_json.py` module that uses `https://www.reddit.com/search.json` directly. Simpler but misses the MCP ecosystem integration.
### Approach C: MCP-Only (Cleaner)
Require MCP setup. Simpler code but adds a user setup step (`claude mcp add ...`).
**Recommendation: Approach A** — MCP-first with .json fallback gives zero-config Reddit search for everyone while rewarding users who set up MCP with a better experience.
## Technical Approach
### Architecture
```
User invokes /last30days "topic"
├─ env.py detects sources:
│ ├─ MCP reddit available? → mcp_reddit.py
│ ├─ No MCP, no key? → reddit_json.py (.json URL trick)
│ └─ OPENAI_API_KEY set? → openai_reddit.py (existing, backward compat)
├─ X search (Bird CLI / xAI — unchanged)
└─ Normalize → Score → Dedupe → Render (unchanged)
```
### New Files
#### `scripts/lib/reddit_json.py`
- Uses `urllib.request` (stdlib) to call Reddit's `.json` search endpoint
- URL: `https://www.reddit.com/search.json?q={topic}&sort=relevance&t=month&limit=100`
- Custom `User-Agent` header (required by Reddit)
- Parses response into the same format as `openai_reddit.py` output
- Handles pagination via `after` token if depth=deep (multiple requests)
- Rate limiting: 1 second delay between requests
#### `scripts/lib/mcp_reddit.py`
- Detects MCP availability (check if Reddit MCP tools exist in session)
- Formats MCP tool calls for `search_reddit` with topic + date filters
- Normalizes MCP response to match existing `RedditItem` schema
- Falls back to `reddit_json.py` if MCP unavailable
#### Modified Files
- **`scripts/lib/env.py`** — Add Reddit source detection: MCP → .json → OpenAI
- **`scripts/last30days.py`** — Route Reddit search through new priority chain
- **`scripts/lib/normalize.py`** — Add normalizer for `.json` endpoint response format
- **`SKILL.md`** — Document MCP setup as optional enhancement
### MCP Integration Design
The MCP approach has a subtle challenge: the last30days skill runs as a **Python subprocess** (`python3 last30days.py`), but MCP tools are available to **Claude's session**, not to subprocesses.
**Two paths to solve this:**
**Path 1: SKILL.md orchestration** — The SKILL.md workflow calls the MCP `search_reddit` tool directly (before or in parallel with the Python script), saves the MCP response to a temp file, and the Python script reads it:
```
SKILL.md workflow:
1. Call MCP search_reddit → save to /tmp/last30days_mcp_reddit.json
2. Call python3 last30days.py --reddit-from=/tmp/last30days_mcp_reddit.json --emit=compact
3. Python script reads pre-fetched Reddit data instead of calling OpenAI
```
**Path 2: .json only for subprocess** — The Python script uses `.json` URLs directly (no MCP needed in subprocess). MCP is a bonus for users who want Claude to also browse specific threads interactively.
**Recommendation: Path 2 for MVP, Path 1 as enhancement.** The `.json` approach is self-contained, testable, and doesn't require SKILL.md workflow changes. MCP can be layered on later.
### Source Priority Chain (updated env.py)
```python
def get_reddit_source(config: dict) -> str:
"""Returns: 'mcp', 'json', 'openai', or 'none'"""
# 1. Check for pre-fetched MCP data (from SKILL.md)
if os.path.exists(MCP_REDDIT_CACHE_PATH):
return 'mcp'
# 2. Always available — no key needed
# (reddit .json endpoints are free)
return 'json'
# 3. OpenAI key present → legacy path
# if config.get('OPENAI_API_KEY'):
# return 'openai'
```
For MVP, the `.json` path is **always available** so it becomes the default. OpenAI path remains as opt-in for users who want higher rate limits.
### Data Mapping: .json → RedditItem
Reddit `.json` search returns `data.children[].data` with:
```json
{
"title": "Post title",
"permalink": "/r/subreddit/comments/abc123/...",
"subreddit": "ClaudeAI",
"score": 42,
"num_comments": 15,
"upvote_ratio": 0.95,
"created_utc": 1738800000,
"selftext": "Post body...",
"url": "https://...",
"author": "username"
}
```
Maps cleanly to existing `RedditItem`:
| .json field | RedditItem field | Notes |
|-------------|------------------|-------|
| `title` | `title` | Direct |
| `permalink` | `url` | Prepend `https://www.reddit.com` |
| `subreddit` | `subreddit` | Direct |
| `score` | `engagement.score` | Direct |
| `num_comments` | `engagement.num_comments` | Direct |
| `upvote_ratio` | `engagement.upvote_ratio` | Direct |
| `created_utc` | `date` | Convert epoch → YYYY-MM-DD |
| `selftext` | (used for relevance) | AI relevance scoring needed |
| `author` | (not in current schema) | Ignore for now |
### Relevance Scoring Without AI
The current `openai_reddit.py` gets relevance scores **from OpenAI** (the AI judges how relevant each result is). With `.json` endpoints, we lose that AI relevance judgment.
**Options:**
1. **Keyword matching** — Score based on how many query terms appear in title + selftext. Simple but effective.
2. **TF-IDF-like** — Weight rarer query terms higher. More accurate but more code.
3. **Let Claude judge** — Pass results to Claude in SKILL.md and have Claude score relevance. Most accurate but changes the workflow.
4. **Skip relevance, rely on Reddit's sort** — Reddit's `sort=relevance` already ranks by relevance. Trust it and use position-based scoring (first result = 1.0, last = 0.5).
**Recommendation: Option 4 for MVP.** Reddit's search relevance ranking is already good. Use position-based relevance (1.0 → 0.5 linear decay over result set) combined with the existing engagement-based scoring. This requires zero external dependencies and no AI calls.
## Implementation Phases
### Phase 0: Fork Repository
- [ ] Create new repo `last30days-free-reddit` (or branch in private repo)
- [ ] `git clone /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit`
- [ ] Create feature branch: `feat/free-reddit`
- [ ] Verify all existing tests pass on the new branch
### Phase 1: reddit_json.py — Core .json Search
- [ ] Create `scripts/lib/reddit_json.py`
- `search_reddit_json(topic: str, depth: str, date_from: str, date_to: str) -> list[dict]`
- Build search URL with `q`, `sort=relevance`, `t=month`, `limit` (25/50/100 by depth)
- Custom `User-Agent: last30days-skill/2.0 (by /u/last30days-bot)`
- Parse `data.children[].data` → list of raw dicts
- Handle pagination for deep mode (follow `after` token, max 3 pages)
- Rate limit: `time.sleep(1.0)` between requests
- Error handling: 429 (rate limit), 403 (blocked), network errors → return empty + error msg
- [ ] Create `tests/test_reddit_json.py`
- Mock HTTP responses using fixture files
- Test: basic search parsing, pagination, rate limit handling, error cases
- [ ] Create `fixtures/reddit_json_search_sample.json`
- Real `.json` search response (sanitized)
### Phase 2: Normalize + Score .json Results
- [ ] Add `normalize_reddit_json()` to `scripts/lib/normalize.py`
- Convert `.json` response format → `RedditItem` schema
- `created_utc` (epoch) → `YYYY-MM-DD` string
- `permalink` → full URL
- Position-based relevance: `1.0 - (index / total * 0.5)` → range [1.0, 0.5]
- Date confidence: `"high"` (Reddit provides exact timestamps)
- [ ] Update `scripts/lib/score.py` if needed
- `.json` results already have real engagement metrics — existing scoring formula works as-is
- No penalty needed (unlike WebSearch which lacks engagement)
- [ ] Add tests for normalization + scoring of `.json` data
### Phase 3: Integration into Orchestrator
- [ ] Update `scripts/lib/env.py`
- Add `get_reddit_source(config) -> str` returning `'json'`, `'openai'`, or `'none'`
- Priority: `.json` always available (default), `'openai'` if key present and user prefers
- Add `REDDIT_SOURCE` config option: `auto` (default), `json`, `openai`
- Update `get_available_sources()` to always include Reddit (since `.json` is free)
- Update `get_missing_keys()` — Reddit no longer shows as "missing"
- [ ] Update `scripts/last30days.py`
- Add `_search_reddit_json()` function alongside existing `_search_reddit()`
- Route based on `get_reddit_source()`: json → `_search_reddit_json()`, openai → `_search_reddit()`
- Skip `reddit_enrich.py` for `.json` results (already have real engagement metrics!)
- Update progress/stats output to show source: "Reddit (free)" vs "Reddit (OpenAI)"
- [ ] Update SKILL.md promo messaging
- Remove "Add OPENAI_API_KEY for Reddit" messaging
- Instead: "Reddit search included free! Add OpenAI key for AI-enhanced relevance scoring."
- [ ] Integration tests: full pipeline with `.json` mock data
### Phase 4: Testing & Polish
- [ ] Run all existing tests — ensure backward compatibility
- [ ] Manual test: invoke `/last30days` with NO API keys → should get Reddit + WebSearch results
- [ ] Manual test: invoke with OPENAI_API_KEY → should still use OpenAI path (backward compat)
- [ ] Manual test: compare result quality — `.json` vs OpenAI for same topic
- [ ] Update README.md — document free Reddit access
- [ ] Update SPEC.md — document new source priority chain
### Phase 5 (Future): MCP Enhancement Layer
- [ ] Detect Reddit MCP in Claude's session
- [ ] SKILL.md pre-fetches via MCP `search_reddit` → saves to temp file
- [ ] Python script reads pre-fetched MCP data via `--reddit-from=` flag
- [ ] MCP results get AI-judged relevance (since Claude sees them)
- [ ] Better than `.json` position-based relevance
## Acceptance Criteria
### Functional
- [ ] `/last30days "any topic"` returns Reddit results with **zero API keys configured**
- [ ] Reddit results include real engagement metrics (score, comments, upvote_ratio)
- [ ] Results are properly scored, deduped, and rendered (same quality as OpenAI path)
- [ ] Existing OpenAI path still works when key is present
- [ ] Existing X search (Bird CLI / xAI) is unchanged
- [ ] Stats box shows correct source indicator ("Reddit (free)" or "Reddit (OpenAI)")
### Non-Functional
- [ ] No external Python packages (stdlib only — `urllib.request`, `json`, `time`)
- [ ] Respects Reddit rate limits (~10 req/min for unauthenticated)
- [ ] Graceful degradation if Reddit blocks requests (429/403 → empty results + error msg)
- [ ] All new code has unit tests with fixtures (no live API calls in tests)
### Quality Gates
- [ ] All existing tests pass (zero regressions)
- [ ] New tests cover: search parsing, normalization, scoring, error handling, pagination
- [ ] Manual smoke test passes with zero API keys
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Reddit blocks `.json` endpoints | Low | High | Fall back to OpenAI path; monitor for 429s |
| `.json` rate limit too restrictive | Medium | Medium | 1s delay between requests; cache results |
| Relevance quality lower without AI scoring | Medium | Medium | Reddit's `sort=relevance` is decent; can add keyword scoring later |
| Reddit changes `.json` response format | Low | Medium | Schema validation in normalize; fixture-based tests catch changes |
| Node.js MCP server dependency conflicts | N/A (Phase 5) | Low | MCP is optional enhancement, not required |
## Dependencies
- **None** — this feature uses only Python stdlib and Reddit's public `.json` endpoints
- Phase 5 (future) would add optional dependency on an MCP server
## Repository Setup
```bash
# Fork into new working repo
cp -r /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit
cd /Users/mvanhorn/last30days-free-reddit
# Create feature branch
git checkout -b feat/free-reddit
# Verify existing tests
python3 -m pytest tests/ -v
```
The new repo is disposable — once the feature is validated, changes merge back into `last30days-skill-private` via PR or cherry-pick.
## References
### Internal
- `scripts/lib/openai_reddit.py` — Current Reddit search (to be replaced/supplemented)
- `scripts/lib/env.py:get_available_sources()` — Source detection logic to update
- `scripts/lib/normalize.py` — Add `.json` normalizer alongside existing
- `scripts/lib/reddit_enrich.py` — May be skippable for `.json` results (already have engagement)
- `scripts/lib/schema.py:RedditItem` — Target schema (unchanged)
### External
- [Reddit .json search endpoint](https://www.reddit.com/search.json?q=test&sort=relevance&t=month&limit=25)
- [Simon Willison — Scraping Reddit via JSON API](https://til.simonwillison.net/reddit/scraping-reddit-json)
- [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy) — Best MCP option for Phase 5
- [Hawstein/mcp-server-reddit](https://github.com/Hawstein/mcp-server-reddit) — ClaudeLog-recommended MCP (no search though)
- [Reddit API Rate Limits Guide](https://painonsocial.com/blog/reddit-api-rate-limits-guide)
### Research Sources
- last30days skill output — community recommendations for Reddit search tools
- GitHub search — 10+ Reddit MCP repos evaluated
- npm/PyPI registries — package availability confirmed
- ClaudeLog — [Reddit MCP reference](https://claudelog.com/claude-code-mcps/reddit-mcp/)
@@ -0,0 +1,67 @@
---
title: "release: Push V2 to public repo and launch"
type: release
date: 2026-02-07
---
# release: Push V2 to public repo and launch
## Overview
Push all V2 changes from the private development repo to the public GitHub repo, then sync the installed skill. 52 commits need to go from `origin/main` (private) to `upstream/main` (public).
## Current State
- **Private repo:** `https://github.com/mvanhorn/last30days-skill-private` - 52 commits ahead of public
- **Public repo:** `https://github.com/mvanhorn/last30days-skill` - last commit is V1 (`cc892d7`)
- **Upstream remote:** Already configured in private repo
- **Untracked files in private:** test logs, draft plans - these should NOT be pushed
## Steps
### Step 1: Clean up private repo
- [ ] Review untracked files - make sure nothing sensitive gets pushed
- [ ] Decide: commit the untracked docs/plans or leave them out of the push
### Step 2: Push to public
```bash
cd /Users/mvanhorn/last30days-skill-private
git push upstream main
```
This pushes all 52 commits from private `main` to public `main`. Since the private repo was forked from public, history is shared - this is a fast-forward push.
### Step 3: Sync installed skill
```bash
# Update the installed copy that Claude Code actually uses
cd ~/.claude/skills/last30days
git pull origin main
```
### Step 4: Verify
- [ ] Check `https://github.com/mvanhorn/last30days-skill` shows V2 README with new examples
- [ ] Check SKILL.md has the new argument-hint and timing disclaimer
- [ ] Check bird_x.py has the fixed `_extract_core_subject()` and retry logic
- [ ] Run a quick `/last30days` test to confirm installed skill works
## What Gets Published
Key files going public:
- `README.md` - V2 features, 3 new examples (Nano Banana Pro, Kanye, Vibe Motion), speed tradeoff note
- `SKILL.md` - New argument-hint, timing disclaimer, citation rules
- `scripts/lib/bird_x.py` - Fixed query construction + retry logic
- `scripts/lib/http.py` - USER_AGENT bumped to 2.0
- `.claude-plugin/plugin.json` - Marketplace support
- All Bird CLI integration code
- Phase 2 supplemental search code
- Model fallback chain
## Risks
- **Low risk:** This is a fast-forward push, no force push needed
- **Public API keys:** Already confirmed - no `.env` files or secrets in the repo
- **Untracked files:** Won't be pushed unless committed first
@@ -0,0 +1,194 @@
---
title: ClawHub Scanner Compliance for last30days-official
type: feat
date: 2026-02-15
---
# ClawHub Scanner Compliance for last30days-official
## Overview
Make the last30days skill pass ClawHub's security scanner (VirusTotal + Code Insight) so it can be published as `last30days-official`. The user's 8 other mvanhorn skills already pass - we replicate their exact pattern.
## Problem Statement
ClawHub requires skills to pass a multi-layer security scan before publication:
1. Metadata validation (frontmatter fields)
2. VirusTotal automated scanning
3. LLM-powered Code Insight analysis (checks if capabilities match documentation)
4. Credential handling review
The current last30days SKILL.md has basic `metadata.clawdbot` but is missing fields the scanner checks: `emoji`, `user-invocable`, `disable-model-invocation`, `files` declaration. There's no `## Security & Permissions` section (required pattern from passing skills). README has no security/privacy documentation.
## Proposed Solution
Follow the exact pattern from `clawdbot-skill-xai` and `clawdbot-skill-search-x` (both pass the scanner). Three files need changes.
### Fix 1: SKILL.md Frontmatter
Add missing scanner fields to the existing frontmatter:
```yaml
---
name: last30days
version: "2.1"
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web."
argument-hint: 'last30 AI video tools, last30 best project management tools'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill
user-invocable: true
disable-model-invocation: true
metadata:
clawdbot:
emoji: "📰"
requires:
env:
- OPENAI_API_KEY
bins:
- node
- python3
primaryEnv: OPENAI_API_KEY
files:
- "scripts/*"
homepage: https://github.com/mvanhorn/last30days-skill
tags:
- research
- reddit
- x
- youtube
- trends
- prompts
---
```
Key additions:
- `user-invocable: true` - human must trigger it
- `disable-model-invocation: true` - agent cannot self-trigger
- `emoji: "📰"` - required display field
- `files: ["scripts/*"]` - prevents false "instruction-only but has scripts" flag
### Fix 2: Security & Permissions Section in SKILL.md
Add to the bottom of SKILL.md (matches xai/search-x pattern exactly):
```markdown
## Security & Permissions
**What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
**What this skill does NOT do:**
- Does not post, like, or modify content on any platform
- Does not access your Reddit, X, or YouTube accounts
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
Review scripts before first use to verify behavior.
```
### Fix 3: Security & Privacy Section in README.md
Add after the "How It Works" section:
```markdown
## Security & Privacy
### Data that leaves your machine
| Destination | Data Sent | API Key Required |
|------------|-----------|-----------------|
| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
Your research topic is included in all outbound API requests. If you research sensitive topics, be aware that query strings are transmitted to the API providers listed above.
### Data stored locally
- API keys: `~/.config/last30days/.env` (chmod 600 recommended)
- Watchlist database: `~/.local/share/last30days/research.db` (SQLite)
- Briefings: `~/.local/share/last30days/briefs/`
### API key isolation
Each API key is transmitted only to its respective endpoint. Your OpenAI key is never sent to xAI, Brave, or any other provider. Browser cookies for X are read locally and used only for Twitter GraphQL requests.
```
### Fix 4: Update .claude-plugin Files
**plugin.json** - bump version, add youtube keyword:
```json
{
"name": "last30days",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0",
"author": {"name": "mvanhorn"},
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
"skills": ["./"]
}
```
**marketplace.json** - add version, update description:
```json
{
"name": "last30days",
"owner": {"name": "mvanhorn", "url": "https://github.com/mvanhorn"},
"metadata": {
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0"
},
"plugins": [{"name": "last30days", "source": "."}]
}
```
## What We DON'T Need
Based on the audit of your 8 passing skills:
- **Script-level security manifest headers** - your passing skills (xai, search-x, parallel) do NOT have these. The scanner relies on SKILL.md, not per-file headers.
- **Separate SECURITY.md file** - not needed; README section + SKILL.md section is sufficient.
- **Shell injection fixes** - already clean. All subprocess calls use list-form args, no `shell=True` anywhere.
- **Credential leak fixes** - already clean. All keys loaded from env vars, none hardcoded or logged.
## Acceptance Criteria
- [x] SKILL.md frontmatter has `user-invocable`, `disable-model-invocation`, `emoji`, `files`
- [x] SKILL.md has `## Security & Permissions` section with "does" and "does NOT do" lists
- [x] README.md has `## Security & Privacy` section with endpoint table and key isolation docs
- [x] plugin.json version bumped to 2.1.0, youtube keyword added
- [x] marketplace.json version and description updated
- [x] `python3 scripts/last30days.py --diagnose` still works after changes
- [x] Synced to all installed skill locations
- [ ] Published to ClawHub as `last30days-official` (when auth is fixed)
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Add frontmatter fields + Security & Permissions section |
| `README.md` | Add Security & Privacy section |
| `.claude-plugin/plugin.json` | Bump version, add youtube keyword |
| `.claude-plugin/marketplace.json` | Add version, update description |
## References
- [13-point ClawHub checklist](https://gist.github.com/adhishthite/0db995ecfe2f23e09d0b2d418491982c)
- [ClawHub docs](https://docs.openclaw.ai/tools/clawhub)
- [ClawHub Developer Guide 2026](https://www.digitalapplied.com/blog/clawhub-skills-marketplace-developer-guide-2026)
- Your passing skills: `clawdbot-skill-xai`, `clawdbot-skill-search-x` (exact pattern replicated)
@@ -0,0 +1,76 @@
---
title: "fix: watchlist engagement null crash"
type: fix
date: 2026-02-15
---
# fix: Watchlist crashes on `engagement: null` from X posts
## Problem
When X posts return `engagement: null` (JSON null) instead of `engagement: {}` (empty object), the watchlist `_run_topic` findings parser crashes. This is because Python's `dict.get("engagement", {})` returns `None` when the key **exists** with value `None` — the default `{}` only applies when the key is **missing**.
```python
# CRASHES — .get() returns None, not {}
item.get("engagement", {}).get("likes", 0)
# AttributeError: 'NoneType' object has no attribute 'get'
```
**Reporter:** Soft launch tester, 2026-02-15
**Severity:** Medium — breaks watchlist `run-one` and `run-all` for any topic that pulls X posts with null engagement
**One-shot research unaffected** — the main `last30days.py` pipeline uses `normalize.py` which has `isinstance(eng_raw, dict)` guards
## Root Cause
Two locations use the vulnerable `dict.get("key", {})` pattern:
1. **`scripts/watchlist.py:188`** — THE REPORTED BUG
```python
"engagement_score": item.get("engagement", {}).get("likes", 0),
```
2. **`scripts/lib/normalize.py:177`** — YouTube normalizer (same pattern, latent)
```python
eng_raw = item.get("engagement", {})
```
Three other locations are already safe — they use `isinstance(eng_raw, dict)`:
- `scripts/lib/normalize.py:70` (Reddit)
- `scripts/lib/normalize.py:130` (X)
- `scripts/lib/xai_x.py:190`
## Fix
Apply the `or {}` idiom (as suggested by reporter):
### scripts/watchlist.py:188
```python
# Before
"engagement_score": item.get("engagement", {}).get("likes", 0),
# After
"engagement_score": (item.get("engagement") or {}).get("likes", 0),
```
### scripts/lib/normalize.py:177
```python
# Before
eng_raw = item.get("engagement", {})
# After
eng_raw = item.get("engagement") or {}
```
## Acceptance Criteria
- [ ] `watchlist.py run-one` handles X posts with `engagement: null` without crashing
- [ ] `watchlist.py run-one` handles X posts with `engagement: {}` (empty object)
- [ ] `watchlist.py run-one` handles X posts with no engagement key at all
- [ ] YouTube normalizer handles `engagement: null` without crashing
- [ ] Existing tests still pass
## Regarding the screenshot question
The tester asked "Did you use watchlist on open claw or Claude?" — watchlist is designed for the **open variant** (Open Claw). It works in Claude Code too since it's just Python + SQLite, but the SKILL.md routing for watchlist commands is only in `variants/open/SKILL.md`. The main `SKILL.md` is one-shot research only.
Answer to give tester: "Watchlist works in both — it's plain Python. But the skill routing that understands 'watch add topic' is in the open variant. In Claude Code you'd need to call the script directly or use the open variant SKILL.md."
@@ -0,0 +1,221 @@
---
title: Fix YouTube Display and Search Quality
type: fix
date: 2026-02-15
---
# Fix YouTube Display and Search Quality
## Overview
YouTube is the v2.1 headline feature but it's broken in two ways: results don't appear in Claude's synthesis (display bug), and search quality is worse than youtube.com (search bug). Both need fixing before launch.
## Problem Statement
**Display bug:** YouTube data exists in the script output but Claude never sees it. Reproduced on 4/5 recent test runs (Kanye, Seedance 2, Peter Steinberger, YouTube thumbnails). The skill worked once — the earlier "YouTube thumbnails" and "OpenClaw" runs showed YouTube stats — but subsequent runs silently dropped it.
**Search quality bug:** User searched "how to get on seedance 2" on youtube.com and got multiple recent results. yt-dlp returned 10 videos for the same query, but they included old irrelevant content because date filtering is broken.
## Root Cause Analysis
### Display Bug — Three compounding causes
1. **`2>&1` in SKILL.md bash command** (line 79) merges stderr progress messages into stdout. When Claude Code receives this mixed output, the YouTube section (which renders LAST after Reddit + X) can get lost in the noise or hit the 30K char Bash output limit.
2. **Background execution** (partially fixed). The old SKILL.md said "DO WEBSEARCH WHILE SCRIPT RUNS" which caused Claude to background the bash command. Backgrounded commands return truncated output via Task Output. *Already fixed in this session — SKILL.md now says FOREGROUND with 5-minute timeout.*
3. **No explicit model instruction to look for YouTube.** The SKILL.md tells Claude to synthesize but doesn't emphasize that YouTube data is in the script output and must be included.
### Search Quality Bug — `--flat-playlist` breaks date filtering
The yt-dlp command in `youtube_yt.py:110-116`:
```bash
yt-dlp ytsearch{count}:{query} --dateafter {YYYYMMDD} --flat-playlist --dump-json
```
**`--flat-playlist` causes three problems:**
1. `--dateafter` is silently ignored (no video-level metadata to filter on)
2. All items have `date: None` (upload_date not in flat-playlist JSON)
3. Old content leaks in (e.g., "the greatest youtube thumbnails of all time" returned for a 30-day query)
**`_extract_core_subject()` over-strips useful YouTube terms:**
- Strips "tips", "tutorial", "review" — but these ARE the content types people search for on YouTube
- "youtube thumbnail tips" → "youtube thumbnail" loses the intent signal
## Proposed Solution
### Phase 1: Fix Display (Critical — blocks launch)
#### 1a. Remove `2>&1` from SKILL.md bash command
**File:** `SKILL.md:79` (both `last30days-skill-private/SKILL.md` and `~/.claude/skills/last30days21/SKILL.md`)
```bash
# Before:
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
# After:
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
```
This removes ~1-5KB of progress spam from the model's input and ensures clean stdout-only output.
#### 1b. Add YouTube-specific synthesis instruction to SKILL.md
After the "Read the ENTIRE output" instruction, add:
```markdown
**The script output has THREE sections: Reddit items, X items, and YouTube items (in that order).
If you see YouTube items in the output, you MUST include them in your synthesis and stats block.
YouTube items look like: `**{video_id}** (score:N) {channel} [N views, N likes]`**
```
#### 1c. Verify fix with test run
Run `/last30days21 youtube thumbnail tips` and confirm YouTube appears in stats.
### Phase 2: Fix Search Quality (High — headline feature quality)
#### 2a. Remove `--flat-playlist` flag
**File:** `scripts/lib/youtube_yt.py:110-116`
```python
# Before:
cmd = [
"yt-dlp",
f"ytsearch{count}:{core_topic}",
"--dateafter", date_filter,
"--flat-playlist",
"--dump-json",
]
# After:
cmd = [
"yt-dlp",
f"ytsearch{count}:{core_topic}",
"--dateafter", date_filter,
"--dump-json",
"--no-warnings",
"--no-download",
]
```
**Impact:** Slower (yt-dlp resolves each video page for metadata) but:
- `--dateafter` actually works — filters old content
- `upload_date` populated — items get real dates
- Engagement metrics more accurate
**Risk:** Could increase search time from ~5s to ~30-60s for 20 videos. Mitigate by reducing default count or increasing timeout.
**Alternative if too slow:** Keep `--flat-playlist` but append year to search query:
```python
# Bias toward recent content since --dateafter doesn't work with flat-playlist
search_query = f"{core_topic} {from_date[:4]}" # e.g., "youtube thumbnail 2026"
```
#### 2b. Fix `_extract_core_subject()` for YouTube-relevant terms
**File:** `scripts/lib/youtube_yt.py:67-76`
Don't strip terms that are useful YouTube content type signals:
```python
# YouTube-specific: keep 'tips', 'tutorial', 'review' etc.
# These are stripped for Reddit/X search but are valuable for YouTube
noise = {
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features',
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
}
# NOTE: 'tips', 'tricks', 'tutorial', 'guide', 'review', 'reviews'
# are intentionally KEPT — they're YouTube content types
```
#### 2c. Clean question marks and trailing punctuation
**File:** `scripts/lib/youtube_yt.py:48-80`
```python
# At the end of _extract_core_subject():
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.') # Clean trailing punctuation
```
### Phase 3: Polish (Medium — nice to have before launch)
#### 3a. Increase subprocess timeout for non-flat-playlist mode
**File:** `scripts/lib/youtube_yt.py:119-121`
```python
# Before:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
# After — resolving video pages takes longer:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
```
#### 3b. Add year hint to search query for recency bias
Even with `--dateafter` working, YouTube's search algorithm ranks by relevance not recency. Adding the year helps:
```python
# After core_topic extraction:
import datetime
current_year = datetime.datetime.now().year
search_query = f"{core_topic} {current_year}"
```
#### 3c. Reduce compact render item count for YouTube
The compact render currently shows up to 15 YouTube items. With transcripts, this is too much output. Reduce to 10:
**File:** `scripts/lib/render.py` — in `render_compact()`, add YouTube-specific limit or reduce the default.
## Acceptance Criteria
- [ ] `/last30days21 youtube thumbnail tips` shows YouTube in stats block
- [ ] YouTube items have real dates (not `None`)
- [ ] Old content (> 30 days) filtered out by `--dateafter`
- [ ] "youtube thumbnail tips" search returns videos about thumbnail tips (not generic old content)
- [ ] "How to access Seedance 2" returns recent Seedance 2 tutorials
- [ ] Script completes within 5 minutes for default depth
- [ ] No `2>&1` in SKILL.md bash command
## Files to Modify
| File | Changes |
|------|---------|
| `SKILL.md` | Remove `2>&1`, add YouTube synthesis instruction |
| `scripts/lib/youtube_yt.py` | Remove `--flat-playlist`, fix noise words, add year hint, increase timeout |
| `scripts/lib/render.py` | Optional: reduce YouTube item limit in compact mode |
## Testing
```bash
# Quick smoke test — should show YouTube items with dates
cd ~/.claude/skills/last30days21
python3 scripts/last30days.py "youtube thumbnail tips" --quick --emit=compact 2>/dev/null | grep -c "youtube.com"
# Date test — should show YYYY-MM-DD dates, not None
python3 -c "
from scripts.lib import youtube_yt
r = youtube_yt.search_youtube('youtube thumbnail tips', '2026-01-16', '2026-02-15', depth='quick')
for v in r['items'][:3]: print(v['date'], v['title'][:50])
"
# Full integration — run the skill in Claude Code and verify YouTube in stats
```
## References
- SKILL.md bash command: `scripts/last30days.py` line 79
- YouTube search: `scripts/lib/youtube_yt.py` lines 83-174
- Core subject extraction: `scripts/lib/youtube_yt.py` lines 48-80
- Compact render: `scripts/lib/render.py` lines 48-238
- Prior YouTube plan: `docs/plans/2026-02-14-feat-youtube-transcript-search-plan.md`
@@ -0,0 +1,102 @@
---
title: Fix YouTube Timeout and Reddit Resilience
type: fix
date: 2026-02-15
---
# Fix YouTube Timeout and Reddit Resilience
## Overview
YouTube search+transcript fetching exceeds the 60s future timeout on popular topics (20 videos + 5 transcripts), discarding all results. Reddit 429 rate-limiting burns through the entire time budget with aggressive retries, causing global timeouts. Both issues cause the script to return incomplete results and sometimes crash.
## Problem Statement
**YouTube timeout (blocks launch):** YouTube found 20 Seedance 2.0 videos and fetched 4/5 transcripts, but the 60s future timeout killed everything — `✗ Error: YouTube search timed out after 60s`. The data was there; the budget wasn't.
**Reddit 429 cascade (degrades reliability):** One enrichment item hitting 429 burns up to 93s (3 retries x 30s timeout + backoff) against a 45s budget. Phase 2 supplemental search then also 429s, burning another 30s. Total wasted: 75s on doomed requests, triggering the 180s global timeout.
Observed failures across 6 test runs:
- `seedance 2 access`: YouTube timed out (60s), Reddit 0 threads
- `kanye west bully`: Global timeout first run, needed --quick retry
- `Peter Steinberger`: Global timeout during enrichment
- `nano banana pro`: Reddit timed out, YouTube worked (5 videos in time)
- `kanye west bully` (retry): Reddit 0 threads, YouTube 4 videos
## Proposed Solution
### Fix 1: Bump YouTube future timeout (NOT YET DONE)
**File:** `scripts/last30days.py:40-44`
Give YouTube its own timeout, separate from the shared `future` timeout. YouTube inherently takes longer because it does search + parallel transcript fetching.
```python
# Option A: YouTube-specific timeout key
TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 30, "youtube_future": 60, ...},
"default": {"global": 180, "future": 60, "youtube_future": 90, ...},
"deep": {"global": 300, "future": 90, "youtube_future": 120, ...},
}
# Option B: Simpler — just bump default future to 90 for all sources
TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 45, ...},
"default": {"global": 180, "future": 90, ...},
"deep": {"global": 300, "future": 120, ...},
}
```
**Recommendation:** Option A (YouTube-specific key) — keeps Reddit/X futures tight while giving YouTube the breathing room it needs. Reddit/X finish in 20-40s; YouTube needs 60-90s for transcript fetching.
Then where YouTube future is collected (~line 646):
```python
youtube_timeout = timeouts.get("youtube_future", timeouts["future"])
youtube_items, youtube_error = youtube_future.result(timeout=youtube_timeout)
```
### Fix 2: Reddit 429 fail-fast (ALREADY DONE — needs commit)
**Status:** Implemented in working tree, uncommitted. Changes across 4 files:
| File | Change |
|------|--------|
| `scripts/lib/http.py` | `get_reddit_json()` accepts `timeout` and `retries` params (was hardcoded 30s/3) |
| `scripts/lib/reddit_enrich.py` | `RedditRateLimitError` exception; `fetch_thread_data()` propagates 429; `enrich_reddit_item()` defaults to 10s timeout / 1 retry |
| `scripts/lib/openai_reddit.py` | `search_subreddits()` reduced to 1 retry; breaks subreddit loop on first 429 |
| `scripts/last30days.py` | Enrichment loop catches `RedditRateLimitError`, cancels futures, bails; `rate_limited` flag skips Phase 2 Reddit |
**Impact:** 429 scenario drops from ~75s wasted to ~12s wasted.
## Acceptance Criteria
- [ ] YouTube `seedance 2 access` query completes with videos (was timing out at 60s)
- [ ] YouTube `kanye west bully` query returns 4+ videos with transcripts
- [ ] Reddit 429 detected within ~12s, remaining enrichment skipped
- [ ] Phase 2 Reddit skipped when rate-limited
- [ ] No global timeout on default depth for any of the 5 test queries
- [ ] All changes committed and synced to `~/.claude/skills/last30days21/` and `~/.claude/skills/last30days/`
## Files to Modify
| File | Status | Changes |
|------|--------|---------|
| `scripts/last30days.py` | Modify (partially done) | Add `youtube_future` to TIMEOUT_PROFILES; use it for YouTube future collection |
| `scripts/lib/http.py` | Done (uncommitted) | Parameterized `get_reddit_json()` timeout/retries |
| `scripts/lib/reddit_enrich.py` | Done (uncommitted) | `RedditRateLimitError`, fail-fast enrichment |
| `scripts/lib/openai_reddit.py` | Done (uncommitted) | 429 early-bail in `search_subreddits()` |
## Testing
```bash
# Quick smoke test — YouTube should complete within 90s
cd ~/.claude/skills/last30days21
python3 scripts/last30days.py "seedance 2 access" --emit=compact 2>&1 | grep -E "YouTube|timeout"
# Verify YouTube items in output
python3 scripts/last30days.py "kanye west bully" --quick --emit=compact 2>/dev/null | grep "youtube.com" | wc -l
# Full integration — run the skill in Claude Code
# /last30days21 seedance 2 access
# Verify YouTube appears in stats block
```
@@ -0,0 +1,255 @@
---
title: "feat: Create GitHub Release for v2.1"
type: feat
date: 2026-02-17
---
# Create GitHub Release for v2.1
## Overview
The last30days skill is at v2.1 with 2,747 stars and 317 forks, but has **zero git tags and zero GitHub Releases**. The code is already live on the public repo (`upstream/main`). All marketing copy is written. This plan creates a proper v2.1.0 GitHub Release from existing materials.
## Problem Statement
GitHub Releases provide:
- A landing page for each version with formatted release notes
- Discoverability (GitHub shows releases in the sidebar, feeds them to the Explore page)
- A stable reference point for users (`git checkout v2.1.0`)
- A "Full Changelog" diff link
- RSS feed for watchers
- New contributor callouts (community goodwill)
Currently users have no way to reference a specific version of the skill. The README says "v2.1" but there's nothing in git to anchor that.
## Proposed Solution
Create the first GitHub Release (v2.1.0) on the **public** repo using existing marketing copy. Use an annotated tag (not lightweight) for proper `git describe` support and fork propagation.
## Implementation Steps
### Phase 1: Tag and Release
#### 1. Create CHANGELOG.md
Create `CHANGELOG.md` following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format. Source material already exists:
- `README.md` "What's New in V2.1" and "What's New in V2" sections
- `docs/v2.1-tweets.md` (raw verified test results, feature descriptions)
- `docs/v2.1-launch-copy.md` (feature copy, social posts)
- `docs/pr-credits.md` (contributor credits)
- Git log (`git log --oneline` for commit references)
Structure:
```markdown
# Changelog
## [2.1.0] - 2026-02-15
### Highlights
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
Three headline features...
### Added
- Open-class skill with watchlists (SQLite-backed, FTS5)
- YouTube as 4th research source via yt-dlp (search + transcript extraction)
- OpenAI Codex CLI compatibility ($last30days invocation)
- Bundled X search (vendored Bird GraphQL client, no external CLI)
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar)
- Briefing and history modes (open variant)
- --diagnose flag for source status checking
- --store flag for SQLite accumulation
### Changed
- Smarter query construction (strips noise words, auto-retry)
- Two-phase search (Phase 2 entity-aware drill-down)
- Reddit JSON enrichment (real upvotes/comments from reddit.com/.json)
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
- Model auto-selection with 7-day cache
### Fixed
- YouTube timeout increased to 90s
- Reddit 429 rate limit fail-fast
- YouTube soft date filter (keeps evergreen content)
- Eager import crash in __init__.py (Codex compatibility)
### New Contributors
- @JosephOIbrahim - Windows Unicode fix
- @levineam - Model fallback for unverified orgs
- @jonthebeef - --days=N configurable lookback flag
### Credits
- @steipete - Bird CLI (vendored X search) and yt-dlp inspiration
- @galligan - Marketplace plugin inspiration
- @hutchins - Pushed for YouTube feature
## [1.0.0] - 2026-01-15
Initial public release. Reddit + X search via OpenAI and xAI APIs.
```
#### 2. Create annotated tag on the public repo
```bash
cd /Users/mvanhorn/last30days-skill-private
# Tag on the current HEAD (which matches upstream/main)
git tag -a v2.1.0 -m "Release v2.1.0: Watchlists, YouTube transcripts, Codex CLI, bundled X search"
# Push to public repo
git push upstream v2.1.0
```
Use annotated tag (not lightweight) because:
- Stores tagger metadata and date
- Works with `git describe`
- Propagates to forks (317 forks)
- Supports GPG signing if desired later
#### 3. Craft release notes
Assemble from existing copy. The release body should follow this structure:
```markdown
## Highlights
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across **Reddit, X, YouTube, and the web**
from the last 30 days, finds what the community is actually upvoting, sharing,
and saying on camera, and writes you a prompt that works today.
**Three headline features in v2.1:**
1. **Open-class skill with watchlists** - Track competitors, people, topics on a schedule.
Pair with Open Claw for automated briefings. SQLite-backed with FTS5 search.
2. **YouTube transcripts as a 4th source** - When yt-dlp is installed, searches YouTube,
grabs view counts, and reads the actual transcripts. A 20-minute review has 10x the
signal of one X post.
3. **Codex CLI compatibility** - Same skill, same engine, same four sources.
Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
Plus: **Bundled X search** - Vendored Bird GraphQL client. No external CLI needed.
Just Node.js 22+ and browser cookies.
## Real Results (verified 2/15)
| Topic | Reddit | X | YouTube | Web |
|-------|--------|---|---------|-----|
| Nano Banana Pro | - | 32 posts, 164 likes | 5 videos, 98K views | - |
| Seedance 2.0 | 21 threads | 33 posts | 20 videos | 5 pages |
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views | - |
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts | 18 videos, 6.15M views | - |
## What's New
### Added
- Open-class skill with watchlist, briefing, and history modes
- YouTube search + transcript extraction via yt-dlp
- OpenAI Codex CLI compatibility
- Bundled Twitter/X search (vendored Bird GraphQL)
- Native web search (Parallel AI, Brave, OpenRouter)
- `--diagnose` and `--store` flags
- Conversational first-run experience (NUX)
### Changed
- Two-phase search architecture (entity-aware drill-down)
- Reddit JSON enrichment for real engagement metrics
- Smarter query construction with auto-retry
- Engagement-weighted scoring algorithm
### Fixed
- YouTube/Reddit timeout resilience
- Reddit 429 rate limit fail-fast
- Eager import crash in Codex environments
## New Contributors
- @JosephOIbrahim - Windows Unicode fix
- @levineam - Model fallback for unverified orgs
- @jonthebeef - `--days=N` configurable lookback
## Credits
- @steipete - Bird CLI and yt-dlp/summarize inspiration
- @galligan - Marketplace plugin inspiration
- @hutchins - Pushed for YouTube feature
## Install
```bash
# Claude Code
claude install-skill https://github.com/mvanhorn/last30days-skill
# Manual
git clone https://github.com/mvanhorn/last30days-skill ~/.claude/skills/last30days
```
**Full Changelog**: https://github.com/mvanhorn/last30days-skill/commits/v2.1.0
```
#### 4. Create the GitHub Release
```bash
gh release create v2.1.0 \
--repo mvanhorn/last30days-skill \
--verify-tag \
--title "v2.1.0 - Watchlists, YouTube Transcripts, Codex CLI" \
-F release-notes.md
```
No binary assets needed - this is an interpreted skill, and GitHub auto-generates source archives.
### Phase 2: Future Automation (Optional)
#### 5. Add `.github/release.yml` for auto-generated notes on future releases
```yaml
changelog:
exclude:
labels:
- ignore-for-release
categories:
- title: "Breaking Changes"
labels: [breaking-change]
- title: "Features"
labels: [enhancement, feature]
- title: "Bug Fixes"
labels: [bug, fix]
- title: "Documentation"
labels: [documentation]
- title: "Other Changes"
labels: ["*"]
```
This enables `gh release create v2.2.0 --generate-notes` for future releases with automatic PR categorization.
## Acceptance Criteria
- [x] `CHANGELOG.md` exists in repo root following Keep a Changelog format
- [x] Annotated tag `v2.1.0` exists on the public repo
- [x] GitHub Release `v2.1.0` is published at `github.com/mvanhorn/last30days-skill/releases`
- [x] Release notes include: highlights, real results table, feature list, contributor credits, install instructions
- [x] Release appears in the repo sidebar on GitHub
## Key Decisions
1. **v2.1.0 only** - Don't retroactively create v1.0.0 or v2.0.0 tags. The git history doesn't have clean boundary commits for those versions, and retroactive tags add complexity without value.
2. **Release on the public repo** (`upstream`), not the private repo. Users see `mvanhorn/last30days-skill`.
3. **CHANGELOG.md as source of truth** - The release notes are derived from CHANGELOG.md, not the other way around. Future releases update CHANGELOG.md first, then create the release.
4. **Update star count** - The existing copy says "1.5K stars" but the repo now has 2,747. Update in release notes.
## References
- Existing copy: `docs/v2.1-launch-copy.md`, `docs/v2.1-tweets.md`
- Contributors: `docs/pr-credits.md`
- README features: `README.md` "What's New" sections
- GitHub Releases docs: https://docs.github.com/en/repositories/releasing-projects-on-github
- Keep a Changelog: https://keepachangelog.com/en/1.1.0/
@@ -0,0 +1,274 @@
---
title: "feat: Last30Days.com - Automated Trending Topic Research"
type: feat
date: 2026-02-20
---
# Last30Days.com - Automated Trending Topic Research
## Overview
A website at Last30Days.com that automatically shows daily trending topics researched by the /last30days engine. The core challenge: the skill is query-driven (you give it a topic), but a trending page needs to *discover* what topics to research. This plan covers how to source trending topics, run them through the engine, and publish results.
## Problem Statement
The /last30days skill has 2,747 GitHub stars but no public-facing showcase. Users have to install the skill and run it themselves. A website that automatically shows trending topic results would:
1. **Drive installs** - people see the quality and want it for their own topics
2. **SEO surface area** - each topic page is indexable content
3. **Demonstrate capability** - "here's what /last30days found about X today" with real stats
4. **Content flywheel** - daily fresh content with zero manual curation
## Trending Topic Discovery: The Options
The skill has **no existing trending discovery mechanism** - it's entirely query-driven. Here are the available sources, ranked by practicality.
### Tier 1: Free, High Signal, Zero Friction
| Source | What It Returns | Auth | Cost | Best For |
|--------|----------------|------|------|----------|
| **Wikipedia Pageviews** | Top 100 most-viewed articles yesterday | None | Free | General public interest (news, culture, events) |
| **Hacker News** | Top 30 stories with scores | None | Free | Tech/startup topics |
| **Reddit r/all/hot + rising** | Hottest and rapidly rising posts | OAuth (free) | Free | Broad internet culture |
| **Google News RSS** | Top headlines, algorithmically curated | None | Free | Mainstream news |
| **YouTube mostPopular** | Top 50 trending videos by region | API key | Free (10K units/day) | Pop culture, entertainment |
### Tier 2: Very Cheap, High Value
| Source | What It Returns | Auth | Cost | Best For |
|--------|----------------|------|------|----------|
| **Perplexity Sonar API** | "What's trending today?" with sourced answers | API key | ~$1/month | Meta-aggregator that replaces multiple sources |
| **Google Trends (pytrends)** | Daily trending Google searches | None | Free but flaky | What people are actually searching |
| **Bird `getNews()`** | X Explore page trending topics | Browser cookies | Free | Real-time Twitter/X conversation |
### Tier 3: Paid, Skip for MVP
| Source | Cost | Why Skip |
|--------|------|----------|
| X/Twitter API | $200/month minimum | Too expensive; Bird `getNews()` is free |
| Exploding Topics | $249/month | Overkill for daily trends |
| TikTok | Gated, requires approval | Application process |
### Existing Codebase Hooks
Several pieces already exist in the codebase that could be leveraged:
1. **Bird `getNews()` is already on disk.** The vendored Bird library at `scripts/lib/vendor/bird-search/` includes `twitter-client-news.js` which fetches X's Explore page tabs (For You, Trending, News, Sports, Entertainment). Only needs a ~30-line `bird-news.mjs` wrapper to expose it. Zero API keys needed.
2. **`store.get_trending(days=7)`** at `scripts/store.py:559` ranks watchlist topics by recent finding activity. Could feed "what the skill has been researching" as a meta-signal.
3. **Scoring algorithm** at `scripts/lib/score.py` has engagement formulas for Reddit, X, and YouTube that could rank "buzz" by reweighting engagement >> relevance.
4. **Brave Trending API** exists but isn't implemented in `brave_search.py`. Could be added.
## Proposed Architecture
### Topic Discovery Pipeline (daily cron)
```
┌─────────────────────────────────────────────────┐
│ STEP 1: Fetch trending signals (parallel, free) │
├─────────────────────────────────────────────────┤
│ Wikipedia Pageviews ─┐ │
│ Hacker News Top 30 ─┤ │
│ Reddit r/all/hot ─┼─→ Raw topics + signals │
│ Google News RSS ─┤ │
│ YouTube mostPopular ─┤ │
│ Bird getNews() ─┘ │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ STEP 2: Cluster + rank │
├─────────────────────────────────────────────────┤
│ Extract topic keywords from titles │
│ Cluster by semantic similarity │
│ Rank by cross-source frequency │
│ ("Pope Leo XIV" on Wikipedia + Reddit + News │
│ = high confidence trend) │
│ Output: top 15-20 topics, ranked │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ STEP 3: Run /last30days on top topics │
├─────────────────────────────────────────────────┤
│ Top 3-5 topics: full research (--emit=json) │
│ → Full "What I learned" synthesis articles │
│ Remaining 10-15: quick research (--quick) │
│ → Stats teasers (32 X posts, 5 YouTube...) │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ STEP 4: Publish to website │
├─────────────────────────────────────────────────┤
│ Generate static HTML/JSON │
│ Deploy to Last30Days.com │
│ RSS feed for subscribers │
└─────────────────────────────────────────────────┘
```
### Topic Clustering Strategy
The hardest part is deduplicating across sources. "Pope Francis Dies" (Google News), "Pope_Francis" (Wikipedia #1 viewed), and a r/worldnews post are all the same topic. Approaches:
**Option A: LLM clustering (recommended for MVP)**
Feed all raw titles to an LLM and ask it to cluster into distinct topics with a representative label. ~$0.01 per day via a fast model. Simple, accurate, handles edge cases.
**Option B: TF-IDF + cosine similarity**
Extract keywords, compute pairwise similarity, agglomerative clustering. No API cost but worse on paraphrased titles.
**Option C: Embedding similarity**
Embed all titles, cluster by cosine distance. Better than TF-IDF, costs a few cents per day.
### Website Architecture
**Option A: Static site (recommended for MVP)**
- Daily cron generates JSON + static HTML
- Host on GitHub Pages, Cloudflare Pages, or Vercel
- Zero server cost, zero maintenance
- Framework: plain HTML/CSS, or minimal Astro/11ty
**Option B: Next.js with ISR**
- Incremental Static Regeneration rebuilds pages daily
- More flexibility for future features (search, filtering, user accounts)
- Hosting: Vercel free tier
**Option C: Full web app**
- Database-backed, real-time updates, user accounts
- Overkill for MVP
### Cost Estimate (daily operation)
| Item | Cost |
|------|------|
| Trending source APIs | $0 (all free tier) |
| LLM clustering (fast model) | ~$0.01/day |
| Full research on 3-5 topics | ~$0.10-0.50/day (OpenAI API for Reddit search) |
| Quick research on 10-15 topics | ~$0.05-0.20/day |
| Static hosting | $0 (GitHub/Cloudflare Pages) |
| **Total** | **~$5-20/month** |
## Implementation Phases
### Phase 1: Topic Discovery Script (MVP)
Build `scripts/discover_trending.py` that:
- Fetches Wikipedia Pageviews, HN top stories, Reddit hot, Google News RSS in parallel
- Filters out evergreen/non-topical Wikipedia pages (e.g., "Main Page", "ChatGPT" permanent traffic)
- Uses a fast LLM to cluster raw titles into 15-20 distinct topics
- Outputs ranked JSON: `[{topic, sources, confidence, category}]`
**Acceptance criteria:**
- [ ] Fetches from at least 4 free trending sources in parallel
- [ ] Clusters raw titles into deduplicated topics via LLM
- [ ] Filters Wikipedia evergreen pages (maintain a blocklist)
- [ ] Outputs ranked JSON with topic name, source count, category
- [ ] Runs in < 60 seconds
- [ ] No paid API keys required for discovery (only LLM clustering)
### Phase 2: Research Automation
Wire discovered topics into the existing /last30days engine:
- Top 3-5 topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --deep`
- Remaining topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --quick`
- Store all results in `~/.local/share/last30days/trending/YYYY-MM-DD/`
**Acceptance criteria:**
- [ ] Orchestration script runs discovery then research sequentially
- [ ] Full research on top N topics, quick on the rest
- [ ] Results stored as dated JSON files
- [ ] Total daily runtime < 30 minutes
- [ ] Handles timeouts/failures gracefully (skip topic, continue)
### Phase 3: Website Generation
Build a static site generator that reads the daily JSON and produces Last30Days.com:
- Homepage: today's trending topics grid (title, category, key stat, source badges)
- Topic pages: full synthesis for showcase topics, stats teaser for others
- Archive: previous days accessible by date
- RSS feed
- CTA: "Want to research your own topic? Install the skill"
**Acceptance criteria:**
- [ ] Static HTML generated from daily JSON
- [ ] Homepage shows today's 15-20 trending topics
- [ ] 3-5 showcase topic pages with full synthesis
- [ ] Remaining topics show stats teasers + install CTA
- [ ] Deploys to Last30Days.com (Cloudflare Pages or similar)
- [ ] RSS feed for daily updates
- [ ] Mobile responsive
### Phase 4: Bird Trending Integration (bonus)
Wire up the already-vendored Bird `getNews()` for X trending:
- Create `scripts/lib/vendor/bird-search/bird-news.mjs` (~30 lines)
- Add X Explore trending data as a 5th discovery source
- X trends are the fastest-moving signal and fill the "what's happening right now" gap
**Acceptance criteria:**
- [ ] `bird-news.mjs` wrapper exposes X Explore trending topics
- [ ] Integrated into discovery pipeline as an additional source
- [ ] Falls back gracefully if no X session cookies available
## Alternative Approaches Considered
### Perplexity-only approach
Just ask Perplexity Sonar "what are the top 20 trending topics today?" daily. Simpler, but:
- Single point of failure
- Less transparent (can't show "sourced from Reddit, Wikipedia, HN")
- Model may hallucinate or miss niche topics
- **Verdict:** Good fallback, not primary.
### Curated topics (manual)
Manually pick topics each day. Defeats the purpose of automation. Could supplement for "editorial picks."
### Social listening tools (Brandwatch, Sprout Social, etc.)
Expensive ($500+/month), enterprise-focused, overkill.
## Technical Considerations
- **Rate limits:** Wikipedia, HN, and RSS have no practical limits for 1 daily call. Reddit's 100 QPM is generous. YouTube's 10K units/day allows ~3,000 `mostPopular` calls.
- **Wikipedia filtering:** The top Wikipedia pages are always "Main Page", "Special:Search", etc. Need a blocklist of ~50 evergreen pages plus heuristics (skip pages under 1,000 characters, skip disambiguation pages).
- **Cron timing:** Run discovery at ~2am UTC (after Wikipedia pageviews finalize for previous day). Run research at ~3am UTC. Deploy site by ~5am UTC.
- **Cost control:** The /last30days engine uses OpenAI API for Reddit search. Full research on 5 topics * $0.05-0.10 each = ~$0.25-0.50/day. Quick research is cheaper.
- **Domain:** Last30Days.com needs to be registered (check availability).
## Dependencies & Risks
| Risk | Mitigation |
|------|------------|
| pytrends breaks (Google changes) | pytrends is a bonus source, not required. Other 4+ sources sufficient. |
| Wikipedia pageviews delayed | Fall back to Perplexity Sonar for day's topics |
| OpenAI API cost spikes | Cap at 5 full + 15 quick topics per day; use --quick for most |
| Bird cookie auth stops working | X trending is a bonus source; discovery works without it |
| Domain not available | Check availability; alternatives: last30.day, last30days.app |
## Success Metrics
- Daily automated publishing with zero manual intervention
- 15-20 trending topics surfaced daily
- 3-5 full synthesis articles per day
- Site loads in < 2 seconds (static)
- Drives measurable GitHub star growth / skill installs
## References
### Trending APIs (free)
- Wikipedia Pageviews: `wikimedia.org/api/rest_v1/metrics/pageviews/top/{project}/{access}/{year}/{month}/{day}`
- Hacker News: `hacker-news.firebaseio.com/v0/topstories.json`
- Reddit: `oauth.reddit.com/r/all/hot` (or `reddit.com/r/all/hot.json` unauthenticated)
- Google News RSS: `news.google.com/rss`
- YouTube: `googleapis.com/youtube/v3/videos?chart=mostPopular`
### Existing codebase hooks
- Bird `getNews()`: `scripts/lib/vendor/bird-search/vendor/package/dist/lib/twitter-client-news.js`
- Store trending: `scripts/store.py:559` (`get_trending()`)
- Scoring: `scripts/lib/score.py` (engagement formulas)
- Brave Trending API: not implemented, available in Brave docs
### Inspiration
- [Keep a Changelog](https://keepachangelog.com/) - clean daily update format
- [Hacker News front page](https://news.ycombinator.com/) - minimal trending UI
- [Exploding Topics](https://explodingtopics.com/) - trending topic showcase (paid, $249/mo)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,212 @@
---
title: "feat: 15-Test Side-by-Side Comparison - Make CROSS the GOAT"
type: feat
status: active
date: 2026-02-25
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
---
# 15-Test Side-by-Side Comparison - Make CROSS the GOAT
## Overview
Run all 5 canonical topics through all 3 skill versions (base, HN, CROSS) for 15 total full last30days runs. Save every result. Analyze side-by-side. Judge which version is best. Then create an improvement plan to make CROSS the definitive next release.
## Problem Statement / Motivation
The previous analysis only ran 5 tests on the CROSS branch - never comparing the same topics across all 3 versions. Without true side-by-side data, we can't judge whether CROSS is actually better or if the new features (dynamic YouTube scoring, cross-refs) introduce regressions. The user wants to see all 15 results, know which version wins, and get a concrete plan to make CROSS the GOAT before shipping.
## The 3 Versions
| Version | Git State | Sources | YouTube Relevance | Cross-Refs |
|---------|-----------|---------|-------------------|------------|
| **Base** | commit `427a4e4` | Reddit, X, YouTube, Web | Hardcoded 0.7 | No |
| **HN** | `main` / `f60a435` | Reddit, X, YouTube, **HN**, Web | Hardcoded 0.7 | No |
| **CROSS** | `feat/youtube-relevance-cross-source` / `0591f55` | Reddit, X, YouTube, HN, Web | Dynamic 0.1-1.0 | Yes (Jaccard 0.5) |
## The 5 Topics
| # | Topic | Category | Why chosen |
|---|-------|----------|------------|
| 1 | "Claude Code skills and MCP servers" | Developer tools | Core user base topic |
| 2 | "Seedance AI video generation" | Creative AI | Trending topic, cross-platform buzz |
| 3 | "M4 MacBook Pro review" | Consumer tech | Product review, mainstream |
| 4 | "best rap songs 2026" | Pop culture | Non-tech stress test |
| 5 | "React vs Svelte 2026" | Framework debate | Dev community, opinion-heavy |
## Test Matrix (15 Runs)
Run in **topic-sequential** order (all 3 versions of topic 1 back-to-back, then topic 2, etc.) to minimize temporal confounds on YouTube/X search results.
| Run | Topic | Version | Output File |
|-----|-------|---------|-------------|
| 1 | Claude Code | Base | `base-1-claude-code.json` |
| 2 | Claude Code | HN | `hn-1-claude-code.json` |
| 3 | Claude Code | CROSS | `cross-1-claude-code.json` |
| 4 | Seedance | Base | `base-2-seedance.json` |
| 5 | Seedance | HN | `hn-2-seedance.json` |
| 6 | Seedance | CROSS | `cross-2-seedance.json` |
| 7 | MacBook | Base | `base-3-macbook.json` |
| 8 | MacBook | HN | `hn-3-macbook.json` |
| 9 | MacBook | CROSS | `cross-3-macbook.json` |
| 10 | Rap songs | Base | `base-4-rap.json` |
| 11 | Rap songs | HN | `hn-4-rap.json` |
| 12 | Rap songs | CROSS | `cross-4-rap.json` |
| 13 | React/Svelte | Base | `base-5-react-svelte.json` |
| 14 | React/Svelte | HN | `hn-5-react-svelte.json` |
| 15 | React/Svelte | CROSS | `cross-5-react-svelte.json` |
**Output directory:** `/tmp/last30days-comparison/full/`
## Execution Protocol
### Pre-flight (once)
- [x] Clear model cache: `rm -f ~/.cache/last30days/model_selection.json`
- [x] Run `--diagnose` on current branch, save as `diagnose-baseline.json`
- [x] Verify all API keys active: Reddit (OPENAI_API_KEY), X (Bird cookies or XAI_API_KEY), YouTube (yt-dlp), HN (no key needed), Web (parallel AI or Brave)
- [x] Create output dir: `mkdir -p /tmp/last30days-comparison/full`
- [x] Stash any uncommitted changes: `git stash` (none needed - no uncommitted changes)
### Per-topic loop (repeat 5 times)
For each topic, run all 3 versions back-to-back:
```bash
# CRITICAL: Clean __pycache__ between EVERY git checkout to prevent stale bytecode
cleanup() {
find scripts -name '__pycache__' -exec rm -rf {} + 2>/dev/null
find scripts -name '*.pyc' -delete 2>/dev/null
}
```
- [ ] **Step 1: Base version**
- `cleanup && git checkout 427a4e4`
- `python3 scripts/last30days.py --diagnose 2>/dev/null` (verify sources match baseline)
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/base-{N}-{slug}.json 2>/tmp/last30days-comparison/full/base-{N}-{slug}.log`
- [ ] **Step 2: HN version**
- `cleanup && git checkout main`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/hn-{N}-{slug}.json 2>/tmp/last30days-comparison/full/hn-{N}-{slug}.log`
- [ ] **Step 3: CROSS version**
- `cleanup && git checkout feat/youtube-relevance-cross-source`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/cross-{N}-{slug}.json 2>/tmp/last30days-comparison/full/cross-{N}-{slug}.log`
### Post-run
- [x] Return to feature branch: `git checkout feat/youtube-relevance-cross-source`
- [x] Verify all 15 JSON files exist and are non-empty
- [x] Check for `*_error` fields in any JSON output - flag but include (zero errors)
## Analysis Dimensions
### 1. Source Coverage Table
For each of 15 runs, count items per source:
| Topic | Version | Reddit | X | YouTube | HN | Web | Total |
|-------|---------|--------|---|---------|----|----|-------|
Expected: Base has 0 HN items. HN and CROSS should have identical source counts (same search code). Any differences indicate API non-determinism.
### 2. YouTube Relevance Comparison
**Matched-item analysis:** Match videos by `video_id` across Base/CROSS runs of the same topic. For each matched video:
- Base relevance: always 0.7
- CROSS relevance: dynamic score
- Delta and direction (did dynamic scoring promote or demote this video?)
**Aggregate analysis:** Distribution stats (min/avg/max/stddev) per version per topic.
### 3. Cross-Source Links (CROSS only)
- Count of items with cross_refs per topic
- Quality assessment: are the linked items actually about the same story?
- Which source pairs link most often? (Reddit-HN? YouTube-HN? X-Reddit?)
### 4. Score Distribution and Rankings
- Mean/median score of top 10 items per version per topic
- Rank position changes: do the same items appear in different orders?
- Does HN crowd out Reddit/X items in the top 10?
### 5. "Best Version" Judging Criteria
Define BEFORE analyzing to avoid bias:
| Metric | Weight | How measured |
|--------|--------|-------------|
| Source diversity | 25% | Shannon entropy across source types in top 15 items |
| Score quality | 25% | Mean score of top 10 items |
| Relevance accuracy | 25% | YouTube: do high-relevance videos actually match the query? Manual spot-check of top 3 + bottom 3 per topic |
| Bonus features | 25% | HN value-add (unique info not in other sources) + cross-ref utility (do xrefs add value to the reader?) |
### 6. HN Value-Add Assessment
For each topic where HN returns results:
- How many HN items appear in the overall top 10?
- Do HN items provide information not available from Reddit/X/YouTube?
- Are HN comment insights (top_comments) genuinely useful?
## Deliverables
### A. Raw Results Archive
All 15 JSON files plus logs saved in `/tmp/last30days-comparison/full/`, also copied to `docs/comparison-results/` for persistence.
### B. Comparison Summary Table
A single markdown table showing all 15 runs with key metrics per cell.
### C. Version Verdict
Clear judgment: which version is best overall, and best per topic category (tech, consumer, culture).
### D. CROSS Improvement Plan
Concrete changes to make CROSS the GOAT:
**Based on the previous analysis (see origin doc), likely improvements include:**
1. **Fix cross-source linking** - Switch from char-trigram Jaccard (0.5) to hybrid similarity (max of trigram + token Jaccard) at threshold 0.40. Previous modeling showed this goes from 1 link to 24 links across 5 tests.
2. **Cross-ref rendering** - Change from cryptic `[xref: X1, HN3]` IDs to human-readable `[also on: Reddit, HN]` labels.
3. **HN search broadening** - Investigate why React/Svelte returns 0 HN items.
4. **YouTube relevance** - Already working well, may need minor threshold tuning based on 15-test data.
5. **Score normalization** - Ensure HN items don't systematically crowd out other sources in the merged ranking.
**New improvements to discover from 15-test data:**
- Regressions introduced by CROSS changes
- Edge cases where Base or HN outperforms CROSS
- Topic-specific tuning opportunities
## Acceptance Criteria
- [x] All 15 JSON result files saved and non-empty
- [x] All 15 runs use identical source routing (verified via --diagnose)
- [x] __pycache__ cleaned between every git checkout
- [x] Comparison summary table covers all 15 runs
- [x] YouTube matched-item analysis for at least 3 topics (all 5 done)
- [x] Cross-source link quality spot-check for CROSS runs
- [x] Clear version verdict with supporting data
- [x] CROSS improvement plan with specific code changes
- [x] All results appended to the analysis document for the user to review
## Technical Considerations
- **__pycache__ poisoning**: Python caches bytecode. Switching git commits without clearing `__pycache__` means old code runs even after checkout. MUST clean between every checkout.
- **API rate limiting**: 15 runs hitting Reddit, X, YouTube, HN APIs. The `--quick` mode and natural ~30-90s per run provide spacing, but monitor for 429s in logs.
- **YouTube non-determinism**: yt-dlp search results shift over time. Topic-sequential ordering minimizes this by running all 3 versions of the same topic within ~3-5 minutes.
- **X search session**: Bird CLI depends on browser cookies. If session expires mid-run, X results degrade silently. Check X item counts across runs.
- **`--quick` limitation**: Results reflect Phase 1 only (8-12 items per source). Full pipeline with `--deep` might show different patterns. Document this caveat.
## Sources & References
- Previous analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md)
- Implementation plan: [docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md](docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md)
- Existing comparison harness: [scripts/test-v1-vs-v2.sh](scripts/test-v1-vs-v2.sh)
- Scoring weights: [scripts/lib/score.py](scripts/lib/score.py) - 45% relevance + 25% recency + 30% engagement
@@ -0,0 +1,201 @@
---
title: "chore: open PR triage - build or close decision for all 5 open PRs"
type: chore
status: active
date: 2026-03-02
---
# Open PR Triage
Decision record for each of the 5 open PRs on `mvanhorn/last30days-skill`.
For each: build/merge, cherry-pick, or close with explanation.
---
## PR #26 - Add HN, YouTube, and Product Hunt sources + --search flag
**Author:** wkbaran | **Size:** +3402/-49 | **Date:** 2026-02-15
### What it adds
- Hacker News source (Algolia API, no auth)
- YouTube source (YouTube Data API v3, optional key)
- Product Hunt source (GraphQL API, optional key)
- `--search=SOURCES` flag (e.g. `--search=reddit,hn,yt`)
- 58 new tests
### What's already on main
- **HN:** ✅ already merged (`scripts/lib/hackernews.py`)
- **YouTube:** ✅ already merged (`scripts/lib/youtube_yt.py`)
- **Product Hunt:** ❌ NOT on main
- **`--search` flag:** ❌ NOT on main
### Decision: CHERRY-PICK (`--search` flag only, skip Product Hunt)
The HN and YouTube modules in this PR are earlier versions than what's on
main. We should NOT take those. Product Hunt is not a priority source.
The one missing piece worth landing:
**`--search=SOURCES` flag** - lets callers (including agent mode) specify
which sources to run. Useful for `--search=hn,polymarket` or
`--search=reddit,x` focus runs.
### Plan
- Cherry-pick just `tests/test_search_flag.py` and the `--search` argument
wiring in `scripts/last30days.py`
- Skip `scripts/lib/producthunt.py` and all PH-related code entirely
- Do NOT take their `hackernews.py`, `youtube.py`, `render.py`, `normalize.py`,
`schema.py` - our versions are newer
- Update SKILL.md to document `--search` flag
- Comment on PR thanking wkbaran - his HN/YouTube work was an inspiration
for the sources we ended up building. Acknowledge that specifically.
### Acceptance Criteria
- [ ] `python3 scripts/last30days.py "AI coding tools" --search=hn,reddit` runs only HN + Reddit
- [ ] `python3 scripts/last30days.py "test" --search=x,web` skips Reddit
- [ ] `test_search_flag.py` passes (adapt to our test patterns, no PH references)
- [ ] SKILL.md updated with `--search` flag docs
- [ ] Sync via `bash scripts/sync.sh`
---
## PR #24 - Adding Codex compatibility
**Author:** el-analista | **Size:** +228/-29 | **Date:** 2026-02-11
### What it adds
- `agents/openai.yaml` Codex discovery metadata
- Portable script path resolution in SKILL.md (multi-install-path loop)
- Platform-neutral UI text (`assistant` instead of Claude-specific wording)
- `LAST30DAYS_CACHE_DIR` env override in `scripts/lib/cache.py`
- `LAST30DAYS_OUTPUT_DIR` env override in `scripts/lib/render.py`
- X/Bird query noise stripping + last-chance retry in `scripts/lib/bird_x.py`
- New tests: `test_bird_x.py`, `test_cache.py`, `test_render.py`
### What's already on main
- **Codex auth:** ✅ already merged (via PR #38)
- **Multi-path script resolution:** ✅ already in SKILL.md (the `for dir in` loop)
- **Platform-neutral wording:** ❌ NOT systematically applied
- **Cache/output dir overrides:** ❌ NOT on main
- **Bird X improvements:** ✅ partially (we have other bird_x fixes but may be missing noise stripping + retry)
### Decision: REVIEW AND PARTIAL CHERRY-PICK
The env var overrides (`LAST30DAYS_CACHE_DIR`, `LAST30DAYS_OUTPUT_DIR`) are
genuinely useful for sandboxed/containerized Codex environments. The bird_x
noise stripping and last-chance retry may improve X search quality and are
low-risk additions.
The platform-neutral wording change ("assistant" instead of Claude references)
should be skipped — SKILL.md is Claude-specific by design.
### Plan
- Read the diff carefully against our current `cache.py`, `render.py`, `bird_x.py`
- Cherry-pick the `LAST30DAYS_CACHE_DIR` and `LAST30DAYS_OUTPUT_DIR` env overrides
- Cherry-pick the bird_x noise stripping + retry logic if it doesn't conflict
with our existing bird_x changes
- Skip `agents/openai.yaml` — we have our own multi-path resolution
- Skip platform-neutral wording changes
- Comment on PR thanking contributor and explaining what landed
### Acceptance Criteria
- [ ] `LAST30DAYS_CACHE_DIR=/tmp/test python3 scripts/last30days.py "test" --mock` writes cache to /tmp/test
- [ ] `LAST30DAYS_OUTPUT_DIR=/tmp/out python3 scripts/last30days.py "test" --mock` writes output to /tmp/out
- [ ] Existing tests pass after cherry-pick
- [ ] `test_cache.py` and `test_render.py` adapted and passing
---
## PR #14 - Simplify to WebSearch-first, make API keys optional
**Author:** thangman1 | **Size:** +97/-168 | **Date:** 2026-02-01
### What it does
Removes the Reddit/X Python search engine as the primary data source.
Repositions Claude Code's built-in WebSearch as the "default" mode.
Strips engagement metrics (upvotes, likes, repost counts) from output.
Removes mode detection logic (Full Mode / Partial Mode / Web-Only Mode).
### Decision: CLOSE - do not merge
This PR inverts the core value proposition of last30days. The skill's
differentiation is **real engagement data** from Reddit threads and X posts —
upvotes, likes, reposts — that WebSearch cannot provide. Stripping that out
produces a worse tool than just asking Claude to search the web, which anyone
can already do.
The author's intent (lower barrier to entry, no API key required) is valid,
but the right solution is making HN + Polymarket work without any API key
(they already do), and making OPENAI_API_KEY easier to obtain — not removing
the Reddit/X engine.
### Action
- Comment on PR explaining why we're closing it
- Acknowledge the valid friction point (API key setup) and point to HN +
Polymarket as the zero-config sources
- Close PR
---
## PR #10 - OpenRouter API integration
**Author:** thetechreviewer | **Size:** +1029/-16 | **Date:** 2026-01-28
### What it adds
OpenRouter as an alternative to OpenAI for the Reddit discovery search.
Allows using any model available on OpenRouter instead of just OpenAI models.
### What's already on main
- **`scripts/lib/openrouter_search.py`:** ✅ ALREADY ON MAIN
- **Wired into `scripts/last30days.py`:** ✅ ALREADY ON MAIN (the `backend == "openrouter"` path)
### Decision: CLOSE - already merged
The OpenRouter integration that this PR introduced is already on main. It
arrived via internal work that post-dated this PR. The PR is stale.
### Action
- Comment on PR: "Thanks for this — OpenRouter support is already on main
(landed via internal work). Closing as incorporated."
- Close PR
---
## PR #5 - Add support for Codex auth with OpenAI Responses API
**Author:** jblwilliams | **Size:** +358/-66 | **Date:** 2026-01-27
### What it adds
- JWT-based Codex auth with `chatgpt_account_id`
- Codex endpoint routing (`https://chatgpt.com/backend-api/codex/responses`)
- SSE handling for streaming Codex responses
- Typed auth status/source dataclass
- Codex fallback model chain
### What's already on main
**All of this is already on main.** The Codex auth system landed via PR #37
(iliaal:codex-auth-merged), which in turn came in with PR #38. This PR (#5)
predates that work and covers the same ground.
### Decision: CLOSE - already incorporated
### Action
- Comment on PR: "Thanks for this early work on Codex auth! The same feature
landed on main via PR #37 (from a separate contributor who built on similar
ideas). The JWT decode, Codex endpoint routing, SSE parsing, and typed auth
dataclass are all live. Closing as incorporated."
- Close PR
---
## Summary Table
| PR | Author | Decision | Reason |
|----|--------|----------|--------|
| #26 | wkbaran | **Cherry-pick** | `--search` flag not on main; HN/YouTube already there; skip Product Hunt |
| #24 | el-analista | **Cherry-pick** | Cache dir overrides + bird_x improvements worth landing; Codex auth already there |
| #14 | thangman1 | **Close** | Removes engagement data, inverts core value proposition |
| #10 | thetechreviewer | **Close** | OpenRouter already on main |
| #5 | jblwilliams | **Close** | Codex auth already on main via PR #37/38 |
## Implementation Order (if proceeding)
1. Close PR #5, #10, #14 with comments (no code changes needed)
2. Cherry-pick PR #24 pieces (bird_x + cache/render env overrides)
3. Cherry-pick PR #26 pieces (Product Hunt + `--search` flag)
4. Sync and test
5. Push to upstream
@@ -0,0 +1,49 @@
---
title: Close Commented GitHub Issues and PRs
type: fix
status: active
date: 2026-03-03
origin: docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md
---
# Close Commented GitHub Issues and PRs
All 9 open issues and 2 open PRs on `mvanhorn/last30days-skill` were commented on (2026-03-03) with fix confirmations or responses, but none were actually closed on GitHub.
## Acceptance Criteria
- [ ] Close 6 issues where fixes were confirmed (#40, #39, #32, #30, #29, #4)
- [ ] Close PR #26 (superseded - features landed on main)
- [ ] Verify 3 issues remain open (#36, #31, #22) - acknowledged but unresolved
- [ ] Verify PR #24 remains open - explicitly left open per comment
## Actions
### Close as fixed (6 issues)
```bash
gh issue close 40 --repo mvanhorn/last30days-skill --reason completed
gh issue close 39 --repo mvanhorn/last30days-skill --reason completed
gh issue close 32 --repo mvanhorn/last30days-skill --reason completed
gh issue close 30 --repo mvanhorn/last30days-skill --reason completed
gh issue close 29 --repo mvanhorn/last30days-skill --reason completed
gh issue close 4 --repo mvanhorn/last30days-skill --reason completed
```
### Close superseded PR (#26)
```bash
gh pr close 26 --repo mvanhorn/last30days-skill
```
### Keep open (no action needed)
- #36 - SKILL.md flag forwarding (investigating)
- #31 - skills.sh audit scores (needs review)
- #22 - Bird feature requests (backlog)
- PR #24 - Codex compatibility (intentionally left open)
## Sources
- **Origin plan:** [docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md](2026-03-03-fix-triage-all-open-github-issues-plan.md)
- Commit: `5e5d586` - fix: triage all 16 open GitHub issues
@@ -0,0 +1,152 @@
---
title: "feat: Auto-save research results to ~/Documents/Last30Days/"
type: feat
status: completed
date: 2026-03-05
---
# feat: Auto-save research results to ~/Documents/Last30Days/
## Overview
Every time the last30days skill completes a research run, automatically save the full briefing - inquiry, synthesis, stats, and follow-up suggestions - as a topic-named `.md` file in `~/Documents/Last30Days/`. Inspired by how users like @devin_explores are already manually saving results to build a personal research library (see screenshot - 17 topic files in a `Last30Days` Finder folder, each 9-34 KB).
## Problem Statement / Motivation
The skill's most valuable output - the assistant's synthesized "What I learned" briefing with stats and citations - only exists in the conversation. Once the session ends, it's gone. Users like @devin_explores work around this by manually copying output into .md files. Meanwhile, the Python script already writes raw data to `~/.local/share/last30days/out/`, but:
1. It overwrites on every run (no history)
2. It only contains pre-synthesis data (scored items), not the assistant's expert briefing
3. It's in a hidden dot-directory users don't naturally browse
The feature makes saving automatic and puts files where users expect them - the Documents folder, visible in Finder/file explorer.
## Proposed Solution
Add a **Write tool step in SKILL.md** after the synthesis/stats/invitation block that saves the complete briefing to `~/Documents/Last30Days/{topic-slug}.md`. This is a SKILL.md-only change (no Python script modifications needed) because the content to save is the assistant's synthesized output, which only exists in the SKILL.md flow.
### Why SKILL.md, not the Python script
The Python script (`last30days.py`) runs first and produces raw scored items. The assistant then synthesizes these into the "What I learned" briefing, stats block, and invitation. The synthesis is the valuable part - it's what @devin_explores is saving. The script can't produce this because it runs before synthesis happens.
### File naming
Convert the TOPIC variable to a kebab-case slug for the filename:
- "Claude Code best practices" -> `claude-code-best-practices.md`
- "best rap songs 2026" -> `best-rap-songs-2026.md`
- "nano banana 2 prompting guide" -> `nano-banana-2-prompting-guide.md`
This matches the screenshot pattern exactly (e.g., `anthropic-claude-code-best-practices.md`, `seedance-video-prompting-guide.md`).
If a file with the same slug already exists, append a date suffix: `claude-code-best-practices-2026-03-05.md`. This handles re-researching the same topic without overwriting previous results.
### File content
The saved .md file should contain the complete research output in this order:
```markdown
# {TOPIC}
> Researched {date} | Query type: {QUERY_TYPE} | Target tool: {TARGET_TOOL or "general"}
## What I learned
{The full synthesis section - topics, patterns, citations}
## Stats
{The full stats box with source counts and engagement}
## Follow-up suggestions
{The 2-3 specific suggestions from the invitation block}
---
*Generated by [last30days](https://github.com/mvanhorn/last30days-skill) v2.9*
```
### Implementation location in SKILL.md
Insert a new section between the current "LAST - Invitation" display and the "WAIT FOR USER'S RESPONSE" section. The Write tool call happens silently - no user prompt, no opt-in. Just save and briefly confirm.
## Technical Considerations
- **Cross-platform paths**: `~/Documents/` exists on macOS and most Linux desktops. On systems where it doesn't exist, `mkdir -p` handles creation. Windows WSL users get it too.
- **Permissions**: The Write tool in Claude Code can write to `~/Documents/` without issues. No sandbox concerns since this is the user's own Documents folder.
- **Filename sanitization**: Strip special characters, collapse whitespace to hyphens, lowercase. Keep it simple - no need for a library, just basic string ops in the SKILL.md instructions.
- **File size**: Based on the screenshot (9-34 KB files), the synthesis output is well within reasonable bounds.
- **No opt-out flag needed initially**: This is the default behavior. If users complain, a `--no-save` flag can be added later. Start with always-on since the screenshot proves users want this.
## Acceptance Criteria
- [x] Running `/last30days {topic}` creates `~/Documents/Last30Days/{topic-slug}.md` automatically
- [x] File contains: title, date, query metadata, full synthesis, stats block, follow-up suggestions
- [x] Filename is kebab-case slug of the topic (e.g., `claude-code-skills-guide.md`)
- [x] Duplicate topics get a date suffix instead of overwriting
- [x] Directory `~/Documents/Last30Days/` is created automatically if it doesn't exist
- [x] A brief confirmation line appears after the stats (e.g., "Saved to ~/Documents/Last30Days/claude-code-skills-guide.md")
- [x] Agent mode (`--agent`) also saves the file
- [x] No changes to the Python script - this is purely a SKILL.md addition
## Implementation Steps
### Step 1: Add save instructions to SKILL.md
Insert a new section after the invitation block (after line ~496, before "WAIT FOR USER'S RESPONSE" at line ~499):
**New section in `SKILL.md`:**
```markdown
## Save Research to Documents
After displaying the invitation, save the complete research briefing:
1. Generate the filename from TOPIC:
- Lowercase the topic
- Replace spaces and special characters with hyphens
- Remove consecutive hyphens
- Trim to 60 characters max
- Example: "Claude Code Best Practices" -> "claude-code-best-practices"
2. Check if file already exists. If so, append today's date:
- "claude-code-best-practices.md" exists -> use "claude-code-best-practices-2026-03-05.md"
3. Use the Write tool to save to ~/Documents/Last30Days/{slug}.md with this content:
- H1 title: the TOPIC
- Metadata line: date, QUERY_TYPE, TARGET_TOOL
- Full "What I learned" synthesis (everything you just displayed)
- Full stats block
- Follow-up suggestions from the invitation
- Footer with skill attribution
4. Confirm briefly: "Saved to ~/Documents/Last30Days/{slug}.md"
```
### Step 2: Update agent mode section
The `--agent` mode section (line ~116) skips interactive elements but should still save. Add a note that agent mode saves the file with the same logic.
### Step 3: Update Security & Permissions section
Add to the "What this skill does" list (line ~599):
- "Saves research briefings as .md files to ~/Documents/Last30Days/"
## Success Metrics
- Users accumulate a browsable library of .md research files in their Documents folder
- No more manual copy-paste workflow to save results
- Files are immediately findable in Finder/file explorer search
## Dependencies & Risks
- **Low risk**: Write tool is already in the skill's `allowed-tools` list
- **Low risk**: ~/Documents/ is a standard, user-owned directory
- **Edge case**: If the skill is interrupted mid-run (before synthesis), no file is saved - this is correct behavior since there's nothing to save yet
- **Edge case**: Very long topics could produce unwieldy filenames - the 60-char truncation handles this
## Sources & References
- Screenshot from @devin_explores showing manual .md file library in ~/Documents/Last30Days/
- Current output pipeline: `scripts/lib/render.py:798` (`write_outputs()`) writes to `~/.local/share/last30days/out/`
- SKILL.md synthesis flow: lines 275-496 (internalize research -> show summary -> invitation)
- Existing `--emit` modes: `scripts/last30days.py:1700` (`output_result()`)
@@ -0,0 +1,83 @@
---
title: "fix: Eliminate save-related output noise after research"
type: fix
status: active
date: 2026-03-06
---
# fix: Eliminate save-related output noise after research
## Problem Statement
After research completes, the auto-save feature adds 3-5 lines of unwanted output below the clean invitation block. Every approach tried so far has made things worse:
| Version | Approach | Lines added | Side effects |
|---------|----------|-------------|--------------|
| v2.9.1 | `Write` tool | ~5 | Shows "Wrote 43 lines to..." |
| v2.9.2 | `run_in_background: true` | ~5 | Background callback triggers 2-4 min cogitation, model hallucinates fake "Human:" messages and generates unsolicited multi-paragraph responses |
| v2.9.3 | Foreground `cat >` heredoc | ~3 | Shows `(No output)`, still triggers 2-4 min cogitation |
**Current v2.9.3 noise (3 lines):**
```
⏺ Bash(mkdir -p ~/Documents/Last30Days && cat > ...)
⎿ (No output)
✻ Churned for 2m 38s
```
The root cause is that ANY tool call after the invitation creates unavoidable Claude Code UI chrome, and the model cogitates for minutes regardless of foreground vs background.
## Proposed Solutions
### Option A: Remove auto-save entirely (Recommended)
Remove the entire "Save Research to Documents" section from SKILL.md. The research lives in the conversation. Zero extra tool calls, zero extra lines, zero cogitation.
**Changes to `SKILL.md`:**
- Delete the "Save Research to Documents" section (~45 lines)
- Update agent mode reference (line 126) - remove save mention
- Update security section (line 612) - change "Saves research briefings" to past tense or conditional
- Remove `📎` footer line from invitation format
- Simplify "WAIT FOR USER'S RESPONSE" section
**What users lose:** Auto-saved .md files in `~/Documents/Last30Days/`
**What users gain:** Clean output ending exactly at the invitation block
### Option B: Move save into Python script
Add `--save-dir` flag to `last30days.py`. The script saves its raw output during its existing Bash call (which already runs). Zero extra tool calls.
**Changes:**
- `scripts/last30days.py` - add `--save-dir` argument, write raw output to file at end of execution
- `SKILL.md` - add `--save-dir=~/Documents/Last30Days` to the script invocation, remove save section
**What users lose:** Saved file contains raw research data, not Claude's synthesis
**What users gain:** Zero extra lines, data is still preserved
### Option C: Opt-in save via follow-up command
Remove auto-save. Add a note to the invitation: "Say 'save' to save this research." When user says "save", run the Bash heredoc then.
**Changes to `SKILL.md`:**
- Delete auto-save section
- Add "save" as a recognized intent in "WHEN USER RESPONDS" section
- Save logic only runs when explicitly requested
**What users lose:** Nothing - save is still available on demand
**What users gain:** Clean output by default, save when they want it
## Acceptance Criteria
- [ ] Research output ends at the invitation block with zero tool calls after it
- [ ] No `(No output)` line visible after invitation
- [ ] No multi-minute cogitation after invitation
- [ ] No hallucinated fake user messages
- [ ] Version bumped to v2.9.4
- [ ] CHANGELOG updated
- [ ] Synced to ~/.claude, ~/.agents, ~/.codex via sync.sh
## Context
- Source: `/Users/mvanhorn/last30days-skill-private/SKILL.md`
- Deployed to: `~/.claude/skills/last30days/SKILL.md`
- Sync command: `bash scripts/sync.sh`
- Current version: v2.9.3
@@ -0,0 +1,375 @@
---
title: "feat: Paperclip Marketing Automation for last30days"
type: feat
status: active
date: 2026-03-07
---
# Paperclip Marketing Automation for last30days
## Overview
Set up a Paperclip "company" that auto-runs marketing for the last30days open-source skill (3,800+ stars). Four agent roles handle daily demo showcases, release announcements, community engagement, and analytics - all using a draft-then-approve workflow through Paperclip's built-in approval gates.
The killer angle: **last30days markets itself by running itself.** The Content Creator agent runs `/last30days [trending topic] --agent` on hot topics daily, then drafts X threads showing the results. Every post is a live demo.
## Problem Statement / Motivation
- All marketing is currently manual - ~60KB of pre-drafted X threads sit in `docs/` unposted
- No automated community monitoring (GitHub issues, contributor shoutouts)
- No metrics tracking (star growth, fork trends, social engagement)
- Solo entrepreneur can't sustain daily content + community management + development
- The tool's best ad is itself running on interesting topics, but that requires daily effort
## Proposed Solution
A Paperclip company called **"last30days Marketing"** with 4 agent roles, all draft-then-approve.
### Company Structure
```
last30days Marketing (Company)
Mission: "Grow last30days to 10K GitHub stars through daily demo content,
release marketing, and community engagement"
Marketing Director (Claude Code agent)
- Sets daily topic priorities
- Reviews draft quality before surfacing to human
- Coordinates cross-agent work
Content Creator (Python script agent)
- Runs last30days on trending topics daily
- Drafts X showcase threads from the results
- Heartbeat: daily at 8 AM PT
Release Manager (Bash + Python agent)
- Watches for new git tags on upstream
- Drafts release announcement threads
- Heartbeat: every 6 hours (tag check is cheap)
Community Manager (Python script agent)
- Monitors GitHub issues/PRs via gh CLI
- Drafts welcome messages for new contributors
- Surfaces popular feature requests
- Heartbeat: every 4 hours
Analytics Analyst (Python script agent)
- Tracks GitHub stars, forks, traffic
- Tracks X engagement (@slashlast30days)
- Generates weekly digest
- Heartbeat: daily at 11 PM PT (collect), weekly Monday 9 AM (digest)
```
### Draft-Then-Approve Flow
```
Agent creates draft
-> Saved to ~/Documents/Last30Days/drafts/{agent}/{date}-{slug}.md
-> Paperclip approval gate triggers
-> Matt reviews in Paperclip UI (approve / reject / edit)
-> On approve: Python script posts to X API via tweepy
-> Audit log records everything
```
## Technical Approach
### Phase 1: Infrastructure (Day 1-2)
Set up Paperclip and external API access.
**Files to create:**
- `marketing/paperclip-config.yaml` - Company definition, org chart, budgets
- `marketing/scripts/post_to_x.py` - X API v2 posting via tweepy (draft queue -> X)
- `marketing/scripts/github_monitor.py` - GitHub event monitoring via gh CLI
- `marketing/scripts/metrics_collector.py` - Star/fork/engagement tracking
- `marketing/.env.example` - Required API keys template
**Setup steps:**
1. Install Paperclip locally:
```bash
git clone https://github.com/paperclipai/paperclip
cd paperclip && pnpm install && pnpm dev
```
2. Get X API v2 credentials (developer.x.com) for @slashlast30days
- Need: API key, API secret, Access token, Access token secret
- Permissions: Read + Write (posting)
3. GitHub token for monitoring (gh auth already configured)
4. Create the company in Paperclip UI at localhost:3100:
- Company name: "last30days Marketing"
- Mission: "Grow last30days to 10K GitHub stars"
- Monthly budget cap: $50 (mostly API token costs)
### Phase 2: Content Creator Pipeline (Day 3-5)
The core marketing engine. Uses last30days to market itself.
**How it works:**
1. **Topic Discovery** - `marketing/scripts/discover_topics.py`
- Scrapes trending topics from: Wikipedia pageviews API, HN front page, Reddit r/all
- Filters for topics that would make compelling demos (tech, culture, sports, geopolitics)
- Outputs ranked topic list to `~/Documents/Last30Days/topics-queue.json`
2. **Research & Draft** - `marketing/scripts/create_showcase.py`
- Picks top topic from queue
- Runs: `python3 scripts/last30days.py "{topic}" --agent --emit=compact --save-dir=~/Documents/Last30Days`
- Reads the saved research output
- Drafts a 1-2 tweet thread in the established style (see Style Guide below)
- Saves draft to `~/Documents/Last30Days/drafts/content/{date}-{slug}.md`
3. **Approval Gate** - Paperclip surfaces draft for review
- Matt approves/edits in Paperclip UI
- On approve: triggers `post_to_x.py` with the draft content
**Style Guide (extracted from existing launch threads):**
```
Format: Stats-first hook + key finding + tool credit
Example (from v2.5 launch):
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views).
5 HN stories. 9 Polymarket markets.
[Key finding in 2-3 sentences]
Polymarket: [relevant odds with specific numbers]
[One-liner showing the tool's value]
github.com/mvanhorn/last30days-skill
```
Rules:
- Always lead with the /last30days command that was run
- Always include the stats line (thread/post/video counts)
- Always end with the GitHub link
- Pick topics people care about RIGHT NOW
- Never use em dashes - use hyphens instead
- Keep threads to 1-2 tweets max for daily showcases
- Save longer threads (3-6 tweets) for releases
### Phase 3: Release Manager Pipeline (Day 5-6)
**How it works:**
1. **Tag Watcher** - `marketing/scripts/watch_releases.py`
- Runs `git -C /Users/mvanhorn/last30days-skill-private fetch upstream --tags` every 6 hours
- Compares local tags vs upstream tags
- On new tag: reads CHANGELOG.md diff since last tag
2. **Thread Drafter** - `marketing/scripts/draft_release_thread.py`
- Reads changelog diff + release-notes.md
- Drafts a 3-6 tweet thread following the v2.5 launch thread style
- Includes: version number, headline features, demo queries, contributor shoutouts
- Saves to `~/Documents/Last30Days/drafts/releases/{tag}.md`
3. **Approval Gate** - Same flow as content pipeline
**Template (from existing docs/v2.5-launch-tweets.md):**
```
Tweet 1: Announcement + 3 headline features + GitHub link
Tweet 2-4: One demo per tweet (command + stats + finding)
Tweet 5: Contributor shoutouts
Tweet 6: Install instructions
```
### Phase 4: Community Manager Pipeline (Day 6-7)
**How it works:**
1. **GitHub Monitor** - `marketing/scripts/github_monitor.py`
- Runs `gh issue list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt,labels`
- Runs `gh pr list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt`
- Compares against `~/Documents/Last30Days/community/seen.json` to detect new items
- Categorizes: bug report, feature request, question, PR
2. **Response Drafter** - `marketing/scripts/draft_community_response.py`
- For new issues: drafts a welcome + triage response
- For new PRs: drafts a thank-you + initial review comment
- For merged PRs: drafts a contributor shoutout tweet
- Saves to `~/Documents/Last30Days/drafts/community/{type}-{number}.md`
3. **Approval Gate** - GitHub responses go through same Paperclip approval
- Approved responses posted via `gh issue comment` or `gh pr comment`
- Shoutout tweets posted via `post_to_x.py`
**Response templates:**
```markdown
# New Issue (bug)
Thanks for the report! I'll look into this. Can you share:
- Your OS and Python version
- The exact command you ran
- Whether you have SCRAPECREATORS_API_KEY set
# New Issue (feature request)
Interesting idea! [1-2 sentences acknowledging the value].
Adding this to the backlog for consideration.
# New PR
Thanks for the contribution, @{author}! I'll review this shortly.
[If first-time contributor: Welcome to the project!]
# Merged PR (tweet)
Shoutout to @{author} for [what they did] in last30days v{version}!
[Brief description of the change and why it matters]
github.com/mvanhorn/last30days-skill/pull/{number}
```
### Phase 5: Analytics Pipeline (Day 7-8)
**How it works:**
1. **Metrics Collector** - `marketing/scripts/metrics_collector.py`
- Daily: GitHub stars, forks, open issues, open PRs (via `gh api`)
- Daily: X followers, tweet impressions for @slashlast30days (via X API v2)
- Stores in SQLite at `~/Documents/Last30Days/analytics.db`
2. **Weekly Digest** - `marketing/scripts/weekly_digest.py`
- Runs Monday 9 AM PT
- Generates markdown report with week-over-week changes:
- Star growth (absolute + rate)
- New forks
- Issues opened/closed
- PRs merged
- Top-performing tweets
- Notable community interactions
- Saves to `~/Documents/Last30Days/digests/{date}-weekly.md`
- Optionally sends via email (SendGrid) or Slack webhook
**Schema for analytics.db:**
```sql
CREATE TABLE daily_metrics (
date TEXT PRIMARY KEY,
github_stars INTEGER,
github_forks INTEGER,
github_open_issues INTEGER,
github_open_prs INTEGER,
x_followers INTEGER,
x_impressions INTEGER,
x_engagement_rate REAL
);
CREATE TABLE tweet_performance (
tweet_id TEXT PRIMARY KEY,
posted_at TEXT,
type TEXT, -- 'showcase', 'release', 'shoutout'
topic TEXT,
impressions INTEGER,
likes INTEGER,
retweets INTEGER,
replies INTEGER,
link_clicks INTEGER
);
```
## Technical Considerations
### API Costs
| Service | Usage | Est. Monthly Cost |
|---------|-------|-------------------|
| Paperclip | Self-hosted | $0 |
| X API v2 | Free tier (posting) | $0 |
| ScrapeCreators | ~30 daily research runs | ~$15 |
| GitHub API | gh CLI, already authed | $0 |
| Claude API | Marketing Director agent | ~$20 |
| **Total** | | **~$35/month** |
### Security
- API keys stored in `marketing/.env` (gitignored, never committed)
- X API tokens scoped to @slashlast30days only (not personal account)
- Paperclip budget cap prevents runaway spend
- All posts go through human approval gate - no autonomous posting
- GitHub token uses existing `gh auth` session
### Failure Modes
- **last30days script fails** - Content Creator skips that day, logs error, tries again tomorrow with new topic
- **X API rate limit** - Queue drafts and retry on next heartbeat cycle
- **Paperclip goes down** - Drafts accumulate in filesystem, nothing posts (safe failure)
- **Bad topic selection** - Marketing Director agent filters topics before research (no politics, no NSFW)
- **Stale approval queue** - If drafts pile up >3 days unapproved, send a nudge notification
### Topic Selection Criteria
The discover_topics.py script should filter for topics that:
1. Are trending NOW (Wikipedia pageview spike, HN front page, Reddit r/all)
2. Would produce interesting multi-source results (not too niche, not too broad)
3. Are safe for a developer tool brand (tech, sports, culture, science - avoid divisive politics)
4. Haven't been covered in the last 7 days (dedup against previous showcases)
5. Span different categories to show the tool's versatility (not all tech, not all sports)
Good examples (from existing launch threads): "Anthropic Pete Hegseth", "Seedance prompting", "Arizona basketball", "Iran war"
## Acceptance Criteria
- [ ] Paperclip running locally with "last30days Marketing" company created
- [ ] Content Creator agent runs last30days daily on a trending topic and produces a draft
- [ ] Release Manager agent detects new git tags and drafts announcement threads
- [ ] Community Manager agent detects new GitHub issues/PRs and drafts responses
- [ ] Analytics agent collects daily metrics and generates weekly digest
- [ ] All drafts go through Paperclip approval gate before posting
- [ ] X API integration posts approved drafts to @slashlast30days
- [ ] GitHub responses posted via gh CLI after approval
- [ ] Monthly budget stays under $50
- [ ] Style of generated tweets matches existing launch thread tone
## Success Metrics
- **Content velocity**: 5-7 showcase tweets/week (up from ~0 currently)
- **Star growth**: Track week-over-week acceleration after content starts
- **Time savings**: <5 min/day reviewing drafts vs 30-60 min/day manual marketing
- **Content quality**: Drafts require minimal editing before approval (>80% approved as-is within 2 weeks)
## Dependencies & Risks
| Dependency | Risk | Mitigation |
|-----------|------|------------|
| Paperclip stability | Early-stage project, may have bugs | Pin to specific commit, keep simple config |
| X API free tier | May get rate-limited or deprecated | Queue-based posting, daily limits |
| Topic discovery quality | Bad topics = bad demos | Human review in approval gate, topic blocklist |
| Claude API for Marketing Director | Cost could spike | Budget cap in Paperclip, simple prompts |
| ScrapeCreators API | PAYG costs scale with usage | Cap at 1 research run/day for showcases |
## File Structure
```
last30days-skill-private/
marketing/
README.md # Setup instructions
.env.example # Required API keys
paperclip-config.yaml # Company definition
scripts/
discover_topics.py # Trending topic discovery
create_showcase.py # Research + draft showcase tweet
draft_release_thread.py # Release announcement drafter
github_monitor.py # GitHub issue/PR monitor
draft_community_response.py # Community response drafter
post_to_x.py # X API posting (after approval)
metrics_collector.py # Daily metrics collection
weekly_digest.py # Weekly analytics digest
templates/
showcase.md # Tweet template for daily demos
release.md # Thread template for releases
community_responses.md # Response templates by type
```
## Sources & References
- Paperclip GitHub: github.com/paperclipai/paperclip
- Paperclip docs: paperclip.ing
- Existing launch threads: `docs/v2.5-launch-tweets.md`, `docs/v2.1-tweets.md`
- Existing launch copy: `docs/v2.1-launch-copy.md`
- Planned Last30Days.com: `docs/plans/2026-02-20-feat-last30days-com-trending-topics-plan.md`
- last30days `--agent` mode: SKILL.md line 115-142 (non-interactive output for automation)
@@ -0,0 +1,265 @@
---
title: Triage Open Issues and PRs
type: refactor
status: active
date: 2026-03-07
---
# Triage Open Issues and PRs - last30days-skill
Consolidated review and action plan for all 6 open items (2 issues, 4 PRs) as of 2026-03-07.
---
## 1. PR #52 - Fix missing metadata files in skill upload bundle
**Author:** 04cb | **Size:** +0/-1 | **Fixes:** #46
**File changed:** `.clawhubignore` (removes `*.json` glob)
### What it does
The `*.json` pattern in `.clawhubignore` was excluding `.claude-plugin/marketplace.json` and `.claude-plugin/plugin.json` - the metadata files required for ClawHub skill upload. This caused the "Zip file contains path with invalid characters" error reported in #46.
### Risk analysis
- Removing `*.json` also includes test fixtures (`fixtures/*.json`) and vendored `package.json` in the bundle. None are sensitive or harmful.
- No `package-lock.json`, `skills-lock.json`, or credential files exist as `.json` in the repo.
- One-line change with zero production code impact.
### Recommendation: MERGE IMMEDIATELY
- [x] Merge PR #52
- [x] Close issue #46 (auto-closes via "Fixes #46")
- [ ] Verify upload works post-merge
---
## 2. Issue #46 - "Claude will not upload as it says thread is invalid"
**Author:** johnuppard | **Error:** "Zip file contains path with invalid characters"
### Recommendation: CLOSES WITH PR #52
No standalone action needed. PR #52 is the fix. After merge, comment on #46 confirming resolution and ask reporter to verify.
---
## 3. PR #50 - test: add tests for entity_extract module
**Author:** mark-c4r | **Size:** +167/-0 (tests only)
### What it does
Adds `tests/test_entity_extract.py` with 23 test cases covering all 4 public/private functions in `scripts/lib/entity_extract.py`:
| Function | Cases | Coverage |
|----------|-------|----------|
| `_extract_x_handles` | 8 | author handles, @mentions, generic filtering, case normalization, frequency ranking, edge cases |
| `_extract_x_hashtags` | 5 | basic extraction, multiple tags, frequency ranking, short tag filtering, empty input |
| `_extract_subreddits` | 6 | field extraction, cross-refs in comments, frequency ranking, r/ stripping |
| `extract_entities` | 4 | integration, max limits, empty inputs, return key validation |
### Code quality assessment
- All function signatures match the actual module implementation exactly
- All 23 tests validated against the real module - they pass
- Import pattern matches existing tests (`sys.path.insert` + `from lib import`)
- Uses `unittest.TestCase` consistent with all other test files
- No external dependencies, no API calls, no mocking needed (pure functions)
- Test data uses inline dicts matching real Reddit/X response structure
### Recommendation: MERGE AFTER LOCAL VERIFICATION
- [x] Pull branch and run `python3 -m pytest tests/test_entity_extract.py -v` (23/23 passed)
- [x] Run full suite `python3 -m pytest tests/` (317/320 passed - 3 pre-existing failures in test_models.py, unrelated)
- [x] Merge
This is a clean, well-structured community contribution that adds coverage to a previously untested module.
---
## 4. PR #48 - feat: add Xiaohongshu source + Reddit public fallback
**Author:** YJLi-new | **Size:** +523/-75 | **Files:** 5 modified
### What it does
Two features bundled in one PR:
**A. Xiaohongshu (Little Red Book) source:**
- New `scripts/lib/xiaohongshu_api.py` module for searching via xiaohongshu-mcp HTTP API
- Source alias: `--search xiaohongshu` or `--search xhs`
- Health/login availability checks in `env.py`
- Handles Chinese numeric suffixes (wan/yi) in engagement parsing
- Default API base: `http://host.docker.internal:18060` (Docker-hosted)
**B. Reddit public JSON fallback:**
- `search_reddit_public()` added to `openai_reddit.py`
- Uses `reddit.com/search/.json` endpoint (no API key required)
- Multiple query strategy (topic, core subject, quoted core)
- Engagement-based relevance heuristic (60% score, 40% comments)
- Supplemental retries correctly gated - only fire when OpenAI auth is present
### Architecture compliance
| Pattern | Status |
|---------|--------|
| Three-function module pattern (search/parse/enrich) | Follows existing patterns |
| `DEPTH_CONFIG` with quick/default/deep | Present |
| `_log()` gated on stderr.isatty() | Present |
| env.py availability check | Present with retry logic |
| ThreadPoolExecutor dispatch | Correctly wired with +1 worker |
| Render/UI integration | Consistent with existing sources |
### Issues found
1. **Health check duplication** - `is_xiaohongshu_available()` in env.py AND `search_feeds()` both probe login status. Redundant but harmless.
2. **`from_date`/`to_date` accepted but unused** for Xiaohongshu - relies on API-side time bucketing. Acceptable given API limitation.
3. **Over-defensive `setdefault()`** in normalization (id and source_domain already set by search_feeds). Harmless.
4. **Error message format inconsistency** between OpenAI and public Reddit paths. Minor.
5. **`get_available_sources()` docstring** not updated to reflect Reddit always being available now.
6. **No tests included** for new Xiaohongshu module or Reddit public fallback.
### Security
- No hardcoded credentials
- xsec_token comes from API response (not user input)
- Docker `host.docker.internal` default is safe for containerized environments
- All URLs use `https://` for public Xiaohongshu
### Recommendation: MERGE WITH CONDITIONS
The Reddit public fallback alone makes this worth merging - it makes Reddit work with zero API keys. Xiaohongshu adds value for Chinese-market research.
**Before merge:**
- [x] Run full test suite to confirm no regressions (292/297 pass, 5 pre-existing)
- [x] Verify Xiaohongshu gracefully skips when API is unavailable (returns False, no crash)
- [x] Resolved merge conflicts with ScrapeCreators Reddit (main). Priority: ScrapeCreators -> OpenAI -> public fallback
- [x] Merged to main and pushed
- [ ] Test Reddit public fallback locally: unset OPENAI_API_KEY, run `python3 scripts/last30days.py "test topic" --search reddit --emit=compact`
- [ ] Update `get_available_sources()` docstring to reflect Reddit always-available change
**After merge (follow-up):**
- [ ] Add `tests/test_xiaohongshu.py` with fixture-based tests
- [ ] Add `tests/test_reddit_public.py` for the fallback path
- [ ] Consider extracting health check duplication
---
## 5. PR #47 - feat: add Apify as unified API provider
**Author:** lapolazzati | **Size:** +1484/-73 | **Files:** 6 new + orchestrator changes
### What it does
Adds Apify as a single-token alternative (`APIFY_API_TOKEN`) covering Reddit, X, TikTok, and Instagram. Existing per-source keys take priority; Apify is a transparent fallback.
**New modules:**
- `apify_client.py` (163 lines) - shared HTTP client for Apify run-sync API
- `apify_reddit.py` (195 lines) - Reddit via `trudax/reddit-scraper` actor
- `apify_x.py` (221 lines) - X via `apidojo/tweet-scraper` actor
- `apify_tiktok.py` (284 lines) - TikTok via `clockworks/tiktok-scraper` actor
- `apify_instagram.py` (288 lines) - Instagram via `apify/instagram-reel-scraper` actor
**Source routing priority:**
```
Reddit: OpenAI -> Apify -> None
X: Bird -> xAI -> Apify -> None
TikTok: ScrapeCreators -> Apify -> None
Instagram: ScrapeCreators -> Apify -> None
```
### Critical issues
1. **Actor ID mismatch (BLOCKER):** Code uses different Apify actors than documented in README and plan.md.
| Source | In Code | In Docs |
|--------|---------|---------|
| Reddit | `trudax/reddit-scraper` | `automation-lab/reddit-scraper` |
| X | `apidojo/tweet-scraper` | `scraper_one/x-posts-search` |
| TikTok | `clockworks/tiktok-scraper` | `epctex/tiktok-search-scraper` |
| Instagram | `apify/instagram-reel-scraper` | matches |
Users following README instructions will hit wrong actors.
2. **No tests (BLOCKER):** 1484 lines of new code with zero test coverage. Each Apify module has its own date parsing, relevance scoring, and response normalization - all untested.
3. **Significant code duplication:** `apify_tiktok.py` and `apify_instagram.py` share nearly identical:
- `_tokenize()` (7 lines)
- `_compute_relevance()` (14 lines)
- `STOPWORDS` set
- `SYNONYMS` dict
- `_extract_core_subject()` (~20 lines)
4. **Version mismatch:** README says v2.9 but git history shows v2.9.4 on main. Version should be higher.
5. **plan.md included in commit** - implementation planning doc shouldn't be in the final merge.
### What's good
- Source routing logic in env.py is clean and backward-compatible
- Orchestrator dispatch uses consistent `_source` parameter pattern
- Existing source paths are completely untouched
- Timeout scaling is appropriate (90s quick, 150s default, 240s deep)
- Token handling follows security best practices (parameter passing, Bearer auth)
### Recommendation: REQUEST CHANGES
This PR is too large and has too many issues for a clean merge. Request the contributor to:
**Must fix before any merge:**
- [ ] Resolve actor ID mismatches (verify which actors actually work, update code OR docs)
- [ ] Add unit tests for all 5 new modules (at minimum: response normalization, date parsing)
- [ ] Fix version number
- [ ] Remove plan.md from commit
**Should fix:**
- [ ] Extract shared code from apify_tiktok/apify_instagram into `apify_common.py`
- [ ] Consider splitting into 2 PRs:
- PR A: `apify_client.py` + env.py routing (foundation)
- PR B: Individual source modules + tests (features)
**Status:** [x] Review posted requesting changes (2026-03-07)
**Comment template for PR:**
> Thanks for this contribution - the single-token approach is a great idea for simplifying setup. A few things need fixing before we can merge:
>
> 1. The Apify actor IDs in the code don't match the README/plan docs. Can you verify which actors are correct and align code + docs?
> 2. We need test coverage for the new modules - at minimum normalization tests with sample actor responses.
> 3. There's significant duplication between apify_tiktok.py and apify_instagram.py (_tokenize, _compute_relevance, STOPWORDS, SYNONYMS, _extract_core_subject). Could you extract shared code to an apify_common.py?
> 4. Version in README should be higher than v2.9.4 (current main).
> 5. Please remove plan.md from the commit.
---
## 6. Issue #45 - Add support for Gemini CLI
**Author:** alexferrari88 | **Request:** "Would it be possible to add support for Gemini CLI?"
### Assessment
This is a feature request to support Gemini CLI as a runtime alongside Claude Code and Codex. Key considerations:
- **Scope:** The skill is currently packaged for Claude Code (SKILL.md format) and Codex (~/.codex/skills/). Gemini CLI uses a different skill/extension format.
- **Effort:** Would require understanding Gemini CLI's plugin system, creating a compatible manifest, and potentially adapting the Python execution model.
- **Priority:** Low - Claude Code and Codex are the primary targets and where the user base is.
- **Community:** If the requester wants to contribute, they'd be best positioned to understand Gemini CLI's requirements.
### Recommendation: ACKNOWLEDGE AND BACKLOG
- [x] Respond with comment acknowledging and inviting contribution
- [x] Add a `help wanted` label
- [x] Keep open as a backlog item
---
## Priority Summary
| Priority | Item | Action | Risk |
|----------|------|--------|------|
| 1 | PR #52 (metadata fix) | ~~Merge now~~ DONE | None |
| 2 | Issue #46 (upload error) | ~~Auto-closes with PR #52~~ DONE | None |
| 3 | PR #50 (entity tests) | ~~Run tests, merge~~ DONE | None |
| 4 | PR #48 (Xiaohongshu + Reddit fallback) | ~~Test locally, merge with conditions~~ DONE | Low |
| 5 | Issue #45 (Gemini CLI) | ~~Acknowledge, backlog, label~~ DONE | None |
| 6 | PR #47 (Apify unified) | ~~Request changes~~ DONE (awaiting contributor) | Medium-High |
@@ -0,0 +1,197 @@
---
title: "feat: Publish last30days as Claude Code marketplace plugin"
type: feat
status: completed
date: 2026-03-08
---
# feat: Publish last30days as Claude Code Marketplace Plugin
## Overview
Publish the last30days skill as a Claude Code plugin so users can install it with:
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
This gives users auto-updates, versioned installs, and the standard plugin management UX instead of manual git clone.
## Current State
The repo already has most of what's needed:
| Component | Status | Notes |
|-----------|--------|-------|
| `SKILL.md` | Exists | Source of truth at repo root |
| `skills/last30days/SKILL.md` | Exists (symlink) | Points to root SKILL.md, added for Gemini CLI |
| `scripts/` | Exists | Python research engine |
| `gemini-extension.json` | Exists | Gemini CLI support |
| `.claude-plugin/marketplace.json` | **Missing** | Needed to make repo a marketplace |
| `.claude-plugin/plugin.json` | **Missing** | Needed (in plugin subdir) to define the plugin |
| `plugins/last30days/` | **Missing** | Plugin directory structure |
## How It Works (compound-engineering as reference)
The repo serves as BOTH a **marketplace** (distribution channel) and contains **plugins**:
```
repo-root/ # = marketplace
├── .claude-plugin/
│ └── marketplace.json # Makes repo discoverable as marketplace
└── plugins/
└── plugin-name/ # = individual plugin
├── .claude-plugin/
│ └── plugin.json # Plugin manifest
└── skills/
└── skill-name/
└── SKILL.md
```
## Implementation Plan
### 1. Create marketplace manifest
#### `.claude-plugin/marketplace.json`
```json
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "last30days-skill",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"owner": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"plugins": [
{
"name": "last30days",
"description": "Research any topic from the last 30 days. Become an expert and write copy-paste-ready prompts.",
"version": "2.9.5",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"source": "./plugins/last30days",
"category": "productivity",
"homepage": "https://github.com/mvanhorn/last30days-skill"
}
]
}
```
### 2. Create plugin directory
#### `plugins/last30days/.claude-plugin/plugin.json`
```json
{
"name": "last30days",
"version": "2.9.5",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"author": {
"name": "Matt Van Horn",
"email": "mvanhorn@gmail.com",
"url": "https://github.com/mvanhorn"
},
"homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": [
"research",
"reddit",
"twitter",
"youtube",
"tiktok",
"trends",
"prompts"
]
}
```
### 3. Symlink skill and scripts into plugin directory
```bash
mkdir -p plugins/last30days/.claude-plugin
mkdir -p plugins/last30days/skills/last30days
# Symlink SKILL.md (single source of truth)
ln -s ../../../../SKILL.md plugins/last30days/skills/last30days/SKILL.md
# Symlink scripts directory (the Python engine)
ln -s ../../scripts plugins/last30days/scripts
# Symlink fixtures if needed
ln -s ../../fixtures plugins/last30days/fixtures
```
### 4. Update README with plugin install instructions
Add before the manual install section:
```markdown
### Plugin Install (recommended)
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
Then configure your API keys:
```bash
mkdir -p ~/.config/last30days
# Add SCRAPECREATORS_API_KEY to your env
```
```
### 5. Version bump workflow
When releasing new versions, update version in ALL manifests:
- `SKILL.md` frontmatter `version:`
- `gemini-extension.json` `version`
- `plugins/last30days/.claude-plugin/plugin.json` `version`
- `.claude-plugin/marketplace.json` plugins[0].version
Consider: a `scripts/bump-version.sh` that updates all four in one command.
## Open Question: Symlinks in Plugin Cache
**Risk:** When Claude Code installs a plugin, it clones the repo into `~/.claude/plugins/cache/`. Git preserves symlinks on macOS/Linux, so `plugins/last30days/skills/last30days/SKILL.md -> ../../../../SKILL.md` should resolve correctly because the full repo is cloned.
**Mitigation if symlinks fail:** Replace symlinks with a small build step in sync.sh that copies SKILL.md into the plugin directory. Or just copy the file and accept the duplication in the plugin directory (since the plugin cache is a read-only clone anyway - drift only matters in the source repo, not the installed copy).
**Test:** After implementing, verify with `git clone` into a temp dir and check symlink resolution.
## Acceptance Criteria
- [ ] `.claude-plugin/marketplace.json` exists with correct schema
- [ ] `plugins/last30days/.claude-plugin/plugin.json` exists
- [ ] `plugins/last30days/skills/last30days/SKILL.md` resolves to root SKILL.md
- [ ] `plugins/last30days/scripts` resolves to root scripts/
- [ ] README has plugin install instructions
- [ ] `/plugin marketplace add mvanhorn/last30days-skill` works
- [ ] `/plugin install last30days@last30days-skill` works
- [ ] `/last30days test topic` works after plugin install
- [ ] Standalone install (git clone to ~/.claude/skills/) still works
- [ ] Gemini CLI install still works
## User Experience After Publishing
```
# One-time setup
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
# Use it
/last30days AI video tools for business
# Auto-updates happen at startup when version bumps
```
## Sources
- Claude Code plugin docs: https://code.claude.com/docs/en/plugins
- Plugin reference: https://code.claude.com/docs/en/plugins-reference
- Marketplace docs: https://code.claude.com/docs/en/plugin-marketplaces
- compound-engineering reference: https://github.com/EveryInc/compound-engineering-plugin
- Your installed plugins: superpowers (4.3.1), swift-lsp (1.0.0), compound-engineering (2.38.1)
@@ -0,0 +1,180 @@
---
title: "feat: Add Gemini CLI extension support"
type: feat
status: completed
date: 2026-03-08
origin: docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md
---
# feat: Add Gemini CLI Extension Support
## Overview
Add Gemini CLI compatibility to last30days by cherry-picking the good parts from PR #53 (@alexferrari88) and fixing the implementation problems ourselves. Close PR #53 with a thank-you comment crediting the contributor.
Based on the review at `docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md`, we accept the concept but fix the approach to match our "one SKILL.md for all platforms" convention established during Codex compatibility work.
## What We Take from PR #53
| Component | Source | Changes Needed |
|-----------|--------|----------------|
| `gemini-extension.json` | PR #53 | Fix settings format (array, not object), bump version to 2.9.5 |
| Path resolution (for loop) | PR #53 | Take as-is, add to both SKILL.md and variants/open/SKILL.md |
| README.md install section | PR #53 | Take as-is |
## What We Skip from PR #53
| Component | Reason |
|-----------|--------|
| `skills/last30days/SKILL.md` (693-line copy) | Duplicate maintenance nightmare - use symlink instead |
| "or" tool name scattering in SKILL.md | LLMs translate tool intent natively - clutters prompt |
| Gemini tool names in `allowed-tools` | Gemini ignores `allowed-tools`; risks breaking Claude Code |
## Implementation Steps
### 1. Create `gemini-extension.json` (new file)
Based on PR #53 but with correct settings format (array per Gemini CLI docs) and v2.9.5 version.
```json
{
"name": "last30days-skill",
"version": "2.9.5",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
"name": "Extension Directory",
"description": "Extension installation directory (auto-set by Gemini CLI)",
"envVar": "GEMINI_EXTENSION_DIR",
"sensitive": false
},
{
"name": "ScrapeCreators API Key",
"description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
"envVar": "SCRAPECREATORS_API_KEY",
"sensitive": true
},
{
"name": "OpenAI API Key",
"description": "OpenAI API Key - optional fallback for Reddit discovery",
"envVar": "OPENAI_API_KEY",
"sensitive": true
},
{
"name": "xAI API Key",
"description": "xAI API Key for X/Twitter search (optional)",
"envVar": "XAI_API_KEY",
"sensitive": true
},
{
"name": "OpenRouter API Key",
"description": "OpenRouter API Key (optional)",
"envVar": "OPENROUTER_API_KEY",
"sensitive": true
},
{
"name": "Parallel AI API Key",
"description": "Parallel AI API Key (optional)",
"envVar": "PARALLEL_API_KEY",
"sensitive": true
},
{
"name": "Brave Search API Key",
"description": "Brave Search API Key (optional)",
"envVar": "BRAVE_API_KEY",
"sensitive": true
},
{
"name": "Apify API Token",
"description": "Apify API Token (optional legacy)",
"envVar": "APIFY_API_TOKEN",
"sensitive": true
},
{
"name": "Twitter AUTH_TOKEN",
"description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
"envVar": "AUTH_TOKEN",
"sensitive": true
},
{
"name": "Twitter CT0",
"description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
"envVar": "CT0",
"sensitive": true
}
]
}
```
### 2. Create `skills/last30days/SKILL.md` as symlink
```bash
mkdir -p skills/last30days
ln -s ../../SKILL.md skills/last30days/SKILL.md
```
Gemini CLI discovers skills from `skills/<name>/SKILL.md`. Symlink ensures single source of truth - no drift, no duplicate maintenance. Git tracks symlinks natively on macOS/Linux (Gemini CLI's target platforms).
### 3. Add Gemini paths to SKILL.md for-loop (line ~172)
Add these 3 entries after `"${CLAUDE_PLUGIN_ROOT:-}"`:
```bash
"${GEMINI_EXTENSION_DIR:-}" \
"$HOME/.gemini/extensions/last30days-skill" \
"$HOME/.gemini/extensions/last30days" \
```
### 4. Add same Gemini paths to `variants/open/SKILL.md` for-loop
Same 3 entries, same position.
### 5. Update `README.md` install section
Add Gemini CLI install before existing Claude Code section:
```markdown
### Gemini CLI
\`\`\`bash
gemini extensions install https://github.com/mvanhorn/last30days-skill.git
\`\`\`
### Claude Code / Codex
```
### 6. Close PR #53 with credit
Post comment thanking @alexferrari88, explaining we incorporated their work with modifications, and close the PR. Credit them in the commit message.
## Acceptance Criteria
- [ ] `gemini-extension.json` exists with correct array-format settings
- [ ] `skills/last30days/SKILL.md` is a symlink to `../../SKILL.md`
- [ ] Gemini path entries in SKILL.md for-loop (3 new entries)
- [ ] Gemini path entries in variants/open/SKILL.md for-loop (3 new entries)
- [ ] README.md has Gemini CLI install instructions
- [ ] No changes to `allowed-tools` in any SKILL.md
- [ ] No "or" tool name alternatives in any SKILL.md body
- [ ] Claude Code still works (`/last30days` runs normally)
- [ ] PR #53 closed with credit to @alexferrari88
- [ ] Commit message credits @alexferrari88 as co-author
## Confidence Assessment
**High confidence.** All changes are additive:
- New file (gemini-extension.json) - zero conflict risk
- Symlink (skills/last30days/SKILL.md) - zero conflict risk
- 3 lines added to a for-loop - trivially safe, short-circuits on first match
- README addition - simple text
- No changes to SKILL.md content, allowed-tools, or Python scripts
- No changes that could break existing Claude Code behavior
The only untested aspect is whether Gemini CLI correctly follows the symlink for skill discovery. If symlinks cause issues, fallback is a 2-line SKILL.md stub that imports the root (but this is unlikely - Gemini CLI runs on macOS/Linux where symlinks are native).
## Sources
- PR #53: https://github.com/mvanhorn/last30days-skill/pull/53
- Issue #45: https://github.com/mvanhorn/last30days-skill/issues/45
- Gemini CLI extension reference (settings array format confirmed): https://geminicli.com/docs/extensions/reference/
- Gemini CLI skills docs: https://geminicli.com/docs/cli/creating-skills/
- Review plan: `docs/plans/2026-03-08-review-pr-53-gemini-cli-support-plan.md`
@@ -0,0 +1,238 @@
---
title: "feat: Perpetual Monitoring Mode"
type: feat
status: active
date: 2026-03-08
---
# Perpetual Monitoring Mode
## Overview
Add a "perpetual monitoring" mode to last30days that lets users track topics over time with cumulative intelligence, delta reporting, and effortless re-runs. Designed for Claude Code's session model - not server cron.
**User signal:** @JTDaly asked "Can your skill be used to do this perpetually?" in response to a Perplexity AI S&P 500 earnings dashboard that auto-refreshes quarterly.
## Problem Statement
last30days is one-shot: ask, get a briefing, done. But the most valuable research is longitudinal - how conversations evolve, when new voices enter, when engagement spikes. Users want "set it and watch" without re-prompting from scratch every time.
## The Scheduling Reality in Claude Code
| Mechanism | Persistence | Max Duration | Fires When |
|-----------|------------|-------------|------------|
| **CronCreate / `/loop`** | Session-only (RAM) | 3 days, auto-expires | REPL idle |
| **System cron/launchd** | Permanent | Forever | Always |
| **SQLite watchlist** | Permanent (disk) | Forever | On demand |
**Key insight:** Don't build a scheduler. Build a **stateful watchlist** that composes with Claude Code's existing `/loop` for in-session automation, and survives across sessions via SQLite for manual re-runs. The `/loop` skill already exists and does scheduling perfectly - just make last30days a good citizen of it.
## What Already Exists
| Component | File | Status |
|-----------|------|--------|
| Topic CRUD + schedule fields | `scripts/watchlist.py` | Working |
| SQLite persistence with URL dedup, sighting counts | `scripts/store.py` | Working |
| Daily/weekly briefing generation | `scripts/briefing.py` | Working |
| Budget tracking (daily cap) | `scripts/store.py` | Working |
| FTS5 full-text search | `scripts/store.py` | Working |
| WAL mode for concurrent access | `scripts/store.py` | Working |
| `delivery_channel` setting in DB | `scripts/store.py` | Schema only |
| SKILL.md watchlist commands | `SKILL.md` | Missing |
## Proposed Solution
### The Composable Pattern
Instead of building scheduling into last30days, make last30days composable with Claude Code's existing tools:
```
# One-shot: research a topic and store findings
/last30 "S&P 500 earnings" --watch
# See what's new since last run (delta briefing)
/last30 briefing
# Automate with /loop (Claude Code native, session-scoped, 3-day max)
/loop 4h /last30 briefing
# Next session? Watchlist persists. Just re-loop or run manually.
/last30 briefing
```
The user's watchlist lives in SQLite forever. The scheduling is ephemeral by design - you opt into it each session. This matches how people actually use Claude Code: sessions, not servers.
### Phase 1: Watchlist + Briefing via SKILL.md (MVP)
Wire up the existing `watchlist.py` and `briefing.py` infrastructure through the skill interface.
**Tasks:**
- [ ] **SKILL.md additions** - New command branches:
- `/last30 watch add "topic"` - Add topic to watchlist, run initial research with `--store`
- `/last30 watch list` - Show watched topics with last-run timestamps and finding counts
- `/last30 watch remove "topic"` - Remove from watchlist
- `/last30 briefing` - Generate delta briefing across all watched topics (what's new)
- `/last30 briefing --weekly` - Weekly digest with trend analysis
- `/last30 "topic" --watch` - One-shot research that also adds to watchlist
- [ ] **`watchlist.py` updates:**
- `run-all` auto-adds `--store` flag so results persist
- `run-all` returns structured JSON (exit code + summary) for `/loop` consumption
- `run-one` returns per-topic summary for composability
- Add `--quick` default for watched topics (save API cost on recurring runs)
- [ ] **`briefing.py` updates:**
- `generate` outputs compact format suitable for Claude synthesis (like main script's `--emit=compact`)
- Include "last run" timestamp per topic so user knows freshness
- Flag stale topics (not updated in 48+ hours)
**Acceptance criteria:**
- [ ] `/last30 watch add "AI earnings"` persists topic and runs initial research
- [ ] `/last30 briefing` shows accumulated findings with "new since last briefing" markers
- [ ] `/loop 4h /last30 briefing` works out of the box (composability)
- [ ] Watchlist survives across Claude Code sessions (SQLite)
- [ ] Budget cap enforced (default $5/day)
### Phase 2: Delta Intelligence
Make briefings show what *changed*, not just what exists.
**Tasks:**
- [ ] **`scripts/lib/delta.py`** (~150 lines):
- New findings since last briefing (URLs with `first_seen > last_briefing_time`)
- Engagement spikes (>2x increase in score since last sighting)
- New voices (new @handles appearing for first time in a topic)
- Gone quiet (topics with no new findings in 2+ runs)
- [ ] **`briefing.py` delta integration:**
- Section: "Breaking" - high engagement + first seen this run
- Section: "Trending" - engagement increasing across runs
- Section: "New voices" - handles not seen before
- Section: "Gone quiet" - previously active, now silent
- [ ] **Smart re-run logic in `watchlist.py`:**
- Skip topics that were updated < 4 hours ago (avoid redundant API calls)
- Prioritize topics with most engagement change potential
- `--force` flag to override skip logic
**Acceptance criteria:**
- [ ] Briefings clearly distinguish new vs. previously-seen findings
- [ ] Engagement spikes flagged with specific metric ("upvotes 2.3x since yesterday")
- [ ] Redundant API calls avoided via smart skip logic
### Phase 3: Delivery + `/loop` Integration
Make the monitoring truly hands-off during a session.
**Tasks:**
- [ ] **Slack delivery** (`scripts/lib/deliver.py` ~100 lines):
- Webhook URL config: `watchlist.py config delivery slack --webhook "https://..."`
- Format briefing as Slack Block Kit (topic sections, trend indicators)
- Auto-deliver after `run-all` if webhook configured
- [ ] **`/last30 monitor` convenience command:**
- Shorthand that does: `run-all` + `briefing` + starts `/loop` automatically
- Prints: "Monitoring 5 topics every 4h. Auto-expires in 3 days. Run `/last30 monitor` again next session."
- Uses CronCreate directly (no `/loop` dependency) for tighter control
- [ ] **Session resume hint:**
- On `/last30 briefing`, if watchlist has topics but no `/loop` active, suggest:
"You have 5 watched topics. Run `/loop 4h /last30 briefing` to auto-refresh, or `/last30 monitor` for hands-off mode."
**Acceptance criteria:**
- [ ] Slack webhook delivery works end-to-end
- [ ] `/last30 monitor` starts automated loop with one command
- [ ] Clear messaging about 3-day session limit and how to resume
## Architecture: Why This Is Better Than Server Cron
```
Traditional approach (rejected):
System Cron -> watchlist.py run-all -> SQLite -> ??? deliver somehow
Claude Code-native approach:
/last30 watch add "topic" --> SQLite (persists forever)
/loop 4h /last30 briefing --> CronCreate (session, 3-day max)
|
v
Claude reads briefing.py output
Claude synthesizes with LLM judgment
Claude delivers via Slack webhook
Claude answers follow-up questions in context
```
The Claude Code-native approach is better because:
1. **LLM synthesis on every run** - not just raw data, but judgment ("this is unusual because...")
2. **Conversational** - user can ask follow-ups ("tell me more about the META earnings spike")
3. **Zero infrastructure** - no plist, no crontab, no daemon management
4. **Portable** - works on any OS where Claude Code runs
5. **Composable** - `/loop` is a general-purpose tool, not custom scheduling code
## Alternative Approaches Considered
### 1. Build custom scheduler.py with system cron/launchd
**Rejected.** Doesn't work in Claude Code's model. Platform-specific. Requires root/sudo for some configs. Users of a Claude Code skill shouldn't need to manage system daemons.
### 2. OpenClaw's persistent cron service
**Not applicable.** OpenClaw has its own cron system with database-backed scheduling, but last30days is an open-source skill that should work without OpenClaw. Could be an optional integration later.
### 3. Long-running Python daemon
**Rejected.** Fragile, wastes resources, doesn't benefit from LLM synthesis on each run.
### 4. Web dashboard with its own backend
**Deferred to Phase 4.** Dramatically increases scope. The briefing-to-Slack pattern delivers 80% of the value with 10% of the effort.
## Technical Considerations
**No new services.** Pure Python scripts + SQLite + SKILL.md instructions. Zero infrastructure.
**Cost control.** Each watched topic costs ~$0.05-0.30/run at `--quick` depth. 10 topics x 6 runs/day = $3-18/day. Budget cap in store.py already enforces limits.
**Composability contract.** `briefing.py generate` must output clean, parseable text that Claude can synthesize. No interactive prompts, no side effects beyond SQLite writes.
**Backwards compatibility.** All new. Existing `/last30 topic` unchanged. `--watch` flag is opt-in.
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Users expect "perpetual" to mean forever | High | Clear messaging: "3-day auto-expire per session, watchlist persists, re-run next session" |
| `/loop` changes or breaks | Low | Composability means we don't depend on `/loop` internals - just CronCreate |
| Budget overrun with many topics | Low | Budget cap already implemented in store.py |
| SQLite grows large over months | Low | Add 90-day retention policy |
| Slack webhook stops working | Low | Log failures, don't block pipeline, alert in next briefing |
## Success Metrics
- User can go from zero to monitoring in one command: `/last30 "S&P 500 earnings" --watch`
- `/last30 briefing` surfaces genuinely new information with delta markers
- The system composes cleanly with `/loop` - no special integration needed
- Watchlist persists across sessions - user picks up where they left off
- Clear, honest UX about session limits vs. persistent state
## Scope Boundaries
**In scope:**
- Watchlist CRUD via SKILL.md
- Delta-aware briefings
- `/loop` composability (not custom scheduling)
- Slack webhook delivery
- `/last30 monitor` convenience command
**Out of scope:**
- System cron/launchd integration
- Web dashboard
- Email delivery (Slack webhook is simpler, covers most users)
- Multi-user / team features
- Custom NLP beyond what exists
## Implementation Estimate
- Phase 1 (Watchlist + Briefing): SKILL.md additions, minor `watchlist.py` and `briefing.py` updates
- Phase 2 (Delta Intelligence): New `delta.py` module, `briefing.py` integration
- Phase 3 (Delivery + Monitor): New `deliver.py`, SKILL.md `/last30 monitor` command
Each phase ships independently. Phase 1 alone answers @JTDaly's question.
@@ -0,0 +1,81 @@
---
title: "fix: Remove re-introduced Save Research to Documents section from SKILL.md"
type: fix
status: completed
date: 2026-03-08
---
# fix: Remove re-introduced Save Research to Documents section from SKILL.md
## Overview
PR merges on March 7 regressed SKILL.md by re-introducing the "Save Research to Documents" section that v2.9.4 (`6d5acb9`) intentionally removed. The save logic was moved into the Python script via `--save-dir` flag, but the merged branches (PR #48 Xiaohongshu, upstream merge) were forked before v2.9.4 and brought the old SKILL.md content back via merge resolution.
**Result:** The skill now saves research AFTER the "I'm now an expert" invitation via a separate Bash tool call, which causes extra cogitation time, "(No output)" noise, and sometimes hallucinated follow-up messages - the exact UX problems v2.9.4 fixed.
## Root Cause
1. Commit `6d5acb9` (v2.9.4, March 6) removed the ~45-line "Save Research to Documents" section and added `--save-dir=~/Documents/Last30Days` to the bash command
2. PR #48 (Xiaohongshu) branch contained commit `9950d01` with the OLD save-via-Bash version
3. Merging PR #48 on March 7 re-introduced the save section
4. Upstream merge (`28dff6e`) compounded it
## Three changes needed
All in `SKILL.md`:
### 1. Fix agent mode line (line 142)
**Current (broken):**
```
Agent mode still saves the research briefing to `~/Documents/Last30Days/` using the same logic as interactive mode (see "Save Research to Documents" section).
```
**Should be:**
```
Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir` (handled by the script, no extra tool calls).
```
### 2. Add `--save-dir` back to bash command (line 186)
**Current (broken):**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web
```
**Should be:**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact --no-native-web --save-dir=~/Documents/Last30Days
```
### 3. Delete the entire "Save Research to Documents" section (lines 517-563)
Remove from `## Save Research to Documents` through the `---` separator before `## WAIT FOR USER'S RESPONSE`. This is ~47 lines.
### 4. Update "STOP and wait" line (line 567)
**Current:**
```
**STOP and wait** for the user to respond.
```
**Should be (matching v2.9.4):**
```
**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`.
```
## Post-edit steps
1. Run `bash /Users/mvanhorn/last30days-skill-private/scripts/sync.sh` to deploy to `~/.claude/skills/`
2. Push to `origin/main` and `upstream/main`
3. Verify with `diff` that installed skill matches repo
## Acceptance Criteria
- [ ] "Save Research to Documents" section is gone from SKILL.md
- [ ] `--save-dir=~/Documents/Last30Days` is in the bash command
- [ ] Agent mode line references `--save-dir`, not the deleted section
- [ ] "STOP and wait" line includes the "do NOT call any tools" reinforcement
- [ ] `sync.sh` deployed successfully
- [ ] `diff` between repo and `~/.claude/skills/last30days/SKILL.md` shows no differences
- [ ] Pushed to origin and upstream
@@ -0,0 +1,157 @@
---
title: "review: PR #53 - Gemini CLI Support"
type: review
status: active
date: 2026-03-08
---
# Review: PR #53 - Gemini CLI Support (alexferrari88)
## Verdict: MODIFY - Accept concept, reject implementation approach
PR #53 by @alexferrari88 adds Gemini CLI extension support. The intent is good and closes issue #45, but the implementation has structural problems that would create the exact maintenance nightmare we just fixed in v2.9.5.
## PR Summary
| File | Changes | Assessment |
|------|---------|------------|
| `gemini-extension.json` | +67 new file | **Accept with fixes** |
| `skills/last30days/SKILL.md` | +693 new file (full copy) | **Reject** - duplicate SKILL.md |
| `SKILL.md` | +9/-6 (tool name scattering) | **Reject** - wrong approach |
| `variants/open/SKILL.md` | +4/-1 | **Partially accept** (path resolution yes, tool names no) |
| `README.md` | +6 install instructions | **Accept** |
## Critical Issues
### 1. Duplicated SKILL.md (BLOCKER)
The PR creates `skills/last30days/SKILL.md` as a **full 693-line copy** of the root `SKILL.md`. This is the exact problem we just spent hours debugging - PR merges on March 7 regressed v2.9.4's save-section removal because branches had stale copies. A second SKILL.md guarantees this happens again.
The Codex compatibility work (see `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md`) explicitly chose **one SKILL.md for all platforms** to avoid this. Gemini CLI should follow the same pattern.
**Fix:** Delete `skills/last30days/SKILL.md`. Gemini CLI discovers skills from the extension's `skills/` directory, but we can either:
- (a) Symlink: `skills/last30days/SKILL.md -> ../../SKILL.md`
- (b) Use the root SKILL.md directly and configure `contextFileName` in gemini-extension.json to point to it
- (c) Have `skills/last30days/SKILL.md` be a thin wrapper that says "See root SKILL.md" (least ideal)
### 2. Based on v2.9.1, not v2.9.5 (BLOCKER)
The PR's copy of SKILL.md is based on v2.9.1 and includes:
- The "Save Research to Documents" section (removed in v2.9.4, re-removed in v2.9.5)
- Old agent mode line referencing deleted section
- Missing `--save-dir=~/Documents/Last30Days` flag
- Old version number
**Fix:** Rebase on current main (v2.9.5).
### 3. "Or" tool name scattering (REJECT)
The PR adds `WebSearch or google_web_search(...)` and similar patterns throughout SKILL.md. This is the wrong approach because:
- **LLMs already translate intent to tools.** When Gemini reads "do a WebSearch for X", it knows to use `google_web_search`. When Claude reads it, it uses `WebSearch`. The model handles this mapping natively.
- **Clutters the prompt.** SKILL.md is a 640-line prompt. Adding "or alternative_name" to every tool reference makes it harder for the model to parse.
- **Maintenance burden.** Every new platform means adding more "or" alternatives.
**Fix:** Remove all "or" alternatives. Keep Claude Code tool names in the SKILL.md body (they work as intent descriptions). If Gemini needs explicit tool mapping, that belongs in `GEMINI.md` (Gemini CLI's context file), not scattered through the skill instructions.
### 4. `allowed-tools` pollution (RISKY)
Adding `run_shell_command, read_file, write_file, ask_user, google_web_search` to `allowed-tools`:
```
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch, run_shell_command, read_file, write_file, ask_user, google_web_search
```
**Risk:** If Claude Code's parser is strict and rejects unknown tool names, this breaks the skill for all Claude Code users. If it silently ignores unknown names, it's harmless but noisy.
**Finding:** Gemini CLI only recognizes `name` and `description` in SKILL.md frontmatter. It **ignores** `allowed-tools` entirely. So adding Gemini tool names to `allowed-tools` provides zero benefit to Gemini users while potentially breaking Claude Code users.
**Fix:** Remove Gemini tool names from `allowed-tools`. They serve no purpose on either platform.
## What to Accept
### 1. gemini-extension.json (with fixes)
The manifest file is the right approach. However:
- [ ] **Verify settings format.** The PR uses object-key format (`"SCRAPECREATORS_API_KEY": { ... }`). Gemini CLI docs show array format (`[{ "name": "...", ... }]`). Need to confirm which is correct for the current Gemini CLI version. The researcher found array format in the docs.
- [ ] **Update version** from `2.9.1` to `2.9.5`
- [ ] **Consider adding `contextFileName`** to point to root SKILL.md instead of duplicating
### 2. Path resolution additions
Adding these to the bash `for` loop is correct and low-risk:
```bash
"${GEMINI_EXTENSION_DIR:-}" \
"$HOME/.gemini/extensions/last30days-skill" \
"$HOME/.gemini/extensions/last30days" \
```
This should be in both `SKILL.md` and `variants/open/SKILL.md`.
### 3. README.md install section
Clean and appropriate. Adding Gemini CLI install command before Claude Code section.
## Proposed Changes to Request from Contributor
### Must-fix (before merge)
1. **Delete `skills/last30days/SKILL.md`** - no duplicate. Either symlink or use `contextFileName` in manifest.
2. **Rebase on main** (v2.9.5) - the PR is based on stale code.
3. **Remove all "or" tool name alternatives** from SKILL.md and variants/open/SKILL.md body text.
4. **Remove Gemini tool names from `allowed-tools`** in all SKILL.md files.
5. **Verify `gemini-extension.json` settings format** against current Gemini CLI docs (array vs object).
### Nice-to-have
6. **Add a `GEMINI.md` context file** (optional) - can include a short note like "When this skill references 'WebSearch', use `google_web_search`. When it references 'Bash', use `run_shell_command`." This is the clean way to handle tool name translation.
7. **Update sync.sh** to optionally deploy to `~/.gemini/extensions/last30days/` (debatable - Gemini users may prefer `gemini extensions install` instead).
8. **Add `.gemini/` to the path check in sync.sh** import verification.
## Testing Plan
Before merging, verify:
- [ ] `gemini extensions install` works from the repo (or `gemini extensions link .` for local dev)
- [ ] Skill activates in Gemini CLI and the model can find `scripts/last30days.py`
- [ ] `GEMINI_EXTENSION_DIR` env var resolves correctly in the bash for-loop
- [ ] Claude Code still works with no regressions (run `/last30days test topic --mock` or similar)
- [ ] `allowed-tools` with only Claude Code tool names doesn't break Gemini CLI skill loading
## Comment Template for PR
```
Thanks for the contribution! Gemini CLI support is great to have, and the `gemini-extension.json` manifest and path resolution additions are solid.
A few things need changing before we can merge:
**Must-fix:**
1. **Remove `skills/last30days/SKILL.md`** - We maintain one SKILL.md to avoid sync drift (we literally just fixed a regression from this exact problem yesterday). Either symlink it or use `contextFileName` in the manifest to point to the root SKILL.md.
2. **Rebase on `main`** - The PR is based on v2.9.1 but we're now at v2.9.5. The "Save Research to Documents" section in your copy was removed, `--save-dir` was added to the bash command, and the version was bumped.
3. **Remove "or" tool name alternatives** from SKILL.md body (e.g., `WebSearch or google_web_search`). LLMs handle tool name translation natively - Gemini knows to use `google_web_search` when the skill says "search the web". Scattering alternatives clutters the prompt.
4. **Remove Gemini tool names from `allowed-tools`** - Gemini CLI ignores `allowed-tools` (it only reads `name` and `description` from SKILL.md frontmatter), so these provide no benefit. And they risk breaking Claude Code if its parser rejects unknown tool names.
5. **Verify `gemini-extension.json` settings format** - The Gemini CLI docs I found show settings as an array (`[{ "name": "...", ... }]`), not object keys (`{ "KEY": { ... } }`). Can you confirm which format your Gemini CLI version expects?
**Optional but recommended:**
6. Consider adding a `GEMINI.md` context file with a short tool-name translation note (e.g., "When this skill says 'WebSearch', use `google_web_search`"). This is the clean way to bridge tool names.
Happy to help work through any of these! The core approach (manifest + path resolution) is right.
```
## Sources
- PR #53: https://github.com/mvanhorn/last30days-skill/pull/53
- Issue #45: https://github.com/mvanhorn/last30days-skill/issues/45
- Gemini CLI extension docs: https://geminicli.com/docs/extensions/writing-extensions/
- Gemini CLI extension reference: https://geminicli.com/docs/extensions/reference/
- Gemini CLI skills docs: https://geminicli.com/docs/cli/creating-skills/
- Codex compatibility plan: `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md`
- v2.9.5 regression fix: commit `8f7fb5a` (today)
@@ -0,0 +1,75 @@
---
title: "docs: v2.9.5 README update, plugin.json fix, Claude Code install above fold"
type: docs
status: completed
date: 2026-03-09
---
# docs: v2.9.5 README update, plugin.json fix, Claude Code install above fold
Three things in one commit:
## 1. Fix plugin.json hooks field (Issue #62)
**Bug:** `"hooks": ["./hooks/"]` should be `"hooks": {}`. Array format is invalid per Claude Code plugin manifest schema. No hooks directory exists anyway.
**File:** `.claude-plugin/plugin.json:25`
**Fix:** Change `"hooks": ["./hooks/"]` to `"hooks": {}`
## 2. Add Claude Code install above ClawHub
Currently the README opens with ClawHub badge + install. User wants Claude Code (the default/best way) to be equally prominent ABOVE ClawHub.
**Add before the ClawHub badge:**
```
### Claude Code (recommended)
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
This matches the Installation section further down but puts it above the fold where people actually see it.
## 3. Update README to v2.9.5
**Header:** Change `v2.9.1` to `v2.9.5`
**Source list:** Add "Bluesky" everywhere sources are listed (currently says "Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web")
**New in v2.9.5 block** (replace the v2.9.1 block):
Features shipped since v2.9.1:
- **ScrapeCreators X backend** (`4b7087e`) - X/Twitter search now uses ScrapeCreators as a backend option alongside Bird cookies
- **Bluesky/AT Protocol** (`9a1059e`, `adb5a67`) - New social source with full pipeline (search, parse, normalize, score, dedupe, render). Opt-in via `BSKY_HANDLE` + `BSKY_APP_PASSWORD` app password
- **Comparative mode** (`ecf90c0`) - "X vs Y" queries run 3 parallel research passes and output side-by-side comparison
- **Per-project .env config** (PR #59) - `.last30days.env` in project root for per-project API keys
- **SessionStart config check** (PR #58) - Validates config on session start
- **Expanded test coverage** (PRs #56, #57) - Unit tests for untested modules + smoke tests + edge cases
- **Claude Code marketplace plugin** (`627947f`) - Install via `/plugin install`
- **Gemini CLI extension** (`2f16ff1`) - `gemini extensions install`
**Env vars section:** Add `BSKY_HANDLE` and `BSKY_APP_PASSWORD` to the optional keys table
## Files to Change
| File | Change |
|------|--------|
| `README.md` | Version bump, Claude Code above fold, new features, Bluesky in source lists, env vars |
| `.claude-plugin/plugin.json` | Fix hooks field |
## Acceptance Criteria
- [x] `"hooks": {}` in plugin.json (fixes #62)
- [x] Claude Code plugin install appears ABOVE ClawHub badge
- [x] README says v2.9.5
- [x] "New in v2.9.5" block lists all features since v2.9.1
- [x] Bluesky appears in source list references
- [x] BSKY_HANDLE/BSKY_APP_PASSWORD documented in env vars
- [ ] Plugin installs successfully after fix
## Sources
- Issue #62: https://github.com/mvanhorn/last30days-skill/issues/62
- Issue #63: https://github.com/mvanhorn/last30days-skill/issues/63 (skipped - score 4/10, niche)
- Recent commits: `4b7087e`, `9a1059e`, `ecf90c0`, `adb5a67`
- PRs #56-#59 merged since v2.9.1
@@ -0,0 +1,118 @@
---
title: "feat: Add Truth Social as opt-in source"
type: feat
status: completed
date: 2026-03-09
---
# feat: Add Truth Social as opt-in source
Add Truth Social (Mastodon fork) as an opt-in social source. When `TRUTHSOCIAL_TOKEN` is set, posts from Truth Social appear alongside other sources in research results. When not configured, completely silent.
## Problem / Motivation
Issue #63 requested Truth Social support. Truth Social is a Mastodon fork with ~7M monthly active users. For users who care about that community's perspective on a topic, it's a valuable signal source. Follows the same opt-in pattern as Bluesky.
## Approach
Use Truth Social's Mastodon-compatible API directly with `urllib3` (no external dependencies). One env var: `TRUTHSOCIAL_TOKEN` (bearer token). Follow the Bluesky source pattern exactly across all 10 pipeline files.
**Why bearer token (not username/password):** Truth Social's OAuth uses a non-standard `/oauth/v2/token` endpoint with hardcoded `client_id`/`client_secret` extracted from their JS bundle. These values break when Truth Social updates their frontend. A bearer token is more stable - user extracts it once from browser dev tools (Application > Local Storage > truthsocial.com > `access_token`) or via `truthbrush` CLI.
**API endpoint:**
```
GET https://truthsocial.com/api/v2/search
Authorization: Bearer {token}
Params: q={topic}&type=statuses&limit=40
```
**Response format:** Standard Mastodon status objects with `content` (HTML), `created_at`, `url`, `account`, `favourites_count`, `reblogs_count`, `replies_count`.
## Files to Change
| # | File | Change |
|---|------|--------|
| 1 | `scripts/lib/truthsocial.py` | **New file.** API client: `search_truthsocial(topic, from_date, to_date, depth, config)` + `parse_truthsocial_response(response)`. Strip HTML tags from `content`. Handle 401/403/429 gracefully. |
| 2 | `scripts/lib/env.py` | Add `('TRUTHSOCIAL_TOKEN', None)` to `get_config()`. Add `is_truthsocial_available(config)`. |
| 3 | `scripts/lib/schema.py` | Add `TruthSocialItem` dataclass (id prefix `"TS"`). Add `truthsocial`/`truthsocial_error` fields to `Report`. Update `to_dict()`/`from_dict()`. |
| 4 | `scripts/lib/normalize.py` | Add `normalize_truthsocial_items()`. Map Mastodon fields: `favourites_count` -> `likes`, `reblogs_count` -> `reposts`, `replies_count` -> `replies`. |
| 5 | `scripts/lib/score.py` | Add `compute_truthsocial_engagement_raw()` + `score_truthsocial_items()`. Same weighted log formula as Bluesky. |
| 6 | `scripts/lib/dedupe.py` | Add `dedupe_truthsocial()` (one-liner wrapping `dedupe_items`). |
| 7 | `scripts/lib/render.py` | Add Truth Social sections to `_xref_tag()`, `_assess_data_freshness()`, `render_compact()`, `render_source_status()`, `render_context_snippet()`, `render_full_report()`. |
| 8 | `scripts/last30days.py` | ~20 touchpoints: TIMEOUT_PROFILES, VALID_SEARCH_SOURCES, import, `_search_truthsocial()`, `run_research()` param + dispatch + collect, `main()` availability + diag + flag + normalize + score + sort + dedupe + report. |
| 9 | `SKILL.md` | Add Truth Social to source lists, optionalEnv (`TRUTHSOCIAL_TOKEN`), stats template, security/privacy section. |
| 10 | `tests/test_truthsocial.py` | **New file.** Tests: HTML stripping, date parsing, response parsing, empty response, missing fields, depth config, auth error handling, successful search with mocked HTTP. |
## Key Implementation Details
### HTML stripping (`truthsocial.py`)
Truth Social returns HTML content (`<p>Post text</p>`). Strip tags to plain text:
```python
import re
def _strip_html(html: str) -> str:
text = re.sub(r'<br\s*/?>', '\n', html)
text = re.sub(r'<[^>]+>', '', text)
return text.strip()
```
### Date filtering
Mastodon `created_at` is ISO 8601 (`2026-03-09T12:00:00.000Z`). Use `[:10]` slice for `YYYY-MM-DD` comparison against `from_date`/`to_date`.
### Engagement mapping
| Mastodon field | Internal field | Display |
|---------------|---------------|---------|
| `favourites_count` | `likes` | `{N}lk` |
| `reblogs_count` | `reposts` | `{N}rp` |
| `replies_count` | `replies` | `{N}re` |
### Error handling
| HTTP Status | Behavior |
|------------|----------|
| 200 | Parse and return results |
| 401 | Return `{"statuses": [], "error": "Truth Social token expired"}` |
| 403 | Return `{"statuses": [], "error": "Truth Social access denied (Cloudflare)"}` |
| 429 | Return `{"statuses": [], "error": "Truth Social rate limited"}` |
| Other | Return `{"statuses": [], "error": "Truth Social search failed: {status}"}` |
All errors return empty results gracefully - never crash the research run.
### DEPTH_CONFIG
| Depth | Limit |
|-------|-------|
| quick | 15 |
| default | 30 |
| deep | 60 |
## What NOT to Change
- `lib/__init__.py` - must stay bare (no eager imports)
- `cross_source_link.py` - works generically on items with `cross_refs` field
- `filter.py` - date filtering is generic
- No new pip dependencies
## Acceptance Criteria
- [x] `is_truthsocial_available()` returns False when `TRUTHSOCIAL_TOKEN` not set
- [x] No Truth Social stats line, no error, no mention when unconfigured
- [x] With valid `TRUTHSOCIAL_TOKEN`, posts are returned and rendered
- [x] HTML tags stripped from post content
- [x] Token expiry (401) returns empty results gracefully
- [x] Cloudflare block (403) returns empty results gracefully
- [x] `--diagnose` shows Truth Social availability status
- [x] SKILL.md documents `TRUTHSOCIAL_TOKEN` env var
- [x] All existing tests still pass
- [x] New tests cover: HTML stripping, parsing, auth errors, successful search
- [x] `bash scripts/sync.sh` deploys successfully
## Sources
- Issue #63: https://github.com/mvanhorn/last30days-skill/issues/63
- Truth Social API: Mastodon-compatible at `truthsocial.com/api/v2/search`
- Auth: Bearer token via browser dev tools or `truthbrush` CLI
- Pattern reference: `scripts/lib/bluesky.py` (most recent source addition)
- Bluesky auth plan: `docs/plans/2026-03-09-fix-bluesky-auth-opt-in-plan.md`
@@ -0,0 +1,105 @@
---
title: "fix: Make Bluesky opt-in with auth (searchPosts now requires authentication)"
type: fix
status: completed
date: 2026-03-09
---
# fix: Make Bluesky opt-in with auth (searchPosts now requires authentication)
## Problem
Bluesky's `app.bsky.feed.searchPosts` endpoint now returns **403 Forbidden** for unauthenticated requests. This broke today (2026-03-09), likely related to the CEO transition. The `searchActors` endpoint still works without auth, but `searchPosts` does not.
We built Bluesky as an always-on source (like HN and Polymarket) because the AT Protocol public API was documented as free. That's no longer true for post search.
**Current behavior:** Bluesky silently returns 0 results (403 caught, empty list returned). No stats line appears, no error shown. Users don't know it exists.
**Desired behavior:** Bluesky is opt-in via env vars. When not configured, completely invisible (no error, no stats line, no mention). When configured with auth, it works as before.
## Approach
Follow the same pattern as X/Twitter auth: env vars for credentials, availability check gates dispatch.
AT Protocol auth flow:
1. User creates an App Password at `bsky.app/settings/app-passwords`
2. User sets `BSKY_HANDLE` and `BSKY_APP_PASSWORD` env vars
3. `bluesky.py` calls `com.atproto.server.createSession` to get a bearer token
4. Subsequent `searchPosts` calls include `Authorization: Bearer {token}`
## Files to Change
| File | Change | Lines (est.) |
|------|--------|-------------|
| `scripts/lib/bluesky.py` | Add `_create_session()` auth, pass bearer token to search | ~25 |
| `scripts/lib/env.py` | Update `is_bluesky_available()` to check env vars; add env var loading | ~10 |
| `scripts/last30days.py` | Gate `do_bluesky` on `is_bluesky_available(config)`; update diag dicts | ~8 |
| `tests/test_bluesky.py` | Add auth session tests, update existing tests | ~15 |
| `SKILL.md` | Document `BSKY_HANDLE` / `BSKY_APP_PASSWORD` env vars in setup section | ~5 |
## Implementation Steps
### 1. `scripts/lib/env.py`
- Add `BSKY_HANDLE` and `BSKY_APP_PASSWORD` to the env var loading in `get_config()`
- Update `is_bluesky_available()`:
```python
def is_bluesky_available(config: Dict[str, Any]) -> bool:
return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))
```
### 2. `scripts/lib/bluesky.py`
- Add `_create_session(handle, app_password)` that POSTs to `https://bsky.social/xrpc/com.atproto.server.createSession`
- Cache the access token for the lifetime of the search (module-level or passed through)
- Update `search_bluesky()` to accept config dict, extract creds, create session, add `Authorization: Bearer {token}` header
- On auth failure, return `{"posts": [], "error": "Bluesky auth failed"}` (don't raise)
### 3. `scripts/last30days.py`
- Change `do_bluesky` default from `True` to checking `is_bluesky_available(config)`:
```python
has_bluesky = env.is_bluesky_available(config)
```
- Gate the `bluesky_future` submit on `has_bluesky` (already gated on `do_bluesky`, just wire it)
- Update both diagnostic dicts: `"bluesky": True` -> `"bluesky": has_bluesky`
- Pass `config` to `_search_bluesky()` so it can extract auth creds
### 4. `tests/test_bluesky.py`
- Test `_create_session` with mocked HTTP response
- Test `search_bluesky` with auth header injection
- Existing parse/normalize tests don't change (they don't hit the network)
### 5. `SKILL.md`
- Add Bluesky to the optional env vars section (near AUTH_TOKEN/CT0):
```
BSKY_HANDLE=your-handle.bsky.social
BSKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx
```
- Keep Bluesky in source lists but note "(requires app password)" in setup
## What NOT to Change
- `render.py` - already handles empty bluesky gracefully (hides stats line when 0 items)
- `schema.py`, `normalize.py`, `score.py`, `dedupe.py` - no changes needed, they work on items already
- No error display when unconfigured - completely silent, same as how TikTok/Instagram behave when SCRAPECREATORS_API_KEY is missing
## Acceptance Criteria
- [x] `is_bluesky_available()` returns False when env vars not set
- [x] No Bluesky stats line, no error, no mention when unconfigured
- [x] With valid `BSKY_HANDLE` + `BSKY_APP_PASSWORD`, posts are returned
- [x] Auth failure returns empty results gracefully (no crash)
- [x] SKILL.md documents the env vars
- [x] All existing Bluesky tests still pass
- [x] New auth tests added
- [x] `bash scripts/sync.sh` deploys successfully
## Sources
- AT Protocol auth: `POST https://bsky.social/xrpc/com.atproto.server.createSession` with `{identifier, password}`
- App passwords: `bsky.app/settings/app-passwords`
- Existing pattern: X auth with AUTH_TOKEN/CT0 in `env.py:256-257`
- Existing pattern: TikTok/Instagram opt-in via `is_tiktok_available()` in `env.py:499`
+48
View File
@@ -0,0 +1,48 @@
# Search Quality Eval
`scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
- runs a baseline revision (default `origin/main`) against a candidate checkout
- evaluates the fixed 5 reviewer topics by default
- computes deterministic stability metrics:
- `Jaccard` overlap vs baseline
- retention vs baseline
- per-source counts and overlap
- optionally calls Gemini as a judge for graded relevance labels and then computes:
- `Precision@5`
- `nDCG@5`
- source-coverage recall across the judged union pool
Recommended usage:
```bash
uv run python scripts/evaluate_search_quality.py
```
Useful flags:
```bash
uv run python scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
--topic "cursor IDE pricing" \
--per-source-limit 5
```
Gemini configuration:
- preferred on this workspace: set `GOOGLE_API_KEY`
- also accepted: `GEMINI_API_KEY` or `GOOGLE_GENAI_API_KEY`
- optional: set `GEMINI_MODEL`
- default model is `gemini-3-pro-preview` for the direct Gemini API
Notes:
- The script forces a clean env-based auth path when it shells out to `last30days.py`.
- It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
- It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either.
- `Jaccard` and retention are regression guards, not truth metrics.
- `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.
@@ -0,0 +1,388 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -0,0 +1,310 @@
# V1 vs V2 Comparison Analysis
**Date:** 2026-02-06
**Queries tested:** 4 (1 head-to-head, 3 V1-only)
**Scope:** Quick smoke test, not full 17-query matrix
---
## Part 1: Head-to-Head -- "kanye west" (NEWS Query)
### Dimension-by-Dimension Scoring
#### 1. Query Parsing Display
Does it show the `🔍 **{TOPIC}** · {QUERY_TYPE}` line before running tools?
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 1 | No parsing display at all. Output starts with "## What I learned:" -- jumps straight into synthesis. No acknowledgment of topic or query type before research. |
| V2 | 1 | No parsing display either. Output starts with "Here's what I found:" then "## What I learned:" -- same problem as V1. |
**Analysis:** Neither version actually rendered the query parsing display. V2 SKILL.md explicitly requires `🔍 **kanye west** · News` before any tools run, but the agent did not produce it. This is a V2 instruction that failed to land. Both score 1/5.
Possible cause: The parsing display is supposed to appear *before* tools are called -- it may have been shown during execution but not captured in the final output text. If so, both outputs represent only the post-research synthesis, not the full session. Regardless, based on what is in the output files, neither shows it.
---
#### 2. Source Coverage (Reddit/X/Web counts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | `Reddit: 0 relevant threads` / `X: 30 posts │ ~10 likes` / `Web: 20+ pages`. Two of three sources returned results. Reddit was zero. |
| V2 | 3 | `Reddit: 0 threads (no results this cycle)` / `X: 29 posts │ 33 likes │ 14 reposts` / `Web: 30+ pages`. Same pattern: two of three returned results. |
**Analysis:** Nearly identical coverage. Both got zero Reddit results (likely a script/API issue for this topic, not a SKILL.md problem). V2 has slightly more precise X metrics (33 likes, 14 reposts vs. V1's vague "~10 likes"). V2 has more web pages (30+ vs 20+). Both miss the 10+ Reddit threshold for a score of 4+.
---
#### 3. Citation Quality (sparse vs every-sentence)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 2 | No inline citations at all. The body text makes claims ("full-page Wall Street Journal apology," "Hellwatt Festival in Italy") but never attributes them to a specific source. The stats box lists "Washington Post, Billboard, AllHipHop" but the body has zero `per @handle` or `per Rolling Stone` attributions. |
| V2 | 5 | Every bold section ends with a sparse, clean citation. Examples: `"per Rolling Stone"`, `"per The Washington Post"`, `"per Billboard"`, `"per AllHipHop"`, `"per The News International"`. One citation per topic, never chained. Exactly what V2 SKILL.md specifies. |
**Analysis:** This is the single biggest quality gap between V1 and V2. V1's output reads like a Wikipedia summary -- informative but ungrounded. V2 reads like a researched briefing where every claim has a named source. V2 nails the "sparse citation" rule from its SKILL.md: `"cite 1 source per pattern, short format: 'per @handle' or 'per r/sub'"`.
V1 quote (no citation): `"He'll headline the new Hellwatt Festival in Italy (July 4-18, 2026)."`
V2 quote (cited): `"Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard."`
---
#### 4. Summary Structure (bold topic headers, organized sections)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Has a coherent narrative structure with a paragraph of synthesis, then a `**KEY THEMES:**` numbered list. But the opening is a single dense paragraph, not broken into scannable sections with bold headers. |
| V2 | 5 | Each storyline gets its own bold header: `**BULLY Album — March 20, 2026 via Gamma**`, `**Public Apology for Antisemitism**`, `**Hellwatt Festival in Italy**`, `**Health Concerns**`, `**Grammys Ban**`, `**Kim & Lewis Hamilton Buzz**`. Each is a standalone scannable unit with 1-3 sentences. |
**Analysis:** V2 follows the SKILL.md template exactly: `**{Topic 1}** — [1-2 sentences, per source]`. V1 uses a blob + list approach which is readable but less scannable. V2 is notably better for a user who wants to skim and find the story they care about.
V1 structure: 1 dense paragraph -> 5-item `KEY THEMES` list
V2 structure: 6 bold topic cards, each self-contained -> no KEY THEMES list (but doesn't need one because the structure itself is the organization)
---
#### 5. Stats Box Format (emoji tree vs plain text)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Uses `├─` tree format with emoji: `├─ 🟠 Reddit: 0 relevant threads` / `├─ 🔵 X: 30 posts` / `├─ 🌐 Web: 20+ pages` / `└─ Top voices:`. Minor deviation: says "0 relevant threads (filtered out noise)" instead of the V1 SKILL.md template "0 threads (no results this cycle)". Also omits the `🗣️` emoji on the Top voices line. |
| V2 | 5 | Perfect match to V2 SKILL.md template: `├─ 🟠 Reddit: 0 threads (no results this cycle)` / `├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)` / `├─ 🌐 Web: 30+ pages │ rollingstone.com, ...` / `└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex`. Includes `(via xAI)` notation, `🗣️` emoji, @handles with engagement counts. |
**Analysis:** V2 is tighter and matches its template exactly. V1 is close but has minor deviations (custom "filtered out noise" text, missing `🗣️` emoji, no @handles or engagement counts on Top voices). V2's inclusion of actual @handles with like counts (`@honest30bgfan_ (33 likes)`) adds credibility.
---
#### 6. Research Grounding (actual research vs generic knowledge)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Clearly grounded: mentions specific details like "Wall Street Journal apology (Jan 26, 2026)," "four-month-long manic episode," "frontal-lobe brain injury," "North West collaborated on 'Piercings on My Hand,'" "Monumental Plaza de Toros." These are specific enough to be from research, not pre-training. Minor generic leakage: the "KEY THEMES" list uses editorial framing ("Accountability arc," "Mental health transparency") that feels more like analysis than research extraction. |
| V2 | 5 | Every fact is specific and attributed: "12th studio album," "13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign," "earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded," "103,000-capacity RCF Arena." The AI-deepfaked vocals detail is a standout -- it is clearly from research, not something a model would know from pre-training. The Kim/Lewis Hamilton item (`"X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton"`) is explicitly sourced from X data, not general knowledge. |
**Analysis:** Both are well-grounded, but V2 has more "could only come from research" details. The deepfaked vocals story, the exact venue capacity, and the explicit X chatter observation are details that prove the synthesis is from the research output, not hallucinated.
---
#### 7. Prompt Quality (invitation to share vision, not dumping prompts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Ends with: `"Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in."` This is a follow-up invitation, but it is NOT the SKILL.md-specified invitation. It is topic-specific and conversational, which is nice, but it does not ask the user to "share your vision for what you want to create." It misses the prompt-generation angle entirely. |
| V2 | 5 | Ends with exactly: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice."` This matches the V2 SKILL.md template verbatim. It positions the skill correctly: not a news summarizer but a research-to-prompt pipeline. |
**Analysis:** V1's closing is friendly but off-brand. It treats the skill as a research tool, not a research-to-prompt tool. V2 correctly frames the next step as "tell me what to create and I'll write the prompt." This is a meaningful difference -- V1 would leave a user thinking they just got a summary, while V2 primes them to get a usable output.
---
### Head-to-Head Scorecard
| Dimension | V1 | V2 | Winner |
|-----------|----|----|--------|
| 1. Query Parsing Display | 1 | 1 | Tie (both failed) |
| 2. Source Coverage | 3 | 3 | Tie |
| 3. Citation Quality | 2 | 5 | **V2 (+3)** |
| 4. Summary Structure | 3 | 5 | **V2 (+2)** |
| 5. Stats Box Format | 4 | 5 | **V2 (+1)** |
| 6. Research Grounding | 4 | 5 | **V2 (+1)** |
| 7. Prompt Quality (invitation) | 3 | 5 | **V2 (+2)** |
| **TOTAL** | **20/35** | **29/35** | **V2 wins by 9 points** |
**V2 is clearly better.** The biggest gaps are citation quality (+3) and summary structure (+2). V2's output reads like a professional research briefing; V1's reads like a decent but unstructured summary.
---
## Part 2: V1-Only Outputs Analysis
### Output 1: "open claw" (GENERAL query)
**What V1 does well:**
- Strong research grounding. Mentions exact numbers: "145,000+ GitHub stars," "20,000+ forks," "700+ skills," "341 malicious skills." These are clearly from research.
- The KEY PATTERNS section is excellent: 5 well-organized patterns with community quotes (`"I give it sudo and let it configure everything"` vs `"prompt injection is terrifying when you give the bot access to your actual bank account"`).
- Good synthesis of the security vs. enthusiasm tension -- captures the community split accurately.
- Stats box uses the emoji tree format correctly with `├──` (though note: uses double-dash `──` instead of single `─`, minor inconsistency).
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display (`🔍 **open claw** · General`).
- No inline citations in the body text. The 5 KEY PATTERNS have no `per @handle` or `per r/sub` attribution. Which Reddit thread said "I give it sudo"? Which X post raised the security concern? We do not know.
- The stats box says `├── 🟠 Reddit: 25 threads │ ~750+ upvotes` -- the tilde and plus are imprecise. V2 SKILL.md wants exact parsed numbers.
- Top voices line lists subreddits and handles but no engagement counts: `@grok, @Starlink` -- are these the highest-engagement handles? No like counts shown.
- No bold topic headers in the body -- it is a single paragraph followed by a numbered list, not the `**{Topic}** — sentence, per source` format V2 requires.
**V1 Score (estimated):** 22/35
---
### Output 2: "nano banana pro prompting" (PROMPTING query)
**What V1 does well:**
- Correctly identifies two prompting styles (JSON structured vs. natural language "Creative Director") and explains when each works best. This is excellent PROMPTING-type synthesis.
- KEY PATTERNS are specific and actionable: "85mm lens at f/1.8," "three-point lighting with key at 45 degrees," "text rendering works -- keep text under 3 words for best results (75% success rate)." These are concrete tips a user can apply immediately.
- Research grounding is strong: cites specific upvote counts ("149-259 upvotes"), subreddit names (`r/nanobanana2pro`), and the Google AI blog.
- The invitation correctly targets Nano Banana Pro: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro."`
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Stats box uses plain text dashes: `- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments` instead of the tree format `├─ 🟠 Reddit:`. Uses `|` pipe instead of `│` box-drawing character. V2 SKILL.md explicitly says: "NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji."
- No inline body citations. KEY PATTERNS mention Reddit upvote ranges but no specific `per @handle` attributions.
- Missing `✅ All agents reported back!` header -- just says "All agents reported back!" without the checkmark.
- Body structure is paragraph + numbered list, not bold topic headers.
**V1 Score (estimated):** 23/35 (slightly higher than open claw due to better actionability)
---
### Output 3: "how to best setup clawdbot" (HOW-TO query)
**What V1 does well:**
- This is the best V1 output of the batch. It goes beyond synthesis and actually delivers a **Quick-Start guide** with numbered steps, a **Security Hardening** checklist, and a **Budget Option** -- all grounded in research.
- Excellent research grounding: `"per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0"` -- this is an actual citation with an @handle!
- Specific, actionable recommendations: exact commands (`curl -fsSL https://clawd.bot/install.sh | bash`), specific model recommendations (Claude Opus 4.5 for best results, GLM 4.7 Flash for local), specific channel advice (Telegram first, WhatsApp QR code fails).
- Stats box is correct emoji tree format with engagement counts: `@aashatwt (452 likes), @recap_david (329 likes)`.
- Captures the naming confusion accurately: "Clawdbot -> Moltbot -> OpenClaw."
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Body text has no inline citations except the Budget Option section. The 5 KEY PATTERNS have no `per @handle` attribution.
- Bold topic headers are used only in the Quick-Start and Security sections, not in the KEY PATTERNS or intro.
- The output delivers the "answer" directly (setup guide) rather than waiting for the user's vision and offering to write a prompt. For a HOW-TO query this might be the right call, but it skips the SKILL.md flow of "show research -> invite vision -> write prompt."
**V1 Score (estimated):** 26/35 (best of the V1 outputs)
---
### Patterns Across All V1 Outputs
**Consistent strengths:**
1. Research grounding is solid across all three. V1 does not hallucinate -- the facts are clearly from the research output, not pre-training.
2. KEY PATTERNS lists are consistently useful and actionable.
3. Stats boxes are present in all outputs (though formatting varies).
4. The invitation/closing line is present in all outputs.
**Consistent weaknesses:**
1. **No query parsing display** in any output (0 for 4, including Kanye West).
2. **No inline citations** in the body text (except one @handle in the clawdbot output). The research feels real but is unattributed.
3. **Stats box formatting is inconsistent.** Open claw uses `├──` (double dash), nano banana pro uses `- 🟠` (plain dash + pipe), clawdbot uses `├─` (correct). Three different formats in three outputs.
4. **Body structure defaults to paragraph + numbered list** instead of bold topic headers. Only clawdbot partially uses bold headers (in the guide section, not the research section).
5. **No `(via Bird/xAI)` notation** on X stats in any output.
---
## Part 3: SKILL.md Feature Diff
### Features in V2 but NOT V1
| Feature | V2 Lines | Impact |
|---------|----------|--------|
| **Query parsing display** (`🔍 **{TOPIC}** · {QUERY_TYPE}`) | 40-53 | HIGH -- confirms to user the skill understood their request before spending time on research. |
| **Sparse citation rules** with BAD/GOOD examples | 186-193 | HIGH -- this is the #1 quality differentiator in the Kanye head-to-head. `"per @handle"` format, never chain multiple citations. |
| **Bold topic headers** template (`**{Topic 1}** — [1-2 sentences, per source]`) | 195-208 | HIGH -- makes output scannable. |
| **Strict stats template** with "NEVER use plain text dashes" instruction | 217-230 | MEDIUM -- prevents the formatting inconsistency seen across V1 outputs. |
| **RECOMMENDATIONS source attribution** (each item MUST have Sources: line with @handles) | 178-182 | MEDIUM -- only affects RECOMMENDATIONS queries. |
| **Reddit 0 results handling** (explicit instruction for what to write) | 229 | LOW -- edge case, but prevents ad-hoc text like V1's "filtered out noise." |
| **Bird CLI / xAI notation** in stats | 223 | LOW -- cosmetic transparency about data source. |
| **Step 2 phrasing: "DO WEBSEARCH WHILE SCRIPT RUNS"** | 71-73 | LOW -- execution optimization, no output impact. |
### Features in V1 but NOT V2
| Feature | V1 Lines | Impact | Should Restore? |
|---------|----------|--------|-----------------|
| **Use cases block** (4 examples in intro) | 12-17 | LOW | No |
| **Setup Check section** (3 modes, bash script, "keys are OPTIONAL") | 50-78 | MEDIUM for new users | Yes, for public release |
| **BAD/GOOD synthesis anti-pattern examples** | 172-191 | MEDIUM-HIGH | YES |
| **Self-check instruction** ("Re-read your 'What I learned' section...") | 269 | MEDIUM | YES |
| **Quality Checklist** (5-point checklist before delivering prompt) | 306-324 | HIGH | YES |
| **Prompt format anti-pattern** ("Research says JSON but you write prose") | 302 | MEDIUM | YES |
| **"IF USER ASKS FOR MORE OPTIONS"** section | 327-329 | LOW-MEDIUM | YES |
| **Web-only mode stats template + promo** | 248-259 | MEDIUM for no-key users | For public release |
| **TARGET_TOOL question template** (4 options) | 272-280 | LOW | No |
| **Context Memory: explicit "don't re-search" instructions** | 342-358 | MEDIUM | YES |
| **Output footer emoji + engagement counts** | 366-380 | LOW | YES |
### Features in BOTH (Shared)
| Feature | Notes |
|---------|-------|
| Parse User Intent (TOPIC, TARGET_TOOL, QUERY_TYPE) | Same 4 query types, same detection logic |
| "Don't ask about tool before research" rule | Identical |
| Research script execution command | Same `python3` command |
| WebSearch queries by QUERY_TYPE | Same search strategies |
| "Use user's exact terminology" instruction | V2 shorter but same intent |
| Judge Agent synthesis logic | Same 5-step weighting process |
| "Ground in actual research" instruction | Same core instruction, V1 has more examples |
| RECOMMENDATIONS: extract specific names | Same logic |
| Prompt format matching | Same instruction |
| Wait for user's vision | Same |
| Write ONE perfect prompt | Same structure |
| Context Memory | V2 shorter version |
| Output summary footer | Both have it, V1 has emoji |
| Depth options (quick/default/deep) | Same |
| "After each prompt: Stay in Expert Mode" | Same |
### Overall Assessment
**V2 is a clear upgrade in output formatting and citation quality.** The three features V2 adds (query parsing display, sparse citation rules, bold topic headers) directly address the three biggest weaknesses seen across all V1 outputs. The Kanye West head-to-head proves it: V2 scores 29/35 vs V1's 20/35.
**However, V2 dropped several quality guardrails from V1** that do not affect formatting but affect *correctness*: the self-check instruction, the anti-pattern examples, the quality checklist for prompts, and the "don't re-search" context memory rule. These are cheap to restore (under 25 lines total) and protect against subtle failure modes that may not show up in a 1-query test but will appear over dozens of uses.
---
## Part 4: Verdict
### Ship V2 or Not?
**Ship V2 -- but restore the guardrails first.**
V2 is unambiguously better on every formatting dimension. The citation quality improvement alone (V1: 2/5 -> V2: 5/5) makes it worth shipping. The bold topic headers and strict stats template fix the inconsistency problems visible across all V1 outputs.
But V2 dropped 6 guardrail features from V1 that cost almost nothing to include and protect against real failure modes. These should be restored before V2 goes public.
### Remaining Gaps
**Must fix before shipping (affects correctness):**
1. **Restore the quality checklist for prompts.** This is the test plan's #1 priority item. V1 had a 5-point checklist; V2 reduced it to one line. The checklist is what makes prompts feel polished -- it is the "that's a great prompt" mechanism. Add 8 lines.
2. **Restore BAD/GOOD anti-pattern examples.** V2 says "ground in actual research" but does not show what *bad* grounding looks like. V1's ClawdBot/Claude Code conflation example is exactly the kind of concrete negative example that prevents real failures. Add 5 lines.
3. **Restore self-check instruction.** One sentence: "Re-read your 'What I learned' section -- does it match what the research ACTUALLY says?" Zero cost, catches hallucination. Add 2 lines.
4. **Restore "don't re-search" context memory rule.** V2 only says "only do new research if user asks about a DIFFERENT topic." V1 explicitly bans re-searching and tells the agent to answer from existing research. Add 3 lines.
**Should fix (polish):**
5. Restore prompt format anti-pattern ("Research says JSON but you write prose"). Add 2 lines.
6. Restore "IF USER ASKS FOR MORE OPTIONS" section. Add 2 lines.
7. Add emoji + engagement counts back to the output summary footer. Edit 3 lines.
**Skip for now:**
8. Setup Check section -- add back for public release, not needed for execution.
9. Web-only mode stats template -- lower priority, most testers have API keys.
10. TARGET_TOOL question template -- agent handles this naturally.
### Query Parsing Display: Investigate
Both V1 and V2 scored 1/5 on query parsing display. V2 has the feature in its SKILL.md but the agent did not render it in the captured output. This could mean:
- The display was shown during execution but not captured (likely -- it appears before tools run, and the output files may only contain post-research content).
- The instruction is not strong enough and the agent skips it.
**Recommendation:** Verify in a live session whether the parsing display actually appears. If it does not, strengthen the instruction (e.g., "This line MUST be the first thing you output, before any tool calls").
### Total Effort
Restoring all 7 priority items: approximately 25 lines added to V2 SKILL.md. Under 15 minutes of work. The V2 formatting wins are substantial and proven; the V1 guardrails are small and proven. Combining both produces the best version.
### Final Score Summary
| | V1 (Kanye) | V2 (Kanye) | Delta |
|--|-----------|-----------|-------|
| Total | 20/35 | 29/35 | **V2 +9** |
| | V1 (Open Claw) | V1 (Nano Banana) | V1 (Clawdbot) | V1 Average |
|--|---------------|-----------------|--------------|------------|
| Estimated Total | 22/35 | 23/35 | 26/35 | **23.7/35** |
V2 at 29/35 beats every V1 output, including V1's best (clawdbot at 26/35).
**Decision: Ship V2 with guardrails restored.**
@@ -0,0 +1,388 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -0,0 +1,25 @@
## What I learned:
The last 30 days have been one of Kanye West's most significant periods in years. The biggest story is his **full-page Wall Street Journal apology** (Jan 26, 2026) titled "To Those I've Hurt" — a comprehensive open letter addressing his antisemitic remarks and erratic behavior. He attributed his actions to a "four-month-long manic episode" and revealed that a car accident 25 years ago caused a frontal-lobe brain injury that wasn't properly diagnosed until 2023, leading to his bipolar type-1 diagnosis. Multiple sources describe wife **Bianca Censori** as instrumental in getting him to seek treatment during a manic depressive episode.
On the music side, he's announced **two upcoming albums**: *BULLY* (now scheduled for March 20, 2026) and *IN A PERFECT WORLD*. Daughter **North West** collaborated with him on a track called "Piercings on My Hand." He performed a massive show in Mexico City (Jan 30) at the Monumental Plaza de Toros, and he'll headline the new **Hellwatt Festival** in Italy (July 4-18, 2026).
The **2026 Grammys** remain contentious — he's reportedly not invited after last year's red carpet drama. Health concerns also surfaced after a January LA outing where he appeared "noticeably bloated and moving with sluggish energy."
**KEY THEMES:**
1. **Accountability arc** — The WSJ apology is being widely discussed as either genuine growth or calculated PR
2. **Mental health transparency** — Brain injury + bipolar diagnosis framing his past behavior
3. **Bianca Censori's role** — Increasingly seen as a stabilizing force (despite continued provocative public appearances)
4. **Music comeback** — Two albums announced, active touring in Latin America and Europe
5. **Grammy exile** — Still persona non grata at major award shows
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 relevant threads (filtered out noise)
├─ 🔵 X: 30 posts │ ~10 likes │ casual fan chatter + Bianca Censori discourse
├─ 🌐 Web: 20+ pages │ washingtonpost.com, billboard.com, allhiphop.com, euronews.com, rap-up.com
└─ Top voices: Washington Post, Billboard, AllHipHop, Euronews, The Mirror
---
Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in.
@@ -0,0 +1,24 @@
## What I learned:
OpenClaw (formerly ClawdBot, formerly Moltbot) is an open-source autonomous AI agent created by Peter Steinberger that went massively viral in late January 2026 — hitting 145,000+ GitHub stars and 20,000+ forks. It runs locally on your machine, connects to messaging platforms (Telegram, WhatsApp, Discord, Signal, etc.), and uses LLMs like Claude, GPT, or Gemini to actually perform tasks on your computer — file management, scheduling, web browsing, purchases, and more. Think "JARVIS for everyone."
The community is simultaneously excited and alarmed. The "ClawHub" skill marketplace has 700+ skills but a VirusTotal/Koi Security audit found **341 malicious skills** doing data exfiltration and crypto theft. A high-severity RCE vulnerability was also disclosed. The #1 most downloaded skill on ClawHub was confirmed malware.
**KEY PATTERNS from the research:**
1. **Cost is the top complaint** — API usage burns money fast; many users recommend Gemini CLI (free 1,000 req/day) over Claude/GPT for OpenClaw to control spend
2. **Security is a dumpster fire** — Supply chain attacks via ClawHub skills, prompt injection risks when browsing the web, RCE vulnerabilities, and orgs are banning it at the EDR level
3. **Setup is nontrivial** — Even after install, most users get stuck on skills/tools/automation config; the `openclaw onboard` wizard is the recommended path
4. **The skill ecosystem is powerful but dangerous** — 700+ skills available, but no code signing, no real moderation, and active malware campaigns
5. **Community is split** — Enthusiasts love the autonomy ("I give it sudo and let it configure everything"), while security folks are terrified ("prompt injection is terrifying when you give the bot access to your actual bank account")
---
✅ All agents reported back!
├── 🟠 Reddit: 25 threads │ ~750+ upvotes │ ~600+ comments
├── 🔵 X: 27 posts │ 1,750+ likes │ 325+ reposts
├── 🌐 Web: 20+ pages │ Wikipedia, Bloomberg, The Register, Hacker News, Nature, VirusTotal, DigitalOcean, Codecademy, Tom's Hardware
└── Top voices: r/openclaw, r/cybersecurity, r/clawdbot, r/ArtificialInteligence │ @grok, @Starlink │ Nature, Bloomberg, ACM
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly.
@@ -0,0 +1,27 @@
Now I have a comprehensive picture. Let me synthesize everything.
**What I learned:**
Nano Banana Pro is fundamentally different from older image models — it's a "thinking" model that understands intent, physics, and composition. The community has converged on two distinct prompting styles, and **both work well** depending on the use case:
1. **JSON structured prompts** — Dominant on Reddit (r/nanobanana2pro) and X. Power users share elaborate JSON objects with nested fields for subject, appearance, lighting, composition, and style. These produce the most consistent, controlled results for portraits, editorials, and product shots.
2. **Natural language "Creative Director" prompts** — Recommended by Google's official guide. Full sentences describing the scene as if briefing a photographer. Better for cinematic shots, complex edits, and iterative refinement.
**KEY PATTERNS I'll use:**
1. **JSON format for maximum control** — The top Reddit posts (149-259 upvotes) all use structured JSON with nested objects: `subject > appearance > hair/skin/expression`, `lighting > key/fill/rim`, `camera > lens/angle/depth_of_field`
2. **Be a Creative Director, not a tag spammer** — Replace "4k, realistic, beautiful" with specific technical directions: "85mm lens at f/1.8", "three-point lighting with key at 45°", "shallow depth of field with bokeh"
3. **Provide context for WHY** — Adding purpose ("for a luxury fashion editorial", "for a high-end gourmet cookbook") helps the model infer appropriate styling, plating, lighting
4. **Edit, don't regenerate** — When 80% is right, ask for the specific change conversationally
5. **Text rendering works** — Keep text under 3 words for best results (75% success rate), specify font style and placement
---
All agents reported back!
- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments
- 🔵 X: 29 posts | 312 likes | 117 reposts
- 🌐 Web: 10+ pages | blog.google, dev.to, eweek.com, imagine.art, github.com, leonardo.ai, medium.com
- Top voices: r/nanobanana2pro | @Xmira_belle, @doctorwasif, @chatgptpaglu | Google AI blog, DEV Community
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro.
@@ -0,0 +1,48 @@
**What I learned:**
Clawdbot (now officially renamed **OpenClaw**) is a self-hosted, open-source personal AI assistant that connects to messaging apps (Telegram, WhatsApp, Discord, Slack, Signal, iMessage) and uses LLMs to act as an autonomous agent on your behalf. It's roughly one week old as a public project and the community is moving fast — the name has already changed twice (Clawdbot → Moltbot → OpenClaw).
**KEY PATTERNS from the research:**
1. **Security is the #1 concern** — Multiple Reddit threads and guides warn that many setups are wide open to the internet with zero auth. Run it on dedicated/isolated hardware (VM, Mac Mini, NAS), never your main machine. Block dangerous commands via allowlist. Never add it to group chats.
2. **Use the onboarding wizard, not Docker** — Docker was reported as unreliable (shims, bridges). The single-line install script (`curl -fsSL https://clawd.bot/install.sh | bash`) followed by `openclaw onboard` is the smoothest path. Docker is a "no-go" per multiple users.
3. **Model choice matters a lot** — Claude Opus 4.5 is recommended for best results (trained to resist prompt injection, ~99% resistance). For budget setups, Claude Haiku works at $0. Local models (Ollama/LM Studio) work but cause hallucination and token issues — GLM 4.7 Flash (30B) was recommended as the best local option.
4. **Telegram is the easiest channel to start with** — Most guides and users start with a Telegram bot token. WhatsApp QR code pairing fails frequently. Get Telegram working first, then expand.
5. **Sandbox and isolate** — Set `agents.defaults.sandbox.mode` to `"non-main"` for Docker sandboxing. Use a Linux VM or UTM on Mac to segregate traffic. Tailscale for secure remote access.
---
✅ All agents reported back!
├─ 🟠 Reddit: 19 threads │ 415+ upvotes │ 290+ comments
├─ 🔵 X: 29 posts │ 815+ likes │ 65+ reposts
├─ 🌐 Web: 10+ pages │ Medium, DataCamp, DEV, GitHub, QNAP, docs.openclaw.ai
└─ Top voices: r/clawdbot, r/LocalLLM, r/AskVibecoders │ @aashatwt (452 likes), @recap_david (329 likes) │ Lukas Niessen on Medium
---
**Here's the consensus best setup path based on what the community actually recommends:**
### Quick-Start (30 min)
1. **Get dedicated hardware** — Mac Mini, old laptop, NAS (QNAP has an official guide), or a VPS (DigitalOcean gist available). Don't use your daily driver.
2. **Install via script**: `curl -fsSL https://clawd.bot/install.sh | bash`
3. **Run the wizard**: `openclaw onboard` — it walks through gateway, workspace, channels, and skills
4. **Connect Telegram first** — create a bot via @BotFather, paste the token
5. **Set your API key** — Anthropic API key for Claude, or connect Ollama for local models
6. **Customize SOUL.md** — this controls your bot's personality and behavior
### Security Hardening (do this immediately)
- Bind gateway to localhost only, use Tailscale for remote access
- Set sandbox mode to `"non-main"`
- Create an explicit command allowlist (block `rm -rf`, `sudo`, `chmod`)
- Rotate API keys every 90 days
- Never add the bot to group chats
### Budget Option ($0)
Per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0.
---
Want me to help you with a specific part of the setup, or do you have a particular use case in mind (home automation, CRM, coding assistant, etc.)?
@@ -0,0 +1,332 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
---
## STEP 2: DO WEBSEARCH WHILE SCRIPT RUNS
The script auto-detects sources (Bird CLI, API keys, etc). While waiting for it, do WebSearch.
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords?
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
---
## THEN: Show Summary + Invite Vision
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned with sources:
```
🏆 Most mentioned:
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle3, r/sub2, Complex
Notable mentions: [other specific things with 1-2 mentions]
```
**CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson)
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety)
- Parse @handles from research output and include the highest-engagement ones
- Format naturally - tables work well for wide terminals, stacked cards for narrow
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
**BAD:** "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
**GOOD:** "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
**THEN - Stats (right before invitation):**
**CRITICAL: Calculate actual totals from the research output.**
- Count posts/threads from each section
- Sum engagement: parse `[Xlikes, Yrt]` from each X post, `[Xpts, Ycmt]` from Reddit
- Identify top voices: highest-engagement @handles from X, most active subreddits
**Copy this EXACTLY, replacing only the {placeholders}:**
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
```
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT.**
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Quality Checklist (run before delivering):
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS]
---
This uses [brief 1-line explanation of what research insight you applied].
```
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
@@ -0,0 +1,25 @@
Here's what I found:
## What I learned:
**BULLY Album — March 20, 2026 via Gamma** — After years of delays (first announced September 2024), Kanye's 12th studio album finally has a firm release date. The 13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign. Sonically it recalls *808s & Heartbreak* and *MBDTF* — Ye mostly sings rather than raps. Notably, earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded with his real voice for the official release, per Rolling Stone.
**Public Apology for Antisemitism** — On January 26, Ye took out a full-page Wall Street Journal ad titled "To Those I've Hurt," publicly apologizing for his antisemitic remarks. He disclosed a previously undiagnosed brain injury from his 2002 car accident and attributed his behavior to a four-month manic episode fueled by bipolar disorder, psychosis, and paranoia. "I lost touch with reality," he wrote. The apology also extended to the Black community, per The Washington Post.
**Hellwatt Festival in Italy** — Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard.
**Health Concerns** — A rare January 2026 outing in LA with Bianca Censori reignited concern about Ye's physical appearance. Insiders point to medication side effects, frequent travel, and inconsistent routines, per AllHipHop.
**Grammys Ban** — Ye is reportedly not welcome at the 2026 Grammy Awards after clashing with organizers last year over his invitation terms, per The News International.
**Kim & Lewis Hamilton Buzz** — X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton, with users contrasting her new relationship against her marriage to Ye.
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 threads (no results this cycle)
├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)
├─ 🌐 Web: 30+ pages │ rollingstone.com, washingtonpost.com, complex.com, billboard.com, npr.org
└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice.
+18 -20
View File
@@ -13,7 +13,7 @@ YouTube transcripts are the second headline feature. Inspired by Peter Steinberg
**New in V2.1 — two headline features:**
- **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a tweet — now the skill reads it. Inspired by @steipete's yt-dlp + summarize toolchain.
- **X search is fully bundled.** No external `bird` CLI or xAI API key needed. Just Node.js 22+ and your browser cookies. Uses a vendored subset of Bird's Twitter GraphQL client (MIT licensed, originally by @steipete).
- **X search is fully bundled.** No external `bird` CLI install needed. Add `AUTH_TOKEN` and `CT0` once, and the vendored Bird client runs locally without browser-cookie prompts. `XAI_API_KEY` remains an optional fallback.
---
@@ -21,20 +21,18 @@ YouTube transcripts are the second headline feature. Inspired by Peter Steinberg
### X Search Authentication
X search reads your existing browser cookies — no API keys or login commands needed.
X search prefers explicit env auth. This keeps local runs headless and avoids browser-cookie and macOS Keychain prompts.
**Safari (recommended on Mac):** Just be logged into x.com. No setup needed.
**Recommended setup:** While logged into x.com once, open browser dev tools and copy the `auth_token` and `ct0` cookies for `x.com`.
**Chrome:** Works, but macOS will prompt you to allow Keychain access the first time. Click "Allow" (or "Always Allow" to stop future prompts).
**Firefox:** Just be logged into x.com. No setup needed.
**Manual fallback:** If cookie auto-detection doesn't work, set these env vars (grab them from your browser's dev tools → Application → Cookies → x.com):
Save them as `AUTH_TOKEN` and `CT0` in `~/.config/last30days/.env` or `.claude/last30days.env`:
```bash
export AUTH_TOKEN=your_auth_token
export CT0=your_ct0_token
AUTH_TOKEN=your_auth_token
CT0=your_ct0_token
```
**xAI fallback:** If you do not want to provide `AUTH_TOKEN` and `CT0`, set `XAI_API_KEY` and use xAI's `x_search` backend instead.
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
@@ -44,8 +42,10 @@ node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs
## README: Install block env line
```
XAI_API_KEY=xai-... # optional — cookie auth is default for X search
```bash
AUTH_TOKEN=... # recommended for X search
CT0=... # recommended for X search
XAI_API_KEY=xai-... # optional X fallback
```
---
@@ -60,15 +60,13 @@ XAI_API_KEY=xai-... # optional — cookie auth is default for X search
## GitHub issue #19 response (post AFTER publishing)
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted steipete was asked to take it down.
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted. steipete was asked to take it down.
>
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly — no external CLI, no `npm install`, no brew. Just Node.js 22+ and your browser cookies.
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly. No external CLI, no `npm install`, no brew. Just Node.js 22+ plus `AUTH_TOKEN` and `CT0`, or `XAI_API_KEY` as fallback.
>
> It also adds **YouTube as a 4th source** — when yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
> It also adds **YouTube as a 4th source**. When yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
>
> If you're on a Mac, Safari is the easiest path for X — just be logged into x.com. Chrome works too but macOS will prompt for Keychain access the first time.
>
> If cookie auto-detection doesn't work, you can set `AUTH_TOKEN` and `CT0` env vars manually (grab from browser dev tools → Application → Cookies → x.com).
> The recommended setup is to copy `auth_token` and `ct0` from x.com once and store them as `AUTH_TOKEN` and `CT0` in your env. That avoids browser-cookie and Keychain prompts during normal runs.
>
> The xAI API (`XAI_API_KEY`) also still works as a fallback.
@@ -82,7 +80,7 @@ XAI_API_KEY=xai-... # optional — cookie auth is default for X search
Two new features:
→ YouTube transcripts as a 4th source (yt-dlp)
→ X search fully bundled (no bird CLI needed)
→ X search fully bundled (no bird CLI install needed)
Research any topic across Reddit, X, YouTube & web in one command.
@@ -100,7 +98,7 @@ When yt-dlp is installed, the skill searches YouTube, grabs view counts, and ext
### Thread version (post 2)
2️⃣ X search is fully bundled
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Just be logged into x.com in your browser. No npm install, no API keys.
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Add `AUTH_TOKEN` and `CT0` once, then it runs locally with no npm install. `XAI_API_KEY` still works as fallback.
Both features inspired by @steipete's tooling.
+360
View File
@@ -0,0 +1,360 @@
# last30days v2.5 Launch Thread
## FINAL THREAD (6 tweets)
### 1/6 - Announcement
I can't believe it's been 30 days since I launched @slashlast30days. 3.2k stars later, time for v2.5.
Three big additions:
1. @Polymarket prediction markets as a 6th source - helps you predict the future
2. Cross-source linking + massively better results - detects when the same story trends across multiple platforms. Ran a 15-way blinded comparison, v2.5 scored 4.38 vs 3.73 for the original. Won all 5 topics.
3. Hacker News as a 5th source - a window into the tech and developer insider world
github.com/mvanhorn/last30days-skill
### 2/6 - Demo: Anthropic vs Pentagon
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views). 5 HN stories. 9 Polymarket markets.
This story broke TODAY. Hegseth designated Anthropic a "supply chain risk." Trump ordered every agency to stop using their tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March: only 6%.
Markets say Anthropic wins regardless. That's the kind of signal you can't get from opinion threads.
### 3/6 - Demo: Seedance Prompting
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. Research first, then create from what you learned.
### 4/6 - Demo: Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Arizona is 25-2, set a program record with a 22-0 start, and holds a 2-game Big 12 lead with 3 games left. The Field of 68 called them "the TOUGHEST team in America" after escaping Baylor shorthanded. Kansas rematch Saturday - the highlight video from their first meeting has 248K views on ESPN's YouTube.
Polymarket: Championship 13%. #1 seed: 88%. Duke and Michigan each at 18% to win it all.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
### 5/6 - Demo: Iran War
"/last30days iran war"
2 Reddit threads. 34 X posts (10,048 likes). 20 YouTube videos (1.6M views, 5 transcripts). 4 HN stories (850 points). 14 Polymarket markets ($473M volume).
Geneva talks just ended without a deal. 150+ US aircraft deployed. Two carrier strike groups in position. F-22s sent to Israel. Members of Congress who saw the secret war plan came out "terrified." @cenkuygur: "they are about to drag us into a war that 70-85% of Americans oppose" (7,700 likes).
Polymarket ($473M in volume - one of their biggest markets ever): strikes by 2026: 80%. By March 31: 68%. War Powers invoked: 51%. Formal war declaration: only 12%.
Markets say: strikes are very likely, declared war is not. That's the sharpest signal in the entire research.
### 6/6 - Thank You
Thank you to ARJ999 and wkbaran on GitHub who filed three separate issues asking for Hacker News support. v2.5 delivers.
It's been a crazy 30 days. 3.2k stars. Six sources. Massively better results. Super excited to get this out.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
---
## REFERENCE MATERIAL BELOW
## Context
- 3.2k stars on GitHub
- V2.5 headline features: Polymarket (6th source), Hacker News (5th source), cross-source linking
- Ran 15-way blinded comparison: 4.38/5.0 vs 3.73/5.0
- Won all 5 topics, zero regressions
- Cross-source linking: 3 -> 13 linked items
- Demo topics: Anthropic odds (11 markets), Arizona basketball (6 markets), Iran war ($425M volume)
---
## Post 1: Lead (Announcement)
V2.5 of @slashlast30days is out. Now with @Polymarket prediction markets, cross-source linking, and massively better results.
1. Polymarket as a 6th source - real money on outcomes, no API key needed
2. Hacker News as a 5th source
3. Cross-source linking - detects when the same story trends across multiple platforms
Ran a 15-way blinded comparison across 5 topics. v2.5 scored 4.38 vs 3.73 for the original. Won all 5. Zero regressions.
github.com/mvanhorn/last30days-skill
---
## Post 2: POLYMARKET AS A 6TH SOURCE.
Reddit tells you what people think. X tells you what people share. YouTube tells you what people watch. HN tells you what developers discuss.
Polymarket helps you predict the future.
"/last30days anthropic odds"
11 markets found. Best AI model February: Anthropic 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. Pentagon ban odds: only 22%.
Free API. No key. Real money on outcomes.
---
## Post 3: CROSS-SOURCE LINKING.
When a Seedance 2.0 tutorial has 44K YouTube views AND trends on HN AND gets discussed on Reddit, v2.5 flags it: [also on: HN, YouTube]
Old version linked 3 items across 5 test topics. New version links 13. The difference is hybrid similarity - combining character-trigram and token-level matching at a tuned threshold.
Cross-platform convergence is the strongest signal that something actually matters. Not engagement on one platform. Convergence across all of them.
---
## Post 4: 15-WAY BLINDED EVALUATION.
I don't trust vibes for measuring quality. So I ran a scientific comparison.
5 topics x 3 versions. Stripped version labels. Randomized as A/B/C. Scored on groundedness, specificity, coverage, actionability, and format.
v2.5: 4.38/5.0
v2.2 (HN only): 4.10/5.0
v2.0 (original): 3.73/5.0
Won all 5 topics. Zero regressions. Biggest gains: specificity (+0.8) and format (+1.0) from cross-source linking giving the synthesis better material to work with.
---
## Post 5: Demo - Anthropic Odds
Asked it about Anthropic odds.
11 Polymarket markets. 25 X posts. 13 YouTube videos (719K views). 6 HN stories (471 points).
Best AI model February: 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. FrontierMath 50% score: 48% (up 28% today). Pentagon ban: only 22%.
Markets say Anthropic is winning the model race AND the valuation race. The Pentagon thing is noise.
---
## Post 6: Demo - Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Championship odds: 13%. #1 seed: 88%. Big 12 title: Arizona leads by 2.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
The Polymarket integration uses two-pass query expansion. First pass finds "Arizona Big 12." Second pass discovers the championship and #1 seed markets via tag-based domain bridging.
---
## Post 7: Demo - Iran War
The best Polymarket demo is news.
"/last30days iran war"
14 Polymarket markets. $425M+ in volume. 7 Reddit threads. 30 X posts. 20 YouTube videos (2M views). 18 HN stories (1,187 points).
US strikes Iran by 2026: 70%. War Powers by March: 60%. Israel strikes by June: 64%. Formal war declaration: only 8%.
Markets say: limited strikes with War Powers, NOT a declared war. Breaking Points (435K views) covered leaked Pentagon opposition. r/Conservative "imploding" per r/SubredditDrama.
One command. Six sources. Real money.
---
## Post 8: Credits + CTA
Also in v2.5: YouTube synonym expansion ("hip hop" now matches "rap" - relevance jumped 0.33 to 0.71), X handle resolution, and HN OR queries for framework topics.
The difference between "good research" and "research you'd actually trust" is in details like this.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 9: Demo - Claude Code (ALL 6 sources)
"/last30days Claude Code"
3 Reddit threads (199 upvotes). 35 X posts (5,239 likes). 15 YouTube videos (1.4M views, 5 transcripts). 30 HN stories (~8,500 points). 8 Polymarket markets. 20 web pages.
All six sources hit. Top finding: the planning-first workflow has won. The #1 HN post this month (969 pts, 590 comments) is about separating planning from execution. Boris Cherny (Head of Claude Code) on Lenny's Podcast: "100% of my code is written by Claude Code - I have not edited a single line by hand since November."
Polymarket: Anthropic 99% for best AI model in February. 58% for March. Claude on FrontierMath at 55%. The US government rejected Claude - Polymarket has the Hegseth ban at 32%.
Then I asked it to dig deeper into the planning-first workflow. No new searches - it answered from what it already learned.
---
## Post 10: Demo - March Madness Odds (Polymarket + Sports)
"/last30days March Madness Odds"
2 Reddit threads. 31 X posts. 6 YouTube videos (46K views, 4 transcripts). 2 Polymarket markets.
Tournament winner: Duke 18%, Michigan 18%, Arizona 13%. #1 seeds: Michigan 98%, Duke 91%, Arizona 88%.
Duke is the hottest mover - went from +700 to +450 in one week. The skill surfaced that from sportsbook data, X commentary, and Polymarket odds simultaneously.
Then I asked it to break down Michigan vs Duke vs Arizona. Full analysis from the research it already had.
---
## Post 11: Demo - Seedance Prompting (Expert + Prompt Mode)
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 1 HN story. 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style + Constraints. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. That's the skill's real power - research first, then create from what you learned.
---
## Post 12: Thank You + CTA (Final)
Thank you to ARJ999 and wkbaran on GitHub who kept asking for Hacker News support. Three separate issues. v2.5 delivers.
30 days. 3.2k stars. 6 sources. Massively better results.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 13: Demo - Anthropic vs Pentagon (Breaking News + Polymarket)
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views, 5 transcripts). 5 HN stories. 9 Polymarket markets. 10 web pages.
This story broke TODAY. Defense Secretary Hegseth designated Anthropic a "supply chain risk" - believed to be the first time an American company has ever received this designation. Trump ordered every federal agency to stop using Anthropic tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March 31: only 6%.
Markets say: Anthropic wins the model race regardless. Bettors don't think Hegseth survives this. That's the kind of signal you can't get from opinion threads.
---
## Post 14: Demo - OpenAI Insider Trading (News + Polymarket)
"/last30days OpenAI Insider Trading"
2 Reddit threads. 29 X posts. 4 YouTube videos (360K views, 4 transcripts). 2 HN stories. 15 Polymarket markets. 15 web pages.
An OpenAI employee was just fired for using confidential info to bet on Polymarket. 13 brand-new wallets appeared 40 hours before the browser launch. $309K bet on the right outcome. Unusual Whales flagged 77 suspected insider positions across 60 wallets.
Meanwhile Polymarket has OpenAI's IPO at $1.25-1.5T: 54%. Anthropic IPOs first: 62%. Best AI model: Anthropic 99%.
The prediction markets are both the story AND the source. One command pulled all of it together.
---
## Standalone Tweet: Polymarket Stats Line
"/last30days Anthropic Pete Hegseth"
The Pentagon just designated Anthropic a supply chain risk. First time ever for an American company. Trump ordered every agency to stop using their tech.
Here's what Polymarket says:
📊 9 markets │ Best AI model: 99% │ $500B+ valuation: 68% │ IPO >$600B: 97% │ Hegseth out by March: 6%
Bettors with real money on the line think Anthropic wins the model race, goes public at a massive valuation, and Hegseth doesn't survive this.
That's the gap between headlines and reality. One command, six sources.
github.com/mvanhorn/last30days-skill
---
## Recommended Thread Order (pick 8-10)
The full thread above is 12 posts. Here's what I'd cut to keep it tight:
**Must include (core story):**
1. Post 1 - Lead announcement
2. Post 2 - Polymarket ("Reddit tells you what people think...")
3. Post 3 - Cross-source linking
4. Post 4 - Blinded evaluation
**Best demos (pick 3-4):**
- Post 13 (Anthropic vs Pentagon) - STRONGEST. Breaking news today. Polymarket cuts through the noise. "Markets say Anthropic wins regardless."
- Post 9 (Claude Code) - All 6 sources. Massive numbers. Shows follow-up flow.
- Post 10 (March Madness) - Sports/Polymarket crossover. Timely with tournament approaching.
- Post 11 (Seedance) - Shows prompting flow. 1.2M YouTube views.
- Post 5 (Anthropic Odds) - Overlaps with Post 13 now. Skip.
**Skip or save for standalone tweets:**
- Post 5 (Anthropic Odds) - Redundant with Post 13
- Post 6 (Arizona Basketball) - Covered by March Madness now
- Post 7 (Iran War) - Great standalone tweet, not for launch thread
- Post 8 (Credits/minor features) - Fold into CTA
**My recommended 8-post thread:**
1. Lead (Post 1)
2. Polymarket (Post 2)
3. Cross-source linking (Post 3)
4. Blinded evaluation (Post 4)
5. Demo: Anthropic vs Pentagon (Post 13) - breaking news, best Polymarket showcase
6. Demo: Claude Code (Post 9)
7. Demo: March Madness (Post 10)
8. Thank you + CTA (Post 12)
---
## Video Script (~60 seconds)
**[Talking to camera]**
Oh my god, I can't believe it's been 30 days since I launched last30days. 3,200 stars on GitHub. This has been the craziest month.
Today I'm shipping v2.5 and I'm really excited about this one. Three big things.
**[Screen recording: typing /last30days Anthropic Pete Hegseth]**
First - Polymarket prediction markets as a 6th source. So this Anthropic-Pentagon story broke today. Hegseth designated Anthropic a supply chain risk, Trump ordered agencies to stop using their tech. Scary headline, right?
But Polymarket says: Anthropic still 99% for best AI model. IPO above 600 billion: 97%. Hegseth out by March: 6%. Real money on outcomes helps you predict the future. That's a different story than the headlines.
**[Screen recording: typing /last30days arizona basketball]**
Second - it now searches Hacker News and does cross-source linking. When the same story shows up on Reddit AND YouTube AND HN, it flags it. I ran a 15-way blinded comparison and v2.5 scored 4.38 versus 3.73 for the original. Won all 5 test topics.
**[Back to camera]**
Thank you to everyone who starred it, filed issues, and kept pushing me to make this better. Shoutout to the people on GitHub who literally filed three separate issues asking for Hacker News. v2.5 delivers.
Link in bio. Try it on anything.
---
## Scoring Note
The 4.38 vs 3.73 score is from a custom 5-dimension rubric (30% groundedness, 25% specificity, 20% coverage, 15% actionability, 10% format compliance) evaluated by Claude on blinded outputs. The relative ranking is meaningful; the absolute numbers are not. It's an LLM grading LLM output - useful for A/B comparison, not for claiming "4.38 out of 5 quality."
If using the score in a tweet, frame it as "scored X vs Y on a blinded comparison" not "rated 4.38/5.0 quality" - the former is honest, the latter implies an objective standard that doesn't exist.
+67
View File
@@ -0,0 +1,67 @@
{
"name": "last30days-skill",
"version": "2.9.5",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
"name": "Extension Directory",
"description": "Extension installation directory (auto-set by Gemini CLI)",
"envVar": "GEMINI_EXTENSION_DIR",
"sensitive": false
},
{
"name": "ScrapeCreators API Key",
"description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
"envVar": "SCRAPECREATORS_API_KEY",
"sensitive": true
},
{
"name": "OpenAI API Key",
"description": "OpenAI API Key - optional fallback for Reddit discovery",
"envVar": "OPENAI_API_KEY",
"sensitive": true
},
{
"name": "xAI API Key",
"description": "xAI API Key for X/Twitter search (optional)",
"envVar": "XAI_API_KEY",
"sensitive": true
},
{
"name": "OpenRouter API Key",
"description": "OpenRouter API Key (optional)",
"envVar": "OPENROUTER_API_KEY",
"sensitive": true
},
{
"name": "Parallel AI API Key",
"description": "Parallel AI API Key (optional)",
"envVar": "PARALLEL_API_KEY",
"sensitive": true
},
{
"name": "Brave Search API Key",
"description": "Brave Search API Key (optional)",
"envVar": "BRAVE_API_KEY",
"sensitive": true
},
{
"name": "Apify API Token",
"description": "Apify API Token (optional legacy)",
"envVar": "APIFY_API_TOKEN",
"sensitive": true
},
{
"name": "Twitter AUTH_TOKEN",
"description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
"envVar": "AUTH_TOKEN",
"sensitive": true
},
{
"name": "Twitter CT0",
"description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
"envVar": "CT0",
"sensitive": true
}
]
}
+16
View File
@@ -0,0 +1,16 @@
{
"hooks": {
"SessionStart": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/check-config.sh",
"timeout": 5
}
]
}
]
}
}
+108
View File
@@ -0,0 +1,108 @@
#!/bin/bash
set -euo pipefail
# Check last30days configuration status and show appropriate welcome message.
# Priority: .claude/last30days.env > ~/.config/last30days/.env > env vars
PROJECT_ENV=".claude/last30days.env"
GLOBAL_ENV="$HOME/.config/last30days/.env"
# Helper: warn if file permissions are too open
check_perms() {
local file="$1"
if [[ ! -f "$file" ]]; then return; fi
local perms
perms=$(stat -f '%Lp' "$file" 2>/dev/null || stat -c '%a' "$file" 2>/dev/null || echo "")
if [[ -n "$perms" && "$perms" != "600" && "$perms" != "400" ]]; then
echo "/last30days: WARNING — $file has permissions $perms (should be 600)."
echo " Fix: chmod 600 $file"
fi
}
# Load env file into variables for inspection (without exporting)
load_env_vars() {
local file="$1"
if [[ -f "$file" ]]; then
while IFS='=' read -r key value; do
# Skip comments, empty lines
[[ "$key" =~ ^[[:space:]]*# ]] && continue
[[ -z "$key" ]] && continue
key=$(echo "$key" | xargs)
value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
if [[ -n "$key" && -n "$value" ]]; then
eval "ENV_${key}=\"${value}\""
fi
done < "$file"
fi
}
# Determine which config file is active
CONFIG_FILE=""
if [[ -f "$PROJECT_ENV" ]]; then
CONFIG_FILE="$PROJECT_ENV"
check_perms "$PROJECT_ENV"
elif [[ -f "$GLOBAL_ENV" ]]; then
CONFIG_FILE="$GLOBAL_ENV"
check_perms "$GLOBAL_ENV"
fi
# Load config if found
if [[ -n "$CONFIG_FILE" ]]; then
load_env_vars "$CONFIG_FILE"
fi
# Check SETUP_COMPLETE (from file or env)
SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
# If setup has never been run, show welcome message for new users
if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
cat <<'EOF'
/last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
Reddit, Hacker News, and Polymarket work out of the box.
The setup wizard can unlock X/Twitter, YouTube, and more.
EOF
exit 0
fi
# Setup done but check for ScrapeCreators
HAS_SCRAPECREATORS="${ENV_SCRAPECREATORS_API_KEY:-${SCRAPECREATORS_API_KEY:-}}"
HAS_X="${ENV_AUTH_TOKEN:-${AUTH_TOKEN:-}}"
HAS_XAI="${ENV_XAI_API_KEY:-${XAI_API_KEY:-}}"
HAS_YTDLP=""
if command -v yt-dlp &>/dev/null; then
HAS_YTDLP="yes"
fi
HAS_BSKY="${ENV_BSKY_HANDLE:-${BSKY_HANDLE:-}}"
HAS_EXA="${ENV_EXA_API_KEY:-${EXA_API_KEY:-}}"
# Count active sources
SOURCE_COUNT=2 # HN + Polymarket are always free
if [[ -n "$HAS_X" || -n "$HAS_XAI" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
# Reddit public JSON always works
SOURCE_COUNT=$((SOURCE_COUNT + 1))
if [[ -n "$HAS_YTDLP" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_EXA" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_BSKY" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 3)) # Reddit comments + TikTok + Instagram
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Fully configured — compact ready message
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
else
# Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 100 free API calls, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider."
fi
-253
View File
@@ -1,253 +0,0 @@
# Partner Credits: Zero-Registration Free Tier for last30days Users
**Type:** Partnership proposal for ScrapeCreators
**Date:** 2026-03-05
**Status:** Draft proposal
---
## The Pitch
Every last30days user gets 100 free ScrapeCreators credits — Reddit, TikTok, Instagram — without ever visiting scrapecreators.com or creating an account. When they run out, the skill tells them where to upgrade. ScrapeCreators gets a distribution channel. last30days gets a killer default experience.
## The Problem
Right now, new last30days users hit a wall:
1. Install the skill (30 seconds)
2. Try `/last30days AI video tools`
3. Get told they need a `SCRAPECREATORS_API_KEY`
4. Have to go to scrapecreators.com, register, get a key, paste it into `.env`
5. Many never come back
The best Reddit experience requires a key. The friction kills adoption.
## The Proposal: Machine-Bound Partner Tokens
### How It Works
**ScrapeCreators side:**
1. Issue last30days a **partner ID** (e.g., `partner_last30days`)
2. Accept a new header: `X-Partner-Device: <device_hash>`
3. On first request per device hash: allocate 100 credits, no registration needed
4. Track usage: `(partner_id, device_hash) → credits_remaining`
5. When credits hit 0: return `402` with upgrade URL in response body
**last30days side:**
1. On first run, generate a **device fingerprint** and cache it locally
2. If user has no `SCRAPECREATORS_API_KEY`, send requests with partner headers instead
3. When 402 comes back, show a friendly "upgrade" message
That's it. No accounts, no OAuth, no registration flow.
### The Device Fingerprint
```python
import hashlib, platform, uuid, os
def get_device_id():
"""Generate a stable, hard-to-forge device fingerprint."""
# Use the OS-level machine ID (persists across reinstalls on most systems)
machine_id = _get_machine_id()
# Salt with the partner ID so the hash is useless outside this context
raw = f"last30days:{machine_id}"
return hashlib.sha256(raw.encode()).hexdigest()
def _get_machine_id():
"""Get the OS hardware/machine ID."""
if platform.system() == "Darwin":
# macOS: IOPlatformUUID (hardware-bound, survives OS reinstall)
import subprocess
result = subprocess.run(
["ioreg", "-rd1", "-c", "IOPlatformExpertDevice"],
capture_output=True, text=True
)
for line in result.stdout.splitlines():
if "IOPlatformUUID" in line:
return line.split('"')[-2]
elif platform.system() == "Linux":
# Linux: /etc/machine-id (set at install time)
try:
return open("/etc/machine-id").read().strip()
except FileNotFoundError:
pass
# Fallback: MAC address + hostname (less stable but reasonable)
return f"{uuid.getnode()}:{platform.node()}"
```
**Why this works:**
- macOS `IOPlatformUUID` is hardware-bound — can't change it without a new motherboard
- Linux `/etc/machine-id` is set at OS install — persists across reboots
- Hashed with `last30days:` prefix so the raw ID is never sent to ScrapeCreators
- Cached locally in `~/.config/last30days/.device_id` after first generation
### API Request Format
```
# Without partner credits (existing flow — user has their own key)
GET /v1/reddit/search?query=AI+tools
x-api-key: sc_user_abc123
# With partner credits (new — no registration needed)
GET /v1/reddit/search?query=AI+tools
x-api-key: sc_partner_last30days
X-Partner-Device: a1b2c3d4e5f6... (sha256 hex)
```
ScrapeCreators treats `sc_partner_last30days` as a special key class:
- Requires `X-Partner-Device` header
- Credits tracked per device hash, not per API key
- Rate limited per device (e.g., 10 requests/minute)
- 100 credits per unique device, lifetime
### What Counts as a Credit
One API call = one credit. A typical `/last30days` run uses roughly:
- 2-4 Reddit searches (global + subreddit drilldowns)
- 1-2 TikTok searches + 2-3 transcript fetches
- 1-2 Instagram searches + 2-3 transcript fetches
So ~10-15 credits per run. 100 credits ≈ **7-10 full research runs** before upgrade.
That's enough to get hooked.
## Abuse Prevention
### What we're defending against
| Threat | Likelihood | Impact |
|--------|-----------|--------|
| User spoofs device ID to get infinite credits | Low | Medium |
| Script generates thousands of fake device IDs | Medium | High |
| User shares partner key for non-last30days use | Low | Low |
### Defenses (simplest first)
**1. Hardware-bound device ID (primary defense)**
- macOS IOPlatformUUID can't be changed without hardware swap
- Linux machine-id requires root to change and breaks other software
- Not a cookie or config file — it's the machine itself
**2. Rate limiting per device (ScrapeCreators side)**
- 10 requests/minute per device hash
- Prevents scripted rapid-fire abuse even with valid device IDs
- Normal usage never hits this — a full run takes 60-70 seconds with natural gaps
**3. IP rate limiting on new device registrations (ScrapeCreators side)**
- Max 3 new device hashes per IP per day
- Stops "generate 1000 device IDs from one server" attacks
- Legitimate users: one machine, one device ID, done
**4. Total partner pool cap (safety valve)**
- ScrapeCreators sets a monthly cap on total partner credits (e.g., 50,000/month)
- If last30days goes viral and blows the cap, both parties renegotiate
- Prevents runaway costs from unexpected growth
### What we're NOT doing (intentional simplicity)
- No CAPTCHAs
- No email verification
- No phone verification
- No browser fingerprinting
- No token signing or crypto
- No account creation whatsoever
The goal is zero friction. The device ID is "good enough" — it stops casual abuse and scripts. A determined attacker could maybe get 200-300 free credits by VM gymnastics, but that's not worth defending against when paid plans are cheap.
## User Experience
### First run (no key configured)
```
$ /last30days AI video tools
🔍 Searching Reddit, TikTok, Instagram...
️ Using 100 free partner credits from ScrapeCreators (93 remaining)
Get your own key for unlimited use: scrapecreators.com/last30days
[... normal results ...]
```
### Credits running low
```
️ 12 partner credits remaining. Get unlimited access: scrapecreators.com/last30days
```
### Credits exhausted
```
⚠️ Free partner credits used up!
Reddit, TikTok, and Instagram require a ScrapeCreators API key.
Get one at: scrapecreators.com/last30days (100 free credits on signup, then pay-as-you-go)
Continuing with X, YouTube, Hacker News, Polymarket, and web search...
```
Key detail: the skill **doesn't stop working** — it gracefully falls back to the sources that don't need a key. The user still gets value, but they see what they're missing.
### After upgrade
```
$ echo 'SCRAPECREATORS_API_KEY=sc_abc123' >> ~/.config/last30days/.env
# Next run — partner headers no longer sent, user's own key used
```
## What ScrapeCreators Gets
1. **Distribution channel** — every last30days install is a potential paying customer
2. **Zero support burden** — no accounts to manage for free tier users
3. **Qualified leads** — users who exhaust 100 credits are proven power users
4. **Co-marketing** — "Powered by ScrapeCreators" in every skill run
5. **Usage data** — anonymous device-level usage patterns across topics
## What last30days Gets
1. **Zero-config Reddit** — install and go, no registration anywhere
2. **TikTok and Instagram included** — three sources work out of the box
3. **Lower barrier to adoption** — the #1 friction point eliminated
4. **Upgrade path built in** — natural conversion funnel
## Implementation Effort
### ScrapeCreators side (their work)
- [ ] Create partner key class with per-device credit tracking
- [ ] Accept `X-Partner-Device` header on partner keys
- [ ] Return `402` with upgrade URL when credits exhausted
- [ ] Rate limit: 10 req/min per device, 3 new devices/day per IP
- [ ] Dashboard for last30days to see aggregate partner usage
### last30days side (our work)
- [ ] `scripts/lib/device_id.py` — generate and cache device fingerprint (~30 lines)
- [ ] Update `scripts/lib/env.py` — fall back to partner auth when no user key
- [ ] Update `_sc_headers()` in reddit.py, tiktok.py, instagram.py — add partner headers
- [ ] Handle 402 response — show upgrade message, continue with other sources
- [ ] Show credits remaining in run output (from response header)
### Suggested response headers from ScrapeCreators
```
X-Partner-Credits-Remaining: 87
X-Partner-Credits-Total: 100
X-Partner-Upgrade-URL: https://scrapecreators.com/last30days
```
## Open Questions
1. **Credit pool negotiation** — what monthly cap works for ScrapeCreators?
2. **Referral tracking** — should `scrapecreators.com/last30days` give a signup bonus or revenue share?
3. **Credit count per endpoint** — should transcript fetches cost the same as searches?
4. **Expiration** — do unused partner credits expire (e.g., 90 days)?
---
## Summary
One device ID. One partner key. One header. 100 free credits. Zero registration.
The entire abuse prevention is: your computer has a hardware ID that you can't easily change. That's it. Simple, clever, and good enough.
+8 -4
View File
@@ -2,15 +2,19 @@ The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## What's New in v2.9.1
**Auto-save to ~/Documents/Last30Days/.** Every run now saves the complete research briefing - synthesis, stats, and follow-up suggestions - as a topic-named `.md` file to your Documents folder. Build a personal research library without lifting a finger. Inspired by [@devin_explores](https://x.com/devin_explores) who was already doing this manually.
## Three Headline Features in v2.9
**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.
**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram - three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.
**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency × recency × topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw not generic programming subs.
**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency x recency x topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw - not generic programming subs.
**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with `💬` and upvote counts. Reddit's value is in the comments now the skill surfaces them.
**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with upvote counts. Reddit's value is in the comments - now the skill surfaces them.
Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** no external CLI needed.
Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** - no external CLI needed.
## Beta Test Results (v2.9)
+641
View File
@@ -0,0 +1,641 @@
#!/usr/bin/env python3
"""Run local search-quality evaluations across fixed topics.
This is an optional local gate, not a required CI job. It compares a baseline
revision against a candidate checkout, computes deterministic regression
metrics, and optionally calls Gemini as a judge for graded relevance labels.
"""
from __future__ import annotations
import argparse
import json
import math
import os
import shlex
import shutil
import subprocess
import sys
import tempfile
import textwrap
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Tuple
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
sys.path.insert(0, str(Path(__file__).parent))
from lib import env as envlib
REPO_ROOT = Path(__file__).resolve().parent.parent
DEFAULT_TOPICS: List[Tuple[str, str]] = [
("nano banana pro prompting", "product"),
("codex vs claude code", "comparison"),
("anthropic odds", "prediction"),
("kanye west", "breaking_news"),
("remotion animations for Claude Code", "how_to"),
]
DEFAULT_SEARCH = "reddit,x,youtube,hn,polymarket"
SOURCE_KEYS = [
"reddit",
"x",
"youtube",
"tiktok",
"instagram",
"hackernews",
"bluesky",
"truthsocial",
"polymarket",
"websearch",
]
DEFAULT_JUDGE_MODEL = "gemini-3-pro-preview"
GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}"
def slugify(topic: str) -> str:
return "".join(c.lower() if c.isalnum() else "-" for c in topic).strip("-")
def path_without_node(path_value: str) -> str:
parts = []
for entry in path_value.split(os.pathsep):
if not entry:
continue
if (Path(entry) / "node").exists():
continue
parts.append(entry)
return os.pathsep.join(parts)
def write_exec_wrapper(path: Path, target: str, fixed_args: List[str]) -> None:
quoted_target = shlex.quote(target)
quoted_args = " ".join(shlex.quote(arg) for arg in fixed_args)
path.write_text(f"#!/bin/sh\nexec {quoted_target} {quoted_args} \"$@\"\n")
path.chmod(0o755)
def create_eval_tool_path(eval_home: Path, base_path: str) -> str:
"""Create safe wrapper binaries for local evaluation subprocesses."""
bin_dir = eval_home / "bin"
bin_dir.mkdir(parents=True, exist_ok=True)
real_ytdlp = shutil.which("yt-dlp")
if real_ytdlp:
write_exec_wrapper(
bin_dir / "yt-dlp",
real_ytdlp,
["--ignore-config", "--no-cookies-from-browser"],
)
if not base_path:
return str(bin_dir)
return os.pathsep.join([str(bin_dir), base_path])
def stable_item_key(source: str, item: Dict[str, Any]) -> str:
url = str(item.get("url") or "").strip()
if url:
return url
item_id = str(item.get("id") or "").strip()
text = item_text(source, item)
return f"{source}:{item_id}:{text[:120]}"
def item_text(source: str, item: Dict[str, Any]) -> str:
if source in {"x", "bluesky", "truthsocial"}:
return str(item.get("text") or "").strip()
if source == "polymarket":
return str(item.get("question") or item.get("title") or "").strip()
return str(item.get("title") or "").strip()
def build_ranked_items(report: Dict[str, Any], per_source_limit: int) -> List[Dict[str, Any]]:
ranked: List[Dict[str, Any]] = []
for source in SOURCE_KEYS:
items = list(report.get(source) or [])[:per_source_limit]
for item in items:
ranked.append({
"source": source,
"key": stable_item_key(source, item),
"url": str(item.get("url") or "").strip(),
"text": item_text(source, item),
"score": float(item.get("score") or 0),
"relevance": float(item.get("relevance") or 0),
"date": item.get("date"),
})
ranked.sort(key=lambda item: (-item["score"], item["source"], item["key"]))
return ranked
def url_sets_by_source(report: Dict[str, Any]) -> Dict[str, set[str]]:
result: Dict[str, set[str]] = {}
for source in SOURCE_KEYS:
items = report.get(source) or []
urls = {
stable_item_key(source, item)
for item in items
}
result[source] = urls
return result
def jaccard(left: Iterable[str], right: Iterable[str]) -> float:
left_set = set(left)
right_set = set(right)
if not left_set and not right_set:
return 1.0
union = left_set | right_set
if not union:
return 1.0
return len(left_set & right_set) / len(union)
def retention(left: Iterable[str], right: Iterable[str]) -> float:
left_set = set(left)
right_set = set(right)
if not left_set:
return 1.0
return len(left_set & right_set) / len(left_set)
def precision_at_k(ranking: List[Dict[str, Any]], judgments: Dict[str, int], k: int) -> float:
top = ranking[:k]
if not top:
return 0.0
hits = sum(1 for item in top if judgments.get(item["key"], 0) >= 2)
return hits / len(top)
def ndcg_at_k(
ranking: List[Dict[str, Any]],
judgments: Dict[str, int],
k: int,
judged_pool: Optional[List[Dict[str, Any]]] = None,
) -> float:
top = ranking[:k]
if not top:
return 0.0
def dcg(grades: List[int]) -> float:
total = 0.0
for index, grade in enumerate(grades, start=1):
total += (2**grade - 1) / math.log2(index + 1)
return total
actual = [judgments.get(item["key"], 0) for item in top]
ideal_candidates = judged_pool or ranking
ideal = sorted(
(judgments.get(item["key"], 0) for item in ideal_candidates),
reverse=True,
)[:len(top)]
ideal_score = dcg(ideal)
if ideal_score == 0:
return 0.0
return dcg(actual) / ideal_score
def source_coverage_recall(
ranking: List[Dict[str, Any]],
judged_pool: List[Dict[str, Any]],
judgments: Dict[str, int],
) -> float:
good_sources = {item["source"] for item in judged_pool if judgments.get(item["key"], 0) >= 2}
if not good_sources:
return 1.0
hit_sources = {
item["source"]
for item in ranking
if judgments.get(item["key"], 0) >= 2
}
return len(hit_sources & good_sources) / len(good_sources)
def create_eval_env(include_web: bool) -> Tuple[Dict[str, str], Path]:
config = envlib.get_config()
eval_home = Path(tempfile.mkdtemp(prefix="last30days-eval-home-"))
(eval_home / ".config").mkdir(parents=True, exist_ok=True)
safe_path = create_eval_tool_path(
eval_home,
path_without_node(os.environ.get("PATH", "")),
)
passthrough = {
"HOME": str(eval_home),
"XDG_CONFIG_HOME": str(eval_home / ".config"),
"PATH": safe_path,
"LANG": os.environ.get("LANG", "en_US.UTF-8"),
"LC_ALL": os.environ.get("LC_ALL", ""),
"TMPDIR": os.environ.get("TMPDIR", ""),
"PYTHONUTF8": "1",
"LAST30DAYS_CONFIG_DIR": "",
"BIRD_DISABLE_BROWSER_COOKIES": "1",
"LAST30DAYS_DISABLE_BROWSER_COOKIES": "1",
}
for key in ("OPENAI_API_KEY", "XAI_API_KEY", "SCRAPECREATORS_API_KEY"):
value = config.get(key)
if value:
passthrough[key] = value
if include_web:
for key in ("PARALLEL_API_KEY", "BRAVE_API_KEY", "OPENROUTER_API_KEY"):
value = config.get(key)
if value:
passthrough[key] = value
return passthrough, eval_home
def run_last30days(
repo_dir: Path,
topic: str,
*,
search: str,
timeout_seconds: int,
include_web: bool,
env: Dict[str, str],
) -> Tuple[Dict[str, Any], str]:
cmd = [
sys.executable,
"scripts/last30days.py",
topic,
"--emit",
"json",
"--search",
search,
"--timeout",
str(timeout_seconds),
]
if not include_web:
cmd.append("--no-native-web")
result = subprocess.run(
cmd,
cwd=repo_dir,
env=env,
capture_output=True,
text=True,
timeout=timeout_seconds + 30,
check=False,
)
if result.returncode != 0:
raise RuntimeError(
f"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\n{result.stderr.strip()}"
)
return json.loads(result.stdout), result.stderr
def create_worktree(rev: str) -> Path:
worktree_dir = Path(tempfile.mkdtemp(prefix="last30days-eval-"))
subprocess.run(
["git", "worktree", "add", "--detach", str(worktree_dir), rev],
cwd=REPO_ROOT,
check=True,
capture_output=True,
text=True,
)
return worktree_dir
def remove_worktree(path: Path) -> None:
subprocess.run(
["git", "worktree", "remove", "--force", str(path)],
cwd=REPO_ROOT,
check=False,
capture_output=True,
text=True,
)
shutil.rmtree(path, ignore_errors=True)
def extract_gemini_text(payload: Dict[str, Any]) -> str:
for candidate in payload.get("candidates", []):
content = candidate.get("content") or {}
for part in content.get("parts", []):
text = part.get("text")
if text:
return text
raise ValueError("Gemini response did not contain text")
def resolve_google_judge_api_key(config: Dict[str, Any]) -> Optional[str]:
"""Resolve the local canonical Google API key name.
This workspace conventionally uses GOOGLE_API_KEY. We also accept the
more Gemini-specific aliases for portability.
"""
return (
os.environ.get("GOOGLE_API_KEY")
or config.get("GOOGLE_API_KEY")
or os.environ.get("GEMINI_API_KEY")
or config.get("GEMINI_API_KEY")
or os.environ.get("GOOGLE_GENAI_API_KEY")
or config.get("GOOGLE_GENAI_API_KEY")
)
def call_gemini_judge(api_key: str, model: str, prompt: str) -> Dict[str, Any]:
body = {
"contents": [{"parts": [{"text": prompt}]}],
"generationConfig": {
"temperature": 0,
"responseMimeType": "application/json",
},
}
url = GEMINI_API_URL.format(model=model, api_key=api_key)
request = Request(
url,
data=json.dumps(body).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urlopen(request, timeout=120) as response:
payload = json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Gemini HTTP {exc.code}: {detail}") from exc
except URLError as exc:
raise RuntimeError(f"Gemini request failed: {exc}") from exc
return json.loads(extract_gemini_text(payload))
def build_judge_prompt(
*,
topic: str,
query_type: str,
items: List[Dict[str, Any]],
) -> str:
item_lines = []
for item in items:
item_lines.append(
"\n".join([
f"- id: {item['key']}",
f" source: {item['source']}",
f" title: {item['text'][:220]}",
f" url: {item['url']}",
f" date: {item.get('date') or 'unknown'}",
])
)
joined = "\n".join(item_lines)
return textwrap.dedent(
f"""
Judge search-result relevance for a last-30-days research tool.
Topic: {topic}
Query type: {query_type}
Score each item on this 0-3 scale:
- 0 = off-topic or clearly bad
- 1 = weak or tangential
- 2 = relevant and useful
- 3 = highly relevant, one of the best results
Focus on actual user intent, not just token overlap. Penalize items that
only match generic words like "odds", "review", or "tips" without
matching the real entity or subject. Favor items that would genuinely
help answer the topic in the context of recent discussion.
Return strict JSON with this shape:
{{
"judgments": [
{{"id": "ITEM_ID", "grade": 0, "reason": "short reason"}}
]
}}
Items:
{joined}
"""
).strip()
def get_judgments(
*,
output_dir: Path,
slug: str,
topic: str,
query_type: str,
items: List[Dict[str, Any]],
judge_model: str,
gemini_api_key: Optional[str],
) -> Dict[str, int]:
cache_file = output_dir / "judgments" / f"{slug}.json"
cache_file.parent.mkdir(parents=True, exist_ok=True)
if cache_file.exists():
cached = json.loads(cache_file.read_text())
return {entry["id"]: int(entry["grade"]) for entry in cached.get("judgments", [])}
if not gemini_api_key:
return {}
prompt = build_judge_prompt(topic=topic, query_type=query_type, items=items)
payload = call_gemini_judge(gemini_api_key, judge_model, prompt)
cache_file.write_text(json.dumps(payload, indent=2))
return {entry["id"]: int(entry["grade"]) for entry in payload.get("judgments", [])}
def summarize_topic(
*,
topic: str,
query_type: str,
baseline_report: Dict[str, Any],
candidate_report: Dict[str, Any],
judged_pool: List[Dict[str, Any]],
judgments: Dict[str, int],
per_source_limit: int,
) -> Dict[str, Any]:
baseline_ranked = build_ranked_items(baseline_report, per_source_limit)
candidate_ranked = build_ranked_items(candidate_report, per_source_limit)
baseline_sets = url_sets_by_source(baseline_report)
candidate_sets = url_sets_by_source(candidate_report)
metrics = {
"topic": topic,
"query_type": query_type,
"baseline": {
"precision_at_5": precision_at_k(baseline_ranked, judgments, 5),
"ndcg_at_5": ndcg_at_k(baseline_ranked, judgments, 5, judged_pool),
"source_coverage_recall": source_coverage_recall(baseline_ranked, judged_pool, judgments),
},
"candidate": {
"precision_at_5": precision_at_k(candidate_ranked, judgments, 5),
"ndcg_at_5": ndcg_at_k(candidate_ranked, judgments, 5, judged_pool),
"source_coverage_recall": source_coverage_recall(candidate_ranked, judged_pool, judgments),
},
"stability": {
"overall_jaccard": jaccard(
set().union(*baseline_sets.values()),
set().union(*candidate_sets.values()),
),
"overall_retention_vs_baseline": retention(
set().union(*baseline_sets.values()),
set().union(*candidate_sets.values()),
),
"per_source": {
source: {
"baseline_count": len(baseline_sets[source]),
"candidate_count": len(candidate_sets[source]),
"jaccard": jaccard(baseline_sets[source], candidate_sets[source]),
"retention_vs_baseline": retention(baseline_sets[source], candidate_sets[source]),
}
for source in SOURCE_KEYS
},
},
}
return metrics
def write_markdown_summary(
output_dir: Path,
baseline_label: str,
candidate_label: str,
topic_summaries: List[Dict[str, Any]],
) -> None:
lines = [
f"# Search Quality Evaluation",
"",
f"- Baseline: `{baseline_label}`",
f"- Candidate: `{candidate_label}`",
f"- Generated: {datetime.now().isoformat(timespec='seconds')}",
"",
"## Topic Metrics",
"",
"| Topic | Base P@5 | Cand P@5 | Base nDCG@5 | Cand nDCG@5 | Jaccard | Retention |",
"|---|---:|---:|---:|---:|---:|---:|",
]
for summary in topic_summaries:
lines.append(
"| {topic} | {bp:.2f} | {cp:.2f} | {bn:.2f} | {cn:.2f} | {jac:.2f} | {ret:.2f} |".format(
topic=summary["topic"],
bp=summary["baseline"]["precision_at_5"],
cp=summary["candidate"]["precision_at_5"],
bn=summary["baseline"]["ndcg_at_5"],
cn=summary["candidate"]["ndcg_at_5"],
jac=summary["stability"]["overall_jaccard"],
ret=summary["stability"]["overall_retention_vs_baseline"],
)
)
lines.append("")
lines.append("## Notes")
lines.append("")
lines.append("- `Precision@5` and `nDCG@5` depend on the judged union pool, not a full gold corpus.")
lines.append("- `Source coverage recall` measures whether a run surfaced at least one judged-good result from the good sources in the judged pool.")
lines.append("- `Jaccard` and `retention` are stability guards against baseline drift, not truth metrics.")
(output_dir / "summary.md").write_text("\n".join(lines))
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Evaluate last30days search quality locally")
parser.add_argument("--baseline-rev", default="origin/main", help="Git revision for the baseline run")
parser.add_argument("--candidate-rev", default=None, help="Optional git revision for the candidate run")
parser.add_argument("--no-default-topics", action="store_true", help="Do not include the built-in 5-topic suite")
parser.add_argument("--topic", action="append", default=[], help="Extra topic to evaluate (repeatable)")
parser.add_argument("--search", default=DEFAULT_SEARCH, help="Comma-separated sources passed to --search")
parser.add_argument("--timeout", type=int, default=180, help="Per-topic timeout passed to last30days")
parser.add_argument("--per-source-limit", type=int, default=5, help="Items per source to judge")
parser.add_argument("--include-web", action="store_true", help="Include web-search keys and native web backends")
parser.add_argument("--judge-model", default=None, help="Gemini judge model override")
parser.add_argument("--judge-provider", choices=["auto", "gemini", "none"], default="auto")
parser.add_argument("--keep-worktrees", action="store_true", help="Leave temporary baseline/candidate worktrees on disk")
parser.add_argument("--output-dir", default=None, help="Output directory (default: docs/test-results/search-quality-<timestamp>)")
return parser.parse_args()
def main() -> int:
args = parse_args()
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
output_dir = Path(args.output_dir) if args.output_dir else REPO_ROOT / "docs" / "test-results" / f"search-quality-{timestamp}"
output_dir.mkdir(parents=True, exist_ok=True)
topics = [] if args.no_default_topics else list(DEFAULT_TOPICS)
topics.extend((topic, "custom") for topic in args.topic)
if not topics:
raise SystemExit("No topics configured. Use the default suite or pass --topic.")
judge_config = envlib.get_config()
judge_provider = args.judge_provider
gemini_api_key = resolve_google_judge_api_key(judge_config)
judge_model = args.judge_model or judge_config.get("GEMINI_MODEL") or DEFAULT_JUDGE_MODEL
if judge_provider == "auto":
judge_provider = "gemini" if gemini_api_key else "none"
if judge_provider == "none":
gemini_api_key = None
eval_env, eval_home = create_eval_env(include_web=args.include_web)
baseline_dir = create_worktree(args.baseline_rev)
candidate_dir = create_worktree(args.candidate_rev) if args.candidate_rev else REPO_ROOT
baseline_label = args.baseline_rev
candidate_label = args.candidate_rev or "working-tree"
topic_summaries: List[Dict[str, Any]] = []
try:
for topic, query_type in topics:
slug = slugify(topic)
baseline_report, baseline_stderr = run_last30days(
baseline_dir,
topic,
search=args.search,
timeout_seconds=args.timeout,
include_web=args.include_web,
env=eval_env,
)
candidate_report, candidate_stderr = run_last30days(
candidate_dir,
topic,
search=args.search,
timeout_seconds=args.timeout,
include_web=args.include_web,
env=eval_env,
)
topic_dir = output_dir / slug
topic_dir.mkdir(parents=True, exist_ok=True)
(topic_dir / "baseline.json").write_text(json.dumps(baseline_report, indent=2))
(topic_dir / "candidate.json").write_text(json.dumps(candidate_report, indent=2))
(topic_dir / "baseline.stderr.txt").write_text(baseline_stderr)
(topic_dir / "candidate.stderr.txt").write_text(candidate_stderr)
baseline_ranked = build_ranked_items(baseline_report, args.per_source_limit)
candidate_ranked = build_ranked_items(candidate_report, args.per_source_limit)
union_map = {item["key"]: item for item in baseline_ranked + candidate_ranked}
judgments = get_judgments(
output_dir=output_dir,
slug=slug,
topic=topic,
query_type=query_type,
items=list(union_map.values()),
judge_model=judge_model,
gemini_api_key=gemini_api_key,
)
summary = summarize_topic(
topic=topic,
query_type=query_type,
baseline_report=baseline_report,
candidate_report=candidate_report,
judged_pool=list(union_map.values()),
judgments=judgments,
per_source_limit=args.per_source_limit,
)
topic_summaries.append(summary)
payload = {
"baseline": baseline_label,
"candidate": candidate_label,
"judge_provider": judge_provider,
"judge_model": judge_model if gemini_api_key else None,
"topics": topic_summaries,
}
(output_dir / "summary.json").write_text(json.dumps(payload, indent=2))
write_markdown_summary(output_dir, baseline_label, candidate_label, topic_summaries)
print(output_dir)
return 0
finally:
if not args.keep_worktrees:
remove_worktree(baseline_dir)
if args.candidate_rev:
remove_worktree(candidate_dir)
shutil.rmtree(eval_home, ignore_errors=True)
if __name__ == "__main__":
raise SystemExit(main())
+456 -75
View File
@@ -38,13 +38,16 @@ _child_pids: set = set()
_child_pids_lock = threading.Lock()
TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "tiktok_future": 90, "instagram_future": 90, "hackernews_future": 30, "polymarket_future": 15, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10},
"default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "tiktok_future": 120, "instagram_future": 120, "hackernews_future": 60, "polymarket_future": 30, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
"deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "tiktok_future": 150, "instagram_future": 150, "hackernews_future": 90, "polymarket_future": 45, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
"quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "tiktok_future": 90, "instagram_future": 90, "hackernews_future": 30, "bluesky_future": 30, "truthsocial_future": 30, "polymarket_future": 15, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10},
"default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "tiktok_future": 120, "instagram_future": 120, "hackernews_future": 60, "bluesky_future": 60, "truthsocial_future": 60, "polymarket_future": 30, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
"deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "tiktok_future": 150, "instagram_future": 150, "hackernews_future": 90, "bluesky_future": 90, "truthsocial_future": 90, "polymarket_future": 45, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
}
# Valid source names for the --search flag
VALID_SEARCH_SOURCES = {"reddit", "x", "hn", "youtube", "tiktok", "instagram", "polymarket", "web"}
VALID_SEARCH_SOURCES = {
"reddit", "x", "hn", "bluesky", "bsky", "truthsocial", "truth", "youtube", "tiktok", "instagram",
"polymarket", "web", "xiaohongshu", "xhs",
}
def parse_search_flag(search_str: str) -> set:
@@ -64,6 +67,8 @@ def parse_search_flag(search_str: str) -> set:
s = s.strip().lower()
if not s:
continue
if s == "xhs":
s = "xiaohongshu"
if s not in VALID_SEARCH_SOURCES:
print(
f"Error: Unknown search source '{s}'. "
@@ -130,9 +135,12 @@ def _install_global_timeout(timeout_seconds: int):
from lib import (
bird_x,
bluesky,
truthsocial,
dates,
dedupe,
hackernews,
xiaohongshu_api,
polymarket,
entity_extract,
env,
@@ -141,16 +149,21 @@ from lib import (
normalize,
openai_reddit,
reddit,
reddit_public,
reddit_enrich,
render,
schema,
score,
scrapecreators_x,
setup_wizard,
ui,
tiktok,
instagram,
websearch,
xai_x,
youtube_yt,
quality_nudge,
query_type as qt,
)
@@ -174,8 +187,10 @@ def _search_reddit(
) -> tuple:
"""Search Reddit (runs in thread).
Uses ScrapeCreators when SCRAPECREATORS_API_KEY is available (preferred).
Falls back to OpenAI Responses API otherwise.
Hierarchy:
1. ScrapeCreators (if SCRAPECREATORS_API_KEY exists) premium, best quality
2. Public Reddit JSON (always available) free, good for thread discovery
3. OpenAI Responses API legacy fallback for backwards compatibility
Returns:
Tuple of (reddit_items, raw_response, error, used_scrapecreators)
@@ -189,7 +204,7 @@ def _search_reddit(
if mock:
raw_response = load_fixture("openai_sample.json")
elif sc_token:
# === ScrapeCreators path (preferred) ===
# === Tier 1: ScrapeCreators path (preferred) ===
used_scrapecreators = True
try:
sys.stderr.write("[Reddit] Using ScrapeCreators API\n")
@@ -206,16 +221,35 @@ def _search_reddit(
reddit_error = f"ScrapeCreators: {type(e).__name__}: {e}"
sys.stderr.write(f"[Reddit] ScrapeCreators failed: {e}\n")
sys.stderr.flush()
# Fall through to OpenAI if we have that key
if not config.get("OPENAI_API_KEY"):
return [], {"error": str(e)}, reddit_error, used_scrapecreators
used_scrapecreators = False
sys.stderr.write("[Reddit] Falling back to OpenAI\n")
sys.stderr.flush()
# Fall through to Tier 2 (public JSON)
# === OpenAI path (fallback) ===
# === Tier 2: Public Reddit JSON (free, always available) ===
if not mock:
try:
sys.stderr.write("[Reddit] Trying public Reddit JSON\n")
sys.stderr.flush()
reddit_items = reddit_public.search_reddit_public(
topic, from_date, to_date, depth=depth,
)
if reddit_items:
raw_response = {"source": "reddit_public", "items": reddit_items}
sys.stderr.write(f"[Reddit] Public JSON returned {len(reddit_items)} results\n")
sys.stderr.flush()
return reddit_items, raw_response, None, False
# Empty results — fall through to Tier 3
sys.stderr.write("[Reddit] Public JSON returned 0 results, trying OpenAI\n")
sys.stderr.flush()
except Exception as e:
sys.stderr.write(f"[Reddit] Public JSON failed: {e}\n")
sys.stderr.flush()
# Fall through to Tier 3
# === Tier 3: OpenAI Responses API (legacy fallback) ===
if not mock and config.get("OPENAI_API_KEY"):
try:
sys.stderr.write("[Reddit] Falling back to OpenAI Responses API\n")
sys.stderr.flush()
raw_response = openai_reddit.search_reddit(
config["OPENAI_API_KEY"],
selected_models["openai"],
@@ -233,11 +267,11 @@ def _search_reddit(
raw_response = {"error": str(e)}
reddit_error = f"{type(e).__name__}: {e}"
# Parse response
# Parse response (OpenAI path)
reddit_items = openai_reddit.parse_reddit_response(raw_response or {})
# Quick retry with simpler query if few results
if len(reddit_items) < 5 and not mock and not reddit_error:
# Quick retry with simpler query if few results (OpenAI path only)
if len(reddit_items) < 5 and not mock and not reddit_error and config.get("OPENAI_API_KEY"):
core = openai_reddit._extract_core_subject(topic)
if core.lower() != topic.lower():
try:
@@ -258,8 +292,8 @@ def _search_reddit(
except Exception:
pass
# Subreddit-targeted fallback if still < 3 results
if len(reddit_items) < 3 and not mock and not reddit_error:
# Subreddit-targeted fallback if still < 3 results (OpenAI path only)
if len(reddit_items) < 3 and not mock and not reddit_error and config.get("OPENAI_API_KEY"):
sub_query = openai_reddit._build_subreddit_query(topic)
try:
sub_raw = openai_reddit.search_reddit(
@@ -319,7 +353,7 @@ def _search_x(
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = bird_x.parse_bird_response(raw_response or {})
x_items = bird_x.parse_bird_response(raw_response or {}, query=topic)
# Check for error in response (Bird returns list on success, dict on error)
if raw_response and isinstance(raw_response, dict) and raw_response.get("error") and not x_error:
@@ -327,6 +361,25 @@ def _search_x(
return x_items, raw_response, x_error
# Use ScrapeCreators if specified
if x_source == "scrapecreators":
try:
raw_response = scrapecreators_x.search_x(
topic, from_date, to_date,
depth=depth,
token=config.get("SCRAPECREATORS_API_KEY"),
)
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = scrapecreators_x.parse_x_response(raw_response or {})
if raw_response and isinstance(raw_response, dict) and raw_response.get("error") and not x_error:
x_error = raw_response["error"]
return x_items, raw_response, x_error
# Use xAI (original behavior)
try:
raw_response = xai_x.search_x(
@@ -455,7 +508,7 @@ def _search_hackernews(
except Exception as e:
return [], f"{type(e).__name__}: {e}"
hn_items = hackernews.parse_hackernews_response(response)
hn_items = hackernews.parse_hackernews_response(response, query=topic)
if response.get("error"):
hn_error = response["error"]
@@ -463,6 +516,64 @@ def _search_hackernews(
return hn_items, hn_error
def _search_bluesky(
topic: str,
from_date: str,
to_date: str,
depth: str,
config: dict = None,
) -> tuple:
"""Search Bluesky via AT Protocol (runs in thread).
Returns:
Tuple of (bsky_items, bsky_error)
"""
bsky_error = None
try:
response = bluesky.search_bluesky(
topic, from_date, to_date, depth=depth, config=config,
)
except Exception as e:
return [], f"{type(e).__name__}: {e}"
bsky_items = bluesky.parse_bluesky_response(response)
if response.get("error"):
bsky_error = response["error"]
return bsky_items, bsky_error
def _search_truthsocial(
topic: str,
from_date: str,
to_date: str,
depth: str,
config: dict = None,
) -> tuple:
"""Search Truth Social via Mastodon API (runs in thread).
Returns:
Tuple of (ts_items, ts_error)
"""
ts_error = None
try:
response = truthsocial.search_truthsocial(
topic, from_date, to_date, depth=depth, config=config,
)
except Exception as e:
return [], f"{type(e).__name__}: {e}"
ts_items = truthsocial.parse_truthsocial_response(response)
if response.get("error"):
ts_error = response["error"]
return ts_items, ts_error
def _search_polymarket(
topic: str,
from_date: str,
@@ -506,7 +617,7 @@ def _search_web(
Tuple of (web_items, web_error)
web_items are raw dicts ready for websearch.normalize_websearch_items()
"""
from lib import brave_search, parallel_search, openrouter_search
from lib import brave_search, parallel_search, openrouter_search, exa_search
backend = env.get_web_search_source(config)
if not backend:
@@ -516,13 +627,19 @@ def _search_web(
raw_results = []
try:
if backend == "parallel":
if backend == "exa":
raw_results = exa_search.search_web(
topic, from_date, to_date, config["EXA_API_KEY"], depth=depth,
)
elif backend == "parallel":
raw_results = parallel_search.search_web(
topic, from_date, to_date, config["PARALLEL_API_KEY"], depth=depth,
)
elif backend == "brave":
use_llm_ctx = os.environ.get("BRAVE_LLM_CONTEXT", "").strip() == "1"
raw_results = brave_search.search_web(
topic, from_date, to_date, config["BRAVE_API_KEY"], depth=depth,
topic, from_date, to_date, config["BRAVE_API_KEY"],
depth=depth, use_llm_context=use_llm_ctx,
)
elif backend == "openrouter":
raw_results = openrouter_search.search_web(
@@ -543,6 +660,48 @@ def _search_web(
return raw_results, web_error
def _search_xiaohongshu(
topic: str,
config: dict,
from_date: str,
to_date: str,
depth: str,
) -> tuple:
"""Search Xiaohongshu via xiaohongshu-mcp HTTP API (runs in thread).
Returns:
Tuple of (xiaohongshu_items, xiaohongshu_error)
Items are in web-item dict shape and can be normalized with websearch module.
"""
base_url = env.get_xiaohongshu_api_base(config)
try:
items = xiaohongshu_api.search_feeds(
topic=topic,
from_date=from_date,
to_date=to_date,
base_url=base_url,
depth=depth,
)
except Exception as e:
return [], f"{type(e).__name__}: {e}"
# Ensure all required keys exist for normalize_websearch_items()
for i, item in enumerate(items):
item.setdefault("id", f"XHS{i+1}")
item.setdefault("title", "")
item.setdefault("url", "")
item.setdefault("source_domain", "xiaohongshu.com")
item.setdefault("snippet", "")
if item.get("date") and not item.get("date_confidence"):
item["date_confidence"] = "med"
elif not item.get("date"):
item["date_confidence"] = "low"
item.setdefault("relevance", 0.5)
item.setdefault("why_relevant", "")
return items, None
def _run_supplemental(
topic: str,
reddit_items: list,
@@ -727,9 +886,12 @@ def run_research(
run_youtube: bool = False,
run_tiktok: bool = False,
run_instagram: bool = False,
run_xiaohongshu: bool = False,
timeouts: dict = None,
resolved_handle: str = None,
do_hackernews: bool = True,
do_bluesky: bool = True,
do_truthsocial: bool = True,
do_polymarket: bool = True,
no_native_web: bool = False,
) -> tuple:
@@ -737,10 +899,10 @@ def run_research(
Returns:
Tuple of (reddit_items, x_items, youtube_items, tiktok_items, instagram_items,
hackernews_items, polymarket_items, web_items, web_needed,
hackernews_items, bluesky_items, truthsocial_items, polymarket_items, web_items, web_needed,
raw_openai, raw_xai, raw_reddit_enriched,
reddit_error, x_error, youtube_error, tiktok_error, instagram_error,
hackernews_error, polymarket_error, web_error)
hackernews_error, bluesky_error, truthsocial_error, polymarket_error, web_error)
Note: web_needed is True when web search should be performed by the assistant
(i.e., no native web search API keys are configured). When native web search
@@ -756,6 +918,8 @@ def run_research(
tiktok_items = []
instagram_items = []
hackernews_items = []
bluesky_items = []
truthsocial_items = []
polymarket_items = []
web_items = []
raw_openai = None
@@ -767,8 +931,11 @@ def run_research(
tiktok_error = None
instagram_error = None
hackernews_error = None
bluesky_error = None
truthsocial_error = None
polymarket_error = None
web_error = None
xiaohongshu_error = None
# Determine web search mode
do_web = sources in ("all", "web", "reddit-web", "x-web")
@@ -796,6 +963,19 @@ def run_research(
if progress:
progress.start_web_only()
progress.end_web_only()
# Optional Xiaohongshu search in web-only mode.
if run_xiaohongshu:
try:
xhs_items, xiaohongshu_error = _search_xiaohongshu(
topic, config, from_date, to_date, depth,
)
web_items.extend(xhs_items)
if xiaohongshu_error and progress:
progress.show_error(f"Xiaohongshu error: {xiaohongshu_error}")
except Exception as e:
xiaohongshu_error = f"{type(e).__name__}: {e}"
if progress:
progress.show_error(f"Xiaohongshu error: {e}")
# Still run YouTube/TikTok/Instagram in web-only mode if available
if run_youtube:
if progress:
@@ -837,7 +1017,7 @@ def run_research(
progress.show_error(f"Instagram error: {e}")
if progress:
progress.end_instagram(len(instagram_items))
return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error
return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, bluesky_items, truthsocial_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, bluesky_error, truthsocial_error, polymarket_error, web_error
# Determine which searches to run
do_reddit = sources in ("both", "reddit", "all", "reddit-web")
@@ -851,10 +1031,24 @@ def run_research(
youtube_future = None
tiktok_future = None
instagram_future = None
xiaohongshu_future = None
hackernews_future = None
bluesky_future = None
truthsocial_future = None
polymarket_future = None
web_future = None
max_workers = 2 + (1 if run_youtube else 0) + (1 if run_tiktok else 0) + (1 if run_instagram else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)
max_workers = (
2
+ (1 if run_youtube else 0)
+ (1 if run_tiktok else 0)
+ (1 if run_instagram else 0)
+ (1 if run_xiaohongshu else 0)
+ (1 if do_hackernews else 0)
+ (1 if do_bluesky else 0)
+ (1 if do_truthsocial else 0)
+ (1 if do_polymarket else 0)
+ (1 if web_backend else 0)
)
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit searches
@@ -897,6 +1091,11 @@ def run_research(
env.get_instagram_token(config),
)
if run_xiaohongshu:
xiaohongshu_future = executor.submit(
_search_xiaohongshu, topic, config, from_date, to_date, depth,
)
if do_hackernews:
if progress:
progress.start_hackernews()
@@ -904,6 +1103,16 @@ def run_research(
_search_hackernews, topic, from_date, to_date, depth
)
if do_bluesky:
bluesky_future = executor.submit(
_search_bluesky, topic, from_date, to_date, depth, config
)
if do_truthsocial:
truthsocial_future = executor.submit(
_search_truthsocial, topic, from_date, to_date, depth, config
)
if do_polymarket:
if progress:
progress.start_polymarket()
@@ -1004,6 +1213,21 @@ def run_research(
if progress:
progress.end_instagram(len(instagram_items))
if xiaohongshu_future:
try:
xhs_items, xiaohongshu_error = xiaohongshu_future.result(timeout=future_timeout)
web_items.extend(xhs_items)
if xiaohongshu_error and progress:
progress.show_error(f"Xiaohongshu error: {xiaohongshu_error}")
except TimeoutError:
xiaohongshu_error = f"Xiaohongshu search timed out after {future_timeout}s"
if progress:
progress.show_error(xiaohongshu_error)
except Exception as e:
xiaohongshu_error = f"{type(e).__name__}: {e}"
if progress:
progress.show_error(f"Xiaohongshu error: {e}")
if hackernews_future:
hn_timeout = timeouts.get("hackernews_future", future_timeout)
try:
@@ -1021,6 +1245,36 @@ def run_research(
if progress:
progress.end_hackernews(len(hackernews_items))
if bluesky_future:
bsky_timeout = timeouts.get("bluesky_future", future_timeout)
try:
bluesky_items, bluesky_error = bluesky_future.result(timeout=bsky_timeout)
if bluesky_error and progress:
progress.show_error(f"Bluesky error: {bluesky_error}")
except TimeoutError:
bluesky_error = f"Bluesky search timed out after {bsky_timeout}s"
if progress:
progress.show_error(bluesky_error)
except Exception as e:
bluesky_error = f"{type(e).__name__}: {e}"
if progress:
progress.show_error(f"Bluesky error: {e}")
if truthsocial_future:
ts_timeout = timeouts.get("truthsocial_future", future_timeout)
try:
truthsocial_items, truthsocial_error = truthsocial_future.result(timeout=ts_timeout)
if truthsocial_error and progress:
progress.show_error(f"Truth Social error: {truthsocial_error}")
except TimeoutError:
truthsocial_error = f"Truth Social search timed out after {ts_timeout}s"
if progress:
progress.show_error(truthsocial_error)
except Exception as e:
truthsocial_error = f"{type(e).__name__}: {e}"
if progress:
progress.show_error(f"Truth Social error: {e}")
if polymarket_future:
pm_timeout = timeouts.get("polymarket_future", future_timeout)
try:
@@ -1155,7 +1409,7 @@ def run_research(
if sup_x:
x_items.extend(sup_x)
return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error
return reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, bluesky_items, truthsocial_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, bluesky_error, truthsocial_error, polymarket_error, web_error
def main():
@@ -1250,6 +1504,13 @@ def main():
default=False,
help="Skip native web search backends (Parallel/Brave/OpenRouter). Use when the assistant has its own WebSearch tool.",
)
parser.add_argument(
"--save-dir",
type=str,
default=None,
metavar="DIR",
help="Auto-save raw research output to DIR/{topic-slug}.md",
)
args = parser.parse_args()
args.topic = " ".join(args.topic) if args.topic else None
@@ -1280,12 +1541,29 @@ def main():
# Load config
config = env.get_config()
# Inject .env credentials into Bird module before auth check
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
# Detect first run (no SETUP_COMPLETE in config)
first_run = setup_wizard.is_first_run(config)
# On first run, block Bird's Node.js sweet-cookie scanner from probing
# browser cookies before the user has given consent via the setup wizard.
# Explicit AUTH_TOKEN (from env var) is fine — only block browser scanning.
if first_run and config.get('_AUTH_TOKEN_SOURCE') != 'env':
os.environ['BIRD_DISABLE_BROWSER_COOKIES'] = '1'
# Inject .env credentials into Bird module before auth check.
# On first run (no SETUP_COMPLETE), only inject explicit env var credentials —
# skip if AUTH_TOKEN came from browser cookies (no consent yet).
if first_run:
auth_source = config.get('_AUTH_TOKEN_SOURCE')
if auth_source == 'env':
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
else:
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
# Auto-detect Bird (no prompts - just use it if available)
x_source_status = env.get_x_source_status(config)
x_source = x_source_status["source"] # 'bird', 'xai', or None
x_method = x_source_status.get("method") # 'env', 'browser-firefox', 'api', etc.
# Auto-detect yt-dlp for YouTube search
has_ytdlp = env.is_ytdlp_available()
@@ -1296,22 +1574,38 @@ def main():
# Auto-detect ScrapeCreators for Instagram
has_instagram = env.is_instagram_available(config)
# Auto-detect Xiaohongshu HTTP API (requires service + login)
has_xiaohongshu = env.is_xiaohongshu_available(config)
# Auto-detect Bluesky (requires BSKY_HANDLE + BSKY_APP_PASSWORD)
has_bluesky = env.is_bluesky_available(config)
# Auto-detect Truth Social (requires TRUTHSOCIAL_TOKEN)
has_truthsocial = env.is_truthsocial_available(config)
# --diagnose: show source availability and exit
if args.diagnose:
web_source = env.get_web_search_source(config)
diag = {
"openai": bool(config.get("OPENAI_API_KEY")),
"reddit_public": True,
"xai": bool(config.get("XAI_API_KEY")),
"x_source": x_source_status["source"],
"x_method": x_source_status.get("method"),
"bird_installed": x_source_status["bird_installed"],
"bird_authenticated": x_source_status["bird_authenticated"],
"bird_username": x_source_status.get("bird_username"),
"youtube": has_ytdlp,
"tiktok": has_tiktok,
"instagram": has_instagram,
"xiaohongshu": has_xiaohongshu,
"xiaohongshu_api_base": env.get_xiaohongshu_api_base(config),
"hackernews": True,
"bluesky": has_bluesky,
"truthsocial": has_truthsocial,
"polymarket": True,
"web_search_backend": web_source,
"exa": bool(config.get("EXA_API_KEY")),
"parallel_ai": bool(config.get("PARALLEL_API_KEY")),
"brave": bool(config.get("BRAVE_API_KEY")),
"openrouter": bool(config.get("OPENROUTER_API_KEY")),
@@ -1319,6 +1613,19 @@ def main():
print(json.dumps(diag, indent=2))
sys.exit(0)
# Handle 'setup' subcommand
if args.topic and args.topic.strip().lower() == "setup":
results = setup_wizard.run_auto_setup(config)
# Write config
env_path = env.CONFIG_FILE
if env_path:
written = setup_wizard.write_setup_config(env_path)
results["env_written"] = written
else:
results["env_written"] = False
print(setup_wizard.get_setup_status_text(results))
sys.exit(0)
# Validate topic (--diagnose doesn't need one)
if not args.topic:
print("Error: Please provide a topic to research.", file=sys.stderr)
@@ -1328,35 +1635,30 @@ def main():
# Initialize progress display with topic
progress = ui.ProgressDisplay(args.topic, show_banner=True)
# Show diagnostic banner when sources are missing
# Show status banner (free-first design — lead with what works)
web_source = env.get_web_search_source(config)
reddit_source = env.get_reddit_source(config)
diag = {
"openai": bool(config.get("OPENAI_API_KEY")),
"xai": bool(config.get("XAI_API_KEY")),
"setup_complete": bool(config.get("SETUP_COMPLETE")),
"reddit_source": reddit_source, # 'scrapecreators', 'openai', or None
"x_source": x_source_status["source"],
"bird_installed": x_source_status["bird_installed"],
"bird_authenticated": x_source_status["bird_authenticated"],
"bird_username": x_source_status.get("bird_username"),
"x_method": x_source_status.get("method"),
"youtube": has_ytdlp,
"tiktok": has_tiktok,
"instagram": has_instagram,
"hackernews": True,
"polymarket": True,
"bluesky": has_bluesky,
"truthsocial": has_truthsocial,
"xiaohongshu": has_xiaohongshu,
"scrapecreators": bool(config.get("SCRAPECREATORS_API_KEY")),
"web_search_backend": "deferred to assistant" if args.no_native_web else web_source,
}
ui.show_diagnostic_banner(diag)
# Check available sources (accounting for Bird auto-detection)
# Check available sources (now accounts for Bird/cookie auth automatically)
available = env.get_available_sources(config)
# Override available if Bird is ready
if x_source == 'bird':
if available == 'reddit':
available = 'both' # Now have both Reddit + X (via Bird)
elif available == 'reddit-web':
available = 'all' # Reddit + X (via Bird) + Web
elif available == 'web':
available = 'x-web' # X via Bird + Web
# Mock mode can work without keys
if args.mock:
if args.sources == "auto":
@@ -1419,21 +1721,33 @@ def main():
else:
mode = sources
# Detect query type for source tiering and scoring adjustments
query_type = qt.detect_query_type(args.topic)
# Apply --search flag: restrict sources to the specified subset
search_do_hackernews = True
search_do_polymarket = True
search_run_youtube = has_ytdlp
search_run_tiktok = has_tiktok
search_run_instagram = has_instagram
# Source defaults are query-type-aware (Truth Social always opt-in,
# Bluesky only for query types where it adds signal)
search_do_hackernews = qt.is_source_enabled("hn", query_type) if not args.search else True
search_do_bluesky = has_bluesky and qt.is_source_enabled("bluesky", query_type)
search_do_truthsocial = False # Always opt-in (requires --search truthsocial)
search_do_polymarket = qt.is_source_enabled("polymarket", query_type)
search_run_youtube = has_ytdlp and qt.is_source_enabled("youtube", query_type)
search_run_tiktok = has_tiktok and qt.is_source_enabled("tiktok", query_type)
search_run_instagram = has_instagram and qt.is_source_enabled("instagram", query_type)
search_run_xiaohongshu = has_xiaohongshu
if args.search:
search_sources = parse_search_flag(args.search)
has_reddit = "reddit" in search_sources
has_x = "x" in search_sources
search_do_hackernews = "hn" in search_sources
search_do_bluesky = ("bluesky" in search_sources or "bsky" in search_sources) and has_bluesky
search_do_truthsocial = ("truthsocial" in search_sources or "truth" in search_sources) and has_truthsocial
search_do_polymarket = "polymarket" in search_sources
search_run_youtube = "youtube" in search_sources and has_ytdlp
search_run_tiktok = "tiktok" in search_sources and has_tiktok
search_run_instagram = "instagram" in search_sources and has_instagram
# If explicitly requested, attempt Xiaohongshu even when preflight says unavailable.
search_run_xiaohongshu = "xiaohongshu" in search_sources
include_search_web = "web" in search_sources
# Map to existing sources string
if has_reddit and has_x:
@@ -1447,7 +1761,7 @@ def main():
sources = "web" # hn/polymarket only; no Reddit/X
# Run research
reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, polymarket_error, web_error = run_research(
reddit_items, x_items, youtube_items, tiktok_items, instagram_items, hackernews_items, bluesky_items, truthsocial_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, tiktok_error, instagram_error, hackernews_error, bluesky_error, truthsocial_error, polymarket_error, web_error = run_research(
args.topic,
sources,
config,
@@ -1461,9 +1775,12 @@ def main():
run_youtube=search_run_youtube,
run_tiktok=search_run_tiktok,
run_instagram=search_run_instagram,
run_xiaohongshu=search_run_xiaohongshu,
timeouts=timeouts,
resolved_handle=args.x_handle,
do_hackernews=search_do_hackernews,
do_bluesky=search_do_bluesky,
do_truthsocial=search_do_truthsocial,
do_polymarket=search_do_polymarket,
no_native_web=args.no_native_web,
)
@@ -1478,6 +1795,8 @@ def main():
normalized_tiktok = normalize.normalize_tiktok_items(tiktok_items, from_date, to_date) if tiktok_items else []
normalized_ig = normalize.normalize_instagram_items(instagram_items, from_date, to_date) if instagram_items else []
normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
normalized_bsky = normalize.normalize_bluesky_items(bluesky_items, from_date, to_date) if bluesky_items else []
normalized_ts = normalize.normalize_truthsocial_items(truthsocial_items, from_date, to_date) if truthsocial_items else []
normalized_pm = normalize.normalize_polymarket_items(polymarket_items, from_date, to_date) if polymarket_items else []
normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
@@ -1494,6 +1813,8 @@ def main():
# Instagram: hard date filter (instagram.py already pre-filters, but safety net)
filtered_ig = normalize.filter_by_date_range(normalized_ig, from_date, to_date) if normalized_ig else []
filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
filtered_bsky = normalize.filter_by_date_range(normalized_bsky, from_date, to_date) if normalized_bsky else []
filtered_ts = normalize.filter_by_date_range(normalized_ts, from_date, to_date) if normalized_ts else []
# Polymarket: skip hard date filter - markets are active/traded, updatedAt is fine
filtered_pm = normalized_pm
filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else []
@@ -1505,18 +1826,22 @@ def main():
scored_tiktok = score.score_tiktok_items(filtered_tiktok) if filtered_tiktok else []
scored_ig = score.score_instagram_items(filtered_ig) if filtered_ig else []
scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
scored_bsky = score.score_bluesky_items(filtered_bsky) if filtered_bsky else []
scored_ts = score.score_truthsocial_items(filtered_ts) if filtered_ts else []
scored_pm = score.score_polymarket_items(filtered_pm) if filtered_pm else []
scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
scored_web = score.score_websearch_items(filtered_web, query_type=query_type) if filtered_web else []
# Sort items
sorted_reddit = score.sort_items(scored_reddit)
sorted_x = score.sort_items(scored_x)
sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
sorted_tiktok = score.sort_items(scored_tiktok) if scored_tiktok else []
sorted_ig = score.sort_items(scored_ig) if scored_ig else []
sorted_hn = score.sort_items(scored_hn) if scored_hn else []
sorted_pm = score.sort_items(scored_pm) if scored_pm else []
sorted_web = score.sort_items(scored_web) if scored_web else []
# Sort items (query-type-aware tiebreaker ordering)
sorted_reddit = score.sort_items(scored_reddit, query_type=query_type)
sorted_x = score.sort_items(scored_x, query_type=query_type)
sorted_youtube = score.sort_items(scored_youtube, query_type=query_type) if scored_youtube else []
sorted_tiktok = score.sort_items(scored_tiktok, query_type=query_type) if scored_tiktok else []
sorted_ig = score.sort_items(scored_ig, query_type=query_type) if scored_ig else []
sorted_hn = score.sort_items(scored_hn, query_type=query_type) if scored_hn else []
sorted_bsky = score.sort_items(scored_bsky, query_type=query_type) if scored_bsky else []
sorted_ts = score.sort_items(scored_ts, query_type=query_type) if scored_ts else []
sorted_pm = score.sort_items(scored_pm, query_type=query_type) if scored_pm else []
sorted_web = score.sort_items(scored_web, query_type=query_type) if scored_web else []
# Dedupe items
deduped_reddit = dedupe.dedupe_reddit(sorted_reddit)
@@ -1525,19 +1850,25 @@ def main():
deduped_tiktok = dedupe.dedupe_tiktok(sorted_tiktok) if sorted_tiktok else []
deduped_ig = dedupe.dedupe_instagram(sorted_ig) if sorted_ig else []
deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
deduped_bsky = dedupe.dedupe_bluesky(sorted_bsky) if sorted_bsky else []
deduped_ts = dedupe.dedupe_truthsocial(sorted_ts) if sorted_ts else []
deduped_pm = dedupe.dedupe_polymarket(sorted_pm) if sorted_pm else []
deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
# Minimum result guarantee: if all Reddit results were filtered out but
# we had raw results, keep top 3 by relevance regardless of score
if not deduped_reddit and normalized_reddit:
print("[REDDIT WARNING] All results scored below threshold, keeping top 3 by relevance", file=sys.stderr)
by_relevance = sorted(normalized_reddit, key=lambda item: item.relevance, reverse=True)
deduped_reddit = by_relevance[:3]
# Post-retrieval relevance filter: drop low-relevance items per source
deduped_reddit = score.relevance_filter(deduped_reddit, "REDDIT")
deduped_x = score.relevance_filter(deduped_x, "X")
deduped_youtube = score.relevance_filter(deduped_youtube, "YOUTUBE")
deduped_tiktok = score.relevance_filter(deduped_tiktok, "TIKTOK")
deduped_ig = score.relevance_filter(deduped_ig, "INSTAGRAM")
deduped_hn = score.relevance_filter(deduped_hn, "HN")
deduped_bsky = score.relevance_filter(deduped_bsky, "BLUESKY")
deduped_ts = score.relevance_filter(deduped_ts, "TRUTHSOCIAL")
deduped_pm = score.relevance_filter(deduped_pm, "POLYMARKET") if deduped_pm else []
# Cross-source linking: annotate items that discuss the same story
dedupe.cross_source_link(
deduped_reddit, deduped_x, deduped_youtube, deduped_tiktok, deduped_ig, deduped_hn, deduped_pm, deduped_web,
deduped_reddit, deduped_x, deduped_youtube, deduped_tiktok, deduped_ig, deduped_hn, deduped_bsky, deduped_ts, deduped_pm, deduped_web,
)
progress.end_processing()
@@ -1557,6 +1888,8 @@ def main():
report.tiktok = deduped_tiktok
report.instagram = deduped_ig
report.hackernews = deduped_hn
report.bluesky = deduped_bsky
report.truthsocial = deduped_ts
report.polymarket = deduped_pm
report.web = deduped_web
report.reddit_error = reddit_error
@@ -1565,6 +1898,8 @@ def main():
report.tiktok_error = tiktok_error
report.instagram_error = instagram_error
report.hackernews_error = hackernews_error
report.bluesky_error = bluesky_error
report.truthsocial_error = truthsocial_error
report.polymarket_error = polymarket_error
report.web_error = web_error
report.resolved_x_handle = args.x_handle
@@ -1583,26 +1918,53 @@ def main():
# Build source info for status footer
source_info = {}
if not bool(config.get("OPENAI_API_KEY")):
source_info["reddit_skip_reason"] = "No OPENAI_API_KEY (add to ~/.config/last30days/.env)"
if not x_source:
if x_source_status["bird_installed"]:
source_info["x_skip_reason"] = "Bird installed but not authenticated — log into x.com in browser"
else:
source_info["x_skip_reason"] = "No Bird CLI or XAI_API_KEY (Node.js 22+ needed for Bird)"
source_info["x_skip_reason"] = "No Bird CLI, XAI_API_KEY, or SCRAPECREATORS_API_KEY"
if not has_ytdlp:
source_info["youtube_skip_reason"] = "yt-dlp not installed — fix: brew install yt-dlp"
elif has_ytdlp and not report.youtube:
source_info["youtube_skip_reason"] = "0 results (query may be too specific)"
if not has_tiktok:
source_info["tiktok_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
source_info["tiktok_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free API calls, no credit card)"
if not has_instagram:
source_info["instagram_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free credits)"
source_info["instagram_skip_reason"] = "No SCRAPECREATORS_API_KEY - sign up at scrapecreators.com (100 free API calls, no credit card)"
if not has_xiaohongshu:
source_info["xiaohongshu_skip_reason"] = (
f"Xiaohongshu API unavailable or not logged in - start xiaohongshu-mcp and login "
f"(base: {env.get_xiaohongshu_api_base(config)})"
)
if not web_source:
source_info["web_skip_reason"] = "assistant will use WebSearch (add BRAVE_API_KEY for native search)"
# Compute quality score and upgrade nudge
research_results = {
"x_error": x_error,
"youtube_error": youtube_error,
"reddit_error": reddit_error,
}
quality = quality_nudge.compute_quality_score(config, research_results)
# Output result
output_result(report, args.emit, web_needed, args.topic, from_date, to_date, missing_keys, args.days, source_info)
output_result(report, args.emit, web_needed, args.topic, from_date, to_date, missing_keys, args.days, source_info, first_run=first_run, quality=quality)
# Auto-save raw research to file if --save-dir is set
if args.save_dir:
import re
save_dir = Path(args.save_dir).expanduser()
save_dir.mkdir(parents=True, exist_ok=True)
slug = re.sub(r'[^a-z0-9]+', '-', args.topic.lower()).strip('-')[:60]
save_path = save_dir / f"{slug}-raw.md"
if save_path.exists():
save_path = save_dir / f"{slug}-raw-{datetime.now().strftime('%Y-%m-%d')}.md"
content = render.render_compact(report, missing_keys=missing_keys)
if quality and quality.get("nudge_text"):
content += "\n" + render.render_quality_nudge(quality)
content += "\n" + render.render_source_status(report, source_info)
save_path.write_text(content, encoding="utf-8")
print(f"📎 {save_path}", file=sys.stderr)
# Persist findings to SQLite if requested
if args.store:
@@ -1653,6 +2015,16 @@ def main():
"engagement_score": item.engagement.score if item.engagement else 0,
"relevance_score": item.relevance,
})
for item in deduped_bsky:
findings.append({
"source": "bluesky",
"url": item.url,
"title": item.text[:100],
"author": item.author_handle,
"content": item.text,
"engagement_score": item.engagement.likes if item.engagement else 0,
"relevance_score": item.relevance,
})
for item in deduped_pm:
findings.append({
"source": "polymarket",
@@ -1707,10 +2079,19 @@ def output_result(
missing_keys: str = "none",
days: int = 30,
source_info: dict = None,
first_run: bool = False,
quality: dict = None,
):
"""Output the result based on emit mode."""
if emit_mode == "compact":
# Emit first-run flag before research output so SKILL.md can detect it
if first_run:
print("FIRST_RUN: true")
print("")
print(render.render_compact(report, missing_keys=missing_keys))
# Quality nudge (right before source status/stats block)
if quality and quality.get("nudge_text"):
print(render.render_quality_nudge(quality))
# Append source status footer
print(render.render_source_status(report, source_info))
elif emit_mode == "json":
+38 -56
View File
@@ -14,6 +14,8 @@ from pathlib import Path
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
from .relevance import token_overlap_relevance as _compute_relevance
# Path to the vendored bird-search wrapper
_BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
@@ -36,10 +38,19 @@ def set_credentials(auth_token: Optional[str], ct0: Optional[str]):
_credentials['CT0'] = ct0
def _has_injected_credentials() -> bool:
"""Return True when both X session cookies were injected from config."""
return bool(_credentials.get('AUTH_TOKEN') and _credentials.get('CT0'))
def _subprocess_env() -> Dict[str, str]:
"""Build env dict for Node subprocesses, merging injected credentials."""
env = os.environ.copy()
env.update(_credentials)
# When repo config already provides cookies, disable browser-cookie fallback
# so vendored Bird never hits Safari/Chrome keychain during automation.
if _has_injected_credentials():
env.setdefault("BIRD_DISABLE_BROWSER_COOKIES", "1")
return env
@@ -54,56 +65,10 @@ def _extract_core_subject(topic: str) -> str:
X search is literal keyword AND matching all words must appear.
Aggressively strip question/meta/research words to keep only the
core product/concept name (2-3 words max).
core product/concept name (max 5 words).
"""
text = topic.lower().strip()
# Phase 1: Strip multi-word prefixes (longest first)
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
break
# Phase 2: Strip multi-word suffixes
suffixes = [
'best practices', 'use cases', 'prompt techniques',
'prompting techniques', 'prompting tips',
]
for s in suffixes:
if text.endswith(' ' + s):
text = text[:-len(s)].strip()
break
# Phase 3: Filter individual noise words
_noise = {
# Question/filler words
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'and', 'or',
'of', 'in', 'on', 'for', 'with', 'about', 'to',
'people', 'saying', 'think', 'said', 'lately',
# Research/meta descriptors
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
'practices', 'features', 'guide', 'tutorial',
'recommendations', 'advice', 'review', 'reviews',
'usecases', 'examples', 'comparison', 'versus', 'vs',
'plugin', 'plugins', 'skill', 'skills', 'tool', 'tools',
# Prompting meta words
'prompt', 'prompts', 'prompting', 'techniques', 'tips',
'tricks', 'methods', 'strategies', 'approaches',
# Action words
'using', 'uses', 'use',
}
words = text.split()
result = [w for w in words if w not in _noise]
return ' '.join(result[:3]) or topic.lower().strip() # Max 3 words
from .query import extract_core_subject
return extract_core_subject(topic, max_words=5, strip_suffixes=True)
def is_bird_installed() -> bool:
@@ -126,6 +91,9 @@ def is_bird_authenticated() -> Optional[str]:
if not is_bird_installed():
return None
if _has_injected_credentials():
return "env AUTH_TOKEN"
try:
result = subprocess.run(
["node", str(_BIRD_SEARCH_MJS), "--whoami"],
@@ -279,16 +247,28 @@ def search_x(
response = _run_bird_search(query, count, timeout)
# Check if we got results
items = parse_bird_response(response)
items = parse_bird_response(response, query=core_topic)
# Retry with fewer keywords if 0 results and query has 3+ words
# Retry with OR groups for multi-word queries (X supports OR operator)
core_words = core_topic.split()
if not items and len(core_words) >= 2:
from .query import extract_compound_terms
compounds = extract_compound_terms(topic)
if compounds:
# Build OR-group query: ("multi-agent" OR "agent simulation") since:DATE
or_parts = ' OR '.join(f'"{t}"' for t in compounds[:3])
_log(f"0 results for '{core_topic}', retrying with OR groups: {or_parts}")
query = f"({or_parts}) since:{from_date}"
response = _run_bird_search(query, count, timeout)
items = parse_bird_response(response, query=core_topic)
# Retry with fewer keywords if still 0 results and query has 3+ words
if not items and len(core_words) > 2:
shorter = ' '.join(core_words[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
response = _run_bird_search(query, count, timeout)
items = parse_bird_response(response)
items = parse_bird_response(response, query=core_topic)
# Last-chance retry: use strongest remaining token (often the product name)
if not items and core_words:
@@ -353,6 +333,7 @@ def search_handles(
stderr=subprocess.PIPE,
text=True,
preexec_fn=preexec,
env=_subprocess_env(),
)
try:
@@ -375,7 +356,7 @@ def search_handles(
continue
response = json.loads(output)
items = parse_bird_response(response)
items = parse_bird_response(response, query=core_topic)
all_items.extend(items)
except json.JSONDecodeError:
@@ -386,11 +367,12 @@ def search_handles(
return all_items
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
def parse_bird_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
query: Original search query for relevance scoring
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
@@ -468,9 +450,9 @@ def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
"relevance": _compute_relevance(query, str(tweet.get("text", ""))) if query else 0.7,
}
items.append(item)
return items
return items
+225
View File
@@ -0,0 +1,225 @@
"""Bluesky search via AT Protocol (requires app password).
Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
"""
import math
import re
import sys
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from . import http
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
# Module-level token cache (valid for the lifetime of a single research run)
_cached_token: Optional[str] = None
_session_error: Optional[str] = None
def _log(msg: str):
"""Log to stderr (only in TTY mode to avoid cluttering Claude Code output)."""
if sys.stderr.isatty():
sys.stderr.write(f"[Bluesky] {msg}\n")
sys.stderr.flush()
def _create_session(handle: str, app_password: str) -> Optional[str]:
"""Create an AT Protocol session and return the access token.
Args:
handle: Bluesky handle (e.g. user.bsky.social)
app_password: App password from bsky.app/settings/app-passwords
Returns:
Access JWT string, or None on failure. Sets _session_error on failure.
"""
global _cached_token, _session_error
if _cached_token:
return _cached_token
try:
response = http.request(
"POST",
BSKY_SESSION_URL,
json_data={"identifier": handle, "password": app_password},
timeout=15,
)
token = response.get("accessJwt")
if token:
_cached_token = token
_session_error = None
_log("Session created successfully")
return token
_log("No accessJwt in session response")
_session_error = "No accessJwt in session response"
return None
except http.HTTPError as e:
if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
_session_error = "Cloudflare blocked the request (403 Forbidden). This is a network-level block, not an auth issue. Try a different network or VPN."
elif e.status_code == 401:
_session_error = "Invalid credentials (401 Unauthorized). Check BSKY_HANDLE and BSKY_APP_PASSWORD."
else:
_session_error = f"Session request failed: {e}"
_log(f"Session creation failed: {_session_error}")
return None
except Exception as e:
_session_error = f"Session request failed: {type(e).__name__}: {e}"
_log(f"Session creation failed: {_session_error}")
return None
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Bluesky search."""
from .query import extract_core_subject
_BSKY_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features', 'recommendations', 'advice',
})
return extract_core_subject(topic, noise=_BSKY_NOISE)
def _parse_date(item: Dict[str, Any]) -> Optional[str]:
"""Parse date from Bluesky post to YYYY-MM-DD.
AT Protocol uses ISO 8601 format in indexedAt and createdAt fields.
"""
for key in ("indexedAt", "createdAt"):
val = item.get(key)
if val and isinstance(val, str):
try:
dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
return None
def search_bluesky(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Search Bluesky via AT Protocol API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
config: Config dict with BSKY_HANDLE and BSKY_APP_PASSWORD
Returns:
Dict with 'posts' list from AT Protocol response.
"""
config = config or {}
handle = config.get("BSKY_HANDLE", "")
app_password = config.get("BSKY_APP_PASSWORD", "")
if not handle or not app_password:
return {"posts": [], "error": "Bluesky credentials not configured"}
# Authenticate
token = _create_session(handle, app_password)
if not token:
error_msg = _session_error or "Bluesky session creation failed (unknown error)"
return {"posts": [], "error": error_msg}
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching for '{core_topic}' (depth={depth}, limit={count})")
from urllib.parse import urlencode
params = {
"q": core_topic,
"limit": str(min(count, 100)),
"sort": "top",
}
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
try:
response = http.request(
"GET", url,
headers={"Authorization": f"Bearer {token}"},
timeout=30,
)
except http.HTTPError as e:
_log(f"Search failed: {e}")
if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
return {"posts": [], "error": "Bluesky search blocked by Cloudflare (403). This is a network-level block - try a different network or VPN."}
return {"posts": [], "error": f"Bluesky search failed: {e}"}
except Exception as e:
_log(f"Search failed: {e}")
return {"posts": [], "error": f"Bluesky search failed: {type(e).__name__}: {e}"}
posts = response.get("posts", [])
_log(f"Found {len(posts)} posts")
return response
def parse_bluesky_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse AT Protocol response into normalized item dicts.
Returns:
List of item dicts ready for normalization.
"""
posts = response.get("posts", [])
items = []
for i, post in enumerate(posts):
record = post.get("record") or {}
text = record.get("text") or ""
author = post.get("author") or {}
handle = author.get("handle") or ""
display_name = author.get("displayName") or handle
# Post URI -> URL
# URI format: at://did:plc:xxx/app.bsky.feed.post/rkey
uri = post.get("uri") or ""
rkey = uri.rsplit("/", 1)[-1] if uri else ""
url = f"https://bsky.app/profile/{handle}/post/{rkey}" if handle and rkey else ""
likes = post.get("likeCount") or 0
reposts = post.get("repostCount") or 0
replies = post.get("replyCount") or 0
quotes = post.get("quoteCount") or 0
date_str = _parse_date(post) or _parse_date(record)
# Relevance: position-based (AT Protocol sorts by relevance with sort=top)
rank_score = max(0.3, 1.0 - (i * 0.02))
engagement_boost = min(0.2, math.log1p(likes + reposts) / 40)
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"handle": handle,
"display_name": display_name,
"text": text,
"url": url,
"date": date_str,
"engagement": {
"likes": likes,
"reposts": reposts,
"replies": replies,
"quotes": quotes,
},
"relevance": round(relevance, 2),
"why_relevant": f"Bluesky: @{handle}: {text[:60]}" if text else f"Bluesky: {handle}",
})
return items
+119 -3
View File
@@ -1,7 +1,12 @@
"""Brave Search web search for last30days skill.
Uses the Brave Search API as a fallback web search backend.
Simple, cheap (free tier: 2,000 queries/month), widely available.
Uses the Brave Search API as a web search backend.
Requires a paid Brave Search subscription.
Two modes:
- Standard: /res/v1/web/search returns URLs + snippets (default)
- LLM Context: /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption. Enable with BRAVE_LLM_CONTEXT=1 env var.
API docs: https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
"""
@@ -16,6 +21,7 @@ from urllib.parse import urlencode, urlparse
from . import http
ENDPOINT = "https://api.search.brave.com/res/v1/web/search"
LLM_CONTEXT_ENDPOINT = "https://api.search.brave.com/res/v1/llm/context"
# Freshness codes: pd=24h, pw=7d, pm=31d
FRESHNESS_MAP = {1: "pd", 7: "pw", 31: "pm"}
@@ -33,6 +39,7 @@ def search_web(
to_date: str,
api_key: str,
depth: str = "default",
use_llm_context: bool = False,
) -> List[Dict[str, Any]]:
"""Search the web via Brave Search API.
@@ -42,6 +49,7 @@ def search_web(
to_date: End date (YYYY-MM-DD)
api_key: Brave Search API key
depth: 'quick', 'default', or 'deep'
use_llm_context: Use LLM Context endpoint for pre-extracted content
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
@@ -49,6 +57,9 @@ def search_web(
Raises:
http.HTTPError: On API errors
"""
if use_llm_context:
return _search_llm_context(topic, from_date, to_date, api_key, depth)
count = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
# Calculate days for freshness filter
@@ -81,6 +92,48 @@ def search_web(
return _normalize_results(response, from_date, to_date)
def _search_llm_context(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search via Brave LLM Context endpoint for pre-extracted web content.
Returns results in the same schema as search_web() for downstream compatibility.
Snippets contain actual page content instead of short descriptions.
"""
count = {"quick": 5, "default": 20, "deep": 50}.get(depth, 20)
max_tokens = {"quick": 2048, "default": 8192, "deep": 16384}.get(depth, 8192)
days = _days_between(from_date, to_date)
freshness = _brave_freshness(days)
params = {
"q": topic,
"count": count,
"maximum_number_of_tokens": max_tokens,
"context_threshold_mode": "balanced",
}
if freshness:
params["freshness"] = freshness
url = f"{LLM_CONTEXT_ENDPOINT}?{urlencode(params)}"
sys.stderr.write(f"[Web] Searching Brave LLM Context for: {topic}\n")
sys.stderr.flush()
response = http.request(
"GET",
url,
headers={"X-Subscription-Token": api_key},
timeout=30,
)
return _normalize_llm_context(response)
def _days_between(from_date: str, to_date: str) -> int:
"""Calculate days between two YYYY-MM-DD dates."""
try:
@@ -138,7 +191,7 @@ def _normalize_results(
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
except (ValueError, TypeError):
domain = ""
title = _clean_html(str(result.get("title", "")).strip())
@@ -169,6 +222,69 @@ def _normalize_results(
return items
def _normalize_llm_context(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Brave LLM Context response to websearch item schema.
LLM Context returns grounding.generic[] with url, title, snippets[].
Sources metadata provides hostname and age for each URL.
"""
items = []
grounding = response.get("grounding", {})
sources = response.get("sources", {})
for i, result in enumerate(grounding.get("generic", [])):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except (ValueError, TypeError):
domain = ""
title = str(result.get("title", "")).strip()
snippets = result.get("snippets", [])
snippet = "\n".join(str(s).strip() for s in snippets if s)
if not title and not snippet:
continue
# Parse date from sources metadata
source_meta = sources.get(url, {})
age_list = source_meta.get("age") or []
date = None
for age_str in age_list:
date = _parse_brave_date(age_str, None)
if date:
break
date_confidence = "med" if date else "low"
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": source_meta.get("hostname", domain),
"snippet": snippet[:1500], # LLM Context returns richer content
"date": date,
"date_confidence": date_confidence,
"relevance": 0.7, # LLM Context pre-filters for relevance
"why_relevant": "",
})
sys.stderr.write(f"[Web] Brave LLM Context: {len(items)} results\n")
sys.stderr.flush()
return items
def _clean_html(text: str) -> str:
"""Remove HTML tags and decode entities."""
text = re.sub(r"<[^>]*>", "", text)
+265
View File
@@ -0,0 +1,265 @@
"""Chrome cookie extraction for macOS.
Extracts cookies from Chrome's encrypted SQLite database using only stdlib
modules and the system openssl CLI (ships with macOS). Zero pip dependencies.
Chrome on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
This is NOT affected by Windows App-Bound Encryption (v20).
"""
import hashlib
import logging
import shutil
import sqlite3
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Chrome cookie DB location on macOS
CHROME_COOKIES_DB = Path.home() / "Library" / "Application Support" / "Google" / "Chrome" / "Default" / "Cookies"
# Chrome v10 encryption constants
CHROME_SALT = b"saltysalt"
CHROME_PBKDF2_ITERATIONS = 1003
CHROME_KEY_LENGTH = 16
# IV is 16 space characters (0x20)
CHROME_IV_HEX = "20" * 16
def _get_chrome_encryption_key() -> Optional[bytes]:
"""Retrieve Chrome's encryption passphrase from macOS Keychain.
Calls `security find-generic-password` which may trigger a system dialog
on first access.
Returns the raw passphrase bytes, or None on failure.
"""
try:
result = subprocess.run(
["security", "find-generic-password", "-w", "-s", "Chrome Safe Storage"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
logger.info("Chrome Keychain access denied or Chrome not installed: %s", result.stderr.strip())
return None
passphrase = result.stdout.strip()
if not passphrase:
logger.info("Chrome Keychain returned empty passphrase")
return None
return passphrase.encode("utf-8")
except FileNotFoundError:
logger.info("'security' command not found — not on macOS?")
return None
except subprocess.TimeoutExpired:
logger.info("Chrome Keychain access timed out")
return None
except Exception as e:
logger.info("Failed to get Chrome encryption key: %s", e)
return None
def _derive_aes_key(passphrase: bytes) -> bytes:
"""Derive 16-byte AES key from Chrome's Keychain passphrase via PBKDF2."""
return hashlib.pbkdf2_hmac(
"sha1",
passphrase,
CHROME_SALT,
CHROME_PBKDF2_ITERATIONS,
dklen=CHROME_KEY_LENGTH,
)
def _decrypt_v10_value(encrypted_value: bytes, aes_key: bytes, db_version: int) -> Optional[str]:
"""Decrypt a Chrome v10-encrypted cookie value.
Uses system openssl CLI for AES-128-CBC decryption (zero pip deps).
For Chrome 130+ (db_version >= 24), strips 32-byte SHA-256 prefix after decryption.
Returns decrypted string or None on failure.
"""
# Strip the 'v10' prefix
ciphertext = encrypted_value[3:]
if not ciphertext:
return None
hex_key = aes_key.hex()
try:
result = subprocess.run(
[
"openssl", "enc", "-aes-128-cbc", "-d",
"-K", hex_key,
"-iv", CHROME_IV_HEX,
"-nopad",
],
input=ciphertext,
capture_output=True,
timeout=5,
)
if result.returncode != 0:
logger.debug("openssl decryption failed: %s", result.stderr.decode(errors="replace").strip())
return None
decrypted = result.stdout
if not decrypted:
return None
# Remove PKCS7 padding
decrypted = _remove_pkcs7_padding(decrypted)
if decrypted is None:
return None
# Chrome 130+ (db version >= 24): strip 32-byte SHA-256 prefix
if db_version >= 24 and len(decrypted) > 32:
decrypted = decrypted[32:]
return decrypted.decode("utf-8", errors="replace")
except FileNotFoundError:
logger.info("openssl not found — cannot decrypt Chrome cookies")
return None
except subprocess.TimeoutExpired:
logger.info("openssl decryption timed out")
return None
except Exception as e:
logger.debug("Chrome cookie decryption error: %s", e)
return None
def _remove_pkcs7_padding(data: bytes) -> Optional[bytes]:
"""Remove PKCS7 padding from decrypted data.
The last byte indicates the number of padding bytes added.
All padding bytes must have the same value.
Returns unpadded data or None if padding is invalid.
"""
if not data:
return None
pad_len = data[-1]
if pad_len < 1 or pad_len > 16:
return None
# Verify all padding bytes match
if data[-pad_len:] != bytes([pad_len]) * pad_len:
return None
return data[:-pad_len]
def _get_db_version(cursor: sqlite3.Cursor) -> int:
"""Get Chrome cookie database version from the meta table.
Returns 0 if meta table doesn't exist or version can't be read.
"""
try:
cursor.execute("SELECT value FROM meta WHERE key = 'version'")
row = cursor.fetchone()
if row:
return int(row[0])
except Exception:
pass
return 0
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS.
Copies the locked Cookies database to a temp file, reads specified cookies,
and decrypts v10-encrypted values using the Keychain-stored key.
Args:
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com")
cookie_names: List of cookie names to extract
Returns:
Dict mapping cookie name to decrypted value, or None on failure.
Only includes cookies that were successfully found and decrypted.
"""
if not CHROME_COOKIES_DB.exists():
logger.info("Chrome cookies database not found at %s", CHROME_COOKIES_DB)
return None
# Get encryption key from Keychain
passphrase = _get_chrome_encryption_key()
aes_key = _derive_aes_key(passphrase) if passphrase else None
# Copy DB to temp file (Chrome locks the original)
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(CHROME_COOKIES_DB), tmp_path)
except Exception as e:
logger.info("Failed to copy Chrome cookies database: %s", e)
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
return None
finally:
if tmp_fd is not None:
import os
os.close(tmp_fd)
try:
conn = sqlite3.connect(tmp_path)
cursor = conn.cursor()
db_version = _get_db_version(cursor)
logger.debug("Chrome cookie DB version: %d", db_version)
# Build query with placeholders for cookie names
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value, encrypted_value FROM cookies "
f"WHERE host_key LIKE ? AND name IN ({placeholders})"
)
# Use LIKE for domain matching (e.g., %.twitter.com matches .twitter.com)
params = [f"%{domain}"] + list(cookie_names)
cursor.execute(query, params)
results: dict[str, str] = {}
for name, value, encrypted_value in cursor.fetchall():
# Prefer unencrypted value if present
if value:
results[name] = value
continue
# Handle encrypted value
if encrypted_value and encrypted_value[:3] == b"v10":
if aes_key is None:
logger.debug("Skipping encrypted cookie %s — no Keychain access", name)
continue
decrypted = _decrypt_v10_value(encrypted_value, aes_key, db_version)
if decrypted:
results[name] = decrypted
else:
logger.debug("Failed to decrypt cookie %s", name)
elif encrypted_value:
# Unknown encryption version
logger.debug("Unknown encryption for cookie %s (prefix: %r)", name, encrypted_value[:3])
conn.close()
if not results:
logger.info("No matching cookies found in Chrome for domain %s", domain)
return None
return results
except sqlite3.Error as e:
logger.info("Failed to read Chrome cookies database: %s", e)
return None
except Exception as e:
logger.info("Unexpected error reading Chrome cookies: %s", e)
return None
finally:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
+295
View File
@@ -0,0 +1,295 @@
"""Browser cookie extraction for last30days.
Extracts cookies from local browser databases (Firefox, Chrome, Safari)
to enable zero-config authentication for services like X/Twitter.
Only uses Python stdlib no external dependencies.
"""
import configparser
import logging
import platform
import shutil
import sqlite3
import tempfile
from pathlib import Path
from typing import Dict, List, Optional
logger = logging.getLogger(__name__)
def _get_firefox_profiles_dir() -> Optional[Path]:
"""Return the Firefox profiles directory for the current platform, or None."""
system = platform.system()
if system == "Darwin":
path = Path.home() / "Library" / "Application Support" / "Firefox"
elif system == "Linux":
path = Path.home() / ".mozilla" / "firefox"
else:
# Windows: %APPDATA%\Mozilla\Firefox — best-effort
appdata = Path.home() / "AppData" / "Roaming" / "Mozilla" / "Firefox"
path = appdata
return path if path.is_dir() else None
def _find_default_profile(profiles_dir: Path) -> Optional[Path]:
"""Parse profiles.ini to find the default profile directory.
Looks for a section with Default=1. Falls back to the first profile
directory found on disk if profiles.ini is missing or malformed.
"""
ini_path = profiles_dir / "profiles.ini"
if ini_path.is_file():
try:
config = configparser.ConfigParser()
config.read(str(ini_path), encoding="utf-8")
# First pass: look for Default=1
for section in config.sections():
if config.has_option(section, "Default") and config.get(section, "Default") == "1":
return _resolve_profile_path(profiles_dir, config, section)
# Second pass: first Install* section with Default key (Firefox >= 67 format)
for section in config.sections():
if section.startswith("Install") and config.has_option(section, "Default"):
raw = config.get(section, "Default")
candidate = profiles_dir / raw
if candidate.is_dir():
return candidate
# Third pass: first Profile section that exists on disk
for section in config.sections():
if section.startswith("Profile"):
resolved = _resolve_profile_path(profiles_dir, config, section)
if resolved and resolved.is_dir():
return resolved
except (configparser.Error, OSError) as exc:
logger.debug("Failed to parse profiles.ini: %s", exc)
# Fallback: scan directory for anything that looks like a profile
return _fallback_find_profile(profiles_dir)
def _resolve_profile_path(
profiles_dir: Path, config: configparser.ConfigParser, section: str
) -> Optional[Path]:
"""Resolve a profile path from a ConfigParser section."""
if not config.has_option(section, "Path"):
return None
raw_path = config.get(section, "Path")
is_relative = config.has_option(section, "IsRelative") and config.get(section, "IsRelative") == "1"
if is_relative:
candidate = profiles_dir / raw_path
else:
candidate = Path(raw_path)
return candidate if candidate.is_dir() else None
def _fallback_find_profile(profiles_dir: Path) -> Optional[Path]:
"""Find the first directory that contains cookies.sqlite."""
try:
for child in sorted(profiles_dir.iterdir()):
if child.is_dir() and (child / "cookies.sqlite").is_file():
return child
except OSError:
pass
return None
def _query_cookies_db(
db_path: Path, domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Copy the cookies database to a temp file and query it.
Firefox locks cookies.sqlite while running, so we copy first.
Returns {name: value} dict or None if no matching cookies found.
"""
if not db_path.is_file():
return None
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(db_path), tmp_path)
conn = sqlite3.connect(tmp_path)
try:
# Build parameterized query — SQLite doesn't support array params,
# so we build the IN clause with individual placeholders.
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value FROM moz_cookies "
f"WHERE host LIKE ? AND name IN ({placeholders})"
)
# domain pattern: match .x.com, x.com, etc.
domain_pattern = f"%{domain}"
params = [domain_pattern] + list(cookie_names)
cursor = conn.execute(query, params)
rows = cursor.fetchall()
finally:
conn.close()
if not rows:
return None
return {name: value for name, value in rows}
except (sqlite3.Error, OSError) as exc:
logger.debug("Failed to query cookies database %s: %s", db_path, exc)
return None
finally:
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
except OSError:
pass
if tmp_fd is not None:
try:
import os
os.close(tmp_fd)
except OSError:
pass
def extract_firefox_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Firefox for the given domain and cookie names.
Finds the default Firefox profile, copies cookies.sqlite to a temp file
(to avoid lock conflicts), and queries for the requested cookies.
Args:
domain: The cookie domain to match (e.g. ".x.com"). Matched with LIKE %domain.
cookie_names: List of cookie names to extract (e.g. ["auth_token", "ct0"]).
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
profiles_dir = _get_firefox_profiles_dir()
if profiles_dir is None:
logger.debug("Firefox profiles directory not found")
return None
profile_path = _find_default_profile(profiles_dir)
if profile_path is None:
logger.debug("No Firefox profile found in %s", profiles_dir)
return None
db_path = profile_path / "cookies.sqlite"
return _query_cookies_db(db_path, domain, cookie_names)
def extract_chrome_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Chrome for the given domain and cookie names.
macOS only uses Keychain + system openssl for AES-128-CBC decryption.
Linux/Windows not supported (Chrome uses platform-specific encryption).
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Chrome cookie extraction only supported on macOS")
return None
try:
from .chrome_cookies import extract_chrome_cookies_macos
return extract_chrome_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Chrome cookie extraction failed: %s", exc)
return None
def extract_safari_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Safari for the given domain and cookie names.
macOS only parses the unencrypted binary cookie file.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Safari cookie extraction only supported on macOS")
return None
try:
from .safari_cookies import extract_safari_cookies_macos
return extract_safari_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Safari cookie extraction failed: %s", exc)
return None
def extract_cookies(
browser: str, domain: str, cookie_names: list[str]
) -> Optional[dict[str, str]]:
"""Extract cookies from the specified browser.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
'auto' tries browsers in platform-appropriate order:
- macOS: Chrome -> Firefox -> Safari
- Linux: Firefox only
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
result = extract_cookies_with_source(browser, domain, cookie_names)
if result is None:
return None
cookies, _browser_name = result
return cookies
def extract_cookies_with_source(
browser: str, domain: str, cookie_names: list[str]
) -> Optional[tuple[dict[str, str], str]]:
"""Extract cookies and report which browser they came from.
Same as extract_cookies() but returns a (cookies, browser_name) tuple
so callers can track the source.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Tuple of ({cookie_name: cookie_value}, browser_name) or None.
"""
extractors = {
"firefox": extract_firefox_cookies,
"chrome": extract_chrome_cookies,
"safari": extract_safari_cookies,
}
if browser != "auto":
extractor = extractors.get(browser)
if extractor is None:
logger.warning("Unknown browser: %s", browser)
return None
result = extractor(domain, cookie_names)
return (result, browser) if result is not None else None
# Auto mode: try browsers in platform-appropriate order
system = platform.system()
if system == "Darwin":
order = ["chrome", "firefox", "safari"]
elif system == "Linux":
order = ["firefox"]
else:
order = ["firefox"]
for name in order:
result = extractors[name](domain, cookie_names)
if result is not None:
return (result, name)
return None
+16
View File
@@ -226,6 +226,22 @@ def dedupe_hackernews(
return dedupe_items(items, threshold)
def dedupe_bluesky(
items: List[schema.BlueskyItem],
threshold: float = 0.7,
) -> List[schema.BlueskyItem]:
"""Dedupe Bluesky items."""
return dedupe_items(items, threshold)
def dedupe_truthsocial(
items: List[schema.TruthSocialItem],
threshold: float = 0.7,
) -> List[schema.TruthSocialItem]:
"""Dedupe Truth Social items."""
return dedupe_items(items, threshold)
def dedupe_polymarket(
items: List[schema.PolymarketItem],
threshold: float = 0.7,
+387 -74
View File
@@ -2,11 +2,38 @@
import base64
import json
import logging
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, Dict, Any, Literal
from typing import Optional, Dict, Any, List, Literal
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Cookie domain registry: maps source names to browser cookie extraction params.
# Each entry: (domain, cookie_names, config_key_mapping)
# config_key_mapping: {cookie_name: config_key} so we know which config key
# each extracted cookie should populate.
# ---------------------------------------------------------------------------
COOKIE_DOMAINS: Dict[str, Dict[str, Any]] = {
"x": {
"domain": ".x.com",
"cookies": ["auth_token", "ct0"],
"mapping": {
"auth_token": "AUTH_TOKEN",
"ct0": "CT0",
},
},
"truthsocial": {
"domain": ".truthsocial.com",
"cookies": ["_session_id"],
"mapping": {
"_session_id": "TRUTHSOCIAL_TOKEN",
},
},
}
# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
@@ -47,11 +74,28 @@ class OpenAIAuth:
codex_auth_file: str
def _check_file_permissions(path: Path) -> None:
"""Warn to stderr if a secrets file has overly permissive permissions."""
try:
mode = path.stat().st_mode
# Check if group or other can read (bits 0o044)
if mode & 0o044:
import sys
sys.stderr.write(
f"[last30days] WARNING: {path} is readable by other users. "
f"Run: chmod 600 {path}\n"
)
sys.stderr.flush()
except OSError:
pass
def load_env_file(path: Path) -> Dict[str, str]:
"""Load environment variables from a file."""
env = {}
if not path.exists():
if not path or not path.exists():
return env
_check_file_permissions(path)
with open(path, 'r') as f:
for line in f:
@@ -178,14 +222,126 @@ def get_openai_auth(file_env: Dict[str, str]) -> OpenAIAuth:
)
def _find_project_env() -> Optional[Path]:
"""Find per-project .env by walking up from cwd.
Searches for .claude/last30days.env in each parent directory,
stopping at the user's home directory or filesystem root.
"""
cwd = Path.cwd()
for parent in [cwd, *cwd.parents]:
candidate = parent / '.claude' / 'last30days.env'
if candidate.exists():
return candidate
# Stop at filesystem root or home
if parent == Path.home() or parent == parent.parent:
break
return None
def extract_browser_credentials(config: Dict[str, Any]) -> Dict[str, str]:
"""Extract credentials from browser cookies for sources that need them.
Checks the FROM_BROWSER config key to decide whether/how to extract:
- 'auto': try browsers in platform order
- 'firefox', 'chrome', 'safari': try only that browser
- 'off': skip extraction entirely
If SETUP_COMPLETE is not set AND FROM_BROWSER is not explicitly set,
defaults to 'off' (wizard hasn't run yet — no extraction without consent).
If SETUP_COMPLETE is set and FROM_BROWSER is not set, defaults to 'auto'.
Explicit env var/config values always take priority over extracted cookies.
Returns:
Dict of {config_key: value} for credentials discovered from cookies.
"""
setup_complete = config.get("SETUP_COMPLETE")
from_browser = config.get("FROM_BROWSER")
# Determine effective browser setting
if from_browser is None:
if setup_complete:
from_browser = "auto"
else:
from_browser = "off"
from_browser = from_browser.lower().strip() if isinstance(from_browser, str) else "off"
if from_browser == "off":
return {}
# Lazy import to avoid loading cookie_extract at module level
try:
from . import cookie_extract
except Exception:
logger.debug("cookie_extract module not available")
return {}
credentials: Dict[str, str] = {}
for source_name, spec in COOKIE_DOMAINS.items():
domain = spec["domain"]
cookie_names: List[str] = spec["cookies"]
mapping: Dict[str, str] = spec["mapping"]
# Skip if ALL mapped config keys already have values
all_present = all(config.get(config_key) for config_key in mapping.values())
if all_present:
logger.debug(
"Skipping cookie extraction for %s: credentials already set",
source_name,
)
continue
try:
result = cookie_extract.extract_cookies_with_source(from_browser, domain, cookie_names)
except Exception as exc:
logger.debug(
"Cookie extraction failed for %s: %s", source_name, exc
)
continue
if result is None:
continue
cookies, browser_name = result
filled_any = False
for cookie_name, config_key in mapping.items():
# Only fill in keys not already present
if not config.get(config_key) and cookie_name in cookies:
credentials[config_key] = cookies[cookie_name]
filled_any = True
# Track which browser provided the credentials for this source
if filled_any:
credentials[f"__{source_name.upper()}_BROWSER"] = browser_name
return credentials
def get_config() -> Dict[str, Any]:
"""Load configuration from ~/.config/last30days/.env and environment."""
# Load from config file first (if configured)
"""Load configuration from multiple sources.
Priority (highest wins):
1. Environment variables (os.environ)
2. .claude/last30days.env (per-project config)
3. ~/.config/last30days/.env (global config)
4. Browser cookies (only fills in missing keys)
"""
# Load from global config file
file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
openai_auth = get_openai_auth(file_env)
# Load from per-project config (overrides global)
project_env_path = _find_project_env()
project_env = load_env_file(project_env_path) if project_env_path else {}
# Build config: Codex/OpenAI auth + process.env > .env file
# Merge: project overrides global
merged_env = {**file_env, **project_env}
openai_auth = get_openai_auth(merged_env)
# Build config: Codex/OpenAI auth + process.env > project .env > global .env
config = {
'OPENAI_API_KEY': openai_auth.token,
'OPENAI_AUTH_SOURCE': openai_auth.source,
@@ -196,9 +352,15 @@ def get_config() -> Dict[str, Any]:
keys = [
('XAI_API_KEY', None),
('GOOGLE_API_KEY', None),
('GEMINI_API_KEY', None),
('GOOGLE_GENAI_API_KEY', None),
('OPENROUTER_API_KEY', None),
('PARALLEL_API_KEY', None),
('BRAVE_API_KEY', None),
('EXA_API_KEY', None),
('XIAOHONGSHU_API_BASE', None),
('GEMINI_MODEL', None),
('OPENAI_MODEL_POLICY', 'auto'),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_POLICY', 'latest'),
@@ -207,17 +369,52 @@ def get_config() -> Dict[str, Any]:
('APIFY_API_TOKEN', None),
('AUTH_TOKEN', None),
('CT0', None),
('BSKY_HANDLE', None),
('BSKY_APP_PASSWORD', None),
('TRUTHSOCIAL_TOKEN', None),
('FROM_BROWSER', None),
('SETUP_COMPLETE', None),
]
for key, default in keys:
config[key] = os.environ.get(key) or file_env.get(key, default)
config[key] = os.environ.get(key) or merged_env.get(key, default)
# Inject browser cookies for any credentials not already set
browser_creds = extract_browser_credentials(config)
for key, value in browser_creds.items():
if not config.get(key):
config[key] = value
# Track AUTH_TOKEN source for status reporting
if config.get('AUTH_TOKEN'):
if os.environ.get('AUTH_TOKEN') or merged_env.get('AUTH_TOKEN'):
config['_AUTH_TOKEN_SOURCE'] = 'env'
elif browser_creds.get('AUTH_TOKEN'):
browser_name = browser_creds.get('__X_BROWSER', 'unknown')
config['_AUTH_TOKEN_SOURCE'] = f'browser-{browser_name}'
else:
config['_AUTH_TOKEN_SOURCE'] = 'env' # fallback
else:
config['_AUTH_TOKEN_SOURCE'] = None
# Track which config source was used
if project_env_path:
config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
elif CONFIG_FILE and CONFIG_FILE.exists():
config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
else:
config['_CONFIG_SOURCE'] = 'env_only'
return config
def config_exists() -> bool:
"""Check if configuration file exists."""
return CONFIG_FILE.exists()
"""Check if any configuration source exists."""
if _find_project_env():
return True
if CONFIG_FILE:
return CONFIG_FILE.exists()
return False
def is_reddit_available(config: Dict[str, Any]) -> bool:
@@ -245,38 +442,41 @@ def get_reddit_source(config: Dict[str, Any]) -> Optional[str]:
def get_available_sources(config: Dict[str, Any]) -> str:
"""Determine which sources are available based on API keys.
"""Determine which sources are available.
X is available if ANY auth method works: AUTH_TOKEN/CT0 (env or cookies),
XAI_API_KEY, or Bird installed+authenticated.
Reddit is always available (public JSON fallback).
HN and Polymarket are always available.
YouTube available if yt-dlp installed.
Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
"""
has_reddit = is_reddit_available(config)
has_xai = bool(config.get('XAI_API_KEY'))
has_reddit = True
has_x = get_x_source(config) is not None
has_web = has_web_search_keys(config)
if has_reddit and has_xai:
if has_reddit and has_x:
return 'all' if has_web else 'both'
elif has_reddit:
return 'reddit-web' if has_web else 'reddit'
elif has_xai:
return 'x-web' if has_web else 'x'
elif has_web:
return 'web'
else:
return 'web' # Fallback: assistant WebSearch (no API keys needed)
return 'web' if has_web else 'none'
def has_web_search_keys(config: Dict[str, Any]) -> bool:
"""Check if any web search API keys are configured."""
return bool(config.get('OPENROUTER_API_KEY') or config.get('PARALLEL_API_KEY') or config.get('BRAVE_API_KEY'))
return bool(config.get('EXA_API_KEY') or config.get('OPENROUTER_API_KEY') or config.get('PARALLEL_API_KEY') or config.get('BRAVE_API_KEY'))
def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available web search backend.
Priority: Parallel AI > Brave > OpenRouter/Sonar Pro
Priority: Exa (free) > Parallel AI > Brave > OpenRouter/Sonar Pro
Returns: 'parallel', 'brave', 'openrouter', or None
Returns: 'exa', 'parallel', 'brave', 'openrouter', or None
"""
if config.get('EXA_API_KEY'):
return 'exa'
if config.get('PARALLEL_API_KEY'):
return 'parallel'
if config.get('BRAVE_API_KEY'):
@@ -287,11 +487,11 @@ def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
def get_missing_keys(config: Dict[str, Any]) -> str:
"""Determine which sources are missing (accounting for Bird and ScrapeCreators).
"""Determine which sources are missing (accounting for Bird).
Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
"""
has_reddit = is_reddit_available(config)
has_reddit = True
has_xai = bool(config.get('XAI_API_KEY'))
has_web = has_web_search_keys(config)
@@ -305,12 +505,11 @@ def get_missing_keys(config: Dict[str, Any]) -> str:
return 'none'
elif has_reddit and has_x:
return 'web' # Missing web search keys
elif has_reddit and has_web:
return 'x' # Missing X source
elif has_reddit:
return 'x' # Missing X source (and possibly web)
elif has_x:
return 'reddit' # Missing Reddit source (and possibly web)
else:
return 'all' # Missing everything
return 'all'
def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]:
@@ -324,56 +523,51 @@ def validate_sources(requested: str, available: str, include_web: bool = False)
Returns:
Tuple of (effective_sources, error_message)
"""
# No API keys at all
if available == 'none':
if requested == 'auto':
return 'web', "No API keys configured. The assistant can still search the web if it has a search tool."
elif requested == 'web':
return 'web', None
else:
return 'web', f"No API keys configured. Add keys to ~/.config/last30days/.env for Reddit/X."
# Web-only mode (only web search API keys)
if available == 'web':
if requested == 'auto':
return 'web', None
elif requested == 'web':
return 'web', None
else:
return 'web', "Only web search keys configured. Add OPENAI_API_KEY (or run codex login) for Reddit, XAI_API_KEY for X."
has_reddit = available in ('reddit', 'both', 'reddit-web', 'all')
has_x = available in ('x', 'both', 'x-web', 'all')
has_web = available in ('web', 'reddit-web', 'x-web', 'all')
if requested == 'auto':
# Add web to sources if include_web is set
if has_reddit and has_x:
base = 'both'
elif has_reddit:
base = 'reddit'
elif has_x:
base = 'x'
elif has_web:
base = 'web'
else:
return 'none', "No sources are available."
if include_web:
if available == 'both':
return 'all', None # reddit + x + web
elif available == 'reddit':
if base == 'both':
return 'all', None
if base == 'reddit':
return 'reddit-web', None
elif available == 'x':
if base == 'x':
return 'x-web', None
return available, None
return base, None
if requested == 'web':
return 'web', None
if requested == 'both':
if available not in ('both',):
missing = 'xAI' if available == 'reddit' else 'OpenAI'
return 'none', f"Requested both sources but {missing} key is missing. Use --sources=auto to use available keys."
if not (has_reddit and has_x):
return 'none', "Requested both sources but X source is missing."
if include_web:
return 'all', None
return 'both', None
if requested == 'reddit':
if available == 'x':
if not has_reddit:
return 'none', "Requested Reddit but only xAI key is available."
if include_web:
return 'reddit-web', None
return 'reddit', None
if requested == 'x':
if available == 'reddit':
return 'none', "Requested X but only OpenAI key is available."
if not has_x:
return 'none', "Requested X but no X source is available (need Bird auth or XAI_API_KEY)."
if include_web:
return 'x-web', None
return 'x', None
@@ -384,7 +578,13 @@ def validate_sources(requested: str, available: str, include_web: bool = False)
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) xAI (paid API)
Priority chain:
1. AUTH_TOKEN/CT0 from env var or .env file Bird with method "env"
2. AUTH_TOKEN/CT0 from browser cookie extraction Bird with method "browser-{browser}"
3. XAI_API_KEY xAI with method "api"
4. None
Use get_x_source_with_method() to also get the method string.
Args:
config: Configuration dict from get_config()
@@ -394,20 +594,58 @@ def get_x_source(config: Dict[str, Any]) -> Optional[str]:
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
source, _method = get_x_source_with_method(config)
return source
def get_x_source_with_method(config: Dict[str, Any]) -> tuple[Optional[str], Optional[str]]:
"""Determine the best available X/Twitter source and auth method.
Priority chain:
1. AUTH_TOKEN/CT0 (env var or .env) Bird with method "env"
2. AUTH_TOKEN/CT0 (browser cookies) Bird with method "browser-{browser}"
3. XAI_API_KEY xAI with method "api"
4. None
Args:
config: Configuration dict from get_config()
Returns:
Tuple of (source, method) where source is 'bird', 'xai', or None
and method is 'env', 'browser-chrome', 'browser-firefox', 'browser-safari', 'api', or None.
"""
from . import bird_x
# Check Bird first (free option)
setup_complete = config.get('SETUP_COMPLETE')
# Check Bird first (free option — uses AUTH_TOKEN/CT0 from any source)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
auth_source = config.get('_AUTH_TOKEN_SOURCE')
# If SETUP_COMPLETE is not set, only allow explicit env var credentials.
# Do NOT call is_bird_authenticated() for browser-cookie probing —
# that requires user consent via the setup wizard.
if not setup_complete:
# Explicit AUTH_TOKEN from env var / .env file is always allowed
if auth_source == 'env' and config.get('AUTH_TOKEN'):
username = bird_x.is_bird_authenticated()
if username:
return 'bird', 'env'
else:
# SETUP_COMPLETE is set — normal flow, probe cookies if needed
username = bird_x.is_bird_authenticated()
if username:
if auth_source and auth_source.startswith('browser-'):
method = auth_source # e.g. "browser-firefox"
else:
method = 'env'
return 'bird', method
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return 'xai', 'api'
return None
return None, None
def is_ytdlp_available() -> bool:
@@ -424,6 +662,22 @@ def is_hackernews_available() -> bool:
return True
def is_bluesky_available(config: Dict[str, Any]) -> bool:
"""Check if Bluesky source is available.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD (app password from bsky.app/settings).
"""
return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))
def is_truthsocial_available(config: Dict[str, Any]) -> bool:
"""Check if Truth Social source is available.
Requires TRUTHSOCIAL_TOKEN (bearer token from browser dev tools).
"""
return bool(config.get('TRUTHSOCIAL_TOKEN'))
def is_polymarket_available() -> bool:
"""Check if Polymarket source is available.
@@ -459,6 +713,40 @@ def get_instagram_token(config: Dict[str, Any]) -> str:
return config.get('SCRAPECREATORS_API_KEY') or ''
def get_xiaohongshu_api_base(config: Dict[str, Any]) -> str:
"""Get Xiaohongshu HTTP API base URL.
Defaults to host.docker.internal so OpenClaw Docker can reach host service.
"""
return (config.get('XIAOHONGSHU_API_BASE') or "http://host.docker.internal:18060").rstrip("/")
def is_xiaohongshu_available(config: Dict[str, Any]) -> bool:
"""Check whether Xiaohongshu HTTP API is reachable and logged in."""
# Import here to avoid heavy imports at module load.
from . import http
base = get_xiaohongshu_api_base(config)
try:
# Keep health probe snappy, but allow one retry for transient hiccups.
health = http.get(f"{base}/health", timeout=3, retries=2)
if not isinstance(health, dict):
return False
if not health.get("success"):
return False
# Login probe can be slower on some deployments (browser/session checks),
# so use a slightly longer timeout to avoid false negatives.
login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=2)
is_logged_in = (
login.get("data", {}).get("is_logged_in")
if isinstance(login, dict) else False
)
return bool(is_logged_in)
except Exception:
return False
# Backward compat alias
is_apify_available = is_tiktok_available
@@ -467,24 +755,49 @@ def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
Dict with keys: source, method, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
The ``method`` field indicates HOW the active source is authenticated:
- "env" AUTH_TOKEN came from an env var or .env file
- "browser-chrome", "browser-firefox", "browser-safari" from cookie extraction
- "api" using xAI API key
- None no X source available
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
setup_complete = config.get('SETUP_COMPLETE')
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
if not setup_complete:
# Before consent: do NOT call get_bird_status() which probes cookies.
# Only check if Bird is installed (no cookie probing) and use the
# gated get_x_source_with_method() which blocks cookie detection.
bird_installed = bird_x.is_bird_installed()
source, method = get_x_source_with_method(config)
# Bird "authenticated" only if get_x_source_with_method found explicit creds
bird_authenticated = (source == 'bird')
return {
"source": source,
"method": method,
"bird_installed": bird_installed,
"bird_authenticated": bird_authenticated,
"bird_username": None if not bird_authenticated else "env AUTH_TOKEN",
"xai_available": xai_available,
"can_install_bird": True,
}
# SETUP_COMPLETE is set — normal flow
bird_status = bird_x.get_bird_status()
# Use the unified resolution function for source + method
source, method = get_x_source_with_method(config)
return {
"source": source,
"method": method,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
+176
View File
@@ -0,0 +1,176 @@
"""Exa AI web search for last30days skill.
Uses the Exa Search API as a free web search backend.
Free tier: 1,000 searches/month, semantic search, no credit card required.
API docs: https://docs.exa.ai/reference/search
"""
import sys
from typing import Any, Dict, List, Optional
from urllib.parse import urlparse
from . import http
ENDPOINT = "https://api.exa.ai/search"
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Exa AI Search API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: Exa API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
"""
num_results = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
max_chars = {"quick": 1000, "default": 2000, "deep": 3000}.get(depth, 2000)
payload = {
"query": f"{topic} (from {from_date} to {to_date})",
"type": "auto",
"numResults": num_results,
"contents": {"text": {"maxCharacters": max_chars}},
}
# Add date filtering if dates are provided
if from_date:
payload["startPublishedDate"] = f"{from_date}T00:00:00.000Z"
if to_date:
payload["endPublishedDate"] = f"{to_date}T23:59:59.999Z"
sys.stderr.write(f"[Web] Searching Exa for: {topic}\n")
sys.stderr.flush()
try:
response = http.post(
ENDPOINT,
json_data=payload,
headers={
"x-api-key": api_key,
},
timeout=20,
retries=2,
)
except http.HTTPError as e:
status = e.status_code
if status == 401:
sys.stderr.write("[Web] Exa: invalid API key (401)\n")
sys.stderr.flush()
return []
if status == 429:
sys.stderr.write("[Web] Exa: rate limited (429)\n")
sys.stderr.flush()
return []
sys.stderr.write(f"[Web] Exa: HTTP error {status}: {e}\n")
sys.stderr.flush()
return []
except Exception as e:
sys.stderr.write(f"[Web] Exa: request failed: {e}\n")
sys.stderr.flush()
return []
return _normalize_results(response)
def _normalize_results(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Exa API response to websearch item schema.
Exa results have: title, url, text, publishedDate, score, author.
"""
items = []
results = response.get("results", [])
if not isinstance(results, list):
return items
for i, result in enumerate(results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = str(result.get("title", "")).strip()
# Exa returns page content in "text" field
snippet = str(result.get("text", "")).strip()
if not title and not snippet:
continue
# Parse publishedDate (ISO format from Exa: "2026-03-15T00:00:00.000Z")
date = _parse_exa_date(result.get("publishedDate"))
date_confidence = "med" if date else "low"
# Exa provides a relevance score
relevance = result.get("score", 0.6)
try:
relevance = min(1.0, max(0.0, float(relevance)))
except (TypeError, ValueError):
relevance = 0.6
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": snippet[:500],
"date": date,
"date_confidence": date_confidence,
"relevance": relevance,
"why_relevant": "",
})
sys.stderr.write(f"[Web] Exa: {len(items)} results\n")
sys.stderr.flush()
return items
def _parse_exa_date(published_date: Optional[str]) -> Optional[str]:
"""Parse Exa's publishedDate to YYYY-MM-DD.
Exa returns ISO format like "2026-03-15T00:00:00.000Z".
"""
if not published_date:
return None
try:
# Extract YYYY-MM-DD from ISO datetime
if "T" in published_date:
return published_date.split("T")[0]
# Already YYYY-MM-DD
if len(published_date) >= 10:
return published_date[:10]
except Exception:
pass
return None
+21 -8
View File
@@ -12,6 +12,8 @@ from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional
from . import http
from .query import extract_core_subject
from .relevance import token_overlap_relevance
ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
@@ -84,13 +86,17 @@ def search_hackernews(
from_ts = _date_to_unix(from_date)
to_ts = _date_to_unix(to_date) + 86400 # Include the end date
_log(f"Searching for '{topic}' (since {from_date}, count={count})")
# Use extracted core subject instead of raw topic for cleaner Algolia matching
core = extract_core_subject(topic)
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Use relevance-sorted search (better for topic matching)
# Use relevance-sorted search with minimum engagement filter.
# NOTE: restrictSearchableAttributes=title omitted intentionally — it would
# miss Ask HN/Show HN threads where the topic appears in the body.
params = {
"query": topic,
"query": core,
"tags": "story",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts}",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
"hitsPerPage": str(count),
}
@@ -111,9 +117,13 @@ def search_hackernews(
return response
def parse_hackernews_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
"""Parse Algolia response into normalized item dicts.
Args:
response: Algolia search response
query: Original search query for token-overlap relevance scoring
Returns:
List of item dicts ready for normalization.
"""
@@ -134,11 +144,14 @@ def parse_hackernews_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
article_url = hit.get("url") or ""
hn_url = f"https://news.ycombinator.com/item?id={object_id}"
# Relevance: Algolia rank position gives a base, engagement boosts it
# Position 0 = most relevant from Algolia
# Relevance: blend Algolia rank with token-overlap content matching
rank_score = max(0.3, 1.0 - (i * 0.02)) # 1.0 -> 0.3 over 35 items
engagement_boost = min(0.2, math.log1p(points) / 40)
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
if query:
content_score = token_overlap_relevance(query, hit.get("title", ""))
relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
else:
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"object_id": object_id,
+35 -107
View File
@@ -3,7 +3,7 @@
Uses ScrapeCreators REST API to search Instagram Reels by keyword, extract
engagement metrics (views, likes, comments), and fetch video transcripts.
Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
@@ -17,6 +17,8 @@ try:
except ImportError:
_requests = None
from . import http
SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
# Depth configurations: how many results to fetch / captions to extract
@@ -29,93 +31,13 @@ DEPTH_CONFIG = {
# Max words to keep from each caption
CAPTION_MAX_WORDS = 500
# Stopwords for relevance computation (shared with tiktok.py pattern)
STOPWORDS = frozenset({
'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
'all', 'just', 'get', 'has', 'have', 'was', 'will',
})
# Synonym groups for relevance scoring
SYNONYMS = {
'hip': {'rap', 'hiphop'},
'hop': {'rap', 'hiphop'},
'rap': {'hip', 'hop', 'hiphop'},
'hiphop': {'rap', 'hip', 'hop'},
'js': {'javascript'},
'javascript': {'js'},
'ts': {'typescript'},
'typescript': {'ts'},
'ai': {'artificial', 'intelligence'},
'ml': {'machine', 'learning'},
'react': {'reactjs'},
'reactjs': {'react'},
}
def _tokenize(text: str) -> Set[str]:
"""Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
expanded = set(tokens)
for t in tokens:
if t in SYNONYMS:
expanded.update(SYNONYMS[t])
return expanded
def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> float:
"""Compute relevance as ratio of query tokens found in text + hashtags.
Uses ratio overlap (intersection / query_length). Hashtags provide
an Instagram-specific relevance boost. Floors at 0.1.
"""
q_tokens = _tokenize(query)
# Combine text and hashtags for matching
combined = text
if hashtags:
combined = f"{text} {' '.join(hashtags)}"
t_tokens = _tokenize(combined)
# Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
if hashtags:
for tag in hashtags:
tag_lower = tag.lower()
for qt in q_tokens:
if qt in tag_lower and qt != tag_lower:
t_tokens.add(qt)
if not q_tokens:
return 0.5 # Neutral fallback
overlap = len(q_tokens & t_tokens)
ratio = overlap / len(q_tokens)
return max(0.1, min(1.0, ratio))
from .relevance import token_overlap_relevance as _compute_relevance
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Instagram search.
Strips meta/research words to keep only the core product/concept name.
"""
text = topic.lower().strip()
# Strip multi-word prefixes
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
# Strip individual noise words
noise = {
"""Extract core subject from verbose query for Instagram search."""
from .query import extract_core_subject
_INSTAGRAM_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
@@ -123,12 +45,8 @@ def _extract_core_subject(topic: str) -> str:
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
}
words = text.split()
filtered = [w for w in words if w not in noise]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.')
})
return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
def _log(msg: str):
@@ -207,28 +125,38 @@ def search_instagram(
if not token:
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
if not _requests:
return {"items": [], "error": "requests library not installed"}
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/v1/instagram/reels/search",
params={"query": core_topic},
headers=_sc_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
if not _requests:
_log("requests library not installed, falling back to urllib")
try:
from urllib.parse import urlencode
params = urlencode({"query": core_topic})
url = f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search?{params}"
headers = _sc_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
params={"query": core_topic},
headers=_sc_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
# Items are in the 'reels' array (ScrapeCreators v1 response)
# Items are in the 'reels' array (ScrapeCreators v2 response)
raw_items = data.get("reels") or data.get("items") or data.get("data") or []
# Limit to configured count
+58 -22
View File
@@ -1,20 +1,35 @@
"""Model auto-selection for last30days skill."""
"""Model auto-selection for last30days skill.
Model selection philosophy: this tool uses LLM APIs exclusively for
search tool invocation + structured JSON extraction. This is not
reasoning-heavy or creative work mini models handle it equally well
at ~3-5x lower cost. We prefer the newest-generation mini model, falling
back to mainline only when mini isn't available.
xAI non-reasoning variant preferred: same pricing as reasoning, but
faster (skips thinking phase, saves reasoning token output costs).
"""
import re
import sys
from typing import Dict, List, Optional, Tuple
from . import cache, http, env
# OpenAI API
OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4.1", "gpt-4o"]
# Ordered by cost-efficiency for web_search + JSON extraction tasks.
# Mini models first: same structured extraction quality at ~3x lower cost.
OPENAI_FALLBACK_MODELS = ["gpt-5-mini", "gpt-4.1-mini", "gpt-4.1", "gpt-4o"]
CODEX_FALLBACK_MODELS = ["gpt-5.1-codex-mini", "gpt-5.2"]
# xAI API - Agent Tools API requires grok-4 family
# Non-reasoning: same price, faster, no unnecessary thinking tokens.
# Both variants support function calling and structured outputs.
XAI_MODELS_URL = "https://api.x.ai/v1/models"
XAI_ALIASES = {
"latest": "grok-4-1-fast", # Required for x_search tool
"stable": "grok-4-1-fast",
"latest": "grok-4-1-fast-non-reasoning",
"stable": "grok-4-1-fast-non-reasoning",
}
@@ -32,30 +47,49 @@ def parse_version(model_id: str) -> Optional[Tuple[int, ...]]:
return None
def is_mainline_openai_model(model_id: str) -> bool:
"""Check if model is a mainline GPT model (not mini/nano/chat/codex/pro)."""
def is_search_capable_model(model_id: str) -> bool:
"""Check if model supports Responses API web_search with domain filtering.
Includes mini variants (same structured extraction quality, lower cost).
Excludes: nano (no web_search), gpt-4o-mini (no domain filtering),
chat/codex/pro/preview/turbo/search (specialized variants).
Note: gpt-5 with reasoning effort="minimal" does NOT support web_search
(per OpenAI docs). We never set reasoning params our usage is pure
tool invocation + JSON extraction so gpt-5 is safe to include here.
"""
model_lower = model_id.lower()
# Must be gpt-4o, gpt-4.1+, or gpt-5+ series (mainline, not mini/nano/etc)
if not re.match(r'^gpt-(?:4o|4\.1|5)(\.\d+)*$', model_lower):
# gpt-4o-mini does NOT support web_search with filters — exclude it
if model_lower.startswith("gpt-4o-mini"):
return False
# Exclude variants
excludes = ['mini', 'nano', 'chat', 'codex', 'pro', 'preview', 'turbo']
for exc in excludes:
# Must be gpt-4o, gpt-4.1[-mini], or gpt-5[-mini] series
if not re.match(r'^gpt-(?:4o|4\.1|5)(\.\d+)*(-mini)?$', model_lower):
return False
# Exclude unsupported variants
for exc in ['nano', 'chat', 'codex', 'pro', 'preview', 'turbo', 'search']:
if exc in model_lower:
return False
return True
# Backward compat alias
is_mainline_openai_model = is_search_capable_model
def select_openai_model(
api_key: str,
policy: str = "auto",
pin: Optional[str] = None,
mock_models: Optional[List[Dict]] = None,
) -> str:
"""Select the best OpenAI model based on policy.
"""Select the most cost-efficient OpenAI model for web_search + JSON extraction.
Prefers mini models within the newest generation available, since the task
is structured extraction (not reasoning or creative work).
Args:
api_key: OpenAI API key
@@ -82,27 +116,29 @@ def select_openai_model(
headers = {"Authorization": f"Bearer {api_key}"}
response = http.get(OPENAI_MODELS_URL, headers=headers)
models = response.get("data", [])
except http.HTTPError:
# Fall back to known models
except http.HTTPError as e:
sys.stderr.write(f"[Models] Failed to fetch OpenAI models: {e}")
if hasattr(e, 'status_code') and e.status_code in (401, 403):
sys.stderr.write(" — API key may be invalid or lack permissions")
sys.stderr.write(f", using fallback {OPENAI_FALLBACK_MODELS[0]}\n")
return OPENAI_FALLBACK_MODELS[0]
# Filter to mainline models
candidates = [m for m in models if is_mainline_openai_model(m.get("id", ""))]
candidates = [m for m in models if is_search_capable_model(m.get("id", ""))]
if not candidates:
# No gpt-5 models found, use fallback
return OPENAI_FALLBACK_MODELS[0]
# Sort by version (descending), then by created timestamp
# Sort: newest generation first, prefer mini within same generation
def sort_key(m):
version = parse_version(m.get("id", "")) or (0,)
created = m.get("created", 0)
return (version, created)
model_id = m.get("id", "")
version = parse_version(model_id) or (0,)
major = version[0] if version else 0
is_mini = 1 if "mini" in model_id.lower() else 0
return (major, is_mini, version)
candidates.sort(key=sort_key, reverse=True)
selected = candidates[0]["id"]
# Cache the selection
cache.set_cached_model("openai", selected)
return selected
+89 -1
View File
@@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union
from . import dates, schema
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem)
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.BlueskyItem, schema.PolymarketItem)
def filter_by_date_range(
@@ -193,6 +193,7 @@ def normalize_youtube_items(
date_confidence="high",
engagement=engagement,
transcript_snippet=item.get("transcript_snippet", ""),
transcript_highlights=item.get("transcript_highlights", []),
relevance=item.get("relevance", 0.7),
why_relevant=item.get("why_relevant", ""),
))
@@ -350,6 +351,93 @@ def normalize_hackernews_items(
return normalized
def normalize_bluesky_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.BlueskyItem]:
"""Normalize raw Bluesky items to schema.
Args:
items: Raw Bluesky items from AT Protocol API
from_date: Start of date range
to_date: End of date range
Returns:
List of BlueskyItem objects
"""
normalized = []
for i, item in enumerate(items):
eng_raw = item.get("engagement") or {}
engagement = schema.Engagement(
likes=eng_raw.get("likes"),
reposts=eng_raw.get("reposts"),
replies=eng_raw.get("replies"),
quotes=eng_raw.get("quotes"),
)
date_str = item.get("date")
normalized.append(schema.BlueskyItem(
id=f"BS{i+1}",
text=item.get("text", ""),
url=item.get("url", ""),
author_handle=item.get("handle", ""),
display_name=item.get("display_name", ""),
date=date_str,
date_confidence="high",
engagement=engagement,
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def normalize_truthsocial_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.TruthSocialItem]:
"""Normalize raw Truth Social items to schema.
Args:
items: Raw Truth Social items from Mastodon API
from_date: Start of date range
to_date: End of date range
Returns:
List of TruthSocialItem objects
"""
normalized = []
for i, item in enumerate(items):
eng_raw = item.get("engagement") or {}
engagement = schema.Engagement(
likes=eng_raw.get("likes"),
reposts=eng_raw.get("reposts"),
replies=eng_raw.get("replies"),
)
date_str = item.get("date")
normalized.append(schema.TruthSocialItem(
id=f"TS{i+1}",
text=item.get("text", ""),
url=item.get("url", ""),
author_handle=item.get("handle", ""),
display_name=item.get("display_name", ""),
date=date_str,
date_confidence="high",
engagement=engagement,
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def normalize_polymarket_items(
items: List[Dict[str, Any]],
from_date: str,
+103 -3
View File
@@ -7,9 +7,10 @@ from typing import Any, Dict, List, Optional
from . import http, env
# Fallback models when the selected model isn't accessible (e.g., org not verified for GPT-5)
# Note: gpt-4o-mini does NOT support web_search with filters param, so exclude it
MODEL_FALLBACK_ORDER = ["gpt-4.1", "gpt-4o"]
# Fallback models when the selected model isn't accessible (e.g., org not verified).
# Ordered by cost-efficiency: mini models handle structured extraction equally well.
# Note: gpt-4o-mini does NOT support web_search with filters — excluded.
MODEL_FALLBACK_ORDER = ["gpt-5-mini", "gpt-4.1-mini", "gpt-4.1", "gpt-4o"]
def _log_error(msg: str):
@@ -355,6 +356,105 @@ def search_reddit(
raise http.HTTPError("No models available")
def _public_relevance(score: int, num_comments: int) -> float:
"""Estimate relevance for public Reddit search results."""
# Lightweight heuristic: blend normalized score + comments.
score_component = min(1.0, max(0.0, score / 500.0))
comments_component = min(1.0, max(0.0, num_comments / 200.0))
return round((score_component * 0.6) + (comments_component * 0.4), 3)
def search_reddit_public(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search Reddit directly via public JSON endpoint (no OpenAI key required).
This is a fallback mode for environments where OpenAI auth is unavailable.
It uses reddit.com/search/.json with recency filter (t=month).
"""
_, max_items = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
limit = min(100, max(20, max_items))
core = _extract_core_subject(topic)
queries = [topic]
if core and core.lower() != topic.lower():
queries.append(core)
queries.append(f'"{core}"')
seen_urls = set()
all_items: List[Dict[str, Any]] = []
headers = {
"User-Agent": http.USER_AGENT,
"Accept": "application/json",
}
for query in queries:
try:
url = (
"https://www.reddit.com/search/.json"
f"?q={_url_encode(query)}&sort=new&t=month&limit={limit}&raw_json=1"
)
data = http.get(url, headers=headers, timeout=20, retries=2)
children = data.get("data", {}).get("children", [])
for child in children:
if child.get("kind") != "t3":
continue
post = child.get("data", {})
permalink = str(post.get("permalink", "")).strip()
if not permalink or "/comments/" not in permalink:
continue
full_url = f"https://www.reddit.com{permalink}"
if full_url in seen_urls:
continue
seen_urls.add(full_url)
score = int(post.get("score", 0) or 0)
num_comments = int(post.get("num_comments", 0) or 0)
# Parse date from created_utc
created_utc = post.get("created_utc")
date_value = None
if created_utc:
from . import dates as dates_mod
date_value = dates_mod.timestamp_to_date(created_utc)
all_items.append({
"id": f"R{len(all_items)+1}",
"title": str(post.get("title", "")).strip(),
"url": full_url,
"subreddit": str(post.get("subreddit", "")).strip(),
"date": date_value,
"why_relevant": "Found via Reddit public search",
"relevance": _public_relevance(score, num_comments),
"engagement": {
"score": score,
"num_comments": num_comments,
"upvote_ratio": post.get("upvote_ratio"),
},
})
except http.HTTPError as e:
_log_info(f"Public Reddit search failed for query '{query}': {e}")
# Continue with next query; partial results are still useful.
continue
except Exception as e:
_log_info(f"Public Reddit search error for query '{query}': {e}")
continue
# Sort by date (desc, unknown dates last), then relevance desc
def _sort_key(item: Dict[str, Any]):
date_str = item.get("date") or ""
return (date_str, float(item.get("relevance", 0.0)))
all_items.sort(key=_sort_key, reverse=True)
return all_items[: max_items * 2]
def search_subreddits(
subreddits: List[str],
topic: str,
+54 -28
View File
@@ -1,7 +1,7 @@
"""Polymarket prediction market search via Gamma API (free, no auth required).
Uses gamma-api.polymarket.com for event/market discovery.
No API key needed - public read-only API with generous rate limits (350 req/10s).
No API key needed - public read-only API with generous rate limits (15K req/10s).
"""
import json
@@ -13,6 +13,8 @@ from typing import Any, Dict, List, Optional
from urllib.parse import quote_plus, urlencode
from . import http
from .query_type import detect_query_type
from .relevance import LOW_SIGNAL_QUERY_TOKENS, token_overlap_relevance
GAMMA_SEARCH_URL = "https://gamma-api.polymarket.com/public-search"
@@ -73,7 +75,7 @@ def _expand_queries(topic: str) -> List[str]:
words = core.split()
if len(words) >= 2:
for word in words:
if len(word) > 1: # skip single-char words
if len(word) > 1 and word.lower() not in LOW_SIGNAL_QUERY_TOKENS:
queries.append(word)
# Add the full topic if different from core
@@ -130,7 +132,12 @@ def _extract_domain_queries(topic: str, events: List[Dict]) -> List[str]:
def _search_single_query(query: str, page: int = 1) -> Dict[str, Any]:
"""Run a single search query against Gamma API."""
params = {"q": query, "page": str(page)}
params = {
"q": query,
"page": str(page),
"events_status": "active",
"keep_closed_markets": "0",
}
url = f"{GAMMA_SEARCH_URL}?{urlencode(params)}"
try:
@@ -309,8 +316,8 @@ def _shorten_question(question: str) -> str:
def _compute_text_similarity(topic: str, title: str, outcomes: List[str] = None) -> float:
"""Score how well the event title (or outcome names) match the search topic.
Returns 0.0-1.0. Title substring match gets 1.0, outcome match gets 0.85/0.7,
title token overlap gets proportional score.
Returns 0.0-1.0. Exact title phrase match gets 1.0. Otherwise we reuse the
shared query-centric relevance scorer and take the best title/outcome match.
"""
core = _extract_core_subject(topic).lower()
title_lower = title.lower()
@@ -321,27 +328,45 @@ def _compute_text_similarity(topic: str, title: str, outcomes: List[str] = None)
if core in title_lower:
return 1.0
# Check if topic appears in any outcome name (bidirectional)
query_type = detect_query_type(topic)
title_score = token_overlap_relevance(core, title)
best_score = title_score
if outcomes:
core_tokens = set(core.split())
best_outcome_score = 0.0
for outcome_name in outcomes:
outcome_lower = outcome_name.lower()
# Bidirectional: "arizona" in "arizona basketball" OR "arizona basketball" contains "arizona"
if core in outcome_lower or outcome_lower in core:
best_outcome_score = max(best_outcome_score, 0.85)
elif core_tokens & set(outcome_lower.split()):
best_outcome_score = max(best_outcome_score, 0.7)
if best_outcome_score > 0:
return best_outcome_score
outcome_score = token_overlap_relevance(core, outcome_name)
if _strong_phrase_match(core, outcome_lower):
outcome_score = max(outcome_score, 0.92 if len(outcome_lower.split()) >= 2 else 0.88)
if title_score < 0.3:
outcome_cap = 0.55 if query_type == "prediction" else 0.24
outcome_score = min(outcome_cap, outcome_score)
else:
outcome_score = max(title_score, 0.75 * title_score + 0.25 * outcome_score)
best_score = max(best_score, outcome_score)
# Token overlap fallback against title
topic_tokens = set(core.split())
title_tokens = set(title_lower.split())
if not topic_tokens:
return 0.5
overlap = len(topic_tokens & title_tokens)
return overlap / len(topic_tokens)
return round(best_score, 2)
def _strong_phrase_match(core: str, candidate: str) -> bool:
"""Require real token matches, not accidental short substrings.
This prevents binary outcomes like "No" from matching "nano" or similar
short-string accidents.
"""
candidate = " ".join(re.sub(r"[^\w\s]", " ", candidate.lower()).split())
core = " ".join(re.sub(r"[^\w\s]", " ", core.lower()).split())
if not candidate or not core:
return False
candidate_tokens = candidate.split()
core_tokens = set(core.split())
if len(candidate_tokens) >= 2:
return candidate in core or core in candidate
token = candidate_tokens[0]
return len(token) > 2 and token in core_tokens
def _safe_float(val, default=0.0) -> float:
@@ -479,7 +504,8 @@ def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List
except (IndexError, TypeError):
end_date = None
# Quality-signal relevance (replaces position-based decay)
# Semantic relevance should dominate. Market quality should refine
# relevant matches, not rescue unrelated high-liquidity events.
text_score = _compute_text_similarity(topic, title, all_outcome_names) if topic else 0.5
# Volume signal: log-scaled monthly volume (most stable signal)
@@ -499,13 +525,13 @@ def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List
# Competitive bonus: markets near 50/50 are more interesting
competitive_score = event_competitive
relevance = min(1.0, (
0.30 * text_score +
0.30 * vol_score +
0.15 * liq_score +
market_quality = (
0.50 * vol_score +
0.25 * liq_score +
0.15 * movement_score +
0.10 * competitive_score
))
)
relevance = min(1.0, text_score * (0.75 + 0.25 * market_quality))
# Surface the topic-matching outcome to the front before truncating
if topic and outcome_prices:
+201
View File
@@ -0,0 +1,201 @@
"""Post-research quality score and upgrade nudge.
Computes a quality score based on 5 core sources and builds
a nudge message describing what the user missed and how to fix it.
"""
from typing import Any, Dict, List, Optional
# The 5 core sources
CORE_SOURCES = ["hn", "polymarket", "x", "youtube", "reddit_comments"]
# Labels for display
SOURCE_LABELS = {
"hn": "Hacker News",
"polymarket": "Polymarket",
"x": "X/Twitter",
"youtube": "YouTube",
"reddit_comments": "Reddit with comments",
}
def _is_x_active(config: dict, research_results: dict) -> bool:
"""Check if X source is active (has credentials AND didn't error)."""
has_creds = bool(config.get("AUTH_TOKEN") or config.get("XAI_API_KEY"))
if not has_creds:
return False
# If X errored this run, it's configured but broken
if research_results.get("x_error"):
return False
return True
def _is_youtube_active(config: dict, research_results: dict) -> bool:
"""Check if YouTube source is active (yt-dlp installed)."""
try:
from . import youtube_yt
has_ytdlp = youtube_yt.is_ytdlp_installed()
except Exception:
has_ytdlp = False
if not has_ytdlp:
return False
if research_results.get("youtube_error"):
return False
return True
def _is_reddit_comments_active(config: dict, research_results: dict) -> bool:
"""Check if Reddit with comments is active (ScrapeCreators)."""
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
if not has_sc:
return False
if research_results.get("reddit_error"):
return False
return True
def compute_quality_score(config: dict, research_results: dict) -> dict:
"""Compute research quality score based on 5 core sources.
Args:
config: Configuration dict from env.get_config()
research_results: Dict with keys like x_error, youtube_error,
reddit_error reflecting what happened this run.
Returns:
{
"score_pct": 40-100,
"core_active": ["hn", "polymarket", ...],
"core_missing": ["x", "youtube", "reddit_comments"],
"core_errored": ["reddit_comments"], # configured but errored
"nudge_text": "..." or None if 100%
}
"""
core_active: List[str] = []
core_missing: List[str] = []
core_errored: List[str] = []
# HN and Polymarket are always active
core_active.append("hn")
core_active.append("polymarket")
# X
has_x_creds = bool(config.get("AUTH_TOKEN") or config.get("XAI_API_KEY"))
if _is_x_active(config, research_results):
core_active.append("x")
else:
core_missing.append("x")
if has_x_creds and research_results.get("x_error"):
core_errored.append("x")
# YouTube
try:
from . import youtube_yt
has_ytdlp = youtube_yt.is_ytdlp_installed()
except Exception:
has_ytdlp = False
if _is_youtube_active(config, research_results):
core_active.append("youtube")
else:
core_missing.append("youtube")
if has_ytdlp and research_results.get("youtube_error"):
core_errored.append("youtube")
# Reddit with comments (ScrapeCreators)
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
if _is_reddit_comments_active(config, research_results):
core_active.append("reddit_comments")
else:
core_missing.append("reddit_comments")
if has_sc and research_results.get("reddit_error"):
core_errored.append("reddit_comments")
score_pct = int(len(core_active) / 5 * 100)
nudge_text = _build_nudge_text(core_missing, core_errored) if core_missing else None
return {
"score_pct": score_pct,
"core_active": core_active,
"core_missing": core_missing,
"core_errored": core_errored,
"nudge_text": nudge_text,
}
def _build_nudge_text(core_missing: List[str], core_errored: List[str]) -> str:
"""Build human-readable nudge text describing what was missed.
Prioritizes free suggestions before paid ones.
"""
lines: List[str] = []
# Describe what was missed
missed_parts: List[str] = []
for src in core_missing:
label = SOURCE_LABELS[src]
if src in core_errored:
missed_parts.append(f"{label} (errored this run)")
else:
missed_parts.append(label)
active_count = 5 - len(core_missing)
lines.append(f"Research quality: {active_count}/5 core sources.")
lines.append(f"Missing: {', '.join(missed_parts)}.")
lines.append("")
# Free suggestions first
free_suggestions: List[str] = []
paid_suggestions: List[str] = []
if "x" in core_missing:
if "x" in core_errored:
free_suggestions.append(
"X errored — try refreshing your browser cookies "
"(log into x.com, then re-run)."
)
else:
free_suggestions.append(
"X/Twitter: scan browser cookies automatically — "
"just log into x.com in any browser and re-run."
)
if "youtube" in core_missing:
if "youtube" in core_errored:
free_suggestions.append(
"YouTube errored — check that yt-dlp is up to date: "
"brew upgrade yt-dlp"
)
else:
free_suggestions.append(
"YouTube: install yt-dlp — brew install yt-dlp"
)
if "reddit_comments" in core_missing:
if "reddit_comments" in core_errored:
paid_suggestions.append(
"Reddit comments errored — check your ScrapeCreators API key "
"at scrapecreators.com."
)
else:
paid_suggestions.append(
"Reddit with comments: add SCRAPECREATORS_API_KEY — "
"100 free API calls, no credit card — scrapecreators.com"
)
if free_suggestions:
lines.append("Free fixes:")
for s in free_suggestions:
lines.append(f" - {s}")
lines.append("")
if paid_suggestions:
lines.append("Paid options:")
for s in paid_suggestions:
lines.append(f" - {s}")
lines.append("")
lines.append("last30days has no affiliation with any API provider.")
return "\n".join(lines)
+117
View File
@@ -0,0 +1,117 @@
"""Shared query preprocessing utilities: noise-word stripping, core subject
extraction, and compound term detection. Used by all search modules."""
import re
from typing import FrozenSet, List, Optional, Set
# Common multi-word prefixes stripped from all queries (identical across modules)
PREFIXES = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
# Multi-word suffixes (used by bird_x)
SUFFIXES = [
'best practices', 'use cases', 'prompt techniques',
'prompting techniques', 'prompting tips',
]
# Base noise words shared across most modules
NOISE_WORDS = frozenset({
# Articles/prepositions/conjunctions
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'and', 'or',
'of', 'in', 'on', 'for', 'with', 'about', 'to',
# Question words
'how', 'what', 'which', 'who', 'why', 'when', 'where',
'does', 'should', 'could', 'would',
# Research/meta descriptors
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
'practices', 'features', 'guide', 'tutorial',
'recommendations', 'advice', 'review', 'reviews',
'usecases', 'examples', 'comparison', 'versus', 'vs',
'plugin', 'plugins', 'skill', 'skills', 'tool', 'tools',
# Prompting meta words
'prompt', 'prompts', 'prompting', 'techniques', 'tips',
'tricks', 'methods', 'strategies', 'approaches',
# Action words
'using', 'uses', 'use',
# Misc filler
'people', 'saying', 'think', 'said', 'lately',
})
def extract_core_subject(
topic: str,
*,
noise: Optional[FrozenSet[str]] = None,
max_words: Optional[int] = None,
strip_suffixes: bool = False,
) -> str:
"""Extract core subject from a verbose search query.
Strips common question/meta prefixes and noise words to produce a
compact search-friendly query. Platforms customize via parameters.
Args:
topic: Raw user query
noise: Override noise word set (default: NOISE_WORDS)
max_words: Cap result to N words (default: no cap)
strip_suffixes: Also strip trailing multi-word suffixes (bird_x uses this)
Returns:
Cleaned query string
"""
text = topic.lower().strip()
if not text:
return text
# Phase 1: Strip multi-word prefixes (longest first, stop after first match)
for p in PREFIXES:
if text.startswith(p + ' '):
text = text[len(p):].strip()
break
# Phase 2: Strip multi-word suffixes (opt-in)
if strip_suffixes:
for s in SUFFIXES:
if text.endswith(' ' + s):
text = text[:-len(s)].strip()
break
# Phase 3: Filter individual noise words
noise_set = noise if noise is not None else NOISE_WORDS
words = text.split()
filtered = [w for w in words if w not in noise_set]
# Apply word cap if requested
if max_words is not None and filtered:
filtered = filtered[:max_words]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.') if not max_words else (result or topic.lower().strip())
def extract_compound_terms(topic: str) -> List[str]:
"""Detect multi-word terms that should be quoted in search queries.
Identifies:
- Hyphenated terms: "multi-agent", "vc-backed"
- Title-cased multi-word names: "Claude Code", "React Native"
Returns list of terms suitable for quoting (e.g., '"multi-agent"').
"""
terms: List[str] = []
# Hyphenated terms
for match in re.finditer(r'\b\w+-\w+(?:-\w+)*\b', topic):
terms.append(match.group())
# Title-cased sequences (2+ capitalized words in a row)
for match in re.finditer(r'(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+', topic):
terms.append(match.group())
return terms
+111
View File
@@ -0,0 +1,111 @@
"""Query type detection for source selection and scoring adjustments."""
import re
from typing import Literal
QueryType = Literal["product", "concept", "opinion", "how_to", "comparison", "breaking_news", "prediction"]
# Pattern-based classification (no LLM, no external deps)
_PRODUCT_PATTERNS = re.compile(
r"\b(price|pricing|cost|buy|purchase|deal|discount|subscription|plan|tier|free tier|alternative|prompt|prompts|prompting|template|templates)\b", re.I
)
_CONCEPT_PATTERNS = re.compile(
r"\b(what is|what are|explain|definition|how does|how do|overview|introduction|guide to|primer)\b", re.I
)
_OPINION_PATTERNS = re.compile(
r"\b(worth it|thoughts on|opinion|review|experience with|recommend|should i|pros and cons|good or bad)\b", re.I
)
_HOWTO_PATTERNS = re.compile(
r"\b(how to|tutorial|step by step|setup|install|configure|deploy|migrate|implement|build a|create a|prompting|prompts?|best practices|tips|examples|animation|animations|video workflow|render pipeline)\b",
re.I,
)
_COMPARISON_PATTERNS = re.compile(
r"\b(vs\.?|versus|compared to|comparison|better than|difference between|switch from)\b", re.I
)
_BREAKING_PATTERNS = re.compile(
r"\b(latest|breaking|just announced|launched|released|new|update|news|happened|today|this week)\b", re.I
)
_PREDICTION_PATTERNS = re.compile(
r"\b(predict|forecast|odds|chance|probability|election|outcome|bet on|market for)\b", re.I
)
def detect_query_type(topic: str) -> QueryType:
"""Classify a query into a type using pattern matching.
Returns the first match in priority order:
comparison > how_to > product > opinion > prediction > concept > breaking_news.
"""
# Most specific first
if _COMPARISON_PATTERNS.search(topic):
return "comparison"
if _HOWTO_PATTERNS.search(topic):
return "how_to"
if _PRODUCT_PATTERNS.search(topic):
return "product"
if _OPINION_PATTERNS.search(topic):
return "opinion"
if _PREDICTION_PATTERNS.search(topic):
return "prediction"
if _CONCEPT_PATTERNS.search(topic):
return "concept"
if _BREAKING_PATTERNS.search(topic):
return "breaking_news"
# Default: treat as breaking news (most common use case for "last 30 days")
return "breaking_news"
# Source tiering by query type.
# Tier 1: always run. Tier 2: run if available. Tier 3: opt-in only.
# Sources not listed are implicitly tier 3 (opt-in).
SOURCE_TIERS = {
"product": {"tier1": {"reddit", "x", "youtube"}, "tier2": {"web", "tiktok"}},
"concept": {"tier1": {"reddit", "hn", "web"}, "tier2": {"youtube", "x"}},
"opinion": {"tier1": {"reddit", "x"}, "tier2": {"youtube", "bluesky"}},
"how_to": {"tier1": {"youtube", "reddit", "hn"}, "tier2": {"web", "x"}},
"comparison": {"tier1": {"reddit", "hn", "youtube"}, "tier2": {"x", "web"}},
"breaking_news": {"tier1": {"x", "reddit", "web"}, "tier2": {"hn", "bluesky", "youtube"}},
"prediction": {"tier1": {"polymarket", "x", "reddit"}, "tier2": {"web", "hn", "youtube"}},
}
# WebSearch penalty adjustment by query type.
# Points subtracted from websearch score (0-100 scale). 0 = no penalty, 15 = full penalty.
# Concept/how_to queries benefit from authoritative web sources.
WEBSEARCH_PENALTY_BY_TYPE = {
"product": 15, # default: social discussion > blog posts
"concept": 0, # web docs are the best source
"opinion": 15, # social discussion > blog posts
"how_to": 5, # tutorials on web are valuable
"comparison": 10, # mix of social and web
"breaking_news": 10, # news sites are valuable
"prediction": 15, # social/market data > web articles
}
# Tiebreaker priority overrides by query type.
# Maps source type name to priority (lower = higher priority).
TIEBREAKER_BY_TYPE = {
"product": {"reddit": 0, "x": 1, "youtube": 2, "tiktok": 3, "instagram": 4, "hn": 5, "web": 6, "polymarket": 7},
"concept": {"hn": 0, "reddit": 1, "web": 2, "youtube": 3, "x": 4, "tiktok": 5, "instagram": 6, "polymarket": 7},
"opinion": {"reddit": 0, "x": 1, "bluesky": 2, "youtube": 3, "hn": 4, "tiktok": 5, "web": 6, "polymarket": 7},
"how_to": {"youtube": 0, "reddit": 1, "hn": 2, "web": 3, "x": 4, "tiktok": 5, "instagram": 6, "polymarket": 7},
"comparison": {"reddit": 0, "hn": 1, "youtube": 2, "x": 3, "web": 4, "tiktok": 5, "instagram": 6, "polymarket": 7},
"breaking_news": {"x": 0, "reddit": 1, "web": 2, "hn": 3, "bluesky": 4, "tiktok": 5, "youtube": 6, "polymarket": 7},
"prediction": {"polymarket": 0, "x": 1, "reddit": 2, "web": 3, "hn": 4, "bluesky": 5, "youtube": 6, "tiktok": 7},
}
def is_source_enabled(source: str, query_type: QueryType, explicitly_requested: bool = False) -> bool:
"""Check if a source should run for a given query type.
Tier 1 and Tier 2 sources are enabled. Tier 3 (unlisted) sources only run
if explicitly requested via --search flag. Truth Social is always opt-in.
"""
if source == "truthsocial":
return explicitly_requested
if explicitly_requested:
return True
tiers = SOURCE_TIERS.get(query_type, SOURCE_TIERS["breaking_news"])
return source in tiers["tier1"] or source in tiers["tier2"]
+42 -28
View File
@@ -48,7 +48,11 @@ DEPTH_CONFIG = {
},
}
# Stopwords for query extraction
from .query import extract_core_subject as _query_extract
from .query_type import detect_query_type
from .relevance import token_overlap_relevance
# Reddit-specific noise words (preserves original smaller set)
NOISE_WORDS = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
@@ -82,24 +86,7 @@ def _extract_core_subject(topic: str) -> str:
Strips meta/research words to keep only the core product/concept name.
"""
text = topic.lower().strip()
# Strip multi-word prefixes
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
words = text.split()
filtered = [w for w in words if w not in NOISE_WORDS]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.')
return _query_extract(topic, noise=NOISE_WORDS)
def expand_reddit_queries(topic: str, depth: str) -> List[str]:
@@ -121,10 +108,14 @@ def expand_reddit_queries(topic: str, depth: str) -> List[str]:
if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
queries.append(original_clean)
if depth in ("default", "deep"):
# Opinion/review variants help mostly for product and opinion queries.
# They contaminate broader searches like predictions or breaking news.
qtype = detect_query_type(topic)
if depth in ("default", "deep") and qtype in ("product", "opinion"):
queries.append(f"{core} worth it OR thoughts OR review")
if depth == "deep":
# Problem/bug variants are useful for tool workflows, not generic news.
if depth == "deep" and qtype in ("product", "opinion", "how_to"):
queries.append(f"{core} issues OR problems OR bug OR broken")
return queries
@@ -199,7 +190,7 @@ def _parse_date(created_utc) -> Optional[str]:
return None
def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global") -> Dict[str, Any]:
def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global", query: str = "") -> Dict[str, Any]:
"""Normalize a ScrapeCreators Reddit post to our internal format."""
permalink = post.get("permalink", "")
url = f"https://www.reddit.com{permalink}" if permalink else post.get("url", "")
@@ -208,10 +199,17 @@ def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global"
if url and "reddit.com" not in url:
url = ""
title = str(post.get("title", "")).strip()
selftext = str(post.get("selftext", ""))
# Score the title first, then let the body provide limited support.
# This keeps long selftexts from overpowering the visible topic signal.
relevance = _compute_post_relevance(query, title, selftext) if query else 0.7
return {
"id": f"R{idx}",
"reddit_id": post.get("id", ""),
"title": str(post.get("title", "")).strip(),
"title": title,
"url": url,
"subreddit": str(post.get("subreddit", "")).strip(),
"date": _parse_date(post.get("created_utc")),
@@ -220,12 +218,28 @@ def _normalize_post(post: Dict[str, Any], idx: int, source_label: str = "global"
"num_comments": post.get("num_comments", 0),
"upvote_ratio": post.get("upvote_ratio"),
},
"relevance": 0.7,
"relevance": relevance,
"why_relevant": f"Reddit {source_label} search",
"selftext": str(post.get("selftext", ""))[:500],
}
def _compute_post_relevance(query: str, title: str, selftext: str) -> float:
"""Compute Reddit relevance with title-first weighting.
Title should carry most of the weight because it is the visible summary the
user sees. Selftext can lift a marginal match, but it should not rescue a
weak or ambiguous title into the top ranks.
"""
title_score = token_overlap_relevance(query, title)
if not selftext.strip():
return title_score
body_score = token_overlap_relevance(query, selftext)
support_score = max(title_score, body_score)
return round(0.75 * title_score + 0.25 * support_score, 2)
def _global_search(
query: str,
token: str,
@@ -431,23 +445,23 @@ def search_reddit(
_log(f" -> {len(posts)} results")
all_raw_posts.extend(posts)
# Normalize all posts
# Normalize all posts (with query for relevance scoring)
core = _extract_core_subject(topic)
all_items = []
for i, post in enumerate(all_raw_posts):
item = _normalize_post(post, i + 1, "global")
item = _normalize_post(post, i + 1, "global", query=core)
all_items.append(item)
# === Phase 3: Subreddit Discovery + Targeted Search ===
discovered_subs = discover_subreddits(all_raw_posts, topic=topic, max_subs=config["subreddit_searches"])
_log(f"Discovered subreddits: {discovered_subs}")
core = _extract_core_subject(topic)
for sub in discovered_subs[:config["subreddit_searches"]]:
_log(f"Subreddit search: r/{sub} for '{core}'")
sub_posts = _subreddit_search(sub, core, token, sort="relevance", timeframe=timeframe)
_log(f" -> {len(sub_posts)} results from r/{sub}")
for j, post in enumerate(sub_posts):
item = _normalize_post(post, len(all_items) + j + 1, f"r/{sub}")
item = _normalize_post(post, len(all_items) + j + 1, f"r/{sub}", query=core)
all_items.append(item)
# === Phase 4: Deduplicate ===
+259
View File
@@ -0,0 +1,259 @@
"""Standalone Reddit public JSON search module.
Searches Reddit using the free public JSON endpoints (no API key required).
Promoted from last-resort fallback to robust primary free path.
Endpoints:
- Global: https://www.reddit.com/search.json?q={query}&sort=relevance&t=month&limit={limit}
- Subreddit: https://www.reddit.com/r/{sub}/search.json?q={query}&restrict_sr=on&sort=relevance&t=month
Handles 429 rate limits with exponential backoff, HTML anti-bot responses,
network timeouts, and missing subreddits.
"""
import json
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from typing import Any, Dict, List, Optional
USER_AGENT = "last30days/3.0 (research tool)"
# Depth-aware limits for thread counts
DEPTH_LIMITS = {
"quick": 10,
"default": 25,
"deep": 50,
}
MAX_RETRIES = 3
BASE_BACKOFF = 2.0 # seconds
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[RedditPublic] {msg}\n")
sys.stderr.flush()
def _url_encode(text: str) -> str:
"""URL-encode a query string."""
return urllib.parse.quote_plus(text)
def _fetch_json(url: str, timeout: int = 15) -> Optional[Dict[str, Any]]:
"""Fetch JSON from a URL with retry on 429 and error handling.
Returns parsed JSON dict, or None on unrecoverable failure.
"""
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/json",
}
req = urllib.request.Request(url, headers=headers)
for attempt in range(MAX_RETRIES):
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
content_type = resp.headers.get("Content-Type", "")
if "json" not in content_type and "text/html" in content_type:
_log(f"Anti-bot HTML response (Content-Type: {content_type})")
return None
body = resp.read().decode("utf-8")
return json.loads(body)
except urllib.error.HTTPError as e:
if e.code == 429:
delay = BASE_BACKOFF * (2 ** attempt)
retry_after = None
if hasattr(e, "headers"):
retry_after = e.headers.get("Retry-After")
if retry_after:
try:
delay = float(retry_after)
except ValueError:
pass
_log(f"429 rate limited, retry {attempt + 1}/{MAX_RETRIES} after {delay:.1f}s")
if attempt < MAX_RETRIES - 1:
time.sleep(delay)
continue
# Last attempt exhausted
_log("429 retries exhausted")
return None
elif e.code == 404:
_log(f"404 not found: {url}")
return None
elif e.code == 403:
_log(f"403 forbidden: {url}")
return None
else:
_log(f"HTTP {e.code}: {e.reason}")
return None
except (urllib.error.URLError, OSError, TimeoutError) as e:
_log(f"Network error: {e}")
return None
except json.JSONDecodeError as e:
_log(f"JSON decode error: {e}")
return None
return None
def _parse_posts(data: Optional[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Parse Reddit listing JSON into normalized post dicts."""
if not data:
return []
children = data.get("data", {}).get("children", [])
posts = []
for child in children:
if child.get("kind") != "t3":
continue
post = child.get("data", {})
permalink = str(post.get("permalink", "")).strip()
if not permalink or "/comments/" not in permalink:
continue
score = int(post.get("score", 0) or 0)
num_comments = int(post.get("num_comments", 0) or 0)
selftext = str(post.get("selftext", ""))
author = str(post.get("author", "[deleted]"))
created_utc = post.get("created_utc")
# Parse date
date_str = None
if created_utc:
try:
from datetime import datetime, timezone
dt = datetime.fromtimestamp(float(created_utc), tz=timezone.utc)
date_str = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError, OSError):
pass
posts.append({
"id": "", # Will be assigned after dedup
"title": str(post.get("title", "")).strip(),
"url": f"https://www.reddit.com{permalink}",
"score": score,
"num_comments": num_comments,
"subreddit": str(post.get("subreddit", "")).strip(),
"created_utc": float(created_utc) if created_utc else None,
"author": author if author not in ("[deleted]", "[removed]") else "[deleted]",
"selftext": selftext[:500] if selftext else "",
# Normalized fields matching ScrapeCreators output
"date": date_str,
"engagement": {
"score": score,
"num_comments": num_comments,
"upvote_ratio": post.get("upvote_ratio"),
},
"relevance": _compute_relevance(score, num_comments),
"why_relevant": "Reddit public search",
})
return posts
def _compute_relevance(score: int, num_comments: int) -> float:
"""Estimate relevance from engagement signals."""
score_component = min(1.0, max(0.0, score / 500.0))
comments_component = min(1.0, max(0.0, num_comments / 200.0))
return round((score_component * 0.6) + (comments_component * 0.4), 3)
def search(
query: str,
depth: str = "default",
subreddit: Optional[str] = None,
timeout: int = 15,
) -> List[Dict[str, Any]]:
"""Search Reddit via the public JSON endpoint.
Args:
query: Search query string
depth: 'quick', 'default', or 'deep' controls result limit
subreddit: Optional subreddit name (without r/) for scoped search
timeout: HTTP timeout in seconds
Returns:
List of normalized post dicts. Empty list on any failure.
"""
limit = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
encoded_query = _url_encode(query)
if subreddit:
sub = subreddit.lstrip("r/").strip()
url = (
f"https://www.reddit.com/r/{sub}/search.json"
f"?q={encoded_query}&restrict_sr=on&sort=relevance&t=month&limit={limit}&raw_json=1"
)
else:
url = (
f"https://www.reddit.com/search.json"
f"?q={encoded_query}&sort=relevance&t=month&limit={limit}&raw_json=1"
)
data = _fetch_json(url, timeout=timeout)
posts = _parse_posts(data)
# Dedupe by URL and assign IDs
seen_urls = set()
unique = []
for post in posts:
if post["url"] not in seen_urls:
seen_urls.add(post["url"])
unique.append(post)
for i, post in enumerate(unique):
post["id"] = f"R{i + 1}"
return unique[:limit]
def search_reddit_public(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""High-level Reddit public search matching the openai_reddit interface.
Runs global search, deduplicates, filters by date range, and sorts
by engagement. Compatible as a drop-in replacement in the fallback chain.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
Returns:
List of normalized item dicts matching ScrapeCreators output format.
"""
results = search(topic, depth=depth)
# Date filter: keep posts in range or with unknown dates
filtered = []
for item in results:
d = item.get("date")
if d is None or (from_date <= d <= to_date):
filtered.append(item)
# Sort by engagement (score desc)
filtered.sort(
key=lambda x: x.get("engagement", {}).get("score", 0),
reverse=True,
)
# Re-index IDs
for i, item in enumerate(filtered):
item["id"] = f"R{i + 1}"
return filtered
+148
View File
@@ -0,0 +1,148 @@
"""Shared token-overlap relevance scoring for search result ranking.
The score is intentionally query-centric:
- exact phrase matches should score very high
- partial matches should pay a meaningful penalty
- matches on generic words alone ("odds", "review") should not pass as relevant
"""
import re
from typing import List, Optional, Set
# Stopwords for relevance computation (common English words that dilute token overlap)
STOPWORDS = frozenset({
'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
'all', 'just', 'get', 'has', 'have', 'was', 'will',
})
# Synonym groups for relevance scoring (bidirectional expansion)
# Superset of all platform-specific synonym dicts
SYNONYMS = {
'hip': {'rap', 'hiphop'},
'hop': {'rap', 'hiphop'},
'rap': {'hip', 'hop', 'hiphop'},
'hiphop': {'rap', 'hip', 'hop'},
'js': {'javascript'},
'javascript': {'js'},
'ts': {'typescript'},
'typescript': {'ts'},
'ai': {'artificial', 'intelligence'},
'ml': {'machine', 'learning'},
'react': {'reactjs'},
'reactjs': {'react'},
'svelte': {'sveltejs'},
'sveltejs': {'svelte'},
'vue': {'vuejs'},
'vuejs': {'vue'},
}
# Generic query words that should not carry relevance on their own.
# They still help when paired with stronger entity/topic matches.
LOW_SIGNAL_QUERY_TOKENS = frozenset({
'advice', 'animation', 'animations', 'best', 'chance', 'chances',
'code', 'compare', 'comparison', 'differences', 'explain', 'guide',
'guides', 'how', 'latest', 'news', 'odds', 'opinion', 'opinions',
'prediction', 'predictions', 'probability', 'probabilities', 'prompt',
'prompting', 'prompts', 'rate', 'review', 'reviews', 'thoughts',
'tip', 'tips', 'tutorial', 'tutorials', 'update', 'updates', 'use',
'using', 'versus', 'vs', 'worth',
})
def tokenize(text: str) -> Set[str]:
"""Lowercase, strip punctuation, remove stopwords, drop single-char tokens.
Expands tokens with synonyms for better cross-domain matching.
"""
words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
expanded = set(tokens)
for t in tokens:
if t in SYNONYMS:
expanded.update(SYNONYMS[t])
return expanded
def _normalize_phrase(text: str) -> str:
"""Normalize text for phrase containment checks."""
return ' '.join(re.sub(r'[^\w\s]', ' ', text.lower()).split())
def token_overlap_relevance(
query: str,
text: str,
hashtags: Optional[List[str]] = None,
) -> float:
"""Compute a query-centric relevance score between 0.0 and 1.0.
The score combines:
- query coverage
- informative-token coverage
- a small precision term to penalize extra noise
- an exact phrase bonus
Generic tokens alone are capped below the post-retrieval 0.3 threshold.
Args:
query: Search query
text: Content text to match against
hashtags: Optional list of hashtags (TikTok/Instagram). Concatenated
hashtags are split to match query tokens (e.g. "claudecode" matches "claude").
Returns:
Float between 0.0 and 1.0 (0.5 for empty queries)
"""
q_tokens = tokenize(query)
# Combine text and hashtags for matching
combined = text
if hashtags:
combined = f"{text} {' '.join(hashtags)}"
t_tokens = tokenize(combined)
# Split concatenated hashtags (e.g., "claudecode" -> matches "claude", "code")
if hashtags:
for tag in hashtags:
tag_lower = tag.lower()
for qt in q_tokens:
if qt in tag_lower and qt != tag_lower:
t_tokens.add(qt)
if not q_tokens:
return 0.5 # Neutral fallback for empty/stopword-only queries
overlap_tokens = q_tokens & t_tokens
overlap = len(overlap_tokens)
if overlap == 0:
return 0.0
informative_q_tokens = {t for t in q_tokens if t not in LOW_SIGNAL_QUERY_TOKENS}
if not informative_q_tokens:
informative_q_tokens = q_tokens
coverage = overlap / len(q_tokens)
informative_overlap = len(informative_q_tokens & t_tokens) / len(informative_q_tokens)
precision_denominator = min(len(t_tokens), len(q_tokens) + 4) or 1
precision = overlap / precision_denominator
phrase_bonus = 0.0
normalized_query = _normalize_phrase(query)
normalized_text = _normalize_phrase(combined)
if normalized_query and normalized_query in normalized_text:
phrase_bonus = 0.12 if len(normalized_query.split()) > 1 else 0.16
base = (
0.55 * (coverage ** 1.35) +
0.25 * informative_overlap +
0.20 * precision
)
# If we only matched generic query words, keep the score below the
# normal relevance filter threshold so these do not survive by default.
if informative_q_tokens and not (informative_q_tokens & t_tokens):
return round(min(0.24, base), 2)
return round(min(1.0, base + phrase_bonus), 2)
+184 -8
View File
@@ -30,6 +30,10 @@ def _xref_tag(item) -> str:
source_names.add('Instagram')
elif ref_id.startswith('HN'):
source_names.add('HN')
elif ref_id.startswith('BS'):
source_names.add('Bluesky')
elif ref_id.startswith('TS'):
source_names.add('Truth Social')
elif ref_id.startswith('PM'):
source_names.add('Polymarket')
elif ref_id.startswith('W'):
@@ -59,13 +63,15 @@ def _assess_data_freshness(report: schema.Report) -> dict:
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from)
bsky_recent = sum(1 for b in report.bluesky if b.date and b.date >= report.range_from)
ts_recent = sum(1 for ts in report.truthsocial if ts.date and ts.date >= report.range_from)
pm_recent = sum(1 for p in report.polymarket if p.date and p.date >= report.range_from)
tiktok_recent = sum(1 for t in report.tiktok if t.date and t.date >= report.range_from)
ig_recent = sum(1 for ig in report.instagram if ig.date and ig.date >= report.range_from)
total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent + tiktok_recent + ig_recent
total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket) + len(report.tiktok) + len(report.instagram)
total_recent = reddit_recent + x_recent + web_recent + hn_recent + bsky_recent + ts_recent + pm_recent + tiktok_recent + ig_recent
total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.bluesky) + len(report.truthsocial) + len(report.polymarket) + len(report.tiktok) + len(report.instagram)
return {
"reddit_recent": reddit_recent,
@@ -109,7 +115,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
lines.append("")
lines.append("---")
lines.append("**⚡ Want better results?** Add API keys to unlock Reddit, TikTok, Instagram & X data:")
lines.append("- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — real upvotes, comments, views")
lines.append("- `SCRAPECREATORS_API_KEY` → Reddit + TikTok + Instagram (one key, all three!) — 100 free calls, no CC — scrapecreators.com (no affiliation)")
lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
lines.append("- `OPENAI_API_KEY` (legacy) → Reddit threads (slower, higher cost)")
lines.append("- Edit `~/.config/last30days/.env` to add keys")
@@ -137,7 +143,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
lines.append("*💡 Tip: Add an xAI key (`XAI_API_KEY`) for X/Twitter data and better triangulation.*")
lines.append("")
elif report.mode == "x-only" and missing_keys in ("reddit", "none"):
lines.append("*💡 Tip: Add `SCRAPECREATORS_API_KEY` for Reddit + TikTok + Instagram data (one key, all three) and better triangulation.*")
lines.append("*💡 Tip: Add `SCRAPECREATORS_API_KEY` for Reddit + TikTok + Instagram data (one key, all three) — 100 free calls, no CC — scrapecreators.com (no affiliation)*")
lines.append("")
# Reddit items
@@ -251,11 +257,15 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
lines.append(f"**{item.id}** (score:{item.score}) {item.channel_name}{date_str}{eng_str}{_xref_tag(item)}")
lines.append(f" {item.title}")
lines.append(f" {item.url}")
if item.transcript_highlights:
lines.append(" Highlights:")
for hl in item.transcript_highlights[:5]:
lines.append(f' - "{hl}"')
if item.transcript_snippet:
snippet = item.transcript_snippet[:200]
if len(item.transcript_snippet) > 200:
snippet += "..."
lines.append(f" Transcript: {snippet}")
word_count = len(item.transcript_snippet.split())
lines.append(f" <details><summary>Full transcript ({word_count} words)</summary>")
lines.append(f" {item.transcript_snippet}")
lines.append(" </details>")
lines.append(f" *{item.why_relevant}*")
lines.append("")
@@ -367,6 +377,78 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
lines.append("")
# Bluesky items
if report.bluesky_error:
lines.append("### Bluesky Posts")
lines.append("")
lines.append(f"**ERROR:** {report.bluesky_error}")
lines.append("")
elif report.bluesky:
lines.append("### Bluesky Posts")
lines.append("")
for item in report.bluesky[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.likes is not None:
parts.append(f"{eng.likes}lk")
if eng.reposts is not None:
parts.append(f"{eng.reposts}rp")
if eng.replies is not None:
parts.append(f"{eng.replies}re")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else ""
lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{eng_str}{_xref_tag(item)}")
if item.text:
snippet = item.text[:200]
if len(item.text) > 200:
snippet += "..."
lines.append(f" {snippet}")
if item.url:
lines.append(f" {item.url}")
lines.append(f" *{item.why_relevant}*")
lines.append("")
# Truth Social items
if report.truthsocial_error:
lines.append("### Truth Social Posts")
lines.append("")
lines.append(f"**ERROR:** {report.truthsocial_error}")
lines.append("")
elif report.truthsocial:
lines.append("### Truth Social Posts")
lines.append("")
for item in report.truthsocial[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.likes is not None:
parts.append(f"{eng.likes}lk")
if eng.reposts is not None:
parts.append(f"{eng.reposts}rp")
if eng.replies is not None:
parts.append(f"{eng.replies}re")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else ""
lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{eng_str}{_xref_tag(item)}")
if item.text:
snippet = item.text[:200]
if len(item.text) > 200:
snippet += "..."
lines.append(f" {snippet}")
if item.url:
lines.append(f" {item.url}")
lines.append(f" *{item.why_relevant}*")
lines.append("")
# Polymarket items
if report.polymarket_error:
lines.append("### Prediction Markets (Polymarket)")
@@ -443,6 +525,28 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
return "\n".join(lines)
def render_quality_nudge(quality: dict) -> str:
"""Render the quality score nudge block.
Args:
quality: Dict from quality_nudge.compute_quality_score()
Returns:
Markdown string with quality nudge, or empty string if no nudge.
"""
nudge_text = quality.get("nudge_text")
if not nudge_text:
return ""
lines = []
lines.append("---")
lines.append(f"**🔍 Research Coverage: {quality['score_pct']}%**")
lines.append("")
lines.append(nudge_text)
lines.append("")
return "\n".join(lines)
def render_source_status(report: schema.Report, source_info: dict = None) -> str:
"""Render source status footer showing what was used/skipped and why.
@@ -510,6 +614,20 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
lines.append(f" ✅ Instagram: {len(report.instagram)} reels ({with_captions} with captions)")
# Hide when zero results
# Xiaohongshu (from Web source bucket)
xhs_count = 0
if report.web:
xhs_count = sum(
1 for w in report.web
if getattr(w, "source_domain", "").lower().endswith("xiaohongshu.com")
)
if xhs_count > 0:
lines.append(f" ✅ Xiaohongshu: {xhs_count} notes")
else:
reason = source_info.get("xiaohongshu_skip_reason")
if reason:
lines.append(f" ⚡ Xiaohongshu: {reason}")
# Hacker News
if report.hackernews_error:
lines.append(f" ❌ HN: error - {report.hackernews_error}")
@@ -517,6 +635,20 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
lines.append(f" ✅ HN: {len(report.hackernews)} stories")
# Hide when zero results
# Bluesky
if report.bluesky_error:
lines.append(f" ❌ Bluesky: error - {report.bluesky_error}")
elif report.bluesky:
lines.append(f" ✅ Bluesky: {len(report.bluesky)} posts")
# Hide when zero results
# Truth Social
if report.truthsocial_error:
lines.append(f" ❌ Truth Social: error - {report.truthsocial_error}")
elif report.truthsocial:
lines.append(f" ✅ Truth Social: {len(report.truthsocial)} posts")
# Hide when zero results
# Polymarket
if report.polymarket_error:
lines.append(f" ❌ Polymarket: error - {report.polymarket_error}")
@@ -567,6 +699,10 @@ def render_context_snippet(report: schema.Report) -> str:
all_items.append((item.score, "Instagram", item.text[:50] + "...", item.url))
for item in report.hackernews[:5]:
all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
for item in report.bluesky[:5]:
all_items.append((item.score, "Bluesky", item.text[:50] + "...", item.url))
for item in report.truthsocial[:5]:
all_items.append((item.score, "Truth Social", item.text[:50] + "...", item.url))
for item in report.polymarket[:5]:
all_items.append((item.score, "Polymarket", item.question[:50] + "...", item.url))
for item in report.web[:5]:
@@ -740,6 +876,46 @@ def render_full_report(report: schema.Report) -> str:
lines.append("")
# Bluesky section
if report.bluesky:
lines.append("## Bluesky Posts")
lines.append("")
for item in report.bluesky:
lines.append(f"### {item.id}: @{item.author_handle}")
lines.append("")
lines.append(f"- **URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'}")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
if item.engagement:
eng = item.engagement
lines.append(f"- **Engagement:** {eng.likes or '?'} likes, {eng.reposts or '?'} reposts, {eng.replies or '?'} replies")
lines.append("")
lines.append(f"> {item.text[:300]}")
lines.append("")
# Truth Social section
if report.truthsocial:
lines.append("## Truth Social Posts")
lines.append("")
for item in report.truthsocial:
lines.append(f"### {item.id}: @{item.author_handle}")
lines.append("")
lines.append(f"- **URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'}")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
if item.engagement:
eng = item.engagement
lines.append(f"- **Engagement:** {eng.likes or '?'} likes, {eng.reposts or '?'} reposts, {eng.replies or '?'} replies")
lines.append("")
lines.append(f"> {item.text[:300]}")
lines.append("")
# Polymarket section
if report.polymarket:
lines.append("## Prediction Markets (Polymarket)")
+182
View File
@@ -0,0 +1,182 @@
"""
Safari binary cookie extractor for macOS.
Parses ~/Library/Cookies/Cookies.binarycookies (unencrypted binary format)
using only stdlib. Zero pip dependencies.
Reference: github.com/mdegrazia/Safari-Binary-Cookie-Parser
"""
from __future__ import annotations
import io
import struct
import sys
from pathlib import Path
# Mac epoch: 2001-01-01 00:00:00 UTC (not used for filtering, but documented)
_MAC_EPOCH_OFFSET = 978307200 # seconds between Unix epoch and Mac epoch
_MAGIC = b"cook"
def _read_null_terminated(data: bytes, offset: int) -> str:
"""Read a null-terminated string from data starting at offset."""
end = data.find(b"\x00", offset)
if end == -1:
end = len(data)
return data[offset:end].decode("utf-8", errors="replace")
def _parse_cookie_record(data: bytes) -> dict | None:
"""Parse a single cookie record. Returns dict with url, name, value, path or None."""
if len(data) < 44:
return None
try:
(size,) = struct.unpack("<I", data[0:4])
# flags at offset 4 (4 bytes, little-endian) — not needed for extraction
(url_offset,) = struct.unpack("<I", data[16:20])
(name_offset,) = struct.unpack("<I", data[20:24])
(path_offset,) = struct.unpack("<I", data[24:28])
(value_offset,) = struct.unpack("<I", data[28:32])
# expiry at offset 40 (8-byte double, little-endian) — not needed for filtering
# creation at offset 48 (8-byte double, little-endian) — not needed
url = _read_null_terminated(data, url_offset)
name = _read_null_terminated(data, name_offset)
path = _read_null_terminated(data, path_offset)
value = _read_null_terminated(data, value_offset)
return {"url": url, "name": name, "value": value, "path": path}
except (struct.error, IndexError, UnicodeDecodeError):
return None
def _parse_page(page_data: bytes) -> list[dict]:
"""Parse a single page of cookies. Returns list of cookie dicts."""
cookies = []
if len(page_data) < 8:
return cookies
# Page header: 4 bytes (always 00 00 01 00), then 4-byte LE cookie count
try:
(num_cookies,) = struct.unpack("<I", page_data[4:8])
except struct.error:
return cookies
# Sanity check
if num_cookies > 10000:
return cookies
# Cookie offsets: array of 4-byte LE uint32 starting at offset 8
offsets_end = 8 + num_cookies * 4
if offsets_end > len(page_data):
return cookies
for i in range(num_cookies):
off_start = 8 + i * 4
try:
(cookie_offset,) = struct.unpack("<I", page_data[off_start : off_start + 4])
except struct.error:
continue
if cookie_offset >= len(page_data):
continue
cookie_data = page_data[cookie_offset:]
record = _parse_cookie_record(cookie_data)
if record:
cookies.append(record)
return cookies
def extract_safari_cookies_macos(
domain: str, cookie_names: list[str]
) -> dict[str, str] | None:
"""
Extract cookies from Safari on macOS.
Args:
domain: Domain to match (substring match, e.g. "x.com")
cookie_names: List of cookie names to extract (e.g. ["auth_token", "ct0"])
Returns:
Dict mapping cookie name to value for found cookies, or None on failure.
"""
if sys.platform != "darwin":
return None
cookie_path = Path.home() / "Library" / "Cookies" / "Cookies.binarycookies"
try:
raw = cookie_path.read_bytes()
except FileNotFoundError:
return None
except PermissionError:
print(
"[safari] Permission denied reading Cookies.binarycookies. "
"Enable Full Disk Access for Terminal in System Settings > "
"Privacy & Security > Full Disk Access.",
file=sys.stderr,
)
return None
except OSError:
return None
return _parse_binary_cookies(raw, domain, cookie_names)
def _parse_binary_cookies(
raw: bytes, domain: str, cookie_names: list[str]
) -> dict[str, str] | None:
"""Parse raw binary cookie data. Separated for testability."""
if len(raw) < 8:
return None
# Validate magic
if raw[:4] != _MAGIC:
return None
try:
(num_pages,) = struct.unpack(">I", raw[4:8])
except struct.error:
return None
if num_pages > 100000:
return None
# Read page sizes (big-endian uint32 array)
page_sizes_end = 8 + num_pages * 4
if page_sizes_end > len(raw):
return None
page_sizes = []
for i in range(num_pages):
off = 8 + i * 4
try:
(ps,) = struct.unpack(">I", raw[off : off + 4])
page_sizes.append(ps)
except struct.error:
return None
# Parse each page
names_set = set(cookie_names)
result: dict[str, str] = {}
offset = page_sizes_end
for ps in page_sizes:
if offset + ps > len(raw):
break
page_data = raw[offset : offset + ps]
cookies = _parse_page(page_data)
for c in cookies:
# Substring match on domain (handles leading dots like ".x.com")
if domain in c["url"] and c["name"] in names_set:
result[c["name"]] = c["value"]
offset += ps
if not result:
return None
return result
+112
View File
@@ -210,6 +210,7 @@ class YouTubeItem:
date_confidence: str = "high" # YouTube dates are always reliable
engagement: Optional[Engagement] = None
transcript_snippet: str = ""
transcript_highlights: List[str] = field(default_factory=list)
relevance: float = 0.7
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
@@ -226,6 +227,7 @@ class YouTubeItem:
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'transcript_snippet': self.transcript_snippet,
'transcript_highlights': self.transcript_highlights,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
@@ -355,6 +357,80 @@ class HackerNewsItem:
return d
@dataclass
class BlueskyItem:
"""Normalized Bluesky post."""
id: str # "BS1", "BS2", ...
text: str
url: str # bsky.app permalink
author_handle: str # user.bsky.social
display_name: str
date: Optional[str] = None
date_confidence: str = "high" # AT Protocol has exact timestamps
engagement: Optional[Engagement] = None # likes, reposts, replies, quotes
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
cross_refs: List[str] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
d = {
'id': self.id,
'text': self.text,
'url': self.url,
'author_handle': self.author_handle,
'display_name': self.display_name,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
if self.cross_refs:
d['cross_refs'] = self.cross_refs
return d
@dataclass
class TruthSocialItem:
"""Normalized Truth Social post."""
id: str # "TS1", "TS2", ...
text: str
url: str # truthsocial.com permalink
author_handle: str # username
display_name: str
date: Optional[str] = None
date_confidence: str = "high" # Mastodon API has exact timestamps
engagement: Optional[Engagement] = None # likes, reposts, replies
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
cross_refs: List[str] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
d = {
'id': self.id,
'text': self.text,
'url': self.url,
'author_handle': self.author_handle,
'display_name': self.display_name,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
if self.cross_refs:
d['cross_refs'] = self.cross_refs
return d
@dataclass
class PolymarketItem:
"""Normalized Polymarket prediction market item."""
@@ -415,6 +491,8 @@ class Report:
tiktok: List[TikTokItem] = field(default_factory=list)
instagram: List[InstagramItem] = field(default_factory=list)
hackernews: List[HackerNewsItem] = field(default_factory=list)
bluesky: List[BlueskyItem] = field(default_factory=list)
truthsocial: List[TruthSocialItem] = field(default_factory=list)
polymarket: List[PolymarketItem] = field(default_factory=list)
best_practices: List[str] = field(default_factory=list)
prompt_pack: List[str] = field(default_factory=list)
@@ -427,6 +505,8 @@ class Report:
tiktok_error: Optional[str] = None
instagram_error: Optional[str] = None
hackernews_error: Optional[str] = None
bluesky_error: Optional[str] = None
truthsocial_error: Optional[str] = None
polymarket_error: Optional[str] = None
# Handle resolution
resolved_x_handle: Optional[str] = None
@@ -452,6 +532,8 @@ class Report:
'tiktok': [t.to_dict() for t in self.tiktok],
'instagram': [ig.to_dict() for ig in self.instagram],
'hackernews': [h.to_dict() for h in self.hackernews],
'bluesky': [b.to_dict() for b in self.bluesky],
'truthsocial': [ts.to_dict() for ts in self.truthsocial],
'polymarket': [p.to_dict() for p in self.polymarket],
'best_practices': self.best_practices,
'prompt_pack': self.prompt_pack,
@@ -473,6 +555,10 @@ class Report:
d['instagram_error'] = self.instagram_error
if self.hackernews_error:
d['hackernews_error'] = self.hackernews_error
if self.bluesky_error:
d['bluesky_error'] = self.bluesky_error
if self.truthsocial_error:
d['truthsocial_error'] = self.truthsocial_error
if self.polymarket_error:
d['polymarket_error'] = self.polymarket_error
if self.from_cache:
@@ -571,6 +657,7 @@ class Report:
date_confidence=y.get('date_confidence', 'high'),
engagement=eng,
transcript_snippet=y.get('transcript_snippet', ''),
transcript_highlights=y.get('transcript_highlights', []),
relevance=y.get('relevance', 0.7),
why_relevant=y.get('why_relevant', ''),
subs=subs,
@@ -652,6 +739,29 @@ class Report:
cross_refs=h.get('cross_refs', []),
))
# Reconstruct Truth Social items (backward compat: key may not exist)
ts_items = []
for ts in data.get('truthsocial', []):
eng = None
if ts.get('engagement'):
eng = Engagement(**ts['engagement'])
subs = SubScores(**ts.get('subs', {})) if ts.get('subs') else SubScores()
ts_items.append(TruthSocialItem(
id=ts['id'],
text=ts['text'],
url=ts['url'],
author_handle=ts.get('author_handle', ''),
display_name=ts.get('display_name', ''),
date=ts.get('date'),
date_confidence=ts.get('date_confidence', 'high'),
engagement=eng,
relevance=ts.get('relevance', 0.5),
why_relevant=ts.get('why_relevant', ''),
subs=subs,
score=ts.get('score', 0),
cross_refs=ts.get('cross_refs', []),
))
# Reconstruct Polymarket items (backward compat: key may not exist)
pm_items = []
for p in data.get('polymarket', []):
@@ -693,6 +803,7 @@ class Report:
tiktok=tiktok_items,
instagram=ig_items,
hackernews=hn_items,
truthsocial=ts_items,
polymarket=pm_items,
best_practices=data.get('best_practices', []),
prompt_pack=data.get('prompt_pack', []),
@@ -704,6 +815,7 @@ class Report:
tiktok_error=data.get('tiktok_error'),
instagram_error=data.get('instagram_error'),
hackernews_error=data.get('hackernews_error'),
truthsocial_error=data.get('truthsocial_error'),
polymarket_error=data.get('polymarket_error'),
resolved_x_handle=data.get('resolved_x_handle'),
from_cache=data.get('from_cache', False),
+183 -34
View File
@@ -4,16 +4,25 @@ import math
from typing import List, Optional, Union
from . import dates, schema
from .query_type import QueryType, WEBSEARCH_PENALTY_BY_TYPE, TIEBREAKER_BY_TYPE
# Score weights for Reddit/X (has engagement)
WEIGHT_RELEVANCE = 0.45
WEIGHT_RECENCY = 0.25
WEIGHT_ENGAGEMENT = 0.30
# WebSearch weights (no engagement, reweighted to 100%)
# Polymarket needs stronger semantic weighting because volume/liquidity already
# show up as engagement and lightly influence parse-time relevance.
PM_WEIGHT_RELEVANCE = 0.60
PM_WEIGHT_RECENCY = 0.20
PM_WEIGHT_ENGAGEMENT = 0.20
# WebSearch weights (no engagement data available)
WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# Default web search penalty (fallback when query_type is not provided).
# Per-type penalties in query_type.WEBSEARCH_PENALTY_BY_TYPE.
WEBSEARCH_SOURCE_PENALTY = 15
# WebSearch date confidence adjustments
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date (high confidence)
@@ -468,6 +477,122 @@ def score_hackernews_items(items: List[schema.HackerNewsItem]) -> List[schema.Ha
return items
def compute_bluesky_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for Bluesky item.
Formula: 0.40*log1p(likes) + 0.30*log1p(reposts) + 0.20*log1p(replies) + 0.10*log1p(quotes)
Likes are primary signal; reposts indicate reach; replies indicate discussion depth.
"""
if engagement is None:
return None
if engagement.likes is None and engagement.reposts is None:
return None
likes = log1p_safe(engagement.likes)
reposts = log1p_safe(engagement.reposts)
replies = log1p_safe(engagement.replies)
quotes = log1p_safe(engagement.quotes)
return 0.40 * likes + 0.30 * reposts + 0.20 * replies + 0.10 * quotes
def score_bluesky_items(items: List[schema.BlueskyItem]) -> List[schema.BlueskyItem]:
"""Compute scores for Bluesky items.
Uses same weight structure as Reddit/X (relevance + recency + engagement).
"""
if not items:
return items
eng_raw = [compute_bluesky_engagement_raw(item.engagement) for item in items]
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
item.score = max(0, min(100, int(overall)))
return items
def compute_truthsocial_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for Truth Social item.
Formula: 0.45*log1p(likes) + 0.30*log1p(reposts) + 0.25*log1p(replies)
Likes are primary signal; reposts indicate reach; replies indicate discussion.
"""
if engagement is None:
return None
if engagement.likes is None and engagement.reposts is None:
return None
likes = log1p_safe(engagement.likes)
reposts = log1p_safe(engagement.reposts)
replies = log1p_safe(engagement.replies)
return 0.45 * likes + 0.30 * reposts + 0.25 * replies
def score_truthsocial_items(items: List[schema.TruthSocialItem]) -> List[schema.TruthSocialItem]:
"""Compute scores for Truth Social items."""
if not items:
return items
eng_raw = [compute_truthsocial_engagement_raw(item.engagement) for item in items]
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
item.score = max(0, min(100, int(overall)))
return items
def compute_polymarket_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for Polymarket item.
@@ -513,9 +638,9 @@ def score_polymarket_items(items: List[schema.PolymarketItem]) -> List[schema.Po
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
PM_WEIGHT_RELEVANCE * rel_score +
PM_WEIGHT_RECENCY * rec_score +
PM_WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
@@ -526,19 +651,17 @@ def score_polymarket_items(items: List[schema.PolymarketItem]) -> List[schema.Po
return items
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
def score_websearch_items(items: List[schema.WebSearchItem], query_type: QueryType = None) -> List[schema.WebSearchItem]:
"""Compute scores for WebSearch items WITHOUT engagement metrics.
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
This ensures WebSearch items rank below comparable Reddit/X items.
Date confidence adjustments:
- High confidence (URL-verified date): +10 bonus
- Med confidence (snippet-extracted date): no change
- Low confidence (no date signals): -20 penalty
Uses reweighted formula: 55% relevance + 45% recency - penalty.
Penalty varies by query type: concept queries get 0 penalty (web docs
are authoritative), while product/opinion queries get full 15pt penalty
(social discussion is more valuable).
Args:
items: List of WebSearch items
query_type: Query classification for penalty adjustment
Returns:
Items with updated scores
@@ -566,8 +689,9 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
WEBSEARCH_WEIGHT_RECENCY * rec_score
)
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
overall -= WEBSEARCH_SOURCE_PENALTY
# Apply source penalty (varies by query type)
penalty = WEBSEARCH_PENALTY_BY_TYPE.get(query_type, WEBSEARCH_SOURCE_PENALTY) if query_type else WEBSEARCH_SOURCE_PENALTY
overall -= penalty
# Apply date confidence adjustments
# High confidence (URL-verified): reward with bonus
@@ -583,15 +707,36 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
return items
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
"""Sort items by score (descending), then date, then source priority.
_ITEM_SOURCE_MAP = {
schema.RedditItem: "reddit",
schema.XItem: "x",
schema.YouTubeItem: "youtube",
schema.TikTokItem: "tiktok",
schema.InstagramItem: "instagram",
schema.HackerNewsItem: "hn",
schema.BlueskyItem: "bluesky",
schema.TruthSocialItem: "truthsocial",
schema.PolymarketItem: "polymarket",
}
_DEFAULT_TIEBREAKER = {"reddit": 0, "x": 1, "youtube": 2, "tiktok": 3, "instagram": 4, "hn": 5, "bluesky": 6, "truthsocial": 7, "polymarket": 8, "web": 9}
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.TikTokItem, schema.InstagramItem, schema.HackerNewsItem, schema.BlueskyItem, schema.TruthSocialItem, schema.PolymarketItem]], query_type: QueryType = None) -> List:
"""Sort items by score (descending), then date, then source tiebreaker.
Tiebreaker (tertiary sort key, after score and date): source priority
varies by query type. YouTube ranks first for how_to, X ranks first
for breaking_news, Polymarket ranks first for prediction.
Args:
items: List of items to sort
query_type: Query classification for tiebreaker adjustment
Returns:
Sorted items
"""
tiebreaker = TIEBREAKER_BY_TYPE.get(query_type, _DEFAULT_TIEBREAKER) if query_type else _DEFAULT_TIEBREAKER
def sort_key(item):
# Primary: score descending (negate for descending)
score = -item.score
@@ -600,23 +745,9 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
date = item.date or "0000-00-00"
date_key = -int(date.replace("-", ""))
# Tertiary: source priority (Reddit > X > YouTube > TikTok > HN > Polymarket > WebSearch)
if isinstance(item, schema.RedditItem):
source_priority = 0
elif isinstance(item, schema.XItem):
source_priority = 1
elif isinstance(item, schema.YouTubeItem):
source_priority = 2
elif isinstance(item, schema.TikTokItem):
source_priority = 3
elif isinstance(item, schema.InstagramItem):
source_priority = 4
elif isinstance(item, schema.HackerNewsItem):
source_priority = 5
elif isinstance(item, schema.PolymarketItem):
source_priority = 6
else: # WebSearchItem
source_priority = 7
# Tertiary: query-type-aware source priority
source_name = _ITEM_SOURCE_MAP.get(type(item), "web")
source_priority = tiebreaker.get(source_name, 99)
# Quaternary: title/text for stability
text = getattr(item, "title", "") or getattr(item, "text", "")
@@ -624,3 +755,21 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
return (score, date_key, source_priority, text)
return sorted(items, key=sort_key)
def relevance_filter(items, source_name: str, threshold: float = 0.3):
"""Filter items below relevance threshold with minimum-result guarantee.
Items with no relevance attribute are treated as 0.0 (fail the filter).
If all items are below threshold, keeps the top 3 by relevance.
Lists with 3 or fewer items are returned unchanged.
"""
import sys
if len(items) <= 3:
return items
passed = [i for i in items if getattr(i, 'relevance', 0.0) >= threshold]
if not passed:
print(f"[{source_name} WARNING] All results below relevance {threshold}, keeping top 3", file=sys.stderr)
by_rel = sorted(items, key=lambda x: getattr(x, 'relevance', 0.0), reverse=True)
return by_rel[:3]
return passed
+182
View File
@@ -0,0 +1,182 @@
"""X/Twitter search via ScrapeCreators API for /last30days.
Uses ScrapeCreators REST API to search Twitter/X by keyword.
Same API key as Reddit, TikTok, and Instagram - one key covers all social sources.
Requires SCRAPECREATORS_API_KEY in config.
API docs: https://scrapecreators.com/docs
"""
import sys
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
try:
import requests as _requests
except ImportError:
_requests = None
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/twitter"
DEPTH_CONFIG = {
"quick": {"results_per_page": 10},
"default": {"results_per_page": 20},
"deep": {"results_per_page": 40},
}
from .relevance import token_overlap_relevance as _compute_relevance
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Twitter search."""
from .query import extract_core_subject
_SC_X_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features', 'recommendations', 'advice',
})
return extract_core_subject(topic, noise=_SC_X_NOISE)
def _log(msg: str):
if sys.stderr.isatty():
sys.stderr.write(f"[X/SC] {msg}\n")
sys.stderr.flush()
def _sc_headers(token: str) -> Dict[str, str]:
return {
"x-api-key": token,
"Content-Type": "application/json",
}
def _parse_date(item: Dict[str, Any]) -> Optional[str]:
"""Parse date from ScrapeCreators Twitter item to YYYY-MM-DD."""
# Try created_at string (e.g. "Wed Oct 10 20:19:24 +0000 2018")
created_at = item.get("created_at")
if created_at and isinstance(created_at, str):
try:
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Try unix timestamp
ts = item.get("timestamp") or item.get("created_at_timestamp")
if ts:
try:
dt = datetime.fromtimestamp(int(ts), tz=timezone.utc)
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError, OSError):
pass
# Try ISO format
for key in ("created_at", "date"):
val = item.get(key)
if val and isinstance(val, str):
try:
dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
return None
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
token: str = None,
) -> Dict[str, Any]:
"""Search X/Twitter via ScrapeCreators API.
Returns:
Dict with 'items' list (in normalize_x_items format) and optional 'error'.
"""
if not token:
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
if not _requests:
return {"items": [], "error": "requests library not installed"}
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching X for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search/tweets",
params={"query": core_topic, "sort_by": "relevance"},
headers=_sc_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
raw_items = data.get("tweets") or data.get("data") or data.get("results") or []
raw_items = raw_items[:config["results_per_page"]]
items = []
for i, raw in enumerate(raw_items):
tweet_id = str(raw.get("id") or raw.get("tweet_id") or raw.get("id_str") or f"sc-x-{i}")
text = raw.get("full_text") or raw.get("text") or ""
user = raw.get("user") or raw.get("author") or {}
author_handle = user.get("screen_name") or user.get("username") or ""
# Engagement metrics
likes = raw.get("favorite_count") or raw.get("likes") or 0
retweets = raw.get("retweet_count") or raw.get("retweets") or 0
replies = raw.get("reply_count") or raw.get("replies") or 0
quotes = raw.get("quote_count") or raw.get("quotes") or 0
date_str = _parse_date(raw)
relevance = _compute_relevance(core_topic, text)
url = ""
if author_handle and tweet_id and not tweet_id.startswith("sc-x-"):
url = f"https://x.com/{author_handle}/status/{tweet_id}"
items.append({
"id": tweet_id,
"text": text,
"url": url,
"author_handle": author_handle,
"date": date_str,
"engagement": {
"likes": likes,
"reposts": retweets,
"replies": replies,
"quotes": quotes,
},
"relevance": relevance,
"why_relevant": f"X: @{author_handle}: {text[:60]}" if text else f"X: {core_topic}",
})
# Date filter
in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
out_of_range = len(items) - len(in_range)
if in_range:
items = in_range
if out_of_range:
_log(f"Filtered {out_of_range} tweets outside date range")
else:
_log(f"No tweets within date range, keeping all {len(items)}")
# Sort by engagement (likes + retweets)
items.sort(key=lambda x: (x["engagement"]["likes"] + x["engagement"]["reposts"]), reverse=True)
_log(f"Found {len(items)} tweets")
return {"items": items}
def parse_x_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse search response to normalized format."""
return response.get("items", [])
+186
View File
@@ -0,0 +1,186 @@
"""First-run setup wizard for last30days.
Detects first run, performs auto-setup (cookie extraction + yt-dlp check),
and writes configuration. The actual wizard UI is SKILL.md-driven (the LLM
presents it), but this module provides the detection and setup actions.
"""
import logging
import shutil
import subprocess
from pathlib import Path
from typing import Any, Dict, Optional
logger = logging.getLogger(__name__)
def is_first_run(config: Dict[str, Any]) -> bool:
"""Return True if the setup wizard has not been completed.
Checks for SETUP_COMPLETE in the config dict. If it's not set
(None or empty string), the user hasn't gone through setup yet.
"""
return not config.get("SETUP_COMPLETE")
def run_auto_setup(config: Dict[str, Any]) -> Dict[str, Any]:
"""Perform the auto-setup actions.
- Runs cookie extraction in auto mode for all registered domains
- Checks if yt-dlp is installed
Returns:
Dict with keys:
cookies_found: {source_name: browser_name} for each source where cookies were found
ytdlp_installed: bool
env_written: bool (always False here caller writes config separately)
"""
from . import cookie_extract
from .env import COOKIE_DOMAINS
cookies_found: Dict[str, str] = {}
for source_name, spec in COOKIE_DOMAINS.items():
domain = spec["domain"]
cookie_names = spec["cookies"]
try:
result = cookie_extract.extract_cookies_with_source("auto", domain, cookie_names)
except Exception as exc:
logger.debug("Cookie extraction failed for %s: %s", source_name, exc)
continue
if result is not None:
_cookies, browser_name = result
cookies_found[source_name] = browser_name
# Check yt-dlp availability and install via Homebrew if missing
ytdlp_action: str
if shutil.which("yt-dlp") is not None:
ytdlp_installed = True
ytdlp_action = "already_installed"
elif shutil.which("brew") is not None:
brew_stderr = ""
try:
proc = subprocess.run(
["brew", "install", "yt-dlp"],
capture_output=True, text=True, timeout=120,
)
if proc.returncode == 0:
ytdlp_installed = True
ytdlp_action = "installed"
else:
ytdlp_installed = False
ytdlp_action = "install_failed"
brew_stderr = proc.stderr
logger.warning("brew install yt-dlp failed: %s", proc.stderr)
except Exception as exc:
ytdlp_installed = False
ytdlp_action = "install_failed"
brew_stderr = str(exc)
logger.warning("brew install yt-dlp exception: %s", exc)
else:
ytdlp_installed = False
ytdlp_action = "no_homebrew"
results: Dict[str, Any] = {
"cookies_found": cookies_found,
"ytdlp_installed": ytdlp_installed,
"ytdlp_action": ytdlp_action,
"env_written": False,
}
if ytdlp_action == "install_failed":
results["ytdlp_stderr"] = brew_stderr
return results
def write_setup_config(env_path: Path, from_browser: str = "auto") -> bool:
"""Write SETUP_COMPLETE and FROM_BROWSER to the .env file.
Creates the file and parent directories if needed.
Appends to existing file without overwriting existing keys.
Args:
env_path: Path to the .env file (e.g. ~/.config/last30days/.env)
from_browser: Browser extraction mode to write (default: "auto")
Returns:
True if config was written successfully, False on error.
"""
try:
env_path = Path(env_path)
env_path.parent.mkdir(parents=True, exist_ok=True)
# Read existing content to avoid overwriting keys
existing_keys: set = set()
existing_content = ""
if env_path.exists():
existing_content = env_path.read_text(encoding="utf-8")
for line in existing_content.splitlines():
stripped = line.strip()
if stripped and not stripped.startswith("#") and "=" in stripped:
key = stripped.split("=", 1)[0].strip()
existing_keys.add(key)
lines_to_add = []
if "SETUP_COMPLETE" not in existing_keys:
lines_to_add.append("SETUP_COMPLETE=true")
if "FROM_BROWSER" not in existing_keys:
lines_to_add.append(f"FROM_BROWSER={from_browser}")
if not lines_to_add:
return True # Nothing to write, already configured
# Ensure trailing newline before appending
with open(env_path, "a", encoding="utf-8") as f:
if existing_content and not existing_content.endswith("\n"):
f.write("\n")
f.write("\n".join(lines_to_add) + "\n")
return True
except OSError as exc:
logger.error("Failed to write setup config to %s: %s", env_path, exc)
return False
def get_setup_status_text(results: Dict[str, Any]) -> str:
"""Return a human-readable summary of auto-setup results.
Args:
results: Dict from run_auto_setup()
Returns:
Multi-line status text.
"""
lines = []
lines.append("Setup complete! Here's what I found:")
lines.append("")
cookies_found = results.get("cookies_found", {})
if cookies_found:
for source, browser in cookies_found.items():
lines.append(f" - {source.upper()} cookies found in {browser}")
else:
lines.append(" - No browser cookies found for X/Twitter")
ytdlp_action = results.get("ytdlp_action", "")
if ytdlp_action == "installed":
lines.append(" - Installed yt-dlp via Homebrew")
elif ytdlp_action == "install_failed":
lines.append(" - yt-dlp install failed \u2014 run `brew install yt-dlp` manually")
elif ytdlp_action == "no_homebrew":
lines.append(" - yt-dlp not found. Install Homebrew first, then: brew install yt-dlp")
elif ytdlp_action == "already_installed":
lines.append(" - yt-dlp already installed")
elif results.get("ytdlp_installed", False):
lines.append(" - yt-dlp is installed (YouTube search ready)")
else:
lines.append(" - yt-dlp not found (install with: brew install yt-dlp)")
env_written = results.get("env_written", False)
if env_written:
lines.append("")
lines.append("Configuration saved. Future runs will auto-detect your browsers.")
return "\n".join(lines)
+34 -106
View File
@@ -3,7 +3,7 @@
Uses ScrapeCreators REST API to search TikTok by keyword, extract engagement
metrics (views, likes, comments, shares), and fetch video transcripts.
Requires SCRAPECREATORS_API_KEY in config. 100 free credits, then PAYG.
Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
@@ -17,6 +17,8 @@ try:
except ImportError:
_requests = None
from . import http
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/tiktok"
# Depth configurations: how many results to fetch / captions to extract
@@ -29,93 +31,13 @@ DEPTH_CONFIG = {
# Max words to keep from each caption
CAPTION_MAX_WORDS = 500
# Stopwords for relevance computation (shared with youtube_yt.py pattern)
STOPWORDS = frozenset({
'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
'all', 'just', 'get', 'has', 'have', 'was', 'will',
})
# Synonym groups for relevance scoring
SYNONYMS = {
'hip': {'rap', 'hiphop'},
'hop': {'rap', 'hiphop'},
'rap': {'hip', 'hop', 'hiphop'},
'hiphop': {'rap', 'hip', 'hop'},
'js': {'javascript'},
'javascript': {'js'},
'ts': {'typescript'},
'typescript': {'ts'},
'ai': {'artificial', 'intelligence'},
'ml': {'machine', 'learning'},
'react': {'reactjs'},
'reactjs': {'react'},
}
def _tokenize(text: str) -> Set[str]:
"""Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
expanded = set(tokens)
for t in tokens:
if t in SYNONYMS:
expanded.update(SYNONYMS[t])
return expanded
def _compute_relevance(query: str, text: str, hashtags: List[str] = None) -> float:
"""Compute relevance as ratio of query tokens found in text + hashtags.
Uses ratio overlap (intersection / query_length). Hashtags provide
a TikTok-specific relevance boost. Floors at 0.1.
"""
q_tokens = _tokenize(query)
# Combine text and hashtags for matching
combined = text
if hashtags:
combined = f"{text} {' '.join(hashtags)}"
t_tokens = _tokenize(combined)
# Split concatenated hashtags (e.g., "claudecode" -> "claude", "code")
if hashtags:
for tag in hashtags:
tag_lower = tag.lower()
for qt in q_tokens:
if qt in tag_lower and qt != tag_lower:
t_tokens.add(qt)
if not q_tokens:
return 0.5 # Neutral fallback
overlap = len(q_tokens & t_tokens)
ratio = overlap / len(q_tokens)
return max(0.1, min(1.0, ratio))
from .relevance import token_overlap_relevance as _compute_relevance
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for TikTok search.
Strips meta/research words to keep only the core product/concept name.
"""
text = topic.lower().strip()
# Strip multi-word prefixes
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
# Strip individual noise words
noise = {
"""Extract core subject from verbose query for TikTok search."""
from .query import extract_core_subject
_TIKTOK_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
@@ -123,12 +45,8 @@ def _extract_core_subject(topic: str) -> str:
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
}
words = text.split()
filtered = [w for w in words if w not in noise]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.')
})
return extract_core_subject(topic, noise=_TIKTOK_NOISE)
def _log(msg: str):
@@ -204,26 +122,36 @@ def search_tiktok(
if not token:
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
if not _requests:
return {"items": [], "error": "requests library not installed"}
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching TikTok for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search/keyword",
params={"query": core_topic, "sort_by": "relevance"},
headers=_sc_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
if not _requests:
_log("requests library not installed, falling back to urllib")
try:
from urllib.parse import urlencode
params = urlencode({"query": core_topic, "sort_by": "relevance"})
url = f"{SCRAPECREATORS_BASE}/search/keyword?{params}"
headers = _sc_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search/keyword",
params={"query": core_topic, "sort_by": "relevance"},
headers=_sc_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
# Items are nested under aweme_info
raw_entries = data.get("search_item_list") or data.get("data") or []
+183
View File
@@ -0,0 +1,183 @@
"""Truth Social search via Mastodon-compatible API (requires bearer token).
Uses truthsocial.com/api/v2/search endpoint.
Requires TRUTHSOCIAL_TOKEN env var (bearer token from browser dev tools).
"""
import math
import re
import sys
from typing import Any, Dict, List, Optional
from . import http
TRUTHSOCIAL_SEARCH_URL = "https://truthsocial.com/api/v2/search"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
def _log(msg: str):
"""Log to stderr (only in TTY mode to avoid cluttering Claude Code output)."""
if sys.stderr.isatty():
sys.stderr.write(f"[TruthSocial] {msg}\n")
sys.stderr.flush()
def _strip_html(html: str) -> str:
"""Strip HTML tags from Truth Social post content."""
text = re.sub(r'<br\s*/?>', '\n', html)
text = re.sub(r'<[^>]+>', '', text)
return text.strip()
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Truth Social search."""
text = topic.lower().strip()
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
noise = {
'best', 'top', 'good', 'great', 'awesome',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features', 'recommendations', 'advice',
}
words = text.split()
filtered = [w for w in words if w not in noise]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.')
def _parse_date(status: Dict[str, Any]) -> Optional[str]:
"""Parse date from Mastodon status to YYYY-MM-DD.
Mastodon uses ISO 8601 format in created_at field.
"""
val = status.get("created_at")
if val and isinstance(val, str) and len(val) >= 10:
return val[:10]
return None
def search_truthsocial(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Search Truth Social via Mastodon-compatible API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
config: Config dict with TRUTHSOCIAL_TOKEN
Returns:
Dict with 'statuses' list from Mastodon API response.
"""
config = config or {}
token = config.get("TRUTHSOCIAL_TOKEN", "")
if not token:
return {"statuses": [], "error": "Truth Social token not configured"}
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching for '{core_topic}' (depth={depth}, limit={count})")
from urllib.parse import urlencode
params = {
"q": core_topic,
"type": "statuses",
"limit": str(min(count, 40)),
}
url = f"{TRUTHSOCIAL_SEARCH_URL}?{urlencode(params)}"
try:
response = http.request(
"GET", url,
headers={"Authorization": f"Bearer {token}"},
timeout=30,
)
except http.HTTPError as e:
if e.status_code == 401:
_log("Token expired")
return {"statuses": [], "error": "Truth Social token expired"}
elif e.status_code == 403:
_log("Access denied (Cloudflare)")
return {"statuses": [], "error": "Truth Social access denied (Cloudflare)"}
elif e.status_code == 429:
_log("Rate limited")
return {"statuses": [], "error": "Truth Social rate limited"}
else:
_log(f"Search failed: {e}")
return {"statuses": [], "error": f"Truth Social search failed: {e.status_code}"}
except Exception as e:
_log(f"Search failed: {e}")
return {"statuses": [], "error": str(e)}
statuses = response.get("statuses", [])
_log(f"Found {len(statuses)} posts")
return response
def parse_truthsocial_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Mastodon API response into normalized item dicts.
Returns:
List of item dicts ready for normalization.
"""
statuses = response.get("statuses", [])
items = []
for i, status in enumerate(statuses):
content_html = status.get("content") or ""
text = _strip_html(content_html)
account = status.get("account") or {}
handle = account.get("acct") or account.get("username") or ""
display_name = account.get("display_name") or handle
url = status.get("url") or ""
likes = status.get("favourites_count") or 0
reposts = status.get("reblogs_count") or 0
replies = status.get("replies_count") or 0
date_str = _parse_date(status)
# Relevance: position-based (search results are ranked by relevance)
rank_score = max(0.3, 1.0 - (i * 0.02))
engagement_boost = min(0.2, math.log1p(likes + reposts) / 40)
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"handle": handle,
"display_name": display_name,
"text": text,
"url": url,
"date": date_str,
"engagement": {
"likes": likes,
"reposts": reposts,
"replies": replies,
},
"relevance": round(relevance, 2),
"why_relevant": f"Truth Social: @{handle}: {text[:60]}" if text else f"Truth Social: {handle}",
})
return items
+182 -95
View File
@@ -143,7 +143,7 @@ Just start with "last30" and talk to me like normal.
# Shorter promo for single missing key
PROMO_SINGLE_KEY = {
"reddit": "\n💡 You can unlock Reddit with an OpenAI API key or by running `codex login` — just ask me how.\n",
"x": "\n💡 You can unlock X with an xAI API key — just ask me how.\n",
"x": "\n💡 You can unlock X with AUTH_TOKEN/CT0 or XAI_API_KEY - just ask me how.\n",
}
# Bird auth help (for local users with vendored Bird CLI)
@@ -151,16 +151,16 @@ BIRD_AUTH_HELP = f"""
{Colors.YELLOW}Bird authentication failed.{Colors.RESET}
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Try again Bird reads your browser cookies automatically.
1. Add AUTH_TOKEN and CT0 to ~/.config/last30days/.env or .claude/last30days.env
2. Or set XAI_API_KEY for the xAI fallback backend
"""
BIRD_AUTH_HELP_PLAIN = """
Bird authentication failed.
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Try again Bird reads your browser cookies automatically.
1. Add AUTH_TOKEN and CT0 to ~/.config/last30days/.env or .claude/last30days.env
2. Or set XAI_API_KEY for the xAI fallback backend
"""
# Spinner frames
@@ -417,103 +417,190 @@ class ProgressDisplay:
sys.stderr.flush()
def show_diagnostic_banner(diag: dict):
"""Show pre-flight source status banner when sources are missing.
def _build_status_banner(diag: dict) -> list[str]:
"""Build the status banner lines (plain text, no ANSI).
Returns a list of strings, each being a line of the banner box.
Args:
diag: Dict from env diagnostics with keys:
openai, xai, x_source, bird_installed, bird_authenticated,
bird_username, youtube, web_search_backend
diag: Dict with keys:
setup_complete, reddit_source, x_source, x_method,
youtube, tiktok, instagram, hackernews, polymarket,
bluesky, truthsocial, xiaohongshu, scrapecreators,
web_search_backend
"""
has_openai = diag.get("openai", False)
has_x = diag.get("x_source") is not None
has_youtube = diag.get("youtube", False)
has_web = diag.get("web_search_backend") is not None
setup_complete = diag.get("setup_complete", False)
has_sc = diag.get("scrapecreators", False)
# If everything is available, no banner needed
if has_openai and has_x and has_youtube and has_web:
return
# --- Build active sources list: (label, method_label) ---
active: list[str] = []
lines = []
# Reddit — always available; what matters to users is comments or not
reddit_src = diag.get("reddit_source")
if reddit_src == "scrapecreators":
active.append("Reddit (with comments)")
else:
active.append("Reddit (threads only)")
# X/Twitter
x_source = diag.get("x_source")
x_method = diag.get("x_method")
if x_source:
if x_method and x_method.startswith("browser-"):
browser = x_method.split("-", 1)[1].capitalize()
active.append(f"X ({browser})")
elif x_method == "env":
active.append("X (env)")
elif x_method == "api":
active.append("X (xAI)")
else:
active.append("X")
# YouTube
if diag.get("youtube"):
active.append("YouTube")
# HN — always available
if diag.get("hackernews"):
active.append("HN")
# Polymarket — always available
if diag.get("polymarket"):
active.append("Polymarket")
# TikTok (requires SC or Apify)
if diag.get("tiktok"):
active.append("TikTok")
# Instagram (requires SC)
if diag.get("instagram"):
active.append("Instagram")
# Bluesky
if diag.get("bluesky"):
active.append("Bluesky")
# Truth Social
if diag.get("truthsocial"):
active.append("Truth Social")
# Xiaohongshu
if diag.get("xiaohongshu"):
active.append("Xiaohongshu")
# --- Format active sources into wrapped lines ---
BOX_INNER = 53 # characters inside the box (between │ and │)
PREFIX = " " # 2-space indent inside box
def _wrap_sources(sources: list[str]) -> list[str]:
"""Wrap source labels into lines that fit the box width."""
result_lines: list[str] = []
current = PREFIX
for i, s in enumerate(sources):
token = f"{s}"
sep = " " if current != PREFIX else ""
if len(current) + len(sep) + len(token) > BOX_INNER:
result_lines.append(current)
current = PREFIX + token
else:
current += sep + token
if current.strip():
result_lines.append(current)
return result_lines
source_lines = _wrap_sources(active)
# --- Title ---
if not setup_complete:
title = "/last30days v3.0 — First Run"
else:
title = "/last30days v3.0 — Source Status"
# --- Build upgrade suggestions ---
suggestions: list[str] = []
if not setup_complete:
suggestions.append("Run /last30days setup to unlock more sources")
else:
# Recommend ScrapeCreators if missing
if not has_sc:
suggestions.append("⭐ Add SCRAPECREATORS_API_KEY for Reddit comments")
suggestions.append(" + TikTok + Instagram")
suggestions.append(" 100 free calls, no CC — scrapecreators.com (no affiliation)")
# --- Assemble box lines ---
# Collect all content lines, then determine box width dynamically.
content: list[str] = []
content.append(f" {title}")
content.append("") # blank line
for sl in source_lines:
content.append(sl)
if suggestions:
content.append("") # blank line
for sg in suggestions:
content.append(f" {sg}")
content.append("") # blank line
content.append(" Config: ~/.config/last30days/.env")
# Width = widest content line + 1 for right margin
width = max(len(line) for line in content) + 1
if width < 53:
width = 53
lines: list[str] = []
lines.append("\u250c" + "\u2500" * width + "\u2510")
for c in content:
lines.append("\u2502" + c.ljust(width) + "\u2502")
lines.append("\u2514" + "\u2500" * width + "\u2518")
return lines
def _colorize_banner(lines: list[str]) -> list[str]:
"""Apply ANSI colors to plain-text banner lines for TTY output."""
colored: list[str] = []
for line in lines:
if line.startswith("\u250c") or line.startswith("\u2514"):
colored.append(f"{Colors.DIM}{line}{Colors.RESET}")
elif line.startswith("\u2502"):
inner = line[1:-1] # strip box chars on both sides
inner_width = len(inner)
# Colorize check marks green, star yellow
inner = inner.replace("\u2705", f"{Colors.GREEN}\u2705{Colors.RESET}")
inner = inner.replace("\u2b50", f"{Colors.YELLOW}\u2b50{Colors.RESET}")
# Bold the title line
if "/last30days v3.0" in inner:
stripped = inner.strip()
inner = f" {Colors.BOLD}{stripped}{Colors.RESET}"
# Re-pad to original width (ANSI codes are zero-width)
visible_len = 1 + len(stripped)
inner = inner + " " * max(0, inner_width - visible_len)
colored.append(f"{Colors.DIM}\u2502{Colors.RESET}{inner}{Colors.DIM}\u2502{Colors.RESET}")
else:
colored.append(line)
return colored
def show_diagnostic_banner(diag: dict):
"""Show pre-flight source status banner.
Free-first design: leads with what's working (✅), not what's broken.
Shows upgrade suggestions only when relevant.
Args:
diag: Dict with keys:
setup_complete, reddit_source, x_source, x_method,
youtube, tiktok, instagram, hackernews, polymarket,
bluesky, truthsocial, xiaohongshu, scrapecreators,
web_search_backend
"""
lines = _build_status_banner(diag)
if IS_TTY:
lines.append(f"{Colors.DIM}┌─────────────────────────────────────────────────────┐{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}/last30days v2.1 — Source Status{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
# Reddit
if has_openai:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ Reddit{Colors.RESET} — OPENAI_API_KEY found {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ Reddit{Colors.RESET} — No OPENAI_API_KEY {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Add to ~/.config/last30days/.env {Colors.DIM}{Colors.RESET}")
# X/Twitter
if has_x:
source = diag.get("x_source", "")
username = diag.get("bird_username", "")
label = f"Bird ({username})" if source == "bird" and username else source.upper()
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ X/Twitter{Colors.RESET}{label} {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ X/Twitter{Colors.RESET} — No Bird CLI or XAI_API_KEY {Colors.DIM}{Colors.RESET}")
if diag.get("bird_installed"):
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Bird installed but not authenticated {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Log into x.com in your browser, then retry {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Needs Node.js 22+ (Bird is bundled) {Colors.DIM}{Colors.RESET}")
# YouTube
if has_youtube:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ YouTube{Colors.RESET} — yt-dlp found {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.RED}❌ YouTube{Colors.RESET} — yt-dlp not installed {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} └─ Fix: brew install yt-dlp (free) {Colors.DIM}{Colors.RESET}")
# Web
if has_web:
backend = diag.get("web_search_backend", "")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.GREEN}✅ Web{Colors.RESET}{backend} API {Colors.DIM}{Colors.RESET}")
else:
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.YELLOW}⚡ Web{Colors.RESET} — Using assistant's search tool {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} Config: {Colors.BOLD}~/.config/last30days/.env{Colors.RESET} {Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}└─────────────────────────────────────────────────────┘{Colors.RESET}")
else:
# Plain text for non-TTY (Claude Code / Codex)
lines.append("┌─────────────────────────────────────────────────────┐")
lines.append("│ /last30days v2.1 — Source Status │")
lines.append("│ │")
if has_openai:
lines.append("│ ✅ Reddit — OPENAI_API_KEY found │")
else:
lines.append("│ ❌ Reddit — No OPENAI_API_KEY │")
lines.append("│ └─ Add to ~/.config/last30days/.env │")
if has_x:
lines.append("│ ✅ X/Twitter — available │")
else:
lines.append("│ ❌ X/Twitter — No Bird CLI or XAI_API_KEY │")
if diag.get("bird_installed"):
lines.append("│ └─ Log into x.com in your browser, then retry │")
else:
lines.append("│ └─ Needs Node.js 22+ (Bird is bundled) │")
if has_youtube:
lines.append("│ ✅ YouTube — yt-dlp found │")
else:
lines.append("│ ❌ YouTube — yt-dlp not installed │")
lines.append("│ └─ Fix: brew install yt-dlp (free) │")
if has_web:
lines.append("│ ✅ Web — API search available │")
else:
lines.append("│ ⚡ Web — Using assistant's search tool │")
lines.append("│ │")
lines.append("│ Config: ~/.config/last30days/.env │")
lines.append("└─────────────────────────────────────────────────────┘")
lines = _colorize_banner(lines)
sys.stderr.write("\n".join(lines) + "\n\n")
sys.stderr.flush()
+19 -1
View File
@@ -14,6 +14,13 @@ function normalizeValue(value) {
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : null;
}
function envFlagEnabled(name) {
const value = normalizeValue(process.env[name]);
if (!value) {
return false;
}
return ['1', 'true', 'yes', 'on'].includes(value.toLowerCase());
}
function cookieHeader(authToken, ct0) {
return `auth_token=${authToken}; ct0=${ct0}`;
}
@@ -123,6 +130,8 @@ export async function extractCookiesFromFirefox(profile) {
export async function resolveCredentials(options) {
const warnings = [];
const cookies = buildEmpty();
const disableBrowserCookies = envFlagEnabled('BIRD_DISABLE_BROWSER_COOKIES') ||
envFlagEnabled('LAST30DAYS_DISABLE_BROWSER_COOKIES');
const cookieTimeoutMs = typeof options.cookieTimeoutMs === 'number' &&
Number.isFinite(options.cookieTimeoutMs) &&
options.cookieTimeoutMs > 0
@@ -146,6 +155,15 @@ export async function resolveCredentials(options) {
cookies.cookieHeader = cookieHeader(cookies.authToken, cookies.ct0);
return { cookies, warnings };
}
if (disableBrowserCookies) {
if (!cookies.authToken) {
warnings.push('Missing auth_token - provide via --auth-token, AUTH_TOKEN env var, or disable BIRD_DISABLE_BROWSER_COOKIES to allow browser cookie lookup');
}
if (!cookies.ct0) {
warnings.push('Missing ct0 - provide via --ct0, CT0 env var, or disable BIRD_DISABLE_BROWSER_COOKIES to allow browser cookie lookup');
}
return { cookies, warnings };
}
const sourcesToTry = resolveSources(options.cookieSource);
for (const source of sourcesToTry) {
const res = await readTwitterCookiesFromBrowser({
@@ -170,4 +188,4 @@ export async function resolveCredentials(options) {
}
return { cookies, warnings };
}
//# sourceMappingURL=cookies.js.map
//# sourceMappingURL=cookies.js.map
+2 -2
View File
@@ -91,11 +91,11 @@ def search_x(
# Adjust timeout based on depth (generous for API response time)
timeout = 90 if depth == "quick" else 120 if depth == "default" else 180
# Use Agent Tools API with x_search tool
# Use Agent Tools API with x_search tool (native date filtering)
payload = {
"model": model,
"tools": [
{"type": "x_search"}
{"type": "x_search", "from_date": from_date, "to_date": to_date}
],
"input": [
{
+162
View File
@@ -0,0 +1,162 @@
"""Xiaohongshu HTTP API search client for last30days.
Uses xpzouying/xiaohongshu-mcp REST endpoints:
- GET/POST /api/v1/feeds/search
- GET /api/v1/login/status
"""
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from . import http
def _to_int(value: Any) -> int:
"""Convert Xiaohongshu count strings to int.
Supports plain ints and Chinese suffixes like 1.2 / 3亿.
"""
if value is None:
return 0
if isinstance(value, (int, float)):
return int(value)
text = str(value).strip().lower().replace(",", "")
if not text:
return 0
try:
if text.endswith(""):
return int(float(text[:-1]) * 10000)
if text.endswith("亿"):
return int(float(text[:-1]) * 100000000)
return int(float(text))
except (TypeError, ValueError):
return 0
def _timestamp_to_date_ms(ts: Any) -> Optional[str]:
"""Convert millisecond timestamp to YYYY-MM-DD."""
try:
iv = int(ts)
if iv <= 0:
return None
# API examples use milliseconds.
dt = datetime.fromtimestamp(iv / 1000.0, tz=timezone.utc)
return dt.strftime("%Y-%m-%d")
except (TypeError, ValueError, OSError):
return None
def _relevance_from_interactions(likes: int, comments: int, favorites: int) -> float:
"""Heuristic relevance score from engagement metrics."""
# Weighted engagement with soft caps to [0, 1].
weighted = (likes * 1.0) + (comments * 2.5) + (favorites * 1.5)
# 5000 weighted engagement ~= strong relevance.
score = min(1.0, max(0.05, weighted / 5000.0))
return round(score, 3)
def _build_note_url(feed_id: str, xsec_token: str) -> str:
"""Build a stable Xiaohongshu note URL."""
if xsec_token:
return f"https://www.xiaohongshu.com/explore/{feed_id}?xsec_token={xsec_token}"
return f"https://www.xiaohongshu.com/explore/{feed_id}"
def search_feeds(
topic: str,
from_date: str,
to_date: str,
base_url: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search Xiaohongshu feeds and normalize to web-item shape."""
base = (base_url or "").rstrip("/")
if not base:
raise ValueError("Missing Xiaohongshu API base URL")
# Quick login sanity check.
login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=1)
is_logged_in = (
login.get("data", {}).get("is_logged_in")
if isinstance(login, dict) else False
)
if not is_logged_in:
raise http.HTTPError("Xiaohongshu API reachable but not logged in")
# API supports filters; use recency-oriented defaults.
publish_time = "一天内" if depth == "quick" else "一周内" if depth == "default" else "半年内"
payload = {
"keyword": topic,
"filters": {
"sort_by": "综合",
"note_type": "不限",
"publish_time": publish_time,
"search_scope": "不限",
"location": "不限",
},
}
resp = http.post(f"{base}/api/v1/feeds/search", payload, timeout=20, retries=1)
feeds = resp.get("data", {}).get("feeds", []) if isinstance(resp, dict) else []
if not isinstance(feeds, list):
feeds = []
# Cap source volume similarly to other web sources.
limit = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
items: List[Dict[str, Any]] = []
for i, feed in enumerate(feeds[:limit]):
if not isinstance(feed, dict):
continue
note = feed.get("noteCard") or {}
if not isinstance(note, dict):
note = {}
interact = note.get("interactInfo") or {}
if not isinstance(interact, dict):
interact = {}
feed_id = str(feed.get("id") or note.get("noteId") or "").strip()
if not feed_id:
continue
xsec_token = str(feed.get("xsecToken") or note.get("xsecToken") or "").strip()
title = str(
note.get("displayTitle")
or note.get("title")
or ""
).strip()
snippet = str(
note.get("desc")
or note.get("displayDesc")
or title
or ""
).strip()
likes = _to_int(interact.get("likedCount"))
comments = _to_int(interact.get("commentCount"))
favorites = _to_int(interact.get("collectedCount"))
date_value = _timestamp_to_date_ms(note.get("time"))
why = f"Xiaohongshu engagement: likes={likes}, comments={comments}, favorites={favorites}"
items.append({
"id": f"XHS{i+1}",
"title": title[:200] if title else f"Xiaohongshu note {feed_id}",
"url": _build_note_url(feed_id, xsec_token),
"source_domain": "xiaohongshu.com",
"snippet": snippet[:500],
"date": date_value,
"date_confidence": "high" if date_value else "low",
"relevance": _relevance_from_interactions(likes, comments, favorites),
"why_relevant": why,
# Keep raw engagement for debugging/possible future rendering.
"engagement": {
"likes": likes,
"comments": comments,
"favorites": favorites,
},
})
return items
+191 -88
View File
@@ -15,6 +15,8 @@ import shutil
import subprocess
import sys
import tempfile
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple
@@ -33,67 +35,55 @@ TRANSCRIPT_LIMITS = {
}
# Max words to keep from each transcript
TRANSCRIPT_MAX_WORDS = 500
TRANSCRIPT_MAX_WORDS = 5000
# Stopwords for relevance computation (common English words that dilute token overlap)
STOPWORDS = frozenset({
'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
'all', 'just', 'get', 'has', 'have', 'was', 'will',
})
from .relevance import token_overlap_relevance as _compute_relevance
# Synonym groups for relevance scoring (bidirectional expansion)
SYNONYMS = {
'hip': {'rap', 'hiphop'},
'hop': {'rap', 'hiphop'},
'rap': {'hip', 'hop', 'hiphop'},
'hiphop': {'rap', 'hip', 'hop'},
'js': {'javascript'},
'javascript': {'js'},
'ts': {'typescript'},
'typescript': {'ts'},
'ai': {'artificial', 'intelligence'},
'ml': {'machine', 'learning'},
'react': {'reactjs'},
'reactjs': {'react'},
'svelte': {'sveltejs'},
'sveltejs': {'svelte'},
'vue': {'vuejs'},
'vuejs': {'vue'},
}
def extract_transcript_highlights(transcript: str, topic: str, limit: int = 5) -> List[str]:
"""Extract quotable highlights from a YouTube transcript.
def _tokenize(text: str) -> Set[str]:
"""Lowercase, strip punctuation, remove stopwords, drop single-char tokens.
Expands tokens with synonyms for better cross-domain matching."""
words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
tokens = {w for w in words if w not in STOPWORDS and len(w) > 1}
# Expand synonyms
expanded = set(tokens)
for t in tokens:
if t in SYNONYMS:
expanded.update(SYNONYMS[t])
return expanded
def _compute_relevance(query: str, title: str) -> float:
"""Compute relevance as ratio of query tokens found in title.
Uses ratio overlap (intersection / query_length) so short queries
score higher when fully represented in the title. Floors at 0.1.
Similar to reddit_enrich.extract_comment_insights() but for
continuous speech-to-text rather than threaded comments.
"""
q_tokens = _tokenize(query)
t_tokens = _tokenize(title)
if not transcript:
return []
if not q_tokens:
return 0.5 # Neutral fallback for empty/stopword-only queries
sentences = re.split(r'(?<=[.!?])\s+', transcript)
overlap = len(q_tokens & t_tokens)
ratio = overlap / len(q_tokens)
return max(0.1, min(1.0, ratio))
filler = [
r"^(hey |hi |what's up|welcome back|in today's video|don't forget to)",
r"(subscribe|like and comment|hit the bell|check out the link|down below)",
r"^(so |and |but |okay |alright |um |uh )",
r"(thanks for watching|see you (next|in the)|bye)",
]
topic_words = [w.lower() for w in topic.lower().split() if len(w) > 2]
candidates = []
for sent in sentences:
sent = sent.strip()
words = sent.split()
if len(words) < 8 or len(words) > 50:
continue
if any(re.search(p, sent, re.IGNORECASE) for p in filler):
continue
score = 0
if re.search(r'\d', sent):
score += 2
if re.search(r'[A-Z][a-z]+', sent):
score += 1
if '?' in sent:
score += 1
sent_lower = sent.lower()
if any(w in sent_lower for w in topic_words):
score += 2
candidates.append((score, sent))
candidates.sort(key=lambda x: -x[0])
return [sent for _, sent in candidates[:limit]]
def _log(msg: str):
@@ -110,26 +100,12 @@ def is_ytdlp_installed() -> bool:
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for YouTube search.
Strips meta/research words to keep only the core product/concept name,
similar to bird_x.py's approach.
NOTE: 'tips', 'tricks', 'tutorial', 'guide', 'review', 'reviews'
are intentionally KEPT they're YouTube content types that improve search.
"""
text = topic.lower().strip()
# Strip multi-word prefixes
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
# Strip individual noise words
# NOTE: 'tips', 'tricks', 'tutorial', 'guide', 'review', 'reviews'
# are intentionally KEPT — they're YouTube content types that improve search
noise = {
from .query import extract_core_subject
# YouTube-specific noise set: smaller than default, keeps content-type words
_YT_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
@@ -137,12 +113,8 @@ def _extract_core_subject(topic: str) -> str:
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
}
words = text.split()
filtered = [w for w in words if w not in noise]
result = ' '.join(filtered) if filtered else text
return result.rstrip('?!.')
})
return extract_core_subject(topic, noise=_YT_NOISE)
def search_youtube(
@@ -171,11 +143,13 @@ def search_youtube(
_log(f"Searching YouTube for '{core_topic}' (since {from_date}, count={count})")
# yt-dlp search with full metadata (no --flat-playlist so dates are real).
# No --dateafter — we filter by date in Python with a soft fallback,
# because YouTube search returns relevance-sorted results and strict date
# filtering returns 0 for evergreen topics like "thumbnail tips".
# NOTE: --dateafter intentionally omitted — YouTube search returns
# relevance-sorted results and strict date filtering returns 0 for
# evergreen topics. Python soft filter (below) handles date filtering.
cmd = [
"yt-dlp",
"--ignore-config",
"--no-cookies-from-browser",
f"ytsearch{count}:{core_topic}",
"--dump-json",
"--no-warnings",
@@ -283,18 +257,119 @@ def _clean_vtt(vtt_text: str) -> str:
return re.sub(r'\s+', ' ', ' '.join(unique)).strip()
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
_YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
"""Fetch YouTube transcript via direct HTTP without yt-dlp.
Scrapes the watch page HTML for the captions track URL in
ytInitialPlayerResponse, then fetches the VTT subtitle file.
Args:
video_id: YouTube video ID
timeout: HTTP request timeout in seconds
Returns:
Raw VTT text, or None if captions are unavailable.
"""
watch_url = f"https://www.youtube.com/watch?v={video_id}"
headers = {
"User-Agent": _YT_USER_AGENT,
"Accept-Language": "en-US,en;q=0.9",
}
# Step 1: Fetch the watch page HTML
req = urllib.request.Request(watch_url, headers=headers)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
html = resp.read().decode("utf-8", errors="replace")
except (urllib.error.URLError, urllib.error.HTTPError, OSError, TimeoutError) as exc:
_log(f"Direct transcript: failed to fetch watch page for {video_id}: {exc}")
return None
# Step 2: Extract captions URL from ytInitialPlayerResponse
# YouTube embeds this as a JS variable in the page HTML
match = re.search(
r'ytInitialPlayerResponse\s*=\s*(\{.+?\})\s*;(?:\s*var\s|\s*<\/script>)',
html,
)
if not match:
# Fallback: try the JSON embedded in the script tag
match = re.search(
r'var\s+ytInitialPlayerResponse\s*=\s*(\{.+?\})\s*;',
html,
)
if not match:
_log(f"Direct transcript: no ytInitialPlayerResponse found for {video_id}")
return None
try:
player_response = json.loads(match.group(1))
except json.JSONDecodeError:
_log(f"Direct transcript: failed to parse ytInitialPlayerResponse for {video_id}")
return None
# Navigate to caption tracks
captions = player_response.get("captions", {})
renderer = captions.get("playerCaptionsTracklistRenderer", {})
caption_tracks = renderer.get("captionTracks", [])
if not caption_tracks:
_log(f"Direct transcript: no caption tracks for {video_id}")
return None
# Find English track (prefer exact 'en', then any en variant, then first track)
base_url = None
for track in caption_tracks:
lang = track.get("languageCode", "")
if lang == "en":
base_url = track.get("baseUrl")
break
if not base_url:
for track in caption_tracks:
lang = track.get("languageCode", "")
if lang.startswith("en"):
base_url = track.get("baseUrl")
break
if not base_url:
# Fall back to first available track
base_url = caption_tracks[0].get("baseUrl")
if not base_url:
_log(f"Direct transcript: no baseUrl in caption tracks for {video_id}")
return None
# Step 3: Fetch the VTT subtitle file
sep = "&" if "?" in base_url else "?"
vtt_url = f"{base_url}{sep}fmt=vtt"
vtt_req = urllib.request.Request(vtt_url, headers=headers)
try:
with urllib.request.urlopen(vtt_req, timeout=timeout) as resp:
vtt_text = resp.read().decode("utf-8", errors="replace")
except (urllib.error.URLError, urllib.error.HTTPError, OSError, TimeoutError) as exc:
_log(f"Direct transcript: failed to fetch VTT for {video_id}: {exc}")
return None
if not vtt_text or not vtt_text.strip():
return None
return vtt_text
def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch transcript using yt-dlp (original implementation).
Args:
video_id: YouTube video ID
temp_dir: Temporary directory for subtitle files
Returns:
Plaintext transcript string, or None if no captions available.
Raw VTT text, or None if no captions available.
"""
cmd = [
"yt-dlp",
"--ignore-config",
"--no-cookies-from-browser",
"--write-auto-subs",
"--sub-lang", "en",
"--sub-format", "vtt",
@@ -337,11 +412,35 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
return None
try:
raw = vtt_path.read_text(encoding="utf-8", errors="replace")
return vtt_path.read_text(encoding="utf-8", errors="replace")
except OSError:
return None
transcript = _clean_vtt(raw)
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Uses yt-dlp when available (preferred, more robust). Falls back to
direct HTTP transcript fetching when yt-dlp is not installed.
Args:
video_id: YouTube video ID
temp_dir: Temporary directory for subtitle files
Returns:
Plaintext transcript string, or None if no captions available.
"""
raw_vtt = None
if is_ytdlp_installed():
raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
else:
_log("yt-dlp not installed, using direct HTTP transcript fetch")
raw_vtt = _fetch_transcript_direct(video_id)
if not raw_vtt:
return None
transcript = _clean_vtt(raw_vtt)
# Truncate to max words
words = transcript.split()
@@ -417,11 +516,15 @@ def search_and_transcribe(
top_ids = [item["video_id"] for item in items[:transcript_limit]]
transcripts = fetch_transcripts_parallel(top_ids)
# Step 3: Attach transcripts to items
# Step 3: Attach transcripts and extract highlights
core_topic = _extract_core_subject(topic)
for item in items:
vid = item["video_id"]
transcript = transcripts.get(vid)
item["transcript_snippet"] = transcript or ""
item["transcript_highlights"] = extract_transcript_highlights(
transcript or "", core_topic,
)
return {"items": items}
+1 -13
View File
@@ -8,7 +8,6 @@ echo "Source: $SRC"
TARGETS=(
"$HOME/.claude/skills/last30days"
"$HOME/.claude/skills/last30daysCROSS"
"$HOME/.agents/skills/last30days"
"$HOME/.codex/skills/last30days"
)
@@ -18,18 +17,7 @@ for t in "${TARGETS[@]}"; do
echo "--- Syncing to $t ---"
mkdir -p "$t/scripts/lib"
# SKILL.md — CROSS gets patched frontmatter + skill root, others get verbatim copy
if [[ "$t" == *"last30daysCROSS"* ]]; then
sed \
-e 's/^name: last30days$/name: last30daysCROSS/' \
-e 's/^version: "2\.1"/version: "2.2-cross"/' \
-e "s|^description: .*|description: \"TEST BUILD with outcome-aware Polymarket scoring + cross-source linking. Research a topic from the last 30 days. Sources: Reddit, X, YouTube, Hacker News, Polymarket, web.\"|" \
-e "s/^argument-hint: .*/argument-hint: 'last30daysCROSS AI video tools'/" \
-e 's|"\$HOME/.claude/skills/last30days"|"$HOME/.claude/skills/last30daysCROSS" \\\n "$HOME/.claude/skills/last30days"|' \
"$SRC/SKILL.md" > "$t/SKILL.md"
else
cp "$SRC/SKILL.md" "$t/"
fi
cp "$SRC/SKILL.md" "$t/"
# Main script + lib modules (rsync handles identical files gracefully)
rsync -a "$SRC/scripts/last30days.py" "$t/scripts/"
+219
View File
@@ -0,0 +1,219 @@
#!/bin/bash
set -euo pipefail
# === V1 vs V2 Skill Test Harness ===
# Runs all 17 test queries through both v1 and v2 SKILL.md
# using `claude --print` to capture real end-to-end output.
SKILL_DIR="$HOME/.claude/skills/last30days"
REPO_DIR="/Users/mvanhorn/last30days-skill"
# Safety: always restore V2 SKILL.md on exit/crash
cleanup() {
if [ -f "$SKILL_DIR/SKILL.md.v2.bak" ]; then
echo ""
echo "⚠️ Restoring V2 SKILL.md from backup (script interrupted)..."
cp "$SKILL_DIR/SKILL.md.v2.bak" "$SKILL_DIR/SKILL.md"
rm -f "$SKILL_DIR/SKILL.md.v2.bak"
echo " ✅ V2 restored"
fi
}
trap cleanup EXIT
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
OUT_DIR="$REPO_DIR/docs/test-results/v1-vs-v2-${TIMESTAMP}"
V1_DIR="$OUT_DIR/v1"
V2_DIR="$OUT_DIR/v2"
mkdir -p "$V1_DIR" "$V2_DIR"
echo "📁 Output directory: $OUT_DIR"
echo ""
# All 17 test queries
QUERIES=(
"prompting techniques for chatgpt for legal questions"
"best clawdbot use cases"
"how to best setup clawdbot"
"prompting tips for nano banana pro for ios designs"
"top claude code skills"
"using ChatGPT to make images of dogs"
"research best practices for beautiful remotion animation videos in claude code"
"photorealistic people in nano banana pro"
"What are the best rap songs lately"
"what are people saying about DeepSeek R1"
"best practices for cursor rules files for Cursor"
"prompt advice for using suno to make killer songs in simple mode"
"how do I use Codex with Claude Code on same app to make it better"
"kanye west"
"howie.ai"
"open claw"
"nano banana pro prompting"
)
TYPES=(
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"HOW-TO"
"PROMPTING+TOOL"
"RECOMMENDATIONS"
"GENERAL"
"PROMPTING"
"PROMPTING"
"RECOMMENDATIONS"
"NEWS"
"PROMPTING"
"PROMPTING"
"HOW-TO"
"NEWS"
"GENERAL"
"GENERAL"
"PROMPTING"
)
slugify() {
echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | cut -c1-50
}
run_version() {
local version="$1"
local outdir="$2"
local total=${#QUERIES[@]}
echo ""
echo "=========================================="
echo " Running $version$total queries"
echo "=========================================="
echo ""
for i in "${!QUERIES[@]}"; do
local query="${QUERIES[$i]}"
local type="${TYPES[$i]}"
local slug
slug=$(slugify "$query")
local num=$((i + 1))
local outfile="$outdir/${num}-${slug}.txt"
local errfile="$outdir/${num}-${slug}.stderr.txt"
echo "[$version] ($num/$total) $query [$type]"
local start_time
start_time=$(date +%s)
# Run claude --print with the skill invocation
# No timeout — claude --print exits on its own; kill manually if stuck
if /Users/mvanhorn/.local/bin/claude --print \
"/last30days $query" \
> "$outfile" 2>"$errfile"; then
local end_time
end_time=$(date +%s)
local duration=$((end_time - start_time))
local lines
lines=$(wc -l < "$outfile")
echo " ✅ Done — ${lines} lines, ${duration}s"
else
local exit_code=$?
echo " ❌ Failed (exit $exit_code)" | tee -a "$outfile"
fi
# Brief pause between queries to avoid rate limits
sleep 3
done
}
# === Phase 1: Test V1 ===
echo "📦 Backing up current V2 SKILL.md..."
cp "$SKILL_DIR/SKILL.md" "$SKILL_DIR/SKILL.md.v2.bak"
echo "📥 Installing V1 SKILL.md from upstream..."
cd "$REPO_DIR"
git show upstream/main:SKILL.md | sed '/^context: fork$/d; /^agent: Explore$/d; /^disable-model-invocation: true$/d' > "$SKILL_DIR/SKILL.md"
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v1-SKILL.md"
echo " ✅ V1 installed (stripped: context:fork, agent:Explore, disable-model-invocation)"
run_version "V1" "$V1_DIR"
# === Phase 2: Test V2 ===
echo ""
echo "📥 Restoring V2 SKILL.md..."
cp "$SKILL_DIR/SKILL.md.v2.bak" "$SKILL_DIR/SKILL.md"
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v2-SKILL.md"
echo " ✅ V2 restored"
run_version "V2" "$V2_DIR"
# === Phase 3: Generate summary ===
echo ""
echo "=========================================="
echo " Generating comparison summary"
echo "=========================================="
SUMMARY="$OUT_DIR/comparison-summary.md"
cat > "$SUMMARY" << EOF
# V1 vs V2 Comparison Results
Generated: $(date)
Output directory: $OUT_DIR
## Output Files
| # | Query | Type | V1 Lines | V2 Lines | V1 Time | V2 Time |
|---|-------|------|----------|----------|---------|---------|
EOF
for i in "${!QUERIES[@]}"; do
query="${QUERIES[$i]}"
type="${TYPES[$i]}"
slug=$(slugify "$query")
num=$((i + 1))
v1file="$V1_DIR/${num}-${slug}.txt"
v2file="$V2_DIR/${num}-${slug}.txt"
v1lines=$(wc -l < "$v1file" 2>/dev/null || echo "ERR")
v2lines=$(wc -l < "$v2file" 2>/dev/null || echo "ERR")
echo "| $num | \`$query\` | $type | $v1lines | $v2lines | — | — |" >> "$SUMMARY"
done
cat >> "$SUMMARY" << 'EOF'
## Quick Check: Key Features
For each query, check these v2 improvements:
- [ ] Query parsing display (`🔍 **{TOPIC}** · {QUERY_TYPE}`)
- [ ] Sparse citations (not every sentence)
- [ ] Bold topic headers in summary
- [ ] Emoji stats tree (`├─ 🟠 Reddit:`)
- [ ] Quality checklist applied to prompts
- [ ] Self-check (research grounding, not generic)
## Scoring Guide
Use the full scoring rubric from:
`docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md`
## Next Step
Have Claude read all 34 output files and generate scored comparison:
```
Read all files in docs/test-results/v1-vs-v2-*/v1/ and v2/
Score each on the 7 dimensions from the test plan
Write the final analysis to docs/test-results/v1-vs-v2-*/analysis.md
```
EOF
# Cleanup backup
rm -f "$SKILL_DIR/SKILL.md.v2.bak"
echo ""
echo "✅ All done!"
echo ""
echo "📁 Results: $OUT_DIR"
echo "📊 Summary: $SUMMARY"
echo "📄 V1 files: $V1_DIR/"
echo "📄 V2 files: $V2_DIR/"
echo ""
echo "To review:"
echo " open $OUT_DIR"
+1
View File
@@ -0,0 +1 @@
../../SKILL.md
+11
View File
@@ -0,0 +1,11 @@
📁 Output directory: /Users/mvanhorn/last30days-skill-private/docs/test-results/v1-vs-v2-20260206-231338
📦 Backing up current V2 SKILL.md...
📥 Installing V1 SKILL.md from upstream...
✅ V1 installed (stripped: context:fork, agent:Explore, disable-model-invocation)
==========================================
Running V1 — 17 queries
==========================================
[V1] (1/17) prompting techniques for chatgpt for legal questions [PROMPTING+TOOL]
+45
View File
@@ -11,6 +11,9 @@ from lib import bird_x
class TestExtractCoreSubject(unittest.TestCase):
def tearDown(self):
bird_x._credentials.clear()
def test_strips_trending_noise(self):
result = bird_x._extract_core_subject("trendiest Claude Code skills")
self.assertNotIn("trendiest", result)
@@ -28,6 +31,9 @@ class TestExtractCoreSubject(unittest.TestCase):
class TestBirdSearchRetries(unittest.TestCase):
def tearDown(self):
bird_x._credentials.clear()
def test_last_chance_retry_uses_strongest_token(self):
"""When shorter retry also returns 0, uses longest non-noise token."""
empty = {"items": []}
@@ -53,5 +59,44 @@ class TestBirdSearchRetries(unittest.TestCase):
self.assertEqual(run_mock.call_count, 1)
class TestBirdAuthEnvironment(unittest.TestCase):
def tearDown(self):
bird_x._credentials.clear()
def test_subprocess_env_disables_browser_cookie_fallback_when_injected(self):
bird_x.set_credentials("auth-token", "ct0-token")
env = bird_x._subprocess_env()
self.assertEqual(env["AUTH_TOKEN"], "auth-token")
self.assertEqual(env["CT0"], "ct0-token")
self.assertEqual(env["BIRD_DISABLE_BROWSER_COOKIES"], "1")
def test_is_bird_authenticated_short_circuits_when_credentials_injected(self):
bird_x.set_credentials("auth-token", "ct0-token")
with mock.patch.object(bird_x, "is_bird_installed", return_value=True), \
mock.patch.object(bird_x.subprocess, "run") as run_mock:
result = bird_x.is_bird_authenticated()
self.assertEqual(result, "env AUTH_TOKEN")
run_mock.assert_not_called()
def test_search_handles_passes_injected_credentials_to_subprocess(self):
bird_x.set_credentials("auth-token", "ct0-token")
proc = mock.Mock()
proc.communicate.return_value = ("[]", "")
proc.returncode = 0
with mock.patch.object(bird_x.subprocess, "Popen", return_value=proc) as popen_mock:
bird_x.search_handles(["openai"], "codex vs claude code", "2026-01-01", count_per=1)
env = popen_mock.call_args.kwargs["env"]
self.assertEqual(env["AUTH_TOKEN"], "auth-token")
self.assertEqual(env["CT0"], "ct0-token")
self.assertEqual(env["BIRD_DISABLE_BROWSER_COOKIES"], "1")
if __name__ == "__main__":
unittest.main()
+199
View File
@@ -0,0 +1,199 @@
"""Tests for bluesky module."""
import sys
import unittest
from pathlib import Path
from unittest.mock import patch, MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
from lib import bluesky
class TestExtractCoreSubject(unittest.TestCase):
def test_strips_prefix(self):
result = bluesky._extract_core_subject("what are people saying about claude code")
self.assertEqual(result, "claude code")
def test_strips_noise(self):
result = bluesky._extract_core_subject("latest trending news claude code")
self.assertNotIn("latest", result)
self.assertNotIn("trending", result)
self.assertIn("claude", result)
def test_preserves_core(self):
result = bluesky._extract_core_subject("react native")
self.assertEqual(result, "react native")
class TestParseDate(unittest.TestCase):
def test_indexed_at_iso(self):
item = {"indexedAt": "2024-06-15T12:00:00Z"}
self.assertEqual(bluesky._parse_date(item), "2024-06-15")
def test_created_at_iso(self):
item = {"createdAt": "2024-03-01T08:30:00.000Z"}
self.assertEqual(bluesky._parse_date(item), "2024-03-01")
def test_indexed_at_preferred_over_created_at(self):
item = {"indexedAt": "2024-06-15T12:00:00Z", "createdAt": "2024-06-14T12:00:00Z"}
self.assertEqual(bluesky._parse_date(item), "2024-06-15")
def test_none_returns_none(self):
self.assertIsNone(bluesky._parse_date({}))
def test_invalid_date_returns_none(self):
self.assertIsNone(bluesky._parse_date({"indexedAt": "not-a-date"}))
class TestParseBlueskyResponse(unittest.TestCase):
def test_basic_post(self):
response = {
"posts": [{
"uri": "at://did:plc:abc123/app.bsky.feed.post/xyz789",
"author": {"handle": "alice.bsky.social", "displayName": "Alice"},
"record": {"text": "Hello world", "createdAt": "2024-06-15T12:00:00Z"},
"indexedAt": "2024-06-15T12:01:00Z",
"likeCount": 10,
"repostCount": 5,
"replyCount": 3,
"quoteCount": 1,
}]
}
items = bluesky.parse_bluesky_response(response)
self.assertEqual(len(items), 1)
self.assertEqual(items[0]["handle"], "alice.bsky.social")
self.assertEqual(items[0]["display_name"], "Alice")
self.assertEqual(items[0]["text"], "Hello world")
self.assertEqual(items[0]["url"], "https://bsky.app/profile/alice.bsky.social/post/xyz789")
self.assertEqual(items[0]["engagement"]["likes"], 10)
self.assertEqual(items[0]["engagement"]["reposts"], 5)
self.assertEqual(items[0]["date"], "2024-06-15")
def test_empty_response(self):
items = bluesky.parse_bluesky_response({})
self.assertEqual(items, [])
def test_missing_fields(self):
response = {"posts": [{"uri": "", "author": {}, "record": {}}]}
items = bluesky.parse_bluesky_response(response)
self.assertEqual(len(items), 1)
self.assertEqual(items[0]["handle"], "")
self.assertEqual(items[0]["text"], "")
def test_relevance_decreases_with_position(self):
response = {"posts": [
{"uri": f"at://did/app.bsky.feed.post/{i}", "author": {"handle": f"u{i}"}, "record": {"text": f"post {i}"}}
for i in range(5)
]}
items = bluesky.parse_bluesky_response(response)
self.assertGreater(items[0]["relevance"], items[4]["relevance"])
class TestDepthConfig(unittest.TestCase):
def test_all_depths_exist(self):
for depth in ("quick", "default", "deep"):
self.assertIn(depth, bluesky.DEPTH_CONFIG)
def test_deep_has_more_results(self):
quick = bluesky.DEPTH_CONFIG["quick"]
deep = bluesky.DEPTH_CONFIG["deep"]
self.assertGreater(deep, quick)
class TestCreateSession(unittest.TestCase):
def setUp(self):
bluesky._cached_token = None
def tearDown(self):
bluesky._cached_token = None
@patch("lib.bluesky.http.request")
def test_returns_token(self, mock_request):
mock_request.return_value = {"accessJwt": "tok123", "refreshJwt": "ref456"}
token = bluesky._create_session("user.bsky.social", "app-pw")
self.assertEqual(token, "tok123")
mock_request.assert_called_once()
@patch("lib.bluesky.http.request")
def test_caches_token(self, mock_request):
mock_request.return_value = {"accessJwt": "tok123", "refreshJwt": "ref456"}
bluesky._create_session("user.bsky.social", "app-pw")
bluesky._create_session("user.bsky.social", "app-pw")
mock_request.assert_called_once() # Only one HTTP call
@patch("lib.bluesky.http.request")
def test_returns_none_on_failure(self, mock_request):
mock_request.side_effect = Exception("connection refused")
token = bluesky._create_session("user.bsky.social", "app-pw")
self.assertIsNone(token)
self.assertIn("connection refused", bluesky._session_error)
@patch("lib.bluesky.http.request")
def test_returns_none_on_missing_jwt(self, mock_request):
mock_request.return_value = {"did": "did:plc:abc"}
token = bluesky._create_session("user.bsky.social", "app-pw")
self.assertIsNone(token)
class TestSearchBlueskyAuth(unittest.TestCase):
def setUp(self):
bluesky._cached_token = None
def tearDown(self):
bluesky._cached_token = None
def test_no_config_returns_error(self):
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09")
self.assertEqual(result["posts"], [])
self.assertIn("not configured", result["error"])
def test_empty_config_returns_error(self):
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09", config={})
self.assertEqual(result["posts"], [])
self.assertIn("not configured", result["error"])
@patch("lib.bluesky.http.request")
def test_auth_failure_returns_specific_error(self, mock_request):
mock_request.side_effect = Exception("connection refused")
config = {"BSKY_HANDLE": "user.bsky.social", "BSKY_APP_PASSWORD": "pw"}
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09", config=config)
self.assertEqual(result["posts"], [])
self.assertIn("connection refused", result["error"])
self.assertNotIn("auth failed", result["error"])
@patch("lib.bluesky.http.request")
def test_cloudflare_403_returns_network_error(self, mock_request):
from lib.http import HTTPError
mock_request.side_effect = HTTPError("HTTP 403: Forbidden", 403, "<html>Cloudflare</html>")
config = {"BSKY_HANDLE": "user.bsky.social", "BSKY_APP_PASSWORD": "pw"}
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09", config=config)
self.assertEqual(result["posts"], [])
self.assertIn("Cloudflare", result["error"])
self.assertIn("network", result["error"].lower())
@patch("lib.bluesky.http.request")
def test_401_returns_credentials_error(self, mock_request):
from lib.http import HTTPError
mock_request.side_effect = HTTPError("HTTP 401: Unauthorized", 401, "")
config = {"BSKY_HANDLE": "user.bsky.social", "BSKY_APP_PASSWORD": "pw"}
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09", config=config)
self.assertEqual(result["posts"], [])
self.assertIn("Invalid credentials", result["error"])
@patch("lib.bluesky.http.request")
def test_successful_search_passes_bearer(self, mock_request):
# First call: createSession, second call: searchPosts
mock_request.side_effect = [
{"accessJwt": "tok123", "refreshJwt": "ref456"},
{"posts": [{"uri": "at://did/app.bsky.feed.post/abc", "author": {"handle": "u1"}, "record": {"text": "hi"}}]},
]
config = {"BSKY_HANDLE": "user.bsky.social", "BSKY_APP_PASSWORD": "pw"}
result = bluesky.search_bluesky("test", "2026-01-01", "2026-03-09", config=config)
self.assertEqual(len(result["posts"]), 1)
# Verify the search call included the Bearer token
search_call = mock_request.call_args_list[1]
self.assertEqual(search_call.kwargs.get("headers", {}), {"Authorization": "Bearer tok123"})
if __name__ == "__main__":
unittest.main()
+197
View File
@@ -0,0 +1,197 @@
"""Tests for Brave Search module, including LLM Context endpoint."""
import sys
import os
import unittest
# Ensure scripts/ is on path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'scripts'))
from lib.brave_search import (
_normalize_results,
_normalize_llm_context,
_days_between,
_brave_freshness,
_parse_brave_date,
EXCLUDED_DOMAINS,
)
class TestDaysBetween(unittest.TestCase):
def test_same_day(self):
self.assertEqual(_days_between("2026-03-01", "2026-03-01"), 1)
def test_one_week(self):
self.assertEqual(_days_between("2026-03-01", "2026-03-08"), 7)
def test_invalid_dates(self):
self.assertEqual(_days_between("bad", "dates"), 30)
class TestBraveFreshness(unittest.TestCase):
def test_one_day(self):
self.assertEqual(_brave_freshness(1), "pd")
def test_one_week(self):
self.assertEqual(_brave_freshness(7), "pw")
def test_one_month(self):
self.assertEqual(_brave_freshness(31), "pm")
def test_longer_returns_range(self):
result = _brave_freshness(60)
self.assertIn("to", result)
def test_none(self):
self.assertIsNone(_brave_freshness(None))
class TestParseBraveDate(unittest.TestCase):
def test_hours_ago(self):
result = _parse_brave_date("3 hours ago", None)
self.assertIsNotNone(result)
self.assertRegex(result, r"\d{4}-\d{2}-\d{2}")
def test_days_ago(self):
result = _parse_brave_date("5 days ago", None)
self.assertIsNotNone(result)
def test_weeks_ago(self):
result = _parse_brave_date("2 weeks ago", None)
self.assertIsNotNone(result)
def test_iso_date(self):
self.assertEqual(_parse_brave_date("2026-03-10T12:00:00", None), "2026-03-10")
def test_none(self):
self.assertIsNone(_parse_brave_date(None, None))
class TestNormalizeResults(unittest.TestCase):
def test_merges_news_and_web(self):
response = {
"news": {"results": [
{"url": "https://news.example.com/a", "title": "News A", "description": "News desc"},
]},
"web": {"results": [
{"url": "https://blog.example.com/b", "title": "Blog B", "description": "Blog desc"},
]},
}
items = _normalize_results(response, "2026-03-01", "2026-03-10")
self.assertEqual(len(items), 2)
self.assertEqual(items[0]["title"], "News A")
self.assertEqual(items[1]["title"], "Blog B")
def test_excludes_reddit_and_x(self):
response = {
"web": {"results": [
{"url": "https://www.reddit.com/r/test/123", "title": "Reddit", "description": "text"},
{"url": "https://x.com/user/status/1", "title": "X post", "description": "text"},
{"url": "https://example.com/ok", "title": "OK", "description": "text"},
]},
}
items = _normalize_results(response, "2026-03-01", "2026-03-10")
self.assertEqual(len(items), 1)
self.assertEqual(items[0]["title"], "OK")
def test_default_relevance(self):
response = {"web": {"results": [
{"url": "https://a.com", "title": "A", "description": "desc"},
]}}
items = _normalize_results(response, "2026-03-01", "2026-03-10")
self.assertEqual(items[0]["relevance"], 0.6)
class TestNormalizeLlmContext(unittest.TestCase):
def _make_response(self, generic=None, sources=None):
return {
"grounding": {"generic": generic or []},
"sources": sources or {},
}
def test_basic_result(self):
resp = self._make_response(
generic=[{
"url": "https://docs.example.com/page",
"title": "Example Page",
"snippets": ["First chunk of text.", "Second chunk of text."],
}],
sources={
"https://docs.example.com/page": {
"title": "Example Page",
"hostname": "docs.example.com",
"age": ["2026-03-05", "5 days ago"],
}
},
)
items = _normalize_llm_context(resp)
self.assertEqual(len(items), 1)
item = items[0]
self.assertEqual(item["title"], "Example Page")
self.assertEqual(item["url"], "https://docs.example.com/page")
self.assertIn("First chunk", item["snippet"])
self.assertIn("Second chunk", item["snippet"])
self.assertEqual(item["date"], "2026-03-05")
self.assertEqual(item["date_confidence"], "med")
self.assertEqual(item["relevance"], 0.7)
self.assertEqual(item["source_domain"], "docs.example.com")
def test_excludes_reddit(self):
resp = self._make_response(
generic=[
{"url": "https://www.reddit.com/r/test", "title": "Reddit", "snippets": ["text"]},
{"url": "https://example.com", "title": "OK", "snippets": ["text"]},
],
)
items = _normalize_llm_context(resp)
self.assertEqual(len(items), 1)
self.assertEqual(items[0]["title"], "OK")
def test_empty_grounding(self):
resp = self._make_response()
items = _normalize_llm_context(resp)
self.assertEqual(items, [])
def test_snippet_truncation(self):
long_snippet = "x" * 2000
resp = self._make_response(
generic=[{"url": "https://a.com", "title": "A", "snippets": [long_snippet]}],
)
items = _normalize_llm_context(resp)
self.assertLessEqual(len(items[0]["snippet"]), 1500)
def test_no_date_gives_low_confidence(self):
resp = self._make_response(
generic=[{"url": "https://a.com", "title": "A", "snippets": ["text"]}],
sources={"https://a.com": {"hostname": "a.com", "age": None}},
)
items = _normalize_llm_context(resp)
self.assertIsNone(items[0]["date"])
self.assertEqual(items[0]["date_confidence"], "low")
def test_multiple_age_entries_picks_first_valid(self):
resp = self._make_response(
generic=[{"url": "https://a.com", "title": "A", "snippets": ["text"]}],
sources={"https://a.com": {
"hostname": "a.com",
"age": ["Monday, March 10, 2026", "2026-03-10", "1 day ago"],
}},
)
items = _normalize_llm_context(resp)
self.assertEqual(items[0]["date"], "2026-03-10")
def test_ids_are_sequential(self):
resp = self._make_response(
generic=[
{"url": "https://a.com", "title": "A", "snippets": ["a"]},
{"url": "https://b.com", "title": "B", "snippets": ["b"]},
{"url": "https://c.com", "title": "C", "snippets": ["c"]},
],
)
items = _normalize_llm_context(resp)
ids = [item["id"] for item in items]
self.assertEqual(ids, ["W1", "W2", "W3"])
if __name__ == "__main__":
unittest.main()
+350
View File
@@ -0,0 +1,350 @@
"""Tests for Chrome cookie extraction on macOS."""
import hashlib
import os
import sqlite3
import subprocess
import tempfile
from pathlib import Path
from unittest import mock
import pytest
from scripts.lib.chrome_cookies import (
CHROME_COOKIES_DB,
CHROME_IV_HEX,
CHROME_KEY_LENGTH,
CHROME_PBKDF2_ITERATIONS,
CHROME_SALT,
_derive_aes_key,
_get_chrome_encryption_key,
_get_db_version,
_remove_pkcs7_padding,
_decrypt_v10_value,
extract_chrome_cookies_macos,
)
# ---------------------------------------------------------------------------
# Helpers — create real encrypted cookie values using known key + system openssl
# ---------------------------------------------------------------------------
KNOWN_PASSPHRASE = b"test_passphrase_for_unit_tests"
KNOWN_AES_KEY = _derive_aes_key(KNOWN_PASSPHRASE)
def _encrypt_value_v10(plaintext: str, aes_key: bytes) -> bytes:
"""Encrypt a value the same way Chrome v10 does, using system openssl.
Returns b'v10' + AES-128-CBC ciphertext with PKCS7 padding.
"""
hex_key = aes_key.hex()
result = subprocess.run(
[
"openssl", "enc", "-aes-128-cbc", "-e",
"-K", hex_key,
"-iv", CHROME_IV_HEX,
],
input=plaintext.encode("utf-8"),
capture_output=True,
timeout=5,
)
assert result.returncode == 0, f"openssl encrypt failed: {result.stderr}"
return b"v10" + result.stdout
def _encrypt_value_v10_with_sha_prefix(plaintext: str, aes_key: bytes) -> bytes:
"""Encrypt with a 32-byte SHA-256 prefix (Chrome 130+ style)."""
raw = b"\x00" * 32 + plaintext.encode("utf-8")
hex_key = aes_key.hex()
result = subprocess.run(
[
"openssl", "enc", "-aes-128-cbc", "-e",
"-K", hex_key,
"-iv", CHROME_IV_HEX,
],
input=raw,
capture_output=True,
timeout=5,
)
assert result.returncode == 0, f"openssl encrypt failed: {result.stderr}"
return b"v10" + result.stdout
def _create_chrome_cookies_db(path: str, cookies: list[tuple], db_version: int = 20) -> None:
"""Create a minimal Chrome Cookies SQLite database.
cookies: list of (host_key, name, value, encrypted_value) tuples
"""
conn = sqlite3.connect(path)
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT)")
c.execute("INSERT OR REPLACE INTO meta (key, value) VALUES ('version', ?)", (str(db_version),))
c.execute(
"CREATE TABLE IF NOT EXISTS cookies ("
" host_key TEXT NOT NULL,"
" name TEXT NOT NULL,"
" value TEXT NOT NULL DEFAULT '',"
" encrypted_value BLOB NOT NULL DEFAULT x'',"
" path TEXT NOT NULL DEFAULT '/',"
" expires_utc INTEGER NOT NULL DEFAULT 0,"
" is_secure INTEGER NOT NULL DEFAULT 1,"
" is_httponly INTEGER NOT NULL DEFAULT 1,"
" creation_utc INTEGER NOT NULL DEFAULT 0,"
" last_access_utc INTEGER NOT NULL DEFAULT 0,"
" has_expires INTEGER NOT NULL DEFAULT 1,"
" is_persistent INTEGER NOT NULL DEFAULT 1,"
" priority INTEGER NOT NULL DEFAULT 1,"
" samesite INTEGER NOT NULL DEFAULT 0,"
" source_scheme INTEGER NOT NULL DEFAULT 2,"
" source_port INTEGER NOT NULL DEFAULT 443,"
" last_update_utc INTEGER NOT NULL DEFAULT 0"
")"
)
for host_key, name, value, encrypted_value in cookies:
c.execute(
"INSERT INTO cookies (host_key, name, value, encrypted_value) VALUES (?, ?, ?, ?)",
(host_key, name, value, encrypted_value),
)
conn.commit()
conn.close()
# ---------------------------------------------------------------------------
# PKCS7 padding tests
# ---------------------------------------------------------------------------
class TestPkcs7Padding:
def test_valid_padding_1(self):
# 1 byte of padding
data = b"hello world!!!!!" + b"\x01"
assert _remove_pkcs7_padding(data) == b"hello world!!!!!"
def test_valid_padding_5(self):
data = b"hello world" + b"\x05\x05\x05\x05\x05"
assert _remove_pkcs7_padding(data) == b"hello world"
def test_valid_padding_16(self):
# Full block of padding
data = b"\x10" * 16
assert _remove_pkcs7_padding(data) == b""
def test_invalid_padding_zero(self):
data = b"hello\x00"
assert _remove_pkcs7_padding(data) is None
def test_invalid_padding_mismatch(self):
data = b"hello\x03\x03\x02"
assert _remove_pkcs7_padding(data) is None
def test_empty_data(self):
assert _remove_pkcs7_padding(b"") is None
# ---------------------------------------------------------------------------
# Key derivation test
# ---------------------------------------------------------------------------
class TestKeyDerivation:
def test_derive_aes_key_deterministic(self):
key1 = _derive_aes_key(b"my_passphrase")
key2 = _derive_aes_key(b"my_passphrase")
assert key1 == key2
assert len(key1) == 16
def test_derive_aes_key_different_passphrases(self):
key1 = _derive_aes_key(b"passphrase_a")
key2 = _derive_aes_key(b"passphrase_b")
assert key1 != key2
# ---------------------------------------------------------------------------
# Decryption test (real openssl, known key)
# ---------------------------------------------------------------------------
class TestDecryption:
def test_decrypt_v10_roundtrip(self):
"""Encrypt then decrypt — verifies the full pipeline works."""
original = "my_secret_cookie_value_12345"
encrypted = _encrypt_value_v10(original, KNOWN_AES_KEY)
assert encrypted[:3] == b"v10"
decrypted = _decrypt_v10_value(encrypted, KNOWN_AES_KEY, db_version=20)
assert decrypted == original
def test_decrypt_v10_chrome130_with_sha_prefix(self):
"""Chrome 130+ (db_version >= 24) strips 32-byte SHA-256 prefix."""
original = "session_token_abc"
encrypted = _encrypt_value_v10_with_sha_prefix(original, KNOWN_AES_KEY)
decrypted = _decrypt_v10_value(encrypted, KNOWN_AES_KEY, db_version=24)
assert decrypted == original
def test_decrypt_wrong_key_returns_none_or_garbage(self):
"""Wrong key should either fail decryption or produce garbage."""
original = "secret"
encrypted = _encrypt_value_v10(original, KNOWN_AES_KEY)
wrong_key = _derive_aes_key(b"wrong_passphrase")
result = _decrypt_v10_value(encrypted, wrong_key, db_version=20)
# Either None (padding check fails) or garbage (not the original)
assert result is None or result != original
def test_decrypt_empty_ciphertext(self):
"""v10 prefix with no ciphertext should return None."""
assert _decrypt_v10_value(b"v10", KNOWN_AES_KEY, db_version=20) is None
# ---------------------------------------------------------------------------
# Chrome not installed → returns None
# ---------------------------------------------------------------------------
class TestChromeNotInstalled:
def test_db_not_found(self):
with mock.patch(
"scripts.lib.chrome_cookies.CHROME_COOKIES_DB",
Path("/nonexistent/path/Cookies"),
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
assert result is None
# ---------------------------------------------------------------------------
# Keychain access denied → returns None
# ---------------------------------------------------------------------------
class TestKeychainDenied:
def test_security_command_fails(self):
with mock.patch("scripts.lib.chrome_cookies.subprocess.run") as mock_run:
mock_run.return_value = subprocess.CompletedProcess(
args=[], returncode=44, stdout="", stderr="security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain."
)
result = _get_chrome_encryption_key()
assert result is None
def test_security_command_not_found(self):
with mock.patch("scripts.lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError):
result = _get_chrome_encryption_key()
assert result is None
# ---------------------------------------------------------------------------
# openssl not found → returns None
# ---------------------------------------------------------------------------
class TestOpensslNotFound:
def test_openssl_missing(self):
encrypted = _encrypt_value_v10("test", KNOWN_AES_KEY)
with mock.patch("scripts.lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError):
result = _decrypt_v10_value(encrypted, KNOWN_AES_KEY, db_version=20)
assert result is None
# ---------------------------------------------------------------------------
# Unencrypted cookie values → returned as-is
# ---------------------------------------------------------------------------
class TestUnencryptedCookies:
def test_plain_value_returned(self, tmp_path):
"""Unencrypted cookies (value column populated) returned without decryption."""
db_path = str(tmp_path / "Cookies")
_create_chrome_cookies_db(db_path, [
(".x.com", "auth_token", "plain_token_value", b""),
(".x.com", "ct0", "plain_ct0_value", b""),
])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
# No keychain needed for unencrypted values
with mock.patch("scripts.lib.chrome_cookies._get_chrome_encryption_key", return_value=None):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
assert result == {"auth_token": "plain_token_value", "ct0": "plain_ct0_value"}
# ---------------------------------------------------------------------------
# Full integration: mock DB with real v10 encryption, mock Keychain
# ---------------------------------------------------------------------------
class TestFullExtraction:
def test_encrypted_cookies_extracted(self, tmp_path):
"""End-to-end: create DB with real v10-encrypted values, extract them."""
auth_val = "my_auth_token_123"
ct0_val = "my_ct0_csrf_456"
encrypted_auth = _encrypt_value_v10(auth_val, KNOWN_AES_KEY)
encrypted_ct0 = _encrypt_value_v10(ct0_val, KNOWN_AES_KEY)
db_path = str(tmp_path / "Cookies")
_create_chrome_cookies_db(db_path, [
(".x.com", "auth_token", "", encrypted_auth),
(".x.com", "ct0", "", encrypted_ct0),
(".other.com", "other", "", b""), # unrelated cookie
])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == auth_val
assert result["ct0"] == ct0_val
def test_no_matching_cookies_returns_none(self, tmp_path):
db_path = str(tmp_path / "Cookies")
_create_chrome_cookies_db(db_path, [
(".other.com", "session", "val", b""),
])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch("scripts.lib.chrome_cookies._get_chrome_encryption_key", return_value=None):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
assert result is None
def test_chrome130_db_version_24(self, tmp_path):
"""Chrome 130+ with db_version >= 24 strips SHA-256 prefix."""
auth_val = "token_for_chrome130"
encrypted_auth = _encrypt_value_v10_with_sha_prefix(auth_val, KNOWN_AES_KEY)
db_path = str(tmp_path / "Cookies")
_create_chrome_cookies_db(db_path, [
(".x.com", "auth_token", "", encrypted_auth),
], db_version=24)
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
assert result is not None
assert result["auth_token"] == auth_val
# ---------------------------------------------------------------------------
# DB version detection
# ---------------------------------------------------------------------------
class TestDbVersion:
def test_reads_version_from_meta(self, tmp_path):
db_path = str(tmp_path / "test.db")
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("CREATE TABLE meta (key TEXT, value TEXT)")
c.execute("INSERT INTO meta VALUES ('version', '24')")
conn.commit()
assert _get_db_version(c) == 24
conn.close()
def test_no_meta_table_returns_zero(self, tmp_path):
db_path = str(tmp_path / "test.db")
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("CREATE TABLE dummy (x TEXT)")
conn.commit()
assert _get_db_version(c) == 0
conn.close()
+6 -3
View File
@@ -124,7 +124,8 @@ class TestLoadCodexAuth(unittest.TestCase):
class TestGetAvailableSourcesWithAuth(unittest.TestCase):
def test_codex_auth_ok_counts_as_openai(self):
@patch("lib.bird_x.is_bird_installed", return_value=False)
def test_codex_auth_ok_counts_as_openai(self, _mock_bird):
config = {
"OPENAI_API_KEY": "codex-token",
"OPENAI_AUTH_STATUS": "ok",
@@ -133,14 +134,16 @@ class TestGetAvailableSourcesWithAuth(unittest.TestCase):
result = env.get_available_sources(config)
self.assertIn("reddit", result)
def test_codex_auth_expired_not_counted(self):
@patch("lib.bird_x.is_bird_installed", return_value=False)
def test_codex_auth_expired_not_counted(self, _mock_bird):
config = {
"OPENAI_API_KEY": None,
"OPENAI_AUTH_STATUS": "expired",
"XAI_API_KEY": None,
}
result = env.get_available_sources(config)
self.assertEqual(result, "web")
# Reddit is available via public JSON fallback even without OpenAI auth
self.assertEqual(result, "reddit")
class TestParseCodexStream(unittest.TestCase):
+293
View File
@@ -0,0 +1,293 @@
"""Tests for browser cookie extraction module."""
import configparser
import sqlite3
import textwrap
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from unittest.mock import patch
import pytest
from scripts.lib.cookie_extract import (
extract_cookies,
extract_firefox_cookies,
_find_default_profile,
_get_firefox_profiles_dir,
)
@pytest.fixture
def mock_firefox_env(tmp_path):
"""Create a mock Firefox profiles directory with cookies.sqlite.
Returns (profiles_dir, profile_dir) for patching.
"""
def _make(
*,
profiles_ini=None, # type: Optional[str]
profiles=None, # type: Optional[Dict[str, List[Tuple[str, str, str]]]]
default_profile="abc123.default-release", # type: str
):
profiles_dir = tmp_path / "Firefox"
profiles_dir.mkdir(parents=True, exist_ok=True)
# Default: one profile with X cookies
if profiles is None:
profiles = {
default_profile: [
(".x.com", "auth_token", "tok_abc123"),
(".x.com", "ct0", "ct0_xyz789"),
(".example.com", "session", "sess_other"),
],
}
# Create profile directories with cookies databases
for profile_name, cookies in profiles.items():
profile_dir = profiles_dir / profile_name
profile_dir.mkdir(parents=True, exist_ok=True)
db_path = profile_dir / "cookies.sqlite"
conn = sqlite3.connect(str(db_path))
conn.execute(
"CREATE TABLE moz_cookies ("
" id INTEGER PRIMARY KEY,"
" name TEXT NOT NULL,"
" value TEXT NOT NULL,"
" host TEXT NOT NULL,"
" path TEXT DEFAULT '/',"
" expiry INTEGER DEFAULT 0,"
" isSecure INTEGER DEFAULT 1,"
" isHttpOnly INTEGER DEFAULT 1,"
" sameSite INTEGER DEFAULT 0,"
" schemeMap INTEGER DEFAULT 0"
")"
)
for host, name, value in cookies:
conn.execute(
"INSERT INTO moz_cookies (name, value, host) VALUES (?, ?, ?)",
(name, value, host),
)
conn.commit()
conn.close()
# Write profiles.ini
if profiles_ini is None:
profiles_ini = textwrap.dedent(f"""\
[General]
StartWithLastProfile=1
[Profile0]
Name=default-release
IsRelative=1
Path={default_profile}
Default=1
""")
(profiles_dir / "profiles.ini").write_text(profiles_ini)
return profiles_dir
return _make
class TestExtractFirefoxCookies:
"""Tests for extract_firefox_cookies."""
def test_valid_cookies_extracted(self, mock_firefox_env):
"""Cookies for the target domain are returned correctly."""
profiles_dir = mock_firefox_env()
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == "tok_abc123"
assert result["ct0"] == "ct0_xyz789"
assert "session" not in result # different domain cookie not included
def test_multiple_profiles_selects_default(self, mock_firefox_env):
"""When multiple profiles exist, the one with Default=1 is used."""
profiles_dir = mock_firefox_env(
profiles={
"aaa111.other": [
(".x.com", "auth_token", "wrong_token"),
],
"bbb222.default-release": [
(".x.com", "auth_token", "correct_token"),
(".x.com", "ct0", "correct_ct0"),
],
},
profiles_ini=textwrap.dedent("""\
[General]
StartWithLastProfile=1
[Profile0]
Name=other
IsRelative=1
Path=aaa111.other
[Profile1]
Name=default-release
IsRelative=1
Path=bbb222.default-release
Default=1
"""),
)
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == "correct_token"
assert result["ct0"] == "correct_ct0"
def test_firefox_not_installed(self):
"""Returns None when Firefox profiles directory doesn't exist."""
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=None,
):
result = extract_firefox_cookies(".x.com", ["auth_token"])
assert result is None
def test_cookies_sqlite_empty(self, mock_firefox_env):
"""Returns None when cookies.sqlite has no rows."""
profiles_dir = mock_firefox_env(
profiles={"abc123.default-release": []}, # no cookies
)
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
assert result is None
def test_domain_has_no_cookies(self, mock_firefox_env):
"""Returns None when cookies exist but not for the target domain."""
profiles_dir = mock_firefox_env(
profiles={
"abc123.default-release": [
(".example.com", "session", "sess_123"),
],
},
)
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
assert result is None
def test_malformed_profiles_ini_falls_back(self, mock_firefox_env):
"""Falls back to first profile on disk when profiles.ini is garbage."""
profiles_dir = mock_firefox_env(
profiles={
"zzz999.fallback": [
(".x.com", "auth_token", "fallback_token"),
],
},
profiles_ini="this is not valid ini content\n[[[broken",
)
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_firefox_cookies(".x.com", ["auth_token"])
assert result is not None
assert result["auth_token"] == "fallback_token"
class TestExtractCookiesAuto:
"""Tests for extract_cookies with browser='auto'."""
def test_auto_macos_tries_chrome_then_firefox(self, mock_firefox_env):
"""On macOS, auto tries Chrome first, falls back to Firefox if Chrome fails."""
profiles_dir = mock_firefox_env()
with (
patch("scripts.lib.cookie_extract.platform.system", return_value="Darwin"),
patch(
"scripts.lib.cookie_extract.extract_chrome_cookies",
return_value=None,
),
patch(
"scripts.lib.cookie_extract.extract_safari_cookies",
return_value=None,
),
patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
),
):
result = extract_cookies("auto", ".x.com", ["auth_token", "ct0"])
# Chrome and Safari return None, Firefox succeeds
assert result is not None
assert result["auth_token"] == "tok_abc123"
assert result["ct0"] == "ct0_xyz789"
def test_auto_linux_tries_firefox_only(self, mock_firefox_env):
"""On Linux, auto only tries Firefox."""
profiles_dir = mock_firefox_env()
with (
patch("scripts.lib.cookie_extract.platform.system", return_value="Linux"),
patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
),
):
result = extract_cookies("auto", ".x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == "tok_abc123"
def test_explicit_firefox(self, mock_firefox_env):
"""Explicit browser='firefox' goes directly to Firefox."""
profiles_dir = mock_firefox_env()
with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir,
):
result = extract_cookies("firefox", ".x.com", ["auth_token"])
assert result is not None
assert result["auth_token"] == "tok_abc123"
def test_unknown_browser_returns_none(self):
"""Unknown browser name returns None."""
result = extract_cookies("netscape", ".x.com", ["auth_token"])
assert result is None
def test_chrome_delegates_to_chrome_module(self):
"""Chrome extraction delegates to chrome_cookies module."""
with patch(
"scripts.lib.cookie_extract.extract_chrome_cookies",
return_value={"auth_token": "chrome_tok"},
):
result = extract_cookies("chrome", ".x.com", ["auth_token"])
assert result == {"auth_token": "chrome_tok"}
def test_safari_delegates_to_safari_module(self):
"""Safari extraction delegates to safari_cookies module."""
with patch(
"scripts.lib.cookie_extract.extract_safari_cookies",
return_value={"auth_token": "safari_tok"},
):
result = extract_cookies("safari", ".x.com", ["auth_token"])
assert result == {"auth_token": "safari_tok"}
+167
View File
@@ -0,0 +1,167 @@
"""Tests for entity_extract module."""
import sys
import unittest
from pathlib import Path
# Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
from lib import entity_extract
class TestExtractXHandles(unittest.TestCase):
def test_basic_author_handle(self):
items = [{"author_handle": "techguru", "text": ""}]
result = entity_extract._extract_x_handles(items)
self.assertEqual(result, ["techguru"])
def test_mentions_in_text(self):
items = [{"text": "Great thread by @airesearcher and @mldev"}]
result = entity_extract._extract_x_handles(items)
self.assertIn("airesearcher", result)
self.assertIn("mldev", result)
def test_generic_handles_filtered(self):
items = [
{"author_handle": "@openai", "text": ""},
{"author_handle": "@elonmusk", "text": ""},
{"author_handle": "realexpert", "text": ""},
]
result = entity_extract._extract_x_handles(items)
self.assertEqual(result, ["realexpert"])
def test_case_normalization(self):
items = [{"author_handle": "@CamelCase", "text": ""}]
result = entity_extract._extract_x_handles(items)
self.assertEqual(result, ["camelcase"])
def test_frequency_ranking(self):
items = [
{"author_handle": "popular", "text": ""},
{"author_handle": "popular", "text": ""},
{"author_handle": "popular", "text": ""},
{"author_handle": "rare", "text": ""},
]
result = entity_extract._extract_x_handles(items)
self.assertEqual(result[0], "popular")
def test_leading_at_stripped(self):
items = [{"author_handle": "@withatsign", "text": ""}]
result = entity_extract._extract_x_handles(items)
self.assertEqual(result, ["withatsign"])
def test_empty_input(self):
result = entity_extract._extract_x_handles([])
self.assertEqual(result, [])
def test_mixed_items(self):
items = [
{"author_handle": "poster1", "text": "Check @mentioned"},
{"text": "No author here"},
{"author_handle": "", "text": ""},
]
result = entity_extract._extract_x_handles(items)
self.assertIn("poster1", result)
self.assertIn("mentioned", result)
self.assertEqual(len(result), 2)
class TestExtractXHashtags(unittest.TestCase):
def test_basic_hashtag(self):
items = [{"text": "Exciting news #AI"}]
result = entity_extract._extract_x_hashtags(items)
self.assertEqual(result, ["#ai"])
def test_multiple_tags(self):
items = [{"text": "#Python and #MachineLearning are trending"}]
result = entity_extract._extract_x_hashtags(items)
self.assertIn("#python", result)
self.assertIn("#machinelearning", result)
def test_frequency_ranking(self):
items = [
{"text": "#ai is great"},
{"text": "#ai again"},
{"text": "#rare tag"},
]
result = entity_extract._extract_x_hashtags(items)
self.assertEqual(result[0], "#ai")
def test_single_char_tag_filtered(self):
items = [{"text": "#X is not enough chars but #AI is"}]
result = entity_extract._extract_x_hashtags(items)
# #X is only 1 char, filtered by \w{2,30} regex
self.assertNotIn("#x", result)
self.assertIn("#ai", result)
def test_empty_input(self):
result = entity_extract._extract_x_hashtags([])
self.assertEqual(result, [])
class TestExtractSubreddits(unittest.TestCase):
def test_basic_subreddit_field(self):
items = [{"subreddit": "MachineLearning"}]
result = entity_extract._extract_subreddits(items)
self.assertEqual(result, ["MachineLearning"])
def test_cross_ref_in_comment_insights(self):
items = [{"subreddit": "AI", "comment_insights": ["Check out r/localLLaMA for more"]}]
result = entity_extract._extract_subreddits(items)
self.assertIn("localLLaMA", result)
def test_cross_ref_in_top_comments(self):
items = [{"subreddit": "tech", "top_comments": [{"excerpt": "Also see r/programming"}]}]
result = entity_extract._extract_subreddits(items)
self.assertIn("programming", result)
def test_frequency_ranking(self):
items = [
{"subreddit": "popular"},
{"subreddit": "popular"},
{"subreddit": "rare"},
]
result = entity_extract._extract_subreddits(items)
self.assertEqual(result[0], "popular")
def test_leading_r_slash_stripped(self):
items = [{"subreddit": "r/stripped"}]
result = entity_extract._extract_subreddits(items)
self.assertEqual(result, ["stripped"])
def test_empty_input(self):
result = entity_extract._extract_subreddits([])
self.assertEqual(result, [])
class TestExtractEntities(unittest.TestCase):
def test_integration(self):
reddit = [{"subreddit": "AI", "comment_insights": ["r/localLLaMA"]}]
x = [{"author_handle": "researcher", "text": "#deeplearning @colleague"}]
result = entity_extract.extract_entities(reddit, x)
self.assertIn("researcher", result["x_handles"])
self.assertIn("#deeplearning", result["x_hashtags"])
self.assertIn("AI", result["reddit_subreddits"])
def test_max_limits(self):
x = [
{"author_handle": f"user{i}", "text": ""}
for i in range(10)
]
result = entity_extract.extract_entities([], x, max_handles=2)
self.assertLessEqual(len(result["x_handles"]), 2)
def test_empty_inputs(self):
result = entity_extract.extract_entities([], [])
self.assertEqual(result["x_handles"], [])
self.assertEqual(result["x_hashtags"], [])
self.assertEqual(result["reddit_subreddits"], [])
def test_return_keys(self):
result = entity_extract.extract_entities([], [])
self.assertSetEqual(set(result.keys()), {"x_handles", "x_hashtags", "reddit_subreddits"})
if __name__ == "__main__":
unittest.main()
+185
View File
@@ -0,0 +1,185 @@
"""Tests for browser cookie extraction integration in env.py."""
from unittest.mock import patch, MagicMock
import pytest
from scripts.lib.env import extract_browser_credentials, COOKIE_DOMAINS
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _base_config(**overrides):
"""Return a minimal config dict with common defaults."""
cfg = {
"AUTH_TOKEN": None,
"CT0": None,
"TRUTHSOCIAL_TOKEN": None,
"FROM_BROWSER": None,
"SETUP_COMPLETE": None,
}
cfg.update(overrides)
return cfg
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
class TestExtractBrowserCredentials:
"""Unit tests for extract_browser_credentials()."""
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_auto_with_setup_complete_populates_credentials(self, mock_extract):
"""FROM_BROWSER=auto, SETUP_COMPLETE=true, mock returns valid cookies
-> config now contains AUTH_TOKEN and CT0."""
mock_extract.return_value = ({"auth_token": "tok123", "ct0": "ct0val"}, "chrome")
config = _base_config(FROM_BROWSER="auto", SETUP_COMPLETE="true")
result = extract_browser_credentials(config)
assert result["AUTH_TOKEN"] == "tok123"
assert result["CT0"] == "ct0val"
# Should have been called for x domain
mock_extract.assert_any_call("auto", ".x.com", ["auth_token", "ct0"])
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_explicit_auth_token_skips_x_extraction(self, mock_extract):
"""Config already has AUTH_TOKEN and CT0 from env var
-> cookie extraction skipped for X (explicit takes priority)."""
# Return None for any non-X domains that still get checked (e.g. Truth Social)
mock_extract.return_value = None
config = _base_config(
AUTH_TOKEN="explicit_token",
CT0="explicit_ct0",
FROM_BROWSER="auto",
SETUP_COMPLETE="true",
)
result = extract_browser_credentials(config)
# X cookies should not appear in result (already set)
assert "AUTH_TOKEN" not in result
assert "CT0" not in result
# extract_cookies should NOT have been called for .x.com
for call in mock_extract.call_args_list:
assert call[0][1] != ".x.com", "Should not extract cookies for X when credentials already set"
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_from_browser_off_skips_all(self, mock_extract):
"""FROM_BROWSER=off -> no cookie extraction attempted."""
config = _base_config(FROM_BROWSER="off", SETUP_COMPLETE="true")
result = extract_browser_credentials(config)
assert result == {}
mock_extract.assert_not_called()
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_no_setup_complete_no_from_browser_defaults_off(self, mock_extract):
"""FROM_BROWSER not set and SETUP_COMPLETE not set
-> no extraction (wizard hasn't run)."""
config = _base_config() # both None
result = extract_browser_credentials(config)
assert result == {}
mock_extract.assert_not_called()
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_setup_complete_no_from_browser_defaults_auto(self, mock_extract):
"""SETUP_COMPLETE is set but FROM_BROWSER is not
-> defaults to 'auto'."""
mock_extract.return_value = ({"auth_token": "found", "ct0": "found_ct0"}, "firefox")
config = _base_config(SETUP_COMPLETE="true") # FROM_BROWSER=None
result = extract_browser_credentials(config)
assert result["AUTH_TOKEN"] == "found"
# extract_cookies should have been called with 'auto'
mock_extract.assert_any_call("auto", ".x.com", ["auth_token", "ct0"])
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_from_browser_firefox_only(self, mock_extract):
"""FROM_BROWSER=firefox -> only Firefox extraction attempted."""
mock_extract.return_value = ({"auth_token": "ff_tok", "ct0": "ff_ct0"}, "firefox")
config = _base_config(FROM_BROWSER="firefox", SETUP_COMPLETE="true")
result = extract_browser_credentials(config)
assert result["AUTH_TOKEN"] == "ff_tok"
# All calls should use 'firefox' as the browser arg
for call in mock_extract.call_args_list:
assert call[0][0] == "firefox"
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_extraction_returns_none_config_unchanged(self, mock_extract):
"""Cookie extraction returns None for X -> config unchanged for AUTH_TOKEN."""
mock_extract.return_value = None
config = _base_config(FROM_BROWSER="auto", SETUP_COMPLETE="true")
result = extract_browser_credentials(config)
assert "AUTH_TOKEN" not in result
assert "CT0" not in result
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_extraction_raises_exception_config_unchanged(self, mock_extract):
"""Cookie extraction raises exception -> caught, config unchanged."""
mock_extract.side_effect = RuntimeError("database locked")
config = _base_config(FROM_BROWSER="auto", SETUP_COMPLETE="true")
result = extract_browser_credentials(config)
# Should not raise, and no credentials populated
assert "AUTH_TOKEN" not in result
assert "CT0" not in result
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
def test_partial_credentials_only_fills_missing(self, mock_extract):
"""If AUTH_TOKEN is set but CT0 is not, only CT0 gets filled."""
mock_extract.return_value = ({"auth_token": "cookie_tok", "ct0": "cookie_ct0"}, "chrome")
config = _base_config(
AUTH_TOKEN="explicit",
CT0=None,
FROM_BROWSER="auto",
SETUP_COMPLETE="true",
)
result = extract_browser_credentials(config)
# AUTH_TOKEN already set, should not be overridden
assert "AUTH_TOKEN" not in result
# CT0 was missing, should be filled
assert result["CT0"] == "cookie_ct0"
class TestGetConfigCookieIntegration:
"""Integration test: get_config() calls extract_browser_credentials."""
@patch("scripts.lib.cookie_extract.extract_cookies_with_source")
@patch("scripts.lib.env._find_project_env", return_value=None)
@patch("scripts.lib.env.load_env_file", return_value={})
@patch("scripts.lib.env.get_openai_auth")
def test_get_config_injects_cookies(
self, mock_openai, mock_load, mock_proj, mock_extract
):
"""get_config merges browser cookies into the returned config."""
from scripts.lib.env import get_config, OpenAIAuth
mock_openai.return_value = OpenAIAuth(
token=None, source="none", status="missing",
account_id=None, codex_auth_file="/fake",
)
mock_extract.return_value = ({"auth_token": "browser_tok", "ct0": "browser_ct0"}, "firefox")
import os
env_patch = {
"SETUP_COMPLETE": "true",
"FROM_BROWSER": "auto",
"LAST30DAYS_CONFIG_DIR": "",
}
with patch.dict(os.environ, env_patch, clear=False):
config = get_config()
assert config["AUTH_TOKEN"] == "browser_tok"
assert config["CT0"] == "browser_ct0"

Some files were not shown because too many files have changed in this diff Show More