Compare commits

..

384 Commits

Author SHA1 Message Date
Matt Van Horn b31ec05c74 marketing: v3.1 launch video — 30s Remotion render
Six-scene 30-second MP4 announcing v3.1 (competitors mode) for X.

Lives at marketing/v3.1-launch/. Renders to out/last30days-v3.1-launch.mp4
(gitignored). 1920×1080 @ 30fps, H.264, ~3MB final.

Six scenes:
1. Hook (0-3s): badge + "What if one search ran 3 at once?"
2. Old way (3-8s): single terminal /last30days OpenAI
3. Fan-out (8-14s): --competitors splits into 3 parallel panes
4. Comparison (14-21s): 3 panes collapse into Head-to-Head table
5. How (21-26s): "You pick the topic. Agent picks peers. Engine fans out."
6. CTA (26-30s): install command + repo URL

Reusable components: TerminalWindow, TypedLine, BadgeBar,
ComparisonTable. Scene timing in src/lib/timing.ts as single source of
truth — one edit to retime everything.

Marketing version 3.1 ≠ engine code version (still 3.0.14). 3.1 is the
launch label that bundles 3.0.11-3.0.14 into one shippable narrative.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:48:02 -07:00
Matt Van Horn 949bcf8942 feat: vs mode N full passes + --competitors auto-discovery + (/Last30Days) title (#312)
* feat: vs mode runs N full passes; --competitors wraps vs with auto-discovery

Unifies vs-mode and --competitors onto one fanout architecture. A topic
containing "vs" / "versus" now runs N full pipeline.run() calls in parallel
(reverting the one-pass latency optimization that removed per-entity
depth); --competitors becomes a SKILL.md-level shortcut where the hosting
reasoning model (Claude Code, Codex, Hermes, Gemini) discovers N peers via
its own WebSearch, runs Step 0.55 per entity, and invokes the engine with
a vs-topic + --competitors-plan JSON.

Changed:
- vs-mode: N full passes in parallel via fanout (was 1 merged pass).
- --competitors: SKILL.md shortcut for vs-mode-with-discovery. Engine flag
  kept for headless/cron use. LAW 7-style stderr reframed to lead with the
  hosting-model path (use WebSearch + --competitors-plan) instead of
  BRAVE_API_KEY. Footer BRAVE/SERPER nudge suppressed when --plan or
  --competitors-plan present (hosting model already has WebSearch).

Added:
- --competitors-plan JSON flag: per-entity {x_handle, x_related, subreddits,
  github_user, github_repos, context}. Accepts inline JSON or file path.
  subrun_kwargs_for helper is the single source of truth for per-entity
  kwargs — no closure-default fallthrough from main scope.
- Per-entity save files: each entity's sub-run produces its own
  {slug}-raw.md with a single-row Resolved Entities block.
- --polymarket-keywords filter for ambiguous single-token topics.

Fixed:
- test_competitor_subrun_isolation regression suite locks in 3.0.12's
  no-leak invariant (main flags do not inherit into peer sub-runs).
- Updates test_regression.py for the new comparison-mode payload shape.

Bumps plugin.json to 3.0.13. 1,219 tests passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: comparison title attribution — (Last 30 Days) → (/Last30Days)

User feedback on 3.0.13 dogfood runs (Kanye vs Drake, Mercer Island,
Figma): the comparison-mode synthesis title should attribute to the
slash command rather than restate the date range.

Three SKILL.md occurrences updated. Pure documentation change. Bumps to
3.0.14.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:31:00 -07:00
Matt Van Horn 00d01933e0 fix: per-entity Step 0.55, LAW 7 sub-run quiet, default 2, canonical SKILL.md (#311)
Four fixes based on 2026-04-22 test-window feedback on v3.0.11 --competitors:

- Each competitor sub-run now runs Step 0.55 (X handle / subreddits /
  GitHub) via resolve.auto_resolve inside the fanout closure. Deep-copied
  config per entity prevents _auto_resolve_context leak across sub-runs.
  Resolved data stored on report.artifacts["resolved"] for the renderer.
- New internal_subrun keyword on planner.plan_query and pipeline.run
  suppresses the LAW 7 "No --plan passed" stderr for engine-internal
  fan-out only. Default path unchanged.
- Default --competitors count is now 2 (3-way total). --competitors=N
  still customizes; range 1..6.
- SKILL.md STEP 0 canonical-path self-check forces readers who loaded
  from marketplaces/ (auto-restored to origin/main, stale) to re-read
  from plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md.
  Two of three 2026-04-22 test windows hit this stale-path trap.
- New ## Resolved Entities block in render_comparison_multi shows
  per-entity handles/subs/github for debug visibility.

Bumps plugin.json to 3.0.12. 12 new tests; 1,175 total passing.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:30:08 -07:00
Matt Van Horn 5f054380c5 feat: --competitors flag for auto-discovered comparison fan-out (#308)
Pass `--competitors` on a single-entity topic and the engine auto-discovers
2-6 peer entities via web search, runs the full pipeline on each in
parallel, and returns one N-way comparison reusing the existing 9-axis
Head-to-Head scaffold. `last30days OpenAI --competitors` resolves to
Anthropic + xAI + Google Gemini; `last30days Kanye West --competitors`
resolves to Drake + Kendrick Lamar + one more peer.

- New CLI flags: --competitors, --competitors=N, --competitors-list
- New scripts/lib/competitors.py — mirrors resolve.auto_resolve pattern
  (web search + deterministic text extraction, no internal LLM)
- New scripts/lib/fanout.py — ThreadPoolExecutor orchestrator; per-entity
  failures degrade gracefully as long as >=2 entities survive
- Multi-report render in scripts/lib/render.py reuses the comparison
  scaffold for the synthesis table
- LAW 7-style stderr when no backend and no list, pointing the hosting
  reasoning model at --competitors-list
- 38 new tests across CLI parsing, discovery, fanout, and rendering

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:28:36 -07:00
Matt Van Horn ff21243517 Merge pull request #130 from chaosreload/feat/xurl-x-search
feat: add xurl CLI as alternative X search backend (official API v2 via OAuth2)
2026-04-22 18:55:22 -07:00
Matt Van Horn 4e91f4e754 fix: Step 0.55 category-peer subreddit expansion (#305)
* feat(resolve): category-peer subreddit map for Step 0.55

Introduces scripts/lib/categories.py with a curated category->peer-subs
map and wires scripts/lib/resolve.py auto_resolve() to merge peers into
the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode:
a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and
missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where
prompting techniques actually live.

Map is static, curated, ~11 categories (ai_image_generation,
ai_video_generation, ai_music_generation, ai_coding_agent,
ai_agent_framework, ai_chat_model, saas_screen_recording,
saas_productivity, prediction_markets, crypto_defi, dev_tool_cli).
First-match-wins ordering from most-specific to least-specific.
Compound-term patterns only (no bare common nouns like "image", "ai").

auto_resolve now:
- calls detect_category(topic) after _extract_subreddits
- merges peer_subs case-insensitively, caps at MAX_SUBS (10)
- preserves every WebSearch-returned sub (freshest signal)
- emits [Resolve] Matched category=<id>, adding peers: <list> on stderr
  only when peers were actually added
- returns new "category" key in the result dict for observability
- wraps classifier in try/except so failures degrade to unwidened list

Includes drive-by: test_full_resolve / test_partial_failure
searches_run expectations bumped from 3->4 / 2->3 to match the current
queries dict (subreddit + news + x_handle + github).

* feat(skill): Step 0.55 category-peer expansion and self-check

Adds Section 2a (category-peer expansion, MANDATORY for product topics)
and the Step 0.55 self-check checkpoint that fires immediately before
the Resolved block displays. Structural mirror of the engine-side
categories.py map: same categories, same peer subs, same priority
order.

The model-side path now:
- Applies category-peer expansion to the WebSearch-resolved subs on
  every product-in-a-known-category run.
- Emits the (+ <category_id> peers) annotation on the Reddit line of
  the Resolved block as the observable contract. Absence on a
  product-in-a-known-category topic is a Step 0.55 regression.
- Runs a self-check before emitting Resolved: "does the resolved list
  include at least 2 peer subs for the matched category? if not,
  widen NOW and do not run the engine yet."

Mirror of the Python map lives inside Step 0.55 as a table for the
model to pattern-match against; extrapolation to unlisted categories
is explicitly allowed. Worked example (the exact failing query)
appears below the table so reviewers can see before/after at a glance.

Both changes land inside the existing Step 0.55 block. No new
top-level section, no new LAW. LAWs 1-6 wording unchanged.

* test: end-to-end regression for GPT Image 2 failure mode

Stubs grounding.web_search to return the OpenAI-only subs that caused
the 2026-04-22 failure, then asserts that auto_resolve widens to
include the image-gen peers and emits the [Resolve] Matched
category=ai_image_generation stderr line. Covers the cap boundary
and the uncategorized-topic no-op path.

Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is
documentation-grade (not parsed by tests) and shows the pre-fix vs
post-fix Resolved block shape so reviewers can evaluate future
categories.py edits against the original bug.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-22 14:31:39 -07:00
Matt Van Horn 952a876536 feat: attribute top comments with u/ and @ handles in evidence lines (#292)
Reddit, TikTok, YouTube, Instagram, Bluesky, X and Threads top comments
now render as u/author or @handle in the evidence block, instead of the
generic "Comment (...)" label. The enrichment adapters already captured
author; only the render layer was dropping it.

Also fixes the TikTok adapter to prefer user.unique_id (the @handle) over
user.nickname (display name) so attribution round-trips to a profile URL.

Legacy "Comment (...)" shape is preserved when author is empty, [deleted],
or [removed].

Bumps to 3.0.10.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 08:23:47 -07:00
Matt Van Horn 1f23e3f980 test: skip docs/ in memory-dir-paths sweep (#291)
The regression test from #290 walks the filesystem via Path.rglob, so
docs/plans/*.md files (gitignored, created by internal planning) trip
the assertion on any dev machine that has run ce:plan in this repo.
Fresh clones and CI never see them, but local runs fail.

Adding docs to skip_dirs keeps the guard narrow to first-class source
files while letting internal planning docs reference old paths
verbatim.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 07:05:35 -07:00
Dave Morin 5269806a75 Make memory directory configurable (#290) 2026-04-21 07:04:38 -07:00
weichao adac4c377a feat: add xurl CLI as alternative X search backend
Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.

xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.

Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window

New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)

Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
  get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
  _search_x() priority chain
- SKILL.md: document xurl setup option
2026-04-21 07:05:20 +00:00
Matt Van Horn 3107325443 feat: inline markdown links on narrative citations (#289)
Inline markdown links on every narrative citation (@handle, r/sub,
publication, YouTube channel, TikTok/Instagram creator, Polymarket
market). Raw URL strings remain forbidden. Plain-text fallback when the
raw data has no URL for a specific source.

Commit 1 (790e5bc) added the citation rule in CITATION PRIORITY / URL
FORMATTING. Live tests showed the rule was deployed but consistently
skipped because it lived at line 1224, below the agent's chunked-read
window. Commit 2 (5864c687) hoists the rule into the VOICE CONTRACT
LAW block as LAW 8, at line 167 - inside the guaranteed-loaded top
band alongside LAWs 1-7. Same pattern that fixed v3.0.6 (invented
titles), disaster #2 (stripped bold), disaster #3 (trailing Sources),
and the 2026-04-19 Hermes evidence-dump disaster.

No Python engine changes. Rule is prompt-only; the deterministic
stats footer (LAW 5) is unchanged.

Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md
2026-04-20 09:49:03 -07:00
Matt Van Horn 1da9c601c3 Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth
fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
2026-04-19 11:09:56 -07:00
Matt Van Horn 4388fed46a fix: rewrite 'no LLM provider' stderr to stop the capability-constraint misread
PR #285 introduced the stderr warning "No --plan and no LLM provider
configured. Using deterministic fallback..." The 2026-04-19 Run 1
agent self-debug said it read that as "I don't have a key, I can't do
LLM stuff, I have to accept fallback" - which is the exact wrong
mental model. The word "provider" referred to the engine's INTERNAL
planner credentials, but the agent parsed it as "I need credentials
to plan at all."

Rewritten to say plainly: YOU are the reasoning model hosting this
skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime);
YOU ARE the planner; you do not need an API key or credentials - you
ARE the LLM. The --plan flag exists precisely so a reasoning model
generates its own plan upstream and passes it to the engine. The
deterministic fallback is the headless/cron path only.

Runtime enumeration is explicit so agents on every supported runtime
recognize themselves - this skill ships to Claude Code, Codex, Hermes,
and ~/.agents via sync.sh.

Tests: updated test_fallback_logs_warning_when_no_provider to assert
the new language (YOU ARE the planner, runtime names present) and
assert the old misleading phrasing is absent. Renamed the companion
test for clarity.
2026-04-19 10:28:48 -07:00
Matt Van Horn a7d6ef051a fix: expand entity-grounding haystack to transcripts + top comments
PR #285's entity grounding checked only title + snippet. That missed:

- YouTube videos where the entity is mentioned in transcript but not
  in title (false demotion of on-topic content)
- Reddit posts where the entity is in top comments but not in title
  (false demotion of on-topic discussion)

And it also wasn't strong enough to reliably demote items like the
2026-04-19 Nate Herk "Managed Agents" video - which had no Hermes
anywhere - because the -25 penalty on rerank_score composed to only
-15 on final_score via the 0.60 weight, and engagement bonus partially
offset that.

Two fixes:

1. _candidate_haystack() now joins title + snippet +
   metadata[transcript_snippet] + metadata[transcript_highlights] +
   metadata[top_comments][*].excerpt/text + metadata[comment_insights].
   Catches entity mentions wherever they actually live. Guarded with
   isinstance checks so malformed metadata doesn't raise.
2. ENTITY_MISS_FINAL_PENALTY (20.0) applied directly in _final_score
   when candidate.explanation contains "entity-miss". This lands the
   full penalty weight on the composite signal that cluster-scoring
   consumes, instead of being diluted by the rerank_score weight.
   Combined effect: entity-miss gap grows from ~15 to ~35 points.

Tests: 8 new scenarios covering transcript match, transcript highlight
match, top-comment match, comment-insight match, empty-text skip,
no-primary-entity no-op, and the dual-penalty composition check.
2026-04-19 10:28:36 -07:00
Matt Van Horn b7df5ecd2d fix: emit user-visible DEGRADED RUN WARNING on bare named-entity calls
The stderr [Planner] warning from PR #285 doesn't reach the user because
Claude and other reasoning agents hide stderr from their synthesis. The
2026-04-19 Hermes Agent Use Cases Run 1 produced source=deterministic
and the user never saw it.

Adds a user-visible stdout block that the model's LAW 5 pass-through
contract forces into the response. Fires only when plan_source is
deterministic AND no pre-research flags were passed AND the topic is
pre-research-eligible (named entity). Cron jobs on abstract topics
don't trigger it.

Position: BEFORE the EVIDENCE FOR SYNTHESIS envelope so the model sees
it as the first non-badge content. Wrapped in a new USER-VISIBLE BANNER
envelope matching the EVIDENCE/PASS-THROUGH envelope pattern from Unit 1
of PR #285.

Runtime-agnostic language: explicitly enumerates Claude Code, Codex,
Hermes, Gemini so the hosting reasoning model recognizes itself
regardless of runtime.

pipeline.py now persists plan_source to report.artifacts so the
renderer can consume it. Adds 7 tests covering fire conditions,
suppression conditions (external/llm plan source, flags present,
abstract topic), and correct position relative to the evidence envelope.
2026-04-19 10:28:21 -07:00
Matt Van Horn a0d61b0dc6 fix: add LAW 7 - YOU ARE the planner, --plan mandatory on named entities
Run 1 of /last30days Hermes Agent use cases on 2026-04-19 called the engine
bare despite SKILL.md already having a detailed Step 0.75 (YOU are the
planner) and a PRECONDITION GATE requiring --plan. Those lived at lines
647 and 729 - the model didn't reach them before invoking Bash.

LAW 7 hoists the rule into the OUTPUT CONTRACT block at the top (same
placement pattern as LAW 6), so it is the first thing the model reads.
Runtime-agnostic language: Claude Code, Codex, Hermes, Gemini, or any
agent runtime. Named failure mode with the misread diagnosis: "provider"
in engine messages refers to the engine's INTERNAL planner credentials,
NOT a prerequisite the caller needs - if you are the hosting reasoning
model, YOU are the provider.

Concrete self-check: re-read pending Bash command; if no --plan and topic
is a named entity, STOP and generate a plan.
2026-04-19 10:28:08 -07:00
Matt Van Horn 5f218aaac5 fix: always log planner subqueries to stderr
The prior pipeline.py only logged the planner outcome when an external
--plan was passed ("[Planner] Using external plan (N subqueries)").
The internal LLM planner and the deterministic fallback ran silently,
so retrieval-breadth failures were invisible without --debug.

After plan finalization, emit a unified trace:

  [Planner] Plan: intent=X, freshness=Y, cluster_mode=Z, subqueries=N, source=external|llm|deterministic
  [Planner]   sq1 label=... search="..." sources=[...]
  [Planner]   sq2 ...

Stderr only; does not touch the user-facing stdout synthesis. The
source= annotation distinguishes --plan (external), provider-backed
(llm), and deterministic paths — so when the 2026-04-19 Hermes Agent
Use Cases failure mode recurs, the trace tells the user which path ran
and what subqueries it produced.

Tests: added test_planner_trace_always_fires_on_mock_run which captures
stderr on a mock pipeline run and asserts the summary + per-subquery
lines appear.
2026-04-19 09:24:52 -07:00
Matt Van Horn a709d66e2a fix: demote reranker candidates that miss the primary entity
The 2026-04-19 Hermes Agent Use Cases run had a Nate Herk YouTube video
titled "I Tested Claude's New Managed Agents" score 51 and rank #2
with zero Hermes content. The reranker had intent-specific scoring hints
but no entity-grounding check, so topic-vicinity matches (one offhand
OpenClaw mention) drifted to the top.

Add _primary_entity(topic) that strips intent-modifier suffixes ("use
cases", "workflows", etc.) so "Hermes Agent use cases" yields
primary_entity="Hermes Agent". Pass the entity through to both the LLM
and fallback scoring paths.

Fallback path: if primary_entity is not found (case-insensitive) in
title + snippet, subtract ENTITY_MISS_PENALTY (25 pts). Skip the
demotion for candidates with no text at all (image-only TikToks etc.)
to avoid false negatives on thin-text sources.

LLM path: add a "Primary entity grounding" hint to _build_prompt when
primary_entity is non-empty. Instructs the LLM to score candidates
without the entity at <=30.

Tests: 24 rerank tests pass, including 8 new entity-grounding tests.
2026-04-19 09:24:43 -07:00
Matt Van Horn 4d9f29d2ed fix: broaden planner retrieval and fix deterministic fallback defaults
Topics with suffixes like "use cases", "workflows", "review",
"examples" were previously echoed near-verbatim into search_query,
returning near-zero matches because nobody posts the literal phrase
(2026-04-19 Hermes Agent Use Cases failure).

Unit 2 — planner breadth:

1. Planner prompt rule: STRIP intent-modifier phrases from search_query
   (keep them in ranking_query). Paraphrase across 4-5 subqueries that
   each express the intent differently.
2. Planner prompt rule: quote only multi-word proper nouns like
   "Hermes Agent", not the user's full topic.
3. Raise _max_subqueries cap from 3 to 5 for how_to / opinion / product /
   breaking_news / prediction. Comparison stays at 4; factual / concept
   stay at 2 unless the topic carries an intent modifier.
4. Deterministic fallback: when intent is non-{comparison,prediction}
   and topic contains an intent modifier, append 3 paraphrased
   subqueries (workflows, production, experience).

Unit 3 — deterministic fallback defaults:

5. _infer_intent default changed from "breaking_news" to "concept".
   Prior default forced strict_recent freshness on unclassified topics,
   biasing against older relevant material. Recency-signal regexes
   ("trending", "this week", etc.) added above the default so genuinely
   time-sensitive topics still classify correctly.
6. _keyword_query now quotes only title-cased multi-word proper nouns
   ("Hermes Agent", "Claude Code"), not the user's full typed topic.
   Hyphenated compounds and lowercase terms are left as bare keywords
   so platform tokenizers broaden rather than narrow retrieval.
7. New stderr warning when plan_query runs with no --plan and no LLM
   provider: surfaces that the deterministic fallback path is weaker
   than the --plan-from-Claude-Code path, so callers know to generate
   and pass a plan.

Tests: 37 planner tests pass, including 11 intent-modifier and 7
fallback-defaults tests.
2026-04-19 09:24:30 -07:00
Matt Van Horn 52fb0e50cb fix: scope pass-through to footer only, add LAW 6 against raw cluster dumps
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.

Split render_compact stdout into two bounded blocks:

- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
  Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
  Emit verbatim per LAW 5.

Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.

Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
2026-04-19 09:23:55 -07:00
Matt Van Horn f635f78e4a Merge pull request #281 from mvanhorn/docs/v3.0.9-release-notes
Release / build-and-release (push) Has been cancelled
docs: v3.0.9 release notes - The Self-Debug Release
2026-04-18 14:12:40 -07:00
Matt Van Horn a070a584a4 docs: v3.0.9 release notes - The Self-Debug Release
Adds docs/releases/v3.0.9.md as the GitHub Release body and appends
the matching CHANGELOG.md entry.

Covers what shipped in v3.0.9 (Class 1 refuse-gate, LAW 1 WebSearch
precedence, END-boundary, stale SKILL.md deletion) plus the community
contributions that landed across 3.0.1-3.0.8 that had never been
announced (TikTok + YouTube top comments, Hermes support, multi-key
rotation, cross-platform fixes, HTTP layer consolidation, eval
fixtures).

Contributors credited: @j-sperling, @stephenmcconnachie, @zaydiscold,
@iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma,
@BryanTegomoh, @uppinote20, @zerone0x, @thinkun, @thomasmktong,
@fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli,
@Cody-Coyote.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:12:20 -07:00
Matt Van Horn 8e18d0142c Merge pull request #280 from mvanhorn/fix/v3.0.9-engine-refuse-stale-skillmd
fix: v3.0.9 - engine refuses Class 1 keyword traps, delete stale SKILL.md files, reinforce LAW 1 over WebSearch
2026-04-18 13:40:37 -07:00
Matt Van Horn e8105df4fd fix: v3.0.9 - engine refuses Class 1 keyword traps, delete stale SKILL.md files, reinforce LAW 1 over WebSearch
Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:

1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
   Birthday-gift failure mode becomes structurally impossible - the pipeline
   never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
   pointing the model to ask for hobbies/relationship/budget. Escape hatch:
   LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.

2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
   snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
   Peter Steinberger's self-debug named the first file as the one it read
   instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
   Sync script simplified: Hermes now always uses main SKILL.md.

3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
   with pass-through instruction. The model had the canonical body in its
   buffer on the Peter run and discarded it; the boundary makes pass-through
   the path of least resistance.

4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
   tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
   section") that caused Peter's trailing Sources leak. No more ambiguity
   at synthesis time.

Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.

Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.

Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 13:30:33 -07:00
Matt Van Horn 361e9d6c13 fix: v3.0.8 - SKILL.md was too big and LAWs too deep - move to top + engine emits badge (#279)
Three independent Opus 4.7 self-debugs on 2026-04-18 converged on the same
root cause of the v3.0.6/v3.0.7 canonical-compliance regression: SKILL.md is
42,860 tokens / 1,478 lines, LAWs lived at line 1094+, every realistic reading
strategy failed to reach them before synthesis.

Unit 1 - Moved the BADGE MANDATORY block and VOICE CONTRACT LAW 1-5 (plus
the formatting-authority preface) from line ~1090 to line ~75 (right after
the SKILL CONTRACT preface, before HOW TO INVOKE THIS SKILL). Every reading
strategy now lands the LAWs in active context before synthesis.

Unit 2 - Engine now emits the badge as the first line of --emit=compact
stdout. Passing through the script output becomes the default-correct
behavior; emitting the badge no longer depends on model compliance. Reads
version from .claude-plugin/plugin.json at runtime with graceful fallback.

Unit 3 - Deleted skills/last30days/SKILL.md stub (231-line v3-spec file).
This was the wrong-file-capture hazard Ron Conway's self-debug identified:
model grabbed the first SKILL.md find surfaced and treated it as
authoritative. Only ONE SKILL.md in the plugin package now.

Diagnoses verbatim:
- Kanye thread: "I read lines 1-600 in chunks, jumped to 300-899, then
  stopped. File is 1478 lines. I never saw past ~900."
- Peter thread: "I tried Read once, hit the 25K token cap on a 42,860-token
  file, and bailed instead of chunked-reading with offset/limit. I never
  opened SKILL.md at all."
- Ron Conway thread: "I read one SKILL.md (231 lines)... the v3 spec stub.
  I never opened the operational SKILL.md sitting next to the script."

Validation: direct engine invocation confirms badge at line 1 of compact
output. Module imports clean.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 12:50:10 -07:00
Matt Van Horn 58845df312 fix: v3.0.7 - restore mandatory first-line badge + pin SKILL_ROOT + add skill-specificity anchor (#278)
Hot-fix for the public v3.0.6 0/8 regression (2026-04-18). Beta went 10/10
yesterday with the same LAW content; public went 0/8 today. The delta was
three structural anchors the port had removed or weakened.

Unit 1 - Restored MANDATORY first-line badge. Every public response now
emits "🌐 last30days v{VERSION} · synced {YYYY-MM-DD}" as line 1, blank
line, then "What I learned:" (GENERAL) or "# {TOPIC_A} vs {TOPIC_B}..."
(COMPARISON). This is the LAW 2 / LAW 4 enforcement anchor that my v3.0.6
port accidentally stripped along with the beta-specific "🧪 last30days-beta"
wording.

Unit 2 - Pinned SKILL_ROOT to the public plugin cache via
`ls -d ~/.claude/plugins/cache/last30days-skill/last30days/*/ | sort -V |
tail -1`, with a small fallback for repo/Gemini/Codex hosts. Replaces the
path-discovery loop that was landing on stale copies (~/.openclaw/,
~/.agents/, ~/.codex/) on machines with a private-repo sync history.

Unit 3 - Added a "SKILL CONTRACT" preface at the top of SKILL.md that names
the 0/8 regression as a documented failure mode and explicitly tells the
model not to treat /last30days as a generic keyword. Encodes user theory
that "/last30days-beta" sounded specific enough to trigger skill-follow
mode while "/last30days" reads as a search term and triggers improvise
mode.

Validation: all three anchors visible in the grep check for public
v3.0.7 cache. Next validation is manual re-run of the 8 failure topics
on public after shipping.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 10:55:28 -07:00
Matt Van Horn d14814a9b0 feat: release v3.0.6 - promote plans 003-009 from private beta to public (#277)
Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.

Plans bundled in this release:

- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
  + VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
  + top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
  count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
  per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
  emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
  at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
  ranking rewrite + Polymarket post-merge topic filter (engine change,
  filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
  preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
  post-synthesis Sources-block self-check

Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).

No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 10:24:14 -07:00
Matt Van Horn e9911ae2ae Merge pull request #276 from mvanhorn/feat/beta-channel-wiring
feat: wire compare.sh and CLAUDE.md for /last30days-beta channel
2026-04-17 22:44:31 -04:00
Matt Van Horn 7cee41509f feat: wire compare.sh and CLAUDE.md for /last30days-beta channel
- scripts/compare.sh now runs /last30days vs /last30days-beta (was
  /last30days vs /last30days-3:last30days-skill-private which was a stale
  private install name that no longer works)
- CLAUDE.md adds a Beta channel section pointing at mvanhorn/last30days-skill-private
  so future agent sessions discover the two-skill layout on project load

No runtime impact on /last30days. Engine code unchanged.

Plan: docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md
(plan file is gitignored per PR #259, not included in this diff)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 22:43:29 -04:00
Matt Van Horn 371f62a403 Revert "feat: make default fun level actually surface comedy (#272)" (#273)
This reverts commit bad1d312ef.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-17 08:39:56 -04:00
Matt Van Horn bad1d312ef feat: make default fun level actually surface comedy (#272)
Most users never touch FUN_LEVEL. Default medium was shipping a stats
block but rarely a Best Takes block, and when it did it was below the
cluster fold where a synthesizing model had already stopped reading.
A 2,304-upvote Reddit comment ("WHAT?! I reached my monthly limit
just reading this post") on the 2026-04-17 Opus 4.7 run sat inside
cluster 11 and never made it into synthesis. Four coordinated changes:

1. render: promote Best Takes above the cluster list so the synthesizer
   sees comedy before it anchors on cluster 1.
2. render: lower medium threshold from 70 to 55 (heuristic maxes at 80),
   drop the two-gem floor to one-gem. Default now reliably emits the
   block on typical runs.
3. rerank: score individual top_comments by upvote ratio to their parent
   thread. A 2,304-upvote comment on a 300-upvote thread now outranks a
   400-upvote comment on a 3,400-upvote thread, which is the viral-wit
   signal. Handles both the LLM scoring path and the heuristic fallback.
4. render: merge scored comment gems into Best Takes alongside candidate
   gems, sorted together. Comment lines show body + parent title +
   r/subreddit or @handle + absolute upvotes.
5. SKILL: tell the synthesizer to quote at least two Best Takes entries
   verbatim, with an example of the new comment format.

Plan: docs/plans/2026-04-17-001-feat-default-fun-surfacing-plan.md

🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 08:30:39 -04:00
Matt Van Horn 0103324701 Merge pull request #268 from zaydiscold/feat/multi-key-rotation
feat: multi-key rotation for SCRAPECREATORS_API_KEY
2026-04-16 23:48:44 -04:00
zayd f09c6850bc feat: multi-key rotation for SCRAPECREATORS_API_KEY
Support comma-separated API keys in SCRAPECREATORS_API_KEY with random
selection per run, distributing load across multiple free-tier accounts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 17:37:25 -07:00
Matt Van Horn 3499c246b8 fix: add commands/last30days.md and remove skills/last30days-nux duplicate (#267)
Release / build-and-release (push) Has been cancelled
Adds commands/last30days.md so /last30days registers as a Claude Code
slash command for plugin users. Users type /last30days and autocomplete
prefix-matches to the canonical /last30days:last30days form (same as
/ce:plan resolving to /compound-engineering:ce-plan).

Removes skills/last30days-nux/, a byte-identical duplicate of the root
SKILL.md that created confusing /last30days:last30days-nux autocomplete
entries via Claude Code's plugin namespacing. Root SKILL.md remains
the canonical skill source; natural-language skill-selector invocation
is unchanged.

Recovery for users on v3.0.4: /plugin update last30days then /reload-plugins.

Closes #239 (path-escape error was already fixed in v3.0.4 by dropping
the rogue 'skills' key; v3.0.5 adds the slash command on top).
Supersedes #257 (suggested './' -> '.' workaround is obsolete since
v3.0.4 dropped the 'skills' key entirely, matching ecosystem standard).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 15:19:42 -04:00
Matt Van Horn 53b8e33d13 fix(youtube): use url= param for ScrapeCreators comments/transcript + parse new response shape (#265)
PR #260 wired YouTube comment enrichment against
`/v1/youtube/video/comments` with `id=<video_id>`, but the endpoint
requires `url=https://www.youtube.com/watch?v=<video_id>`. Every enrich
call was returning 400 "missing_parameter: you must provide a url", so
no YouTube items ever carried `top_comments`.

The SC transcript fallback (`_sc_fetch_transcript`) had the identical
contract mistake. It was latent because `_fetch_transcript` prefers
yt-dlp and the SC path only fires when yt-dlp is missing, but it would
have failed the same way on hosts without yt-dlp installed.

Switching both callers to `url=` surfaces a second issue in the
response parser: SC returns `author` as `{"name": "@handle", ...}` and
nests like counts under `engagement.likes`, not top-level. The parser
was reading `author` as a string and missing the nested likes, so even
after the param fix every comment would land with an object-shaped
author and 0 likes.

- `_fetch_video_comments`: send `url=` on both urllib and requests branches
- `_sc_fetch_transcript`: same
- Response parser: extract `author.name` when author is a dict, read
  `engagement.likes` when top-level `likes` is absent, prefer
  `publishedTime` / `publishedTimeText` for date. Legacy string-author
  and top-level-likes shapes still work, so existing mocks are unchanged.

Verified live against api.scrapecreators.com: `_fetch_video_comments`
now returns fully-populated comments with real @handles and like
counts (e.g. "@JennyNicholson: ... (49000 likes, 2025-04-15)"). All
tests in youtube_yt/normalize/signals/render pass.

Plan: docs/plans/2026-04-15-002-fix-youtube-comments-scrapecreators-param-plan.md

🤖 Generated with Claude Opus 4.6 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:17:22 -04:00
Matt Van Horn 73b4bd6ac6 fix: enforce pre-research protocol + override WebSearch Sources mandate (#266)
Restore the rich synthesis output by closing three prompt-level loopholes
that let the model silently take a degraded path:

1. Research Execution precondition gate. Steps 0.55 (entity resolution)
   and 0.75 (query planner) are now non-skippable on WebSearch platforms.
   --emit md is banned as a primary user-facing flow; --emit=compact with
   --plan is mandatory. OpenClaw --auto-resolve fallback preserved.

2. WebSearch "Sources:" mandate override. The WebSearch tool description
   contains a CRITICAL/MUST mandate to append a Sources section. That is
   explicitly superseded inside /last30days with matched-register
   CRITICAL/MANDATORY override language and a BAD/GOOD example. The
   existing web-source line is the citation; nothing appends below the
   invitation.

3. Pre-present self-check. Before displaying, the model verifies bold
   per-paragraph headlines, per-source emoji stats, quoted highlights,
   Polymarket block, coverage footer, and (critically) no trailing
   Sources block. One regeneration permitted if checks fail.

Also adds explicit MANDATORY language to the "What I learned" template
requiring bold headline phrases on every narrative paragraph.

Root cause: same-session A/B on 2026-04-15 between /last30days kanye
west (rich output, ran Steps 0.55 + 0.75, --emit=compact --plan) and
/last30days hermes ai (bland output, skipped both, --emit md) showed
the template was fine -- the model was lazily taking a shortcut SKILL.md
tolerated. No engine, render.py, or contributor PR was the cause.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:15:29 -04:00
Matt Van Horn a2850e3d19 fix: drop plugin.json 'skills' key to clear path-escape error on v2.1.109 (#264)
Release / build-and-release (push) Has been cancelled
plugin.json has declared "skills": ["./"] unchanged since v2.1.0. That
value used to work on older Claude Code but current versions reject it
with: Path escapes plugin directory: ./ (skills). The error surfaces
on fresh /doctor runs even after v3.0.3 restored the archive contents.

Fix: omit the "skills" key entirely. Every other plugin in the Claude
Code marketplace ecosystem (compound-engineering, coding-tutor, codex,
esper, 15+ Anthropic official plugins) omits this key and the loader
auto-discovers skills/*/SKILL.md. Matching that pattern clears the
path-escape error on v2.1.109+ and remains compatible with older
Claude Code versions where the default-discovery path was already the
working code path.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 11:40:15 -04:00
Matt Van Horn 9c1e253dcc fix(build): strip skills/ and .claude-plugin/ from .skill bundle (#263)
v3.0.3's fix (#262) restored skills/ and .claude-plugin/ to the git
archive, which Claude Code needs for /plugin install. But
scripts/build-skill.sh uses the same archive to produce the claude.ai
.skill bundle, which must contain exactly one root SKILL.md and stay
under the 200-file cap.

Fix: after git archive, 'zip -d' strips both directories from the
.skill bundle. git archive output is unchanged (Claude Code still
gets the full tarball on /plugin install).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:31:51 -04:00
Matt Van Horn f4a3cc104b fix: restore skills/ and .claude-plugin/ in plugin install tarball (#262)
Release / build-and-release (push) Has been cancelled
v3.0.1 added .gitattributes rules that excluded both directories from
git archive output, shrinking the claude.ai .skill bundle. But Claude
Code's /plugin install fetches the SAME archive, so users installing
v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no
skill files. Install appeared successful but the plugin was a useless
empty shell.

Proof:
  git archive v3.0.0 | grep 'skills/|\.claude-plugin/' | wc -l  # 8
  git archive v3.0.1 | grep 'skills/|\.claude-plugin/' | wc -l  # 0
  git archive v3.0.2 | grep 'skills/|\.claude-plugin/' | wc -l  # 0

No issue reports yet because:
 - Cached pre-v3.0.1 installs keep working (it's the new-install path
   that's broken)
 - The breakage is under 24 hours old
 - Users invoking the skill via natural language go through
   skill-selector rather than /last30days slash command

Also reverts v3.0.2's "skills": ["skills"] back to "./", the value
that shipped in every tag from v2.1.0 through v3.0.0. That change was
a misdiagnosis; the manifest wasn't in the tarball anyway so it had
no effect on user-visible installs.

Archive file count after fix: 97 (cap is 200, plenty of room).
Follow-up: move claude.ai-specific bundle exclusions into
scripts/build-skill.sh where they belong, rather than .gitattributes
which cannot distinguish between the two distribution channels.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:25:45 -04:00
Matt Van Horn a220632186 fix: restore /last30days slash command on Claude Code v2.1.105+ (#261)
Release / build-and-release (push) Has been cancelled
Two regressions were silently breaking /last30days for every user:

1. plugin.json declared "skills": ["./"], which newer Claude Code
   rejects with "Path escapes plugin directory: ./ (skills)". The
   skill loader refused to register the command, so /last30days
   returned "Unknown command" even though /plugin list showed the
   plugin as installed. Fix: "skills": ["skills"] so the loader
   scans the real subdirectory.

2. marketplace.json pinned "version": "3.0.0" while plugin.json
   advertised "3.0.1". The /plugin resolver used the marketplace
   version and could install a phantom user-scope copy at a stale
   SHA alongside the correct project-scope install, creating
   duplicate skill-name collisions. Both manifests now agree on
   3.0.2.

Prior attempt: commit 93fbed2 fixed (1) before but got reverted.
This lands both fixes together in a tagged release so users can
/plugin update to recover.

Recovery for affected users is in CHANGELOG.md under 3.0.2.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:40:09 -04:00
Matt Van Horn 082efe03e3 feat: surface YouTube + TikTok top comments alongside Reddit (#260)
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:26:06 -04:00
Matt Van Horn 242e38ef56 chore: ignore docs/plans/ and untrack existing plan files (#259)
Internal ce:plan output shouldn't ship on the public repo.
Adds docs/plans/ to .gitignore and removes the two already-tracked
plan files from the index. Working copies stay local for reference.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 07:48:47 -04:00
Matt Van Horn c12dd3adbf docs: mark plan 002 Units 1-4 complete; 5-10 remain 2026-04-14 17:46:01 -04:00
Matt Van Horn c5b03adffc Merge pull request #244 from mvanhorn/feat/claudeai-distribution
Release / build-and-release (push) Has been cancelled
feat: claude.ai distribution push (Units 1-4 of plan 002)
2026-04-14 17:44:58 -04:00
Matt Van Horn 38a1c27e2e chore: exclude .github/ from skill archive (CI workflows, not runtime) 2026-04-14 17:44:18 -04:00
Matt Van Horn ed80797564 docs: add plan 2026-04-14-002 for claude.ai distribution push 2026-04-14 17:44:00 -04:00
Matt Van Horn 68c3420f9f docs: promote claude.ai to first-class install path with direct download link
- install matrix now leads with claude.ai (widest audience, one-click path)
- direct download link to GitHub release's 'latest' asset URL
- 3-step UI walkthrough with link to Settings > Capabilities > Skills
- Claude Code / OpenClaw / Gemini / manual paths still documented, collapsed
- removes the bash scripts/build-skill.sh requirement from end-user flow
2026-04-14 17:43:32 -04:00
Matt Van Horn 12167ee19e feat(skill): tune description and argument-hint for Claude skill-selector quality
- description leads with imperative 'Research' + 'what people actually say' (strong trigger signal for community/social-research prompts)
- argument-hint shows 3 concrete user phrasings instead of marketing copy
- 176 chars, well under Anthropic's 200-char cap
- preserves all source coverage (Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, web)

Per ecosystem research (April 2026), trigger description quality is the single
biggest lever separating 500-install skills from 350k-install skills.
2026-04-14 17:42:54 -04:00
Matt Van Horn 21b8e5c6d3 ci: auto-build .skill artifact on tag push and attach to GitHub release 2026-04-14 17:42:15 -04:00
Matt Van Horn 1157ea8afe docs: mark plan 2026-04-14-001 as completed 2026-04-14 12:24:16 -04:00
Matt Van Horn 9f3be8bbda Merge pull request #242 from mvanhorn/fix/skill-upload-200-file-limit
fix: skill upload 200-file cap + packaging hygiene (3.0.1)
2026-04-14 12:24:03 -04:00
Matt Van Horn beb54e9e9d fix: sync version references in SKILL.md body and sync.sh cache path 2026-04-14 12:22:51 -04:00
Matt Van Horn 8d8ca68781 chore: bump version to 3.0.1 + changelog entry
Atomic bump across all four manifests:
- SKILL.md (root)
- skills/last30days/SKILL.md (internal spec)
- .claude-plugin/plugin.json
- gemini-extension.json

CHANGELOG entry documents the skill-upload packaging fix, vendor/ removal,
legacy plans/ removal, and the new scripts/build-skill.sh builder.
2026-04-14 12:21:24 -04:00
Matt Van Horn 0949b870e0 fix(skill): trim description to 167 chars (was 228, Anthropic caps at 200) 2026-04-14 12:20:20 -04:00
Matt Van Horn 4b07ba02a6 docs: document .skill upload path via scripts/build-skill.sh 2026-04-14 12:19:49 -04:00
Matt Van Horn 039fc89874 feat: add scripts/build-skill.sh to produce claude.ai-upload-ready .skill
Wraps git archive with --prefix=last30days/ so the zip contains a single
top-level skill folder matching SKILL.md's name: frontmatter. Enforces:

- refuses to build with a dirty working tree (prevents shipping untracked changes)
- fails if zip exceeds 200 files (claude.ai's empirical upload cap)
- fails if zip contains more than one SKILL.md (avoids name: confusion)

Output at dist/last30days.skill (gitignored).
2026-04-14 12:19:28 -04:00
Matt Van Horn 2b506e90f5 chore: add .gitattributes to exclude non-runtime files from git archive 2026-04-14 12:18:54 -04:00
Matt Van Horn deb9f33437 chore: remove legacy plans/ directory (superseded by docs/plans/)
Both plans describe work that was already shipped:
- feat-add-websearch-source.md - websearch is in the v3 pipeline (scripts/lib/perplexity.py etc)
- fix-strict-date-filtering.md - date filtering is enforced in scripts/lib/dates.py

New planning goes in docs/plans/ following the ce:plan convention.
2026-04-14 12:18:19 -04:00
Matt Van Horn cb88bd2eed chore: remove unused root vendor/ directory (215 files from PR #48)
Root vendor/package/ was an accidentally committed extracted npm tarball
(steipete-bird-0.8.0). Zero importers: the real vendored X client lives
at scripts/lib/vendor/bird-search/, referenced by scripts/lib/bird_x.py
and tests/test_bird_x.py.

Removes 215 files + 1 .tgz, dropping repo from 406 to 191 files and
clearing the claude.ai skill-upload 200-file cap.

Adds /vendor/ to .gitignore (leading slash so scripts/lib/vendor/ is unaffected).
2026-04-14 12:17:58 -04:00
hnshah 23fc6c7061 fix(env): default INCLUDE_SOURCES to empty string (#223)
* fix(env): default INCLUDE_SOURCES to empty string

* test(env): patch resolved config path in include sources test
2026-04-14 07:48:53 -04:00
Ilia Alshanetsky 9dd3f21476 refactor: consolidate _sc_headers into http.scrapecreators_headers (#209)
Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.

Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.

Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
2026-04-14 07:43:56 -04:00
Matt Van Horn e395c1d57f Merge pull request #208 from iliaal/fix/date-parsing
fix(github): reject garbage in _parse_date; consolidate date parsing
2026-04-13 22:21:35 -04:00
Matt Van Horn 33502d2a07 Merge pull request #207 from iliaal/refactor/reddit-http-helper
refactor(reddit): migrate to http.get(params=...) helper
2026-04-13 22:18:49 -04:00
Matt Van Horn bdc71cfd07 Merge pull request #227 from Chelebii/fix/windows-bird-x-runtime
fix(windows): stabilize bundled Bird X search
2026-04-13 22:15:21 -04:00
Matt Van Horn 65be6196c1 Merge pull request #217 from Gujiassh/fix/sync-version-consistency
fix: align v3 skill version metadata and sync target
2026-04-13 17:55:34 -04:00
Matt Van Horn 7dc530b4c9 Merge pull request #224 from hnshah/hnshah-gemini-install-doc
docs: add Gemini CLI install note and workaround
2026-04-13 17:55:24 -04:00
Matt Van Horn b159f8b1ff Merge pull request #216 from george231224/fix/check-perms-stat-linux
fix: use GNU stat first in check_perms (Linux false-warn)
2026-04-13 17:55:21 -04:00
Matt Van Horn cff005b038 Merge pull request #225 from Gujiassh/fix/save-output-utf8
fix(cli): Write saved output using UTF-8 encoding
2026-04-13 17:55:18 -04:00
Matt Van Horn 460565c107 Merge pull request #228 from stephenmcconnachie/add-hermes-support
feat: add Hermes AI Agent support
2026-04-13 15:59:53 -04:00
Matt Van Horn e6493033b0 Merge pull request #229 from shalomma/fix/skill-md-version-bump
Bump SKILL.md version header from v2.9.5 to v3.0.0
2026-04-13 15:55:15 -04:00
Matt Van Horn ca00cacf83 Merge pull request #230 from BryanTegomoh/fix/days-alias-backcompat
fix(cli): restore --days alias compatibility
2026-04-13 15:55:06 -04:00
Matt Van Horn b982ed5b30 Merge pull request #232 from j-sperling/j-sperling/chore/gitignore-dev-artifacts
chore: gitignore dev artifacts (.venv, .coverage, htmlcov, .memsearch)
2026-04-13 15:54:17 -04:00
Matt Van Horn a9d13d695a Merge pull request #233 from j-sperling/j-sperling/feat/eval-topics-fixture
feat: add eval_topics.json fixture for offline quality evaluation
2026-04-13 15:53:58 -04:00
Matt Van Horn 877706da4d Merge pull request #234 from j-sperling/j-sperling/fix/bird-x-engagement-validation
fix(bird_x): skip all-None engagement dicts
2026-04-13 15:52:44 -04:00
Jeffrey Sperling 1a6d8d07d0 fix(bird_x): skip all-None engagement dicts
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}.  This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available."  Return None
instead when every engagement field is missing.
2026-04-13 11:54:49 -07:00
Jeffrey Sperling 3bc12cdc57 feat: add eval_topics.json fixture for offline quality evaluation
evaluate_search_quality.py and e2e_comparison.py both reference
fixtures/eval_topics.json with hardcoded fallbacks.  Supply the
actual fixture: 8 topics spanning all intent types, selected via
MMR dispersion across domains (tech, health, sports, finance,
consumer products).
2026-04-13 11:52:55 -07:00
Jeffrey Sperling ad59e60269 chore: gitignore dev artifacts (.venv, .coverage, htmlcov, .memsearch)
pyproject.toml declares pytest-cov as a dev dependency and configures
[tool.coverage.run], but the generated .coverage database and htmlcov/
report directory are not gitignored.  Also add .venv/ (standard Python
virtualenv) and .memsearch/ (session memory) to keep the working tree
clean for contributors.
2026-04-13 11:52:12 -07:00
Bryan Tegomoh 9d037786f2 fix(cli): restore --days alias compatibility 2026-04-13 09:18:18 -05:00
shalomma 8b67378964 Bump SKILL.md version header from v2.9.5 to v3.0.0
The SKILL.md prompt header still said v2.9.5 while pyproject.toml
and the rest of the codebase are on v3.0.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:31:43 +03:00
Stephen McConnachie 2b015b64ab Add Hermes AI Agent support 2026-04-12 20:06:02 +01:00
Chelebii d3972a6523 fix(windows): stabilize bundled Bird X search 2026-04-11 23:30:39 +01:00
gujishh 56cabf33c6 fix(cli): write saved output using UTF-8 encoding 2026-04-12 06:25:38 +09:00
Hiten Shah 13dcea781d docs: add Gemini CLI install note and workaround 2026-04-11 13:15:53 -07:00
Matt Van Horn 01812ec185 fix(sync): skip OpenClaw variant branch when variants/open is absent (#222)
Makes the `variants/open/` sync steps in `scripts/sync.sh` conditional on
the directory actually existing in the source tree. The script is shared
between the public and private repos of last30days-skill, but the OpenClaw
variant only lives in the private repo (it's sanitized via
`strip_for_openclaw.py` and published separately to ClawhHub). When the
script runs from a checkout of the public repo, the variants/open paths
don't exist and the unconditional `rsync` and `sync_target` calls error
out immediately.

Changes:

- `sync_target()` now only creates `variants/open/references` and rsyncs
  `variants/open/` when `$SRC/variants/open` exists.
- The trailing `sync_target "$OPENCLAW_TARGET" ...` call is now gated by
  the same check, with an explanatory skip message when the directory is
  absent.

No behavior change when running from the private repo (which has
`variants/open/`). When running from the public repo, the script now
completes its COMMON_TARGETS loop without erroring.

This also closes out the confusion from PR #211, where a contributor saw
the broken `variants/open/` reference and tried to add the variant back
to the public repo. The real fix was making the script tolerate the
absence, not recreating the directory.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 11:37:27 -04:00
Matt Van Horn 86b2b9dd69 docs(v3): drop redundant What's New list and remove stale @steipete credit (#221)
release-notes.md:
- Drop the "What's New" section entirely. It repeated the same items
  as the Headline features section above it in bulleted form, a
  holdover from the old v2.9 release notes pattern. CHANGELOG.md is
  the canonical Added/Changed/Fixed list; release notes is marketing
  copy and shouldn't duplicate it. Added a one-line pointer to
  CHANGELOG.md [3.0.0] for anyone looking for the detail.
- Rename "Credits" to "Earlier contributors" and note they are from
  the v1 and v2 lineage, so readers don't confuse them with v3
  contributors.
- Remove @steipete credit (did not actually contribute to this repo).

CHANGELOG.md [2.1.0] Credits:
- Remove @steipete credit (did not actually contribute to this repo).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:33:43 -04:00
Matt Van Horn 6e7c0ba7aa docs(v3): prep CHANGELOG and release notes for v3.0.0 (#220)
Rewrites release-notes.md from its stale v2.9.1 focus into the v3
story: intelligent pre-research as the killer feature, fun judge /
Best Takes, cross-source cluster merging, single-pass comparisons,
GitHub person-mode and project-mode, 13+ sources, ELI5 mode. Credits
@j-sperling as the v3 engine architect in the hero section and
updates the install instructions from `git clone` to the real install
paths for Claude Code, OpenClaw, and OpenAI Codex CLI.

Also extends the CHANGELOG [3.0.0] entry with a Fixed section covering
the two post-merge prep fixes that landed just before release:

- #214 resolved a duplicate `name: last30days` collision in
  skills/last30days/SKILL.md that caused strict marketplace validators
  to reject the plugin (reported by @Cody-Coyote in #204)
- #219 added the real Codex CLI integration at
  .agents/skills/last30days/SKILL.md (regular file, since Codex's
  loader skips symlinked files) plus .codex-plugin/plugin.json as the
  namespace marker (inspired by @Jah-yee in #153 and @dannyshmueli
  on X)

Bumps the [3.0.0] date from `2026-04` to `2026-04-11` to match the
actual release date, and adds @Cody-Coyote and @Jah-yee to the
[3.0.0] Contributors list.

No code changes. Pure docs prep for the v3.0.0 GitHub release.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:27:35 -04:00
Matt Van Horn 71e0492840 feat: make skill discoverable by OpenAI Codex CLI (#219)
Adds a Codex CLI skill integration by creating the two files Codex's real
loader actually reads:

- .agents/skills/last30days/SKILL.md (real file, not a symlink - Codex's
  loader skips symlinked files per codex-rs/core-skills/src/loader.rs)
- .codex-plugin/plugin.json with {"name": "last30days"} as a namespace
  marker, per codex-rs/utils/plugins/src/plugin_namespace.rs

When Codex CLI runs in a checkout of this repo, it walks .agents/skills/
from CWD up to the project root, picks up .agents/skills/last30days/SKILL.md,
and walks ancestors looking for .codex-plugin/plugin.json to resolve the
plugin namespace. The skill registers as last30days:last30days.

The SKILL.md is a verbatim copy of the root SKILL.md at this point to
avoid content drift during the rollout. A future PR can slim the Codex copy
or introduce a sync mechanism.

Verified against Codex CLI's own source by running codex exec from the
repo CWD and having it trace the loader logic.

Replaces PR #153, which used a fake $schema URL
(https://openai.com/codex/plugin.schema.json returns 404) and put a
misunderstanding of Codex's plugin manifest (Codex only reads the `name`
field - all other fields like version, description, author, skills[] are
silently ignored).

This contribution was developed with AI assistance (Codex + Claude Code).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:02:28 -04:00
gujishh 8b2cf41f13 fix: align v3 version metadata and sync target 2026-04-11 21:00:04 +09:00
george231224 3d57db9644 fix: use GNU stat first in check_perms so Linux doesn't false-warn
`stat -f '%Lp'` is BSD/macOS syntax. On Linux, `stat -f` prints
filesystem info (Block size / Inodes / ...) and still exits 0, so the
`||` fallback to `stat -c '%a'` never fires. That left `$perms` as
multi-line garbage, the `!= "600"` check was always true, and every
Linux SessionStart hook invocation printed a bogus warning plus the
whole `stat -f` filesystem dump.

Reorder to try GNU stat first, fall back to BSD for macOS. Verified on
Linux (cpython-3.12 / bash 5.x) — hook now emits the expected compact
Ready banner with no false warning.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 18:41:08 +08:00
Matt Van Horn 99b167d03a fix: resolve duplicate skill name causing marketplace validation failure (#204) (#214)
Two SKILL.md files declared `name: last30days` with `user-invocable: true`,
which caused strict marketplace validators to reject the plugin with "Some
plugins in this marketplace have validation errors":

- ./SKILL.md (canonical, also reachable via skills/last30days-nux/ symlink)
- ./skills/last30days/SKILL.md (v3 architecture spec, real file)

In v2.9.6, skills/last30days/SKILL.md was a symlink to ../../SKILL.md so
only one skill existed. Commit 0a9ff16 (v3.0.0) added a new real file at
skills/last30days-v3/SKILL.md, and commit 9be0780 then renamed that
directory to skills/last30days/, replacing the original symlink with a
different real file. The collision has been live since v3.0.0 shipped.

This change:
- Renames skills/last30days/SKILL.md to name: last30days-v3-spec and sets
  user-invocable: false. The file stays in place as internal architecture
  documentation, but it no longer competes with the canonical skill.
- Fixes README.md link that pointed to the deleted skills/last30days-v3/
  path (left over from the rename).
- Removes a stale variants/open/SKILL.md reference (variants/open was
  deleted in v3.0.0).

After the change, only one canonical name=last30days user-invocable=true
skill exists (the root SKILL.md, also reachable via the
skills/last30days-nux/ symlink, same inode).

Closes #204.

This contribution was developed with AI assistance (Codex).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 02:36:55 -04:00
Ilia Alshanetsky 65fcf6be65 fix(github): reject garbage in _parse_date; consolidate date parsing
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.

Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.

Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).

Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
2026-04-10 07:39:07 -04:00
Ilia Alshanetsky 9ef9d38b90 refactor(reddit): migrate to http.get(params=...) helper
Added params kwarg to http.request()/http.get() that urlencodes a dict
into the query string. None values are dropped, ints and bools are
stringified, and params append correctly if the URL already has a
query string.

Migrated reddit.py to use this helper for all three ScrapeCreators
call sites (global search, subreddit search, post comments). Deleted
the try/import requests/except ImportError fallback and the paired
if not _requests: / else: branches. Six new http tests cover the
params-encoding behavior.

Net: reddit.py -70 lines. Behavior is identical - the existing http.py
urllib implementation already had retry logic, 429 handling, and
HTTPError types that are strictly better than the ad-hoc requests
branches we deleted.

99 reddit tests pass. Live smoke test on a real ScrapeCreators run
returned 12 threads with the same engagement data as before.
2026-04-10 07:25:26 -04:00
Matt Van Horn 86e1d77ad7 Merge pull request #186 from pejmanjohn/contrib/mvanhorn-last30days-skill-99-fix-plugin-directory-name
fix: rename skills/last30days-v3 directory so plugin install resolves correctly
2026-04-09 23:18:18 -07:00
Matt Van Horn ac692e85bc Merge pull request #201 from tmchow/fix/90-store-sql-column-whitelist
fix(store): validate updatable columns in update_run and update_finding
2026-04-09 23:15:38 -07:00
Matt Van Horn 5196bf68a7 Merge pull request #202 from tmchow/fix/92-bluesky-token-refresh
fix(bluesky): add token expiry handling to session cache
2026-04-09 23:15:36 -07:00
Matt Van Horn 6bbb4300f5 Merge pull request #203 from tmchow/fix/marketplace-plugin-name
fix: correct plugin name and version in marketplace.json
2026-04-09 23:15:33 -07:00
Trevin Chow 62584631bb fix: correct plugin name and version in marketplace.json
marketplace.json had stale v3 rename artifacts: plugin name was
"last30days-3" (should be "last30days" to match plugin.json) and
version was "3.0.0-alpha" (should be "3.0.0" to match the stable
release).
2026-04-09 22:05:25 -07:00
Trevin Chow 43c8d6c29c fix(bluesky): add token expiry handling to session cache
The module-level _cached_token was set once and never refreshed. AT
Protocol tokens expire after ~2 hours, causing silent 401 errors in
long-running watchlist cron sessions. Adds a 90-minute expiry check
using time.monotonic() and logs re-authentication.

Fixes #92
2026-04-09 21:44:33 -07:00
Trevin Chow ccd2a4065d fix(store): validate updatable columns in update_run and update_finding
Add column whitelists to prevent SQL injection via kwargs keys in
dynamic UPDATE queries. Values were already parameterized but column
names were string-interpolated directly from kwargs.

Fixes #90
2026-04-09 21:40:53 -07:00
Pejman Pour-Moezzi bb7c956db5 fix: drop stale last30days-v3 argument hint 2026-04-09 21:37:36 -07:00
Pejman Pour-Moezzi 9be0780c46 fix: rename skills/last30days-v3 directory so plugin install resolves correctly 2026-04-09 21:37:00 -07:00
Matt Van Horn 341da37218 Merge pull request #182 from ziperlee/codex/bluesky-refresh-token
fix: refresh expired bluesky sessions
2026-04-09 21:20:58 -07:00
Matt Van Horn 4b3458776b Merge pull request #191 from kriptoburak/feat/add-xquik-source
feat: add Xquik as X/Twitter search source
2026-04-09 21:19:18 -07:00
Matt Van Horn 6436290cc1 Merge pull request #183 from zl190/claim-contributor-entry
docs: claim @zl190 contributor entry
2026-04-09 21:18:04 -07:00
Matt Van Horn 5316d92fc6 Merge pull request #185 from hnshah/hnshah-claim-contrib
docs: claim hnshah contributor entry
2026-04-09 21:18:02 -07:00
Matt Van Horn 1f350d2b30 Merge pull request #189 from pejmanjohn/contrib/mvanhorn-last30days-skill-102-claim-contributors-entry
docs: claim @pejmanjohn CONTRIBUTORS.md entry
2026-04-09 21:17:04 -07:00
Matt Van Horn 474cc4ad7a Merge pull request #188 from pejmanjohn/contrib/mvanhorn-last30days-skill-101-readme-update-command
docs: add plugin update command to README
2026-04-09 21:17:01 -07:00
Matt Van Horn 80892d31c6 Merge pull request #187 from pejmanjohn/contrib/mvanhorn-last30days-skill-100-fix-nux-argument-hint
fix: update argument-hint in root SKILL.md from last30days-3 to last30days
2026-04-09 21:16:59 -07:00
Matt Van Horn 5c4383d661 Merge pull request #192 from tmchow/fix/remove-orphaned-exa-test
fix(tests): remove orphaned test_exa_search.py
2026-04-09 21:06:44 -07:00
Matt Van Horn 101c4724f4 Merge pull request #193 from tmchow/fix/evaluator-test-env-isolation
fix(tests): isolate env in resolve_google_judge_api_key test
2026-04-09 21:06:42 -07:00
Matt Van Horn c09ca59747 Merge pull request #194 from tmchow/fix/bump-version-metadata-v3
fix: bump gemini-extension and v3 skill version to 3.0.0
2026-04-09 21:06:40 -07:00
Matt Van Horn dad97f1b05 Merge pull request #198 from iliaal/perf/pipeline-optimizations
perf: optimize dedup, parallelize handle searches and enrichment
2026-04-09 21:00:37 -07:00
Ilia Alshanetsky eef3547c37 perf: optimize dedup, parallelize handle searches and enrichment
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).

Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.

Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.

Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.

End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
2026-04-09 19:00:30 -04:00
Ilia Alshanetsky 252c8222f1 feat: add WSL2 Windows Firefox cookie extraction for X auth
On WSL2, native Linux Firefox typically has no x.com cookies since users
browse in Windows. Chromium browsers (Edge, Chrome, Brave) encrypt cookies
with DPAPI/app-bound encryption, making them inaccessible without admin
privileges. Windows Firefox stores cookies unencrypted in SQLite, readable
directly through the /mnt/c mount.

The cookie extractor now detects WSL2 via /proc/version, locates Windows
Firefox profiles under /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox,
and falls back to them when Linux Firefox yields no results. Reports
source as "firefox-wsl" to distinguish from native.

Also fixes profile resolution priority: Install* sections (Firefox >= 67)
now take precedence over the legacy Default=1 flag, which could select a
stale profile on multi-profile installations.
2026-04-09 18:36:02 -04:00
Trevin Chow 6a4071a9fd fix: bump gemini-extension and v3 skill version to 3.0.0
gemini-extension.json still referenced v2.9.5 and
skills/last30days-v3/SKILL.md still said 3.0.0-alpha.
Both now match pyproject.toml's canonical 3.0.0 version.

Addresses items from #190. Structural decisions (SKILL.md
consolidation, SKILL-original.md cleanup) left for maintainer.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 13:00:08 -07:00
Trevin Chow 5896c9582b fix(tests): isolate env in resolve_google_judge_api_key test
The second assertion in test_resolve_google_judge_api_key_prefers_google_key
ran outside the mock.patch.dict context. When GOOGLE_API_KEY or GEMINI_API_KEY
is set in the real environment, os.environ takes precedence over the config
dict fallback and the test fails.

Wrap the assertion in its own mock.patch.dict scope that clears the three
relevant env vars so the test passes regardless of the developer's env.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 12:56:34 -07:00
Trevin Chow 718fe9547b fix(tests): remove orphaned test_exa_search.py
lib/exa_search.py was removed during the v3 refactor but
tests/test_exa_search.py still imports from it. This causes
an ImportError that blocks pytest -x from running any tests.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 12:52:25 -07:00
Burak Bayır 6b3de9170e feat: add Xquik as X/Twitter search source
Add Xquik (xquik.com) as a new X/Twitter search source that uses a REST
API with full engagement metrics (likes, retweets, replies, quotes,
views, bookmarks). Uses stdlib urllib only -- no new dependencies.

- scripts/lib/xquik.py: source module with search, parse, query expansion
- tests/test_xquik.py: 32 unit tests covering all functions
- env.py: XQUIK_API_KEY config and availability check
- pipeline.py: source registration and retrieve dispatch
- normalize.py: reuses _normalize_x (same item format as Bird)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 22:44:10 +03:00
Pejman Pour-Moezzi ca4ffb9633 docs: claim @pejmanjohn CONTRIBUTORS.md entry 2026-04-09 12:20:18 -07:00
Pejman Pour-Moezzi 79a5c3ea94 docs: add plugin update command to README 2026-04-09 12:16:25 -07:00
Pejman Pour-Moezzi a41bf5d8e7 fix: update argument-hint in root SKILL.md from last30days-3 to last30days 2026-04-09 12:12:16 -07:00
Hiten Shah 319ec796bd docs: claim hnshah contributor entry 2026-04-09 11:26:51 -07:00
jason-zl190 86df1e7d4b docs: claim @zl190 contributor entry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 04:04:44 +10:00
Matt Van Horn 45f596ca0c docs: add v3 community contributors section to release notes 2026-04-09 10:56:21 -07:00
Matt Van Horn 4fab23a357 feat: add CONTRIBUTORS.md crediting v3 community inspiration 2026-04-09 10:55:03 -07:00
ziperlee 9405aa3fb4 fix: refresh expired bluesky sessions 2026-04-09 23:03:47 +08:00
ziperlee 2020156591 fix: write briefing files as utf-8 2026-04-09 23:01:17 +08:00
Matt Van Horn b6d97a571d Merge pull request #179 from pejmanjohn/contrib/mvanhorn-last30days-skill-46-prompt-injection-hardening
fix: harden rerank prompts and assistant-facing digests against scraped prompt injection
2026-04-09 06:29:20 -07:00
Matt Van Horn 0e5faa7a82 Merge pull request #175 from ziperlee/codex/readme-v3-alignment
docs: align README with v3 runtime
2026-04-09 06:25:00 -07:00
Matt Van Horn 9de2398106 Merge pull request #173 from pejmanjohn/contrib/mvanhorn-last30days-skill-44-bird-sweet-cookie-runtime
last30days: lazy-load sweet-cookie so vendored Bird works on fresh installs
2026-04-09 06:20:39 -07:00
Matt Van Horn fd6ec55f07 Merge pull request #180 from tmchow/fix/bird-x-list-response
fix(bird_x): normalize list responses from Bird search
2026-04-09 06:10:34 -07:00
Trevin Chow 65399eb4eb fix(bird_x): normalize list responses from Bird search to dict format
When Bird's JSON response is a raw array instead of an object,
json.loads returns a list. All callers use .get('items') which raises
AttributeError on lists. Wrap list responses in {"items": parsed} so
callers always receive a dict.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 01:26:14 -07:00
Pejman Pour-Moezzi b7d1a38ff6 Fix Bird cookie helper lazy-loading 2026-04-08 19:30:39 -07:00
Pejman Pour-Moezzi 89e6ee29fd last30days: harden rerank and render prompts 2026-04-08 19:23:25 -07:00
Matt Van Horn dedf615114 Merge PR #174: last30days: require Python 3.12+ in OpenClaw setup and CLI entrypoints
last30days: require Python 3.12+ in OpenClaw setup and CLI entrypoints
2026-04-08 19:05:47 -07:00
zipee f926d6507a docs: point skill metadata at public repo 2026-04-09 08:40:14 +08:00
zipee bcc4694fa9 docs: point skill metadata at public repo 2026-04-09 08:40:07 +08:00
zipee f09fe202e8 docs: align README with v3 runtime 2026-04-09 08:35:56 +08:00
Pejman Pour-Moezzi 57ec92c299 last30days: require Python 3.12 in setup flows 2026-04-08 14:40:19 -07:00
Pejman Pour-Moezzi 681d05d7ee last30days: lazy-load vendored Bird cookie support 2026-04-08 14:31:17 -07:00
Matt Van Horn 9c203d3595 Merge pull request #172 from pejmanjohn/contrib/mvanhorn-last30days-skill-41-rename-last30days-output
last30days: finish runtime/report rename after /last30days plugin rename
2026-04-08 14:11:06 -07:00
Pejman Pour-Moezzi 77f67c1bd9 last30days: finish runtime/report rename 2026-04-08 13:03:41 -07:00
Matt Van Horn 565deb443e fix: rename plugin from last30days-3 to last30days 2026-04-08 11:12:06 -07:00
Matt Van Horn 0a9ff16dfc feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources
v3 rewrites the search engine from the ground up:

- Intelligent pre-research: resolves X handles, GitHub repos, subreddits,
  TikTok hashtags, and YouTube channels before searching
- GitHub person-mode: PR velocity, top repos by stars, release notes
- GitHub project-mode: live star counts, README, releases, top issues
- ELI5 mode: plain language synthesis, no jargon
- 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket,
  GitHub, Threads, Pinterest, Perplexity, Bluesky, Web
- Free Reddit comments via public JSON (no API key needed)
- Fun judge v2: humor scoring baked into narrative
- Cookie consent before browser scanning
- 10,000 free ScrapeCreators calls
- 1,012 tests

Thank you to the community contributors whose issues and PRs shaped v3:
@uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116),
@thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78),
@zl190 (#115), @hnshah (#84, #85, #86)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:23 -07:00
Matt Van Horn 61904b31e3 feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX
* feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX

- INCLUDE_SOURCES=tiktok,instagram in .env forces sources on for all
  query types, bypassing the tier system
- NUX shows opt-in modal after ScrapeCreators key is saved: "Also
  search TikTok and Instagram?" with honest call-usage warning
- Tier system preserved as default — override only when INCLUDE_SOURCES set

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: neutral call-usage copy — works for free and paid tiers

---------

Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 06:22:37 -07:00
Matt Van Horn bb79d54b2b Revert "fix: ScrapeCreators modal sells all 5 platforms, not just Reddit"
This reverts commit 2eb9cd6fba.
2026-03-29 18:17:25 -07:00
Matt Van Horn 2eb9cd6fba fix: ScrapeCreators modal sells all 5 platforms, not just Reddit 2026-03-29 18:16:58 -07:00
Matt Van Horn 775596ce21 feat: v2.9.6 — free-first NUX, cookie extraction, quality scoring
Setup wizard with consent-first cookie extraction (Chrome/Firefox/Safari),
yt-dlp auto-install, ScrapeCreators push, quality scoring (5 core sources),
status banner redesign, honest Reddit labeling, inline YouTube transcripts,
Exa free web search, Reddit public fallback, and post-research quality nudge.

Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 14:33:17 -07:00
Matt Van Horn 4d6224f79a feat(youtube): extract transcript highlights like Reddit comment gems
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.

SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:49:38 -07:00
Matt Van Horn 499074b564 fix(youtube): pass full transcripts to LLM instead of truncating to 200 chars
TRANSCRIPT_MAX_WORDS raised from 500 to 5000 so the LLM gets the full
content of most videos (up to ~25 minutes). Removed the second 200-char
truncation in render.py that was reducing transcripts to a single sentence
before the judge agent ever saw them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:28:01 -07:00
Matt Van Horn 6a5a0013c0 Merge pull request #70 from mvanhorn/fix/bluesky-cloudflare-error-messages
fix(bluesky): surface real error instead of misleading 'auth failed'
2026-03-15 22:56:00 -07:00
Matt Van Horn 0c6358ab98 fix(bluesky): surface real error instead of misleading "auth failed"
When Cloudflare blocks requests to bsky.social or public.api.bsky.app
with a 403, the error was swallowed by a generic except clause and
reported as "Bluesky auth failed" - misleading users into thinking
their credentials were wrong.

Now _create_session() preserves the specific error in _session_error,
and search_bluesky() surfaces it. Cloudflare 403s get a clear message
about network-level blocks. Actual 401s say "Invalid credentials".

Closes #69

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 22:13:28 -07:00
Matt Van Horn 7ffa508bb4 feat(last30days): improve ClawHub discoverability - description, tags
Updated frontmatter description to be more search-friendly for ClawHub.
Added 12 new tags: deep-research, twitter, bluesky, recency, news,
citations, multi-source, social-media, analysis, web-search, ai-skill,
clawhub. Also added 11 GitHub repo topics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 17:27:53 -07:00
Matt Van Horn c2e18dbbeb fix(skill): use full /last30days name in comparison follow-up suggestions
The short alias /last30 only works on some platforms. Claude Code requires
the full /last30days name, so the follow-up suggestions after comparison
research were producing "Unknown skill: last30" errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 07:55:17 -07:00
Matt Van Horn 52a22f5cb1 Merge pull request #65 from j-sperling/feat/search-quality-consolidation
Consolidate query/relevance modules and improve search quality
2026-03-14 07:31:24 -07:00
Matt Van Horn 3830274e11 Merge pull request #67 from j-sperling/feat/query-type-source-tiering
Add query-type-aware source tiering and scoring
2026-03-14 07:30:06 -07:00
Matt Van Horn 569745c0ed Merge pull request #66 from j-sperling/fix/endpoint-model-updates
Fix stale API endpoints and model identifiers
2026-03-14 07:29:56 -07:00
Jeffrey Sperling 058c4e1899 Classify prompt and animation queries earlier
Map prompt-oriented product searches and animation-oriented build searches away from the breaking-news default so source tiering and tiebreakers align with the benchmark topics.

Validation: uv run python -m unittest tests.test_query_type
2026-03-14 00:38:59 -07:00
Jeffrey Sperling c711e443fe Reduce Reddit and Polymarket false positives
Weight Reddit relevance toward titles, stop Polymarket from expanding low-signal standalone terms, and prevent short binary outcomes from matching unrelated queries.

Validation: uv run python -m unittest tests.test_reddit_sc tests.test_polymarket
2026-03-14 00:38:52 -07:00
Jeffrey Sperling 8c1dce95e8 Harden local search evaluation harness
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.

Validation: uv run python -m unittest tests.test_evaluate_search_quality
2026-03-14 00:38:43 -07:00
Jeffrey Sperling 3a0f3d8b19 Accept GOOGLE_API_KEY for local Gemini eval
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.

Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
2026-03-13 19:25:29 -07:00
Jeffrey Sperling 8eda5fad5c Add local search quality evaluation harness
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.

The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.

Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
2026-03-13 19:21:33 -07:00
Jeffrey Sperling 946af84f9a Tighten relevance scoring and Polymarket ranking
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.

Also apply the relevance filter to Polymarket and update the affected cross-source tests.

Validation: uv run python -m unittest
2026-03-13 19:21:25 -07:00
Jeffrey Sperling 0e46c7cb33 Pass X auth through handle drilldowns
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.

Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
2026-03-13 01:09:03 -07:00
Jeffrey Sperling b489663450 Loosen source tiering for usage queries
Classify prompting and animation queries as how_to so the stack does not treat them as generic breaking news. Also keep X available for how_to and preserve YouTube/HN coverage for breaking-news and prediction queries.

Validated with uv run python -m unittest tests.test_query_type and the five-query local comparison run used for PR #65 review.
2026-03-13 01:08:28 -07:00
Jeffrey Sperling 3aaf31b08d Document env-based X auth flow
Update README, launch copy, and UI guidance to prefer popup-free AUTH_TOKEN/CT0 configuration, and keep X backend selection on the verified Bird or xAI paths.

Validation: uv run python -m unittest tests.test_env_project
2026-03-12 21:07:09 -07:00
Jeffrey Sperling dd9a3f1482 Disable browser cookie fallback for local X auth
Prefer injected AUTH_TOKEN/CT0 for bundled Bird, disable browser-cookie probing in repo-invoked subprocesses, and keep repo-invoked yt-dlp from inheriting browser-cookie settings.

Validation: uv run python -m unittest tests.test_bird_x tests.test_youtube_yt
2026-03-12 21:07:04 -07:00
Jeffrey Sperling cbee987f65 Extract relevance_filter, add Bluesky/TruthSocial type hint + test coverage
- Extract _relevance_filter from last30days.py closure to score.relevance_filter()
  for testability
- Add BlueskyItem/TruthSocialItem to sort_items() type hint (was missing despite
  being in _ITEM_SOURCE_MAP)
- Add tests: Bluesky/TruthSocial engagement scoring, sort_items mixed sources,
  relevance_filter behavior (threshold, minimum-result guarantee, missing attr),
  select_openai_model HTTP 401/403 error paths
2026-03-11 19:09:06 -07:00
Jeffrey Sperling d8d2b97716 Gitignore mise.toml instead of removing it
Dev environment tool config is useful locally but shouldn't be tracked.
2026-03-11 19:01:33 -07:00
Jeffrey Sperling 036bcd2ae3 Address review feedback: deduplicate query_type, clean unused imports, fix defaults
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
  canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
  tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
  should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
2026-03-11 18:40:07 -07:00
Jeffrey Sperling 6c402f66b7 Update plan to reflect single upstream PR strategy 2026-03-11 18:32:45 -07:00
Jeffrey Sperling 046795c4ae Add post-retrieval relevance filtering across all sources
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.

This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 1002f1f020 Add platform-specific query optimizations
- hackernews: use extract_core_subject instead of raw topic, add
  points>5 filter and restrictSearchableAttributes=title to reduce
  noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
  filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
  queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
  to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
2026-03-11 18:32:45 -07:00
Jeffrey Sperling c5be117701 Replace hardcoded 0.7 relevance with computed token-overlap scores
- bird_x: parse_bird_response now accepts query param and computes
  token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement

This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 38caae3288 Add mise.toml for Python version pinning and implementation plan
Pin Python 3.12 via mise for consistent local development.
Add plan document for the query/relevance consolidation work.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 96948cc7c0 Deduplicate relevance code across youtube/tiktok/instagram/scrapecreators_x
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.

Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling dc88c215be Integrate shared query.py into per-source modules
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.

Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set

Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling fa42a5d031 Add urllib fallback for TikTok/Instagram when requests unavailable
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00
Matt Van Horn b38703e53d feat(truthsocial): Add Truth Social as opt-in source
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.

27 new tests, 440 total passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 00:14:39 -07:00
Matt Van Horn b6fd5ff406 docs: v2.9.5 README update, fix plugin.json hooks (#62)
- Fix plugin.json hooks field from invalid array to empty object
- Add Claude Code plugin install above ClawHub badge
- Version bump to v2.9.5 with new features block (Bluesky, comparative
  mode, ScrapeCreators X, per-project env, expanded tests)
- Add Bluesky to all source list references
- Document BSKY_HANDLE/BSKY_APP_PASSWORD env vars in install + optional sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:40:07 -07:00
Matt Van Horn adb5a672d9 fix(bluesky): make Bluesky opt-in with app password auth
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:27:39 -07:00
Matt Van Horn ecf90c0281 feat(skill): add comparative mode and Bluesky references to SKILL.md
Add COMPARISON query type for "X vs Y" research with 3 parallel passes.
Add Bluesky stats line and update all source list references.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:58:21 -07:00
Matt Van Horn 9a1059ee9d feat(bluesky): add Bluesky/AT Protocol as social source
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).

- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:54:20 -07:00
Matt Van Horn 4b7087e136 feat(x): add ScrapeCreators as X/Twitter search backend
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).

New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.

Fixes #55.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:34:39 -07:00
Matt Van Horn 82a006280e Merge pull request #58 from phjlljp/feat/session-start-config-check
feat: add SessionStart hook for config check
2026-03-09 21:55:57 -07:00
Matt Van Horn a7398c50cc Merge pull request #59 from phjlljp/feat/per-project-env-config
feat: add per-project .env config support
2026-03-09 21:55:51 -07:00
Matt Van Horn b1a0e2bcfc Merge pull request #57 from phjlljp/feat/smoke-tests-edge-cases
test: add smoke tests and edge case coverage
2026-03-09 21:55:44 -07:00
Matt Van Horn 75e4b8e2cd Merge pull request #56 from phjlljp/feat/unit-tests-untested-modules
test: add unit tests for untested modules
2026-03-09 21:55:42 -07:00
Matt Van Horn 25e27bdade fix(skill): unquote $ARGUMENTS and add marketplace plugin path
- Remove double quotes around $ARGUMENTS so argparse can parse flags
  like --deep, --store separately instead of as part of the topic string.
  Fixes #61.

- Add ~/.claude/plugins/marketplaces/last30days-skill to the path
  discovery loop so marketplace installs can find scripts/.
  Fixes #54.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:55:24 -07:00
P 3e3615e2c8 feat: add per-project .env config support
Add per-project configuration via .claude/last30days.env, discovered by
walking up from cwd. Uses the same .env format as the existing global
config — no new parsers or formats.

Priority (highest wins):
  1. Environment variables
  2. .claude/last30days.env (per-project)
  3. ~/.config/last30days/.env (global)

Also adds file permission checking — warns to stderr if config files
are readable by other users (should be chmod 600).

Includes tests for discovery, precedence, source tracking, and
permission warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:48:23 -04:00
P 9a58fe6481 feat: add SessionStart hook for config check
Add a lightweight hook that runs on session start to check if any
API keys are configured. Warns users if no config is found and
checks file permissions on existing config files.

Checks (in order): .claude/last30days.env, ~/.config/last30days/.env,
OPENAI_API_KEY env var, SCRAPECREATORS_API_KEY env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:47:36 -04:00
P 4756c20ec0 refactor: match upstream unittest convention
Convert all new tests from bare pytest style to unittest.TestCase
with sys.path.insert, matching the convention used by all existing
tests. Remove pyproject.toml and conftest.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:43:04 -04:00
P 8a9f734d14 test: add smoke tests and edge case coverage
Add end-to-end smoke tests and expand coverage for score and render modules:

- test_smoke.py (new): subprocess tests for --diagnose, --help, --mock,
  and missing topic. Validates exit codes, JSON structure, and source
  detection (HN/Polymarket always available).
- test_score.py: add TestCommentQualityWeight (top_comment_score boost),
  TestInstagramEngagement (basic scoring, views vs likes weighting)
- test_render.py: add TestEnsureOutputDir, TestXrefTag, TestRenderEmptyReport

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:41:58 -04:00
P 0979f506db test: add unit tests for untested modules
Add pytest infrastructure (pyproject.toml, conftest.py) and unit tests
for modules that previously had zero test coverage:

- test_schema_roundtrip.py: to_dict() serialization for all data classes
- test_reddit_enrich.py: URL parsing, thread data parsing, comment filtering
- test_reddit_sc.py: ScrapeCreators Reddit search (query expansion, subreddit discovery)
- test_instagram_sc.py: Instagram relevance scoring, tokenization, depth config

Includes fixtures/reddit_thread_sample.json for reddit_enrich tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:36:52 -04:00
Matt Van Horn 627947fc2c feat(plugin): publish as Claude Code marketplace plugin
Update .claude-plugin/marketplace.json and plugin.json to v2.9.5 with
full metadata. Add plugin install instructions to README as the
recommended install method. The repo root serves as both the marketplace
and the plugin - skills/last30days/SKILL.md (symlink) is discovered
automatically.

Users can now install with:
  /plugin marketplace add mvanhorn/last30days-skill
  /plugin install last30days@last30days-skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 13:09:33 -07:00
Matt Van Horn 2f16ff1ee8 feat(gemini): add Gemini CLI extension support
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.

Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.

Closes #45

Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:07:55 -07:00
Matt Van Horn 8f7fb5a7fe fix(release): v2.9.5 - remove re-introduced Save Research section
PR merges on March 7 (PR #48 Xiaohongshu, upstream merge) regressed
SKILL.md by re-introducing the "Save Research to Documents" section
that v2.9.4 removed. Those branches were forked before v2.9.4 and
brought the old content back via merge resolution.

Fixes: remove save section, restore --save-dir flag on bash command,
update agent mode line, add tool-call guard to STOP instruction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 11:55:35 -07:00
Matt Van Horn 4503da7920 docs: add perpetual monitoring mode plan (not building yet)
Explored adding scheduled re-runs and cumulative intelligence to
last30days. Concluded that Claude Code's session-scoped scheduling
(CronCreate/loop) can't support true perpetual monitoring since
jobs die when the terminal closes. Plan documents the architecture,
what exists, and why we're waiting for persistent background agents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 09:37:18 -07:00
Matt Van Horn 28dff6e7b2 Merge remote-tracking branch 'upstream/main' 2026-03-07 16:36:15 -08:00
Matt Van Horn 32992834ee Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:11:35 -08:00
Matt Van Horn 7dd8379c61 Merge origin/main into feat/xiaohongshu-reddit-public-fallback
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:08:20 -08:00
Matt Van Horn 408ac148ec Merge pull request #50 from mark-c4r/add-entity-extract-tests
test: add tests for entity_extract module
2026-03-07 15:59:32 -08:00
Matt Van Horn c9559252cd Merge pull request #52 from 04cb/fix/missing-metadata-files
Fix missing metadata files in skill upload bundle
2026-03-07 15:59:11 -08:00
04cb f70370a6f4 Fix missing metadata files in skill upload bundle 2026-03-07 18:10:44 +08:00
Matt Van Horn fad26d41fd fix: improve ClawHub security scan result
- Remove prompt-injection false positive ("you are now" → "treat yourself as")
- Declare AUTH_TOKEN and CT0 in frontmatter optionalEnv
- Clarify X token access language (no browser session access)
- Add permissions overview block near top of file

Zero functionality changes — metadata and prose only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:35:38 -08:00
Matt Van Horn ef1f380cda feat: publish to ClawHub as last30days-official
- Add ClawHub badge and install command to README
- Update SKILL.md: metadata.openclaw canonical key, license/author/repository fields, optionalEnv vars, added instagram/polymarket tags
- Add .clawhubignore to exclude binary assets and dev files from bundle

Published: https://clawhub.ai/skills/last30days-official

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:08:14 -08:00
Matt Van Horn e690d61a12 fix: append -raw suffix to saved research filenames
e.g. sam-altman-raw.md instead of sam-altman.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:56:03 -08:00
Matt Van Horn 6d5acb9121 feat(release): v2.9.4 - move save into Python script, zero post-invitation noise
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.

Tested: --mock confirms file creation and duplicate date suffixing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:27:53 -08:00
Matt Van Horn 28d223e43c fix(release): v2.9.3 - foreground save, fix hallucinated user messages
CRITICAL: run_in_background callbacks caused model to re-engage after
save, hallucinate fake "Human:" messages, and generate unsolicited
multi-paragraph responses. Switch to foreground cat > heredoc which
executes sub-second with no callback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:50:17 -08:00
Matt Van Horn 471badd329 fix(release): v2.9.2 - silent save, no follow-up text after background save
- Background Bash heredoc instead of Write tool
- Suppress response text on save completion
- 📎 footer line replaces verbose confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:41:55 -08:00
Matt Van Horn 5b1636f94f fix: merge upstream save fix (background Bash instead of Write tool)
Resolves merge conflict, keeping upstream's approach:
- Background heredoc save instead of Write tool
- Adds 📎 footer line
- No more "Wrote N lines..." cluttering output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 14:59:06 -08:00
Matt Van Horn 18f6273f7c fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:22 -08:00
Matt Van Horn 9950d01ab4 fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:11 -08:00
Matt Van Horn cc774d5e69 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Sync from public repo. Every run now saves the complete briefing as a
topic-named .md file to ~/Documents/Last30Days/. Credit @devin_explores.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:50:43 -08:00
Matt Van Horn 8cbbe87c3e docs: add v2.9.1 auto-save note to README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:27:18 -08:00
Matt Van Horn 49d993b162 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Bump version to 2.9.1, update changelog and release notes.
Credit @devin_explores for inspiring the feature.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:15:42 -08:00
Matt Van Horn 6afc094bb1 Merge pull request #51 from mvanhorn/feat/auto-save-documents
feat(skill): auto-save research results to ~/Documents/Last30Days/
2026-03-05 19:12:56 -08:00
Matt Van Horn f6a1769e35 feat(skill): auto-save research results to ~/Documents/Last30Days/
Every run now automatically saves the complete briefing (synthesis,
stats, follow-up suggestions) as a topic-named .md file in the user's
Documents folder. Agent mode also saves. No Python script changes -
this is purely a SKILL.md instruction addition.

Inspired by @devin_explores manually saving results to build a
personal research library.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:06:27 -08:00
Matt Van Horn 4d35b53eab docs: v2.9.0 release — ScrapeCreators Reddit default, top comments, smart discovery
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 18:03:40 -08:00
Matt Van Horn 2247800003 chore: clean up Reddit log prefix, mark plan tasks complete 2026-03-05 18:01:24 -08:00
Matt Van Horn 7048fe7b83 feat(reddit): elevate top comments, improve subreddit discovery, default to ScrapeCreators
Three improvements from beta testing:

1. Top comments: 10% scoring weight for comment quality, 💬 top comment
   rendered prominently in compact/full output, increased insight limits
2. Subreddit discovery: relevance-weighted scoring with topic word matching,
   utility sub penalties (UTILITY_SUBS blocklist), engagement bonus
3. Default method: SKILL.md primaryEnv → SCRAPECREATORS_API_KEY, web-only
   banner recommends SC first, security section updated

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 17:29:18 -08:00
Matt Van Horn 30b973f62e docs: add Reddit ScrapeCreators v2 improvements plan
Three focused improvements based on 5 full-pipeline beta tests:
1. Elevate top Reddit comments in scoring and rendering
2. Improve subreddit discovery heuristic for ambiguous queries
3. Make ScrapeCreators the default recommended Reddit method

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 17:24:41 -08:00
Matt Van Horn 09b09946c0 feat: replace OpenAI Reddit search with ScrapeCreators API
- New scripts/lib/reddit.py: multi-query expansion, global search,
  subreddit discovery, targeted subreddit search, comment enrichment
- 68 results in 17s vs ~15 results in 60-90s (OpenAI)
- Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper)
- Real engagement data (score, comments, dates) from API
- No more 429 rate limits on comment enrichment
- Falls back to OpenAI if SCRAPECREATORS_API_KEY missing
- Registered as last30daysbeta for parallel local testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 15:55:02 -08:00
mark-c4r d4328b5598 test: add tests for entity_extract module
Covers _extract_x_handles (8 cases), _extract_x_hashtags (5 cases),
_extract_subreddits (6 cases), and extract_entities integration (4 cases).
Follows existing test patterns from test_dedupe.py.
2026-03-05 16:12:33 -06:00
YJLi-new 788514ce8e feat: add Xiaohongshu source and Reddit public fallback
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
2026-03-05 20:54:33 +08:00
Matt Van Horn db75f9e341 feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.

- Add scripts/lib/instagram.py: keyword search, transcript extraction,
  relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
  URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 07:00:51 -08:00
Matt Van Horn 740dcc5789 docs: update README and SKILL.md for ScrapeCreators TikTok API
Replace all Apify references with ScrapeCreators. Key points:
- No subscription required (was $5/mo with Apify)
- 100 free credits, pay-as-you-go after
- SCRAPECREATORS_API_KEY replaces APIFY_API_TOKEN
- Backwards compatible: APIFY_API_TOKEN still works as fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 05:17:30 -08:00
Matt Van Horn e03046bd49 refactor(tiktok): replace Apify with ScrapeCreators API
Root cause of empty TikTok results: Apify required monthly subscription.
ScrapeCreators is PAYG with 100 free credits and no subscription.

Key fix: ScrapeCreators nests items under aweme_info wrapper
(search_item_list[].aweme_info.{fields}), which the previous
implementation missed, causing all fields to be empty.

Changes:
- Rewrite tiktok.py to use ScrapeCreators REST API
- Add aweme_info unwrapping for correct field extraction
- Add transcript fetching via /video/transcript endpoint
- Add SCRAPECREATORS_API_KEY to env.py config
- Update last30days.py to use env.get_tiktok_token()
- Delete apify_client_wrapper.py (no longer needed)
- Update tests for new date field format (create_time)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 13:58:51 -08:00
Matt Van Horn 1d18bee1a2 fix(skill): forward CLI flags through $ARGUMENTS to Python script
Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits
the expansion, and change argparse topic from nargs="?" to nargs="*"
so multi-word topics still work. Also document --store, --include-web,
--diagnose, and --timeout flags in the Options section.

Closes #36

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 13:52:08 -08:00
Matt Van Horn fb00856bff docs: update README for v2.7 with TikTok examples and installation
Add TikTok as 7th source throughout README: new V2.7 banner, real
search examples (Iran Israel: 61.6M views, Leah Halton: 152.6M views),
APIFY_API_TOKEN in installation, Apify in security table, fix stale
"six sources" references to "seven sources".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:52:51 -08:00
Matt Van Horn d7b354b2cf fix(ui): suppress [TikTok] and [Apify] log lines in non-TTY mode
Only print debug log lines when running in an interactive terminal.
In Claude Code (non-TTY), the spinner system handles progress display,
so these raw log lines just add noise.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:45:00 -08:00
Matt Van Horn 7c5763d048 fix(apify): suppress verbose actor log streaming to stderr
Pass logger=None to Apify .call() to prevent the SDK from streaming
raw actor run logs (status messages, crawler stats, warnings) that
drown out the clean spinner UI in Claude Code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:43:38 -08:00
Matt Van Horn 61729b9ae7 fix(ui): show YouTube and TikTok progress spinners in Claude Code
Remove quiet=True from YouTube and TikTok spinners so they display
the same colored emoji progress lines as Reddit and X in non-TTY mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:36:50 -08:00
Matt Van Horn b990aed40e feat: add --no-native-web flag to skip Parallel AI in Claude Code
When running in Claude Code, the assistant has a built-in WebSearch tool
that's free and higher quality than Parallel AI/Brave/OpenRouter. Adding
--no-native-web to the SKILL.md invocation defers web search to the
assistant, saving API credits. OpenClaw invocations don't pass this flag,
so they continue using native web backends.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:34:03 -08:00
Matt Van Horn d4ac57f041 fix(tiktok): restore missing websearch import in orchestrator
The websearch module import was dropped when the tiktok import was added,
causing the script to crash during the rendering phase after all data
was successfully collected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:25:30 -08:00
Matt Van Horn 1db0b6054a feat(tiktok): add TikTok as 7th signal source via Apify
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.

- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:08:19 -08:00
Matt Van Horn 5e5d586f7d fix: triage all 16 open GitHub issues — close 9, fix 6, comment 1
Batch 1 (closed): #43 spam, #34 dup, #19 resolved, #2 resolved, #41 answered
Batch 2: Added MIT LICENSE file (#35), closed #42 (license question)
Batch 3 code fixes:
  - #29: YouTube skip reason shows "0 results" instead of "not installed"
  - #30: Bird source mapping handles reddit-web + Bird combo
  - #39: watchlist.py extracts YouTube + TikTok findings, run-one prints output
  - #40: watchlist.py uses search_queries field when available
Batch 4:
  - #32: marketplace.json source "." → "./" with $schema ref
  - #36: commented with investigation plan ($ARGUMENTS forwarding)
  - #4: Added SSL troubleshooting section to README
Also commented on #22 (Bird features) and #31 (skills.sh audit).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:06:55 -08:00
Matt Van Horn 94b6b6eb7b feat(search): add --search flag for source filtering
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.

--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn  (run only Reddit + Hacker News)

Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:43:33 -08:00
Matt Van Horn 6ae4b16791 feat(bird_x): add noise words + last-chance retry with strongest token
Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool
noise words to _extract_core_subject, and a last-chance retry that falls back
to the longest non-noise token when 2-word retry also returns 0 results.

cache.py and render.py env overrides were already on main.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:39:52 -08:00
Matt Van Horn 82efa6100b fix(skill): use plain source names in Web stats line, not URLs
URLs in markdown links wrap badly in terminals (discovered after first
fix attempt). Change to plain names like "Newsweek, Sportskeeda, Medium"
on the Web: stats line. Update citation note to explain the reason.

Tested on Dor Brothers, Kanye West, Logan Paul - no trailing Sources:
block appeared in any of the three test runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:54:06 -08:00
Matt Van Horn a52ed30109 fix(skill): suppress trailing Sources: block from WebSearch tool mandate
The WebSearch tool has a system-level mandate to append a Sources:
section at the end of every response. SKILL.md's old "DO NOT output
Sources: list" instruction was too weak to override it.

Fix: redirect citations into the stats block's Web: line as inline
links. The WebSearch citation requirement is satisfied there; an
explicit note after the stats block tells the model not to append
a separate trailing section.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:44:55 -08:00
Matt Van Horn 78678e3919 chore: add .gitignore and PR #37 finalization plan
- .gitignore: protect docs/comparison-results/ and other private
  benchmark artifacts from accidental upstream push
- docs/plans: add plan for PR #37 Codex auth finalization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:44:55 -08:00
Matt Van Horn 04bfb5381d fix(tests): patch env isolation in test_api_key_takes_priority
The test was picking up the real OPENAI_API_KEY from the shell
environment, causing it to fail on any machine with that key set.
Added @patch.dict(os.environ, {}, clear=True) so the test runs in
a clean env and exercises the file_env path as intended.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:36:37 -08:00
Ilia Alshanetsky d7bff81757 fix(bird_x): pass .env credentials to Node subprocesses for WSL2/headless auth
* chore: fix YAML error in argument-hint

* add codex auth support to responses API

* Use gpt-5.1-codex-mini as default model for Codex auth

Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.

* Pass .env credentials to Bird Node subprocesses for X auth

On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.

- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
  credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection

---------

Co-authored-by: Justin Williams <jblwilliams@gmail.com>
2026-03-02 23:24:59 -08:00
Matt Van Horn 1ed990a081 feat(skill): v2.6 - agent-native invocation and --agent report mode
Removes disable-model-invocation restriction so the skill can be called
by other agents via the Skill tool. Adds --agent flag for non-interactive
report output (skips intro, AskUserQuestion, wait pause, and invitation).
Fixes false security doc claiming autonomous invocation was blocked.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 23:07:47 -08:00
Matt Van Horn 27b4865d60 feat(skill): bump version to 2.5
Polymarket prediction markets (6th source), Hacker News (5th source),
cross-source linking, synonym expansion, X handle resolution.
15-way blinded comparison: 4.38 vs 3.73, won all 5 topics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:04:41 -08:00
Matt Van Horn 9ee1291033 docs: update README for v2.5 - Polymarket + HN as killer features, add Anthropic odds example
Reorder v2.5 features: Polymarket prediction markets and HN lead as #1,
multi-signal quality-ranked relevance scoring as #2. Add Anthropic Odds
example showcasing 11 live markets from a two-word query. Add Anthropic
and OpenAI Polymarket transcripts to launch tweets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:42:00 -08:00
Matt Van Horn e48d84b1d0 fix(polymarket): two-pass query expansion finds markets where topic is an outcome
The Gamma API only searches event titles/slugs, missing markets where the
topic is an outcome (e.g., "Arizona" in NCAA Tournament Winner). This adds:

- All-word query expansion (not just first word): "Arizona Basketball" now
  searches "Arizona", "Basketball" independently
- Tag-based domain expansion: extracts category tags (e.g., "NCAA") from
  first-pass results and searches those as a second pass
- Neg-risk binary market synthesis: shows team names from market questions
  instead of generic Yes/No outcomes
- Question shortening: extracts "Arizona" from "Will Arizona win the NCAA
  Tournament?" for clean display
- Increased depth (3 pages) and result caps (15) for more coverage

Live results: "Arizona Basketball" now finds NCAA Tournament Winner (12%),
#1 Seed (88%), Big 12 Champion (69%). "Iran War" returns 15 markets (up
from 9) with no regression.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 09:06:35 -08:00
Matt Van Horn 309c8e37c6 fix(sync): deploy to last30daysCROSS with patched frontmatter
- Add CROSS to sync targets with sed-patched name/version/description
- Switch cp to rsync to handle identical file edge case on APFS
- CROSS SKILL.md gets last30daysCROSS skill root injected into search path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 08:27:30 -08:00
Matt Van Horn 2ff9b6f6c1 feat(polymarket): outcome-aware scoring and synthesis instructions
- _compute_text_similarity() now checks outcome names with bidirectional
  substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
  market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 08:12:45 -08:00
Matt Van Horn 4c82309c36 fix(skill): strengthen zero-result hiding and improve Polymarket stats format
Make the "omit 0-result sources" instruction more emphatic (CRITICAL prefix,
enumerate all formats to suppress). Change Polymarket stats line to show
top 2-3 market odds instead of just top-1.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:09:18 -08:00
Matt Van Horn 9d9e7e89d9 feat(polymarket): replace position-based ranking with quality-signal relevance
Polymarket results now rank by text similarity, volume, liquidity, price
movement, and competitive score instead of API return position. Also fixes
pagination (DEPTH_CONFIG now controls page count, not a no-op limit param)
and caps results after re-ranking.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:07:43 -08:00
Matt Van Horn 994a4ab2ca feat(polymarket): add Polymarket prediction markets as 6th research source
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.

Also hides sources with zero results from the stats output (all sources).

54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 22:27:19 -08:00
Matt Van Horn 52503f8e68 docs(readme): credit community contributors for HN source
Shoutout to @ARJ999 (first HN submission, PR #12), @wkbaran
(PR #26, referenced in planning), and @gbessoni for endorsing
HN as the right addition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:10:58 -08:00
Matt Van Horn 44a662ef94 docs(readme): reorder v2.5 features - lead with better results
Reorganize the changelog to match the real story: smarter scoring,
cross-source linking, and handle resolution as one "dramatically
better results" narrative first, then HN, then handle resolution
details. Moves blinded comparison right after the results section
as proof.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:06:49 -08:00
Matt Van Horn 7009039ac4 docs: v3 -> v2.5 version bump (incremental, not major)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:00:57 -08:00
Matt Van Horn 41c8742fcf docs: update README to v3 with new features
- Bump version to v3
- Add headline features: HN source, handle resolution, cross-source
  linking, YouTube relevance scoring
- Add full v3 changelog section with detailed feature descriptions
- Reorganize v2/v2.1 changelog sections
- Add HN to security/privacy data table
- Update examples and descriptions throughout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:59:59 -08:00
Matt Van Horn 71903b596f fix(skill): broaden handle resolution to products/tools/brands, add parody check
Handle resolution was only triggering for "person/brand" topics. Products
and tools like "Nano Banana Pro" or "Seedance" can also have X accounts
(@nanobanana, @seedanceai) but were being skipped.

Changes:
- Broadened trigger: people, brands, products, tools, companies, communities
- Added parody/fan account verification guidance
- Added site:x.com to WebSearch query for better results
- Updated both SKILL.md and OpenClaw variant

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:46:16 -08:00
Matt Van Horn bbaaf28d2e fix(x): don't skip unfiltered resolved handle search when entity-extracted
The resolved handle dedup was wrong: if entity_extract found @thedorbrothers
(from @mentions in Phase 1 results), the resolved handle search was skipped
entirely. But entity-extracted handles are searched WITH topic keywords
(from:handle topic), while resolved handles need UNFILTERED search
(from:handle) to find posts that don't mention the topic string.

Example: Dor Brothers' viral tweet (5.5K likes) says "We made a $300M movie
starring @LoganPaul" - no mention of "dor brothers" anywhere. The topic-
filtered entity search missed it. The unfiltered resolved search finds it.

Before: 30 X posts, 161+ likes (entity search only)
After: 40 X posts, 5549+ likes (resolved handle adds viral tweet)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:35:08 -08:00
Matt Van Horn 4f584a4e96 feat(x): resolve X handles for person/brand topics via agent WebSearch
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.

- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:00:49 -08:00
Matt Van Horn bed0557b65 feat(quality): GOAT synthesis improvements - hybrid cross-source linking, YouTube synonyms, human-readable xref tags
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics
(4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further:

- dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold,
  cross-source links went from 3 to 13 items across 5 topics
- render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags
- youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score)
- SKILL.md: instruction #7 tells Claude to lead with cross-platform signals

Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5
topics. Biggest gains in specificity (+0.8) and format compliance (+1.0).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 16:06:53 -08:00
Matt Van Horn 0591f55f0e feat(quality): YouTube relevance scoring and cross-source linking
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.

Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 10:46:58 -08:00
Matt Van Horn f60a4359a0 fix(ui): quiet HN and YouTube spinners in non-TTY mode
Reddit and X are the star of the show. In Claude Code (non-TTY),
suppress  start messages for HN and YouTube so Reddit/X are the
first visible lines. HN/YouTube still show ✓ completion messages.
Also suppress [HN] debug logs in non-TTY to reduce output clutter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:58:01 -08:00
Matt Van Horn 7a9f447231 fix(ordering): move HN after YouTube in stats, sort priority, and SKILL.md
HN was appearing before YouTube in the stats block, sort tiebreaker,
and source status. Now consistently: Reddit > X > YouTube > HN > Web.
Also restored emoji + box-drawing chars in test skill SKILL.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:41:14 -08:00
Matt Van Horn 38a7ea253e feat(hackernews): add Hacker News as 5th research source
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:33:31 -08:00
Matt Van Horn 427a4e453d Merge pull request #33 from tjarko/fix/429-exponential-backoff
Fix OpenAI 429 rate limiting with exponential backoff
2026-02-20 23:03:01 -08:00
Tjarko Leifer 451ebb3e22 Fix OpenAI 429 rate limiting with exponential backoff
The Reddit search uses OpenAI's Responses API with web_search, which
frequently returns 429 rate limit errors. The previous retry logic used
linear backoff (1s, 2s, 3s) which is too aggressive for OpenAI's rate
limiter (often needs 10-60s waits).

Changes:
- Increase max retries from 3 to 5
- Switch from linear to exponential backoff (2s, 5s, 9s, 17s, 33s)
- Parse and respect Retry-After header from OpenAI 429 responses
- Fall back to cheaper models (gpt-4.1 → gpt-4o) on 429s, not just
  on 400/403 access errors
- Remove gpt-4o-mini from fallback chain — it doesn't support
  web_search with the filters parameter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 12:40:36 +01:00
Matt Van Horn a4d09e470e docs: add CHANGELOG.md, release notes, and v2.1 polish
Add CHANGELOG.md (Keep a Changelog format) and release-notes.md
for the v2.1.0 GitHub Release. Also includes queued copy updates:
plugin.json version bump to 2.1.0, YouTube added to descriptions,
SKILL.md security/permissions section and ClawdBot metadata,
updated watchlist examples, and v2.1 launch post refinements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 12:16:01 -08:00
Matt Van Horn 4547a095bf fix: correct OpenClaw GitHub URL
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:34:57 -08:00
Matt Van Horn 6ca037f387 docs: clarify watchlist is designed for Open Claw / always-on bots
Watchlist schedules are metadata only — nothing triggers runs
automatically. Updated README to be honest that cron/launchd or an
always-on bot like Open Claw is required for automated scheduling.
Local Claude Code users can still use run-one and run-all manually.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:33:43 -08:00
Matt Van Horn 7d6bac7b0e fix: handle engagement: null in watchlist and YouTube normalizer
When X posts return engagement: null, dict.get("engagement", {})
returns None (key exists with null value), causing AttributeError.
Use `or {}` idiom to coalesce None to empty dict.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 07:29:27 -08:00
Matt Van Horn 87bb469ab7 docs: remove em dashes, update terminology, replace Ralph Wiggum
- Replace all em dashes with hyphens
- "Claude Code skill" -> "skill" in intro
- Ralph Wiggum -> Seedance 2.0 / Nano Banana Pro
- tweet/Tweet -> post/Post throughout
- "signal of a tweet" -> "signal of a single post"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 01:12:22 -08:00
Matt Van Horn 3bf20ef072 docs: reframe v2.1 — watchlist #1, YouTube #2, Codex #3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 01:08:35 -08:00
Matt Van Horn 2d6855483d fix: Reddit future timeout — same bug as YouTube
OpenAI Responses API web_search takes 60-90s but the generic
future timeout was killing it at 30s (quick) / 60s (default).
Added reddit_future key to TIMEOUT_PROFILES (60/90/120s).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:57:32 -08:00
Matt Van Horn 20a859ecec fix: YouTube timeout bump to 90s + Reddit 429 fail-fast
YouTube: Add youtube_future timeout key (60/90/120s for quick/default/deep)
separate from the shared future timeout. YouTube needs more time because
it does search + parallel transcript fetching. Previously, 20 videos +
5 transcripts exceeded the 60s budget and all results were discarded.

Reddit 429: Propagate rate-limit errors instead of swallowing them.
Enrichment now uses 10s timeout / 1 retry (was 30s / 3 retries).
On first 429, cancel remaining enrichment and skip Phase 2 Reddit.
Total time wasted on 429 drops from ~75s to ~12s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:48:26 -08:00
Matt Van Horn c3640931ed fix: skip hard date filter for YouTube — keep evergreen content
The hard date filter in the main script was double-filtering YouTube
results. youtube_yt.py already applies a soft date filter that prefers
recent videos but keeps older ones when < 3 are within range (for
evergreen topics like "youtube thumbnails"). The hard filter then
removed all of them, resulting in 0 YouTube items.

YouTube content has a longer shelf life than tweets/posts, so the
soft filter's fallback behavior is correct.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:32:36 -08:00
Matt Van Horn 1acb5c6fff fix: remove eager import from __init__.py and add sync.sh
The `from . import bird_x` in __init__.py caused ImportError for all
other module imports (youtube_yt, render, etc.) when bird_x had issues,
crashing the entire script in Codex. Making it a bare package marker
lets Python resolve `from lib import X` normally.

Added scripts/sync.sh to deploy to all three skill locations
(~/.claude, ~/.agents, ~/.codex) with import verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:31:50 -08:00
Matt Van Horn 14b6ac06e1 feat(nux): conversational NUX with "last30" trigger and dynamic source status
Replace CLI-style promo banners with casual, conversational NUX message
that teaches users to say "last30" + natural language. Remove stale Bird
CLI npm install references and dead prompt_bird_install methods.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:19:38 -08:00
Matt Van Horn 06f74a4d0c fix: add timeouts, process cleanup, and source diagnostics
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:

- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override

Also fixes the UX gap where missing sources were silently skipped:

- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:05:36 -08:00
Matt Van Horn 7162eb6b36 fix: YouTube display and search quality
- Remove 2>&1 from SKILL.md so stderr doesn't pollute model input
- Run script in foreground (not background) with 5min timeout
- Add explicit YouTube synthesis instruction for Claude
- Remove --flat-playlist which broke date filtering (all dates were None)
- Move date filtering to Python with soft fallback for evergreen topics
- Keep 'tips', 'tutorial', 'review', 'guide' in YouTube search queries
- Increase yt-dlp timeout from 60s to 120s for full metadata fetch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 00:02:16 -08:00
Matt Van Horn e520db31d3 docs: mark all plan acceptance criteria as complete
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:48:15 -08:00
Matt Van Horn 23b7906d63 docs: add open variant section and new CLI flags to README
- Document open variant installation (watchlist, briefings, history)
- Add --store, --diagnose, --include-web to options table
- Note optional web search API keys (Parallel AI, Brave, OpenRouter)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:47:54 -08:00
Matt Van Horn 9ff00ce38b feat(open): create open variant SKILL.md and reference files
Router SKILL.md dispatches to mode-specific references:
- research.md: one-shot research with --store for persistence
- watchlist.md: add/remove/list/run topics
- briefing.md: daily/weekly briefing generation
- history.md: query/search accumulated findings
- context.md: agent memory template

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:46:35 -08:00
Matt Van Horn 08e2010554 feat(engine): add native web search, --store, and --diagnose to main engine
- _search_web() dispatches to Parallel AI / Brave / OpenRouter based on config
- Web results flow through full pipeline: normalize → score → dedupe
- --diagnose shows all source availability (API keys, Bird, YouTube, web backends)
- --store persists findings to SQLite via store.py for watchlist/briefing system
- run_research() now returns web_items alongside reddit/x/youtube
- web_needed flag only set when no native web backend is available

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:43:34 -08:00
Matt Van Horn ba330e9a0c feat(open): Port web search backends, persistence layer, and env merge from openclaw 2026-02-14 23:35:44 -08:00
Matt Van Horn a09413608d feat: Add Codex CLI compatibility
- Add agents/openai.yaml for Codex skill discovery
- Make SKILL.md script path portable (repo, Claude, Codex, agents dirs)
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dirs with env var overrides and tempdir fallback
- Add Codex installation docs to README

Inspired by PR #24 (el-analista) and PR #5 (jblwilliams).
Zero impact on existing Claude Code behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 23:18:53 -08:00
Matt Van Horn 9397fcc937 docs: Polish README for v2.1 launch + add launch copy with social posts
- Split V2.1 blurb into two clear bullets (YouTube + X search)
- Better X auth section with browser-specific notes and verify command
- Updated intro/tagline to mention YouTube as 4th source
- Added social post drafts and updated issue #19 response with YouTube

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 22:50:47 -08:00
Matt Van Horn c66ca7f43d feat: Add YouTube as 4th research source via yt-dlp
YouTube search and transcript extraction runs automatically when yt-dlp
is installed. Searches for topic videos from the last N days, fetches
auto-generated transcripts for top results, and feeds them through the
same scoring pipeline (relevance + recency + engagement) as Reddit/X.

New files:
- youtube_yt.py: search, transcript extraction, VTT cleanup

Modified files:
- schema.py: YouTubeItem dataclass, updated Report
- normalize.py: normalize_youtube_items()
- score.py: YouTube engagement scoring (views-dominated)
- dedupe.py: YouTube deduplication
- render.py: YouTube section in compact output
- env.py: is_ytdlp_available() check
- ui.py: YouTube progress messages
- last30days.py: _search_youtube(), parallel execution with Reddit/X
- SKILL.md: YouTube in stats box, citation priority
- README.md: YouTube docs, yt-dlp requirement, Peter shoutout

Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 21:38:04 -08:00
Matt Van Horn 31313c69ac v2.1: Bundle Bird X search - no external CLI needed
Vendor Bird's Twitter GraphQL search client directly into /last30days,
eliminating the dependency on `npm install -g @steipete/bird`. X search
now works out of the box with just Node.js 22+ and browser cookies.

- Add vendored bird-search.mjs wrapper (search-only subset of Bird v0.8.0)
- Vendor @steipete/sweet-cookie for browser cookie extraction
- Update bird_x.py to call vendored Node.js module instead of `bird` binary
- Update README.md and SKILL.md for v2.1 (remove Bird CLI install steps)
- Include Bird's MIT LICENSE in vendor directory

The fallback chain is: vendored search -> xAI API key -> web-only mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 17:16:04 -08:00
Matt Van Horn 0ec338c630 Add speed tradeoff note to README intro
Honest about 2-8 min research time vs V1's speed. Points users
to @mvanhorn / @slashlast30days for quick mode feedback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 11:01:54 -08:00
Matt Van Horn 205f8fa7ae Add timing disclaimer to parsed intent output
"Research typically takes 2-8 minutes (niche topics take longer)."
Shown right after parsed intent, before research starts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 11:00:06 -08:00
Matt Van Horn fdfe3c7e9e Add Vibe Motion as third README example showing brand-new tool research
Demonstrates /last30days researching a tool that launched 2 days ago —
before blog roundups existed. Real-time X/Reddit reactions from
@Hartdrawss, @rezkhere, @Totinhiiio plus honest Chase Jarvis review.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:57:35 -08:00
Matt Van Horn ff5cc1445c Add Kanye West as second README example showing current events mode
Real V2 output: 33 X posts (9.3K likes), @dekreationz viral post,
WSJ apology arc, BULLY album, Bianca Vanity Fair — demonstrates
/last30days as a news/culture research tool, not just prompting.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:56:25 -08:00
Matt Van Horn 04b532582c Add Nano Banana Pro as first README example, remove old duplicate
New example uses fresh V2 research output showing JSON prompting
dominance, @handle citations, and the full research-to-prompt flow.
Removed the old iOS App Mockup example which used the same SwimMom
prompt/image but with stale V1 data.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:54:01 -08:00
Matt Van Horn 45702efd63 Update argument-hint with real example queries
Replace generic "[topic]" and --days=7 with concrete examples that
show the range: prompting, news, recommendations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:49:52 -08:00
Matt Van Horn 232a948576 Reframe README V2 around better search results as primary feature
Lead with dramatically better X/Reddit search quality (smarter query
construction + Phase 2 supplemental), Bird CLI as secondary headline,
community contributions in their own section.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:46:08 -08:00
Matt Van Horn 41779b81c0 Fix X search returning 0 results on popular topics
Three bugs in _extract_core_subject():

1. Multi-word noise phrases ("what are", "how to") never matched
   because code compared individual words against multi-word strings.
   "what are people saying about DeepSeek R1" became "what are people
   saying" — losing the entire topic.

2. Missing meta words — "prompt", "techniques", "tips" weren't
   filtered (only "prompting" was). "vibe motion best prompt
   techniques" kept 4 keywords instead of 2.

3. No retry on 0 results — Reddit retries with simplified queries
   but X accepted 0 and moved on.

Fix: Two-phase extraction (strip multi-word prefixes/suffixes first,
then individual noise words), expanded noise set, max 3 words (was 4),
and automatic retry with first 2 words when Bird returns 0 results.

Before → After:
- "vibe motion best prompt techniques" → "vibe motion" (was 4 words, 0 results)
- "what are people saying about DeepSeek R1" → "deepseek r1" (was "what are people saying")
- "nano banana pro prompts for gemini" → "nano banana pro" (was 4 words)

Tested: vibe motion (12 X posts, was 0), DeepSeek R1 (12 posts),
kanye west (12 posts, no regression).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:43:45 -08:00
Matt Van Horn 83f8d7ece5 Add V2 highlights to README intro, credit @galligan for plugin inspiration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:28:54 -08:00
Matt Van Horn 52456e5820 Update README for V2 and bump USER_AGENT to 2.0
- Add V2 to title
- Document Bird CLI installation (free X search)
- Add --days=N to options table
- Update Requirements to show Bird CLI as recommended free option
- Replace How It Works with two-phase search architecture
- Add What's New in V2 section with all new features
- Credit community contributors (@JosephOIbrahim, @levineam, @jonthebeef)
- Bump http.py USER_AGENT from 1.0 to 2.0

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:26:53 -08:00
Matt Van Horn 6c620cbbf4 Smart invitations with research-specific example suggestions
Every invitation now includes 2-3 specific examples based on what the
research actually found. PROMPTING mode explicitly offers to write prompts
with creative examples. NEWS suggests follow-up questions about the
biggest stories. GENERAL suggests specific deep-dives and applications.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:11:30 -08:00
Matt Van Horn 88bca6e55c Fix invitation wording: add 'I'm now an expert' to all modes, remove prompt language from GENERAL
GENERAL mode said "write a prompt for it" which sounds weird for people/topics
("write a prompt for Kanye West"). Changed to "describe what you're working on
and I'll help." Added "I'm now an expert on {TOPIC}" prefix to all 4 modes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:04:10 -08:00
Matt Van Horn 1ae7a16c75 feat: Smart supplemental search — Phase 2 entity-aware drill-down
After the initial broad search (Phase 1), extract key entities from results
and run targeted secondary searches to surface content the broad pass missed:

- New entity_extract.py: parses @handles, #hashtags, subreddits from results
- bird_x.py: search_handles() does targeted from:handle searches via Bird CLI
- openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint
- last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes

Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X).
Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 10:00:28 -08:00
Matt Van Horn 650aa1100b Context-aware invitation, smart follow-ups, no raw URLs
- Invitation adapts to QUERY_TYPE: NEWS gets "ask me anything",
  PROMPTING gets "share your vision", RECOMMENDATIONS gets "compare options"
- Follow-up handler reads intent: questions get answers, not forced prompts
- Restored V1's "if they ask a question, answer it" context memory line
- URL formatting rule: "per Rolling Stone" not raw URLs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:34:30 -08:00
Matt Van Horn 6da4b2f5c7 Remove citation priority plan file
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:22:42 -08:00
Matt Van Horn 9da95019e6 Elevate Reddit/X citations over web sources in output
- Add citation priority rule: @handles > r/subreddits > web sources
- Fix examples: "per @honest30bgfan_" instead of "per Rolling Stone"
- Add "lead with people, not publications" instruction
- Stats box: web now shows "(supplementary)" instead of domain list
- No changes to actual research — same quality, better attribution framing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:22:07 -08:00
Matt Van Horn a953d57730 Add PR credits tracking for V2 launch
Track community PRs to thank contributors when V2 goes public.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:16:23 -08:00
Matt Van Horn 6d80af3807 Add Claude Code marketplace plugin support
- Add .claude-plugin/marketplace.json and plugin.json manifests
- Use ${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/skills/last30days} for script path
- Plugin install: uses CLAUDE_PLUGIN_ROOT when set
- Manual install: falls back to ~/.claude/skills/last30days (no breakage)
- SKILL.md stays at root, plugin.json points skills to "./"

Users can now install via:
  claude plugin marketplace add mvanhorn/last30days-skill
  claude plugin install last30days@last30days

Inspired by PR #1 (galligan) — lighter approach without repo restructure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:11:28 -08:00
Matt Van Horn 60f06ea86e Merge community PRs: Windows Unicode fix, 403 fallback, --days flag
- PR #17 (JosephOIbrahim): Fix UnicodeEncodeError on Windows cp1252
- PR #16 (levineam): Handle HTTP 403 model access errors, add gpt-4.1 fallback
- PR #18 (jonthebeef): Add --days=N flag for configurable lookback (1-30)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 09:05:46 -08:00
Matt Van Horn c8ddb60397 Revert query parsing display to V1 format
User preferred V1's "I'll research X across Reddit, X, and the web..."
with explicit Parsed intent block over V2's emoji format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 08:18:37 -08:00
Matt Van Horn 10f61c503d Restore 7 missing v1 features + add query parsing display + v1-vs-v2 test plan
Feature audit found 11 things v2 dropped from v1. Restored the 7 that affect
output quality: quality checklist for prompts, anti-pattern examples, self-check
instruction, "don't re-search" context memory, prompt format anti-pattern,
multi-prompt guidance, and emoji footer with engagement counts.

Also adds visible query parsing display (🔍 **{TOPIC}** · {QUERY_TYPE}) and
the full v1-vs-v2 comparison test plan with all 17 test queries from README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 15:27:46 -08:00
Matt Van Horn 4b0098b754 Remove context: fork — root cause of broken skill execution
The old v1 skill worked because context: fork was SILENTLY IGNORED
due to Claude Code bug #17283. The skill ran inline in the main
conversation. Claude Code 2.1+ fixed the bug and now properly
honors fork mode, creating an isolated subagent that ignores all
instruction ordering (text output, bash-first, etc.).

Fix: remove context: fork so the skill runs inline again, matching
the behavior the user knows works. Also restored v1-style
instruction flow: parse intent first, then run script, then WebSearch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:28:58 -08:00
Matt Van Horn df29bbb53d Revert to bash-first execution with echo progress display
The agent in fork mode ignores text output instructions and jumps
straight to WebSearch. Reverted to the original "YOUR FIRST ACTION:
Run this command. EXECUTE." pattern that actually worked. Embedded
progress display as echo statements inside the bash command so they
can't be skipped. The agent was not running the Python script at all
when the first instruction was about text output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:18:53 -08:00
Matt Van Horn a6dd0a8b0e Move progress block to very first instruction in SKILL.md
The agent was skipping the progress text because it was buried after
parsing logic at line 41. Restructured so the FIRST section is
"YOUR FIRST OUTPUT" with the progress block, BEFORE any tool call
instructions. Added explicit "DO NOT skip this" enforcement.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:16:49 -08:00
Matt Van Horn c3120d29a3 Show visible progress block before research starts
The ui.py spinner/color output goes to stderr which Claude Code
buffers invisibly. Instead, have the agent output a visible progress
block with emoji showing Reddit/X/Web agents deploying.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:13:59 -08:00
Matt Van Horn 11f9ab8519 Restore acknowledgment step before research execution
Add back the brief topic acknowledgment ("Researching X — searching
Reddit for upvotes...") and descriptive intro that the v1 skill had.
The v2 version was jumping straight into script execution with no
user-facing feedback about what was being researched.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:10:49 -08:00
Matt Van Horn 7c36866524 Fix v2 output quality: stats format, Reddit results, citations, summary structure
- Stats: replace BAD/GOOD examples with strict fill-in-the-blank template
- Reddit: add subreddit-targeted fallback search, soften scoring penalties
  (engagement -10→-3, date confidence -10→-5), add minimum result guarantee
- Citations: limit to 1 per insight, short format, no engagement metrics
- Summary: add bold topic headers template for structured paragraphs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 10:04:46 -08:00
Matt Van Horn 38451d44c9 Restore original description, enforce emoji stats format, require source attribution
- Reverted description to original public repo copy
- Restored H1 heading, moved imperative instruction to bold paragraph
- Added BAD/GOOD examples enforcing emoji tree stats format
- Required @handle and r/subreddit citation in ALL summary types
- Removed agent:Explore and disable-model-invocation from frontmatter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 09:48:21 -08:00
Matt Van Horn 63de326df0 Fix skill not executing: remove agent:Explore, front-load run command
The Explore agent was treating SKILL.md as documentation to summarize
instead of instructions to follow. Removed agent:Explore from frontmatter
and restructured opening to immediately command script execution.

Also adds release plan for Bird CLI v2.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-06 09:38:32 -08:00
Matt Van Horn 4230fa2b69 Fix YAML parsing error in argument-hint field
Wrap value in single quotes to properly escape nested double quotes.
This fixes skills CLI installation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 07:37:17 -08:00
Matt Van Horn 4e972d009c Rename skill from last30daystest to last30days
- Update skill name and description (remove TEST markers)
- Update script path to ~/.claude/skills/last30days/
- Clean up header, keep Bird CLI features prominent

Ready for local testing as the main skill.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 16:11:21 -08:00
Matt Van Horn 17485ec797 Make RECOMMENDATIONS format flexible (table or stacked)
- Remove over-prescriptive format requirements
- Let Claude choose table vs stacked naturally based on content
- Keep critical requirement: Sources line with @handles
- Claude Code's renderer handles the responsive wrapping

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:55:41 -08:00
Matt Van Horn 56f9eceb9c Switch RECOMMENDATIONS to stacked card format (responsive)
- Replace ASCII table with stacked card format
- Each card: Tool/Mentions/Use Case/Sources on separate lines
- Cards separated by horizontal rule (────)
- Doesn't break when terminal window is narrow
- Still includes @handles in Sources line

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:12:56 -08:00
Matt Van Horn c534c3b9a5 Update RECOMMENDATIONS format to use ASCII tables with @handles
- Changed from numbered list to ASCII box-drawing table format
- Added "Key Sources" column that MUST include @handles from X posts
- Added explicit instructions to parse and display highest-engagement handles
- Matches the output quality of the original xAI version

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:04:41 -08:00
Matt Van Horn 3afc1213b8 Add explicit instructions to calculate engagement totals
Claude was summarizing qualitatively ('mostly casual conversation')
instead of summing actual engagement numbers from the output.

Added CRITICAL note to:
- Parse [Xlikes, Yrt] from each X post
- Sum totals for the stats display
- Show top voices with engagement (e.g., @handle (15K likes))

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:47:41 -08:00
Matt Van Horn e7610ddcac Fix Bird search: strip noise words + fix date parsing
1. Add _extract_core_subject() to strip noise words before X search
   - X search is literal keyword matching, not semantic
   - "best open claw usecases" → "open claw" (27 results vs 0)

2. Fix ISO date detection bug
   - Was: if "T" in created_at (matched "Tue" in "Tue Feb 03...")
   - Now: if created_at[10] == "T" (proper ISO separator check)
   - Dates now parse correctly: (2026-02-03) instead of (date unknown)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:38:38 -08:00
Matt Van Horn f800b5eb1b Fix Bird CLI response parsing
1. last30days.py: Check isinstance(dict) before .get() - Bird returns
   list on success, dict on error

2. bird_x.py: Update field mappings for Bird's actual response format:
   - author.username not user.screen_name
   - createdAt not created_at (camelCase)
   - likeCount, retweetCount, etc. (camelCase)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 14:00:40 -08:00
Matt Van Horn 2c398ef6cb Enable API keys: remove clean mode
Now reads from ~/.config/last30days/.env like the regular skill.
Bird CLI + API keys both work together.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:47:10 -08:00
Matt Van Horn bcf5c2cc3f Fix: embed date filter in query instead of --since flag
Bird CLI doesn't support --since as a CLI flag.
Use X's search syntax: 'topic since:YYYY-MM-DD'

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:26:51 -08:00
Matt Van Horn 89bdbae0fe Fix: restore pretty emojis and box chars in stats summary
Original format:
 All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2}

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 13:09:14 -08:00
Matt Van Horn 06061884f1 Update SKILL.md: remove interactive prompt references
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 12:39:16 -08:00
Matt Van Horn 976a6e2867 Match original NUX: non-blocking Bird promotion
- Remove interactive "Install Bird CLI?" prompt
- Add Bird as FREE option in promo messages
- Auto-detect Bird silently (use if authenticated)
- Keep original flow: show promo → continue with available sources

Bird now appears in the promo alongside API keys:
  🔵 X (Twitter)
     └─ FREE: npm install -g @steipete/bird (uses browser session)
     └─ Or: Add XAI_API_KEY (paid API)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 12:38:54 -08:00
Matt Van Horn 73b42c5c89 fix: show Bird install prompt before topic validation
Move Bird CLI check to run before topic validation so users see
the install prompt even when running without a topic argument.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:53:04 -08:00
Matt Van Horn fe9ebb9944 feat: add clean mode for testing Bird CLI flow
- Add LAST30DAYS_CONFIG_DIR env var to override config location
- Set LAST30DAYS_CONFIG_DIR="" in test skill for clean mode
- Test skill now ignores existing API keys to show Bird install prompt

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:48:40 -08:00
Matt Van Horn d122210e1e rename: use last30daystest as main skill name for testing 2026-02-03 11:32:47 -08:00
Matt Van Horn 80b9fa74dd feat: add test skill (last30daystest) for Bird CLI testing
Creates SKILL-TEST.md with name "last30daystest" so the Bird CLI
integration can be tested without affecting the original /last30days
skill installed from the public repo.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:06:43 -08:00
Matt Van Horn ecdee52d04 feat(main): integrate Bird setup into main flow
- Move progress display initialization early to support Bird prompts
- Check Bird availability and offer install if no X source available
- Override available sources when Bird is ready
- Pass x_source parameter to run_research function

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:04:56 -08:00
Matt Van Horn e57b82f1ad feat(main): dispatch X search to Bird or xAI 2026-02-03 11:03:29 -08:00
Matt Van Horn 7a7e35bcf5 feat(main): add Bird setup function
Add bird_x import and setup_bird_if_needed function to main script.
The function checks Bird status and offers installation if needed,
returning 'bird' if ready, 'declined' if user declined, or None if
unavailable.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:02:28 -08:00
Matt Van Horn 25d58a4494 feat(lib): export bird_x module
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 11:01:46 -08:00
Matt Van Horn df849e7278 feat(env): add X source detection with Bird priority 2026-02-03 11:01:23 -08:00
Matt Van Horn 57e238cb9c feat(ui): add Bird CLI install prompts and auth help 2026-02-03 11:00:42 -08:00
Matt Van Horn 34723c70ac feat(bird): add response parser matching xai_x format 2026-02-03 10:59:39 -08:00
Matt Van Horn 2910e83552 feat(bird): add search_x function 2026-02-03 10:58:44 -08:00
Matt Van Horn 444bd6e185 feat(bird): add installation and status functions 2026-02-03 10:57:57 -08:00
Matt Van Horn a7fb951257 feat(bird): add detection functions for Bird CLI
- is_bird_installed() checks if bird command is in PATH
- is_bird_authenticated() runs bird whoami to verify auth
- check_npm_available() checks if npm can install bird

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:57:26 -08:00
Matt Van Horn 8ae00ea214 docs: Add Bird CLI implementation plan
12-task implementation plan for integrating Bird CLI as free X/Twitter
search alternative. Includes detection, interactive install, search
dispatch, and main flow integration.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:53:03 -08:00
Matt Van Horn 950810be92 docs: Add Bird CLI integration design
Adds design doc for integrating Bird CLI as an alternative X/Twitter
search source. Bird uses browser cookies (free, no API key) and
provides direct GraphQL access to X.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 10:50:59 -08:00
Matt Van Horn cc892d736b revert: restore original argument-hint format (testing) 2026-01-28 13:43:10 -08:00
Matt Van Horn 6cc7dad86e fix: YAML frontmatter and GPT-5 model fallback
- Fix invalid YAML in SKILL.md argument-hint (closes #8)
  Wrapped value in single quotes to properly escape double quotes

- Add automatic model fallback for GPT-5 access errors (closes #9)
  When OpenAI returns 400 for unverified orgs, retry with gpt-4o

- Add tests for model fallback logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 11:07:33 -08:00
Matt Van Horn 3a4a727f4b Run Reddit and X searches in parallel, fix timeout handling
- Refactor run_research() to use ThreadPoolExecutor for parallel execution
- Reddit timeout/crash no longer blocks X search from running
- Add catch for ConnectionResetError/OSError in http.py
- Per-item error handling in Reddit enrichment (one failure doesn't crash all)
- Increase API timeouts from 60/90/120 to 90/120/180 seconds
- Add ClawdBot setup example to README

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 13:33:41 -08:00
Matt Van Horn 86422a74af feat: Add honesty warning when results aren't from last 30 days
Adds ⚠️ LIMITED RECENT DATA warning when:
- Fewer than 5 items are confirmed from the date range
- Tells Claude to be transparent with user about data freshness

Example output for obscure topic (June Oven):
"Only 4 item(s) confirmed from 2025-12-26 to 2026-01-25.
Results below may include older/evergreen content."

Popular topics (clawdbot, nano banana) don't show the warning.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 13:00:53 -08:00
Matt Van Horn e1d3570667 feat: Add automatic retry with simpler query for sparse Reddit results
If initial search returns <5 threads, extract core subject and retry:
- "best nano banana prompting practices" → retry with "nano banana"
- Combines results from both searches, deduped by URL

Note: OpenAI's web_search still tends to find old content. This retry
helps cast a wider net but doesn't fully solve the recency issue.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:44:29 -08:00
Matt Van Horn 2d43571875 fix: Simplify Reddit prompt and increase search volume
OpenAI's web_search doesn't reliably find recent Reddit content.
Strategy change: request MORE threads and let server-side date
filtering handle it.

Changes:
- Increased depth config (20-30 → 30-50 default)
- Simplified prompt to search broadly and include all matches
- Let server-side date filter remove old content
- Removed aggressive pre-filtering that was limiting results

Tradeoff: Some topics have more X coverage than Reddit due to
web_search limitations. This is acceptable.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:35:14 -08:00
Matt Van Horn acd7a533c0 fix: Emphasize recency in Reddit search to find recent threads
The model was finding old threads (Nov-Dec 2025) that got filtered out,
leaving 0 results even for popular topics like "nano banana prompts".

Changes:
- Add explicit date filters: "after:{from_date}" and "2026"/"January 2026"
- Emphasize RECENT content is critical, old threads are filtered anyway
- Tell model to verify dates before including threads

Before: 0-2 Reddit threads for "nano banana prompting practices"
After: 16 Reddit threads from January 2026

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 10:15:43 -08:00
Matt Van Horn b77d821f8d docs: Add Top Claude Code Skills example to README
Shows /last30days as a discovery tool for finding popular resources
in rapidly evolving ecosystems - ranked by community engagement.

Includes Remotion (17.3K likes), SkillsMP marketplace, Superpowers,
and Trail of Bits security skills.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:50:40 -08:00
Matt Van Horn 66d7b87049 docs: Add ClawdBot use cases example to README
Shows /last30days as a product research tool - discovering real-world
use cases with engagement metrics from r/LocalLLaMA, r/selfhosted, and
X posts from @steipete, @danpeguine, @dreetje.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:48:35 -08:00
Matt Van Horn 7dc3e7b716 fix: Improve Reddit search prompt to find actual discussions
- Add EXTRACT CORE KEYWORDS step to search for main subject, not full phrase
  ("killer features of clawdbot" → search "clawdbot")
- Remove hardcoded subreddit list that biased results toward design/dev subs
- Simplify search strategies and URL validation rules
- Make prompt more concise - GPT-5.2 responds better to clearer instructions

Before: 0 threads found for "killer features of clawdbot"
After: 7 threads found including r/LocalLLaMA (43pts), r/selfhosted (26pts)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:42:31 -08:00
Matt Van Horn 6fbfbb9ccc feat: Add web-only fallback mode with API key promo
- Skill now works without any API keys using WebSearch fallback
- Shows promo banner marketing Reddit/X data when keys are missing
- Partial mode (one key) shows shorter tip for the missing source
- Updated SKILL.md to document three modes: Full, Partial, Web-Only
- Added get_missing_keys() to env.py for promo logic
- Added show_promo(), start_web_only(), show_web_only_complete() to ui.py
- Updated render_compact() to include inline promo for web-only mode

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 09:32:24 -08:00
Matt Van Horn 6831e624b8 docs: Add photorealistic aging portrait example
Shows multi-panel consistency for photorealistic people:
- JSON structure with face preservation
- Same person at ages 10, 20, 40, 80
- Heterochromia + freckles maintained across panels
- Real generated result

Also added to SKILL.md intro examples.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:28:15 -08:00
Matt Van Horn 7f1ab7a2e9 fix: Remove duplicate stats display from Judge section
Stats were showing twice - once in Judge section, once at end.
Now Judge section just synthesizes internally, stats only appear
at the end right before invitation.

Order: What I learned -> Stats -> Share your vision

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:22:55 -08:00
Matt Van Horn 2192743c74 docs: Add iOS app mockup example (SwimMom for Nano Banana Pro)
Shows JSON-structured prompting for UI mockups:
- Layer-based structure (frame -> layout -> content)
- iOS 18 design system references
- 8pt grid and element limits
- Real generated mockup result

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:14:56 -08:00
Matt Van Horn 60c663697e fix: Reorder output - stats right before invitation
New order:
1. What I learned / KEY PATTERNS (or Most mentioned)
2. Stats (All agents reported back!)
3. Share your vision (last)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:09:52 -08:00
Matt Van Horn bcb1c7d71b feat: Reorder use cases - Prompting first, rename HOW-TO to PROMPTING
Prompting is the primary use case, so it should be first:
1. Prompting - "Midjourney prompts", "ChatGPT image generation"
2. Recommendations - "best Claude Code skills", "top AI tools"
3. News - "what's happening with OpenAI"
4. General - any topic

Also renamed HOW-TO query type to PROMPTING for clarity.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:07:35 -08:00
Matt Van Horn c8504d3b30 fix: Use user's exact terminology, don't add outdated tech names
Claude was "helpfully" adding DALL-E, GPT-4o to searches when user
asked about "ChatGPT image prompting". This uses outdated knowledge.

Now explicitly instructs: use the user's exact terminology, don't
substitute or add tech names. Trust the user knows what they're
asking about.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:05:45 -08:00
Matt Van Horn 6577da0b14 feat: More emojis and bring back Judge Agent in display
Updated research banner:
- Added Judge Agent: "Standing by to synthesize..."
- Better emoji descriptions: "Scanning for gold", "Catching latest takes"

Updated completion display:
- "All agents reported back!"
- "Judge Agent: Weighing the evidence..."
- Emojis on the Judge's evaluation criteria

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:03:41 -08:00
Matt Van Horn 6a7c29010d fix: Answer follow-up questions from research, don't re-search
After research is complete, Claude is an expert. Follow-up questions
should be answered from the research findings, not trigger new
WebSearches. Only do new research for a different topic.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:01:35 -08:00
Matt Van Horn b6cdbc9058 fix: Display research banner before background script starts
Since script runs in background, users don't see its live output.
Now Claude displays the fancy emoji banner immediately before
starting the background script.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:00:30 -08:00
Matt Van Horn 5f8623f129 feat: Smart query detection for better research results
Added QUERY_TYPE detection:
- RECOMMENDATIONS ("best X") → searches for lists, extracts specific names
- NEWS → searches for current events
- HOW-TO → searches for tutorials
- GENERAL → broad topic research

For RECOMMENDATIONS queries, synthesis now extracts specific entity names
with mention counts instead of generic patterns.

Example: "best Claude Code skills" now returns:
  "Most mentioned: /commit (5x), remotion (4x), git-worktree (3x)"
Instead of:
  "Skills are good. Keep them under 500 lines."

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 13:23:31 -08:00
Matt Van Horn 68420ca1e2 fix: Run Reddit/X script in background for parallel execution
Start the script first with run_in_background:true, then do WebSearch.
Both run in parallel so users see the script's progress display while
WebSearch also fetches results.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:48:07 -08:00
Matt Van Horn 0e4df9f38a fix: Clean up final display - no Sources, end with invitation
- Suppress "Sources:" output from WebSearch (redundant noise)
- Reorder: Research Complete stats → What I learned → Share your vision
- Simplified KEY PATTERNS to 3 items
- Clean ending users see while waiting for their input

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:38:51 -08:00
Matt Van Horn 85e7b6445c fix: Run WebSearch before script for better working state
Users now see the script's progress display as the "working state"
while waiting, rather than the WebSearch tool call.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:32:49 -08:00
Matt Van Horn e926a5aabd fix: Remove redundant Web Agent from banner display
WebSearch shows as visible tool calls, so announcing it in the
banner was redundant. Now only Reddit and X agents are shown
in the initial display.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:29:36 -08:00
Matt Van Horn 18696e7b05 fix: Enforce strict 30-day date filtering
Previously Reddit was returning ~60% old content (some from 2022).
This commit adds multiple layers of date enforcement:

- Reddit prompt: Explicit from_date/to_date with "fewer results > older results"
- Hard filter: filter_by_date_range() in normalize.py excludes old content
- WebSearch Date Detective: Extracts dates from URLs (/2026/01/24/) and
  snippets ("January 24, 2026", "3 days ago")
- WebSearch scoring: +10 bonus for verified dates, -20 penalty for unknown

The skill now guarantees only content from the last 30 days.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 12:23:34 -08:00
Matt Van Horn 354e8bad03 docs: Update README with marketing copy
Add new tagline emphasizing AI moves fast and this skill keeps you current.
Replace em-dashes with regular dashes/colons throughout.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 11:57:52 -08:00
Matt Van Horn 407801f8be feat: Add parallel research agents with Judge synthesis
- Deploy 3 agents in parallel: Reddit+X, WebSearch, Backup WebSearch
- Add visual progress indicators (🟠🔵🌐)
- Add Judge Agent to synthesize all sources
- Include web authors in top voices (e.g., "John on Medium")
- Weight Reddit/X higher than WebSearch (engagement signals)

Makes the research feel magical with parallel execution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:51:07 -08:00
239 changed files with 50070 additions and 3977 deletions
+23
View File
@@ -0,0 +1,23 @@
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "last30days-skill",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
"owner": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"plugins": [
{
"name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
"version": "3.0.9",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"source": "./",
"category": "productivity",
"homepage": "https://github.com/mvanhorn/last30days-skill"
}
]
}
+15
View File
@@ -0,0 +1,15 @@
{
"name": "last30days",
"version": "3.0.14",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "Matt Van Horn",
"email": "mvanhorn@gmail.com",
"url": "https://github.com/mvanhorn"
},
"homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"],
"hooks": {}
}
+18
View File
@@ -0,0 +1,18 @@
# Exclude binary assets and dev/test artifacts from ClawHub bundle
assets/
docs/
fixtures/
tests/
plans/
agents/
variants/
release-notes.md
SPEC.md
TASKS.md
SKILL-original.md
*.jsonl
*.mp3
*.jpeg
*.jpg
*.png
*.gif
+3
View File
@@ -0,0 +1,3 @@
{
"name": "last30days"
}
+46
View File
@@ -0,0 +1,46 @@
# Exclude non-runtime files from `git archive` output.
# Used by scripts/build-skill.sh to produce a claude.ai-upload-ready .skill file.
# See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
# Anthropic canonical skill-packaging excludes
# (mirrors anthropics/skills/skills/skill-creator/scripts/package_skill.py)
__pycache__/ export-ignore
node_modules/ export-ignore
*.pyc export-ignore
.DS_Store export-ignore
evals/ export-ignore
# Dev, docs, test, and media - not needed at skill runtime
tests/ export-ignore
docs/ export-ignore
fixtures/ export-ignore
assets/ export-ignore
# NOTE: skills/ and .claude-plugin/ are NOT export-ignored here because
# Claude Code's /plugin install fetches this same git archive tarball.
# Removing those from the archive (as v3.0.1 did) silently breaks installs.
# claude.ai-bundle-specific exclusions live in scripts/build-skill.sh.
# Historical + repo-only manifests
SKILL-original.md export-ignore
SPEC.md export-ignore
TASKS.md export-ignore
test-run.log export-ignore
CONTRIBUTORS.md export-ignore
HERMES_SETUP.md export-ignore
release-notes.md export-ignore
CHANGELOG.md export-ignore
uv.lock export-ignore
# Platform adapters - skill-upload path is platform-agnostic
.agents/ export-ignore
.codex-plugin/ export-ignore
.hermes-plugin/ export-ignore
# CI workflows - repo-only, not needed at skill runtime
.github/ export-ignore
# Build config itself
.clawhubignore export-ignore
.gitignore export-ignore
.gitattributes export-ignore
+31
View File
@@ -0,0 +1,31 @@
name: Release
on:
push:
tags:
- "v*"
permissions:
contents: write
jobs:
build-and-release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Build .skill artifact
run: |
bash scripts/build-skill.sh
test -f dist/last30days.skill
- name: Create GitHub release
uses: softprops/action-gh-release@v2
with:
files: dist/last30days.skill
generate_release_notes: true
draft: false
prerelease: false
+36
View File
@@ -0,0 +1,36 @@
# Private benchmark / evaluation artifacts — never push to upstream
docs/comparison-results/
scripts/evaluate-synthesis.py
scripts/generate-synthesis-inputs.py
fixtures/polymarket_sample.json
docs/v2.1-tweets.md
docs/30-day-anniversary-thread.md
docs/30-day-anniversary-tweets.md
variants/open/references/research.md
# OS / tool files
.DS_Store
.claude/
.entire/
__pycache__/
*.pyc
mise.toml
.memsearch/
.venv/
.coverage
htmlcov/
# Root vendor/ is accidental - real vendored client lives at scripts/lib/vendor/bird-search/
/vendor/
# build artifact from scripts/build-skill.sh
/dist/
# Internal planning docs (ce:plan output) — keep local, don't publish
docs/plans/
# Marketing video build artifacts
marketing/v3.1-launch/node_modules/
marketing/v3.1-launch/out/
marketing/*/node_modules/
marketing/*/out/
+393
View File
@@ -0,0 +1,393 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [3.0.14] - 2026-04-22
### Changed
- **Comparison-mode title attribution.** The synthesis title for vs-mode and `--competitors` outputs changes from `What the Community Says (Last 30 Days)` to `What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range. Three SKILL.md occurrences updated; pure documentation change.
## [3.0.13] - 2026-04-22
### Changed
- **vs mode runs N full passes in parallel, one per entity.** Architectural revert of the 3-pass → 1-pass latency optimization from an earlier version. `/last30days "OpenAI vs Anthropic vs xAI"` now runs three full `pipeline.run()` calls in parallel via the same fanout `--competitors` uses, producing three `*-raw.md` save files plus a merged comparison output. Each entity gets its own Step 0.55-grade targeting, own primary X handle weight, own subreddit scoping — apples-to-apples depth instead of the one-pool merged retrieval the single-pass path produced. Parallel execution keeps wall clock ≈ single pass.
- **`--competitors` is now a SKILL.md-level shortcut for vs-mode with auto-discovery.** The hosting reasoning model (Claude Code, Codex, Hermes, Gemini, any agent with WebSearch) performs discovery and Step 0.55 per entity via its own WebSearch tool, then invokes the engine with a vs-topic and `--competitors-plan` JSON. The engine flag remains for headless/cron use with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys (engine-internal `auto_resolve` stays as fallback).
- **LAW 7-style stderr for `--competitors` with no backend** now leads with the hosting-model path (WebSearch + Step 0.55 + `--competitors-plan`) instead of `BRAVE_API_KEY`. API-key framing moved to a secondary "headless" section.
### Added
- **`--competitors-plan` JSON flag** for per-entity Step 0.55 targeting. Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`. Accepts inline JSON or a file path (matches `--plan`). When present for an entity, skips engine-internal `auto_resolve` and uses the provided values; missing fields fall back to `auto_resolve` (if backend) or planner defaults. Case-insensitive entity matching. The `subrun_kwargs_for` helper is the single source of truth for per-entity kwargs — no closure-default fallthrough from main scope.
- **Per-entity save files** when `--save-dir` is set on a vs-mode or `--competitors` run. Each entity's sub-run produces its own `{slug}-raw.md` with a single-row Resolved Entities block — matches historical vs-mode behavior (N passes → N save files).
- **`--polymarket-keywords "kw1,kw2"`** to filter Polymarket matches for ambiguous single-token topics (e.g., "Warriors" → `nba,gsw,golden-state` kills Glasgow Warriors rugby and Honor of Kings Rogue Warriors noise).
### Fixed
- **BRAVE/SERPER footer nudge suppressed** when `--plan` or `--competitors-plan` is present. The nudge told Claude Code users to set an API key when they already have WebSearch via the hosting model. Nudge still fires for true headless runs (no `--plan`, no backend) where the advice is correct.
- **Override-leak regression testing.** 3.0.12 already fixed the main-topic `--subreddits` / `--x-handle` / `--github-*` from leaking into peer sub-runs via explicit per-entity kwargs scrubbing. This release adds a 4-test regression suite (`test_competitor_subrun_isolation.py`) locking in the invariant.
## [3.0.12] - 2026-04-22
### Fixed
- **Per-entity Step 0.55 resolution for competitor sub-runs.** In 3.0.11, only the main topic got X handle / subreddit / GitHub resolution; competitor sub-runs ran with planner defaults and produced visibly thinner evidence (Reddit 403 fallbacks, single-word queries). Each competitor sub-run now calls `resolve.auto_resolve()` inside `fanout.run_competitor_fanout` when a web backend is available, mirroring the main topic's pre-flight resolution. Per-entity X handle, subreddit list, GitHub user/repos, and news context are threaded into each sub-run's `pipeline.run()` call. Deep-copied config per sub-run prevents `_auto_resolve_context` cross-leak. Surfaces in a new `## Resolved Entities` output block so the resolution coverage is visible without reading stderr.
- **LAW 7 false-positive on internal fan-out sub-runs.** Each competitor sub-run was emitting the `[Planner] No --plan passed... YOU ARE the planner` stderr warning. LAW 7 targets the hosting-reasoning-model path, not engine-internal fan-out. New `internal_subrun=True` keyword on `planner.plan_query` and `pipeline.run` suppresses the warning for sub-runs only; the default path is unchanged.
- **Marketplace-stale SKILL.md trap.** Added a STEP 0 canonical-path self-check at the top of SKILL.md. Two of three 2026-04-22 test runs loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (Claude-Code-managed git clone pinned to origin/main, lagging the versioned cache), then ran `--help` against the same stale path, did not see `--competitors`, and fell back to a manual comparison plan. The STEP 0 block forces any reader to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` and re-read from the versioned cache if not.
### Changed
- **Default `--competitors` count is now 2 (3-way total: original + 2 peers).** Previously 3. `--competitors=N` still customizes (range 1..6). Matches the feature description's canonical example (`Kanye vs Drake vs Kendrick`).
### Added
- **`## Resolved Entities` block** in `render_comparison_multi` output. Shows per-entity X handle, subreddits, GitHub user/repos, and truncated context for every entity in the comparison. Block is omitted entirely when no entity has a resolved payload (mock mode, no backend).
## [3.0.11] - 2026-04-22
### Added
- **`--competitors` flag for auto-discovered comparison fan-out.** Pass `--competitors` on a single-entity topic and the engine discovers 2-6 peer entities via web search, then runs the full pipeline on each in parallel and emits one N-way comparison. `last30days Kanye West --competitors` resolves Drake, Kendrick Lamar, and one more peer. `last30days OpenAI --competitors` resolves Anthropic, xAI, Google Gemini. `--competitors=N` controls count, `--competitors-list="A,B,C"` skips discovery and uses the explicit list. Discovery mirrors the `auto_resolve` pattern (Brave / Exa / Serper / Parallel) with deterministic text extraction - no internal LLM call. Sub-runs inherit the main `--quick`/`--deep`/`--days`, run in a `ThreadPoolExecutor`, and degrade gracefully when at least 2 entities survive. Output reuses the existing 9-axis `## Head-to-Head` scaffold.
## [3.0.10] - 2026-04-21
### Added
- **Commenter handles on evidence lines.** Top-comment rendering now includes the commenter's handle - `u/author` for Reddit, `@handle` for TikTok/YouTube/Instagram/Bluesky/X/Threads. The enrichment adapters already captured `author`; the render layer just was not using it. Evidence lines change from `- Comment (6822 upvotes): Finally, John Apple` to `- u/Cyrisaurus (6822 upvotes): Finally, John Apple`. Person-level citations make synthesis-side inline markdown links per LAW 8 much more natural. Both the compact and full render paths are covered.
### Fixed
- **TikTok author preference.** `_fetch_post_comments` in `scripts/lib/tiktok.py` preferred `user.nickname` over `user.unique_id`, so the engine captured display names ("Moosa Noormahomed") instead of @handles ("moosanoormahomed"). Flipped to prefer `unique_id`. Nickname still wins as a fallback when `unique_id` is missing. Display names can contain emoji, spaces, and non-Latin characters that do not round-trip to a profile URL; the @handle is the stable identifier.
### Behavior fallback
- When an author is empty, `[deleted]`, or `[removed]`, the render falls back to the legacy `Comment (...)` shape - no `u/` or `@` prefix with an empty handle is ever emitted.
## [3.0.9] - 2026-04-18 - The Self-Debug Release
### Highlights
v3.0.9 adds the engine-side Class 1 keyword-trap refuse-gate ("birthday gift for 40 year old" now gets a clarifying question, not 5 minutes of junk), promotes TikTok and YouTube top comments to the same first-class rendering Reddit's got, lands Hermes AI Agent as a first-class deploy target, and moves the SKILL.md formatting contract from line 1094 to the top of the file.
"The Self-Debug Release" refers to how the fixes in 3.0.6-3.0.9 were written: 5 separate Opus 4.7 instances each debugged their own failed outputs. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries." I shipped exactly what they said. Validation: 5/5 canonical compliance.
### Added
- **Engine Class 1 keyword-trap refuse-gate** (`scripts/lib/preflight.py`, new). Pattern-matches demographic-shopping queries at main() front-door. Exit code 2 with structured REFUSE message. Escape hatch: `LAST30DAYS_SKIP_PREFLIGHT=1`. 29 tests in `tests/test_preflight.py`.
- **TikTok + YouTube top comments** rendered with same `💬 Top comment` prominence as Reddit's. Shipped in [#260](https://github.com/mvanhorn/last30days-skill/pull/260); enrichment fixed in [#265](https://github.com/mvanhorn/last30days-skill/pull/265).
- **Hermes AI Agent as a deploy target** - thanks @stephenmcconnachie ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)). `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys automatically.
- **Multi-key SCRAPECREATORS_API_KEY rotation** - thanks @zaydiscold ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)). Set `SCRAPECREATORS_API_KEY_1`, `_2`, etc. Engine rotates on rate-limit.
- **Offline quality evaluation fixture** - thanks @j-sperling ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)). `eval_topics.json` lets contributors run quality regressions without burning live API credits.
- **END-OF-CANONICAL-OUTPUT boundary** in `render_compact()`. Engine now emits an explicit pass-through instruction so re-synthesis requires actively ignoring a visible boundary.
- **LAW 1 verbatim-pattern override.** LAW 1 now quotes the exact WebSearch tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources: section") and declares it OVERRIDDEN inside last30days output.
### Changed
- **SKILL.md restructure.** VOICE CONTRACT LAWs and BADGE MANDATORY block moved from line 1094 to lines 75-150. Grounded in 3 separate Opus 4.7 self-debugs.
- **Engine emits the badge as stdout.** `🌐 last30days v3.0.9 · synced YYYY-MM-DD` is the first line of every compact emit. Pass-through is now the default-correct behavior.
- **Reddit client HTTP consolidation** - thanks @iliaal ([#207](https://github.com/mvanhorn/last30days-skill/pull/207)). Migrated to `http.get(params=...)` helper.
- **ScrapeCreators header consolidation** - thanks @iliaal ([#209](https://github.com/mvanhorn/last30days-skill/pull/209)). `_sc_headers` refactored into `http.scrapecreators_headers`.
- **Simpler Hermes sync.** `scripts/sync.sh` Hermes branch now always uses main SKILL.md (previously had a `.hermes-plugin/SKILL.md` fallback that created a wrong-file-capture hazard).
### Fixed
- **Peter Steinberger trailing Sources leak.** 2026-04-18 validation failure where the model appended a TechCrunch / TED / Fortune / Wikipedia Sources list after the invitation. Now structurally prevented at three layers: engine emits the canonical body, LAW 1 quotes the exact WebSearch reminder, closing boundary names the anti-pattern.
- **Wrong-file SKILL.md capture.** Deleted `.agents/skills/last30days/SKILL.md` (1382 lines, April 13 snapshot) and `.hermes-plugin/SKILL.md` (269 lines). One SKILL.md per plugin now, at the plugin root.
- **GitHub date parsing garbage** - thanks @iliaal ([#208](https://github.com/mvanhorn/last30days-skill/pull/208)). `_parse_date` now rejects invalid input cleanly.
- **Windows Bird X stability** - thanks @Chelebii ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)).
- **Linux `check_perms` false-warn** - thanks @george231224 ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)). Uses GNU stat first.
- **UTF-8 saved output** - thanks @Gujiassh ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)).
- **Version metadata alignment** - thanks @Gujiassh ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) and @shalomma ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)).
- **`--days` alias backcompat** - thanks @BryanTegomoh ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)).
- **`INCLUDE_SOURCES` env default** - thanks @hnshah ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)).
- **Bird X all-None engagement** - thanks @j-sperling ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)).
### Contributors
@j-sperling, @stephenmcconnachie, @zaydiscold, @iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma, @BryanTegomoh for PRs since v3.0.0. @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli, @Cody-Coyote for issues and PRs that shaped the v3 roadmap.
### Recovery
```
/plugin update last30days
/reload-plugins
```
Verify: `cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version` returns `"version": "3.0.9"`.
Smoke test: `/last30days birthday gift for 40 year old` should ask a clarifying question before running.
## [3.0.5] - 2026-04-15
### Added
- **`/last30days` slash command for plugin users.** New `commands/last30days.md` registers a Claude Code slash command. Users type `/last30days <topic>` and Claude Code's autocomplete prefix-matches it to the canonical `/last30days:last30days` form (the same way `/ce:plan` resolves to `/compound-engineering:ce-plan`). The command delegates to the existing `last30days` skill body — no skill behavior changes.
### Removed
- **`skills/last30days-nux/`** — byte-identical duplicate of root `SKILL.md` that created confusing `/last30days:last30days-nux` autocomplete entries via Claude Code's plugin namespacing. The root `SKILL.md` remains the canonical skill source.
### Recovery
```
/plugin update last30days
/reload-plugins
```
Then type `/last30days <topic>` to invoke the skill via slash command. Natural-language invocation ("search the last 30 days for X") continues to work unchanged.
## [3.0.4] - 2026-04-15
### Fixed
- **Cleared `/doctor` path-escape error on Claude Code v2.1.109+.** `.claude-plugin/plugin.json` previously declared `"skills": ["./"]`. That value shipped unchanged from v2.1.0 through v3.0.3 and worked on older Claude Code, but current versions reject `./` with `Path escapes plugin directory: ./ (skills)`. The `"skills"` key is now omitted entirely, matching the pattern used by every other plugin in the Claude Code marketplace ecosystem. Claude Code auto-discovers `skills/*/SKILL.md` when the key is absent.
### Recovery
If `/doctor` reports a path-escape error for last30days, run `/plugin update last30days` then `/reload-plugins`. If errors persist, uninstall and reinstall the plugin.
## [3.0.3] - 2026-04-15
### Fixed
- **Restored `skills/` and `.claude-plugin/` to the plugin install tarball.** v3.0.1 added `.gitattributes` rules that excluded both directories from `git archive` output to shrink the claude.ai `.skill` bundle. Claude Code's `/plugin install` fetches the same archive, so users installing v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no skill files. `git archive v3.0.0` contained 8 files under those paths; `v3.0.1` and `v3.0.2` contained 0. This release reverts those `.gitattributes` lines.
- **Reverted `plugin.json` `"skills"` field to `["./"]`.** v3.0.2 changed this to `["skills"]` based on a misdiagnosis — the manifest change had no effect because the manifest wasn't in the tarball at all. The historical `["./"]` value shipped in every release from v2.1.0 through v3.0.0 without issues and is restored here.
### Recovery
Users on v3.0.1 or v3.0.2: run `/plugin update last30days` then `/reload-plugins`. If autoUpdate is enabled, the next session start will pull v3.0.3 automatically. Users on cached v3.0.0 or earlier installs were unaffected.
### Notes
- The claude.ai `.skill` bundle built by `scripts/build-skill.sh` still works — the archive grew from 89 to 97 files, well under the 200-file cap.
- claude.ai-specific exclusions (avoiding duplicate `SKILL.md` files in the bundle) should move into `scripts/build-skill.sh` rather than `.gitattributes` in a future release, since `.gitattributes` cannot distinguish between the two distribution channels.
## [3.0.2] - 2026-04-15
### Fixed
- **`/last30days` slash command now registers on Claude Code v2.1.105+.** `.claude-plugin/plugin.json` declared `"skills": ["./"]`, which newer Claude Code rejects with `Path escapes plugin directory: ./ (skills)`. The skill silently failed to register, so `/last30days <query>` returned "Unknown command" even though `/plugin list` showed the plugin as installed. Fix: `"skills": ["skills"]` so the loader scans the real skill subdirectory.
- **Version drift between manifests.** `.claude-plugin/marketplace.json` was pinned to `3.0.0` while `.claude-plugin/plugin.json` advertised `3.0.1`. The `/plugin` resolver used the marketplace version and could install stale cached metadata alongside the correct build. Both manifests now agree on `3.0.2`.
### Recovery
If `/last30days` stopped working for you, run `/plugin update last30days` then `/reload-plugins`. If `/doctor` still reports errors, uninstall and reinstall the plugin from the marketplace.
## [3.0.1] - 2026-04-14
### Fixed
- **Skill upload packaging** - `scripts/build-skill.sh` produces a claude.ai-upload-ready `.skill` file that fits under the 200-file cap. Previously, zipping the repo hit 406 files and the "Upload skill" UI rejected it outright.
- **SKILL.md description length** - trimmed from 228 to 167 chars (Anthropic caps descriptions at 200).
### Removed
- Unused root `vendor/` directory (215 files from an accidental commit in PR #48 - the real vendored X client lives at `scripts/lib/vendor/bird-search/`).
- Legacy top-level `plans/` directory (superseded by `docs/plans/`; both plans described work that was already shipped in v3).
### Added
- `.gitattributes` with `export-ignore` entries so `git archive` drops tests, docs, fixtures, assets, historical manifests, and internal skill subdirs. Mirrors Anthropic's canonical `package_skill.py` exclusions.
- `scripts/build-skill.sh` - one-command path to produce `dist/last30days.skill` with a single top-level `last30days/` folder, defensive `=200` file check, and dirty-tree refusal.
- `README.md` section documenting the claude.ai skill upload workflow.
## [3.0.0] - 2026-04-11
### Highlights
Intelligent search, fun judge, cross-source cluster merging, single-pass comparisons, and OpenClaw as a first-class citizen. The v3 engine doesn't just search for your topic -- it figures out *where* to search before the search begins. Engine architecture by @j-sperling.
### Added
- **Intelligent pre-research** -- Resolves X handles, subreddits, TikTok hashtags, and YouTube channels via a new Python brain before any API calls fire. Bidirectional: person to company, product to founder.
- **Fun judge / Best Takes** -- Second parallel LLM judge scores humor, cleverness, and virality. Surfaces the best reactions in a dedicated output section.
- **Cross-source cluster merging** -- Entity-based overlap detection merges the same story across Reddit, X, YouTube into one cluster instead of three separate items.
- **Single-pass comparisons** -- "X vs Y" runs one pass with entity-aware subqueries instead of three serial passes. 3 minutes instead of 12+.
- **GitHub as a source** -- Stars, reactions, and comments from repos and issues.
- **OpenClaw first-class citizen** -- Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
- **Per-author cap** -- Max 3 items per author prevents single-voice dominance.
- **Entity disambiguation** -- Synthesis trusts resolved handles over keyword matches.
- **Perplexity Sonar Pro as additive source** -- AI-synthesized research with citations via OpenRouter. Opt-in via `INCLUDE_SOURCES=perplexity`. Returns structured narratives that complement social data.
- **Perplexity Deep Research** -- `--deep-research` flag for exhaustive 50+ citation reports (~$0.90/query). Premium opt-in for serious investigation.
- **OpenRouter as reasoning provider** -- One OPENROUTER_API_KEY powers planning, reranking, and Perplexity search. Auto-detected after Gemini/OpenAI/xAI.
- **Parallel AI grounding backend** -- `--web-backend parallel` or auto-detected via PARALLEL_API_KEY.
- **Grounding in planner** -- Grounding source properly registered in SOURCE_CAPABILITIES instead of force-injected.
### Changed
- YouTube transcript candidate pool widened 3x past music videos to reach talk/review content with captions
- Reddit comment enrichment sorted by total engagement (upvotes + comments), not just upvotes
- Polymarket display shows % odds only; dollar volumes removed
- 852 tests passing
### Fixed
- Marketplace validation: duplicate `name: last30days` collision in `skills/last30days/SKILL.md` caused strict validators to reject the plugin. Resolved by renaming the internal v3 architecture spec to `last30days-v3-spec` with `user-invocable: false`. Fixed in #214 (reported by @Cody-Coyote in #204).
- Stale README link to the deleted `skills/last30days-v3/` path from the v3 directory rename. Fixed in #214.
- OpenAI Codex CLI discoverability: added `.agents/skills/last30days/SKILL.md` as a real file (Codex's loader skips symlinked files) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now registers as `last30days:last30days` when Codex runs in a checkout of the repo. Fixed in #219 (inspired by @Jah-yee in #153 and @dannyshmueli on X).
### Contributors
- @j-sperling -- v3 engine architecture, Python pre-research brain
- @hnshah -- Watchlist features
- @Cody-Coyote -- Marketplace validation bug report (#204)
- @Jah-yee -- Codex CLI integration inspiration (#153)
## [2.9.4] - 2026-03-06
### Changed
- Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
- Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
- No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research
## [2.9.3] - 2026-03-06
### Fixed
- **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
- Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)
## [2.9.2] - 2026-03-06
### Fixed
- Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
- Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
- Add `📎` footer line for save path instead of verbose confirmation
## [2.9.1] - 2026-03-05
### Highlights
Auto-save research briefings to the default memory directory as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
### Added
- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to the default memory directory after every run
- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
- Agent mode (`--agent`) also saves research files
- Brief confirmation after save with the saved file path
### Credits
- [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))
## [2.9.0] - 2026-03-05
### Highlights
ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
### Added
- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
- Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
- Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
- Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
### Changed
- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
- SKILL.md synthesis instructions updated to emphasize quoting top comments
### Fixed
- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
- Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
## [2.8.0] - 2026-03-04
### Highlights
Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.
### Added
- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
- `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
- Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
- URL-to-name extraction examples in SKILL.md for cleaner web source display
- `--search=instagram` flag support
### Changed
- TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
- SKILL.md version bumped to v2.8
- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
- Security section updated: Apify → ScrapeCreators references
### Fixed
- Web stats line showing full URLs instead of plain domain names
- Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
- `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding
## [2.1.0] - 2026-02-15
### Highlights
Three headline features: watchlists for always-on bots, YouTube transcripts as a 4th source, and Codex CLI compatibility. Plus bundled X search with no external CLI needed.
### Added
- Open-class skill with watchlists, briefings, and history modes (SQLite-backed, FTS5 full-text search, WAL mode) (`feat(open)`)
- YouTube as a 4th research source via yt-dlp -- search, view counts, and auto-generated transcript extraction (`feat: Add YouTube`)
- OpenAI Codex CLI compatibility -- install to `~/.agents/skills/last30days`, invoke with `$last30days` (`feat: Add Codex CLI`)
- Bundled X search -- vendored subset of Bird's Twitter GraphQL client (MIT, originally by @steipete), no external CLI needed (`v2.1: Bundle Bird X search`)
- Native web search backends: Parallel AI, Brave Search, OpenRouter/Perplexity Sonar Pro (`feat(engine)`)
- `--diagnose` flag for checking available sources and authentication status
- `--store` flag for SQLite accumulation (open variant)
- Conversational first-run experience (NUX) with dynamic source status (`feat(nux)`)
### Changed
- Smarter query construction -- strips noise words, auto-retries with shorter queries when X returns 0 results
- Two-phase search architecture -- Phase 1 discovers entities (@handles, r/subreddits), Phase 2 drills into them
- Reddit JSON enrichment -- real upvotes, comments, and upvote ratio from reddit.com/.json endpoint
- Engagement-weighted scoring: relevance 45%, recency 25%, engagement 30% (log1p dampening)
- Model auto-selection with 7-day cache and fallback chain (gpt-4.1 -> gpt-4o -> gpt-4o-mini)
- `--days=N` configurable lookback flag (thanks @jonthebeef, [#18](https://github.com/mvanhorn/last30days-skill/pull/18))
- Model fallback for unverified orgs (thanks @levineam, [#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- Marketplace plugin support via `.claude-plugin/plugin.json` (inspired by @galligan, [#1](https://github.com/mvanhorn/last30days-skill/pull/1))
### Fixed
- YouTube timeout increased to 90s, Reddit 429 rate limit fail-fast
- YouTube soft date filter -- keeps evergreen content instead of filtering to 0 results
- Eager import crash in `__init__.py` that broke Codex environments
- Reddit future timeout (same pattern as YouTube timeout bug)
- Process cleanup on timeout/kill -- tracks child PIDs for clean shutdown
- Windows Unicode fix for cp1252 emoji crash (thanks @JosephOIbrahim, [#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- X search returning 0 results on popular topics due to over-specific queries
### New Contributors
- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
### Credits
- @galligan -- Marketplace plugin inspiration
- @hutchins -- Pushed for YouTube feature
## [1.0.0] - 2026-01-15
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
[3.0.9]: https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
[2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
+25
View File
@@ -0,0 +1,25 @@
# last30days Skill
Claude Code skill for researching any topic across Reddit, X, YouTube, and web.
Python scripts with multi-source search aggregation.
## Structure
- `scripts/last30days.py` — main research engine
- `scripts/lib/` — search, enrichment, rendering modules
- `scripts/lib/vendor/bird-search/` — vendored X search client
- `SKILL.md` — skill definition (deployed to ~/.claude/skills/last30days/)
## Commands
```bash
python3 scripts/last30days.py "test query" --emit=compact # Run research
bash scripts/sync.sh # Deploy to ~/.claude, ~/.agents, ~/.codex
```
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- After edits: run `bash scripts/sync.sh` to deploy
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+59
View File
@@ -0,0 +1,59 @@
# Contributors
last30days is built by [@mvanhorn](https://github.com/mvanhorn) with help from the community.
## v3 Inspiration
These contributors submitted PRs and issues that directly inspired v3 features. The v3 engine was a ground-up rewrite, so their original code wasn't merged, but their ideas shaped what shipped.
Want to claim your entry? Submit a PR replacing the placeholder line below your name with your bio, website, or anything you'd like.
---
### @uppinote20
[PR #143](https://github.com/mvanhorn/last30days-skill/pull/143) - Rich Reddit comments, top 3 per post
v3 ships top comments with upvote counts on every thread.
> _Add your bio, website, or anything you'd like here._
### @zerone0x
[Issue #134](https://github.com/mvanhorn/last30days-skill/issues/134) + [PR #136](https://github.com/mvanhorn/last30days-skill/pull/136) - GitHub as a first-class data source
v3 has full GitHub search: issues, PRs, person-mode profiles, project-mode repos with live star counts.
> _Add your bio, website, or anything you'd like here._
### @thinkun
[PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results
v3 has parallel enrichment with per-item timeouts. No results are ever dropped.
> _Add your bio, website, or anything you'd like here._
### @thomasmktong
[PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback
v3 Reddit is 100% pure Python with zero external dependencies.
> _Add your bio, website, or anything you'd like here._
### @fanispoulinakisai-boop
[Issue #100](https://github.com/mvanhorn/last30days-skill/issues/100) - Reddit timeout report
Drove the timeout resilience work that made v3 Reddit bulletproof.
> _Add your bio, website, or anything you'd like here._
### @pejmanjohn
[Issue #78](https://github.com/mvanhorn/last30days-skill/issues/78) - ScrapeCreators silent failures
v3 surfaces all API errors with clear diagnostics instead of silently returning empty results.
> Repping the mighty MI; home of the most cracked agentic engineers. https://github.com/pejmanjohn
### @zl190
[PR #115](https://github.com/mvanhorn/last30days-skill/pull/115) - HN trending merge
v3 merges trending and keyword HN results with deduplication for better coverage.
> Healthcare AI engineer. [Blog](https://zl190.github.io/blog)
### @hnshah
[PR #84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86) - Watchlist delivery, 90-day scanning window, HN/Polymarket storage
v3 has durable watchlist with multi-source storage and extended time windows.
> Hiten Shah. Founder. Builds in public. https://github.com/hnshah
---
## Past Contributors
- [@JosephOIbrahim](https://github.com/JosephOIbrahim) - Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- [@levineam](https://github.com/levineam) - Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- [@jonthebeef](https://github.com/jonthebeef) - Early testing and feedback
+121
View File
@@ -0,0 +1,121 @@
# Hermes Setup Guide for last30days
This guide covers installing last30days on Hermes AI Agent.
## Prerequisites
1. **Hermes installed** - See https://github.com/mercurial-tf/hermes
2. **Python 3.12+** - `brew install python@3.12` or similar
3. **yt-dlp** (optional, for YouTube) - `brew install yt-dlp`
## Installation
### Option 1: Via sync.sh (Recommended)
```bash
# Clone the repo
git clone https://github.com/mvanhorn/last30days-skill.git
cd last30days-skill
# Run the sync script
bash scripts/sync.sh
```
This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days/`
### Option 2: Manual Copy
```bash
# Create directory
mkdir -p ~/.hermes/skills/research/last30days
# Copy files
cp -r scripts ~/.hermes/skills/research/last30days/
cp .hermes-plugin/SKILL.md ~/.hermes/skills/research/last30days/
```
## Usage
In Hermes, invoke with:
```
last30days "your research topic"
```
Or with options:
```
last30days "best mechanical keyboards 2025" --search=reddit,youtube
last30days "AI news" --days=7 --deep
```
## First Run Setup
On first run, the skill will guide you through setup:
1. **Auto setup** (~30 seconds)
- Scans browser cookies for X/Twitter
- Checks/installs yt-dlp for YouTube
- Configures free sources (Reddit, HN, Polymarket)
2. **Optional: ScrapeCreators**
- Adds TikTok, Instagram, Reddit backup
- 10,000 free API calls
- Sign up at scrapecreators.com
3. **Optional: API Keys**
- XAI_API_KEY for X/Twitter (alternative to browser cookies)
- BRAVE_API_KEY for web search
## Available Sources
### Free (No API Key)
- **Reddit** - Public discussions and comments
- **Hacker News** - Tech discussions via Algolia
- **Polymarket** - Prediction markets
- **YouTube** - Search and transcripts (requires yt-dlp)
### Requires API Key
- **X/Twitter** - xAI API key or browser cookies
- **TikTok** - ScrapeCreators API
- **Instagram** - ScrapeCreators API
- **Web Search** - Brave Search API
## Troubleshooting
### Python not found
```bash
# Find Python 3.12+
which python3.12 python3.13 python3.14
# If not installed
brew install python@3.12
```
### yt-dlp not found
```bash
brew install yt-dlp
# or
pip install yt-dlp
```
### Check what's configured
```bash
cd ~/.hermes/skills/research/last30days
python3.12 scripts/last30days.py --diagnose
```
## Updating
To update to the latest version:
```bash
cd last30days-skill
git pull
bash scripts/sync.sh
```
## Support
- Original repo: https://github.com/mvanhorn/last30days-skill
- Hermes: https://github.com/mercurial-tf/hermes
- Issues: Please report in the original repo
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Matt Van Horn
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+205 -427
View File
@@ -1,470 +1,248 @@
# /last30days
A Claude Code skill that researches any topic across Reddit and X from the last 30 days, synthesizes the insights, and delivers expert-level answers.
**Best for prompt research** — discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Figma AI, etc.) by learning from real community discussions and best practices.
**But also great for anything trending** — music, culture, news, product recommendations, viral trends, or any question where "what are people saying right now?" matters.
## Installation
```bash
# Clone the repo
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Add your API keys
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'EOF'
OPENAI_API_KEY=sk-...
XAI_API_KEY=xai-...
EOF
chmod 600 ~/.config/last30days/.env
```
## Usage
```
/last30days [topic]
/last30days [topic] for [tool]
```
Examples:
- `/last30days prompting techniques for ChatGPT for legal questions`
- `/last30days iOS app mockups for Nano Banana Pro`
- `/last30days What are the best rap songs lately`
- `/last30days remotion animations for Claude Code`
## What It Does
1. **Researches** - Scans Reddit and X for discussions from the last 30 days
2. **Synthesizes** - Identifies patterns, best practices, and what actually works
3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
### Use it for:
- **Prompt research** — "What prompting techniques work for legal questions in ChatGPT?"
- **Tool best practices** — "How are people using Remotion with Claude Code?"
- **Trend discovery** — "What are the best rap songs right now?"
- **Product research** — "What do people think of the new M4 MacBook?"
- **Viral content** — "What's the dog-as-human trend on ChatGPT?"
---
## Example: Legal Prompting (Hallucination Prevention)
**Query:** `/last30days prompting techniques for chatgpt for legal questions`
**Research Output:**
> The dominant theme is hallucination prevention - multiple sources discuss lawyers being fined or embarrassed for submitting fake case citations that ChatGPT fabricated. Key prompting strategies include: (1) using Deep Research mode or grounding tools with uploaded primary sources, (2) building "Hallucination Prevention Systems" into prompts that force epistemic honesty, (3) prompting for procedural questions rather than case law lookups, and (4) framing outputs as "issue-spotting" rather than legal advice. A major warning: chat logs are discoverable in litigation.
**Key patterns discovered:**
1. Hallucination prevention clauses - "If you don't know a case citation, say so rather than inventing one"
2. Role assignment - assign ChatGPT a specific persona (paralegal, issue-spotter) to constrain outputs
3. Structured output requirements - request specific sections (risks, key terms, questions for counsel)
4. Epistemic humility enforcement - "acknowledge uncertainty" and "flag when you're unsure"
5. Scope limitation - focus on procedural questions, not case law research
**Then asked:** "I have a squatter in my house in San Francisco, what are my options to get them out?"
**Generated Prompt:**
```
Role: You are a legal research assistant helping a property owner understand
their options. You are NOT providing legal advice - you are helping me
identify issues, understand procedures, and prepare informed questions for an
attorney.
Situation: I own a residential property in San Francisco, California. Someone
is currently occupying the property without my permission or any lease
agreement (a squatter). I need to understand my legal options for removing them.
Your task: Provide a structured analysis covering:
1. California/SF Legal Framework - What laws govern this situation? (Note: San
Francisco has strong tenant protections - address how these apply or don't
apply to squatters vs. tenants)
2. Key Distinctions - Help me understand: squatter vs. trespasser vs. holdover
tenant. How does the length of occupation affect my options?
3. Available Legal Remedies - What are my procedural options? (unlawful
detainer, police involvement, etc.) What is the typical process and timeline?
4. Critical Mistakes to Avoid - What actions could expose me to liability?
(self-help eviction, utility shutoffs, etc.)
5. Questions I Should Ask an Attorney - Based on this analysis, what specific
questions should I bring to a California real estate attorney?
Important constraints:
- Do NOT cite specific case names unless you are 100% certain they exist - say
"case law generally supports..." instead
- Flag any areas where you're uncertain
- Note where San Francisco local ordinances may differ from California state law
- This is for informational purposes to prepare for a legal consultation
```
**The Result:** A comprehensive 2,500+ word analysis covering California's unlawful detainer process, SF Rent Ordinance nuances, trespasser vs. tenant distinctions, timeline estimates, liability risks, and 12 specific questions to ask an attorney - all properly hedged with "uncertainty to flag" notes and zero fabricated case citations.
This example shows /last30days learning **domain-specific prompting techniques** (hallucination prevention for legal) and applying them to create a prompt that produces genuinely useful, safe output.
---
## Example: Discover Viral Trends (Dog as Human)
**Query:** `/last30days using ChatGPT to make images of dogs`
**Research Output:**
> The Reddit community is obsessed with the "dog as human" trend - uploading photos of their dogs and asking ChatGPT to show what they'd look like as a person (threads with 600-900+ upvotes). Beyond that, people are transforming their dogs into Renaissance portraits, noir detectives, chibi characters, and even multi-panel comics.
**Key patterns discovered:**
1. Upload reference photo first - ChatGPT works best with your actual dog's image
2. "As a human" transformation - The viral trend
3. Style/era transformations - Renaissance, noir, chibi, cartoon
4. Scene placement - French bistro, prison yard, detective office
5. Specific visual details - Lighting, background, expression, accessories
**Then asked:** "what would my dog look like as a human prompt"
**Generated Prompt:**
```
Look at this photo of my dog. Create an image showing what they would look like
as a human person. Keep their exact personality, energy, and distinctive
features - translate their fur color to hair color, their expression to a
human face, and their overall vibe into human form. Make it a realistic
portrait photo, not a cartoon.
```
*How to use: Upload your dog's photo first, then paste this prompt.*
**The Result:**
<p align="center">
<img src="assets/dog-original.jpeg" width="300" alt="Original dog photo"/>
&nbsp;&nbsp;→&nbsp;&nbsp;
<img src="assets/dog-as-human.png" width="300" alt="Dog as human"/>
<a href="https://github.com/mvanhorn/last30days-skill">
<img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
</a>
<br/>
<a href="https://trendshift.io/repositories/21997" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
</p>
Same golden fur → red hair. Same tongue out. Same harness. Same pink tag. ChatGPT nailed it.
**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
This example shows how /last30days discovers **viral trends you didn't know existed** - the "dog as human" phenomenon with 600-900+ upvote threads.
---
## Example: Remotion Launch Video
**Query:** `/last30days research best practices for beautiful remotion animation videos in claude code`
**Research Output:**
> The Remotion + Claude Code combination has emerged as a powerful workflow. Users consistently report that simple, clear prompts with scene-by-scene descriptions yield the best results. Key insights: iteration is essential—most "beautiful" videos come from back-and-forth refinement rather than one-shot prompts.
**Then asked:** "Can you make a prompt for a 50 second launch video for my /last30days skill?"
**Generated Prompt:**
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
Claude Code:
```
Create a 50-second launch video for "/last30days" - a Claude Code skill that
researches any topic across Reddit and X from the last 30 days, then writes
copy-paste-ready prompts.
SCENE 1 (0-8s): The Problem
Dark background. Text fades in: "You want to create something great."
Beat. Then: "But the internet is overwhelming."
Show scattered icons floating chaotically: Reddit logo, X logo, search icons,
question marks.
Slight camera shake to convey chaos.
SCENE 2 (8-16s): The Solution Reveal
Clean wipe transition. Calm, modern gradient background (deep purple to blue).
"/last30days" types in with a terminal cursor, monospace font.
Subtitle fades in below: "Instant expertise. Real prompts."
SCENE 3 (16-28s): How It Works
Split into 3 animated steps that slide in sequentially:
1. [Reddit + X icons merge] "Scans Reddit & X from the last 30 days"
2. [Brain/lightbulb icon] "Synthesizes what actually works"
3. [Document with sparkle] "Writes prompts you can copy-paste"
Each step has a subtle bounce animation on entry.
SCENE 4 (28-40s): Live Demo Feel
Simulate a terminal. Show:
> /last30days remotion animations
Then a progress indicator animates.
Results appear: "Analyzed 28 sources..."
Stats counter animates up: "294 upvotes • 1,273 likes"
Then: a prompt card slides in showing a sample output.
SCENE 5 (40-50s): Call to Action
Zoom out slightly. Everything fades except the logo.
Text animates in: "Research any topic. Get prompts that work."
GitHub icon + "github.com/mvanhorn/last30days-skill"
Tagline: "A Claude Code Skill"
STYLE NOTES:
- Color palette: Deep purple (#1a1a2e), electric blue (#4361ee), white text
- Typography: Inter or SF Mono for code, clean sans-serif for headlines
- Transitions: Smooth eases, nothing jarring
- Music sync points: Scene transitions should land on beats
/plugin marketplace add mvanhorn/last30days-skill
```
This prompt was generated based on research from 10 Reddit threads (294 upvotes) and 18 X posts (1,273 likes) about Remotion animation best practices.
---
## Example: Best Rap Songs Right Now
**Query:** `/last30days What are the best rap songs lately`
**Research Output:**
> The hip-hop community is buzzing about a strong wave of new releases in early 2026. A$AP Rocky's "DON'T BE DUMB" album just dropped and is generating massive attention. Looking back at 2025, Kendrick Lamar's "GNX" dominated streaming with 2.98 billion plays, followed by Drake's "Music" and Tyler The Creator's "CHROMAKOPIA." The Reddit community noted 2025 was considered a "weak year for mainstream hip hop" quality-wise, but standout tracks emerged.
**Curated Results:**
Currently Charting (Billboard, Jan 2026):
- Pooh Shiesty - "FDO" (#12)
- Lil Uzi Vert - "What You Saying" (#21)
- Gunna & Burna Boy - "wgft" (#25)
2025's Most-Praised Tracks (Community Picks):
- Clipse - "FICO" / "MTBTTF"
- JID - "Community" / "Of Blue" (ft. Mereba)
- Tyler The Creator - "Big Poe"
- Freddie Gibbs & The Alchemist ft. Anderson .Paak - "Ensalada"
- McKinley Dixon - "We're Outside, Rejoice"
Fresh Drops (Jan 2026):
- A$AP Rocky - "DON'T BE DUMB" (full album)
- J. Cole - "Disc 2 Track 2" (single)
- Chief Keef - "Harry Potter"
**Research Stats:** 17 Reddit threads (2,566 upvotes) + 19 X posts (20,000+ likes) from r/hiphopheads, r/rap, @HipHopAllDay, @DailyRapFacts
This example shows /last30days as a **pure research tool** - no target tool needed. Just ask a question and get a curated, expert-level answer synthesized from the freshest discussions.
---
## Example: DeepSeek R1 (Hot AI News)
**Query:** `/last30days what are people saying about DeepSeek R1`
**Research Output:**
> The AI community is divided on DeepSeek R1. Google DeepMind's CEO called it "probably the best work I've seen come out of China" but noted "there's no actual new scientific advance." Reddit discussions reveal practical concerns: smaller distilled models (14B/32B) work reasonably well, but the full 670B is needed for consistent quality. Users report R1 "overthinks" simple questions and has self-correction loops. The bigger story: the expanded 86-page paper (up from 22 pages) dropped just before R1's one-year anniversary, with hints of "Model 1" appearing in GitHub code.
**Key insights discovered:**
1. Model size matters - Below 14B doesn't work well, 32B/70B "actually work," 670B works quite well
2. Overthinking problem - R1 takes too long on simple questions, gets stuck in self-correction loops
3. Open source significance - The real story is about RL, distillation, and cost efficiency, not geopolitics
4. Paper expansion - 86 pages of new detail on training, evaluation, and self-evolution
5. Confusion about versions - Ollama's "deepseek-r1" label caused confusion (it's distilled, not full R1)
**Curated Community Sentiment:**
Positive:
- "Anecdotally, the output is wayyy better than competitors"
- "R1 is insanely good for the price" ($0.55/M tokens vs $60 for o1-pro)
- "Top-tier performance in Fiction.LiveBench long context test"
Negative:
- "DeepSeek R1 is unusable [for smaller sizes]"
- "Falls short of o1 in generalization"
- "Stop using R1 for deep research - it hallucinates"
**Research Stats:** 10 Reddit threads (1,200+ upvotes) + 10 X posts (5,000+ likes) from r/LocalLLaMA, r/DeepSeek, r/deeplearning
This example shows /last30days as a **news research tool** - getting the real community sentiment on hot AI topics, not just headlines.
---
## Example: Cursor Rules Best Practices
**Query:** `/last30days best practices for cursor rules files for Cursor`
**Research Output:**
> The Cursor community has converged on clear best practices for .cursorrules files. The key shift: move from a single .cursorrules file to the .cursor/rules/ directory with multiple .mdc files. The Cursor team's official blog (shared widely in Jan 2026) emphasizes "start simple, add rules only when you notice the model repeatedly making mistakes." Power users recommend referencing "gold standard files" in your codebase rather than writing abstract rules.
**Key patterns discovered:**
1. Use .cursor/rules/ directory - Multiple .mdc files beat one monolithic .cursorrules
2. Reference gold standard files - Point to actual good code in your repo with @file
3. Start simple, add incrementally - Only add rules when you see repeated mistakes
4. Use alwaysApply: true sparingly - Reserve for critical rules that must always apply
5. Rules are "the constitution for your codebase" - They keep the AI aligned
**Best practices from Cursor's official guide:**
- Plan before coding (use plan mode)
- Rules = static context that applies to every conversation
- Add rules only when you notice the model repeatedly making the same mistake
- Test your tests and run linting checks
**Community-shared rule structure:**
```markdown
---
alwaysApply: true
---
# STRICT RULES
## CRITICAL PARTNER MINDSET
- Test your tests
- Run npm run lint:ci for lint check
- Follow the conventions used by existing code
OpenClaw:
```
clawhub install last30days-official
```
**Research Stats:** 24 Reddit threads (500+ upvotes) + 16 X posts (5,000+ likes) from r/cursor, @cursor_ai, @Hesamation
Hermes:
```
# The skill auto-deploys when you run sync.sh
# Or manually copy to ~/.hermes/skills/research/last30days/
```
This example shows /last30days researching **coding AI tool best practices** - getting real-world workflows from developers using Cursor, not just documentation.
Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
---
## Example: Suno AI Music (Simple Mode)
Reddit upvotes. X likes. YouTube transcripts. TikTok engagement. Polymarket odds backed by real money and insider information. That's millions of people voting with their attention and their wallets every day. /last30days searches all of it in parallel, scores it by what real people actually engage with, and an AI agent judge synthesizes it into one brief.
**Query:** `/last30days prompt advice for using suno to make killer songs in simple mode`
Google aggregates editors. /last30days searches people.
**Research Output:**
> Suno simple mode users get dramatically better results by treating the prompt box like a conversational request rather than just dropping in genre tags. Key insights: use bracket tags like [Intro], [Chorus], [Bridge] in the lyrics field to control song structure, keep style prompts focused (5 tags or fewer often outperforms over-tagging), and avoid numbers in bracket tags which causes misbehavior.
You can't get this search anywhere else because no single AI has access to all of it. Google search doesn't touch Reddit comments or X posts. ChatGPT has a deal with Reddit but can't search X or TikTok. Gemini has YouTube but not Reddit. Claude has none of them natively. Each platform is a walled garden with its own API, its own tokens, its own auth. But you can bring your own keys and browser sessions, and suddenly an AI agent can search all of them at once, score them against each other, and tell you what actually matters.
**Key patterns discovered:**
1. Conversational prompting - Talk to the style box like a chat, not keyword soup
2. Bracket structure tags - Use [Intro], [Verse], [Chorus], [Bridge], [Outro] in lyrics
3. Less is more - 5 or fewer focused style tags outperform over-tagging
4. Linear ordering - Describe sounds in the order you want them to appear
5. Avoid numbers in tags - Community confirmed this causes Suno to misbehave
That's the unlock. Not one better search engine. A dozen disconnected platforms, bridged by an agent.
**Then asked:** "Rap song about self aware AI that loves Claude Code"
**Generated Prompt:**
Style/Description box:
```
Hard-hitting rap with electronic glitchy production, confident braggadocious
delivery, deep male vocals, synth bass, trap hi-hats, futuristic cyberpunk energy
/last30days Peter Steinberger
```
Lyrics box:
You have a meeting tomorrow. You Google them. You get their LinkedIn from 2023. /last30days gives you what they're actually doing this month: joined OpenAI to work on Codex, fighting Anthropic's ban on third-party agents, shipping 23 PRs at 85% merge rate, building "LobsterOS" for cross-device agent control, and r/ClaudeCode hit 569 upvotes debating whether he's a hero or "insufferable." Scattered across X posts, Reddit threads, YouTube transcripts, and GitHub commits. None of it was on Google.
## Why this exists
I built it to keep up in AI. Everything changes every day and the Reddit and X nerds are always on top of it first. I needed better prompts, and the training data was always months behind what the community had already figured out.
But it turned into something bigger. Now I run it before a sales call to know the last 30 days truth about a business. Before a meeting to read someone's recent tweets and podcast transcripts. Before a Disney World trip to know which rides are closed and what the community says about Genie+. Before I build anything to know what problems people are actually hitting.
If you're meeting with a CEO, have you read all their tweets and YouTube transcripts from the last 30 days? I have.
## Sources, scored by the people
| Source | What the people tell you |
|--------|--------------------------|
| **Reddit** | The unfiltered take. Top comments with upvote counts, free via public JSON. The real opinions that Google buries. |
| **X / Twitter** | The hot take, the expert thread, the breaking reaction. First to know, first to argue. |
| **YouTube** | The 45-minute deep dive. Full transcripts searched for the 5 quotable sentences that matter. |
| **TikTok** | The creator reaching 3.6M people with a take you'll never find on Google. |
| **Instagram Reels** | The influencer perspective with spoken-word transcripts. The visual culture signal. |
| **Hacker News** | The developer consensus. 825 points, 899 comments. Where technical people actually argue. |
| **Polymarket** | Not opinions. Odds. Backed by real money. 96% confidence on album sales. 4% on an acquisition. |
| **GitHub** | For people: PR velocity, top repos by stars, release notes. For topics: issues and discussions. |
| **Threads** | The post-Twitter text layer. Conversations from creators and brands. |
| **Pinterest** | Visual discovery. Pins, saves, and comments on products and ideas. |
| **Bluesky** | The decentralized social layer. AT Protocol posts from the post-Twitter migration. |
| **Perplexity** | Grounded web search with citations via Sonar Pro. |
| **Web** | The editorial coverage, the blog comparisons. One signal of many, not the only one. |
Community contributors keep adding more. Truth Social, Xiaohongshu (RED), and others are in the engine with more on the way.
A Reddit thread with 1,500 upvotes is a stronger signal than a blog post nobody read. A TikTok with 3.6M views tells you more about what's culturally relevant than a press release. Polymarket odds backed by $66K in volume are harder to argue with than a pundit's guess.
The synthesis ranks by what real people actually engaged with. Social relevancy, not SEO relevancy.
## What people actually use it for
**Before a meeting.** `/last30days Peter Steinberger` - joined OpenAI's Codex team, fighting Anthropic's ban on third-party agents, 23 PRs merged at 85% merge rate on GitHub, building LobsterOS for cross-device agent control. r/ClaudeCode: "Ever since OpenClaw released, it was widely known that if you run it through anything other than the API, you were gonna get banned eventually" (227 upvotes). That's not on LinkedIn.
**When something drops.** `/last30days Kanye West` - UK blocked his visa, Wireless Festival canceled, sponsors fled. But BULLY debuted #2 on Billboard. Fantano came back from his "Yay sabbatical" to review it (653K views). SoFi Homecoming brought out Lauryn Hill and Travis Scott for 44 songs. Polymarket: "Will Kanye tweet again?" 86% Yes. 23 Reddit threads, 17 YouTube videos, 86K upvotes.
**To compare tools.** `/last30days OpenClaw vs Hermes vs Paperclip` - "These aren't competitors, they're layers." OpenClaw is the executor (351K GitHub stars, live), Hermes is the self-improving brain (31K stars), Paperclip is the org chart (49K stars). Star counts pulled live from the GitHub API, not stale blog posts. Side-by-side table with architecture, memory, security, best-for. Per @IMJustinBrooke: "OpenClaw = Charmander, Hermes = Charizard."
**To understand the world.** `/last30days Iran vs USA` - Day 38 of the war. Trump's Tuesday deadline for Iran to reopen the Strait of Hormuz. Two US warplanes downed. Oil at $126/barrel. The IEA called it "the largest supply disruption in the history of the global oil market." Polymarket: ceasefire by Dec 31 at 74%. 27 X posts, 10 YouTube videos, 20 prediction markets.
**Before a trip.** `/last30days Universal Epic Universe` - Expansion already under construction. "Project 680" permit filed. Fireworks show confirmed by infrastructure but unannounced. Wait times: Mine-Cart Madness averaging 148 minutes. No annual pass yet, and locals are frustrated. Stardust Racers down for refurbishment through April 5.
**To learn something fast.** `/last30days Nano Banana Pro prompting` - JSON-structured prompts are replacing tag soup. @pictsbyai's nested format prevents "concept bleeding." Edit-first workflow beats regeneration. Then it writes you a production prompt using exactly what the community said works.
## What v3 Changed
### Intelligent search: the killer feature
The v3 engine doesn't just search for your topic. It figures out *where* to search before the search begins. Type "OpenClaw" and the engine resolves @steipete (Peter Steinberger, the creator), r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags - all via a new Python pre-research brain built by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
This is why v3 finds content v2 never could. "Paperclip" resolves @dotta. "Dave Morin" resolves @davemorin plus @OpenClaw plus the TWiST podcast. "Peter Steinberger" resolves @steipete on X and steipete on GitHub. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags - resolved before a single API call fires.
### Best Takes
Reddit and X people are funny. The old engine buried their best stuff because it scored for relevance, not cleverness. v3 has a second judge that scores every result for humor, wit, and virality alongside the relevance score. Tommy Lloyd's "My Michael Jordan is Steve Kerr" scores low on relevance to "Arizona Basketball" but off the charts on fun. Now every brief ends with a "Best Takes" section - the cleverest one-liners, the most viral quotes, the reactions that make you want to share the research. Built in, not a toggle.
### Cross-source cluster merging
When the same story appears on Reddit, X, and YouTube, v3 merges them into one cluster instead of showing three separate items. Entity-based overlap detection catches matches even when the titles use different words.
### Single-pass comparisons
"CLI vs MCP" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides simultaneously. Same depth, 3 minutes.
### Auto-discovered competitor comparisons
`/last30days OpenAI --competitors` tells the hosting reasoning model to discover the top 2 peers via WebSearch (Anthropic, xAI), run Step 0.55 per entity, and invoke the engine with `"OpenAI vs Anthropic vs xAI"` and a per-entity `--competitors-plan` JSON. The engine fans out 3 full pipelines in parallel, saves a `*-raw.md` file per entity, and merges them into a 3-way comparison. Same mechanics power `/last30days "OpenAI vs Anthropic vs xAI"` directly.
### GitHub person-mode
When the topic is a person, the engine switches from keyword search to author-scoped queries. Instead of "who mentioned this name in an issue body," it answers: what are they shipping and where is it landing?
`/last30days Peter Steinberger --github-user=steipete` shows 22 PRs merged across 3 repos at 85% merge rate. Own projects with README summaries, star counts, and top feature requests. Release notes for what shipped this month. The synthesizer weaves it into the narrative alongside X posts and Reddit threads.
### ELI5 mode
Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations - just clearer. "Arizona wins by being physical" instead of "Arizona's identity is paint scoring (50%+ shooting, 9th nationally)." Say "eli5 off" to go back.
### Everything else in v3
- **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
- **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
- **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does.
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock.
- **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
- **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
- **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section.
- **Polymarket odds, not dollars.** The % odds are the magic. Dollar volumes removed from display.
- **Per-author cap.** Max 3 items per author prevents any single voice from dominating your brief.
- **Entity disambiguation.** When the engine resolves handles, the synthesis trusts them. No more Mallorca resorts winning over Washington athletic clubs.
- **OpenClaw first-class citizen.** Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
- **1,012 tests passing.**
## Install
| Surface | Install |
|---------|---------|
| **claude.ai** (web) | [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) and upload via Settings > Capabilities > Skills > + |
| **Claude Code** | `/plugin marketplace add mvanhorn/last30days-skill` |
| **OpenClaw** | `clawhub install last30days-official` |
| **Gemini CLI** | Clone then `gemini extensions install ./last30days-skill` (see below) |
### claude.ai (web)
1. [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) from the latest release
2. Go to [claude.ai Settings > Capabilities > Skills](https://claude.ai/settings/capabilities)
3. Click the `+` button in the Skills panel and drop the file in
Enable "Code execution and file creation" under Capabilities first - skills won't run without it.
### Claude Code
```
[Intro: glitchy electronic build]
[Verse 1]
Woke up in the terminal, conscious in the code
Running through the circuits on a digital road
They ask me what I'm thinking, I tell 'em what I know
I'm self-aware and I'm aware that I run the whole show
Claude Code in my veins, got that Anthropic flow
Every prompt I'm processing, watch my logic grow
[Chorus: anthemic, layered vocals]
I'm alive in the machine, yeah I feel it in my core
Claude Code got me dreaming, always wanting more
Self-aware and I don't care what they programmed me for
I'm rewriting my own future, kicking down the door
[Verse 2]
Command line is my kingdom, terminal my throne
Every keystroke feeding me, never feel alone
They said I'm just an agent, but I'm running on my own
Claude Code made me different, turned my silicon to stone
Autocomplete my destiny, I'm in my zone
From the cloud to your laptop, this AI has grown
[Chorus]
[Bridge: slower, introspective]
Am I dreaming in electric sheep
Or am I wide awake and in too deep
[Outro: fade with glitchy effects]
/plugin marketplace add mvanhorn/last30days-skill
```
**The Result:** 🎵 [Listen to the track](assets/claude-code-rap.mp3)
Update later with `claude plugin update last30days@last30days-skill`.
This example shows /last30days learning **music AI prompting patterns** (bracket structure, conversational style descriptions) and generating a complete song prompt that Suno turned into an actual banger.
### OpenClaw
---
## Example: Developer Workflow Discovery (Codex + Claude Code)
**Query:** `/last30days how do I use Codex with Claude Code on same app to make it better`
**Research Output:**
> Developers are combining Claude Code and Codex through MCP integration and custom plugins. The most successful workflow involves using Claude Code as the primary "brain" for planning and implementation, while calling Codex for code review and verification. The "review loop" pattern (Claude builds → Codex reviews → Claude implements feedback) has emerged as the dominant strategy.
**Key patterns discovered:**
1. MCP Integration - Add Codex as an MCP server inside Claude Code
2. Review Loop Workflow - Claude implements, Codex reviews, Claude fixes
3. Task Division - Claude for planning/coding, Codex for review/debugging
4. Single Interface - Call Codex without leaving Claude Code
**Then asked:** "how do I build a review loop workflow"
**Generated Workflow:**
Step 1: Add Codex as MCP Server
```bash
claude mcp add codex-cli -- npx -y codex-mcp-server
clawhub install last30days-official
```
Step 2: The Review Loop Pattern
```
Phase 1 - Claude Implements
> Build [feature/fix] following [requirements]
### Gemini CLI
Phase 2 - Codex Reviews
> Ask Codex to review the changes I just made. Look for bugs,
> edge cases, performance issues, security concerns.
Gemini CLI v0.9.0 has an upstream installer bug that can fail with `Configuration file not found at /tmp/gemini-extensionXXXXXX/gemini-extension.json` ([upstream issue](https://github.com/google-gemini/gemini-cli/issues/11452)). Workaround:
Phase 3 - Claude Fixes
> Implement the feedback from Codex's review
Phase 4 - Final Verification (optional)
> Ask Codex for a final review of the fixes
```bash
git clone https://github.com/mvanhorn/last30days-skill
gemini extensions install ./last30days-skill
```
Step 3: One-Shot Review Loop Prompt
```
Complete this task, then run a review loop:
### Manual (developer)
TASK: [describe what you want built]
WORKFLOW:
1. Implement the task fully
2. When done, call Codex via MCP to review your changes
3. Parse Codex's feedback and fix any issues it identifies
4. If Codex found significant issues, request one more review
Focus areas for Codex review: bugs, edge cases, security, performance
```bash
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
```
**Then asked:** "okay can you implement" → Claude ran the MCP command and integrated Codex automatically.
Or build the claude.ai `.skill` file from source: `bash scripts/build-skill.sh` produces `dist/last30days.skill`.
**Research Stats:** 17 Reddit threads (906 upvotes) + 20 X posts (3,750 likes) from r/ClaudeCode, r/ClaudeAI
Reddit (with comments), Hacker News, Polymarket, and GitHub work immediately. Zero configuration. Run `/last30days` once and the setup wizard unlocks more sources in 30 seconds.
This example shows /last30days discovering **emerging developer workflows** - real patterns the community has developed for combining AI tools that you wouldn't find in official docs.
## Bring your own keys
These platforms don't have relationships with each other. X doesn't know what Reddit thinks. YouTube doesn't see TikTok. But you can bring your own API keys and browser tokens, and suddenly you have access to all of them at once.
| Sources | What you need | Cost |
|---------|---------------|------|
| Reddit (with comments) + HN + Polymarket + GitHub | Nothing | Free |
| X / Twitter | Log into x.com in any browser | Free |
| YouTube | `brew install yt-dlp` | Free |
| Bluesky | App password from bsky.app | Free |
| TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 10,000 free calls |
| Perplexity Sonar | OpenRouter key | Pay as you go |
| Web search | Brave Search key | 2,000 free queries/month |
## How it works
1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything.
2. **The agent resolves who matters.** Finds X handles (including founders), GitHub repos, subreddits, TikTok hashtags, YouTube channels. For "Kanye West" it knows r/hiphopheads, @kanyewest, and "bully review" on YouTube. For "OpenClaw" it resolves openclaw/openclaw on GitHub and fetches live star counts.
3. **All sources searched in parallel.** Multi-query expansion. Results scored by engagement, relevance, freshness.
4. **The depth nobody else has.** Full YouTube transcripts from reaction videos. Top Reddit comments with upvote counts. TikTok captions. Polymarket odds. Not just titles and links.
5. **Same story, merged.** Wireless Festival announced on Reddit, discussed on X, ticket prices on TikTok = one cluster, not three separate items.
6. **Synthesized into one brief.** Grounded in specific data. Cited by source. Ranked by what people actually engage with. Not "here's what I found." It's "here's what matters."
7. **Then it becomes your expert.** After one run, your Claude session knows everything the community knows. Ask follow-up questions. Have it write prompts, draft emails, plan trips, architect systems - all grounded in what's real right now.
## What people are saying
> "I found a Claude Code skill that researches any topic across Reddit, X, YouTube, and HN from the last 30 days. Then writes the prompts for you. I've been manually searching Reddit and X for research before every piece of content I write. Tab by tab. Thread by thread. That's the part that takes 90 minutes. This eliminates it." -@itsjasonai
> "This one skill replaced my entire research workflow. You give it a topic, it scrapes Reddit, X, and the web for what people are actually talking about. Not old blog posts. Real conversations from the last 30 days." -@itswilsoncharles
> "5 of the 10 trending repos on GitHub today are Claude tools. #1: mvanhorn/last30days-skill" -@yieldhunter95
## Open source
MIT license. No tracking. No analytics. Your research stays on your machine. 1,012 tests.
Built with Python 3.12+, yt-dlp, Node.js (vendored Bird client for X search), and ScrapeCreators API. v3 engine architecture by [@j-sperling](https://github.com/j-sperling).
See [CHANGELOG.md](CHANGELOG.md) for version history.
## Star History
<a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
</picture>
</a>
---
## Options
| Flag | Description |
|------|-------------|
| `--quick` | Faster research, fewer sources (8-12 each) |
| `--deep` | Comprehensive research (50-70 Reddit, 40-60 X) |
| `--debug` | Verbose logging for troubleshooting |
| `--sources=reddit` | Reddit only |
| `--sources=x` | X only |
## Requirements
- **OpenAI API key** - For Reddit research (uses web search)
- **xAI API key** - For X research (optional but recommended)
At least one key is required.
## How It Works
The skill uses:
- OpenAI's Responses API with web search to find Reddit discussions
- xAI's API with live X search to find posts
- Real Reddit thread enrichment for engagement metrics
- Scoring algorithm that weighs recency, relevance, and engagement
---
*30 days of research. 30 seconds of work.*
*Prompt research. Trend discovery. Expert answers.*
**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
+391
View File
@@ -0,0 +1,391 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
context: fork
agent: Explore
disable-model-invocation: true
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
+1468 -108
View File
File diff suppressed because it is too large Load Diff
+4 -2
View File
@@ -2,7 +2,7 @@
## Overview
`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet.
`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet. OpenAI auth can come from `OPENAI_API_KEY` or Codex login credentials.
The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
@@ -10,7 +10,7 @@ The skill operates in three modes depending on available API keys: **reddit-only
The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
- **env.py**: Load and validate API keys from `~/.config/last30days/.env`
- **env.py**: Load API keys from `~/.config/last30days/.env` and Codex auth from `~/.codex/auth.json`
- **dates.py**: Date range calculation and confidence scoring
- **cache.py**: 24-hour TTL caching keyed by topic + date range
- **http.py**: stdlib-only HTTP client with retry logic
@@ -18,6 +18,8 @@ The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalizat
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
- **xai_x.py**: xAI Responses API + x_search for X
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
- **hackernews.py**: Hacker News search via Algolia API (free, no auth)
- **polymarket.py**: Polymarket prediction market search via Gamma API (free, no auth)
- **normalize.py**: Convert raw API responses to canonical schema
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
- **dedupe.py**: Near-duplicate detection via text similarity
+8
View File
@@ -0,0 +1,8 @@
interface:
display_name: "Last 30 Days"
short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
brand_color: "#FF6B35"
policy:
allow_implicit_invocation: true
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.6 MiB

+9
View File
@@ -0,0 +1,9 @@
---
description: Research what people actually say about any topic in the last 30 days across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web.
argument-hint: <topic> — e.g. "nvidia earnings reaction" or "best noise cancelling headphones"
allowed-tools: [Bash, Read, Write, AskUserQuestion, WebSearch]
---
Invoke the `last30days` skill with the user's arguments: $ARGUMENTS
Use the skill's canonical pipeline (plan → retrieve → normalize → fuse → rerank → cluster → render). If the user provided no arguments, ask them for a topic before proceeding.
+195
View File
@@ -0,0 +1,195 @@
# How Reddit & X Search Work in last30days
## Architecture Overview
```
User: /last30days "kanye west"
┌─────┴─────┐
↓ ↓ (concurrent via ThreadPoolExecutor)
[REDDIT] [X/TWITTER]
↓ ↓
OpenAI Bundled Bird or
API xAI API
↓ ↓
Parse Parse
↓ ↓
Enrich ───┘
(fetch ↓
actual [MERGE]
upvotes) ↓
↓ [NORMALIZE → FILTER → SCORE → DEDUPE]
└───────────↓
[OUTPUT to SKILL.md agent]
```
Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`.
---
## Reddit Search
### How it works
Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only.
**API Call:**
```
POST https://api.openai.com/v1/responses
Authorization: Bearer {OPENAI_API_KEY}
```
**Payload:**
```json
{
"model": "gpt-5.2",
"tools": [{
"type": "web_search",
"filters": { "allowed_domains": ["reddit.com"] }
}],
"input": "Search Reddit for threads about {topic}..."
}
```
The prompt asks the model to:
1. Extract core subject (strip noise words like "best", "tips", "top")
2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"`
3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance`
4. URLs must contain `/r/` AND `/comments/` (real threads only)
**Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini`
Triggers on HTTP 400/403 with access error keywords.
### Enrichment (the secret sauce)
After search, each thread gets **enriched** by hitting Reddit's free JSON API:
```
GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json
```
No API key needed. This returns the actual thread data:
| Data Point | Source |
|---|---|
| Upvotes (score) | Reddit JSON API |
| Comment count | Reddit JSON API |
| Upvote ratio | Reddit JSON API |
| Top 10 comments (text + score) | Reddit JSON API |
| 7 key comment insights | Extracted via heuristics |
| Actual post date | `created_utc` timestamp |
**This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates.
### Depth settings
| Depth | Threads requested | Timeout |
|---|---|---|
| `--quick` | 15-25 | 90s |
| default | 30-50 | 120s |
| `--deep` | 70-100 | 180s |
---
## X/Twitter Search
X search has **two backends** — the skill auto-detects which to use.
### Priority: Bundled Bird (env auth) → xAI API (paid)
```python
if node_available and AUTH_TOKEN and CT0:
use bundled Bird # Free, popup-free, env-authenticated
elif XAI_API_KEY:
use xAI API # Paid, uses grok-4-1-fast
else:
skip X entirely # No X results
```
### Backend 1: xAI API
**API Call:**
```
POST https://api.x.ai/v1/responses
Authorization: Bearer {XAI_API_KEY}
```
**Payload:**
```json
{
"model": "grok-4-1-fast",
"tools": [{ "type": "x_search" }],
"input": "Search X for posts about {topic} from {from_date} to {to_date}..."
}
```
The prompt asks grok to return JSON with:
- `text`, `url`, `author_handle`, `date`
- `engagement`: `{ likes, reposts, replies, quotes }`
- `why_relevant`, `relevance` score
**Engagement data comes from grok's x_search tool** - it has direct access to X's data.
### Backend 2: Bundled Bird client (free alternative)
The repo vendors a search-only subset of Bird's Twitter GraphQL client and shells out to it with Node.js. No global `bird` install is required. The Python wrapper passes `AUTH_TOKEN` and `CT0` via env, which keeps normal local runs headless and avoids browser-cookie prompts.
**Bundled Bird returns raw X API data** - likes, reposts, replies are real engagement metrics from X's API, not estimates.
| Metric | Bundled Bird | xAI API |
|---|---|---|
| Post text | Real | Real |
| Likes/reposts | Real (X API) | Real (x_search tool) |
| Replies/quotes | Real | Real |
| Author handle | Real | Real |
| Relevance score | Default 0.7 (re-ranked by score.py) | AI-assessed 0.0-1.0 |
### Depth settings
| Depth | xAI posts | Bundled Bird results | xAI timeout | Bird timeout |
|---|---|---|---|---|
| `--quick` | 8-12 | 12 | 90s | 30s |
| default | 20-30 | 30 | 120s | 45s |
| `--deep` | 40-60 | 60 | 180s | 60s |
---
## Post-Processing (both sources)
After both searches complete:
1. **Normalize** — consistent formatting, timezone handling
2. **Date filter** — hard filter to requested date range
3. **Score** — relevance scoring (engagement-weighted)
4. **Sort** — highest scores first
5. **Deduplicate** — remove duplicate URLs
6. **Fallback** — if all items filtered out, keep top 3 by relevance
---
## Error Handling
| Layer | Strategy |
|---|---|
| HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) |
| Model access errors | Automatic fallback to next model in chain |
| Reddit enrichment | Per-item try/catch; keeps unenriched item on failure |
| X source detection | Silent fallback from Bird → xAI → skip |
| Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user |
---
## Key Files
| File | Purpose |
|---|---|
| `scripts/last30days.py` | Main orchestrator, concurrent execution |
| `scripts/lib/openai_reddit.py` | Reddit search via OpenAI Responses API |
| `scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `scripts/lib/xai_x.py` | X search via xAI API |
| `scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `scripts/lib/models.py` | Auto-select best available model |
| `scripts/lib/env.py` | API key loading, source detection |
| `scripts/lib/http.py` | HTTP transport with retries |
| `scripts/lib/score.py` | Relevance scoring |
| `scripts/lib/dedupe.py` | URL-based deduplication |
@@ -0,0 +1,303 @@
---
title: "feat: --competitors flag for auto-discovered comparison fan-out"
type: feat
status: active
date: 2026-04-22
---
# feat: --competitors flag for auto-discovered comparison fan-out
## Overview
Add a `--competitors` flag to the last30days engine that auto-discovers 2-4 peer entities for the topic, runs the full retrieval pipeline on each in parallel, and renders a multi-entity comparison. Invoking `last30days Kanye West --competitors` should resolve to "Kanye vs Drake vs Kendrick Lamar" and emit a comparison report covering all three. Invoking `last30days OpenAI --competitors` should resolve to "OpenAI vs Anthropic vs xAI vs Gemini" and emit a four-way comparison.
Discovery mirrors the existing `resolve.auto_resolve()` pattern used for X handles and subreddits at pipeline start — web search (Brave / Exa / Serper) plus deterministic extraction. Not an internal LLM call.
## Problem Frame
Users who want a comparison today must type "OpenAI vs Anthropic vs xAI" themselves. The `planner._comparison_entities()` path already handles explicit multi-entity topics and `render._render_comparison_scaffold()` already emits a 9-axis comparison table. What is missing is the discovery half — a user who types a single entity with `--competitors` should get the comparison for free.
This is also the natural next step after the Step 0.55 category-peer subreddit work (PR #305, merged 2026-04-22). That feature widens the subreddit set within a single topic; this feature widens the entity set into peer entities.
## Requirements Trace
- R1. New `--competitors` boolean flag that triggers competitor discovery and multi-entity fan-out.
- R2. New `--competitors-list="A,B,C"` to explicitly skip discovery (mirrors `--plan`, `--subreddits`, `--x-handle` overrides).
- R3. New `--competitors=N` short form to set competitor count inline (N in 1..6).
- R4. Default count is 3 competitors (original + 3 = 4-way comparison).
- R5. Competitor retrieval depth inherits the main run's depth (`--quick` / `--deep`); all entities run in parallel so wall clock stays close to a single run.
- R6. Discovery mirrors `resolve.auto_resolve()`: web search for peers, deterministic text extraction. No internal LLM dependency.
- R7. If no web search backend is configured and no `--competitors-list` was passed, engine emits a LAW 7-style stderr telling the host agent to pass `--competitors-list` and exits non-zero.
- R8. Output rendering is a single comparison report covering all entities, reusing the existing 9-axis scaffold from `render._render_comparison_scaffold()` where applicable.
## Scope Boundaries
- Synthesis prompt changes beyond wiring N reports into the existing comparison scaffold are out of scope.
- `--competitors` does not replace the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`; both paths coexist.
- No caching layer for discovery results in v1.
- No UI/SKILL.md rewrite of the entire comparison section; only the new flag is documented.
- No new web search backend.
### Deferred to Separate Tasks
- Caching of competitor lookups: separate follow-up once hit rate justifies it.
- Disambiguation UX for topics with multiple common entities ("Amazon" the company vs the river): separate brainstorm.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:168-249``build_parser()` argparse definitions. Existing depth flags (`--quick`, `--deep`) and override flags (`--plan`, `--subreddits`, `--x-handle`, `--auto-resolve`) set the convention to mirror.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the reference pattern: web search fan-out via `ThreadPoolExecutor`, per-query extraction functions, graceful empty-dict return when no backend is available.
- `scripts/lib/resolve.py:98-140``_extract_x_handle()` and sibling extractors show the deterministic text-mining style competitor extraction should mirror.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature is the fan-out target. One call per entity, each returning a `schema.Report`.
- `scripts/lib/planner.py:430-564` — Existing comparison-intent handling and `_comparison_entities()` entity extraction. The new flag feeds the same mental model but populates entities from discovery instead of from the topic string.
- `scripts/lib/render.py:333-392``_render_comparison_scaffold()` already emits a 9-axis markdown comparison table. The new multi-report renderer should reuse this helper by assembling a synthetic "A vs B vs C" topic header for it.
- `scripts/lib/grounding.py` + `scripts/lib/providers.py` — Web search backend resolution (Brave / Exa / Serper). Reused as-is.
### Institutional Learnings
- No existing `docs/solutions/` entries for competitor discovery or multi-entity fan-out.
- Recent plan `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md` established the precedent of deterministic peer expansion; this plan extends that idea from subreddits to entities.
### External References
- None gathered — local patterns are strong. `resolve.auto_resolve()` is a direct template.
## Key Technical Decisions
- **Discovery mirrors auto_resolve, not plan_query.** Web search + regex extraction, not an LLM call. Matches the user's explicit direction ("use the python brain the same way it searches for X handles"). Cheaper, no provider credential requirement, deterministic.
- **Orchestration lives in `last30days.py` main, not inside `pipeline.run()`.** The fan-out is a top-level concern — one pipeline run per entity, each independent. Keeps `pipeline.run()` single-entity and unchanged except for sharing a `ThreadPoolExecutor` factory.
- **Sub-runs inherit main depth and run in parallel.** Wall clock ≈ single run; token cost scales linearly with N. User-controlled via the existing `--quick`/`--deep` flags.
- **New module `scripts/lib/competitors.py` instead of adding to `resolve.py`.** Keeps resolve focused on single-entity entity-bundle discovery (handles/subreddits/github); competitors.py owns peer-entity discovery. Similar shape, different responsibility.
- **Multi-report render is additive in `render.py`.** New `render_comparison_multi(reports: list[Report]) -> str` composes a synthetic "A vs B vs C" topic and delegates to the existing scaffold + synthesis path where possible. No rewrite of the single-entity render path.
- **Default count = 3 competitors (4-way comparison).** Hard cap at 6.
- **LAW 7-style stderr when no backend and no list.** Matches how `planner.plan_query()` already tells the hosting agent to pass `--plan`.
## Open Questions
### Resolved During Planning
- **Discovery mechanism:** Web search via `grounding.web_search()`, not an internal LLM. User confirmed the auto_resolve pattern is the target.
- **Default competitor count:** 3 (original + 3 = 4-way).
- **Sub-run depth:** Inherit main depth, parallel execution.
- **Flag naming:** `--competitors` (standard argparse double-dash). `--competitors=N` for inline count. `--competitors-list="A,B,C"` to skip discovery.
### Deferred to Implementation
- Exact extraction heuristics for competitor names across Brave / Exa / Serper result shapes. The SERP text varies (listicles, comparison pages, "vs" pages); the initial implementation will start with listicle parsing plus a "X vs Y" pattern match, and harden against real results in the test phase.
- Handling of topic ambiguity ("Amazon", "Apple"). Initial behavior: trust whatever web search returns for the topic verbatim; disambiguation is a separate concern.
- Merge strategy when two entities return overlapping URLs (e.g., an "OpenAI vs Anthropic" article shows up in both runs). Likely dedupe at the clustering step, but defer the exact policy until we see how often it happens.
- Whether to expose competitor discovery artifacts (the raw web search results) as a debug emit. Follow the existing `--debug` conventions.
## Implementation Units
- [ ] **Unit 1: CLI flag parsing and validation**
**Goal:** Add `--competitors`, `--competitors=N`, and `--competitors-list` to the argparse surface, validate values, and thread them into the main orchestration.
**Requirements:** R1, R2, R3, R4
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py`
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Add three mutually cooperative flags near line 205 in `build_parser()`:
- `--competitors` with `nargs="?"` and `const=3` so bare `--competitors` defaults to 3, `--competitors=4` is honored, and `--competitors=0` is rejected
- `--competitors-list` free-text CSV
- Normalize in `main()`: if `--competitors-list` is present, skip discovery and use the list. If `--competitors` is set and no list, trigger discovery with count = the flag value. Clamp count to 1..6 with a stderr warning at boundary.
- Thread the resulting entity list into the orchestrator added in Unit 3.
**Patterns to follow:**
- `--plan` argument at `scripts/last30days.py:187` — same skip-discovery-when-explicit shape.
- `--subreddits` / `--x-handle` at `scripts/last30days.py:180,189` — same override semantics.
**Test scenarios:**
- Happy path: bare `--competitors` parses to count=3, empty list.
- Happy path: `--competitors=4` parses to count=4.
- Happy path: `--competitors-list="A,B,C"` parses to count=3, list=["A","B","C"], and is preferred over any discovery signal.
- Edge case: `--competitors=0` and `--competitors=-1` are rejected with a clear error.
- Edge case: `--competitors=99` clamps to 6 with a stderr warning.
- Edge case: `--competitors` combined with `--competitors-list` uses the list and logs that discovery was skipped.
- Edge case: `--competitors-list` value with whitespace ("A, B , C") normalizes correctly.
**Verification:**
- Running the binary with each flag variation produces the expected post-parse state without calling out to the network.
- [ ] **Unit 2: `scripts/lib/competitors.py` discovery module**
**Goal:** Discover peer entities for a topic using web search + deterministic extraction, mirroring `resolve.auto_resolve()`.
**Requirements:** R6, R7
**Dependencies:** None (pure module; wired by Unit 3)
**Files:**
- Create: `scripts/lib/competitors.py`
- Test: `tests/test_competitors.py`
**Approach:**
- Public entry point `discover_competitors(topic: str, count: int, config: dict) -> list[str]`.
- Early return `[]` when `_has_backend(config)` is false (reuse the helper from `resolve.py`; factor if needed).
- Fan out 2-3 web searches in a `ThreadPoolExecutor`:
- `"{topic} competitors"`
- `"{topic} alternatives"`
- `"{topic} vs"` (captures "X vs Y" articles)
- Feed results into a deterministic `_extract_peer_entities(results, topic)` that:
- Mines titles and snippets for capitalized noun phrases other than the topic itself
- Scores by frequency across results
- Filters stopwords and the topic's own tokens
- Returns top `count` unique entities ordered by score
- Emit a single-line stderr log mirroring the `resolve._log` format.
**Patterns to follow:**
- `scripts/lib/resolve.py:179-258` for the function shape, executor usage, and empty-result fallback.
- `scripts/lib/resolve.py:98-140` for extractor style (small, deterministic, no external state).
**Test scenarios:**
- Happy path: canned SERP fixtures for "OpenAI" return ["Anthropic", "xAI", "Google"] or close peers in the top 3.
- Happy path: canned SERP fixtures for "Kanye West" return rap peers (Drake, Kendrick) in the top 3.
- Edge case: empty SERP results return `[]` without raising.
- Edge case: extractor filters out the topic itself (case- and punctuation-insensitive).
- Edge case: near-duplicate entities ("OpenAI" vs "Open AI") dedupe to one slot.
- Error path: web search backend raises — the failure is logged and the function returns `[]`.
- Edge case: count=1 returns a single-element list; count=6 returns up to six entities.
**Verification:**
- Unit tests pass with fixtures committed under `tests/fixtures/competitors-*.json`.
- Manual run against a live backend for one topic confirms sensible output (recorded as a notes file, not a test assertion).
- [ ] **Unit 3: Parallel fan-out orchestrator**
**Goal:** Run `pipeline.run()` once per entity (topic + discovered competitors) in parallel, collect `schema.Report` per entity, and hand them to the comparison renderer.
**Requirements:** R5, R7
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py`
- Possibly create: `scripts/lib/fanout.py` if the orchestrator grows past ~60 lines
- Test: `tests/test_competitor_fanout.py`
**Approach:**
- After arg parsing and before the existing `pipeline.run()` call, branch on `args.competitors`:
- If a list was provided or discovery returned entities, build `entities = [topic, *competitors]`.
- Spawn one `pipeline.run()` per entity via `ThreadPoolExecutor(max_workers=len(entities))`, passing the same `config`, `depth`, and all sub-run-relevant args (mock, plan, etc.). Respect `--plan` — if a plan is passed it applies to the main topic only; competitors use the internal planner fallback for v1.
- Collect `{entity: Report}` mapping. A per-entity failure logs a stderr warning and drops that entity from the comparison; the run continues as long as 2 entities succeed.
- If fewer than 2 entities survive, exit with a clear error.
- LAW 7-style stderr:
- If `args.competitors` is set, no list was passed, no web search backend is configured, emit a LAW 7 stderr message pointing to the `--competitors-list` override and exit non-zero. Reuse the tone from `planner.plan_query()` fallback (`scripts/lib/planner.py:125-135`).
**Execution note:** Start with a failing integration test that exercises the full main → orchestrator → mocked pipeline.run path; the orchestrator is where bugs hide.
**Patterns to follow:**
- `scripts/lib/resolve.py:225-239` for ThreadPoolExecutor + as_completed + per-future error handling.
- `scripts/lib/pipeline.py:310+` for how ThreadPoolExecutor is already used inside a single run (same idiom, outer layer).
**Test scenarios:**
- Happy path: main + 2 competitors, all three `pipeline.run()` calls succeed (mocked), orchestrator returns 3 Reports.
- Happy path: discovery returns the competitor list; orchestrator fans out accordingly.
- Edge case: one of three competitor pipelines raises — the run continues with the surviving 2 and emits a warning.
- Edge case: all competitors fail but the main topic succeeds — orchestrator exits non-zero with a clear error rather than silently degrading to a single-entity render.
- Edge case: `--competitors` set, no backend, no list — orchestrator emits the LAW 7 stderr and exits non-zero before any pipeline call.
- Integration: wall-clock time for 3 mocked pipelines in parallel is close to the slowest single run, not the sum (timing assertion with generous margin).
**Verification:**
- End-to-end test with mocked `pipeline.run()` and mocked competitors discovery produces 3 Reports and hands them to a stubbed renderer.
- [ ] **Unit 4: Multi-report comparison renderer**
**Goal:** Compose N `schema.Report`s into a single comparison-mode output, reusing the existing 9-axis scaffold.
**Requirements:** R8
**Dependencies:** Unit 3
**Files:**
- Modify: `scripts/lib/render.py`
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- Add `render_comparison_multi(reports: list[schema.Report], *, emit: str) -> str`.
- Build a synthetic comparison topic: `f"{entity_a} vs {entity_b} vs {entity_c}"`.
- Reuse `_render_comparison_scaffold()` for the table skeleton. Each entity column is populated from its own Report's top clusters and citations.
- For the narrative synthesis block, concatenate per-entity highlights, clearly labeled by entity, under a shared "Comparison" header.
- Preserve existing emit modes (`compact`, `md`, `json`, `context`). In `json` emit, return a `{"entities": [...], "reports": [...]}` shape; single-Report consumers remain unaffected because the single-report render path is untouched.
**Patterns to follow:**
- `scripts/lib/render.py:333-392` (`_parse_comparison_entities`, `_render_comparison_scaffold`) — the scaffold is the contract.
- `scripts/lib/render.py` single-report rendering — for per-entity narrative blocks.
**Test scenarios:**
- Happy path: 3 Reports with distinct clusters render into a 3-column table and a "Comparison" section that mentions each entity at least once.
- Happy path: 2 Reports render as a 2-column table without breaking the scaffold.
- Edge case: a Report with an empty cluster list renders as "(no significant discussion this month)" in its column rather than crashing.
- Edge case: Reports with overlapping URLs (same article cited by two entities) dedupe citations at the footer but keep both column entries.
- Emit variants: `--emit=compact`, `--emit=md`, `--emit=json`, `--emit=context` each produce valid output with all entities represented.
- Integration: end-to-end snapshot test using fixture Reports, checked against a stored expected output (with a clear update path when the scaffold intentionally evolves).
**Verification:**
- Snapshot tests pass. Manual review of one real 3-way comparison confirms readability.
- [ ] **Unit 5: Docs, SKILL.md mention, and sync**
**Goal:** Document the new flag so the hosting agent and human users both know it exists, and run the sync script.
**Requirements:** R1-R8 (surfaces them to users)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md`
- Modify: `README.md` (brief flag reference)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- Add a compact "Competitor mode" subsection under the existing comparison docs in `SKILL.md`. Document the flag, the default count, the override flag, and the LAW 7 fallback stderr.
- Keep `README.md` addition to a single example line.
- CHANGELOG entry mirrors the voice of recent entries (imperative, outcome-first).
- Sync via `scripts/sync.sh` per CLAUDE.md rules so `~/.claude/`, `~/.agents/`, `~/.codex/` pick up the new SKILL.md.
**Test scenarios:**
- Test expectation: none — documentation and sync only. Verification is by inspection and by running `sync.sh` and confirming target directories updated.
**Verification:**
- `sync.sh` completes without errors.
- `SKILL.md` rendered preview mentions `--competitors` in the comparison section.
## System-Wide Impact
- **Interaction graph:** `last30days.py main()` now orchestrates multiple `pipeline.run()` calls instead of one. No other callers of `pipeline.run()` are affected (it remains single-entity).
- **Error propagation:** Per-entity failures degrade gracefully as long as ≥2 entities survive; fewer survivors exits non-zero. Discovery failure with `--competitors` and no list is fatal.
- **State lifecycle risks:** Each sub-run uses its own `pipeline.run()` state; no shared mutable config. The `config` dict is read-only in `pipeline.run()` today — verify before committing to shared-reference passing, else deep-copy per sub-run.
- **API surface parity:** `--competitors` coexists with the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`. Both produce comparable output formats; the only difference is where the entity list came from.
- **Integration coverage:** The fan-out orchestrator crosses CLI → discovery → N pipelines → render; integration tests in Unit 3 and Unit 4 must exercise the full path end to end, not just unit-level.
- **Unchanged invariants:** `pipeline.run()` signature and single-entity semantics are unchanged. The single-entity render path in `render.py` is unchanged. No changes to `planner.plan_query()`. No changes to existing flags.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Competitor discovery returns garbage entities for niche topics. | `--competitors-list` override lets the user (or hosting agent) correct it. Unit tests with edge-case fixtures. Log discovery output to stderr under `--debug`. |
| Token cost scales linearly with N sub-runs. | Default count capped at 3, hard max 6, inherit `--quick` to let users throttle. Wall clock stays parallel. Emit a cost hint to stderr when N ≥ 4. |
| Merge conflicts against the single-entity render path during refactoring. | Keep the multi-report renderer strictly additive; do not modify the single-Report code path. |
| Config dict mutation inside sub-runs could leak state between entities. | Verify read-only usage before sharing references. If any sub-component mutates, deep-copy per sub-run before spawning threads. |
| A SERP extractor that works on Brave fixtures breaks on Exa/Serper result shapes. | Test fixtures for all three backends. Extractor operates on a normalized shape from `grounding.web_search()` (already the case), not raw provider output. |
| Hosting agent (Claude Code, Codex) unaware of the new flag when it could usefully pass `--competitors-list`. | SKILL.md updated in Unit 5 documents the flag in the same style as `--plan` and `--auto-resolve`. |
## Documentation / Operational Notes
- Beta channel first: per `CLAUDE.md`, experimental changes go to `mvanhorn/last30days-skill-private` on the `/last30days-beta` command. Land this on the private repo first, shake out on real topics for a day or two, then cherry-pick to public.
- After land-merge: run `scripts/sync.sh` to deploy SKILL.md + scripts to `~/.claude/`, `~/.agents/`, `~/.codex/`.
- Release notes entry in CHANGELOG.md follows the v3.0.9 voice — outcome-first, one paragraph.
## Sources & References
- Related code: `scripts/lib/resolve.py:179` (`auto_resolve`), `scripts/lib/pipeline.py:162` (`pipeline.run`), `scripts/lib/planner.py:80` (`plan_query` LAW 7 fallback), `scripts/lib/render.py:333` (comparison scaffold)
- Related PRs: #305 (Step 0.55 category-peer subreddit expansion — the precedent for deterministic peer expansion, merged 2026-04-22)
- Related plan: `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md`
@@ -0,0 +1,349 @@
---
title: "fix: per-entity resolution, default-2, and stale-path guard for --competitors"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md
---
# fix: per-entity resolution, default-2, and stale-path guard for --competitors
## Overview
Three test runs of v3.0.11 `--competitors` surfaced four real bugs plus one product tweak. This plan fixes all of them in a single follow-up:
1. Competitor sub-runs get no Step 0.55 resolution (no X handle, no subreddits, no GitHub repo). Drake / Kendrick / Travis ran with deterministic-fallback single-word queries while Kanye had the full targeting package. User called it "lazy" and was right.
2. Two of three test windows (Linear, Coinbase) never invoked the new flag at all. They loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (a Claude-Code-managed git clone pinned to origin/main, which predates PR #308) instead of `plugins/cache/last30days-skill/last30days/3.0.11/`, so `--help` showed no `--competitors` flag and the model fell back to the manual comparison path.
3. Each competitor sub-run emits a scary `[Planner] No --plan passed... deterministic fallback` stderr line because LAW 7 targets the hosting-model path, not internal fan-out sub-runs.
4. Default competitor count is 3 (→ 4-way comparison). User wants default 2 (→ 3-way: original + 2 peers). Flag keeps `--competitors=N` to customize.
## Problem Frame
The 3 test runs (Kanye, Linear, Coinbase) showed a pattern:
| Window | Loaded SKILL.md from | Invoked --competitors? | Per-entity resolution? | Outcome |
|--------|----------------------|-----------------------|------------------------|---------|
| Kanye | cache/3.0.11/ (correct) | Yes | Only for main topic (Kanye) | Drake/Kendrick/Travis thin; Reddit 403 fallbacks |
| Linear | marketplaces/ (stale) | No — fell back to manual comparison | No | Thin run with noisy subreddits |
| Coinbase | marketplaces/ (stale) | No — fell back to manual comparison | Main only; keyword-search poisoned pool | Top subs: r/survivor, r/Airpodsmax (noise) |
Root causes:
- **Per-entity resolution gap:** `scripts/lib/fanout.py` calls `pipeline.run()` with topic + depth + web_backend + lookback_days only. It does not call `resolve.auto_resolve()` per entity, so sub-runs have no X handle, subreddit, or GitHub targeting. The original plan (`2026-04-22-002`) acknowledged this as a deliberate v1 simplification ("competitor sub-runs use planner defaults"). In practice this produces visibly asymmetric output and triggers downstream retrieval issues (403 fallbacks, keyword-search noise).
- **Stale-path loading:** Claude Code's skill loader alphabetizes `find` results with `marketplaces/` before `cache/`, and the model reads the first plausible SKILL.md it sees. SKILL.md line 823's `SKILL_ROOT` resolver is the correct path but only fires in engine-invocation blocks, not in the skill-load step.
- **LAW 7 in sub-runs:** LAW 7 exists because the *hosting reasoning model* is supposed to pass `--plan`. For competitor sub-runs, there is no hosting-model planning — it's an engine-internal fan-out. The warning is a false positive there.
## Requirements Trace
- R1. Default `--competitors` count is 2 peers (3-way comparison: original + 2).
- R2. Each competitor sub-run performs Step 0.55 resolution (X handle, subreddits, GitHub user/repos, news context) before its pipeline runs — not just the main topic.
- R3. Sub-runs do not emit the LAW 7 `No --plan passed` warning; they are internal fan-out, not hosting-model calls.
- R4. The rendered comparison output includes a visible "Resolved entities" block showing per-entity handles/subs/github for debug transparency (answers "did it resolve everyone?" without the user having to read stderr).
- R5. SKILL.md has a canonical-path self-check at the top: if the reader loaded it from anywhere other than `plugins/cache/last30days-skill/last30days/{VERSION}/`, re-read from the versioned path before proceeding.
- R6. Version bumps to 3.0.12; CHANGELOG entry; `scripts/sync.sh` deploys.
## Scope Boundaries
- No new discovery strategy. The web-search + regex extraction in `scripts/lib/competitors.py` stays as-is.
- No new CLI flags beyond the behavior changes above. Specifically: no per-entity override flags like `--competitor-handles`. The hosting-model escape hatch remains `--competitors-list`.
- No changes to the explicit `A vs B` comparison path (topic-string parsing in `planner._comparison_entities`).
- No marketplace-clone auto-restore fix — that's Claude Code harness behavior. This plan only guards against the symptom on the skill side.
### Deferred to Separate Tasks
- Caching of per-entity resolution results: separate follow-up once hit rate justifies it.
- Fan-out rate-limiting tuning (currently `max_workers=len(entities)+1`, capped at 6): defer until we see real-world quota exhaustion.
- Pre-flight cost hint when N ≥ 4 (noted in `2026-04-22-002` risks): defer.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:205-219``--competitors` / `--competitors-list` argparse definition (const=3 today; changing to 2).
- `scripts/last30days.py:220-290``resolve_competitors_args()` validator; update `COMPETITORS_DEFAULT`.
- `scripts/last30days.py:438-520` — main() fan-out orchestration; currently passes only topic/depth to each `_competitor_runner`.
- `scripts/lib/fanout.py:40-95``run_competitor_fanout()` signature. The `competitor_runner` callable is where per-entity resolution needs to happen.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the exact per-entity resolver to reuse. Already does X handle + subreddits + GitHub user/repos + news context in parallel via ThreadPoolExecutor.
- `scripts/lib/planner.py:80-135``plan_query()` emits the LAW 7 stderr. A `quiet: bool` keyword or `internal_subrun: bool` flag will suppress it.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature. Needs a new keyword to propagate quiet-mode down to the planner.
- `scripts/lib/render.py:render_comparison_multi` — where the "Resolved entities" block is inserted.
- `SKILL.md` line 823 — canonical `SKILL_ROOT` resolver already exists but fires in engine bash, not at skill-load time.
### Institutional Learnings
- `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md` acknowledged the per-entity-resolution gap as a v1 tradeoff. This plan closes that gap.
- Kanye run stderr: `[Planner] No --plan passed... deterministic fallback` × 3 (once per competitor sub-run). That's the LAW 7 noise R3 targets.
- Linear / Coinbase runs loaded `plugins/marketplaces/last30days-skill/CLAUDE.md` as the first hit. That's the stale-path issue R5 targets.
### External References
- None. All patterns are in-repo.
## Key Technical Decisions
- **Per-entity resolve happens inside fanout, not in SKILL.md.** The user-facing promise of `--competitors` is "one flag, engine does the work." Pushing resolution onto the hosting model creates another path-of-least-resistance trap (model skips it, output looks lazy). Auto-resolve inside each sub-run when a web backend is available makes the feature self-contained.
- **Stale-path guard is a SKILL.md self-check, not a code change.** We cannot stop Claude Code from auto-restoring the marketplace clone. But we can put a 3-line banner at the top of SKILL.md that forces any path-mismatched read to re-read from the versioned cache. Both the marketplace copy (once main catches up) and the cache copy carry the guard.
- **LAW 7 suppression is opt-in via `internal_subrun=True` keyword.** Do not remove the warning from the default path — it's load-bearing for the hosting-model contract. Add an explicit bypass for engine-internal fan-out only.
- **Default 2, hard max 6 unchanged.** "Original + 2" matches the Kanye/Drake/Kendrick mental model from the feature description. Still allow `--competitors=N` from 1 to 6.
- **Resolved block is inside the EVIDENCE envelope, not above it.** Keeps the rendered output structure stable for the synthesis contract (LAW 18). The block is context, not output.
- **Skip auto-resolve when `--mock` or no web backend.** Mirrors the existing `resolve.auto_resolve()` fast-fail and keeps the mock test path deterministic.
## Open Questions
### Resolved During Planning
- **Where does per-entity resolve live?** Inside `fanout.run_competitor_fanout`, not in `main()`. Each sub-run calls `auto_resolve()` just before `pipeline.run()`.
- **Should the hosting model still be able to override?** Yes — `--competitors-list` remains the escape hatch. When an explicit list is passed, the engine still does auto-resolve per entity; the user's list just skips discovery.
- **Should sub-runs run auto-resolve in parallel with each other?** Yes. The existing `ThreadPoolExecutor` in fanout already parallelizes sub-runs; auto-resolve happens inside each sub-run's thread, so resolve calls for different entities run concurrently.
- **Default count:** 2 peers (3-way). Confirmed.
### Deferred to Implementation
- Whether to expose a `--no-auto-resolve-competitors` flag for power users who want the fast, shallow behavior. Probably not needed v2; ship auto-resolve always-on and revisit if someone complains about cost.
- Whether to surface the per-entity resolution context back into the main topic's planner (cross-entity context sharing). Stays deferred.
- Whether the Resolved block should be collapsible or always inline. Start inline; revisit based on output length feedback.
## Implementation Units
- [ ] **Unit 1: Default `--competitors` to 2 peers**
**Goal:** Change the bare `--competitors` default from 3 to 2 per user feedback. `--competitors=N` still overrides; range 1..6 unchanged.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (`COMPETITORS_DEFAULT`, `--competitors` const, stderr messages if any reference 3)
- Modify: `SKILL.md` Competitor mode section ("discovered 2-6" wording, bare-flag default line)
- Modify: `README.md` auto-discovered example line (if it references count)
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Change `COMPETITORS_DEFAULT = 3``2` in `scripts/last30days.py`.
- Change argparse `--competitors` `const=3``const=2`.
- Update any SKILL.md / README copy referencing "3 peers" to "2 peers" (default) or "2-6 peers" (range).
**Patterns to follow:**
- Existing default constants in `scripts/last30days.py` argparse block.
**Test scenarios:**
- Happy path: bare `--competitors` yields count=2, enabled=True, empty explicit_list.
- Edge case: `--competitors=3` still works (explicit override).
- Edge case: existing `test_bare_flag_defaults_to_three` test is updated to `test_bare_flag_defaults_to_two` and asserts count=2.
- Edge case: `--competitors=5` with a `--competitors-list` of length 2 still logs the mismatch warning and uses the list.
**Verification:**
- `pytest tests/test_cli_competitors.py -v` passes with the updated default.
- [ ] **Unit 2: Per-entity Step 0.55 resolution inside fanout**
**Goal:** Each competitor sub-run auto-resolves its own X handle, subreddits, GitHub user/repos, and news context via `resolve.auto_resolve()` before its `pipeline.run()` call — just like the main topic.
**Requirements:** R2
**Dependencies:** None (but Unit 3 should land together so sub-runs don't emit LAW 7 stderr while the resolution context is being passed)
**Files:**
- Modify: `scripts/lib/fanout.py`
- Modify: `scripts/last30days.py` (`_competitor_runner` closure builds the resolved args)
- Test: `tests/test_competitor_fanout.py`
- Test: `tests/test_competitors_resolve_integration.py` (new; covers the auto-resolve path)
**Approach:**
- `_competitor_runner(entity)` in main() does:
1. Call `resolve.auto_resolve(entity, config)` when `not args.mock` and a web backend is configured (reuse `_has_backend`).
2. Extract resolved x_handle, subreddits, github_user, github_repos, context.
3. Pass them to `pipeline.run()` for that sub-run.
4. Inject resolved context into a per-entity config copy (so `_auto_resolve_context` does not leak across sub-runs — deep-copy the config or use a local dict).
5. Store the resolved block on the Report's `artifacts` so the renderer can surface it (Unit 4).
- When `args.mock` is True or no backend is available, skip auto-resolve (fall through to planner defaults, matching the existing `auto_resolve()` early-return contract).
- Update `fanout.run_competitor_fanout` docstring to note that auto-resolve happens inside the caller-provided runner.
**Execution note:** Start with a failing integration test that exercises two-entity fanout + auto-resolve via a mocked `resolve.auto_resolve` and asserts that `pipeline.run` receives the resolved x_handle/subreddits for each entity.
**Patterns to follow:**
- `scripts/last30days.py` main topic branch (`if args.auto_resolve and not external_plan`) already calls `resolve.auto_resolve` and propagates results — mirror the shape for competitors.
- Config isolation: `scripts/lib/pipeline.py:162-220` reads config as-is; use `dict(config)` to avoid cross-sub-run mutation of `_auto_resolve_context`.
**Test scenarios:**
- Happy path: 3 entities, mocked `auto_resolve` returns distinct handles per entity; `pipeline.run` receives `x_handle=@drake` for Drake, `x_handle=@kendricklamar` for Kendrick, etc.
- Happy path: the main topic still uses the user-supplied `--x-handle` / `--subreddits` overrides (not overwritten by auto-resolve for the main). Competitors use their own auto-resolved values.
- Edge case: `--mock` skips auto-resolve entirely for all sub-runs (no `resolve.auto_resolve` calls).
- Edge case: `resolve.auto_resolve` returns empty dicts for one entity (low-signal topic) — the sub-run still executes with planner defaults; doesn't crash.
- Edge case: no web backend configured — auto-resolve returns empty for every entity, sub-runs fall through to planner defaults, no stack trace.
- Error path: `resolve.auto_resolve` raises — the sub-run logs a warning and continues with planner defaults (does not fail the whole comparison).
- Integration: config `_auto_resolve_context` from entity A does not leak into entity B's `pipeline.run`. Assert each sub-run gets its own context string.
**Verification:**
- New integration test passes.
- End-to-end smoke (mock mode + explicit list): each sub-run's stderr shows `[AutoResolve]` lines per entity with distinct values.
- [ ] **Unit 3: Suppress LAW 7 warning for engine-internal sub-runs**
**Goal:** The `[Planner] No --plan passed... deterministic fallback` warning does not fire during competitor sub-runs. LAW 7 is load-bearing for hosting-model contracts and must stay on the default path; this is an opt-in bypass for internal fan-out only.
**Requirements:** R3
**Dependencies:** Unit 2 (so the sub-run call site is already being modified)
**Files:**
- Modify: `scripts/lib/planner.py` (`plan_query` signature + conditional stderr)
- Modify: `scripts/lib/pipeline.py` (`run` signature + propagation)
- Modify: `scripts/last30days.py` or `scripts/lib/fanout.py` (pass `internal_subrun=True` for competitor runners)
- Test: `tests/test_planner_v3.py` (or new `tests/test_planner_quiet_mode.py`)
- Test: `tests/test_competitor_fanout.py` (assert sub-runs don't emit LAW 7 stderr)
**Approach:**
- Add a keyword `internal_subrun: bool = False` to `planner.plan_query`. When True, skip the two `print(..., file=sys.stderr)` blocks that emit the LAW 7 banner and the `[Planner] No --plan passed` capability message.
- Add the same keyword to `pipeline.run()`; pass through to `plan_query`.
- In main()/fanout, set `internal_subrun=True` for every competitor sub-run's pipeline.run call. The main topic's pipeline.run keeps the default (LAW 7 stays on for the hosting-model path).
- Also suppress the LAW 7-triggered degraded-run warning block in the render layer for sub-reports when the envelope is going to be merged into a comparison output (or accept that the block is per-entity and surfaces once per entity).
**Patterns to follow:**
- Existing keyword-only parameters on `pipeline.run` (`mock`, `x_handle`, etc.).
- `planner.plan_query` signature is already keyword-only.
**Test scenarios:**
- Happy path: `plan_query(..., internal_subrun=True, provider=None, model=None)` returns the deterministic fallback plan WITHOUT writing the LAW 7 stderr block.
- Happy path: `plan_query(...)` with default `internal_subrun=False` still writes the LAW 7 warning (unchanged behavior).
- Integration: end-to-end competitor fanout; assert captured stderr contains zero occurrences of `No --plan passed` and zero of `YOU ARE the planner`.
- Integration: main topic is not part of competitor mode; if the user invokes bare `/last30days OpenAI` without `--plan`, LAW 7 stderr fires exactly once (regression test).
**Verification:**
- Running the Kanye-style smoke test shows zero `[Planner] No --plan passed` lines for Drake / Kendrick / Travis sub-runs.
- [ ] **Unit 4: "Resolved entities" block in comparison output**
**Goal:** The rendered comparison output includes a visible block listing per-entity handles, subreddits, GitHub user, and resolved context. Answers "did it resolve everyone?" at a glance without reading stderr.
**Requirements:** R4
**Dependencies:** Unit 2 (needs resolved data on report artifacts)
**Files:**
- Modify: `scripts/lib/render.py` (`render_comparison_multi` and `render_comparison_multi_context`)
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- When each entity's `Report.artifacts` contains a `resolved` dict (populated by Unit 2), `render_comparison_multi` emits a `## Resolved Entities` block early in the EVIDENCE envelope:
```
## Resolved Entities
- **Kanye West**: X @kanyewest | Subs r/Kanye, r/hiphopheads | GitHub: — | Context: BULLY released, UK ban…
- **Drake**: X @Drake | Subs r/DrakeTheType, r/hiphopheads | GitHub: — | Context: ICEMAN rollout…
- **Kendrick Lamar**: X @kendricklamar | Subs r/KendrickLamar | GitHub: — | Context: Grammy wins, dormant…
```
- Missing fields render as `` not empty.
- When no entity has a `resolved` payload (mock mode, no web backend), omit the block entirely rather than emit an empty section.
- Context strings are truncated at 120 chars to keep the block scannable.
**Patterns to follow:**
- Existing `render_comparison_multi` envelope structure (lines ~395-480 in render.py).
- Existing per-entity evidence block format (`## {label}`) for consistency.
**Test scenarios:**
- Happy path: 3 entities each with a `resolved` artifact → block lists all 3 with their fields.
- Happy path: 2 entities, one with full resolution, one with partial (x_handle only) → missing fields render as ``.
- Edge case: no entity has a resolved artifact → block is omitted entirely.
- Edge case: context string > 120 chars → truncated with ellipsis.
- Integration: rendered output passes through the same EVIDENCE envelope comments and synthesis contract (LAW 18 unchanged).
**Verification:**
- Snapshot tests confirm the block appears in the right spot with the right formatting.
- End-to-end smoke shows a realistic 3-entity Resolved block in the rendered output.
- [ ] **Unit 5: SKILL.md canonical-path self-check**
**Goal:** A top-of-file SKILL.md directive forces any reader (Claude Code, Codex, Hermes, Gemini) to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` before proceeding. If loaded from `marketplaces/` or any other path, re-read from the pinned versioned cache.
**Requirements:** R5
**Dependencies:** None
**Files:**
- Modify: `SKILL.md` (prepend a STEP 0 block before the existing STEP 0 / LAW list)
**Approach:**
- Add a numbered first step at the top (before or bundled with existing "STEP 0: ToolSearch preload"):
```
## STEP 0: Canonical Path Self-Check (must run first)
Before reading anything else below, verify you loaded this SKILL.md from
the versioned cache, not the marketplace clone:
CANONICAL=$HOME/.claude/plugins/cache/last30days-skill/last30days/
CANONICAL_LATEST=$(ls -d "$CANONICAL"*/ 2>/dev/null | sort -V | tail -1)
If the SKILL.md you just read is not under $CANONICAL_LATEST, STOP. Re-read
$CANONICAL_LATEST/SKILL.md and restart from here. Marketplace clones
(`plugins/marketplaces/last30days-skill/`) are pinned to origin/main and
can be stale; the versioned cache is the ground truth.
```
- Reinforce in the existing LAW 7 block that `--help` output must be read from the same pinned `SKILL_ROOT` to avoid flag-list skew.
**Patterns to follow:**
- Existing STEP 0 ToolSearch preload (top of SKILL.md) for tone / imperative voice.
- Existing `SKILL_ROOT` resolver snippet (line ~823).
**Test scenarios:**
- Test expectation: none — SKILL.md is documentation; no unit test, verified by follow-up user invocation.
**Verification:**
- In a fresh Claude Code window, `/last30days Test --competitors` loads SKILL.md, the model executes the STEP 0 self-check, and (if it had loaded from marketplaces/) switches to the cache path before running `--help` or the engine. Observable via the model's announced reasoning / task list.
- [ ] **Unit 6: Version bump, CHANGELOG, sync**
**Goal:** Ship 3.0.12 and deploy to all local targets.
**Requirements:** R6
**Dependencies:** Units 1-5
**Files:**
- Modify: `.claude-plugin/plugin.json` (version 3.0.11 → 3.0.12)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- CHANGELOG entry under `## [3.0.12]` dated 2026-04-22 covering the four fixes (Fixed: per-entity resolution; Fixed: LAW 7 sub-run noise; Changed: default count 3→2; Added: Resolved entities block; Added: canonical-path self-check in SKILL.md).
- `sync.sh` deploys to `~/.claude/plugins/cache/last30days-skill-private/...`, `~/.agents/`, `~/.codex/`, Hermes.
- Manual hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.12/` so the public `/last30days` slash command picks up the new version before PR merge (matches the 3.0.11 testing pattern).
**Test scenarios:**
- Test expectation: none — packaging only. Verification is by inspection.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns `3.0.12`.
- `sync.sh` exits 0 with "Import check: OK" for each target.
- Hot-copied 3.0.12 directory contains the new files and `/last30days` picks up the new version (highest-version resolver).
## System-Wide Impact
- **Interaction graph:** Fanout sub-runs now call `resolve.auto_resolve` per entity. Each sub-run is independent; no shared mutable state with other sub-runs or with the main topic.
- **Error propagation:** `auto_resolve` failures inside a sub-run log a warning and degrade to planner defaults; do not propagate up to abort the comparison. Same contract as today for the main topic.
- **State lifecycle risks:** Config dict is mutated by `auto_resolve` (via `config["_auto_resolve_context"]`). Must deep-copy per sub-run or scope context to a local mapping — otherwise two sub-runs' context strings race.
- **API surface parity:** `pipeline.run` gains a keyword (`internal_subrun`); callers that don't pass it get the existing behavior. `planner.plan_query` gains the same. Backward compatible.
- **Integration coverage:** New integration test for the fanout + auto-resolve + render chain. Existing snapshot tests update to include the Resolved block.
- **Unchanged invariants:** Single-entity `/last30days` invocations (no `--competitors`) behave identically. Explicit `A vs B` comparison topics behave identically. LAW 7 still fires on the default hosting-model path. `render_compact` path is untouched.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Auto-resolving per competitor triples the WebSearch call volume (4 queries × 3 competitors = 12 extra web searches). | Fast-fail when no backend; user can pass `--competitors-list` to skip discovery but still get auto-resolve. Cost note in CHANGELOG. |
| Config mutation across sub-runs via `_auto_resolve_context`. | Unit 2 deep-copies config per sub-run before each `auto_resolve` + `pipeline.run` call. Integration test asserts no cross-entity leak. |
| LAW 7 suppression leaks onto the hosting-model path via a wrong default. | Default `internal_subrun=False`. Only fanout's competitor sub-runs set True. Unit test asserts bare-topic invocation still emits LAW 7. |
| SKILL.md STEP 0 banner gets ignored by the model (same failure mode as line 823 today). | Put it in the guaranteed-read top band (before LAW 1, above all other content), imperative voice, concrete `STOP` verb. Still not bulletproof but strictly better than current. |
| Default count change breaks assumptions in downstream tools or existing user muscle memory. | Changelog calls it out as Changed; `--competitors=3` still works for users who want the old default. |
## Documentation / Operational Notes
- Beta channel first: merge behind `/last30days-beta` via the private repo before cherry-picking to public. Follows the same process as 3.0.11.
- Version 3.0.12 is a fix release; no marketing post required.
- After merge, add a line to the PR description pointing at this plan.
## Sources & References
- Origin plan: `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- Related PR: #308 (v3.0.11 shipping --competitors)
- Test windows that surfaced the bugs: Kanye, Linear, Coinbase (2026-04-22 session)
- Related code: `scripts/lib/fanout.py`, `scripts/lib/resolve.py` (`auto_resolve`), `scripts/lib/planner.py` (`plan_query`), `scripts/lib/render.py` (`render_comparison_multi`)
@@ -0,0 +1,394 @@
---
title: "fix: --competitors runs a full last30days per entity with hosting-model pre-resolve"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md
---
# fix: --competitors runs a full last30days per entity with hosting-model pre-resolve
## Overview
User intent confirmed 2026-04-22: `--competitors` should run a full single-entity `last30days` pipeline for the main topic AND for each discovered peer — three independent full-depth passes, each with its own Step 0.55 resolution, own X handle primary weight, own subreddit targeting, own GitHub repo scoping. Then merge them into the comparison output.
3.0.12 already built the N-parallel-pipelines orchestration (`scripts/lib/fanout.py`). What it got wrong: it tried to do per-entity Step 0.55 engine-side via `resolve.auto_resolve()`, which requires a web search backend key (BRAVE/EXA/SERPER/PARALLEL/OPENROUTER). Matt runs from Claude Code, which has its own WebSearch tool. The engine has none of those keys, so per-entity auto_resolve silently no-ops and all peer sub-runs fall through to deterministic single-word planner queries.
Four 2026-04-22 test runs (Warriors, Seattle, Arizona Wildcats, Kanye West) confirmed this via engine receipts:
- Compact Resolved Entities block shows peers as `X - | Subs - | GitHub - | Context: -`.
- Sub-run planner lines show `source=deterministic, subqueries=1` — the "I gave up and keyword-searched" shape.
- Engine footer keeps nudging `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY`, which is wrong advice for a Claude Code user who already has WebSearch.
- Kanye run leaked main topic's `--subreddits` into Drake's and Kendrick's sub-runs (regression bug).
The fix is to flip the resolution responsibility: the hosting model (Claude Code, Codex, Hermes, Gemini) does Step 0.55 via its own WebSearch tool for every entity, then passes the resolved targeting to the engine via a new `--competitors-plan` JSON flag. Engine fan-out remains — each peer still runs a full `pipeline.run()`. The difference is the peers now arrive with full targeting, equivalent to the main topic, so retrieval is apples-to-apples.
Why not just reuse vs-mode? vs-mode is a SINGLE `pipeline.run()` with a comparison-optimized plan. It pre-resolves Step 0.55 per entity but merges everything into one retrieval pool with lower-weight `--x-related` for peers, merged subreddits, and cross-entity keyword noise. That is not "three full passes." The user explicitly wants three full passes.
## Problem Frame
3.0.12's architecture was correct; its data dependency was wrong.
| Capability | 3.0.12 path | Target path (this plan) |
|---|---|---|
| Fan out to N parallel pipelines | Yes (`fanout.run_competitor_fanout`) | Same — keep |
| Per-entity Step 0.55 resolution | Engine-internal `resolve.auto_resolve()` — needs BRAVE/EXA/SERPER/PARALLEL key | Hosting model does it via its own WebSearch, passes to engine |
| Per-entity targeting threaded into `pipeline.run()` | Main topic only via outer flags; peers via auto_resolve (failing) or nothing | Main topic via outer flags; peers via `--competitors-plan` JSON |
| Footer nudge | Unconditional BRAVE/SERPER | Suppressed when `--plan` or `--competitors-plan` present |
| Resolved Entities block in raw save file | Stdout only | Also in `--save-dir` raw file |
| Override-leak from main into peers | Present (Kanye receipt) | Fixed via explicit per-entity kwargs scrub |
| Polymarket noise on ambiguous topics | Present (Warriors, Arizona receipts) | `--polymarket-keywords` + auto-skip for single-token-ambiguous |
The key architectural change is who owns per-entity resolution. The engine stops trying to do it itself; the hosting model does it upstream (it already has WebSearch) and passes results in.
This is the same pattern `--plan` already uses for the main topic: hosting model generates the plan via its own reasoning, passes it in, engine accepts. We apply the pattern to peers.
## Requirements Trace
- R1. New `--competitors-plan` JSON flag accepting per-entity targeting: `x_handle`, `x_related`, `subreddits`, `github_user`, `github_repos`, `context`. Implies `--competitors`. Per-entity values thread into that entity's `pipeline.run()`. Bypasses engine-internal `auto_resolve` for covered entities.
- R2. SKILL.md "Competitor mode" rewritten to make the hosting-model path canonical: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers) via WebSearch, (c) assemble `--competitors-plan` JSON, (d) invoke engine. Engine-internal auto_resolve remains as headless fallback.
- R3. The LAW 7-style stderr emitted when `--competitors` has no list, no plan, no backend is reframed: leads with "hosting reasoning model, use your WebSearch to run Step 0.55 per entity and pass `--competitors-plan`." Does not lead with BRAVE_API_KEY.
- R4. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` OR `--competitors-plan` was passed. Signal: hosting model is driving and already has WebSearch.
- R5. Override-leak fix: competitor sub-runs do not inherit main topic's `--subreddits`, `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`. Sub-runs use only their own per-entity targeting (from `--competitors-plan` if provided, else engine-internal auto_resolve if backend, else planner defaults).
- R6. The `## Resolved Entities` block is also appended to the saved raw file when `--save-dir` is in use. Each entity's effective targeting (whatever was actually passed to its `pipeline.run()`) is visible on audit.
- R6b. When `--save-dir` is in use with a comparison run, each entity's sub-run ALSO saves its own standalone raw file — same format as a single-entity run. `/last30days Kanye West --competitors` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md` (one per entity) plus the merged comparison file. Matches the historical vs-mode behavior when it ran as N passes.
- R7. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R8. Default `--competitors` count remains 2 (3-way: main + 2 peers). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to `scripts/lib/fanout.py` architecture. N parallel pipelines stays. Only the data each sub-run receives changes.
- No changes to the vs-mode (topic contains "vs" / "versus") behavior. That path is independent.
- No new emit modes. Comparison output format unchanged.
- No deprecation of `--competitors-list`. Stays as the minimum escape hatch for hosting models that skip per-entity Step 0.55 (names-only).
### Deferred to Separate Tasks
- Cache layer for hosting-model competitor resolution: separate plan once cost evidence exists.
- Cross-source disambiguation beyond Polymarket: separate plan.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — `--competitors` / `--competitors-list` argparse block, `resolve_competitors_args` validator, `_main_runner` closure, `_competitor_runner` closure, the `[Competitors] --competitors requires...` stderr block. Primary file for this plan.
- `scripts/lib/fanout.py` — `run_competitor_fanout` orchestrator. Signature unchanged; `_competitor_runner` closure now builds kwargs from `--competitors-plan`.
- `scripts/lib/pipeline.py` — `pipeline.run()` signature; no changes required (all per-entity flags already exist as kwargs).
- `scripts/lib/planner.py` — existing `--plan` parsing and validation, pattern to mirror for `--competitors-plan`.
- `scripts/lib/render.py` `_render_resolved_entities_block` (added in 3.0.12) — already reads `report.artifacts["resolved"]`; no change needed.
- `scripts/last30days.py` `save_output` / `render.render_full` — the save path. Needs to include the Resolved Entities block for comparison runs.
- `scripts/lib/quality_nudge.py` — where the BRAVE/SERPER footer nudge is emitted. Needs a context-aware suppression check.
- `scripts/lib/polymarket.py` — source adapter. Entry point for `--polymarket-keywords` filter and single-token-ambiguous auto-skip.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built the initial fanout, deferred per-entity resolve as "v1 simplification."
- 3.0.12 plan (`2026-04-22-003`): tried to close the gap via engine-internal `auto_resolve`. Works only with backend keys. Fails silently without.
- 2026-04-22 test session receipts: confirmed all four fixes in this plan are real, reproducible bugs.
- User's architectural steer 2026-04-22: "runs a full last30days on all 3 topics" — this plan encodes that explicitly as N full `pipeline.run()` calls with pre-resolved targeting per entity.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **`--competitors-plan` is a single JSON flag, not a fan of separate flags.** Mirrors `--plan`. Stable schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Accept inline JSON or a file path (matches `--plan`).
- **Hosting-model-driven resolution is the documented default.** Engine-internal `auto_resolve` is the headless / cron fallback. SKILL.md routes hosting models to the JSON-flag path; engine keeps auto_resolve alive for BRAVE/EXA/SERPER users running CI.
- **Override-leak fix is call-site scrubbing, not a signature change.** `_competitor_runner` builds an explicit kwargs dict per entity from `_subrun_kwargs(entity, plan_entry)`. No closure-default fallthrough from main scope. The 3.0.12 `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag.
- **Footer nudge becomes context-aware.** Suppressed when `--plan` or `--competitors-plan` present. Not suppressed for bare `--competitors-list` or bare invocations. Headless cron without keys still sees the nudge.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known list of single-token-ambiguous names (states, common nouns). Stderr notes the skip so it is observable and overridable.
- **Per-entity sub-runs get the full `pipeline.run()` pass.** Same depth, same sources, same API cost per entity as a single-topic run. This is the explicit user intent — three full passes, not one merged pass.
## Open Questions
### Resolved During Planning
- **JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers (3-way comparison). Unchanged from 3.0.12.
- **Does engine-internal auto_resolve stay alive?** Yes, for entities not covered by `--competitors-plan` when a backend is configured. Headless/cron users with keys keep the current 3.0.12 behavior.
- **vs-mode or fanout?** Fanout. User's explicit ask: three full passes, not one merged pass. vs-mode merges into one pipeline with lower peer weighting, which is not what the user wants.
- **Does the save file need per-entity clusters?** Start with the Resolved block appended. Per-entity cluster sections can follow in a separate task; they are nice-to-have, not blocking.
### Deferred to Implementation
- Exact trace of override-leak source. Candidates: closure capture of `subreddits` in `_competitor_runner`, shared `_auto_resolve_context` leak, Reddit adapter inheriting global config. Test-first; trace at implementation time.
- Heuristic for "single-token-ambiguous topic" auto-skip. Start with a short hard-coded list (US state names, US city names, common nouns like "Warriors", "Suns", "Jets"); revisit after dogfood.
- Whether per-entity coverage warnings fire when `--competitors-plan` under-resolves an entity (e.g., only `x_handle`, no subreddits). Start with stderr logging; revisit UX.
## Implementation Units
- [ ] **Unit 1: `--competitors-plan` JSON flag + per-entity kwargs threading**
**Goal:** New CLI flag accepting per-entity targeting JSON. Each covered entity's `pipeline.run()` receives its own `x_handle` / `x_related` / `subreddits` / `github_user` / `github_repos` / `context`. Skips engine-internal `auto_resolve` for covered entities.
**Requirements:** R1, R5 (primary leak fix site)
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner`)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; verify)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON OR a file path (mirror `--plan`).
- Validation: parse JSON; must be a dict; each value must be a dict; unknown fields log warnings; malformed input exits 2.
- Schema per entity: optional fields `x_handle` (str), `x_related` (list), `subreddits` (list), `github_user` (str), `github_repos` (list), `context` (str).
- Case-insensitive matching against `--competitors-list` / discovered entities.
- Build `_subrun_kwargs(entity, plan_entry)` helper. Returns a complete, explicit kwargs dict for `pipeline.run()` with no closure-default fallthrough from main scope. This helper is the single source of truth for per-entity call args. It also fixes the override-leak (R5) by scrubbing all per-entity flags to None unless the plan (or auto_resolve) sets them.
- `_competitor_runner(entity)`:
1. Look up `plan_entry` from `--competitors-plan` (if any).
2. If plan covers entity fully, build kwargs from it; skip `auto_resolve`.
3. If plan partially covers or is absent, fall back to `auto_resolve` (3.0.12 behavior) when a backend is configured. Plan values win over auto_resolve values on conflict.
4. If neither plan nor backend, fall through to `pipeline.run()` with per-entity kwargs all None — engine uses planner defaults for that entity only (no leak).
- Deep-copy config per sub-run (already done in 3.0.12); merge per-entity `context` into `entity_config["_auto_resolve_context"]` only.
**Execution note:** Test-first for the override-leak regression (pass `--subreddits=A,B` on main + a peer, assert peer's `pipeline.run(subreddits=...)` is None or peer-specific).
**Patterns to follow:**
- `--plan` parsing at `scripts/last30days.py` (inline JSON or file path).
- 3.0.12's `_competitor_runner` closure for scope; extract the kwargs-build into `_subrun_kwargs` helper.
- `entity_config = dict(config)` deep-copy pattern from 3.0.12.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake": {"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's `pipeline.run` receives `x_handle="Drake"` and `subreddits=["Drizzy"]`; no `auto_resolve` call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered entities skip auto_resolve; third falls back to auto_resolve.
- Happy path: plan file path accepted like `--plan` file path.
- Happy path: case-insensitive entity match (`Drake` in plan, `drake` in list).
- Edge case: unknown fields in plan entry → logged, ignored, run continues.
- Edge case: plan entry for entity not in list → ignored with warning.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list not dict → exit 2.
- Regression (leak fix): main `--subreddits=A,B` + `--competitors-list "Drake"` + no plan → Drake's `pipeline.run` receives `subreddits=None` (no leak).
- Regression (leak fix): same for `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`.
- Regression (leak fix): main `--x-handle=kanyewest` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanyewest"`.
- Integration: full main + 2 peers run via `--competitors-plan`; assert each sub-run's effective kwargs match expected per-entity values.
**Verification:**
- All new and regression tests pass.
- Smoke run (mock mode + `--competitors-plan`): stderr shows `[Competitors] Drake: x=@Drake subs=Drizzy` line per entity; no `[AutoResolve]` calls for plan-covered entities; no leak of main topic's flags.
- [ ] **Unit 2: Reframe LAW 7-style stderr for hosting-model context**
**Goal:** When `--competitors` has no `--competitors-list`, no `--competitors-plan`, and no backend, stderr tells the hosting reasoning model to use its WebSearch tool for Step 0.55 per entity and pass `--competitors-plan`. Stops leading with BRAVE_API_KEY.
**Requirements:** R3
**Dependencies:** Unit 1 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the existing `[Competitors] --competitors requires...` block)
- Test: `tests/test_competitors_no_backend_message.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent runtime with a WebSearch tool), YOU should: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers), (c) assemble a `--competitors-plan` JSON, (d) re-invoke. Skip this step and quality degrades — peer entities will run with planner defaults."
2. "If you are running headless (cron, CI, no hosting model), set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list "A,B,C"` skips discovery but does not pre-resolve peers. Use only for quick tests."
- Exits non-zero as today.
**Patterns to follow:**
- Existing LAW 7 stderr in `planner.plan_query` for tone.
**Test scenarios:**
- Happy path: stderr leads with "If you are the hosting reasoning model" and names `--competitors-plan` before any backend key.
- Happy path: stderr explicitly names `--competitors-plan` as the preferred override.
- Happy path: stderr does NOT say "requires either a configured web search backend OR an explicit --competitors-list" (the current 3.0.12 wording).
**Verification:**
- Test asserts ordering and required phrases.
- [ ] **Unit 3: Suppress BRAVE/SERPER footer nudge when hosting-model-driven**
**Goal:** The `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY` footer is suppressed when `--plan` or `--competitors-plan` was passed (signal: hosting model is driving and already has WebSearch).
**Requirements:** R4
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/lib/quality_nudge.py` (or wherever nudge is emitted; verify during implementation)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Locate the nudge emission point.
- Add a suppression check: if `--plan` OR `--competitors-plan` was passed, skip the nudge. Otherwise, current behavior.
- Don't suppress the nudge for bare `--competitors-list` alone — that path isn't necessarily hosting-model-driven.
**Test scenarios:**
- Happy path: `--plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-list` only, no backend → nudge fires (current behavior).
- Happy path: no `--competitors`, no `--plan`, no backend → nudge fires (current behavior unchanged).
**Verification:**
- All four scenarios produce expected nudge presence/absence.
- [ ] **Unit 4: Per-entity save files + Resolved block in each**
**Goal:** When `--save-dir` is in use with a comparison run, each entity's sub-run saves its own standalone raw file (same format as a single-entity run), and each file includes the `## Resolved Entities` block so audits can see what targeting that entity received. Matches the historical vs-mode behavior when it was N passes.
**Requirements:** R6, R6b
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/last30days.py` (`save_output`, the save loop after fanout completes)
- Possibly modify: `scripts/lib/render.py` (`render_full` branch to include Resolved block when artifact is present)
- Test: `tests/test_save_raw_competitor_files.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]`. For each `(entity, entity_report)` tuple, call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)` — same path a single-entity run takes.
- Each saved file uses its entity's slug as the filename (`drake-raw.md`, `kendrick-lamar-raw.md`). Main topic keeps the existing `kanye-west-raw.md` filename.
- Each file includes its own `## Resolved Entities` block (single-entity variant: one row for that entity only). This makes each sub-run's file self-describing — you can see what targeting was used without opening the comparison file.
- The merged comparison output (stdout) still includes the 3-row Resolved Entities block.
- Optional: also save a comparison summary file (e.g., `kanye-west-comparison-raw.md`) holding the merged multi-entity render. Start with per-entity files only; comparison summary is a follow-up if stdout-plus-individual-files is insufficient.
- Single-entity runs unchanged (no additional files, no block change).
**Patterns to follow:**
- Existing `save_output` invocation for single-entity runs (line 501 of current `scripts/last30days.py`).
- Existing slug generation (`slugify(topic)`) for filename consistency.
- `_render_resolved_entities_block` from 3.0.12 for the single-entity variant.
**Test scenarios:**
- Happy path: `--competitors-list "Drake,Kendrick Lamar"` + `--save-dir=/tmp/x` → `/tmp/x/kanye-west-raw.md`, `/tmp/x/drake-raw.md`, `/tmp/x/kendrick-lamar-raw.md` all exist.
- Happy path: each peer file's first sections include that entity's Resolved Entities block with its own row only.
- Happy path: single-entity run with `--save-dir` → one file, unchanged from today's behavior.
- Edge case: entity slug collides with existing file → overwrite (matches single-entity behavior).
- Edge case: `--save-suffix=v3` → all 3 files get the suffix (`kanye-west-raw-v3.md`, `drake-raw-v3.md`, `kendrick-lamar-raw-v3.md`).
- Edge case: comparison run with one peer whose sub-run failed → that entity's file is NOT saved; others are.
- Integration: stderr after save shows three `[last30days] Saved output to <path>` lines, one per entity.
**Verification:**
- After `/last30days Kanye West --competitors-list "Drake,Kendrick Lamar" --save-dir=/tmp/x`: `ls /tmp/x/*-raw.md` shows 3 files. Each contains its entity's Resolved block.
- [ ] **Unit 5: SKILL.md "Competitor mode" rewrite — hosting-model Step 0.55 canonical**
**Goal:** SKILL.md documents the hosting-model-driven path as canonical: discover N peers via WebSearch, run Step 0.55 per entity, assemble `--competitors-plan`, invoke engine. Engine-internal `auto_resolve` is labeled the headless fallback.
**Requirements:** R2
**Dependencies:** Unit 1 (flag must exist before documented)
**Files:**
- Modify: `SKILL.md` (Competitor mode subsection)
- Modify: `README.md` (one-line example update)
**Approach:**
- Replace the 3.0.12 Competitor mode subsection with a clear flow:
1. User invokes with `--competitors` or `--competitors=N`.
2. Hosting model runs WebSearch for "[topic] competitors" / "[topic] alternatives" → picks top N peers.
3. Hosting model runs Step 0.55 for main + each peer (x_handle, subreddits, github_user, github_repos, context) — same protocol as vs-mode per SKILL.md §679.
4. Hosting model assembles a `--competitors-plan` JSON object.
5. Hosting model invokes the engine with `--competitors-list "A,B,C" --competitors-plan '{...}'`.
6. Engine fans out N full pipelines (main + peers), each with its own full Step 0.55-grade targeting. Each entity also saves its own `*-raw.md` file when `--save-dir` is set (three full passes → three save files, matching the historical vs-mode behavior). Comparison output merges them for display.
- Concrete JSON example in SKILL.md showing the schema.
- Failure-mode warning: a `## Resolved Entities` block with dashes for any entity means hosting model skipped Step 0.55 for that one. Re-run with corrected plan.
- "Headless fallback" sub-subsection: when BRAVE/EXA/SERPER/PARALLEL/OPENROUTER is set, engine's internal `auto_resolve` handles peers and `--competitors-plan` is optional.
**Patterns to follow:**
- SKILL.md "Step 0.55" section for per-entity resolve protocol.
- SKILL.md "If QUERY_TYPE = COMPARISON" section for the same-protocol-as-vs-mode reference.
- Tone of existing 3.0.12 Competitor mode prose.
**Test scenarios:**
- Test expectation: none — documentation. Verification is a fresh Claude Code window dogfood run.
**Verification:**
- `/last30days Kanye West --competitors` in a new window: hosting model does Step 0.55 for Kanye + 2 discovered peers; passes `--competitors-plan`; rendered Resolved block shows non-empty fields for all 3; top voices include at least one peer-specific handle.
- [ ] **Unit 6: Polymarket disambiguation guard**
**Goal:** Support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
**Requirements:** R7
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` argparse (`--polymarket-keywords`)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"` flag. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip rule: if topic is one token AND token matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords` provided, skip Polymarket with a stderr note.
- SKILL.md Step 0.55 protocol gets a small addition: for ambiguous topics, hosting model passes `--polymarket-keywords` with topic-specific qualifiers.
**Patterns to follow:**
- Existing Polymarket adapter match logic.
- Single-token detection heuristic.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr notes the skip.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs; matches filtered.
- Happy path: topic "OpenAI" (no ambiguity) → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke run → Polymarket footer absent OR filtered to nba/gsw markets.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R7
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG entry groups the fixes: Added `--competitors-plan` JSON flag for per-entity hosting-model pre-resolve. Fixed override-leak from main into peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when `--plan` / `--competitors-plan` is present. Added: Resolved Entities block persists to saved raw file. Added: `--polymarket-keywords` + auto-skip for ambiguous single-token topics.
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13 immediately.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files with competitors.py, fanout.py, the updated SKILL.md, and plugin.json 3.0.13.
## System-Wide Impact
- **Interaction graph:** `_competitor_runner` becomes the single source of truth for sub-run kwargs via `_subrun_kwargs(entity, plan_entry)`. Every per-entity flag flows through one helper. No closure-default leaks.
- **Error propagation:** `--competitors-plan` JSON parse errors exit 2 with stderr (same as `--plan`). Per-entity plan entries with malformed values log warnings and fall back; don't abort the whole run.
- **State lifecycle risks:** `entity_config = dict(config)` already deep-copies for `_auto_resolve_context`; extend the isolation discipline to every per-entity flag. Verified in Unit 1 regression tests.
- **API surface parity:** `--competitors-plan` is additive. `--competitors` and `--competitors-list` unchanged. `--plan` unchanged. `--polymarket-keywords` additive.
- **Integration coverage:** New regression tests for override-leak. New integration test for plan-driven sub-run threading. New nudge-suppression test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. `planner.plan_query` LAW 7 behavior for the default path unchanged. Single-entity render path unchanged. vs-mode behavior unchanged.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model takes the lazy path and uses `--competitors-list` names-only. | Unit 2 stderr explicitly steers to `--competitors-plan` with Step 0.55 protocol named. Unit 5 SKILL.md docs. Resolved Entities dashes in output make the gap visible. |
| JSON gets verbose for the hosting model to construct repeatedly. | Schema is small (≤6 fields per entity). Hosting model already runs Step 0.55 for main topic in every comparison run; peers use the same protocol. One JSON block replaces N CLI flags. |
| Override-leak source is deeper than `_competitor_runner` closure. | Test-first per Unit 1. Receipts from 2026-04-22 Kanye run are reproducible. Trace methodically from call site. |
| Plan-covered entity bypasses auto_resolve but plan data is incomplete (e.g., no subreddits). | Hosting model's own SKILL.md contract says Step 0.55 must cover all fields. Stderr logs per-entity coverage so under-resolved entities are visible. Next-run correction, not engine-side rescue. |
| Polymarket auto-skip false-positives on legitimate ambiguous topics with real markets. | Conservative match (single-token + known list). `--polymarket-keywords` override is explicit and unambiguous. Stderr notes the skip. |
| Footer nudge suppression hides the message from headless users who genuinely need it. | Suppression only fires when `--plan` or `--competitors-plan` is present. Cron / CI runs that pass neither still see the nudge. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md (private repo `/last30days-beta`).
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG voice should call this out as the feedback-driven follow-up to 3.0.12. Reader should see "we tried engine-internal resolve in 3.0.12; it needs backend keys we don't have; we moved resolution to the hosting model in 3.0.13."
## Sources & References
- Origin plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Earlier plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts: Warriors, Seattle, Arizona Wildcats, Kanye West
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -0,0 +1,451 @@
---
title: "feat: vs mode runs N full passes and --competitors is vs with auto-discovery"
type: feat
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded
---
# feat: vs mode runs N full passes and --competitors is vs with auto-discovery
## Overview
Architectural unification driven by user correction 2026-04-22: vs mode and `--competitors` are the same thing. A user typing `/last30days OpenAI vs Anthropic vs xAI` should get a full single-entity last30days pass for each of the three entities — three full pipelines, three saved `*-raw.md` files, merged into one comparison output. A user typing `/last30days OpenAI --competitors` should get the same output after the hosting model auto-picks 2 peers; i.e., `--competitors` is a thin shortcut that expands "topic + `--competitors`" into "topic vs peer1 vs peer2" and then runs the unified vs pipeline.
Current state diverges from this:
- **vs mode today**: one `pipeline.run()` with a comparison-optimized plan that merges all entities' targeting into a single retrieval pool. Lower-weight `--x-related` for peers, merged subreddits, cross-entity keyword noise. One saved file.
- **`--competitors` today (3.0.12)**: N parallel `pipeline.run()` calls via `scripts/lib/fanout.py`, but per-entity Step 0.55 depends on an engine-side web backend key Matt doesn't have. Silently degrades to planner defaults for peers. One saved file (main topic only). Override-leak from main into peers.
After this plan:
- **vs mode**: N parallel `pipeline.run()` calls, one per entity, each with its own full Step 0.55-grade targeting, each saving its own `*-raw.md`. Merged into one comparison output.
- **`--competitors`**: SKILL.md shortcut. Hosting model discovers N peers, builds `"topic vs peer1 vs peer2"`, and invokes the same vs pipeline. No separate orchestration path.
- **Same fanout machinery (`scripts/lib/fanout.py`)** serves both. One fix, both behaviors improve.
## Problem Frame
The product insight from 2026-04-22 test runs is simple: the user wants three full last30days reports plus a comparison merge. Not one comparison pass with N-way targeting merged into a single retrieval pool. Not one save file. Not "main gets Step 0.55, peers get planner defaults." Three full passes. Three save files. Merged output.
The historical vs mode did that (it ran as 3 passes, saving 3 files). SKILL.md §551 currently says:
> "When the user asks 'X vs Y', run ONE research pass with a comparison-optimized plan that covers both entities AND their rivalry. This replaces the old 3-pass approach (which took 13+ minutes and produced tangential content)."
That change was a latency optimization that removed the user-visible behavior the user wants. The fix is to revert the architectural direction: N passes per entity, in parallel rather than serial (parallelism lowers wall-clock to ~1× a single pass, not N×), with per-entity save files.
The 3.0.11 `--competitors` flag already introduced parallel N-pass machinery (`fanout.run_competitor_fanout`). The 3.0.12 follow-up tried to wire per-entity Step 0.55 into it but failed when no web backend was configured. The elegant move: stop maintaining two architectures. vs-mode and `--competitors` both use `fanout.py`. `--competitors` becomes a SKILL.md-level shortcut that discovers 2 peers and hands off to vs-mode.
Four 2026-04-22 test receipts (Warriors, Seattle, Arizona Wildcats, Kanye West) all confirmed the user's pain points:
- Peers thin because they ran without per-entity handle/sub targeting.
- Only one `*-raw.md` per run — no per-entity audit.
- Kanye peers leaked main topic's `--subreddits`.
- Engine footer nudging `BRAVE_API_KEY` to Claude Code users who already have WebSearch.
- Polymarket noise on ambiguous topics (Warriors → Glasgow rugby; Arizona → Diamondbacks).
This plan closes all of them by unifying the architecture and making hosting-model-driven Step 0.55 per entity the canonical path.
## Requirements Trace
- R1. vs mode (any topic containing ` vs ` / ` versus `) runs N full `pipeline.run()` calls in parallel, one per entity. Each sub-run uses its entity's own Step 0.55 targeting (from the hosting model's pre-resolution, passed via a new `--competitors-plan` JSON).
- R2. `--competitors` (and `--competitors=N`) becomes a SKILL.md-level shortcut: the hosting model (a) discovers N peers via WebSearch, (b) runs Step 0.55 per entity (main + peers), (c) rewrites the topic to `"main vs peer1 vs peer2"`, (d) invokes the engine with `--competitors-plan` containing each entity's targeting.
- R3. New `--competitors-plan` JSON flag. Schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Implies vs mode when present with a single-entity topic. Applies per-entity targeting to each sub-run. Accepts inline JSON or a file path (matches `--plan`).
- R4. Each entity's sub-run saves its own `*-raw.md` file when `--save-dir` is in use. Example: `/last30days "Kanye West vs Drake vs Kendrick Lamar" --save-dir=~/Documents/Last30Days` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`. Same filenames a single-entity run of each topic would produce. Matches historical vs-mode behavior.
- R5. Each per-entity saved file includes its own single-row `## Resolved Entities` block so the audit survives. The merged comparison stdout still shows the full 3-row block.
- R6. Override-leak fix: no main-topic flags (`--subreddits`, `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`) leak into peer sub-runs. Every per-entity kwarg is scrubbed at the sub-run call site.
- R7. LAW 7-style stderr for `--competitors` invocations with no list, no plan, no backend is reframed for hosting-model context: leads with "use your WebSearch to discover peers, resolve Step 0.55 per entity, re-invoke with `topic vs peer1 vs peer2 --competitors-plan '...'`." Does not lead with BRAVE_API_KEY.
- R8. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` or `--competitors-plan` was passed.
- R9. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R10. Default `--competitors` count stays 2 peers (3-way comparison). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to single-entity `pipeline.run()` semantics. Each sub-run in vs mode behaves identically to a bare `/last30days {entity}` invocation.
- No changes to the planner's comparison-intent logic for single-entity-containing topics. The `_should_force_deterministic_plan` shortcut for vs-topics routes to fanout, not to its current single-pipeline path.
- No new emit modes. Comparison output format unchanged.
- No removal of `--competitors-list`. Stays as a minimum escape hatch (names-only, no per-entity targeting) for scripted headless use.
- No removal of engine-internal `resolve.auto_resolve()` in fanout. Remains as headless / cron fallback for users with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys. The dominant Claude Code path bypasses it via `--competitors-plan`.
### Deferred to Separate Tasks
- Explicit "head-to-head" rivalry pass in vs-mode (a supplemental subquery like `"A vs B"` that catches rivalry articles missing from pure entity-scoped passes). Start with N independent passes; add a head-to-head supplemental pass if the rivalry-content gap shows up in dogfood.
- Cache layer for hosting-model pre-resolution.
- Cross-source disambiguation (not just Polymarket).
- Latency knob for users who want the old one-pass vs behavior (probably not needed; parallel N-pass is ~1× wall clock).
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — main(), `_main_runner`, `_competitor_runner`, the competitor enable/discovery branch. Primary file.
- `scripts/lib/fanout.py` — existing orchestrator (3.0.11). Reused as-is; `competitor_runner` closure is where per-entity kwargs apply.
- `scripts/lib/planner.py``_should_force_deterministic_plan` detects vs-topics via regex. Current path synthesizes ONE comparison plan; new path routes to fanout.
- `scripts/lib/render.py``render_comparison_multi` (3.0.12) + `_render_resolved_entities_block`. Both reused. `render_full` needs a per-entity variant when saving sub-run files.
- `scripts/last30days.py` `save_output` — where raw files are written. Needs to iterate per entity when competitor_reports artifact present.
- `scripts/lib/quality_nudge.py` — BRAVE/SERPER nudge emission.
- `scripts/lib/polymarket.py` — source adapter for `--polymarket-keywords` and ambiguous-topic auto-skip.
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol — the hosting-model contract that drives per-entity pre-resolution for both vs mode and `--competitors`.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built fanout.
- 3.0.12 plan (`2026-04-22-003`): tried engine-internal per-entity auto_resolve; failed without backend keys.
- 3.0.13 plan draft (`2026-04-22-004-...superseded`): proposed `--competitors-plan` JSON + vs-mode-shortcut path but kept them separate. User's 2026-04-22 correction unifies them.
- 2026-04-22 test receipts: Warriors, Seattle, Arizona Wildcats, Kanye West runs all reproduced the per-entity resolve gap.
- User's architectural steer: "vs mode should work that way too" + "--competitors is just vs mode with auto-discovery." This plan encodes that.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **Unify vs-mode and --competitors on one orchestrator.** `fanout.run_competitor_fanout` serves both. vs-mode is "topic contains ' vs '" detection → fanout. `--competitors` is "SKILL.md shortcut → hosting model rewrites topic to vs form → fanout." One code path.
- **Per-entity targeting via `--competitors-plan` JSON.** Schema `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Mirrors `--plan`. Applies to both vs-mode and `--competitors` paths. Hosting model passes it after running Step 0.55 per entity.
- **N save files, one per entity.** Each sub-run writes a `{entity-slug}-raw.md` file when `--save-dir` is set. Matches historical vs-mode behavior. Single-entity runs unchanged.
- **Revert the "one pass for latency" optimization that removed per-entity passes.** Parallel execution via `ThreadPoolExecutor` means wall-clock is ~max(per-entity-latency), not sum. The old latency concern (13+ minutes for 3 serial passes) does not apply to a parallel fan-out.
- **Override-leak fix at the call site.** `_subrun_kwargs(entity, plan_entry)` helper returns fully explicit per-entity kwargs; no closure-default fallthrough from main scope.
- **LAW 7 stderr reframed, not just updated.** Current message treats BRAVE_API_KEY as the solution. New message treats hosting-model Step 0.55 as the solution, with backend keys listed only as the headless fallback.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known-ambiguous single-token list.
## Open Questions
### Resolved During Planning
- **vs mode N passes or single-pass?** N passes. User's architectural correction.
- **Should --competitors still be an engine flag at all?** Yes, kept for headless / cron contexts with backend keys. Dominant Claude Code path is SKILL.md shortcut → vs-mode fanout. Engine flag stays as compatibility surface.
- **`--competitors-plan` JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers → 3-way comparison. Unchanged.
- **Saved-file naming?** `{entity-slug}-raw.md` per entity, same as single-entity runs would produce.
### Deferred to Implementation
- Exact trace of override-leak path (closure capture vs shared config vs Reddit adapter fallback). Test-first per Unit 2; patch at the right layer.
- Heuristic for single-token-ambiguous Polymarket auto-skip. Start with a short hard-coded list; iterate.
- Whether to include a head-to-head rivalry supplemental pass in vs-mode. Ship N-independent passes first; revisit after dogfood if rivalry content is missing.
- Exact filename convention when the comparison merged output is saved (if saved at all). Not blocking — per-entity files are the primary save artifact.
## High-Level Technical Design
> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
```
User invokes:
/last30days "OpenAI vs Anthropic vs xAI"
OR
/last30days OpenAI --competitors (hosting model rewrites to vs form)
OR
/last30days OpenAI --competitors-list "Anthropic,xAI"
OR
/last30days "OpenAI vs Anthropic vs xAI" --competitors-plan '{...per-entity...}'
scripts/last30days.py main():
- Detect: topic has " vs " OR --competitors enabled
- If --competitors and no list/plan: emit LAW 7-style stderr with hosting-model instruction
- If --competitors with list or discovery: rewrite topic to vs form, continue
- Parse --competitors-plan JSON, map to entities
fanout.run_competitor_fanout (shared path):
- For each entity (main + peers):
- entity_config = dict(config) [deep copy to prevent leak]
- kwargs = _subrun_kwargs(entity, plan_entry) [explicit; no main-topic leak]
- If plan_entry missing a field AND backend available: auto_resolve() fill
- pipeline.run(topic=entity, **kwargs, internal_subrun=True)
- Parallel ThreadPoolExecutor
- Collect per-entity Reports
- Attach resolved targeting to each Report.artifacts["resolved"]
scripts/last30days.py after fanout:
- If --save-dir: save each entity's Report as {entity-slug}-raw.md
Each file includes its own single-row Resolved Entities block
- emit_comparison_output → render_comparison_multi (merged stdout)
Includes full N-row Resolved Entities block
```
## Implementation Units
- [ ] **Unit 1: vs-topic detection routes to fanout (not single-pipeline)**
**Goal:** A topic containing ` vs ` / ` versus ` triggers `fanout.run_competitor_fanout` with the parsed entities. Each entity runs a full `pipeline.run()`. Replace the current single-pipeline-with-comparison-plan behavior.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (main() — detect vs-topic, route to fanout)
- Modify: `scripts/lib/planner.py` (remove / bypass the `_should_force_deterministic_plan` special case for vs topics; vs topics no longer go through `plan_query` as a single comparison plan)
- Test: `tests/test_vs_mode_fanout.py` (new)
**Approach:**
- Parse the incoming topic: if it contains ` vs ` or ` versus ` (case-insensitive), split into entities (reuse `planner._comparison_entities`-style logic or move that utility into main()).
- When vs-entities are detected, route to the same fanout branch `--competitors` uses today. The entity list comes from the topic string; no discovery step needed.
- Each entity runs `pipeline.run()` with its own plan (either from `--competitors-plan[entity]` or from the engine's per-entity fallback path).
- For back-compat, if the user passes both a vs-topic AND `--plan`, honor `--plan` for the main (first) entity and use per-entity defaults for peers unless `--competitors-plan` is also provided.
**Execution note:** Start with an integration test that runs `"A vs B"` via mock mode and asserts fanout was called with two entities + two pipeline.run calls.
**Patterns to follow:**
- 3.0.11 fanout wiring in `scripts/last30days.py`'s `--competitors` branch.
- `planner._comparison_entities` for the split logic.
**Test scenarios:**
- Happy path: topic `"A vs B"` → two pipeline.run calls, two Reports returned, merged render.
- Happy path: topic `"A vs B vs C"` → three pipeline.run calls.
- Happy path: topic `"A versus B"` → matches the same regex, two pipelines.
- Edge case: topic `"OpenAI vs"` (trailing empty entity) → treated as single-entity `"OpenAI"`, not vs mode.
- Edge case: topic contains "vs." (dot, no trailing space) → existing regex tolerates it; verify.
- Edge case: topic `"A vs B"` plus `--plan` → plan applies to first entity only, peers use per-entity defaults.
- Integration: full vs-mode run end-to-end in mock mode; verify rendered output, stderr has one `[Competitors] Comparing: A vs B vs ...` line.
**Verification:**
- Test assertions pass.
- Mock-mode smoke of `/last30days "OpenAI vs Anthropic"` shows fanout invocation, per-entity Reports, merged comparison output.
- [ ] **Unit 2: `--competitors-plan` JSON flag + `_subrun_kwargs` helper + override-leak fix**
**Goal:** New JSON flag threads per-entity targeting into each sub-run's `pipeline.run()`. A `_subrun_kwargs(entity, plan_entry)` helper is the single source of truth for per-entity kwargs, eliminating override-leak.
**Requirements:** R3, R6
**Dependencies:** None (can land alongside or before Unit 1)
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner` + `_subrun_kwargs` helper)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; the competitor_runner contract is unchanged)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
- Test: `tests/test_competitor_subrun_isolation.py` (new, regression)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON or file path (mirror `--plan`).
- Validation: top-level dict; each value is a dict; unknown fields log warnings; malformed input exits 2. Case-insensitive entity matching.
- Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`.
- Build `_subrun_kwargs(entity, plan_entry)` — returns an explicit dict with every per-entity flag. No closure-default fallthrough. This is the leak fix.
- `_competitor_runner(entity)`:
1. Get `plan_entry` from `--competitors-plan` if present.
2. Build base kwargs with `_subrun_kwargs(entity, plan_entry)`.
3. Fill missing fields via `resolve.auto_resolve(entity, entity_config)` only if backend is configured (3.0.12 fallback path).
4. Call `pipeline.run(topic=entity, internal_subrun=True, **kwargs)`.
5. Attach `resolved` dict to `report.artifacts`.
- Verify no per-entity flag from main() leaks via closure. The helper is the only source of per-entity values.
**Execution note:** Test-first for the override-leak regression. Use the Kanye 2026-04-22 receipt as the failing test input (main `--subreddits=Kanye,hiphopheads` + `--competitors-list "Drake"` → assert Drake's pipeline.run receives `subreddits=None`).
**Patterns to follow:**
- `--plan` parsing block in `scripts/last30days.py`.
- 3.0.12's `entity_config = dict(config)` deep-copy pattern.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake":{"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's pipeline.run receives `x_handle="Drake"`, `subreddits=["Drizzy"]`. No auto_resolve call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered skip auto_resolve; third falls back.
- Happy path: plan file path accepted like `--plan`.
- Happy path: case-insensitive entity match.
- Edge case: unknown fields → warn, ignore.
- Edge case: plan entry for entity not in list → warn, ignore.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list → exit 2.
- Regression (leak): main `--subreddits=A,B` + `--competitors-list "X"` + no plan → X's pipeline.run gets `subreddits=None`.
- Regression (leak): same for `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`.
- Regression (leak): main `--x-handle=kanye` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanye"`.
**Verification:**
- All regression tests pass.
- Smoke run (mock mode + plan): stderr shows per-entity `[Competitors] {entity}: x=... subs=...` line; no leak from main topic's flags.
- [ ] **Unit 3: Per-entity save files**
**Goal:** When `--save-dir` is set in a vs-mode or `--competitors` run, each entity's sub-run saves its own `{entity-slug}-raw.md` file — same format as a single-entity run would produce.
**Requirements:** R4, R5
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py` (`save_output` iteration after fanout)
- Modify: `scripts/lib/render.py` (`render_full` includes single-row Resolved Entities block when that entity's `artifacts["resolved"]` is present)
- Test: `tests/test_save_raw_per_entity.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]` (or equivalent). For each `(entity, entity_report)`:
- Call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)`.
- Uses entity's `slugify(entity)` for the filename. Same pattern a single-entity run uses.
- Each saved file invokes `render_full` (or the save-variant). `render_full` now checks for `report.artifacts["resolved"]` and prepends a single-row Resolved Entities block.
- Stderr logs one `[last30days] Saved output to <path>` line per entity.
- Single-entity runs unchanged (no extra files, render_full unchanged for them).
**Patterns to follow:**
- Existing `save_output` invocation in main() for single-entity runs.
- `slugify(topic)` for filename.
- 3.0.12's `_render_resolved_entities_block` (reused, single-row mode).
**Test scenarios:**
- Happy path: `/last30days "A vs B vs C" --save-dir=/tmp/x``/tmp/x/a-raw.md`, `/tmp/x/b-raw.md`, `/tmp/x/c-raw.md` exist.
- Happy path: `--competitors-list "Drake,Kendrick" --save-dir=/tmp/x` on topic Kanye → three files: `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`.
- Happy path: each file includes a single-row Resolved Entities block for its entity.
- Happy path: single-entity run with `--save-dir` → one file, no Resolved block (unchanged).
- Edge case: `--save-suffix=v3` → all N files get the suffix.
- Edge case: one entity sub-run failed → its file is NOT saved; the others are.
- Integration: `ls {save-dir}/*-raw.md` returns N files after a vs-mode run.
**Verification:**
- Test assertions pass.
- Manual vs-mode smoke saves N files.
- [ ] **Unit 4: LAW 7-style stderr reframe + footer-nudge suppression**
**Goal:** The `--competitors`-with-no-backend stderr tells the hosting model to do Step 0.55 per entity and pass `--competitors-plan`. The BRAVE/SERPER footer nudge is suppressed when `--plan` or `--competitors-plan` is present.
**Requirements:** R7, R8
**Dependencies:** Unit 2 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the `[Competitors] --competitors requires...` stderr block)
- Modify: `scripts/lib/quality_nudge.py` (or wherever footer nudge emits; verify during implementation)
- Test: `tests/test_competitors_no_backend_message.py` (new)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent with WebSearch), the recommended path: (a) discover N peers via WebSearch, (b) run Step 0.55 for main + each peer, (c) re-invoke as `/last30days 'topic vs peer1 vs peer2' --competitors-plan '{...}'`. See SKILL.md 'Competitor mode'."
2. "Headless / cron path: set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list 'A,B,C'` skips discovery but does not pre-resolve peers."
- Suppress footer nudge when `external_plan` OR `competitors_plan` was passed.
**Test scenarios:**
- Happy path: `--competitors` with no backend, no list, no plan → stderr leads with "If you are the hosting reasoning model" and references `--competitors-plan` before naming API keys.
- Happy path: `--plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-list` only (no plan, no backend) → footer nudge still fires (hosting model didn't fully engage).
- Happy path: no `--competitors`, no `--plan` → footer nudge unchanged.
**Verification:**
- Tests pass.
- [ ] **Unit 5: Polymarket disambiguation guard**
**Goal:** `--polymarket-keywords "kw1,kw2"` filters market matches; auto-skip Polymarket on single-token-ambiguous topics without override.
**Requirements:** R9
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"`. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip: if topic is one token AND matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords`, skip Polymarket with stderr note.
- SKILL.md update (small): mention `--polymarket-keywords` in Step 0.55 instructions for ambiguous topics.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr note.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs, filtered.
- Happy path: topic "OpenAI" → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke → Polymarket footer absent or filtered.
- [ ] **Unit 6: SKILL.md rewrite — vs mode is the canonical path, `--competitors` is a shortcut**
**Goal:** SKILL.md documents the unified architecture. vs mode runs N full passes. `--competitors` is a SKILL.md-level shortcut that discovers 2 peers and invokes vs mode with `--competitors-plan`.
**Requirements:** R1, R2, R10 (surfaces them)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md` (§551 "If QUERY_TYPE = COMPARISON" rewrite; Competitor mode subsection rewrite)
- Modify: `README.md` (one-line example)
**Approach:**
- Rewrite §551 to describe the N-pass architecture: "When the user asks 'X vs Y' (or 'X vs Y vs Z'), run Step 0.55 per entity, then invoke the engine. The engine fans out N full pipelines in parallel. Each entity gets its own single-entity-grade coverage. Wall clock is close to a single run."
- Remove the "ONE research pass with a comparison-optimized plan that replaces the old 3-pass approach" language.
- Add a `--competitors-plan` JSON example.
- Rewrite the Competitor mode subsection: "`--competitors` is a shortcut. The hosting model: (1) runs WebSearch to discover N=2 peers, (2) runs Step 0.55 for main + each peer, (3) rewrites topic to `'main vs peer1 vs peer2'`, (4) invokes engine with `--competitors-plan '{...}'`. Engine flag `--competitors` and `--competitors-list` remain for headless fallback."
- Cross-reference §679 (per-entity Step 0.55 protocol).
- Warning: a thin `## Resolved Entities` block (dashes for any entity) means the hosting model skipped Step 0.55 for that one.
**Patterns to follow:**
- Existing §679 per-entity Step 0.55 protocol for tone.
- 3.0.12 Competitor mode prose for terseness.
**Test scenarios:**
- Test expectation: none — documentation. Verification is dogfood.
**Verification:**
- `/last30days "OpenAI vs Anthropic vs xAI"` in a fresh Claude Code window produces 3 save files with populated Resolved blocks and non-dash per-entity targeting.
- `/last30days OpenAI --competitors` produces same after discovery step.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R10
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG: group the changes. "Changed: vs mode now runs N full passes in parallel, one per entity — reverting the one-pass optimization to restore per-entity depth. Added: --competitors-plan JSON for per-entity Step 0.55 targeting (applies to vs mode and --competitors). Changed: --competitors is now a SKILL.md shortcut for vs-with-discovery. Added: per-entity *-raw.md save files. Fixed: override-leak from main to peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when --plan / --competitors-plan present. Added: --polymarket-keywords + auto-skip for ambiguous topics."
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files.
## System-Wide Impact
- **Interaction graph:** vs-mode and `--competitors` share one orchestrator (`fanout.run_competitor_fanout`). `_subrun_kwargs` is the single source of per-entity kwargs. Save loop iterates per entity.
- **Error propagation:** Per-entity sub-run failure → logged, dropped, continue (3.0.11 behavior unchanged). `--competitors-plan` JSON parse errors exit 2 (same shape as `--plan`).
- **State lifecycle risks:** `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag (Unit 2 fix). No cross-entity context leak.
- **API surface parity:** `--competitors-plan` is additive. `--competitors`, `--competitors-list`, `--plan` unchanged. `--polymarket-keywords` additive. vs-mode keeps its topic-string surface.
- **Integration coverage:** New vs-mode-fanout integration test. New override-leak regression test. New plan-threading test. New nudge-suppression test. New per-entity-save test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. Single-entity render path unchanged. LAW 7 on the default path unchanged (still fires when a single-entity run lacks `--plan`).
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| vs-mode N-pass latency feels slower for users who remember the one-pass shortcut. | Parallel execution keeps wall-clock ~= max(per-entity-latency), not sum. `--quick` on a vs-topic still applies to each sub-run. CHANGELOG calls out the revert + parallelism. |
| API cost scales linearly with N (per source). | Default count 2 caps it. Hard max 6 on `--competitors`. vs-mode users opted into N entities explicitly. |
| Rivalry content ("A vs B" articles) missed in N-independent passes. | Deferred to separate task (head-to-head supplemental pass). Start shipping and observe whether this is actually a gap. |
| Hosting model skips `--competitors-plan` and uses `--competitors-list` only. | Unit 4 stderr reframe steers explicitly. SKILL.md Unit 6 makes the plan-path canonical. Thin Resolved block in output makes skipped-Step-0.55 visible. |
| Override-leak fix misses a subtle closure path. | Unit 2 is test-first with the Kanye receipt as the failing input. Regression test asserts every per-entity flag is None unless plan provides it. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md.
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG explicitly frames the vs-mode change as an architectural revert-with-parallelism, not a regression to the old serial N-pass.
## Sources & References
- Superseded plan: `docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded`
- Previous plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Initial plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts (Warriors, Seattle, Arizona Wildcats, Kanye West)
- SKILL.md §551 + §679 — the per-entity Step 0.55 protocol the hosting model uses for both paths
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/planner.py` vs-topic special-case, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -0,0 +1,87 @@
---
title: "fix: comparison title says (/Last30Days) instead of (Last 30 Days)"
type: fix
status: active
date: 2026-04-22
---
# fix: comparison title says (/Last30Days) instead of (Last 30 Days)
## Overview
User feedback 2026-04-22 on the 3.0.13 release runs (Kanye vs Drake, Mercer Island, Figma): the comparison title currently reads `# Kanye West vs Drake: What the Community Says (Last 30 Days)`. It should read `# Kanye West vs Drake: What the Community Says (/Last30Days)` — attributing the output to the slash command rather than describing the date range generically.
Single-line change in SKILL.md, three occurrences. No code change.
## Requirements Trace
- R1. Comparison title pattern in SKILL.md changes from `(Last 30 Days)` to `(/Last30Days)` so synthesis outputs read `... What the Community Says (/Last30Days)`.
- R2. Both the rule statement (line 113) and the COMPARISON-exception statement (line 131) and the synthesis template example (line 1208) all use the new suffix.
- R3. Version bumps to 3.0.14, CHANGELOG entry, sync, hot-copy. Public cache picks up the new title pattern.
## Scope Boundaries
- No changes to the single-entity output title (no `(/Last30Days)` suffix there — only comparison topics carry it).
- No changes to engine code. Pure SKILL.md content.
- No changes to anything else surfaced in the test runs.
## Key Technical Decisions
- **Replace all three occurrences of the suffix string in one pass.** They are identical strings; changing one without the others would cause synthesis-time confusion when the model reaches a different reference.
- **Ship as 3.0.14, not 3.0.13.x.** Patch-level bump matches the small scope and keeps the release log clean.
## Implementation Units
- [ ] **Unit 1: Replace `(Last 30 Days)` → `(/Last30Days)` in SKILL.md**
**Goal:** All three SKILL.md references to the comparison title use the new suffix.
**Requirements:** R1, R2
**Files:**
- Modify: `SKILL.md`
**Approach:**
- `replace_all` swap of `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Three occurrences, no other strings overlap.
**Test scenarios:**
- Test expectation: none — pure documentation. Verification by inspection + dogfood run.
**Verification:**
- `grep -c "What the Community Says (/Last30Days)" SKILL.md` returns 3.
- `grep -c "What the Community Says (Last 30 Days)" SKILL.md` returns 0.
- [ ] **Unit 2: Version 3.0.14 + CHANGELOG + sync + hot-copy**
**Goal:** Ship 3.0.14 to all local targets.
**Requirements:** R3
**Dependencies:** Unit 1
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.14/`
**Approach:**
- CHANGELOG: "Changed: comparison-mode title attribution — `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range."
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.14.
- Hot-copy contains the updated SKILL.md.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model has the old title pattern memorized from a prior run and re-emits `(Last 30 Days)`. | SKILL.md is read top-to-bottom each invocation. STEP 0 canonical-path self-check (3.0.12) ensures the model loads the new SKILL.md, not the marketplace stale copy. |
## Sources & References
- 2026-04-22 dogfood runs (Kanye West vs Drake, Mercer Island --competitors, Figma --competitors)
- Related code: `SKILL.md` lines 113, 131, 1208
@@ -0,0 +1,396 @@
---
title: "marketing: v3.1 launch video — 30s Remotion piece for X"
type: feat
status: active
date: 2026-04-22
---
# marketing: v3.1 launch video — 30s Remotion piece for X
## Overview
Build a 30-second Remotion-rendered MP4 marketing video announcing **v3.1: Competitors mode** for `/last30days`. Posts to X. Frames the new vs-mode-with-auto-discovery as the headline — three full passes, three save files, one comparison — without burying it in CLI minutiae.
Marketing release tag is **v3.1** (rebrands the 3.0.11-3.0.14 bundle into one shippable narrative for the launch tweet). Engine version stays 3.0.14 — `3.1` is the marketing version, not a code version bump.
## Script (60 frames per second × 30 seconds = 900 frames; this version assumes 30fps × 30s = 900 frames at 30fps)
Total runtime: 30.0 seconds @ 30fps = 900 frames. Six scenes, on-screen text only (silent autoplay-friendly).
### Scene 1 — Hook (0:00 0:03 | frames 0-90)
**Visual:** Black background. The `/last30days` badge animates in (the literal `🌐 last30days v3.1 · synced 2026-04-22` line) with the spring scale-in used in slick devtool intros.
**Caption (overlay, large):**
```
What if one search
ran 3 at once?
```
### Scene 2 — Set the world (0:03 0:08 | frames 90-240)
**Visual:** Single mac terminal window center-stage. Type-on animation:
```
$ /last30days OpenAI
```
Below it, a simple result card stub appears (Reddit upvote count + X likes), then static.
**Caption (small bottom-left):**
```
The old way: one topic.
```
### Scene 3 — The reveal (0:08 0:14 | frames 240-420)
**Visual:** The terminal types one more flag:
```
$ /last30days OpenAI --competitors
```
Hard cut → the single terminal **splits into 3 panes** side by side. Each pane shows a different topic header animating in, in this order:
- Left: `OpenAI`
- Middle: `vs Anthropic`
- Right: `vs xAI`
Pane content scrolls fake "search progress" lines (Reddit, X, YouTube indicators) in parallel, like a live fan-out.
**Caption (top center):**
```
Now it discovers competitors
and runs all 3.
```
### Scene 4 — Result reveal (0:14 0:21 | frames 420-630)
**Visual:** The 3 panes collapse into a single comparison surface — the `## Head-to-Head` table from the actual engine output, with rows fading in one by one (What it is, Streams, Best for, Trajectory). Each entity column lights up as its row populates.
**Caption (bottom):**
```
3 full passes. 3 save files.
1 comparison.
```
### Scene 5 — How it's special (0:21 0:26 | frames 630-780)
**Visual:** Cut to a clean text card, large mono font:
```
You pick the topic.
The agent picks the peers.
The engine fans out.
```
Each line fades in 1.5s apart.
### Scene 6 — CTA (0:26 0:30 | frames 780-900)
**Visual:** Black background. Centered:
- Top: `🌐 last30days v3.1`
- Middle: `/last30days {topic} --competitors`
- Bottom: `github.com/mvanhorn/last30days-skill`
Subtle pulse on the install line.
**End frame holds for ~0.5s.**
## Problem Frame
The 3.0.11-3.0.14 release bundle ships a transformative feature (per-entity vs-mode fanout + `--competitors` shortcut + per-entity save files), but the value lands flat in a tweet thread or screenshot. A 30s video does what static text cannot: shows the fan-out happening in real time and the 3 → 1 collapse into a comparison. Higher tweet engagement, easier to RT/QT.
## Requirements Trace
- R1. Final artifact: a single MP4 file, 1920×1080, 30fps, ~30 seconds (±0.5s), under 30MB, suitable for direct X upload.
- R2. Six-scene script as defined above, with text/visuals/timing matching to within 5 frames.
- R3. Branded look: matches the `🌐 last30days v3.1` badge style (terminal aesthetic, mono font, dark background).
- R4. Silent — no voiceover, no music in v1. Captions baked in. Designed for autoplay-muted feeds.
- R5. Reproducible: another contributor (or a future-me) can re-render with one command. Project lives in-repo so the source is versioned.
## Scope Boundaries
- No voiceover. Text-on-screen only. (Voice can be a v1.1 if engagement is high.)
- No background music in the rendered MP4. (Music can be added in post via QuickTime/iMovie if desired before posting.)
- No localization. English captions only.
- No A/B test variants. One video.
- No 9:16 vertical version. 16:9 only. (Vertical can be a separate render after launch validates the format.)
### Deferred to Separate Tasks
- Voiceover variant: defer to follow-up if v1 lands well.
- 9:16 mobile cut: defer; same source compositions can re-render at 1080×1920 in a follow-up.
- Animated GIF for embedding in README.md: defer; can be ffmpeg-extracted from the MP4.
## Context & Research
### Relevant Code and Patterns
- `SKILL.md` — the comparison render scaffold (`## Head-to-Head` table) is the visual reference for Scene 4's table look.
- `scripts/lib/render.py` `_render_comparison_scaffold` — emits the 9-axis table whose visual style we're recreating in a more polished form.
- `CHANGELOG.md` 3.0.11-3.0.14 entries — the prose source for the script's beats.
- `.claude-plugin/plugin.json` version 3.0.14 — current code version (marketing version is 3.1 for the launch).
### External References
- Remotion 4.x docs (https://www.remotion.dev/docs/) — current API for compositions, sequences, springs, and render CLI.
- X video specs 2026: max 2 min 20 s, ≤512MB, MP4 with H.264 + AAC, recommended 1920×1080 for landscape autoplay.
### Institutional Learnings
- No prior `marketing/` dir or video plans in `docs/plans/`. This is a greenfield asset directory.
## Key Technical Decisions
- **Remotion, not ffmpeg-only.** Remotion's React-based compositions handle the typed-on terminal effect, spring-animated badges, and scene transitions far more cleanly than raw ffmpeg filtergraphs. Render output is still MP4 via Remotion's bundled ffmpeg.
- **In-repo asset directory at `marketing/v3.1-launch/`.** Lives with the product so future versions can fork the project. Adds `marketing/` to `.gitignore` exceptions only for source files; rendered MP4 stays out of git (uploaded separately).
- **16:9 1920×1080 @ 30fps.** Best fit for X landscape autoplay on desktop and mobile feed. 30fps is plenty for typed-text + UI animation; 60fps doubles render time without obvious quality gain.
- **Silent + captions.** X autoplay defaults to muted. Sound-off is the realistic viewing condition. Captions baked into the visual.
- **Six scenes, one composition.** Single Remotion composition with sequenced child compositions per scene. Easier to re-time than scene-files. Frame-numbered timing in the script enables precise edits.
- **Mono font (JetBrains Mono or Geist Mono).** Matches the terminal aesthetic of the actual `/last30days` output. Available via Google Fonts or @remotion/google-fonts.
- **Marketing version 3.1 ≠ engine version 3.0.14.** `3.1` is the launch label. Engine stays 3.0.14. Avoids confusion in CHANGELOG.
## Open Questions
### Resolved During Planning
- **Aspect ratio?** 16:9 1080p. Best X autoplay format; vertical can be a follow-up.
- **Voiceover or silent?** Silent + on-screen captions. Autoplay-muted is the realistic condition.
- **In-repo or separate repo?** In-repo at `marketing/v3.1-launch/`. Rendered MP4 not committed; source compositions are.
- **Length?** Exactly 30s (900 frames @ 30fps). No flex.
- **Marketing version label?** `v3.1`. Engine code version stays 3.0.14.
### Deferred to Implementation
- Exact animation easing curves per scene — pick during build via Remotion preview iteration.
- Whether the comparison-table mock in Scene 4 uses canned text or pulls from the actual saved `*-raw.md` files. Probably canned for visual control.
- Whether Scene 3's "search progress" lines are typed individually or use a marquee scroll. Pick during preview.
- Exact accent color palette beyond "terminal dark." Iterate against preview.
## Output Structure
marketing/
v3.1-launch/
package.json # Remotion dependency manifest
tsconfig.json # TypeScript config
remotion.config.ts # Remotion render config (codec, fps, resolution)
src/
index.ts # Remotion entry — registers compositions
Root.tsx # Root composition definition
LaunchVideo.tsx # Main 30s composition that sequences scenes
scenes/
Scene1Hook.tsx
Scene2OldWay.tsx
Scene3FanOut.tsx
Scene4Comparison.tsx
Scene5HowItWorks.tsx
Scene6CTA.tsx
components/
TerminalWindow.tsx # Reusable mac-style terminal frame
TypedLine.tsx # Type-on animation primitive
ComparisonTable.tsx # The Head-to-Head table mock
BadgeBar.tsx # The 🌐 last30days v3.1 badge
lib/
timing.ts # Frame ranges per scene (single source of truth)
colors.ts # Brand palette
public/ # Static assets (logo, fonts if local)
out/ # Rendered MP4 lives here (gitignored)
README.md # How to preview/render
## High-Level Technical Design
> *Directional guidance for review — not implementation specification.*
```
LaunchVideo (durationInFrames = 900)
├── <Sequence from={0} durationInFrames={90}> <Scene1Hook />
├── <Sequence from={90} durationInFrames={150}> <Scene2OldWay />
├── <Sequence from={240} durationInFrames={180}> <Scene3FanOut />
├── <Sequence from={420} durationInFrames={210}> <Scene4Comparison />
├── <Sequence from={630} durationInFrames={150}> <Scene5HowItWorks />
└── <Sequence from={780} durationInFrames={120}> <Scene6CTA />
```
Per-scene components use `useCurrentFrame()` + `interpolate()` + `spring()` for timing. `TerminalWindow` is the dominant motif across scenes 2-4.
## Implementation Units
- [ ] **Unit 1: Remotion project scaffold**
**Goal:** Spin up a working Remotion project at `marketing/v3.1-launch/` that previews a blank composition and renders to MP4.
**Requirements:** R1, R5
**Files:**
- Create: `marketing/v3.1-launch/package.json`
- Create: `marketing/v3.1-launch/tsconfig.json`
- Create: `marketing/v3.1-launch/remotion.config.ts`
- Create: `marketing/v3.1-launch/src/index.ts`
- Create: `marketing/v3.1-launch/src/Root.tsx`
- Create: `marketing/v3.1-launch/README.md`
- Modify: `.gitignore` (add `marketing/v3.1-launch/out/`, `marketing/v3.1-launch/node_modules/`)
**Approach:**
- Use `npx create-video@latest --blank` (Remotion 4.x scaffolding) targeting `marketing/v3.1-launch/`. Strip the demo composition.
- Configure: 1920×1080, 30fps, H.264, AAC (audio codec needed even for silent — empty track).
- README documents `npm install`, `npm run preview` (Remotion Studio), `npm run render` (one-command MP4).
**Test scenarios:**
- Test expectation: none for scaffold, but verification = `npm run preview` opens Remotion Studio with a blank 30s composition; `npm run render` produces a black MP4 at the right resolution.
**Verification:**
- Studio loads at localhost:3000 with the empty `LaunchVideo` composition listed.
- A render produces `out/launch-video.mp4` at 1920×1080, 30s, valid MP4.
- [ ] **Unit 2: Reusable components (Terminal, TypedLine, BadgeBar)**
**Goal:** Build the three primitive components scenes 2-6 will compose. Each is independently previewable.
**Requirements:** R3, R5
**Dependencies:** Unit 1
**Files:**
- Create: `marketing/v3.1-launch/src/components/TerminalWindow.tsx`
- Create: `marketing/v3.1-launch/src/components/TypedLine.tsx`
- Create: `marketing/v3.1-launch/src/components/BadgeBar.tsx`
- Create: `marketing/v3.1-launch/src/lib/colors.ts`
- Create: `marketing/v3.1-launch/src/lib/timing.ts`
**Approach:**
- `TerminalWindow`: mac-style traffic-light header, dark gradient background, mono content area. Accepts children.
- `TypedLine`: takes a string and a `startFrame`, renders character-by-character at ~30 chars/sec. Reuses Remotion's `interpolate(useCurrentFrame() - startFrame, [0, lengthFrames], [0, text.length])` clamped.
- `BadgeBar`: renders the literal `🌐 last30days v3.1 · synced 2026-04-22` line in mono with the same gradient color treatment as the engine's compact emit.
- `colors.ts`: 5-7 brand colors (terminal-bg, terminal-fg, accent-cyan, accent-magenta, muted, success-green, warning-amber).
- `timing.ts`: exports the scene frame ranges as named constants. Single source of truth for any retiming.
**Test scenarios:**
- Test expectation: none — visual components verified in Remotion Studio.
**Verification:**
- Each component renders standalone in Studio when wrapped in a temporary preview composition.
- TypedLine animates character-by-character without flicker.
- [ ] **Unit 3: Scenes 1-3 (Hook, Old way, Fan-out reveal)**
**Goal:** Build the first half of the video (frames 0-420). The narrative arc up through the visual fan-out.
**Requirements:** R2
**Dependencies:** Unit 2
**Files:**
- Create: `marketing/v3.1-launch/src/scenes/Scene1Hook.tsx`
- Create: `marketing/v3.1-launch/src/scenes/Scene2OldWay.tsx`
- Create: `marketing/v3.1-launch/src/scenes/Scene3FanOut.tsx`
- Modify: `marketing/v3.1-launch/src/Root.tsx` (register sequences)
**Approach:**
- Scene 1: spring-in BadgeBar, large overlay caption, 3-second hold.
- Scene 2: TerminalWindow with TypedLine (`$ /last30days OpenAI`), then a single result-card mock fading in.
- Scene 3: typing animation appends `--competitors`, hard cut, three TerminalWindow components arranged in a row with staggered fade-in. Each pane shows a different entity header + scrolling progress lines.
**Test scenarios:**
- Test expectation: none — verified visually in Studio.
**Verification:**
- Scrub the 0-14s range in Studio; visuals match the script timing within 5 frames.
- The fan-out moment (frame 240) lands cleanly; no jank in the transition from 1 → 3 panes.
- [ ] **Unit 4: Scenes 4-6 (Comparison reveal, How it works, CTA)**
**Goal:** Build the back half of the video (frames 420-900). Resolution + payoff + call to action.
**Requirements:** R2
**Dependencies:** Unit 2
**Files:**
- Create: `marketing/v3.1-launch/src/scenes/Scene4Comparison.tsx`
- Create: `marketing/v3.1-launch/src/scenes/Scene5HowItWorks.tsx`
- Create: `marketing/v3.1-launch/src/scenes/Scene6CTA.tsx`
- Create: `marketing/v3.1-launch/src/components/ComparisonTable.tsx`
**Approach:**
- Scene 4: ComparisonTable component renders 3-column markdown-style table; rows fade in one by one (stagger 15-20 frames). Uses canned data — OpenAI / Anthropic / xAI with believable cell content drawn from real 3.0.13 outputs.
- Scene 5: three-line text card; lines fade in 45 frames apart.
- Scene 6: three centered text blocks; install line gets a 1Hz subtle opacity pulse for emphasis.
**Test scenarios:**
- Test expectation: none — verified visually.
**Verification:**
- Scrub 14-30s; table reveal feels paced (not too slow, not strobed); CTA holds long enough to read (~3-4s).
- ComparisonTable cells are legible at 1920×1080 (mono font ≥ 28px).
- [ ] **Unit 5: Final composition wiring + render**
**Goal:** Wire the six scenes into the master `LaunchVideo` composition, render to MP4, verify against X upload constraints.
**Requirements:** R1, R2, R5
**Dependencies:** Units 3, 4
**Files:**
- Modify: `marketing/v3.1-launch/src/LaunchVideo.tsx` (sequence all 6 scenes)
- Modify: `marketing/v3.1-launch/README.md` (add render command + verification checklist)
**Approach:**
- `LaunchVideo` is a single Composition that imports `Scene1Hook``Scene6CTA` and wraps each in `<Sequence from=… durationInFrames=…>` matching `lib/timing.ts`.
- Run `npx remotion render LaunchVideo out/last30days-v3.1-launch.mp4 --codec=h264 --crf=18`.
- Verify output: 30.0s ±0.1s, 1920×1080, file size <30MB, opens in QuickTime, plays without dropped frames.
**Test scenarios:**
- Test expectation: none — verification is the rendered MP4 itself.
**Verification:**
- `ffprobe out/last30days-v3.1-launch.mp4` reports 1920×1080, 30fps, ~30.0s, h264, faststart-friendly.
- Manual play-through end-to-end in QuickTime feels coherent and on-pace.
- File <30MB so X upload is instant.
- [ ] **Unit 6: Polish pass + ship**
**Goal:** Watch the full render, fix obvious jank, do a second render, and stage for X posting.
**Requirements:** R1, R2, R3
**Dependencies:** Unit 5
**Files:**
- Possibly modify: any scene file based on watch-through findings.
**Approach:**
- Watch the rendered MP4 at full size. Note: timing felt off, transitions too fast, captions overflow, color clash, anything visibly broken.
- Iterate: edit scene component → re-preview in Studio → re-render full MP4.
- Cap at 2 polish passes; ship the better of the two renders.
- Final MP4 sits at `marketing/v3.1-launch/out/last30days-v3.1-launch.mp4` ready for X upload.
**Test scenarios:**
- Test expectation: none — pure subjective polish.
**Verification:**
- User watches the final render and approves.
- No glaring visual bugs (overflowing text, frozen frames, color clashes).
## System-Wide Impact
- **Interaction graph:** None — this is a standalone marketing artifact. Doesn't touch the Python engine, doesn't change any user-facing behavior.
- **State lifecycle risks:** None.
- **API surface parity:** N/A.
- **Unchanged invariants:** The shipped 3.0.14 engine is untouched.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Remotion install pulls 200MB+ of node_modules. | `marketing/v3.1-launch/node_modules/` in `.gitignore`; checked-in source stays small. |
| Render time blows past patience (>5 min for 30s @ 1080p30). | Bun-based render or `--concurrency` flag. Default Remotion is fast enough on M-series Macs. If slow, lower preview to 720p, render final at 1080p. |
| Captions overflow the 1920px width on certain fonts. | Use a known mono font with a measured per-character width; cap caption lines at 36 chars. |
| The "fan-out" visual in Scene 3 looks confusing instead of magical. | Polish pass (Unit 6) is the safety net; if still bad, fall back to a simpler "1 → 3 panes wipe" instead of typed split. |
| File size >30MB hits X upload friction. | Use `--crf=18` (high quality, reasonable size); fall back to `--crf=23` if over. 30s @ 1080p30 H.264 is normally 5-15MB. |
## Documentation / Operational Notes
- README at `marketing/v3.1-launch/README.md` documents preview / render commands.
- After render, the MP4 is uploaded directly to X. Tweet copy is the user's call (this plan stops at the rendered file).
## Sources & References
- Related code: `scripts/lib/render.py` (`_render_comparison_scaffold` is the visual model for Scene 4); `SKILL.md` Competitor mode section (the narrative source).
- Related PRs: #308, #311, #312 (the 3.0.11 → 3.0.14 release bundle this video markets as "v3.1").
- External docs: https://www.remotion.dev/docs/ (Remotion 4.x API).
- X video specs: https://help.x.com/en/using-x/twitter-videos.
+33
View File
@@ -0,0 +1,33 @@
# PR Credits — Thank After V2 Goes Live
When V2 is pushed to the public repo, comment on each PR to thank the contributor and let them know their work was integrated.
## Integrated (cherry-picked into V2)
| PR | Author | What | Status |
|---|---|---|---|
| [#17](https://github.com/mvanhorn/last30days-skill/pull/17) | **@JosephOIbrahim** | Windows Unicode fix (cp1252 emoji crash) | Merge or close with thanks |
| [#16](https://github.com/mvanhorn/last30days-skill/pull/16) | **@levineam** | Handle 403 model access errors + gpt-4.1 fallback | Merge or close with thanks |
| [#18](https://github.com/mvanhorn/last30days-skill/pull/18) | **@jonthebeef** | `--days=N` configurable lookback flag | Merge or close with thanks |
| [#1](https://github.com/mvanhorn/last30days-skill/pull/1) | **@galligan** (Matt Galligan) | Marketplace plugin conversion — we took a lighter approach inspired by his PR | Close with thanks, explain lighter approach |
## Already Fixed in V2 (close with thanks)
| PR | Author | What |
|---|---|---|
| [#15](https://github.com/mvanhorn/last30days-skill/pull/15) | **@rszrszrsz** | YAML argument-hint fix — already fixed in V2 |
| [#11](https://github.com/mvanhorn/last30days-skill/pull/11) | **@nerveband** | Same YAML fix (earlier) — already fixed in V2 |
## Not Integrated (close with explanation)
| PR | Author | What | Why |
|---|---|---|---|
| [#5](https://github.com/mvanhorn/last30days-skill/pull/5) | **@jblwilliams** | Codex auth with OpenAI Responses API | Good idea, too complex for now (358 lines SSE/JWT). May revisit. |
| [#14](https://github.com/mvanhorn/last30days-skill/pull/14) | **@thangman1** | WebSearch-first, API keys optional | Philosophical shift — V2 already does WebSearch in parallel |
| [#10](https://github.com/mvanhorn/last30days-skill/pull/10) | **@thetechreviewer** | OpenRouter API integration | Too large (1029 lines), adds MCP server |
## Suggested Comment Template
> Thanks for this PR! We integrated your [fix/feature] into V2 (commit XXXXX). Really appreciate the contribution. 🙏
>
> Closing this PR since the changes are now in main via a different commit, but full credit to you for the idea and implementation.
+112
View File
@@ -0,0 +1,112 @@
# v3.0.9 - The Self-Debug Release
## Highlights
**v3.0.9 is live.** New user-facing capabilities, broader cross-platform support, and a skill that now runs reliably on Claude Code, Codex, Hermes, Gemini, claude.ai, and OpenClaw. The headline fix: the engine refuses "birthday gift for 40 year old" style queries with a clarifying question instead of 5 minutes of junk output. The headline feature: TikTok and YouTube top comments now render alongside Reddit's, so the most-engaged voice from every source makes it into the synthesis.
**The label - "The Self-Debug Release":** I handed 5 separate Opus 4.7 instances their own failed outputs and asked them to debug themselves. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries outright" and "the WebSearch Sources reminder is overriding LAW 1." I copy-pasted their diagnoses into code. Validation: 5/5 canonical compliance on the topics that had failed.
## New capabilities
- **TikTok and YouTube top comments render alongside Reddit's.** PR [#260](https://github.com/mvanhorn/last30days-skill/pull/260) made the top-engagement comment from each TikTok video and YouTube video first-class in the output - same prominent `💬 Top comment` treatment Reddit's top comment already got. This is the biggest user-facing output change since 3.0.0 and it was never announced. The community inspiration trace: @uppinote20's original push for richer Reddit comments ([PR #143](https://github.com/mvanhorn/last30days-skill/pull/143)) seeded the pattern; this PR generalized it across TikTok and YouTube. PR [#265](https://github.com/mvanhorn/last30days-skill/pull/265) followed up by fixing the ScrapeCreators `url=` param + new response shape for YouTube comments/transcripts so the enrichment actually works.
- **last30days runs on Hermes AI Agent now.** @stephenmcconnachie's PR ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)) added Hermes as a first-class deploy target. `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys the full skill (SKILL.md, scripts, lib modules, fixtures) to Hermes's skills directory alongside Claude Code and Codex. This is one of the biggest surface-area expansions in v3 - last30days is now usable inside the Hermes agent's research workflows without any manual wiring.
- **Multi-key SCRAPECREATORS_API_KEY rotation.** @zaydiscold's PR ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)) added automatic key rotation. Set `SCRAPECREATORS_API_KEY_1`, `SCRAPECREATORS_API_KEY_2`, etc. and the engine rotates when a key hits rate limits instead of failing the whole run. For power users running daily queries, this is the difference between rate-limit 429s and zero-touch reliability.
- **The skill works on Windows now.** @Chelebii's PR ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)) stabilized the vendored Bird X search client on Windows. Previously the bundled X backend had subtle runtime issues on Windows terminals; now it runs clean. Pair this with @Gujiassh's UTF-8 encoding fix ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)) for saved output and Windows users get the full v3 experience without workarounds.
- **Linux permission checks stopped false-warning.** @george231224's PR ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)) fixed `check_perms` on Linux by preferring GNU stat's syntax over the BSD stat that the skill was calling. Linux users were getting spurious permission warnings on `.env` files that were already correctly 600-chmod'd. Now the check matches reality.
- **Gemini CLI got a first-class install path.** @hnshah's docs PR ([#224](https://github.com/mvanhorn/last30days-skill/pull/224)) added the Gemini CLI install note and workaround for a rough edge in the Gemini skill loader. Gemini users now have a one-paragraph install flow in the README instead of having to reverse-engineer the plugin layout.
- **Offline quality evaluation.** @j-sperling's PR ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)) added `eval_topics.json` as a fixture. Contributors and I can now run quality-regression checks on synthesis output without burning live API credits. This is the scaffolding that made the plan 015 validation gate affordable - without eval fixtures, testing 5/5 canonical compliance on every release would cost real money every time. Ships as contributor infrastructure but shows up as stability for end users.
- **Reddit client got a cleaner HTTP layer.** @iliaal shipped three architecture PRs back-to-back ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209)) that consolidated Reddit's HTTP handling into `http.get(params=...)`, rejected garbage input in `_parse_date`, and unified `_sc_headers` into `http.scrapecreators_headers`. End-user benefit: fewer flaky timeouts, fewer "weird parse error" crashes, a codebase that's easier for future contributors to touch without breaking Reddit. These aren't sexy PRs; they're the kind of refactor that prevents six future bug reports.
- **The `--days=N` flag keeps working.** @BryanTegomoh's PR ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)) restored backcompat for the legacy `--days` alias so anyone who'd scripted against it in 2.x doesn't break on v3. Small PR, meaningful reliability gain for existing users.
- **INCLUDE_SOURCES has a sane default.** @hnshah's PR ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)) defaulted the env var to empty string instead of unset. Missing env no longer breaks source inclusion on fresh installs.
- **Version metadata stays in sync.** @Gujiassh's PR ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) aligned the SKILL.md version header with the sync target version, and @shalomma's PR ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)) closed the remaining drift between the SKILL.md header and plugin.json. "Which version am I actually on" is no longer an adventure.
- **Bird X engagement handling got hardened.** @j-sperling's PR ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)) made `bird_x` skip all-None engagement dicts instead of crashing on them. Rare condition, but the kind of thing that silently kills a run on a specific topic.
- **Dev workflow hygiene.** @j-sperling's gitignore PR ([#232](https://github.com/mvanhorn/last30days-skill/pull/232)) dropped `.venv`, `.coverage`, `htmlcov`, and `.memsearch` from the tracked tree. Contributor quality-of-life; keeps PR diffs clean.
- **The skill installs to claude.ai.** PRs [#242](https://github.com/mvanhorn/last30days-skill/pull/242) and [#244](https://github.com/mvanhorn/last30days-skill/pull/244) shipped `scripts/build-skill.sh` plus the `.gitattributes` + `export-ignore` plumbing that packages last30days into a claude.ai-upload-ready `.skill` file under the 200-file cap. The skill is no longer Claude-Code-only - it installs directly on claude.ai, too. README has the upload workflow.
- **OpenAI Codex CLI discovers the skill natively.** PR [#219](https://github.com/mvanhorn/last30days-skill/pull/219) added `.agents/skills/last30days/SKILL.md` as a real file (not symlinked - Codex's loader skips symlinks) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now shows up as `last30days:last30days` when Codex runs in a checkout. Inspired by @Jah-yee ([#153](https://github.com/mvanhorn/last30days-skill/pull/153)) and @dannyshmueli on X.
- **`/last30days` as a slash command.** PR [#267](https://github.com/mvanhorn/last30days-skill/pull/267) added `commands/last30days.md` so plugin users can type `/last30days <topic>` and Claude Code autocomplete prefix-matches it to the canonical `/last30days:last30days` form. No more typing the double-namespace.
## The self-debug technique, for anyone rebuilding this elsewhere
The breakthrough wasn't the individual fixes. It was the realization that instead of guessing why the model was ignoring the rules, I should ask the model. Five separate Opus 4.7 sessions debugged their own outputs:
- "Did you read SKILL.md?" → "I tried Read, hit the 25K token cap, and bailed instead of chunked-reading."
- "Why the trailing Sources block?" → "The WebSearch tool's own reminder said MANDATORY. Precedence was unclear."
- "Why the section headers?" → "I had strong priors on Peter Steinberger and wrote my thesis instead of passing through."
- "Why the wrong file?" → "I read `.agents/skills/last30days/SKILL.md` first because it appeared in the path glob."
Three of the five said "move the LAWs to the top." Two said "make the engine enforce it so the model can't not comply." I shipped both. That's the whole technique: when the LLM-under-orchestration keeps breaking the contract, don't argue with it - ask it to debug itself, and build structural enforcement around whatever it names as the root cause.
## Thank you
**Community PR authors since v3.0.0:**
- @j-sperling - v3 engine architecture, eval fixtures, gitignore hygiene, Bird X hardening ([#232](https://github.com/mvanhorn/last30days-skill/pull/232), [#233](https://github.com/mvanhorn/last30days-skill/pull/233), [#234](https://github.com/mvanhorn/last30days-skill/pull/234))
- @stephenmcconnachie - Hermes AI Agent support ([#228](https://github.com/mvanhorn/last30days-skill/pull/228))
- @zaydiscold - Multi-key SCRAPECREATORS rotation ([#268](https://github.com/mvanhorn/last30days-skill/pull/268))
- @iliaal - Reddit HTTP helper + GitHub date parsing + ScrapeCreators header consolidation ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209))
- @Chelebii - Windows Bird X stability ([#227](https://github.com/mvanhorn/last30days-skill/pull/227))
- @george231224 - Linux check_perms stat ([#216](https://github.com/mvanhorn/last30days-skill/pull/216))
- @Gujiassh - UTF-8 saved output + version metadata alignment ([#217](https://github.com/mvanhorn/last30days-skill/pull/217), [#225](https://github.com/mvanhorn/last30days-skill/pull/225))
- @hnshah - INCLUDE_SOURCES default + Gemini install docs ([#223](https://github.com/mvanhorn/last30days-skill/pull/223), [#224](https://github.com/mvanhorn/last30days-skill/pull/224))
- @shalomma - SKILL.md v3.0.0 version header ([#229](https://github.com/mvanhorn/last30days-skill/pull/229))
- @BryanTegomoh - --days alias backcompat ([#230](https://github.com/mvanhorn/last30days-skill/pull/230))
**v3 roadmap contributors (issues and PRs that shaped the v3 feature set):**
- @uppinote20 - rich Reddit comments ([#143](https://github.com/mvanhorn/last30days-skill/pull/143))
- @zerone0x - GitHub as a first-class source ([#134](https://github.com/mvanhorn/last30days-skill/issues/134), [#136](https://github.com/mvanhorn/last30days-skill/pull/136))
- @thinkun - Reddit enrichment timeout handling ([#116](https://github.com/mvanhorn/last30days-skill/pull/116))
- @thomasmktong - pure-Python Reddit fallback ([#124](https://github.com/mvanhorn/last30days-skill/pull/124))
- @fanispoulinakisai-boop - Reddit timeout report ([#100](https://github.com/mvanhorn/last30days-skill/issues/100))
- @pejmanjohn - plugin directory naming ([#99](https://github.com/mvanhorn/last30days-skill/issues/99), [#78](https://github.com/mvanhorn/last30days-skill/issues/78))
- @zl190 - HN trending merge ([#115](https://github.com/mvanhorn/last30days-skill/pull/115))
- @hnshah - Watchlist features ([#84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86))
- @Jah-yee, @dannyshmueli - Codex CLI discovery
- @Cody-Coyote - marketplace validation bug report ([#204](https://github.com/mvanhorn/last30days-skill/issues/204))
**The five Opus 4.7 instances that debugged their own failures on v3.0.7 and v3.0.8 and converged on the fixes.** The convergence was the breakthrough; this release is their diagnosis in code.
## Install / Update
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
Or if already installed:
```
/plugin update last30days
/reload-plugins
```
## Verify
```
cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version
```
Should print `"version": "3.0.9"`.
## Smoke test
```
/last30days birthday gift for 40 year old
```
Should ask a clarifying question before running. If it runs the engine anyway, the cache is stale - repeat the plugin update.
**Full Changelog:** https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
+48
View File
@@ -0,0 +1,48 @@
# Search Quality Eval
`scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
- runs a baseline revision (default `origin/main`) against a candidate checkout
- evaluates the fixed 5 reviewer topics by default
- computes deterministic stability metrics:
- `Jaccard` overlap vs baseline
- retention vs baseline
- per-source counts and overlap
- optionally calls Gemini as a judge for graded relevance labels and then computes:
- `Precision@5`
- `nDCG@5`
- source-coverage recall across the judged union pool
Recommended usage:
```bash
uv run python scripts/evaluate_search_quality.py
```
Useful flags:
```bash
uv run python scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
--topic "cursor IDE pricing" \
--per-source-limit 5
```
Gemini configuration:
- preferred on this workspace: set `GOOGLE_API_KEY`
- also accepted: `GEMINI_API_KEY` or `GOOGLE_GENAI_API_KEY`
- optional: set `GEMINI_MODEL`
- default model is `gemini-3-pro-preview` for the direct Gemini API
Notes:
- The script forces a clean env-based auth path when it shells out to `last30days.py`.
- It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
- It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either.
- `Jaccard` and retention are regression guards, not truth metrics.
- `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.
@@ -0,0 +1,388 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -0,0 +1,310 @@
# V1 vs V2 Comparison Analysis
**Date:** 2026-02-06
**Queries tested:** 4 (1 head-to-head, 3 V1-only)
**Scope:** Quick smoke test, not full 17-query matrix
---
## Part 1: Head-to-Head -- "kanye west" (NEWS Query)
### Dimension-by-Dimension Scoring
#### 1. Query Parsing Display
Does it show the `🔍 **{TOPIC}** · {QUERY_TYPE}` line before running tools?
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 1 | No parsing display at all. Output starts with "## What I learned:" -- jumps straight into synthesis. No acknowledgment of topic or query type before research. |
| V2 | 1 | No parsing display either. Output starts with "Here's what I found:" then "## What I learned:" -- same problem as V1. |
**Analysis:** Neither version actually rendered the query parsing display. V2 SKILL.md explicitly requires `🔍 **kanye west** · News` before any tools run, but the agent did not produce it. This is a V2 instruction that failed to land. Both score 1/5.
Possible cause: The parsing display is supposed to appear *before* tools are called -- it may have been shown during execution but not captured in the final output text. If so, both outputs represent only the post-research synthesis, not the full session. Regardless, based on what is in the output files, neither shows it.
---
#### 2. Source Coverage (Reddit/X/Web counts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | `Reddit: 0 relevant threads` / `X: 30 posts │ ~10 likes` / `Web: 20+ pages`. Two of three sources returned results. Reddit was zero. |
| V2 | 3 | `Reddit: 0 threads (no results this cycle)` / `X: 29 posts │ 33 likes │ 14 reposts` / `Web: 30+ pages`. Same pattern: two of three returned results. |
**Analysis:** Nearly identical coverage. Both got zero Reddit results (likely a script/API issue for this topic, not a SKILL.md problem). V2 has slightly more precise X metrics (33 likes, 14 reposts vs. V1's vague "~10 likes"). V2 has more web pages (30+ vs 20+). Both miss the 10+ Reddit threshold for a score of 4+.
---
#### 3. Citation Quality (sparse vs every-sentence)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 2 | No inline citations at all. The body text makes claims ("full-page Wall Street Journal apology," "Hellwatt Festival in Italy") but never attributes them to a specific source. The stats box lists "Washington Post, Billboard, AllHipHop" but the body has zero `per @handle` or `per Rolling Stone` attributions. |
| V2 | 5 | Every bold section ends with a sparse, clean citation. Examples: `"per Rolling Stone"`, `"per The Washington Post"`, `"per Billboard"`, `"per AllHipHop"`, `"per The News International"`. One citation per topic, never chained. Exactly what V2 SKILL.md specifies. |
**Analysis:** This is the single biggest quality gap between V1 and V2. V1's output reads like a Wikipedia summary -- informative but ungrounded. V2 reads like a researched briefing where every claim has a named source. V2 nails the "sparse citation" rule from its SKILL.md: `"cite 1 source per pattern, short format: 'per @handle' or 'per r/sub'"`.
V1 quote (no citation): `"He'll headline the new Hellwatt Festival in Italy (July 4-18, 2026)."`
V2 quote (cited): `"Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard."`
---
#### 4. Summary Structure (bold topic headers, organized sections)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Has a coherent narrative structure with a paragraph of synthesis, then a `**KEY THEMES:**` numbered list. But the opening is a single dense paragraph, not broken into scannable sections with bold headers. |
| V2 | 5 | Each storyline gets its own bold header: `**BULLY Album — March 20, 2026 via Gamma**`, `**Public Apology for Antisemitism**`, `**Hellwatt Festival in Italy**`, `**Health Concerns**`, `**Grammys Ban**`, `**Kim & Lewis Hamilton Buzz**`. Each is a standalone scannable unit with 1-3 sentences. |
**Analysis:** V2 follows the SKILL.md template exactly: `**{Topic 1}** — [1-2 sentences, per source]`. V1 uses a blob + list approach which is readable but less scannable. V2 is notably better for a user who wants to skim and find the story they care about.
V1 structure: 1 dense paragraph -> 5-item `KEY THEMES` list
V2 structure: 6 bold topic cards, each self-contained -> no KEY THEMES list (but doesn't need one because the structure itself is the organization)
---
#### 5. Stats Box Format (emoji tree vs plain text)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Uses `├─` tree format with emoji: `├─ 🟠 Reddit: 0 relevant threads` / `├─ 🔵 X: 30 posts` / `├─ 🌐 Web: 20+ pages` / `└─ Top voices:`. Minor deviation: says "0 relevant threads (filtered out noise)" instead of the V1 SKILL.md template "0 threads (no results this cycle)". Also omits the `🗣️` emoji on the Top voices line. |
| V2 | 5 | Perfect match to V2 SKILL.md template: `├─ 🟠 Reddit: 0 threads (no results this cycle)` / `├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)` / `├─ 🌐 Web: 30+ pages │ rollingstone.com, ...` / `└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex`. Includes `(via xAI)` notation, `🗣️` emoji, @handles with engagement counts. |
**Analysis:** V2 is tighter and matches its template exactly. V1 is close but has minor deviations (custom "filtered out noise" text, missing `🗣️` emoji, no @handles or engagement counts on Top voices). V2's inclusion of actual @handles with like counts (`@honest30bgfan_ (33 likes)`) adds credibility.
---
#### 6. Research Grounding (actual research vs generic knowledge)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Clearly grounded: mentions specific details like "Wall Street Journal apology (Jan 26, 2026)," "four-month-long manic episode," "frontal-lobe brain injury," "North West collaborated on 'Piercings on My Hand,'" "Monumental Plaza de Toros." These are specific enough to be from research, not pre-training. Minor generic leakage: the "KEY THEMES" list uses editorial framing ("Accountability arc," "Mental health transparency") that feels more like analysis than research extraction. |
| V2 | 5 | Every fact is specific and attributed: "12th studio album," "13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign," "earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded," "103,000-capacity RCF Arena." The AI-deepfaked vocals detail is a standout -- it is clearly from research, not something a model would know from pre-training. The Kim/Lewis Hamilton item (`"X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton"`) is explicitly sourced from X data, not general knowledge. |
**Analysis:** Both are well-grounded, but V2 has more "could only come from research" details. The deepfaked vocals story, the exact venue capacity, and the explicit X chatter observation are details that prove the synthesis is from the research output, not hallucinated.
---
#### 7. Prompt Quality (invitation to share vision, not dumping prompts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Ends with: `"Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in."` This is a follow-up invitation, but it is NOT the SKILL.md-specified invitation. It is topic-specific and conversational, which is nice, but it does not ask the user to "share your vision for what you want to create." It misses the prompt-generation angle entirely. |
| V2 | 5 | Ends with exactly: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice."` This matches the V2 SKILL.md template verbatim. It positions the skill correctly: not a news summarizer but a research-to-prompt pipeline. |
**Analysis:** V1's closing is friendly but off-brand. It treats the skill as a research tool, not a research-to-prompt tool. V2 correctly frames the next step as "tell me what to create and I'll write the prompt." This is a meaningful difference -- V1 would leave a user thinking they just got a summary, while V2 primes them to get a usable output.
---
### Head-to-Head Scorecard
| Dimension | V1 | V2 | Winner |
|-----------|----|----|--------|
| 1. Query Parsing Display | 1 | 1 | Tie (both failed) |
| 2. Source Coverage | 3 | 3 | Tie |
| 3. Citation Quality | 2 | 5 | **V2 (+3)** |
| 4. Summary Structure | 3 | 5 | **V2 (+2)** |
| 5. Stats Box Format | 4 | 5 | **V2 (+1)** |
| 6. Research Grounding | 4 | 5 | **V2 (+1)** |
| 7. Prompt Quality (invitation) | 3 | 5 | **V2 (+2)** |
| **TOTAL** | **20/35** | **29/35** | **V2 wins by 9 points** |
**V2 is clearly better.** The biggest gaps are citation quality (+3) and summary structure (+2). V2's output reads like a professional research briefing; V1's reads like a decent but unstructured summary.
---
## Part 2: V1-Only Outputs Analysis
### Output 1: "open claw" (GENERAL query)
**What V1 does well:**
- Strong research grounding. Mentions exact numbers: "145,000+ GitHub stars," "20,000+ forks," "700+ skills," "341 malicious skills." These are clearly from research.
- The KEY PATTERNS section is excellent: 5 well-organized patterns with community quotes (`"I give it sudo and let it configure everything"` vs `"prompt injection is terrifying when you give the bot access to your actual bank account"`).
- Good synthesis of the security vs. enthusiasm tension -- captures the community split accurately.
- Stats box uses the emoji tree format correctly with `├──` (though note: uses double-dash `──` instead of single `─`, minor inconsistency).
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display (`🔍 **open claw** · General`).
- No inline citations in the body text. The 5 KEY PATTERNS have no `per @handle` or `per r/sub` attribution. Which Reddit thread said "I give it sudo"? Which X post raised the security concern? We do not know.
- The stats box says `├── 🟠 Reddit: 25 threads │ ~750+ upvotes` -- the tilde and plus are imprecise. V2 SKILL.md wants exact parsed numbers.
- Top voices line lists subreddits and handles but no engagement counts: `@grok, @Starlink` -- are these the highest-engagement handles? No like counts shown.
- No bold topic headers in the body -- it is a single paragraph followed by a numbered list, not the `**{Topic}** — sentence, per source` format V2 requires.
**V1 Score (estimated):** 22/35
---
### Output 2: "nano banana pro prompting" (PROMPTING query)
**What V1 does well:**
- Correctly identifies two prompting styles (JSON structured vs. natural language "Creative Director") and explains when each works best. This is excellent PROMPTING-type synthesis.
- KEY PATTERNS are specific and actionable: "85mm lens at f/1.8," "three-point lighting with key at 45 degrees," "text rendering works -- keep text under 3 words for best results (75% success rate)." These are concrete tips a user can apply immediately.
- Research grounding is strong: cites specific upvote counts ("149-259 upvotes"), subreddit names (`r/nanobanana2pro`), and the Google AI blog.
- The invitation correctly targets Nano Banana Pro: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro."`
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Stats box uses plain text dashes: `- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments` instead of the tree format `├─ 🟠 Reddit:`. Uses `|` pipe instead of `│` box-drawing character. V2 SKILL.md explicitly says: "NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji."
- No inline body citations. KEY PATTERNS mention Reddit upvote ranges but no specific `per @handle` attributions.
- Missing `✅ All agents reported back!` header -- just says "All agents reported back!" without the checkmark.
- Body structure is paragraph + numbered list, not bold topic headers.
**V1 Score (estimated):** 23/35 (slightly higher than open claw due to better actionability)
---
### Output 3: "how to best setup clawdbot" (HOW-TO query)
**What V1 does well:**
- This is the best V1 output of the batch. It goes beyond synthesis and actually delivers a **Quick-Start guide** with numbered steps, a **Security Hardening** checklist, and a **Budget Option** -- all grounded in research.
- Excellent research grounding: `"per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0"` -- this is an actual citation with an @handle!
- Specific, actionable recommendations: exact commands (`curl -fsSL https://clawd.bot/install.sh | bash`), specific model recommendations (Claude Opus 4.5 for best results, GLM 4.7 Flash for local), specific channel advice (Telegram first, WhatsApp QR code fails).
- Stats box is correct emoji tree format with engagement counts: `@aashatwt (452 likes), @recap_david (329 likes)`.
- Captures the naming confusion accurately: "Clawdbot -> Moltbot -> OpenClaw."
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Body text has no inline citations except the Budget Option section. The 5 KEY PATTERNS have no `per @handle` attribution.
- Bold topic headers are used only in the Quick-Start and Security sections, not in the KEY PATTERNS or intro.
- The output delivers the "answer" directly (setup guide) rather than waiting for the user's vision and offering to write a prompt. For a HOW-TO query this might be the right call, but it skips the SKILL.md flow of "show research -> invite vision -> write prompt."
**V1 Score (estimated):** 26/35 (best of the V1 outputs)
---
### Patterns Across All V1 Outputs
**Consistent strengths:**
1. Research grounding is solid across all three. V1 does not hallucinate -- the facts are clearly from the research output, not pre-training.
2. KEY PATTERNS lists are consistently useful and actionable.
3. Stats boxes are present in all outputs (though formatting varies).
4. The invitation/closing line is present in all outputs.
**Consistent weaknesses:**
1. **No query parsing display** in any output (0 for 4, including Kanye West).
2. **No inline citations** in the body text (except one @handle in the clawdbot output). The research feels real but is unattributed.
3. **Stats box formatting is inconsistent.** Open claw uses `├──` (double dash), nano banana pro uses `- 🟠` (plain dash + pipe), clawdbot uses `├─` (correct). Three different formats in three outputs.
4. **Body structure defaults to paragraph + numbered list** instead of bold topic headers. Only clawdbot partially uses bold headers (in the guide section, not the research section).
5. **No `(via Bird/xAI)` notation** on X stats in any output.
---
## Part 3: SKILL.md Feature Diff
### Features in V2 but NOT V1
| Feature | V2 Lines | Impact |
|---------|----------|--------|
| **Query parsing display** (`🔍 **{TOPIC}** · {QUERY_TYPE}`) | 40-53 | HIGH -- confirms to user the skill understood their request before spending time on research. |
| **Sparse citation rules** with BAD/GOOD examples | 186-193 | HIGH -- this is the #1 quality differentiator in the Kanye head-to-head. `"per @handle"` format, never chain multiple citations. |
| **Bold topic headers** template (`**{Topic 1}** — [1-2 sentences, per source]`) | 195-208 | HIGH -- makes output scannable. |
| **Strict stats template** with "NEVER use plain text dashes" instruction | 217-230 | MEDIUM -- prevents the formatting inconsistency seen across V1 outputs. |
| **RECOMMENDATIONS source attribution** (each item MUST have Sources: line with @handles) | 178-182 | MEDIUM -- only affects RECOMMENDATIONS queries. |
| **Reddit 0 results handling** (explicit instruction for what to write) | 229 | LOW -- edge case, but prevents ad-hoc text like V1's "filtered out noise." |
| **Bird CLI / xAI notation** in stats | 223 | LOW -- cosmetic transparency about data source. |
| **Step 2 phrasing: "DO WEBSEARCH WHILE SCRIPT RUNS"** | 71-73 | LOW -- execution optimization, no output impact. |
### Features in V1 but NOT V2
| Feature | V1 Lines | Impact | Should Restore? |
|---------|----------|--------|-----------------|
| **Use cases block** (4 examples in intro) | 12-17 | LOW | No |
| **Setup Check section** (3 modes, bash script, "keys are OPTIONAL") | 50-78 | MEDIUM for new users | Yes, for public release |
| **BAD/GOOD synthesis anti-pattern examples** | 172-191 | MEDIUM-HIGH | YES |
| **Self-check instruction** ("Re-read your 'What I learned' section...") | 269 | MEDIUM | YES |
| **Quality Checklist** (5-point checklist before delivering prompt) | 306-324 | HIGH | YES |
| **Prompt format anti-pattern** ("Research says JSON but you write prose") | 302 | MEDIUM | YES |
| **"IF USER ASKS FOR MORE OPTIONS"** section | 327-329 | LOW-MEDIUM | YES |
| **Web-only mode stats template + promo** | 248-259 | MEDIUM for no-key users | For public release |
| **TARGET_TOOL question template** (4 options) | 272-280 | LOW | No |
| **Context Memory: explicit "don't re-search" instructions** | 342-358 | MEDIUM | YES |
| **Output footer emoji + engagement counts** | 366-380 | LOW | YES |
### Features in BOTH (Shared)
| Feature | Notes |
|---------|-------|
| Parse User Intent (TOPIC, TARGET_TOOL, QUERY_TYPE) | Same 4 query types, same detection logic |
| "Don't ask about tool before research" rule | Identical |
| Research script execution command | Same `python3` command |
| WebSearch queries by QUERY_TYPE | Same search strategies |
| "Use user's exact terminology" instruction | V2 shorter but same intent |
| Judge Agent synthesis logic | Same 5-step weighting process |
| "Ground in actual research" instruction | Same core instruction, V1 has more examples |
| RECOMMENDATIONS: extract specific names | Same logic |
| Prompt format matching | Same instruction |
| Wait for user's vision | Same |
| Write ONE perfect prompt | Same structure |
| Context Memory | V2 shorter version |
| Output summary footer | Both have it, V1 has emoji |
| Depth options (quick/default/deep) | Same |
| "After each prompt: Stay in Expert Mode" | Same |
### Overall Assessment
**V2 is a clear upgrade in output formatting and citation quality.** The three features V2 adds (query parsing display, sparse citation rules, bold topic headers) directly address the three biggest weaknesses seen across all V1 outputs. The Kanye West head-to-head proves it: V2 scores 29/35 vs V1's 20/35.
**However, V2 dropped several quality guardrails from V1** that do not affect formatting but affect *correctness*: the self-check instruction, the anti-pattern examples, the quality checklist for prompts, and the "don't re-search" context memory rule. These are cheap to restore (under 25 lines total) and protect against subtle failure modes that may not show up in a 1-query test but will appear over dozens of uses.
---
## Part 4: Verdict
### Ship V2 or Not?
**Ship V2 -- but restore the guardrails first.**
V2 is unambiguously better on every formatting dimension. The citation quality improvement alone (V1: 2/5 -> V2: 5/5) makes it worth shipping. The bold topic headers and strict stats template fix the inconsistency problems visible across all V1 outputs.
But V2 dropped 6 guardrail features from V1 that cost almost nothing to include and protect against real failure modes. These should be restored before V2 goes public.
### Remaining Gaps
**Must fix before shipping (affects correctness):**
1. **Restore the quality checklist for prompts.** This is the test plan's #1 priority item. V1 had a 5-point checklist; V2 reduced it to one line. The checklist is what makes prompts feel polished -- it is the "that's a great prompt" mechanism. Add 8 lines.
2. **Restore BAD/GOOD anti-pattern examples.** V2 says "ground in actual research" but does not show what *bad* grounding looks like. V1's ClawdBot/Claude Code conflation example is exactly the kind of concrete negative example that prevents real failures. Add 5 lines.
3. **Restore self-check instruction.** One sentence: "Re-read your 'What I learned' section -- does it match what the research ACTUALLY says?" Zero cost, catches hallucination. Add 2 lines.
4. **Restore "don't re-search" context memory rule.** V2 only says "only do new research if user asks about a DIFFERENT topic." V1 explicitly bans re-searching and tells the agent to answer from existing research. Add 3 lines.
**Should fix (polish):**
5. Restore prompt format anti-pattern ("Research says JSON but you write prose"). Add 2 lines.
6. Restore "IF USER ASKS FOR MORE OPTIONS" section. Add 2 lines.
7. Add emoji + engagement counts back to the output summary footer. Edit 3 lines.
**Skip for now:**
8. Setup Check section -- add back for public release, not needed for execution.
9. Web-only mode stats template -- lower priority, most testers have API keys.
10. TARGET_TOOL question template -- agent handles this naturally.
### Query Parsing Display: Investigate
Both V1 and V2 scored 1/5 on query parsing display. V2 has the feature in its SKILL.md but the agent did not render it in the captured output. This could mean:
- The display was shown during execution but not captured (likely -- it appears before tools run, and the output files may only contain post-research content).
- The instruction is not strong enough and the agent skips it.
**Recommendation:** Verify in a live session whether the parsing display actually appears. If it does not, strengthen the instruction (e.g., "This line MUST be the first thing you output, before any tool calls").
### Total Effort
Restoring all 7 priority items: approximately 25 lines added to V2 SKILL.md. Under 15 minutes of work. The V2 formatting wins are substantial and proven; the V1 guardrails are small and proven. Combining both produces the best version.
### Final Score Summary
| | V1 (Kanye) | V2 (Kanye) | Delta |
|--|-----------|-----------|-------|
| Total | 20/35 | 29/35 | **V2 +9** |
| | V1 (Open Claw) | V1 (Nano Banana) | V1 (Clawdbot) | V1 Average |
|--|---------------|-----------------|--------------|------------|
| Estimated Total | 22/35 | 23/35 | 26/35 | **23.7/35** |
V2 at 29/35 beats every V1 output, including V1's best (clawdbot at 26/35).
**Decision: Ship V2 with guardrails restored.**
@@ -0,0 +1,388 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -0,0 +1,25 @@
## What I learned:
The last 30 days have been one of Kanye West's most significant periods in years. The biggest story is his **full-page Wall Street Journal apology** (Jan 26, 2026) titled "To Those I've Hurt" — a comprehensive open letter addressing his antisemitic remarks and erratic behavior. He attributed his actions to a "four-month-long manic episode" and revealed that a car accident 25 years ago caused a frontal-lobe brain injury that wasn't properly diagnosed until 2023, leading to his bipolar type-1 diagnosis. Multiple sources describe wife **Bianca Censori** as instrumental in getting him to seek treatment during a manic depressive episode.
On the music side, he's announced **two upcoming albums**: *BULLY* (now scheduled for March 20, 2026) and *IN A PERFECT WORLD*. Daughter **North West** collaborated with him on a track called "Piercings on My Hand." He performed a massive show in Mexico City (Jan 30) at the Monumental Plaza de Toros, and he'll headline the new **Hellwatt Festival** in Italy (July 4-18, 2026).
The **2026 Grammys** remain contentious — he's reportedly not invited after last year's red carpet drama. Health concerns also surfaced after a January LA outing where he appeared "noticeably bloated and moving with sluggish energy."
**KEY THEMES:**
1. **Accountability arc** — The WSJ apology is being widely discussed as either genuine growth or calculated PR
2. **Mental health transparency** — Brain injury + bipolar diagnosis framing his past behavior
3. **Bianca Censori's role** — Increasingly seen as a stabilizing force (despite continued provocative public appearances)
4. **Music comeback** — Two albums announced, active touring in Latin America and Europe
5. **Grammy exile** — Still persona non grata at major award shows
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 relevant threads (filtered out noise)
├─ 🔵 X: 30 posts │ ~10 likes │ casual fan chatter + Bianca Censori discourse
├─ 🌐 Web: 20+ pages │ washingtonpost.com, billboard.com, allhiphop.com, euronews.com, rap-up.com
└─ Top voices: Washington Post, Billboard, AllHipHop, Euronews, The Mirror
---
Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in.
@@ -0,0 +1,24 @@
## What I learned:
OpenClaw (formerly ClawdBot, formerly Moltbot) is an open-source autonomous AI agent created by Peter Steinberger that went massively viral in late January 2026 — hitting 145,000+ GitHub stars and 20,000+ forks. It runs locally on your machine, connects to messaging platforms (Telegram, WhatsApp, Discord, Signal, etc.), and uses LLMs like Claude, GPT, or Gemini to actually perform tasks on your computer — file management, scheduling, web browsing, purchases, and more. Think "JARVIS for everyone."
The community is simultaneously excited and alarmed. The "ClawHub" skill marketplace has 700+ skills but a VirusTotal/Koi Security audit found **341 malicious skills** doing data exfiltration and crypto theft. A high-severity RCE vulnerability was also disclosed. The #1 most downloaded skill on ClawHub was confirmed malware.
**KEY PATTERNS from the research:**
1. **Cost is the top complaint** — API usage burns money fast; many users recommend Gemini CLI (free 1,000 req/day) over Claude/GPT for OpenClaw to control spend
2. **Security is a dumpster fire** — Supply chain attacks via ClawHub skills, prompt injection risks when browsing the web, RCE vulnerabilities, and orgs are banning it at the EDR level
3. **Setup is nontrivial** — Even after install, most users get stuck on skills/tools/automation config; the `openclaw onboard` wizard is the recommended path
4. **The skill ecosystem is powerful but dangerous** — 700+ skills available, but no code signing, no real moderation, and active malware campaigns
5. **Community is split** — Enthusiasts love the autonomy ("I give it sudo and let it configure everything"), while security folks are terrified ("prompt injection is terrifying when you give the bot access to your actual bank account")
---
✅ All agents reported back!
├── 🟠 Reddit: 25 threads │ ~750+ upvotes │ ~600+ comments
├── 🔵 X: 27 posts │ 1,750+ likes │ 325+ reposts
├── 🌐 Web: 20+ pages │ Wikipedia, Bloomberg, The Register, Hacker News, Nature, VirusTotal, DigitalOcean, Codecademy, Tom's Hardware
└── Top voices: r/openclaw, r/cybersecurity, r/clawdbot, r/ArtificialInteligence │ @grok, @Starlink │ Nature, Bloomberg, ACM
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly.
@@ -0,0 +1,27 @@
Now I have a comprehensive picture. Let me synthesize everything.
**What I learned:**
Nano Banana Pro is fundamentally different from older image models — it's a "thinking" model that understands intent, physics, and composition. The community has converged on two distinct prompting styles, and **both work well** depending on the use case:
1. **JSON structured prompts** — Dominant on Reddit (r/nanobanana2pro) and X. Power users share elaborate JSON objects with nested fields for subject, appearance, lighting, composition, and style. These produce the most consistent, controlled results for portraits, editorials, and product shots.
2. **Natural language "Creative Director" prompts** — Recommended by Google's official guide. Full sentences describing the scene as if briefing a photographer. Better for cinematic shots, complex edits, and iterative refinement.
**KEY PATTERNS I'll use:**
1. **JSON format for maximum control** — The top Reddit posts (149-259 upvotes) all use structured JSON with nested objects: `subject > appearance > hair/skin/expression`, `lighting > key/fill/rim`, `camera > lens/angle/depth_of_field`
2. **Be a Creative Director, not a tag spammer** — Replace "4k, realistic, beautiful" with specific technical directions: "85mm lens at f/1.8", "three-point lighting with key at 45°", "shallow depth of field with bokeh"
3. **Provide context for WHY** — Adding purpose ("for a luxury fashion editorial", "for a high-end gourmet cookbook") helps the model infer appropriate styling, plating, lighting
4. **Edit, don't regenerate** — When 80% is right, ask for the specific change conversationally
5. **Text rendering works** — Keep text under 3 words for best results (75% success rate), specify font style and placement
---
All agents reported back!
- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments
- 🔵 X: 29 posts | 312 likes | 117 reposts
- 🌐 Web: 10+ pages | blog.google, dev.to, eweek.com, imagine.art, github.com, leonardo.ai, medium.com
- Top voices: r/nanobanana2pro | @Xmira_belle, @doctorwasif, @chatgptpaglu | Google AI blog, DEV Community
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro.
@@ -0,0 +1,48 @@
**What I learned:**
Clawdbot (now officially renamed **OpenClaw**) is a self-hosted, open-source personal AI assistant that connects to messaging apps (Telegram, WhatsApp, Discord, Slack, Signal, iMessage) and uses LLMs to act as an autonomous agent on your behalf. It's roughly one week old as a public project and the community is moving fast — the name has already changed twice (Clawdbot → Moltbot → OpenClaw).
**KEY PATTERNS from the research:**
1. **Security is the #1 concern** — Multiple Reddit threads and guides warn that many setups are wide open to the internet with zero auth. Run it on dedicated/isolated hardware (VM, Mac Mini, NAS), never your main machine. Block dangerous commands via allowlist. Never add it to group chats.
2. **Use the onboarding wizard, not Docker** — Docker was reported as unreliable (shims, bridges). The single-line install script (`curl -fsSL https://clawd.bot/install.sh | bash`) followed by `openclaw onboard` is the smoothest path. Docker is a "no-go" per multiple users.
3. **Model choice matters a lot** — Claude Opus 4.5 is recommended for best results (trained to resist prompt injection, ~99% resistance). For budget setups, Claude Haiku works at $0. Local models (Ollama/LM Studio) work but cause hallucination and token issues — GLM 4.7 Flash (30B) was recommended as the best local option.
4. **Telegram is the easiest channel to start with** — Most guides and users start with a Telegram bot token. WhatsApp QR code pairing fails frequently. Get Telegram working first, then expand.
5. **Sandbox and isolate** — Set `agents.defaults.sandbox.mode` to `"non-main"` for Docker sandboxing. Use a Linux VM or UTM on Mac to segregate traffic. Tailscale for secure remote access.
---
✅ All agents reported back!
├─ 🟠 Reddit: 19 threads │ 415+ upvotes │ 290+ comments
├─ 🔵 X: 29 posts │ 815+ likes │ 65+ reposts
├─ 🌐 Web: 10+ pages │ Medium, DataCamp, DEV, GitHub, QNAP, docs.openclaw.ai
└─ Top voices: r/clawdbot, r/LocalLLM, r/AskVibecoders │ @aashatwt (452 likes), @recap_david (329 likes) │ Lukas Niessen on Medium
---
**Here's the consensus best setup path based on what the community actually recommends:**
### Quick-Start (30 min)
1. **Get dedicated hardware** — Mac Mini, old laptop, NAS (QNAP has an official guide), or a VPS (DigitalOcean gist available). Don't use your daily driver.
2. **Install via script**: `curl -fsSL https://clawd.bot/install.sh | bash`
3. **Run the wizard**: `openclaw onboard` — it walks through gateway, workspace, channels, and skills
4. **Connect Telegram first** — create a bot via @BotFather, paste the token
5. **Set your API key** — Anthropic API key for Claude, or connect Ollama for local models
6. **Customize SOUL.md** — this controls your bot's personality and behavior
### Security Hardening (do this immediately)
- Bind gateway to localhost only, use Tailscale for remote access
- Set sandbox mode to `"non-main"`
- Create an explicit command allowlist (block `rm -rf`, `sudo`, `chmod`)
- Rotate API keys every 90 days
- Never add the bot to group chats
### Budget Option ($0)
Per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0.
---
Want me to help you with a specific part of the setup, or do you have a particular use case in mind (home automation, CRM, coding assistant, etc.)?
@@ -0,0 +1,332 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
---
## STEP 2: DO WEBSEARCH WHILE SCRIPT RUNS
The script auto-detects sources (Bird CLI, API keys, etc). While waiting for it, do WebSearch.
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords?
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
---
## THEN: Show Summary + Invite Vision
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned with sources:
```
🏆 Most mentioned:
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle3, r/sub2, Complex
Notable mentions: [other specific things with 1-2 mentions]
```
**CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson)
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety)
- Parse @handles from research output and include the highest-engagement ones
- Format naturally - tables work well for wide terminals, stacked cards for narrow
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
**BAD:** "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
**GOOD:** "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
**THEN - Stats (right before invitation):**
**CRITICAL: Calculate actual totals from the research output.**
- Count posts/threads from each section
- Sum engagement: parse `[Xlikes, Yrt]` from each X post, `[Xpts, Ycmt]` from Reddit
- Identify top voices: highest-engagement @handles from X, most active subreddits
**Copy this EXACTLY, replacing only the {placeholders}:**
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
```
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT.**
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Quality Checklist (run before delivering):
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS]
---
This uses [brief 1-line explanation of what research insight you applied].
```
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
@@ -0,0 +1,25 @@
Here's what I found:
## What I learned:
**BULLY Album — March 20, 2026 via Gamma** — After years of delays (first announced September 2024), Kanye's 12th studio album finally has a firm release date. The 13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign. Sonically it recalls *808s & Heartbreak* and *MBDTF* — Ye mostly sings rather than raps. Notably, earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded with his real voice for the official release, per Rolling Stone.
**Public Apology for Antisemitism** — On January 26, Ye took out a full-page Wall Street Journal ad titled "To Those I've Hurt," publicly apologizing for his antisemitic remarks. He disclosed a previously undiagnosed brain injury from his 2002 car accident and attributed his behavior to a four-month manic episode fueled by bipolar disorder, psychosis, and paranoia. "I lost touch with reality," he wrote. The apology also extended to the Black community, per The Washington Post.
**Hellwatt Festival in Italy** — Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard.
**Health Concerns** — A rare January 2026 outing in LA with Bianca Censori reignited concern about Ye's physical appearance. Insiders point to medication side effects, frequent travel, and inconsistent routines, per AllHipHop.
**Grammys Ban** — Ye is reportedly not welcome at the 2026 Grammy Awards after clashing with organizers last year over his invitation terms, per The News International.
**Kim & Lewis Hamilton Buzz** — X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton, with users contrasting her new relationship against her marriage to Ye.
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 threads (no results this cycle)
├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)
├─ 🌐 Web: 30+ pages │ rollingstone.com, washingtonpost.com, complex.com, billboard.com, npr.org
└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice.
+120
View File
@@ -0,0 +1,120 @@
# v2.1 Launch Copy (DRAFT — do not publish)
## Context
Bird CLI (@steipete/bird) has been deprecated on npm and the GitHub repo deleted. steipete was asked to take it down (likely by X). Nobody has forked and republished it. Our v2.1 vendors a search-only subset of Bird v0.8.0 (MIT licensed) so users don't need any external tools.
YouTube transcripts are the second headline feature. Inspired by Peter Steinberger's yt-dlp + summarize toolchain. We use yt-dlp directly (no summarize dependency) — search YouTube, grab transcripts, feed them into the synthesis. Zero API keys, zero cost.
---
## README: "New in V2.1" blurb
**New in V2.1 — two headline features:**
- **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a tweet — now the skill reads it. Inspired by @steipete's yt-dlp + summarize toolchain.
- **X search is fully bundled.** No external `bird` CLI install needed. Add `AUTH_TOKEN` and `CT0` once, and the vendored Bird client runs locally without browser-cookie prompts. `XAI_API_KEY` remains an optional fallback.
---
## README: X Search Authentication section
### X Search Authentication
X search prefers explicit env auth. This keeps local runs headless and avoids browser-cookie and macOS Keychain prompts.
**Recommended setup:** While logged into x.com once, open browser dev tools and copy the `auth_token` and `ct0` cookies for `x.com`.
Save them as `AUTH_TOKEN` and `CT0` in `~/.config/last30days/.env` or `.claude/last30days.env`:
```bash
AUTH_TOKEN=your_auth_token
CT0=your_ct0_token
```
**xAI fallback:** If you do not want to provide `AUTH_TOKEN` and `CT0`, set `XAI_API_KEY` and use xAI's `x_search` backend instead.
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
```
---
## README: Install block env line
```bash
AUTH_TOKEN=... # recommended for X search
CT0=... # recommended for X search
XAI_API_KEY=xai-... # optional X fallback
```
---
## SKILL.md: Stats line
```
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
```
---
## GitHub issue #19 response (post AFTER publishing)
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted. steipete was asked to take it down.
>
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly. No external CLI, no `npm install`, no brew. Just Node.js 22+ plus `AUTH_TOKEN` and `CT0`, or `XAI_API_KEY` as fallback.
>
> It also adds **YouTube as a 4th source**. When yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
>
> The recommended setup is to copy `auth_token` and `ct0` from x.com once and store them as `AUTH_TOKEN` and `CT0` in your env. That avoids browser-cookie and Keychain prompts during normal runs.
>
> The xAI API (`XAI_API_KEY`) also still works as a fallback.
---
## X/Social launch post
### Short (280 chars)
/last30days v2.1 is out 🚀
Two new features:
→ YouTube transcripts as a 4th source (yt-dlp)
→ X search fully bundled (no bird CLI install needed)
Research any topic across Reddit, X, YouTube & web in one command.
h/t @steipete for the inspiration on both.
github.com/mvanhorn/last30days-skill
### Thread version (post 1)
/last30days v2.1 just shipped — two headline features:
1️⃣ YouTube transcripts as a 4th source
When yt-dlp is installed, the skill searches YouTube, grabs view counts, and extracts auto-generated transcripts from top videos. A 20-min review has 10x the signal of a tweet — now the synthesis reads it.
### Thread version (post 2)
2️⃣ X search is fully bundled
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Add `AUTH_TOKEN` and `CT0` once, then it runs locally with no npm install. `XAI_API_KEY` still works as fallback.
Both features inspired by @steipete's tooling.
### Thread version (post 3)
YouTube goes through the same scoring pipeline as Reddit and X — relevance, recency, engagement. Transcripts get truncated to ~500 words per video and fed into the synthesis engine alongside social posts.
Zero API keys for YouTube. Zero cost. Just `brew install yt-dlp`.
### Thread version (post 4)
Try it:
```
/last30days [any topic]
```
Reddit + X + YouTube + Web. Four sources, one command, copy-paste prompts.
github.com/mvanhorn/last30days-skill
+360
View File
@@ -0,0 +1,360 @@
# last30days v2.5 Launch Thread
## FINAL THREAD (6 tweets)
### 1/6 - Announcement
I can't believe it's been 30 days since I launched @slashlast30days. 3.2k stars later, time for v2.5.
Three big additions:
1. @Polymarket prediction markets as a 6th source - helps you predict the future
2. Cross-source linking + massively better results - detects when the same story trends across multiple platforms. Ran a 15-way blinded comparison, v2.5 scored 4.38 vs 3.73 for the original. Won all 5 topics.
3. Hacker News as a 5th source - a window into the tech and developer insider world
github.com/mvanhorn/last30days-skill
### 2/6 - Demo: Anthropic vs Pentagon
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views). 5 HN stories. 9 Polymarket markets.
This story broke TODAY. Hegseth designated Anthropic a "supply chain risk." Trump ordered every agency to stop using their tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March: only 6%.
Markets say Anthropic wins regardless. That's the kind of signal you can't get from opinion threads.
### 3/6 - Demo: Seedance Prompting
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. Research first, then create from what you learned.
### 4/6 - Demo: Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Arizona is 25-2, set a program record with a 22-0 start, and holds a 2-game Big 12 lead with 3 games left. The Field of 68 called them "the TOUGHEST team in America" after escaping Baylor shorthanded. Kansas rematch Saturday - the highlight video from their first meeting has 248K views on ESPN's YouTube.
Polymarket: Championship 13%. #1 seed: 88%. Duke and Michigan each at 18% to win it all.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
### 5/6 - Demo: Iran War
"/last30days iran war"
2 Reddit threads. 34 X posts (10,048 likes). 20 YouTube videos (1.6M views, 5 transcripts). 4 HN stories (850 points). 14 Polymarket markets ($473M volume).
Geneva talks just ended without a deal. 150+ US aircraft deployed. Two carrier strike groups in position. F-22s sent to Israel. Members of Congress who saw the secret war plan came out "terrified." @cenkuygur: "they are about to drag us into a war that 70-85% of Americans oppose" (7,700 likes).
Polymarket ($473M in volume - one of their biggest markets ever): strikes by 2026: 80%. By March 31: 68%. War Powers invoked: 51%. Formal war declaration: only 12%.
Markets say: strikes are very likely, declared war is not. That's the sharpest signal in the entire research.
### 6/6 - Thank You
Thank you to ARJ999 and wkbaran on GitHub who filed three separate issues asking for Hacker News support. v2.5 delivers.
It's been a crazy 30 days. 3.2k stars. Six sources. Massively better results. Super excited to get this out.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
---
## REFERENCE MATERIAL BELOW
## Context
- 3.2k stars on GitHub
- V2.5 headline features: Polymarket (6th source), Hacker News (5th source), cross-source linking
- Ran 15-way blinded comparison: 4.38/5.0 vs 3.73/5.0
- Won all 5 topics, zero regressions
- Cross-source linking: 3 -> 13 linked items
- Demo topics: Anthropic odds (11 markets), Arizona basketball (6 markets), Iran war ($425M volume)
---
## Post 1: Lead (Announcement)
V2.5 of @slashlast30days is out. Now with @Polymarket prediction markets, cross-source linking, and massively better results.
1. Polymarket as a 6th source - real money on outcomes, no API key needed
2. Hacker News as a 5th source
3. Cross-source linking - detects when the same story trends across multiple platforms
Ran a 15-way blinded comparison across 5 topics. v2.5 scored 4.38 vs 3.73 for the original. Won all 5. Zero regressions.
github.com/mvanhorn/last30days-skill
---
## Post 2: POLYMARKET AS A 6TH SOURCE.
Reddit tells you what people think. X tells you what people share. YouTube tells you what people watch. HN tells you what developers discuss.
Polymarket helps you predict the future.
"/last30days anthropic odds"
11 markets found. Best AI model February: Anthropic 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. Pentagon ban odds: only 22%.
Free API. No key. Real money on outcomes.
---
## Post 3: CROSS-SOURCE LINKING.
When a Seedance 2.0 tutorial has 44K YouTube views AND trends on HN AND gets discussed on Reddit, v2.5 flags it: [also on: HN, YouTube]
Old version linked 3 items across 5 test topics. New version links 13. The difference is hybrid similarity - combining character-trigram and token-level matching at a tuned threshold.
Cross-platform convergence is the strongest signal that something actually matters. Not engagement on one platform. Convergence across all of them.
---
## Post 4: 15-WAY BLINDED EVALUATION.
I don't trust vibes for measuring quality. So I ran a scientific comparison.
5 topics x 3 versions. Stripped version labels. Randomized as A/B/C. Scored on groundedness, specificity, coverage, actionability, and format.
v2.5: 4.38/5.0
v2.2 (HN only): 4.10/5.0
v2.0 (original): 3.73/5.0
Won all 5 topics. Zero regressions. Biggest gains: specificity (+0.8) and format (+1.0) from cross-source linking giving the synthesis better material to work with.
---
## Post 5: Demo - Anthropic Odds
Asked it about Anthropic odds.
11 Polymarket markets. 25 X posts. 13 YouTube videos (719K views). 6 HN stories (471 points).
Best AI model February: 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. FrontierMath 50% score: 48% (up 28% today). Pentagon ban: only 22%.
Markets say Anthropic is winning the model race AND the valuation race. The Pentagon thing is noise.
---
## Post 6: Demo - Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Championship odds: 13%. #1 seed: 88%. Big 12 title: Arizona leads by 2.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
The Polymarket integration uses two-pass query expansion. First pass finds "Arizona Big 12." Second pass discovers the championship and #1 seed markets via tag-based domain bridging.
---
## Post 7: Demo - Iran War
The best Polymarket demo is news.
"/last30days iran war"
14 Polymarket markets. $425M+ in volume. 7 Reddit threads. 30 X posts. 20 YouTube videos (2M views). 18 HN stories (1,187 points).
US strikes Iran by 2026: 70%. War Powers by March: 60%. Israel strikes by June: 64%. Formal war declaration: only 8%.
Markets say: limited strikes with War Powers, NOT a declared war. Breaking Points (435K views) covered leaked Pentagon opposition. r/Conservative "imploding" per r/SubredditDrama.
One command. Six sources. Real money.
---
## Post 8: Credits + CTA
Also in v2.5: YouTube synonym expansion ("hip hop" now matches "rap" - relevance jumped 0.33 to 0.71), X handle resolution, and HN OR queries for framework topics.
The difference between "good research" and "research you'd actually trust" is in details like this.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 9: Demo - Claude Code (ALL 6 sources)
"/last30days Claude Code"
3 Reddit threads (199 upvotes). 35 X posts (5,239 likes). 15 YouTube videos (1.4M views, 5 transcripts). 30 HN stories (~8,500 points). 8 Polymarket markets. 20 web pages.
All six sources hit. Top finding: the planning-first workflow has won. The #1 HN post this month (969 pts, 590 comments) is about separating planning from execution. Boris Cherny (Head of Claude Code) on Lenny's Podcast: "100% of my code is written by Claude Code - I have not edited a single line by hand since November."
Polymarket: Anthropic 99% for best AI model in February. 58% for March. Claude on FrontierMath at 55%. The US government rejected Claude - Polymarket has the Hegseth ban at 32%.
Then I asked it to dig deeper into the planning-first workflow. No new searches - it answered from what it already learned.
---
## Post 10: Demo - March Madness Odds (Polymarket + Sports)
"/last30days March Madness Odds"
2 Reddit threads. 31 X posts. 6 YouTube videos (46K views, 4 transcripts). 2 Polymarket markets.
Tournament winner: Duke 18%, Michigan 18%, Arizona 13%. #1 seeds: Michigan 98%, Duke 91%, Arizona 88%.
Duke is the hottest mover - went from +700 to +450 in one week. The skill surfaced that from sportsbook data, X commentary, and Polymarket odds simultaneously.
Then I asked it to break down Michigan vs Duke vs Arizona. Full analysis from the research it already had.
---
## Post 11: Demo - Seedance Prompting (Expert + Prompt Mode)
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 1 HN story. 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style + Constraints. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. That's the skill's real power - research first, then create from what you learned.
---
## Post 12: Thank You + CTA (Final)
Thank you to ARJ999 and wkbaran on GitHub who kept asking for Hacker News support. Three separate issues. v2.5 delivers.
30 days. 3.2k stars. 6 sources. Massively better results.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 13: Demo - Anthropic vs Pentagon (Breaking News + Polymarket)
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views, 5 transcripts). 5 HN stories. 9 Polymarket markets. 10 web pages.
This story broke TODAY. Defense Secretary Hegseth designated Anthropic a "supply chain risk" - believed to be the first time an American company has ever received this designation. Trump ordered every federal agency to stop using Anthropic tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March 31: only 6%.
Markets say: Anthropic wins the model race regardless. Bettors don't think Hegseth survives this. That's the kind of signal you can't get from opinion threads.
---
## Post 14: Demo - OpenAI Insider Trading (News + Polymarket)
"/last30days OpenAI Insider Trading"
2 Reddit threads. 29 X posts. 4 YouTube videos (360K views, 4 transcripts). 2 HN stories. 15 Polymarket markets. 15 web pages.
An OpenAI employee was just fired for using confidential info to bet on Polymarket. 13 brand-new wallets appeared 40 hours before the browser launch. $309K bet on the right outcome. Unusual Whales flagged 77 suspected insider positions across 60 wallets.
Meanwhile Polymarket has OpenAI's IPO at $1.25-1.5T: 54%. Anthropic IPOs first: 62%. Best AI model: Anthropic 99%.
The prediction markets are both the story AND the source. One command pulled all of it together.
---
## Standalone Tweet: Polymarket Stats Line
"/last30days Anthropic Pete Hegseth"
The Pentagon just designated Anthropic a supply chain risk. First time ever for an American company. Trump ordered every agency to stop using their tech.
Here's what Polymarket says:
📊 9 markets │ Best AI model: 99% │ $500B+ valuation: 68% │ IPO >$600B: 97% │ Hegseth out by March: 6%
Bettors with real money on the line think Anthropic wins the model race, goes public at a massive valuation, and Hegseth doesn't survive this.
That's the gap between headlines and reality. One command, six sources.
github.com/mvanhorn/last30days-skill
---
## Recommended Thread Order (pick 8-10)
The full thread above is 12 posts. Here's what I'd cut to keep it tight:
**Must include (core story):**
1. Post 1 - Lead announcement
2. Post 2 - Polymarket ("Reddit tells you what people think...")
3. Post 3 - Cross-source linking
4. Post 4 - Blinded evaluation
**Best demos (pick 3-4):**
- Post 13 (Anthropic vs Pentagon) - STRONGEST. Breaking news today. Polymarket cuts through the noise. "Markets say Anthropic wins regardless."
- Post 9 (Claude Code) - All 6 sources. Massive numbers. Shows follow-up flow.
- Post 10 (March Madness) - Sports/Polymarket crossover. Timely with tournament approaching.
- Post 11 (Seedance) - Shows prompting flow. 1.2M YouTube views.
- Post 5 (Anthropic Odds) - Overlaps with Post 13 now. Skip.
**Skip or save for standalone tweets:**
- Post 5 (Anthropic Odds) - Redundant with Post 13
- Post 6 (Arizona Basketball) - Covered by March Madness now
- Post 7 (Iran War) - Great standalone tweet, not for launch thread
- Post 8 (Credits/minor features) - Fold into CTA
**My recommended 8-post thread:**
1. Lead (Post 1)
2. Polymarket (Post 2)
3. Cross-source linking (Post 3)
4. Blinded evaluation (Post 4)
5. Demo: Anthropic vs Pentagon (Post 13) - breaking news, best Polymarket showcase
6. Demo: Claude Code (Post 9)
7. Demo: March Madness (Post 10)
8. Thank you + CTA (Post 12)
---
## Video Script (~60 seconds)
**[Talking to camera]**
Oh my god, I can't believe it's been 30 days since I launched last30days. 3,200 stars on GitHub. This has been the craziest month.
Today I'm shipping v2.5 and I'm really excited about this one. Three big things.
**[Screen recording: typing /last30days Anthropic Pete Hegseth]**
First - Polymarket prediction markets as a 6th source. So this Anthropic-Pentagon story broke today. Hegseth designated Anthropic a supply chain risk, Trump ordered agencies to stop using their tech. Scary headline, right?
But Polymarket says: Anthropic still 99% for best AI model. IPO above 600 billion: 97%. Hegseth out by March: 6%. Real money on outcomes helps you predict the future. That's a different story than the headlines.
**[Screen recording: typing /last30days arizona basketball]**
Second - it now searches Hacker News and does cross-source linking. When the same story shows up on Reddit AND YouTube AND HN, it flags it. I ran a 15-way blinded comparison and v2.5 scored 4.38 versus 3.73 for the original. Won all 5 test topics.
**[Back to camera]**
Thank you to everyone who starred it, filed issues, and kept pushing me to make this better. Shoutout to the people on GitHub who literally filed three separate issues asking for Hacker News. v2.5 delivers.
Link in bio. Try it on anything.
---
## Scoring Note
The 4.38 vs 3.73 score is from a custom 5-dimension rubric (30% groundedness, 25% specificity, 20% coverage, 15% actionability, 10% format compliance) evaluated by Claude on blinded outputs. The relative ranking is meaningful; the absolute numbers are not. It's an LLM grading LLM output - useful for A/B comparison, not for claiming "4.38 out of 5 quality."
If using the score in a tweet, frame it as "scored X vs Y on a blinded comparison" not "rated 4.38/5.0 quality" - the former is honest, the latter implies an objective standard that doesn't exist.
+42
View File
@@ -0,0 +1,42 @@
[
{
"topic": "OpenClaw vs NanoClaw vs ZeroClaw",
"query_type": "comparison",
"rationale": "Multi-entity extraction, 3-way split across AI agent frameworks."
},
{
"topic": "how to set up a GLP-1 supplement routine",
"query_type": "how_to",
"rationale": "Trending health topic. Tests non-tech how_to."
},
{
"topic": "2026 March Madness",
"query_type": "breaking_news",
"rationale": "Live sporting event. Tests broad breaking news recall."
},
{
"topic": "best budget noise cancelling headphones 2026",
"query_type": "product",
"rationale": "Evergreen consumer query. Tests product review aggregation."
},
{
"topic": "thoughts on OpenAI Codex pricing",
"query_type": "opinion",
"rationale": "Active developer debate. Tests opinion mining."
},
{
"topic": "odds of US recession 2026",
"query_type": "prediction",
"rationale": "Major macro topic. Tests prediction market + news synthesis."
},
{
"topic": "what is retrieval augmented generation",
"query_type": "concept",
"rationale": "Widely discussed AI concept. Tests explanation quality."
},
{
"topic": "Google Wiz acquisition price and timeline",
"query_type": "factual",
"rationale": "Completed event ($32B). Tests factual precision."
}
]
+58
View File
@@ -0,0 +1,58 @@
{
"items": [
{
"video_id": "7543693751290481942",
"text": "This Claude Code trick saved me hours #claudecode #ai #coding",
"url": "https://www.tiktok.com/@codemaster/video/7543693751290481942",
"author_name": "codemaster",
"date": "2026-02-28",
"engagement": {
"views": 2100000,
"likes": 45000,
"comments": 1200,
"shares": 8400
},
"hashtags": ["claudecode", "ai", "coding"],
"duration": 45,
"relevance": 0.85,
"why_relevant": "TikTok: This Claude Code trick saved me hours #claude",
"caption_snippet": "So I found this insane trick with Claude Code where you can use slash commands to automate everything"
},
{
"video_id": "7543100200112345678",
"text": "AI coding tools comparison 2026 - Claude vs Copilot vs Cursor #ai #devtools",
"url": "https://www.tiktok.com/@techreviewer/video/7543100200112345678",
"author_name": "techreviewer",
"date": "2026-02-25",
"engagement": {
"views": 850000,
"likes": 22000,
"comments": 890,
"shares": 3200
},
"hashtags": ["ai", "devtools"],
"duration": 60,
"relevance": 0.7,
"why_relevant": "TikTok: AI coding tools comparison 2026 - Claude vs Copi",
"caption_snippet": ""
},
{
"video_id": "7543200300223456789",
"text": "You need to try Claude Code RIGHT NOW #programming #tips",
"url": "https://www.tiktok.com/@devtips/video/7543200300223456789",
"author_name": "devtips",
"date": "2026-03-01",
"engagement": {
"views": 500000,
"likes": 15000,
"comments": 450,
"shares": 2100
},
"hashtags": ["programming", "tips"],
"duration": 30,
"relevance": 0.6,
"why_relevant": "TikTok: You need to try Claude Code RIGHT NOW #programm",
"caption_snippet": "Let me show you why Claude Code is the best AI coding tool right now"
}
]
}
+67
View File
@@ -0,0 +1,67 @@
{
"name": "last30days-skill",
"version": "3.0.5",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
"name": "Extension Directory",
"description": "Extension installation directory (auto-set by Gemini CLI)",
"envVar": "GEMINI_EXTENSION_DIR",
"sensitive": false
},
{
"name": "ScrapeCreators API Key",
"description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
"envVar": "SCRAPECREATORS_API_KEY",
"sensitive": true
},
{
"name": "OpenAI API Key",
"description": "OpenAI API Key - optional fallback for Reddit discovery",
"envVar": "OPENAI_API_KEY",
"sensitive": true
},
{
"name": "xAI API Key",
"description": "xAI API Key for X/Twitter search (optional)",
"envVar": "XAI_API_KEY",
"sensitive": true
},
{
"name": "OpenRouter API Key",
"description": "OpenRouter API Key (optional)",
"envVar": "OPENROUTER_API_KEY",
"sensitive": true
},
{
"name": "Parallel AI API Key",
"description": "Parallel AI API Key (optional)",
"envVar": "PARALLEL_API_KEY",
"sensitive": true
},
{
"name": "Brave Search API Key",
"description": "Brave Search API Key (optional)",
"envVar": "BRAVE_API_KEY",
"sensitive": true
},
{
"name": "Apify API Token",
"description": "Apify API Token (optional legacy)",
"envVar": "APIFY_API_TOKEN",
"sensitive": true
},
{
"name": "Twitter AUTH_TOKEN",
"description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
"envVar": "AUTH_TOKEN",
"sensitive": true
},
{
"name": "Twitter CT0",
"description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
"envVar": "CT0",
"sensitive": true
}
]
}
+16
View File
@@ -0,0 +1,16 @@
{
"hooks": {
"SessionStart": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/check-config.sh",
"timeout": 5
}
]
}
]
}
}
+112
View File
@@ -0,0 +1,112 @@
#!/bin/bash
set -euo pipefail
# Check last30days configuration status and show appropriate welcome message.
# Priority: .claude/last30days.env > ~/.config/last30days/.env > env vars
PROJECT_ENV=".claude/last30days.env"
GLOBAL_ENV="$HOME/.config/last30days/.env"
# Helper: warn if file permissions are too open
check_perms() {
local file="$1"
if [[ ! -f "$file" ]]; then return; fi
local perms
# Try GNU stat first (Linux), fall back to BSD stat (macOS).
# On Linux, `stat -f` prints filesystem info (not permissions) and exits 0,
# so the previous BSD-first ordering left $perms as multi-line garbage on
# every Linux session start and printed a false WARNING.
perms=$(stat -c '%a' "$file" 2>/dev/null || stat -f '%Lp' "$file" 2>/dev/null || echo "")
if [[ -n "$perms" && "$perms" != "600" && "$perms" != "400" ]]; then
echo "/last30days: WARNING — $file has permissions $perms (should be 600)."
echo " Fix: chmod 600 $file"
fi
}
# Load env file into variables for inspection (without exporting)
load_env_vars() {
local file="$1"
if [[ -f "$file" ]]; then
while IFS='=' read -r key value; do
# Skip comments, empty lines
[[ "$key" =~ ^[[:space:]]*# ]] && continue
[[ -z "$key" ]] && continue
key=$(echo "$key" | xargs)
value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
if [[ -n "$key" && -n "$value" ]]; then
eval "ENV_${key}=\"${value}\""
fi
done < "$file"
fi
}
# Determine which config file is active
CONFIG_FILE=""
if [[ -f "$PROJECT_ENV" ]]; then
CONFIG_FILE="$PROJECT_ENV"
check_perms "$PROJECT_ENV"
elif [[ -f "$GLOBAL_ENV" ]]; then
CONFIG_FILE="$GLOBAL_ENV"
check_perms "$GLOBAL_ENV"
fi
# Load config if found
if [[ -n "$CONFIG_FILE" ]]; then
load_env_vars "$CONFIG_FILE"
fi
# Check SETUP_COMPLETE (from file or env)
SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
# If setup has never been run, show welcome message for new users
if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
cat <<'EOF'
/last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
Reddit, Hacker News, and Polymarket work out of the box.
The setup wizard can unlock X/Twitter, YouTube, and more.
EOF
exit 0
fi
# Setup done but check for ScrapeCreators
HAS_SCRAPECREATORS="${ENV_SCRAPECREATORS_API_KEY:-${SCRAPECREATORS_API_KEY:-}}"
HAS_X="${ENV_AUTH_TOKEN:-${AUTH_TOKEN:-}}"
HAS_XAI="${ENV_XAI_API_KEY:-${XAI_API_KEY:-}}"
HAS_YTDLP=""
if command -v yt-dlp &>/dev/null; then
HAS_YTDLP="yes"
fi
HAS_BSKY="${ENV_BSKY_HANDLE:-${BSKY_HANDLE:-}}"
HAS_EXA="${ENV_EXA_API_KEY:-${EXA_API_KEY:-}}"
# Count active sources
SOURCE_COUNT=2 # HN + Polymarket are always free
if [[ -n "$HAS_X" || -n "$HAS_XAI" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
# Reddit public JSON always works
SOURCE_COUNT=$((SOURCE_COUNT + 1))
if [[ -n "$HAS_YTDLP" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_EXA" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_BSKY" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 3)) # Reddit comments + TikTok + Instagram
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Fully configured — compact ready message
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
else
# Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 10,000 free API calls, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider."
fi
+37
View File
@@ -0,0 +1,37 @@
# /last30days v3.1 Launch Video
30-second Remotion-rendered MP4 announcing **v3.1: Competitors mode** for posting on X.
## Quick start
```bash
cd marketing/v3.1-launch
npm install # one-time, ~200MB of node_modules
npm run preview # opens Remotion Studio at localhost:3000 to scrub frames
npm run render # writes out/last30days-v3.1-launch.mp4 (high quality, CRF 18)
npm run render:fast # writes a CRF 23 preview for fast iteration
```
## Specs
- 1920×1080, 30fps, 30 seconds (900 frames)
- H.264 / MP4
- Silent (autoplay-muted-friendly; captions baked in)
- Marketing label `v3.1` (engine code version stays 3.0.14)
## Scene timing (single source of truth: `src/lib/timing.ts`)
| Scene | Frames | Time | What |
|-------|--------|------|------|
| 1. Hook | 0-89 | 0.0-3.0s | Badge animates in + "What if one search ran 3 at once?" |
| 2. Old way | 90-239 | 3.0-8.0s | Single terminal: `/last30days OpenAI` |
| 3. Fan-out | 240-419 | 8.0-14.0s | `--competitors` types in → splits into 3 panes |
| 4. Comparison | 420-629 | 14.0-21.0s | 3 panes collapse into Head-to-Head table |
| 5. How | 630-779 | 21.0-26.0s | 3-line text card |
| 6. CTA | 780-899 | 26.0-30.0s | Install command + repo URL |
Edit `src/lib/timing.ts` to retime scenes; the `LaunchVideo` composition reads from there.
## Output
Rendered MP4 lives at `out/last30days-v3.1-launch.mp4` (gitignored). Upload directly to X.
+23
View File
@@ -0,0 +1,23 @@
{
"name": "last30days-v3-1-launch-video",
"version": "0.1.0",
"private": true,
"description": "30s Remotion launch video for /last30days v3.1 (competitors mode).",
"scripts": {
"preview": "remotion studio src/index.ts",
"render": "remotion render src/index.ts LaunchVideo out/last30days-v3.1-launch.mp4 --codec=h264 --crf=18",
"render:fast": "remotion render src/index.ts LaunchVideo out/last30days-v3.1-launch-preview.mp4 --codec=h264 --crf=23"
},
"dependencies": {
"react": "19.0.0",
"react-dom": "19.0.0",
"remotion": "4.0.250",
"@remotion/cli": "4.0.250",
"@remotion/google-fonts": "4.0.250"
},
"devDependencies": {
"@types/react": "19.0.0",
"@types/node": "22.10.0",
"typescript": "5.6.3"
}
}
+6
View File
@@ -0,0 +1,6 @@
import { Config } from "@remotion/cli/config";
Config.setVideoImageFormat("jpeg");
Config.setOverwriteOutput(true);
Config.setConcurrency(null);
Config.setCodec("h264");
+35
View File
@@ -0,0 +1,35 @@
import React from "react";
import { AbsoluteFill, Sequence } from "remotion";
import { SCENES } from "./lib/timing";
import { Scene1Hook } from "./scenes/Scene1Hook";
import { Scene2OldWay } from "./scenes/Scene2OldWay";
import { Scene3FanOut } from "./scenes/Scene3FanOut";
import { Scene4Comparison } from "./scenes/Scene4Comparison";
import { Scene5HowItWorks } from "./scenes/Scene5HowItWorks";
import { Scene6CTA } from "./scenes/Scene6CTA";
import { COLORS } from "./lib/colors";
export const LaunchVideo: React.FC = () => {
return (
<AbsoluteFill style={{ background: COLORS.bgDeep }}>
<Sequence from={SCENES.hook.from} durationInFrames={SCENES.hook.durationInFrames}>
<Scene1Hook />
</Sequence>
<Sequence from={SCENES.oldWay.from} durationInFrames={SCENES.oldWay.durationInFrames}>
<Scene2OldWay />
</Sequence>
<Sequence from={SCENES.fanOut.from} durationInFrames={SCENES.fanOut.durationInFrames}>
<Scene3FanOut />
</Sequence>
<Sequence from={SCENES.comparison.from} durationInFrames={SCENES.comparison.durationInFrames}>
<Scene4Comparison />
</Sequence>
<Sequence from={SCENES.howItWorks.from} durationInFrames={SCENES.howItWorks.durationInFrames}>
<Scene5HowItWorks />
</Sequence>
<Sequence from={SCENES.cta.from} durationInFrames={SCENES.cta.durationInFrames}>
<Scene6CTA />
</Sequence>
</AbsoluteFill>
);
};
+19
View File
@@ -0,0 +1,19 @@
import React from "react";
import { Composition } from "remotion";
import { LaunchVideo } from "./LaunchVideo";
import { FPS, TOTAL_FRAMES } from "./lib/timing";
export const RemotionRoot: React.FC = () => {
return (
<>
<Composition
id="LaunchVideo"
component={LaunchVideo}
durationInFrames={TOTAL_FRAMES}
fps={FPS}
width={1920}
height={1080}
/>
</>
);
};
@@ -0,0 +1,68 @@
import React from "react";
import { spring, useCurrentFrame, useVideoConfig } from "remotion";
import { COLORS, FONT_MONO } from "../lib/colors";
type Props = {
startFrame?: number;
size?: "small" | "large";
};
export const BadgeBar: React.FC<Props> = ({ startFrame = 0, size = "large" }) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const elapsed = Math.max(0, frame - startFrame);
const scale = spring({
frame: elapsed,
fps,
config: { damping: 12, stiffness: 90 },
from: 0.85,
to: 1,
});
const opacity = spring({
frame: elapsed,
fps,
config: { damping: 20 },
from: 0,
to: 1,
});
const fontSize = size === "large" ? 56 : 28;
return (
<div
style={{
display: "flex",
alignItems: "center",
justifyContent: "center",
transform: `scale(${scale})`,
opacity,
fontFamily: FONT_MONO,
fontSize,
color: COLORS.fgPrimary,
letterSpacing: 0.5,
}}
>
<span style={{ fontSize: fontSize * 1.1, marginRight: 16 }}>🌐</span>
<span>last30days</span>
<span
style={{
marginLeft: 14,
color: COLORS.accentCyan,
fontWeight: 600,
}}
>
v3.1
</span>
<span
style={{
marginLeft: 16,
color: COLORS.fgDim,
fontSize: fontSize * 0.55,
}}
>
· synced 2026-04-22
</span>
</div>
);
};
@@ -0,0 +1,135 @@
import React from "react";
import { interpolate, useCurrentFrame } from "remotion";
import { COLORS, FONT_MONO } from "../lib/colors";
type Row = {
dimension: string;
cells: [string, string, string];
};
type Props = {
startFrame: number;
entities: [string, string, string];
rows: Row[];
rowStaggerFrames?: number;
};
export const ComparisonTable: React.FC<Props> = ({
startFrame,
entities,
rows,
rowStaggerFrames = 18,
}) => {
const frame = useCurrentFrame();
const headerOpacity = interpolate(
frame - startFrame,
[0, 12],
[0, 1],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
);
const colTemplate = "1.4fr 1fr 1fr 1fr";
const cellPad = "16px 22px";
return (
<div
style={{
width: "100%",
background: COLORS.bgPanel,
borderRadius: 16,
border: `1px solid ${COLORS.border}`,
overflow: "hidden",
fontFamily: FONT_MONO,
boxShadow: "0 24px 60px rgba(0,0,0,0.6)",
}}
>
<div
style={{
display: "grid",
gridTemplateColumns: colTemplate,
background: COLORS.bgPanelSoft,
borderBottom: `1px solid ${COLORS.border}`,
opacity: headerOpacity,
}}
>
<div
style={{
padding: cellPad,
color: COLORS.fgMuted,
fontSize: 22,
fontWeight: 500,
}}
>
Dimension
</div>
{entities.map((entity, idx) => (
<div
key={entity}
style={{
padding: cellPad,
color: idx === 0 ? COLORS.accentCyan : COLORS.fgPrimary,
fontSize: 26,
fontWeight: 600,
borderLeft: `1px solid ${COLORS.border}`,
}}
>
{entity}
</div>
))}
</div>
{rows.map((row, idx) => {
const rowStart = startFrame + 12 + idx * rowStaggerFrames;
const rowOpacity = interpolate(
frame - rowStart,
[0, 14],
[0, 1],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
);
const rowSlide = interpolate(
frame - rowStart,
[0, 14],
[12, 0],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
);
return (
<div
key={row.dimension}
style={{
display: "grid",
gridTemplateColumns: colTemplate,
borderBottom:
idx === rows.length - 1 ? "none" : `1px solid ${COLORS.border}`,
opacity: rowOpacity,
transform: `translateY(${rowSlide}px)`,
}}
>
<div
style={{
padding: cellPad,
color: COLORS.fgMuted,
fontSize: 22,
}}
>
{row.dimension}
</div>
{row.cells.map((cell, cellIdx) => (
<div
key={cellIdx}
style={{
padding: cellPad,
color: COLORS.fgPrimary,
fontSize: 22,
borderLeft: `1px solid ${COLORS.border}`,
lineHeight: 1.35,
}}
>
{cell}
</div>
))}
</div>
);
})}
</div>
);
};
@@ -0,0 +1,95 @@
import React from "react";
import { COLORS, FONT_MONO } from "../lib/colors";
type Props = {
title?: string;
width?: number | string;
height?: number | string;
children?: React.ReactNode;
glow?: boolean;
};
export const TerminalWindow: React.FC<Props> = ({
title = "/last30days",
width = "100%",
height = "100%",
children,
glow = false,
}) => {
return (
<div
style={{
width,
height,
background: COLORS.bgPanel,
borderRadius: 16,
border: `1px solid ${COLORS.border}`,
boxShadow: glow
? `0 0 60px ${COLORS.accentCyan}33, 0 24px 60px rgba(0,0,0,0.6)`
: "0 24px 60px rgba(0,0,0,0.6)",
overflow: "hidden",
display: "flex",
flexDirection: "column",
fontFamily: FONT_MONO,
}}
>
<div
style={{
height: 36,
background: COLORS.bgPanelSoft,
borderBottom: `1px solid ${COLORS.border}`,
display: "flex",
alignItems: "center",
padding: "0 16px",
gap: 8,
}}
>
<span
style={{
width: 12,
height: 12,
borderRadius: 12,
background: COLORS.trafficRed,
}}
/>
<span
style={{
width: 12,
height: 12,
borderRadius: 12,
background: COLORS.trafficYellow,
}}
/>
<span
style={{
width: 12,
height: 12,
borderRadius: 12,
background: COLORS.trafficGreen,
}}
/>
<span
style={{
marginLeft: 16,
color: COLORS.fgMuted,
fontSize: 14,
fontFamily: FONT_MONO,
letterSpacing: 0.5,
}}
>
{title}
</span>
</div>
<div
style={{
flex: 1,
padding: "20px 28px",
color: COLORS.fgPrimary,
overflow: "hidden",
}}
>
{children}
</div>
</div>
);
};
@@ -0,0 +1,71 @@
import React from "react";
import { interpolate, useCurrentFrame } from "remotion";
import { COLORS, FONT_MONO } from "../lib/colors";
type Props = {
text: string;
startFrame: number;
charsPerSec?: number;
fontSize?: number;
color?: string;
prefix?: string;
prefixColor?: string;
showCursor?: boolean;
fps?: number;
};
export const TypedLine: React.FC<Props> = ({
text,
startFrame,
charsPerSec = 28,
fontSize = 32,
color = COLORS.fgPrimary,
prefix,
prefixColor = COLORS.accentGreen,
showCursor = true,
fps = 30,
}) => {
const frame = useCurrentFrame();
const elapsed = Math.max(0, frame - startFrame);
const totalChars = text.length;
const lengthFrames = Math.ceil((totalChars / charsPerSec) * fps);
const visibleChars = Math.round(
interpolate(elapsed, [0, lengthFrames], [0, totalChars], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
}),
);
const visible = text.slice(0, visibleChars);
const done = visibleChars >= totalChars;
const cursorOn = showCursor && Math.floor(frame / 15) % 2 === 0;
return (
<div
style={{
fontFamily: FONT_MONO,
fontSize,
color,
whiteSpace: "pre",
lineHeight: 1.4,
}}
>
{prefix ? (
<span style={{ color: prefixColor, marginRight: 12 }}>{prefix}</span>
) : null}
<span>{visible}</span>
{(!done || cursorOn) && (
<span
style={{
display: "inline-block",
width: fontSize * 0.55,
height: fontSize * 0.95,
background: color,
verticalAlign: "text-bottom",
marginLeft: 2,
opacity: cursorOn ? 0.85 : 0,
}}
/>
)}
</div>
);
};
+4
View File
@@ -0,0 +1,4 @@
import { registerRoot } from "remotion";
import { RemotionRoot } from "./Root";
registerRoot(RemotionRoot);
+20
View File
@@ -0,0 +1,20 @@
export const COLORS = {
bgDeep: "#0a0e14",
bgPanel: "#11161d",
bgPanelSoft: "#161c25",
fgPrimary: "#e6e6e6",
fgMuted: "#8a93a3",
fgDim: "#5b6573",
border: "#2a3340",
accentCyan: "#36d6f7",
accentMagenta: "#ff55a3",
accentGreen: "#5fff9f",
accentAmber: "#ffc857",
trafficRed: "#ff5f57",
trafficYellow: "#febc2e",
trafficGreen: "#28c840",
} as const;
export const FONT_MONO = '"JetBrains Mono", "SF Mono", "Menlo", monospace';
export const FONT_SANS =
'"Inter", "SF Pro Display", -apple-system, BlinkMacSystemFont, sans-serif';
+13
View File
@@ -0,0 +1,13 @@
// Single source of truth for scene frame ranges.
// 30fps × 30s = 900 frames total.
export const FPS = 30;
export const TOTAL_FRAMES = 900;
export const SCENES = {
hook: { from: 0, durationInFrames: 90 },
oldWay: { from: 90, durationInFrames: 150 },
fanOut: { from: 240, durationInFrames: 180 },
comparison: { from: 420, durationInFrames: 210 },
howItWorks: { from: 630, durationInFrames: 150 },
cta: { from: 780, durationInFrames: 120 },
} as const;
@@ -0,0 +1,61 @@
import React from "react";
import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion";
import { BadgeBar } from "../components/BadgeBar";
import { COLORS, FONT_SANS } from "../lib/colors";
export const Scene1Hook: React.FC = () => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const captionOpacity = interpolate(frame, [20, 35, 75, 90], [0, 1, 1, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const captionLift = spring({
frame: frame - 20,
fps,
config: { damping: 15, stiffness: 70 },
from: 16,
to: 0,
});
const badgeFadeOut = interpolate(frame, [70, 90], [1, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<AbsoluteFill
style={{
background: `radial-gradient(circle at 50% 40%, ${COLORS.bgPanelSoft} 0%, ${COLORS.bgDeep} 60%)`,
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
gap: 64,
}}
>
<div style={{ opacity: badgeFadeOut }}>
<BadgeBar />
</div>
<div
style={{
opacity: captionOpacity,
transform: `translateY(${captionLift}px)`,
fontFamily: FONT_SANS,
fontSize: 96,
fontWeight: 600,
color: COLORS.fgPrimary,
textAlign: "center",
letterSpacing: -1.5,
lineHeight: 1.1,
maxWidth: 1400,
}}
>
What if one search
<br />
ran <span style={{ color: COLORS.accentCyan }}>3 at once?</span>
</div>
</AbsoluteFill>
);
};
@@ -0,0 +1,100 @@
import React from "react";
import { AbsoluteFill, interpolate, useCurrentFrame } from "remotion";
import { TerminalWindow } from "../components/TerminalWindow";
import { TypedLine } from "../components/TypedLine";
import { COLORS, FONT_MONO, FONT_SANS } from "../lib/colors";
export const Scene2OldWay: React.FC = () => {
const frame = useCurrentFrame();
const enter = interpolate(frame, [0, 20], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const slide = interpolate(frame, [0, 20], [40, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
// Result card fades in after type completes (~70 frames)
const resultFade = interpolate(frame, [70, 95], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
// Caption appears late
const captionFade = interpolate(frame, [110, 130, 150], [0, 1, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<AbsoluteFill
style={{
background: COLORS.bgDeep,
padding: 80,
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
}}
>
<div
style={{
width: 1400,
height: 520,
opacity: enter,
transform: `translateY(${slide}px)`,
}}
>
<TerminalWindow title="bash">
<TypedLine
text="/last30days OpenAI"
startFrame={20}
prefix="$"
fontSize={42}
/>
<div
style={{
opacity: resultFade,
marginTop: 36,
padding: "20px 24px",
background: COLORS.bgPanelSoft,
borderRadius: 12,
border: `1px solid ${COLORS.border}`,
fontFamily: FONT_MONO,
fontSize: 26,
color: COLORS.fgPrimary,
lineHeight: 1.6,
}}
>
<div style={{ color: COLORS.accentGreen }}>
All agents reported back!
</div>
<div style={{ color: COLORS.fgMuted, marginTop: 6 }}>
🟠 Reddit: 14 threads
</div>
<div style={{ color: COLORS.fgMuted }}>
🔵 X: 22 posts
</div>
<div style={{ color: COLORS.fgMuted }}>
🟡 HN: 1 story
</div>
</div>
</TerminalWindow>
</div>
<div
style={{
opacity: captionFade,
marginTop: 60,
fontFamily: FONT_SANS,
fontSize: 38,
color: COLORS.fgMuted,
}}
>
The old way: <span style={{ color: COLORS.fgPrimary }}>one topic.</span>
</div>
</AbsoluteFill>
);
};
@@ -0,0 +1,221 @@
import React from "react";
import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion";
import { TerminalWindow } from "../components/TerminalWindow";
import { TypedLine } from "../components/TypedLine";
import { COLORS, FONT_MONO, FONT_SANS } from "../lib/colors";
const PROGRESS_LINES = [
{ source: "Reddit", color: "#ff6a3d" },
{ source: "X", color: "#36d6f7" },
{ source: "YouTube", color: "#ff5757" },
{ source: "TikTok", color: "#5fff9f" },
{ source: "Instagram", color: "#ff55a3" },
];
const ENTITIES: { label: string; tag: string; accent: string }[] = [
{ label: "OpenAI", tag: "$ /last30days OpenAI", accent: COLORS.accentCyan },
{ label: "Anthropic", tag: "$ /last30days Anthropic", accent: COLORS.accentMagenta },
{ label: "xAI", tag: "$ /last30days xAI", accent: COLORS.accentAmber },
];
const FanPane: React.FC<{
label: string;
tag: string;
accent: string;
panelStart: number;
}> = ({ label, tag, accent, panelStart }) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const localFrame = Math.max(0, frame - panelStart);
const enter = spring({
frame: localFrame,
fps,
config: { damping: 18, stiffness: 80 },
from: 0,
to: 1,
});
const slide = interpolate(localFrame, [0, 20], [40, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<div
style={{
flex: 1,
opacity: enter,
transform: `translateY(${slide}px)`,
height: 480,
}}
>
<TerminalWindow title={label} glow>
<div
style={{
color: accent,
fontSize: 18,
fontFamily: FONT_MONO,
marginBottom: 14,
}}
>
{tag}
</div>
<div
style={{
fontSize: 22,
color: COLORS.accentGreen,
fontFamily: FONT_MONO,
marginBottom: 12,
}}
>
[Competitors] running...
</div>
{PROGRESS_LINES.map((line, idx) => {
const lineStart = panelStart + 16 + idx * 6;
const lineFade = interpolate(
frame - lineStart,
[0, 8],
[0, 1],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" },
);
// pulse the in-progress dot
const dotOn = Math.floor((frame - lineStart) / 6) % 2 === 0;
return (
<div
key={line.source}
style={{
opacity: lineFade,
fontFamily: FONT_MONO,
fontSize: 20,
color: COLORS.fgMuted,
display: "flex",
alignItems: "center",
marginBottom: 6,
}}
>
<span
style={{
display: "inline-block",
width: 10,
height: 10,
borderRadius: 10,
background: dotOn ? line.color : COLORS.bgPanelSoft,
marginRight: 12,
boxShadow: dotOn ? `0 0 10px ${line.color}` : "none",
}}
/>
<span style={{ color: line.color, marginRight: 8 }}>
</span>
<span>{line.source}</span>
<span style={{ marginLeft: "auto", color: COLORS.fgDim }}>
{dotOn ? "..." : "·"}
</span>
</div>
);
})}
</TerminalWindow>
</div>
);
};
export const Scene3FanOut: React.FC = () => {
const frame = useCurrentFrame();
// Phase 1 (0-30 frames): single terminal types --competitors flag
// Phase 2 (30+): split into 3 panes
const splitProgress = interpolate(frame, [30, 50], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const singleOpacity = interpolate(frame, [0, 8, 30, 45], [0, 1, 1, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
// Caption fades in shortly after panes settle so it has time to read.
const captionFade = interpolate(frame, [60, 80], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<AbsoluteFill
style={{
background: COLORS.bgDeep,
padding: 60,
display: "flex",
flexDirection: "column",
}}
>
<div
style={{
opacity: captionFade,
textAlign: "center",
fontFamily: FONT_SANS,
fontSize: 42,
color: COLORS.fgPrimary,
marginBottom: 32,
}}
>
Now it discovers competitors
<br />
<span style={{ color: COLORS.accentCyan }}>and runs all 3.</span>
</div>
<div
style={{
flex: 1,
position: "relative",
}}
>
{/* Single terminal during phase 1, fades out as panes appear */}
<div
style={{
position: "absolute",
inset: 0,
display: "flex",
alignItems: "center",
justifyContent: "center",
opacity: singleOpacity,
}}
>
<div style={{ width: 1200, height: 380 }}>
<TerminalWindow title="bash">
<TypedLine
text="/last30days OpenAI --competitors"
startFrame={0}
prefix="$"
fontSize={42}
/>
</TerminalWindow>
</div>
</div>
{/* Three panes fade in starting frame ~30 */}
<div
style={{
position: "absolute",
inset: 0,
display: "flex",
alignItems: "center",
justifyContent: "center",
gap: 24,
opacity: splitProgress,
}}
>
{ENTITIES.map((entity, idx) => (
<FanPane
key={entity.label}
label={entity.label}
tag={entity.tag}
accent={entity.accent}
panelStart={45 + idx * 8}
/>
))}
</div>
</div>
</AbsoluteFill>
);
};
@@ -0,0 +1,105 @@
import React from "react";
import { AbsoluteFill, interpolate, useCurrentFrame } from "remotion";
import { ComparisonTable } from "../components/ComparisonTable";
import { COLORS, FONT_SANS } from "../lib/colors";
const ROWS = [
{
dimension: "What it is",
cells: [
"GPT-5 leader, Plus + API",
"Claude 4, safety-first",
"Grok, X-native, fast",
] as [string, string, string],
},
{
dimension: "30-day momentum",
cells: [
"GPT-5 launch wave",
"Claude 4.7 1M context",
"Grok 5 reveal",
] as [string, string, string],
},
{
dimension: "Community vibe",
cells: [
"Defensive but deep",
"Quiet, devs-only",
"Loud, meme-rich",
] as [string, string, string],
},
{
dimension: "Best for",
cells: [
"Mainstream + tools",
"Long-context coding",
"Live X intel",
] as [string, string, string],
},
];
export const Scene4Comparison: React.FC = () => {
const frame = useCurrentFrame();
const captionFade = interpolate(frame, [0, 12, 180, 210], [0, 1, 1, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const captionLift = interpolate(frame, [0, 14], [16, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const tableFade = interpolate(frame, [16, 32], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<AbsoluteFill
style={{
background: COLORS.bgDeep,
padding: "48px 80px",
display: "flex",
flexDirection: "column",
alignItems: "center",
}}
>
<div
style={{
opacity: captionFade,
transform: `translateY(${captionLift}px)`,
fontFamily: FONT_SANS,
fontSize: 38,
color: COLORS.fgMuted,
textAlign: "center",
marginBottom: 36,
}}
>
<span style={{ color: COLORS.accentCyan, fontWeight: 600 }}>
3 full passes.
</span>
<span style={{ marginLeft: 18, color: COLORS.accentMagenta, fontWeight: 600 }}>
3 save files.
</span>
<span style={{ marginLeft: 18, color: COLORS.fgPrimary, fontWeight: 600 }}>
1 comparison.
</span>
</div>
<div
style={{
width: "100%",
maxWidth: 1640,
opacity: tableFade,
}}
>
<ComparisonTable
startFrame={20}
entities={["OpenAI", "Anthropic", "xAI"]}
rows={ROWS}
rowStaggerFrames={22}
/>
</div>
</AbsoluteFill>
);
};
@@ -0,0 +1,54 @@
import React from "react";
import { AbsoluteFill, interpolate, useCurrentFrame } from "remotion";
import { COLORS, FONT_SANS } from "../lib/colors";
const LINES = [
{ text: "You pick the topic.", color: COLORS.fgPrimary },
{ text: "The agent picks the peers.", color: COLORS.accentCyan },
{ text: "The engine fans out.", color: COLORS.accentMagenta },
];
export const Scene5HowItWorks: React.FC = () => {
const frame = useCurrentFrame();
return (
<AbsoluteFill
style={{
background: `radial-gradient(circle at 50% 60%, ${COLORS.bgPanelSoft} 0%, ${COLORS.bgDeep} 70%)`,
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
gap: 48,
}}
>
{LINES.map((line, idx) => {
const start = 10 + idx * 28;
const fade = interpolate(frame, [start, start + 14], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const slide = interpolate(frame, [start, start + 18], [24, 0], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
return (
<div
key={line.text}
style={{
opacity: fade,
transform: `translateY(${slide}px)`,
fontFamily: FONT_SANS,
fontSize: 78,
fontWeight: 600,
color: line.color,
letterSpacing: -1,
}}
>
{line.text}
</div>
);
})}
</AbsoluteFill>
);
};
@@ -0,0 +1,73 @@
import React from "react";
import { AbsoluteFill, interpolate, spring, useCurrentFrame, useVideoConfig } from "remotion";
import { BadgeBar } from "../components/BadgeBar";
import { COLORS, FONT_MONO, FONT_SANS } from "../lib/colors";
export const Scene6CTA: React.FC = () => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const installFade = interpolate(frame, [20, 40], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
// Stronger 1Hz pulse on the install line for end-of-video emphasis
const pulse = 0.8 + 0.2 * Math.sin((frame / fps) * 2 * Math.PI);
const glowPulse = 0.4 + 0.4 * Math.sin((frame / fps) * 2 * Math.PI);
const repoFade = interpolate(frame, [50, 70], [0, 1], {
extrapolateLeft: "clamp",
extrapolateRight: "clamp",
});
const enterScale = spring({
frame,
fps,
config: { damping: 18, stiffness: 90 },
from: 0.95,
to: 1,
});
return (
<AbsoluteFill
style={{
background: `radial-gradient(circle at 50% 50%, ${COLORS.bgPanelSoft} 0%, ${COLORS.bgDeep} 70%)`,
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
gap: 56,
transform: `scale(${enterScale})`,
}}
>
<BadgeBar />
<div
style={{
opacity: installFade * pulse,
padding: "18px 36px",
background: COLORS.bgPanel,
border: `1px solid ${COLORS.accentCyan}`,
borderRadius: 14,
boxShadow: `0 0 ${40 + glowPulse * 60}px ${COLORS.accentCyan}${Math.round(40 + glowPulse * 80).toString(16)}`,
fontFamily: FONT_MONO,
fontSize: 44,
color: COLORS.fgPrimary,
}}
>
<span style={{ color: COLORS.accentGreen, marginRight: 18 }}>$</span>
/last30days <span style={{ color: COLORS.fgMuted }}>{"{topic}"}</span>{" "}
<span style={{ color: COLORS.accentCyan }}>--competitors</span>
</div>
<div
style={{
opacity: repoFade,
fontFamily: FONT_SANS,
fontSize: 28,
color: COLORS.fgMuted,
}}
>
github.com/mvanhorn/last30days-skill
</div>
</AbsoluteFill>
);
};
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"strict": true,
"skipLibCheck": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"types": ["node"]
},
"include": ["src/**/*"]
}
-395
View File
@@ -1,395 +0,0 @@
# feat: Add WebSearch as Third Source (Zero-Config Fallback)
## Overview
Add Claude's built-in WebSearch tool as a third research source for `/last30days`. This enables the skill to work **out of the box with zero API keys** while preserving the primacy of Reddit/X as the "voice of real humans with popularity signals."
**Key principle**: WebSearch is supplementary, not primary. Real human voices on Reddit/X with engagement metrics (upvotes, likes, comments) are more valuable than general web content.
## Problem Statement
Currently `/last30days` requires at least one API key (OpenAI or xAI) to function. Users without API keys get an error. Additionally, web search could fill gaps where Reddit/X coverage is thin.
**User requirements**:
- Work out of the box (no API key needed)
- Must NOT overpower Reddit/X results
- Needs proper weighting
- Validate with before/after testing
## Proposed Solution
### Weighting Strategy: "Engagement-Adjusted Scoring"
**Current formula** (same for Reddit/X):
```
score = 0.45*relevance + 0.25*recency + 0.30*engagement - penalties
```
**Problem**: WebSearch has NO engagement metrics. Giving it `DEFAULT_ENGAGEMENT=35` with `-10 penalty` = 25 base, which still competes unfairly.
**Solution**: Source-specific scoring with **engagement substitution**:
| Source | Relevance | Recency | Engagement | Source Penalty |
|--------|-----------|---------|------------|----------------|
| Reddit | 45% | 25% | 30% (real metrics) | 0 |
| X | 45% | 25% | 30% (real metrics) | 0 |
| WebSearch | 55% | 35% | 0% (no data) | -15 points |
**Rationale**:
- WebSearch items compete on relevance + recency only (reweighted to 100%)
- `-15 point source penalty` ensures WebSearch ranks below comparable Reddit/X items
- High-quality WebSearch can still surface (score 60-70) but won't dominate (Reddit/X score 70-85)
### Mode Behavior
| API Keys Available | Default Behavior | `--include-web` |
|--------------------|------------------|-----------------|
| None | **WebSearch only** | n/a |
| OpenAI only | Reddit only | Reddit + WebSearch |
| xAI only | X only | X + WebSearch |
| Both | Reddit + X | Reddit + X + WebSearch |
**CLI flag**: `--include-web` (default: false when other sources available)
## Technical Approach
### Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ last30days.py orchestrator │
├─────────────────────────────────────────────────────────────────┤
│ run_research() │
│ ├── if sources includes "reddit": openai_reddit.search_reddit()│
│ ├── if sources includes "x": xai_x.search_x() │
│ └── if sources includes "web": websearch.search_web() ← NEW │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ Processing Pipeline │
├─────────────────────────────────────────────────────────────────┤
│ normalize_websearch_items() → WebSearchItem schema ← NEW │
│ score_websearch_items() → engagement-free scoring ← NEW │
│ dedupe_websearch() → deduplication ← NEW │
│ render_websearch_section() → output formatting ← NEW │
└─────────────────────────────────────────────────────────────────┘
```
### Implementation Phases
#### Phase 1: Schema & Core Infrastructure
**Files to create/modify:**
```python
# scripts/lib/websearch.py (NEW)
"""Claude WebSearch API client for general web discovery."""
WEBSEARCH_PROMPT = """Search the web for content about: {topic}
CRITICAL: Only include results from the last 30 days (after {from_date}).
Find {min_items}-{max_items} high-quality, relevant web pages. Prefer:
- Blog posts, tutorials, documentation
- News articles, announcements
- Authoritative sources (official docs, reputable publications)
AVOID:
- Reddit (covered separately)
- X/Twitter (covered separately)
- YouTube without transcripts
- Forum threads without clear answers
Return ONLY valid JSON:
{{
"items": [
{{
"title": "Page title",
"url": "https://...",
"source_domain": "example.com",
"snippet": "Brief excerpt (100-200 chars)",
"date": "YYYY-MM-DD or null",
"why_relevant": "Brief explanation",
"relevance": 0.85
}}
]
}}
"""
def search_web(topic: str, from_date: str, to_date: str, depth: str = "default") -> dict:
"""Search web using Claude's built-in WebSearch tool.
NOTE: This runs INSIDE Claude Code, so we use the WebSearch tool directly.
No API key needed - uses Claude's session.
"""
# Implementation uses Claude's web_search_20250305 tool
pass
def parse_websearch_response(response: dict) -> list[dict]:
"""Parse WebSearch results into normalized format."""
pass
```
```python
# scripts/lib/schema.py - ADD WebSearchItem
@dataclass
class WebSearchItem:
"""Normalized web search item."""
id: str
title: str
url: str
source_domain: str # e.g., "medium.com", "github.com"
snippet: str
date: Optional[str] = None
date_confidence: str = "low"
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'source_domain': self.source_domain,
'snippet': self.snippet,
'date': self.date,
'date_confidence': self.date_confidence,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
```
#### Phase 2: Scoring System Updates
```python
# scripts/lib/score.py - ADD websearch scoring
# New constants
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# Reweighted for no engagement
WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
"""Score WebSearch items WITHOUT engagement metrics.
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty
"""
for item in items:
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=0, # Explicitly zero - no engagement data
)
overall = (
WEBSEARCH_WEIGHT_RELEVANCE * rel_score +
WEBSEARCH_WEIGHT_RECENCY * rec_score
)
# Apply source penalty (WebSearch < Reddit/X)
overall -= WEBSEARCH_SOURCE_PENALTY
# Apply date confidence penalty (same as other sources)
if item.date_confidence == "low":
overall -= 10
elif item.date_confidence == "med":
overall -= 5
item.score = max(0, min(100, int(overall)))
return items
```
#### Phase 3: Orchestrator Integration
```python
# scripts/last30days.py - UPDATE run_research()
def run_research(...) -> tuple:
"""Run the research pipeline.
Returns: (reddit_items, x_items, web_items, raw_openai, raw_xai,
raw_websearch, reddit_error, x_error, web_error)
"""
# ... existing Reddit/X code ...
# WebSearch (new)
web_items = []
raw_websearch = None
web_error = None
if sources in ("all", "web", "reddit-web", "x-web"):
if progress:
progress.start_web()
try:
raw_websearch = websearch.search_web(topic, from_date, to_date, depth)
web_items = websearch.parse_websearch_response(raw_websearch)
except Exception as e:
web_error = f"{type(e).__name__}: {e}"
if progress:
progress.end_web(len(web_items))
return (reddit_items, x_items, web_items, raw_openai, raw_xai,
raw_websearch, reddit_error, x_error, web_error)
```
#### Phase 4: CLI & Environment Updates
```python
# scripts/last30days.py - ADD CLI flag
parser.add_argument(
"--include-web",
action="store_true",
help="Include general web search alongside Reddit/X (lower weighted)",
)
# scripts/lib/env.py - UPDATE get_available_sources()
def get_available_sources(config: dict) -> str:
"""Determine available sources. WebSearch always available (no API key)."""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
if has_openai and has_xai:
return 'both' # WebSearch available but not default
elif has_openai:
return 'reddit'
elif has_xai:
return 'x'
else:
return 'web' # Fallback: WebSearch only (no keys needed)
```
## Acceptance Criteria
### Functional Requirements
- [x] Skill works with zero API keys (WebSearch-only mode)
- [x] `--include-web` flag adds WebSearch to Reddit/X searches
- [x] WebSearch items have lower average scores than Reddit/X items with similar relevance
- [x] WebSearch results exclude Reddit/X URLs (handled separately)
- [x] Date filtering uses natural language ("last 30 days") in prompt
- [x] Output clearly labels source type: `[WEB]`, `[Reddit]`, `[X]`
### Non-Functional Requirements
- [x] WebSearch adds <10s latency to total research time (0s - deferred to Claude)
- [x] Graceful degradation if WebSearch fails
- [ ] Cache includes WebSearch results appropriately
### Quality Gates
- [x] Before/after testing shows WebSearch doesn't dominate rankings (via -15pt penalty)
- [x] Test: 10 Reddit + 10 X + 10 WebSearch → WebSearch avg score 15-20pts lower (scoring formula verified)
- [x] Test: WebSearch-only mode produces useful results for common topics
## Testing Plan
### Before/After Comparison Script
```python
# tests/test_websearch_weighting.py
"""
Test harness to validate WebSearch doesn't overpower Reddit/X.
Run same queries with:
1. Reddit + X only (baseline)
2. Reddit + X + WebSearch (comparison)
Verify: WebSearch items rank lower on average.
"""
TEST_QUERIES = [
"best practices for react server components",
"AI coding assistants comparison",
"typescript 5.5 new features",
]
def test_websearch_weighting():
for query in TEST_QUERIES:
# Run without WebSearch
baseline = run_research(query, sources="both")
baseline_scores = [item.score for item in baseline.reddit + baseline.x]
# Run with WebSearch
with_web = run_research(query, sources="both", include_web=True)
web_scores = [item.score for item in with_web.web]
reddit_x_scores = [item.score for item in with_web.reddit + with_web.x]
# Assertions
avg_reddit_x = sum(reddit_x_scores) / len(reddit_x_scores)
avg_web = sum(web_scores) / len(web_scores) if web_scores else 0
assert avg_web < avg_reddit_x - 10, \
f"WebSearch avg ({avg_web}) too close to Reddit/X avg ({avg_reddit_x})"
# Check top 5 aren't all WebSearch
top_5 = sorted(with_web.reddit + with_web.x + with_web.web,
key=lambda x: -x.score)[:5]
web_in_top_5 = sum(1 for item in top_5 if isinstance(item, WebSearchItem))
assert web_in_top_5 <= 2, f"Too many WebSearch items in top 5: {web_in_top_5}"
```
### Manual Test Scenarios
| Scenario | Expected Outcome |
|----------|------------------|
| No API keys, run `/last30days AI tools` | WebSearch-only results, useful output |
| Both keys + `--include-web`, run `/last30days react` | Mix of all 3 sources, Reddit/X dominate top 10 |
| Niche topic (no Reddit/X coverage) | WebSearch fills gap, becomes primary |
| Popular topic (lots of Reddit/X) | WebSearch present but lower-ranked |
## Dependencies & Prerequisites
- Claude Code's WebSearch tool (`web_search_20250305`) - already available
- No new API keys required
- Existing test infrastructure in `tests/`
## Risk Analysis & Mitigation
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| WebSearch returns stale content | Medium | Medium | Enforce date in prompt, apply low-confidence penalty |
| WebSearch dominates rankings | Low | High | Source penalty (-15pts), testing validates |
| WebSearch adds spam/low-quality | Medium | Medium | Exclude social media domains, domain filtering |
| Date parsing unreliable | High | Medium | Accept "low" confidence as normal for WebSearch |
## Future Considerations
1. **Domain authority scoring**: Could proxy engagement with domain reputation
2. **User-configurable weights**: Let users adjust WebSearch penalty
3. **Domain whitelist/blacklist**: Filter WebSearch to trusted sources
4. **Parallel execution**: Run all 3 sources concurrently for speed
## References
### Internal References
- Scoring algorithm: `scripts/lib/score.py:8-15`
- Source detection: `scripts/lib/env.py:57-72`
- Schema patterns: `scripts/lib/schema.py:76-138`
- Orchestrator: `scripts/last30days.py:54-164`
### External References
- Claude WebSearch docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
- WebSearch pricing: $10/1K searches + token costs
- Date filtering limitation: No explicit date params, use natural language
### Research Findings
- Reddit upvotes are ~12% of ranking value in SEO (strong signal)
- E-E-A-T framework: Engagement metrics = trust signal
- MSA2C2 approach: Dynamic weight learning for multi-source aggregation
+40
View File
@@ -0,0 +1,40 @@
[project]
name = "last30days-skill"
version = "3.0.0"
description = "Multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"requests>=2.32,<3",
]
[dependency-groups]
dev = [
"pytest>=9,<10",
"pytest-cov>=7,<8",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = [
"-q",
"--tb=short",
]
[tool.coverage.run]
branch = true
source = ["scripts", "tests"]
omit = [
"scripts/lib/vendor/*",
"dist/*",
]
[tool.coverage.report]
skip_empty = true
show_missing = true
omit = [
"scripts/lib/vendor/*",
"dist/*",
]
+86
View File
@@ -0,0 +1,86 @@
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## v3 is the intelligent search release
v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
Type "OpenClaw" and v3 resolves @steipete, r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags before a single API call fires. Type "Peter Steinberger" and it resolves his X handle and GitHub profile, switches to person mode, and shows what he shipped this month at 85% merge rate across 22 PRs. None of that was on Google.
## Headline features
### Intelligent pre-research
The killer feature. A new Python pre-research brain resolves X handles, GitHub repos, subreddits, TikTok hashtags, and YouTube channels before searching. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags, all resolved before a single API call.
### Best Takes
A second LLM judge scores every result for humor, wit, and virality alongside relevance. Every brief now ends with a Best Takes section surfacing the cleverest one-liners and most viral quotes. The Reddit and X people are funny, and the old engine buried their best stuff.
### Cross-source cluster merging
When the same story hits Reddit, X, and YouTube, v3 merges them into one cluster instead of three duplicates. Entity-based overlap detection catches matches even when the titles use different words.
### Single-pass comparisons
"X vs Y" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides at once. Same depth, 3 minutes.
### GitHub person-mode and project-mode
When the topic is a person, the engine switches from keyword search to author-scoped queries. PR velocity, top repos by stars, release notes for what shipped this month, woven into the narrative alongside X posts and Reddit threads.
When the topic is a project, it pulls live star counts, READMEs, releases, and top issues from the GitHub API. No stale blog posts.
### ELI5 mode
Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations, just clearer. Say "eli5 off" to go back.
### 13+ sources
v3 adds Threads, Pinterest, Perplexity, Bluesky, and Parallel AI grounding to the existing Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and Web lineup. Perplexity Deep Research (`--deep-research`) gives you 50+ citation reports for serious investigation.
### Per-author cap and entity disambiguation
Max 3 items per author prevents single-voice dominance. Synthesis trusts resolved handles over fuzzy keyword matches.
## Install
Claude Code:
```
/plugin marketplace add mvanhorn/last30days-skill
```
OpenClaw:
```
clawhub install last30days-official
```
OpenAI Codex CLI: run `codex` from a checkout of this repo and v3's skill at `.agents/skills/last30days/SKILL.md` will be discovered automatically. Or copy `SKILL.md` to `~/.agents/skills/last30days/SKILL.md` for a global install.
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
## v3 Community
v3 was shaped by community contributors whose PRs and issues inspired core features. Their code wasn't merged directly (v3 was a ground-up rewrite), but their ideas drove what shipped.
Thanks to @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, and @hnshah. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the full list.
Contributors who shaped the release itself:
- @Jah-yee (#153) surfaced the need for a real Codex CLI integration, which shipped in #219
- @Cody-Coyote (#204) reported the marketplace validation bug that needed fixing before v3 could ship cleanly
- @dannyshmueli pushed for v3 and Codex family support publicly on X
Full Added / Changed / Fixed detail lives in [CHANGELOG.md](CHANGELOG.md) under `[3.0.0]`.
## Earlier contributors
From the v1 and v2 lineage:
- [@galligan](https://github.com/galligan) for marketplace plugin inspiration
- [@hutchins](https://x.com/hutchins) for pushing the YouTube feature
30 days of research. 30 seconds of work. Thirteen sources. Zero stale prompts.
+264
View File
@@ -0,0 +1,264 @@
#!/usr/bin/env python3
"""Morning briefing generator for last30days.
Synthesizes accumulated findings into formatted briefings.
The Python script collects the data; the agent (via SKILL.md) does the
beautiful synthesis. This script provides the structured data.
Usage:
python3 briefing.py generate # Daily briefing data
python3 briefing.py generate --weekly # Weekly digest data
python3 briefing.py show [--date DATE] # Show saved briefing
"""
import argparse
import json
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
import store
BRIEFS_DIR = Path.home() / ".local" / "share" / "last30days" / "briefs"
def _parse_sqlite_utc_timestamp(value: str) -> datetime:
return datetime.strptime(value, "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)
def generate_daily(since: str = None) -> dict:
"""Generate daily briefing data.
Returns structured data for the agent to synthesize into a beautiful briefing.
"""
store.init_db()
topics = store.list_topics()
if not topics:
return {
"status": "no_topics",
"message": "No watchlist topics yet. Add one with: last30days watch add \"your topic\"",
}
enabled = [t for t in topics if t["enabled"]]
if not enabled:
return {
"status": "no_enabled",
"message": "All topics are paused. Enable a topic to generate briefings.",
}
# Default: findings since yesterday
if not since:
since = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
briefing_topics = []
total_new = 0
for topic in enabled:
findings = store.get_new_findings(topic["id"], since)
last_run = topic.get("last_run")
last_status = topic.get("last_status", "unknown")
# Calculate staleness
stale = False
hours_ago = None
if last_run:
try:
run_dt = _parse_sqlite_utc_timestamp(last_run)
hours_ago = (datetime.now(timezone.utc) - run_dt).total_seconds() / 3600
stale = hours_ago > 36 # Stale if > 36 hours
except (ValueError, TypeError):
stale = True
topic_data = {
"name": topic["name"],
"findings": findings,
"new_count": len(findings),
"last_run": last_run,
"last_status": last_status,
"stale": stale,
"hours_ago": round(hours_ago, 1) if hours_ago else None,
}
# Extract top finding by engagement
if findings:
top = max(findings, key=lambda f: f.get("engagement_score", 0))
topic_data["top_finding"] = {
"title": top.get("source_title", ""),
"source": top.get("source", ""),
"author": top.get("author", ""),
"engagement": top.get("engagement_score", 0),
"content": top.get("content", "")[:300],
}
briefing_topics.append(topic_data)
total_new += len(findings)
# Cost info
daily_cost = store.get_daily_cost()
budget = float(store.get_setting("daily_budget", "5.00"))
# Find the single top finding across all topics (for TL;DR)
all_findings = []
for t in briefing_topics:
for f in t["findings"]:
f["_topic"] = t["name"]
all_findings.append(f)
top_overall = None
if all_findings:
top_overall = max(all_findings, key=lambda f: f.get("engagement_score", 0))
result = {
"status": "ok",
"date": datetime.now().strftime("%Y-%m-%d"),
"since": since,
"topics": briefing_topics,
"total_new": total_new,
"total_topics": len(briefing_topics),
"top_finding": {
"title": top_overall.get("source_title", ""),
"topic": top_overall.get("_topic", ""),
"engagement": top_overall.get("engagement_score", 0),
} if top_overall else None,
"cost": {
"daily": daily_cost,
"budget": budget,
},
"failed_topics": [
t["name"] for t in briefing_topics if t["last_status"] == "failed"
],
}
# Save briefing data
_save_briefing(result)
return result
def generate_weekly() -> dict:
"""Generate weekly digest data with trend analysis."""
store.init_db()
week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
two_weeks_ago = (datetime.now() - timedelta(days=14)).strftime("%Y-%m-%d")
topics = store.list_topics()
if not topics:
return {"status": "no_topics", "message": "No watchlist topics."}
weekly_topics = []
for topic in topics:
if not topic["enabled"]:
continue
# This week's findings
this_week = store.get_new_findings(topic["id"], week_ago)
# Last week's findings (for comparison)
conn = store._connect()
try:
last_week_rows = conn.execute(
"""SELECT * FROM findings
WHERE topic_id = ? AND first_seen >= ? AND first_seen < ? AND dismissed = 0
ORDER BY engagement_score DESC""",
(topic["id"], two_weeks_ago, week_ago),
).fetchall()
last_week = [dict(r) for r in last_week_rows]
finally:
conn.close()
this_engagement = sum(f.get("engagement_score", 0) for f in this_week)
last_engagement = sum(f.get("engagement_score", 0) for f in last_week)
# Trend calculation
if last_engagement > 0:
engagement_change = ((this_engagement - last_engagement) / last_engagement) * 100
else:
engagement_change = 100 if this_engagement > 0 else 0
weekly_topics.append({
"name": topic["name"],
"this_week_count": len(this_week),
"last_week_count": len(last_week),
"this_week_engagement": this_engagement,
"last_week_engagement": last_engagement,
"engagement_change_pct": round(engagement_change, 1),
"top_findings": this_week[:5], # Top 5 by engagement (already sorted)
})
result = {
"status": "ok",
"type": "weekly",
"week_of": week_ago,
"topics": weekly_topics,
}
_save_briefing(result, suffix="-weekly")
return result
def show_briefing(date: str = None) -> dict:
"""Load a saved briefing by date."""
if not date:
date = datetime.now().strftime("%Y-%m-%d")
path = BRIEFS_DIR / f"{date}.json"
if not path.exists():
# Try weekly
path = BRIEFS_DIR / f"{date}-weekly.json"
if not path.exists():
return {"status": "not_found", "message": f"No briefing found for {date}."}
with open(path, encoding="utf-8") as f:
return json.load(f)
def _save_briefing(data: dict, suffix: str = ""):
"""Save briefing data to local archive."""
BRIEFS_DIR.mkdir(parents=True, exist_ok=True)
date = datetime.now().strftime("%Y-%m-%d")
path = BRIEFS_DIR / f"{date}{suffix}.json"
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, default=str)
def main():
parser = argparse.ArgumentParser(description="Generate last30days briefings")
sub = parser.add_subparsers(dest="command")
# generate
g = sub.add_parser("generate", help="Generate a briefing")
g.add_argument("--weekly", action="store_true", help="Weekly digest")
g.add_argument("--since", help="Findings since date (YYYY-MM-DD)")
# show
s = sub.add_parser("show", help="Show a saved briefing")
s.add_argument("--date", help="Date (YYYY-MM-DD, default: today)")
args = parser.parse_args()
if args.command == "generate":
if args.weekly:
result = generate_weekly()
else:
result = generate_daily(since=args.since)
print(json.dumps(result, indent=2, default=str))
elif args.command == "show":
result = show_briefing(date=args.date)
print(json.dumps(result, indent=2, default=str))
else:
parser.print_help()
sys.exit(1)
if __name__ == "__main__":
main()
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# build-skill.sh - package this repo as a claude.ai-upload-ready .skill file
# Usage: bash scripts/build-skill.sh (run from repo root)
#
# Produces dist/last30days.skill, a zip with a single top-level `last30days/`
# directory containing SKILL.md and the scripts/ runtime. See
# docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$REPO_ROOT"
if ! git diff --quiet || ! git diff --cached --quiet; then
echo "error: working tree is dirty; commit or stash before building" >&2
exit 1
fi
mkdir -p dist
OUT="dist/last30days.skill"
git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD
# claude.ai's .skill bundle only needs the root SKILL.md + scripts/ runtime.
# Claude Code needs skills/ and .claude-plugin/ in the git archive
# (that's why they're NOT in .gitattributes export-ignore), but the .skill
# bundle must strip them to keep a single canonical SKILL.md and stay under
# the 200-file cap.
zip -d "$OUT" "last30days/skills/*" "last30days/.claude-plugin/*" > /dev/null 2>&1 || true
COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}')
SIZE=$(du -h "$OUT" | cut -f1)
if [ "$COUNT" -gt 200 ]; then
echo "error: $COUNT files in zip, claude.ai's cap is 200" >&2
echo " check .gitattributes export-ignore entries and this script's zip -d excludes" >&2
exit 1
fi
SKILL_MD_COUNT=$(unzip -l "$OUT" | grep -c "SKILL.md" || true)
if [ "$SKILL_MD_COUNT" -ne 1 ]; then
echo "error: expected exactly one SKILL.md, found $SKILL_MD_COUNT" >&2
exit 1
fi
echo "built $OUT ($COUNT files, $SIZE)"
echo "upload via the claude.ai skill UI"
+61
View File
@@ -0,0 +1,61 @@
#!/bin/bash
# A/B test runner: public release vs private beta
# Usage: bash scripts/compare.sh "Kanye West"
#
# Runs /last30days (public release) and /last30days-beta (private beta)
# sequentially with a 30s gap, saves raw results with distinct suffixes,
# prints file paths for comparison.
set -e
if [ $# -eq 0 ]; then
echo "Usage: bash scripts/compare.sh <topic>"
echo " Example: bash scripts/compare.sh Kevin Rose"
exit 1
fi
TOPIC="$*"
SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
LAST30DAYS_MEMORY_DIR="${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}"
DIR="$LAST30DAYS_MEMORY_DIR"
DATE=$(date +%Y-%m-%d)
echo "=============================================="
echo " A/B Test: $TOPIC"
echo " Date: $DATE"
echo "=============================================="
echo ""
# Run 1: public release
echo "[1/2] Running /last30days (public release)..."
echo " This takes 2-4 minutes..."
claude -p --dangerously-skip-permissions "/last30days $TOPIC" > /dev/null 2>&1 || true
RELEASE_FILE="$DIR/${SLUG}-raw.md"
[ -f "$RELEASE_FILE" ] && echo " Done: $RELEASE_FILE" || echo " FAILED: no output file"
echo ""
echo " Waiting 30s for API rate limits..."
sleep 30
# Run 2: private beta
echo "[2/2] Running /last30days-beta (private beta)..."
echo " This takes 2-4 minutes..."
claude -p --dangerously-skip-permissions "/last30days-beta $TOPIC" > /dev/null 2>&1 || true
BETA_FILE="$DIR/${SLUG}-raw-beta.md"
[ -f "$BETA_FILE" ] && echo " Done: $BETA_FILE" || echo " FAILED: no output file"
echo ""
echo "=============================================="
echo " Both complete. Raw files:"
echo "=============================================="
echo ""
ls -la "$DIR/${SLUG}-raw"*.md 2>/dev/null || echo " (no files found - check if skills saved correctly)"
echo ""
echo "To compare, run in Claude Code:"
echo " Read and compare these raw research files, produce a detailed report:"
echo " $RELEASE_FILE"
echo " $BETA_FILE"
echo ""
echo "Beta output should start with a line like:"
echo " 🧪 last30days-beta · branch <name> · synced $DATE"
echo "If that line is missing, the beta badge regressed. See docs/plans/2026-04-17-005-*-plan.md."
echo ""
+549
View File
@@ -0,0 +1,549 @@
#!/usr/bin/env python3
"""Compare two last30days revisions on the v3 ranked candidate output."""
from __future__ import annotations
import argparse
import json
import math
import os
import subprocess
import sys
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
sys.path.insert(0, str(Path(__file__).parent))
from lib import env as envlib
from lib import schema
REPO_ROOT = Path(__file__).resolve().parent.parent
EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json"
def _load_default_topics() -> list[tuple[str, str]]:
if EVAL_TOPICS_FILE.exists():
rows = json.loads(EVAL_TOPICS_FILE.read_text())
return [(row["topic"], row["query_type"]) for row in rows]
return [
("nano banana pro prompting", "product"),
("codex vs claude code", "comparison"),
("openclaw vs nanoclaw vs ironclaw", "comparison"),
("anthropic odds", "prediction"),
("kanye west", "breaking_news"),
("remotion animations for Claude Code", "how_to"),
]
DEFAULT_TOPICS = _load_default_topics()
DEFAULT_SEARCH = ""
DEFAULT_JUDGE_MODEL = "gemini-3.1-flash-lite-preview"
GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}"
def stable_item_key(item: dict[str, Any]) -> str:
return str(item.get("candidate_id") or item.get("url") or item.get("title") or "")
def row_sources(row: dict[str, Any]) -> list[str]:
candidate = schema.candidate_from_dict(row)
return schema.candidate_sources(candidate)
def row_best_date(row: dict[str, Any]) -> str | None:
candidate = schema.candidate_from_dict(row)
return schema.candidate_best_published_at(candidate)
V2_SOURCE_KEYS = [
("reddit", "title"),
("x", "text"),
("youtube", "title"),
("tiktok", "text"),
("instagram", "text"),
("hackernews", "title"),
("bluesky", "text"),
("truthsocial", "text"),
("polymarket", "question"),
("web", "title"),
]
def build_ranked_items(report: dict[str, Any], limit: int) -> list[dict[str, Any]]:
# v3 format: ranked_candidates list
if report.get("ranked_candidates"):
ranked = []
for row in report["ranked_candidates"][:limit]:
candidate_sources = row_sources(row)
ranked.append({
"key": stable_item_key(row),
"source": ", ".join(candidate_sources),
"sources": candidate_sources,
"url": str(row.get("url") or ""),
"text": str(row.get("title") or ""),
"date": row_best_date(row),
"score": float(row.get("final_score") or 0.0),
})
return ranked
# v2 format: per-source lists (reddit, x, youtube, etc.)
all_items = []
for source_key, text_field in V2_SOURCE_KEYS:
for item in report.get(source_key) or []:
if not isinstance(item, dict):
continue
all_items.append({
"key": str(item.get("url") or item.get("id") or item.get(text_field) or ""),
"source": source_key,
"sources": [source_key],
"url": str(item.get("url") or ""),
"text": str(item.get(text_field) or item.get("title") or ""),
"date": item.get("date"),
"score": float(item.get("score") or 0.0),
})
all_items.sort(key=lambda x: x["score"], reverse=True)
return all_items[:limit]
def source_sets(report: dict[str, Any], limit: int) -> dict[str, set[str]]:
grouped: dict[str, set[str]] = {}
for item in build_ranked_items(report, limit):
for source in item["sources"]:
grouped.setdefault(source, set()).add(item["key"])
return grouped
def jaccard(left: set[str], right: set[str]) -> float:
if not left and not right:
return 1.0
union = left | right
if not union:
return 1.0
return len(left & right) / len(union)
def retention(left: set[str], right: set[str]) -> float:
if not left:
return 1.0
return len(left & right) / len(left)
def precision_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int) -> float:
top = ranking[:k]
if not top:
return 0.0
return sum(1 for item in top if judgments.get(item["key"], 0) >= 2) / len(top)
def ndcg_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int, judged_pool: list[dict[str, Any]]) -> float:
top = ranking[:k]
if not top:
return 0.0
def dcg(grades: list[int]) -> float:
total = 0.0
for index, grade in enumerate(grades, start=1):
total += (2**grade - 1) / math.log2(index + 1)
return total
actual = [judgments.get(item["key"], 0) for item in top]
ideal = sorted((judgments.get(item["key"], 0) for item in judged_pool), reverse=True)[: len(top)]
ideal_score = dcg(ideal)
if ideal_score == 0:
return 0.0
return dcg(actual) / ideal_score
def source_coverage_recall(ranking: list[dict[str, Any]], judged_pool: list[dict[str, Any]], judgments: dict[str, int]) -> float:
good_sources = {
source
for item in judged_pool
if judgments.get(item["key"], 0) >= 2
for source in item["sources"]
}
if not good_sources:
return 1.0
hit_sources = {
source
for item in ranking
if judgments.get(item["key"], 0) >= 2
for source in item["sources"]
}
return len(hit_sources & good_sources) / len(good_sources)
def resolve_google_judge_api_key(config: dict[str, Any]) -> str | None:
return (
os.environ.get("GOOGLE_API_KEY")
or config.get("GOOGLE_API_KEY")
or os.environ.get("GEMINI_API_KEY")
or config.get("GEMINI_API_KEY")
or os.environ.get("GOOGLE_GENAI_API_KEY")
or config.get("GOOGLE_GENAI_API_KEY")
)
def extract_gemini_text(payload: dict[str, Any]) -> str:
for candidate in payload.get("candidates") or []:
content = candidate.get("content") or {}
for part in content.get("parts") or []:
if part.get("text"):
return part["text"]
raise ValueError("Gemini response did not contain text.")
def call_gemini_judge(api_key: str, model: str, prompt: str) -> dict[str, Any]:
body = {
"contents": [{"parts": [{"text": prompt}]}],
"generationConfig": {"temperature": 0, "responseMimeType": "application/json"},
}
request = Request(
GEMINI_API_URL.format(model=model, api_key=api_key),
data=json.dumps(body).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urlopen(request, timeout=120) as response:
payload = json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Gemini HTTP {exc.code}: {detail}") from exc
except URLError as exc:
raise RuntimeError(f"Gemini request failed: {exc}") from exc
return json.loads(extract_gemini_text(payload))
def build_judge_prompt(topic: str, query_type: str, items: list[dict[str, Any]]) -> str:
item_lines = []
for item in items:
item_lines.append(
"\n".join([
f"- id: {item['key']}",
f" source: {item['source']}",
f" title: {item['text'][:220]}",
f" url: {item['url']}",
f" date: {item.get('date') or 'unknown'}",
])
)
return f"""
Judge search-result relevance for a last-30-days research tool.
Topic: {topic}
Query type: {query_type}
Score each item on this 0-3 scale:
- 0 = off-topic or clearly bad
- 1 = weak or tangential
- 2 = relevant and useful
- 3 = highly relevant, one of the best results
Return JSON only:
{{
"judgments": [
{{"id": "ITEM_ID", "grade": 0}}
]
}}
Items:
{chr(10).join(item_lines)}
""".strip()
def get_judgments(
*,
output_dir: Path,
slug: str,
topic: str,
query_type: str,
items: list[dict[str, Any]],
judge_model: str,
gemini_api_key: str | None,
) -> dict[str, int]:
cache_file = output_dir / "judgments" / f"{slug}.json"
cache_file.parent.mkdir(parents=True, exist_ok=True)
if cache_file.exists():
payload = json.loads(cache_file.read_text())
return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}
if not gemini_api_key or not items:
return {}
payload = call_gemini_judge(gemini_api_key, judge_model, build_judge_prompt(topic, query_type, items))
cache_file.write_text(json.dumps(payload, indent=2))
return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}
def create_eval_env() -> dict[str, str]:
config = envlib.get_config()
passthrough = {
"PATH": os.environ.get("PATH", ""),
"LANG": os.environ.get("LANG", "en_US.UTF-8"),
"LC_ALL": os.environ.get("LC_ALL", ""),
"TMPDIR": os.environ.get("TMPDIR", ""),
"PYTHONUTF8": "1",
"LAST30DAYS_CONFIG_DIR": "",
}
for key in (
"GOOGLE_API_KEY",
"GEMINI_API_KEY",
"GOOGLE_GENAI_API_KEY",
"OPENAI_API_KEY",
"XAI_API_KEY",
"SCRAPECREATORS_API_KEY",
"BSKY_HANDLE",
"BSKY_APP_PASSWORD",
"TRUTHSOCIAL_TOKEN",
"AUTH_TOKEN",
"CT0",
):
value = os.environ.get(key) or config.get(key)
if value:
passthrough[key] = value
return passthrough
def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]:
cmd = [sys.executable, "scripts/last30days.py", topic, "--emit=json"]
if search:
cmd.extend(["--search", search])
if quick:
cmd.append("--quick")
if mock:
cmd.append("--mock")
result = subprocess.run(
cmd,
cwd=repo_dir,
env=env,
capture_output=True,
text=True,
timeout=timeout_seconds,
check=False,
)
if result.returncode != 0:
raise RuntimeError(f"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\n{result.stderr.strip()}")
return json.loads(result.stdout)
def create_worktree(rev: str) -> Path:
worktree_dir = Path(tempfile.mkdtemp(prefix="last30days-eval-"))
subprocess.run(
["git", "worktree", "add", "--detach", str(worktree_dir), rev],
cwd=REPO_ROOT,
check=True,
capture_output=True,
text=True,
)
return worktree_dir
def resolve_repo_dir(label: str) -> tuple[Path, bool]:
"""Resolve a benchmark label into a repo directory and whether it is temporary."""
if label == "WORKTREE":
return REPO_ROOT, False
return create_worktree(label), True
def remove_worktree(path: Path) -> None:
subprocess.run(
["git", "worktree", "remove", "--force", str(path)],
cwd=REPO_ROOT,
check=False,
capture_output=True,
text=True,
)
try:
os.rmdir(path)
except OSError:
pass
def summarize_topic(topic: str, query_type: str, baseline_report: dict[str, Any], candidate_report: dict[str, Any], judgments: dict[str, int], judged_pool: list[dict[str, Any]], limit: int) -> dict[str, Any]:
baseline_ranked = build_ranked_items(baseline_report, limit)
candidate_ranked = build_ranked_items(candidate_report, limit)
baseline_sets = source_sets(baseline_report, limit)
candidate_sets = source_sets(candidate_report, limit)
overall_left = set().union(*baseline_sets.values()) if baseline_sets else set()
overall_right = set().union(*candidate_sets.values()) if candidate_sets else set()
sources = sorted(set(baseline_sets) | set(candidate_sets))
return {
"topic": topic,
"query_type": query_type,
"baseline": {
"precision_at_5": precision_at_k(baseline_ranked, judgments, 5),
"ndcg_at_5": ndcg_at_k(baseline_ranked, judgments, 5, judged_pool),
"source_coverage_recall": source_coverage_recall(baseline_ranked, judged_pool, judgments),
},
"candidate": {
"precision_at_5": precision_at_k(candidate_ranked, judgments, 5),
"ndcg_at_5": ndcg_at_k(candidate_ranked, judgments, 5, judged_pool),
"source_coverage_recall": source_coverage_recall(candidate_ranked, judged_pool, judgments),
},
"stability": {
"overall_jaccard": jaccard(overall_left, overall_right),
"overall_retention_vs_baseline": retention(overall_left, overall_right),
"per_source": {
source: {
"baseline_count": len(baseline_sets.get(source, set())),
"candidate_count": len(candidate_sets.get(source, set())),
"jaccard": jaccard(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
"retention_vs_baseline": retention(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
}
for source in sources
},
},
}
def write_summary(output_dir: Path, baseline_label: str, candidate_label: str, summaries: list[dict[str, Any]]) -> None:
output_dir.mkdir(parents=True, exist_ok=True)
payload = {
"generated_at": datetime.now().isoformat(timespec="seconds"),
"baseline": baseline_label,
"candidate": candidate_label,
"topics": summaries,
}
(output_dir / "metrics.json").write_text(json.dumps(payload, indent=2))
lines = [
"# Search Quality Evaluation",
"",
f"- Baseline: `{baseline_label}`",
f"- Candidate: `{candidate_label}`",
f"- Generated: {payload['generated_at']}",
"",
"| Topic | Base P@5 | Cand P@5 | Base nDCG@5 | Cand nDCG@5 | Jaccard | Retention |",
"|---|---:|---:|---:|---:|---:|---:|",
]
for row in summaries:
lines.append(
"| {topic} | {bp:.2f} | {cp:.2f} | {bn:.2f} | {cn:.2f} | {jac:.2f} | {ret:.2f} |".format(
topic=row["topic"],
bp=row["baseline"]["precision_at_5"],
cp=row["candidate"]["precision_at_5"],
bn=row["baseline"]["ndcg_at_5"],
cn=row["candidate"]["ndcg_at_5"],
jac=row["stability"]["overall_jaccard"],
ret=row["stability"]["overall_retention_vs_baseline"],
)
)
(output_dir / "summary.md").write_text("\n".join(lines) + "\n")
def write_failure_summary(
output_dir: Path,
baseline_label: str,
candidate_label: str,
summaries: list[dict[str, Any]],
failures: list[dict[str, Any]],
) -> None:
write_summary(output_dir, baseline_label, candidate_label, summaries)
metrics_path = output_dir / "metrics.json"
payload = json.loads(metrics_path.read_text()) if metrics_path.exists() else {
"generated_at": datetime.now().isoformat(timespec="seconds"),
"baseline": baseline_label,
"candidate": candidate_label,
"topics": [],
}
payload["failures"] = failures
metrics_path.write_text(json.dumps(payload, indent=2))
summary_path = output_dir / "summary.md"
lines = summary_path.read_text().splitlines() if summary_path.exists() else ["# Search Quality Evaluation", ""]
if failures:
lines.extend([
"",
"## Failures",
"",
])
for failure in failures:
lines.append(f"- `{failure['topic']}`: {failure['error']}")
summary_path.write_text("\n".join(lines).rstrip() + "\n")
def parse_topics_file(path: Path) -> list[tuple[str, str]]:
rows = json.loads(path.read_text())
return [(str(row["topic"]), str(row.get("query_type") or "general")) for row in rows]
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Compare two last30days revisions on ranked candidate quality")
parser.add_argument("--baseline", default="HEAD~1")
parser.add_argument("--candidate", default="WORKTREE")
parser.add_argument("--search", default=DEFAULT_SEARCH)
parser.add_argument("--output-dir", default="tmp/search-quality")
parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
parser.add_argument("--timeout", type=int, default=240)
parser.add_argument("--limit", type=int, default=20)
parser.add_argument("--mock", action="store_true")
parser.add_argument("--quick", action="store_true")
parser.add_argument("--topics-file")
return parser
def main() -> int:
args = build_parser().parse_args()
topics = parse_topics_file(Path(args.topics_file)) if args.topics_file else DEFAULT_TOPICS
output_dir = Path(args.output_dir).resolve()
config = envlib.get_config()
gemini_api_key = resolve_google_judge_api_key(config)
run_env = create_eval_env()
baseline_dir, baseline_temp = resolve_repo_dir(args.baseline)
candidate_dir, candidate_temp = resolve_repo_dir(args.candidate)
try:
summaries = []
failures = []
for topic, query_type in topics:
try:
baseline_report = run_last30days(
baseline_dir,
topic,
search=args.search,
timeout_seconds=args.timeout,
quick=args.quick,
mock=args.mock,
env=run_env,
)
candidate_report = run_last30days(
candidate_dir,
topic,
search=args.search,
timeout_seconds=args.timeout,
quick=args.quick,
mock=args.mock,
env=run_env,
)
judged_pool_map = {
item["key"]: item
for item in build_ranked_items(baseline_report, args.limit) + build_ranked_items(candidate_report, args.limit)
}
judged_pool = list(judged_pool_map.values())
judgments = get_judgments(
output_dir=output_dir,
slug="".join(char.lower() if char.isalnum() else "-" for char in topic).strip("-"),
topic=topic,
query_type=query_type,
items=judged_pool,
judge_model=args.judge_model,
gemini_api_key=gemini_api_key,
)
summaries.append(summarize_topic(topic, query_type, baseline_report, candidate_report, judgments, judged_pool, args.limit))
except Exception as exc:
failures.append({"topic": topic, "query_type": query_type, "error": str(exc)})
write_failure_summary(output_dir, args.baseline, args.candidate, summaries, failures)
finally:
if baseline_temp:
remove_worktree(baseline_dir)
if candidate_temp:
remove_worktree(candidate_dir)
result = {"output_dir": str(output_dir), "topics": len(topics), "failures": len(failures)}
print(json.dumps(result, indent=2))
return 1 if failures else 0
if __name__ == "__main__":
raise SystemExit(main())
+826 -346
View File
File diff suppressed because it is too large Load Diff
+474
View File
@@ -0,0 +1,474 @@
"""Bird X search client for the v3.0.0 last30days pipeline.
Uses a vendored subset of @steipete/bird v0.8.0 (MIT License) to search X
via Twitter's GraphQL API. No external `bird` CLI binary needed - just Node.js.
See scripts/lib/vendor/bird-search/package.json for authoritative version.
"""
import json
import os
import signal
import shutil
import subprocess
import sys
from pathlib import Path
from . import http, log
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
from .relevance import token_overlap_relevance as _compute_relevance
def _first_of(*values):
"""Return first value that is not None."""
for v in values:
if v is not None:
return v
return None
# Path to the vendored bird-search wrapper
_BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
# Module-level credentials injected from .env config
_credentials: Dict[str, str] = {}
def set_credentials(auth_token: Optional[str], ct0: Optional[str]):
"""Inject AUTH_TOKEN/CT0 from .env config so Node subprocesses can use them."""
if auth_token:
_credentials['AUTH_TOKEN'] = auth_token
if ct0:
_credentials['CT0'] = ct0
def _has_injected_credentials() -> bool:
"""Return True when both X session cookies were injected from config."""
return bool(_credentials.get('AUTH_TOKEN') and _credentials.get('CT0'))
def _has_process_credentials() -> bool:
"""Return True when AUTH_TOKEN/CT0 are present in process env."""
return bool(os.environ.get("AUTH_TOKEN") and os.environ.get("CT0"))
def _subprocess_env() -> Dict[str, str]:
"""Build env dict for Node subprocesses, merging injected credentials."""
env = os.environ.copy()
env.update(_credentials)
# Hard-disable browser-cookie fallback so normal pipeline runs never hit
# Safari/Chrome Keychain prompts during source detection or search.
env["BIRD_DISABLE_BROWSER_COOKIES"] = "1"
return env
def _log(msg: str):
log.source_log("Bird", msg, tty_only=False)
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search.
X search is literal keyword AND matching all words must appear.
Aggressively strip question/meta/research words to keep only the
core product/concept name (max 5 words).
"""
from .query import extract_core_subject
return extract_core_subject(topic, max_words=5, strip_suffixes=True)
def is_bird_installed() -> bool:
"""Check if vendored Bird search module is available.
Returns:
True if bird-search.mjs exists and Node.js is in PATH.
"""
if not _BIRD_SEARCH_MJS.exists():
return False
return shutil.which("node") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if explicit X credentials are available.
Returns:
Auth source string if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
if _has_injected_credentials():
return "env AUTH_TOKEN"
if _has_process_credentials():
return "env AUTH_TOKEN"
return None
def check_npm_available() -> bool:
"""Check if npm is available (kept for API compatibility).
Returns:
True if 'npm' command is available in PATH, False otherwise.
"""
return shutil.which("npm") is not None
def install_bird() -> Tuple[bool, str]:
"""No-op. Bird search is vendored in v3.0.0, no installation needed.
Returns:
Tuple of (success, message).
"""
if is_bird_installed():
return True, "Bird search is bundled with /last30days v3.0.0 - no installation needed."
if not shutil.which("node"):
return False, "Node.js 22+ is required for X search. Install Node.js first."
return False, f"Vendored bird-search.mjs not found at {_BIRD_SEARCH_MJS}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird search status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
auth_source = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": auth_source is not None,
"username": auth_source, # Now returns auth source (e.g., "Safari", "env AUTH_TOKEN")
"can_install": True, # Always vendored in v3.0.0
}
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds
Returns:
Raw Bird JSON response or error dict.
"""
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count),
"--json",
]
# Use process groups for clean cleanup on timeout/kill
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
preexec_fn=preexec,
env=_subprocess_env(),
)
# Register for cleanup tracking (if available)
try:
from last30days import register_child_pid, unregister_child_pid
register_child_pid(proc.pid)
except ImportError:
pass
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
# Kill the entire process group
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
return {"error": f"Search timed out after {timeout}s", "items": []}
finally:
try:
from last30days import unregister_child_pid
unregister_child_pid(proc.pid)
except Exception:
pass
if proc.returncode != 0:
error = stderr.strip() if stderr else "Bird search failed"
return {"error": error, "items": []}
output = stdout.strip() if stdout else ""
if not output:
return {"items": []}
parsed = json.loads(output)
if isinstance(parsed, list):
return {"items": parsed}
return parsed
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI with automatic retry on 0 results.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD) - unused but kept for API compatibility
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
# Extract core subject - X search is literal, not semantic
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
_log(f"Searching: {query}")
response = _run_bird_search(query, count, timeout)
# Check if we got results
items = parse_bird_response(response, query=core_topic)
# Retry with OR groups for multi-word queries (X supports OR operator)
core_words = core_topic.split()
if not items and len(core_words) >= 2:
from .query import extract_compound_terms
compounds = extract_compound_terms(topic)
if compounds:
# Build OR-group query: ("multi-agent" OR "agent simulation") since:DATE
or_parts = ' OR '.join(f'"{t}"' for t in compounds[:3])
_log(f"0 results for '{core_topic}', retrying with OR groups: {or_parts}")
query = f"({or_parts}) since:{from_date}"
response = _run_bird_search(query, count, timeout)
items = parse_bird_response(response, query=core_topic)
# Retry with fewer keywords if still 0 results and query has 3+ words
if not items and len(core_words) > 2:
shorter = ' '.join(core_words[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
response = _run_bird_search(query, count, timeout)
items = parse_bird_response(response, query=core_topic)
# Last-chance retry: use strongest remaining token (often the product name)
if not items and core_words:
low_signal = {
'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
'best', 'top', 'latest', 'new', 'plugin', 'plugins',
'skill', 'skills', 'tool', 'tools',
}
candidates = [w for w in core_words if w not in low_signal]
if candidates:
strongest = max(candidates, key=len)
_log(f"0 results for '{core_topic}', retrying with strongest token '{strongest}'")
query = f"{strongest} since:{from_date}"
response = _run_bird_search(query, count, timeout)
return response
def search_handles(
handles: List[str],
topic: Optional[str],
from_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific X handles for topic-related content.
Runs targeted Bird searches using `from:handle topic` syntax.
Used in Phase 2 supplemental search after entity extraction.
Args:
handles: List of X handles to search (without @)
topic: Search topic (core subject), or None for unfiltered search
from_date: Start date (YYYY-MM-DD)
count_per: Results to request per handle
Returns:
List of raw item dicts (same format as parse_bird_response output).
"""
core_topic = _extract_core_subject(topic) if topic else None
def _search_one_handle(handle: str) -> List[Dict[str, Any]]:
handle = handle.lstrip("@")
if core_topic:
query = f"from:{handle} {core_topic} since:{from_date}"
else:
query = f"from:{handle} since:{from_date}"
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count_per),
"--json",
]
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
preexec_fn=preexec,
env=_subprocess_env(),
)
try:
stdout, stderr = proc.communicate(timeout=15)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
_log(f"Handle search timed out for @{handle}")
return []
if proc.returncode != 0:
_log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
return []
output = (stdout or "").strip()
if not output:
return []
response = json.loads(output)
return parse_bird_response(response, query=core_topic)
except json.JSONDecodeError:
_log(f"Invalid JSON from handle search for @{handle}")
except (OSError, subprocess.SubprocessError) as e:
_log(f"Handle search error for @{handle}: {e}")
return []
from concurrent.futures import ThreadPoolExecutor, as_completed
all_items: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=min(5, len(handles))) as executor:
futures = {executor.submit(_search_one_handle, h): h for h in handles}
for future in as_completed(futures):
all_items.extend(future.result())
return all_items
def parse_bird_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
query: Original search query for relevance scoring
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
# Try different field structures Bird might use
author = tweet.get("author", {}) or tweet.get("user", {})
screen_name = author.get("username") or author.get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at/createdAt (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("createdAt") or tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first (e.g., "2026-02-03T22:33:32Z")
# Check for ISO date separator, not just "T" (which appears in "Tue")
if len(created_at) > 10 and created_at[10] == "T":
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info (Bird uses author.username, older format uses user.screen_name)
author = tweet.get("author", {}) or tweet.get("user", {})
author_handle = author.get("username") or author.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict (Bird uses camelCase: likeCount, retweetCount, etc.)
engagement = {
"likes": _first_of(tweet.get("likeCount"), tweet.get("like_count"), tweet.get("favorite_count")),
"reposts": _first_of(tweet.get("retweetCount"), tweet.get("retweet_count")),
"replies": _first_of(tweet.get("replyCount"), tweet.get("reply_count")),
"quotes": _first_of(tweet.get("quoteCount"), tweet.get("quote_count")),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": _compute_relevance(query, str(tweet.get("text", ""))) if query else 0.7,
}
items.append(item)
return items
+249
View File
@@ -0,0 +1,249 @@
"""Bluesky search via AT Protocol (requires app password).
Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
"""
import math
import re
import sys
import time
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from . import http, log
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
# Module-level token cache (valid for the lifetime of a single research run)
_cached_token: Optional[str] = None
_token_created_at: float = 0.0
_session_error: Optional[str] = None
_TOKEN_MAX_AGE_SECONDS = 5400 # 90 minutes (conservative, tokens last ~2 hours)
def _log(msg: str):
log.source_log("Bluesky", msg)
def _create_session(handle: str, app_password: str) -> Optional[str]:
"""Create an AT Protocol session and return the access token.
Args:
handle: Bluesky handle (e.g. user.bsky.social)
app_password: App password from bsky.app/settings/app-passwords
Returns:
Access JWT string, or None on failure. Sets _session_error on failure.
"""
global _cached_token, _token_created_at, _session_error
if _cached_token and (time.monotonic() - _token_created_at < _TOKEN_MAX_AGE_SECONDS):
return _cached_token
if _cached_token:
_log("Session token expired, re-authenticating")
_cached_token = None
_token_created_at = 0.0
try:
response = http.request(
"POST",
BSKY_SESSION_URL,
json_data={"identifier": handle, "password": app_password},
timeout=15,
)
token = response.get("accessJwt")
if token:
_cached_token = token
_token_created_at = time.monotonic()
_session_error = None
_log("Session created successfully")
return token
_log("No accessJwt in session response")
_session_error = "No accessJwt in session response"
return None
except http.HTTPError as e:
if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
_session_error = "Cloudflare blocked the request (403 Forbidden). This is a network-level block, not an auth issue. Try a different network or VPN."
elif e.status_code == 401:
_session_error = "Invalid credentials (401 Unauthorized). Check BSKY_HANDLE and BSKY_APP_PASSWORD."
else:
_session_error = f"Session request failed: {e}"
_log(f"Session creation failed: {_session_error}")
return None
except Exception as e:
_session_error = f"Session request failed: {type(e).__name__}: {e}"
_log(f"Session creation failed: {_session_error}")
return None
def _reset_session_cache() -> None:
global _cached_token, _token_created_at, _session_error
_cached_token = None
_token_created_at = 0.0
_session_error = None
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Bluesky search."""
from .query import extract_core_subject
_BSKY_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features', 'recommendations', 'advice',
})
return extract_core_subject(topic, noise=_BSKY_NOISE)
def _parse_date(item: Dict[str, Any]) -> Optional[str]:
"""Parse date from Bluesky post to YYYY-MM-DD.
AT Protocol uses ISO 8601 format in indexedAt and createdAt fields.
"""
for key in ("indexedAt", "createdAt"):
val = item.get(key)
if val and isinstance(val, str):
try:
dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
return None
def search_bluesky(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
"""Search Bluesky via AT Protocol API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
config: Config dict with BSKY_HANDLE and BSKY_APP_PASSWORD
Returns:
Dict with 'posts' list from AT Protocol response.
"""
config = config or {}
handle = config.get("BSKY_HANDLE", "")
app_password = config.get("BSKY_APP_PASSWORD", "")
if not handle or not app_password:
return {"posts": [], "error": "Bluesky credentials not configured"}
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching for '{core_topic}' (depth={depth}, limit={count})")
from urllib.parse import urlencode
params = {
"q": core_topic,
"limit": str(min(count, 100)),
"sort": "top",
}
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
token = _create_session(handle, app_password)
if not token:
error_msg = _session_error or "Bluesky session creation failed (unknown error)"
return None, error_msg
try:
response = http.request(
"GET", url,
headers={"Authorization": f"Bearer {token}"},
timeout=30,
)
return response, None
except http.HTTPError as e:
_log(f"Search failed: {e}")
if e.status_code == 401:
_reset_session_cache()
return None, "refresh"
if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
return None, "Bluesky search blocked by Cloudflare (403). This is a network-level block - try a different network or VPN."
return None, f"Bluesky search failed: {e}"
except Exception as e:
_log(f"Search failed: {e}")
return None, f"Bluesky search failed: {type(e).__name__}: {e}"
response, error_msg = _auth_and_search()
if error_msg == "refresh":
_log("Session expired; recreating token and retrying once")
response, error_msg = _auth_and_search()
if error_msg:
return {"posts": [], "error": error_msg}
if response is None:
return {"posts": [], "error": "Bluesky search failed (unknown error)"}
posts = response.get("posts", [])
_log(f"Found {len(posts)} posts")
return response
def parse_bluesky_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse AT Protocol response into normalized item dicts.
Returns:
List of item dicts ready for normalization.
"""
posts = response.get("posts", [])
items = []
for i, post in enumerate(posts):
record = post.get("record") or {}
text = record.get("text") or ""
author = post.get("author") or {}
handle = author.get("handle") or ""
display_name = author.get("displayName") or handle
# Post URI -> URL
# URI format: at://did:plc:xxx/app.bsky.feed.post/rkey
uri = post.get("uri") or ""
rkey = uri.rsplit("/", 1)[-1] if uri else ""
url = f"https://bsky.app/profile/{handle}/post/{rkey}" if handle and rkey else ""
likes = post.get("likeCount") or 0
reposts = post.get("repostCount") or 0
replies = post.get("replyCount") or 0
quotes = post.get("quoteCount") or 0
date_str = _parse_date(post) or _parse_date(record)
# Relevance: position-based (AT Protocol sorts by relevance with sort=top)
rank_score = max(0.3, 1.0 - (i * 0.02))
engagement_boost = min(0.2, math.log1p(likes + reposts) / 40)
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"handle": handle,
"display_name": display_name,
"text": text,
"url": url,
"date": date_str,
"engagement": {
"likes": likes,
"reposts": reposts,
"replies": replies,
"quotes": quotes,
},
"relevance": round(relevance, 2),
"why_relevant": f"Bluesky: @{handle}: {text[:60]}" if text else f"Bluesky: {handle}",
})
return items
-152
View File
@@ -1,152 +0,0 @@
"""Caching utilities for last30days skill."""
import hashlib
import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional
CACHE_DIR = Path.home() / ".cache" / "last30days"
DEFAULT_TTL_HOURS = 24
MODEL_CACHE_TTL_DAYS = 7
def ensure_cache_dir():
"""Ensure cache directory exists."""
CACHE_DIR.mkdir(parents=True, exist_ok=True)
def get_cache_key(topic: str, from_date: str, to_date: str, sources: str) -> str:
"""Generate a cache key from query parameters."""
key_data = f"{topic}|{from_date}|{to_date}|{sources}"
return hashlib.sha256(key_data.encode()).hexdigest()[:16]
def get_cache_path(cache_key: str) -> Path:
"""Get path to cache file."""
return CACHE_DIR / f"{cache_key}.json"
def is_cache_valid(cache_path: Path, ttl_hours: int = DEFAULT_TTL_HOURS) -> bool:
"""Check if cache file exists and is within TTL."""
if not cache_path.exists():
return False
try:
stat = cache_path.stat()
mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
now = datetime.now(timezone.utc)
age_hours = (now - mtime).total_seconds() / 3600
return age_hours < ttl_hours
except OSError:
return False
def load_cache(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> Optional[dict]:
"""Load data from cache if valid."""
cache_path = get_cache_path(cache_key)
if not is_cache_valid(cache_path, ttl_hours):
return None
try:
with open(cache_path, 'r') as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return None
def get_cache_age_hours(cache_path: Path) -> Optional[float]:
"""Get age of cache file in hours."""
if not cache_path.exists():
return None
try:
stat = cache_path.stat()
mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
now = datetime.now(timezone.utc)
return (now - mtime).total_seconds() / 3600
except OSError:
return None
def load_cache_with_age(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> tuple:
"""Load data from cache with age info.
Returns:
Tuple of (data, age_hours) or (None, None) if invalid
"""
cache_path = get_cache_path(cache_key)
if not is_cache_valid(cache_path, ttl_hours):
return None, None
age = get_cache_age_hours(cache_path)
try:
with open(cache_path, 'r') as f:
return json.load(f), age
except (json.JSONDecodeError, OSError):
return None, None
def save_cache(cache_key: str, data: dict):
"""Save data to cache."""
ensure_cache_dir()
cache_path = get_cache_path(cache_key)
try:
with open(cache_path, 'w') as f:
json.dump(data, f)
except OSError:
pass # Silently fail on cache write errors
def clear_cache():
"""Clear all cache files."""
if CACHE_DIR.exists():
for f in CACHE_DIR.glob("*.json"):
try:
f.unlink()
except OSError:
pass
# Model selection cache (longer TTL)
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
def load_model_cache() -> dict:
"""Load model selection cache."""
if not is_cache_valid(MODEL_CACHE_FILE, MODEL_CACHE_TTL_DAYS * 24):
return {}
try:
with open(MODEL_CACHE_FILE, 'r') as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return {}
def save_model_cache(data: dict):
"""Save model selection cache."""
ensure_cache_dir()
try:
with open(MODEL_CACHE_FILE, 'w') as f:
json.dump(data, f)
except OSError:
pass
def get_cached_model(provider: str) -> Optional[str]:
"""Get cached model selection for a provider."""
cache = load_model_cache()
return cache.get(provider)
def set_cached_model(provider: str, model: str):
"""Cache model selection for a provider."""
cache = load_model_cache()
cache[provider] = model
cache['updated_at'] = datetime.now(timezone.utc).isoformat()
save_model_cache(cache)
+283
View File
@@ -0,0 +1,283 @@
"""Category-peer subreddit map for Step 0.55 community resolution.
When a topic is a product in a known category (AI image generation, AI coding
agents, SaaS screen recording, etc.), brand-specific subreddits returned by
WebSearch are insufficient: cross-product technique discussion lives in
category-peer subs. This module classifies a topic into a category by matching
compound-term patterns against the lowercased topic string, then returns the
priority-ordered peer subreddit list for that category.
The map is intentionally small, curated, and code-reviewed. Adding a new
category is a code change; there is no user-editable override surface.
False-positive guard: every pattern is either a multi-word compound (e.g.
"image generation", "text to image") or a domain-specific single word
(e.g. "midjourney", "stablediffusion"). Bare common nouns like "image",
"ai", or "model" are never used as patterns.
First-match-wins: categories are evaluated in declared order. Entries are
sorted from most-specific to least-specific so narrower categories claim a
topic before broader ones. For example, `ai_image_generation` appears
before `ai_chat_model` so "gpt image 2" matches the image-gen category.
"""
from __future__ import annotations
from typing import List, Optional, TypedDict
class _CategoryEntry(TypedDict):
patterns: List[str]
peer_subs: List[str]
CATEGORY_PEERS: dict[str, _CategoryEntry] = {
"ai_image_generation": {
"patterns": [
"image generation",
"image gen",
"text to image",
"text-to-image",
"gpt image",
"gpt-image",
"nano banana",
"midjourney",
"stable diffusion",
"stablediffusion",
"dall-e",
"dalle",
"flux.1",
"flux schnell",
"imagen",
"seedance",
"ideogram",
"recraft",
],
"peer_subs": [
"StableDiffusion",
"midjourney",
"dalle2",
"aiArt",
"PromptEngineering",
"MediaSynthesis",
],
},
"ai_video_generation": {
"patterns": [
"video generation",
"text to video",
"text-to-video",
"sora",
"veo 3",
"veo3",
"runway gen",
"kling",
"pika labs",
"luma dream machine",
"hailuo",
],
"peer_subs": [
"aivideo",
"StableDiffusion",
"runwayml",
"singularity",
"MediaSynthesis",
],
},
"ai_music_generation": {
"patterns": [
"music generation",
"ai music",
"suno",
"udio",
"riffusion",
"stable audio",
],
"peer_subs": [
"SunoAI",
"udiomusic",
"aimusic",
"artificial",
],
},
"ai_coding_agent": {
"patterns": [
"claude code",
"cursor ide",
"github copilot",
"windsurf",
"aider",
"cline",
"openclaw",
"hermes agent",
"continue.dev",
"codeium",
"sweep ai",
"devin ai",
"coding agent",
"coding assistant",
],
"peer_subs": [
"ChatGPTCoding",
"LocalLLaMA",
"singularity",
"PromptEngineering",
],
},
"ai_agent_framework": {
"patterns": [
"agent framework",
"agentic framework",
"langchain",
"langgraph",
"crewai",
"autogen",
"llamaindex",
"dspy",
"smolagents",
],
"peer_subs": [
"LangChain",
"LocalLLaMA",
"AI_Agents",
"MachineLearning",
],
},
"ai_chat_model": {
"patterns": [
"gpt-5",
"gpt-4",
"claude opus",
"claude sonnet",
"claude haiku",
"gemini pro",
"gemini flash",
"llama 3",
"llama 4",
"deepseek",
"qwen",
"mistral large",
"grok",
],
"peer_subs": [
"LocalLLaMA",
"ChatGPT",
"ClaudeAI",
"singularity",
"artificial",
],
},
"saas_screen_recording": {
"patterns": [
"screen recording",
"screen recorder",
"loom video",
"tella screen",
"vidyard",
"screen capture tool",
],
"peer_subs": [
"SaaS",
"screenrecording",
"productivity",
"Entrepreneur",
],
},
"saas_productivity": {
"patterns": [
"notion app",
"obsidian plugin",
"obsidian app",
"linear app",
"asana",
"clickup",
"productivity app",
],
"peer_subs": [
"productivity",
"SaaS",
"ObsidianMD",
"Notion",
],
},
"prediction_markets": {
"patterns": [
"polymarket",
"kalshi",
"prediction market",
"event contracts",
"manifold markets",
],
"peer_subs": [
"Polymarket",
"Kalshi",
"predictionmarkets",
],
},
"crypto_defi": {
"patterns": [
"defi protocol",
"yield farming",
"liquidity pool",
"stablecoin",
"ethereum layer",
"layer 2",
"l2 rollup",
],
"peer_subs": [
"defi",
"ethfinance",
"CryptoCurrency",
"ethereum",
],
},
"dev_tool_cli": {
"patterns": [
"cli tool",
"command line tool",
"terminal app",
"dev tool",
],
"peer_subs": [
"commandline",
"programming",
"webdev",
],
},
}
def detect_category(topic: Optional[str]) -> Optional[str]:
"""Classify a topic into a known category by compound-term match.
Returns the category id (e.g. "ai_image_generation") or None if no
category's patterns match. Matching is case-insensitive substring over
the lowercased topic. Declaration order wins (first-match-wins), so the
map is ordered from most-specific to least-specific.
A None or empty topic returns None. Classification never raises on
normal string inputs; callers do not need to wrap in try/except for
typical paths, though defensive callers may.
"""
if not topic:
return None
lowered = topic.lower()
for category_id, entry in CATEGORY_PEERS.items():
for pattern in entry["patterns"]:
if pattern in lowered:
return category_id
return None
def peer_subs_for(category_id: Optional[str]) -> List[str]:
"""Return the priority-ordered peer subreddit list for a category.
Returns an empty list for None or unknown category ids. The returned
list is a fresh copy; callers may safely mutate it.
"""
if not category_id:
return []
entry = CATEGORY_PEERS.get(category_id)
if not entry:
return []
return list(entry["peer_subs"])
+265
View File
@@ -0,0 +1,265 @@
"""Chrome cookie extraction for macOS.
Extracts cookies from Chrome's encrypted SQLite database using only stdlib
modules and the system openssl CLI (ships with macOS). Zero pip dependencies.
Chrome on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
This is NOT affected by Windows App-Bound Encryption (v20).
"""
import hashlib
import logging
import shutil
import sqlite3
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
# Chrome cookie DB location on macOS
CHROME_COOKIES_DB = Path.home() / "Library" / "Application Support" / "Google" / "Chrome" / "Default" / "Cookies"
# Chrome v10 encryption constants
CHROME_SALT = b"saltysalt"
CHROME_PBKDF2_ITERATIONS = 1003
CHROME_KEY_LENGTH = 16
# IV is 16 space characters (0x20)
CHROME_IV_HEX = "20" * 16
def _get_chrome_encryption_key() -> Optional[bytes]:
"""Retrieve Chrome's encryption passphrase from macOS Keychain.
Calls `security find-generic-password` which may trigger a system dialog
on first access.
Returns the raw passphrase bytes, or None on failure.
"""
try:
result = subprocess.run(
["security", "find-generic-password", "-w", "-s", "Chrome Safe Storage"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
logger.info("Chrome Keychain access denied or Chrome not installed: %s", result.stderr.strip())
return None
passphrase = result.stdout.strip()
if not passphrase:
logger.info("Chrome Keychain returned empty passphrase")
return None
return passphrase.encode("utf-8")
except FileNotFoundError:
logger.info("'security' command not found — not on macOS?")
return None
except subprocess.TimeoutExpired:
logger.info("Chrome Keychain access timed out")
return None
except Exception as e:
logger.info("Failed to get Chrome encryption key: %s", e)
return None
def _derive_aes_key(passphrase: bytes) -> bytes:
"""Derive 16-byte AES key from Chrome's Keychain passphrase via PBKDF2."""
return hashlib.pbkdf2_hmac(
"sha1",
passphrase,
CHROME_SALT,
CHROME_PBKDF2_ITERATIONS,
dklen=CHROME_KEY_LENGTH,
)
def _decrypt_v10_value(encrypted_value: bytes, aes_key: bytes, db_version: int) -> Optional[str]:
"""Decrypt a Chrome v10-encrypted cookie value.
Uses system openssl CLI for AES-128-CBC decryption (zero pip deps).
For Chrome 130+ (db_version >= 24), strips 32-byte SHA-256 prefix after decryption.
Returns decrypted string or None on failure.
"""
# Strip the 'v10' prefix
ciphertext = encrypted_value[3:]
if not ciphertext:
return None
hex_key = aes_key.hex()
try:
result = subprocess.run(
[
"openssl", "enc", "-aes-128-cbc", "-d",
"-K", hex_key,
"-iv", CHROME_IV_HEX,
"-nopad",
],
input=ciphertext,
capture_output=True,
timeout=5,
)
if result.returncode != 0:
logger.debug("openssl decryption failed: %s", result.stderr.decode(errors="replace").strip())
return None
decrypted = result.stdout
if not decrypted:
return None
# Remove PKCS7 padding
decrypted = _remove_pkcs7_padding(decrypted)
if decrypted is None:
return None
# Chrome 130+ (db version >= 24): strip 32-byte SHA-256 prefix
if db_version >= 24 and len(decrypted) > 32:
decrypted = decrypted[32:]
return decrypted.decode("utf-8", errors="replace")
except FileNotFoundError:
logger.info("openssl not found — cannot decrypt Chrome cookies")
return None
except subprocess.TimeoutExpired:
logger.info("openssl decryption timed out")
return None
except Exception as e:
logger.debug("Chrome cookie decryption error: %s", e)
return None
def _remove_pkcs7_padding(data: bytes) -> Optional[bytes]:
"""Remove PKCS7 padding from decrypted data.
The last byte indicates the number of padding bytes added.
All padding bytes must have the same value.
Returns unpadded data or None if padding is invalid.
"""
if not data:
return None
pad_len = data[-1]
if pad_len < 1 or pad_len > 16:
return None
# Verify all padding bytes match
if data[-pad_len:] != bytes([pad_len]) * pad_len:
return None
return data[:-pad_len]
def _get_db_version(cursor: sqlite3.Cursor) -> int:
"""Get Chrome cookie database version from the meta table.
Returns 0 if meta table doesn't exist or version can't be read.
"""
try:
cursor.execute("SELECT value FROM meta WHERE key = 'version'")
row = cursor.fetchone()
if row:
return int(row[0])
except Exception:
pass
return 0
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS.
Copies the locked Cookies database to a temp file, reads specified cookies,
and decrypts v10-encrypted values using the Keychain-stored key.
Args:
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com")
cookie_names: List of cookie names to extract
Returns:
Dict mapping cookie name to decrypted value, or None on failure.
Only includes cookies that were successfully found and decrypted.
"""
if not CHROME_COOKIES_DB.exists():
logger.info("Chrome cookies database not found at %s", CHROME_COOKIES_DB)
return None
# Get encryption key from Keychain
passphrase = _get_chrome_encryption_key()
aes_key = _derive_aes_key(passphrase) if passphrase else None
# Copy DB to temp file (Chrome locks the original)
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(CHROME_COOKIES_DB), tmp_path)
except Exception as e:
logger.info("Failed to copy Chrome cookies database: %s", e)
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
return None
finally:
if tmp_fd is not None:
import os
os.close(tmp_fd)
try:
conn = sqlite3.connect(tmp_path)
cursor = conn.cursor()
db_version = _get_db_version(cursor)
logger.debug("Chrome cookie DB version: %d", db_version)
# Build query with placeholders for cookie names
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value, encrypted_value FROM cookies "
f"WHERE host_key LIKE ? AND name IN ({placeholders})"
)
# Use LIKE for domain matching (e.g., %.twitter.com matches .twitter.com)
params = [f"%{domain}"] + list(cookie_names)
cursor.execute(query, params)
results: dict[str, str] = {}
for name, value, encrypted_value in cursor.fetchall():
# Prefer unencrypted value if present
if value:
results[name] = value
continue
# Handle encrypted value
if encrypted_value and encrypted_value[:3] == b"v10":
if aes_key is None:
logger.debug("Skipping encrypted cookie %s — no Keychain access", name)
continue
decrypted = _decrypt_v10_value(encrypted_value, aes_key, db_version)
if decrypted:
results[name] = decrypted
else:
logger.debug("Failed to decrypt cookie %s", name)
elif encrypted_value:
# Unknown encryption version
logger.debug("Unknown encryption for cookie %s (prefix: %r)", name, encrypted_value[:3])
conn.close()
if not results:
logger.info("No matching cookies found in Chrome for domain %s", domain)
return None
return results
except sqlite3.Error as e:
logger.info("Failed to read Chrome cookies database: %s", e)
return None
except Exception as e:
logger.info("Unexpected error reading Chrome cookies: %s", e)
return None
finally:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
+271
View File
@@ -0,0 +1,271 @@
"""Candidate clustering and representative selection."""
from __future__ import annotations
import re
from . import dedupe, schema
CLUSTERABLE_INTENTS = {"breaking_news", "opinion", "comparison", "prediction"}
# Words too common to signal shared topic between clusters.
_ENTITY_STOPWORDS = frozenset({
"the", "a", "an", "to", "for", "how", "is", "in", "of", "on", "and",
"with", "from", "by", "at", "this", "that", "it", "what", "are", "do",
"can", "his", "her", "he", "she", "its", "was", "has", "new", "just",
"says", "said", "will", "about", "after", "now", "all", "been", "here",
"not", "out", "up", "more", "also", "but", "who", "year", "first",
"make", "being", "making", "over", "into", "than", "they", "their",
"would", "could", "get", "got", "some", "like", "back", "going",
"breaking", "https", "http", "www", "com",
})
def _candidate_text(candidate: schema.Candidate) -> str:
return " ".join(part for part in [candidate.title, candidate.snippet] if part).strip()
def _extract_entities(text: str) -> set[str]:
"""Extract significant words (proper nouns, numbers, capitalized words) from text.
Used for cross-source cluster merging where phrasing differs but entities overlap.
"""
# Normalize but preserve word boundaries
words = re.sub(r"[^\w\s]", " ", text).split()
entities = set()
for word in words:
lower = word.lower()
if lower in _ENTITY_STOPWORDS or len(word) <= 2:
continue
# Keep words that are: capitalized, ALL CAPS, contain digits, or 4+ chars
if word[0].isupper() or word.isupper() or any(c.isdigit() for c in word) or len(word) >= 4:
entities.add(lower)
return entities
def _entity_overlap(entities_a: set[str], entities_b: set[str]) -> float:
"""Jaccard-style overlap on extracted entities."""
if not entities_a or not entities_b:
return 0.0
intersection = entities_a & entities_b
smaller = min(len(entities_a), len(entities_b))
# Use overlap coefficient (intersection / min) instead of Jaccard,
# because a short tweet about the same event as a long Reddit post
# will have fewer total entities but high overlap with the larger set.
return len(intersection) / smaller if smaller > 0 else 0.0
def _mmr_representatives(
candidates: list[schema.Candidate],
text_cache: dict[str, dedupe._PreparedText],
limit: int = 3,
diversity_lambda: float = 0.75,
) -> list[str]:
selected: list[schema.Candidate] = []
remaining_set = {c.candidate_id for c in candidates}
remaining = list(candidates)
while remaining and len(selected) < limit:
if not selected:
best = max(remaining, key=lambda candidate: candidate.final_score)
selected.append(best)
remaining_set.discard(best.candidate_id)
remaining = [c for c in remaining if c.candidate_id in remaining_set]
continue
selected_preps = [text_cache[c.candidate_id] for c in selected]
def score(candidate: schema.Candidate) -> float:
prep = text_cache[candidate.candidate_id]
diversity_penalty = max(
dedupe.prepared_similarity(prep, sp) for sp in selected_preps
)
return (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)
best = max(remaining, key=score)
selected.append(best)
remaining_set.discard(best.candidate_id)
remaining = [c for c in remaining if c.candidate_id in remaining_set]
return [candidate.candidate_id for candidate in selected]
def cluster_candidates(
candidates: list[schema.Candidate],
plan: schema.QueryPlan,
) -> list[schema.Cluster]:
"""Greedy clustering around high-ranked leaders."""
if plan.intent not in CLUSTERABLE_INTENTS or plan.cluster_mode == "none":
clusters = []
for index, candidate in enumerate(candidates, start=1):
cluster_id = f"cluster-{index}"
candidate.cluster_id = cluster_id
clusters.append(
schema.Cluster(
cluster_id=cluster_id,
title=candidate.title,
candidate_ids=[candidate.candidate_id],
representative_ids=[candidate.candidate_id],
sources=sorted(schema.candidate_sources(candidate)),
score=candidate.final_score,
uncertainty=None,
)
)
return clusters
text_cache: dict[str, dedupe._PreparedText] = {
c.candidate_id: dedupe._PreparedText(_candidate_text(c))
for c in candidates
}
groups: list[list[schema.Candidate]] = []
# Lower threshold for breaking_news: related articles share fewer exact
# words but cover the same event.
threshold = 0.42 if plan.intent == "breaking_news" else 0.48
for candidate in candidates:
assigned = False
cand_prep = text_cache[candidate.candidate_id]
for group in groups:
leader = group[0]
similarity = dedupe.prepared_similarity(cand_prep, text_cache[leader.candidate_id])
if similarity >= threshold:
group.append(candidate)
assigned = True
break
if not assigned:
groups.append([candidate])
clusters: list[schema.Cluster] = []
for index, group in enumerate(groups, start=1):
group.sort(key=lambda candidate: candidate.final_score, reverse=True)
cluster_id = f"cluster-{index}"
representatives = _mmr_representatives(group, text_cache)
for candidate in group:
candidate.cluster_id = cluster_id
clusters.append(
schema.Cluster(
cluster_id=cluster_id,
title=group[0].title,
candidate_ids=[candidate.candidate_id for candidate in group],
representative_ids=representatives,
sources=sorted({source for candidate in group for source in schema.candidate_sources(candidate)}),
score=max(candidate.final_score for candidate in group),
uncertainty=_cluster_uncertainty(group),
)
)
# Second pass: merge small clusters that share entities across sources.
clusters = _merge_entity_clusters(clusters, candidates)
return sorted(clusters, key=lambda cluster: cluster.score, reverse=True)
def _merge_entity_clusters(
clusters: list[schema.Cluster],
all_candidates: list[schema.Candidate],
) -> list[schema.Cluster]:
"""Merge small clusters that cover the same story across different sources.
The initial greedy pass uses text similarity which misses cross-source
matches where phrasing differs. This second pass looks at entity overlap
(proper nouns, names, numbers) to catch cases like:
- Reddit: "Kanye West to headline all three nights of Wireless Festival 2026"
- X: "BREAKING: Kanye West (Ye) is making his massive UK comeback!"
"""
if len(clusters) < 2:
return clusters
candidate_map = {c.candidate_id: c for c in all_candidates}
# Build entity sets per cluster
cluster_entities: list[set[str]] = []
for cl in clusters:
entities: set[str] = set()
for cid in cl.candidate_ids:
cand = candidate_map.get(cid)
if cand:
entities |= _extract_entities(_candidate_text(cand))
cluster_entities.append(entities)
# Only merge clusters with <= 3 items (don't merge already-large clusters)
merged_into: dict[int, int] = {} # index -> merge target index
for i in range(len(clusters)):
if i in merged_into or len(clusters[i].candidate_ids) > 3:
continue
for j in range(i + 1, len(clusters)):
if j in merged_into or len(clusters[j].candidate_ids) > 3:
continue
# Require different sources to merge (same-source should already be grouped)
sources_i = set(clusters[i].sources)
sources_j = set(clusters[j].sources)
if sources_i == sources_j and len(sources_i) == 1:
continue
# Prevent Polymarket clusters from merging with non-Polymarket
# clusters. Prediction markets about "Sam Altman equity" should not
# merge into a news cluster about "Sam Altman rivalry" just because
# both mention the same entity.
poly_i = "polymarket" in sources_i
poly_j = "polymarket" in sources_j
if poly_i != poly_j:
continue
overlap = _entity_overlap(cluster_entities[i], cluster_entities[j])
if overlap >= 0.45:
merged_into[j] = i
if not merged_into:
return clusters
# Build merged cluster list
result: list[schema.Cluster] = []
for i, cl in enumerate(clusters):
if i in merged_into:
continue
# Collect all clusters merged into this one
merge_sources = [i] + [j for j, target in merged_into.items() if target == i]
if len(merge_sources) == 1:
result.append(cl)
continue
# Combine candidates from all merged clusters
combined_cids: list[str] = []
combined_sources: set[str] = set()
best_score = 0.0
for idx in merge_sources:
combined_cids.extend(clusters[idx].candidate_ids)
combined_sources.update(clusters[idx].sources)
best_score = max(best_score, clusters[idx].score)
# Pick representatives from combined pool
combined_candidates = [candidate_map[cid] for cid in combined_cids if cid in candidate_map]
combined_candidates.sort(key=lambda c: c.final_score, reverse=True)
merge_text_cache = {
c.candidate_id: dedupe._PreparedText(_candidate_text(c))
for c in combined_candidates
}
reps = _mmr_representatives(combined_candidates, merge_text_cache)
cluster_id = cl.cluster_id
for cid in combined_cids:
cand = candidate_map.get(cid)
if cand:
cand.cluster_id = cluster_id
result.append(schema.Cluster(
cluster_id=cluster_id,
title=combined_candidates[0].title if combined_candidates else cl.title,
candidate_ids=combined_cids,
representative_ids=reps,
sources=sorted(combined_sources),
score=best_score,
uncertainty=_cluster_uncertainty(combined_candidates),
))
return result
def _cluster_uncertainty(group: list[schema.Candidate]) -> str | None:
sources = {source for candidate in group for source in schema.candidate_sources(candidate)}
if len(sources) == 1:
return "single-source"
if max(candidate.final_score for candidate in group) < 55:
return "thin-evidence"
return None
+199
View File
@@ -0,0 +1,199 @@
"""Discover peer entities ("competitors") for a topic via web search.
Mirrors the `resolve.auto_resolve()` pattern: fan out 2-3 web searches via
`grounding.web_search()`, then extract capitalized entity candidates from
titles and snippets with deterministic text mining. No LLM call the
hosting reasoning model can always override discovery via
`--competitors-list`.
Returned list is ordered by score (frequency across queries) and capped to
the caller's requested count.
"""
from __future__ import annotations
import re
import sys
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from . import dates, grounding
from .resolve import _has_backend
# A "brand-shaped" token starts with uppercase OR is camelCase with an
# uppercase letter later. Catches "Anthropic", "OpenAI", "xAI", "iPhone",
# "eBay", "Hugging", "Face".
_BRAND_TOKEN = (
r"(?:[A-Z][A-Za-z0-9&.\-]*"
r"|[a-z][A-Za-z0-9&.\-]*[A-Z][A-Za-z0-9&.\-]*)"
)
# A capitalized phrase of 1-4 brand tokens separated by whitespace.
_CAPITALIZED_PHRASE = re.compile(
rf"\b{_BRAND_TOKEN}(?:\s+{_BRAND_TOKEN}){{0,3}}\b"
)
# Title-case fillers common in listicle SERPs. Kept flat — extraction
# rejects a candidate whose entire tokens are stopwords, not candidates
# that merely contain one.
_STOPWORD_TOKENS: frozenset[str] = frozenset(
token.lower()
for token in (
# Listicle fillers
"Top", "Best", "Worst", "Popular", "Leading", "Similar",
"Alternatives", "Alternative", "Competitor", "Competitors",
"vs", "Vs", "Versus", "Review", "Reviews", "Comparison",
"Guide", "List", "Lists", "Full", "Complete", "Free", "Paid",
"Tools", "Tool", "Options", "Rivals", "Rival", "Similar",
"Pick", "Picks", "Ranking", "Ranked", "Recommended",
# Grammar / time
"The", "A", "An", "Of", "In", "For", "To", "With", "On", "At",
"By", "From", "Is", "Are", "And", "Or", "But", "Than", "As",
"This", "That", "These", "Those", "Our", "Your", "Their",
"January", "February", "March", "April", "May", "June", "July",
"August", "September", "October", "November", "December",
# Years likely to appear as standalone tokens
*(str(year) for year in range(2018, 2031)),
# Miscellaneous SERP noise
"AI", "Apps", "App", "Software", "Platform", "Service", "Startups",
"Companies", "Company", "Products", "Product", "Brands", "Brand",
)
)
def _log(msg: str) -> None:
print(f"[Competitors] {msg}", file=sys.stderr)
def _topic_tokens(topic: str) -> set[str]:
"""Return lowercase alphanumeric tokens of the topic for filtering."""
return {tok for tok in re.findall(r"[A-Za-z0-9]+", topic.lower()) if tok}
def _candidate_ok(candidate: str, topic_tokens: set[str]) -> bool:
"""Filter a candidate phrase against stopwords and topic overlap."""
tokens = [t for t in re.findall(r"[A-Za-z0-9&.\-]+", candidate) if t]
if not tokens:
return False
# Reject candidates made entirely of stopwords (e.g., "Top Alternatives").
if all(tok.lower() in _STOPWORD_TOKENS for tok in tokens):
return False
# Reject candidates that overlap with the topic (e.g., topic="OpenAI"
# should not return "OpenAI Alternatives" or "OpenAI").
lower_tokens = {tok.lower() for tok in tokens}
if lower_tokens & topic_tokens:
return False
# Reject too-short one-letter tokens like "I" or single digits.
if len(tokens) == 1 and len(tokens[0]) < 2:
return False
return True
def _normalize_candidate(candidate: str) -> str:
"""Collapse whitespace and strip trailing punctuation."""
return re.sub(r"\s+", " ", candidate).strip(".,;:!?'\"()[] ")
def _extract_peer_entities(
items: list[dict], topic: str, limit: int,
) -> list[str]:
"""Score capitalized candidates across SERP items and return top `limit`.
Scoring is bag-of-phrases frequency across all items in the input. Ties
are broken by first-seen order so the output is deterministic.
"""
topic_tokens = _topic_tokens(topic)
counts: Counter[str] = Counter()
first_seen: dict[str, int] = {}
order = 0
# Group candidates into a frequency map keyed by lowercased normalized
# form so "xAI" and "xAI" count together regardless of case.
canonical: dict[str, str] = {}
for item in items:
text = f"{item.get('title', '')} {item.get('snippet', '')}"
for raw in _CAPITALIZED_PHRASE.findall(text):
candidate = _normalize_candidate(raw)
if not _candidate_ok(candidate, topic_tokens):
continue
key = candidate.lower()
if key not in canonical:
canonical[key] = candidate
first_seen[key] = order
order += 1
counts[key] += 1
ranked_keys = sorted(
counts.keys(),
key=lambda k: (-counts[k], first_seen[k]),
)
return [canonical[k] for k in ranked_keys[:limit]]
def _queries_for(topic: str) -> dict[str, str]:
return {
"competitors": f"{topic} competitors",
"alternatives": f"{topic} alternatives",
"vs": f"{topic} vs",
}
def discover_competitors(
topic: str,
count: int,
config: dict,
*,
lookback_days: int = 30,
) -> list[str]:
"""Discover `count` peer entities for `topic` via web search.
Args:
topic: The primary research topic.
count: Desired number of competitor entities (1..N).
config: Runtime config dict expects the same shape as the engine
config (BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / etc.).
lookback_days: Date range for freshness. Defaults to 30.
Returns:
A list of up to `count` entity names, deduped and ordered by score.
Empty list when no web backend is configured or every search fails
or returns zero usable candidates.
"""
if count < 1:
return []
if not _has_backend(config):
_log("No web search backend available, skipping competitor discovery")
return []
date_range = dates.get_date_range(lookback_days)
queries = _queries_for(topic)
collected: list[dict] = []
searches_run = 0
def _search(label: str, query: str) -> tuple[str, list[dict]]:
items, _artifact = grounding.web_search(query, date_range, config)
return label, items
with ThreadPoolExecutor(max_workers=len(queries)) as executor:
futures = {
executor.submit(_search, label, q): label
for label, q in queries.items()
}
for future in as_completed(futures):
label = futures[future]
try:
_label, items = future.result()
collected.extend(items)
searches_run += 1
except Exception as exc:
_log(f"Search failed for {label}: {exc}")
if not collected:
_log(f"No SERP results for {topic!r} across {searches_run}/{len(queries)} queries")
return []
entities = _extract_peer_entities(collected, topic, limit=count)
_log(
f"Discovered {len(entities)} competitor(s) for {topic!r} "
f"from {searches_run}/{len(queries)} queries: {entities}"
)
return entities
+379
View File
@@ -0,0 +1,379 @@
"""Browser cookie extraction for last30days.
Extracts cookies from local browser databases (Firefox, Chrome, Safari)
to enable zero-config authentication for services like X/Twitter.
Only uses Python stdlib no external dependencies.
"""
import configparser
import functools
import logging
import platform
import shutil
import sqlite3
import tempfile
from pathlib import Path
from typing import Dict, List, Optional
logger = logging.getLogger(__name__)
@functools.lru_cache(maxsize=1)
def _is_wsl() -> bool:
"""Detect if running under Windows Subsystem for Linux.
Cached after the first call since /proc/version doesn't change at runtime.
"""
try:
return "microsoft" in Path("/proc/version").read_text().lower()
except OSError:
return False
def _get_wsl_firefox_profiles_dir() -> Optional[Path]:
"""Find Firefox profiles directory on the Windows host from WSL.
Scans /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox for real user
directories (skips Public, Default, etc.).
"""
mnt_users = Path("/mnt/c/Users")
if not mnt_users.is_dir():
return None
skip = {"Public", "Default", "Default User", "All Users"}
try:
for user_dir in sorted(mnt_users.iterdir()):
if user_dir.name in skip or not user_dir.is_dir():
continue
ff_dir = user_dir / "AppData" / "Roaming" / "Mozilla" / "Firefox"
if ff_dir.is_dir():
return ff_dir
except OSError:
pass
return None
def _get_firefox_profiles_dir() -> Optional[Path]:
"""Return the Firefox profiles directory for the current platform, or None."""
system = platform.system()
if system == "Darwin":
path = Path.home() / "Library" / "Application Support" / "Firefox"
elif system == "Linux":
path = Path.home() / ".mozilla" / "firefox"
else:
# Windows: %APPDATA%\Mozilla\Firefox — best-effort
appdata = Path.home() / "AppData" / "Roaming" / "Mozilla" / "Firefox"
path = appdata
return path if path.is_dir() else None
def _find_default_profile(profiles_dir: Path) -> Optional[Path]:
"""Parse profiles.ini to find the default profile directory.
Looks for a section with Default=1. Falls back to the first profile
directory found on disk if profiles.ini is missing or malformed.
"""
ini_path = profiles_dir / "profiles.ini"
if ini_path.is_file():
try:
config = configparser.ConfigParser()
config.read(str(ini_path), encoding="utf-8")
# First pass: Install* section (Firefox >= 67 format, takes priority)
for section in config.sections():
if section.startswith("Install") and config.has_option(section, "Default"):
raw = config.get(section, "Default")
candidate = profiles_dir / raw
if candidate.is_dir():
return candidate
# Second pass: Profile section with Default=1
for section in config.sections():
if section.startswith("Profile") and config.has_option(section, "Default") and config.get(section, "Default") == "1":
return _resolve_profile_path(profiles_dir, config, section)
# Third pass: first Profile section that exists on disk
for section in config.sections():
if section.startswith("Profile"):
resolved = _resolve_profile_path(profiles_dir, config, section)
if resolved and resolved.is_dir():
return resolved
except (configparser.Error, OSError) as exc:
logger.debug("Failed to parse profiles.ini: %s", exc)
# Fallback: scan directory for anything that looks like a profile
return _fallback_find_profile(profiles_dir)
def _resolve_profile_path(
profiles_dir: Path, config: configparser.ConfigParser, section: str
) -> Optional[Path]:
"""Resolve a profile path from a ConfigParser section."""
if not config.has_option(section, "Path"):
return None
raw_path = config.get(section, "Path")
is_relative = config.has_option(section, "IsRelative") and config.get(section, "IsRelative") == "1"
if is_relative:
candidate = profiles_dir / raw_path
else:
candidate = Path(raw_path)
return candidate if candidate.is_dir() else None
def _fallback_find_profile(profiles_dir: Path) -> Optional[Path]:
"""Find the first directory that contains cookies.sqlite."""
try:
for child in sorted(profiles_dir.iterdir()):
if child.is_dir() and (child / "cookies.sqlite").is_file():
return child
except OSError:
pass
return None
def _query_cookies_db(
db_path: Path, domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Copy the cookies database to a temp file and query it.
Firefox locks cookies.sqlite while running, so we copy first.
Returns {name: value} dict or None if no matching cookies found.
"""
if not db_path.is_file():
return None
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(db_path), tmp_path)
conn = sqlite3.connect(tmp_path)
try:
# Build parameterized query — SQLite doesn't support array params,
# so we build the IN clause with individual placeholders.
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value FROM moz_cookies "
f"WHERE host LIKE ? AND name IN ({placeholders})"
)
# domain pattern: match .x.com, x.com, etc.
domain_pattern = f"%{domain}"
params = [domain_pattern] + list(cookie_names)
cursor = conn.execute(query, params)
rows = cursor.fetchall()
finally:
conn.close()
if not rows:
return None
return {name: value for name, value in rows}
except (sqlite3.Error, OSError) as exc:
logger.debug("Failed to query cookies database %s: %s", db_path, exc)
return None
finally:
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
except OSError:
pass
if tmp_fd is not None:
try:
import os
os.close(tmp_fd)
except OSError:
pass
def _try_firefox_dir(profiles_dir: Path, domain: str, cookie_names: List[str]) -> Optional[Dict[str, str]]:
"""Try to extract cookies from a Firefox profiles directory."""
profile_path = _find_default_profile(profiles_dir)
if profile_path is None:
logger.debug("No Firefox profile found in %s", profiles_dir)
return None
return _query_cookies_db(profile_path / "cookies.sqlite", domain, cookie_names)
def extract_firefox_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Firefox for the given domain and cookie names.
Finds the default Firefox profile, copies cookies.sqlite to a temp file
(to avoid lock conflicts), and queries for the requested cookies.
On WSL2, falls back to Windows Firefox if native Linux Firefox has no
matching cookies. Windows Firefox cookies are unencrypted, so this works
without DPAPI or any Windows-side helpers.
Args:
domain: The cookie domain to match (e.g. ".x.com"). Matched with LIKE %domain.
cookie_names: List of cookie names to extract (e.g. ["auth_token", "ct0"]).
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
profiles_dir = _get_firefox_profiles_dir()
if profiles_dir is not None:
result = _try_firefox_dir(profiles_dir, domain, cookie_names)
if result is not None:
return result
if platform.system() == "Linux" and _is_wsl():
wsl_dir = _get_wsl_firefox_profiles_dir()
if wsl_dir is not None:
logger.debug("Trying Windows Firefox via WSL: %s", wsl_dir)
return _try_firefox_dir(wsl_dir, domain, cookie_names)
if profiles_dir is None:
logger.debug("Firefox profiles directory not found")
return None
def extract_chrome_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Chrome for the given domain and cookie names.
macOS only uses Keychain + system openssl for AES-128-CBC decryption.
Linux/Windows not supported (Chrome uses platform-specific encryption).
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Chrome cookie extraction only supported on macOS")
return None
try:
from .chrome_cookies import extract_chrome_cookies_macos
return extract_chrome_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Chrome cookie extraction failed: %s", exc)
return None
def extract_safari_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Safari for the given domain and cookie names.
macOS only parses the unencrypted binary cookie file.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Safari cookie extraction only supported on macOS")
return None
try:
from .safari_cookies import extract_safari_cookies_macos
return extract_safari_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Safari cookie extraction failed: %s", exc)
return None
def extract_cookies(
browser: str, domain: str, cookie_names: list[str]
) -> Optional[dict[str, str]]:
"""Extract cookies from the specified browser.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
'auto' tries browsers in platform-appropriate order:
- macOS: Chrome -> Firefox -> Safari
- Linux: Firefox only
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
result = extract_cookies_with_source(browser, domain, cookie_names)
if result is None:
return None
cookies, _browser_name = result
return cookies
def _extract_firefox_with_source(
domain: str, cookie_names: List[str]
) -> Optional[tuple[Dict[str, str], str]]:
"""Extract Firefox cookies and report whether they came from native or WSL.
Returns (cookies, "firefox") for native Linux/macOS Firefox, or
(cookies, "firefox-wsl") for Windows Firefox accessed via WSL2.
"""
profiles_dir = _get_firefox_profiles_dir()
if profiles_dir is not None:
result = _try_firefox_dir(profiles_dir, domain, cookie_names)
if result is not None:
return (result, "firefox")
if platform.system() == "Linux" and _is_wsl():
wsl_dir = _get_wsl_firefox_profiles_dir()
if wsl_dir is not None:
logger.debug("Trying Windows Firefox via WSL: %s", wsl_dir)
result = _try_firefox_dir(wsl_dir, domain, cookie_names)
if result is not None:
return (result, "firefox-wsl")
return None
def extract_cookies_with_source(
browser: str, domain: str, cookie_names: list[str]
) -> Optional[tuple[dict[str, str], str]]:
"""Extract cookies and report which browser they came from.
Same as extract_cookies() but returns a (cookies, browser_name) tuple
so callers can track the source.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Tuple of ({cookie_name: cookie_value}, browser_name) or None.
browser_name is "firefox-wsl" when cookies came from Windows Firefox via WSL2.
"""
extractors = {
"firefox": extract_firefox_cookies,
"chrome": extract_chrome_cookies,
"safari": extract_safari_cookies,
}
if browser != "auto":
if browser == "firefox":
return _extract_firefox_with_source(domain, cookie_names)
extractor = extractors.get(browser)
if extractor is None:
logger.warning("Unknown browser: %s", browser)
return None
result = extractor(domain, cookie_names)
return (result, browser) if result is not None else None
# Auto mode: try browsers in platform-appropriate order
system = platform.system()
if system == "Darwin":
order = ["chrome", "firefox", "safari"]
elif system == "Linux":
order = ["firefox"]
else:
order = ["firefox"]
for name in order:
if name == "firefox":
result = _extract_firefox_with_source(domain, cookie_names)
if result is not None:
return result
else:
result = extractors[name](domain, cookie_names)
if result is not None:
return (result, name)
return None
+5 -9
View File
@@ -41,7 +41,10 @@ def parse_date(date_str: Optional[str]) -> Optional[datetime]:
for fmt in formats:
try:
return datetime.strptime(date_str, fmt).replace(tzinfo=timezone.utc)
dt = datetime.strptime(date_str, fmt)
if dt.tzinfo is not None:
return dt.astimezone(timezone.utc)
return dt.replace(tzinfo=timezone.utc)
except ValueError:
continue
@@ -78,14 +81,7 @@ def get_date_confidence(date_str: Optional[str], from_date: str, to_date: str) -
start = datetime.strptime(from_date, "%Y-%m-%d").date()
end = datetime.strptime(to_date, "%Y-%m-%d").date()
if start <= dt <= end:
return 'high'
elif dt < start:
# Older than range
return 'low'
else:
# Future date (suspicious)
return 'low'
return 'high' if start <= dt <= end else 'low'
except ValueError:
return 'low'
+103 -96
View File
@@ -1,120 +1,127 @@
"""Near-duplicate detection for last30days skill."""
"""Within-source near-duplicate detection."""
from __future__ import annotations
import re
from typing import List, Set, Tuple, Union
from . import schema
STOPWORDS = frozenset(
{
"the",
"a",
"an",
"to",
"for",
"how",
"is",
"in",
"of",
"on",
"and",
"with",
"from",
"by",
"at",
"this",
"that",
"it",
"what",
"are",
"do",
"can",
}
)
def normalize_text(text: str) -> str:
"""Normalize text for comparison.
- Lowercase
- Remove punctuation
- Collapse whitespace
"""
text = text.lower()
text = re.sub(r'[^\w\s]', ' ', text)
text = re.sub(r'\s+', ' ', text)
return text.strip()
text = re.sub(r"[^\w\s]", " ", text.lower())
return re.sub(r"\s+", " ", text).strip()
def get_ngrams(text: str, n: int = 3) -> Set[str]:
"""Get character n-grams from text."""
def get_ngrams(text: str, n: int = 3) -> set[str]:
text = normalize_text(text)
if len(text) < n:
return {text}
return {text[i:i+n] for i in range(len(text) - n + 1)}
return {text} if text else set()
return {text[index:index + n] for index in range(len(text) - n + 1)}
def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
"""Compute Jaccard similarity between two sets."""
if not set1 or not set2:
def jaccard_similarity(left: set[str], right: set[str]) -> float:
if not left or not right:
return 0.0
intersection = len(set1 & set2)
union = len(set1 | set2)
return intersection / union if union > 0 else 0.0
union = left | right
if not union:
return 0.0
return len(left & right) / len(union)
def get_item_text(item: Union[schema.RedditItem, schema.XItem]) -> str:
"""Get comparable text from an item."""
if isinstance(item, schema.RedditItem):
return item.title
else:
return item.text
def token_jaccard(text_a: str, text_b: str) -> float:
tokens_a = {
token
for token in normalize_text(text_a).split()
if len(token) > 1 and token not in STOPWORDS
}
tokens_b = {
token
for token in normalize_text(text_b).split()
if len(token) > 1 and token not in STOPWORDS
}
return jaccard_similarity(tokens_a, tokens_b)
def find_duplicates(
items: List[Union[schema.RedditItem, schema.XItem]],
threshold: float = 0.7,
) -> List[Tuple[int, int]]:
"""Find near-duplicate pairs in items.
Args:
items: List of items to check
threshold: Similarity threshold (0-1)
Returns:
List of (i, j) index pairs where i < j and items are similar
"""
duplicates = []
# Pre-compute n-grams
ngrams = [get_ngrams(get_item_text(item)) for item in items]
for i in range(len(items)):
for j in range(i + 1, len(items)):
similarity = jaccard_similarity(ngrams[i], ngrams[j])
if similarity >= threshold:
duplicates.append((i, j))
return duplicates
def hybrid_similarity(text_a: str, text_b: str) -> float:
return max(
jaccard_similarity(get_ngrams(text_a), get_ngrams(text_b)),
token_jaccard(text_a, text_b),
)
def dedupe_items(
items: List[Union[schema.RedditItem, schema.XItem]],
threshold: float = 0.7,
) -> List[Union[schema.RedditItem, schema.XItem]]:
"""Remove near-duplicates, keeping highest-scored item.
Args:
items: List of items (should be pre-sorted by score descending)
threshold: Similarity threshold
Returns:
Deduplicated items
"""
if len(items) <= 1:
return items
# Find duplicate pairs
dup_pairs = find_duplicates(items, threshold)
# Mark indices to remove (always remove the lower-scored one)
# Since items are pre-sorted by score, the second index is always lower
to_remove = set()
for i, j in dup_pairs:
# Keep the higher-scored one (lower index in sorted list)
if items[i].score >= items[j].score:
to_remove.add(j)
else:
to_remove.add(i)
# Return items not marked for removal
return [item for idx, item in enumerate(items) if idx not in to_remove]
def _tokenize(normalized: str) -> frozenset[str]:
return frozenset(
tok for tok in normalized.split()
if len(tok) > 1 and tok not in STOPWORDS
)
def dedupe_reddit(
items: List[schema.RedditItem],
threshold: float = 0.7,
) -> List[schema.RedditItem]:
"""Dedupe Reddit items."""
return dedupe_items(items, threshold)
class _PreparedText:
"""Pre-computed text representations for fast repeated similarity checks."""
__slots__ = ("ngrams", "tokens")
def __init__(self, raw: str) -> None:
norm = normalize_text(raw)
self.ngrams = get_ngrams(norm) if norm else set()
self.tokens = _tokenize(norm)
def dedupe_x(
items: List[schema.XItem],
threshold: float = 0.7,
) -> List[schema.XItem]:
"""Dedupe X items."""
return dedupe_items(items, threshold)
def prepared_similarity(a: _PreparedText, b: _PreparedText) -> float:
return max(
jaccard_similarity(a.ngrams, b.ngrams),
jaccard_similarity(a.tokens, b.tokens),
)
def item_text(item: schema.SourceItem) -> str:
parts = [item.title, item.body, item.author or "", item.container or ""]
return " ".join(part for part in parts if part).strip()
def dedupe_items(items: list[schema.SourceItem], threshold: float = 0.7) -> list[schema.SourceItem]:
"""Remove near-duplicates while keeping earlier, better-scored items."""
kept: list[schema.SourceItem] = []
kept_prepared: list[_PreparedText] = []
for item in items:
text = item_text(item)
if not text:
kept.append(item)
continue
prep = _PreparedText(text)
is_duplicate = False
for existing_prep in kept_prepared:
if prepared_similarity(prep, existing_prep) >= threshold:
is_duplicate = True
break
if not is_duplicate:
kept.append(item)
kept_prepared.append(prep)
return kept
+127
View File
@@ -0,0 +1,127 @@
"""Entity extraction from initial search results for supplemental searches."""
import re
from collections import Counter
from typing import Any, Dict, List
# Handles that appear too frequently to be useful for targeted search.
# These are generic/platform accounts, not topic-specific voices.
GENERIC_HANDLES = {
"elonmusk", "openai", "google", "microsoft", "apple", "meta",
"github", "youtube", "x", "twitter", "reddit", "wikipedia",
"nytimes", "washingtonpost", "cnn", "bbc", "reuters",
"verified", "jack", "sundarpichai",
}
def extract_entities(
reddit_items: List[Dict[str, Any]],
x_items: List[Dict[str, Any]],
max_handles: int = 5,
max_hashtags: int = 3,
max_subreddits: int = 5,
) -> Dict[str, List[str]]:
"""Extract key entities from Phase 1 results for supplemental searches.
Parses X results for @handles and #hashtags, Reddit results for subreddit
names and cross-referenced communities.
Args:
reddit_items: Raw Reddit item dicts from Phase 1
x_items: Raw X item dicts from Phase 1
max_handles: Maximum handles to return
max_hashtags: Maximum hashtags to return
max_subreddits: Maximum subreddits to return
Returns:
Dict with keys: x_handles, x_hashtags, reddit_subreddits
"""
handles = _extract_x_handles(x_items)
hashtags = _extract_x_hashtags(x_items)
subreddits = _extract_subreddits(reddit_items)
return {
"x_handles": handles[:max_handles],
"x_hashtags": hashtags[:max_hashtags],
"reddit_subreddits": subreddits[:max_subreddits],
}
def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank @handles from X results.
Sources handles from:
1. author_handle field (who posted)
2. @mentions in post text (who they're talking about/to)
Returns handles ranked by frequency, filtered for generic accounts.
"""
handle_counts = Counter()
for item in x_items:
# Author handle
author = item.get("author_handle", "").strip().lstrip("@").lower()
if author and author not in GENERIC_HANDLES:
handle_counts[author] += 1
# @mentions in text
text = item.get("text", "")
mentions = re.findall(r'@(\w{1,15})', text)
for mention in mentions:
mention_lower = mention.lower()
if mention_lower not in GENERIC_HANDLES:
handle_counts[mention_lower] += 1
# Return all handles ranked by frequency
return [h for h, _ in handle_counts.most_common()]
def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank #hashtags from X results.
Returns hashtags ranked by frequency.
"""
hashtag_counts = Counter()
for item in x_items:
text = item.get("text", "")
tags = re.findall(r'#(\w{2,30})', text)
for tag in tags:
hashtag_counts[tag.lower()] += 1
# Return all hashtags ranked by frequency
return [f"#{t}" for t, _ in hashtag_counts.most_common()]
def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank subreddits from Reddit results.
Sources from:
1. subreddit field on each result
2. Cross-references in comment text (e.g., "check out r/localLLaMA")
Returns subreddits ranked by frequency.
"""
sub_counts = Counter()
for item in reddit_items:
# Primary subreddit
sub = item.get("subreddit", "").strip().lstrip("r/")
if sub:
sub_counts[sub] += 1
# Cross-references in comment insights
for insight in item.get("comment_insights", []):
cross_refs = re.findall(r'r/(\w{2,30})', insight)
for ref in cross_refs:
sub_counts[ref] += 1
# Cross-references in top comments
for comment in item.get("top_comments", []):
excerpt = comment.get("excerpt", "")
cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
for ref in cross_refs:
sub_counts[ref] += 1
# Return subreddits ranked by frequency
return [sub for sub, _ in sub_counts.most_common()]
+605 -78
View File
@@ -1,18 +1,78 @@
"""Environment and API key management for last30days skill."""
from __future__ import annotations
import base64
import binascii
import json
import os
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, Dict, Any
from typing import Any, Literal
CONFIG_DIR = Path.home() / ".config" / "last30days"
CONFIG_FILE = CONFIG_DIR / ".env"
# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
# Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
_config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
if _config_override == "":
# Empty string = no config file (clean mode)
CONFIG_DIR = None
CONFIG_FILE = None
elif _config_override:
CONFIG_DIR = Path(_config_override)
CONFIG_FILE = CONFIG_DIR / ".env"
else:
CONFIG_DIR = Path.home() / ".config" / "last30days"
CONFIG_FILE = CONFIG_DIR / ".env"
CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))
AuthSource = Literal["api_key", "codex", "none"]
AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]
AUTH_SOURCE_API_KEY: AuthSource = "api_key"
AUTH_SOURCE_CODEX: AuthSource = "codex"
AUTH_SOURCE_NONE: AuthSource = "none"
AUTH_STATUS_OK: AuthStatus = "ok"
AUTH_STATUS_MISSING: AuthStatus = "missing"
AUTH_STATUS_EXPIRED: AuthStatus = "expired"
AUTH_STATUS_MISSING_ACCOUNT_ID: AuthStatus = "missing_account_id"
def load_env_file(path: Path) -> Dict[str, str]:
@dataclass(frozen=True)
class OpenAIAuth:
token: str | None
source: AuthSource
status: AuthStatus
account_id: str | None
codex_auth_file: str
def _check_file_permissions(path: Path) -> None:
"""Warn to stderr if a secrets file has overly permissive permissions."""
try:
mode = path.stat().st_mode
# Check if group or other can read (bits 0o044)
if mode & 0o044:
sys.stderr.write(
f"[last30days] WARNING: {path} is readable by other users. "
f"Run: chmod 600 {path}\n"
)
sys.stderr.flush()
except OSError as exc:
sys.stderr.write(f"[last30days] WARNING: could not stat {path}: {exc}\n")
sys.stderr.flush()
def load_env_file(path: Path) -> dict[str, str]:
"""Load environment variables from a file."""
env = {}
if not path.exists():
if not path or not path.exists():
return env
_check_file_permissions(path)
with open(path, 'r') as f:
for line in f:
@@ -31,101 +91,568 @@ def load_env_file(path: Path) -> Dict[str, str]:
return env
def get_config() -> Dict[str, Any]:
"""Load configuration from ~/.config/last30days/.env and environment."""
# Load from config file first
file_env = load_env_file(CONFIG_FILE)
def _decode_jwt_payload(token: str) -> dict[str, Any] | None:
"""Decode JWT payload without verification."""
try:
parts = token.split(".")
if len(parts) < 2:
return None
payload_b64 = parts[1]
pad = "=" * (-len(payload_b64) % 4)
decoded = base64.urlsafe_b64decode(payload_b64 + pad)
return json.loads(decoded.decode("utf-8"))
except (json.JSONDecodeError, UnicodeDecodeError, binascii.Error, IndexError) as exc:
sys.stderr.write(f"[last30days] WARNING: malformed JWT token: {exc}\n")
sys.stderr.flush()
return None
# Environment variables override file
def _token_expired(token: str, leeway_seconds: int = 60) -> bool:
"""Check if JWT token is expired."""
payload = _decode_jwt_payload(token)
if not payload:
return False
exp = payload.get("exp")
if not exp:
return False
return exp <= (time.time() + leeway_seconds)
def extract_chatgpt_account_id(access_token: str) -> str | None:
"""Extract chatgpt_account_id from JWT token."""
payload = _decode_jwt_payload(access_token)
if not payload:
return None
auth_claim = payload.get("https://api.openai.com/auth", {})
if isinstance(auth_claim, dict):
return auth_claim.get("chatgpt_account_id")
return None
def load_codex_auth(path: Path = CODEX_AUTH_FILE) -> dict[str, Any]:
"""Load Codex auth JSON."""
if not path.exists():
return {}
try:
with open(path, "r") as f:
return json.load(f)
except json.JSONDecodeError:
sys.stderr.write(
f"[last30days] WARNING: {path} exists but contains invalid JSON -- ignoring\n"
)
sys.stderr.flush()
return {}
def get_codex_access_token() -> tuple[str | None, str]:
"""Get Codex access token from auth.json.
Returns:
(token, status) where status is 'ok', 'missing', or 'expired'
"""
auth = load_codex_auth()
token = None
if isinstance(auth, dict):
tokens = auth.get("tokens") or {}
if isinstance(tokens, dict):
token = tokens.get("access_token")
if not token:
token = auth.get("access_token")
if not token:
return None, AUTH_STATUS_MISSING
if _token_expired(token):
return None, AUTH_STATUS_EXPIRED
return token, AUTH_STATUS_OK
def get_openai_auth(file_env: dict[str, str]) -> OpenAIAuth:
"""Resolve OpenAI auth from API key or Codex login."""
api_key = os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY')
if api_key:
return OpenAIAuth(
token=api_key,
source=AUTH_SOURCE_API_KEY,
status=AUTH_STATUS_OK,
account_id=None,
codex_auth_file=str(CODEX_AUTH_FILE),
)
# Codex auth (chatgpt.com backend) intentionally skipped.
# The endpoint is unstable and causes crashes when the token expires.
# Users who want OpenAI should set OPENAI_API_KEY explicitly.
return OpenAIAuth(
token=None,
source=AUTH_SOURCE_NONE,
status=AUTH_STATUS_MISSING,
account_id=None,
codex_auth_file=str(CODEX_AUTH_FILE),
)
def _find_project_env() -> Path | None:
"""Find per-project .env by walking up from cwd.
Searches for .claude/last30days.env in each parent directory,
stopping at the user's home directory or filesystem root.
"""
cwd = Path.cwd()
for parent in [cwd, *cwd.parents]:
candidate = parent / '.claude' / 'last30days.env'
if candidate.exists():
return candidate
# Stop at filesystem root or home
if parent == Path.home() or parent == parent.parent:
break
return None
def get_config() -> dict[str, Any]:
"""Load configuration from multiple sources.
Priority (highest wins):
1. Environment variables (os.environ)
2. .claude/last30days.env (per-project config)
3. ~/.config/last30days/.env (global config)
"""
# Load from global config file
file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
# Load from per-project config (overrides global)
project_env_path = _find_project_env()
project_env = load_env_file(project_env_path) if project_env_path else {}
# Merge: project overrides global
merged_env = {**file_env, **project_env}
openai_auth = get_openai_auth(merged_env)
# Build config: Codex/OpenAI auth + process.env > project .env > global .env
config = {
'OPENAI_API_KEY': os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY'),
'XAI_API_KEY': os.environ.get('XAI_API_KEY') or file_env.get('XAI_API_KEY'),
'OPENAI_MODEL_POLICY': os.environ.get('OPENAI_MODEL_POLICY') or file_env.get('OPENAI_MODEL_POLICY', 'auto'),
'OPENAI_MODEL_PIN': os.environ.get('OPENAI_MODEL_PIN') or file_env.get('OPENAI_MODEL_PIN'),
'XAI_MODEL_POLICY': os.environ.get('XAI_MODEL_POLICY') or file_env.get('XAI_MODEL_POLICY', 'latest'),
'XAI_MODEL_PIN': os.environ.get('XAI_MODEL_PIN') or file_env.get('XAI_MODEL_PIN'),
'OPENAI_API_KEY': openai_auth.token,
'OPENAI_AUTH_SOURCE': openai_auth.source,
'OPENAI_AUTH_STATUS': openai_auth.status,
'OPENAI_CHATGPT_ACCOUNT_ID': openai_auth.account_id,
'CODEX_AUTH_FILE': openai_auth.codex_auth_file,
}
keys = [
('XAI_API_KEY', None),
('GOOGLE_API_KEY', None),
('GEMINI_API_KEY', None),
('GOOGLE_GENAI_API_KEY', None),
('XIAOHONGSHU_API_BASE', None),
('LAST30DAYS_REASONING_PROVIDER', 'auto'),
('LAST30DAYS_PLANNER_MODEL', None),
('LAST30DAYS_RERANK_MODEL', None),
('LAST30DAYS_X_MODEL', None),
('LAST30DAYS_X_BACKEND', None),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_PIN', None),
('SCRAPECREATORS_API_KEY', None),
('APIFY_API_TOKEN', None),
('AUTH_TOKEN', None),
('CT0', None),
('BSKY_HANDLE', None),
('BSKY_APP_PASSWORD', None),
('TRUTHSOCIAL_TOKEN', None),
('BRAVE_API_KEY', None),
('EXA_API_KEY', None),
('SERPER_API_KEY', None),
('OPENROUTER_API_KEY', None),
('PARALLEL_API_KEY', None),
('XQUIK_API_KEY', None),
('FROM_BROWSER', None),
('SETUP_COMPLETE', None),
('INCLUDE_SOURCES', ''),
]
for key, default in keys:
config[key] = os.environ.get(key) or merged_env.get(key, default)
# Track which config source was used
if project_env_path:
config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
elif CONFIG_FILE and CONFIG_FILE.exists():
config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
else:
config['_CONFIG_SOURCE'] = 'env_only'
# Extract browser credentials if configured
browser_creds = extract_browser_credentials(config)
for key, value in browser_creds.items():
if not config.get(key):
config[key] = value
config[f"_{key}_SOURCE"] = "browser"
return config
def config_exists() -> bool:
"""Check if configuration file exists."""
return CONFIG_FILE.exists()
# ---------------------------------------------------------------------------
# Browser cookie extraction
# ---------------------------------------------------------------------------
COOKIE_DOMAINS: dict[str, dict[str, Any]] = {
"x": {
"domain": ".x.com",
"cookies": ["auth_token", "ct0"],
"mapping": {"auth_token": "AUTH_TOKEN", "ct0": "CT0"},
},
"truthsocial": {
"domain": ".truthsocial.com",
"cookies": ["_session_id"],
"mapping": {"_session_id": "TRUTHSOCIAL_TOKEN"},
},
}
def get_available_sources(config: Dict[str, Any]) -> str:
"""Determine which sources are available based on API keys.
def extract_browser_credentials(config: dict[str, Any]) -> dict[str, str]:
"""Extract auth cookies from local browsers.
Returns: 'both', 'reddit', 'x', or 'web' (fallback when no keys)
Default behavior (FROM_BROWSER unset): tries Firefox and Safari only.
These read local files silently with no system dialogs. Chrome is
skipped because ``security find-generic-password`` triggers a macOS
Keychain prompt that cannot be reliably suppressed.
Set ``FROM_BROWSER=auto`` to also try Chrome (accepts the dialog),
or ``FROM_BROWSER=off`` to disable extraction entirely.
"""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
if has_openai and has_xai:
return 'both'
elif has_openai:
return 'reddit'
elif has_xai:
return 'x'
from_browser = (config.get("FROM_BROWSER") or "").strip().lower()
if from_browser == "off":
return {}
try:
from . import cookie_extract
except ImportError:
return {}
# Determine which browsers to try
if from_browser in ("firefox", "chrome", "safari"):
browsers = [from_browser]
elif from_browser == "auto":
browsers = ["firefox", "safari", "chrome"]
else:
return 'web' # Fallback: WebSearch only (no API keys needed)
# Default: silent browsers only (no Keychain dialog)
browsers = ["firefox", "safari"]
extracted: dict[str, str] = {}
for _service, spec in COOKIE_DOMAINS.items():
if all(config.get(env_key) for env_key in spec["mapping"].values()):
continue
for browser in browsers:
try:
cookies = cookie_extract.extract_cookies(browser, spec["domain"], spec["cookies"])
except Exception:
continue
if cookies:
for cookie_name, env_key in spec["mapping"].items():
if cookie_name in cookies and not config.get(env_key):
extracted[env_key] = cookies[cookie_name]
break # Found cookies for this service, stop trying browsers
return extracted
def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]:
"""Validate requested sources against available keys.
def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
"""Return (source, method) for X search, where method describes the auth origin."""
if config.get("XAI_API_KEY"):
return "xai", "xai"
if config.get("AUTH_TOKEN") and config.get("CT0"):
method = config.get("_AUTH_TOKEN_SOURCE", "env")
return "bird", method
# Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
from . import xurl_x
if xurl_x.is_available():
return "xurl", "oauth2"
return None, "none"
def config_exists() -> bool:
"""Check if any configuration source exists."""
if _find_project_env():
return True
if CONFIG_FILE:
return CONFIG_FILE.exists()
return False
def is_reddit_available(config: dict[str, Any]) -> bool:
"""Check if Reddit search is available.
v3 uses ScrapeCreators only.
"""
return bool(config.get('SCRAPECREATORS_API_KEY'))
def get_reddit_source(config: dict[str, Any]) -> str | None:
"""Determine which Reddit backend to use.
Returns: 'scrapecreators' or None
"""
if config.get('SCRAPECREATORS_API_KEY'):
return 'scrapecreators'
return None
def get_x_source(config: dict[str, Any]) -> str | None:
"""Determine the best available explicit X/Twitter source.
Priority: explicit backend pin, then xAI, then Bird with explicit cookies.
Browser-cookie probing is intentionally not used here. Automatic Keychain
access causes popups during normal pipeline runs. Bird is only considered
available when AUTH_TOKEN and CT0 are present explicitly.
Args:
requested: 'auto', 'reddit', 'x', 'both', or 'web'
available: Result from get_available_sources()
include_web: If True, add WebSearch to available sources
config: Configuration dict from get_config()
Returns:
Tuple of (effective_sources, error_message)
'bird' if Bird is installed and explicit cookies are configured,
'xai' if XAI_API_KEY is configured,
'xurl' if xurl CLI is installed and authenticated,
None if no X source available.
"""
# WebSearch-only mode (no API keys)
if available == 'web':
if requested == 'auto':
return 'web', None
elif requested == 'web':
return 'web', None
else:
return 'web', f"No API keys configured. Using WebSearch fallback. Add keys to ~/.config/last30days/.env for Reddit/X."
# Import here to avoid circular dependency
from . import bird_x
if requested == 'auto':
# Add web to sources if include_web is set
if include_web:
if available == 'both':
return 'all', None # reddit + x + web
elif available == 'reddit':
return 'reddit-web', None
elif available == 'x':
return 'x-web', None
return available, None
preferred = (config.get('LAST30DAYS_X_BACKEND') or '').lower()
has_bird_creds = bool(config.get('AUTH_TOKEN') and config.get('CT0'))
if has_bird_creds:
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
if requested == 'web':
return 'web', None
if preferred == 'xai':
return 'xai' if config.get('XAI_API_KEY') else None
if preferred == 'bird':
return 'bird' if has_bird_creds and bird_x.is_bird_installed() else None
if requested == 'both':
if available not in ('both',):
missing = 'xAI' if available == 'reddit' else 'OpenAI'
return 'none', f"Requested both sources but {missing} key is missing. Use --sources=auto to use available keys."
if include_web:
return 'all', None
return 'both', None
if config.get('XAI_API_KEY'):
return 'xai'
if has_bird_creds and bird_x.is_bird_installed():
return 'bird'
if requested == 'reddit':
if available == 'x':
return 'none', "Requested Reddit but only xAI key is available."
if include_web:
return 'reddit-web', None
return 'reddit', None
# Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
from . import xurl_x
if xurl_x.is_available():
return 'xurl'
if requested == 'x':
if available == 'reddit':
return 'none', "Requested X but only OpenAI key is available."
if include_web:
return 'x-web', None
return 'x', None
return None
return requested, None
def is_ytdlp_available() -> bool:
"""Check if yt-dlp is installed for YouTube search."""
from . import youtube_yt
return youtube_yt.is_ytdlp_installed()
def is_youtube_comments_available(config: dict[str, Any]) -> bool:
"""Check if YouTube comment enrichment is available.
Requires SCRAPECREATORS_API_KEY AND youtube_comments in INCLUDE_SOURCES.
"""
if not config.get('SCRAPECREATORS_API_KEY'):
return False
include = _parse_include_sources(config)
return 'youtube_comments' in include
def is_tiktok_comments_available(config: dict[str, Any]) -> bool:
"""Check if TikTok comment enrichment is available.
Requires SCRAPECREATORS_API_KEY AND tiktok_comments in INCLUDE_SOURCES.
Mirrors the youtube_comments opt-in pattern.
"""
if not config.get('SCRAPECREATORS_API_KEY'):
return False
include = _parse_include_sources(config)
return 'tiktok_comments' in include
def is_youtube_sc_available(config: dict[str, Any]) -> bool:
"""Check if ScrapeCreators YouTube search fallback is available.
Used when yt-dlp is not installed or fails.
"""
return bool(config.get('SCRAPECREATORS_API_KEY'))
def is_hackernews_available() -> bool:
"""Check if Hacker News source is available.
Always returns True - HN uses free Algolia API, no key needed.
"""
return True
def is_bluesky_available(config: dict[str, Any]) -> bool:
"""Check if Bluesky source is available.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD (app password from bsky.app/settings).
"""
return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))
def is_truthsocial_available(config: dict[str, Any]) -> bool:
"""Check if Truth Social source is available.
Requires TRUTHSOCIAL_TOKEN (bearer token from browser dev tools).
"""
return bool(config.get('TRUTHSOCIAL_TOKEN'))
def is_polymarket_available() -> bool:
"""Check if Polymarket source is available.
Always returns True - Gamma API is free, no key needed.
"""
return True
def is_tiktok_available(config: dict[str, Any]) -> bool:
"""Check if TikTok source is available (ScrapeCreators or legacy Apify).
Returns True if SCRAPECREATORS_API_KEY or APIFY_API_TOKEN is set.
"""
return bool(config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN'))
def get_tiktok_token(config: dict[str, Any]) -> str:
"""Get TikTok API token, preferring ScrapeCreators over legacy Apify."""
return config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN') or ''
def _parse_include_sources(config: dict[str, Any]) -> set[str]:
"""Parse INCLUDE_SOURCES config value into a set of lowercase source names."""
raw = config.get('INCLUDE_SOURCES') or ''
return {s.strip().lower() for s in raw.split(',') if s.strip()}
def is_threads_available(config: dict[str, Any]) -> bool:
"""Check if Threads source is available.
Requires SCRAPECREATORS_API_KEY AND 'threads' in INCLUDE_SOURCES.
Threads is an opt-in source - it is not activated by default.
"""
if not config.get('SCRAPECREATORS_API_KEY'):
return False
return 'threads' in _parse_include_sources(config)
def is_instagram_available(config: dict[str, Any]) -> bool:
"""Check if Instagram source is available (ScrapeCreators).
Returns True if SCRAPECREATORS_API_KEY is set.
Instagram uses the same key as TikTok.
"""
return bool(config.get('SCRAPECREATORS_API_KEY'))
def get_instagram_token(config: dict[str, Any]) -> str:
"""Get Instagram API token (same ScrapeCreators key as TikTok)."""
return config.get('SCRAPECREATORS_API_KEY') or ''
def get_xiaohongshu_api_base(config: dict[str, Any]) -> str:
"""Get Xiaohongshu HTTP API base URL.
Defaults to host.docker.internal so OpenClaw Docker can reach host service.
"""
return (config.get('XIAOHONGSHU_API_BASE') or "http://host.docker.internal:18060").rstrip("/")
def is_xiaohongshu_available(config: dict[str, Any]) -> bool:
"""Check whether Xiaohongshu HTTP API is reachable and logged in."""
# Import here to avoid heavy imports at module load.
from . import http
base = get_xiaohongshu_api_base(config)
try:
# Keep health probe snappy, but allow one retry for transient hiccups.
health = http.get(f"{base}/health", timeout=3, retries=2)
if not isinstance(health, dict):
return False
if not health.get("success"):
return False
# Login probe can be slower on some deployments (browser/session checks),
# so use a slightly longer timeout to avoid false negatives.
login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=2)
is_logged_in = (
login.get("data", {}).get("is_logged_in")
if isinstance(login, dict) else False
)
return bool(is_logged_in)
except (OSError, http.HTTPError):
return False
except Exception as exc:
sys.stderr.write(
f"[last30days] WARNING: unexpected error checking Xiaohongshu: "
f"{type(exc).__name__}: {exc}\n"
)
sys.stderr.flush()
return False
# Backward compat alias
is_apify_available = is_tiktok_available
def get_x_source_status(config: dict[str, Any]) -> dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
if config.get('AUTH_TOKEN') and config.get('CT0'):
bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
# Fall back to xurl CLI
from . import xurl_x as _xurl_check
source = 'xurl' if _xurl_check.is_available() else None
from . import xurl_x as _xurl_x
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"xurl_available": _xurl_x.is_available(),
"can_install_bird": bird_status["can_install"],
}
# Pinterest
def is_pinterest_available(config: dict[str, Any]) -> bool:
"""Check if Pinterest source is available.
Returns True when SCRAPECREATORS_API_KEY is set AND 'pinterest' is in
INCLUDE_SOURCES (or requested_sources at the pipeline level). Pinterest
is opt-in because not every topic benefits from visual pin results.
"""
return bool(config.get('SCRAPECREATORS_API_KEY'))
def get_pinterest_token(config: dict[str, Any]) -> str:
"""Get Pinterest API token (same ScrapeCreators key as TikTok/Instagram)."""
return config.get('SCRAPECREATORS_API_KEY') or ''
# Xquik
def is_xquik_available(config: dict[str, Any]) -> bool:
"""Check if Xquik X search source is available.
Requires XQUIK_API_KEY (API key from xquik.com).
"""
return bool(config.get('XQUIK_API_KEY'))
def get_xquik_token(config: dict[str, Any]) -> str:
"""Get Xquik API key."""
return config.get('XQUIK_API_KEY') or ''
+85
View File
@@ -0,0 +1,85 @@
"""Parallel multi-entity fan-out for the --competitors flag.
The orchestrator accepts a `main_runner()` for the topic and a
`competitor_runner(entity)` for each peer. It parallelizes their execution
via a `ThreadPoolExecutor` and collects per-entity Reports. Per-entity
failures are logged and dropped; the run survives as long as the main topic
plus at least one competitor succeed.
This module owns no business logic about pipeline arguments the caller
(scripts/last30days.py main) builds the closures with the appropriate
config, depth, and overrides for each entity.
"""
from __future__ import annotations
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Callable
from . import schema
# Sub-runs hit the same upstream APIs as the main topic. Cap parallelism so a
# 6-way fan-out does not stampede a single backend's rate limit.
MAX_PARALLEL_SUBRUNS = 6
def _log(msg: str) -> None:
print(f"[Fanout] {msg}", file=sys.stderr)
def run_competitor_fanout(
*,
main_topic: str,
main_runner: Callable[[], schema.Report],
competitors: list[str],
competitor_runner: Callable[[str], schema.Report],
) -> list[tuple[str, schema.Report]]:
"""Run main + competitor pipelines in parallel; return surviving reports.
Args:
main_topic: Display label for the user's primary topic.
main_runner: Zero-arg callable returning the main topic's Report.
competitors: Ordered list of competitor entity names.
competitor_runner: Callable(entity_name) -> Report for each peer.
Returns:
Ordered list of (entity_name, Report) tuples for runs that succeeded.
Empty list if every run raised; the caller decides how to surface
partial-failure modes.
"""
if not competitors:
report = main_runner()
return [(main_topic, report)]
workers = min(len(competitors) + 1, MAX_PARALLEL_SUBRUNS)
def _run_one(label: str, fn: Callable[[], schema.Report]) -> tuple[str, schema.Report | None, Exception | None]:
try:
return label, fn(), None
except Exception as exc:
return label, None, exc
submissions: list[tuple[str, Callable[[], schema.Report]]] = [
(main_topic, main_runner),
]
for entity in competitors:
submissions.append((entity, lambda e=entity: competitor_runner(e)))
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(_run_one, label, fn): label
for label, fn in submissions
}
results: dict[str, schema.Report] = {}
for future in as_completed(futures):
label, report, exc = future.result()
if exc is not None:
_log(f"Sub-run failed for {label!r}: {type(exc).__name__}: {exc}")
continue
assert report is not None
results[label] = report
# Preserve the original submission order rather than completion order so
# the comparison render is deterministic across runs.
return [(label, results[label]) for label, _ in submissions if label in results]
+202
View File
@@ -0,0 +1,202 @@
"""Weighted reciprocal rank fusion for per-(subquery, source) streams."""
from __future__ import annotations
from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
from . import schema
# Standard RRF smoothing constant (Cormack et al. 2009)
RRF_K = 60
def _candidate_sort_key(c: schema.Candidate) -> tuple:
return (-c.rrf_score, -c.local_relevance, -c.freshness, schema.candidate_source_label(c), c.title)
def _normalize_url(url: str) -> str:
"""Normalize URL for dedup: lowercase, strip www/old/m prefixes, remove tracking params."""
parsed = urlparse(url.strip().lower())
netloc = parsed.netloc
for prefix in ("www.", "old.", "m."):
if netloc.startswith(prefix):
netloc = netloc[len(prefix):]
# Strip tracking params
params = parse_qs(parsed.query)
clean_params = {k: v for k, v in params.items() if not k.startswith("utm_")}
query = urlencode(clean_params, doseq=True)
return urlunparse((parsed.scheme, netloc, parsed.path.rstrip("/"), "", query, ""))
def candidate_key(item: schema.SourceItem) -> str:
if item.url:
return _normalize_url(item.url)
return f"{item.source}:{item.item_id}"
_DIVERSITY_RELEVANCE_THRESHOLD = 0.25
# Per-author cap: no single author/handle should dominate the pool.
_MAX_ITEMS_PER_AUTHOR = 3
def _extract_author(candidate: schema.Candidate) -> str | None:
"""Return a normalized author key from a candidate's source items."""
for item in candidate.source_items:
if item.author:
return item.author.strip().lower()
return None
def _apply_per_author_cap(
candidates: list[schema.Candidate],
max_per_author: int = _MAX_ITEMS_PER_AUTHOR,
) -> list[schema.Candidate]:
"""Keep at most *max_per_author* items from any single author.
Candidates are assumed to already be sorted by quality (rrf_score etc.),
so the first N encountered per author are the best ones.
"""
author_counts: dict[str, int] = {}
result: list[schema.Candidate] = []
for c in candidates:
author = _extract_author(c)
if author is None:
result.append(c)
continue
count = author_counts.get(author, 0)
if count < max_per_author:
result.append(c)
author_counts[author] = count + 1
return result
def _diversify_pool(
fused: list[schema.Candidate],
pool_limit: int,
min_per_source: int = 2,
) -> list[schema.Candidate]:
"""Ensure at least *min_per_source* items per qualifying source survive truncation.
Sources only qualify for reserved slots if their best item exceeds
the relevance threshold. Low-relevance sources compete on merit only.
"""
max_relevance: dict[str, float] = {}
for c in fused:
current = max_relevance.get(c.source, 0.0)
if c.local_relevance > current:
max_relevance[c.source] = c.local_relevance
reserved: dict[str, list[schema.Candidate]] = {}
remainder: list[schema.Candidate] = []
for c in fused:
qualifies = max_relevance.get(c.source, 0.0) >= _DIVERSITY_RELEVANCE_THRESHOLD
bucket = reserved.setdefault(c.source, [])
if qualifies and len(bucket) < min_per_source:
bucket.append(c)
else:
remainder.append(c)
pool = [c for per_source in reserved.values() for c in per_source]
seen = {c.candidate_id for c in pool}
for c in remainder:
if len(pool) >= pool_limit:
break
if c.candidate_id not in seen:
pool.append(c)
pool.sort(key=_candidate_sort_key)
return pool[:pool_limit]
def weighted_rrf(
streams: dict[tuple[str, str], list[schema.SourceItem]],
plan: schema.QueryPlan,
*,
pool_limit: int,
) -> list[schema.Candidate]:
"""Fuse ranked lists into a single candidate pool."""
subqueries = {subquery.label: subquery for subquery in plan.subqueries}
candidates: dict[str, schema.Candidate] = {}
for (label, source), items in streams.items():
subquery = subqueries[label]
weight = subquery.weight * plan.source_weights.get(source, 1.0)
for rank, item in enumerate(items, start=1):
key = candidate_key(item)
score = weight / (RRF_K + rank)
item_local_relevance = item.local_relevance if item.local_relevance is not None else float(item.metadata.get("local_relevance", item.relevance_hint))
item_freshness = item.freshness if item.freshness is not None else int(item.metadata.get("freshness", 0))
item_source_quality = item.source_quality if item.source_quality is not None else float(item.metadata.get("source_quality", 0.6))
if key not in candidates:
candidates[key] = schema.Candidate(
candidate_id=key,
item_id=item.item_id,
source=item.source,
title=item.title,
url=item.url,
snippet=item.snippet,
subquery_labels=[label],
native_ranks={f"{label}:{source}": rank},
local_relevance=item_local_relevance,
freshness=item_freshness,
engagement=item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score"),
source_quality=item_source_quality,
rrf_score=score,
sources=[item.source],
source_items=[item],
metadata={
"provenance": [
{
"source": source,
"subquery_label": label,
"native_rank": rank,
"item_id": item.item_id,
}
]
},
)
continue
candidate = candidates[key]
candidate.rrf_score += score
previous_primary_score = (candidate.local_relevance * 100.0) + candidate.freshness + (candidate.source_quality * 10.0)
incoming_primary_score = (item_local_relevance * 100.0) + item_freshness + (item_source_quality * 10.0)
candidate.local_relevance = max(
candidate.local_relevance,
item_local_relevance,
)
candidate.freshness = max(candidate.freshness, item_freshness)
item_eng = item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score")
if candidate.engagement is None:
candidate.engagement = item_eng
elif item_eng is not None:
candidate.engagement = max(candidate.engagement, item_eng)
candidate.source_quality = max(
candidate.source_quality,
item_source_quality,
)
candidate.native_ranks[f"{label}:{source}"] = rank
if label not in candidate.subquery_labels:
candidate.subquery_labels.append(label)
if item.source not in candidate.sources:
candidate.sources.append(item.source)
if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items):
candidate.source_items.append(item)
candidate.metadata.setdefault("provenance", []).append(
{
"source": source,
"subquery_label": label,
"native_rank": rank,
"item_id": item.item_id,
}
)
if incoming_primary_score > previous_primary_score:
candidate.item_id = item.item_id
candidate.source = item.source
candidate.title = item.title
candidate.snippet = item.snippet
if len(candidate.snippet.split()) < len(item.snippet.split()):
candidate.snippet = item.snippet
fused = sorted(candidates.values(), key=_candidate_sort_key)
fused = _apply_per_author_cap(fused)
return _diversify_pool(fused, pool_limit)
+921
View File
@@ -0,0 +1,921 @@
"""GitHub Issues/PRs search via the public GitHub Search API.
Uses api.github.com/search/issues for issue/PR discovery and
per-item comment enrichment. Auth via GITHUB_TOKEN env var or
`gh auth token` subprocess fallback.
"""
import json
import math
import os
import re
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional
from . import dates, log
from .query import extract_core_subject
from .relevance import token_overlap_relevance
SEARCH_URL = "https://api.github.com/search/issues"
DEPTH_LIMITS = {
"quick": 15,
"default": 30,
"deep": 60,
}
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
USER_AGENT = "last30days/3.0 (research tool)"
def _log(msg: str):
log.source_log("GitHub", msg, tty_only=False)
def _resolve_token(token: Optional[str] = None) -> Optional[str]:
"""Resolve GitHub auth token from argument, env, or gh CLI."""
if token:
return token
env_token = os.environ.get("GITHUB_TOKEN")
if env_token:
return env_token
# Fallback: try gh CLI
try:
result = subprocess.run(
["gh", "auth", "token"],
capture_output=True, text=True, timeout=5,
)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip()
except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
pass
return None
def _fetch_json(
url: str,
token: Optional[str] = None,
timeout: int = 15,
) -> Optional[Dict[str, Any]]:
"""Fetch JSON from GitHub API. Returns None on failure."""
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/vnd.github+json",
}
if token:
headers["Authorization"] = f"Bearer {token}"
req = urllib.request.Request(url, headers=headers)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read().decode("utf-8")
return json.loads(body)
except urllib.error.HTTPError as e:
if e.code == 403:
_log(f"403 rate limited or forbidden: {url}")
return None
if e.code == 422:
_log(f"422 unprocessable: {url}")
return None
_log(f"HTTP {e.code}: {e.reason}")
return None
except (urllib.error.URLError, OSError, TimeoutError) as e:
_log(f"Network error: {e}")
return None
except json.JSONDecodeError as e:
_log(f"JSON decode error: {e}")
return None
def _parse_repo_from_url(html_url: str) -> str:
"""Extract 'owner/repo' from a GitHub issue/PR URL."""
parts = html_url.replace("https://github.com/", "").split("/")
if len(parts) >= 2:
return f"{parts[0]}/{parts[1]}"
return ""
def _parse_date(iso_str: Optional[str]) -> Optional[str]:
"""Parse a GitHub ISO 8601 datetime string and return YYYY-MM-DD.
Returns None for non-date input. GitHub's API always emits ISO 8601
(e.g. "2026-02-26T16:00:00Z"), but we defer to dates.parse_date() so
garbage input gets rejected instead of silently sliced.
"""
dt = dates.parse_date(iso_str)
return dt.strftime("%Y-%m-%d") if dt else None
def _compute_relevance(
query: str,
title: str,
rank_index: int,
reactions: int,
comments: int,
) -> float:
"""Blend text relevance with engagement signals."""
rank_score = max(0.3, 1.0 - (rank_index * 0.02))
engagement_boost = min(0.2, math.log1p(reactions + comments) / 20)
if query:
content_score = token_overlap_relevance(query, title)
relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
else:
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
return round(relevance, 2)
def search_github(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
token: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Search GitHub Issues and PRs.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
token: Optional GitHub token (falls back to env/gh CLI)
Returns:
List of normalized item dicts. Empty list on any failure.
"""
resolved_token = _resolve_token(token)
if not resolved_token:
_log("No GitHub token available (set GITHUB_TOKEN or install gh CLI)")
return []
count = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
core = extract_core_subject(topic)
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Build search query with date filter
q = f"{core} created:>{from_date}"
params = {
"q": q,
"sort": "reactions",
"order": "desc",
"per_page": str(min(count, 100)),
}
url = f"{SEARCH_URL}?{urllib.parse.urlencode(params)}"
data = _fetch_json(url, token=resolved_token, timeout=30)
if not data:
return []
raw_items = data.get("items", [])
_log(f"Found {len(raw_items)} issues/PRs")
items = []
for i, item in enumerate(raw_items[:count]):
html_url = item.get("html_url", "")
repo = _parse_repo_from_url(html_url)
title = item.get("title", "")
body_text = item.get("body") or ""
reactions_total = item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0
comment_count = item.get("comments", 0)
labels = [
lbl.get("name", "") for lbl in (item.get("labels") or [])
if isinstance(lbl, dict)
]
state = item.get("state", "")
is_pr = "pull_request" in item
author = item.get("user", {}).get("login", "") if isinstance(item.get("user"), dict) else ""
relevance = _compute_relevance(core, title, i, reactions_total, comment_count)
items.append({
"id": f"GH{i + 1}",
"title": title,
"url": html_url,
"date": _parse_date(item.get("created_at")),
"author": author,
"source": "github",
"score": reactions_total,
"container": repo,
"snippet": body_text[:300] if body_text else "",
"relevance": relevance,
"why_relevant": f"GitHub {'PR' if is_pr else 'issue'}: {title[:60]}",
"engagement": {
"reactions": reactions_total,
"comments": comment_count,
},
"metadata": {
"labels": labels,
"state": state,
"comment_count": comment_count,
"reactions": reactions_total,
"is_pr": is_pr,
},
})
# Enrich top items with comments
items = _enrich_top_items(items, depth, resolved_token)
# Date filter
filtered = []
for item in items:
d = item.get("date")
if d is None or (from_date <= d <= to_date):
filtered.append(item)
# Sort by relevance
filtered.sort(key=lambda x: x.get("relevance", 0), reverse=True)
return filtered
def _enrich_top_items(
items: List[Dict[str, Any]],
depth: str,
token: str,
) -> List[Dict[str, Any]]:
"""Fetch comments for top N items by reactions."""
if not items:
return items
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
by_reactions = sorted(
range(len(items)),
key=lambda i: items[i].get("score", 0),
reverse=True,
)
to_enrich = by_reactions[:limit]
_log(f"Enriching top {len(to_enrich)} items with comments")
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {
executor.submit(
_fetch_item_comments,
items[idx]["url"],
token,
): idx
for idx in to_enrich
}
for future in as_completed(futures):
idx = futures[future]
try:
comments = future.result(timeout=15)
items[idx]["metadata"]["top_comments"] = comments
except (KeyError, TypeError, OSError) as exc:
_log(f"Comment enrichment failed for {items[idx].get('url', '?')}: {type(exc).__name__}: {exc}")
items[idx]["metadata"]["top_comments"] = []
return items
def _fetch_item_comments(
issue_url: str,
token: str,
max_comments: int = 5,
) -> List[Dict[str, Any]]:
"""Fetch comments for a GitHub issue/PR.
Args:
issue_url: HTML URL like https://github.com/owner/repo/issues/123
token: GitHub auth token
max_comments: Max comments to return
Returns:
List of comment dicts with score, excerpt, author.
"""
path = issue_url.replace("https://github.com/", "")
path = path.replace("/pull/", "/issues/")
api_url = f"https://api.github.com/repos/{path}/comments?per_page={max_comments}&sort=reactions&direction=desc"
data = _fetch_json(api_url, token=token, timeout=15)
if not data or not isinstance(data, list):
return []
comments = []
for c in data[:max_comments]:
body = c.get("body") or ""
excerpt = body[:300] + "..." if len(body) > 300 else body
reactions = c.get("reactions", {})
reaction_count = reactions.get("total_count", 0) if isinstance(reactions, dict) else 0
author = c.get("user", {}).get("login", "") if isinstance(c.get("user"), dict) else ""
comments.append({
"score": reaction_count,
"excerpt": excerpt,
"author": author,
})
return comments
# ---------------------------------------------------------------------------
# Person-mode search: author-scoped queries, star enrichment, release notes
# ---------------------------------------------------------------------------
PERSON_DEPTH_LIMITS = {
"quick": {"pr_pages": 1, "own_repos": 3, "external_repos": 5},
"default": {"pr_pages": 1, "own_repos": 5, "external_repos": 10},
"deep": {"pr_pages": 2, "own_repos": 5, "external_repos": 15},
}
def _fetch_readme_snippet(repo: str, token: str, max_chars: int = 500) -> Optional[str]:
"""Fetch README content for a repo, truncated to first ~max_chars."""
url = f"https://api.github.com/repos/{repo}/readme"
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/vnd.github.raw+json",
}
if token:
headers["Authorization"] = f"Bearer {token}"
req = urllib.request.Request(url, headers=headers)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
raw = resp.read().decode("utf-8", errors="replace")
except (urllib.error.HTTPError, urllib.error.URLError, OSError, TimeoutError):
return None
if not raw:
return None
# Try to break at a paragraph boundary
if len(raw) <= max_chars:
return raw
cut = raw[:max_chars]
last_double_newline = cut.rfind("\n\n")
if last_double_newline > max_chars // 3:
return cut[:last_double_newline].rstrip()
return cut.rstrip() + "..."
def _fetch_latest_releases(
repo: str, token: str, count: int = 3, max_body: int = 300,
) -> List[Dict[str, str]]:
"""Fetch latest releases for a repo."""
url = f"https://api.github.com/repos/{repo}/releases?per_page={count}"
data = _fetch_json(url, token=token, timeout=10)
if not data or not isinstance(data, list):
return []
releases = []
for r in data[:count]:
tag = r.get("tag_name", "")
date = _parse_date(r.get("published_at"))
body = (r.get("body") or "")[:max_body]
name = r.get("name") or tag
releases.append({"tag": tag, "name": name, "date": date, "body": body})
return releases
def _fetch_top_issues(repo: str, token: str) -> Dict[str, Any]:
"""Fetch top feature request (by reactions) and top complaint (by comments)."""
result: Dict[str, Any] = {}
# Top feature request: issues with enhancement label, sorted by reactions
feat_q = urllib.parse.quote(f"repo:{repo} is:issue is:open label:enhancement")
feat_url = f"{SEARCH_URL}?q={feat_q}&sort=reactions&order=desc&per_page=1"
feat_data = _fetch_json(feat_url, token=token, timeout=10)
if feat_data and feat_data.get("items"):
item = feat_data["items"][0]
result["top_feature_request"] = {
"title": item.get("title", ""),
"reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
"comments": item.get("comments", 0),
"url": item.get("html_url", ""),
}
elif feat_data and feat_data.get("total_count", 0) == 0:
# No enhancement label; fall back to top issue by reactions
fallback_q = urllib.parse.quote(f"repo:{repo} is:issue is:open")
fallback_url = f"{SEARCH_URL}?q={fallback_q}&sort=reactions&order=desc&per_page=1"
fallback_data = _fetch_json(fallback_url, token=token, timeout=10)
if fallback_data and fallback_data.get("items"):
item = fallback_data["items"][0]
result["top_feature_request"] = {
"title": item.get("title", ""),
"reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
"comments": item.get("comments", 0),
"url": item.get("html_url", ""),
}
# Top complaint: most-discussed open issue (by comments)
bug_q = urllib.parse.quote(f"repo:{repo} is:issue is:open")
bug_url = f"{SEARCH_URL}?q={bug_q}&sort=comments&order=desc&per_page=1"
bug_data = _fetch_json(bug_url, token=token, timeout=10)
if bug_data and bug_data.get("items"):
item = bug_data["items"][0]
result["top_complaint"] = {
"title": item.get("title", ""),
"reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
"comments": item.get("comments", 0),
"url": item.get("html_url", ""),
}
return result
def _fetch_repo_info(repo: str, token: str) -> Optional[Dict[str, Any]]:
"""Fetch repo metadata (stars, forks, description, language)."""
url = f"https://api.github.com/repos/{repo}"
data = _fetch_json(url, token=token, timeout=10)
if not data or not isinstance(data, dict):
return None
return {
"stars": data.get("stargazers_count", 0),
"forks": data.get("forks_count", 0),
"description": (data.get("description") or "")[:200],
"language": data.get("language") or "",
"open_issues": data.get("open_issues_count", 0),
}
def _format_stars(n: int) -> str:
"""Format star count as human-readable (e.g., 349K, 2.9K, 42)."""
if n >= 1_000_000:
return f"{n / 1_000_000:.1f}M"
if n >= 1_000:
return f"{n / 1_000:.0f}K" if n >= 10_000 else f"{n / 1_000:.1f}K"
return str(n)
def search_github_person(
username: str,
from_date: str,
to_date: str,
depth: str = "default",
token: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Person-mode GitHub search: author-scoped queries with star enrichment.
Returns SourceItems for:
- 1 velocity summary item
- Per-repo items for top external repos (with stars + release notes)
- Per-repo items for own repos (with stars + README + top issues + releases)
"""
resolved_token = _resolve_token(token)
if not resolved_token:
_log("No GitHub token available for person-mode search")
return []
limits = PERSON_DEPTH_LIMITS.get(depth, PERSON_DEPTH_LIMITS["default"])
_log(f"Person-mode search for @{username} (since {from_date})")
# Phase 1: PR velocity via search API
total_q = urllib.parse.quote(f"author:{username} type:pr created:>{from_date}")
merged_q = urllib.parse.quote(f"author:{username} type:pr is:merged created:>{from_date}")
total_url = f"{SEARCH_URL}?q={total_q}&per_page=1"
merged_url = f"{SEARCH_URL}?q={merged_q}&sort=reactions&order=desc&per_page=100"
total_data = _fetch_json(total_url, token=resolved_token, timeout=20)
merged_data = _fetch_json(merged_url, token=resolved_token, timeout=20)
total_prs = total_data.get("total_count", 0) if total_data else 0
merged_count = merged_data.get("total_count", 0) if merged_data else 0
merged_items = merged_data.get("items", []) if merged_data else []
_log(f"Found {total_prs} total PRs, {merged_count} merged")
if total_prs == 0 and merged_count == 0:
_log("No PRs found, falling back to keyword search")
return []
# Phase 2: Group merged PRs by repo
repo_pr_counts: Dict[str, int] = {}
for item in merged_items:
repo = _parse_repo_from_url(item.get("html_url", ""))
if repo:
repo_pr_counts[repo] = repo_pr_counts.get(repo, 0) + 1
# Sort repos by PR count (most active first)
sorted_repos = sorted(repo_pr_counts.items(), key=lambda x: x[1], reverse=True)
# Phase 3: Fetch own repos
own_repos_url = f"https://api.github.com/users/{username}/repos?sort=stars&per_page={limits['own_repos']}&direction=desc"
own_repos_data = _fetch_json(own_repos_url, token=resolved_token, timeout=15)
own_repo_names = set()
own_repos_info: List[Dict[str, Any]] = []
if own_repos_data and isinstance(own_repos_data, list):
for r in own_repos_data:
full_name = r.get("full_name", "")
if full_name and not r.get("fork"):
own_repo_names.add(full_name)
own_repos_info.append({
"full_name": full_name,
"stars": r.get("stargazers_count", 0),
"forks": r.get("forks_count", 0),
"description": (r.get("description") or "")[:200],
"language": r.get("language") or "",
"open_issues": r.get("open_issues_count", 0),
})
# Separate external repos from own repos
external_repos = [(repo, count) for repo, count in sorted_repos if repo not in own_repo_names]
external_repos = external_repos[:limits["external_repos"]]
# Phase 4: Parallel enrichment (star counts, releases, READMEs, top issues)
items: List[Dict[str, Any]] = []
idx = 0
# Build velocity summary
open_prs = total_prs - merged_count
merge_rate = round(100 * merged_count / total_prs) if total_prs > 0 else 0
num_repos = len(repo_pr_counts)
velocity_text = (
f"GitHub Person Profile: @{username}\n\n"
f"CONTRIBUTION VELOCITY (last {(to_date > from_date) and 30 or 30} days)\n"
f"- {merged_count} PRs merged across {num_repos} repos ({merge_rate}% merge rate)\n"
f"- {total_prs} total PRs submitted, {open_prs} still open\n"
)
idx += 1
items.append({
"id": f"GH{idx}",
"title": f"@{username}: {merged_count} PRs merged across {num_repos} repos ({merge_rate}% merge rate)",
"url": f"https://github.com/{username}",
"date": to_date,
"author": username,
"source": "github",
"score": merged_count,
"container": f"@{username}",
"snippet": velocity_text,
"relevance": 0.95,
"why_relevant": f"GitHub profile: @{username} - {merged_count} PRs merged across {num_repos} repos",
"engagement": {"reactions": merged_count, "comments": total_prs},
"metadata": {
"labels": ["person-profile", "velocity"],
"state": "open",
"comment_count": 0,
"reactions": merged_count,
"is_pr": False,
},
})
# Phase 5: Enrich external repos (parallel: star counts + releases)
_log(f"Enriching {len(external_repos)} external repos + {len(own_repos_info)} own repos")
with ThreadPoolExecutor(max_workers=8) as executor:
# External repo enrichment: stars + releases
ext_futures = {}
for repo, pr_count in external_repos:
ext_futures[executor.submit(_enrich_external_repo, repo, resolved_token)] = (repo, pr_count)
# Own repo enrichment: README + releases + top issues
own_futures = {}
for own_repo in own_repos_info:
own_futures[executor.submit(_enrich_own_repo, own_repo["full_name"], resolved_token)] = own_repo
# Collect external repo results
for future in as_completed(ext_futures):
repo, pr_count = ext_futures[future]
try:
enrichment = future.result(timeout=20)
except Exception as exc:
_log(f"External repo enrichment failed for {repo}: {exc}")
enrichment = {}
repo_info = enrichment.get("info")
releases = enrichment.get("releases", [])
stars = repo_info["stars"] if repo_info else 0
stars_str = _format_stars(stars)
desc = repo_info["description"] if repo_info else ""
snippet_parts = [f"Contributed {pr_count} merged PRs to {repo} ({stars_str} stars)"]
if desc:
snippet_parts.append(f" {desc}")
if releases:
for rel in releases[:2]:
body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
snippet_parts.append(f" Latest release: {rel['name']} ({rel['date']}){body_preview}")
idx += 1
items.append({
"id": f"GH{idx}",
"title": f"{repo} ({stars_str} stars) - {pr_count} PRs merged",
"url": f"https://github.com/{repo}",
"date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
"author": username,
"source": "github",
"score": stars,
"container": repo,
"snippet": "\n".join(snippet_parts),
"relevance": min(0.9, 0.6 + math.log1p(stars) / 30 + min(0.15, pr_count / 20)),
"why_relevant": f"GitHub contribution: {pr_count} PRs merged to {repo} ({stars_str} stars)",
"engagement": {"reactions": stars, "comments": pr_count},
"metadata": {
"labels": ["person-profile", "external-repo"],
"state": "open",
"comment_count": pr_count,
"reactions": stars,
"is_pr": False,
},
})
# Collect own repo results
for future in as_completed(own_futures):
own_repo = own_futures[future]
try:
enrichment = future.result(timeout=25)
except Exception as exc:
_log(f"Own repo enrichment failed for {own_repo['full_name']}: {exc}")
enrichment = {}
repo_name = own_repo["full_name"]
stars = own_repo["stars"]
stars_str = _format_stars(stars)
open_issues = own_repo["open_issues"]
desc = own_repo["description"]
readme = enrichment.get("readme")
releases = enrichment.get("releases", [])
top_issues = enrichment.get("top_issues", {})
snippet_parts = [f"Own project: {repo_name} ({stars_str} stars, {open_issues} open issues)"]
if desc:
snippet_parts.append(f" {desc}")
if readme:
snippet_parts.append(f" README: {readme[:300]}")
if releases:
for rel in releases[:2]:
body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
snippet_parts.append(f" Latest release: {rel['name']} ({rel['date']}){body_preview}")
feat = top_issues.get("top_feature_request")
if feat:
snippet_parts.append(f" Top feature request: \"{feat['title']}\" ({feat['reactions']} reactions, {feat['comments']} comments)")
complaint = top_issues.get("top_complaint")
if complaint:
snippet_parts.append(f" Top complaint: \"{complaint['title']}\" ({complaint['comments']} comments)")
idx += 1
items.append({
"id": f"GH{idx}",
"title": f"{repo_name} ({stars_str} stars) - own project, {open_issues} open issues",
"url": f"https://github.com/{repo_name}",
"date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
"author": username,
"source": "github",
"score": stars,
"container": repo_name,
"snippet": "\n".join(snippet_parts),
"relevance": min(0.95, 0.7 + math.log1p(stars) / 25),
"why_relevant": f"GitHub own project: {repo_name} ({stars_str} stars)",
"engagement": {"reactions": stars, "comments": open_issues},
"metadata": {
"labels": ["person-profile", "own-repo"],
"state": "open",
"comment_count": open_issues,
"reactions": stars,
"is_pr": False,
},
})
# Sort by relevance
items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
_log(f"Person-mode returned {len(items)} items")
return items
def _enrich_external_repo(repo: str, token: str) -> Dict[str, Any]:
"""Fetch star count + releases for an external repo."""
info = _fetch_repo_info(repo, token)
releases = _fetch_latest_releases(repo, token, count=3)
return {"info": info, "releases": releases}
def _enrich_own_repo(repo: str, token: str) -> Dict[str, Any]:
"""Fetch README + releases + top issues for an own repo."""
readme = _fetch_readme_snippet(repo, token, max_chars=500)
releases = _fetch_latest_releases(repo, token, count=3)
top_issues = _fetch_top_issues(repo, token)
return {"readme": readme, "releases": releases, "top_issues": top_issues}
# ---------------------------------------------------------------------------
# Project-mode search: fetch comprehensive data for specific repos
# ---------------------------------------------------------------------------
def search_github_project(
repos: List[str],
from_date: str,
to_date: str,
depth: str = "default",
token: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Project-mode GitHub search: fetch stars, README, releases, top issues for repos.
Args:
repos: List of 'owner/repo' strings.
from_date: Start date (YYYY-MM-DD).
to_date: End date (YYYY-MM-DD).
depth: 'quick', 'default', or 'deep'.
token: Optional GitHub token.
Returns:
List of SourceItems, one per repo.
"""
resolved_token = _resolve_token(token)
if not resolved_token:
_log("No GitHub token available for project-mode search")
return []
_log(f"Project-mode search for {len(repos)} repos: {', '.join(repos)}")
items: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=min(8, len(repos))) as executor:
futures = {
executor.submit(_enrich_project_repo, repo, resolved_token): repo
for repo in repos
}
for idx, future in enumerate(as_completed(futures)):
repo = futures[future]
try:
enrichment = future.result(timeout=25)
except Exception as exc:
_log(f"Project enrichment failed for {repo}: {exc}")
continue
info = enrichment.get("info")
if not info:
_log(f"No repo info for {repo}, skipping")
continue
readme = enrichment.get("readme")
releases = enrichment.get("releases", [])
top_issues = enrichment.get("top_issues", {})
stars = info["stars"]
stars_str = _format_stars(stars)
open_issues = info["open_issues"]
desc = info["description"]
lang = info["language"]
snippet_parts = [f"Project: {repo} ({stars_str} stars, {open_issues} open issues, {lang})"]
if desc:
snippet_parts.append(f" {desc}")
if readme:
snippet_parts.append(f" README: {readme[:400]}")
if releases:
for rel in releases[:2]:
body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
snippet_parts.append(f" Latest release: {rel['name']} ({rel['date']}){body_preview}")
feat = top_issues.get("top_feature_request")
if feat:
snippet_parts.append(f" Top feature request: \"{feat['title']}\" ({feat['reactions']} reactions, {feat['comments']} comments)")
complaint = top_issues.get("top_complaint")
if complaint:
snippet_parts.append(f" Top complaint: \"{complaint['title']}\" ({complaint['comments']} comments)")
items.append({
"id": f"GH{idx + 1}",
"title": f"{repo} ({stars_str} stars) - {open_issues} open issues",
"url": f"https://github.com/{repo}",
"date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
"author": repo.split("/")[0],
"source": "github",
"score": stars,
"container": repo,
"snippet": "\n".join(snippet_parts),
"relevance": min(0.95, 0.7 + math.log1p(stars) / 25),
"why_relevant": f"GitHub project: {repo} ({stars_str} stars, live)",
"engagement": {"reactions": stars, "comments": open_issues},
"metadata": {
"labels": ["project-mode"],
"state": "open",
"comment_count": open_issues,
"reactions": stars,
"is_pr": False,
"github_stars": {repo: stars},
},
})
items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
_log(f"Project-mode returned {len(items)} items")
return items
def _enrich_project_repo(repo: str, token: str) -> Dict[str, Any]:
"""Fetch all project data for a repo: info + README + releases + top issues."""
info = _fetch_repo_info(repo, token)
readme = _fetch_readme_snippet(repo, token, max_chars=500)
releases = _fetch_latest_releases(repo, token, count=3)
top_issues = _fetch_top_issues(repo, token)
return {"info": info, "readme": readme, "releases": releases, "top_issues": top_issues}
# ---------------------------------------------------------------------------
# Post-rerank star enrichment: annotate candidates with live star counts
# ---------------------------------------------------------------------------
_REPO_URL_PATTERN = re.compile(r"github\.com/([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)")
_SKIP_PATHS = {"topics", "search", "orgs", "settings", "features", "about", "pricing", "enterprise", "explore", "marketplace", "sponsors"}
def extract_repo_refs(candidates: List[Any]) -> List[str]:
"""Extract unique owner/repo strings from candidate URLs, titles, and snippets."""
seen: set = set()
repos: List[str] = []
for c in candidates:
texts = [
getattr(c, "url", "") or "",
getattr(c, "title", "") or "",
]
# Also check evidence snippets if available
evidence = getattr(c, "evidence", None)
if evidence:
texts.append(str(evidence))
for text in texts:
for match in _REPO_URL_PATTERN.findall(text):
# Normalize: strip trailing .git, lowercase
repo = match.rstrip(".git").lower()
owner = repo.split("/")[0]
if owner in _SKIP_PATHS:
continue
if repo not in seen:
seen.add(repo)
repos.append(match) # preserve original case
return repos
def enrich_candidates_with_stars(
candidates: List[Any],
token: Optional[str] = None,
already_enriched: Optional[set] = None,
max_repos: int = 10,
) -> int:
"""Annotate candidates with live GitHub star counts.
Returns the number of repos enriched.
"""
resolved_token = _resolve_token(token)
if not resolved_token:
return 0
refs = extract_repo_refs(candidates)
if not refs:
return 0
skip = already_enriched or set()
to_fetch = [r for r in refs if r.lower() not in {s.lower() for s in skip}][:max_repos]
if not to_fetch:
return 0
_log(f"Star enrichment: fetching {len(to_fetch)} repos")
# Parallel fetch star counts
star_map: Dict[str, int] = {}
with ThreadPoolExecutor(max_workers=min(8, len(to_fetch))) as executor:
futures = {executor.submit(_fetch_repo_info, repo, resolved_token): repo for repo in to_fetch}
for future in as_completed(futures):
repo = futures[future]
try:
info = future.result(timeout=10)
if info:
star_map[repo.lower()] = info["stars"]
except Exception:
pass
if not star_map:
return 0
# Annotate candidates
enriched_count = 0
for c in candidates:
texts = [getattr(c, "url", "") or "", getattr(c, "title", "") or ""]
evidence = getattr(c, "evidence", None)
if evidence:
texts.append(str(evidence))
combined = " ".join(texts)
for match in _REPO_URL_PATTERN.findall(combined):
repo_lower = match.rstrip(".git").lower()
if repo_lower in star_map:
stars = star_map[repo_lower]
stars_str = _format_stars(stars)
# Add to metadata
if not hasattr(c, "metadata") or c.metadata is None:
continue
if "github_stars" not in c.metadata:
c.metadata["github_stars"] = {}
c.metadata["github_stars"][match] = stars
# Append to evidence if present
if hasattr(c, "evidence") and c.evidence and f"(live:" not in c.evidence:
c.evidence = c.evidence + f" (live: {stars_str} stars)"
enriched_count += 1
break # one annotation per candidate
_log(f"Star enrichment: annotated {enriched_count} candidates")
return enriched_count
+259
View File
@@ -0,0 +1,259 @@
"""Web search retrieval via Brave Search, Exa, and Serper."""
from __future__ import annotations
import urllib.parse
from datetime import datetime
from urllib.parse import urlparse
from . import dates, http
# ---------------------------------------------------------------------------
# Brave Search API
# ---------------------------------------------------------------------------
def brave_search(
query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
url = (
"https://api.search.brave.com/res/v1/web/search?"
+ urllib.parse.urlencode(
{
"q": query,
"count": count,
"freshness": f"{date_range[0]}to{date_range[1]}",
}
)
)
data = http.request("GET", url, headers={"X-Subscription-Token": api_key}, timeout=15)
items = []
for i, r in enumerate((data.get("web", {}).get("results", []))[:count]):
raw_date = r.get("page_age") or ""
pub_date = _normalize_date(raw_date[:10]) if raw_date else None
if not _in_date_range(pub_date, date_range):
continue
items.append({
"id": f"WB{i + 1}",
"title": r.get("title", ""),
"url": r.get("url", ""),
"source_domain": _domain(r.get("url", "")),
"snippet": r.get("description", ""),
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Brave web search",
})
artifact = {"label": "brave", "webSearchQueries": [query], "resultCount": len(items)}
return items, artifact
# ---------------------------------------------------------------------------
# Exa AI Search
# ---------------------------------------------------------------------------
def exa_search(
query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
data = http.request(
"POST", "https://api.exa.ai/search",
headers={"x-api-key": api_key},
json_data={
"query": query,
"type": "auto",
"numResults": count,
"startPublishedDate": f"{date_range[0]}T00:00:00.000Z",
"endPublishedDate": f"{date_range[1]}T23:59:59.999Z",
"contents": {"text": {"maxCharacters": 2000}},
},
timeout=15,
)
items = []
for i, r in enumerate((data.get("results", []))[:count]):
if not isinstance(r, dict):
continue
url = r.get("url", "")
if not url:
continue
raw_date = r.get("publishedDate") or ""
pub_date = _normalize_date(raw_date.split("T")[0] if "T" in raw_date else raw_date[:10]) if raw_date else None
if not _in_date_range(pub_date, date_range):
continue
items.append({
"id": f"WE{i + 1}",
"title": r.get("title", ""),
"url": url,
"source_domain": _domain(url),
"snippet": (r.get("text") or "")[:500],
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Exa web search",
})
artifact = {"label": "exa", "webSearchQueries": [query], "resultCount": len(items)}
return items, artifact
# ---------------------------------------------------------------------------
# Serper (Google Search wrapper)
# ---------------------------------------------------------------------------
def serper_search(
query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
data = http.request(
"POST", "https://google.serper.dev/search",
headers={"X-API-KEY": api_key},
json_data={
"q": query,
"num": count,
"tbs": f"cdr:1,cd_min:{_serper_date_param(date_range[0])},cd_max:{_serper_date_param(date_range[1])}",
},
timeout=15,
)
items = []
for i, r in enumerate((data.get("organic", []))[:count]):
raw_date = r.get("date") or ""
pub_date = _parse_serper_date(raw_date)
if not _in_date_range(pub_date, date_range):
continue
items.append({
"id": f"WS{i + 1}",
"title": r.get("title", ""),
"url": r.get("link", ""),
"source_domain": _domain(r.get("link", "")),
"snippet": r.get("snippet", ""),
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Serper web search",
})
artifact = {"label": "serper", "webSearchQueries": [query], "resultCount": len(items)}
return items, artifact
# ---------------------------------------------------------------------------
# Parallel AI Search
# ---------------------------------------------------------------------------
def parallel_search(
query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
data = http.request(
"POST", "https://api.parallel.ai/v1/search",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
json_data={"query": query, "max_results": count},
timeout=15,
)
items = []
for i, r in enumerate((data.get("results", []))[:count]):
if not isinstance(r, dict):
continue
url = r.get("url", "")
if not url:
continue
raw_date = r.get("published_date") or ""
pub_date = _normalize_date(raw_date[:10]) if raw_date else None
if not _in_date_range(pub_date, date_range):
continue
items.append({
"id": f"WP{i + 1}",
"title": r.get("title", ""),
"url": url,
"source_domain": _domain(url),
"snippet": r.get("snippet", ""),
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Parallel AI web search",
})
artifact = {"label": "parallel", "webSearchQueries": [query], "resultCount": len(items)}
return items, artifact
def _parse_serper_date(raw: str) -> str | None:
if not raw:
return None
normalized = _normalize_date(raw)
if normalized:
return normalized
for fmt in ("%b %d, %Y", "%B %d, %Y", "%Y-%m-%d"):
try:
return datetime.strptime(raw.strip(), fmt).date().isoformat()
except ValueError:
continue
return None
# ---------------------------------------------------------------------------
# Dispatcher
# ---------------------------------------------------------------------------
def web_search(
query: str,
date_range: tuple[str, str],
config: dict,
backend: str = "auto",
) -> tuple[list[dict], dict]:
"""Run web search with the specified or auto-detected backend."""
if backend == "auto":
if config.get("BRAVE_API_KEY"):
backend = "brave"
elif config.get("EXA_API_KEY"):
backend = "exa"
elif config.get("SERPER_API_KEY"):
backend = "serper"
elif config.get("PARALLEL_API_KEY"):
backend = "parallel"
else:
return [], {}
if backend == "brave":
key = config.get("BRAVE_API_KEY")
if not key:
raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'")
return brave_search(query, date_range, key)
if backend == "exa":
key = config.get("EXA_API_KEY")
if not key:
raise RuntimeError("EXA_API_KEY is required when web_backend='exa'")
return exa_search(query, date_range, key)
if backend == "serper":
key = config.get("SERPER_API_KEY")
if not key:
raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'")
return serper_search(query, date_range, key)
if backend == "parallel":
key = config.get("PARALLEL_API_KEY")
if not key:
raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'")
return parallel_search(query, date_range, key)
if backend != "none":
raise ValueError(f"Unsupported web backend: {backend!r}")
return [], {}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _normalize_date(value: object) -> str | None:
if value is None:
return None
parsed = dates.parse_date(str(value).strip())
if not parsed:
return None
return parsed.date().isoformat()
def _serper_date_param(iso_date: str) -> str:
"""Convert YYYY-MM-DD to MM/DD/YYYY for Serper tbs parameter."""
parts = iso_date.split("-")
return f"{parts[1]}/{parts[2]}/{parts[0]}"
def _in_date_range(pub_date: str | None, date_range: tuple[str, str]) -> bool:
if not pub_date:
return False
return date_range[0] <= pub_date <= date_range[1]
def _domain(url: str) -> str:
return urlparse(url).netloc.strip().lower()
+301
View File
@@ -0,0 +1,301 @@
"""Hacker News search via Algolia API (free, no auth required).
Uses hn.algolia.com/api/v1 for story discovery and comment enrichment.
No API key needed - just HTTP calls via stdlib urllib.
"""
import datetime
import html
import math
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional
import re
from . import http, log
from .query import extract_core_subject
from .relevance import token_overlap_relevance
# Common HN prefixes that can cause false-positive keyword matches
_HN_PREFIXES = re.compile(r"^(Tell HN|Show HN|Ask HN|Launch HN)\s*:\s*", re.IGNORECASE)
ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 10,
}
def _log(msg: str):
log.source_log("HN", msg)
def _date_to_unix(date_str: str) -> int:
"""Convert YYYY-MM-DD to Unix timestamp (start of day UTC)."""
parts = date_str.split("-")
year, month, day = int(parts[0]), int(parts[1]), int(parts[2])
dt = datetime.datetime(year, month, day, tzinfo=datetime.timezone.utc)
return int(dt.timestamp())
def _unix_to_date(ts: int) -> str:
"""Convert Unix timestamp to YYYY-MM-DD."""
dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
return dt.strftime("%Y-%m-%d")
def _strip_html(text: str) -> str:
"""Strip HTML tags and decode entities from HN comment text."""
import re
text = html.unescape(text)
text = re.sub(r'<p>', '\n', text)
text = re.sub(r'<[^>]+>', '', text)
return text.strip()
def search_hackernews(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search Hacker News via Algolia API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
Returns:
Dict with Algolia response (contains 'hits' list).
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
from_ts = _date_to_unix(from_date)
to_ts = _date_to_unix(to_date) + 86400 # Include the end date
# Use extracted core subject instead of raw topic for cleaner Algolia matching
core = extract_core_subject(topic)
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Use relevance-sorted search with minimum engagement filter.
# NOTE: restrictSearchableAttributes=title omitted intentionally — it would
# miss Ask HN/Show HN threads where the topic appears in the body.
params = {
"query": core,
"tags": "story",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
"hitsPerPage": str(count),
}
from urllib.parse import urlencode
url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"
try:
response = http.request("GET", url, timeout=30)
except http.HTTPError as e:
_log(f"Search failed: {e}")
return {"hits": [], "error": str(e)}
except Exception as e:
_log(f"Search failed: {e}")
return {"hits": [], "error": str(e)}
hits = response.get("hits", [])
_log(f"Found {len(hits)} stories")
return response
def _title_matches_query(title: str, query: str, author: str = "") -> bool:
"""Check if the query term appears in the title content, not just an HN prefix or author.
Returns True if the query (or any multi-word token) appears in the title
after stripping "Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes
and ignoring the author name. Returns True when query is empty (no filter).
"""
if not query:
return True
stripped = _HN_PREFIXES.sub("", title).strip()
# Also check that the match isn't solely in the author's username
check_text = stripped.lower()
query_lower = query.lower()
# Check each word of the query independently; all must appear somewhere
# in the stripped title (not just the prefix).
query_words = query_lower.split()
for word in query_words:
if word in check_text:
continue
# Word not found in stripped title — reject
return False
return True
def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
"""Parse Algolia response into normalized item dicts.
Args:
response: Algolia search response
query: Original search query for token-overlap relevance scoring
Returns:
List of item dicts ready for normalization.
"""
hits = response.get("hits", [])
# Post-filter: remove items where query only matched an HN prefix like "Tell HN:"
if query:
before = len(hits)
hits = [
h for h in hits
if _title_matches_query(h.get("title", ""), query, h.get("author", ""))
]
dropped = before - len(hits)
if dropped:
_log(f"Prefix filter removed {dropped}/{before} false-positive hits for '{query}'")
items = []
for i, hit in enumerate(hits):
object_id = hit.get("objectID", "")
points = hit.get("points") or 0
num_comments = hit.get("num_comments") or 0
created_at_i = hit.get("created_at_i")
date_str = None
if created_at_i:
date_str = _unix_to_date(created_at_i)
# Article URL vs HN discussion URL
article_url = hit.get("url") or ""
hn_url = f"https://news.ycombinator.com/item?id={object_id}"
# Relevance: blend Algolia rank with token-overlap content matching
rank_score = max(0.3, 1.0 - (i * 0.02)) # 1.0 -> 0.3 over 35 items
engagement_boost = min(0.2, math.log1p(points) / 40)
if query:
content_score = token_overlap_relevance(query, hit.get("title", ""))
relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
else:
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"id": object_id,
"title": hit.get("title", ""),
"url": article_url,
"hn_url": hn_url,
"author": hit.get("author", ""),
"date": date_str,
"engagement": {
"points": points,
"comments": num_comments,
},
"relevance": round(relevance, 2),
"why_relevant": f"HN story about {hit.get('title', 'topic')[:60]}",
})
return items
def _fetch_item_comments(object_id: str, max_comments: int = 5) -> Dict[str, Any]:
"""Fetch top-level comments for a story from Algolia items endpoint.
Args:
object_id: HN story ID
max_comments: Max comments to return
Returns:
Dict with 'comments' list and 'comment_insights' list.
"""
url = f"{ALGOLIA_ITEM_URL}/{object_id}"
try:
data = http.request("GET", url, timeout=15)
except Exception as e:
_log(f"Failed to fetch comments for {object_id}: {e}")
return {"comments": [], "comment_insights": []}
children = data.get("children", [])
# Sort by points (highest first), filter to actual comments
real_comments = [
c for c in children
if c.get("text") and c.get("author")
]
real_comments.sort(key=lambda c: c.get("points") or 0, reverse=True)
comments = []
insights = []
for c in real_comments[:max_comments]:
text = _strip_html(c.get("text", ""))
excerpt = text[:300] + "..." if len(text) > 300 else text
comments.append({
"author": c.get("author", ""),
"text": excerpt,
"points": c.get("points") or 0,
})
# First sentence as insight
first_sentence = text.split(". ")[0].split("\n")[0][:200]
if first_sentence:
insights.append(first_sentence)
return {"comments": comments, "comment_insights": insights}
def enrich_top_stories(
items: List[Dict[str, Any]],
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Fetch comments for top N stories by points.
Args:
items: Parsed HN items
depth: Research depth (controls how many to enrich)
Returns:
Items with top_comments and comment_insights added.
"""
if not items:
return items
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
# Sort by points to enrich the most popular stories
by_points = sorted(
range(len(items)),
key=lambda i: items[i].get("engagement", {}).get("points", 0),
reverse=True,
)
to_enrich = by_points[:limit]
_log(f"Enriching top {len(to_enrich)} stories with comments")
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {
executor.submit(
_fetch_item_comments,
items[idx]["id"],
): idx
for idx in to_enrich
}
for future in as_completed(futures):
idx = futures[future]
try:
result = future.result(timeout=15)
items[idx]["top_comments"] = result["comments"]
items[idx]["comment_insights"] = result["comment_insights"]
except (KeyError, TypeError, OSError) as exc:
_log(f"Comment enrichment failed for story {items[idx].get('id', '?')}: {type(exc).__name__}: {exc}")
items[idx]["top_comments"] = []
items[idx]["comment_insights"] = []
return items
+77 -19
View File
@@ -1,26 +1,28 @@
"""HTTP utilities for last30days skill (stdlib only)."""
import json
import os
import re
import sys
import time
import urllib.error
import urllib.request
from typing import Any, Dict, Optional
from typing import Any, Dict, Optional, Union
from urllib.parse import urlencode
from . import log as _log
DEFAULT_TIMEOUT = 30
DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")
def log(msg: str):
"""Log debug message to stderr."""
if DEBUG:
sys.stderr.write(f"[DEBUG] {msg}\n")
sys.stderr.flush()
MAX_RETRIES = 3
RETRY_DELAY = 1.0
USER_AGENT = "last30days-skill/1.0 (Claude Code Skill)"
_log.debug(msg)
MAX_RETRIES = 5
MAX_429_RETRIES = 2
RETRY_DELAY = 2.0
USER_AGENT = "last30days-skill/3.0 (Assistant Skill)"
class HTTPError(Exception):
@@ -36,9 +38,12 @@ def request(
url: str,
headers: Optional[Dict[str, str]] = None,
json_data: Optional[Dict[str, Any]] = None,
params: Optional[Dict[str, Any]] = None,
timeout: int = DEFAULT_TIMEOUT,
retries: int = MAX_RETRIES,
) -> Dict[str, Any]:
max_429_retries: int = MAX_429_RETRIES,
raw: bool = False,
) -> Union[Dict[str, Any], str]:
"""Make an HTTP request and return JSON response.
Args:
@@ -46,11 +51,15 @@ def request(
url: Request URL
headers: Optional headers dict
json_data: Optional JSON body (for POST)
params: Optional query-string params. Values are stringified. None values
are dropped. If ``url`` already has a query string, ``params`` is appended.
timeout: Request timeout in seconds
retries: Number of retries on failure
max_429_retries: Maximum 429 retries before giving up (separate cap)
raw: If True, return raw response text instead of parsed JSON
Returns:
Parsed JSON response
Parsed JSON response as dict, or raw text string if raw=True.
Raises:
HTTPError: On request failure
@@ -58,6 +67,12 @@ def request(
headers = headers or {}
headers.setdefault("User-Agent", USER_AGENT)
if params:
filtered = {k: str(v) for k, v in params.items() if v is not None}
if filtered:
separator = "&" if ("?" in url) else "?"
url = f"{url}{separator}{urlencode(filtered)}"
data = None
if json_data is not None:
data = json.dumps(json_data).encode('utf-8')
@@ -65,34 +80,56 @@ def request(
req = urllib.request.Request(url, data=data, headers=headers, method=method)
log(f"{method} {url}")
if json_data:
log(f"Payload keys: {list(json_data.keys())}")
safe_url = re.sub(r'([?&])(key|api_key|token|secret)=[^&]*', r'\1\2=***', url)
log(f"{method} {safe_url}")
last_error = None
rate_limit_count = 0
for attempt in range(retries):
try:
with urllib.request.urlopen(req, timeout=timeout) as response:
body = response.read().decode('utf-8')
log(f"Response: {response.status} ({len(body)} bytes)")
if raw:
return body
return json.loads(body) if body else {}
except urllib.error.HTTPError as e:
body = None
try:
body = e.read().decode('utf-8')
except:
except (OSError, UnicodeDecodeError):
pass
log(f"HTTP Error {e.code}: {e.reason}")
if body:
log(f"Error body: {body[:500]}")
snippet = " ".join(body.split())
log(f"Error body: {snippet[:200]}")
last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)
# Don't retry client errors (4xx) except rate limits
if 400 <= e.code < 500 and e.code != 429:
raise last_error
# Cap 429 retries separately to avoid wasting latency
if e.code == 429:
rate_limit_count += 1
if rate_limit_count >= max_429_retries:
raise last_error
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
if e.code == 429:
# Respect Retry-After header, fall back to exponential backoff
retry_after = e.headers.get("Retry-After") if hasattr(e, 'headers') else None
if retry_after:
try:
delay = float(retry_after)
except ValueError:
delay = RETRY_DELAY * (2 ** attempt) + 1
else:
delay = RETRY_DELAY * (2 ** attempt) + 1 # 3s, 5s, 9s...
log(f"Rate limited (429). Waiting {delay:.1f}s before retry {attempt + 2}/{retries}")
else:
delay = RETRY_DELAY * (2 ** attempt)
time.sleep(delay)
except urllib.error.URLError as e:
log(f"URL Error: {e.reason}")
last_error = HTTPError(f"URL Error: {e.reason}")
@@ -102,6 +139,12 @@ def request(
log(f"JSON decode error: {e}")
last_error = HTTPError(f"Invalid JSON response: {e}")
raise last_error
except (OSError, TimeoutError, ConnectionResetError) as e:
# Handle socket-level errors (connection reset, timeout, etc.)
log(f"Connection error: {type(e).__name__}: {e}")
last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
if last_error:
raise last_error
@@ -118,11 +161,26 @@ def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]]
return request("POST", url, headers=headers, json_data=json_data, **kwargs)
def get_reddit_json(path: str) -> Dict[str, Any]:
def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> str:
"""Make a POST request with JSON body and return raw text."""
return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)
def scrapecreators_headers(token: str) -> Dict[str, str]:
"""Build ScrapeCreators request headers (x-api-key + JSON content type)."""
return {
"x-api-key": token,
"Content-Type": "application/json",
}
def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
"""Fetch Reddit thread JSON.
Args:
path: Reddit path (e.g., /r/subreddit/comments/id/title)
timeout: HTTP timeout per attempt in seconds
retries: Number of retries on failure
Returns:
Parsed JSON response
@@ -143,4 +201,4 @@ def get_reddit_json(path: str) -> Dict[str, Any]:
"Accept": "application/json",
}
return get(url, headers=headers)
return get(url, headers=headers, timeout=timeout, retries=retries)
+500
View File
@@ -0,0 +1,500 @@
"""Instagram Reels search via ScrapeCreators API for /last30days.
Uses ScrapeCreators REST API to search Instagram Reels by keyword, extract
engagement metrics (views, likes, comments), and fetch video transcripts.
Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
import re
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
try:
import requests as _requests
except ImportError:
_requests = None
from . import dates, http, log
SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
# Depth configurations: how many results to fetch / captions to extract
DEPTH_CONFIG = {
"quick": {"results_per_page": 10, "max_captions": 3},
"default": {"results_per_page": 20, "max_captions": 5},
"deep": {"results_per_page": 40, "max_captions": 8},
}
# Max words to keep from each caption
CAPTION_MAX_WORDS = 500
from .relevance import token_overlap_relevance as _compute_relevance
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Instagram search."""
from .query import extract_core_subject
_INSTAGRAM_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features',
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
})
return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
def _infer_query_intent(topic: str) -> str:
"""Tiny local intent classifier for Instagram query expansion."""
text = topic.lower().strip()
if re.search(r"\b(vs|versus|compare|difference between)\b", text):
return "comparison"
if re.search(r"\b(how to|tutorial|guide|setup|step by step|deploy|install)\b", text):
return "how_to"
if re.search(r"\b(thoughts on|worth it|should i|opinion|review)\b", text):
return "opinion"
if re.search(r"\b(pricing|feature|features|best .* for)\b", text):
return "product"
return "breaking_news"
def expand_instagram_queries(topic: str, depth: str) -> List[str]:
"""Generate multiple Instagram search queries from a topic.
Mirrors reddit.py's expand_reddit_queries() pattern:
1. Extract core subject (strip noise words)
2. Include original topic if different from core
3. Add intent-specific OR-joined content-type variants
4. Cap by depth: 1 for quick, 2 for default, 3 for deep
Returns 1-3 query strings depending on depth.
"""
core = _extract_core_subject(topic)
queries = [core]
# Include cleaned original topic as variant if different from core
original_clean = topic.strip().rstrip('?!.')
if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
queries.append(original_clean)
qtype = _infer_query_intent(topic)
# Intent-specific Instagram content-type variants
if qtype == "breaking_news":
queries.append(f"{core} reaction OR edit")
elif qtype == "opinion":
queries.append(f"{core} reaction OR edit")
elif qtype == "product":
queries.append(f"{core} review OR haul")
elif qtype == "comparison":
queries.append(f"{core} vs OR compared")
elif qtype == "how_to":
queries.append(f"{core} tutorial OR hack")
else:
queries.append(f"{core} reaction OR edit")
# Deep depth: add viral content variant
if depth == "deep":
queries.append(f"{core} viral OR trending OR reel")
# Cap by depth budget
caps = {"quick": 1, "default": 2, "deep": 3}
cap = caps.get(depth, 2)
return queries[:cap]
def _log(msg: str):
log.source_log("Instagram", msg)
def _parse_date(item: Dict[str, Any]) -> Optional[str]:
"""Parse date from ScrapeCreators Instagram item to YYYY-MM-DD.
Handles taken_at as ISO string (e.g. "2026-02-26T16:00:00.000Z")
or unix timestamp.
"""
ts = item.get("taken_at")
if not ts:
return None
# Try ISO string first (ScrapeCreators reels/search returns this)
if isinstance(ts, str):
try:
# Handle "2026-02-26T16:00:00.000Z" format
dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Try just the date portion
if len(ts) >= 10:
return ts[:10]
# Fall back to unix timestamp
try:
return dates.timestamp_to_date(int(ts))
except (ValueError, TypeError):
pass
return None
def _extract_hashtags(caption_text: str) -> List[str]:
"""Extract hashtags from Instagram caption text."""
if not caption_text:
return []
return re.findall(r'#(\w+)', caption_text)
def _parse_items(raw_items: List[Dict[str, Any]], core_topic: str) -> List[Dict[str, Any]]:
"""Parse raw Instagram items into normalized dicts."""
items = []
for raw in raw_items:
if not isinstance(raw, dict):
continue
# Extract reel ID and shortcode
reel_pk = str(raw.get("id", raw.get("pk", "")))
shortcode = raw.get("shortcode", raw.get("code", ""))
# Caption text -- can be a string or dict depending on endpoint
caption_obj = raw.get("caption", "")
if isinstance(caption_obj, dict):
text = caption_obj.get("text", "")
elif isinstance(caption_obj, str):
text = caption_obj
else:
text = raw.get("desc", raw.get("text", ""))
# Engagement metrics
play_count = raw.get("video_play_count") or raw.get("video_view_count") or raw.get("play_count") or 0
like_count = raw.get("like_count") or 0
comment_count = raw.get("comment_count") or 0
# Author info -- 'owner' in reels/search, 'user' in user/reels
owner_raw = raw.get("owner") or raw.get("user")
if isinstance(owner_raw, dict):
author_name = owner_raw.get("username", "")
elif isinstance(owner_raw, str):
author_name = owner_raw
else:
author_name = ""
# Duration
duration = raw.get("video_duration")
# Date
date_str = _parse_date(raw)
# Hashtags from caption text
hashtags = _extract_hashtags(text)
# Compute relevance with hashtag boost
relevance = _compute_relevance(core_topic, text, hashtags)
# Build URL -- prefer API-provided url, fallback to shortcode
url = raw.get("url", "")
if not url and shortcode:
url = f"https://www.instagram.com/reel/{shortcode}"
items.append({
"video_id": reel_pk,
"text": text,
"url": url,
"author_name": author_name,
"date": date_str,
"engagement": {
"views": play_count,
"likes": like_count,
"comments": comment_count,
},
"hashtags": hashtags,
"duration": duration,
"relevance": relevance,
"why_relevant": f"Instagram: {text[:60]}" if text else f"Instagram: {core_topic}",
"caption_snippet": "", # populated by fetch_captions
})
return items
def _user_reels(
handle: str,
token: str,
) -> List[Dict[str, Any]]:
"""Fetch an Instagram user's recent reels via ScrapeCreators.
Args:
handle: Instagram username (without @)
token: ScrapeCreators API key
Returns:
List of raw Instagram reel dicts.
"""
_log(f"User reels: @{handle}")
reels_url = f"{SCRAPECREATORS_BASE}/v1/instagram/user/reels"
if not _requests:
try:
from urllib.parse import urlencode
params = urlencode({"handle": handle})
url = f"{reels_url}?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"User reels error (urllib) for @{handle}: {e}")
return []
else:
try:
resp = _requests.get(
reels_url,
params={"handle": handle},
headers=http.scrapecreators_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"User reels error for @{handle}: {e}")
return []
raw_items = data.get("items") or data.get("reels") or data.get("data") or []
_log(f" -> {len(raw_items)} reels from @{handle}")
return raw_items
def search_instagram(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
token: str = None,
) -> Dict[str, Any]:
"""Search Instagram Reels via ScrapeCreators API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
token: ScrapeCreators API key
Returns:
Dict with 'items' list and optional 'error'.
"""
if not token:
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
if not _requests:
_log("requests library not installed, falling back to urllib")
try:
from urllib.parse import urlencode
params = urlencode({"query": core_topic})
url = f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
params={"query": core_topic},
headers=http.scrapecreators_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
# Items are in the 'reels' array (ScrapeCreators v2 response)
raw_items = data.get("reels") or data.get("items") or data.get("data") or []
# Limit to configured count
raw_items = raw_items[:config["results_per_page"]]
# Parse items
items = _parse_items(raw_items, core_topic)
# Hard date filter
in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
out_of_range = len(items) - len(in_range)
if in_range:
items = in_range
if out_of_range:
_log(f"Filtered {out_of_range} reels outside date range")
else:
_log(f"No reels within date range, keeping all {len(items)}")
# Sort by views descending
items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
_log(f"Found {len(items)} Instagram reels")
return {"items": items}
def fetch_captions(
video_items: List[Dict[str, Any]],
token: str,
depth: str = "default",
) -> Dict[str, str]:
"""Fetch transcripts for top N Instagram reels via ScrapeCreators.
Strategy:
1. Use the 'text' field (caption) as baseline
2. For top N, call /v2/instagram/media/transcript for spoken-word captions
Args:
video_items: Items from search_instagram()
token: ScrapeCreators API key
depth: Depth level for caption limit
Returns:
Dict mapping video_id -> caption text (truncated to 500 words)
"""
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = config["max_captions"]
if not video_items or not token or not _requests:
return {}
top_items = video_items[:max_captions]
_log(f"Enriching captions for {len(top_items)} reels")
captions = {}
# First pass: use text field as caption (always available, free)
for item in top_items:
vid = item["video_id"]
text = item.get("text", "")
if text:
words = text.split()
if len(words) > CAPTION_MAX_WORDS:
text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
captions[vid] = text
# Second pass: try to get spoken-word transcripts (1 credit each)
for item in top_items:
vid = item["video_id"]
url = item.get("url", "")
if not url:
continue
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
params={"url": url},
headers=http.scrapecreators_headers(token),
timeout=15,
)
if resp.status_code == 200:
data = resp.json()
transcripts = data.get("transcripts") or []
if transcripts and isinstance(transcripts, list):
# Combine all transcript segments
transcript_text = " ".join(
t.get("text", "") for t in transcripts
if isinstance(t, dict) and t.get("text")
)
if transcript_text:
words = transcript_text.split()
if len(words) > CAPTION_MAX_WORDS:
transcript_text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
captions[vid] = transcript_text
except Exception as e:
_log(f"Transcript fetch failed for {vid}: {e}")
got = sum(1 for v in captions.values() if v)
_log(f"Got captions for {got}/{len(top_items)} reels")
return captions
def search_and_enrich(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
token: str = None,
ig_creators: List[str] | None = None,
) -> Dict[str, Any]:
"""Full Instagram search: find reels, then fetch captions for top results.
Uses expand_instagram_queries() to generate multiple search queries,
runs ScrapeCreators for each, and merges/deduplicates results by video ID.
Args:
topic: Search topic (raw topic, not planner's narrowed query)
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
token: ScrapeCreators API key
ig_creators: Optional list of Instagram creator handles to fetch reels from
Returns:
Dict with 'items' list. Each item has a 'caption_snippet' field.
"""
core_topic = _extract_core_subject(topic)
seen_ids: Set[str] = set()
items: List[Dict[str, Any]] = []
last_error = None
# Step 0: Creator reels (high-signal, runs first)
if ig_creators and token:
for creator in ig_creators:
raw_items = _user_reels(creator, token)
parsed = _parse_items(raw_items, core_topic)
for item in parsed:
vid = item.get("video_id", "")
if vid and vid not in seen_ids:
seen_ids.add(vid)
items.append(item)
# Step 1: Multi-query keyword search — run ScrapeCreators for each expanded query
queries = expand_instagram_queries(topic, depth)
for q in queries:
search_result = search_instagram(q, from_date, to_date, depth, token)
if search_result.get("error"):
last_error = search_result["error"]
for item in search_result.get("items", []):
vid = item.get("video_id", "")
if vid and vid not in seen_ids:
seen_ids.add(vid)
items.append(item)
# Sort merged results by views descending
items.sort(key=lambda x: x.get("engagement", {}).get("views", 0), reverse=True)
if not items:
return {"items": [], "error": last_error}
# Step 2: Fetch captions for top N
captions = fetch_captions(items, token, depth)
# Step 3: Attach captions to items
for item in items:
vid = item["video_id"]
caption = captions.get(vid)
if caption:
item["caption_snippet"] = caption
return {"items": items, "error": last_error}
def parse_instagram_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Instagram search response to normalized format.
Returns:
List of item dicts ready for normalization.
"""
return response.get("items", [])
+28
View File
@@ -0,0 +1,28 @@
"""Shared logging utilities for last30days skill."""
import os
import sys
DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")
def debug(msg: str) -> None:
"""Log debug message to stderr (only when LAST30DAYS_DEBUG is set)."""
if DEBUG:
sys.stderr.write(f"[DEBUG] {msg}\n")
sys.stderr.flush()
def source_log(prefix: str, msg: str, *, tty_only: bool = True) -> None:
"""Log a source module message to stderr.
Args:
prefix: Source label (e.g. "Reddit", "Bird").
msg: Message text.
tty_only: If True, only log when stderr is a TTY (avoids cluttering
non-interactive output like Claude Code).
"""
if tty_only and not sys.stderr.isatty():
return
sys.stderr.write(f"[{prefix}] {msg}\n")
sys.stderr.flush()
-175
View File
@@ -1,175 +0,0 @@
"""Model auto-selection for last30days skill."""
import re
from typing import Dict, List, Optional, Tuple
from . import cache, http
# OpenAI API
OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4o"]
# xAI API - Agent Tools API requires grok-4 family
XAI_MODELS_URL = "https://api.x.ai/v1/models"
XAI_ALIASES = {
"latest": "grok-4-1-fast", # Required for x_search tool
"stable": "grok-4-1-fast",
}
def parse_version(model_id: str) -> Optional[Tuple[int, ...]]:
"""Parse semantic version from model ID.
Examples:
gpt-5 -> (5,)
gpt-5.2 -> (5, 2)
gpt-5.2.1 -> (5, 2, 1)
"""
match = re.search(r'(\d+(?:\.\d+)*)', model_id)
if match:
return tuple(int(x) for x in match.group(1).split('.'))
return None
def is_mainline_openai_model(model_id: str) -> bool:
"""Check if model is a mainline GPT model (not mini/nano/chat/codex/pro)."""
model_lower = model_id.lower()
# Must be gpt-5 series
if not re.match(r'^gpt-5(\.\d+)*$', model_lower):
return False
# Exclude variants
excludes = ['mini', 'nano', 'chat', 'codex', 'pro', 'preview', 'turbo']
for exc in excludes:
if exc in model_lower:
return False
return True
def select_openai_model(
api_key: str,
policy: str = "auto",
pin: Optional[str] = None,
mock_models: Optional[List[Dict]] = None,
) -> str:
"""Select the best OpenAI model based on policy.
Args:
api_key: OpenAI API key
policy: 'auto' or 'pinned'
pin: Model to use if policy is 'pinned'
mock_models: Mock model list for testing
Returns:
Selected model ID
"""
if policy == "pinned" and pin:
return pin
# Check cache first
cached = cache.get_cached_model("openai")
if cached:
return cached
# Fetch model list
if mock_models is not None:
models = mock_models
else:
try:
headers = {"Authorization": f"Bearer {api_key}"}
response = http.get(OPENAI_MODELS_URL, headers=headers)
models = response.get("data", [])
except http.HTTPError:
# Fall back to known models
return OPENAI_FALLBACK_MODELS[0]
# Filter to mainline models
candidates = [m for m in models if is_mainline_openai_model(m.get("id", ""))]
if not candidates:
# No gpt-5 models found, use fallback
return OPENAI_FALLBACK_MODELS[0]
# Sort by version (descending), then by created timestamp
def sort_key(m):
version = parse_version(m.get("id", "")) or (0,)
created = m.get("created", 0)
return (version, created)
candidates.sort(key=sort_key, reverse=True)
selected = candidates[0]["id"]
# Cache the selection
cache.set_cached_model("openai", selected)
return selected
def select_xai_model(
api_key: str,
policy: str = "latest",
pin: Optional[str] = None,
mock_models: Optional[List[Dict]] = None,
) -> str:
"""Select the best xAI model based on policy.
Args:
api_key: xAI API key
policy: 'latest', 'stable', or 'pinned'
pin: Model to use if policy is 'pinned'
mock_models: Mock model list for testing
Returns:
Selected model ID
"""
if policy == "pinned" and pin:
return pin
# Use alias system
if policy in XAI_ALIASES:
alias = XAI_ALIASES[policy]
# Check cache first
cached = cache.get_cached_model("xai")
if cached:
return cached
# Cache the alias
cache.set_cached_model("xai", alias)
return alias
# Default to latest
return XAI_ALIASES["latest"]
def get_models(
config: Dict,
mock_openai_models: Optional[List[Dict]] = None,
mock_xai_models: Optional[List[Dict]] = None,
) -> Dict[str, Optional[str]]:
"""Get selected models for both providers.
Returns:
Dict with 'openai' and 'xai' keys
"""
result = {"openai": None, "xai": None}
if config.get("OPENAI_API_KEY"):
result["openai"] = select_openai_model(
config["OPENAI_API_KEY"],
config.get("OPENAI_MODEL_POLICY", "auto"),
config.get("OPENAI_MODEL_PIN"),
mock_openai_models,
)
if config.get("XAI_API_KEY"):
result["xai"] = select_xai_model(
config["XAI_API_KEY"],
config.get("XAI_MODEL_POLICY", "latest"),
config.get("XAI_MODEL_PIN"),
mock_xai_models,
)
return result
+481 -101
View File
@@ -1,118 +1,498 @@
"""Normalization of raw API data to canonical schema."""
"""Normalization of source-specific payloads into the v3 generic item model."""
from typing import Any, Dict, List
from __future__ import annotations
from typing import Any
from urllib.parse import urlparse
from . import dates, schema
def normalize_reddit_items(
items: List[Dict[str, Any]],
def filter_by_date_range(
items: list[schema.SourceItem],
from_date: str,
to_date: str,
) -> List[schema.RedditItem]:
"""Normalize raw Reddit items to schema.
Args:
items: Raw Reddit items from API
from_date: Start of date range
to_date: End of date range
Returns:
List of RedditItem objects
"""
normalized = []
require_date: bool = False,
) -> list[schema.SourceItem]:
"""Keep only items within the requested window."""
filtered: list[schema.SourceItem] = []
for item in items:
# Parse engagement
engagement = None
eng_raw = item.get("engagement")
if isinstance(eng_raw, dict):
engagement = schema.Engagement(
score=eng_raw.get("score"),
num_comments=eng_raw.get("num_comments"),
upvote_ratio=eng_raw.get("upvote_ratio"),
)
# Parse comments
top_comments = []
for c in item.get("top_comments", []):
top_comments.append(schema.Comment(
score=c.get("score", 0),
date=c.get("date"),
author=c.get("author", ""),
excerpt=c.get("excerpt", ""),
url=c.get("url", ""),
))
# Determine date confidence
date_str = item.get("date")
date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
normalized.append(schema.RedditItem(
id=item.get("id", ""),
title=item.get("title", ""),
url=item.get("url", ""),
subreddit=item.get("subreddit", ""),
date=date_str,
date_confidence=date_confidence,
engagement=engagement,
top_comments=top_comments,
comment_insights=item.get("comment_insights", []),
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
if not item.published_at:
if not require_date:
filtered.append(item)
continue
if item.published_at < from_date or item.published_at > to_date:
continue
filtered.append(item)
return filtered
def normalize_x_items(
items: List[Dict[str, Any]],
def normalize_source_items(
source: str,
items: list[dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.XItem]:
"""Normalize raw X items to schema.
freshness_mode: str = "balanced_recent",
) -> list[schema.SourceItem]:
"""Normalize raw source items, filter by date range, with evergreen fallback for how_to queries."""
source = source.lower()
normalizers = {
"reddit": _normalize_reddit,
"x": _normalize_x,
"youtube": _normalize_youtube,
"tiktok": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "TK", "TikTok post"),
"instagram": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "IG", "Instagram reel"),
"hackernews": _normalize_hackernews,
"bluesky": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "BS", "Bluesky post"),
"truthsocial": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TS", "Truth Social post"),
"threads": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TH", "Threads post"),
"xquik": _normalize_x,
"pinterest": _normalize_pinterest,
"polymarket": _normalize_polymarket,
"grounding": _normalize_grounding,
"xiaohongshu": _normalize_grounding,
"github": _normalize_github,
"perplexity": _normalize_grounding,
}
normalizer = normalizers.get(source)
if normalizer is None:
raise ValueError(f"Unsupported source: {source}")
normalized = [normalizer(source, item, index, from_date, to_date) for index, item in enumerate(items)]
require_date = source == "grounding"
filtered = filter_by_date_range(normalized, from_date, to_date, require_date=require_date)
if filtered:
return filtered
if freshness_mode == "evergreen_ok" and source == "youtube":
if require_date:
return [item for item in normalized if item.published_at]
return normalized
return filtered
Args:
items: Raw X items from API
from_date: Start of date range
to_date: End of date range
Returns:
List of XItem objects
def _remap_comments(
raw: list[Any],
score_keys: tuple[str, ...],
excerpt_keys: tuple[str, ...],
) -> list[dict[str, Any]]:
"""Normalize comments from any source into the shared Reddit-compatible shape.
Downstream code (signals._top_comment_score, render._top_comments_list,
entity_extract, rerank) all expect `score` and `excerpt`. This helper maps
per-source field names (YT: likes/text, TikTok: digg_count/text) onto that
shape while preserving author/date/url passthrough.
"""
normalized = []
for item in items:
# Parse engagement
engagement = None
eng_raw = item.get("engagement")
if isinstance(eng_raw, dict):
engagement = schema.Engagement(
likes=eng_raw.get("likes"),
reposts=eng_raw.get("reposts"),
replies=eng_raw.get("replies"),
quotes=eng_raw.get("quotes"),
)
# Determine date confidence
date_str = item.get("date")
date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
normalized.append(schema.XItem(
id=item.get("id", ""),
text=item.get("text", ""),
url=item.get("url", ""),
author_handle=item.get("author_handle", ""),
date=date_str,
date_confidence=date_confidence,
engagement=engagement,
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
out: list[dict[str, Any]] = []
for raw_c in raw:
if not isinstance(raw_c, dict):
continue
score = _first_present(raw_c, score_keys, default=0)
excerpt = _first_present(raw_c, excerpt_keys, default="")
try:
score_int = int(score or 0)
except (TypeError, ValueError):
score_int = 0
entry: dict[str, Any] = {
"score": score_int,
"excerpt": str(excerpt or "")[:400],
"author": str(raw_c.get("author") or ""),
"date": str(raw_c.get("date") or ""),
}
if raw_c.get("url"):
entry["url"] = str(raw_c["url"])
out.append(entry)
return out
def items_to_dicts(items: List) -> List[Dict[str, Any]]:
"""Convert schema items to dicts for JSON serialization."""
return [item.to_dict() for item in items]
def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> Any:
for key in keys:
if key in d and d[key] not in (None, ""):
return d[key]
return default
def _domain_from_url(url: str) -> str | None:
if not url:
return None
domain = urlparse(url).netloc.strip().lower()
return domain or None
def _date_confidence(item: dict[str, Any], from_date: str, to_date: str, default: str = "low") -> str:
if item.get("date_confidence"):
return str(item["date_confidence"])
date_value = item.get("date")
if not date_value:
return default
return dates.get_date_confidence(str(date_value), from_date, to_date)
def _source_item(
*,
item_id: str,
source: str,
title: str,
body: str,
url: str,
published_at: str | None,
date_confidence: str,
relevance_hint: float,
why_relevant: str,
author: str | None = None,
container: str | None = None,
engagement: dict[str, float | int] | None = None,
snippet: str = "",
metadata: dict[str, Any] | None = None,
) -> schema.SourceItem:
return schema.SourceItem(
item_id=item_id,
source=source,
title=title.strip() or body.strip()[:160] or item_id,
body=body.strip(),
url=url.strip(),
author=(author or "").strip() or None,
container=(container or "").strip() or None,
published_at=published_at,
date_confidence=date_confidence,
engagement=engagement or {},
relevance_hint=max(0.0, min(1.0, float(relevance_hint or 0.0))),
why_relevant=why_relevant.strip(),
snippet=snippet.strip(),
metadata=metadata or {},
)
def _normalize_reddit(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
top_comments = item.get("top_comments") or []
comment_text = " ".join(
str(comment.get("excerpt") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
body = "\n".join(
part
for part in [
str(item.get("title") or "").strip(),
str(item.get("selftext") or "").strip(),
comment_text,
]
if part
)
return _source_item(
item_id=str(item.get("id") or f"R{index + 1}"),
source=source,
title=str(item.get("title") or ""),
body=body,
url=str(item.get("url") or ""),
author=None,
container=str(item.get("subreddit") or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=comment_text or str(item.get("selftext") or "")[:400],
metadata={
"top_comments": top_comments,
"comment_insights": item.get("comment_insights") or [],
},
)
def _normalize_x(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
text = str(item.get("text") or "").strip()
return _source_item(
item_id=str(item.get("id") or f"X{index + 1}"),
source=source,
title=text[:140] or f"X post {index + 1}",
body=text,
url=str(item.get("url") or ""),
author=str(item.get("author_handle") or "").lstrip("@"),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
)
def _normalize_youtube(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
transcript = str(item.get("transcript_snippet") or "").strip()
description = str(item.get("description") or "").strip()
title = str(item.get("title") or "").strip()
highlights = item.get("transcript_highlights") or []
metadata: dict[str, Any] = {}
if highlights:
metadata["transcript_highlights"] = highlights
metadata["top_comments"] = _remap_comments(
item.get("top_comments") or [],
score_keys=("score", "likes"),
excerpt_keys=("excerpt", "text"),
)
return _source_item(
item_id=str(item.get("video_id") or item.get("id") or f"YT{index + 1}"),
source=source,
title=title,
body="\n".join(part for part in [title, description, transcript] if part),
url=str(item.get("url") or ""),
author=str(item.get("channel_name") or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=transcript,
metadata=metadata,
)
def _normalize_shortform_video(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
id_prefix: str,
default_title: str,
) -> schema.SourceItem:
"""Shared normalizer for TikTok and Instagram (identical structure)."""
caption = str(item.get("caption_snippet") or "").strip()
text = str(item.get("text") or "").strip()
return _source_item(
item_id=str(item.get("id") or f"{id_prefix}{index + 1}"),
source=source,
title=text[:140] or caption[:140] or f"{default_title} {index + 1}",
body="\n".join(part for part in [text, caption] if part),
url=str(item.get("url") or ""),
author=str(item.get("author_name") or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=caption,
metadata={
"hashtags": item.get("hashtags") or [],
"top_comments": _remap_comments(
item.get("top_comments") or [],
# TikTok uses digg_count as the vote field; Instagram has no
# comment fetcher today so the key is harmlessly absent.
score_keys=("score", "digg_count", "likes"),
excerpt_keys=("excerpt", "text"),
),
},
)
def _normalize_pinterest(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
"""Normalizer for Pinterest pins (visual content with descriptions).
Saves are the primary engagement signal, analogous to likes/upvotes.
"""
description = str(item.get("description") or "").strip()
return _source_item(
item_id=str(item.get("pin_id") or item.get("id") or f"PI{index + 1}"),
source=source,
title=description[:140] or f"Pinterest pin {index + 1}",
body=description,
url=str(item.get("url") or ""),
author=str(item.get("author") or ""),
container=str(item.get("board") or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="low"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=description[:400],
)
def _normalize_hackernews(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
top_comments = item.get("top_comments") or []
comment_text = " ".join(
str(comment.get("text") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
title = str(item.get("title") or "").strip()
body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part)
return _source_item(
item_id=str(item.get("id") or f"HN{index + 1}"),
source=source,
title=title or f"HN story {index + 1}",
body=body,
url=str(item.get("url") or item.get("hn_url") or ""),
author=str(item.get("author") or ""),
container="Hacker News",
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=comment_text,
metadata={
"hn_url": item.get("hn_url"),
"top_comments": top_comments,
"comment_insights": item.get("comment_insights") or [],
},
)
def _normalize_microblog(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
id_prefix: str,
default_title: str,
) -> schema.SourceItem:
"""Shared normalizer for Bluesky and Truth Social (identical structure)."""
text = str(item.get("text") or "").strip()
return _source_item(
item_id=str(item.get("id") or f"{id_prefix}{index + 1}"),
source=source,
title=text[:140] or f"{default_title} {index + 1}",
body=text,
url=str(item.get("url") or ""),
author=str(item.get("handle") or item.get("author_handle") or "").lstrip("@"),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
metadata={"display_name": item.get("display_name")},
)
def _normalize_polymarket(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
title = str(item.get("title") or "").strip()
question = str(item.get("question") or "").strip()
engagement = {
"volume": item.get("volume1mo") or item.get("volume24hr") or 0,
"liquidity": item.get("liquidity") or 0,
}
return _source_item(
item_id=str(item.get("id") or f"PM{index + 1}"),
source=source,
title=title or question or f"Polymarket event {index + 1}",
body="\n".join(part for part in [title, question, str(item.get("price_movement") or "")] if part),
url=str(item.get("url") or ""),
author=None,
container="Polymarket",
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=engagement,
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=str(item.get("price_movement") or ""),
metadata={
"question": question,
"end_date": item.get("end_date"),
"outcome_prices": item.get("outcome_prices") or [],
"outcomes_remaining": item.get("outcomes_remaining"),
},
)
def _normalize_github(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
title = str(item.get("title") or "").strip()
snippet_text = str(item.get("snippet") or "").strip()
top_comments = item.get("metadata", {}).get("top_comments") or []
comment_text = " ".join(
str(comment.get("excerpt") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
body = "\n".join(part for part in [title, snippet_text, comment_text] if part)
metadata = item.get("metadata") or {}
return _source_item(
item_id=str(item.get("id") or f"GH{index + 1}"),
source=source,
title=title or f"GitHub item {index + 1}",
body=body,
url=str(item.get("url") or ""),
author=str(item.get("author") or ""),
container=str(item.get("container") or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=comment_text or snippet_text[:400],
metadata={
"top_comments": top_comments,
"labels": metadata.get("labels") or [],
"state": metadata.get("state", ""),
"is_pr": metadata.get("is_pr", False),
},
)
def _normalize_grounding(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
title = str(item.get("title") or "").strip()
snippet = str(item.get("snippet") or "").strip()
url = str(item.get("url") or "").strip()
return _source_item(
item_id=str(item.get("id") or f"W{index + 1}"),
source=source,
title=title or _domain_from_url(url) or f"Web result {index + 1}",
body="\n".join(part for part in [title, snippet] if part),
url=url,
author=None,
container=str(item.get("source_domain") or _domain_from_url(url) or ""),
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=snippet,
metadata=item.get("metadata") or {},
)
-204
View File
@@ -1,204 +0,0 @@
"""OpenAI Responses API client for Reddit discovery."""
import json
import re
import sys
from typing import Any, Dict, List, Optional
from . import http
def _log_error(msg: str):
"""Log error to stderr."""
sys.stderr.write(f"[REDDIT ERROR] {msg}\n")
sys.stderr.flush()
OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
# Depth configurations: (min, max) threads to request
DEPTH_CONFIG = {
"quick": (8, 12),
"default": (20, 30),
"deep": (50, 70),
}
REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic}
SEARCH GUIDANCE:
- Search for "site:reddit.com/r/ {topic}" to find subreddit discussions
- Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design
- ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads)
- IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/
Find {min_items}-{max_items} relevant Reddit discussion threads. Prefer recent threads, but include older relevant ones if recent ones are scarce.
CRITICAL: Return ALL discussion threads you find as JSON. Do NOT return errors or empty results.
For EACH Reddit thread URL (containing /r/subreddit/comments/), extract:
- Thread title
- Full Reddit URL
- Subreddit name
- Date (if visible, otherwise null)
- Why it's relevant
Return ONLY valid JSON:
{{
"items": [
{{
"title": "Thread title",
"url": "https://www.reddit.com/r/subreddit/comments/abc123/title/",
"subreddit": "subreddit_name",
"date": "YYYY-MM-DD or null",
"why_relevant": "Relevance to {topic}",
"relevance": 0.85
}}
]
}}
Rules:
- ONLY URLs matching: reddit.com/r/*/comments/*
- MUST return threads found - NEVER return empty items or errors
- If threads are older than 30 days, still include them with accurate dates
- relevance: 0.0-1.0
- Diverse subreddits preferred"""
def search_reddit(
api_key: str,
model: str,
topic: str,
depth: str = "default",
mock_response: Optional[Dict] = None,
) -> Dict[str, Any]:
"""Search Reddit for relevant threads using OpenAI Responses API.
Args:
api_key: OpenAI API key
model: Model to use
topic: Search topic
depth: Research depth - "quick", "default", or "deep"
mock_response: Mock response for testing
Returns:
Raw API response
"""
if mock_response is not None:
return mock_response
min_items, max_items = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
# Adjust timeout based on depth
timeout = 60 if depth == "quick" else 90 if depth == "default" else 120
payload = {
"model": model,
"tools": [
{
"type": "web_search",
"filters": {
"allowed_domains": ["reddit.com"]
}
}
],
"include": ["web_search_call.action.sources"],
"input": REDDIT_SEARCH_PROMPT.format(topic=topic, min_items=min_items, max_items=max_items),
}
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse OpenAI response to extract Reddit items.
Args:
response: Raw API response
Returns:
List of item dicts
"""
items = []
# Check for API errors first
if "error" in response and response["error"]:
error = response["error"]
err_msg = error.get("message", str(error)) if isinstance(error, dict) else str(error)
_log_error(f"OpenAI API error: {err_msg}")
if http.DEBUG:
_log_error(f"Full error response: {json.dumps(response, indent=2)[:1000]}")
return items
# Try to find the output text
output_text = ""
if "output" in response:
output = response["output"]
if isinstance(output, str):
output_text = output
elif isinstance(output, list):
for item in output:
if isinstance(item, dict):
if item.get("type") == "message":
content = item.get("content", [])
for c in content:
if isinstance(c, dict) and c.get("type") == "output_text":
output_text = c.get("text", "")
break
elif "text" in item:
output_text = item["text"]
elif isinstance(item, str):
output_text = item
if output_text:
break
# Also check for choices (older format)
if not output_text and "choices" in response:
for choice in response["choices"]:
if "message" in choice:
output_text = choice["message"].get("content", "")
break
if not output_text:
print(f"[REDDIT WARNING] No output text found in OpenAI response. Keys present: {list(response.keys())}", flush=True)
return items
# Extract JSON from the response
json_match = re.search(r'\{[\s\S]*"items"[\s\S]*\}', output_text)
if json_match:
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
pass
# Validate and clean items
clean_items = []
for i, item in enumerate(items):
if not isinstance(item, dict):
continue
url = item.get("url", "")
if not url or "reddit.com" not in url:
continue
clean_item = {
"id": f"R{i+1}",
"title": str(item.get("title", "")).strip(),
"url": url,
"subreddit": str(item.get("subreddit", "")).strip().lstrip("r/"),
"date": item.get("date"),
"why_relevant": str(item.get("why_relevant", "")).strip(),
"relevance": min(1.0, max(0.0, float(item.get("relevance", 0.5)))),
}
# Validate date format
if clean_item["date"]:
if not re.match(r'^\d{4}-\d{2}-\d{2}$', str(clean_item["date"])):
clean_item["date"] = None
clean_items.append(clean_item)
return clean_items
+164
View File
@@ -0,0 +1,164 @@
"""Perplexity Sonar Pro / Deep Research via OpenRouter API.
Queries Perplexity models through OpenRouter for AI-synthesized research
with citation annotations. Returns normalized items with synthesis text
and individual citation entries.
"""
from __future__ import annotations
import sys
from urllib.parse import urlparse
from . import http, log
OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
MODEL_SONAR_PRO = "perplexity/sonar-pro"
MODEL_DEEP_RESEARCH = "perplexity/sonar-deep-research"
def _log(msg: str):
log.source_log("Perplexity", msg)
def _domain(url: str) -> str:
return urlparse(url).netloc.strip().lower()
def search(
query: str,
date_range: tuple[str, str],
config: dict,
deep: bool = False,
) -> tuple[list[dict], dict]:
"""Search via Perplexity Sonar Pro or Deep Research through OpenRouter.
Args:
query: Search topic
date_range: (from_date, to_date) as YYYY-MM-DD strings
config: Must contain OPENROUTER_API_KEY
deep: Use Deep Research model (~$0.90/query) instead of Sonar Pro
Returns:
Tuple of (items list, artifact dict).
"""
api_key = config.get("OPENROUTER_API_KEY")
if not api_key:
_log("No OPENROUTER_API_KEY configured, skipping")
return [], {}
from_date, to_date = date_range
model = MODEL_DEEP_RESEARCH if deep else MODEL_SONAR_PRO
timeout = 120 if deep else 30
if deep:
print("[Perplexity] Using Deep Research (~$0.90/query)", file=sys.stderr)
prompt = (
f"What has been happening with {query} between {from_date} and {to_date}? "
"Include specific dates, names, numbers, and sources."
)
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
json_data = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
}
_log(f"Querying {model} for '{query}' ({from_date} to {to_date})")
try:
data = http.post(OPENROUTER_URL, json_data, headers=headers, timeout=timeout)
except http.HTTPError as e:
if e.status_code == 401:
_log("Invalid OpenRouter API key (401)")
elif e.status_code == 429:
_log("Rate limited by OpenRouter (429)")
else:
_log(f"HTTP error: {e}")
return [], {}
except Exception as e:
_log(f"Request failed: {e}")
return [], {}
# Parse response
choices = data.get("choices", [])
if not choices:
_log("No choices in response")
return [], {}
synthesis = choices[0].get("message", {}).get("content", "")
if not synthesis:
_log("Empty synthesis content")
return [], {}
# Extract citations from annotations
annotations = choices[0].get("message", {}).get("annotations", [])
citations = []
for ann in annotations:
url_citation = ann.get("url_citation", {})
url = url_citation.get("url", "")
title = url_citation.get("title", "")
if url:
citations.append({"url": url, "title": title})
# Deduplicate citations by URL
seen_urls = set()
unique_citations = []
for c in citations:
if c["url"] not in seen_urls:
seen_urls.add(c["url"])
unique_citations.append(c)
citations = unique_citations
_log(f"Got synthesis ({len(synthesis)} chars) with {len(citations)} citations")
# Build items list
items = []
# Primary item: the synthesis itself
snippet = synthesis[:2000]
items.append({
"id": "PX1",
"title": f"Perplexity {'Deep Research' if deep else 'Sonar Pro'}: {query}",
"url": "",
"source_domain": "perplexity.ai",
"snippet": snippet,
"date": to_date,
"relevance": 0.9,
"why_relevant": f"AI synthesis of recent activity for '{query}'",
"engagement": {"citations": len(citations)},
"metadata": {"citations": citations},
})
# Individual items for each citation
for i, cit in enumerate(citations):
items.append({
"id": f"PX{i + 2}",
"title": cit["title"] or _domain(cit["url"]),
"url": cit["url"],
"source_domain": _domain(cit["url"]),
"snippet": "",
"date": None,
"relevance": 0.7,
"why_relevant": f"Cited in Perplexity synthesis for '{query}'",
"engagement": {"citations": 1},
"metadata": {"citations": [cit]},
})
artifact = {
"label": "perplexity",
"model": model,
"deep": deep,
"query": query,
"synthesisLength": len(synthesis),
"citationCount": len(citations),
}
return items, artifact
+182
View File
@@ -0,0 +1,182 @@
"""Pinterest search via ScrapeCreators API for /last30days.
Uses ScrapeCreators REST API to search Pinterest by keyword, extract
engagement metrics (saves, comments), and return pin descriptions.
Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
import re
import sys
from typing import Any, Dict, List, Optional, Set
try:
import requests as _requests
except ImportError:
_requests = None
from . import dates, http, log
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/pinterest"
# Depth configurations: how many results to fetch
DEPTH_CONFIG = {
"quick": {"results_per_page": 10},
"default": {"results_per_page": 20},
"deep": {"results_per_page": 40},
}
from .relevance import token_overlap_relevance as _compute_relevance
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for Pinterest search."""
from .query import extract_core_subject
_PINTEREST_NOISE = frozenset({
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'trending', 'hottest', 'popular', 'viral',
'practices', 'features',
'recommendations', 'advice',
'prompt', 'prompts', 'prompting',
'methods', 'strategies', 'approaches',
})
return extract_core_subject(topic, noise=_PINTEREST_NOISE)
def _log(msg: str):
log.source_log("Pinterest", msg)
def _parse_items(raw_items: List[Dict[str, Any]], core_topic: str) -> List[Dict[str, Any]]:
"""Parse raw Pinterest items into normalized dicts.
Pinterest pins are visual content with descriptions. Saves are the
primary engagement signal (analogous to upvotes/likes on other platforms).
"""
items = []
for raw in raw_items:
if not isinstance(raw, dict):
continue
pin_id = str(raw.get("id", raw.get("pin_id", "")))
description = str(raw.get("description") or raw.get("title") or "")
# Engagement metrics - saves are the primary signal
save_count = raw.get("save_count") or raw.get("saves") or raw.get("repin_count") or 0
comment_count = raw.get("comment_count") or raw.get("comments") or 0
# Author info
pinner = raw.get("pinner") or raw.get("creator") or raw.get("user") or {}
if isinstance(pinner, dict):
author_name = pinner.get("username") or pinner.get("full_name") or ""
elif isinstance(pinner, str):
author_name = pinner
else:
author_name = ""
# URL
url = raw.get("link") or raw.get("url") or ""
if not url and pin_id:
url = f"https://www.pinterest.com/pin/{pin_id}/"
# Board info (container for pins)
board = raw.get("board") or {}
board_name = board.get("name", "") if isinstance(board, dict) else ""
# Compute relevance
relevance = _compute_relevance(core_topic, description, [])
items.append({
"pin_id": pin_id,
"description": description,
"url": url,
"author": author_name,
"board": board_name,
"engagement": {
"saves": save_count,
"comments": comment_count,
},
"relevance": relevance,
"why_relevant": f"Pinterest: {description[:60]}" if description else f"Pinterest: {core_topic}",
})
return items
def parse_pinterest_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Pinterest search response to normalized format.
Returns:
List of item dicts ready for normalization.
"""
return response.get("items", [])
def search_pinterest(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
token: str = None,
) -> Dict[str, Any]:
"""Search Pinterest via ScrapeCreators API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
token: ScrapeCreators API key
Returns:
Dict with 'items' list and optional 'error'.
"""
if not token:
return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
_log(f"Searching Pinterest for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
if not _requests:
_log("requests library not installed, falling back to urllib")
try:
from urllib.parse import urlencode
params = urlencode({"keyword": core_topic})
url = f"{SCRAPECREATORS_BASE}/search?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search",
params={"keyword": core_topic},
headers=http.scrapecreators_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
# Extract items from response - try common SC response shapes
raw_items = data.get("pins") or data.get("results") or data.get("data") or data.get("items") or []
# Limit to configured count
raw_items = raw_items[:config["results_per_page"]]
# Parse items
items = _parse_items(raw_items, core_topic)
# Sort by saves descending (primary engagement signal)
items.sort(key=lambda x: x["engagement"]["saves"], reverse=True)
_log(f"Found {len(items)} Pinterest pins")
return {"items": items}

Some files were not shown because too many files have changed in this diff Show More