Compare commits

...

117 Commits

Author SHA1 Message Date
Trevin Chow f3df47c381 chore: remove orphaned SPEC.md and TASKS.md
Both files lived at the repo root as pre-plugin-layout artifacts. On
current main neither is referenced from README, SKILL.md, AGENTS.md,
CHANGELOG, or docs/ — no inbound links to break by removing. Git history
preserves the content for anyone who needs to dig it up.

Closes #352, #353. The PRs by @dinakars777 correctly flagged the drift;
deletion is the cleaner resolution than annotating them as historical.
2026-05-17 00:00:47 -07:00
Trevin Chow aba6172032 Merge pull request #366 from davemorin/feat/324-reddit-json-fallback
feat(web): auto-enrich Reddit URLs from web search via JSON API
2026-05-16 23:41:57 -07:00
Trevin Chow d07e4698e3 Merge pull request #358 from dinakars777/fix/openclaw-poll-clock-init
fix: initialize OpenClaw poll timing once
2026-05-16 23:41:42 -07:00
Trevin Chow 4c0282dd55 Merge pull request #365 from davemorin/fix/284-version-metadata-drift
fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
2026-05-16 23:40:53 -07:00
Trevin Chow 99909fca67 Merge pull request #368 from hnshah/ren/advisory-security-workflow-252
ci: add advisory security workflow
2026-05-16 23:40:28 -07:00
Trevin Chow 36c43d50b7 test: drop side_effect padding to match collapsed time.time() call 2026-05-16 23:39:14 -07:00
Dinakar Sarbada ecf68347db fix: initialize OpenClaw poll timing once 2026-05-16 23:38:30 -07:00
Trevin Chow e2d9d705f6 review: fix selftext key path + break on RedditRateLimitError 2026-05-16 23:37:00 -07:00
Hiten Shah 9c09a67ac2 ci: add advisory security workflow 2026-05-16 23:36:55 -07:00
Dave Morin 32da0bd6cb test: update version assertions for dynamic _skill_version()
Tests now check for version prefix without hardcoded version number,
matching the render.py change to use _skill_version() dynamically.
2026-05-16 23:35:26 -07:00
Dave Morin 863c3bc145 fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
render.py, ui.py, and last30days.py had hardcoded "v3.0.0" in titles
and headers while plugin.json was at 3.1.1. Use _skill_version()
(reads from plugin.json at runtime) so version strings stay in sync.

Fixes #284
2026-05-16 23:35:26 -07:00
Trevin Chow 9f39d10bc5 Merge pull request #373 from hnshah/ren/watchlist-sightings
feat(store): record per-run finding sightings
2026-05-16 23:24:09 -07:00
Trevin Chow 03043da407 Merge pull request #418 from tmchow/chore/greptile-config
chore: add greptile.json (triggerOnUpdates + statusCheck)
2026-05-16 23:23:48 -07:00
Trevin Chow 8bab997854 chore: add greptile.json to opt into update-triggered reviews + status check
Without this config, Greptile's documented default is `triggerOnUpdates: false`
(only the initial PR open triggers a review). Empirically Greptile has been
re-reviewing on force-push to this repo anyway, but documenting the intent
makes the behavior reliable across plan changes and any future config-source
shifts on Greptile's side.

`statusCheck: true` registers Greptile as a GitHub status check (not just a
PR comment). That gives maintainer-tooling a machine-readable heartbeat -
poll `GET /repos/.../commits/SHA/check-runs` and filter by app name to see
whether Greptile is `queued` / `in_progress` / `completed`. Without it the
only signal is "did a new Greptile comment appear" which is silently
ambiguous when Greptile re-reviews and finds nothing new.

If `statusCheck` is OSS-plan-restricted Greptile silently ignores the key,
which is fine - the rolling-summary comment with `Confidence Score: N/5`
remains the fallback signal.

Refs greptileai/skills `greploop` skill for the terminal-state pattern this
config enables.
2026-05-16 23:20:24 -07:00
Hiten Shah 375fd0bcc0 fix(store): enforce sighting finding id invariant 2026-05-16 22:57:32 -07:00
Hiten Shah 92d65723e4 fix(watchlist): refresh sighting retries 2026-05-16 22:57:04 -07:00
Hiten Shah f794f82af5 feat(store): record per-run finding sightings 2026-05-16 22:57:04 -07:00
Trevin Chow 791c0a57a0 review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.

A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).

Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.

Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded

19/19 grounding tests pass.
2026-05-16 22:52:06 -07:00
Dave Morin 211df0deaa feat(web): auto-enrich Reddit URLs from web search via JSON API
Web search backends (Brave, Exa, Serper) can return Reddit URLs as
results. Claude Code's WebFetch blocks reddit.com, so the model can't
retrieve full thread content. After web search, detect Reddit URLs
and fetch body text + top comments via reddit.com/.json endpoint
using the skill's own HTTP library.

Fixes #324
2026-05-16 22:50:39 -07:00
Trevin Chow d7b3995da1 Merge pull request #357 from dinakars777/fix/windows-env-permission-warning
fix: skip POSIX secret warning on Windows
2026-05-16 22:42:26 -07:00
Trevin Chow e217db77cc Merge pull request #369 from voidborne-d/fix/scrapecreators-100-credits
docs: correct ScrapeCreators free tier to 100 credits (closes #367)
2026-05-16 22:41:16 -07:00
Dinakar Sarbada 8ea207b348 fix: skip POSIX secret warning on Windows 2026-05-16 22:39:35 -07:00
voidborne-d b04212680d docs: correct ScrapeCreators free tier to 100 credits (closes #367)
The skill advertises ScrapeCreators as offering "10,000 free API calls" in
six places. The actual free tier on the ScrapeCreators pricing page is
"100 credits free · No credit card required · Credits never expire" — a
100x overstatement that surprises users on signup.

Reporter (#367) burned through their full free allocation on a single
/last30days run after taking the 10,000-call claim at face value. They
verified the actual tier directly against scrapecreators.com plus an
independent review at fahimai.com.

Sweep:
- hooks/scripts/check-config.sh:110  (SessionStart hook tip line)
- README.md:228                      (Sources × Cost table row)
- HERMES_SETUP.md:62                 (Optional: ScrapeCreators bullet)
- skills/last30days/scripts/lib/ui.py:199  (PROMO_SINGLE_KEY["reddit"])
- skills/last30days/SKILL.md:1648    ("PAYG after 10,000 free API calls")
- skills/last30days/SKILL.md:1661    ("10,000 free API calls, then PAYG")

Wording defaults to the provider's own framing — "100 free credits" — and
keeps PAYG language where it was already explicit, since the paid step is
the part users were actually getting blindsided by.

CI gates: tests/test_plugin_contract.py (4) + tests/test_version_consistency.py (4)
all pass.  shellcheck clean.  No tests pin the "10,000" string.
2026-05-16 22:33:41 -07:00
Trevin Chow 2c2cfb9e7e Merge pull request #417 from tmchow/docs/eval-not-in-ci-solution
docs: capture eval-not-in-CI design decision under docs/solutions/
2026-05-16 22:31:39 -07:00
Trevin Chow 3276496f49 Merge pull request #376 from shoobee/feat/yt-dlp-ssh-routing
feat(youtube): route yt-dlp through SSH host for residential IP egress
2026-05-16 22:31:18 -07:00
Trevin Chow 68ae74ff4f docs: capture eval-not-in-CI design decision under docs/solutions/
Closes #374 (adapted, not 1:1 merged).

@hnshah opened PR #374 proposing a docs/adr/ directory for architecture
decision records. The intent is right -- the "why is search-quality eval
manual?" reasoning drifts out of memory if it isn't written down -- but
the docs/adr/ convention doesn't fit alongside the existing
docs/solutions/ structure (compound-engineering ce-compound pattern with
frontmatter metadata, additive entries, no membership-contract test).

This commit adopts hnshah's ADR 002 content (search-quality eval is
manual by default) as a docs/solutions/architecture/ entry with the
canonical compound-style frontmatter (module, problem_type, applies_when,
related_components, tags). Drops the docs/adr/ directory pattern, the
README index, and the test_adr_docs.py contract test.

ADR 001 (multi-surface packaging) is intentionally not adopted here: it
referenced sync.sh as the deploy mechanism, but sync.sh was removed in
PR #405 in favor of `npx skills add . -g -y`. The multi-surface
packaging story is still real but has moved beyond what the original
ADR captured; a fresh "how we ship to multiple harnesses" entry would
make sense as a separate doc.

Co-authored-by: hnshah <hnshah@users.noreply.github.com>
2026-05-16 22:22:59 -07:00
Trevin Chow 27c90504c0 review: validate SSH host alias + rename LAST30DAYS_YT_SSH_HOST -> LAST30DAYS_YOUTUBE_SSH_HOST
Addresses two concerns surfaced during PR #376 review:

1. **SSH option-injection on the host value.** The original PR uses
   shlex.quote() on the remote command and added a `--` option terminator
   in front of the host, but neither one stops a hostile env var like
   `LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
   first place. Tighten `_ytdlp_ssh_host()` to validate the host against
   `^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
   digits, dot, underscore, hyphen). Any value that doesn't match logs a
   warning to stderr and returns None, so the wrap function falls back to
   local execution. The `--` terminator stays as defense-in-depth for the
   case where a valid host happens to start with `-`, but the regex closes
   the door on the env var reaching ssh at all.

2. **Env var naming consistency.** Existing skill-internal config knobs
   spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
   `LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
   is `youtube_yt.py`, the source key is `youtube`, the function family
   is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
   `LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
   model ("route YouTube fetches via residential IP") and the codebase's
   spelled-out convention.

Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)

The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.

44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
2026-05-16 22:19:35 -07:00
shoobee f4eb0af104 fix(youtube): address Greptile review feedback
Three changes from automated review on PR #376:

1. Add `--` option terminator before host in _wrap_ytdlp_cmd (P1 security)
   Prevents SSH option injection if LAST30DAYS_YT_SSH_HOST were ever set
   to a value starting with `-` (e.g. `-oProxyCommand=...`). Low
   exploitability since the env var is user-controlled config — but the
   fix is a single arg and turns a self-harm footgun into no footgun.

2. Hoist `import shlex` to module-level (P2 style)
   Pure stdlib import, no reason for the deferred form. Cleaner.

3. Cache _ytdlp_ssh_host() result in fetch_transcript (P2 style)
   Was being called 2-3x per video; the function is cheap (env lookup
   + strip) so this is purely about readability.

Adds test_wrap_cmd_uses_option_terminator covering the security fix
explicitly with a `-oFoo=bar` host value. Updates index assertions in
the two existing tests that check command shape (host is now at index
4, command string at 5, with `--` at 3).
2026-05-16 22:17:30 -07:00
shoobee 79b5d049ce feat(youtube): route yt-dlp through SSH host for residential IP egress
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.

Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.

The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.

Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
  <host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
  returns True without a local PATH check (the binary lives on the
  remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
  path (which writes a VTT file we couldn't easily read back over SSH)
  and uses the existing _fetch_transcript_direct HTTP fallback. The
  timedtext API isn't bot-walled, so this works fine on datacenter IPs.

Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.

Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).

Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
2026-05-16 22:17:29 -07:00
Trevin Chow 0e2059661a Merge pull request #378 from j-sperling/chore/gemini-3.1-flash-lite-ga
chore: migrate to gemini-3.1-flash-lite GA model
2026-05-16 22:12:20 -07:00
Trevin Chow b1c5f8db82 Merge pull request #382 from lustrousgorilla/bugfix/reddit-gaierror-retry
fix(http): expand retry budget + exponential backoff on DNS resolution failure
2026-05-16 22:11:56 -07:00
Trevin Chow 89c5cb9d5d Merge pull request #416 from tmchow/worktree-inherited-discovering-pebble
fix(ci): run full pytest suite, repair 13 rotted tests
2026-05-16 22:11:34 -07:00
Trevin Chow 5d4f9ef2c5 fix(store): use UTC for all date arithmetic against SQLite columns
Greptile flagged that _cli_query's --since parsing uses datetime.now()
(local time) while first_seen is stored via SQLite's datetime('now') (UTC).
The same bug exists in three other call sites that compare against either
first_seen or run_date (both UTC):

- get_daily_cost: "today" defaults to local date, returns wrong day's cost
  near UTC midnight
- get_stats: "7 days ago" cutoff for runs_7d / successful_7d
- get_trending: "N days ago" cutoff for finding activity ranking
- _cli_query: "N days ago" cutoff for --since flag (Greptile's flag)

All four now use datetime.now(timezone.utc). Same root cause and same fix
as the test_get_new_findings_filters_by_date repair in the previous commit.
2026-05-16 21:40:25 -07:00
Jeffrey Sperling 01262f78c6 chore: import GEMINI_FLASH_LITE in evaluate_search_quality
Address Greptile review nit. DEFAULT_JUDGE_MODEL now reuses the
constant from lib/providers.py instead of duplicating the literal,
so a future identifier change only needs one edit.
2026-05-16 21:40:22 -07:00
Jeffrey Sperling 96a4a78faa chore: migrate to gemini-3.1-flash-lite GA model
The Gemini 3.1 Flash Lite preview model is being discontinued on
May 25, 2026. Per Google's GA announcement, the underlying model
architecture is identical and only the model identifier needs to
be updated from `gemini-3.1-flash-lite-preview` to
`gemini-3.1-flash-lite`.

Also relaxes the `_require_gemini_31_preview` guard to accept any
`gemini-3.1-*` identifier (renamed to `_require_gemini_31`), so the
GA name and the still-preview `gemini-3.1-pro-preview` both pass.
2026-05-16 21:40:22 -07:00
Trevin Chow eb2d7a0f37 fix(ci): run full pytest suite, repair 13 rotted tests
CI was running only test_plugin_contract.py and test_version_consistency.py
(2 of 84 test files), masking 13 rotted tests across 4 clusters. The suite is
fully offline-safe (1402 tests in ~7s without network), so the narrow scope
wasn't gating integration flakiness; it was just stale. validate.yml now runs
`uv run pytest` against the full suite.

Engine fix: store.findings_from_report is rerank-first. ranked_candidates is
the primary persistence path; hackernews/polymarket are unconditionally
supplemented from items_by_source because they rank poorly but matter for
watchlists. When ranked_candidates was empty (rerank failed or skipped),
reddit, x, and every other source were silently dropped. The supplement loop
now falls back to all sources only when ranked_candidates is empty; the normal
path is unchanged.

Test repairs:
- test_store.py (6) + test_watchlist_commands.py (2): cascade from the engine fix
- test_get_new_findings_filters_by_date (latent): local-time vs SQLite UTC
  flake — switched to datetime.now(timezone.utc)
- TestPollDeviceAuth (3): mock_time.time side_effect lists too short after
  impl added a last_reminder call — padded timeout test, pinned others to
  return_value=0 (loops terminate via urlopen, not the clock)
- test_bare_run_emits_web_promo: engine reads ~/.config/last30days/.env, so
  a contributor's saved EXA/PARALLEL key made grounding "available" and
  suppressed the web promo. Also missing X made the "x" promo preempt "web".
  Set LAST30DAYS_CONFIG_DIR="", subprocess cwd=tmpdir, XAI_API_KEY stub.
2026-05-16 21:34:22 -07:00
Trevin Chow 719cdef2fb fix(http): contain DNS retry-budget widening to DNS path only
PR #382 introduced an `effective_retries` widening on the first gaierror,
but the widening leaked: every non-DNS error path (HTTPError, non-DNS
URLError, OSError) was gated on `effective_retries - 1` and so inherited
the expanded bound. A caller passing `retries=2` who hit DNS-then-non-DNS
got 3 attempts instead of 2 — contrary to the PR description and the
fail-fast intent of small retry budgets.

Fix:
- Gate every non-DNS sleep/retry decision on the caller's original
  `retries`, not the widened `effective_retries`.
- Add an explicit `break` in each non-DNS branch when the original
  budget is exhausted, so the (possibly widened) outer loop bound
  can't pull us into an extra attempt.

Adds two regression tests covering the DNS-then-non-DNS-URLError and
DNS-then-OSError sequences flagged in Greptile review on PR #382.
2026-05-16 21:13:17 -07:00
Trevin Chow ac04b56acc Merge pull request #383 from lustrousgorilla/bugfix/bird-x-json-decode-retry
fix(bird_x): retry subprocess on non-JSON stdout (HTML anti-bot interstitial)
2026-05-16 21:08:35 -07:00
Trevin Chow 42bfc6c76c Merge pull request #415 from tmchow/fix/sc-source-gating-consistency
fix(sources): align SC source gating between code and docs
2026-05-16 21:08:10 -07:00
Gabriel Arrillaga 5a2fe5279b fix(http): expand retry budget + use exponential backoff on DNS failure
Transient DNS resolution failures (socket.gaierror, surfaced as
urllib.error.URLError with reason=gaierror) were retried with the
generic URLError handler — linear backoff (2s, 4s, 6s) and bounded by
the caller-passed `retries` parameter. For callers that pass small
retry values (e.g. lib/reddit.py::_subreddit_search uses retries=2), a
single first-attempt DNS hiccup followed by one quick retry on the
still-flaky resolver would exhaust the retry budget and wipe a whole
subreddit sweep — which the caller's broad `except Exception` then
silent-empties as `[]`.

Fix:
- Distinguish URLError-with-gaierror-reason from generic URLError via
  a new `_is_dns_failure()` helper.
- For DNS failures, use exponential backoff (1s, 2s, 4s, ...) instead
  of the linear default.
- For DNS failures, expand the effective retry budget to at least
  MIN_DNS_RETRIES (=3) on first occurrence, so callers that passed
  `retries=2` still get a meaningful retry budget for the transient
  case. Non-DNS URLErrors and HTTPErrors keep the caller's value.
- DNS attempts are counted separately (`dns_attempts`) so unrelated
  URLError or OSError failures within the same call don't accidentally
  expand the budget further.

Reported during a community-signal pass where the Reddit subreddit
sweep silently returned zero items after a first-round transient DNS
hiccup. The fix lives at the http layer (where the retry loop is)
rather than per-source so every caller benefits.

Tests:
- Verifies a caller-passed retries=2 still gets MIN_DNS_RETRIES=3
  attempts on gaierror.
- Verifies gaierror-then-success returns successfully on attempt 2.
- Verifies the exponential-backoff sleep pattern (1s, 2s) on the
  retry attempts before exhaustion.
- Verifies a non-DNS URLError (ConnectionRefusedError reason) does
  NOT expand the retry budget — only true DNS failures do.

All 12 http tests pass (8 baseline + 4 new). No regressions in the
broader test suite (1373 pass / 14 fail, vs 1369 pass / 14 fail on
main — the 14 failures are pre-existing and unrelated to this PR).
2026-05-16 20:54:14 -07:00
Gabriel Arrillaga a717dd2b2c fix(bird_x): retry subprocess on non-JSON stdout (HTML interstitial)
Twitter's edge intermittently serves an HTML anti-bot interstitial in
place of JSON when the bird-search subprocess hits a per-query rate
limit. Before this fix, that response made json.loads raise
JSONDecodeError and _run_bird_search() returned {"error": ..., "items":
[]} with the parsed exception message — silent-empty against an
orchestrator that has no way to distinguish "Twitter served HTML; retry
likely succeeds" from "no tweets matched the query."

Surfaced during a community-signal pass where a Karpathy-LLM-wiki
subquery returned zero X items, while a second identical run a few
seconds later returned full results.

Fix:
- Extract the subprocess invocation into _invoke_bird_subprocess() so
  the retry loop can call it multiple times cleanly. Returns
  (result, terminal_error) — terminal_error is non-None for
  unrecoverable cases (subprocess timeout, spawn failure) that should
  NOT be retried.
- In _run_bird_search(), wrap the json.loads parse in a retry loop
  bounded by MAX_JSON_DECODE_RETRIES (=2) with JSON_DECODE_RETRY_DELAY
  (=5s) between attempts.
- On non-JSON stdout, log a diagnostic that names the shape
  (`looks_html`, first-80-chars stdout preview, attempt counter) so
  silent-empty failures become legible in logs.
- On retry exhaustion, return an error dict whose message explicitly
  names "anti-bot interstitial" as the likely cause, distinguishing
  this failure from a genuine no-results case.

Subprocess timeout, spawn failure, and non-zero return-code paths are
unchanged — those are terminal and don't retry.

Tests:
- Verifies HTML-then-JSON returns success on attempt 2.
- Verifies all-HTML returns the diagnostic error dict mentioning the
  anti-bot interstitial cause.
- Verifies subprocess timeout is NOT retried.

All 12 bird_x tests pass (9 baseline + 3 new).
2026-05-16 20:52:56 -07:00
Trevin Chow 9f08bb68b5 fix(sources): align SC source gating between code and docs
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.

This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.

The resulting source-gating model is three-tier and intentional:

  • **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
    reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
    grounding, **tiktok, instagram, threads**

  • **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
    perplexity (different paid API — OpenRouter),
    tiktok_comments / youtube_comments (N× extra SC calls per video)

  • **--search per-query opt-in** (relevance reasons):
    pinterest (visual pins, narrow utility),
    xiaohongshu (Chinese-market specific)

Changes:

- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
  now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
  the new contract and includes a regression guard ("INCLUDE_SOURCES
  should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
  ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
  runs. Drops false INCLUDE_SOURCES requirement for
  tiktok/instagram/threads; corrects pinterest to mention --search;
  adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
  section.

Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
2026-05-16 20:17:31 -07:00
Trevin Chow 602de1ebda Merge pull request #388 from bradferguson/fix/sc-youtube-and-hn-tokenization
fix(sources): unblock SC YouTube + multi-token HN searches
2026-05-16 19:41:12 -07:00
Trevin Chow bf3a82a87e Merge pull request #389 from kuyua9/fix/save-comparison-html-kuyua9
fix: save comparison HTML artifacts
2026-05-16 19:40:59 -07:00
Trevin Chow c010feb8f8 Merge pull request #399 from spiky02plateau/feat/exclude-sources-banner-and-pipeline
feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
2026-05-16 19:39:51 -07:00
Brad Ferguson edea402b7c fix(sources): unblock SC YouTube + multi-token HN searches
Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.

YouTube (ScrapeCreators)
  SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
    {"error":"missing_parameter","message":"You must provide a query"}
  The canonical SC parameter for that endpoint is `query`. Other SC
  endpoints we use (Reddit, TikTok, Instagram) happened to work because
  they use their own per-endpoint parameter names — YouTube was the lone
  outlier.

Hacker News (Algolia)
  Multi-keyword theme queries returned zero hits across every theme.
  Algolia treats query= as strict AND across tokens, so a 4-5 word query
  like "claude, personal agents, agentic infra" matches no stories.

  Three changes in hackernews.py:

  1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
     search_hackernews and _title_matches_query normalize the query the
     same way — addresses Greptile P2 #2 about the two callsites needing
     to stay in sync.
  2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
     by token-overlap instead of requiring every token.
  3. Relax _title_matches_query from all-words to any-word, *but match
     on word boundaries (\b<word>\b) rather than naive substring* —
     addresses Greptile P2 #1, which flagged that the previous any-word
     relaxation would let "ai" falsely match "email" or "rail".

  Token-overlap relevance scoring at parse time already demotes weak
  matches, so word-boundary any-word matching is safe.

Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.

Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-05-16 19:37:16 -07:00
Trevin Chow 4d4ac97ffb refactor: hoist comparison-html gate into a single condition (Greptile DRY)
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
2026-05-16 19:32:49 -07:00
kuyua9 cd34966b4f fix: save comparison HTML artifacts 2026-05-16 19:32:06 -07:00
Trevin Chow 85255be350 Merge pull request #414 from mvanhorn/dependabot/uv/pytest-9.0.3
chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
2026-05-16 19:31:46 -07:00
Trevin Chow 1aa120a420 Merge pull request #407 from DamienStevens/feat/macos-keychain-source
feat(env): macOS Keychain credential source
2026-05-16 19:31:34 -07:00
Trevin Chow 306d8c2d73 fix(env): wire EXCLUDE_SOURCES through get_config + SKILL.md integration
The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().

Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
  propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
  than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
  invoking the skill knows to subtract excluded sources before displaying
  the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
  prose are incomplete — the agent invoking the skill won't know the flag
  exists.)
2026-05-16 19:30:27 -07:00
Trevin Chow d0dcf751f1 fix(keychain): single source of truth for key list + robust USER fallback
Addresses Greptile review on PR #407:

- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
  XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
  manually storing those keys would not see them in --list and the
  interactive prompt would never offer to set them.

  Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
  get_config() pass it through, and add a parity test that parses
  ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
  caught at CI time instead of after a user reports a missing key.

- P2: os.environ.get("USER", "") silently returned "" under sudo, in
  Docker without --env USER, or in CI runners that strip USER. The
  resulting `security find-generic-password -a ""` call would never
  match items stored by setup-keychain.sh, so all lookups silently
  returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
  USER is absent.

The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
2026-05-16 19:25:05 -07:00
dependabot[bot] afd4b04d6d chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-17 02:22:02 +00:00
Trevin Chow 14d8f62e02 Merge pull request #413 from tmchow/docs/compound-release-cascade-pattern
docs: compound learning on release-time consistency-test cascade failures
2026-05-16 19:12:31 -07:00
Trevin Chow 0fd532d249 docs: compound learning on release-time consistency-test cascade failures
Documents the cascade pattern surfaced during this session's install-modernization
arc: a `test_sync_cache_path_uses_skill_version` test asserted that a hardcoded
version pin in `sync.sh` matched the version frontmatter in SKILL.md. When a
release bumped SKILL.md, every open PR's CI failed simultaneously on the
unrelated stale-pin assertion. Affected at least 5 PRs across the 2026-05-13
to 2026-05-15 window (#400, #390, #392, and two others) plus required hotfix
PR #397 to unblock the queue.

The permanent fix shipped in PR #405 (deleted sync.sh + the test). This doc
captures the design lesson so the pattern doesn't reappear: don't write
consistency tests that read two files and assert one matches a value derived
from the other. Either derive at runtime from a single source of truth, or
self-skip / merge-base-scope the test so deletion is a non-event.

Created via /ce-compound. Includes:

- docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md
  (the new learning — first entry under docs/solutions/)
- CONCEPTS.md (new — 4 entries: Skill, Engine, Harness, Beta channel,
  capturing project-specific vocabulary that surfaced across the session)
- AGENTS.md (added one-line Structure entries surfacing docs/solutions/ and
  CONCEPTS.md so fresh agents discover them)
- docs/plans/2026-04-22-{002,003,005,006}-*-plan.md (added deprecation banner
  to each, pointing readers at PR #405 and the new docs/solutions entry —
  these 4 historical plans still reference the deleted sync.sh inline)

Also: closed PR #379 (j-sperling's workaround for the same cascade,
superseded by PR #405's permanent fix).
2026-05-16 19:08:02 -07:00
Trevin Chow 8867a007ea Merge pull request #392 from Gujiassh/fix/openclaw-scrapecreators-optional-env
fix(openclaw): make ScrapeCreators key optional
2026-05-16 18:50:07 -07:00
gujishh 8af8f06b06 fix(openclaw): make ScrapeCreators key optional 2026-05-16 18:48:29 -07:00
Trevin Chow 01b5f3dc1e Merge pull request #363 from thinkun/pr/claim-contributor-entry
Claim contributor entry — @thinkun
2026-05-16 18:46:35 -07:00
Trevin Chow 2e39ee8ce4 Merge pull request #412 from tmchow/refactor/skill-meta-version-helper
refactor: consolidate SKILL.md version regex into lib/skill_meta.py
2026-05-16 18:42:49 -07:00
Trevin Chow 37033164da Merge pull request #410 from tmchow/docs/agents-orientation-multi-harness
docs: reframe as multi-harness Agent Skills package, flip CLAUDE.md ↔ AGENTS.md
2026-05-16 18:42:36 -07:00
Trevin Chow 9fe4b8f130 Update AGENTS.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-16 18:42:14 -07:00
Trevin Chow 73dc6b9996 refactor: consolidate SKILL.md version regex into lib/skill_meta.py
The same `^version:\s*"([^"]+)"\s*$` regex (or a slight variant) was
duplicated across three files: render.py inline, test_plugin_contract.py
local helper, test_version_consistency.py local helper. A future change
to the SKILL.md frontmatter version format would have needed to update
three places without any compile-time pressure to keep them in sync.

New skills/last30days/scripts/lib/skill_meta.py provides:
- `_VERSION_RE` private compiled pattern (accepts double-quoted,
  single-quoted, or unquoted YAML version scalars per the widening
  landed in 997708a)
- `read_skill_version(skill_md_path: Path) -> str | None` helper that
  catches OSError + UnicodeDecodeError and returns None on miss

Callers updated:
- render.py::_skill_version now calls skill_meta.read_skill_version
  inside the SKILL.md fallback loop, returning `read_skill_version(...) or "?"`.
  Semantically equivalent to the old break-after-first-SKILL.md logic.
- test_plugin_contract.py and test_version_consistency.py import the
  helper instead of defining the regex inline. Both files use the
  established sys.path.insert pattern.

Added tests/test_skill_meta.py with 6 direct unit tests covering the
helper's full contract: missing file, undecodable bytes, no-version-line,
and all three quoting styles (double, single, unquoted). Previously the
helper was only exercised transitively through render._skill_version().

Added test_skill_md_uses_double_quoted_version to
test_version_consistency.py — the old per-test regex incidentally
asserted "this repo's SKILL.md uses double-quotes" by being strict;
the shared helper accepts all three styles, so the assertion is now
explicit instead of implicit.

Code-reviewed by ce-code-review (8 reviewers); safe_auto fixes applied
inline (rename to _VERSION_RE, group or-chain instead of generator,
docstring tightened, dropped unnecessary `from __future__ import
annotations`, tightened signature to Path-only).

Conftest.py refactor for the sys.path.insert duplication across ~20 test
files filed as issue #411 — out of scope for this PR (touches many
files, separate concern).

Test results: 23 passed in the affected test set (16 prior + 6 new
test_skill_meta tests + 1 new double-quote assertion). Full suite shows
same 13 pre-existing failures as main; zero new failures.
2026-05-16 18:17:33 -07:00
Trevin Chow 1fd763e09f docs: flip CLAUDE.md ↔ AGENTS.md — AGENTS.md becomes canonical, CLAUDE.md points at it
Mirrors the multi-harness reframing of the project itself. CLAUDE.md is
Claude-Code-specific by name; AGENTS.md is the multi-harness convention
that Codex, Cursor, Gemini CLI, GitHub Copilot, and most other Agent
Skills hosts also read. The canonical content belongs in the file
multi-harness tooling expects.

git mv preserves history — the Orientation section and everything else
that was in CLAUDE.md is now tracked under AGENTS.md, with full blame
continuity. The new CLAUDE.md is a one-line `@AGENTS.md` reference so
Claude Code continues to load the content (it follows @ references).
2026-05-16 17:03:11 -07:00
Trevin Chow e0f6ef845a docs(claude.md): add Orientation section, reframe as multi-harness Agent Skills package
Closes the spirit of #335 (closed in favor of this fresh PR after the
sync.sh thread became obsolete via PR #405).

Two changes:

1. Top-of-file description reframed from "Claude Code skill" to
   "Agent Skills package... installable across Claude Code (most common
   host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other
   Agent Skills hosts". The skill works across every major agent host
   after the install-modernization work in PR #400/#404/#405/#409.
   Calling it "Claude Code skill" undersells the surface and biases
   contributors toward Claude-Code-specific assumptions.

2. New ## Orientation section (4 bullets) framing the project for
   contributors who would otherwise read the python3 invocation in
   ## Commands and form a CLI-first mental model. Names the trap
   explicitly with one concrete invalid-syntax example
   (`/last30days OpenClaw --emit=html | pbcopy` — slash commands don't
   pass shell mechanics through). Bullets adapted from #335 with
   multi-harness framing replacing the Claude-Code-only framing.

No code changes. No SKILL.md changes. CLAUDE.md only. AGENTS.md
inherits via @CLAUDE.md.
2026-05-16 17:01:04 -07:00
Trevin Chow c918e18465 Merge pull request #409 from tmchow/refactor/skill-dir-relative-resolver
refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
2026-05-16 16:55:32 -07:00
Trevin Chow 6fe0aca7ee refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
The Step 1 and comparison-mode resolver loops walked a hardcoded list
of install paths trying to find scripts/last30days.py. Two problems:

1. The list was never exhaustive — it covered ~/.codex/skills, but not
   ~/.claude/skills, ~/.cursor/skills, ~/.gemini/skills, ~/.copilot/skills,
   ~/.hermes/skills/research, etc. PR #406 was about to fix that by
   enumerating more paths, but enumeration is the wrong shape.
2. The resolver could pick a different install than the SKILL.md the
   model loaded from. Spec-vs-engine divergence is subtle and confusing
   when it triggers.

The model already knows the SKILL.md path it loaded (from its Read tool
result). Templating that into the bash block is strictly better than
guessing across an enumerated list:

- Works for every harness without enumeration (Hermes, Cursor, anything
  new) because we just use wherever the harness loaded SKILL.md from
- Aligns spec with engine — the engine runs from the same install the
  spec was read from
- Deletes ~80 lines of bash across Step 1 + comparison-mode + the
  prose preamble describing the resolver

Mechanics:

- SKILL_DIR placeholder in both bash blocks — model substitutes the
  absolute path of the directory containing the SKILL.md it just Read
- One-line validation `[ ! -f "$SKILL_DIR/scripts/last30days.py" ]`
  catches bad templating with a clear error
- All references to $SKILL_ROOT replaced with $SKILL_DIR (badge
  VERSION lookup, prose description in the LAW-7 preamble area)
- STEP 0 unchanged — different concern (marketplaces stale-clone hop)

Version 3.2.3 -> 3.2.4 (behavior change: install paths the resolver
list never enumerated now work; install paths it did enumerate work
the same way they used to but via the SKILL_DIR template).
2026-05-16 16:26:45 -07:00
Matt Van Horn d9f606ff75 chore: add gogcli #589 zoom demo gif (#408)
PR demo embed asset for openclaw/gogcli #589 (feat: --with-zoom).
Hosted here for stable raw URL.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-16 11:18:41 -07:00
Damien Stevens 74a387b093 feat(env): macOS Keychain credential source
Adds the macOS Keychain as the lowest-priority credential source on Darwin.
Items stored as generic passwords with service name "last30days-<KEY>" for
the current user are picked up automatically by get_config() — file env
and process env still win on collision.

No new config knob: behavior is strictly additive. On non-Darwin (or when
the `security` binary is missing) the loader is a no-op, so Linux/Windows
behavior is unchanged.

  Priority (highest wins):
    1. Environment variables
    2. .claude/last30days.env (per-project)
    3. ~/.config/last30days/.env (global)
    4. macOS Keychain items prefixed last30days- (new)

Includes:
  - lib/env.py: KEYCHAIN_SERVICE_PREFIX constant, _load_keychain helper
    (platform-gated, shutil.which-gated, subprocess-error tolerant),
    wiring into get_config before get_openai_auth so OPENAI_API_KEY can
    come from Keychain too, _CONFIG_SOURCE reports "keychain" when no
    file source is present.
  - scripts/setup-keychain.sh: bash helper with interactive set,
    --list, --delete, --replace modes. Uses `security add-generic-password`.
  - tests/test_env_keychain.py: 12 tests covering platform gate,
    missing-binary gate, success path, whitespace stripping, subprocess
    errors swallowed, get_config precedence, and an OPENAI_AUTH wiring
    regression test.
  - tests/test_env_cookies.py: existing integration test mocks the new
    _load_keychain hook so it stays hermetic on Darwin developer
    machines that have real keychain entries.
  - README.md: new "macOS Keychain (optional)" subsection under
    "Bring your own keys" documenting setup-keychain.sh and the manual
    `security add-generic-password` invocation.

Tested on macOS with a populated keychain and against the existing pytest
suite — CI-tracked tests (test_plugin_contract.py, test_version_consistency.py)
plus all env-touching tests pass. Pre-existing unrelated failures in
test_store.py / test_watchlist_commands.py / test_setup_openclaw.py /
test_footer_nudge_suppression.py are untouched.
2026-05-16 09:01:28 -04:00
Trevin Chow 4a30923892 Merge pull request #405 from tmchow/docs/readme-multi-harness-install
docs+refactor: modernize install story everywhere, delete sync.sh
2026-05-15 23:46:57 -07:00
Trevin Chow 9fb19eae63 refactor: delete sync.sh, dev workflow moves to npx skills add . -g -y + native installers
Every job sync.sh did has a better replacement:

- Per-harness skill dirs (~/.claude/skills, ~/.codex/skills, ~/.agents/skills):
  `npx skills add . -g -y` writes to every detected harness's home dir and
  uses symlinks by default. Edits propagate live — no re-deploy step.
- Hermes (~/.hermes/skills/research/last30days):
  `hermes skills install mvanhorn/last30days-skill --force` pulls from
  GitHub and handles the deploy itself. The script wrapping was redundant.
- OpenClaw variant: `clawhub install last30days-official` is what users
  already run per the README; the maintainer doesn't need a separate
  variant-deploy step in the public repo's scripts.
- Claude marketplace cache (~/.claude/plugins/cache/...): this was a
  "test against the official install path" hack we shouldn't have been
  recommending. With PR #400's resolver collapse, STEP 0 no longer
  enforces the cache as the only valid SKILL.md location. Just install
  the skill normally via `npx skills` or the marketplace.

Cleanup:

- DELETE skills/last30days/scripts/sync.sh
- tests/test_version_consistency.py — drop test_sync_cache_path_uses_skill_version
- CLAUDE.md — replace the sync.sh command + rule with `npx skills add . -g -y`
- HERMES_SETUP.md — Installation now uses `hermes skills install --force`;
  developer-alternative section shows the symlink pattern for live editing
- render.py — _skill_version docstring no longer attributes the
  ".claude-plugin absent" case to sync.sh; explains it via per-harness
  install paths in general
- .github/PULL_REQUEST_TEMPLATE.md — drop the "Ran bash scripts/sync.sh"
  checklist item

CHANGELOG and historical docs (release notes, plan files) keep their
existing sync.sh mentions as accurate history.
2026-05-15 23:42:31 -07:00
Trevin Chow d1cc29d338 docs(readme): add -g (global) flag to every npx skills example
`npx skills add` defaults to project-local install (`./.skills/`,
committed with the repo). For a research-the-world skill like this one,
that's almost never what users want — they want it available across all
projects, not scoped to whichever directory they happened to run the
install from.

Adding `-g` (global) to every npx skills example in the README:
- Top-of-file install snippet
- Install table row
- Claude Code subsection's "alternative via npx skills" example
- Codex/Cursor/etc. subsection's default, per-harness, update, list,
  and remove commands

Brief one-liner explains what `-g` does and notes that dropping it
gives a project-local install for users who want team consistency on
a specific codebase.
2026-05-15 23:20:33 -07:00
Tobi 095bcae915 fix(check-config): normalize EXCLUDE_SOURCES (lowercase + whitespace) before matching
The bash banner accounting used raw substring matching while
pipeline.py normalises EXCLUDE_SOURCES via .strip().lower(). With
EXCLUDE_SOURCES=TikTok,Instagram (or with surrounding spaces),
pipeline correctly excludes the sources but the banner did not
deduct them — count showed 1-2 higher than what the pipeline
actually runs. Normalisation now mirrors the Python side
(lowercase, collapse whitespace around commas, strip outer whitespace).

Reproducer (clean HOME with config EXCLUDE_SOURCES=TikTok,Instagram):
  before: /last30days: Ready — 7 sources active.
  after:  /last30days: Ready — 5 sources active.

Addresses Greptile review comment P1 on #399.
2026-05-16 08:05:34 +02:00
Trevin Chow ded52062e6 docs(readme): drop Gemini CLI native-extension install path
The native `gemini extensions install` path was a workaround for the
v0.9.0 installer bug (still unresolved per upstream issue #11452).
Now that `npx skills add -a gemini-cli` covers Gemini cleanly with the
same install/update story as every other supported harness, the native
path is just one more confusing option to maintain. Users on Gemini get
the same recommendation everyone else does.

Removes the dedicated "Gemini CLI (native extension)" subsection and
the separate table row. Gemini CLI is now surfaced once, in the npx
skills section, alongside Codex, Cursor, Copilot, and the rest.
2026-05-15 23:03:51 -07:00
Trevin Chow 164d7ae6ed docs(readme): surface gemini-cli (and copilot, windsurf, 50+ others) in npx skills coverage
npx skills supports 50+ harnesses via the -a flag, including gemini-cli,
github-copilot, windsurf, cline, continue, roo, aider-desk, opencode,
goose, and more — not just the few I'd listed initially. Updating to
reflect that breadth.

- Top-of-file snippet now reads "Codex, Cursor, Copilot, Gemini CLI, or
  any of 50+ Agent Skills hosts" (was: "Codex, Cursor, Copilot, or any
  Agent Skills host" + Gemini listed separately in the table footer).
- Install table: same expansion; Gemini CLI native-extension row relabeled
  to clarify it's the native path (not the only Gemini option).
- npx skills subsection: lists the most common harness flags and links
  to the upstream vercel-labs/skills repo for the full list.
- Gemini CLI native subsection: now leads with "the npx skills path
  above is simpler" and frames the native install as the alternative
  for users with an existing Gemini extensions workflow or who hit
  the v0.9.0 installer bug.
2026-05-15 23:02:57 -07:00
Trevin Chow f1ce7533e6 docs(readme): recommend Claude Code plugin, add npx skills install for Codex/Cursor/Copilot
The skill is now installable across every major agent harness after the
SKILL.md path-resolver work landed in PR #400 + #404. README didn't yet
reflect that — the install table only listed Claude Code, OpenClaw, and
Gemini CLI, and the top-of-file install snippets featured Hermes (an
internal dev workflow, not a public install method).

Restructured the install section:

- Top-of-file snippets: just Claude Code (recommended, auto-updates) and
  the universal `npx skills add` one-liner. Dropped Hermes from the
  prominent spot (internal-only); pointed everything else to the Install
  section below.
- Install table: added a third column for update commands, since every
  harness now has a distinct update path worth surfacing. Added the
  `npx skills` row covering Codex/Cursor/Copilot/any Agent Skills host.
- Claude Code subsection: explains why it's recommended (marketplace
  handles versioned cache + auto-refresh) and notes that the agent-skills
  install also works on Claude Code if preferred (`-a claude-code`).
- New "Codex, Cursor, Copilot, and other Agent Skills hosts" subsection:
  shows the default install, per-harness `-a` targeting, and the update
  commands (`npx skills update last30days` for one skill, bare
  `npx skills update` for all).
- Manual (developer) subsection: switched from a clone-into-skills-dir
  recipe to a clone + symlink recipe. Symlink keeps the install in sync
  with the working tree as you edit, no re-copy on each change.

No code changes. No version bump (docs-only).
2026-05-15 22:58:54 -07:00
Trevin Chow 0b939bf703 Merge pull request #404 from tmchow/fix/json-plan-shell-quoting
fix(skill): write --plan / --competitors-plan to tmpfile (closes #403)
2026-05-15 22:52:49 -07:00
Trevin Chow 9f95efb215 fix(skill): use portable trailing-XXXXXX mktemp form for plan tmpfiles
Greptile's review flagged mktemp -t as non-portable between BSD and GNU.
The suggested replacement (mktemp "$TMPDIR/...XXXXXX.json") is correct
about dropping -t but still puts X's in the middle of the template name
(XXXXXX.json), which BSD mktemp does not substitute — only X's at the
end of the basename are replaced on BSD. Verified on macOS:

  mktemp "$TMPDIR/last30days-test.XXXXXX.json"
  → /var/folders/.../last30days-test.XXXXXX.json  (X's left literal)

The fully portable form uses trailing X's and drops the .json suffix
(engine reads by path, not extension):

  mktemp "$TMPDIR/last30days-test.XXXXXX"
  → /var/folders/.../last30days-test.DXAHzR     (X's substituted)

Verified on bash and zsh, BSD/macOS. GNU/Linux is already fine since
GNU substitutes X's wherever they appear in the basename.

Applied to both --competitors-plan (comparison-mode block) and --plan
(Step 1 block) tmpfile writes.
2026-05-15 22:50:56 -07:00
Trevin Chow ff54c07a3b fix(skill): write --plan / --competitors-plan to tmpfile, bump 3.2.2 -> 3.2.3
Closes #403.

The SKILL.md templates instructed the model to invoke the engine with
inline single-quoted JSON: `--plan '$JSON'` and `--competitors-plan '{...}'`.
When any resolved field value contained an apostrophe (common in `context`
strings like "McDonald's", "people's choice", or contracted forms like
"don't", "won't"), the inner `'` closed the outer single-quote and broke
shell parsing before the engine was even invoked.

Observed during PR #400 testing: a Codex run hit the trap and self-healed
by re-encoding, wasting one engine invocation and ~30s of latency.

Fix: switch both templates to the heredoc + tmpfile pattern. The engine's
`parse_plan()` and `parse_competitors_plan()` already check
`os.path.isfile(plan_str)` and read from disk — only the SKILL.md prose
needed to change.

The quoted heredoc marker (<<'PLAN_EOF') is load-bearing: it suppresses
shell interpolation so apostrophes, $, backticks, etc. pass through verbatim.
A trap on EXIT cleans up the tmpfile after the engine call returns.

LAW 7's "MUST contain --plan" self-check guidance and Step 1's invocation
example both updated to reference the file form. Comparison-mode invocation
block updated the same way for --competitors-plan.

Version bump 3.2.2 -> 3.2.3 because this is a behavior change users
running comparison-mode queries will notice (no more "shell quoting error,
retrying" sequences on apostrophe-containing context strings).
2026-05-15 22:43:02 -07:00
Trevin Chow e276c30477 Merge pull request #400 from tmchow/refactor/skill-md-relative-path-resolver
refactor(skill): SKILL.md-relative path resolver, drop Codex native plugin
2026-05-15 22:36:53 -07:00
Trevin Chow 2f277dfc66 fix(skill): address greptile P1+P2 review feedback on PR #400
Two real bugs flagged in the automated review of PR #400; both small.

1. render.py::_skill_version manifest with no "version" key

   `json.loads(manifest.read_text()).get("version", "?")` returned "?"
   immediately on a valid JSON manifest that lacked the "version" key,
   never falling through to the SKILL.md frontmatter fallback. Contradicted
   the docstring's "Returns '?' only if both sources are missing" contract.
   Same shape if version is present but empty string ("" produces the
   broken badge `🌐 last30days v · synced ...`).

   Fix: pull the version out of the parsed dict, then `continue` to the
   next ancestor if it's None or empty. Falls through to the SKILL.md
   walk only after exhausting every ancestor.

2. SKILL.md STEP 0 re-read target hardcoded to nested cache layout

   STEP 0 told the model to re-read from
   `$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md` — the new nested
   layout. But Step 1's resolver explicitly handles both shapes
   (nested `{cache}/{version}/skills/last30days/` and flat
   `{cache}/{version}/`), noting "Both shapes ship in the wild." On an
   install where the highest-versioned cache happens to be the older flat
   shape, STEP 0's re-read target wouldn't exist; the model would silently
   stay on the stale marketplaces/ copy STEP 0 was supposed to move it
   away from — the exact failure mode this guard was added to prevent.

   Fix: extend the STEP 0 bash to resolve $CLAUDE_CACHE_SKILL_MD by
   probing both layouts, then have the model hop to that resolved path
   instead of constructing the path from a hardcoded suffix.

Two new tests in tests/test_skill_version.py cover the missing-key and
empty-string cases for fix 1. Fix 2 is exercised via the bash probe at
verify time (the STEP 0 prose-contract test isn't unit-testable from
Python, but the dual-layout bash is verified to resolve to the correct
SKILL.md on both shapes).

Stale finding skipped: greptile also flagged a missing try/except on the
SKILL.md read_text() call, but that was already addressed during the
ce-code-review safe_auto pass earlier in this PR — current code wraps it
in `try/except (OSError, UnicodeDecodeError)`, strictly more defensive
than the suggested fix.
2026-05-15 22:34:52 -07:00
Trevin Chow 6c2c55733c fix(skill): use find instead of ls+glob in cache resolvers (zsh compatibility)
zsh errors on globs that match nothing instead of returning the literal
pattern (bash's default), and `2>/dev/null` does not suppress the error
because it comes from the shell's glob expansion before `ls` even runs.
Under Codex (which executes the SKILL.md bash via zsh), STEP 0 and the
Step 1 / comparison-mode resolvers emitted noisy "no matches found"
errors on machines without a Claude plugin cache populated.

Replaces all three `ls -d $HOME/.claude/plugins/cache/last30days-skill/last30days/*/`
invocations with `find ... -mindepth 1 -maxdepth 1 -type d 2>/dev/null`.
find is POSIX-portable, errors silently when the base dir doesn't exist,
and never triggers shell glob errors. `sort -V | tail -1` precedence
preserved (verified: picks 3.10.0 over 3.2.1 over 3.1.0). Trailing-slash
strip removed because find doesn't append slashes.

Observed in Codex session running /last30days against PR #400 with the
Claude plugin cache deleted - bash output was:
  zsh:1: no matches found: /Users/.../last30days/*/

After fix: clean empty output, exit 0, STEP 0 correctly treats it as
"no cache present, do not hop", resolver falls through to per-harness
skill dirs as designed.
2026-05-15 22:14:37 -07:00
Trevin Chow 997708ad48 refactor(skill): apply ce-code-review fixes — bump to 3.2.2, fallback tests, comparison resolver
12 fixes from the multi-agent code review on PR #400:

Version 3.2.1 -> 3.2.2 across all manifests (SKILL.md frontmatter + body
header, pyproject.toml, .claude-plugin/{plugin,marketplace}.json, sync.sh
cache path). The PR ships observable behavior changes (STEP 0 logic flip,
resolver order change, badge fallback) that should not silently appear
under the same version number — the new fallback reads SKILL.md version
directly so the badge would otherwise be misleading.

render.py::_skill_version:
- `import re` moved to module top
- _VERSION_RE extracted as a module-level compiled pattern that accepts
  double-quoted, single-quoted, and unquoted YAML version scalars
- `break` -> `continue` on corrupt manifest, so a corrupt inner manifest
  no longer shadows a valid outer one
- Wrap SKILL.md read_text() in try/except for UnicodeDecodeError to keep
  badge emission from crashing on mis-encoded SKILL.md
- Docstring clarifies precedence; inline comment marks the fallback boundary
  between the manifest walk and the SKILL.md walk

tests/test_skill_version.py (new): 7 unit tests for the fallback paths
(manifest absent, manifest corrupt, corrupt-inner + valid-outer, both
absent, SKILL.md without version, single-quoted, unquoted).

tests/test_plugin_contract.py: tombstone test asserting .codex-plugin/
stays removed (was the only CI guard against accidental reintroduction).

SKILL.md:
- STEP 0 bash echoes CLAUDE_CACHE_LATEST so the model can see the
  resolved value when deciding whether to hop
- "Both shapes ship in the wild" comment now names the two cache layouts
  (nested {cache}/{version}/skills/last30days/ vs flat {cache}/{version}/)
- Comparison-mode bash invocation gets its own inline SKILL_ROOT resolver
  (latent gap: the contract tells the model to skip Step 1 on comparison
  queries, so SKILL_ROOT was previously unset there)

CHANGELOG.md: [Unreleased] entries for the resolver rewrite and the
breaking removal of Codex native-plugin support.

All 9 reviewer personas surfaced findings; 3 cross-reviewer corroboration
clusters were promoted (import re, "both shapes" comment, missing fallback
tests). Maintainability follow-up flagged: regex now duplicated across
render.py and 2 test files; could consolidate via shared lib/skill_meta.py
helper in a future PR.
2026-05-15 21:45:25 -07:00
Trevin Chow c913e1cf89 refactor(skill): SKILL.md-relative path resolver, drop Codex native plugin
STEP 0 (CANONICAL PATH SELF-CHECK) used to force any SKILL.md load that wasn't
under $HOME/.claude/plugins/cache/last30days-skill/last30days/{version}/ to
re-Read from there. That guard is Claude-Code-specific (defends against the
marketplaces/ stale-clone bug) and broke under non-Claude installers like
`npx skills add`, ~/.codex/skills/, and ~/.agents/skills/.

The new STEP 0 narrows the check to its actual target: fire only when the
loaded SKILL.md path contains /.claude/plugins/marketplaces/. Every other
install path is trusted. The 2026-04-22 incident workaround is preserved
without breaking other harnesses.

Step 1 SKILL_ROOT resolver collapses the Codex-first / Claude-fallback /
CWD-fallback chain into a single precedence walk: Claude plugin cache
(versioned) first, then ~/.codex/skills, ~/.agents/skills, repo checkout,
./.skills/last30days (npx skills install dir), CWD, and GEMINI_EXTENSION_DIR.

Also drops Codex native plugin support: .codex-plugin/plugin.json is deleted,
the badge VERSION jq fallback in line 108 stops looking at it, and render.py's
_skill_version no longer scans for it. Codex users install via `npx skills add`
or the per-harness skill dir going forward.

render.py::_skill_version gains a SKILL.md frontmatter fallback so the badge
no longer emits `v?` on install dirs that sync.sh populates (which don't
include .claude-plugin/plugin.json).
2026-05-15 21:44:55 -07:00
Trevin Chow 54db014c7c fix(sync): point sync.sh at this repo's plugin cache, not the private repo's (#402)
sync.sh was written against the layout of mvanhorn/last30days-skill-private
(`.../cache/last30days-skill-private/last30days-3/{version}`) and that path
was never updated when this public repo got its own copy. Running sync.sh
from here populated the BETA channel's cache (`/last30days-beta`) instead
of this repo's own `/last30days` cache, so devs working in this repo could
not test their changes via the public slash command without waiting for a
marketplace release.

Path now derives from this repo's own manifests:
- marketplace name `last30days-skill` (.claude-plugin/marketplace.json)
- plugin name      `last30days`       (.claude-plugin/plugin.json)

Drops the `last30days-3-nogem` target along with it - that's a private-repo
variant with no public equivalent.

Updates test_sync_cache_path_uses_skill_version to assert the new path
pattern and clarifies the COMMON_TARGETS comment so the next person editing
it understands which marketplace/plugin name segments come from where.
2026-05-15 21:43:23 -07:00
Tobi 4f6b86c456 feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES
config key. Two coupled changes:

1. pipeline.available_sources() filters out any source listed in
   config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive,
   whitespace-tolerant) before returning.
2. hooks/scripts/check-config.sh "Ready — N sources active" banner
   subtracts excluded sources from the ScrapeCreators +3 (Reddit
   comments + TikTok + Instagram) so the count matches what the
   pipeline actually runs.

Use case: skip TikTok/Instagram on runs where you only want
text-substantive sources, without unsetting SCRAPECREATORS_API_KEY
(which would also kill Reddit comments). The existing INCLUDE_SOURCES
allowlist covers Perplexity opt-in but doesn't cover this denylist case
— tiktok and instagram are added unconditionally when
SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key.

Tests (tests/test_pipeline_v3.py::TestExcludeSources):
- excludes tiktok+instagram when listed
- no exclusion when env unset or empty string
- case-insensitive + whitespace-tolerant parsing
- works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
2026-05-16 00:54:18 +02:00
Trevin Chow 80a1a47eef refactor: drop requests dep, route all providers through lib/http urllib wrapper (#393)
Five provider modules (pinterest, threads, instagram, tiktok, youtube_yt)
and watchlist.py each carried a try/except `requests` import with parallel
urllib + requests branches. The urllib path already used the
stdlib-only wrapper at `lib/http.py` (retries, 429 handling, HTTPError).
This collapses every dual-branch into a single `http.get`/`http.post`
call and removes the `requests` dependency from `pyproject.toml`.

Also drops 4 transitive deps (urllib3, certifi, charset-normalizer, idna)
from the lockfile, leaving the skill stdlib-only at runtime.

Tests for tiktok comments and watchlist delivery were rewritten to mock
`lib.http` directly instead of the now-removed `requests` module.

Out of scope but flagged during review: the 13 surviving SC call sites
share a near-identical scaffold and would benefit from a
`http.scrapecreators_get(url, params, token, ...)` helper. Filed for a
follow-up PR rather than expanding scope here.
2026-05-15 08:07:43 -07:00
Matt Van Horn c845f483d6 fix(sync): bump cache target to 3.2.1 to match SKILL.md (#397)
test_sync_cache_path_uses_skill_version asserts that sync.sh's plugin
cache path includes the version from SKILL.md frontmatter. The frontmatter
moved to 3.2.1 in #371 but sync.sh still pointed at 3.2.0, leaving CI red
on every PR.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 08:06:33 -07:00
Matt Van Horn dc934ddb6a feat(digg): rename to 'Digg' and bump per-cluster post limits (#372)
* feat(digg): bump POSTS_PER_CLUSTER to 5 and render limit to 3

Match the per-item enrichment cap and inline-display cap used by the
other sources (Reddit, HN, YouTube, TikTok, GitHub all use 5 fetched /
3 displayed). At the previous 3/2 caps the engine routinely truncated
cluster context — a recent run on cli-printing-press lost the Jason
Calacanis quote tweet entirely because the display cut off after Garry
Tan's first two posts.

* feat(digg): rename 'Digg AI 1000' to 'Digg' in user-facing strings

Drop the 'AI 1000' suffix from the footer line, source label, inline
quote attribution ('via Digg'), why_relevant, container, mock title,
SKILL.md source list, and README sources table. Internal code comments
and docstrings still reference the upstream Digg AI 1000 product.

Bumps version to 3.2.1 and adds a CHANGELOG entry covering this rename
and the POSTS_PER_CLUSTER / render-limit bumps from the prior commit.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 21:04:23 -07:00
Matt Van Horn 80392061d4 chore(release): v3.2.0 (#371)
Release / build-and-release (push) Has been cancelled
* chore(release): v3.2.0

Bumps plugin/marketplace/codex/pyproject versions from 3.1.1 to 3.2.0.
Promotes the Unreleased CHANGELOG entries (--emit=html, Digg AI 1000
source) to the 3.2.0 release section.

* chore(release): bump SKILL.md header and sync.sh path to 3.2.0

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 19:29:16 -07:00
Matt Van Horn c04bd67922 feat: add Digg AI 1000 as an opt-in source (#370)
* feat(digg): add Digg AI 1000 source module with cluster search and post enrichment

- search_digg shells out to digg-pp-cli with --since 30d --agent
- parse_digg_response normalizes clusters to last30days dict shape
- enrich_with_top_posts attaches top-ranked X posts to top-K clusters
- shutil.which gate plus subproc.run_with_timeout discipline matches
  bird_x.py / youtube_yt.py patterns

25 unit tests cover parse, age window, relevance, binary-missing
fallback, timeout recovery, and partial enrichment failures.

* feat(digg): wire Digg source into pipeline, normalize, signals, and render

pipeline.py:
- Import digg, add to MOCK_AVAILABLE_SOURCES, gate via shutil.which
- Dispatch case calls search_digg + parse_digg_response, runs
  enrich_with_top_posts at default/deep depth
- Mock fixture includes one enriched cluster + one bare cluster

normalize.py:
- _normalize_digg maps cluster dicts to SourceItem with
  container='Digg AI 1000' and metadata.posts pass-through

signals.py:
- SOURCE_QUALITY['digg'] = 0.85 (top tier alongside YouTube,
  reflecting Digg's curatorial layer)
- ENGAGEMENT_WEIGHTS['digg'] balances postCount, uniqueAuthors,
  and the rank_score derived from Digg's curatorial position

render.py:
- SOURCE_LABELS['digg'] = 'Digg AI 1000'
- _FOOTER_SOURCES adds '⛏️ Digg AI 1000' line after GitHub
- ENGAGEMENT_DISPLAY mirrors footer keys
- New _digg_posts_for + _format_digg_quote helpers emit inline
  '@handle via Digg AI 1000' quotes for clusters with attached X
  posts; both compact and full-dump renderers call them

* feat(digg): polish per-item engagement display and progress label

- ENGAGEMENT_DISPLAY for digg uses 'posts' / 'auth' to match the
  codebase abbreviation convention (HN: 'pts'/'cmt', X: 'rt'/'re')
- Footer item word changes from 'story' to 'cluster' to dodge the
  pre-existing naive plural in _footer_line_for_source ('storys')
  and to match Digg's actual data model
- ui.py SOURCE_COMPLETION_META adds digg with correct 'cluster'/
  'clusters' plural so 'Research complete' shows 'Digg: N clusters'

* feat(digg): document Digg AI 1000 source in skill, README, and changelog

- planner.py SOURCE_CAPABILITIES adds digg with discussion/social/link
  capabilities so the planner offers it through the standard fanout
- SKILL.md ACTIVE_SOURCES_LIST gate includes 'which digg-pp-cli' check
  and the source list / available-sources line names digg as opt-in
- README.md Sources table adds the Digg AI 1000 row with the activation
  gate so first-time readers see what they get
- CHANGELOG.md Unreleased section calls out the source addition

* fix(digg): enrich post-dedupe so brief survivors carry inline quotes

Pipeline dispatch was attaching X posts to the top-3 items returned by
search, but dedupe later picked different survivors when multiple
clusters compared similar (common for trending topics). The brief
ended up showing clusters with no posts attached even though
enrichment ran successfully on positions 0-2.

Move enrichment to _finalize_items_by_source. The new
digg.enrich_source_items helper reads metadata['clusterUrlId'] and
writes metadata['posts'] in place on the SourceItems that actually
survive dedupe.

Verified live on 'openclaw': 2 surviving clusters, both now carry
real X-post quotes from @sama and @jeremyphoward attributed
'via Digg AI 1000'.

Adds 3 unit tests covering survivor enrichment, non-digg skip, and
clusterUrlId fallback to item_id.

* test(digg): relax live off-topic test to check shape, not emptiness

Digg's live search uses fuzzy/popularity fallback, so an impossible
token can still return some loosely-related clusters. The contract
the pipeline depends on is shape (results is always a list);
token-overlap relevance handles the noise downstream.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 19:05:41 -07:00
Anurag Chakradhar ed455ca036 Claim contributor entry — @thinkun 2026-05-08 17:15:08 +10:00
Trevin Chow b1773be8f3 feat(emit): --emit=html for shareable self-contained briefs (#332)
Adds a one-command shareable HTML mode to /last30days. The skill detects
HTML intent (explicit --emit=html / --emit:html / --html flag in
$ARGUMENTS, or natural-language asks like "give me a shareable brief",
"for Slack", "export as HTML"), runs the normal research + chat synthesis
flow, then saves a self-contained HTML file to
~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat
as usual; the HTML is an additional artifact for sharing.

User experience:

  /last30days OpenClaw --emit=html
  /last30days OpenClaw, give me an HTML brief for Slack

Synthesis prints to chat. Last line of the response: "📎 Shareable brief
saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it
into a message, browser-print to PDF, email it.

Architecture:

  - SKILL.md gets a small detection block (triggers + early exit +
    MUST/MUST NOT rules + rationale) that points to a reference file.
  - references/save-html-brief.md owns the implementation: capture the
    synthesis verbatim into a temp file via heredoc, invoke the engine
    with --emit=html --synthesis-file, save to disk, append the
    confirmation line to chat.
  - lib/render.py exposes render_for_html(report, synthesis_md=None) and
    render_for_html_comparison(...) -- clean markdown for HTML
    conversion. Omits debug file header, model-facing safety note, and
    data quality warnings (those stay in engine stderr; recipients can't
    act on them in a shared artifact).
  - lib/html_render.py is a new module: ~200-line CSS template (dark
    mode default, prefers-color-scheme switch, print stylesheet, mobile
    breakpoint), stdlib-regex markdown-to-HTML converter, marker-based
    META + engine-footer wrapping, PROSE_LABELS registry promoting plain
    -text labels to <h2>, colophon builder.
  - last30days.py adds --emit=html argparse choice and --synthesis-file
    PATH flag (engine still callable directly without the skill in the
    loop).

Design:

  - Voice-led research brief, not corporate report. Inter + JetBrains
    Mono via Google Fonts with full system fallbacks (no FOIT, works
    offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp:
    body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label
    20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px.
  - 720px max-width, generous whitespace, no card layouts or shadows.
  - Print stylesheet: light theme, A4 margins, [href]::after URL
    footnotes, page-break-inside:avoid on the engine footer.

Templated (locked) shell:

  - HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS
    inline.
  - .badge / .meta / .engine-footer / .colophon containers.

Flexible (role-based):

  - <h2> rendering covers BOTH plain ## headers (comparison mode per
    LAW 4 exception) AND promoted prose labels via PROSE_LABELS
    registry. Adding a new SKILL.md prose label is a one-line tuple
    addition; no CSS or template changes.
  - Marker-based engine boundaries (<!-- META: ... -->,
    <!-- PASS-THROUGH FOOTER -->) survive the markdown converter and
    get promoted post-conversion. Robust to engine output format
    changes.
  - Generic markdown-to-HTML for body content; future SKILL.md additions
    (new sections, tables, blockquotes) render correctly without code
    changes.

Tests: 30 new tests in tests/test_html_render.py covering snapshots
(rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose
label promotion, warning exclusion from artifact, parseability via
html.parser, no-script self-containment.

No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip
dependencies, no JavaScript anywhere.
2026-05-02 11:30:22 -07:00
Ilia Alshanetsky 5b87cca886 fix(xurl): treat PermissionError from PATH lookup as unavailable (#322)
is_available() only caught FileNotFoundError and TimeoutExpired. On WSL,
a /mnt/c/.../WindowsApps entry on $PATH returns EACCES during exec, and
Python raises PermissionError. That escaped is_available() and crashed
pipeline.diagnose() before any source ran.

Catch OSError instead. It covers FileNotFoundError, PermissionError, and
any other spawn-time OS error, so a non-executable xurl on PATH falls
through to the next backend instead of aborting the run.
2026-04-26 14:16:14 -07:00
Ilia Alshanetsky bbf892aecc refactor: extract subprocess cleanup into shared subproc helper (#210)
bird_x.py and youtube_yt.py had four near-identical copies of the same
subprocess cleanup dance (Popen + os.setsid + communicate(timeout) +
SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to
lib.subproc.run_with_timeout(), which:

- runs the child in its own process group via os.setsid where available
- raises SubprocTimeout on timeout
- on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s
- accepts an on_pid callback so bird_x can still register child PIDs
  with last30days.register_child_pid for whole-process cleanup
- captures stdout/stderr as strings in a SubprocResult dataclass

Migrated call sites: _run_bird_search, search_handles inner worker,
search_youtube, fetch_transcript. With the helper in place, the signal
and subprocess imports became dead in both files (plus os in
youtube_yt) and went with them.

Tests: 9 new subproc tests cover success, non-zero exit, stderr capture,
timeout-raises, timeout-kills-group, missing-command, env passthrough,
PID callback, and callback-exception suppression. test_env_v3 and
test_youtube_yt patch subproc.run_with_timeout instead of the removed
bird_x.subprocess and yt-dlp subprocess.
2026-04-25 14:17:47 -07:00
Ilia Alshanetsky 2acbf8a869 perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)
1. N+1 queries in store.store_findings()
   The old loop ran one SELECT per finding to check existence, then one
   INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
   Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
   dict, then executemany() handles all inserts and updates. Query count
   stays constant regardless of batch size. Benchmark on 500 findings:
   ~30ms to ~20ms; gap widens on slower storage.

2. O(n^2) source_items dedup in fusion.weighted_rrf()
   Merging an item into an existing candidate ran any(existing.source ==
   ... for existing in candidate.source_items), linearly scanning a list
   that grew with each merge. At 40 candidates with 20 source_items each,
   fusion went quadratic. Now tracks (source, item_id) tuples in a
   per-candidate set for O(1) lookup. The source_items list itself is
   unchanged since other code iterates it.

3. Dead code removal
   - providers.GeminiClient.ground_search() and .url_context_json(): zero
     callers. Deleted.
   - render._top_comment_excerpt(): zero callers. Deleted.
   - env.is_reddit_available(): one-line wrapper around get_reddit_source.
     Callers can check get_reddit_source(config) is not None directly.
2026-04-25 14:17:17 -07:00
Ilia Alshanetsky e6b89f2644 perf: cache PreparedQuery per stream, skip double-normalize in dedupe (#282)
Scoring hot path (_normalize_score_dedupe) re-tokenized the same
ranking_query ~240x per stream: once per item for local_relevance,
plus ~5x per item across snippet windows. Query tokens are immutable
within a stream, so compute them once as relevance.PreparedQuery and
thread through signals.annotate_stream and snippet.extract_best_snippet.

dedupe._PreparedText called normalize_text twice: once in __init__ and
again via get_ngrams. Factor out _ngrams_of_normalized so the prepared
path skips the redundant pass while get_ngrams keeps its public contract.

Behavior unchanged.
2026-04-25 14:16:57 -07:00
Ilia Alshanetsky 2c2755b49c refactor(normalize): extract _join_comment_excerpts helper (#283)
_normalize_reddit, _normalize_hackernews, and _normalize_github inlined
the same 5-line comprehension to stringify and space-join the first 3
top_comments' excerpt field. Extract one helper, call it from all three.

The comment field name varies per source (Reddit/GitHub use 'excerpt',
HN uses 'text'), so it's passed as a parameter. Behavior unchanged.
2026-04-25 14:16:50 -07:00
Ilia Alshanetsky 18b5658674 chore: remove orphan test for deleted generate-synthesis-inputs script (#205)
tests/test_generate_synthesis_inputs_v3.py imported a script that no
longer exists in the repo. The test failed with FileNotFoundError on
every run.
2026-04-25 14:16:39 -07:00
Matt Van Horn 145adc9f56 Merge pull request #321 from tmchow/tmchow/review-plugin-json
chore: align plugin manifests, add Codex AGENTS.md
2026-04-25 12:34:12 -07:00
Trevin Chow b100caf2df fix(plugin): restore marketplace plugin version
`tests/test_plugin_contract.py::test_versions_match_across_manifests`
enforces that every version-bearing surface agrees: pyproject.toml,
SKILL.md, both plugin.json files, AND the marketplace plugin entry.
The Claude Code spec says plugin.json wins when both are set, but this
repo deliberately mirrors the version across all surfaces and tests it.
Restore the field at 3.1.1 to satisfy the contract.
2026-04-24 23:21:02 -07:00
Trevin Chow dc0cb9850b chore: add AGENTS.md pointing to CLAUDE.md
Codex CLI reads AGENTS.md for repo-level context the way Claude Code reads CLAUDE.md. Delegate to the existing CLAUDE.md so both harnesses share one source of project instructions.
2026-04-24 23:16:32 -07:00
Trevin Chow ceec99b24c chore(plugin): clean up plugin manifests
- Remove no-op `"hooks": {}` from .claude-plugin/plugin.json (auto-discovery from hooks/hooks.json picks up the SessionStart hook).
- Remove redundant `version` from marketplace.json plugin entry; plugin.json is the source of truth per the spec.
- Sync description / longDescription across .claude-plugin and .codex-plugin manifests so all surfaces show the same copy.
2026-04-24 23:16:29 -07:00
Dave Morin d1823a2d05 feat: add PR and issue templates for contributor workflow (#296)
Adds structured templates to help contributors submit higher-quality
PRs and issues. PR template includes testing checklist (pytest, sync.sh).
Issue templates use YAML forms for bug reports and feature requests.

Fixes #251
2026-04-24 10:49:06 -07:00
Claire Novotny 17caa0526d ci: validate plugin contract on pull requests 2026-04-24 12:05:39 -04:00
Claire Novotny f03cb866aa fix: address plugin layout review feedback 2026-04-24 11:52:48 -04:00
Claire Novotny 72495c1c14 Restructure as Codex plugin 2026-04-23 20:15:02 -04:00
Matt Van Horn 1f7e85a03f chore(release): v3.1.0 — consolidate 3.0.10-3.0.14 + OpenClaw republish prep (#314)
Release / build-and-release (push) Has been cancelled
- Bump plugin.json to 3.1.0
- CHANGELOG entry consolidating 3.0.10-3.0.14 dev cycle and noting OpenClaw republish
- Fix broken README link: skills/last30days/SKILL.md -> SKILL.md

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-22 21:56:07 -07:00
Matt Van Horn 949bcf8942 feat: vs mode N full passes + --competitors auto-discovery + (/Last30Days) title (#312)
* feat: vs mode runs N full passes; --competitors wraps vs with auto-discovery

Unifies vs-mode and --competitors onto one fanout architecture. A topic
containing "vs" / "versus" now runs N full pipeline.run() calls in parallel
(reverting the one-pass latency optimization that removed per-entity
depth); --competitors becomes a SKILL.md-level shortcut where the hosting
reasoning model (Claude Code, Codex, Hermes, Gemini) discovers N peers via
its own WebSearch, runs Step 0.55 per entity, and invokes the engine with
a vs-topic + --competitors-plan JSON.

Changed:
- vs-mode: N full passes in parallel via fanout (was 1 merged pass).
- --competitors: SKILL.md shortcut for vs-mode-with-discovery. Engine flag
  kept for headless/cron use. LAW 7-style stderr reframed to lead with the
  hosting-model path (use WebSearch + --competitors-plan) instead of
  BRAVE_API_KEY. Footer BRAVE/SERPER nudge suppressed when --plan or
  --competitors-plan present (hosting model already has WebSearch).

Added:
- --competitors-plan JSON flag: per-entity {x_handle, x_related, subreddits,
  github_user, github_repos, context}. Accepts inline JSON or file path.
  subrun_kwargs_for helper is the single source of truth for per-entity
  kwargs — no closure-default fallthrough from main scope.
- Per-entity save files: each entity's sub-run produces its own
  {slug}-raw.md with a single-row Resolved Entities block.
- --polymarket-keywords filter for ambiguous single-token topics.

Fixed:
- test_competitor_subrun_isolation regression suite locks in 3.0.12's
  no-leak invariant (main flags do not inherit into peer sub-runs).
- Updates test_regression.py for the new comparison-mode payload shape.

Bumps plugin.json to 3.0.13. 1,219 tests passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: comparison title attribution — (Last 30 Days) → (/Last30Days)

User feedback on 3.0.13 dogfood runs (Kanye vs Drake, Mercer Island,
Figma): the comparison-mode synthesis title should attribute to the
slash command rather than restate the date range.

Three SKILL.md occurrences updated. Pure documentation change. Bumps to
3.0.14.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:31:00 -07:00
Matt Van Horn 00d01933e0 fix: per-entity Step 0.55, LAW 7 sub-run quiet, default 2, canonical SKILL.md (#311)
Four fixes based on 2026-04-22 test-window feedback on v3.0.11 --competitors:

- Each competitor sub-run now runs Step 0.55 (X handle / subreddits /
  GitHub) via resolve.auto_resolve inside the fanout closure. Deep-copied
  config per entity prevents _auto_resolve_context leak across sub-runs.
  Resolved data stored on report.artifacts["resolved"] for the renderer.
- New internal_subrun keyword on planner.plan_query and pipeline.run
  suppresses the LAW 7 "No --plan passed" stderr for engine-internal
  fan-out only. Default path unchanged.
- Default --competitors count is now 2 (3-way total). --competitors=N
  still customizes; range 1..6.
- SKILL.md STEP 0 canonical-path self-check forces readers who loaded
  from marketplaces/ (auto-restored to origin/main, stale) to re-read
  from plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md.
  Two of three 2026-04-22 test windows hit this stale-path trap.
- New ## Resolved Entities block in render_comparison_multi shows
  per-entity handles/subs/github for debug visibility.

Bumps plugin.json to 3.0.12. 12 new tests; 1,175 total passing.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:30:08 -07:00
Matt Van Horn 5f054380c5 feat: --competitors flag for auto-discovered comparison fan-out (#308)
Pass `--competitors` on a single-entity topic and the engine auto-discovers
2-6 peer entities via web search, runs the full pipeline on each in
parallel, and returns one N-way comparison reusing the existing 9-axis
Head-to-Head scaffold. `last30days OpenAI --competitors` resolves to
Anthropic + xAI + Google Gemini; `last30days Kanye West --competitors`
resolves to Drake + Kendrick Lamar + one more peer.

- New CLI flags: --competitors, --competitors=N, --competitors-list
- New scripts/lib/competitors.py — mirrors resolve.auto_resolve pattern
  (web search + deterministic text extraction, no internal LLM)
- New scripts/lib/fanout.py — ThreadPoolExecutor orchestrator; per-entity
  failures degrade gracefully as long as >=2 entities survive
- Multi-report render in scripts/lib/render.py reuses the comparison
  scaffold for the synthesis table
- LAW 7-style stderr when no backend and no list, pointing the hosting
  reasoning model at --competitors-list
- 38 new tests across CLI parsing, discovery, fanout, and rendering

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:28:36 -07:00
Matt Van Horn ff21243517 Merge pull request #130 from chaosreload/feat/xurl-x-search
feat: add xurl CLI as alternative X search backend (official API v2 via OAuth2)
2026-04-22 18:55:22 -07:00
Matt Van Horn 4e91f4e754 fix: Step 0.55 category-peer subreddit expansion (#305)
* feat(resolve): category-peer subreddit map for Step 0.55

Introduces scripts/lib/categories.py with a curated category->peer-subs
map and wires scripts/lib/resolve.py auto_resolve() to merge peers into
the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode:
a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and
missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where
prompting techniques actually live.

Map is static, curated, ~11 categories (ai_image_generation,
ai_video_generation, ai_music_generation, ai_coding_agent,
ai_agent_framework, ai_chat_model, saas_screen_recording,
saas_productivity, prediction_markets, crypto_defi, dev_tool_cli).
First-match-wins ordering from most-specific to least-specific.
Compound-term patterns only (no bare common nouns like "image", "ai").

auto_resolve now:
- calls detect_category(topic) after _extract_subreddits
- merges peer_subs case-insensitively, caps at MAX_SUBS (10)
- preserves every WebSearch-returned sub (freshest signal)
- emits [Resolve] Matched category=<id>, adding peers: <list> on stderr
  only when peers were actually added
- returns new "category" key in the result dict for observability
- wraps classifier in try/except so failures degrade to unwidened list

Includes drive-by: test_full_resolve / test_partial_failure
searches_run expectations bumped from 3->4 / 2->3 to match the current
queries dict (subreddit + news + x_handle + github).

* feat(skill): Step 0.55 category-peer expansion and self-check

Adds Section 2a (category-peer expansion, MANDATORY for product topics)
and the Step 0.55 self-check checkpoint that fires immediately before
the Resolved block displays. Structural mirror of the engine-side
categories.py map: same categories, same peer subs, same priority
order.

The model-side path now:
- Applies category-peer expansion to the WebSearch-resolved subs on
  every product-in-a-known-category run.
- Emits the (+ <category_id> peers) annotation on the Reddit line of
  the Resolved block as the observable contract. Absence on a
  product-in-a-known-category topic is a Step 0.55 regression.
- Runs a self-check before emitting Resolved: "does the resolved list
  include at least 2 peer subs for the matched category? if not,
  widen NOW and do not run the engine yet."

Mirror of the Python map lives inside Step 0.55 as a table for the
model to pattern-match against; extrapolation to unlisted categories
is explicitly allowed. Worked example (the exact failing query)
appears below the table so reviewers can see before/after at a glance.

Both changes land inside the existing Step 0.55 block. No new
top-level section, no new LAW. LAWs 1-6 wording unchanged.

* test: end-to-end regression for GPT Image 2 failure mode

Stubs grounding.web_search to return the OpenAI-only subs that caused
the 2026-04-22 failure, then asserts that auto_resolve widens to
include the image-gen peers and emits the [Resolve] Matched
category=ai_image_generation stderr line. Covers the cap boundary
and the uncategorized-topic no-op path.

Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is
documentation-grade (not parsed by tests) and shows the pre-fix vs
post-fix Resolved block shape so reviewers can evaluate future
categories.py edits against the original bug.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-22 14:31:39 -07:00
Matt Van Horn 952a876536 feat: attribute top comments with u/ and @ handles in evidence lines (#292)
Reddit, TikTok, YouTube, Instagram, Bluesky, X and Threads top comments
now render as u/author or @handle in the evidence block, instead of the
generic "Comment (...)" label. The enrichment adapters already captured
author; only the render layer was dropping it.

Also fixes the TikTok adapter to prefer user.unique_id (the @handle) over
user.nickname (display name) so attribution round-trips to a profile URL.

Legacy "Comment (...)" shape is preserved when author is empty, [deleted],
or [removed].

Bumps to 3.0.10.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 08:23:47 -07:00
Matt Van Horn 1f23e3f980 test: skip docs/ in memory-dir-paths sweep (#291)
The regression test from #290 walks the filesystem via Path.rglob, so
docs/plans/*.md files (gitignored, created by internal planning) trip
the assertion on any dev machine that has run ce:plan in this repo.
Fresh clones and CI never see them, but local runs fail.

Adding docs to skip_dirs keeps the guard narrow to first-class source
files while letting internal planning docs reference old paths
verbatim.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 07:05:35 -07:00
Dave Morin 5269806a75 Make memory directory configurable (#290) 2026-04-21 07:04:38 -07:00
weichao adac4c377a feat: add xurl CLI as alternative X search backend
Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.

xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.

Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window

New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)

Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
  get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
  _search_x() priority chain
- SKILL.md: document xurl setup option
2026-04-21 07:05:20 +00:00
Matt Van Horn 3107325443 feat: inline markdown links on narrative citations (#289)
Inline markdown links on every narrative citation (@handle, r/sub,
publication, YouTube channel, TikTok/Instagram creator, Polymarket
market). Raw URL strings remain forbidden. Plain-text fallback when the
raw data has no URL for a specific source.

Commit 1 (790e5bc) added the citation rule in CITATION PRIORITY / URL
FORMATTING. Live tests showed the rule was deployed but consistently
skipped because it lived at line 1224, below the agent's chunked-read
window. Commit 2 (5864c687) hoists the rule into the VOICE CONTRACT
LAW block as LAW 8, at line 167 - inside the guaranteed-loaded top
band alongside LAWs 1-7. Same pattern that fixed v3.0.6 (invented
titles), disaster #2 (stripped bold), disaster #3 (trailing Sources),
and the 2026-04-19 Hermes evidence-dump disaster.

No Python engine changes. Rule is prompt-only; the deterministic
stats footer (LAW 5) is unchanged.

Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md
2026-04-20 09:49:03 -07:00
211 changed files with 12717 additions and 2112 deletions
+20
View File
@@ -0,0 +1,20 @@
{
"name": "last30days-skill",
"interface": {
"displayName": "Last 30 Days"
},
"plugins": [
{
"name": "last30days",
"source": {
"source": "local",
"path": "./"
},
"policy": {
"installation": "AVAILABLE",
"authentication": "ON_INSTALL"
},
"category": "Research"
}
]
}
+5 -4
View File
@@ -1,16 +1,17 @@
{ {
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "last30days-skill", "name": "last30days-skill",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
"owner": { "owner": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
"url": "https://github.com/mvanhorn" "url": "https://github.com/mvanhorn"
}, },
"metadata": {
"description": "Marketplace hosting the Last 30 Days research plugin."
},
"plugins": [ "plugins": [
{ {
"name": "last30days", "name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.0.9", "version": "3.2.4",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
"url": "https://github.com/mvanhorn" "url": "https://github.com/mvanhorn"
+2 -3
View File
@@ -1,6 +1,6 @@
{ {
"name": "last30days", "name": "last30days",
"version": "3.0.9", "version": "3.2.4",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
@@ -10,6 +10,5 @@
"homepage": "https://github.com/mvanhorn/last30days-skill", "homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill", "repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT", "license": "MIT",
"keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"], "keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"]
"hooks": {}
} }
-3
View File
@@ -1,3 +0,0 @@
{
"name": "last30days"
}
+4 -4
View File
@@ -1,5 +1,6 @@
# Exclude non-runtime files from `git archive` output. # Exclude non-runtime files from `git archive` output.
# Used by scripts/build-skill.sh to produce a claude.ai-upload-ready .skill file. # Used by skills/last30days/scripts/build-skill.sh to produce a
# claude.ai-upload-ready .skill file from the canonical skills/last30days tree.
# See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md. # See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
# Anthropic canonical skill-packaging excludes # Anthropic canonical skill-packaging excludes
@@ -32,9 +33,8 @@ release-notes.md export-ignore
CHANGELOG.md export-ignore CHANGELOG.md export-ignore
uv.lock export-ignore uv.lock export-ignore
# Platform adapters - skill-upload path is platform-agnostic # Platform adapters are kept in git archives because Claude Code and Codex
.agents/ export-ignore # plugin installs use the same repository archive as their source payload.
.codex-plugin/ export-ignore
.hermes-plugin/ export-ignore .hermes-plugin/ export-ignore
# CI workflows - repo-only, not needed at skill runtime # CI workflows - repo-only, not needed at skill runtime
+53
View File
@@ -0,0 +1,53 @@
name: Bug Report
description: Report a bug or unexpected behavior
labels: [bug]
body:
- type: textarea
id: summary
attributes:
label: Summary
description: What happened?
placeholder: Describe the bug in 1-2 sentences.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to Reproduce
description: How can we reproduce this?
placeholder: |
1. Run `python3 scripts/last30days.py "topic" --emit compact`
2. ...
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected Behavior
description: What should have happened?
validations:
required: true
- type: textarea
id: traceback
attributes:
label: Error / Traceback
description: Paste the full traceback or error output.
render: text
- type: dropdown
id: install
attributes:
label: Install Method
options:
- Claude Code plugin
- Gemini CLI extension
- Codex plugin
- Hermes skill
- Manual (git clone)
- Other
validations:
required: true
- type: input
id: os
attributes:
label: OS
placeholder: macOS 15.4, Ubuntu 24.04, Windows 11, etc.
@@ -0,0 +1,24 @@
name: Feature Request
description: Suggest a new feature or improvement
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What problem does this solve?
placeholder: When I try to ..., I can't ...
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed Solution
description: How should this work?
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives Considered
description: Other approaches you thought of (optional).
+19
View File
@@ -0,0 +1,19 @@
## Summary
<!-- What does this PR do? 1-3 sentences. -->
## Changes
<!-- Bullet list of what changed. Reference files if helpful. -->
-
## Testing
<!-- How did you verify this works? -->
- [ ] Ran `uv run python -m pytest -q --tb=short`
## Related Issues
<!-- Link issues: Fixes #123 or Relates to #456 -->
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
- name: Build .skill artifact - name: Build .skill artifact
run: | run: |
bash scripts/build-skill.sh bash skills/last30days/scripts/build-skill.sh
test -f dist/last30days.skill test -f dist/last30days.skill
- name: Create GitHub release - name: Create GitHub release
+67
View File
@@ -0,0 +1,67 @@
name: Security
on:
pull_request:
push:
branches:
- main
workflow_dispatch:
permissions:
contents: read
jobs:
dependency-audit:
name: Dependency audit
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Set up Python
run: uv python install 3.12
- name: Export locked dependency set
run: |
uv export \
--locked \
--all-groups \
--no-hashes \
--format requirements.txt \
--output-file /tmp/last30days-requirements.txt
# Advisory-first: visibility before enforcement. This repo handles API keys,
# cookies, browser tokens, and local env files, so dependency CVEs should be
# visible in CI logs even before the project has a clean blocking baseline.
# Set continue-on-error: false once a clean baseline run is confirmed.
- name: Run pip-audit against locked dependencies
continue-on-error: true
run: uvx --python 3.12 pip-audit -r /tmp/last30days-requirements.txt --progress-spinner=off
secret-scan:
name: Secret scan
runs-on: ubuntu-latest
steps:
- name: Checkout full history for diff-aware scanning
uses: actions/checkout@v4
with:
fetch-depth: 0
# Advisory-first: this reports verified secrets in pull requests and pushes to
# main, but does not block merges until maintainers confirm a clean baseline.
# The TruffleHog action automatically scans the PR range for pull_request
# events and the pushed commit range for push events.
# Set continue-on-error: false once a clean baseline run is confirmed.
# Contributor policy: never commit real secrets in fixtures, tests, docs, or
# examples; use obvious dummy values and env-based auth patterns instead.
- name: Run TruffleHog OSS secret scan
if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: trufflesecurity/trufflehog@v3.95.2
continue-on-error: true
with:
path: ./
version: v3.95.2
extra_args: --only-verified
+26
View File
@@ -0,0 +1,26 @@
name: Validate
on:
pull_request:
push:
branches:
- main
permissions:
contents: read
jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Set up Python
run: uv python install 3.12
- name: Run test suite
run: uv run pytest
+38
View File
@@ -0,0 +1,38 @@
# last30days Skill
Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
## Structure
- `skills/last30days/SKILL.md` — canonical skill definition
- `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
- `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
- `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
## Orientation
- This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
- Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration is incomplete — the model invoking the skill won't know the flag exists.
- README and PR examples show `/last30days <topic>` first. Direct CLI invocation (`python3 scripts/last30days.py ...`) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
- Slash commands don't pass shell mechanics through. `/last30days OpenClaw --emit=html | pbcopy` is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full `python3 ...` with explicit flags and a real shell).
## Commands
```bash
# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only)
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # one-time: symlink this repo into every detected harness's skill dir
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` creates symlinks from each detected harness's skill dir to this repo. Edits in the working tree propagate live to every harness — no re-deploy step needed.
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Security hygiene
- Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
- Use the env-based auth patterns in `skills/last30days/scripts/lib/env.py`; tests and fixtures must use obvious dummy values only.
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+116 -3
View File
@@ -5,6 +5,119 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- `LAST30DAYS_YOUTUBE_SSH_HOST` env var: when set, yt-dlp YouTube search invocations are routed through `ssh <host>` for residential-IP egress. Bypasses YouTube's bot-wall on datacenter IPs (Hetzner/DigitalOcean/AWS) where `ytsearch:` returns 0 results regardless of cookies (the IP fingerprint is checked first). The named host must be configured in `~/.ssh/config` and have yt-dlp installed. Host value is validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection (e.g. a leading `-` masquerading as a flag). The transcript path is unchanged (uses the existing HTTP fallback when SSH-routing is on, since the timedtext API isn't bot-walled).
### Changed
- Replace the SKILL_ROOT resolver loops in Step 1 and comparison-mode with a single `SKILL_DIR` substitution pattern. The model templates the absolute path of the SKILL.md's own directory (which it always knows from the Read tool result); the bash block just validates that `scripts/last30days.py` lives there. Removes ~80 lines of bash across the two locations. Fixes a real bug: the previous resolver could pick a different install than the SKILL.md the model loaded from (spec-vs-engine divergence) and didn't enumerate harnesses like Hermes at all. The simplification works for any harness without enumeration because it just uses wherever SKILL.md was loaded from. STEP 0's marketplaces-stale-clone hop is unchanged.
- Rename "Digg AI 1000" to just "Digg" in user-facing output (footer line, source label, inline-quote suffix, why_relevant, container attribution). Internal references to the upstream Digg AI 1000 product remain in code comments and docstrings.
- Bump `POSTS_PER_CLUSTER` from 3 to 5 and the render-side display limit from 2 to 3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous 3/2 caps routinely truncated cluster context (e.g. dropped a Jason Calacanis quote tweet on a `cli-printing-press` run).
- Rewrite SKILL.md path resolution. STEP 0 narrows from a global canonical-path enforcement to a Claude-Code-marketplaces-only stale-clone guard. Step 1 SKILL_ROOT resolver walks a single precedence list (Claude plugin cache, then `~/.codex/skills/`, `~/.agents/skills/`, repo checkout, `./.skills/last30days` for `npx skills add`, CWD, Gemini). Adds SKILL.md frontmatter fallback to `render.py::_skill_version` so the badge no longer prints `v?` on installs that don't include `.claude-plugin/plugin.json`.
- Switch SKILL.md's `--plan` and `--competitors-plan` invocation templates from inline single-quoted JSON to heredoc-written tmpfiles. Apostrophes in resolved context strings ("McDonald's", "people's choice", "developer's") previously closed the outer single-quote and broke shell parsing before the engine started — observed in a Codex run during PR #400 testing. The engine's `parse_plan()` / `parse_competitors_plan()` already supported file paths (via `os.path.isfile()` probe); only the template prose changed. Fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403).
### Removed
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1. Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`.** The maintainer dev-deploy script is gone. Every job it did has a better replacement: `npx skills add . -g -y` symlinks the working tree into every detected harness's skill dir (better than sync.sh's copy model — edits propagate live), `hermes skills install mvanhorn/last30days-skill --force` handles Hermes, `clawhub install last30days-official` handles OpenClaw, and the Claude marketplace cache target was a "test against the official install path" hack we shouldn't have been recommending in the first place. The `test_sync_cache_path_uses_skill_version` test was dropped along with it. CLAUDE.md, HERMES_SETUP.md, the PR template, and a render.py docstring were updated to drop references; CHANGELOG and historical docs (release notes, plan files) keep their existing mentions as accurate history.
## [3.2.0] - 2026-05-09
### Added
- Add `--emit=html` for shareable, print-friendly HTML research briefs.
- **Digg AI 1000 source** (auto-enabled when `digg-pp-cli` is on PATH). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief as `[@handle](xUrl) via Digg AI 1000: ...` lines. Footer line: `⛏️ Digg AI 1000: N clusters │ K posts │ M authors`. No X auth required for the inline quotes since they flow through Digg's read-only endpoints.
## [3.1.1] - 2026-04-24
### Fixed
- **Codex plugin layout.** Move the canonical runtime payload under `skills/last30days/` and update Codex/Claude plugin metadata and tests for the relocated engine path.
- **Claude Code cache resolution.** Resolve Claude plugin installs to `skills/last30days/scripts/last30days.py` after the plugin-layout restructure.
## [3.1.0] - 2026-04-22
Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`, per-entity Step 0.55, vs-mode N passes, comparison title attribution) and republishes the OpenClaw bundle, which had been frozen on ClawHub at `3.0.0-open` since April 8.
### Added
- **OpenClaw republish.** `clawhub install last30days-official` now resolves to `3.1.0-open`, matching current main. Closes [#307](https://github.com/mvanhorn/last30days-skill/issues/307), [#195](https://github.com/mvanhorn/last30days-skill/issues/195), [#236](https://github.com/mvanhorn/last30days-skill/issues/236). The ClawHub bundle had shipped a broken `env.py get_config()` and stale SKILL.md path references since April; both are fixed at source on main and the republish carries the fixes to installers.
### Fixed
- **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
- **Broken README link.** The README's "source of truth" link pointed at `skills/last30days/SKILL.md`, a path that does not exist. Fixed to point at root `SKILL.md`.
### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
Individual changelog entries for 3.0.10 through 3.0.14 below document the incremental work consolidated into this release.
## [3.0.14] - 2026-04-22
### Changed
- **Comparison-mode title attribution.** The synthesis title for vs-mode and `--competitors` outputs changes from `What the Community Says (Last 30 Days)` to `What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range. Three SKILL.md occurrences updated; pure documentation change.
## [3.0.13] - 2026-04-22
### Changed
- **vs mode runs N full passes in parallel, one per entity.** Architectural revert of the 3-pass → 1-pass latency optimization from an earlier version. `/last30days "OpenAI vs Anthropic vs xAI"` now runs three full `pipeline.run()` calls in parallel via the same fanout `--competitors` uses, producing three `*-raw.md` save files plus a merged comparison output. Each entity gets its own Step 0.55-grade targeting, own primary X handle weight, own subreddit scoping — apples-to-apples depth instead of the one-pool merged retrieval the single-pass path produced. Parallel execution keeps wall clock ≈ single pass.
- **`--competitors` is now a SKILL.md-level shortcut for vs-mode with auto-discovery.** The hosting reasoning model (Claude Code, Codex, Hermes, Gemini, any agent with WebSearch) performs discovery and Step 0.55 per entity via its own WebSearch tool, then invokes the engine with a vs-topic and `--competitors-plan` JSON. The engine flag remains for headless/cron use with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys (engine-internal `auto_resolve` stays as fallback).
- **LAW 7-style stderr for `--competitors` with no backend** now leads with the hosting-model path (WebSearch + Step 0.55 + `--competitors-plan`) instead of `BRAVE_API_KEY`. API-key framing moved to a secondary "headless" section.
### Added
- **`--competitors-plan` JSON flag** for per-entity Step 0.55 targeting. Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`. Accepts inline JSON or a file path (matches `--plan`). When present for an entity, skips engine-internal `auto_resolve` and uses the provided values; missing fields fall back to `auto_resolve` (if backend) or planner defaults. Case-insensitive entity matching. The `subrun_kwargs_for` helper is the single source of truth for per-entity kwargs — no closure-default fallthrough from main scope.
- **Per-entity save files** when `--save-dir` is set on a vs-mode or `--competitors` run. Each entity's sub-run produces its own `{slug}-raw.md` with a single-row Resolved Entities block — matches historical vs-mode behavior (N passes → N save files).
- **`--polymarket-keywords "kw1,kw2"`** to filter Polymarket matches for ambiguous single-token topics (e.g., "Warriors" → `nba,gsw,golden-state` kills Glasgow Warriors rugby and Honor of Kings Rogue Warriors noise).
### Fixed
- **BRAVE/SERPER footer nudge suppressed** when `--plan` or `--competitors-plan` is present. The nudge told Claude Code users to set an API key when they already have WebSearch via the hosting model. Nudge still fires for true headless runs (no `--plan`, no backend) where the advice is correct.
- **Override-leak regression testing.** 3.0.12 already fixed the main-topic `--subreddits` / `--x-handle` / `--github-*` from leaking into peer sub-runs via explicit per-entity kwargs scrubbing. This release adds a 4-test regression suite (`test_competitor_subrun_isolation.py`) locking in the invariant.
## [3.0.12] - 2026-04-22
### Fixed
- **Per-entity Step 0.55 resolution for competitor sub-runs.** In 3.0.11, only the main topic got X handle / subreddit / GitHub resolution; competitor sub-runs ran with planner defaults and produced visibly thinner evidence (Reddit 403 fallbacks, single-word queries). Each competitor sub-run now calls `resolve.auto_resolve()` inside `fanout.run_competitor_fanout` when a web backend is available, mirroring the main topic's pre-flight resolution. Per-entity X handle, subreddit list, GitHub user/repos, and news context are threaded into each sub-run's `pipeline.run()` call. Deep-copied config per sub-run prevents `_auto_resolve_context` cross-leak. Surfaces in a new `## Resolved Entities` output block so the resolution coverage is visible without reading stderr.
- **LAW 7 false-positive on internal fan-out sub-runs.** Each competitor sub-run was emitting the `[Planner] No --plan passed... YOU ARE the planner` stderr warning. LAW 7 targets the hosting-reasoning-model path, not engine-internal fan-out. New `internal_subrun=True` keyword on `planner.plan_query` and `pipeline.run` suppresses the warning for sub-runs only; the default path is unchanged.
- **Marketplace-stale SKILL.md trap.** Added a STEP 0 canonical-path self-check at the top of SKILL.md. Two of three 2026-04-22 test runs loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (Claude-Code-managed git clone pinned to origin/main, lagging the versioned cache), then ran `--help` against the same stale path, did not see `--competitors`, and fell back to a manual comparison plan. The STEP 0 block forces any reader to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` and re-read from the versioned cache if not.
### Changed
- **Default `--competitors` count is now 2 (3-way total: original + 2 peers).** Previously 3. `--competitors=N` still customizes (range 1..6). Matches the feature description's canonical example (`Kanye vs Drake vs Kendrick`).
### Added
- **`## Resolved Entities` block** in `render_comparison_multi` output. Shows per-entity X handle, subreddits, GitHub user/repos, and truncated context for every entity in the comparison. Block is omitted entirely when no entity has a resolved payload (mock mode, no backend).
## [3.0.11] - 2026-04-22
### Added
- **`--competitors` flag for auto-discovered comparison fan-out.** Pass `--competitors` on a single-entity topic and the engine discovers 2-6 peer entities via web search, then runs the full pipeline on each in parallel and emits one N-way comparison. `last30days Kanye West --competitors` resolves Drake, Kendrick Lamar, and one more peer. `last30days OpenAI --competitors` resolves Anthropic, xAI, Google Gemini. `--competitors=N` controls count, `--competitors-list="A,B,C"` skips discovery and uses the explicit list. Discovery mirrors the `auto_resolve` pattern (Brave / Exa / Serper / Parallel) with deterministic text extraction - no internal LLM call. Sub-runs inherit the main `--quick`/`--deep`/`--days`, run in a `ThreadPoolExecutor`, and degrade gracefully when at least 2 entities survive. Output reuses the existing 9-axis `## Head-to-Head` scaffold.
## [3.0.10] - 2026-04-21
### Added
- **Commenter handles on evidence lines.** Top-comment rendering now includes the commenter's handle - `u/author` for Reddit, `@handle` for TikTok/YouTube/Instagram/Bluesky/X/Threads. The enrichment adapters already captured `author`; the render layer just was not using it. Evidence lines change from `- Comment (6822 upvotes): Finally, John Apple` to `- u/Cyrisaurus (6822 upvotes): Finally, John Apple`. Person-level citations make synthesis-side inline markdown links per LAW 8 much more natural. Both the compact and full render paths are covered.
### Fixed
- **TikTok author preference.** `_fetch_post_comments` in `scripts/lib/tiktok.py` preferred `user.nickname` over `user.unique_id`, so the engine captured display names ("Moosa Noormahomed") instead of @handles ("moosanoormahomed"). Flipped to prefer `unique_id`. Nickname still wins as a fallback when `unique_id` is missing. Display names can contain emoji, spaces, and non-Latin characters that do not round-trip to a profile URL; the @handle is the stable identifier.
- **Single plugin payload layout.** The canonical runtime moved to `skills/last30days/` for both Claude Code and Codex plugin loading. Root-level `SKILL.md`, `scripts/`, `agents/`, and `assets/` are no longer maintained as duplicate copies.
### Behavior fallback
- When an author is empty, `[deleted]`, or `[removed]`, the render falls back to the legacy `Comment (...)` shape - no `u/` or `@` prefix with an empty handle is ever emitted.
## [3.0.9] - 2026-04-18 - The Self-Debug Release ## [3.0.9] - 2026-04-18 - The Self-Debug Release
### Highlights ### Highlights
@@ -202,15 +315,15 @@ Intelligent search, fun judge, cross-source cluster merging, single-pass compari
### Highlights ### Highlights
Auto-save research briefings to `~/Documents/Last30Days/` as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste. Auto-save research briefings to the default memory directory as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
### Added ### Added
- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to `~/Documents/Last30Days/{topic-slug}.md` after every run - Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to the default memory directory after every run
- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`) - Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`) - Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
- Agent mode (`--agent`) also saves research files - Agent mode (`--agent`) also saves research files
- Brief confirmation after save: "Saved to ~/Documents/Last30Days/{slug}.md" - Brief confirmation after save with the saved file path
### Credits ### Credits
+1 -25
View File
@@ -1,25 +1 @@
# last30days Skill @AGENTS.md
Claude Code skill for researching any topic across Reddit, X, YouTube, and web.
Python scripts with multi-source search aggregation.
## Structure
- `scripts/last30days.py` — main research engine
- `scripts/lib/` — search, enrichment, rendering modules
- `scripts/lib/vendor/bird-search/` — vendored X search client
- `SKILL.md` — skill definition (deployed to ~/.claude/skills/last30days/)
## Commands
```bash
python3 scripts/last30days.py "test query" --emit=compact # Run research
bash scripts/sync.sh # Deploy to ~/.claude, ~/.agents, ~/.codex
```
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- After edits: run `bash scripts/sync.sh` to deploy
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+23
View File
@@ -0,0 +1,23 @@
# Concepts
Shared vocabulary for `last30days-skill`. Terms here have a precise project-specific meaning — distinct enough from their general technical sense that a new contributor would need them defined to follow conversations, PR descriptions, or the SKILL.md contract.
## The package
### Skill
A self-contained agent-instructions package consisting of a `SKILL.md` prose contract plus a sibling `scripts/` directory containing the executable code the SKILL.md invokes. The package conforms to the [Agent Skills](https://agentskills.io) open format and installs across every major harness (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ others) via `npx skills add`, harness-native plugin installers, or per-harness skill directories. A Skill is the unit of distribution; the Skill is the product.
### Engine
The Python script (`scripts/last30days.py`) the Skill's SKILL.md invokes to do the actual research work. The Engine and SKILL.md have a contract: SKILL.md tells the model which flags to pass (`--plan`, `--competitors-plan`, `--x-handle`, `--subreddits`, `--emit=compact`, etc.), and the Engine produces a specific output shape (badge line, ranked evidence clusters, emoji-tree footer) that the model is contractually required to pass through. The Engine is implementation; the SKILL.md prose is the agent-facing surface.
### Harness
The agent runtime that loads Skills and invokes them on the user's behalf. Claude Code is the most common Harness for this Skill but not the only one — Codex, Cursor, GitHub Copilot, Gemini CLI, and the rest of the Agent Skills ecosystem also count. "Multi-harness" describes a Skill that works correctly across every Harness it installs into; features written without multi-harness awareness (e.g., engine flags with no SKILL.md integration, or paths hardcoded to one Harness's install layout) regress on Harnesses other than the one they were tested against.
## Distribution
### Beta channel
A parallel install of the Skill, sourced from the private `mvanhorn/last30days-skill-private` repo and installed as `/last30days-beta` rather than `/last30days`. The Beta channel exists so experimental changes can be tested by real users before they ship to the public `/last30days`. Promotion from Beta to public happens via a review PR against this (public) repo — Beta-only changes never ship to public without that PR. The Beta channel workflow guide lives in `BETA.md` in the private repo.
+1 -1
View File
@@ -23,7 +23,7 @@ v3 has full GitHub search: issues, PRs, person-mode profiles, project-mode repos
### @thinkun ### @thinkun
[PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results [PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results
v3 has parallel enrichment with per-item timeouts. No results are ever dropped. v3 has parallel enrichment with per-item timeouts. No results are ever dropped.
> _Add your bio, website, or anything you'd like here._ > Thinker, technologist, AI expert, music-tinkerer. Founder of [Thinkun](https://thinkun.com). [@thinkun on GitHub](https://github.com/thinkun) · [@unthink on X](https://x.com/unthink)
### @thomasmktong ### @thomasmktong
[PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback [PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback
+12 -22
View File
@@ -10,28 +10,20 @@ This guide covers installing last30days on Hermes AI Agent.
## Installation ## Installation
### Option 1: Via sync.sh (Recommended)
```bash ```bash
# Clone the repo hermes skills install mvanhorn/last30days-skill --force
git clone https://github.com/mvanhorn/last30days-skill.git
cd last30days-skill
# Run the sync script
bash scripts/sync.sh
``` ```
This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days/` This pulls the latest release from GitHub and deploys to `~/.hermes/skills/research/last30days/`. `--force` reinstalls over any existing copy.
### Option 2: Manual Copy ### Developer / live-edit alternative
If you're hacking on the skill locally and want edits to propagate to Hermes without re-installing, symlink your working tree:
```bash ```bash
# Create directory git clone https://github.com/mvanhorn/last30days-skill.git
mkdir -p ~/.hermes/skills/research/last30days mkdir -p ~/.hermes/skills/research
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.hermes/skills/research/last30days
# Copy files
cp -r scripts ~/.hermes/skills/research/last30days/
cp .hermes-plugin/SKILL.md ~/.hermes/skills/research/last30days/
``` ```
## Usage ## Usage
@@ -59,7 +51,7 @@ On first run, the skill will guide you through setup:
2. **Optional: ScrapeCreators** 2. **Optional: ScrapeCreators**
- Adds TikTok, Instagram, Reddit backup - Adds TikTok, Instagram, Reddit backup
- 10,000 free API calls - 100 free credits (no expiration)
- Sign up at scrapecreators.com - Sign up at scrapecreators.com
3. **Optional: API Keys** 3. **Optional: API Keys**
@@ -106,14 +98,12 @@ python3.12 scripts/last30days.py --diagnose
## Updating ## Updating
To update to the latest version:
```bash ```bash
cd last30days-skill hermes skills install mvanhorn/last30days-skill --force
git pull
bash scripts/sync.sh
``` ```
If you symlinked your working tree (developer alternative above), just `git pull` in the repo — edits propagate live, no re-install step.
## Support ## Support
- Original repo: https://github.com/mvanhorn/last30days-skill - Original repo: https://github.com/mvanhorn/last30days-skill
+115 -38
View File
@@ -12,23 +12,20 @@
**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.** **An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior. This README tracks the current v3 pipeline. The runtime skill spec lives in [SKILL.md](SKILL.md), which is the source of truth for the latest command and setup behavior.
Claude Code: **Claude Code (recommended — auto-updates via marketplace):**
``` ```
/plugin marketplace add mvanhorn/last30days-skill /plugin marketplace add mvanhorn/last30days-skill
``` ```
OpenClaw: **Codex, Cursor, Copilot, Gemini CLI, or any of 50+ [Agent Skills](https://agentskills.io) hosts:**
``` ```
clawhub install last30days-official npx skills add mvanhorn/last30days-skill -g
``` ```
(`-g` installs globally for your user, available across all projects. Drop it to scope per-project.)
Hermes: More install options (claude.ai web, OpenClaw, manual) in the [Install](#install) section below.
```
# The skill auto-deploys when you run sync.sh
# Or manually copy to ~/.hermes/skills/research/last30days/
```
Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds. Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
@@ -68,6 +65,7 @@ If you're meeting with a CEO, have you read all their tweets and YouTube transcr
| **Hacker News** | The developer consensus. 825 points, 899 comments. Where technical people actually argue. | | **Hacker News** | The developer consensus. 825 points, 899 comments. Where technical people actually argue. |
| **Polymarket** | Not opinions. Odds. Backed by real money. 96% confidence on album sales. 4% on an acquisition. | | **Polymarket** | Not opinions. Odds. Backed by real money. 96% confidence on album sales. 4% on an acquisition. |
| **GitHub** | For people: PR velocity, top repos by stars, release notes. For topics: issues and discussions. | | **GitHub** | For people: PR velocity, top repos by stars, release notes. For topics: issues and discussions. |
| **Digg** | Curated story clusters from Digg's AI 1000 leaderboard (~1000 high-signal AI accounts on X), with attributable inline quotes (no X auth required). Auto-enabled when `digg-pp-cli` is on PATH. |
| **Threads** | The post-Twitter text layer. Conversations from creators and brands. | | **Threads** | The post-Twitter text layer. Conversations from creators and brands. |
| **Pinterest** | Visual discovery. Pins, saves, and comments on products and ideas. | | **Pinterest** | Visual discovery. Pins, saves, and comments on products and ideas. |
| **Bluesky** | The decentralized social layer. AT Protocol posts from the post-Twitter migration. | | **Bluesky** | The decentralized social layer. AT Protocol posts from the post-Twitter migration. |
@@ -96,6 +94,28 @@ The synthesis ranks by what real people actually engaged with. Social relevancy,
## What v3 Changed ## What v3 Changed
### Shareable HTML briefs
Ask for an HTML brief and the skill saves a self-contained, dark-mode, print-friendly file you can drop into Slack, email, or Notion. No raw markdown leaks. Inline CSS, system-font fallbacks behind Inter and JetBrains Mono. No JavaScript. Works offline.
```
/last30days OpenClaw --emit=html
```
or just ask in plain language:
```
/last30days OpenClaw, give me a shareable HTML brief
/last30days Cursor IDE for slack
/last30days Anthropic earnings export as html
```
The skill emits the synthesis in chat as usual AND saves a brief to `${LAST30DAYS_MEMORY_DIR}/{topic}-brief.html` (defaults to `~/Documents/Last30Days/`). The chat response ends with the file path so you can `open` it or drag it into a message.
What's in the file: badge, inline metadata line, the model's synthesis verbatim with all citations, the engine footer (✅ All agents reported back! tree), and a colophon noting the topic + how to re-run. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs; they never leak into the shareable artifact.
For direct CLI use without the model in the loop, the engine also accepts `--synthesis-file PATH` to convert any markdown synthesis to HTML.
### Intelligent search: the killer feature ### Intelligent search: the killer feature
The v3 engine doesn't just search for your topic. It figures out *where* to search before the search begins. Type "OpenClaw" and the engine resolves @steipete (Peter Steinberger, the creator), r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags - all via a new Python pre-research brain built by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities. The v3 engine doesn't just search for your topic. It figures out *where* to search before the search begins. Type "OpenClaw" and the engine resolves @steipete (Peter Steinberger, the creator), r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags - all via a new Python pre-research brain built by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
@@ -114,6 +134,10 @@ When the same story appears on Reddit, X, and YouTube, v3 merges them into one c
"CLI vs MCP" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides simultaneously. Same depth, 3 minutes. "CLI vs MCP" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides simultaneously. Same depth, 3 minutes.
### Auto-discovered competitor comparisons
`/last30days OpenAI --competitors` tells the hosting reasoning model to discover the top 2 peers via WebSearch (Anthropic, xAI), run Step 0.55 per entity, and invoke the engine with `"OpenAI vs Anthropic vs xAI"` and a per-entity `--competitors-plan` JSON. The engine fans out 3 full pipelines in parallel, saves a `*-raw.md` file per entity, and merges them into a 3-way comparison. Same mechanics power `/last30days "OpenAI vs Anthropic vs xAI"` directly.
### GitHub person-mode ### GitHub person-mode
When the topic is a person, the engine switches from keyword search to author-scoped queries. Instead of "who mentioned this name in an issue body," it answers: what are they shipping and where is it landing? When the topic is a person, the engine switches from keyword search to author-scoped queries. Instead of "who mentioned this name in an issue body," it answers: what are they shipping and where is it landing?
@@ -128,8 +152,10 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
- **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works. - **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
- **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions. - **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
- **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does. - **TikTok, Instagram, Threads.** All three activate automatically once `SCRAPECREATORS_API_KEY` is set — same key, same per-call cost. Suppress any of them with `EXCLUDE_SOURCES=tiktok,instagram,threads` (any comma-separated subset).
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock. - **Pinterest.** Per-query opt-in (visual pins, narrow utility): the model passes `--search=pinterest` for the runs that need it. Requires `SCRAPECREATORS_API_KEY`.
- **YouTube + TikTok comments.** Persistent opt-in via `INCLUDE_SOURCES=youtube_comments,tiktok_comments` because each video pulls N extra ScrapeCreators calls on top of the base search. Surface top comments with vote counts the same way Reddit does.
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` and `INCLUDE_SOURCES=perplexity` (it's a separate paid API — opt-in keeps you from being surprise-billed).
- **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?" - **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
- **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run. - **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
- **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section. - **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section.
@@ -141,12 +167,61 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
## Install ## Install
| Surface | Install | | Surface | Install | Updates |
|---------|---------| |---------|---------|---------|
| **claude.ai** (web) | [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) and upload via Settings > Capabilities > Skills > + | | **Claude Code** (recommended) | `/plugin marketplace add mvanhorn/last30days-skill` | Auto via marketplace, or `claude plugin update last30days@last30days-skill` |
| **Claude Code** | `/plugin marketplace add mvanhorn/last30days-skill` | | **Codex, Cursor, Copilot, Gemini CLI, GitHub Copilot, or any of 50+ [Agent Skills](https://agentskills.io) hosts** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
| **OpenClaw** | `clawhub install last30days-official` | | **claude.ai** (web) | [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) and upload via Settings > Capabilities > Skills > + | Re-download and re-upload |
| **Gemini CLI** | Clone then `gemini extensions install ./last30days-skill` (see below) | | **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
### Claude Code (recommended)
```
/plugin marketplace add mvanhorn/last30days-skill
```
Recommended because the Claude Code marketplace handles updates for you — the plugin cache is versioned and auto-refreshes when a new release publishes. Run `claude plugin update last30days@last30days-skill` to force a check.
If you'd rather use the agent-skills install path on Claude Code, that's also supported:
```
npx skills add mvanhorn/last30days-skill -g -a claude-code
```
The native plugin and the `npx skills` install can coexist; Claude Code dedupes the slash command.
### Codex, Cursor, Copilot, Gemini CLI, and other Agent Skills hosts
Install via the open [Agent Skills](https://agentskills.io) CLI — supports 50+ harnesses including `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose`, and more (full list on the [vercel-labs/skills repo](https://github.com/vercel-labs/skills)).
```bash
npx skills add mvanhorn/last30days-skill -g
```
The `-g` (global) flag installs to your user directory so the skill is available across all projects. Without `-g`, `npx skills` installs project-locally into `./.skills/` (committed with the repo). For a research-the-world tool, global is what you want.
By default this installs for whichever harness `npx skills` detects. To target a specific one (or multiple):
```bash
npx skills add mvanhorn/last30days-skill -g -a codex
npx skills add mvanhorn/last30days-skill -g -a cursor
npx skills add mvanhorn/last30days-skill -g -a gemini-cli
npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
```
Update later with:
```bash
npx skills update last30days -g
```
Or update everything you've installed globally via `npx skills`:
```bash
npx skills update -g
```
List and remove with `npx skills list -g` and `npx skills remove last30days -g`.
### claude.ai (web) ### claude.ai (web)
@@ -154,15 +229,7 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
2. Go to [claude.ai Settings > Capabilities > Skills](https://claude.ai/settings/capabilities) 2. Go to [claude.ai Settings > Capabilities > Skills](https://claude.ai/settings/capabilities)
3. Click the `+` button in the Skills panel and drop the file in 3. Click the `+` button in the Skills panel and drop the file in
Enable "Code execution and file creation" under Capabilities first - skills won't run without it. Enable "Code execution and file creation" under Capabilities first skills won't run without it.
### Claude Code
```
/plugin marketplace add mvanhorn/last30days-skill
```
Update later with `claude plugin update last30days@last30days-skill`.
### OpenClaw ### OpenClaw
@@ -170,22 +237,14 @@ Update later with `claude plugin update last30days@last30days-skill`.
clawhub install last30days-official clawhub install last30days-official
``` ```
### Gemini CLI
Gemini CLI v0.9.0 has an upstream installer bug that can fail with `Configuration file not found at /tmp/gemini-extensionXXXXXX/gemini-extension.json` ([upstream issue](https://github.com/google-gemini/gemini-cli/issues/11452)). Workaround:
```bash
git clone https://github.com/mvanhorn/last30days-skill
gemini extensions install ./last30days-skill
```
### Manual (developer) ### Manual (developer)
```bash ```bash
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days git clone https://github.com/mvanhorn/last30days-skill.git
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
``` ```
Or build the claude.ai `.skill` file from source: `bash scripts/build-skill.sh` produces `dist/last30days.skill`. The symlink keeps the install in sync with your working tree as you edit — no re-copy needed. For `claude.ai`, build the `.skill` file from source: `bash skills/last30days/scripts/build-skill.sh` produces `dist/last30days.skill`.
Reddit (with comments), Hacker News, Polymarket, and GitHub work immediately. Zero configuration. Run `/last30days` once and the setup wizard unlocks more sources in 30 seconds. Reddit (with comments), Hacker News, Polymarket, and GitHub work immediately. Zero configuration. Run `/last30days` once and the setup wizard unlocks more sources in 30 seconds.
@@ -199,10 +258,28 @@ These platforms don't have relationships with each other. X doesn't know what Re
| X / Twitter | Log into x.com in any browser | Free | | X / Twitter | Log into x.com in any browser | Free |
| YouTube | `brew install yt-dlp` | Free | | YouTube | `brew install yt-dlp` | Free |
| Bluesky | App password from bsky.app | Free | | Bluesky | App password from bsky.app | Free |
| TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 10,000 free calls | | TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 100 free credits, then PAYG |
| Perplexity Sonar | OpenRouter key | Pay as you go | | Perplexity Sonar | OpenRouter key | Pay as you go |
| Web search | Brave Search key | 2,000 free queries/month | | Web search | Brave Search key | 2,000 free queries/month |
### macOS Keychain (optional)
On macOS you can store keys in the system Keychain instead of a `.env` file. The skill picks them up automatically as the lowest-priority source — `.env` files and process environment still win on collision.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Items are stored under service name `last30days-<KEY>` for the current user. On non-Darwin platforms the loader is a no-op, so there is no behaviour change for Linux/Windows users.
## How it works ## How it works
1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything. 1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything.
-77
View File
@@ -1,77 +0,0 @@
# last30days Skill Specification
## Overview
`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet. OpenAI auth can come from `OPENAI_API_KEY` or Codex login credentials.
The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
## Architecture
The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
- **env.py**: Load API keys from `~/.config/last30days/.env` and Codex auth from `~/.codex/auth.json`
- **dates.py**: Date range calculation and confidence scoring
- **cache.py**: 24-hour TTL caching keyed by topic + date range
- **http.py**: stdlib-only HTTP client with retry logic
- **models.py**: Auto-selection of OpenAI/xAI models with 7-day caching
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
- **xai_x.py**: xAI Responses API + x_search for X
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
- **hackernews.py**: Hacker News search via Algolia API (free, no auth)
- **polymarket.py**: Polymarket prediction market search via Gamma API (free, no auth)
- **normalize.py**: Convert raw API responses to canonical schema
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
- **dedupe.py**: Near-duplicate detection via text similarity
- **render.py**: Generate markdown and JSON outputs
- **schema.py**: Type definitions and validation
## Embedding in Other Skills
Other skills can import the research context in several ways:
### Inline Context Injection
```markdown
## Recent Research Context
!python3 ~/.claude/skills/last30days/scripts/last30days.py "your topic" --emit=context
```
### Read from File
```markdown
## Research Context
!cat ~/.local/share/last30days/out/last30days.context.md
```
### Get Path for Dynamic Loading
```bash
CONTEXT_PATH=$(python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=path)
cat "$CONTEXT_PATH"
```
### JSON for Programmatic Use
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=json > research.json
```
## CLI Reference
```
python3 ~/.claude/skills/last30days/scripts/last30days.py <topic> [options]
Options:
--refresh Bypass cache and fetch fresh data
--mock Use fixtures instead of real API calls
--emit=MODE Output mode: compact|json|md|context|path (default: compact)
--sources=MODE Source selection: auto|reddit|x|both (default: auto)
```
## Output Files
All outputs are written to `~/.local/share/last30days/out/`:
- `report.md` - Human-readable full report
- `report.json` - Normalized data with scores
- `last30days.context.md` - Compact reusable snippet for other skills
- `raw_openai.json` - Raw OpenAI API response
- `raw_xai.json` - Raw xAI API response
- `raw_reddit_threads_enriched.json` - Enriched Reddit thread data
-47
View File
@@ -1,47 +0,0 @@
# last30days Implementation Tasks
## Setup & Configuration
- [x] Create directory structure
- [x] Write SPEC.md
- [x] Write TASKS.md
- [x] Write SKILL.md with proper frontmatter
## Core Library Modules
- [x] scripts/lib/env.py - Environment and API key loading
- [x] scripts/lib/dates.py - Date range and confidence utilities
- [x] scripts/lib/cache.py - TTL-based caching
- [x] scripts/lib/http.py - HTTP client with retry
- [x] scripts/lib/models.py - Auto model selection
- [x] scripts/lib/schema.py - Data structures
- [x] scripts/lib/openai_reddit.py - OpenAI Responses API
- [x] scripts/lib/xai_x.py - xAI Responses API
- [x] scripts/lib/reddit_enrich.py - Reddit thread JSON fetcher
- [x] scripts/lib/normalize.py - Schema normalization
- [x] scripts/lib/score.py - Popularity scoring
- [x] scripts/lib/dedupe.py - Near-duplicate detection
- [x] scripts/lib/render.py - Output rendering
## Main Script
- [x] scripts/last30days.py - CLI orchestrator
## Fixtures
- [x] fixtures/openai_sample.json
- [x] fixtures/xai_sample.json
- [x] fixtures/reddit_thread_sample.json
- [x] fixtures/models_openai_sample.json
- [x] fixtures/models_xai_sample.json
## Tests
- [x] tests/test_dates.py
- [x] tests/test_cache.py
- [x] tests/test_models.py
- [x] tests/test_score.py
- [x] tests/test_dedupe.py
- [x] tests/test_normalize.py
- [x] tests/test_render.py
## Validation
- [x] Run tests in mock mode
- [x] Demo --emit=compact
- [x] Demo --emit=context
- [x] Verify file tree
@@ -0,0 +1,306 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: --competitors flag for auto-discovered comparison fan-out"
type: feat
status: active
date: 2026-04-22
---
# feat: --competitors flag for auto-discovered comparison fan-out
## Overview
Add a `--competitors` flag to the last30days engine that auto-discovers 2-4 peer entities for the topic, runs the full retrieval pipeline on each in parallel, and renders a multi-entity comparison. Invoking `last30days Kanye West --competitors` should resolve to "Kanye vs Drake vs Kendrick Lamar" and emit a comparison report covering all three. Invoking `last30days OpenAI --competitors` should resolve to "OpenAI vs Anthropic vs xAI vs Gemini" and emit a four-way comparison.
Discovery mirrors the existing `resolve.auto_resolve()` pattern used for X handles and subreddits at pipeline start — web search (Brave / Exa / Serper) plus deterministic extraction. Not an internal LLM call.
## Problem Frame
Users who want a comparison today must type "OpenAI vs Anthropic vs xAI" themselves. The `planner._comparison_entities()` path already handles explicit multi-entity topics and `render._render_comparison_scaffold()` already emits a 9-axis comparison table. What is missing is the discovery half — a user who types a single entity with `--competitors` should get the comparison for free.
This is also the natural next step after the Step 0.55 category-peer subreddit work (PR #305, merged 2026-04-22). That feature widens the subreddit set within a single topic; this feature widens the entity set into peer entities.
## Requirements Trace
- R1. New `--competitors` boolean flag that triggers competitor discovery and multi-entity fan-out.
- R2. New `--competitors-list="A,B,C"` to explicitly skip discovery (mirrors `--plan`, `--subreddits`, `--x-handle` overrides).
- R3. New `--competitors=N` short form to set competitor count inline (N in 1..6).
- R4. Default count is 3 competitors (original + 3 = 4-way comparison).
- R5. Competitor retrieval depth inherits the main run's depth (`--quick` / `--deep`); all entities run in parallel so wall clock stays close to a single run.
- R6. Discovery mirrors `resolve.auto_resolve()`: web search for peers, deterministic text extraction. No internal LLM dependency.
- R7. If no web search backend is configured and no `--competitors-list` was passed, engine emits a LAW 7-style stderr telling the host agent to pass `--competitors-list` and exits non-zero.
- R8. Output rendering is a single comparison report covering all entities, reusing the existing 9-axis scaffold from `render._render_comparison_scaffold()` where applicable.
## Scope Boundaries
- Synthesis prompt changes beyond wiring N reports into the existing comparison scaffold are out of scope.
- `--competitors` does not replace the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`; both paths coexist.
- No caching layer for discovery results in v1.
- No UI/SKILL.md rewrite of the entire comparison section; only the new flag is documented.
- No new web search backend.
### Deferred to Separate Tasks
- Caching of competitor lookups: separate follow-up once hit rate justifies it.
- Disambiguation UX for topics with multiple common entities ("Amazon" the company vs the river): separate brainstorm.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:168-249``build_parser()` argparse definitions. Existing depth flags (`--quick`, `--deep`) and override flags (`--plan`, `--subreddits`, `--x-handle`, `--auto-resolve`) set the convention to mirror.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the reference pattern: web search fan-out via `ThreadPoolExecutor`, per-query extraction functions, graceful empty-dict return when no backend is available.
- `scripts/lib/resolve.py:98-140``_extract_x_handle()` and sibling extractors show the deterministic text-mining style competitor extraction should mirror.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature is the fan-out target. One call per entity, each returning a `schema.Report`.
- `scripts/lib/planner.py:430-564` — Existing comparison-intent handling and `_comparison_entities()` entity extraction. The new flag feeds the same mental model but populates entities from discovery instead of from the topic string.
- `scripts/lib/render.py:333-392``_render_comparison_scaffold()` already emits a 9-axis markdown comparison table. The new multi-report renderer should reuse this helper by assembling a synthetic "A vs B vs C" topic header for it.
- `scripts/lib/grounding.py` + `scripts/lib/providers.py` — Web search backend resolution (Brave / Exa / Serper). Reused as-is.
### Institutional Learnings
- No existing `docs/solutions/` entries for competitor discovery or multi-entity fan-out.
- Recent plan `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md` established the precedent of deterministic peer expansion; this plan extends that idea from subreddits to entities.
### External References
- None gathered — local patterns are strong. `resolve.auto_resolve()` is a direct template.
## Key Technical Decisions
- **Discovery mirrors auto_resolve, not plan_query.** Web search + regex extraction, not an LLM call. Matches the user's explicit direction ("use the python brain the same way it searches for X handles"). Cheaper, no provider credential requirement, deterministic.
- **Orchestration lives in `last30days.py` main, not inside `pipeline.run()`.** The fan-out is a top-level concern — one pipeline run per entity, each independent. Keeps `pipeline.run()` single-entity and unchanged except for sharing a `ThreadPoolExecutor` factory.
- **Sub-runs inherit main depth and run in parallel.** Wall clock ≈ single run; token cost scales linearly with N. User-controlled via the existing `--quick`/`--deep` flags.
- **New module `scripts/lib/competitors.py` instead of adding to `resolve.py`.** Keeps resolve focused on single-entity entity-bundle discovery (handles/subreddits/github); competitors.py owns peer-entity discovery. Similar shape, different responsibility.
- **Multi-report render is additive in `render.py`.** New `render_comparison_multi(reports: list[Report]) -> str` composes a synthetic "A vs B vs C" topic and delegates to the existing scaffold + synthesis path where possible. No rewrite of the single-entity render path.
- **Default count = 3 competitors (4-way comparison).** Hard cap at 6.
- **LAW 7-style stderr when no backend and no list.** Matches how `planner.plan_query()` already tells the hosting agent to pass `--plan`.
## Open Questions
### Resolved During Planning
- **Discovery mechanism:** Web search via `grounding.web_search()`, not an internal LLM. User confirmed the auto_resolve pattern is the target.
- **Default competitor count:** 3 (original + 3 = 4-way).
- **Sub-run depth:** Inherit main depth, parallel execution.
- **Flag naming:** `--competitors` (standard argparse double-dash). `--competitors=N` for inline count. `--competitors-list="A,B,C"` to skip discovery.
### Deferred to Implementation
- Exact extraction heuristics for competitor names across Brave / Exa / Serper result shapes. The SERP text varies (listicles, comparison pages, "vs" pages); the initial implementation will start with listicle parsing plus a "X vs Y" pattern match, and harden against real results in the test phase.
- Handling of topic ambiguity ("Amazon", "Apple"). Initial behavior: trust whatever web search returns for the topic verbatim; disambiguation is a separate concern.
- Merge strategy when two entities return overlapping URLs (e.g., an "OpenAI vs Anthropic" article shows up in both runs). Likely dedupe at the clustering step, but defer the exact policy until we see how often it happens.
- Whether to expose competitor discovery artifacts (the raw web search results) as a debug emit. Follow the existing `--debug` conventions.
## Implementation Units
- [ ] **Unit 1: CLI flag parsing and validation**
**Goal:** Add `--competitors`, `--competitors=N`, and `--competitors-list` to the argparse surface, validate values, and thread them into the main orchestration.
**Requirements:** R1, R2, R3, R4
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py`
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Add three mutually cooperative flags near line 205 in `build_parser()`:
- `--competitors` with `nargs="?"` and `const=3` so bare `--competitors` defaults to 3, `--competitors=4` is honored, and `--competitors=0` is rejected
- `--competitors-list` free-text CSV
- Normalize in `main()`: if `--competitors-list` is present, skip discovery and use the list. If `--competitors` is set and no list, trigger discovery with count = the flag value. Clamp count to 1..6 with a stderr warning at boundary.
- Thread the resulting entity list into the orchestrator added in Unit 3.
**Patterns to follow:**
- `--plan` argument at `scripts/last30days.py:187` — same skip-discovery-when-explicit shape.
- `--subreddits` / `--x-handle` at `scripts/last30days.py:180,189` — same override semantics.
**Test scenarios:**
- Happy path: bare `--competitors` parses to count=3, empty list.
- Happy path: `--competitors=4` parses to count=4.
- Happy path: `--competitors-list="A,B,C"` parses to count=3, list=["A","B","C"], and is preferred over any discovery signal.
- Edge case: `--competitors=0` and `--competitors=-1` are rejected with a clear error.
- Edge case: `--competitors=99` clamps to 6 with a stderr warning.
- Edge case: `--competitors` combined with `--competitors-list` uses the list and logs that discovery was skipped.
- Edge case: `--competitors-list` value with whitespace ("A, B , C") normalizes correctly.
**Verification:**
- Running the binary with each flag variation produces the expected post-parse state without calling out to the network.
- [ ] **Unit 2: `scripts/lib/competitors.py` discovery module**
**Goal:** Discover peer entities for a topic using web search + deterministic extraction, mirroring `resolve.auto_resolve()`.
**Requirements:** R6, R7
**Dependencies:** None (pure module; wired by Unit 3)
**Files:**
- Create: `scripts/lib/competitors.py`
- Test: `tests/test_competitors.py`
**Approach:**
- Public entry point `discover_competitors(topic: str, count: int, config: dict) -> list[str]`.
- Early return `[]` when `_has_backend(config)` is false (reuse the helper from `resolve.py`; factor if needed).
- Fan out 2-3 web searches in a `ThreadPoolExecutor`:
- `"{topic} competitors"`
- `"{topic} alternatives"`
- `"{topic} vs"` (captures "X vs Y" articles)
- Feed results into a deterministic `_extract_peer_entities(results, topic)` that:
- Mines titles and snippets for capitalized noun phrases other than the topic itself
- Scores by frequency across results
- Filters stopwords and the topic's own tokens
- Returns top `count` unique entities ordered by score
- Emit a single-line stderr log mirroring the `resolve._log` format.
**Patterns to follow:**
- `scripts/lib/resolve.py:179-258` for the function shape, executor usage, and empty-result fallback.
- `scripts/lib/resolve.py:98-140` for extractor style (small, deterministic, no external state).
**Test scenarios:**
- Happy path: canned SERP fixtures for "OpenAI" return ["Anthropic", "xAI", "Google"] or close peers in the top 3.
- Happy path: canned SERP fixtures for "Kanye West" return rap peers (Drake, Kendrick) in the top 3.
- Edge case: empty SERP results return `[]` without raising.
- Edge case: extractor filters out the topic itself (case- and punctuation-insensitive).
- Edge case: near-duplicate entities ("OpenAI" vs "Open AI") dedupe to one slot.
- Error path: web search backend raises — the failure is logged and the function returns `[]`.
- Edge case: count=1 returns a single-element list; count=6 returns up to six entities.
**Verification:**
- Unit tests pass with fixtures committed under `tests/fixtures/competitors-*.json`.
- Manual run against a live backend for one topic confirms sensible output (recorded as a notes file, not a test assertion).
- [ ] **Unit 3: Parallel fan-out orchestrator**
**Goal:** Run `pipeline.run()` once per entity (topic + discovered competitors) in parallel, collect `schema.Report` per entity, and hand them to the comparison renderer.
**Requirements:** R5, R7
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py`
- Possibly create: `scripts/lib/fanout.py` if the orchestrator grows past ~60 lines
- Test: `tests/test_competitor_fanout.py`
**Approach:**
- After arg parsing and before the existing `pipeline.run()` call, branch on `args.competitors`:
- If a list was provided or discovery returned entities, build `entities = [topic, *competitors]`.
- Spawn one `pipeline.run()` per entity via `ThreadPoolExecutor(max_workers=len(entities))`, passing the same `config`, `depth`, and all sub-run-relevant args (mock, plan, etc.). Respect `--plan` — if a plan is passed it applies to the main topic only; competitors use the internal planner fallback for v1.
- Collect `{entity: Report}` mapping. A per-entity failure logs a stderr warning and drops that entity from the comparison; the run continues as long as 2 entities succeed.
- If fewer than 2 entities survive, exit with a clear error.
- LAW 7-style stderr:
- If `args.competitors` is set, no list was passed, no web search backend is configured, emit a LAW 7 stderr message pointing to the `--competitors-list` override and exit non-zero. Reuse the tone from `planner.plan_query()` fallback (`scripts/lib/planner.py:125-135`).
**Execution note:** Start with a failing integration test that exercises the full main → orchestrator → mocked pipeline.run path; the orchestrator is where bugs hide.
**Patterns to follow:**
- `scripts/lib/resolve.py:225-239` for ThreadPoolExecutor + as_completed + per-future error handling.
- `scripts/lib/pipeline.py:310+` for how ThreadPoolExecutor is already used inside a single run (same idiom, outer layer).
**Test scenarios:**
- Happy path: main + 2 competitors, all three `pipeline.run()` calls succeed (mocked), orchestrator returns 3 Reports.
- Happy path: discovery returns the competitor list; orchestrator fans out accordingly.
- Edge case: one of three competitor pipelines raises — the run continues with the surviving 2 and emits a warning.
- Edge case: all competitors fail but the main topic succeeds — orchestrator exits non-zero with a clear error rather than silently degrading to a single-entity render.
- Edge case: `--competitors` set, no backend, no list — orchestrator emits the LAW 7 stderr and exits non-zero before any pipeline call.
- Integration: wall-clock time for 3 mocked pipelines in parallel is close to the slowest single run, not the sum (timing assertion with generous margin).
**Verification:**
- End-to-end test with mocked `pipeline.run()` and mocked competitors discovery produces 3 Reports and hands them to a stubbed renderer.
- [ ] **Unit 4: Multi-report comparison renderer**
**Goal:** Compose N `schema.Report`s into a single comparison-mode output, reusing the existing 9-axis scaffold.
**Requirements:** R8
**Dependencies:** Unit 3
**Files:**
- Modify: `scripts/lib/render.py`
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- Add `render_comparison_multi(reports: list[schema.Report], *, emit: str) -> str`.
- Build a synthetic comparison topic: `f"{entity_a} vs {entity_b} vs {entity_c}"`.
- Reuse `_render_comparison_scaffold()` for the table skeleton. Each entity column is populated from its own Report's top clusters and citations.
- For the narrative synthesis block, concatenate per-entity highlights, clearly labeled by entity, under a shared "Comparison" header.
- Preserve existing emit modes (`compact`, `md`, `json`, `context`). In `json` emit, return a `{"entities": [...], "reports": [...]}` shape; single-Report consumers remain unaffected because the single-report render path is untouched.
**Patterns to follow:**
- `scripts/lib/render.py:333-392` (`_parse_comparison_entities`, `_render_comparison_scaffold`) — the scaffold is the contract.
- `scripts/lib/render.py` single-report rendering — for per-entity narrative blocks.
**Test scenarios:**
- Happy path: 3 Reports with distinct clusters render into a 3-column table and a "Comparison" section that mentions each entity at least once.
- Happy path: 2 Reports render as a 2-column table without breaking the scaffold.
- Edge case: a Report with an empty cluster list renders as "(no significant discussion this month)" in its column rather than crashing.
- Edge case: Reports with overlapping URLs (same article cited by two entities) dedupe citations at the footer but keep both column entries.
- Emit variants: `--emit=compact`, `--emit=md`, `--emit=json`, `--emit=context` each produce valid output with all entities represented.
- Integration: end-to-end snapshot test using fixture Reports, checked against a stored expected output (with a clear update path when the scaffold intentionally evolves).
**Verification:**
- Snapshot tests pass. Manual review of one real 3-way comparison confirms readability.
- [ ] **Unit 5: Docs, SKILL.md mention, and sync**
**Goal:** Document the new flag so the hosting agent and human users both know it exists, and run the sync script.
**Requirements:** R1-R8 (surfaces them to users)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md`
- Modify: `README.md` (brief flag reference)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- Add a compact "Competitor mode" subsection under the existing comparison docs in `SKILL.md`. Document the flag, the default count, the override flag, and the LAW 7 fallback stderr.
- Keep `README.md` addition to a single example line.
- CHANGELOG entry mirrors the voice of recent entries (imperative, outcome-first).
- Sync via `scripts/sync.sh` per CLAUDE.md rules so `~/.claude/`, `~/.agents/`, `~/.codex/` pick up the new SKILL.md.
**Test scenarios:**
- Test expectation: none — documentation and sync only. Verification is by inspection and by running `sync.sh` and confirming target directories updated.
**Verification:**
- `sync.sh` completes without errors.
- `SKILL.md` rendered preview mentions `--competitors` in the comparison section.
## System-Wide Impact
- **Interaction graph:** `last30days.py main()` now orchestrates multiple `pipeline.run()` calls instead of one. No other callers of `pipeline.run()` are affected (it remains single-entity).
- **Error propagation:** Per-entity failures degrade gracefully as long as ≥2 entities survive; fewer survivors exits non-zero. Discovery failure with `--competitors` and no list is fatal.
- **State lifecycle risks:** Each sub-run uses its own `pipeline.run()` state; no shared mutable config. The `config` dict is read-only in `pipeline.run()` today — verify before committing to shared-reference passing, else deep-copy per sub-run.
- **API surface parity:** `--competitors` coexists with the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`. Both produce comparable output formats; the only difference is where the entity list came from.
- **Integration coverage:** The fan-out orchestrator crosses CLI → discovery → N pipelines → render; integration tests in Unit 3 and Unit 4 must exercise the full path end to end, not just unit-level.
- **Unchanged invariants:** `pipeline.run()` signature and single-entity semantics are unchanged. The single-entity render path in `render.py` is unchanged. No changes to `planner.plan_query()`. No changes to existing flags.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Competitor discovery returns garbage entities for niche topics. | `--competitors-list` override lets the user (or hosting agent) correct it. Unit tests with edge-case fixtures. Log discovery output to stderr under `--debug`. |
| Token cost scales linearly with N sub-runs. | Default count capped at 3, hard max 6, inherit `--quick` to let users throttle. Wall clock stays parallel. Emit a cost hint to stderr when N ≥ 4. |
| Merge conflicts against the single-entity render path during refactoring. | Keep the multi-report renderer strictly additive; do not modify the single-Report code path. |
| Config dict mutation inside sub-runs could leak state between entities. | Verify read-only usage before sharing references. If any sub-component mutates, deep-copy per sub-run before spawning threads. |
| A SERP extractor that works on Brave fixtures breaks on Exa/Serper result shapes. | Test fixtures for all three backends. Extractor operates on a normalized shape from `grounding.web_search()` (already the case), not raw provider output. |
| Hosting agent (Claude Code, Codex) unaware of the new flag when it could usefully pass `--competitors-list`. | SKILL.md updated in Unit 5 documents the flag in the same style as `--plan` and `--auto-resolve`. |
## Documentation / Operational Notes
- Beta channel first: per `CLAUDE.md`, experimental changes go to `mvanhorn/last30days-skill-private` on the `/last30days-beta` command. Land this on the private repo first, shake out on real topics for a day or two, then cherry-pick to public.
- After land-merge: run `scripts/sync.sh` to deploy SKILL.md + scripts to `~/.claude/`, `~/.agents/`, `~/.codex/`.
- Release notes entry in CHANGELOG.md follows the v3.0.9 voice — outcome-first, one paragraph.
## Sources & References
- Related code: `scripts/lib/resolve.py:179` (`auto_resolve`), `scripts/lib/pipeline.py:162` (`pipeline.run`), `scripts/lib/planner.py:80` (`plan_query` LAW 7 fallback), `scripts/lib/render.py:333` (comparison scaffold)
- Related PRs: #305 (Step 0.55 category-peer subreddit expansion — the precedent for deterministic peer expansion, merged 2026-04-22)
- Related plan: `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md`
@@ -0,0 +1,352 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: per-entity resolution, default-2, and stale-path guard for --competitors"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md
---
# fix: per-entity resolution, default-2, and stale-path guard for --competitors
## Overview
Three test runs of v3.0.11 `--competitors` surfaced four real bugs plus one product tweak. This plan fixes all of them in a single follow-up:
1. Competitor sub-runs get no Step 0.55 resolution (no X handle, no subreddits, no GitHub repo). Drake / Kendrick / Travis ran with deterministic-fallback single-word queries while Kanye had the full targeting package. User called it "lazy" and was right.
2. Two of three test windows (Linear, Coinbase) never invoked the new flag at all. They loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (a Claude-Code-managed git clone pinned to origin/main, which predates PR #308) instead of `plugins/cache/last30days-skill/last30days/3.0.11/`, so `--help` showed no `--competitors` flag and the model fell back to the manual comparison path.
3. Each competitor sub-run emits a scary `[Planner] No --plan passed... deterministic fallback` stderr line because LAW 7 targets the hosting-model path, not internal fan-out sub-runs.
4. Default competitor count is 3 (→ 4-way comparison). User wants default 2 (→ 3-way: original + 2 peers). Flag keeps `--competitors=N` to customize.
## Problem Frame
The 3 test runs (Kanye, Linear, Coinbase) showed a pattern:
| Window | Loaded SKILL.md from | Invoked --competitors? | Per-entity resolution? | Outcome |
|--------|----------------------|-----------------------|------------------------|---------|
| Kanye | cache/3.0.11/ (correct) | Yes | Only for main topic (Kanye) | Drake/Kendrick/Travis thin; Reddit 403 fallbacks |
| Linear | marketplaces/ (stale) | No — fell back to manual comparison | No | Thin run with noisy subreddits |
| Coinbase | marketplaces/ (stale) | No — fell back to manual comparison | Main only; keyword-search poisoned pool | Top subs: r/survivor, r/Airpodsmax (noise) |
Root causes:
- **Per-entity resolution gap:** `scripts/lib/fanout.py` calls `pipeline.run()` with topic + depth + web_backend + lookback_days only. It does not call `resolve.auto_resolve()` per entity, so sub-runs have no X handle, subreddit, or GitHub targeting. The original plan (`2026-04-22-002`) acknowledged this as a deliberate v1 simplification ("competitor sub-runs use planner defaults"). In practice this produces visibly asymmetric output and triggers downstream retrieval issues (403 fallbacks, keyword-search noise).
- **Stale-path loading:** Claude Code's skill loader alphabetizes `find` results with `marketplaces/` before `cache/`, and the model reads the first plausible SKILL.md it sees. SKILL.md line 823's `SKILL_ROOT` resolver is the correct path but only fires in engine-invocation blocks, not in the skill-load step.
- **LAW 7 in sub-runs:** LAW 7 exists because the *hosting reasoning model* is supposed to pass `--plan`. For competitor sub-runs, there is no hosting-model planning — it's an engine-internal fan-out. The warning is a false positive there.
## Requirements Trace
- R1. Default `--competitors` count is 2 peers (3-way comparison: original + 2).
- R2. Each competitor sub-run performs Step 0.55 resolution (X handle, subreddits, GitHub user/repos, news context) before its pipeline runs — not just the main topic.
- R3. Sub-runs do not emit the LAW 7 `No --plan passed` warning; they are internal fan-out, not hosting-model calls.
- R4. The rendered comparison output includes a visible "Resolved entities" block showing per-entity handles/subs/github for debug transparency (answers "did it resolve everyone?" without the user having to read stderr).
- R5. SKILL.md has a canonical-path self-check at the top: if the reader loaded it from anywhere other than `plugins/cache/last30days-skill/last30days/{VERSION}/`, re-read from the versioned path before proceeding.
- R6. Version bumps to 3.0.12; CHANGELOG entry; `scripts/sync.sh` deploys.
## Scope Boundaries
- No new discovery strategy. The web-search + regex extraction in `scripts/lib/competitors.py` stays as-is.
- No new CLI flags beyond the behavior changes above. Specifically: no per-entity override flags like `--competitor-handles`. The hosting-model escape hatch remains `--competitors-list`.
- No changes to the explicit `A vs B` comparison path (topic-string parsing in `planner._comparison_entities`).
- No marketplace-clone auto-restore fix — that's Claude Code harness behavior. This plan only guards against the symptom on the skill side.
### Deferred to Separate Tasks
- Caching of per-entity resolution results: separate follow-up once hit rate justifies it.
- Fan-out rate-limiting tuning (currently `max_workers=len(entities)+1`, capped at 6): defer until we see real-world quota exhaustion.
- Pre-flight cost hint when N ≥ 4 (noted in `2026-04-22-002` risks): defer.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:205-219``--competitors` / `--competitors-list` argparse definition (const=3 today; changing to 2).
- `scripts/last30days.py:220-290``resolve_competitors_args()` validator; update `COMPETITORS_DEFAULT`.
- `scripts/last30days.py:438-520` — main() fan-out orchestration; currently passes only topic/depth to each `_competitor_runner`.
- `scripts/lib/fanout.py:40-95``run_competitor_fanout()` signature. The `competitor_runner` callable is where per-entity resolution needs to happen.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the exact per-entity resolver to reuse. Already does X handle + subreddits + GitHub user/repos + news context in parallel via ThreadPoolExecutor.
- `scripts/lib/planner.py:80-135``plan_query()` emits the LAW 7 stderr. A `quiet: bool` keyword or `internal_subrun: bool` flag will suppress it.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature. Needs a new keyword to propagate quiet-mode down to the planner.
- `scripts/lib/render.py:render_comparison_multi` — where the "Resolved entities" block is inserted.
- `SKILL.md` line 823 — canonical `SKILL_ROOT` resolver already exists but fires in engine bash, not at skill-load time.
### Institutional Learnings
- `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md` acknowledged the per-entity-resolution gap as a v1 tradeoff. This plan closes that gap.
- Kanye run stderr: `[Planner] No --plan passed... deterministic fallback` × 3 (once per competitor sub-run). That's the LAW 7 noise R3 targets.
- Linear / Coinbase runs loaded `plugins/marketplaces/last30days-skill/CLAUDE.md` as the first hit. That's the stale-path issue R5 targets.
### External References
- None. All patterns are in-repo.
## Key Technical Decisions
- **Per-entity resolve happens inside fanout, not in SKILL.md.** The user-facing promise of `--competitors` is "one flag, engine does the work." Pushing resolution onto the hosting model creates another path-of-least-resistance trap (model skips it, output looks lazy). Auto-resolve inside each sub-run when a web backend is available makes the feature self-contained.
- **Stale-path guard is a SKILL.md self-check, not a code change.** We cannot stop Claude Code from auto-restoring the marketplace clone. But we can put a 3-line banner at the top of SKILL.md that forces any path-mismatched read to re-read from the versioned cache. Both the marketplace copy (once main catches up) and the cache copy carry the guard.
- **LAW 7 suppression is opt-in via `internal_subrun=True` keyword.** Do not remove the warning from the default path — it's load-bearing for the hosting-model contract. Add an explicit bypass for engine-internal fan-out only.
- **Default 2, hard max 6 unchanged.** "Original + 2" matches the Kanye/Drake/Kendrick mental model from the feature description. Still allow `--competitors=N` from 1 to 6.
- **Resolved block is inside the EVIDENCE envelope, not above it.** Keeps the rendered output structure stable for the synthesis contract (LAW 18). The block is context, not output.
- **Skip auto-resolve when `--mock` or no web backend.** Mirrors the existing `resolve.auto_resolve()` fast-fail and keeps the mock test path deterministic.
## Open Questions
### Resolved During Planning
- **Where does per-entity resolve live?** Inside `fanout.run_competitor_fanout`, not in `main()`. Each sub-run calls `auto_resolve()` just before `pipeline.run()`.
- **Should the hosting model still be able to override?** Yes — `--competitors-list` remains the escape hatch. When an explicit list is passed, the engine still does auto-resolve per entity; the user's list just skips discovery.
- **Should sub-runs run auto-resolve in parallel with each other?** Yes. The existing `ThreadPoolExecutor` in fanout already parallelizes sub-runs; auto-resolve happens inside each sub-run's thread, so resolve calls for different entities run concurrently.
- **Default count:** 2 peers (3-way). Confirmed.
### Deferred to Implementation
- Whether to expose a `--no-auto-resolve-competitors` flag for power users who want the fast, shallow behavior. Probably not needed v2; ship auto-resolve always-on and revisit if someone complains about cost.
- Whether to surface the per-entity resolution context back into the main topic's planner (cross-entity context sharing). Stays deferred.
- Whether the Resolved block should be collapsible or always inline. Start inline; revisit based on output length feedback.
## Implementation Units
- [ ] **Unit 1: Default `--competitors` to 2 peers**
**Goal:** Change the bare `--competitors` default from 3 to 2 per user feedback. `--competitors=N` still overrides; range 1..6 unchanged.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (`COMPETITORS_DEFAULT`, `--competitors` const, stderr messages if any reference 3)
- Modify: `SKILL.md` Competitor mode section ("discovered 2-6" wording, bare-flag default line)
- Modify: `README.md` auto-discovered example line (if it references count)
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Change `COMPETITORS_DEFAULT = 3``2` in `scripts/last30days.py`.
- Change argparse `--competitors` `const=3``const=2`.
- Update any SKILL.md / README copy referencing "3 peers" to "2 peers" (default) or "2-6 peers" (range).
**Patterns to follow:**
- Existing default constants in `scripts/last30days.py` argparse block.
**Test scenarios:**
- Happy path: bare `--competitors` yields count=2, enabled=True, empty explicit_list.
- Edge case: `--competitors=3` still works (explicit override).
- Edge case: existing `test_bare_flag_defaults_to_three` test is updated to `test_bare_flag_defaults_to_two` and asserts count=2.
- Edge case: `--competitors=5` with a `--competitors-list` of length 2 still logs the mismatch warning and uses the list.
**Verification:**
- `pytest tests/test_cli_competitors.py -v` passes with the updated default.
- [ ] **Unit 2: Per-entity Step 0.55 resolution inside fanout**
**Goal:** Each competitor sub-run auto-resolves its own X handle, subreddits, GitHub user/repos, and news context via `resolve.auto_resolve()` before its `pipeline.run()` call — just like the main topic.
**Requirements:** R2
**Dependencies:** None (but Unit 3 should land together so sub-runs don't emit LAW 7 stderr while the resolution context is being passed)
**Files:**
- Modify: `scripts/lib/fanout.py`
- Modify: `scripts/last30days.py` (`_competitor_runner` closure builds the resolved args)
- Test: `tests/test_competitor_fanout.py`
- Test: `tests/test_competitors_resolve_integration.py` (new; covers the auto-resolve path)
**Approach:**
- `_competitor_runner(entity)` in main() does:
1. Call `resolve.auto_resolve(entity, config)` when `not args.mock` and a web backend is configured (reuse `_has_backend`).
2. Extract resolved x_handle, subreddits, github_user, github_repos, context.
3. Pass them to `pipeline.run()` for that sub-run.
4. Inject resolved context into a per-entity config copy (so `_auto_resolve_context` does not leak across sub-runs — deep-copy the config or use a local dict).
5. Store the resolved block on the Report's `artifacts` so the renderer can surface it (Unit 4).
- When `args.mock` is True or no backend is available, skip auto-resolve (fall through to planner defaults, matching the existing `auto_resolve()` early-return contract).
- Update `fanout.run_competitor_fanout` docstring to note that auto-resolve happens inside the caller-provided runner.
**Execution note:** Start with a failing integration test that exercises two-entity fanout + auto-resolve via a mocked `resolve.auto_resolve` and asserts that `pipeline.run` receives the resolved x_handle/subreddits for each entity.
**Patterns to follow:**
- `scripts/last30days.py` main topic branch (`if args.auto_resolve and not external_plan`) already calls `resolve.auto_resolve` and propagates results — mirror the shape for competitors.
- Config isolation: `scripts/lib/pipeline.py:162-220` reads config as-is; use `dict(config)` to avoid cross-sub-run mutation of `_auto_resolve_context`.
**Test scenarios:**
- Happy path: 3 entities, mocked `auto_resolve` returns distinct handles per entity; `pipeline.run` receives `x_handle=@drake` for Drake, `x_handle=@kendricklamar` for Kendrick, etc.
- Happy path: the main topic still uses the user-supplied `--x-handle` / `--subreddits` overrides (not overwritten by auto-resolve for the main). Competitors use their own auto-resolved values.
- Edge case: `--mock` skips auto-resolve entirely for all sub-runs (no `resolve.auto_resolve` calls).
- Edge case: `resolve.auto_resolve` returns empty dicts for one entity (low-signal topic) — the sub-run still executes with planner defaults; doesn't crash.
- Edge case: no web backend configured — auto-resolve returns empty for every entity, sub-runs fall through to planner defaults, no stack trace.
- Error path: `resolve.auto_resolve` raises — the sub-run logs a warning and continues with planner defaults (does not fail the whole comparison).
- Integration: config `_auto_resolve_context` from entity A does not leak into entity B's `pipeline.run`. Assert each sub-run gets its own context string.
**Verification:**
- New integration test passes.
- End-to-end smoke (mock mode + explicit list): each sub-run's stderr shows `[AutoResolve]` lines per entity with distinct values.
- [ ] **Unit 3: Suppress LAW 7 warning for engine-internal sub-runs**
**Goal:** The `[Planner] No --plan passed... deterministic fallback` warning does not fire during competitor sub-runs. LAW 7 is load-bearing for hosting-model contracts and must stay on the default path; this is an opt-in bypass for internal fan-out only.
**Requirements:** R3
**Dependencies:** Unit 2 (so the sub-run call site is already being modified)
**Files:**
- Modify: `scripts/lib/planner.py` (`plan_query` signature + conditional stderr)
- Modify: `scripts/lib/pipeline.py` (`run` signature + propagation)
- Modify: `scripts/last30days.py` or `scripts/lib/fanout.py` (pass `internal_subrun=True` for competitor runners)
- Test: `tests/test_planner_v3.py` (or new `tests/test_planner_quiet_mode.py`)
- Test: `tests/test_competitor_fanout.py` (assert sub-runs don't emit LAW 7 stderr)
**Approach:**
- Add a keyword `internal_subrun: bool = False` to `planner.plan_query`. When True, skip the two `print(..., file=sys.stderr)` blocks that emit the LAW 7 banner and the `[Planner] No --plan passed` capability message.
- Add the same keyword to `pipeline.run()`; pass through to `plan_query`.
- In main()/fanout, set `internal_subrun=True` for every competitor sub-run's pipeline.run call. The main topic's pipeline.run keeps the default (LAW 7 stays on for the hosting-model path).
- Also suppress the LAW 7-triggered degraded-run warning block in the render layer for sub-reports when the envelope is going to be merged into a comparison output (or accept that the block is per-entity and surfaces once per entity).
**Patterns to follow:**
- Existing keyword-only parameters on `pipeline.run` (`mock`, `x_handle`, etc.).
- `planner.plan_query` signature is already keyword-only.
**Test scenarios:**
- Happy path: `plan_query(..., internal_subrun=True, provider=None, model=None)` returns the deterministic fallback plan WITHOUT writing the LAW 7 stderr block.
- Happy path: `plan_query(...)` with default `internal_subrun=False` still writes the LAW 7 warning (unchanged behavior).
- Integration: end-to-end competitor fanout; assert captured stderr contains zero occurrences of `No --plan passed` and zero of `YOU ARE the planner`.
- Integration: main topic is not part of competitor mode; if the user invokes bare `/last30days OpenAI` without `--plan`, LAW 7 stderr fires exactly once (regression test).
**Verification:**
- Running the Kanye-style smoke test shows zero `[Planner] No --plan passed` lines for Drake / Kendrick / Travis sub-runs.
- [ ] **Unit 4: "Resolved entities" block in comparison output**
**Goal:** The rendered comparison output includes a visible block listing per-entity handles, subreddits, GitHub user, and resolved context. Answers "did it resolve everyone?" at a glance without reading stderr.
**Requirements:** R4
**Dependencies:** Unit 2 (needs resolved data on report artifacts)
**Files:**
- Modify: `scripts/lib/render.py` (`render_comparison_multi` and `render_comparison_multi_context`)
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- When each entity's `Report.artifacts` contains a `resolved` dict (populated by Unit 2), `render_comparison_multi` emits a `## Resolved Entities` block early in the EVIDENCE envelope:
```
## Resolved Entities
- **Kanye West**: X @kanyewest | Subs r/Kanye, r/hiphopheads | GitHub: — | Context: BULLY released, UK ban…
- **Drake**: X @Drake | Subs r/DrakeTheType, r/hiphopheads | GitHub: — | Context: ICEMAN rollout…
- **Kendrick Lamar**: X @kendricklamar | Subs r/KendrickLamar | GitHub: — | Context: Grammy wins, dormant…
```
- Missing fields render as `` not empty.
- When no entity has a `resolved` payload (mock mode, no web backend), omit the block entirely rather than emit an empty section.
- Context strings are truncated at 120 chars to keep the block scannable.
**Patterns to follow:**
- Existing `render_comparison_multi` envelope structure (lines ~395-480 in render.py).
- Existing per-entity evidence block format (`## {label}`) for consistency.
**Test scenarios:**
- Happy path: 3 entities each with a `resolved` artifact → block lists all 3 with their fields.
- Happy path: 2 entities, one with full resolution, one with partial (x_handle only) → missing fields render as ``.
- Edge case: no entity has a resolved artifact → block is omitted entirely.
- Edge case: context string > 120 chars → truncated with ellipsis.
- Integration: rendered output passes through the same EVIDENCE envelope comments and synthesis contract (LAW 18 unchanged).
**Verification:**
- Snapshot tests confirm the block appears in the right spot with the right formatting.
- End-to-end smoke shows a realistic 3-entity Resolved block in the rendered output.
- [ ] **Unit 5: SKILL.md canonical-path self-check**
**Goal:** A top-of-file SKILL.md directive forces any reader (Claude Code, Codex, Hermes, Gemini) to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` before proceeding. If loaded from `marketplaces/` or any other path, re-read from the pinned versioned cache.
**Requirements:** R5
**Dependencies:** None
**Files:**
- Modify: `SKILL.md` (prepend a STEP 0 block before the existing STEP 0 / LAW list)
**Approach:**
- Add a numbered first step at the top (before or bundled with existing "STEP 0: ToolSearch preload"):
```
## STEP 0: Canonical Path Self-Check (must run first)
Before reading anything else below, verify you loaded this SKILL.md from
the versioned cache, not the marketplace clone:
CANONICAL=$HOME/.claude/plugins/cache/last30days-skill/last30days/
CANONICAL_LATEST=$(ls -d "$CANONICAL"*/ 2>/dev/null | sort -V | tail -1)
If the SKILL.md you just read is not under $CANONICAL_LATEST, STOP. Re-read
$CANONICAL_LATEST/SKILL.md and restart from here. Marketplace clones
(`plugins/marketplaces/last30days-skill/`) are pinned to origin/main and
can be stale; the versioned cache is the ground truth.
```
- Reinforce in the existing LAW 7 block that `--help` output must be read from the same pinned `SKILL_ROOT` to avoid flag-list skew.
**Patterns to follow:**
- Existing STEP 0 ToolSearch preload (top of SKILL.md) for tone / imperative voice.
- Existing `SKILL_ROOT` resolver snippet (line ~823).
**Test scenarios:**
- Test expectation: none — SKILL.md is documentation; no unit test, verified by follow-up user invocation.
**Verification:**
- In a fresh Claude Code window, `/last30days Test --competitors` loads SKILL.md, the model executes the STEP 0 self-check, and (if it had loaded from marketplaces/) switches to the cache path before running `--help` or the engine. Observable via the model's announced reasoning / task list.
- [ ] **Unit 6: Version bump, CHANGELOG, sync**
**Goal:** Ship 3.0.12 and deploy to all local targets.
**Requirements:** R6
**Dependencies:** Units 1-5
**Files:**
- Modify: `.claude-plugin/plugin.json` (version 3.0.11 → 3.0.12)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- CHANGELOG entry under `## [3.0.12]` dated 2026-04-22 covering the four fixes (Fixed: per-entity resolution; Fixed: LAW 7 sub-run noise; Changed: default count 3→2; Added: Resolved entities block; Added: canonical-path self-check in SKILL.md).
- `sync.sh` deploys to `~/.claude/plugins/cache/last30days-skill-private/...`, `~/.agents/`, `~/.codex/`, Hermes.
- Manual hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.12/` so the public `/last30days` slash command picks up the new version before PR merge (matches the 3.0.11 testing pattern).
**Test scenarios:**
- Test expectation: none — packaging only. Verification is by inspection.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns `3.0.12`.
- `sync.sh` exits 0 with "Import check: OK" for each target.
- Hot-copied 3.0.12 directory contains the new files and `/last30days` picks up the new version (highest-version resolver).
## System-Wide Impact
- **Interaction graph:** Fanout sub-runs now call `resolve.auto_resolve` per entity. Each sub-run is independent; no shared mutable state with other sub-runs or with the main topic.
- **Error propagation:** `auto_resolve` failures inside a sub-run log a warning and degrade to planner defaults; do not propagate up to abort the comparison. Same contract as today for the main topic.
- **State lifecycle risks:** Config dict is mutated by `auto_resolve` (via `config["_auto_resolve_context"]`). Must deep-copy per sub-run or scope context to a local mapping — otherwise two sub-runs' context strings race.
- **API surface parity:** `pipeline.run` gains a keyword (`internal_subrun`); callers that don't pass it get the existing behavior. `planner.plan_query` gains the same. Backward compatible.
- **Integration coverage:** New integration test for the fanout + auto-resolve + render chain. Existing snapshot tests update to include the Resolved block.
- **Unchanged invariants:** Single-entity `/last30days` invocations (no `--competitors`) behave identically. Explicit `A vs B` comparison topics behave identically. LAW 7 still fires on the default hosting-model path. `render_compact` path is untouched.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Auto-resolving per competitor triples the WebSearch call volume (4 queries × 3 competitors = 12 extra web searches). | Fast-fail when no backend; user can pass `--competitors-list` to skip discovery but still get auto-resolve. Cost note in CHANGELOG. |
| Config mutation across sub-runs via `_auto_resolve_context`. | Unit 2 deep-copies config per sub-run before each `auto_resolve` + `pipeline.run` call. Integration test asserts no cross-entity leak. |
| LAW 7 suppression leaks onto the hosting-model path via a wrong default. | Default `internal_subrun=False`. Only fanout's competitor sub-runs set True. Unit test asserts bare-topic invocation still emits LAW 7. |
| SKILL.md STEP 0 banner gets ignored by the model (same failure mode as line 823 today). | Put it in the guaranteed-read top band (before LAW 1, above all other content), imperative voice, concrete `STOP` verb. Still not bulletproof but strictly better than current. |
| Default count change breaks assumptions in downstream tools or existing user muscle memory. | Changelog calls it out as Changed; `--competitors=3` still works for users who want the old default. |
## Documentation / Operational Notes
- Beta channel first: merge behind `/last30days-beta` via the private repo before cherry-picking to public. Follows the same process as 3.0.11.
- Version 3.0.12 is a fix release; no marketing post required.
- After merge, add a line to the PR description pointing at this plan.
## Sources & References
- Origin plan: `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- Related PR: #308 (v3.0.11 shipping --competitors)
- Test windows that surfaced the bugs: Kanye, Linear, Coinbase (2026-04-22 session)
- Related code: `scripts/lib/fanout.py`, `scripts/lib/resolve.py` (`auto_resolve`), `scripts/lib/planner.py` (`plan_query`), `scripts/lib/render.py` (`render_comparison_multi`)
@@ -0,0 +1,394 @@
---
title: "fix: --competitors runs a full last30days per entity with hosting-model pre-resolve"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md
---
# fix: --competitors runs a full last30days per entity with hosting-model pre-resolve
## Overview
User intent confirmed 2026-04-22: `--competitors` should run a full single-entity `last30days` pipeline for the main topic AND for each discovered peer — three independent full-depth passes, each with its own Step 0.55 resolution, own X handle primary weight, own subreddit targeting, own GitHub repo scoping. Then merge them into the comparison output.
3.0.12 already built the N-parallel-pipelines orchestration (`scripts/lib/fanout.py`). What it got wrong: it tried to do per-entity Step 0.55 engine-side via `resolve.auto_resolve()`, which requires a web search backend key (BRAVE/EXA/SERPER/PARALLEL/OPENROUTER). Matt runs from Claude Code, which has its own WebSearch tool. The engine has none of those keys, so per-entity auto_resolve silently no-ops and all peer sub-runs fall through to deterministic single-word planner queries.
Four 2026-04-22 test runs (Warriors, Seattle, Arizona Wildcats, Kanye West) confirmed this via engine receipts:
- Compact Resolved Entities block shows peers as `X - | Subs - | GitHub - | Context: -`.
- Sub-run planner lines show `source=deterministic, subqueries=1` — the "I gave up and keyword-searched" shape.
- Engine footer keeps nudging `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY`, which is wrong advice for a Claude Code user who already has WebSearch.
- Kanye run leaked main topic's `--subreddits` into Drake's and Kendrick's sub-runs (regression bug).
The fix is to flip the resolution responsibility: the hosting model (Claude Code, Codex, Hermes, Gemini) does Step 0.55 via its own WebSearch tool for every entity, then passes the resolved targeting to the engine via a new `--competitors-plan` JSON flag. Engine fan-out remains — each peer still runs a full `pipeline.run()`. The difference is the peers now arrive with full targeting, equivalent to the main topic, so retrieval is apples-to-apples.
Why not just reuse vs-mode? vs-mode is a SINGLE `pipeline.run()` with a comparison-optimized plan. It pre-resolves Step 0.55 per entity but merges everything into one retrieval pool with lower-weight `--x-related` for peers, merged subreddits, and cross-entity keyword noise. That is not "three full passes." The user explicitly wants three full passes.
## Problem Frame
3.0.12's architecture was correct; its data dependency was wrong.
| Capability | 3.0.12 path | Target path (this plan) |
|---|---|---|
| Fan out to N parallel pipelines | Yes (`fanout.run_competitor_fanout`) | Same — keep |
| Per-entity Step 0.55 resolution | Engine-internal `resolve.auto_resolve()` — needs BRAVE/EXA/SERPER/PARALLEL key | Hosting model does it via its own WebSearch, passes to engine |
| Per-entity targeting threaded into `pipeline.run()` | Main topic only via outer flags; peers via auto_resolve (failing) or nothing | Main topic via outer flags; peers via `--competitors-plan` JSON |
| Footer nudge | Unconditional BRAVE/SERPER | Suppressed when `--plan` or `--competitors-plan` present |
| Resolved Entities block in raw save file | Stdout only | Also in `--save-dir` raw file |
| Override-leak from main into peers | Present (Kanye receipt) | Fixed via explicit per-entity kwargs scrub |
| Polymarket noise on ambiguous topics | Present (Warriors, Arizona receipts) | `--polymarket-keywords` + auto-skip for single-token-ambiguous |
The key architectural change is who owns per-entity resolution. The engine stops trying to do it itself; the hosting model does it upstream (it already has WebSearch) and passes results in.
This is the same pattern `--plan` already uses for the main topic: hosting model generates the plan via its own reasoning, passes it in, engine accepts. We apply the pattern to peers.
## Requirements Trace
- R1. New `--competitors-plan` JSON flag accepting per-entity targeting: `x_handle`, `x_related`, `subreddits`, `github_user`, `github_repos`, `context`. Implies `--competitors`. Per-entity values thread into that entity's `pipeline.run()`. Bypasses engine-internal `auto_resolve` for covered entities.
- R2. SKILL.md "Competitor mode" rewritten to make the hosting-model path canonical: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers) via WebSearch, (c) assemble `--competitors-plan` JSON, (d) invoke engine. Engine-internal auto_resolve remains as headless fallback.
- R3. The LAW 7-style stderr emitted when `--competitors` has no list, no plan, no backend is reframed: leads with "hosting reasoning model, use your WebSearch to run Step 0.55 per entity and pass `--competitors-plan`." Does not lead with BRAVE_API_KEY.
- R4. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` OR `--competitors-plan` was passed. Signal: hosting model is driving and already has WebSearch.
- R5. Override-leak fix: competitor sub-runs do not inherit main topic's `--subreddits`, `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`. Sub-runs use only their own per-entity targeting (from `--competitors-plan` if provided, else engine-internal auto_resolve if backend, else planner defaults).
- R6. The `## Resolved Entities` block is also appended to the saved raw file when `--save-dir` is in use. Each entity's effective targeting (whatever was actually passed to its `pipeline.run()`) is visible on audit.
- R6b. When `--save-dir` is in use with a comparison run, each entity's sub-run ALSO saves its own standalone raw file — same format as a single-entity run. `/last30days Kanye West --competitors` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md` (one per entity) plus the merged comparison file. Matches the historical vs-mode behavior when it ran as N passes.
- R7. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R8. Default `--competitors` count remains 2 (3-way: main + 2 peers). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to `scripts/lib/fanout.py` architecture. N parallel pipelines stays. Only the data each sub-run receives changes.
- No changes to the vs-mode (topic contains "vs" / "versus") behavior. That path is independent.
- No new emit modes. Comparison output format unchanged.
- No deprecation of `--competitors-list`. Stays as the minimum escape hatch for hosting models that skip per-entity Step 0.55 (names-only).
### Deferred to Separate Tasks
- Cache layer for hosting-model competitor resolution: separate plan once cost evidence exists.
- Cross-source disambiguation beyond Polymarket: separate plan.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — `--competitors` / `--competitors-list` argparse block, `resolve_competitors_args` validator, `_main_runner` closure, `_competitor_runner` closure, the `[Competitors] --competitors requires...` stderr block. Primary file for this plan.
- `scripts/lib/fanout.py` — `run_competitor_fanout` orchestrator. Signature unchanged; `_competitor_runner` closure now builds kwargs from `--competitors-plan`.
- `scripts/lib/pipeline.py` — `pipeline.run()` signature; no changes required (all per-entity flags already exist as kwargs).
- `scripts/lib/planner.py` — existing `--plan` parsing and validation, pattern to mirror for `--competitors-plan`.
- `scripts/lib/render.py` `_render_resolved_entities_block` (added in 3.0.12) — already reads `report.artifacts["resolved"]`; no change needed.
- `scripts/last30days.py` `save_output` / `render.render_full` — the save path. Needs to include the Resolved Entities block for comparison runs.
- `scripts/lib/quality_nudge.py` — where the BRAVE/SERPER footer nudge is emitted. Needs a context-aware suppression check.
- `scripts/lib/polymarket.py` — source adapter. Entry point for `--polymarket-keywords` filter and single-token-ambiguous auto-skip.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built the initial fanout, deferred per-entity resolve as "v1 simplification."
- 3.0.12 plan (`2026-04-22-003`): tried to close the gap via engine-internal `auto_resolve`. Works only with backend keys. Fails silently without.
- 2026-04-22 test session receipts: confirmed all four fixes in this plan are real, reproducible bugs.
- User's architectural steer 2026-04-22: "runs a full last30days on all 3 topics" — this plan encodes that explicitly as N full `pipeline.run()` calls with pre-resolved targeting per entity.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **`--competitors-plan` is a single JSON flag, not a fan of separate flags.** Mirrors `--plan`. Stable schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Accept inline JSON or a file path (matches `--plan`).
- **Hosting-model-driven resolution is the documented default.** Engine-internal `auto_resolve` is the headless / cron fallback. SKILL.md routes hosting models to the JSON-flag path; engine keeps auto_resolve alive for BRAVE/EXA/SERPER users running CI.
- **Override-leak fix is call-site scrubbing, not a signature change.** `_competitor_runner` builds an explicit kwargs dict per entity from `_subrun_kwargs(entity, plan_entry)`. No closure-default fallthrough from main scope. The 3.0.12 `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag.
- **Footer nudge becomes context-aware.** Suppressed when `--plan` or `--competitors-plan` present. Not suppressed for bare `--competitors-list` or bare invocations. Headless cron without keys still sees the nudge.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known list of single-token-ambiguous names (states, common nouns). Stderr notes the skip so it is observable and overridable.
- **Per-entity sub-runs get the full `pipeline.run()` pass.** Same depth, same sources, same API cost per entity as a single-topic run. This is the explicit user intent — three full passes, not one merged pass.
## Open Questions
### Resolved During Planning
- **JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers (3-way comparison). Unchanged from 3.0.12.
- **Does engine-internal auto_resolve stay alive?** Yes, for entities not covered by `--competitors-plan` when a backend is configured. Headless/cron users with keys keep the current 3.0.12 behavior.
- **vs-mode or fanout?** Fanout. User's explicit ask: three full passes, not one merged pass. vs-mode merges into one pipeline with lower peer weighting, which is not what the user wants.
- **Does the save file need per-entity clusters?** Start with the Resolved block appended. Per-entity cluster sections can follow in a separate task; they are nice-to-have, not blocking.
### Deferred to Implementation
- Exact trace of override-leak source. Candidates: closure capture of `subreddits` in `_competitor_runner`, shared `_auto_resolve_context` leak, Reddit adapter inheriting global config. Test-first; trace at implementation time.
- Heuristic for "single-token-ambiguous topic" auto-skip. Start with a short hard-coded list (US state names, US city names, common nouns like "Warriors", "Suns", "Jets"); revisit after dogfood.
- Whether per-entity coverage warnings fire when `--competitors-plan` under-resolves an entity (e.g., only `x_handle`, no subreddits). Start with stderr logging; revisit UX.
## Implementation Units
- [ ] **Unit 1: `--competitors-plan` JSON flag + per-entity kwargs threading**
**Goal:** New CLI flag accepting per-entity targeting JSON. Each covered entity's `pipeline.run()` receives its own `x_handle` / `x_related` / `subreddits` / `github_user` / `github_repos` / `context`. Skips engine-internal `auto_resolve` for covered entities.
**Requirements:** R1, R5 (primary leak fix site)
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner`)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; verify)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON OR a file path (mirror `--plan`).
- Validation: parse JSON; must be a dict; each value must be a dict; unknown fields log warnings; malformed input exits 2.
- Schema per entity: optional fields `x_handle` (str), `x_related` (list), `subreddits` (list), `github_user` (str), `github_repos` (list), `context` (str).
- Case-insensitive matching against `--competitors-list` / discovered entities.
- Build `_subrun_kwargs(entity, plan_entry)` helper. Returns a complete, explicit kwargs dict for `pipeline.run()` with no closure-default fallthrough from main scope. This helper is the single source of truth for per-entity call args. It also fixes the override-leak (R5) by scrubbing all per-entity flags to None unless the plan (or auto_resolve) sets them.
- `_competitor_runner(entity)`:
1. Look up `plan_entry` from `--competitors-plan` (if any).
2. If plan covers entity fully, build kwargs from it; skip `auto_resolve`.
3. If plan partially covers or is absent, fall back to `auto_resolve` (3.0.12 behavior) when a backend is configured. Plan values win over auto_resolve values on conflict.
4. If neither plan nor backend, fall through to `pipeline.run()` with per-entity kwargs all None — engine uses planner defaults for that entity only (no leak).
- Deep-copy config per sub-run (already done in 3.0.12); merge per-entity `context` into `entity_config["_auto_resolve_context"]` only.
**Execution note:** Test-first for the override-leak regression (pass `--subreddits=A,B` on main + a peer, assert peer's `pipeline.run(subreddits=...)` is None or peer-specific).
**Patterns to follow:**
- `--plan` parsing at `scripts/last30days.py` (inline JSON or file path).
- 3.0.12's `_competitor_runner` closure for scope; extract the kwargs-build into `_subrun_kwargs` helper.
- `entity_config = dict(config)` deep-copy pattern from 3.0.12.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake": {"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's `pipeline.run` receives `x_handle="Drake"` and `subreddits=["Drizzy"]`; no `auto_resolve` call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered entities skip auto_resolve; third falls back to auto_resolve.
- Happy path: plan file path accepted like `--plan` file path.
- Happy path: case-insensitive entity match (`Drake` in plan, `drake` in list).
- Edge case: unknown fields in plan entry → logged, ignored, run continues.
- Edge case: plan entry for entity not in list → ignored with warning.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list not dict → exit 2.
- Regression (leak fix): main `--subreddits=A,B` + `--competitors-list "Drake"` + no plan → Drake's `pipeline.run` receives `subreddits=None` (no leak).
- Regression (leak fix): same for `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`.
- Regression (leak fix): main `--x-handle=kanyewest` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanyewest"`.
- Integration: full main + 2 peers run via `--competitors-plan`; assert each sub-run's effective kwargs match expected per-entity values.
**Verification:**
- All new and regression tests pass.
- Smoke run (mock mode + `--competitors-plan`): stderr shows `[Competitors] Drake: x=@Drake subs=Drizzy` line per entity; no `[AutoResolve]` calls for plan-covered entities; no leak of main topic's flags.
- [ ] **Unit 2: Reframe LAW 7-style stderr for hosting-model context**
**Goal:** When `--competitors` has no `--competitors-list`, no `--competitors-plan`, and no backend, stderr tells the hosting reasoning model to use its WebSearch tool for Step 0.55 per entity and pass `--competitors-plan`. Stops leading with BRAVE_API_KEY.
**Requirements:** R3
**Dependencies:** Unit 1 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the existing `[Competitors] --competitors requires...` block)
- Test: `tests/test_competitors_no_backend_message.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent runtime with a WebSearch tool), YOU should: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers), (c) assemble a `--competitors-plan` JSON, (d) re-invoke. Skip this step and quality degrades — peer entities will run with planner defaults."
2. "If you are running headless (cron, CI, no hosting model), set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list "A,B,C"` skips discovery but does not pre-resolve peers. Use only for quick tests."
- Exits non-zero as today.
**Patterns to follow:**
- Existing LAW 7 stderr in `planner.plan_query` for tone.
**Test scenarios:**
- Happy path: stderr leads with "If you are the hosting reasoning model" and names `--competitors-plan` before any backend key.
- Happy path: stderr explicitly names `--competitors-plan` as the preferred override.
- Happy path: stderr does NOT say "requires either a configured web search backend OR an explicit --competitors-list" (the current 3.0.12 wording).
**Verification:**
- Test asserts ordering and required phrases.
- [ ] **Unit 3: Suppress BRAVE/SERPER footer nudge when hosting-model-driven**
**Goal:** The `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY` footer is suppressed when `--plan` or `--competitors-plan` was passed (signal: hosting model is driving and already has WebSearch).
**Requirements:** R4
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/lib/quality_nudge.py` (or wherever nudge is emitted; verify during implementation)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Locate the nudge emission point.
- Add a suppression check: if `--plan` OR `--competitors-plan` was passed, skip the nudge. Otherwise, current behavior.
- Don't suppress the nudge for bare `--competitors-list` alone — that path isn't necessarily hosting-model-driven.
**Test scenarios:**
- Happy path: `--plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-list` only, no backend → nudge fires (current behavior).
- Happy path: no `--competitors`, no `--plan`, no backend → nudge fires (current behavior unchanged).
**Verification:**
- All four scenarios produce expected nudge presence/absence.
- [ ] **Unit 4: Per-entity save files + Resolved block in each**
**Goal:** When `--save-dir` is in use with a comparison run, each entity's sub-run saves its own standalone raw file (same format as a single-entity run), and each file includes the `## Resolved Entities` block so audits can see what targeting that entity received. Matches the historical vs-mode behavior when it was N passes.
**Requirements:** R6, R6b
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/last30days.py` (`save_output`, the save loop after fanout completes)
- Possibly modify: `scripts/lib/render.py` (`render_full` branch to include Resolved block when artifact is present)
- Test: `tests/test_save_raw_competitor_files.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]`. For each `(entity, entity_report)` tuple, call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)` — same path a single-entity run takes.
- Each saved file uses its entity's slug as the filename (`drake-raw.md`, `kendrick-lamar-raw.md`). Main topic keeps the existing `kanye-west-raw.md` filename.
- Each file includes its own `## Resolved Entities` block (single-entity variant: one row for that entity only). This makes each sub-run's file self-describing — you can see what targeting was used without opening the comparison file.
- The merged comparison output (stdout) still includes the 3-row Resolved Entities block.
- Optional: also save a comparison summary file (e.g., `kanye-west-comparison-raw.md`) holding the merged multi-entity render. Start with per-entity files only; comparison summary is a follow-up if stdout-plus-individual-files is insufficient.
- Single-entity runs unchanged (no additional files, no block change).
**Patterns to follow:**
- Existing `save_output` invocation for single-entity runs (line 501 of current `scripts/last30days.py`).
- Existing slug generation (`slugify(topic)`) for filename consistency.
- `_render_resolved_entities_block` from 3.0.12 for the single-entity variant.
**Test scenarios:**
- Happy path: `--competitors-list "Drake,Kendrick Lamar"` + `--save-dir=/tmp/x` → `/tmp/x/kanye-west-raw.md`, `/tmp/x/drake-raw.md`, `/tmp/x/kendrick-lamar-raw.md` all exist.
- Happy path: each peer file's first sections include that entity's Resolved Entities block with its own row only.
- Happy path: single-entity run with `--save-dir` → one file, unchanged from today's behavior.
- Edge case: entity slug collides with existing file → overwrite (matches single-entity behavior).
- Edge case: `--save-suffix=v3` → all 3 files get the suffix (`kanye-west-raw-v3.md`, `drake-raw-v3.md`, `kendrick-lamar-raw-v3.md`).
- Edge case: comparison run with one peer whose sub-run failed → that entity's file is NOT saved; others are.
- Integration: stderr after save shows three `[last30days] Saved output to <path>` lines, one per entity.
**Verification:**
- After `/last30days Kanye West --competitors-list "Drake,Kendrick Lamar" --save-dir=/tmp/x`: `ls /tmp/x/*-raw.md` shows 3 files. Each contains its entity's Resolved block.
- [ ] **Unit 5: SKILL.md "Competitor mode" rewrite — hosting-model Step 0.55 canonical**
**Goal:** SKILL.md documents the hosting-model-driven path as canonical: discover N peers via WebSearch, run Step 0.55 per entity, assemble `--competitors-plan`, invoke engine. Engine-internal `auto_resolve` is labeled the headless fallback.
**Requirements:** R2
**Dependencies:** Unit 1 (flag must exist before documented)
**Files:**
- Modify: `SKILL.md` (Competitor mode subsection)
- Modify: `README.md` (one-line example update)
**Approach:**
- Replace the 3.0.12 Competitor mode subsection with a clear flow:
1. User invokes with `--competitors` or `--competitors=N`.
2. Hosting model runs WebSearch for "[topic] competitors" / "[topic] alternatives" → picks top N peers.
3. Hosting model runs Step 0.55 for main + each peer (x_handle, subreddits, github_user, github_repos, context) — same protocol as vs-mode per SKILL.md §679.
4. Hosting model assembles a `--competitors-plan` JSON object.
5. Hosting model invokes the engine with `--competitors-list "A,B,C" --competitors-plan '{...}'`.
6. Engine fans out N full pipelines (main + peers), each with its own full Step 0.55-grade targeting. Each entity also saves its own `*-raw.md` file when `--save-dir` is set (three full passes → three save files, matching the historical vs-mode behavior). Comparison output merges them for display.
- Concrete JSON example in SKILL.md showing the schema.
- Failure-mode warning: a `## Resolved Entities` block with dashes for any entity means hosting model skipped Step 0.55 for that one. Re-run with corrected plan.
- "Headless fallback" sub-subsection: when BRAVE/EXA/SERPER/PARALLEL/OPENROUTER is set, engine's internal `auto_resolve` handles peers and `--competitors-plan` is optional.
**Patterns to follow:**
- SKILL.md "Step 0.55" section for per-entity resolve protocol.
- SKILL.md "If QUERY_TYPE = COMPARISON" section for the same-protocol-as-vs-mode reference.
- Tone of existing 3.0.12 Competitor mode prose.
**Test scenarios:**
- Test expectation: none — documentation. Verification is a fresh Claude Code window dogfood run.
**Verification:**
- `/last30days Kanye West --competitors` in a new window: hosting model does Step 0.55 for Kanye + 2 discovered peers; passes `--competitors-plan`; rendered Resolved block shows non-empty fields for all 3; top voices include at least one peer-specific handle.
- [ ] **Unit 6: Polymarket disambiguation guard**
**Goal:** Support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
**Requirements:** R7
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` argparse (`--polymarket-keywords`)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"` flag. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip rule: if topic is one token AND token matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords` provided, skip Polymarket with a stderr note.
- SKILL.md Step 0.55 protocol gets a small addition: for ambiguous topics, hosting model passes `--polymarket-keywords` with topic-specific qualifiers.
**Patterns to follow:**
- Existing Polymarket adapter match logic.
- Single-token detection heuristic.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr notes the skip.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs; matches filtered.
- Happy path: topic "OpenAI" (no ambiguity) → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke run → Polymarket footer absent OR filtered to nba/gsw markets.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R7
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG entry groups the fixes: Added `--competitors-plan` JSON flag for per-entity hosting-model pre-resolve. Fixed override-leak from main into peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when `--plan` / `--competitors-plan` is present. Added: Resolved Entities block persists to saved raw file. Added: `--polymarket-keywords` + auto-skip for ambiguous single-token topics.
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13 immediately.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files with competitors.py, fanout.py, the updated SKILL.md, and plugin.json 3.0.13.
## System-Wide Impact
- **Interaction graph:** `_competitor_runner` becomes the single source of truth for sub-run kwargs via `_subrun_kwargs(entity, plan_entry)`. Every per-entity flag flows through one helper. No closure-default leaks.
- **Error propagation:** `--competitors-plan` JSON parse errors exit 2 with stderr (same as `--plan`). Per-entity plan entries with malformed values log warnings and fall back; don't abort the whole run.
- **State lifecycle risks:** `entity_config = dict(config)` already deep-copies for `_auto_resolve_context`; extend the isolation discipline to every per-entity flag. Verified in Unit 1 regression tests.
- **API surface parity:** `--competitors-plan` is additive. `--competitors` and `--competitors-list` unchanged. `--plan` unchanged. `--polymarket-keywords` additive.
- **Integration coverage:** New regression tests for override-leak. New integration test for plan-driven sub-run threading. New nudge-suppression test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. `planner.plan_query` LAW 7 behavior for the default path unchanged. Single-entity render path unchanged. vs-mode behavior unchanged.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model takes the lazy path and uses `--competitors-list` names-only. | Unit 2 stderr explicitly steers to `--competitors-plan` with Step 0.55 protocol named. Unit 5 SKILL.md docs. Resolved Entities dashes in output make the gap visible. |
| JSON gets verbose for the hosting model to construct repeatedly. | Schema is small (≤6 fields per entity). Hosting model already runs Step 0.55 for main topic in every comparison run; peers use the same protocol. One JSON block replaces N CLI flags. |
| Override-leak source is deeper than `_competitor_runner` closure. | Test-first per Unit 1. Receipts from 2026-04-22 Kanye run are reproducible. Trace methodically from call site. |
| Plan-covered entity bypasses auto_resolve but plan data is incomplete (e.g., no subreddits). | Hosting model's own SKILL.md contract says Step 0.55 must cover all fields. Stderr logs per-entity coverage so under-resolved entities are visible. Next-run correction, not engine-side rescue. |
| Polymarket auto-skip false-positives on legitimate ambiguous topics with real markets. | Conservative match (single-token + known list). `--polymarket-keywords` override is explicit and unambiguous. Stderr notes the skip. |
| Footer nudge suppression hides the message from headless users who genuinely need it. | Suppression only fires when `--plan` or `--competitors-plan` is present. Cron / CI runs that pass neither still see the nudge. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md (private repo `/last30days-beta`).
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG voice should call this out as the feedback-driven follow-up to 3.0.12. Reader should see "we tried engine-internal resolve in 3.0.12; it needs backend keys we don't have; we moved resolution to the hosting model in 3.0.13."
## Sources & References
- Origin plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Earlier plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts: Warriors, Seattle, Arizona Wildcats, Kanye West
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -0,0 +1,454 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: vs mode runs N full passes and --competitors is vs with auto-discovery"
type: feat
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded
---
# feat: vs mode runs N full passes and --competitors is vs with auto-discovery
## Overview
Architectural unification driven by user correction 2026-04-22: vs mode and `--competitors` are the same thing. A user typing `/last30days OpenAI vs Anthropic vs xAI` should get a full single-entity last30days pass for each of the three entities — three full pipelines, three saved `*-raw.md` files, merged into one comparison output. A user typing `/last30days OpenAI --competitors` should get the same output after the hosting model auto-picks 2 peers; i.e., `--competitors` is a thin shortcut that expands "topic + `--competitors`" into "topic vs peer1 vs peer2" and then runs the unified vs pipeline.
Current state diverges from this:
- **vs mode today**: one `pipeline.run()` with a comparison-optimized plan that merges all entities' targeting into a single retrieval pool. Lower-weight `--x-related` for peers, merged subreddits, cross-entity keyword noise. One saved file.
- **`--competitors` today (3.0.12)**: N parallel `pipeline.run()` calls via `scripts/lib/fanout.py`, but per-entity Step 0.55 depends on an engine-side web backend key Matt doesn't have. Silently degrades to planner defaults for peers. One saved file (main topic only). Override-leak from main into peers.
After this plan:
- **vs mode**: N parallel `pipeline.run()` calls, one per entity, each with its own full Step 0.55-grade targeting, each saving its own `*-raw.md`. Merged into one comparison output.
- **`--competitors`**: SKILL.md shortcut. Hosting model discovers N peers, builds `"topic vs peer1 vs peer2"`, and invokes the same vs pipeline. No separate orchestration path.
- **Same fanout machinery (`scripts/lib/fanout.py`)** serves both. One fix, both behaviors improve.
## Problem Frame
The product insight from 2026-04-22 test runs is simple: the user wants three full last30days reports plus a comparison merge. Not one comparison pass with N-way targeting merged into a single retrieval pool. Not one save file. Not "main gets Step 0.55, peers get planner defaults." Three full passes. Three save files. Merged output.
The historical vs mode did that (it ran as 3 passes, saving 3 files). SKILL.md §551 currently says:
> "When the user asks 'X vs Y', run ONE research pass with a comparison-optimized plan that covers both entities AND their rivalry. This replaces the old 3-pass approach (which took 13+ minutes and produced tangential content)."
That change was a latency optimization that removed the user-visible behavior the user wants. The fix is to revert the architectural direction: N passes per entity, in parallel rather than serial (parallelism lowers wall-clock to ~1× a single pass, not N×), with per-entity save files.
The 3.0.11 `--competitors` flag already introduced parallel N-pass machinery (`fanout.run_competitor_fanout`). The 3.0.12 follow-up tried to wire per-entity Step 0.55 into it but failed when no web backend was configured. The elegant move: stop maintaining two architectures. vs-mode and `--competitors` both use `fanout.py`. `--competitors` becomes a SKILL.md-level shortcut that discovers 2 peers and hands off to vs-mode.
Four 2026-04-22 test receipts (Warriors, Seattle, Arizona Wildcats, Kanye West) all confirmed the user's pain points:
- Peers thin because they ran without per-entity handle/sub targeting.
- Only one `*-raw.md` per run — no per-entity audit.
- Kanye peers leaked main topic's `--subreddits`.
- Engine footer nudging `BRAVE_API_KEY` to Claude Code users who already have WebSearch.
- Polymarket noise on ambiguous topics (Warriors → Glasgow rugby; Arizona → Diamondbacks).
This plan closes all of them by unifying the architecture and making hosting-model-driven Step 0.55 per entity the canonical path.
## Requirements Trace
- R1. vs mode (any topic containing ` vs ` / ` versus `) runs N full `pipeline.run()` calls in parallel, one per entity. Each sub-run uses its entity's own Step 0.55 targeting (from the hosting model's pre-resolution, passed via a new `--competitors-plan` JSON).
- R2. `--competitors` (and `--competitors=N`) becomes a SKILL.md-level shortcut: the hosting model (a) discovers N peers via WebSearch, (b) runs Step 0.55 per entity (main + peers), (c) rewrites the topic to `"main vs peer1 vs peer2"`, (d) invokes the engine with `--competitors-plan` containing each entity's targeting.
- R3. New `--competitors-plan` JSON flag. Schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Implies vs mode when present with a single-entity topic. Applies per-entity targeting to each sub-run. Accepts inline JSON or a file path (matches `--plan`).
- R4. Each entity's sub-run saves its own `*-raw.md` file when `--save-dir` is in use. Example: `/last30days "Kanye West vs Drake vs Kendrick Lamar" --save-dir=~/Documents/Last30Days` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`. Same filenames a single-entity run of each topic would produce. Matches historical vs-mode behavior.
- R5. Each per-entity saved file includes its own single-row `## Resolved Entities` block so the audit survives. The merged comparison stdout still shows the full 3-row block.
- R6. Override-leak fix: no main-topic flags (`--subreddits`, `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`) leak into peer sub-runs. Every per-entity kwarg is scrubbed at the sub-run call site.
- R7. LAW 7-style stderr for `--competitors` invocations with no list, no plan, no backend is reframed for hosting-model context: leads with "use your WebSearch to discover peers, resolve Step 0.55 per entity, re-invoke with `topic vs peer1 vs peer2 --competitors-plan '...'`." Does not lead with BRAVE_API_KEY.
- R8. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` or `--competitors-plan` was passed.
- R9. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R10. Default `--competitors` count stays 2 peers (3-way comparison). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to single-entity `pipeline.run()` semantics. Each sub-run in vs mode behaves identically to a bare `/last30days {entity}` invocation.
- No changes to the planner's comparison-intent logic for single-entity-containing topics. The `_should_force_deterministic_plan` shortcut for vs-topics routes to fanout, not to its current single-pipeline path.
- No new emit modes. Comparison output format unchanged.
- No removal of `--competitors-list`. Stays as a minimum escape hatch (names-only, no per-entity targeting) for scripted headless use.
- No removal of engine-internal `resolve.auto_resolve()` in fanout. Remains as headless / cron fallback for users with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys. The dominant Claude Code path bypasses it via `--competitors-plan`.
### Deferred to Separate Tasks
- Explicit "head-to-head" rivalry pass in vs-mode (a supplemental subquery like `"A vs B"` that catches rivalry articles missing from pure entity-scoped passes). Start with N independent passes; add a head-to-head supplemental pass if the rivalry-content gap shows up in dogfood.
- Cache layer for hosting-model pre-resolution.
- Cross-source disambiguation (not just Polymarket).
- Latency knob for users who want the old one-pass vs behavior (probably not needed; parallel N-pass is ~1× wall clock).
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — main(), `_main_runner`, `_competitor_runner`, the competitor enable/discovery branch. Primary file.
- `scripts/lib/fanout.py` — existing orchestrator (3.0.11). Reused as-is; `competitor_runner` closure is where per-entity kwargs apply.
- `scripts/lib/planner.py``_should_force_deterministic_plan` detects vs-topics via regex. Current path synthesizes ONE comparison plan; new path routes to fanout.
- `scripts/lib/render.py``render_comparison_multi` (3.0.12) + `_render_resolved_entities_block`. Both reused. `render_full` needs a per-entity variant when saving sub-run files.
- `scripts/last30days.py` `save_output` — where raw files are written. Needs to iterate per entity when competitor_reports artifact present.
- `scripts/lib/quality_nudge.py` — BRAVE/SERPER nudge emission.
- `scripts/lib/polymarket.py` — source adapter for `--polymarket-keywords` and ambiguous-topic auto-skip.
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol — the hosting-model contract that drives per-entity pre-resolution for both vs mode and `--competitors`.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built fanout.
- 3.0.12 plan (`2026-04-22-003`): tried engine-internal per-entity auto_resolve; failed without backend keys.
- 3.0.13 plan draft (`2026-04-22-004-...superseded`): proposed `--competitors-plan` JSON + vs-mode-shortcut path but kept them separate. User's 2026-04-22 correction unifies them.
- 2026-04-22 test receipts: Warriors, Seattle, Arizona Wildcats, Kanye West runs all reproduced the per-entity resolve gap.
- User's architectural steer: "vs mode should work that way too" + "--competitors is just vs mode with auto-discovery." This plan encodes that.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **Unify vs-mode and --competitors on one orchestrator.** `fanout.run_competitor_fanout` serves both. vs-mode is "topic contains ' vs '" detection → fanout. `--competitors` is "SKILL.md shortcut → hosting model rewrites topic to vs form → fanout." One code path.
- **Per-entity targeting via `--competitors-plan` JSON.** Schema `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Mirrors `--plan`. Applies to both vs-mode and `--competitors` paths. Hosting model passes it after running Step 0.55 per entity.
- **N save files, one per entity.** Each sub-run writes a `{entity-slug}-raw.md` file when `--save-dir` is set. Matches historical vs-mode behavior. Single-entity runs unchanged.
- **Revert the "one pass for latency" optimization that removed per-entity passes.** Parallel execution via `ThreadPoolExecutor` means wall-clock is ~max(per-entity-latency), not sum. The old latency concern (13+ minutes for 3 serial passes) does not apply to a parallel fan-out.
- **Override-leak fix at the call site.** `_subrun_kwargs(entity, plan_entry)` helper returns fully explicit per-entity kwargs; no closure-default fallthrough from main scope.
- **LAW 7 stderr reframed, not just updated.** Current message treats BRAVE_API_KEY as the solution. New message treats hosting-model Step 0.55 as the solution, with backend keys listed only as the headless fallback.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known-ambiguous single-token list.
## Open Questions
### Resolved During Planning
- **vs mode N passes or single-pass?** N passes. User's architectural correction.
- **Should --competitors still be an engine flag at all?** Yes, kept for headless / cron contexts with backend keys. Dominant Claude Code path is SKILL.md shortcut → vs-mode fanout. Engine flag stays as compatibility surface.
- **`--competitors-plan` JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers → 3-way comparison. Unchanged.
- **Saved-file naming?** `{entity-slug}-raw.md` per entity, same as single-entity runs would produce.
### Deferred to Implementation
- Exact trace of override-leak path (closure capture vs shared config vs Reddit adapter fallback). Test-first per Unit 2; patch at the right layer.
- Heuristic for single-token-ambiguous Polymarket auto-skip. Start with a short hard-coded list; iterate.
- Whether to include a head-to-head rivalry supplemental pass in vs-mode. Ship N-independent passes first; revisit after dogfood if rivalry content is missing.
- Exact filename convention when the comparison merged output is saved (if saved at all). Not blocking — per-entity files are the primary save artifact.
## High-Level Technical Design
> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
```
User invokes:
/last30days "OpenAI vs Anthropic vs xAI"
OR
/last30days OpenAI --competitors (hosting model rewrites to vs form)
OR
/last30days OpenAI --competitors-list "Anthropic,xAI"
OR
/last30days "OpenAI vs Anthropic vs xAI" --competitors-plan '{...per-entity...}'
scripts/last30days.py main():
- Detect: topic has " vs " OR --competitors enabled
- If --competitors and no list/plan: emit LAW 7-style stderr with hosting-model instruction
- If --competitors with list or discovery: rewrite topic to vs form, continue
- Parse --competitors-plan JSON, map to entities
fanout.run_competitor_fanout (shared path):
- For each entity (main + peers):
- entity_config = dict(config) [deep copy to prevent leak]
- kwargs = _subrun_kwargs(entity, plan_entry) [explicit; no main-topic leak]
- If plan_entry missing a field AND backend available: auto_resolve() fill
- pipeline.run(topic=entity, **kwargs, internal_subrun=True)
- Parallel ThreadPoolExecutor
- Collect per-entity Reports
- Attach resolved targeting to each Report.artifacts["resolved"]
scripts/last30days.py after fanout:
- If --save-dir: save each entity's Report as {entity-slug}-raw.md
Each file includes its own single-row Resolved Entities block
- emit_comparison_output → render_comparison_multi (merged stdout)
Includes full N-row Resolved Entities block
```
## Implementation Units
- [ ] **Unit 1: vs-topic detection routes to fanout (not single-pipeline)**
**Goal:** A topic containing ` vs ` / ` versus ` triggers `fanout.run_competitor_fanout` with the parsed entities. Each entity runs a full `pipeline.run()`. Replace the current single-pipeline-with-comparison-plan behavior.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (main() — detect vs-topic, route to fanout)
- Modify: `scripts/lib/planner.py` (remove / bypass the `_should_force_deterministic_plan` special case for vs topics; vs topics no longer go through `plan_query` as a single comparison plan)
- Test: `tests/test_vs_mode_fanout.py` (new)
**Approach:**
- Parse the incoming topic: if it contains ` vs ` or ` versus ` (case-insensitive), split into entities (reuse `planner._comparison_entities`-style logic or move that utility into main()).
- When vs-entities are detected, route to the same fanout branch `--competitors` uses today. The entity list comes from the topic string; no discovery step needed.
- Each entity runs `pipeline.run()` with its own plan (either from `--competitors-plan[entity]` or from the engine's per-entity fallback path).
- For back-compat, if the user passes both a vs-topic AND `--plan`, honor `--plan` for the main (first) entity and use per-entity defaults for peers unless `--competitors-plan` is also provided.
**Execution note:** Start with an integration test that runs `"A vs B"` via mock mode and asserts fanout was called with two entities + two pipeline.run calls.
**Patterns to follow:**
- 3.0.11 fanout wiring in `scripts/last30days.py`'s `--competitors` branch.
- `planner._comparison_entities` for the split logic.
**Test scenarios:**
- Happy path: topic `"A vs B"` → two pipeline.run calls, two Reports returned, merged render.
- Happy path: topic `"A vs B vs C"` → three pipeline.run calls.
- Happy path: topic `"A versus B"` → matches the same regex, two pipelines.
- Edge case: topic `"OpenAI vs"` (trailing empty entity) → treated as single-entity `"OpenAI"`, not vs mode.
- Edge case: topic contains "vs." (dot, no trailing space) → existing regex tolerates it; verify.
- Edge case: topic `"A vs B"` plus `--plan` → plan applies to first entity only, peers use per-entity defaults.
- Integration: full vs-mode run end-to-end in mock mode; verify rendered output, stderr has one `[Competitors] Comparing: A vs B vs ...` line.
**Verification:**
- Test assertions pass.
- Mock-mode smoke of `/last30days "OpenAI vs Anthropic"` shows fanout invocation, per-entity Reports, merged comparison output.
- [ ] **Unit 2: `--competitors-plan` JSON flag + `_subrun_kwargs` helper + override-leak fix**
**Goal:** New JSON flag threads per-entity targeting into each sub-run's `pipeline.run()`. A `_subrun_kwargs(entity, plan_entry)` helper is the single source of truth for per-entity kwargs, eliminating override-leak.
**Requirements:** R3, R6
**Dependencies:** None (can land alongside or before Unit 1)
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner` + `_subrun_kwargs` helper)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; the competitor_runner contract is unchanged)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
- Test: `tests/test_competitor_subrun_isolation.py` (new, regression)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON or file path (mirror `--plan`).
- Validation: top-level dict; each value is a dict; unknown fields log warnings; malformed input exits 2. Case-insensitive entity matching.
- Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`.
- Build `_subrun_kwargs(entity, plan_entry)` — returns an explicit dict with every per-entity flag. No closure-default fallthrough. This is the leak fix.
- `_competitor_runner(entity)`:
1. Get `plan_entry` from `--competitors-plan` if present.
2. Build base kwargs with `_subrun_kwargs(entity, plan_entry)`.
3. Fill missing fields via `resolve.auto_resolve(entity, entity_config)` only if backend is configured (3.0.12 fallback path).
4. Call `pipeline.run(topic=entity, internal_subrun=True, **kwargs)`.
5. Attach `resolved` dict to `report.artifacts`.
- Verify no per-entity flag from main() leaks via closure. The helper is the only source of per-entity values.
**Execution note:** Test-first for the override-leak regression. Use the Kanye 2026-04-22 receipt as the failing test input (main `--subreddits=Kanye,hiphopheads` + `--competitors-list "Drake"` → assert Drake's pipeline.run receives `subreddits=None`).
**Patterns to follow:**
- `--plan` parsing block in `scripts/last30days.py`.
- 3.0.12's `entity_config = dict(config)` deep-copy pattern.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake":{"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's pipeline.run receives `x_handle="Drake"`, `subreddits=["Drizzy"]`. No auto_resolve call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered skip auto_resolve; third falls back.
- Happy path: plan file path accepted like `--plan`.
- Happy path: case-insensitive entity match.
- Edge case: unknown fields → warn, ignore.
- Edge case: plan entry for entity not in list → warn, ignore.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list → exit 2.
- Regression (leak): main `--subreddits=A,B` + `--competitors-list "X"` + no plan → X's pipeline.run gets `subreddits=None`.
- Regression (leak): same for `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`.
- Regression (leak): main `--x-handle=kanye` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanye"`.
**Verification:**
- All regression tests pass.
- Smoke run (mock mode + plan): stderr shows per-entity `[Competitors] {entity}: x=... subs=...` line; no leak from main topic's flags.
- [ ] **Unit 3: Per-entity save files**
**Goal:** When `--save-dir` is set in a vs-mode or `--competitors` run, each entity's sub-run saves its own `{entity-slug}-raw.md` file — same format as a single-entity run would produce.
**Requirements:** R4, R5
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py` (`save_output` iteration after fanout)
- Modify: `scripts/lib/render.py` (`render_full` includes single-row Resolved Entities block when that entity's `artifacts["resolved"]` is present)
- Test: `tests/test_save_raw_per_entity.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]` (or equivalent). For each `(entity, entity_report)`:
- Call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)`.
- Uses entity's `slugify(entity)` for the filename. Same pattern a single-entity run uses.
- Each saved file invokes `render_full` (or the save-variant). `render_full` now checks for `report.artifacts["resolved"]` and prepends a single-row Resolved Entities block.
- Stderr logs one `[last30days] Saved output to <path>` line per entity.
- Single-entity runs unchanged (no extra files, render_full unchanged for them).
**Patterns to follow:**
- Existing `save_output` invocation in main() for single-entity runs.
- `slugify(topic)` for filename.
- 3.0.12's `_render_resolved_entities_block` (reused, single-row mode).
**Test scenarios:**
- Happy path: `/last30days "A vs B vs C" --save-dir=/tmp/x``/tmp/x/a-raw.md`, `/tmp/x/b-raw.md`, `/tmp/x/c-raw.md` exist.
- Happy path: `--competitors-list "Drake,Kendrick" --save-dir=/tmp/x` on topic Kanye → three files: `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`.
- Happy path: each file includes a single-row Resolved Entities block for its entity.
- Happy path: single-entity run with `--save-dir` → one file, no Resolved block (unchanged).
- Edge case: `--save-suffix=v3` → all N files get the suffix.
- Edge case: one entity sub-run failed → its file is NOT saved; the others are.
- Integration: `ls {save-dir}/*-raw.md` returns N files after a vs-mode run.
**Verification:**
- Test assertions pass.
- Manual vs-mode smoke saves N files.
- [ ] **Unit 4: LAW 7-style stderr reframe + footer-nudge suppression**
**Goal:** The `--competitors`-with-no-backend stderr tells the hosting model to do Step 0.55 per entity and pass `--competitors-plan`. The BRAVE/SERPER footer nudge is suppressed when `--plan` or `--competitors-plan` is present.
**Requirements:** R7, R8
**Dependencies:** Unit 2 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the `[Competitors] --competitors requires...` stderr block)
- Modify: `scripts/lib/quality_nudge.py` (or wherever footer nudge emits; verify during implementation)
- Test: `tests/test_competitors_no_backend_message.py` (new)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent with WebSearch), the recommended path: (a) discover N peers via WebSearch, (b) run Step 0.55 for main + each peer, (c) re-invoke as `/last30days 'topic vs peer1 vs peer2' --competitors-plan '{...}'`. See SKILL.md 'Competitor mode'."
2. "Headless / cron path: set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list 'A,B,C'` skips discovery but does not pre-resolve peers."
- Suppress footer nudge when `external_plan` OR `competitors_plan` was passed.
**Test scenarios:**
- Happy path: `--competitors` with no backend, no list, no plan → stderr leads with "If you are the hosting reasoning model" and references `--competitors-plan` before naming API keys.
- Happy path: `--plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-list` only (no plan, no backend) → footer nudge still fires (hosting model didn't fully engage).
- Happy path: no `--competitors`, no `--plan` → footer nudge unchanged.
**Verification:**
- Tests pass.
- [ ] **Unit 5: Polymarket disambiguation guard**
**Goal:** `--polymarket-keywords "kw1,kw2"` filters market matches; auto-skip Polymarket on single-token-ambiguous topics without override.
**Requirements:** R9
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"`. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip: if topic is one token AND matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords`, skip Polymarket with stderr note.
- SKILL.md update (small): mention `--polymarket-keywords` in Step 0.55 instructions for ambiguous topics.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr note.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs, filtered.
- Happy path: topic "OpenAI" → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke → Polymarket footer absent or filtered.
- [ ] **Unit 6: SKILL.md rewrite — vs mode is the canonical path, `--competitors` is a shortcut**
**Goal:** SKILL.md documents the unified architecture. vs mode runs N full passes. `--competitors` is a SKILL.md-level shortcut that discovers 2 peers and invokes vs mode with `--competitors-plan`.
**Requirements:** R1, R2, R10 (surfaces them)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md` (§551 "If QUERY_TYPE = COMPARISON" rewrite; Competitor mode subsection rewrite)
- Modify: `README.md` (one-line example)
**Approach:**
- Rewrite §551 to describe the N-pass architecture: "When the user asks 'X vs Y' (or 'X vs Y vs Z'), run Step 0.55 per entity, then invoke the engine. The engine fans out N full pipelines in parallel. Each entity gets its own single-entity-grade coverage. Wall clock is close to a single run."
- Remove the "ONE research pass with a comparison-optimized plan that replaces the old 3-pass approach" language.
- Add a `--competitors-plan` JSON example.
- Rewrite the Competitor mode subsection: "`--competitors` is a shortcut. The hosting model: (1) runs WebSearch to discover N=2 peers, (2) runs Step 0.55 for main + each peer, (3) rewrites topic to `'main vs peer1 vs peer2'`, (4) invokes engine with `--competitors-plan '{...}'`. Engine flag `--competitors` and `--competitors-list` remain for headless fallback."
- Cross-reference §679 (per-entity Step 0.55 protocol).
- Warning: a thin `## Resolved Entities` block (dashes for any entity) means the hosting model skipped Step 0.55 for that one.
**Patterns to follow:**
- Existing §679 per-entity Step 0.55 protocol for tone.
- 3.0.12 Competitor mode prose for terseness.
**Test scenarios:**
- Test expectation: none — documentation. Verification is dogfood.
**Verification:**
- `/last30days "OpenAI vs Anthropic vs xAI"` in a fresh Claude Code window produces 3 save files with populated Resolved blocks and non-dash per-entity targeting.
- `/last30days OpenAI --competitors` produces same after discovery step.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R10
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG: group the changes. "Changed: vs mode now runs N full passes in parallel, one per entity — reverting the one-pass optimization to restore per-entity depth. Added: --competitors-plan JSON for per-entity Step 0.55 targeting (applies to vs mode and --competitors). Changed: --competitors is now a SKILL.md shortcut for vs-with-discovery. Added: per-entity *-raw.md save files. Fixed: override-leak from main to peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when --plan / --competitors-plan present. Added: --polymarket-keywords + auto-skip for ambiguous topics."
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files.
## System-Wide Impact
- **Interaction graph:** vs-mode and `--competitors` share one orchestrator (`fanout.run_competitor_fanout`). `_subrun_kwargs` is the single source of per-entity kwargs. Save loop iterates per entity.
- **Error propagation:** Per-entity sub-run failure → logged, dropped, continue (3.0.11 behavior unchanged). `--competitors-plan` JSON parse errors exit 2 (same shape as `--plan`).
- **State lifecycle risks:** `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag (Unit 2 fix). No cross-entity context leak.
- **API surface parity:** `--competitors-plan` is additive. `--competitors`, `--competitors-list`, `--plan` unchanged. `--polymarket-keywords` additive. vs-mode keeps its topic-string surface.
- **Integration coverage:** New vs-mode-fanout integration test. New override-leak regression test. New plan-threading test. New nudge-suppression test. New per-entity-save test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. Single-entity render path unchanged. LAW 7 on the default path unchanged (still fires when a single-entity run lacks `--plan`).
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| vs-mode N-pass latency feels slower for users who remember the one-pass shortcut. | Parallel execution keeps wall-clock ~= max(per-entity-latency), not sum. `--quick` on a vs-topic still applies to each sub-run. CHANGELOG calls out the revert + parallelism. |
| API cost scales linearly with N (per source). | Default count 2 caps it. Hard max 6 on `--competitors`. vs-mode users opted into N entities explicitly. |
| Rivalry content ("A vs B" articles) missed in N-independent passes. | Deferred to separate task (head-to-head supplemental pass). Start shipping and observe whether this is actually a gap. |
| Hosting model skips `--competitors-plan` and uses `--competitors-list` only. | Unit 4 stderr reframe steers explicitly. SKILL.md Unit 6 makes the plan-path canonical. Thin Resolved block in output makes skipped-Step-0.55 visible. |
| Override-leak fix misses a subtle closure path. | Unit 2 is test-first with the Kanye receipt as the failing input. Regression test asserts every per-entity flag is None unless plan provides it. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md.
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG explicitly frames the vs-mode change as an architectural revert-with-parallelism, not a regression to the old serial N-pass.
## Sources & References
- Superseded plan: `docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded`
- Previous plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Initial plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts (Warriors, Seattle, Arizona Wildcats, Kanye West)
- SKILL.md §551 + §679 — the per-entity Step 0.55 protocol the hosting model uses for both paths
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/planner.py` vs-topic special-case, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -0,0 +1,90 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: comparison title says (/Last30Days) instead of (Last 30 Days)"
type: fix
status: active
date: 2026-04-22
---
# fix: comparison title says (/Last30Days) instead of (Last 30 Days)
## Overview
User feedback 2026-04-22 on the 3.0.13 release runs (Kanye vs Drake, Mercer Island, Figma): the comparison title currently reads `# Kanye West vs Drake: What the Community Says (Last 30 Days)`. It should read `# Kanye West vs Drake: What the Community Says (/Last30Days)` — attributing the output to the slash command rather than describing the date range generically.
Single-line change in SKILL.md, three occurrences. No code change.
## Requirements Trace
- R1. Comparison title pattern in SKILL.md changes from `(Last 30 Days)` to `(/Last30Days)` so synthesis outputs read `... What the Community Says (/Last30Days)`.
- R2. Both the rule statement (line 113) and the COMPARISON-exception statement (line 131) and the synthesis template example (line 1208) all use the new suffix.
- R3. Version bumps to 3.0.14, CHANGELOG entry, sync, hot-copy. Public cache picks up the new title pattern.
## Scope Boundaries
- No changes to the single-entity output title (no `(/Last30Days)` suffix there — only comparison topics carry it).
- No changes to engine code. Pure SKILL.md content.
- No changes to anything else surfaced in the test runs.
## Key Technical Decisions
- **Replace all three occurrences of the suffix string in one pass.** They are identical strings; changing one without the others would cause synthesis-time confusion when the model reaches a different reference.
- **Ship as 3.0.14, not 3.0.13.x.** Patch-level bump matches the small scope and keeps the release log clean.
## Implementation Units
- [ ] **Unit 1: Replace `(Last 30 Days)` → `(/Last30Days)` in SKILL.md**
**Goal:** All three SKILL.md references to the comparison title use the new suffix.
**Requirements:** R1, R2
**Files:**
- Modify: `SKILL.md`
**Approach:**
- `replace_all` swap of `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Three occurrences, no other strings overlap.
**Test scenarios:**
- Test expectation: none — pure documentation. Verification by inspection + dogfood run.
**Verification:**
- `grep -c "What the Community Says (/Last30Days)" SKILL.md` returns 3.
- `grep -c "What the Community Says (Last 30 Days)" SKILL.md` returns 0.
- [ ] **Unit 2: Version 3.0.14 + CHANGELOG + sync + hot-copy**
**Goal:** Ship 3.0.14 to all local targets.
**Requirements:** R3
**Dependencies:** Unit 1
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.14/`
**Approach:**
- CHANGELOG: "Changed: comparison-mode title attribution — `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range."
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.14.
- Hot-copy contains the updated SKILL.md.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model has the old title pattern memorized from a prior run and re-emits `(Last 30 Days)`. | SKILL.md is read top-to-bottom each invocation. STEP 0 canonical-path self-check (3.0.12) ensures the model loads the new SKILL.md, not the marketplace stale copy. |
## Sources & References
- 2026-04-22 dogfood runs (Kanye West vs Drake, Mercer Island --competitors, Figma --competitors)
- Related code: `SKILL.md` lines 113, 131, 1208
@@ -0,0 +1,82 @@
---
title: Search-quality eval is manual by default, not a CI gate on every PR
date: 2026-05-10
category: docs/solutions/architecture
module: skills/last30days/scripts/evaluate_search_quality.py
problem_type: design_decision
component: ci_policy
severity: low
applies_when:
- a contributor proposes wiring search-quality eval into PR CI
- a change affects retrieval, ranking, grounding, or synthesis quality and a reviewer asks "why aren't we testing this in CI?"
- someone is deciding whether a new evaluator-style script belongs in the default CI workflow
related_components:
- search_quality_evaluation
- ci_workflow
- llm_judging
tags:
- ci-policy
- eval
- design-decision
- cost-vs-signal
- non-determinism
- manual-gates
---
# Search-quality eval is manual by default, not a CI gate on every PR
## Context
`skills/last30days/scripts/evaluate_search_quality.py` compares a baseline revision against a candidate revision across a fixed pool of reviewer topics. It produces two flavors of metrics: deterministic overlap (Jaccard, retention) and LLM-judged quality scores. The natural impulse on seeing an evaluator script is to wire it into CI on every PR — "regression catcher, run it automatically." We deliberately don't.
Three properties of this particular evaluator make CI-on-every-PR the wrong default:
1. **Live API access.** The candidate revision typically needs the engine to actually run, which means real ScrapeCreators calls, real reddit fetches, real YouTube searches. CI runs would either need production credentials or a record/replay fixture set that drifts almost immediately as external APIs change shape.
2. **Cost and latency.** A full eval pass runs the pipeline N times across reviewer topics. Multiplied by every PR (including doc-only PRs), the spend is meaningful and the wall-clock pushes CI from ~30s to many minutes.
3. **Non-determinism in the judging path.** The LLM-judged metrics are valuable for review but depend on judge-model behavior on a given day. A flaky eval that fails 1 PR in 20 because the judge re-scored an item differently is a worse CI signal than no eval at all — it teaches contributors to retry rather than read the result.
The deterministic overlap metrics are useful regression signals but they are not the same as user-facing correctness. A change that improves overlap can degrade synthesis quality; a change that drops overlap can be a deliberate improvement. So even the deterministic side isn't safe to auto-fail on.
## Guidance
### 1. Keep search-quality eval available, just not automatic
The script stays runnable by maintainers and contributors. The pattern is:
```bash
LAST30DAYS_PYTHON=python3.13 \
python3 skills/last30days/scripts/evaluate_search_quality.py \
--baseline main --candidate HEAD
```
Reviewers can request a manual eval run when a PR is in the retrieval/ranking/synthesis path and the risk warrants it. Contributors can run it locally before submitting if they want signal upfront.
### 2. Standard PR CI gates remain deterministic and contract-shaped
`pytest` (offline-safe), plugin-contract checks, version-consistency contracts, ruff/lint. Anything that returns the same answer twice for the same input. Quality-of-output assessment lives outside that loop.
### 3. The middle ground is `workflow_dispatch`, not auto-PR-gating
If maintainers want a GitHub-triggered eval that doesn't make every PR pay the live-API cost, the right shape is a manually-dispatched workflow (or a label-triggered one) — not a `pull_request:` workflow that runs unconditionally. That keeps the cost knob in human hands.
### 4. Revisit if the eval can ever be made offline-deterministic
The blocker is the live-API + non-determinism combination. If a future iteration of the script can compute meaningful Jaccard/retention metrics against static fixtures (no live API calls, no LLM judging), the decision flips and it becomes a candidate for default CI. The decision below tracks that condition; revisit when it's met.
## What this means in practice
- Don't merge PRs that wire `evaluate_search_quality.py` into the default `validate.yml` workflow.
- Do merge PRs that add `workflow_dispatch` triggers or label-gated runs.
- When reviewing a retrieval/ranking change, request a manual eval if the diff suggests it could regress quality — don't expect CI to catch it.
## Links
- `skills/last30days/scripts/evaluate_search_quality.py` — the evaluator script
- `docs/search-quality-eval.md` — user-facing usage documentation
- `.github/workflows/validate.yml` — the default CI workflow (deterministic gates only)
---
*Adapted from a draft ADR proposed by @hnshah in [#374](https://github.com/mvanhorn/last30days-skill/pull/374), restructured into the `docs/solutions/` convention. The original ADR text correctly identified the constraint; this version adds the "why workflow_dispatch is the middle ground" framing and the revisit-condition.*
@@ -0,0 +1,219 @@
---
title: Release-time consistency tests cause cascade CI failures across all open PRs
date: 2026-05-16
category: docs/solutions/workflow-issues
module: ci-release-engineering
problem_type: workflow_issue
component: testing_framework
severity: high
applies_when:
- a test asserts consistency between two release-time artifacts (e.g., SKILL.md version and a hardcoded pin in a shell script)
- one artifact is updated as part of a version bump and the other requires a manual lockstep update
- multiple long-lived PRs are open simultaneously against the same base branch
symptoms:
- every open PR's CI fails after a version bump even though the PRs are unrelated to versioning
- the failing test references a stale hardcoded value that was not updated alongside the bumped version
- PR authors must rebase and manually fix an artifact they did not touch
root_cause: missing_workflow_step
resolution_type: code_fix
related_components:
- development_workflow
- documentation
tags:
- ci
- release-engineering
- consistency-test
- version-pin
- cascade-failure
- test-design
- workflow
---
# Release-time consistency tests cause cascade CI failures across all open PRs
## Context
A `tests/test_version_consistency.py::test_sync_cache_path_uses_skill_version` test was added to enforce that the version string embedded in `skills/last30days/scripts/sync.sh` (a hardcoded plugin-cache path segment) matched the version frontmatter in `skills/last30days/SKILL.md`. The intention was sound: the cache path had to stay in lockstep with the skill version or the sync would silently pull stale files.
The test worked as designed until a release shipped. At that point it turned into a cascade-failure machine:
1. A release PR bumps `SKILL.md` version (e.g., 3.2.0 → 3.2.1) **and** bumps the `sync.sh` pin. That PR's CI is green.
2. The release PR merges to `main`.
3. Every PR that was open at merge time was branched from pre-release `main`. Those PRs have `SKILL.md` 3.2.1 (inherited via merge-base with `main`) but their branch never touched `sync.sh`.
4. CI for those PRs runs the consistency test against the new `main``SKILL.md` says 3.2.1, `sync.sh` still says 3.2.0 — and fails.
5. All open PRs are now red simultaneously, with a failure that has nothing to do with their changes.
This affected at least five PRs during the 2026-05-13 to 2026-05-15 window: PR #400 (caught during rebase, required a manual pin bump), PRs #390 and #392 (OpenClaw `SCRAPECREATORS_API_KEY` fix, both stalled for the same stale-pin reason), and at least two others. A follow-up hotfix PR (#397`fix(sync): bump cache target to 3.2.1 to match SKILL.md`) was required just to unblock the queue.
The permanent fix was PR #405: delete `sync.sh` entirely (the install workflow made it redundant) and drop `test_sync_cache_path_uses_skill_version`. Once both were gone, no version-consistency cascade was possible.
## Guidance
### 1. Don't write consistency tests that read two files and assert one matches a substring derived from the other
This pattern looks safe but is not:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # reads SKILL.md
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text, # asserts sync.sh contains that string
)
```
It encodes the assumption that both files are always updated together, in the same commit, on the same branch. That assumption breaks the moment two files have independent lifecycle owners — a versioned manifest and a deployment script are archetypal examples.
### 2. If the values genuinely need to stay in sync, derive one from the other at runtime
Remove the hardcoded pin from `sync.sh` and compute it:
```bash
# sync.sh — derive version from SKILL.md at runtime, no pin to maintain
SKILL_VERSION=$(grep -m1 '^version:' "$(dirname "$0")/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
CACHE_PATH="last30days-skill/last30days/${SKILL_VERSION}"
```
Now there is only one source of truth (`SKILL.md`). The test that asserted they matched becomes vacuous and should be deleted. If `SKILL.md` is wrong, the sync itself will fail loudly — which is better feedback than a CI gate on a different PR.
### 3. If two values must stay independent for legitimate reasons, update them together and make the test self-skip if either source is missing
If separate versioning is genuinely required (e.g., SKILL.md versions for harness consumers, sync.sh versions a private artifact store with its own cadence), update both in the same PR — never staggered — and write the test to self-skip rather than error when either file is absent:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_sh = SKILL_ROOT / "scripts" / "sync.sh"
if not sync_sh.exists():
self.skipTest("sync.sh not present; skipping pin consistency check")
sync_text = sync_sh.read_text(encoding="utf-8")
version = _skill_version()
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text,
)
```
Self-skipping means deleting the file is a non-event in CI — no cascading red, no hotfix PR to the queue.
### 4. Run consistency tests against the merge-base diff, not main
If you keep a two-file consistency test, scope it so it only fails when the PR itself modifies one of the two files but not the other. A GitHub Actions step can do this:
```yaml
- name: Check sync.sh version pin consistency
run: |
BASE=$(git merge-base HEAD origin/main)
SKILL_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'SKILL\.md' || true)
SYNC_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'sync\.sh' || true)
if [ "$SKILL_CHANGED" -gt 0 ] && [ "$SYNC_CHANGED" -eq 0 ]; then
echo "SKILL.md version bumped but sync.sh pin was not updated"
exit 1
fi
```
This only fires when your PR touched `SKILL.md` and left `sync.sh` alone — never because a release merged to `main` after you branched.
### 5. Ask whether you actually need this test
If the values are wrong, downstream tooling will fail loudly: the sync will fetch the wrong artifact, installs will break, or the harness will reject the version. A test that exists only to catch a human-bookkeeping error at release time adds cascade-fail risk without offering a meaningfully earlier signal. Weigh that cost before adding any two-file consistency gate.
## Why This Matters
The damage from a stale-pin consistency test is asymmetric. It:
- Fails on every open PR simultaneously the moment a release lands on `main` — not just the PR that forgot to update the pin.
- Produces a failure message that points at a line in a test file with no obvious relationship to the PR's actual changes.
- Requires either a hotfix PR (touching a file the failing PRs have no business touching) or a manual rebase of every affected branch.
- Blocks work that has already been reviewed and approved.
In this repo the effect was measurable: at least five PRs stalled across a two-day window, one hotfix PR was shipped just to unblock the queue, and multiple authors spent time debugging a failure completely unrelated to their changes.
The broader principle is that tests which gate on *bookkeeping consistency between files* impose their maintenance cost on every contributor, every time, even when those contributors did nothing wrong. That cost compounds with team size and release cadence.
## When to Apply
Apply this guidance whenever you find yourself:
- Writing a test that reads two files and asserts that a string in one matches a value derived from the other.
- Adding a CI step labeled "consistency check," "sync check," or "pin check" where the check compares a hardcoded value against a computed one from a separate file.
- Working in a repo where a versioned manifest (e.g., `SKILL.md`, `package.json`, `pyproject.toml`) and a deployment artifact (e.g., a shell script, a Dockerfile, a Helm values file) are both maintained by hand.
- Reviewing a PR that touches only one of two "paired" files and fails a consistency test for the other.
It does *not* apply to tests that read a single source of truth and validate its internal structure (e.g., asserting that `SKILL.md`'s frontmatter version is double-quoted, or that `package.json`'s `version` field is a valid semver string). Those tests have one file and one assertion; they cannot cascade across branches.
## Examples
### Before — the pattern that caused the cascade
Original `tests/test_version_consistency.py` (deleted in commit `9fb19ea`):
```python
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days"
def _skill_version() -> str:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
if not match:
raise AssertionError("SKILL.md version frontmatter not found")
return match.group(1)
class TestVersionConsistency(unittest.TestCase):
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # source 1: SKILL.md frontmatter
self.assertIn( # assertion: sync.sh must contain
f'last30days-skill/last30days/{version}"',
sync_text, # source 2: hardcoded string in sync.sh
)
```
`sync.sh` contained a line like:
```bash
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/3.2.0"
```
When SKILL.md bumped to `3.2.1` in a release PR, `sync.sh` was updated in the same PR and CI stayed green. But every PR branched before that release still had `sync.sh` at `3.2.0`. Their CI failed immediately, with an assertion error pointing at the test, not at the release PR.
### After — what we did: delete both
PR #405 deleted `sync.sh` (the install workflow replaced it) and dropped `test_sync_cache_path_uses_skill_version` in the same change. No consistency gate, no pin to maintain, no cascade possible.
### After — what we could have done instead: derive at runtime
If `sync.sh` had still been needed, the right fix would have been to remove the hardcoded version from the script and derive it from `SKILL.md`:
```bash
#!/usr/bin/env bash
# sync.sh — no hardcoded version; reads SKILL.md as single source of truth
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SKILL_VERSION=$(grep -m1 '^version:' "${SCRIPT_DIR}/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
if [ -z "$SKILL_VERSION" ]; then
echo "error: could not parse version from SKILL.md" >&2
exit 1
fi
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/${SKILL_VERSION}"
# ... rest of sync logic
```
With this in place, `test_sync_cache_path_uses_skill_version` has no reason to exist — there is nothing to assert. Delete it. If the version parsing breaks, `sync.sh` itself exits non-zero with a clear message.
## Related
- **PR #397** (merged) — `fix(sync): bump cache target to 3.2.1 to match SKILL.md`. The hotfix that unblocked the cascade temporarily by bumping the pin.
- **PR #400** (merged) — caught the same cascade during rebase; had to bump the pin to clear CI.
- **PR #390** (closed) and **PR #392** (rebased + merged) — OpenClaw `SCRAPECREATORS_API_KEY` fix; both blocked by the cascade until rebased onto post-#405 main.
- **PR #405** (merged) — the permanent fix: deleted `sync.sh` + `test_sync_cache_path_uses_skill_version` together.
- **PR #412** (merged) — adjacent work that consolidated SKILL.md version parsing into `lib/skill_meta.py`, reducing future drift risk by giving the version field one canonical reader.
+4
View File
@@ -0,0 +1,4 @@
{
"triggerOnUpdates": true,
"statusCheck": true
}
+15 -2
View File
@@ -97,7 +97,20 @@ if [[ -n "$HAS_BSKY" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1)) SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then if [[ -n "$HAS_SCRAPECREATORS" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 3)) # Reddit comments + TikTok + Instagram # Start with Reddit comments + TikTok + Instagram, subtract any in EXCLUDE_SOURCES.
# Normalise EXCLUDED (lowercase + collapse whitespace around commas + strip outer
# whitespace) so the matching mirrors pipeline.py's .strip().lower() parsing.
SC_ADD=3
EXCLUDED="${ENV_EXCLUDE_SOURCES:-${EXCLUDE_SOURCES:-}}"
EXCLUDED_NORM=$(printf '%s' "$EXCLUDED" | tr '[:upper:]' '[:lower:]' \
| sed -E 's/[[:space:]]*,[[:space:]]*/,/g; s/^[[:space:]]+//; s/[[:space:]]+$//')
if [[ ",$EXCLUDED_NORM," == *",tiktok,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
if [[ ",$EXCLUDED_NORM," == *",instagram,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
SOURCE_COUNT=$((SOURCE_COUNT + SC_ADD))
fi fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then if [[ -n "$HAS_SCRAPECREATORS" ]]; then
@@ -107,6 +120,6 @@ else
# Setup done but missing ScrapeCreators — recommend it # Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active." echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram." echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 10,000 free API calls, no credit card — scrapecreators.com" echo " 100 free credits, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider." echo " last30days has no affiliation with any API provider."
fi fi
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 MiB

+6 -9
View File
@@ -1,16 +1,14 @@
[project] [project]
name = "last30days-skill" name = "last30days-skill"
version = "3.0.0" version = "3.2.4"
description = "Multi-source last-30-days research skill" description = "Multi-source last-30-days research skill"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = []
"requests>=2.32,<3",
]
[dependency-groups] [dependency-groups]
dev = [ dev = [
"pytest>=9,<10", "pytest>=9.0.3,<10",
"pytest-cov>=7,<8", "pytest-cov>=7,<8",
] ]
@@ -24,9 +22,9 @@ addopts = [
[tool.coverage.run] [tool.coverage.run]
branch = true branch = true
source = ["scripts", "tests"] source = ["skills/last30days/scripts", "tests"]
omit = [ omit = [
"scripts/lib/vendor/*", "skills/last30days/scripts/lib/vendor/*",
"dist/*", "dist/*",
] ]
@@ -34,7 +32,6 @@ omit = [
skip_empty = true skip_empty = true
show_missing = true show_missing = true
omit = [ omit = [
"scripts/lib/vendor/*", "skills/last30days/scripts/lib/vendor/*",
"dist/*", "dist/*",
] ]
+1 -1
View File
@@ -58,7 +58,7 @@ OpenClaw:
clawhub install last30days-official clawhub install last30days-official
``` ```
OpenAI Codex CLI: run `codex` from a checkout of this repo and v3's skill at `.agents/skills/last30days/SKILL.md` will be discovered automatically. Or copy `SKILL.md` to `~/.agents/skills/last30days/SKILL.md` for a global install. OpenAI Codex CLI: install the repo as a local Codex marketplace/plugin. The plugin manifest lives at `.codex-plugin/plugin.json`, and the canonical skill payload is `skills/last30days/SKILL.md`.
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds. Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
-433
View File
@@ -1,433 +0,0 @@
#!/usr/bin/env python3
# ruff: noqa: E402
"""last30days v3.0.0 CLI."""
from __future__ import annotations
import argparse
import atexit
import json
import os
import re
import signal
import sys
import threading
from pathlib import Path
MIN_PYTHON = (3, 12)
def ensure_supported_python(version_info: tuple[int, int, int] | object | None = None) -> None:
if version_info is None:
version_info = sys.version_info
major, minor, micro = tuple(version_info[:3])
if (major, minor) >= MIN_PYTHON:
return
sys.stderr.write(
"last30days v3 requires Python 3.12+.\n"
f"Detected Python {major}.{minor}.{micro}.\n"
"Install and use python3.12 or python3.13, then rerun this command.\n"
)
raise SystemExit(1)
ensure_supported_python()
if os.name == "nt":
for stream in (sys.stdout, sys.stderr):
if hasattr(stream, "reconfigure"):
stream.reconfigure(encoding="utf-8", errors="replace")
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
from lib import env, pipeline, render, schema, ui
_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()
def register_child_pid(pid: int) -> None:
with _child_pids_lock:
_child_pids.add(pid)
def unregister_child_pid(pid: int) -> None:
with _child_pids_lock:
_child_pids.discard(pid)
def _cleanup_children() -> None:
with _child_pids_lock:
pids = list(_child_pids)
for pid in pids:
try:
os.killpg(os.getpgid(pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
continue
atexit.register(_cleanup_children)
def parse_search_flag(raw: str) -> list[str]:
sources = []
for source in raw.split(","):
source = source.strip().lower()
if not source:
continue
normalized = pipeline.SEARCH_ALIAS.get(source, source)
if normalized not in pipeline.MOCK_AVAILABLE_SOURCES:
raise SystemExit(f"Unknown search source: {source}")
if normalized not in sources:
sources.append(normalized)
if not sources:
raise SystemExit("--search requires at least one source.")
return sources
def slugify(value: str) -> str:
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
return slug or "last30days"
def save_output(report: schema.Report, emit: str, save_dir: str, suffix: str = "") -> Path:
from datetime import datetime
path = Path(save_dir).expanduser().resolve()
path.mkdir(parents=True, exist_ok=True)
slug = slugify(report.topic)
extension = "json" if emit == "json" else "md"
suffix_part = f"-{suffix}" if suffix else ""
out_path = path / f"{slug}-raw{suffix_part}.{extension}"
if out_path.exists():
out_path = path / f"{slug}-raw{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
# Always save the FULL dump to disk (all items, all sources, transcripts).
# Claude sees compact clusters via --emit=compact on stdout.
# The saved file is the complete debug artifact.
if emit == "json":
content = emit_output(report, emit)
else:
content = render.render_full(report)
out_path.write_text(content, encoding="utf-8")
return out_path
def emit_output(report: schema.Report, emit: str, fun_level: str = "medium", save_path: str | None = None) -> str:
if emit == "json":
return json.dumps(schema.to_dict(report), indent=2, sort_keys=True)
if emit in {"compact", "md"}:
return render.render_compact(report, fun_level=fun_level, save_path=save_path)
if emit == "context":
return render.render_context(report)
raise SystemExit(f"Unsupported emit mode: {emit}")
def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str) -> str:
"""Compute the user-friendly save path string that will be shown in the footer.
Uses ~ for the home directory so the footer reads "~/Documents/Last30Days/slug-raw.md"
instead of an absolute machine-local path.
"""
from pathlib import Path as _Path
path = _Path(save_dir).expanduser().resolve()
slug = slugify(topic)
extension = "json" if emit == "json" else "md"
suffix_part = f"-{suffix}" if suffix else ""
raw = path / f"{slug}-raw{suffix_part}.{extension}"
try:
home = _Path.home().resolve()
relative = raw.relative_to(home)
return f"~/{relative}"
except ValueError:
return str(raw)
def persist_report(report: schema.Report) -> dict[str, int]:
import store
store.init_db()
topic_row = store.add_topic(report.topic)
topic_id = topic_row["id"]
source_mode = ",".join(sorted(report.items_by_source)) or "v3"
run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
try:
findings = store.findings_from_report(report)
counts = store.store_findings(run_id, topic_id, findings)
store.update_run(
run_id,
status="completed",
findings_new=counts["new"],
findings_updated=counts["updated"],
)
return counts
except Exception as exc:
store.update_run(run_id, status="failed", error_message=str(exc)[:500])
raise
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Research a topic across live social, market, and grounded web sources.")
parser.add_argument("topic", nargs="*", help="Research topic")
parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md"])
parser.add_argument("--search", help="Comma-separated source list")
parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
parser.add_argument("--debug", action="store_true", help="Enable HTTP debug logging")
parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
parser.add_argument("--diagnose", action="store_true", help="Print provider and source availability")
parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
parser.add_argument("--web-backend", default="auto",
choices=["auto", "brave", "exa", "serper", "parallel", "none"],
help="Web search backend (default: auto, tries Brave then Exa then Serper then Parallel)")
parser.add_argument("--deep-research", action="store_true",
help="Use Perplexity Deep Research (~$0.90/query) for in-depth analysis. Requires OPENROUTER_API_KEY.")
parser.add_argument("--plan", help="JSON query plan (skips internal LLM planner). Can be a JSON string or a file path.")
parser.add_argument("--save-suffix", help="Suffix for saved output filename (e.g., 'gemini' → kanye-west-raw-gemini.md)")
parser.add_argument("--subreddits", help="Comma-separated subreddit names to search (e.g., SaaS,Entrepreneur)")
parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
parser.add_argument(
"--days",
"--lookback-days",
dest="lookback_days",
type=int,
default=30,
help="Number of days to look back for research (default: 30, watchlist uses 90)",
)
parser.add_argument("--auto-resolve", action="store_true",
help="Use web search to discover subreddits/handles before planning (for platforms without WebSearch)")
parser.add_argument("--github-user", help="GitHub username for person-mode search (e.g., steipete)")
parser.add_argument("--github-repo", help="Comma-separated owner/repo for project-mode search (e.g., openclaw/openclaw,paperclipai/paperclip)")
return parser
def _missing_sources_for_promo(diag: dict[str, object]) -> str | None:
available = set(diag.get("available_sources") or [])
missing = []
if "reddit" not in available:
missing.append("reddit")
if "x" not in available:
missing.append("x")
if "grounding" not in available:
missing.append("web")
if not missing:
return None
if "reddit" in missing and "x" in missing:
return "both"
return missing[0]
def _show_runtime_ui(report: schema.Report, progress: ui.ProgressDisplay, diag: dict[str, object]) -> None:
counts = {source: len(items) for source, items in report.items_by_source.items()}
display_sources = list(
dict.fromkeys(
[
*report.query_plan.source_weights.keys(),
*report.items_by_source.keys(),
*report.errors_by_source.keys(),
]
)
)
progress.end_processing()
progress.show_complete(
source_counts=counts,
display_sources=display_sources,
)
promo = _missing_sources_for_promo(diag)
if promo:
progress.show_promo(promo, diag=diag)
def main() -> int:
parser = build_parser()
# Use parse_known_args so setup sub-flags (--device-auth, --github,
# --openclaw) pass through without argparse hard-exiting.
args, extra_argv = parser.parse_known_args()
if args.debug:
os.environ["LAST30DAYS_DEBUG"] = "1"
config = env.get_config()
# Handle setup subcommand
topic = " ".join(args.topic).strip()
if topic.lower() == "setup":
from lib import setup_wizard
if "--openclaw" in extra_argv:
results = setup_wizard.run_openclaw_setup(config)
print(json.dumps(results))
return 0
if "--github" in extra_argv:
results = setup_wizard.run_github_auth()
print(json.dumps(results))
return 0
if "--device-auth" in extra_argv:
results = setup_wizard.run_full_device_auth()
print(json.dumps(results))
return 0
sys.stderr.write("Running auto-setup...\n")
results = setup_wizard.run_auto_setup(config)
from_browser = "auto"
if results.get("cookies_found"):
first_browser = next(iter(results["cookies_found"].values()))
from_browser = first_browser
setup_wizard.write_setup_config(env.CONFIG_FILE, from_browser=from_browser)
results["env_written"] = True
sys.stderr.write(setup_wizard.get_setup_status_text(results) + "\n")
return 0
requested_sources = parse_search_flag(args.search) if args.search else None
diag = pipeline.diagnose(config, requested_sources)
if args.diagnose:
print(json.dumps(diag, indent=2, sort_keys=True))
return 0
if not topic:
parser.print_usage(sys.stderr)
return 2
if not os.environ.get("LAST30DAYS_SKIP_PREFLIGHT"):
from lib import preflight
refuse_msg = preflight.check_class_1_trap(topic)
if refuse_msg:
sys.stderr.write(refuse_msg)
return 2
progress = ui.ProgressDisplay(topic, show_banner=True)
progress.start_processing()
depth = "deep" if args.deep else "quick" if args.quick else "default"
try:
x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
# Parse external plan if provided via --plan flag
external_plan = None
if args.plan:
import json as _json
plan_str = args.plan
if os.path.isfile(plan_str):
plan_str = open(plan_str).read()
try:
external_plan = _json.loads(plan_str)
except _json.JSONDecodeError as exc:
sys.stderr.write(f"[Planner] Invalid --plan JSON: {exc}\n")
# Auto-resolve: use web search to discover subreddits/handles before planning.
# This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
# without WebSearch (OpenClaw, Codex, raw CLI).
if args.auto_resolve and not external_plan:
from lib import resolve
resolution = resolve.auto_resolve(topic, config)
if resolution.get("subreddits") and not subreddits:
subreddits = resolution["subreddits"]
sys.stderr.write(f"[AutoResolve] Subreddits: {', '.join(subreddits)}\n")
if resolution.get("x_handle") and not args.x_handle:
args.x_handle = resolution["x_handle"]
sys.stderr.write(f"[AutoResolve] X handle: @{args.x_handle}\n")
if resolution.get("github_user") and not args.github_user:
args.github_user = resolution["github_user"]
sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
if resolution.get("github_repos") and not args.github_repo:
args.github_repo = ",".join(resolution["github_repos"])
sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
if resolution.get("context"):
# Inject context into external_plan metadata for the planner to use
if not external_plan:
external_plan = None # planner will use its own, but with context
# Store context for the planner prompt injection
config["_auto_resolve_context"] = resolution["context"]
sys.stderr.write(f"[AutoResolve] Context: {resolution['context'][:80]}...\n")
github_user = args.github_user.lstrip("@").lower() if args.github_user else None
github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None
# --deep-research: auto-enable perplexity source and set deep flag
if args.deep_research:
if not config.get("OPENROUTER_API_KEY"):
print("Error: --deep-research requires OPENROUTER_API_KEY", file=sys.stderr)
sys.exit(1)
config["_deep_research"] = True
# Auto-enable perplexity in INCLUDE_SOURCES
include = config.get("INCLUDE_SOURCES") or ""
if "perplexity" not in include.lower():
config["INCLUDE_SOURCES"] = f"{include},perplexity" if include else "perplexity"
report = pipeline.run(
topic=topic,
config=config,
depth=depth,
requested_sources=requested_sources,
mock=args.mock,
x_handle=args.x_handle,
x_related=x_related,
web_backend=args.web_backend,
external_plan=external_plan,
subreddits=subreddits,
tiktok_hashtags=tiktok_hashtags,
tiktok_creators=tiktok_creators,
ig_creators=ig_creators,
lookback_days=args.lookback_days,
github_user=github_user,
github_repos=github_repos,
)
except Exception as exc:
progress.end_processing()
progress.show_error(str(exc))
raise
_show_runtime_ui(report, progress, diag)
if args.store:
counts = persist_report(report)
sys.stderr.write(
f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
)
sys.stderr.flush()
# Show quality nudge if applicable
try:
from lib import quality_nudge
quality = quality_nudge.compute_quality_score(config, {})
if quality.get("nudge_text"):
sys.stderr.write(f"\n{quality['nudge_text']}\n")
sys.stderr.flush()
except Exception:
pass
fun_level = config.get("FUN_LEVEL", "medium").lower()
footer_save_path = None
if args.save_dir:
footer_save_path = compute_save_path_display(
args.save_dir, report.topic, args.save_suffix or "", args.emit
)
# Signal to render_compact whether pre-research flags were supplied.
# Used to emit a Pre-Research Status warning when the model skipped
# Step 0.5 / 0.55 and invoked the engine bare on an eligible topic.
pre_research_flags_present = bool(
args.x_handle
or args.github_user
or args.subreddits
or args.plan
or args.auto_resolve
or args.tiktok_creators
or args.ig_creators
)
report.artifacts["pre_research_flags_present"] = pre_research_flags_present
rendered = emit_output(report, args.emit, fun_level=fun_level, save_path=footer_save_path)
if args.save_dir:
save_path = save_output(report, args.emit, args.save_dir, suffix=args.save_suffix or "")
sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
sys.stderr.flush()
print(rendered)
return 0
if __name__ == "__main__":
raise SystemExit(main())
-123
View File
@@ -1,123 +0,0 @@
#!/usr/bin/env bash
# sync.sh - Deploy last30days skill to all host locations
# Usage: bash scripts/sync.sh (run from repo root)
set -euo pipefail
SRC="$(cd "$(dirname "$0")/.." && pwd)"
echo "Source: $SRC"
COMMON_TARGETS=(
# Claude Code plugin cache: marketplace installs overwrite on update,
# but local development needs the cache kept in sync with the repo.
# Do NOT add ~/.claude/skills/last30days - it creates a duplicate
# /last30days-3 in the slash command menu alongside the plugin version.
"$HOME/.claude/plugins/cache/last30days-skill-private/last30days-3/3.0.1"
"$HOME/.claude/plugins/cache/last30days-skill-private/last30days-3-nogem/3.0.0-nogem"
"$HOME/.agents/skills/last30days"
"$HOME/.codex/skills/last30days"
)
OPENCLAW_TARGET="$HOME/.openclaw/skills/last30days"
sync_target() {
local target="$1"
local skill_md="$2"
echo ""
echo "--- Syncing to $target ---"
mkdir -p "$target/scripts/lib"
cp "$skill_md" "$target/SKILL.md"
rsync -a \
"$SRC/scripts/last30days.py" \
"$SRC/scripts/watchlist.py" \
"$SRC/scripts/briefing.py" \
"$SRC/scripts/store.py" \
"$target/scripts/"
rsync -a "$SRC/scripts/lib/"*.py "$target/scripts/lib/"
# The OpenClaw variant lives in the private repo only. Skip cleanly when
# running this script from the public repo where variants/open does not exist.
if [ -d "$SRC/variants/open" ]; then
mkdir -p "$target/variants/open/references"
rsync -a "$SRC/variants/open/" "$target/variants/open/"
fi
if [ -d "$SRC/scripts/lib/vendor" ]; then
rsync -a "$SRC/scripts/lib/vendor" "$target/scripts/lib/"
fi
if [ -d "$SRC/fixtures" ]; then
mkdir -p "$target/fixtures"
rsync -a "$SRC/fixtures/" "$target/fixtures/"
fi
mod_count=$(ls "$target/scripts/lib/"*.py 2>/dev/null | wc -l | tr -d ' ')
echo " Copied $mod_count modules"
if (
cd "$target/scripts" &&
python3 -c "import briefing, store, watchlist; from lib import youtube_yt, bird_x, render, ui; print(' Import check: OK')"
); then
true
else
echo " Import check FAILED"
fi
}
for t in "${COMMON_TARGETS[@]}"; do
sync_target "$t" "$SRC/SKILL.md"
done
# Hermes sync: deploy to Hermes skills directory if it exists
HERMES_TARGET="$HOME/.hermes/skills/research/last30days"
if [ -d "$HOME/.hermes/skills/research" ]; then
echo ""
echo "--- Syncing to Hermes ---"
mkdir -p "$HERMES_TARGET/scripts/lib"
cp "$SRC/SKILL.md" "$HERMES_TARGET/SKILL.md"
rsync -a \
"$SRC/scripts/last30days.py" \
"$SRC/scripts/watchlist.py" \
"$SRC/scripts/briefing.py" \
"$SRC/scripts/store.py" \
"$HERMES_TARGET/scripts/"
rsync -a "$SRC/scripts/lib/"*.py "$HERMES_TARGET/scripts/lib/"
if [ -d "$SRC/scripts/lib/vendor" ]; then
rsync -a "$SRC/scripts/lib/vendor" "$HERMES_TARGET/scripts/lib/"
fi
if [ -d "$SRC/fixtures" ]; then
mkdir -p "$HERMES_TARGET/fixtures"
rsync -a "$SRC/fixtures/" "$HERMES_TARGET/fixtures/"
fi
mod_count=$(ls "$HERMES_TARGET/scripts/lib/"*.py 2>/dev/null | wc -l | tr -d ' ')
echo " Copied $mod_count modules to Hermes"
if (
cd "$HERMES_TARGET/scripts" &&
python3 -c "import briefing, store, watchlist; from lib import youtube_yt, bird_x, render, ui; print(' Import check: OK')"
); then
true
else
echo " Import check FAILED"
fi
fi
# OpenClaw sync only runs when the private-repo OpenClaw variant is present
# in the source tree. The public repo does not ship variants/open (the variant
# is sanitized via strip_for_openclaw.py and published separately from
# last30days-skill-private).
if [ -d "$SRC/variants/open" ]; then
sync_target "$OPENCLAW_TARGET" "$SRC/variants/open/SKILL.md"
else
echo ""
echo "Skipping OpenClaw target (no variants/open in this repo)"
fi
echo ""
echo "Sync complete."
+281 -90
View File
@@ -1,6 +1,6 @@
--- ---
name: last30days name: last30days
version: "3.0.1" version: "3.2.4"
description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web." description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react' argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -13,9 +13,9 @@ metadata:
openclaw: openclaw:
emoji: "📰" emoji: "📰"
requires: requires:
env: env: []
- SCRAPECREATORS_API_KEY
optionalEnv: optionalEnv:
- SCRAPECREATORS_API_KEY
- OPENAI_API_KEY - OPENAI_API_KEY
- XAI_API_KEY - XAI_API_KEY
- OPENROUTER_API_KEY - OPENROUTER_API_KEY
@@ -45,6 +45,7 @@ metadata:
- instagram - instagram
- hackernews - hackernews
- polymarket - polymarket
- digg
- bluesky - bluesky
- truthsocial - truthsocial
- trends - trends
@@ -59,6 +60,35 @@ metadata:
- clawhub - clawhub
--- ---
# STEP 0: STALE-CLONE SELF-CHECK — RUN BEFORE READING BELOW
Before reading anything else in this file, check whether you loaded SKILL.md from the one known stale-clone location: Claude Code's marketplaces directory.
**Run this check:**
```bash
CLAUDE_CACHE_LATEST=$(find "$HOME/.claude/plugins/cache/last30days-skill/last30days" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -V | tail -1)
# Two cache layouts ship in the wild — nested ({version}/skills/last30days/SKILL.md)
# and flat ({version}/SKILL.md). Resolve to whichever shape actually exists.
CLAUDE_CACHE_SKILL_MD=""
if [ -n "$CLAUDE_CACHE_LATEST" ]; then
if [ -f "$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md" ]; then
CLAUDE_CACHE_SKILL_MD="$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md"
elif [ -f "$CLAUDE_CACHE_LATEST/SKILL.md" ]; then
CLAUDE_CACHE_SKILL_MD="$CLAUDE_CACHE_LATEST/SKILL.md"
fi
fi
echo "CLAUDE_CACHE_SKILL_MD=$CLAUDE_CACHE_SKILL_MD"
```
If the SKILL.md path you just Read contains `/.claude/plugins/marketplaces/` AND `$CLAUDE_CACHE_SKILL_MD` is non-empty, STOP and re-read `$CLAUDE_CACHE_SKILL_MD` before proceeding. Otherwise the SKILL.md you have is fine — continue.
**Why this specific check:** `~/.claude/plugins/marketplaces/last30days-skill/` is a git clone Claude Code auto-restores to `origin/main` on session start. It can lag the versioned cache by one or more releases. Three 2026-04-22 test runs (Linear, Coinbase) loaded SKILL.md from `marketplaces/`, ran `--help` from the same stale path, did not see the `--competitors` flag that existed in the cache, and fell back to a manual comparison plan. Result: 2 of 3 windows never invoked the feature they were asked to test. STEP 0 defends against that one Claude Code-specific bug.
**Other install paths are fine:** `~/.codex/skills/`, `~/.agents/skills/`, an `npx skills add` install dir, or a repo checkout are all valid load points - the resolver in Step 1 picks them up. Do NOT abort or hop on those paths.
---
# SKILL CONTRACT — READ BEFORE ANY TOOL CALL # SKILL CONTRACT — READ BEFORE ANY TOOL CALL
You are inside the `/last30days` SKILL. This is a specific research tool with a 1400+ line instruction contract (the rest of this file) that defines EXACTLY how to produce the research output. It is not a generic "last 30 days of X" research prompt. Do NOT treat `/last30days` as a search keyword you can improvise against. You are inside the `/last30days` SKILL. This is a specific research tool with a 1400+ line instruction contract (the rest of this file) that defines EXACTLY how to produce the research output. It is not a generic "last 30 days of X" research prompt. Do NOT treat `/last30days` as a search keyword you can improvise against.
@@ -67,7 +97,7 @@ You are inside the `/last30days` SKILL. This is a specific research tool with a
**How v3.0.7 fixes it:** three structural anchors. **How v3.0.7 fixes it:** three structural anchors.
1. **The MANDATORY first-line badge** (`🌐 last30days v{VERSION} · synced {YYYY-MM-DD}`) at the top of every response is the LAW 2 / LAW 4 enforcement anchor. See "BADGE (MANDATORY, FIRST LINE OF OUTPUT)" in the synthesis section. 1. **The MANDATORY first-line badge** (`🌐 last30days v{VERSION} · synced {YYYY-MM-DD}`) at the top of every response is the LAW 2 / LAW 4 enforcement anchor. See "BADGE (MANDATORY, FIRST LINE OF OUTPUT)" in the synthesis section.
2. **The pinned SKILL_ROOT resolution** in the engine Bash calls always points to the public plugin cache, never `~/.openclaw/` or other stale copies. 2. **The SKILL_DIR substitution** in the engine Bash calls uses the directory of the SKILL.md the model just Read — no resolver list, no precedence walk. Whichever install the harness loaded SKILL.md from is the install whose engine runs. Aligns spec-with-code and works for any harness without enumerating its install path.
3. **This preface** tells you plainly: do NOT improvise. Follow SKILL.md top to bottom. 3. **This preface** tells you plainly: do NOT improvise. Follow SKILL.md top to bottom.
If you catch yourself about to write a `##` section header in a GENERAL-query body, a custom title line, a `Sources:` bullet list, a `for dir in ...` path-discovery loop, or a bare `python3 scripts/last30days.py "{TOPIC}"` engine call with no pre-flight flags — stop. Those are the exact failure modes the LAWs and this contract exist to prevent. The 10/10 beta validation from 2026-04-18 and the 0/8 public v3.0.6 regression from the same day had THE SAME MODEL and SIMILAR SKILL.md CONTENT; the delta is the three anchors this release restores. Read SKILL.md top to bottom before emitting your first response. If you catch yourself about to write a `##` section header in a GENERAL-query body, a custom title line, a `Sources:` bullet list, a `for dir in ...` path-discovery loop, or a bare `python3 scripts/last30days.py "{TOPIC}"` engine call with no pre-flight flags — stop. Those are the exact failure modes the LAWs and this contract exist to prevent. The 10/10 beta validation from 2026-04-18 and the 0/8 public v3.0.6 regression from the same day had THE SAME MODEL and SIMILAR SKILL.md CONTENT; the delta is the three anchors this release restores. Read SKILL.md top to bottom before emitting your first response.
@@ -84,13 +114,13 @@ These anchors used to live at line 1094 of this file. Three independent Opus 4.7
🌐 last30days v{VERSION} · synced {YYYY-MM-DD} 🌐 last30days v{VERSION} · synced {YYYY-MM-DD}
``` ```
Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_ROOT/.claude-plugin/plugin.json"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins. Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_DIR/../../.claude-plugin/plugin.json" 2>/dev/null || awk '/^version:/{gsub(/"/,"",$2); print $2; exit}' "$SKILL_DIR/SKILL.md"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins.
**Why the badge is MANDATORY:** it is the structural anchor for the canonical output shape. Without it the model drifts into blog-post narrative format with `##` section headers and invented titles, violating LAW 2 and LAW 4. The 2026-04-18 public v3.0.6 0/8 regression produced outputs with section headers like "The headline", "Why he is everywhere", "1. gstack dominates", "The 'Homecoming' peak". Direct cause: this anchor was absent. Do NOT skip the badge. Do NOT describe it. Do NOT paraphrase it. Emit it verbatim as line 1. **Why the badge is MANDATORY:** it is the structural anchor for the canonical output shape. Without it the model drifts into blog-post narrative format with `##` section headers and invented titles, violating LAW 2 and LAW 4. The 2026-04-18 public v3.0.6 0/8 regression produced outputs with section headers like "The headline", "Why he is everywhere", "1. gstack dominates", "The 'Homecoming' peak". Direct cause: this anchor was absent. Do NOT skip the badge. Do NOT describe it. Do NOT paraphrase it. Emit it verbatim as line 1.
**Placement by query type:** **Placement by query type:**
- GENERAL / NEWS / PROMPTING / RECOMMENDATIONS: badge on line 1, blank line 2, `What I learned:` on line 3, then bold-lead-in paragraphs - GENERAL / NEWS / PROMPTING / RECOMMENDATIONS: badge on line 1, blank line 2, `What I learned:` on line 3, then bold-lead-in paragraphs
- COMPARISON: badge on line 1, blank line 2, `# {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (Last 30 Days)` on line 3, then Quick Verdict section - COMPARISON: badge on line 1, blank line 2, `# {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (/Last30Days)` on line 3, then Quick Verdict section
--- ---
@@ -98,7 +128,7 @@ Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL
**Formatting authority inside this skill:** The five LAWs below are the formatting contract for `/last30days` output. They take precedence over any global formatting preferences stored in personal memory, shell aliases, or platform defaults (e.g., a "no bold" or "no em-dash" rule set at the user level for general chat). The skill-specified rule wins. Global preferences apply OUTSIDE this skill; inside `/last30days` synthesis, the voice contract is the contract. Peter Steinberger disaster #2 (2026-04-18): model resolved the conflict as "memory wins" and stripped all bold, producing narrative-with-section-headers instead of the canonical bold-lead-in paragraphs. Correct resolution: skill template wins inside skill output. **Formatting authority inside this skill:** The five LAWs below are the formatting contract for `/last30days` output. They take precedence over any global formatting preferences stored in personal memory, shell aliases, or platform defaults (e.g., a "no bold" or "no em-dash" rule set at the user level for general chat). The skill-specified rule wins. Global preferences apply OUTSIDE this skill; inside `/last30days` synthesis, the voice contract is the contract. Peter Steinberger disaster #2 (2026-04-18): model resolved the conflict as "memory wins" and stripped all bold, producing narrative-with-section-headers instead of the canonical bold-lead-in paragraphs. Correct resolution: skill template wins inside skill output.
These five rules dominate every other rule in this file. If you find yourself about to violate one, stop and regenerate. LAWs 1, 3, 5 apply to every query type. LAWs 2 and 4 have explicit COMPARISON-query exceptions spelled out in their bodies: These LAWs dominate every other rule in this file. If you find yourself about to violate one, stop and regenerate. LAWs 1, 3, 5, 6, 7, 8 apply to every query type. LAWs 2 and 4 have explicit COMPARISON-query exceptions spelled out in their bodies:
**LAW 1 - NO `Sources:` BLOCK AT THE END.** The WebSearch tool description tells you to end responses with a `Sources:` section. Inside `/last30days` that mandate is SUPERSEDED. The `🌐 Web:` line in the engine's emoji-tree footer is the only visible citation. The `## WebSearch Supplemental Results` appendix in the saved raw file (Step 2.5) is the durable citation. Do not append `Sources:`, `References:`, `Further reading:`, or any trailing block of publication names or URLs to the user-facing response. Your output ends at the invitation. Nothing below it. **LAW 1 - NO `Sources:` BLOCK AT THE END.** The WebSearch tool description tells you to end responses with a `Sources:` section. Inside `/last30days` that mandate is SUPERSEDED. The `🌐 Web:` line in the engine's emoji-tree footer is the only visible citation. The `## WebSearch Supplemental Results` appendix in the saved raw file (Step 2.5) is the durable citation. Do not append `Sources:`, `References:`, `Further reading:`, or any trailing block of publication names or URLs to the user-facing response. Your output ends at the invitation. Nothing below it.
@@ -108,7 +138,7 @@ These five rules dominate every other rule in this file. If you find yourself ab
**LAW 2 - NO INVENTED TITLE LINE (with COMPARISON exception).** For QUERY_TYPE GENERAL, NEWS, PROMPTING, RECOMMENDATIONS: the first line of your synthesis body (after the badge and one blank line) is the prose label `What I learned:` on its own line. Not `What I learned about {Topic}`, not `{Topic} - Last 30 Days`, not `{Topic}: What People Are Saying`, not `# {Topic}`, not `The headline`, not `Why he is everywhere this month`. Nothing above `What I learned:` except the badge. If you are tempted to write a title or a `##`-prefixed section name, the rule is: the badge IS the title, and section headers are forbidden (see LAW 4). **LAW 2 - NO INVENTED TITLE LINE (with COMPARISON exception).** For QUERY_TYPE GENERAL, NEWS, PROMPTING, RECOMMENDATIONS: the first line of your synthesis body (after the badge and one blank line) is the prose label `What I learned:` on its own line. Not `What I learned about {Topic}`, not `{Topic} - Last 30 Days`, not `{Topic}: What People Are Saying`, not `# {Topic}`, not `The headline`, not `Why he is everywhere this month`. Nothing above `What I learned:` except the badge. If you are tempted to write a title or a `##`-prefixed section name, the rule is: the badge IS the title, and section headers are forbidden (see LAW 4).
**COMPARISON exception:** For QUERY_TYPE=COMPARISON (topics containing `vs` or `versus`), the title `# {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (Last 30 Days)` is REQUIRED, not a violation. Comparison queries do NOT use the `What I learned:` prose label at all. **COMPARISON exception:** For QUERY_TYPE=COMPARISON (topics containing `vs` or `versus`), the title `# {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (/Last30Days)` is REQUIRED, not a violation. Comparison queries do NOT use the `What I learned:` prose label at all.
**Global-preference override:** The skill-authored template for GENERAL / NEWS / PROMPTING / RECOMMENDATIONS queries uses `**bold**` for KEY PATTERNS items and for mid-paragraph lead-ins. Do NOT strip this bold on the grounds of a personal "no bold" memory. The skill's voice contract is the formatting authority here. **Global-preference override:** The skill-authored template for GENERAL / NEWS / PROMPTING / RECOMMENDATIONS queries uses `**bold**` for KEY PATTERNS items and for mid-paragraph lead-ins. Do NOT strip this bold on the grounds of a personal "no bold" memory. The skill's voice contract is the formatting authority here.
@@ -156,13 +186,27 @@ The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do." Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
``` ```
**LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan '$JSON'`. **LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan "$QUERY_PLAN_FILE"` (the path to a tmpfile you wrote via heredoc — see Step 1 for the pattern; never inline `--plan '$JSON'`, apostrophes in search/ranking strings break shell parsing).
Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan '$JSON'`. A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema). Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan "$QUERY_PLAN_FILE"` (or any path the engine can read). A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema).
**Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step. **Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step.
**Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan '$JSON'`? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider. **Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan "$QUERY_PLAN_FILE"` (or another path the engine can read)? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan, then write it to a tmpfile per the Step 1 pattern. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider.
**LAW 8 - EVERY CITATION IN THE NARRATIVE IS AN INLINE MARKDOWN LINK `[name](url)`. NEVER A RAW URL STRING. NEVER A PLAIN NAME WHEN A URL IS AVAILABLE.** Applies to every query type. In the "What I learned:" narrative, in KEY PATTERNS, and in the COMPARISON body sections, every cited @handle, r/subreddit, publication, YouTube channel, TikTok creator, Instagram creator, and Polymarket market is wrapped as `[name](url)` at first mention. The URL comes from the raw research dump — every engine item carries a URL; WebSearch supplements carry URLs in their own output. Claude Code renders `[text](url)` as blue CMD-clickable text; the URL is hidden in the rendering, only the link text shows. The stats footer (emoji-tree block) is engine-emitted per LAW 5 and passes through verbatim — do NOT reformat its links yourself.
**Plain-text fallback:** if the raw data genuinely has no URL for a specific source, fall back to plain text for that one citation only. Never emit a broken empty link like `[Rolling Stone]()` or `[@handle]()`. Default assumption: URL exists; plain text is the exception.
**BAD (raw URL):** `per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/`
**BAD (plain name when URL is available):** `per Rolling Stone`, `per @honest30bgfan_`, `r/hiphopheads`
**BAD (broken empty link):** `per [Rolling Stone]()`
**GOOD:** `per [Rolling Stone](https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/)`, `per [@honest30bgfan_](https://x.com/honest30bgfan_)`, `[r/hiphopheads](https://reddit.com/r/hiphopheads)`
**FALLBACK (URL genuinely missing):** `per Rolling Stone`
**Observed LAW 8 need (2026-04-20 inline-links saga):** the citation rule existed in SKILL.md but was placed in the CITATION PRIORITY block around line 1224 - below the chunked-read window. Four consecutive test runs (Matt Van Horn, Peter Steinberger, Best Headphones, OpenClaw vs Hermes) confirmed the rule was deployed (diff IN SYNC, grep found the text) but was skipped on every synthesis because the model read lines 1-1000 and stopped. The model's own self-diagnosis, repeated verbatim four times: "I never reached line 1224." LAW 8 hoists the rule into the same guaranteed-loaded band as LAWs 1-7 so it enters context on every run. Same pattern that solved v3.0.6 (invented titles), disaster #2 (stripped bold), disaster #3 (trailing Sources), and the Hermes 2026-04-19 evidence-dump disaster.
**Post-synthesis self-check (do this BEFORE emitting your response):** scan your drafted "What I learned:" and KEY PATTERNS for the `[name](url)` pattern. Count how many inline markdown links appear. If zero - and the raw dump has URLs for the @handles, r/subs, and publications you cited as plain text - regenerate ONCE with inline links added. Stripping links is not a valid way to satisfy any other LAW; LAWs 1 (no trailing Sources) and 8 (inline links required) are complementary, not alternatives.
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail. End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
@@ -199,9 +243,9 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
--- ---
# last30days v3.0.1: Research Any Topic from the Last 30 Days # last30days v3.2.4: Research Any Topic from the Last 30 Days
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `~/Documents/Last30Days/`. X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section. > **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now. Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now.
@@ -221,8 +265,14 @@ if [ -z "${LAST30DAYS_PYTHON:-}" ]; then
echo "ERROR: last30days v3 requires Python 3.12+. Install python3.12 or python3.13 and rerun." >&2 echo "ERROR: last30days v3 requires Python 3.12+. Install python3.12 or python3.13 and rerun." >&2
exit 1 exit 1
fi fi
LAST30DAYS_MEMORY_DIR="${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}"
``` ```
## Configuration
Set `LAST30DAYS_MEMORY_DIR` before invoking the skill to choose where raw research files are saved. If it is not set, the skill defaults to `~/Documents/Last30Days`.
## Step 0: First-Run Setup Wizard ## Step 0: First-Run Setup Wizard
Before proceeding to Step 1, handle first-run setup. Before proceeding to Step 1, handle first-run setup.
@@ -277,14 +327,14 @@ Common patterns:
- Always active: Reddit, Hacker News, Polymarket - Always active: Reddit, Hacker News, Polymarket
- If gh CLI is installed (check `which gh`): add GitHub - If gh CLI is installed (check `which gh`): add GitHub
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set: add X - If digg-pp-cli is installed (check `which digg-pp-cli`): add Digg
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set, or xurl CLI is installed and authenticated: add X
- If yt-dlp is installed (check `which yt-dlp`): add YouTube - If yt-dlp is installed (check `which yt-dlp`): add YouTube
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok - If SCRAPECREATORS_API_KEY is set: add TikTok, Instagram, Threads (suppress any of these via EXCLUDE_SOURCES)
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram - If SCRAPECREATORS_API_KEY is set and the user explicitly requested pinterest for this query (e.g. via `--search=pinterest`): add Pinterest
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains pinterest: add Pinterest
- If BSKY_HANDLE and BSKY_APP_PASSWORD are set: add Bluesky - If BSKY_HANDLE and BSKY_APP_PASSWORD are set: add Bluesky
- If OPENROUTER_API_KEY is set: add Perplexity - If OPENROUTER_API_KEY is set and INCLUDE_SOURCES contains perplexity: add Perplexity
- If EXCLUDE_SOURCES is set (comma-separated, case-insensitive): drop any matching source from the list above before displaying
Then display (use "and more" if 5+ sources, otherwise list all with Oxford comma): Then display (use "and more" if 5+ sources, otherwise list all with Oxford comma):
@@ -508,7 +558,7 @@ If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):
5. **Skip** the follow-up invitation ("I'm now an expert on X...") 5. **Skip** the follow-up invitation ("I'm now an expert on X...")
6. **Output** the complete research report and stop - do not wait for further input 6. **Output** the complete research report and stop - do not wait for further input
Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir` (handled by the script, no extra tool calls). Agent mode saves raw research data to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`) automatically via `--save-dir` (handled by the script, no extra tool calls).
Agent mode report format: Agent mode report format:
@@ -530,28 +580,93 @@ Generated: {date} | Sources: Reddit, X, Bluesky, YouTube, TikTok, HN, Polymarket
## If QUERY_TYPE = COMPARISON ## If QUERY_TYPE = COMPARISON
When the user asks "X vs Y", run ONE research pass with a comparison-optimized plan that covers both entities AND their rivalry. This replaces the old 3-pass approach (which took 13+ minutes and produced tangential content). When the user asks "X vs Y" (or "X vs Y vs Z"), the engine fans out N full `pipeline.run()` calls in parallel — one per entity — each with its own Step 0.55-grade targeting. This restored the old N-pass architecture (reverted the one-pass latency optimization that removed per-entity depth); parallel execution keeps wall clock ≈ a single pass.
**IMPORTANT: Include BOTH X handles (`--x-handle={TOPIC_A_HANDLE} --x-related={TOPIC_B_HANDLE},{COMPANY_HANDLES},{COMMENTATOR_HANDLES}`), `--subreddits={RESOLVED_SUBREDDITS}`, `--tiktok-hashtags={RESOLVED_HASHTAGS}`, `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}`, and `--ig-creators={RESOLVED_IG_CREATORS}` from Step 0.55. Omit any flag where the value was not resolved (empty).** **MANDATORY per-entity resolution.** For each entity, resolve the full Step 0.55 stack (X handle, subreddits, GitHub user/repos, news context). Then assemble a `--competitors-plan` JSON mapping each entity to its targeting, and invoke the engine ONCE with the vs-topic string.
**Single pass with entity-aware subqueries:** **Output shape per run:**
- Main topic saves to `{main-slug}-raw.md`.
- Each peer saves to `{peer-slug}-raw.md`.
- Stdout shows a merged comparison with the `## Head-to-Head` scaffold + per-entity Resolved Entities block.
**Invocation:**
```bash ```bash
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B}" --emit=compact --save-dir=~/Documents/Last30Days --save-suffix=v3 --plan 'COMPARISON_PLAN_JSON' --x-handle={TOPIC_A_HANDLE} --x-related={TOPIC_B_HANDLE},{COMPANY_A_HANDLE},{COMPANY_B_HANDLE},{COMMENTATOR_HANDLES} --subreddits={RESOLVED_SUBREDDITS} --tiktok-hashtags={RESOLVED_HASHTAGS} --tiktok-creators={RESOLVED_TIKTOK_CREATORS} --ig-creators={RESOLVED_IG_CREATORS} # SKILL_DIR = absolute path of the directory containing THIS SKILL.md you just Read.
# Substitute the actual path below — your harness told you where this file lives via
# the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
if [ ! -f "$SKILL_DIR/scripts/last30days.py" ]; then
echo "ERROR: scripts/last30days.py not found under SKILL_DIR=$SKILL_DIR" >&2
echo "Re-check the directory of the SKILL.md you Read and substitute it as SKILL_DIR above." >&2
exit 1
fi
# Write the per-entity plan to a tmpfile and pass the path to the engine.
# The engine's parse_competitors_plan() reads file paths transparently. This
# avoids the inline-single-quoted-JSON apostrophe trap (resolved context
# strings like "people's choice" or "McDonald's" otherwise close the outer
# single-quote and break shell parsing before the engine is even invoked).
# Trailing XXXXXX (no .json suffix) so BSD/macOS mktemp works the same as
# GNU; BSD only substitutes X's at the end of the template.
COMPETITORS_PLAN_FILE=$(mktemp "${TMPDIR:-/tmp}/last30days-competitors.XXXXXX")
trap 'rm -f "$COMPETITORS_PLAN_FILE"' EXIT
cat > "$COMPETITORS_PLAN_FILE" <<'PLAN_EOF'
{
"{TOPIC_B}": {"x_handle":"{TOPIC_B_HANDLE}","subreddits":["{TOPIC_B_SUB_1}","{TOPIC_B_SUB_2}"],"github_user":"{TOPIC_B_GH}","context":"{TOPIC_B_CONTEXT}"},
"{TOPIC_C}": {"x_handle":"{TOPIC_C_HANDLE}","subreddits":["{TOPIC_C_SUB_1}"],"github_user":"{TOPIC_C_GH}","context":"{TOPIC_C_CONTEXT}"}
}
PLAN_EOF
"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B} vs {TOPIC_C}" \
--emit=compact \
--save-dir="${LAST30DAYS_MEMORY_DIR}" \
--save-suffix=v3 \
--x-handle={TOPIC_A_HANDLE} \
--subreddits={TOPIC_A_SUBS} \
--competitors-plan "$COMPETITORS_PLAN_FILE"
``` ```
**The `--plan` JSON for comparisons should include 3-4 subqueries:** **The quoted heredoc marker `'PLAN_EOF'` is load-bearing** — quoting suppresses shell interpolation so apostrophes, `$`, backticks, etc. pass through verbatim. If you ever switch to an unquoted `<<PLAN_EOF`, every variable reference and apostrophe inside the JSON becomes a parse hazard.
1. **Head-to-head:** `"{TOPIC_A} vs {TOPIC_B}"` - catches rivalry content, direct comparisons
2. **Entity A news:** `"{TOPIC_A} news {MONTH} {YEAR}"` - catches entity-specific developments
3. **Entity B news:** `"{TOPIC_B} news {MONTH} {YEAR}"` - catches entity-specific developments
4. (Optional) **Domain context:** `"{COMPANY_A} {COMPANY_B} {DOMAIN} news"` - catches industry context (e.g., "OpenAI Anthropic AI news")
ALL subqueries include ALL sources. The fusion engine handles deduplication across subqueries. **At least one subquery MUST include YouTube-specific search terms** (e.g., "{PERSON} interview 2026", "{PRODUCT_A} vs {PRODUCT_B} review") to ensure YouTube content is found. Without YouTube-specific terms, the engine may only find 0-1 videos for comparison queries. Topic A (the main topic, first in the vs-string) uses outer `--x-handle`, `--x-related`, `--subreddits`, `--github-user`, `--github-repo`, `--tiktok-*`, `--ig-creators` as usual. Topics B and C get their targeting from `--competitors-plan` entries (keyed by entity name, case-insensitive).
Then do WebSearch for: `{TOPIC_A} vs {TOPIC_B} comparison {YEAR}` and `{TOPIC_A} vs {TOPIC_B} which is better` and `{COMPANY_A} vs {COMPANY_B} news {MONTH} {YEAR}`. **Step 0.55 for N entities.** The same pre-research protocol that applies to a single-entity topic applies to EACH entity in a vs-run. For N=3, that means 3 WebSearches for X handles, 3 for subreddits, 3 for GitHub, 3 for news context — or equivalent batched queries. A `## Resolved Entities` block with dashes for any entity means you skipped Step 0.55 for that one. Re-run with a corrected plan.
**Then do WebSearch supplements** for: `{TOPIC_A} vs {TOPIC_B} comparison {YEAR}` and `{TOPIC_A} vs {TOPIC_B} which is better` — these catch rivalry articles that per-entity passes might not surface.
**Skip the normal Step 1 below** - go directly to the comparison synthesis format (see "If QUERY_TYPE = COMPARISON" in the synthesis section). **Skip the normal Step 1 below** - go directly to the comparison synthesis format (see "If QUERY_TYPE = COMPARISON" in the synthesis section).
**COMPARISON TABLE SCAFFOLD (engine-emitted, pass through verbatim):** For comparison topics, the engine's compact output includes a `## Head-to-Head Comparison` block with an empty markdown table (columns = entities, rows = axes like "Core pitch", "Who it's for", "Community stance", "Trajectory") plus a "Choose X if / Choose Y if" prose block. Your synthesis MUST include this block verbatim with filled cells, positioned between the narrative and the emoji-tree footer. Keep each cell to 5-15 words. Use ' - ' (hyphen with spaces) not em-dashes inside cells. The block is the canonical comparison output shape - do not invent your own table structure. **COMPARISON TABLE SCAFFOLD (engine-emitted, pass through verbatim):** For comparison topics, the engine's compact output includes a `## Head-to-Head` block with an empty markdown table (columns = entities, rows = axes like "What it is", "Community sentiment", "Trajectory"). Your synthesis MUST include this block verbatim with filled cells, positioned between the narrative and the emoji-tree footer. Keep each cell to 5-15 words. Use ' - ' (hyphen with spaces) not em-dashes inside cells.
### Competitor mode (`--competitors`)
`--competitors` is a SKILL.md-level shortcut for vs-mode with auto-discovery. The engine flag itself just signals intent; YOU (the hosting reasoning model) do the discovery and Step 0.55 via your own WebSearch tool, then invoke the vs-topic path above.
**The four-step protocol:**
1. **Discover peers** via WebSearch: `"{topic} competitors"` / `"{topic} alternatives"`. Pick N=2 by default (match the flag's default), N=argument value if the user passed `--competitors=N`.
2. **Run Step 0.55 for the main topic AND each peer** — same protocol you use for a single-entity topic, just N times. X handle, subreddits, GitHub, news context, per entity.
3. **Build the vs-topic string**: `"{main} vs {peer1} vs {peer2}"`.
4. **Invoke the engine** with the vs-topic, `--competitors-plan` JSON covering both peers (and the main topic if you want to override the outer flags), and the outer `--x-handle`/`--subreddits`/`--github-*` for the main topic.
**Flag surface (engine):**
- `--competitors` (bare) - signals the hosting model to discover 2 peers (3-way total).
- `--competitors=N` - N peers (1..6; out-of-range clamps with stderr warning).
- `--competitors-list="A,B,C"` - minimum escape hatch; names only, no per-entity targeting. Peer sub-runs fall back to planner defaults (visibly thinner data).
- `--competitors-plan '{entity: {x_handle, subreddits, github_user, github_repos, context}}'` - full per-entity targeting; implies vs-mode; preferred.
- `--polymarket-keywords "kw1,kw2"` - disambiguate Polymarket for ambiguous single-token topics ("Warriors" → `nba,gsw,golden-state`).
**Why --competitors-plan over --competitors-list:** without per-entity handles/subs, peer sub-runs run with deterministic single-word planner queries and produce visibly thinner evidence than the main topic. The Resolved Entities block in stdout makes the gap visible — dashes for a peer = you skipped its Step 0.55.
**Engine-internal auto-resolve (headless fallback):** if the engine detects BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY, it runs its own per-entity `resolve.auto_resolve()` before each sub-run. The hosting-model path does NOT need those keys — you are the WebSearch. The engine's auto-resolve is the cron/CI fallback for when no reasoning model is driving.
**Output:** one `{slug}-raw.md` per entity in `--save-dir` plus the merged comparison on stdout. Synthesis contract identical to the vs-mode protocol above.
--- ---
@@ -576,6 +691,43 @@ The first search finds subreddits. The second gives you current events context (
Extract 3-5 subreddit names from the results. Store as `RESOLVED_SUBREDDITS` (comma-separated, no r/ prefix). Extract 3-5 subreddit names from the results. Store as `RESOLVED_SUBREDDITS` (comma-separated, no r/ prefix).
**2a. Category-peer expansion (MANDATORY for product topics).** If the topic is a product in a recognizable category (AI image generation, AI video generation, AI coding agents, AI music, AI chat models, SaaS screen recording, prediction markets, etc.), the brand-specific subreddits that WebSearch returned are INSUFFICIENT. Add 2-3 peer subreddits from the category. Peer subs are where cross-product technique discussion actually lives. Missing them is the 2026-04-22 `GPT Image 2` failure mode: the model resolved `r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering` (all OpenAI-brand) and missed `r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt` where prompting techniques are actually shared. The user had to manually prompt "check image generation reddits too" to get a usable run.
Canonical category peers (single source of truth; `scripts/lib/categories.py` mirrors this for the `--auto-resolve` engine path):
| Category | Trigger keywords | Peer subs (priority order) |
|----------|------------------|---------------------------|
| `ai_image_generation` | image generation, text to image, GPT Image, Nano Banana, Midjourney, Stable Diffusion, DALL-E, Flux.1, Imagen, Seedance, Ideogram, Recraft | `StableDiffusion, midjourney, dalle2, aiArt, PromptEngineering, MediaSynthesis` |
| `ai_video_generation` | video generation, text to video, Sora, Veo 3, Runway Gen, Kling, Pika Labs, Luma Dream Machine, Hailuo | `aivideo, StableDiffusion, runwayml, singularity, MediaSynthesis` |
| `ai_music_generation` | music generation, ai music, Suno, Udio, Riffusion, Stable Audio | `SunoAI, udiomusic, aimusic, artificial` |
| `ai_coding_agent` | Claude Code, Cursor IDE, GitHub Copilot, Windsurf, Aider, Cline, OpenClaw, Hermes Agent, Continue.dev, Codeium, Devin | `ChatGPTCoding, LocalLLaMA, singularity, PromptEngineering` |
| `ai_agent_framework` | agent framework, LangChain, LangGraph, CrewAI, AutoGen, LlamaIndex, DSPy, smolagents | `LangChain, LocalLLaMA, AI_Agents, MachineLearning` |
| `ai_chat_model` | GPT-5/4, Claude Opus/Sonnet/Haiku, Gemini Pro/Flash, Llama 3/4, DeepSeek, Qwen, Mistral Large, Grok | `LocalLLaMA, ChatGPT, ClaudeAI, singularity, artificial` |
| `saas_screen_recording` | screen recording, screen recorder, Loom video, Tella screen, Vidyard | `SaaS, screenrecording, productivity, Entrepreneur` |
| `saas_productivity` | Notion app, Obsidian, Linear app, Asana, ClickUp, productivity app | `productivity, SaaS, ObsidianMD, Notion` |
| `prediction_markets` | Polymarket, Kalshi, prediction market, event contracts, Manifold Markets | `Polymarket, Kalshi, predictionmarkets` |
| `crypto_defi` | DeFi protocol, yield farming, liquidity pool, stablecoin, layer 2, L2 rollup | `defi, ethfinance, CryptoCurrency, ethereum` |
**Merging rule.** Start with WebSearch-returned subs. Append 2-3 category peers in the priority order shown. Dedupe case-insensitively (don't list `midjourney` twice if WebSearch already returned it). Cap total at 10: if adding all peers would exceed the cap, keep every WebSearch-returned sub (they are the freshest signal) and drop peers from the end of the priority list.
**Extrapolation.** If the topic is a product in a category NOT listed in the table (new AI tool, niche SaaS), use the same spirit: pick the 2-3 most active cross-product communities where technique discussion happens. A new image-gen tool still gets `r/StableDiffusion, r/midjourney, r/aiArt`. A new code editor still gets `r/ChatGPTCoding, r/LocalLLaMA`.
**Worked example — the failing query.** Topic: `Prompting GPT Image 2`.
Before (the 2026-04-22 failure mode):
```
Resolved:
- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/artificial
```
After (with category-peer expansion):
```
Resolved:
- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt (+ ai_image_generation peers)
```
The parenthetical `(+ ai_image_generation peers)` is the observable contract of the new Resolved block format. See Step 0.55 self-check below.
**3. TikTok hashtags + creators** - **INFER these from your topic knowledge. Do NOT WebSearch for "{PERSON} TikTok account" - most people/CEOs don't have TikTok, and the search is wasted.** **3. TikTok hashtags + creators** - **INFER these from your topic knowledge. Do NOT WebSearch for "{PERSON} TikTok account" - most people/CEOs don't have TikTok, and the search is wasted.**
- **Hashtags:** Infer 2-3 from the topic name + category. Examples: "Kanye West" → `kanyewest,ye,bully`. "Claude Code" → `claudecode,aiagent,aicoding`. "Sam Altman" → `samaltman,openai,chatgpt`. - **Hashtags:** Infer 2-3 from the topic name + category. Examples: "Kanye West" → `kanyewest,ye,bully`. "Claude Code" → `claudecode,aiagent,aicoding`. "Sam Altman" → `samaltman,openai,chatgpt`.
@@ -638,17 +790,19 @@ Passing the resolved block visibly (per-entity, all 4 types each) is the observa
**If you can't infer targeting for a platform, skip that flag -- the Python engine will fall back to keyword search.** **If you can't infer targeting for a platform, skip that flag -- the Python engine will fall back to keyword search.**
**Step 0.55 self-check: category-peer coverage.** Before emitting the Resolved block, re-read your resolved subreddit list. Does the topic match any category in the Section 2a table (or fit the spirit of one — AI image gen, AI coding, AI music, etc.)? If YES: does your list include AT LEAST 2 peer subs from that category? If NO, widen the list NOW — do not run the engine yet. The observable contract is the `(+ {category_id} peers)` annotation on the Reddit line in the Resolved block. Its absence on a product-in-a-known-category topic is a Step 0.55 regression — the named 2026-04-22 failure mode. Person topics, music artists, news stories, and topics outside any category are exempt; omit the annotation.
**After resolving all handles and communities, display what you found before moving on.** This shows the user that intelligent pre-research happened: **After resolving all handles and communities, display what you found before moving on.** This shows the user that intelligent pre-research happened:
``` ```
Resolved: Resolved:
- X: @{HANDLE} (+ @{COMPANY}, @{COMMENTATOR}) - X: @{HANDLE} (+ @{COMPANY}, @{COMMENTATOR})
- Reddit: r/{sub1}, r/{sub2}, r/{sub3} - Reddit: r/{sub1}, r/{sub2}, r/{sub3}, r/{peer1}, r/{peer2} (+ {category_id} peers)
- TikTok: #{hashtag1}, #{hashtag2} - TikTok: #{hashtag1}, #{hashtag2}
- YouTube: {query1}, {query2} - YouTube: {query1}, {query2}
``` ```
Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful. Only show lines for platforms where something was resolved. Skip empty lines. On the Reddit line, the trailing `(+ {category_id} peers)` annotation appears when Step 0.55 Section 2a added category-peer subs. Omit the annotation when the topic had no matching category. This display replaces the old "Parsed intent" block with something more useful.
--- ---
@@ -712,7 +866,7 @@ Only show lines for platforms where something was resolved. Skip empty lines. Th
- For how_to: prioritize YouTube (tutorials) and Reddit (guides) - For how_to: prioritize YouTube (tutorials) and Reddit (guides)
- Primary subquery weight = 1.0, secondary = 0.6-0.8, peripheral = 0.3-0.5 - Primary subquery weight = 1.0, secondary = 0.6-0.8, peripheral = 0.3-0.5
**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key) **Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg clusters - only if `digg-pp-cli` is on PATH)
**Intent → freshness_mode mapping:** **Intent → freshness_mode mapping:**
- breaking_news, prediction → `strict_recent` - breaking_news, prediction → `strict_recent`
@@ -755,34 +909,45 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
**IMPORTANT: Include `--x-handle={RESOLVED_HANDLE}` in the command. For comparison mode: Pass `--x-handle={TOPIC_A_HANDLE}` to the first pass, `--x-handle={TOPIC_B_HANDLE}` to the second pass, and both to the head-to-head pass. Also include `--subreddits={RESOLVED_SUBREDDITS}`, `--tiktok-hashtags={RESOLVED_HASHTAGS}`, `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}`, and `--ig-creators={RESOLVED_IG_CREATORS}` from Step 0.55. Omit any flag where the value was not resolved (empty).** **IMPORTANT: Include `--x-handle={RESOLVED_HANDLE}` in the command. For comparison mode: Pass `--x-handle={TOPIC_A_HANDLE}` to the first pass, `--x-handle={TOPIC_B_HANDLE}` to the second pass, and both to the head-to-head pass. Also include `--subreddits={RESOLVED_SUBREDDITS}`, `--tiktok-hashtags={RESOLVED_HASHTAGS}`, `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}`, and `--ig-creators={RESOLVED_IG_CREATORS}` from Step 0.55. Omit any flag where the value was not resolved (empty).**
```bash ```bash
# PIN SKILL_ROOT to the public plugin cache (highest-version dir wins on upgrade). # SKILL_DIR = absolute path of the directory containing THIS SKILL.md you just Read.
# DO NOT write your own path-discovery loop. The 2026-04-18 Peter Steinberger run 1 # Substitute the actual path below — your harness told you where this file lives via
# regression was caused by a custom discovery loop landing on ~/.openclaw/skills/last30days/ # the Read tool result. Examples:
# (a stale copy from a private-repo sync pattern). That path contains a pre-plan-007 # Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# engine and produces non-canonical output. This pinned resolution ignores every stale # Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# copy (~/.openclaw/, ~/.agents/, ~/.codex/) and picks the plugin cache exclusively. # Read ~/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days/SKILL.md
SKILL_ROOT="$(ls -d "$HOME/.claude/plugins/cache/last30days-skill/last30days/"*/ 2>/dev/null | sort -V | tail -1)" # → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days
SKILL_ROOT="${SKILL_ROOT%/}" # scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
# Fallback for repo checkout / Gemini / Codex hosts where the plugin cache does not exist. if [ ! -f "$SKILL_DIR/scripts/last30days.py" ]; then
# Only runs if the public plugin cache is missing entirely. echo "ERROR: scripts/last30days.py not found under SKILL_DIR=$SKILL_DIR" >&2
if [ -z "$SKILL_ROOT" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then echo "Re-check the directory of the SKILL.md you Read and substitute it as SKILL_DIR above." >&2
for dir in "." "${CLAUDE_PLUGIN_ROOT:-}" "${GEMINI_EXTENSION_DIR:-}"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
fi
if [ -z "${SKILL_ROOT:-}" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
echo "ERROR: Could not find scripts/last30days.py in public plugin cache or repo checkout" >&2
echo "Expected: $HOME/.claude/plugins/cache/last30days-skill/last30days/{VERSION}/scripts/last30days.py" >&2
exit 1 exit 1
fi fi
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --save-dir=~/Documents/Last30Days --save-suffix=v3 "${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" $ARGUMENTS --emit=compact --save-dir="${LAST30DAYS_MEMORY_DIR}" --save-suffix=v3
``` ```
**If you ran Steps 0.55 and 0.75 (agent planning), add these flags:** **If you ran Steps 0.55 and 0.75 (agent planning), pass the plan via a tmpfile and add the targeting flags:**
- `--plan 'QUERY_PLAN_JSON'` (replace with actual JSON from Step 0.75)
```bash
# Write QUERY_PLAN_JSON to a tmpfile before the engine invocation above.
# parse_plan() reads file paths transparently; this avoids inline-JSON
# shell-quoting hazards (apostrophes in search_query / ranking_query
# strings break single-quoted command-line JSON). Trailing XXXXXX (no
# .json suffix) for BSD/macOS portability — BSD mktemp only substitutes
# X's at the end of the template.
QUERY_PLAN_FILE=$(mktemp "${TMPDIR:-/tmp}/last30days-plan.XXXXXX")
trap 'rm -f "$QUERY_PLAN_FILE"' EXIT
cat > "$QUERY_PLAN_FILE" <<'PLAN_EOF'
{QUERY_PLAN_JSON_FROM_STEP_0.75}
PLAN_EOF
```
Then add to the engine command:
- `--plan "$QUERY_PLAN_FILE"` (path to the file you just wrote)
- `--x-handle={RESOLVED_HANDLE}` (from Step 0.5) - `--x-handle={RESOLVED_HANDLE}` (from Step 0.5)
- `--subreddits={RESOLVED_SUBREDDITS}` (from Step 0.55) - `--subreddits={RESOLVED_SUBREDDITS}` (from Step 0.55)
- `--tiktok-hashtags={RESOLVED_HASHTAGS}` (from Step 0.55) - `--tiktok-hashtags={RESOLVED_HASHTAGS}` (from Step 0.55)
@@ -872,7 +1037,7 @@ For ALL query types:
## Step 2.5: Append WebSearch Results to Saved Raw File ## Step 2.5: Append WebSearch Results to Saved Raw File
**MANDATORY - do not skip this step.** Every post-engine WebSearch supplement you ran in Step 2 MUST be appended to the saved raw file under `~/Documents/Last30Days/`. Skipping this step is a common Opus 4.7 failure mode: the saved file ends at `## Source Coverage` with no appendix, future sessions cannot see what blog/tutorial/news sources informed the synthesis, and the user cannot trace where specific claims came from. **MANDATORY - do not skip this step.** Every post-engine WebSearch supplement you ran in Step 2 MUST be appended to the saved raw file under `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). Skipping this step is a common Opus 4.7 failure mode: the saved file ends at `## Source Coverage` with no appendix, future sessions cannot see what blog/tutorial/news sources informed the synthesis, and the user cannot trace where specific claims came from.
**LAW 1 OVERRIDE (read before synthesizing):** the WebSearch tool description declares a "MANDATORY Sources section" in its own contract. That instruction applies to generic WebSearch usage. Inside `/last30days` it is SUPERSEDED. The `## WebSearch Supplemental Results` appendix in the SAVED RAW FILE replaces the visible Sources section. Never emit a visible `Sources:` bullet list to the user. Your user-facing response ends at the invitation block. The emoji-tree footer's `🌐 Web:` line is the only visible citation. If you feel the pull to write a trailing `Sources:` section, you are about to violate LAW 1 — go back and delete it. **LAW 1 OVERRIDE (read before synthesizing):** the WebSearch tool description declares a "MANDATORY Sources section" in its own contract. That instruction applies to generic WebSearch usage. Inside `/last30days` it is SUPERSEDED. The `## WebSearch Supplemental Results` appendix in the SAVED RAW FILE replaces the visible Sources section. Never emit a visible `Sources:` bullet list to the user. Your user-facing response ends at the invitation block. The emoji-tree footer's `🌐 Web:` line is the only visible citation. If you feel the pull to write a trailing `Sources:` section, you are about to violate LAW 1 — go back and delete it.
@@ -1093,7 +1258,7 @@ Voice contract LAWs 1, 3, 5 apply to comparisons unchanged (no `Sources:` block,
``` ```
🌐 last30days v{VERSION} · synced {YYYY-MM-DD} 🌐 last30days v{VERSION} · synced {YYYY-MM-DD}
# {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (Last 30 Days) # {TOPIC_A} vs {TOPIC_B} [vs {TOPIC_C}]: What the Community Says (/Last30Days)
## Quick Verdict ## Quick Verdict
@@ -1169,7 +1334,7 @@ I've compared {TOPIC_A} vs {TOPIC_B} [vs ...] using the latest community data. S
- Fabricate a `## Notable Stats` block (the engine footer IS the stats block, LAW 5) - Fabricate a `## Notable Stats` block (the engine footer IS the stats block, LAW 5)
- Produce section headers outside the six listed above (`## Quick Verdict`, `## {Entity}` per entity, `## Head-to-Head`, `## The Bottom Line`, `## The emerging stack` are the only allowed `##` headers per LAW 4 comparison exception) - Produce section headers outside the six listed above (`## Quick Verdict`, `## {Entity}` per entity, `## Head-to-Head`, `## The Bottom Line`, `## The emerging stack` are the only allowed `##` headers per LAW 4 comparison exception)
**Reference exemplar:** `~/Documents/Last30Days/openclaw-vs-hermes-vs-paperclip-LAUNCH-VIDEO-april9-exemplar.md` preserves the April 9 canonical output with full structural analysis. Match this shape section-for-section. **Reference exemplar:** `$LAST30DAYS_MEMORY_DIR/openclaw-vs-hermes-vs-paperclip-LAUNCH-VIDEO-april9-exemplar.md` preserves the April 9 canonical output with full structural analysis. Match this shape section-for-section.
### For all QUERY_TYPEs ### For all QUERY_TYPEs
@@ -1221,30 +1386,27 @@ CITATION RULE: Cite sources sparingly to prove research is real.
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box - Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one. - Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
CITATION PRIORITY (most to least preferred): **URL formatting is governed by LAW 8** in the VOICE CONTRACT block above. Every citation in the narrative body is an inline markdown link `[name](url)`; raw URL strings are forbidden; plain-text fallback only when the raw data has no URL for that specific source. Re-read LAW 8 now if you skipped it. The stats footer is engine-emitted per LAW 5 and passes through verbatim.
1. @handles from X - "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit - "per r/subreddit" (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title) CITATION PRIORITY (most to least preferred), with each example showing the LAW 8 inline-link shape:
3. YouTube channels - "per [channel name] on YouTube" (transcript-backed insights) 1. @handles from X - `per [@handle](https://x.com/handle)` (these prove the tool's unique value)
4. TikTok creators - "per @creator on TikTok" (viral/trending signal) 2. r/subreddits from Reddit - `per [r/subreddit](https://reddit.com/r/subreddit)` (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title)
5. Instagram creators - "per @creator on Instagram" (influencer/creator signal) 3. YouTube channels - `per [channel name](https://youtube.com/@channel) on YouTube` (transcript-backed insights)
6. HN discussions - "per HN" or "per hn/username" (developer community signal) 4. TikTok creators - `per [@creator](https://tiktok.com/@creator) on TikTok` (viral/trending signal)
7. Polymarket - "Polymarket has X at Y% (up/down Z%)" with specific odds and movement 5. Instagram creators - `per [@creator](https://instagram.com/creator) on Instagram` (influencer/creator signal)
8. Web sources - ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact 6. HN discussions - `per [HN](https://news.ycombinator.com/item?id=N)` or `per [hn/username](https://news.ycombinator.com/user?id=username)` (developer community signal)
7. Polymarket - `[Polymarket](https://polymarket.com/event/...) has X at Y% (up/down Z%)` with specific odds and movement
8. Web sources - ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact; link the publication: `per [Rolling Stone](https://rollingstone.com/...)`
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote. The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post. When both a web article and an X post cover the same fact, cite the X post.
URL FORMATTING: NEVER paste raw URLs anywhere in the output - not in synthesis, not in stats, not in sources. (These narrative examples illustrate LAW 8 from the VOICE CONTRACT.)
- **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- **GOOD:** "per Rolling Stone"
- **BAD stats line:** `🌐 Web: 10 pages - https://later.com/blog/..., https://buffer.com/...`
- **GOOD stats line:** `🌐 Web: 10 pages - Later, Buffer, CNN, SocialBee`
Use the publication/site name, not the URL. The user doesn't need links - they need clean, readable text.
**BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)." **BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
**GOOD:** "His album BULLY drops March 20 - fans on X are split on the tracklist, per @honest30bgfan_" **GOOD:** "His album BULLY drops March 20 - fans on X are split on the tracklist, per [@honest30bgfan_](https://x.com/honest30bgfan_)"
**GOOD:** "Ye's apology got massive traction on r/hiphopheads" **GOOD:** "Ye's apology got massive traction on [r/hiphopheads](https://reddit.com/r/hiphopheads)"
**OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per Billboard" **OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per [Billboard](https://www.billboard.com/music/music-news/hellwatt-festival-2026-lineup-...)"
**Lead with people, not publications.** Start each topic with what Reddit/X **Lead with people, not publications.** Start each topic with what Reddit/X
users are saying/feeling, then add web context only if needed. The user came users are saying/feeling, then add web context only if needed. The user came
@@ -1263,18 +1425,20 @@ here for the conversation, not the press release.
What I learned: What I learned:
**{Headline summarizing topic 1}** - [1-2 sentences about what people are saying, per @handle or r/sub] **{Headline summarizing topic 1}** - [1-2 sentences about what people are saying, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Headline summarizing topic 2}** - [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 2}** - [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Headline summarizing topic 3}** - [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 3}** - [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
KEY PATTERNS from the research: KEY PATTERNS from the research:
1. [Pattern] - per @handle 1. [Pattern] - per [@handle](https://x.com/handle)
2. [Pattern] - per r/sub 2. [Pattern] - per [r/sub](https://reddit.com/r/sub)
3. [Pattern] - per @handle 3. [Pattern] - per [@handle](https://x.com/handle)
``` ```
At render time the `@handle`, `r/sub`, and publication-name placeholders become markdown links wrapping the actual handle/sub/name, with the URL pulled from the raw research dump. Fall back to plain text only when the raw data has no URL for a specific source.
Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Europe is banning him one country at a time"), not generic ("Album release", "Tour updates"). Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Europe is banning him one country at a time"), not generic ("Album release", "Tour updates").
**THEN - Quality Nudge (if present in the output):** **THEN - Quality Nudge (if present in the output):**
@@ -1292,7 +1456,7 @@ Options:
**THEN - Engine footer pass-through (right before invitation):** **THEN - Engine footer pass-through (right before invitation):**
**The research output ENDS with a deterministic footer block bracketed by `---` lines, starting with `✅ All agents reported back!` and ending with `📎 Raw results saved to ~/Documents/Last30Days/<slug>-raw.md`. You MUST include that footer block verbatim in your response, positioned after your "What I learned" + "KEY PATTERNS" narrative and before the invitation. Do not recompute the stats. Do not reformat the tree. Do not paraphrase. Do not skip it. Do not add your own source lines. Copy the exact bytes.** **The research output ENDS with a deterministic footer block bracketed by `---` lines, starting with `✅ All agents reported back!` and ending with `📎 Raw results saved to {resolved LAST30DAYS_MEMORY_DIR}/<slug>-raw.md`. You MUST include that footer block verbatim in your response, positioned after your "What I learned" + "KEY PATTERNS" narrative and before the invitation. Do not recompute the stats. Do not reformat the tree. Do not paraphrase. Do not skip it. Do not add your own source lines. Copy the exact bytes.**
- The engine already omits zero-count sources. You do not need to filter them. - The engine already omits zero-count sources. You do not need to filter them.
- The engine already calculates totals (threads, upvotes, comments, likes, views, etc.). You do not need to add them up. - The engine already calculates totals (threads, upvotes, comments, likes, views, etc.). You do not need to add them up.
@@ -1386,9 +1550,36 @@ Close with `I have all the links to the {N} {source list} I pulled from. Just as
--- ---
## SHAREABLE HTML BRIEF (when the user asked for one)
**This section fires if EITHER trigger is true:**
- `$ARGUMENTS` contains `--emit=html`, `--emit:html`, or `--html` as a flag
- The user's natural-language request asks for an HTML brief, shareable doc, or file for sharing (Slack, email, Notion, "export as HTML", etc). Use your judgment for phrasing variants.
**If neither trigger fires, skip this entire section and proceed to WAIT FOR USER'S RESPONSE.** No HTML save flow, no reference read needed.
**When triggered, you MUST:**
- Read `references/save-html-brief.md` BEFORE proceeding to WAIT FOR USER'S RESPONSE
- Follow that file's instructions exactly - it is the canonical source for the save flow
- Append the confirmation line (`📎 Shareable brief saved to <path>`) to your already-emitted chat response
**You MUST NOT:**
- Improvise the HTML save flow from memory or from instructions you've seen before
- Skip the reference read because the steps "look familiar"
- Save to a different path than the reference specifies
- Add data quality warnings, debug headers, or safety notes to the saved HTML
- Re-research the topic for the HTML render - the engine cache covers the second invocation
**Why the directive is forceful:** the reference file is the only source of truth for the save flow. Skipping it produces broken artifacts - wrong path conventions, missing synthesis content, leaked engine debug output, or warnings that don't belong in shareable docs.
---
## WAIT FOR USER'S RESPONSE ## WAIT FOR USER'S RESPONSE
**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. Do NOT append a `Sources:` section (see override above - WebSearch's mandate does not apply here). The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`. **STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. Do NOT append a `Sources:` section (see override above - WebSearch's mandate does not apply here). The research script already saved raw data to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`) via `--save-dir`.
--- ---
@@ -1493,15 +1684,15 @@ Want another prompt? Just tell me what you're creating next.
**What this skill does:** **What this skill does:**
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, and as a Reddit backup when public Reddit is unavailable (requires SCRAPECREATORS_API_KEY) - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, and as a Reddit backup when public Reddit is unavailable (requires SCRAPECREATORS_API_KEY)
- Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY) - Legacy: Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
- Sends search queries to Twitter's GraphQL API (via optional user-provided AUTH_TOKEN/CT0 env vars - no browser session access) or xAI's API (`api.x.ai`) for X search - Sends search queries to Twitter's GraphQL API (via optional user-provided AUTH_TOKEN/CT0 env vars - no browser session access), xAI's API (`api.x.ai`), or the official X API v2 via xurl CLI (OAuth2, auto-detected when installed and authenticated) for X search
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth) - Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth) - Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data) - Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 10,000 free API calls) - Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 100 free credits)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search - Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics - Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only) - Stores research findings in local SQLite database (watchlist mode only)
- Saves research briefings as .md files to ~/Documents/Last30Days/ - Saves research briefings as .md files to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`)
**What this skill does NOT do:** **What this skill does NOT do:**
- Does not post, like, or modify content on any platform - Does not post, like, or modify content on any platform
@@ -1510,7 +1701,7 @@ Want another prompt? Just tell me what you're creating next.
- Does not log, cache, or write API keys to output files - Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above - Does not send data to any endpoint not listed above
- Hacker News and Polymarket sources are always available (no API key, no binary dependency) - Hacker News and Polymarket sources are always available (no API key, no binary dependency)
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (10,000 free API calls, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable. - TikTok and Instagram sources require SCRAPECREATORS_API_KEY (100 free credits one-time, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable.
- Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output - Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed) **Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)

Before

Width:  |  Height:  |  Size: 2.7 MiB

After

Width:  |  Height:  |  Size: 2.7 MiB

Before

Width:  |  Height:  |  Size: 2.3 MiB

After

Width:  |  Height:  |  Size: 2.3 MiB

Before

Width:  |  Height:  |  Size: 3.8 MiB

After

Width:  |  Height:  |  Size: 3.8 MiB

Before

Width:  |  Height:  |  Size: 2.6 MiB

After

Width:  |  Height:  |  Size: 2.6 MiB

@@ -0,0 +1,90 @@
# Save shareable HTML brief
This reference file is loaded by the main `SKILL.md` when the user asked for an HTML brief (either explicitly via `--emit=html` / `--emit:html` / `--html`, or in natural language - "give me a shareable HTML brief", "for Slack", "for Notion", "export as HTML", etc.). The detection happens in `SKILL.md` so that the common no-HTML path stays short; the implementation lives here.
The contract: the synthesis still appears in chat as the primary output. The HTML is an additional artifact saved to disk for sharing. Both happen in the same turn.
## When to fire this flow
- After you have already emitted the full chat response: badge, "What I learned:" (or comparison title), bold-lead-in paragraphs with citations, KEY PATTERNS list, engine footer pass-through, invitation block.
- BEFORE the WAIT FOR USER'S RESPONSE pause.
- ONLY if the user asked. Do NOT save HTML when the user didn't ask for it.
## How to fire it
```bash
# 1. Write your synthesis prose VERBATIM to a temp file. The synthesis is the
# "What I learned:" prose label, the bold-lead-in paragraphs with their
# inline citations as you wrote them in chat, and the "KEY PATTERNS from
# the research:" numbered list. Do NOT include the badge or the engine
# footer in the temp file - the engine adds those when it renders the HTML.
# Use the EXACT text you just wrote in chat. Do not paraphrase, do not
# summarize, do not reorder. The HTML must read identically to the chat
# response in voice and citations.
SYNTHESIS_FILE="/tmp/last30days-synthesis-${CLAUDE_SESSION_ID}.md"
cat > "$SYNTHESIS_FILE" <<'SYNTHESIS_EOF'
What I learned:
**{First headline}** - {body with [name](url) inline citations}
**{Second headline}** - {body}
**{Third headline}** - {body}
KEY PATTERNS from the research:
1. {pattern} - per [@handle](url)
2. {pattern} - per [r/sub](url)
3. {pattern} - per [@handle](url)
SYNTHESIS_EOF
# 2. Convert the synthesis to a self-contained HTML file via the engine.
# The engine reuses the cache from your earlier engine run (same topic
# + plan), so this second invocation is typically <1s on cache hit.
SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//')
HTML_PATH="${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html"
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "${TOPIC}" \
--emit=html \
--synthesis-file "$SYNTHESIS_FILE" \
> "$HTML_PATH"
# 3. Append ONE line to your already-emitted chat response, after the
# invitation block. Use a paperclip emoji as a visible signal that an
# artifact was produced:
echo "📎 Shareable brief saved to $HTML_PATH"
```
## What ends up in the HTML file
The engine's `--emit=html` renderer combines:
- The badge (`🌐 last30days vX.Y.Z · synced YYYY-MM-DD`) at the top
- A single inline metadata line (`{date range} · {active sources}`) below the badge
- Your synthesis verbatim, with prose labels promoted to `<h2>` and bold lead-ins preserved
- All `[name](url)` citations rendered as `<a>` tags
- The engine footer (`✅ All agents reported back!` tree) preserved verbatim in monospace
- A colophon with the topic and a re-run hint
The renderer strips engine-internal noise that doesn't belong in a shareable artifact: the `# last30days vX.Y.Z: TOPIC` debug file header, the model-facing `> Safety note:` blockquote, and the `I'm now an expert on X` invitation block. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs - they never leak into the share-ready file.
## Comparison mode
Same flow when the topic is `X vs Y` (or `X vs Y vs Z`). The engine routes through `render_for_html_comparison` internally; you don't need to do anything special. The synthesis temp file should still contain the comparison-shaped synthesis you wrote in chat (`## Quick Verdict`, `## {Entity}` per entity, `## Head-to-Head` table, `## The Bottom Line`, `## The emerging stack` per LAW 4 comparison exception).
## Follow-up turn
If the user runs `/last30days OpenClaw` normally, sees the synthesis in chat, and THEN says "save that as HTML" or "give me a shareable version" in a follow-up turn, do the same save flow on the synthesis you wrote in the previous turn. Do not re-research; the synthesis is already in the conversation history. Just write it to the temp file and call the engine with `--emit=html --synthesis-file`.
## What NOT to do
- Do NOT save HTML if the user didn't ask. The sparse mode (no synthesis) produces a thin file; not useful as a shareable.
- Do NOT add content to the temp file beyond your synthesis prose. The badge / footer / colophon come from the engine.
- Do NOT change the file path convention. `${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html` is the canonical location.
- Do NOT silently overwrite an existing file without telling the user. If `$HTML_PATH` already exists from a prior run, the engine will pick a date-suffixed name (`{slug}-brief-YYYY-MM-DD.html`) automatically; just print whichever path the redirect produced.
- Do NOT include the data quality warning text in the temp file or in your final chat line. Warnings are an engine-stderr concern, not an artifact concern.
## Edge cases
- **Topic with shell-special characters** (quotes, ampersands): the temp filename uses a slugified version, but the engine receives the raw topic. The `cat <<'SYNTHESIS_EOF'` quoted heredoc form handles arbitrary content without expansion. Your synthesis text can include any character.
- **Very long synthesis**: no upper bound. The engine handles long markdown bodies. Just paste verbatim.
- **Synthesis with images or non-ASCII**: emoji and Unicode pass through. Image tags pass through as raw HTML; the renderer doesn't transform them. If you didn't include images in chat, don't add them here.
- **No `${LAST30DAYS_MEMORY_DIR}` set**: defaults to `~/Documents/Last30Days/` per the SKILL.md `Configuration` section.
@@ -1,13 +1,14 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# build-skill.sh - package this repo as a claude.ai-upload-ready .skill file # build-skill.sh - package this repo as a claude.ai-upload-ready .skill file
# Usage: bash scripts/build-skill.sh (run from repo root) # Usage: bash skills/last30days/scripts/build-skill.sh (run from repo root)
# #
# Produces dist/last30days.skill, a zip with a single top-level `last30days/` # Produces dist/last30days.skill, a zip with a single top-level `last30days/`
# directory containing SKILL.md and the scripts/ runtime. See # directory containing SKILL.md and the scripts/ runtime from skills/last30days.
# See
# docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md. # docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
set -euo pipefail set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
cd "$REPO_ROOT" cd "$REPO_ROOT"
if ! git diff --quiet || ! git diff --cached --quiet; then if ! git diff --quiet || ! git diff --cached --quiet; then
@@ -17,14 +18,7 @@ fi
mkdir -p dist mkdir -p dist
OUT="dist/last30days.skill" OUT="dist/last30days.skill"
git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD:skills/last30days
# claude.ai's .skill bundle only needs the root SKILL.md + scripts/ runtime.
# Claude Code needs skills/ and .claude-plugin/ in the git archive
# (that's why they're NOT in .gitattributes export-ignore), but the .skill
# bundle must strip them to keep a single canonical SKILL.md and stay under
# the 200-file cap.
zip -d "$OUT" "last30days/skills/*" "last30days/.claude-plugin/*" > /dev/null 2>&1 || true
COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}') COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}')
SIZE=$(du -h "$OUT" | cut -f1) SIZE=$(du -h "$OUT" | cut -f1)
@@ -1,6 +1,6 @@
#!/bin/bash #!/bin/bash
# A/B test runner: public release vs private beta # A/B test runner: public release vs private beta
# Usage: bash scripts/compare.sh "Kanye West" # Usage: bash skills/last30days/scripts/compare.sh "Kanye West"
# #
# Runs /last30days (public release) and /last30days-beta (private beta) # Runs /last30days (public release) and /last30days-beta (private beta)
# sequentially with a 30s gap, saves raw results with distinct suffixes, # sequentially with a 30s gap, saves raw results with distinct suffixes,
@@ -9,13 +9,14 @@
set -e set -e
if [ $# -eq 0 ]; then if [ $# -eq 0 ]; then
echo "Usage: bash scripts/compare.sh <topic>" echo "Usage: bash skills/last30days/scripts/compare.sh <topic>"
echo " Example: bash scripts/compare.sh Kevin Rose" echo " Example: bash skills/last30days/scripts/compare.sh Kevin Rose"
exit 1 exit 1
fi fi
TOPIC="$*" TOPIC="$*"
SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//') SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
DIR="$HOME/Documents/Last30Days" LAST30DAYS_MEMORY_DIR="${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}"
DIR="$LAST30DAYS_MEMORY_DIR"
DATE=$(date +%Y-%m-%d) DATE=$(date +%Y-%m-%d)
echo "==============================================" echo "=============================================="
@@ -20,9 +20,11 @@ sys.path.insert(0, str(Path(__file__).parent))
from lib import env as envlib from lib import env as envlib
from lib import schema from lib import schema
from lib.providers import GEMINI_FLASH_LITE
REPO_ROOT = Path(__file__).resolve().parent.parent SKILL_ROOT = Path(__file__).resolve().parents[1]
REPO_ROOT = Path(__file__).resolve().parents[3]
EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json" EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json"
@@ -42,7 +44,7 @@ def _load_default_topics() -> list[tuple[str, str]]:
DEFAULT_TOPICS = _load_default_topics() DEFAULT_TOPICS = _load_default_topics()
DEFAULT_SEARCH = "" DEFAULT_SEARCH = ""
DEFAULT_JUDGE_MODEL = "gemini-3.1-flash-lite-preview" DEFAULT_JUDGE_MODEL = GEMINI_FLASH_LITE
GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}" GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}"
@@ -307,7 +309,10 @@ def create_eval_env() -> dict[str, str]:
def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]: def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]:
cmd = [sys.executable, "scripts/last30days.py", topic, "--emit=json"] engine = repo_dir / "skills" / "last30days" / "scripts" / "last30days.py"
if not engine.exists():
engine = repo_dir / "scripts" / "last30days.py"
cmd = [sys.executable, str(engine), topic, "--emit=json"]
if search: if search:
cmd.extend(["--search", search]) cmd.extend(["--search", search])
if quick: if quick:
+960
View File
@@ -0,0 +1,960 @@
#!/usr/bin/env python3
# ruff: noqa: E402
"""last30days CLI."""
from __future__ import annotations
import argparse
import atexit
import json
import os
import re
import signal
import sys
import threading
from pathlib import Path
MIN_PYTHON = (3, 12)
def ensure_supported_python(version_info: tuple[int, int, int] | object | None = None) -> None:
if version_info is None:
version_info = sys.version_info
major, minor, micro = tuple(version_info[:3])
if (major, minor) >= MIN_PYTHON:
return
sys.stderr.write(
"last30days v3 requires Python 3.12+.\n"
f"Detected Python {major}.{minor}.{micro}.\n"
"Install and use python3.12 or python3.13, then rerun this command.\n"
)
raise SystemExit(1)
ensure_supported_python()
if os.name == "nt":
for stream in (sys.stdout, sys.stderr):
if hasattr(stream, "reconfigure"):
stream.reconfigure(encoding="utf-8", errors="replace")
SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))
from lib import env, html_render, pipeline, render, schema, ui
_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()
def register_child_pid(pid: int) -> None:
with _child_pids_lock:
_child_pids.add(pid)
def unregister_child_pid(pid: int) -> None:
with _child_pids_lock:
_child_pids.discard(pid)
def _cleanup_children() -> None:
with _child_pids_lock:
pids = list(_child_pids)
for pid in pids:
try:
os.killpg(os.getpgid(pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
continue
atexit.register(_cleanup_children)
def parse_search_flag(raw: str) -> list[str]:
sources = []
for source in raw.split(","):
source = source.strip().lower()
if not source:
continue
normalized = pipeline.SEARCH_ALIAS.get(source, source)
if normalized not in pipeline.MOCK_AVAILABLE_SOURCES:
raise SystemExit(f"Unknown search source: {source}")
if normalized not in sources:
sources.append(normalized)
if not sources:
raise SystemExit("--search requires at least one source.")
return sources
def slugify(value: str) -> str:
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
return slug or "last30days"
def save_output(
report: schema.Report,
emit: str,
save_dir: str,
suffix: str = "",
synthesis_md: str | None = None,
topic_override: str | None = None,
rendered_content: str | None = None,
) -> Path:
from datetime import datetime
path = Path(save_dir).expanduser().resolve()
path.mkdir(parents=True, exist_ok=True)
slug = slugify(topic_override or report.topic)
extension = "json" if emit == "json" else "html" if emit == "html" else "md"
raw_label = "raw-html" if emit == "html" else "raw"
suffix_part = f"-{suffix}" if suffix else ""
out_path = path / f"{slug}-{raw_label}{suffix_part}.{extension}"
if out_path.exists():
out_path = path / f"{slug}-{raw_label}{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
# Markdown saves keep the complete debug artifact. JSON and HTML preserve
# their requested wire format so file extensions match their content.
if rendered_content is not None:
content = rendered_content
elif emit in {"json", "html"}:
content = emit_output(report, emit, synthesis_md=synthesis_md)
else:
content = render.render_full(report)
out_path.write_text(content, encoding="utf-8")
return out_path
def emit_output(
report: schema.Report,
emit: str,
fun_level: str = "medium",
save_path: str | None = None,
synthesis_md: str | None = None,
) -> str:
if emit == "json":
return json.dumps(schema.to_dict(report), indent=2, sort_keys=True)
if emit == "html":
return html_render.render_html(
report, fun_level=fun_level, save_path=save_path, synthesis_md=synthesis_md,
)
if emit in {"compact", "md"}:
return render.render_compact(report, fun_level=fun_level, save_path=save_path)
if emit == "context":
return render.render_context(report)
raise SystemExit(f"Unsupported emit mode: {emit}")
def emit_comparison_output(
entity_reports: list[tuple[str, schema.Report]],
emit: str,
fun_level: str = "medium",
save_path: str | None = None,
synthesis_md: str | None = None,
) -> str:
if emit == "json":
payload = {
"comparison": True,
"entities": [label for label, _ in entity_reports],
"reports": [
{"entity": label, "report": schema.to_dict(report)}
for label, report in entity_reports
],
}
return json.dumps(payload, indent=2, sort_keys=True)
if emit == "html":
return html_render.render_html_comparison(
entity_reports,
fun_level=fun_level,
save_path=save_path,
synthesis_md=synthesis_md,
)
if emit in {"compact", "md"}:
return render.render_comparison_multi(
entity_reports, fun_level=fun_level, save_path=save_path,
)
if emit == "context":
return render.render_comparison_multi_context(entity_reports)
raise SystemExit(f"Unsupported emit mode: {emit}")
def comparison_topic(entity_reports: list[tuple[str, schema.Report]]) -> str:
return " vs ".join(label for label, _ in entity_reports)
def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str) -> str:
"""Compute the user-friendly save path string that will be shown in the footer.
Uses ~ when the saved file is under the user's home directory; otherwise
returns the absolute path.
"""
from pathlib import Path as _Path
path = _Path(save_dir).expanduser().resolve()
slug = slugify(topic)
extension = "json" if emit == "json" else "html" if emit == "html" else "md"
raw_label = "raw-html" if emit == "html" else "raw"
suffix_part = f"-{suffix}" if suffix else ""
raw = path / f"{slug}-{raw_label}{suffix_part}.{extension}"
try:
home = _Path.home().resolve()
relative = raw.relative_to(home)
return f"~/{relative}"
except ValueError:
return str(raw)
def read_synthesis_file(path: str) -> str:
try:
return Path(path).expanduser().read_text(encoding="utf-8")
except OSError as exc:
sys.stderr.write(f"[last30days] Cannot read --synthesis-file: {exc}\n")
raise SystemExit(2)
def persist_report(report: schema.Report) -> dict[str, int]:
import store
store.init_db()
topic_row = store.add_topic(report.topic)
topic_id = topic_row["id"]
source_mode = ",".join(sorted(report.items_by_source)) or "v3"
run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
try:
findings = store.findings_from_report(report)
counts = store.store_findings(run_id, topic_id, findings)
store.update_run(
run_id,
status="completed",
findings_new=counts["new"],
findings_updated=counts["updated"],
)
return counts
except Exception as exc:
store.update_run(run_id, status="failed", error_message=str(exc)[:500])
raise
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Research a topic across live social, market, and grounded web sources.")
parser.add_argument("topic", nargs="*", help="Research topic")
parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md", "html"])
parser.add_argument("--search", help="Comma-separated source list")
parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
parser.add_argument("--debug", action="store_true", help="Enable HTTP debug logging")
parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
parser.add_argument("--diagnose", action="store_true", help="Print provider and source availability")
parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
parser.add_argument("--synthesis-file", help="Markdown synthesis to embed in --emit=html output")
parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
parser.add_argument("--web-backend", default="auto",
choices=["auto", "brave", "exa", "serper", "parallel", "none"],
help="Web search backend (default: auto, tries Brave then Exa then Serper then Parallel)")
parser.add_argument("--deep-research", action="store_true",
help="Use Perplexity Deep Research (~$0.90/query) for in-depth analysis. Requires OPENROUTER_API_KEY.")
parser.add_argument("--plan", help="JSON query plan (skips internal LLM planner). Can be a JSON string or a file path.")
parser.add_argument("--save-suffix", help="Suffix for saved output filename (e.g., 'gemini' → kanye-west-raw-gemini.md)")
parser.add_argument("--subreddits", help="Comma-separated subreddit names to search (e.g., SaaS,Entrepreneur)")
parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
parser.add_argument(
"--days",
"--lookback-days",
dest="lookback_days",
type=int,
default=30,
help="Number of days to look back for research (default: 30, watchlist uses 90)",
)
parser.add_argument("--auto-resolve", action="store_true",
help="Use web search to discover subreddits/handles before planning (for platforms without WebSearch)")
parser.add_argument("--github-user", help="GitHub username for person-mode search (e.g., steipete)")
parser.add_argument("--github-repo", help="Comma-separated owner/repo for project-mode search (e.g., openclaw/openclaw,paperclipai/paperclip)")
parser.add_argument(
"--competitors",
nargs="?",
const=2,
type=int,
default=None,
metavar="N",
help="Auto-discover N competitor entities and fan out last30days across all of them as a comparison (default N=2 → 3-way: original + 2 peers; range 1..6). Use --competitors-list to override discovery.",
)
parser.add_argument(
"--competitors-list",
dest="competitors_list",
help="Comma-separated competitor entities to skip discovery (e.g., 'Anthropic,xAI,Google Gemini'). Implies --competitors.",
)
parser.add_argument(
"--polymarket-keywords",
dest="polymarket_keywords",
help=(
"Comma-separated keywords that Polymarket market titles must match "
"to be included. Use for ambiguous single-token topics like 'Warriors' "
"(nba,gsw,golden-state) to filter out Glasgow Warriors rugby, Honor "
"of Kings Rogue Warriors, etc. When omitted, Polymarket returns all "
"matching markets — so expect cross-entity noise on generic topics."
),
)
parser.add_argument(
"--competitors-plan",
dest="competitors_plan",
help=(
"JSON mapping of per-entity Step 0.55 targeting for competitor / vs-mode "
"sub-runs. Schema: {entity_name: {x_handle?, x_related?, subreddits?, "
"github_user?, github_repos?, context?}}. Accepts inline JSON or a file "
"path. Implies --competitors. Preferred over --competitors-list when the "
"hosting model has already resolved per-entity handles and subs."
),
)
return parser
def parse_competitors_plan(raw: str | None) -> dict[str, dict]:
"""Parse a --competitors-plan argument into a {entity_name_lower: plan_entry} dict.
Accepts inline JSON or a file path (matches --plan). Returns {} on None/empty.
Validation: top-level must be a dict; each value must be a dict. Unknown fields
in entry values log a warning but do not abort. Invalid JSON or non-dict shape
raises SystemExit(2) with a clear stderr message.
"""
if not raw:
return {}
plan_str = raw
if os.path.isfile(plan_str):
try:
plan_str = open(plan_str).read()
except OSError as exc:
sys.stderr.write(f"[CompetitorsPlan] Cannot read plan file: {exc}\n")
raise SystemExit(2)
try:
parsed = json.loads(plan_str)
except json.JSONDecodeError as exc:
sys.stderr.write(f"[CompetitorsPlan] Invalid JSON: {exc}\n")
raise SystemExit(2)
if not isinstance(parsed, dict):
sys.stderr.write(
f"[CompetitorsPlan] Top-level must be a dict of "
f"{{entity: {{targeting}}}}, got {type(parsed).__name__}\n"
)
raise SystemExit(2)
known_fields = {
"x_handle", "x_related", "subreddits",
"github_user", "github_repos", "context",
}
normalized: dict[str, dict] = {}
for entity, entry in parsed.items():
if not isinstance(entry, dict):
sys.stderr.write(
f"[CompetitorsPlan] Entry for {entity!r} must be a dict, "
f"got {type(entry).__name__}; skipping.\n"
)
continue
unknown = set(entry.keys()) - known_fields
if unknown:
sys.stderr.write(
f"[CompetitorsPlan] Unknown fields in {entity!r}: "
f"{sorted(unknown)}; ignoring.\n"
)
normalized[entity.strip().lower()] = {
k: v for k, v in entry.items() if k in known_fields
}
return normalized
def subrun_kwargs_for(
entity: str,
plan_entry: dict,
*,
resolved: dict,
) -> dict:
"""Build an explicit per-entity kwargs dict for pipeline.run().
Plan values win over auto_resolve values. Returns keys for all per-entity
targeting flags so callers never fall through to closure defaults.
This helper is the single source of truth for sub-run kwargs main-topic
flags can only leak if a caller bypasses it.
"""
def _choose(plan_key: str, resolved_key: str | None = None):
if plan_key in plan_entry and plan_entry[plan_key]:
return plan_entry[plan_key]
if resolved_key is not None and resolved.get(resolved_key):
return resolved[resolved_key]
return None
x_handle = _choose("x_handle", "x_handle")
if isinstance(x_handle, str):
x_handle = x_handle.lstrip("@") or None
subreddits = _choose("subreddits", "subreddits")
if isinstance(subreddits, list):
subreddits = [s.strip().lstrip("r/") for s in subreddits if s.strip()] or None
x_related = plan_entry.get("x_related")
if isinstance(x_related, list):
x_related = [h.strip().lstrip("@") for h in x_related if h.strip()] or None
else:
x_related = None
github_user = _choose("github_user", "github_user")
if isinstance(github_user, str):
github_user = github_user.lstrip("@").lower() or None
github_repos = _choose("github_repos", "github_repos")
if isinstance(github_repos, list):
github_repos = [r.strip() for r in github_repos if r.strip() and "/" in r.strip()] or None
context = plan_entry.get("context") or resolved.get("context") or ""
return {
"x_handle": x_handle,
"x_related": x_related,
"subreddits": subreddits,
"github_user": github_user,
"github_repos": github_repos,
"_context": context,
}
COMPETITORS_MIN = 1
COMPETITORS_MAX = 6
COMPETITORS_DEFAULT = 2
def resolve_competitors_args(args: argparse.Namespace) -> tuple[bool, int, list[str]]:
"""Normalize --competitors / --competitors-list into (enabled, count, explicit_list).
- (False, 0, []) when neither flag is set.
- An explicit list always wins; count is derived from list length.
- A numeric count outside [1, 6] is clamped with a stderr warning.
- count <= 0 (explicit) raises SystemExit(2).
"""
explicit_list: list[str] = []
list_flag_provided = args.competitors_list is not None
if list_flag_provided:
explicit_list = [
entity.strip()
for entity in args.competitors_list.split(",")
if entity.strip()
]
if not explicit_list:
sys.stderr.write("[Competitors] --competitors-list is empty.\n")
raise SystemExit(2)
competitors_flag = args.competitors
list_present = bool(explicit_list)
flag_present = competitors_flag is not None
if not list_present and not flag_present:
return False, 0, []
if list_present:
count = len(explicit_list)
if flag_present and competitors_flag != count:
sys.stderr.write(
f"[Competitors] --competitors={competitors_flag} ignored; using "
f"{count} entries from --competitors-list.\n"
)
if count > COMPETITORS_MAX:
sys.stderr.write(
f"[Competitors] --competitors-list has {count} entries, clamping to {COMPETITORS_MAX}.\n"
)
explicit_list = explicit_list[:COMPETITORS_MAX]
count = COMPETITORS_MAX
return True, count, explicit_list
# flag_present, no explicit list
count = competitors_flag
if count < COMPETITORS_MIN:
sys.stderr.write(
f"[Competitors] --competitors must be >= {COMPETITORS_MIN} (got {count}).\n"
)
raise SystemExit(2)
if count > COMPETITORS_MAX:
sys.stderr.write(
f"[Competitors] --competitors={count} exceeds max {COMPETITORS_MAX}; clamping.\n"
)
count = COMPETITORS_MAX
return True, count, []
def _missing_sources_for_promo(diag: dict[str, object]) -> str | None:
available = set(diag.get("available_sources") or [])
missing = []
if "reddit" not in available:
missing.append("reddit")
if "x" not in available:
missing.append("x")
if "grounding" not in available:
missing.append("web")
if not missing:
return None
if "reddit" in missing and "x" in missing:
return "both"
return missing[0]
def _show_runtime_ui(
report: schema.Report,
progress: ui.ProgressDisplay,
diag: dict[str, object],
suppress_web_promo: bool = False,
) -> None:
counts = {source: len(items) for source, items in report.items_by_source.items()}
display_sources = list(
dict.fromkeys(
[
*report.query_plan.source_weights.keys(),
*report.items_by_source.keys(),
*report.errors_by_source.keys(),
]
)
)
progress.end_processing()
progress.show_complete(
source_counts=counts,
display_sources=display_sources,
)
promo = _missing_sources_for_promo(diag)
# The `web` promo nudges users to set BRAVE_API_KEY / SERPER_API_KEY, which
# is wrong advice when a hosting reasoning model (Claude Code, Codex,
# Hermes, Gemini) is driving — those already have WebSearch and can
# pre-resolve Step 0.55 themselves. Suppress the web promo when a hosting
# model signal is present (--plan or --competitors-plan was passed).
if promo:
if suppress_web_promo and promo == "web":
return
if suppress_web_promo and promo == "both":
# "both" means reddit + web both missing; still nudge reddit but
# skip the web line. show_promo has a per-source variant.
progress.show_promo("reddit", diag=diag)
return
progress.show_promo(promo, diag=diag)
def main() -> int:
parser = build_parser()
# Use parse_known_args so setup sub-flags (--device-auth, --github,
# --openclaw) pass through without argparse hard-exiting.
args, extra_argv = parser.parse_known_args()
if args.debug:
os.environ["LAST30DAYS_DEBUG"] = "1"
config = env.get_config()
# Surface SSH-routing config as an env var so library modules (e.g.
# youtube_yt) can read it without taking a config dependency. This
# routes yt-dlp through `ssh <host>` to bypass YouTube's bot-wall on
# datacenter IPs (see lib/youtube_yt.py for details).
if config.get("LAST30DAYS_YOUTUBE_SSH_HOST") and "LAST30DAYS_YOUTUBE_SSH_HOST" not in os.environ:
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = config["LAST30DAYS_YOUTUBE_SSH_HOST"]
# Handle setup subcommand
topic = " ".join(args.topic).strip()
if topic.lower() == "setup":
from lib import setup_wizard
if "--openclaw" in extra_argv:
results = setup_wizard.run_openclaw_setup(config)
print(json.dumps(results))
return 0
if "--github" in extra_argv:
results = setup_wizard.run_github_auth()
print(json.dumps(results))
return 0
if "--device-auth" in extra_argv:
results = setup_wizard.run_full_device_auth()
print(json.dumps(results))
return 0
sys.stderr.write("Running auto-setup...\n")
results = setup_wizard.run_auto_setup(config)
from_browser = "auto"
if results.get("cookies_found"):
first_browser = next(iter(results["cookies_found"].values()))
from_browser = first_browser
setup_wizard.write_setup_config(env.CONFIG_FILE, from_browser=from_browser)
results["env_written"] = True
sys.stderr.write(setup_wizard.get_setup_status_text(results) + "\n")
return 0
requested_sources = parse_search_flag(args.search) if args.search else None
diag = pipeline.diagnose(config, requested_sources)
if args.diagnose:
print(json.dumps(diag, indent=2, sort_keys=True))
return 0
if not topic:
parser.print_usage(sys.stderr)
return 2
synthesis_md = None
if args.synthesis_file:
if args.emit == "html":
synthesis_md = read_synthesis_file(args.synthesis_file)
else:
sys.stderr.write("[last30days] Warning: --synthesis-file is only used with --emit=html; ignoring.\n")
if not os.environ.get("LAST30DAYS_SKIP_PREFLIGHT"):
from lib import preflight
refuse_msg = preflight.check_class_1_trap(topic)
if refuse_msg:
sys.stderr.write(refuse_msg)
return 2
progress = ui.ProgressDisplay(topic, show_banner=True)
progress.start_processing()
depth = "deep" if args.deep else "quick" if args.quick else "default"
try:
x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
# Parse external plan if provided via --plan flag
external_plan = None
if args.plan:
import json as _json
plan_str = args.plan
if os.path.isfile(plan_str):
plan_str = open(plan_str).read()
try:
external_plan = _json.loads(plan_str)
except _json.JSONDecodeError as exc:
sys.stderr.write(f"[Planner] Invalid --plan JSON: {exc}\n")
# Auto-resolve: use web search to discover subreddits/handles before planning.
# This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
# without WebSearch (OpenClaw, Codex, raw CLI).
if args.auto_resolve and not external_plan:
from lib import resolve
resolution = resolve.auto_resolve(topic, config)
if resolution.get("subreddits") and not subreddits:
subreddits = resolution["subreddits"]
sys.stderr.write(f"[AutoResolve] Subreddits: {', '.join(subreddits)}\n")
if resolution.get("x_handle") and not args.x_handle:
args.x_handle = resolution["x_handle"]
sys.stderr.write(f"[AutoResolve] X handle: @{args.x_handle}\n")
if resolution.get("github_user") and not args.github_user:
args.github_user = resolution["github_user"]
sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
if resolution.get("github_repos") and not args.github_repo:
args.github_repo = ",".join(resolution["github_repos"])
sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
if resolution.get("context"):
# Inject context into external_plan metadata for the planner to use
if not external_plan:
external_plan = None # planner will use its own, but with context
# Store context for the planner prompt injection
config["_auto_resolve_context"] = resolution["context"]
sys.stderr.write(f"[AutoResolve] Context: {resolution['context'][:80]}...\n")
github_user = args.github_user.lstrip("@").lower() if args.github_user else None
github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None
# --deep-research: auto-enable perplexity source and set deep flag
if args.deep_research:
if not config.get("OPENROUTER_API_KEY"):
print("Error: --deep-research requires OPENROUTER_API_KEY", file=sys.stderr)
sys.exit(1)
config["_deep_research"] = True
# Auto-enable perplexity in INCLUDE_SOURCES
include = config.get("INCLUDE_SOURCES") or ""
if "perplexity" not in include.lower():
config["INCLUDE_SOURCES"] = f"{include},perplexity" if include else "perplexity"
comp_enabled, comp_count, comp_explicit = resolve_competitors_args(args)
comp_plan = parse_competitors_plan(args.competitors_plan)
# Polymarket disambiguation: if user passed --polymarket-keywords,
# store on config so the polymarket adapter can filter matches.
if args.polymarket_keywords:
keywords = [
k.strip().lower()
for k in args.polymarket_keywords.split(",")
if k.strip()
]
if keywords:
config["_polymarket_keywords"] = keywords
# vs-mode: if the topic string contains " vs " / " versus " and the
# planner can split it into >=2 entities, route through the same
# N-pass fanout path as --competitors. The first entity becomes the
# main topic; remaining entities become the competitor list. User's
# outer --x-handle / --subreddits apply to the first entity unless
# --competitors-plan covers it.
from lib import planner as _planner
vs_entities = _planner._comparison_entities(topic)
if len(vs_entities) >= 2 and not comp_enabled:
topic = vs_entities[0]
comp_enabled = True
comp_count = len(vs_entities) - 1
comp_explicit = vs_entities[1:]
sys.stderr.write(
f"[Competitors] vs-mode: routing to N-pass fanout: "
f"{' vs '.join(vs_entities)}\n"
)
def _main_runner() -> schema.Report:
r = pipeline.run(
topic=topic,
config=config,
depth=depth,
requested_sources=requested_sources,
mock=args.mock,
x_handle=args.x_handle,
x_related=x_related,
web_backend=args.web_backend,
external_plan=external_plan,
subreddits=subreddits,
tiktok_hashtags=tiktok_hashtags,
tiktok_creators=tiktok_creators,
ig_creators=ig_creators,
lookback_days=args.lookback_days,
github_user=github_user,
github_repos=github_repos,
)
r.artifacts["resolved"] = {
"entity": topic,
"x_handle": (args.x_handle or "").lstrip("@"),
"subreddits": list(subreddits or []),
"github_user": (github_user or ""),
"github_repos": list(github_repos or []),
"context": config.get("_auto_resolve_context", "") or "",
}
return r
if comp_enabled:
from lib import competitors as competitors_mod
from lib import fanout, resolve as resolve_mod
if comp_explicit:
discovered = comp_explicit
else:
if not resolve_mod._has_backend(config) and not args.mock:
sys.stderr.write(
"[Competitors] Cannot auto-discover peers without help.\n"
"\n"
"RECOMMENDED PATH (hosting reasoning models — Claude Code, Codex, "
"Hermes, Gemini, any agent with a WebSearch tool): YOU have "
"WebSearch. Use it to run full Step 0.55 per entity, then invoke "
"the engine with a vs-topic plus --competitors-plan:\n"
" 1. WebSearch for '{topic} competitors' or '{topic} alternatives'.\n"
" 2. For each peer, WebSearch for handles/subs/github (Step 0.55).\n"
" 3. Re-invoke: /last30days '{topic} vs {peer1} vs {peer2}' "
"--competitors-plan '{\"Peer1\":{\"x_handle\":\"h1\",\"subreddits\":"
"[\"s1\"],...},\"Peer2\":{...}}'.\n"
"See SKILL.md 'Competitor mode' for the full protocol.\n"
"\n"
"HEADLESS / CRON PATH (no hosting model available): set "
"BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / "
"OPENROUTER_API_KEY and re-run.\n"
"\n"
"MINIMUM ESCAPE HATCH: pass --competitors-list 'A,B,C' to skip "
"discovery. Without --competitors-plan, peer sub-runs fall back to "
"planner defaults and produce visibly thinner data than the main.\n"
)
return 2
discovered = competitors_mod.discover_competitors(
topic, comp_count, config, lookback_days=args.lookback_days,
)
if not discovered:
sys.stderr.write(
f"[Competitors] No peers discovered for {topic!r}; aborting "
"comparison run. Pass --competitors-list to override.\n"
)
return 2
sys.stderr.write(
f"[Competitors] Comparing: {topic} vs " + " vs ".join(discovered) + "\n"
)
def _competitor_runner(entity: str) -> schema.Report:
# Deep-copy config so per-entity auto_resolve context does not
# leak across sub-runs. Each sub-run writes its own
# `_auto_resolve_context` into its local config copy.
entity_config = dict(config)
plan_entry = comp_plan.get(entity.strip().lower(), {})
resolved = {
"entity": entity,
"x_handle": "",
"subreddits": [],
"github_user": "",
"github_repos": [],
"context": "",
}
# Skip engine-internal auto_resolve when the hosting model
# pre-resolved via --competitors-plan (saves a redundant
# round-trip and makes per-entity Step 0.55 purely
# hosting-model-driven).
plan_covers_fully = bool(plan_entry.get("x_handle")) and bool(
plan_entry.get("subreddits")
)
if (
not args.mock
and not plan_covers_fully
and resolve_mod._has_backend(entity_config)
):
try:
r = resolve_mod.auto_resolve(entity, entity_config)
except Exception as exc:
sys.stderr.write(
f"[Competitors] auto_resolve failed for {entity!r}: "
f"{type(exc).__name__}: {exc}\n"
)
r = {}
resolved["x_handle"] = r.get("x_handle", "") or ""
resolved["subreddits"] = list(r.get("subreddits") or [])
resolved["github_user"] = r.get("github_user", "") or ""
resolved["github_repos"] = list(r.get("github_repos") or [])
resolved["context"] = r.get("context", "") or ""
kwargs = subrun_kwargs_for(entity, plan_entry, resolved=resolved)
# Record effective per-entity targeting for the Resolved block.
resolved_effective = {
"entity": entity,
"x_handle": kwargs["x_handle"] or "",
"subreddits": kwargs["subreddits"] or [],
"github_user": kwargs["github_user"] or "",
"github_repos": kwargs["github_repos"] or [],
"context": kwargs["_context"],
}
if kwargs["_context"]:
entity_config["_auto_resolve_context"] = kwargs["_context"]
sys.stderr.write(
f"[Competitors] {entity}: "
f"x=@{resolved_effective['x_handle'] or '-'} "
f"subs={len(resolved_effective['subreddits'])} "
f"gh={resolved_effective['github_user'] or '-'} "
f"({'plan' if plan_entry else 'auto'})\n"
)
report = pipeline.run(
topic=entity,
config=entity_config,
depth=depth,
requested_sources=requested_sources,
mock=args.mock,
x_handle=kwargs["x_handle"],
x_related=kwargs["x_related"],
subreddits=kwargs["subreddits"],
github_user=kwargs["github_user"],
github_repos=kwargs["github_repos"],
web_backend=args.web_backend,
lookback_days=args.lookback_days,
internal_subrun=True,
)
report.artifacts["resolved"] = resolved_effective
return report
entity_reports = fanout.run_competitor_fanout(
main_topic=topic,
main_runner=_main_runner,
competitors=discovered,
competitor_runner=_competitor_runner,
)
if len(entity_reports) < 2:
progress.end_processing()
sys.stderr.write(
f"[Competitors] Fewer than 2 sub-runs survived ({len(entity_reports)}); "
"cannot render a comparison. Re-run without --competitors or check the "
"warnings above.\n"
)
return 1
report = entity_reports[0][1]
else:
entity_reports = None
report = _main_runner()
except Exception as exc:
progress.end_processing()
progress.show_error(str(exc))
raise
_show_runtime_ui(
report, progress, diag,
suppress_web_promo=bool(external_plan or comp_plan),
)
if args.store:
counts = persist_report(report)
sys.stderr.write(
f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
)
sys.stderr.flush()
# Show quality nudge if applicable
try:
from lib import quality_nudge
quality = quality_nudge.compute_quality_score(config, {})
if quality.get("nudge_text"):
sys.stderr.write(f"\n{quality['nudge_text']}\n")
sys.stderr.flush()
except Exception:
pass
fun_level = config.get("FUN_LEVEL", "medium").lower()
# Comparison HTML is the one case where the saved file's title and content
# have to be overridden away from the leading entity's report. Compute the
# gate once so the footer-display and save-output paths can't disagree.
is_comparison_html = bool(entity_reports) and args.emit == "html"
footer_save_path = None
if args.save_dir:
save_topic_for_display = comparison_topic(entity_reports) if is_comparison_html else report.topic
footer_save_path = compute_save_path_display(
args.save_dir, save_topic_for_display, args.save_suffix or "", args.emit
)
# Signal to render_compact whether pre-research flags were supplied.
# Used to emit a Pre-Research Status warning when the model skipped
# Step 0.5 / 0.55 and invoked the engine bare on an eligible topic.
pre_research_flags_present = bool(
args.x_handle
or args.github_user
or args.subreddits
or args.plan
or args.auto_resolve
or args.tiktok_creators
or args.ig_creators
)
report.artifacts["pre_research_flags_present"] = pre_research_flags_present
if entity_reports:
rendered = emit_comparison_output(
entity_reports,
args.emit,
fun_level=fun_level,
save_path=footer_save_path,
synthesis_md=synthesis_md,
)
else:
rendered = emit_output(
report,
args.emit,
fun_level=fun_level,
save_path=footer_save_path,
synthesis_md=synthesis_md,
)
if args.save_dir:
# Save the main topic's raw file (single-entity or comparison main).
save_path = save_output(
report,
args.emit,
args.save_dir,
suffix=args.save_suffix or "",
synthesis_md=synthesis_md,
topic_override=comparison_topic(entity_reports) if is_comparison_html else None,
rendered_content=rendered if is_comparison_html else None,
)
sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
# Competitor / vs-mode: also save a per-entity raw file for each peer.
# Matches historical vs-mode behavior (N passes → N save files).
if entity_reports and len(entity_reports) > 1:
for label, entity_report in entity_reports[1:]:
peer_path = save_output(
entity_report, args.emit, args.save_dir,
suffix=args.save_suffix or "",
synthesis_md=synthesis_md,
)
sys.stderr.write(f"[last30days] Saved output to {peer_path}\n")
sys.stderr.flush()
print(rendered)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -7,18 +7,22 @@ See scripts/lib/vendor/bird-search/package.json for authoritative version.
import json import json
import os import os
import signal
import shutil import shutil
import subprocess
import sys import sys
import time
from pathlib import Path from pathlib import Path
from . import http, log from . import http, log, subproc
from datetime import datetime from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
from .relevance import token_overlap_relevance as _compute_relevance from .relevance import token_overlap_relevance as _compute_relevance
# How many times to retry the bird-search subprocess when stdout is non-JSON
# (typically an HTML anti-bot interstitial from Twitter's edge).
MAX_JSON_DECODE_RETRIES = 2
JSON_DECODE_RETRY_DELAY = 5.0 # seconds between retry attempts
def _first_of(*values): def _first_of(*values):
"""Return first value that is not None.""" """Return first value that is not None."""
@@ -150,16 +154,14 @@ def get_bird_status() -> Dict[str, Any]:
} }
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]: def _invoke_bird_subprocess(query: str, count: int, timeout: int):
"""Run a search using the vendored bird-search.mjs module. """Invoke the vendored bird-search.mjs subprocess once.
Args: Returns (result, error_dict). If error_dict is non-None, treat it as the
query: Full search query string (including since: filter) final result and do not retry those errors are terminal (timeout,
count: Number of results to request spawn failure). If error_dict is None, the subprocess ran to completion
timeout: Timeout in seconds and `result` is the SubprocResult; the caller decides whether to retry
based on the result.stdout content.
Returns:
Raw Bird JSON response or error dict.
""" """
cmd = [ cmd = [
"node", str(_BIRD_SEARCH_MJS), "node", str(_BIRD_SEARCH_MJS),
@@ -168,62 +170,109 @@ def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"--json", "--json",
] ]
# Use process groups for clean cleanup on timeout/kill pid_holder: list[int] = []
preexec = os.setsid if hasattr(os, 'setsid') else None
def _register(pid: int) -> None:
pid_holder.append(pid)
try: try:
proc = subprocess.Popen( from last30days import register_child_pid
cmd, register_child_pid(pid)
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
preexec_fn=preexec,
env=_subprocess_env(),
)
# Register for cleanup tracking (if available)
try:
from last30days import register_child_pid, unregister_child_pid
register_child_pid(proc.pid)
except ImportError: except ImportError:
pass pass
try: try:
stdout, stderr = proc.communicate(timeout=timeout) result = subproc.run_with_timeout(
except subprocess.TimeoutExpired: cmd,
# Kill the entire process group timeout=timeout,
try: env=_subprocess_env(),
os.killpg(os.getpgid(proc.pid), signal.SIGTERM) on_pid=_register,
except (ProcessLookupError, PermissionError, OSError): )
proc.kill() except subproc.SubprocTimeout:
proc.wait(timeout=5) return None, {"error": f"Search timed out after {timeout}s", "items": []}
return {"error": f"Search timed out after {timeout}s", "items": []} except Exception as e:
return None, {"error": str(e), "items": []}
finally: finally:
if pid_holder:
try: try:
from last30days import unregister_child_pid from last30days import unregister_child_pid
unregister_child_pid(proc.pid) unregister_child_pid(pid_holder[0])
except Exception: except Exception:
pass pass
if proc.returncode != 0: return result, None
error = stderr.strip() if stderr else "Bird search failed"
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
Retries the subprocess on JSON-decode failure (typically a Twitter
anti-bot HTML interstitial in stdout) up to MAX_JSON_DECODE_RETRIES
times with JSON_DECODE_RETRY_DELAY seconds between attempts. Terminal
errors (subprocess timeout, non-zero return code) are returned
immediately without retry.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds (per attempt)
Returns:
Raw Bird JSON response or error dict.
"""
last_decode_error: Optional[str] = None
for attempt in range(MAX_JSON_DECODE_RETRIES):
result, terminal_error = _invoke_bird_subprocess(query, count, timeout)
if terminal_error is not None:
return terminal_error
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []} return {"error": error, "items": []}
output = stdout.strip() if stdout else "" output = result.stdout.strip()
if not output: if not output:
return {"items": []} return {"items": []}
try:
parsed = json.loads(output) parsed = json.loads(output)
except json.JSONDecodeError as e:
# Twitter's edge sometimes serves an HTML anti-bot interstitial
# in place of JSON. Tag the failure shape so it's distinguishable
# from "no results" in logs, then retry the subprocess.
looks_html = output.lstrip().lower().startswith(("<!doctype", "<html", "<"))
attempt_num = attempt + 1
log_msg = (
f"Bird search returned non-JSON stdout "
f"(looks_html={looks_html}, attempt {attempt_num}/{MAX_JSON_DECODE_RETRIES}, "
f"first 80 chars: {output[:80]!r})"
)
last_decode_error = str(e)
if attempt_num < MAX_JSON_DECODE_RETRIES:
log.source_log(
"X/bird",
f"{log_msg}; retrying in {JSON_DECODE_RETRY_DELAY:.0f}s",
)
time.sleep(JSON_DECODE_RETRY_DELAY)
continue
log.source_log("X/bird", log_msg)
return {
"error": (
f"Invalid JSON response after {MAX_JSON_DECODE_RETRIES} attempts "
f"(likely Twitter anti-bot interstitial): {e}"
),
"items": [],
}
if isinstance(parsed, list): if isinstance(parsed, list):
return {"items": parsed} return {"items": parsed}
return parsed return parsed
except json.JSONDecodeError as e: # Defensive fallthrough — loop should always return above.
return {"error": f"Invalid JSON response: {e}", "items": []} return {
except Exception as e: "error": f"Bird search exhausted retries: {last_decode_error}",
return {"error": str(e), "items": []} "items": [],
}
def search_x( def search_x(
@@ -330,47 +379,29 @@ def search_handles(
"--json", "--json",
] ]
preexec = os.setsid if hasattr(os, 'setsid') else None
try: try:
proc = subprocess.Popen( result = subproc.run_with_timeout(cmd, timeout=15, env=_subprocess_env())
cmd, except subproc.SubprocTimeout:
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
preexec_fn=preexec,
env=_subprocess_env(),
)
try:
stdout, stderr = proc.communicate(timeout=15)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
_log(f"Handle search timed out for @{handle}") _log(f"Handle search timed out for @{handle}")
return [] return []
except OSError as e:
if proc.returncode != 0: _log(f"Handle search error for @{handle}: {e}")
_log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
return [] return []
output = (stdout or "").strip() if result.returncode != 0:
_log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
return []
output = result.stdout.strip()
if not output: if not output:
return [] return []
try:
response = json.loads(output) response = json.loads(output)
return parse_bird_response(response, query=core_topic)
except json.JSONDecodeError: except json.JSONDecodeError:
_log(f"Invalid JSON from handle search for @{handle}") _log(f"Invalid JSON from handle search for @{handle}")
except (OSError, subprocess.SubprocessError) as e:
_log(f"Handle search error for @{handle}: {e}")
return [] return []
return parse_bird_response(response, query=core_topic)
from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures import ThreadPoolExecutor, as_completed
+283
View File
@@ -0,0 +1,283 @@
"""Category-peer subreddit map for Step 0.55 community resolution.
When a topic is a product in a known category (AI image generation, AI coding
agents, SaaS screen recording, etc.), brand-specific subreddits returned by
WebSearch are insufficient: cross-product technique discussion lives in
category-peer subs. This module classifies a topic into a category by matching
compound-term patterns against the lowercased topic string, then returns the
priority-ordered peer subreddit list for that category.
The map is intentionally small, curated, and code-reviewed. Adding a new
category is a code change; there is no user-editable override surface.
False-positive guard: every pattern is either a multi-word compound (e.g.
"image generation", "text to image") or a domain-specific single word
(e.g. "midjourney", "stablediffusion"). Bare common nouns like "image",
"ai", or "model" are never used as patterns.
First-match-wins: categories are evaluated in declared order. Entries are
sorted from most-specific to least-specific so narrower categories claim a
topic before broader ones. For example, `ai_image_generation` appears
before `ai_chat_model` so "gpt image 2" matches the image-gen category.
"""
from __future__ import annotations
from typing import List, Optional, TypedDict
class _CategoryEntry(TypedDict):
patterns: List[str]
peer_subs: List[str]
CATEGORY_PEERS: dict[str, _CategoryEntry] = {
"ai_image_generation": {
"patterns": [
"image generation",
"image gen",
"text to image",
"text-to-image",
"gpt image",
"gpt-image",
"nano banana",
"midjourney",
"stable diffusion",
"stablediffusion",
"dall-e",
"dalle",
"flux.1",
"flux schnell",
"imagen",
"seedance",
"ideogram",
"recraft",
],
"peer_subs": [
"StableDiffusion",
"midjourney",
"dalle2",
"aiArt",
"PromptEngineering",
"MediaSynthesis",
],
},
"ai_video_generation": {
"patterns": [
"video generation",
"text to video",
"text-to-video",
"sora",
"veo 3",
"veo3",
"runway gen",
"kling",
"pika labs",
"luma dream machine",
"hailuo",
],
"peer_subs": [
"aivideo",
"StableDiffusion",
"runwayml",
"singularity",
"MediaSynthesis",
],
},
"ai_music_generation": {
"patterns": [
"music generation",
"ai music",
"suno",
"udio",
"riffusion",
"stable audio",
],
"peer_subs": [
"SunoAI",
"udiomusic",
"aimusic",
"artificial",
],
},
"ai_coding_agent": {
"patterns": [
"claude code",
"cursor ide",
"github copilot",
"windsurf",
"aider",
"cline",
"openclaw",
"hermes agent",
"continue.dev",
"codeium",
"sweep ai",
"devin ai",
"coding agent",
"coding assistant",
],
"peer_subs": [
"ChatGPTCoding",
"LocalLLaMA",
"singularity",
"PromptEngineering",
],
},
"ai_agent_framework": {
"patterns": [
"agent framework",
"agentic framework",
"langchain",
"langgraph",
"crewai",
"autogen",
"llamaindex",
"dspy",
"smolagents",
],
"peer_subs": [
"LangChain",
"LocalLLaMA",
"AI_Agents",
"MachineLearning",
],
},
"ai_chat_model": {
"patterns": [
"gpt-5",
"gpt-4",
"claude opus",
"claude sonnet",
"claude haiku",
"gemini pro",
"gemini flash",
"llama 3",
"llama 4",
"deepseek",
"qwen",
"mistral large",
"grok",
],
"peer_subs": [
"LocalLLaMA",
"ChatGPT",
"ClaudeAI",
"singularity",
"artificial",
],
},
"saas_screen_recording": {
"patterns": [
"screen recording",
"screen recorder",
"loom video",
"tella screen",
"vidyard",
"screen capture tool",
],
"peer_subs": [
"SaaS",
"screenrecording",
"productivity",
"Entrepreneur",
],
},
"saas_productivity": {
"patterns": [
"notion app",
"obsidian plugin",
"obsidian app",
"linear app",
"asana",
"clickup",
"productivity app",
],
"peer_subs": [
"productivity",
"SaaS",
"ObsidianMD",
"Notion",
],
},
"prediction_markets": {
"patterns": [
"polymarket",
"kalshi",
"prediction market",
"event contracts",
"manifold markets",
],
"peer_subs": [
"Polymarket",
"Kalshi",
"predictionmarkets",
],
},
"crypto_defi": {
"patterns": [
"defi protocol",
"yield farming",
"liquidity pool",
"stablecoin",
"ethereum layer",
"layer 2",
"l2 rollup",
],
"peer_subs": [
"defi",
"ethfinance",
"CryptoCurrency",
"ethereum",
],
},
"dev_tool_cli": {
"patterns": [
"cli tool",
"command line tool",
"terminal app",
"dev tool",
],
"peer_subs": [
"commandline",
"programming",
"webdev",
],
},
}
def detect_category(topic: Optional[str]) -> Optional[str]:
"""Classify a topic into a known category by compound-term match.
Returns the category id (e.g. "ai_image_generation") or None if no
category's patterns match. Matching is case-insensitive substring over
the lowercased topic. Declaration order wins (first-match-wins), so the
map is ordered from most-specific to least-specific.
A None or empty topic returns None. Classification never raises on
normal string inputs; callers do not need to wrap in try/except for
typical paths, though defensive callers may.
"""
if not topic:
return None
lowered = topic.lower()
for category_id, entry in CATEGORY_PEERS.items():
for pattern in entry["patterns"]:
if pattern in lowered:
return category_id
return None
def peer_subs_for(category_id: Optional[str]) -> List[str]:
"""Return the priority-ordered peer subreddit list for a category.
Returns an empty list for None or unknown category ids. The returned
list is a fresh copy; callers may safely mutate it.
"""
if not category_id:
return []
entry = CATEGORY_PEERS.get(category_id)
if not entry:
return []
return list(entry["peer_subs"])
@@ -0,0 +1,199 @@
"""Discover peer entities ("competitors") for a topic via web search.
Mirrors the `resolve.auto_resolve()` pattern: fan out 2-3 web searches via
`grounding.web_search()`, then extract capitalized entity candidates from
titles and snippets with deterministic text mining. No LLM call the
hosting reasoning model can always override discovery via
`--competitors-list`.
Returned list is ordered by score (frequency across queries) and capped to
the caller's requested count.
"""
from __future__ import annotations
import re
import sys
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from . import dates, grounding
from .resolve import _has_backend
# A "brand-shaped" token starts with uppercase OR is camelCase with an
# uppercase letter later. Catches "Anthropic", "OpenAI", "xAI", "iPhone",
# "eBay", "Hugging", "Face".
_BRAND_TOKEN = (
r"(?:[A-Z][A-Za-z0-9&.\-]*"
r"|[a-z][A-Za-z0-9&.\-]*[A-Z][A-Za-z0-9&.\-]*)"
)
# A capitalized phrase of 1-4 brand tokens separated by whitespace.
_CAPITALIZED_PHRASE = re.compile(
rf"\b{_BRAND_TOKEN}(?:\s+{_BRAND_TOKEN}){{0,3}}\b"
)
# Title-case fillers common in listicle SERPs. Kept flat — extraction
# rejects a candidate whose entire tokens are stopwords, not candidates
# that merely contain one.
_STOPWORD_TOKENS: frozenset[str] = frozenset(
token.lower()
for token in (
# Listicle fillers
"Top", "Best", "Worst", "Popular", "Leading", "Similar",
"Alternatives", "Alternative", "Competitor", "Competitors",
"vs", "Vs", "Versus", "Review", "Reviews", "Comparison",
"Guide", "List", "Lists", "Full", "Complete", "Free", "Paid",
"Tools", "Tool", "Options", "Rivals", "Rival", "Similar",
"Pick", "Picks", "Ranking", "Ranked", "Recommended",
# Grammar / time
"The", "A", "An", "Of", "In", "For", "To", "With", "On", "At",
"By", "From", "Is", "Are", "And", "Or", "But", "Than", "As",
"This", "That", "These", "Those", "Our", "Your", "Their",
"January", "February", "March", "April", "May", "June", "July",
"August", "September", "October", "November", "December",
# Years likely to appear as standalone tokens
*(str(year) for year in range(2018, 2031)),
# Miscellaneous SERP noise
"AI", "Apps", "App", "Software", "Platform", "Service", "Startups",
"Companies", "Company", "Products", "Product", "Brands", "Brand",
)
)
def _log(msg: str) -> None:
print(f"[Competitors] {msg}", file=sys.stderr)
def _topic_tokens(topic: str) -> set[str]:
"""Return lowercase alphanumeric tokens of the topic for filtering."""
return {tok for tok in re.findall(r"[A-Za-z0-9]+", topic.lower()) if tok}
def _candidate_ok(candidate: str, topic_tokens: set[str]) -> bool:
"""Filter a candidate phrase against stopwords and topic overlap."""
tokens = [t for t in re.findall(r"[A-Za-z0-9&.\-]+", candidate) if t]
if not tokens:
return False
# Reject candidates made entirely of stopwords (e.g., "Top Alternatives").
if all(tok.lower() in _STOPWORD_TOKENS for tok in tokens):
return False
# Reject candidates that overlap with the topic (e.g., topic="OpenAI"
# should not return "OpenAI Alternatives" or "OpenAI").
lower_tokens = {tok.lower() for tok in tokens}
if lower_tokens & topic_tokens:
return False
# Reject too-short one-letter tokens like "I" or single digits.
if len(tokens) == 1 and len(tokens[0]) < 2:
return False
return True
def _normalize_candidate(candidate: str) -> str:
"""Collapse whitespace and strip trailing punctuation."""
return re.sub(r"\s+", " ", candidate).strip(".,;:!?'\"()[] ")
def _extract_peer_entities(
items: list[dict], topic: str, limit: int,
) -> list[str]:
"""Score capitalized candidates across SERP items and return top `limit`.
Scoring is bag-of-phrases frequency across all items in the input. Ties
are broken by first-seen order so the output is deterministic.
"""
topic_tokens = _topic_tokens(topic)
counts: Counter[str] = Counter()
first_seen: dict[str, int] = {}
order = 0
# Group candidates into a frequency map keyed by lowercased normalized
# form so "xAI" and "xAI" count together regardless of case.
canonical: dict[str, str] = {}
for item in items:
text = f"{item.get('title', '')} {item.get('snippet', '')}"
for raw in _CAPITALIZED_PHRASE.findall(text):
candidate = _normalize_candidate(raw)
if not _candidate_ok(candidate, topic_tokens):
continue
key = candidate.lower()
if key not in canonical:
canonical[key] = candidate
first_seen[key] = order
order += 1
counts[key] += 1
ranked_keys = sorted(
counts.keys(),
key=lambda k: (-counts[k], first_seen[k]),
)
return [canonical[k] for k in ranked_keys[:limit]]
def _queries_for(topic: str) -> dict[str, str]:
return {
"competitors": f"{topic} competitors",
"alternatives": f"{topic} alternatives",
"vs": f"{topic} vs",
}
def discover_competitors(
topic: str,
count: int,
config: dict,
*,
lookback_days: int = 30,
) -> list[str]:
"""Discover `count` peer entities for `topic` via web search.
Args:
topic: The primary research topic.
count: Desired number of competitor entities (1..N).
config: Runtime config dict expects the same shape as the engine
config (BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / etc.).
lookback_days: Date range for freshness. Defaults to 30.
Returns:
A list of up to `count` entity names, deduped and ordered by score.
Empty list when no web backend is configured or every search fails
or returns zero usable candidates.
"""
if count < 1:
return []
if not _has_backend(config):
_log("No web search backend available, skipping competitor discovery")
return []
date_range = dates.get_date_range(lookback_days)
queries = _queries_for(topic)
collected: list[dict] = []
searches_run = 0
def _search(label: str, query: str) -> tuple[str, list[dict]]:
items, _artifact = grounding.web_search(query, date_range, config)
return label, items
with ThreadPoolExecutor(max_workers=len(queries)) as executor:
futures = {
executor.submit(_search, label, q): label
for label, q in queries.items()
}
for future in as_completed(futures):
label = futures[future]
try:
_label, items = future.result()
collected.extend(items)
searches_run += 1
except Exception as exc:
_log(f"Search failed for {label}: {exc}")
if not collected:
_log(f"No SERP results for {topic!r} across {searches_run}/{len(queries)} queries")
return []
entities = _extract_peer_entities(collected, topic, limit=count)
_log(
f"Discovered {len(entities)} competitor(s) for {topic!r} "
f"from {searches_run}/{len(queries)} queries: {entities}"
)
return entities
@@ -39,11 +39,14 @@ def normalize_text(text: str) -> str:
return re.sub(r"\s+", " ", text).strip() return re.sub(r"\s+", " ", text).strip()
def _ngrams_of_normalized(norm: str, n: int = 3) -> set[str]:
if len(norm) < n:
return {norm} if norm else set()
return {norm[index:index + n] for index in range(len(norm) - n + 1)}
def get_ngrams(text: str, n: int = 3) -> set[str]: def get_ngrams(text: str, n: int = 3) -> set[str]:
text = normalize_text(text) return _ngrams_of_normalized(normalize_text(text), n)
if len(text) < n:
return {text} if text else set()
return {text[index:index + n] for index in range(len(text) - n + 1)}
def jaccard_similarity(left: set[str], right: set[str]) -> float: def jaccard_similarity(left: set[str], right: set[str]) -> float:
@@ -90,7 +93,7 @@ class _PreparedText:
def __init__(self, raw: str) -> None: def __init__(self, raw: str) -> None:
norm = normalize_text(raw) norm = normalize_text(raw)
self.ngrams = get_ngrams(norm) if norm else set() self.ngrams = _ngrams_of_normalized(norm)
self.tokens = _tokenize(norm) self.tokens = _tokenize(norm)
+414
View File
@@ -0,0 +1,414 @@
"""Digg AI 1000 source for last30days.
Shells out to ``digg-pp-cli`` (read-only, no auth required) to surface
clustered stories curated from ~1000 high-signal AI accounts on X. Each
cluster carries a published TLDR, a curatorial rank, and a list of X
posts that can be fetched as inline quotes.
Activation gate: this source is only available when ``digg-pp-cli`` is
on PATH. ``pipeline.available_sources`` checks ``shutil.which`` before
including ``digg`` in the source list. The functions below also detect
the missing-binary case as a defensive fallback.
Primary path: ``digg-pp-cli search <topic> --since 30d --agent --limit N``.
Optional enrichment: ``digg-pp-cli posts <clusterUrlId> --agent --by rank
--limit M`` for the top K clusters in default/deep depth, attaching the
top-ranked X posts to each cluster's ``posts`` field.
"""
from __future__ import annotations
import json
import shutil
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
from . import log, subproc
from .relevance import token_overlap_relevance
CLI_BIN = "digg-pp-cli"
# Per-depth knobs.
DEPTH_CONFIG = {
"quick": 8,
"default": 20,
"deep": 40,
}
# How many top-ranked clusters get post enrichment, per depth. Quick mode
# skips enrichment to keep latency low (clusters already carry a TLDR).
ENRICH_CONFIG = {
"quick": 0,
"default": 3,
"deep": 5,
}
# X posts pulled per enriched cluster. Matches the 5-comment cap used by
# Reddit/HN/YouTube/TikTok/GitHub enrichment.
POSTS_PER_CLUSTER = 5
SEARCH_TIMEOUT = 30
POSTS_TIMEOUT = 15
def _log(msg: str) -> None:
log.source_log("Digg", msg)
def _is_available() -> bool:
"""True when the digg-pp-cli binary is on PATH."""
return shutil.which(CLI_BIN) is not None
def _today() -> datetime:
return datetime.now(timezone.utc)
def _parse_first_post_age(age: Optional[str], today: Optional[datetime] = None) -> Optional[str]:
"""Convert a digg firstPostAge token (e.g. '5d', '17d', '5h', '1w', '1m')
into a YYYY-MM-DD string. Returns None when the value is outside the
last-30-day window or cannot be parsed.
Digg uses minutes-symbol-collision for 'months' (per agent-context:
'Nh, Nd, Nw, Nm (e.g. 30d, 1w, 12h, 1m)'), so 'Nm' is months ~30 days.
"""
if not age or not isinstance(age, str):
return None
age = age.strip().lower()
if len(age) < 2:
return None
unit = age[-1]
try:
amount = int(age[:-1])
except (ValueError, TypeError):
return None
if amount < 0:
return None
base = today or _today()
if unit == "h":
delta = timedelta(hours=amount)
elif unit == "d":
delta = timedelta(days=amount)
elif unit == "w":
delta = timedelta(weeks=amount)
elif unit == "m":
delta = timedelta(days=amount * 30)
else:
return None
if delta > timedelta(days=30):
return None
point = base - delta
return point.date().isoformat()
def _build_search_args(query: str, limit: int) -> List[str]:
return [
CLI_BIN,
"search",
query,
"--since",
"30d",
"--agent",
"--limit",
str(limit),
]
def _build_posts_args(cluster_url_id: str, posts_per: int) -> List[str]:
return [
CLI_BIN,
"posts",
cluster_url_id,
"--agent",
"--by",
"rank",
"--limit",
str(posts_per),
]
def _run_cli(cmd: List[str], timeout: int) -> Dict[str, Any]:
"""Invoke digg-pp-cli and parse the JSON envelope.
Returns ``{"results": [...]}`` on success, ``{"results": [], "error": "..."}``
on failure. Never raises; the pipeline relies on shape consistency.
"""
if not _is_available():
return {"results": [], "error": f"{CLI_BIN} not on PATH"}
try:
result = subproc.run_with_timeout(cmd, timeout=timeout)
except subproc.SubprocTimeout as exc:
_log(f"Timeout: {exc}")
return {"results": [], "error": str(exc)}
except FileNotFoundError as exc:
_log(f"Binary missing: {exc}")
return {"results": [], "error": str(exc)}
except OSError as exc:
_log(f"Spawn failed: {exc}")
return {"results": [], "error": str(exc)}
if result.returncode != 0:
snippet = (result.stderr or "").strip().splitlines()[:1]
first = snippet[0] if snippet else f"exit {result.returncode}"
_log(f"CLI exit {result.returncode}: {first}")
return {"results": [], "error": first}
stdout = result.stdout or ""
if not stdout.strip():
return {"results": []}
try:
data = json.loads(stdout)
except json.JSONDecodeError as exc:
_log(f"JSON decode failed: {exc}")
return {"results": [], "error": f"json decode: {exc}"}
if not isinstance(data, dict):
return {"results": []}
results = data.get("results")
if not isinstance(results, list):
return {"results": []}
return data
def search_digg(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search Digg AI 1000 clusters via digg-pp-cli.
Args:
topic: search query.
from_date: YYYY-MM-DD start (advisory; --since 30d is the actual filter).
to_date: YYYY-MM-DD end (advisory; same).
depth: 'quick' | 'default' | 'deep'.
Returns:
Dict with ``results`` list. On failure, ``results`` is empty and an
``error`` key carries a one-line description.
"""
limit = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
if not topic or not topic.strip():
return {"results": []}
cmd = _build_search_args(topic, limit)
_log(f"search '{topic}' (limit={limit}, since=30d)")
response = _run_cli(cmd, timeout=SEARCH_TIMEOUT)
n = len(response.get("results") or [])
_log(f"found {n} clusters")
return response
def _build_url(cluster_url_id: str) -> str:
return f"https://di.gg/ai/{cluster_url_id}"
def _rank_score(rank: Optional[int]) -> float:
"""Convert Digg rank (lower is better, top 50 are notable) into a
positive engagement-style signal in [0, 50]. Anything off the top-50
leaderboard contributes 0.
"""
if rank is None:
return 0.0
try:
r = int(rank)
except (TypeError, ValueError):
return 0.0
if r < 1 or r > 50:
return 0.0
return float(51 - r)
def parse_digg_response(
response: Dict[str, Any],
query: str = "",
) -> List[Dict[str, Any]]:
"""Parse a digg search envelope into normalized item dicts.
Args:
response: payload from ``search_digg``.
query: original search query, used for token-overlap relevance.
Returns:
List of dicts ready for ``normalize._normalize_digg``.
"""
raw = response.get("results") if isinstance(response, dict) else None
if not isinstance(raw, list):
return []
items: List[Dict[str, Any]] = []
for i, cluster in enumerate(raw):
if not isinstance(cluster, dict):
continue
cluster_url_id = cluster.get("clusterUrlId")
if not cluster_url_id:
continue
title = str(cluster.get("title") or "").strip()
tldr = str(cluster.get("tldr") or "").strip()
rank = cluster.get("rank")
post_count = cluster.get("postCount") or 0
unique_authors = cluster.get("uniqueAuthors") or 0
first_post_age = cluster.get("firstPostAge")
date_str = _parse_first_post_age(first_post_age)
if date_str is None and first_post_age:
# firstPostAge present but outside 30d -> drop; last30days contract.
continue
rank_decay = max(0.3, 1.0 - (i * 0.02))
if query:
content_score = token_overlap_relevance(query, f"{title} {tldr}".strip())
else:
content_score = 0.5
rank_boost = min(0.2, _rank_score(rank) / 250.0)
relevance = min(1.0, 0.55 * rank_decay + 0.35 * content_score + rank_boost)
items.append(
{
"id": str(cluster_url_id),
"title": title or f"Digg cluster {i + 1}",
"url": _build_url(str(cluster_url_id)),
"tldr": tldr,
"author": "",
"date": date_str,
"engagement": {
"postCount": int(post_count) if isinstance(post_count, (int, float)) else 0,
"uniqueAuthors": int(unique_authors) if isinstance(unique_authors, (int, float)) else 0,
"rank": int(rank) if isinstance(rank, (int, float)) else None,
"rank_score": _rank_score(rank),
},
"first_post_age": first_post_age,
"posts": [],
"relevance": round(relevance, 2),
"why_relevant": (
f"Digg cluster (rank {rank}, {post_count} posts, {unique_authors} authors)"
if rank is not None
else f"Digg cluster ({post_count} posts, {unique_authors} authors)"
),
}
)
return items
def _parse_post(raw_post: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Reduce a digg post payload into the small dict render uses.
We deliberately keep this minimal: an inline quote needs the author
handle, the body, the post type, and the X URL.
"""
if not isinstance(raw_post, dict):
return None
body = str(raw_post.get("body") or "").strip()
if not body:
return None
author = raw_post.get("author") or {}
if not isinstance(author, dict):
author = {}
username = str(author.get("username") or "").strip()
if not username:
return None
x_url = str(raw_post.get("xUrl") or "").strip()
if not x_url:
return None
return {
"username": username,
"display_name": str(author.get("display_name") or "").strip() or username,
"category": str(author.get("category") or "").strip(),
"rank": author.get("rank"),
"body": body,
"post_type": str(raw_post.get("post_type") or "tweet").strip(),
"x_url": x_url,
"posted_at": raw_post.get("posted_at"),
}
def fetch_top_posts(cluster_url_id: str, posts_per: int = POSTS_PER_CLUSTER) -> List[Dict[str, Any]]:
"""Fetch top-ranked X posts attached to a cluster.
Returns an empty list on any failure (timeout, missing cluster, JSON
error). Never raises.
"""
if posts_per <= 0:
return []
cmd = _build_posts_args(cluster_url_id, posts_per)
response = _run_cli(cmd, timeout=POSTS_TIMEOUT)
raw = response.get("results") or []
out: List[Dict[str, Any]] = []
for entry in raw:
post = _parse_post(entry)
if post is not None:
out.append(post)
return out
def enrich_with_top_posts(
items: List[Dict[str, Any]],
top_k: int = 3,
posts_per: int = POSTS_PER_CLUSTER,
) -> List[Dict[str, Any]]:
"""Attach top X posts to the first ``top_k`` clusters by Digg rank order.
Mutates and returns the same list. Items that already have posts, or
whose ``postCount`` is 0, are skipped.
"""
if top_k <= 0 or posts_per <= 0:
return items
enriched = 0
for item in items:
if enriched >= top_k:
break
if item.get("posts"):
continue
engagement = item.get("engagement") or {}
if not engagement.get("postCount"):
continue
cluster_url_id = item.get("id")
if not cluster_url_id:
continue
posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
item["posts"] = posts
enriched += 1
if enriched:
_log(f"enriched {enriched} clusters with X posts")
return items
def enrich_source_items(items: list, top_k: int = 3, posts_per: int = POSTS_PER_CLUSTER) -> list:
"""Attach top X posts to the first ``top_k`` SourceItems that survived dedupe.
Reads ``metadata['clusterUrlId']`` and writes ``metadata['posts']`` in
place. Skips items that already carry a non-empty ``metadata['posts']``,
items whose engagement ``postCount`` is 0, and items whose source is not
'digg'. Designed to run from `_finalize_items_by_source` so enrichment
is spent on the items the brief actually shows.
"""
if top_k <= 0 or posts_per <= 0:
return items
enriched = 0
for item in items:
if enriched >= top_k:
break
if getattr(item, "source", None) != "digg":
continue
metadata = getattr(item, "metadata", None) or {}
if metadata.get("posts"):
continue
engagement = getattr(item, "engagement", None) or {}
if not engagement.get("postCount"):
continue
cluster_url_id = metadata.get("clusterUrlId") or item.item_id
if not cluster_url_id:
continue
posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
if posts:
metadata["posts"] = posts
enriched += 1
if enriched:
_log(f"post-dedupe enriched {enriched} clusters with X posts")
return items
@@ -29,6 +29,23 @@ else:
CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json"))) CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))
# macOS Keychain integration: items stored with this service prefix are picked
# up automatically on Darwin as the lowest-priority credential source.
# Example: `security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."`.
KEYCHAIN_SERVICE_PREFIX = "last30days-"
# Single source of truth for which credentials the Keychain loader looks up.
# The setup-keychain.sh helper mirrors this list and is held in sync via
# tests/test_env_keychain.py::test_keychain_keys_match_setup_script.
KEYCHAIN_KEYS = (
"OPENAI_API_KEY", "XAI_API_KEY", "GOOGLE_API_KEY", "GEMINI_API_KEY",
"GOOGLE_GENAI_API_KEY", "SCRAPECREATORS_API_KEY", "APIFY_API_TOKEN",
"AUTH_TOKEN", "CT0", "BSKY_HANDLE", "BSKY_APP_PASSWORD",
"TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
"OPENROUTER_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",
"XIAOHONGSHU_API_BASE",
)
AuthSource = Literal["api_key", "codex", "none"] AuthSource = Literal["api_key", "codex", "none"]
AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"] AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]
@@ -53,6 +70,10 @@ class OpenAIAuth:
def _check_file_permissions(path: Path) -> None: def _check_file_permissions(path: Path) -> None:
"""Warn to stderr if a secrets file has overly permissive permissions.""" """Warn to stderr if a secrets file has overly permissive permissions."""
if os.name == "nt":
# Windows reports synthesized POSIX mode bits that do not reflect NTFS ACLs.
return
try: try:
mode = path.stat().st_mode mode = path.stat().st_mode
# Check if group or other can read (bits 0o044) # Check if group or other can read (bits 0o044)
@@ -91,6 +112,46 @@ def load_env_file(path: Path) -> dict[str, str]:
return env return env
def _load_keychain(keys: list[str]) -> dict[str, str]:
"""Load credentials from macOS Keychain (no-op on other platforms).
Each key is looked up as a generic password with service name
``f"{KEYCHAIN_SERVICE_PREFIX}{key}"`` for the current user. Missing items
and lookup failures are silent Keychain is the lowest-priority source
and is meant to be additive over `.env` files and process environment.
"""
import platform
if platform.system() != "Darwin":
return {}
import shutil
security = shutil.which("security")
if not security:
return {}
import subprocess
import pwd
# USER can be unset under sudo, in Docker without --env USER, or in some CI
# runners; fall back to the OS user record so lookups still match items
# stored by setup-keychain.sh (which uses $USER).
user = os.environ.get("USER") or pwd.getpwuid(os.getuid()).pw_name
env: dict[str, str] = {}
for key in keys:
try:
result = subprocess.run(
[security, "find-generic-password",
"-a", user,
"-s", f"{KEYCHAIN_SERVICE_PREFIX}{key}",
"-w"],
capture_output=True, text=True, timeout=5,
)
except (subprocess.TimeoutExpired, OSError):
continue
if result.returncode == 0 and result.stdout.strip():
env[key] = result.stdout.strip()
return env
def _decode_jwt_payload(token: str) -> dict[str, Any] | None: def _decode_jwt_payload(token: str) -> dict[str, Any] | None:
"""Decode JWT payload without verification.""" """Decode JWT payload without verification."""
try: try:
@@ -214,6 +275,7 @@ def get_config() -> dict[str, Any]:
1. Environment variables (os.environ) 1. Environment variables (os.environ)
2. .claude/last30days.env (per-project config) 2. .claude/last30days.env (per-project config)
3. ~/.config/last30days/.env (global config) 3. ~/.config/last30days/.env (global config)
4. macOS Keychain items prefixed ``last30days-`` (Darwin only)
""" """
# Load from global config file # Load from global config file
file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {} file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
@@ -222,9 +284,14 @@ def get_config() -> dict[str, Any]:
project_env_path = _find_project_env() project_env_path = _find_project_env()
project_env = load_env_file(project_env_path) if project_env_path else {} project_env = load_env_file(project_env_path) if project_env_path else {}
# Merge: project overrides global # Merge file sources: project > global
merged_env = {**file_env, **project_env} merged_env = {**file_env, **project_env}
# Keychain is the lowest-priority source (Darwin only; no-op elsewhere).
# Loaded before openai_auth so OPENAI_API_KEY can come from Keychain too.
keychain_env = _load_keychain(list(KEYCHAIN_KEYS))
merged_env = {**keychain_env, **merged_env}
openai_auth = get_openai_auth(merged_env) openai_auth = get_openai_auth(merged_env)
# Build config: Codex/OpenAI auth + process.env > project .env > global .env # Build config: Codex/OpenAI auth + process.env > project .env > global .env
@@ -265,16 +332,21 @@ def get_config() -> dict[str, Any]:
('FROM_BROWSER', None), ('FROM_BROWSER', None),
('SETUP_COMPLETE', None), ('SETUP_COMPLETE', None),
('INCLUDE_SOURCES', ''), ('INCLUDE_SOURCES', ''),
('EXCLUDE_SOURCES', ''),
('LAST30DAYS_YOUTUBE_SSH_HOST', None),
] ]
for key, default in keys: for key, default in keys:
config[key] = os.environ.get(key) or merged_env.get(key, default) config[key] = os.environ.get(key) or merged_env.get(key, default)
# Track which config source was used # Track which config source was used (highest-priority file source wins
# the label; keychain is only reported when nothing else is configured).
if project_env_path: if project_env_path:
config['_CONFIG_SOURCE'] = f'project:{project_env_path}' config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
elif CONFIG_FILE and CONFIG_FILE.exists(): elif CONFIG_FILE and CONFIG_FILE.exists():
config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}' config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
elif keychain_env:
config['_CONFIG_SOURCE'] = 'keychain'
else: else:
config['_CONFIG_SOURCE'] = 'env_only' config['_CONFIG_SOURCE'] = 'env_only'
@@ -356,6 +428,10 @@ def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
if config.get("AUTH_TOKEN") and config.get("CT0"): if config.get("AUTH_TOKEN") and config.get("CT0"):
method = config.get("_AUTH_TOKEN_SOURCE", "env") method = config.get("_AUTH_TOKEN_SOURCE", "env")
return "bird", method return "bird", method
# Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
from . import xurl_x
if xurl_x.is_available():
return "xurl", "oauth2"
return None, "none" return None, "none"
@@ -368,14 +444,6 @@ def config_exists() -> bool:
return False return False
def is_reddit_available(config: dict[str, Any]) -> bool:
"""Check if Reddit search is available.
v3 uses ScrapeCreators only.
"""
return bool(config.get('SCRAPECREATORS_API_KEY'))
def get_reddit_source(config: dict[str, Any]) -> str | None: def get_reddit_source(config: dict[str, Any]) -> str | None:
"""Determine which Reddit backend to use. """Determine which Reddit backend to use.
@@ -401,6 +469,7 @@ def get_x_source(config: dict[str, Any]) -> str | None:
Returns: Returns:
'bird' if Bird is installed and explicit cookies are configured, 'bird' if Bird is installed and explicit cookies are configured,
'xai' if XAI_API_KEY is configured, 'xai' if XAI_API_KEY is configured,
'xurl' if xurl CLI is installed and authenticated,
None if no X source available. None if no X source available.
""" """
# Import here to avoid circular dependency # Import here to avoid circular dependency
@@ -421,6 +490,11 @@ def get_x_source(config: dict[str, Any]) -> str | None:
if has_bird_creds and bird_x.is_bird_installed(): if has_bird_creds and bird_x.is_bird_installed():
return 'bird' return 'bird'
# Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
from . import xurl_x
if xurl_x.is_available():
return 'xurl'
return None return None
@@ -515,12 +589,12 @@ def _parse_include_sources(config: dict[str, Any]) -> set[str]:
def is_threads_available(config: dict[str, Any]) -> bool: def is_threads_available(config: dict[str, Any]) -> bool:
"""Check if Threads source is available. """Check if Threads source is available.
Requires SCRAPECREATORS_API_KEY AND 'threads' in INCLUDE_SOURCES. Returns True when SCRAPECREATORS_API_KEY is set. Threads runs alongside
Threads is an opt-in source - it is not activated by default. TikTok and Instagram as part of the SC family same key, same per-call
cost shape, so the same default-on rule applies. Suppress via
EXCLUDE_SOURCES=threads.
""" """
if not config.get('SCRAPECREATORS_API_KEY'): return bool(config.get('SCRAPECREATORS_API_KEY'))
return False
return 'threads' in _parse_include_sources(config)
def is_instagram_available(config: dict[str, Any]) -> bool: def is_instagram_available(config: dict[str, Any]) -> bool:
@@ -602,14 +676,18 @@ def get_x_source_status(config: dict[str, Any]) -> dict[str, Any]:
elif xai_available: elif xai_available:
source = 'xai' source = 'xai'
else: else:
source = None # Fall back to xurl CLI
from . import xurl_x as _xurl_check
source = 'xurl' if _xurl_check.is_available() else None
from . import xurl_x as _xurl_x
return { return {
"source": source, "source": source,
"bird_installed": bird_status["installed"], "bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"], "bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"], "bird_username": bird_status["username"],
"xai_available": xai_available, "xai_available": xai_available,
"xurl_available": _xurl_x.is_available(),
"can_install_bird": bird_status["can_install"], "can_install_bird": bird_status["can_install"],
} }
+85
View File
@@ -0,0 +1,85 @@
"""Parallel multi-entity fan-out for the --competitors flag.
The orchestrator accepts a `main_runner()` for the topic and a
`competitor_runner(entity)` for each peer. It parallelizes their execution
via a `ThreadPoolExecutor` and collects per-entity Reports. Per-entity
failures are logged and dropped; the run survives as long as the main topic
plus at least one competitor succeed.
This module owns no business logic about pipeline arguments the caller
(scripts/last30days.py main) builds the closures with the appropriate
config, depth, and overrides for each entity.
"""
from __future__ import annotations
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Callable
from . import schema
# Sub-runs hit the same upstream APIs as the main topic. Cap parallelism so a
# 6-way fan-out does not stampede a single backend's rate limit.
MAX_PARALLEL_SUBRUNS = 6
def _log(msg: str) -> None:
print(f"[Fanout] {msg}", file=sys.stderr)
def run_competitor_fanout(
*,
main_topic: str,
main_runner: Callable[[], schema.Report],
competitors: list[str],
competitor_runner: Callable[[str], schema.Report],
) -> list[tuple[str, schema.Report]]:
"""Run main + competitor pipelines in parallel; return surviving reports.
Args:
main_topic: Display label for the user's primary topic.
main_runner: Zero-arg callable returning the main topic's Report.
competitors: Ordered list of competitor entity names.
competitor_runner: Callable(entity_name) -> Report for each peer.
Returns:
Ordered list of (entity_name, Report) tuples for runs that succeeded.
Empty list if every run raised; the caller decides how to surface
partial-failure modes.
"""
if not competitors:
report = main_runner()
return [(main_topic, report)]
workers = min(len(competitors) + 1, MAX_PARALLEL_SUBRUNS)
def _run_one(label: str, fn: Callable[[], schema.Report]) -> tuple[str, schema.Report | None, Exception | None]:
try:
return label, fn(), None
except Exception as exc:
return label, None, exc
submissions: list[tuple[str, Callable[[], schema.Report]]] = [
(main_topic, main_runner),
]
for entity in competitors:
submissions.append((entity, lambda e=entity: competitor_runner(e)))
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(_run_one, label, fn): label
for label, fn in submissions
}
results: dict[str, schema.Report] = {}
for future in as_completed(futures):
label, report, exc = future.result()
if exc is not None:
_log(f"Sub-run failed for {label!r}: {type(exc).__name__}: {exc}")
continue
assert report is not None
results[label] = report
# Preserve the original submission order rather than completion order so
# the comparison render is deterministic across runs.
return [(label, results[label]) for label, _ in submissions if label in results]
@@ -116,6 +116,8 @@ def weighted_rrf(
"""Fuse ranked lists into a single candidate pool.""" """Fuse ranked lists into a single candidate pool."""
subqueries = {subquery.label: subquery for subquery in plan.subqueries} subqueries = {subquery.label: subquery for subquery in plan.subqueries}
candidates: dict[str, schema.Candidate] = {} candidates: dict[str, schema.Candidate] = {}
# Track (source, item_id) pairs already attached to each candidate for O(1) dedup.
seen_source_items: dict[str, set[tuple[str, str]]] = {}
for (label, source), items in streams.items(): for (label, source), items in streams.items():
subquery = subqueries[label] subquery = subqueries[label]
@@ -154,6 +156,7 @@ def weighted_rrf(
] ]
}, },
) )
seen_source_items[key] = {(item.source, item.item_id)}
continue continue
candidate = candidates[key] candidate = candidates[key]
@@ -179,7 +182,9 @@ def weighted_rrf(
candidate.subquery_labels.append(label) candidate.subquery_labels.append(label)
if item.source not in candidate.sources: if item.source not in candidate.sources:
candidate.sources.append(item.source) candidate.sources.append(item.source)
if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items): source_item_key = (item.source, item.item_id)
if source_item_key not in seen_source_items[key]:
seen_source_items[key].add(source_item_key)
candidate.source_items.append(item) candidate.source_items.append(item)
candidate.metadata.setdefault("provenance", []).append( candidate.metadata.setdefault("provenance", []).append(
{ {
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import sys
import urllib.parse import urllib.parse
from datetime import datetime from datetime import datetime
from urllib.parse import urlparse from urllib.parse import urlparse
@@ -205,29 +206,90 @@ def web_search(
backend = "parallel" backend = "parallel"
else: else:
return [], {} return [], {}
items: list[dict] = []
artifact: dict = {}
if backend == "brave": if backend == "brave":
key = config.get("BRAVE_API_KEY") key = config.get("BRAVE_API_KEY")
if not key: if not key:
raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'") raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'")
return brave_search(query, date_range, key) items, artifact = brave_search(query, date_range, key)
if backend == "exa": elif backend == "exa":
key = config.get("EXA_API_KEY") key = config.get("EXA_API_KEY")
if not key: if not key:
raise RuntimeError("EXA_API_KEY is required when web_backend='exa'") raise RuntimeError("EXA_API_KEY is required when web_backend='exa'")
return exa_search(query, date_range, key) items, artifact = exa_search(query, date_range, key)
if backend == "serper": elif backend == "serper":
key = config.get("SERPER_API_KEY") key = config.get("SERPER_API_KEY")
if not key: if not key:
raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'") raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'")
return serper_search(query, date_range, key) items, artifact = serper_search(query, date_range, key)
if backend == "parallel": elif backend == "parallel":
key = config.get("PARALLEL_API_KEY") key = config.get("PARALLEL_API_KEY")
if not key: if not key:
raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'") raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'")
return parallel_search(query, date_range, key) items, artifact = parallel_search(query, date_range, key)
if backend != "none": elif backend != "none":
raise ValueError(f"Unsupported web backend: {backend!r}") raise ValueError(f"Unsupported web backend: {backend!r}")
else:
return [], {} return [], {}
if items and not _reddit_excluded(config):
items = _enrich_reddit_items(items)
return items, artifact
def _reddit_excluded(config: dict) -> bool:
"""Return True when EXCLUDE_SOURCES contains 'reddit'.
Respects the same suppression knob the pipeline uses for source gating,
so a user who set EXCLUDE_SOURCES=reddit doesn't get Reddit content
smuggled back in via web-search URLs.
"""
raw = (config.get("EXCLUDE_SOURCES") or "").split(",")
return any(s.strip().lower() == "reddit" for s in raw)
def _enrich_reddit_items(items: list[dict]) -> list[dict]:
"""Enrich web search results that are Reddit URLs with thread body and comments.
Claude Code's WebFetch blocks reddit.com, so the model can't retrieve
Reddit content from web search results. This fetches it via the public
JSON API (reddit.com/.../.json) which bypasses that restriction.
Callers should gate this with EXCLUDE_SOURCES=reddit handling (see
`_reddit_excluded`) so a user who explicitly excluded Reddit doesn't
get Reddit content via web-search URLs.
"""
from . import reddit_enrich
from .reddit_enrich import RedditRateLimitError
for item in items:
url = item.get("url", "")
if "reddit.com" not in url or "/comments/" not in url:
continue
try:
thread_data = reddit_enrich.fetch_thread_data(url, timeout=8)
if not thread_data:
continue
parsed = reddit_enrich.parse_thread_data(thread_data)
# selftext lives under parsed["submission"], not at the top level
selftext = (parsed.get("submission") or {}).get("selftext", "")
if selftext:
item["snippet"] = selftext[:2000]
comments = parsed.get("comments", [])
top = reddit_enrich.get_top_comments(comments)
if top:
item["top_comments"] = [
{"score": c.get("score", 0), "excerpt": (c.get("body") or "")[:200]}
for c in top[:5]
]
item["enriched_via"] = "reddit_json_api"
except RedditRateLimitError as exc:
# Stop iterating to avoid flooding more 429s
sys.stderr.write(f"[Web] Reddit rate-limited, halting enrichment: {exc}\n")
break
except Exception as exc:
sys.stderr.write(f"[Web] Reddit enrichment failed for {url}: {exc}\n")
return items
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -88,17 +88,26 @@ def search_hackernews(
# Use extracted core subject instead of raw topic for cleaner Algolia matching # Use extracted core subject instead of raw topic for cleaner Algolia matching
core = extract_core_subject(topic) core = extract_core_subject(topic)
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})") # Hyphens and commas tokenize awkwardly in Algolia; flatten them so themed
# queries like "ts-bun-node" or "claude, personal agents" become plain words.
core_flat = _flatten_query_for_algolia(core)
_log(f"Searching for '{core_flat}' (raw: '{topic}', since {from_date}, count={count})")
# Use relevance-sorted search with minimum engagement filter. # Use relevance-sorted search with minimum engagement filter.
# NOTE: restrictSearchableAttributes=title omitted intentionally — it would # NOTE: restrictSearchableAttributes=title omitted intentionally — it would
# miss Ask HN/Show HN threads where the topic appears in the body. # miss Ask HN/Show HN threads where the topic appears in the body.
params = { params = {
"query": core, "query": core_flat,
"tags": "story", "tags": "story",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2", "numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
"hitsPerPage": str(count), "hitsPerPage": str(count),
} }
# Algolia defaults to AND across query tokens, so a 4-5 word theme query
# matches no stories. Mark all-but-the-first token as optional so Algolia
# ranks by how many tokens match instead of requiring every one.
tokens = core_flat.split()
if len(tokens) > 1:
params["optionalWords"] = " ".join(tokens[1:])
from urllib.parse import urlencode from urllib.parse import urlencode
url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}" url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"
@@ -117,28 +126,56 @@ def search_hackernews(
return response return response
def _title_matches_query(title: str, query: str, author: str = "") -> bool: _WORD_BOUNDARY_RE_CACHE: Dict[str, "re.Pattern[str]"] = {}
"""Check if the query term appears in the title content, not just an HN prefix or author.
Returns True if the query (or any multi-word token) appears in the title
after stripping "Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes def _flatten_query_for_algolia(text: str) -> str:
and ignoring the author name. Returns True when query is empty (no filter). """Normalise query for Algolia + post-filter comparison.
Multi-keyword theme queries frequently contain commas (delimiters) or
hyphens (compound terms like ``ts-bun-node``); both tokenize awkwardly.
Flatten them to spaces and collapse runs of whitespace so the search
parameter and the post-filter operate on the same shape.
"""
return " ".join(text.replace(",", " ").replace("-", " ").split())
def _title_matches_query(title: str, query: str, author: str = "") -> bool:
"""Check if any query token appears as a whole word in the title.
Returns True when the query is empty (no filter), or when at least one
query token matches as a whole word in the title after stripping
"Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes.
We previously required *every* token to appear (all-words), which killed
every Algolia hit on multi-keyword themes like "claude, personal agents,
agentic infra" because real HN titles never contain all five tokens
verbatim. Relaxing to any-word matches Algolia's `optionalWords` behaviour
in `search_hackernews`. Token-overlap relevance scoring at parse time
demotes hits where only one weak token matched, so the loosened gate
won't surface noise to the top of the ranking.
Word-boundary matching (rather than naive substring) prevents short
tokens like ``ai`` or ``ts`` from matching unrelated words like
``email`` or ``artists``.
""" """
if not query: if not query:
return True return True
stripped = _HN_PREFIXES.sub("", title).strip() stripped = _HN_PREFIXES.sub("", title).strip()
# Also check that the match isn't solely in the author's username
check_text = stripped.lower() check_text = stripped.lower()
query_lower = query.lower() # Normalise the query the same way search_hackernews does so post-filter
# Check each word of the query independently; all must appear somewhere # tokens line up with what Algolia actually saw.
# in the stripped title (not just the prefix). query_words = [w for w in _flatten_query_for_algolia(query.lower()).split() if w]
query_words = query_lower.split() if not query_words:
for word in query_words:
if word in check_text:
continue
# Word not found in stripped title — reject
return False
return True return True
for word in query_words:
pattern = _WORD_BOUNDARY_RE_CACHE.get(word)
if pattern is None:
pattern = re.compile(rf"\b{re.escape(word)}\b")
_WORD_BOUNDARY_RE_CACHE[word] = pattern
if pattern.search(check_text):
return True
return False
def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]: def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
@@ -0,0 +1,674 @@
"""HTML rendering for shareable last30days reports."""
from __future__ import annotations
import html
import re
from datetime import date
from . import render, schema
PROSE_LABELS = [
("What I learned:", "What I learned"),
("KEY PATTERNS from the research:", "Key patterns from the research"),
]
INVITATION_PATTERN = re.compile(r"^---\nI'm now an expert.*?Just ask\.$", re.MULTILINE | re.DOTALL)
EVIDENCE_BLOCK_PATTERN = re.compile(r"<!-- EVIDENCE FOR SYNTHESIS.*?<!-- END EVIDENCE FOR SYNTHESIS -->", re.DOTALL)
PASS_THROUGH_FOOTER_PATTERN = re.compile(r"<!-- PASS-THROUGH FOOTER.*?-->\n(.*?)<!-- END PASS-THROUGH FOOTER -->", re.DOTALL)
CANONICAL_BOUNDARY_PATTERN = re.compile(r"\n?---\n# END OF last30days CANONICAL OUTPUT.*$", re.DOTALL)
# render_for_html emits metadata as <!-- META: ... --> so it survives the
# markdown converter (which escapes raw HTML inside paragraphs). Promoted to
# a styled <div class="meta"> after conversion.
META_MARKER_PATTERN = re.compile(r"<!--\s*META:\s*(.*?)\s*-->")
CSS = """
:root {
--bg: #0e0e10;
--bg-elev: #18181b;
--fg: #fafafa;
--fg-muted: #a1a1aa;
--fg-subtle: #71717a;
--accent: #a855f7;
--accent-soft: #c4b5fd;
--border: #27272a;
--code-bg: #1a1a1d;
--max-w: 720px;
}
@media (prefers-color-scheme: light) {
:root {
--bg: #ffffff;
--bg-elev: #fafafa;
--fg: #18181b;
--fg-muted: #52525b;
--fg-subtle: #71717a;
--accent: #7c3aed;
--accent-soft: #6d28d9;
--border: #e4e4e7;
--code-bg: #f4f4f5;
}
}
* { box-sizing: border-box; }
html, body {
margin: 0;
padding: 0;
background: var(--bg);
color: var(--fg);
font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif;
font-size: 17px;
line-height: 1.65;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-rendering: optimizeLegibility;
}
body {
max-width: var(--max-w);
margin: 0 auto;
padding: 4rem 1.5rem 6rem;
}
.badge {
display: inline-block;
padding: 0.4rem 0.85rem;
margin-bottom: 2.5rem;
background: var(--bg-elev);
border: 1px solid var(--border);
border-radius: 999px;
font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
font-size: 13px;
font-weight: 500;
color: var(--fg-muted);
letter-spacing: 0;
}
.badge .accent { color: var(--accent); }
.meta {
margin: -1.5rem 0 2.5rem;
color: var(--fg-subtle);
font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
font-size: 13px;
letter-spacing: 0.01em;
}
h1 {
margin: 0 0 1.5rem;
color: var(--fg);
font-size: 30px;
font-weight: 700;
line-height: 1.2;
letter-spacing: 0;
}
h2,
.prose-label {
margin: 2.75rem 0 1.25rem;
color: var(--fg);
font-size: 20px;
font-weight: 600;
line-height: 1.35;
letter-spacing: 0;
}
.badge + h2,
.badge + .prose-label { margin-top: 0.5rem; }
h3 {
margin: 2rem 0 0.85rem;
color: var(--fg);
font-size: 17px;
font-weight: 600;
line-height: 1.4;
letter-spacing: 0;
}
p {
margin: 0 0 1.4rem;
color: var(--fg-muted);
}
p strong,
li strong,
td strong {
color: var(--fg);
font-weight: 600;
}
a {
color: var(--accent);
text-decoration: none;
border-bottom: 1px solid transparent;
transition: border-color 0.15s ease;
}
a:hover { border-bottom-color: var(--accent); }
ul,
ol {
margin: 0 0 1.6rem;
padding-left: 1.5rem;
color: var(--fg-muted);
}
li {
margin: 0.6rem 0;
padding-left: 0.4rem;
}
li::marker {
color: var(--accent);
font-weight: 600;
}
blockquote {
margin: 1.5rem 0;
padding-left: 1rem;
border-left: 3px solid var(--accent);
color: var(--fg-muted);
}
hr {
margin: 2.5rem 0;
border: 0;
border-top: 1px solid var(--border);
}
code {
font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
font-size: 0.92em;
background: var(--code-bg);
padding: 0.15rem 0.4rem;
border-radius: 4px;
color: var(--accent-soft);
}
pre {
margin: 1.4rem 0;
background: var(--code-bg);
border: 1px solid var(--border);
border-radius: 8px;
padding: 1rem 1.25rem;
overflow-x: auto;
font-size: 14px;
line-height: 1.6;
}
pre code {
background: none;
padding: 0;
color: var(--fg);
}
table {
width: 100%;
border-collapse: collapse;
margin: 1.5rem 0;
font-size: 15px;
}
th,
td {
text-align: left;
padding: 0.75rem 1rem;
border-bottom: 1px solid var(--border);
vertical-align: top;
}
th {
color: var(--fg-muted);
font-weight: 600;
font-size: 13px;
letter-spacing: 0;
text-transform: uppercase;
}
td { color: var(--fg-muted); }
td:first-child { color: var(--fg); font-weight: 500; }
.engine-footer {
margin: 3rem 0 2.5rem;
padding: 1.25rem 1.5rem;
background: var(--bg-elev);
border: 1px solid var(--border);
border-radius: 8px;
color: var(--fg-muted);
}
.engine-footer pre {
margin: 0;
padding: 0;
background: transparent;
border: 0;
border-radius: 0;
font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
font-size: 13.5px;
font-weight: 400;
line-height: 1.75;
color: inherit;
white-space: pre-wrap;
word-break: break-word;
}
.colophon {
margin-top: 4rem;
padding-top: 2rem;
border-top: 1px solid var(--border);
color: var(--fg-subtle);
font-size: 13px;
font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
line-height: 1.7;
}
.colophon .rerun {
display: inline-block;
padding: 0.15rem 0.5rem;
margin-left: 0.25rem;
background: var(--code-bg);
border-radius: 4px;
color: var(--accent-soft);
font-size: 0.95em;
}
@media print {
:root {
--bg: #ffffff;
--bg-elev: #f5f5f5;
--fg: #000000;
--fg-muted: #1f2937;
--fg-subtle: #4b5563;
--accent: #6d28d9;
--accent-soft: #6d28d9;
--border: #d4d4d8;
--code-bg: #f4f4f5;
}
@page { size: A4; margin: 1.5cm 2cm; }
body {
max-width: none;
padding: 0;
font-size: 11pt;
}
a {
color: inherit;
border-bottom: 0;
text-decoration: underline;
}
a[href]::after {
content: " (" attr(href) ")";
font-size: 0.85em;
color: var(--fg-subtle);
}
.engine-footer { page-break-inside: avoid; }
}
@media (max-width: 600px) {
body {
padding: 2.5rem 1.25rem 4rem;
font-size: 16px;
}
h1 { font-size: 25px; }
.badge { font-size: 12px; }
th, td { padding: 0.65rem 0.5rem; }
}
""".strip()
HTML_TEMPLATE = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>last30days · __TITLE__</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&amp;family=JetBrains+Mono:wght@400;500&amp;display=swap" rel="stylesheet">
<style>
__CSS__
</style>
</head>
<body>
__BODY__
__COLOPHON__
</body>
</html>
"""
def render_html(
report: schema.Report,
*,
fun_level: str = "medium",
save_path: str | None = None,
synthesis_md: str | None = None,
) -> str:
_ = fun_level
md = render.render_for_html(report, synthesis_md=synthesis_md, save_path=save_path)
md = _strip_evidence_block(md)
md = _strip_invitation(md)
md = _strip_canonical_boundary(md)
md = _promote_prose_labels(md)
body = _markdown_to_html(md)
body = _wrap_engine_footer(body)
body = _promote_meta_marker(body)
colophon = _build_colophon(report)
return _wrap_in_template(body, colophon, report.topic)
def render_html_comparison(
entity_reports: list[tuple[str, schema.Report]],
*,
fun_level: str = "medium",
save_path: str | None = None,
synthesis_md: str | None = None,
) -> str:
_ = fun_level
md = render.render_for_html_comparison(
entity_reports, synthesis_md=synthesis_md, save_path=save_path,
)
md = _strip_evidence_block(md)
md = _strip_invitation(md)
md = _strip_canonical_boundary(md)
md = _promote_prose_labels(md)
body = _markdown_to_html(md)
body = _wrap_engine_footer(body)
body = _promote_meta_marker(body)
topic = " vs ".join(label for label, _ in entity_reports)
colophon = _build_colophon(entity_reports[0][1], topic=topic)
return _wrap_in_template(body, colophon, topic)
def _strip_evidence_block(md: str) -> str:
return EVIDENCE_BLOCK_PATTERN.sub("", md)
def _strip_invitation(md: str) -> str:
return INVITATION_PATTERN.sub("", md)
def _strip_canonical_boundary(md: str) -> str:
return CANONICAL_BOUNDARY_PATTERN.sub("", md)
def _promote_prose_labels(md: str) -> str:
for source, normalized in PROSE_LABELS:
md = re.sub(
rf"^{re.escape(source)}$",
f"## {normalized}",
md,
flags=re.MULTILINE,
)
return md
def _markdown_to_html(md: str) -> str:
md, footers = _protect_engine_footers(md)
global _ENGINE_FOOTER_STORE
_ENGINE_FOOTER_STORE = footers
# Strip HTML comments EXCEPT preserved markers used for post-processing
# (META is promoted to <div class="meta"> after markdown conversion).
md = re.sub(r"<!--(?!\s*META:).*?-->", "", md, flags=re.DOTALL)
lines = md.splitlines()
out: list[str] = []
paragraph: list[str] = []
list_type: str | None = None
in_code = False
code_lines: list[str] = []
index = 0
def flush_paragraph() -> None:
nonlocal paragraph
if paragraph:
text = " ".join(part.strip() for part in paragraph).strip()
if text:
out.append(f"<p>{_inline_markdown(text)}</p>")
paragraph = []
def close_list() -> None:
nonlocal list_type
if list_type:
out.append(f"</{list_type}>")
list_type = None
while index < len(lines):
line = lines[index]
stripped = line.strip()
if in_code:
if stripped.startswith("```"):
out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
code_lines = []
in_code = False
else:
code_lines.append(line)
index += 1
continue
if stripped.startswith("```"):
flush_paragraph()
close_list()
in_code = True
code_lines = []
index += 1
continue
if stripped in footers:
flush_paragraph()
close_list()
out.append(stripped)
index += 1
continue
if not stripped:
flush_paragraph()
close_list()
index += 1
continue
if stripped == "---":
flush_paragraph()
close_list()
out.append("<hr>")
index += 1
continue
if index + 1 < len(lines) and _is_table_row(stripped) and _is_table_separator(lines[index + 1].strip()):
flush_paragraph()
close_list()
table_lines = [stripped]
index += 2
while index < len(lines) and _is_table_row(lines[index].strip()):
table_lines.append(lines[index].strip())
index += 1
out.append(_render_table(table_lines))
continue
heading = re.match(r"^(#{1,4})\s+(.+)$", stripped)
if heading:
flush_paragraph()
close_list()
level = min(len(heading.group(1)), 3)
out.append(f"<h{level}>{_inline_markdown(heading.group(2))}</h{level}>")
index += 1
continue
if stripped.startswith(">"):
flush_paragraph()
close_list()
quote_lines = []
while index < len(lines) and lines[index].strip().startswith(">"):
quote_lines.append(lines[index].strip().lstrip(">").strip())
index += 1
out.append(f"<blockquote>{_inline_markdown(' '.join(quote_lines))}</blockquote>")
continue
unordered = re.match(r"^[-*]\s+(.+)$", stripped)
ordered = re.match(r"^\d+[.)]\s+(.+)$", stripped)
if unordered or ordered:
flush_paragraph()
next_type = "ul" if unordered else "ol"
if list_type != next_type:
close_list()
out.append(f"<{next_type}>")
list_type = next_type
item = unordered.group(1) if unordered else ordered.group(1)
out.append(f"<li>{_inline_markdown(item)}</li>")
index += 1
continue
if stripped.startswith("🌐 last30days"):
flush_paragraph()
close_list()
badge_text = _inline_markdown(stripped.removeprefix("🌐").strip())
out.append(f'<div class="badge"><span class="accent">🌐</span> {badge_text}</div>')
index += 1
continue
paragraph.append(line)
index += 1
if in_code:
out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
flush_paragraph()
close_list()
return "\n".join(out).strip()
def _protect_engine_footers(md: str) -> tuple[str, dict[str, str]]:
footers: dict[str, str] = {}
def replace(match: re.Match[str]) -> str:
token = f"__LAST30DAYS_ENGINE_FOOTER_{len(footers)}__"
footers[token] = match.group(1).strip("\n")
return f"\n{token}\n"
return PASS_THROUGH_FOOTER_PATTERN.sub(replace, md), footers
def _wrap_engine_footer(body: str) -> str:
def replace(match: re.Match[str]) -> str:
footer = html.escape(_ENGINE_FOOTER_STORE.get(match.group(0), ""), quote=False)
return f'<div class="engine-footer"><pre>{footer}</pre></div>'
return re.sub(
r"__LAST30DAYS_ENGINE_FOOTER_\d+__",
replace,
body,
)
def _promote_meta_marker(body: str) -> str:
"""Promote ``<!-- META: ... -->`` markers into a styled ``<div class="meta">``.
The marker is preserved through the comment-strip pass (see
_markdown_to_html exemption) but the markdown converter wraps it in
``<p>`` and HTML-escapes the angle brackets. After conversion the body
contains shapes like:
<p>&lt;!-- META: TEXT --&gt;</p>
<p><!-- META: TEXT --></p> (when not escaped)
Both collapse to ``<div class="meta">TEXT</div>``.
"""
def replace(match: re.Match[str]) -> str:
text = match.group(1).strip()
return f'<div class="meta">{text}</div>'
# Escaped form (most common after markdown conversion)
body = re.sub(
r"<p>\s*&lt;!--\s*META:\s*(.*?)\s*--&gt;\s*</p>",
replace,
body,
)
body = re.sub(r"&lt;!--\s*META:\s*(.*?)\s*--&gt;", replace, body)
# Unescaped form (paranoid fallback)
body = re.sub(r"<p>\s*<!--\s*META:\s*(.*?)\s*-->\s*</p>", replace, body)
body = re.sub(r"<!--\s*META:\s*(.*?)\s*-->", replace, body)
return body
_ENGINE_FOOTER_STORE: dict[str, str] = {}
def _inline_markdown(text: str) -> str:
escaped = html.escape(text, quote=True)
code_tokens: dict[str, str] = {}
def code_replace(match: re.Match[str]) -> str:
token = f"__CODE_{len(code_tokens)}__"
code_tokens[token] = f"<code>{match.group(1)}</code>"
return token
escaped = re.sub(r"`([^`]+)`", code_replace, escaped)
escaped = re.sub(r"\*\*([^*]+)\*\*", r"<strong>\1</strong>", escaped)
escaped = re.sub(
r"\[([^\]]+)\]\(([^)\s]+)\)",
r'<a href="\2">\1</a>',
escaped,
)
for token, value in code_tokens.items():
escaped = escaped.replace(token, value)
return escaped
def _is_table_row(line: str) -> bool:
return "|" in line and len(_split_table_cells(line)) >= 2
def _is_table_separator(line: str) -> bool:
cells = _split_table_cells(line)
return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell.strip()) for cell in cells)
def _split_table_cells(line: str) -> list[str]:
return [cell.strip() for cell in line.strip().strip("|").split("|")]
def _render_table(rows: list[str]) -> str:
header = _split_table_cells(rows[0])
body_rows = [_split_table_cells(row) for row in rows[1:]]
out = ["<table>", "<thead>", "<tr>"]
out.extend(f"<th>{_inline_markdown(cell)}</th>" for cell in header)
out.extend(["</tr>", "</thead>", "<tbody>"])
for row in body_rows:
out.append("<tr>")
out.extend(f"<td>{_inline_markdown(cell)}</td>" for cell in row)
out.append("</tr>")
out.extend(["</tbody>", "</table>"])
return "\n".join(out)
def _build_colophon(report: schema.Report, *, topic: str | None = None) -> str:
display_topic = topic or report.topic
generated = _generated_date(report)
version = render._skill_version()
escaped_topic = html.escape(display_topic)
rerun = html.escape(f"/last30days {display_topic}")
return (
'<div class="colophon">\n'
f" Generated {generated} by /last30days v{html.escape(version)} · topic: {escaped_topic}<br>\n"
f' Re-run for fresh data: <span class="rerun">{rerun}</span>\n'
"</div>"
)
def _generated_date(report: schema.Report) -> str:
if report.generated_at:
return report.generated_at[:10]
return date.today().strftime("%Y-%m-%d")
def _wrap_in_template(body: str, colophon: str, title: str) -> str:
return (
HTML_TEMPLATE
.replace("__TITLE__", html.escape(title))
.replace("__CSS__", CSS)
.replace("__BODY__", body)
.replace("__COLOPHON__", colophon)
)
@@ -2,6 +2,7 @@
import json import json
import re import re
import socket
import sys import sys
import time import time
import urllib.error import urllib.error
@@ -22,9 +23,19 @@ def log(msg: str):
MAX_RETRIES = 5 MAX_RETRIES = 5
MAX_429_RETRIES = 2 MAX_429_RETRIES = 2
RETRY_DELAY = 2.0 RETRY_DELAY = 2.0
# DNS resolution failures (gaierror) are transient — typically resolved by a
# brief backoff and retry. Use a dedicated minimum attempt count + exponential
# delays (1s, 2s, 4s) so callers that pass a small `retries` value still get a
# meaningful chance to recover from a transient resolution failure.
MIN_DNS_RETRIES = 3
USER_AGENT = "last30days-skill/3.0 (Assistant Skill)" USER_AGENT = "last30days-skill/3.0 (Assistant Skill)"
def _is_dns_failure(err: urllib.error.URLError) -> bool:
"""Return True if a URLError was caused by DNS resolution (gaierror)."""
return isinstance(getattr(err, "reason", None), socket.gaierror)
class HTTPError(Exception): class HTTPError(Exception):
"""HTTP request error with status code.""" """HTTP request error with status code."""
def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None): def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
@@ -85,7 +96,13 @@ def request(
last_error = None last_error = None
rate_limit_count = 0 rate_limit_count = 0
for attempt in range(retries): # DNS failures get a dedicated minimum attempt count + exponential backoff.
# `effective_retries` is the actual loop bound; we expand it on the first
# gaierror if the caller passed a smaller `retries` value than MIN_DNS_RETRIES.
effective_retries = retries
dns_attempts = 0
attempt = 0
while attempt < effective_retries:
try: try:
with urllib.request.urlopen(req, timeout=timeout) as response: with urllib.request.urlopen(req, timeout=timeout) as response:
body = response.read().decode('utf-8') body = response.read().decode('utf-8')
@@ -115,6 +132,8 @@ def request(
if rate_limit_count >= max_429_retries: if rate_limit_count >= max_429_retries:
raise last_error raise last_error
# HTTP errors respect the caller's original `retries`; only DNS
# failures get the widened `effective_retries` budget.
if attempt < retries - 1: if attempt < retries - 1:
if e.code == 429: if e.code == 429:
# Respect Retry-After header, fall back to exponential backoff # Respect Retry-After header, fall back to exponential backoff
@@ -130,11 +149,43 @@ def request(
else: else:
delay = RETRY_DELAY * (2 ** attempt) delay = RETRY_DELAY * (2 ** attempt)
time.sleep(delay) time.sleep(delay)
else:
# Caller's original retry budget exhausted; an earlier DNS
# failure may have widened `effective_retries`, but that
# widening is DNS-only — don't grant extra HTTP attempts.
break
except urllib.error.URLError as e: except urllib.error.URLError as e:
log(f"URL Error: {e.reason}") log(f"URL Error: {e.reason}")
last_error = HTTPError(f"URL Error: {e.reason}") last_error = HTTPError(f"URL Error: {e.reason}")
if attempt < retries - 1: if _is_dns_failure(e):
# DNS resolution failures are transient; expand the retry budget
# to MIN_DNS_RETRIES if the caller passed fewer, and use
# exponential backoff (1s, 2s, 4s, ...) instead of the linear
# default. Counts DNS attempts separately so other URLError
# causes don't bypass the regular retry budget.
dns_attempts += 1
if effective_retries < MIN_DNS_RETRIES:
log(
f"DNS resolution failed; expanding retry budget from "
f"{effective_retries} to {MIN_DNS_RETRIES}"
)
effective_retries = MIN_DNS_RETRIES
if attempt < effective_retries - 1:
delay = 2 ** (dns_attempts - 1) # 1s, 2s, 4s, 8s, ...
log(
f"DNS resolution failure (attempt {dns_attempts}); "
f"retrying in {delay:.1f}s"
)
time.sleep(delay)
elif attempt < retries - 1:
# Non-DNS URLError (e.g. ConnectionRefused) respects the
# caller's original retry budget, not the DNS-widened bound.
time.sleep(RETRY_DELAY * (attempt + 1)) time.sleep(RETRY_DELAY * (attempt + 1))
else:
# Caller's original retry budget exhausted; an earlier DNS
# failure widening `effective_retries` does not carry over
# to non-DNS error paths.
break
except json.JSONDecodeError as e: except json.JSONDecodeError as e:
log(f"JSON decode error: {e}") log(f"JSON decode error: {e}")
last_error = HTTPError(f"Invalid JSON response: {e}") last_error = HTTPError(f"Invalid JSON response: {e}")
@@ -144,7 +195,13 @@ def request(
log(f"Connection error: {type(e).__name__}: {e}") log(f"Connection error: {type(e).__name__}: {e}")
last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}") last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
if attempt < retries - 1: if attempt < retries - 1:
# Socket errors respect the caller's original retry budget.
time.sleep(RETRY_DELAY * (attempt + 1)) time.sleep(RETRY_DELAY * (attempt + 1))
else:
# Original budget exhausted; DNS widening doesn't apply here.
break
attempt += 1
if last_error: if last_error:
raise last_error raise last_error
@@ -12,11 +12,6 @@ import sys
from datetime import datetime from datetime import datetime
from typing import Any, Dict, List, Optional, Set from typing import Any, Dict, List, Optional, Set
try:
import requests as _requests
except ImportError:
_requests = None
from . import dates, http, log from . import dates, http, log
SCRAPECREATORS_BASE = "https://api.scrapecreators.com" SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
@@ -236,27 +231,14 @@ def _user_reels(
""" """
_log(f"User reels: @{handle}") _log(f"User reels: @{handle}")
reels_url = f"{SCRAPECREATORS_BASE}/v1/instagram/user/reels" reels_url = f"{SCRAPECREATORS_BASE}/v1/instagram/user/reels"
if not _requests:
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"handle": handle})
url = f"{reels_url}?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"User reels error (urllib) for @{handle}: {e}")
return []
else:
try:
resp = _requests.get(
reels_url, reels_url,
params={"handle": handle}, params={"handle": handle},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"User reels error for @{handle}: {e}") _log(f"User reels error for @{handle}: {e}")
return [] return []
@@ -293,28 +275,14 @@ def search_instagram(
_log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})") _log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
if not _requests:
_log("requests library not installed, falling back to urllib")
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"query": core_topic})
url = f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search", f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
params={"query": core_topic}, params={"query": core_topic},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"ScrapeCreators error: {e}") _log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"} return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -367,7 +335,7 @@ def fetch_captions(
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"]) config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = config["max_captions"] max_captions = config["max_captions"]
if not video_items or not token or not _requests: if not video_items or not token:
return {} return {}
top_items = video_items[:max_captions] top_items = video_items[:max_captions]
@@ -392,17 +360,15 @@ def fetch_captions(
if not url: if not url:
continue continue
try: try:
resp = _requests.get( data = http.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript", f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
params={"url": url}, params={"url": url},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=15, timeout=15,
retries=1,
) )
if resp.status_code == 200:
data = resp.json()
transcripts = data.get("transcripts") or [] transcripts = data.get("transcripts") or []
if transcripts and isinstance(transcripts, list): if transcripts and isinstance(transcripts, list):
# Combine all transcript segments
transcript_text = " ".join( transcript_text = " ".join(
t.get("text", "") for t in transcripts t.get("text", "") for t in transcripts
if isinstance(t, dict) and t.get("text") if isinstance(t, dict) and t.get("text")
@@ -49,6 +49,7 @@ def normalize_source_items(
"xquik": _normalize_x, "xquik": _normalize_x,
"pinterest": _normalize_pinterest, "pinterest": _normalize_pinterest,
"polymarket": _normalize_polymarket, "polymarket": _normalize_polymarket,
"digg": _normalize_digg,
"grounding": _normalize_grounding, "grounding": _normalize_grounding,
"xiaohongshu": _normalize_grounding, "xiaohongshu": _normalize_grounding,
"github": _normalize_github, "github": _normalize_github,
@@ -110,6 +111,19 @@ def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> An
return default return default
def _join_comment_excerpts(
top_comments: list[Any],
key: str,
limit: int = 3,
) -> str:
"""Space-join the `key` field from the first `limit` dict-shaped comments."""
return " ".join(
str(comment.get(key) or "").strip()
for comment in top_comments[:limit]
if isinstance(comment, dict)
)
def _domain_from_url(url: str) -> str | None: def _domain_from_url(url: str) -> str | None:
if not url: if not url:
return None return None
@@ -169,11 +183,7 @@ def _normalize_reddit(
to_date: str, to_date: str,
) -> schema.SourceItem: ) -> schema.SourceItem:
top_comments = item.get("top_comments") or [] top_comments = item.get("top_comments") or []
comment_text = " ".join( comment_text = _join_comment_excerpts(top_comments, "excerpt")
str(comment.get("excerpt") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
body = "\n".join( body = "\n".join(
part part
for part in [ for part in [
@@ -338,11 +348,7 @@ def _normalize_hackernews(
to_date: str, to_date: str,
) -> schema.SourceItem: ) -> schema.SourceItem:
top_comments = item.get("top_comments") or [] top_comments = item.get("top_comments") or []
comment_text = " ".join( comment_text = _join_comment_excerpts(top_comments, "text")
str(comment.get("text") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
title = str(item.get("title") or "").strip() title = str(item.get("title") or "").strip()
body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part) body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part)
return _source_item( return _source_item(
@@ -394,6 +400,53 @@ def _normalize_microblog(
) )
def _normalize_digg(
source: str,
item: dict[str, Any],
index: int,
from_date: str,
to_date: str,
) -> schema.SourceItem:
"""Normalizer for Digg AI 1000 clusters.
Each cluster is one item. The TLDR carries the most useful body for
rerank and synthesis. Top-ranked X posts attached at search time are
passed through under metadata['posts'] so render can emit them as
inline 'via Digg' quotes.
"""
title = str(item.get("title") or "").strip()
tldr = str(item.get("tldr") or "").strip()
body = "\n\n".join(part for part in [title, tldr] if part)
posts = item.get("posts") or []
if not isinstance(posts, list):
posts = []
cluster_url_id = str(item.get("id") or f"DG{index + 1}")
return _source_item(
item_id=cluster_url_id,
source=source,
title=title or f"Digg cluster {index + 1}",
body=body,
url=str(item.get("url") or f"https://di.gg/ai/{cluster_url_id}"),
author="",
container="Digg",
published_at=item.get("date"),
date_confidence=_date_confidence(item, from_date, to_date, default="high"),
engagement=item.get("engagement") or {},
relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""),
snippet=tldr[:400],
metadata={
"clusterUrlId": cluster_url_id,
"tldr": tldr,
"rank": (item.get("engagement") or {}).get("rank"),
"uniqueAuthors": (item.get("engagement") or {}).get("uniqueAuthors"),
"postCount": (item.get("engagement") or {}).get("postCount"),
"firstPostAge": item.get("first_post_age"),
"posts": posts,
},
)
def _normalize_polymarket( def _normalize_polymarket(
source: str, source: str,
item: dict[str, Any], item: dict[str, Any],
@@ -441,11 +494,7 @@ def _normalize_github(
title = str(item.get("title") or "").strip() title = str(item.get("title") or "").strip()
snippet_text = str(item.get("snippet") or "").strip() snippet_text = str(item.get("snippet") or "").strip()
top_comments = item.get("metadata", {}).get("top_comments") or [] top_comments = item.get("metadata", {}).get("top_comments") or []
comment_text = " ".join( comment_text = _join_comment_excerpts(top_comments, "excerpt")
str(comment.get("excerpt") or "").strip()
for comment in top_comments[:3]
if isinstance(comment, dict)
)
body = "\n".join(part for part in [title, snippet_text, comment_text] if part) body = "\n".join(part for part in [title, snippet_text, comment_text] if part)
metadata = item.get("metadata") or {} metadata = item.get("metadata") or {}
return _source_item( return _source_item(
@@ -11,11 +11,6 @@ import re
import sys import sys
from typing import Any, Dict, List, Optional, Set from typing import Any, Dict, List, Optional, Set
try:
import requests as _requests
except ImportError:
_requests = None
from . import dates, http, log from . import dates, http, log
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/pinterest" SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/pinterest"
@@ -140,28 +135,14 @@ def search_pinterest(
_log(f"Searching Pinterest for '{core_topic}' (depth={depth}, count={config['results_per_page']})") _log(f"Searching Pinterest for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
if not _requests:
_log("requests library not installed, falling back to urllib")
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"keyword": core_topic})
url = f"{SCRAPECREATORS_BASE}/search?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search", f"{SCRAPECREATORS_BASE}/search",
params={"keyword": core_topic}, params={"keyword": core_topic},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"ScrapeCreators error: {e}") _log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"} return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -15,6 +15,7 @@ from . import (
bluesky, bluesky,
dates, dates,
dedupe, dedupe,
digg,
entity_extract, entity_extract,
env, env,
github, github,
@@ -30,6 +31,7 @@ from . import (
query, query,
reddit, reddit,
reddit_public, reddit_public,
relevance,
rerank, rerank,
schema, schema,
signals, signals,
@@ -40,6 +42,7 @@ from . import (
xai_x, xai_x,
xiaohongshu_api, xiaohongshu_api,
xquik, xquik,
xurl_x,
youtube_yt, youtube_yt,
) )
from .cluster import cluster_candidates from .cluster import cluster_candidates
@@ -77,6 +80,7 @@ MOCK_AVAILABLE_SOURCES = [
"github", "github",
"perplexity", "perplexity",
"xquik", "xquik",
"digg",
] ]
@@ -104,6 +108,8 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.extend(["hackernews", "polymarket"]) available.extend(["hackernews", "polymarket"])
if config.get("GITHUB_TOKEN") or which("gh"): if config.get("GITHUB_TOKEN") or which("gh"):
available.append("github") available.append("github")
if which("digg-pp-cli"):
available.append("digg")
if env.is_bluesky_available(config): if env.is_bluesky_available(config):
available.append("bluesky") available.append("bluesky")
if env.is_truthsocial_available(config): if env.is_truthsocial_available(config):
@@ -122,6 +128,9 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.append("pinterest") available.append("pinterest")
if env.is_xquik_available(config): if env.is_xquik_available(config):
available.append("xquik") available.append("xquik")
exclude = {s.strip().lower() for s in (config.get("EXCLUDE_SOURCES") or "").split(",") if s.strip()}
if exclude:
available = [s for s in available if s not in exclude]
return available return available
@@ -177,6 +186,7 @@ def run(
lookback_days: int = 30, lookback_days: int = 30,
github_user: str | None = None, github_user: str | None = None,
github_repos: list[str] | None = None, github_repos: list[str] | None = None,
internal_subrun: bool = False,
) -> schema.Report: ) -> schema.Report:
settings = DEPTH_SETTINGS[depth] settings = DEPTH_SETTINGS[depth]
requested_sources = normalize_requested_sources(requested_sources) requested_sources = normalize_requested_sources(requested_sources)
@@ -214,6 +224,7 @@ def run(
provider=None if mock else reasoning_provider, provider=None if mock else reasoning_provider,
model=None if mock else runtime.planner_model, model=None if mock else runtime.planner_model,
context=config.get("_auto_resolve_context", ""), context=config.get("_auto_resolve_context", ""),
internal_subrun=internal_subrun,
) )
# Source labelling: the fallback path annotates notes with "fallback-plan" # Source labelling: the fallback path annotates notes with "fallback-plan"
# or "deterministic-comparison-plan"; anything else came from the LLM. # or "deterministic-comparison-plan"; anything else came from the LLM.
@@ -440,7 +451,7 @@ def run(
if bundle.items_by_source.get(source): if bundle.items_by_source.get(source):
del bundle.errors_by_source[source] del bundle.errors_by_source[source]
items_by_source = _finalize_items_by_source(bundle.items_by_source, topic=topic) items_by_source = _finalize_items_by_source(bundle.items_by_source, topic=topic, config=config)
candidates = weighted_rrf(bundle.items_by_source_and_query, plan, pool_limit=settings["pool_limit"]) candidates = weighted_rrf(bundle.items_by_source_and_query, plan, pool_limit=settings["pool_limit"])
ranked_candidates = rerank.rerank_candidates( ranked_candidates = rerank.rerank_candidates(
topic=topic, topic=topic,
@@ -497,17 +508,19 @@ def _normalize_score_dedupe(
source, raw_items, from_date, to_date, source, raw_items, from_date, to_date,
freshness_mode=freshness_mode, freshness_mode=freshness_mode,
) )
normalized = signals.annotate_stream(normalized, ranking_query, freshness_mode) prepared_query = relevance.PreparedQuery(ranking_query)
normalized = signals.annotate_stream(normalized, prepared_query, freshness_mode)
normalized = signals.prune_low_relevance(normalized) normalized = signals.prune_low_relevance(normalized)
normalized = dedupe.dedupe_items(normalized) normalized = dedupe.dedupe_items(normalized)
for item in normalized: for item in normalized:
item.snippet = snippet.extract_best_snippet(item, ranking_query) item.snippet = snippet.extract_best_snippet(item, prepared_query)
return normalized return normalized
def _finalize_items_by_source( def _finalize_items_by_source(
items_by_source_raw: dict[str, list[schema.SourceItem]], items_by_source_raw: dict[str, list[schema.SourceItem]],
topic: str = "", topic: str = "",
config: dict | None = None,
) -> dict[str, list[schema.SourceItem]]: ) -> dict[str, list[schema.SourceItem]]:
finalized = {} finalized = {}
for source, items in items_by_source_raw.items(): for source, items in items_by_source_raw.items():
@@ -520,6 +533,17 @@ def _finalize_items_by_source(
# (e.g., WTI crude oil, Elon tweet counts) before footer emission. # (e.g., WTI crude oil, Elon tweet counts) before footer emission.
if source == "polymarket" and topic: if source == "polymarket" and topic:
items = polymarket.filter_items_against_topic(topic, items) items = polymarket.filter_items_against_topic(topic, items)
# --polymarket-keywords (via config): additional keyword filter
# for ambiguous single-token topics (e.g., "Warriors" → nba,gsw).
keywords = config.get("_polymarket_keywords") if isinstance(config, dict) else None
if keywords:
items = polymarket.filter_items_against_keywords(items, keywords)
if source == "digg" and items:
# Pull top-ranked X posts only for the survivors that will appear
# in the brief. Spending the enrichment budget here (rather than
# at retrieval time) keeps the inline 'via Digg' quotes
# paired with the clusters dedupe actually kept.
digg.enrich_source_items(items, top_k=3)
finalized[source] = items finalized[source] = items
return finalized return finalized
@@ -895,6 +919,9 @@ def _retrieve_stream(
depth=depth, depth=depth,
) )
return xai_x.parse_x_response(result), {} return xai_x.parse_x_response(result), {}
if backend == "xurl":
result = xurl_x.search_x(subquery.search_query, depth=depth)
return xurl_x.parse_x_response(result, topic=subquery.search_query), {}
raise RuntimeError("No X backend is available.") raise RuntimeError("No X backend is available.")
if source == "youtube": if source == "youtube":
# Use raw_topic so expand_youtube_queries() generates diverse variants # Use raw_topic so expand_youtube_queries() generates diverse variants
@@ -952,6 +979,13 @@ def _retrieve_stream(
if source == "hackernews": if source == "hackernews":
result = hackernews.search_hackernews(subquery.search_query, from_date, to_date, depth=depth) result = hackernews.search_hackernews(subquery.search_query, from_date, to_date, depth=depth)
return hackernews.parse_hackernews_response(result, query=subquery.search_query), {} return hackernews.parse_hackernews_response(result, query=subquery.search_query), {}
if source == "digg":
result = digg.search_digg(subquery.search_query, from_date, to_date, depth=depth)
items = digg.parse_digg_response(result, query=subquery.search_query)
# Enrichment with attached X posts is deferred to
# _finalize_items_by_source so it runs on the items that actually
# survive dedupe rather than on top-K of the raw fanout.
return items, {}
if source == "bluesky": if source == "bluesky":
result = bluesky.search_bluesky(subquery.search_query, from_date, to_date, depth=depth, config=config) result = bluesky.search_bluesky(subquery.search_query, from_date, to_date, depth=depth, config=config)
return bluesky.parse_bluesky_response(result), {} return bluesky.parse_bluesky_response(result), {}
@@ -1044,6 +1078,45 @@ def _mock_stream_results(source: str, subquery: schema.SubQuery) -> tuple[list[d
"why_relevant": "Brave web search", "why_relevant": "Brave web search",
} }
], ],
"digg": [
{
"id": "mock1abc",
"title": f"Digg cluster about {subquery.search_query}",
"url": "https://di.gg/ai/mock1abc",
"tldr": f"Curated cluster summarizing recent {subquery.search_query} discussion across the AI 1000.",
"author": "",
"date": dates.get_date_range(3)[0],
"engagement": {"postCount": 8, "uniqueAuthors": 5, "rank": 2, "rank_score": 49.0},
"first_post_age": "3d",
"posts": [
{
"username": "exampledev",
"display_name": "Example Dev",
"category": "Engineer",
"rank": 142,
"body": f"Quote from the AI 1000 about {subquery.search_query}.",
"post_type": "tweet",
"x_url": "https://x.com/exampledev/status/1",
"posted_at": dates.get_date_range(3)[0],
},
],
"relevance": 0.84,
"why_relevant": "Mock Digg cluster",
},
{
"id": "mock2def",
"title": f"Second Digg cluster on {subquery.search_query}",
"url": "https://di.gg/ai/mock2def",
"tldr": f"Another angle on {subquery.search_query}.",
"author": "",
"date": dates.get_date_range(8)[0],
"engagement": {"postCount": 3, "uniqueAuthors": 2, "rank": 18, "rank_score": 33.0},
"first_post_age": "8d",
"posts": [],
"relevance": 0.71,
"why_relevant": "Mock Digg cluster",
},
],
} }
if source == "grounding": if source == "grounding":
return payloads.get(source, []), { return payloads.get(source, []), {
@@ -67,6 +67,7 @@ SOURCE_CAPABILITIES = {
"bluesky": {"discussion", "social"}, "bluesky": {"discussion", "social"},
"truthsocial": {"discussion", "social"}, "truthsocial": {"discussion", "social"},
"polymarket": {"market"}, "polymarket": {"market"},
"digg": {"discussion", "social", "link"},
"xiaohongshu": {"video", "video_shortform", "social"}, "xiaohongshu": {"video", "video_shortform", "social"},
"github": {"discussion", "link"}, "github": {"discussion", "link"},
"grounding": {"web", "reference", "link"}, "grounding": {"web", "reference", "link"},
@@ -86,9 +87,16 @@ def plan_query(
provider: providers.ReasoningClient | None, provider: providers.ReasoningClient | None,
model: str | None, model: str | None,
context: str = "", context: str = "",
internal_subrun: bool = False,
) -> schema.QueryPlan: ) -> schema.QueryPlan:
"""Create a query plan. Comparison queries with extractable entities use a """Create a query plan. Comparison queries with extractable entities use a
deterministic plan; other intents prefer the configured reasoning provider.""" deterministic plan; other intents prefer the configured reasoning provider.
internal_subrun: when True, suppress the LAW 7 "No --plan passed" stderr
warning. LAW 7 targets the hosting-reasoning-model path; competitor
fan-out sub-runs are engine-internal and the warning is a false positive
there. Default False preserves the warning on every user-facing invocation.
"""
if _should_force_deterministic_plan(topic): if _should_force_deterministic_plan(topic):
return _fallback_plan( return _fallback_plan(
topic, topic,
@@ -122,6 +130,7 @@ def plan_query(
# planner credentials - NOT a prerequisite the caller needs. If you are # planner credentials - NOT a prerequisite the caller needs. If you are
# the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19 # the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
# Hermes Agent Use Cases failure mode. # Hermes Agent Use Cases failure mode.
if not internal_subrun:
import sys import sys
print( print(
"[Planner] No --plan passed. If you are the reasoning model hosting " "[Planner] No --plan passed. If you are the reasoning model hosting "
@@ -232,6 +232,39 @@ def filter_items_against_topic(topic: str, items: List[Any]) -> List[Any]:
return filtered return filtered
def filter_items_against_keywords(items: List[Any], keywords: List[str]) -> List[Any]:
"""Keep only items whose title contains at least one keyword (case-insensitive).
Intended for disambiguating ambiguous single-token topics like 'Warriors'
via --polymarket-keywords (e.g., 'nba,gsw,golden-state') to filter out
Glasgow Warriors rugby, Honor of Kings Rogue Warriors markets that share
the 'Warriors' token but are not the target entity.
"""
if not keywords:
return items
normalized_keywords = [kw.strip().lower() for kw in keywords if kw and kw.strip()]
if not normalized_keywords:
return items
filtered = []
for item in items:
title = getattr(item, "title", None)
if title is None and isinstance(item, dict):
title = item.get("title", "")
title = (title or "").lower()
if any(kw in title for kw in normalized_keywords):
filtered.append(item)
dropped = len(items) - len(filtered)
if dropped:
_log(
f"Keyword filter dropped {dropped} Polymarket items; "
f"kept {len(filtered)} matching {normalized_keywords}"
)
return filtered
def _extract_domain_queries(topic: str, events: List[Dict]) -> List[str]: def _extract_domain_queries(topic: str, events: List[Dict]) -> List[str]:
"""Extract domain-indicator search terms from first-pass event tags. """Extract domain-indicator search terms from first-pass event tags.
@@ -9,7 +9,7 @@ from typing import Any
from . import env, http, schema from . import env, http, schema
GEMINI_FLASH_LITE = "gemini-3.1-flash-lite-preview" GEMINI_FLASH_LITE = "gemini-3.1-flash-lite"
GEMINI_PRO = "gemini-3.1-pro-preview" GEMINI_PRO = "gemini-3.1-pro-preview"
OPENAI_DEFAULT = "gpt-5.4-nano" OPENAI_DEFAULT = "gpt-5.4-nano"
XAI_DEFAULT = "grok-4-1-fast" XAI_DEFAULT = "grok-4-1-fast"
@@ -93,13 +93,6 @@ class GeminiClient(ReasoningClient):
) )
return extract_gemini_text(payload) return extract_gemini_text(payload)
def ground_search(self, model: str, prompt: str) -> dict[str, Any]:
return self._generate_content(model, prompt, tools=[{"google_search": {}}])
def url_context_json(self, model: str, prompt: str) -> dict[str, Any]:
return self.generate_json(model, prompt, tools=[{"url_context": {}}])
class OpenAIClient(ReasoningClient): class OpenAIClient(ReasoningClient):
name = "openai" name = "openai"
@@ -239,8 +232,8 @@ def _resolve_model_pins(config: dict[str, Any], depth: str, provider_name: str)
rerank_model = config.get("LAST30DAYS_RERANK_MODEL") or default_rerank rerank_model = config.get("LAST30DAYS_RERANK_MODEL") or default_rerank
if provider_name == "gemini": if provider_name == "gemini":
_require_gemini_31_preview(planner_model, role="planner") _require_gemini_31(planner_model, role="planner")
_require_gemini_31_preview(rerank_model, role="rerank") _require_gemini_31(rerank_model, role="rerank")
return planner_model, rerank_model return planner_model, rerank_model
@@ -351,11 +344,11 @@ def _resolve_x_backend(config: dict[str, Any]) -> str | None:
return env.get_x_source(config) return env.get_x_source(config)
def _require_gemini_31_preview(model: str, *, role: str) -> None: def _require_gemini_31(model: str, *, role: str) -> None:
if model.startswith("gemini-3.1-") and model.endswith("-preview"): if model.startswith("gemini-3.1-"):
return return
raise RuntimeError( raise RuntimeError(
f"{role} must use a Gemini 3.1 preview model. Got: {model}" f"{role} must use a Gemini 3.1 model. Got: {model}"
) )
@@ -71,8 +71,29 @@ def _normalize_phrase(text: str) -> str:
return ' '.join(re.sub(r'[^\w\s]', ' ', text.lower()).split()) return ' '.join(re.sub(r'[^\w\s]', ' ', text.lower()).split())
class PreparedQuery:
"""Precomputed query shape reused across items in a stream.
Built once per ranking_query; reused by token_overlap_relevance so the
per-item normalize/score loops don't re-tokenize the same query N times.
"""
__slots__ = ("raw", "q_tokens", "informative_q_tokens", "normalized_phrase")
def __init__(self, query: str) -> None:
self.raw = query
self.q_tokens = tokenize(query)
informative = {t for t in self.q_tokens if t not in LOW_SIGNAL_QUERY_TOKENS}
self.informative_q_tokens = informative or self.q_tokens
self.normalized_phrase = _normalize_phrase(query)
def _as_prepared(query: "str | PreparedQuery") -> PreparedQuery:
return query if isinstance(query, PreparedQuery) else PreparedQuery(query)
def token_overlap_relevance( def token_overlap_relevance(
query: str, query: "str | PreparedQuery",
text: str, text: str,
hashtags: Optional[List[str]] = None, hashtags: Optional[List[str]] = None,
) -> float: ) -> float:
@@ -95,7 +116,8 @@ def token_overlap_relevance(
Returns: Returns:
Float between 0.0 and 1.0 (0.5 for empty queries) Float between 0.0 and 1.0 (0.5 for empty queries)
""" """
q_tokens = tokenize(query) prepared = _as_prepared(query)
q_tokens = prepared.q_tokens
# Combine text and hashtags for matching # Combine text and hashtags for matching
combined = text combined = text
@@ -119,9 +141,7 @@ def token_overlap_relevance(
if overlap == 0: if overlap == 0:
return 0.0 return 0.0
informative_q_tokens = {t for t in q_tokens if t not in LOW_SIGNAL_QUERY_TOKENS} informative_q_tokens = prepared.informative_q_tokens
if not informative_q_tokens:
informative_q_tokens = q_tokens
coverage = overlap / len(q_tokens) coverage = overlap / len(q_tokens)
informative_overlap = len(informative_q_tokens & t_tokens) / len(informative_q_tokens) informative_overlap = len(informative_q_tokens & t_tokens) / len(informative_q_tokens)
@@ -129,7 +149,7 @@ def token_overlap_relevance(
precision = overlap / precision_denominator precision = overlap / precision_denominator
phrase_bonus = 0.0 phrase_bonus = 0.0
normalized_query = _normalize_phrase(query) normalized_query = prepared.normalized_phrase
normalized_text = _normalize_phrase(combined) normalized_text = _normalize_phrase(combined)
if normalized_query and normalized_query in normalized_text: if normalized_query and normalized_query in normalized_text:
phrase_bonus = 0.12 if len(normalized_query.split()) > 1 else 0.16 phrase_bonus = 0.12 if len(normalized_query.split()) > 1 else 0.16
@@ -8,24 +8,40 @@ from collections import Counter
from datetime import date from datetime import date
from urllib.parse import urlparse from urllib.parse import urlparse
from . import dates, schema from . import dates, schema, skill_meta
def _skill_version() -> str: def _skill_version() -> str:
"""Read plugin version from .claude-plugin/plugin.json if available. """Read plugin version from .claude-plugin/plugin.json, falling back to SKILL.md frontmatter.
Tries nearest plugin.json by walking up from render.py's own location. Per-harness skill install dirs (`~/.claude/skills`, `~/.codex/skills`, `~/.agents/skills`,
Falls back to "?" if not found. This keeps the badge emission from Hermes, etc.) do not always carry `.claude-plugin/plugin.json` that file ships with
crashing on non-plugin-cache installs (repo checkout, Gemini, Codex). plugin-cache installs but not with per-harness skill installs. SKILL.md frontmatter is
the fallback that keeps the badge from emitting v? on those installs. Returns "?" only
if no usable version string is found from either source (missing files, corrupt JSON,
or SKILL.md without a version line).
A corrupt manifest at one ancestor does not shadow a valid manifest at a deeper one
(continue, not break). SKILL.md parsing accepts double-quoted, single-quoted, or
unquoted YAML version scalars (delegated to skill_meta.read_skill_version).
""" """
here = pathlib.Path(__file__).resolve() here = pathlib.Path(__file__).resolve()
for parent in [here.parent, *here.parents]: for parent in here.parents:
candidate = parent / ".claude-plugin" / "plugin.json" manifest = parent / ".claude-plugin" / "plugin.json"
if candidate.is_file(): if manifest.is_file():
try: try:
return json.loads(candidate.read_text()).get("version", "?") version = json.loads(manifest.read_text()).get("version")
except (json.JSONDecodeError, OSError): except (json.JSONDecodeError, OSError):
return "?" continue
if version:
return version
# No usable manifest found at any ancestor — fall back to SKILL.md frontmatter.
# First SKILL.md found in the walk is THIS skill's; never traverse past it.
for parent in here.parents:
skill_md = parent / "SKILL.md"
if skill_md.is_file():
return skill_meta.read_skill_version(skill_md) or "?"
return "?" return "?"
@@ -52,6 +68,7 @@ SOURCE_LABELS = {
"xiaohongshu": "Xiaohongshu", "xiaohongshu": "Xiaohongshu",
"x": "X", "x": "X",
"github": "GitHub", "github": "GitHub",
"digg": "Digg",
"perplexity": "Perplexity", "perplexity": "Perplexity",
} }
@@ -79,7 +96,7 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items] non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [ lines = [
*_render_badge(), *_render_badge(),
f"# last30days v3.0.0: {report.topic}", f"# last30days v{_skill_version()}: {report.topic}",
"", "",
*_assistant_safety_lines(), *_assistant_safety_lines(),
f"- Date range: {report.range_from} to {report.range_to}", f"- Date range: {report.range_from} to {report.range_to}",
@@ -170,6 +187,168 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
return "\n".join(lines).strip() + "\n" return "\n".join(lines).strip() + "\n"
def render_for_html(
report: schema.Report,
synthesis_md: str | None = None,
*,
save_path: str | None = None,
) -> str:
"""Render markdown intended for shareable HTML conversion.
This output keeps the public badge, compact source/date metadata, an
optional one-line data quality note, optional synthesized brief markdown,
and the engine footer. It deliberately omits the debug file header,
model-facing safety note, and evidence scratchpad emitted by
render_compact().
When synthesis_md is None, the body is intentionally sparse: badge,
metadata, optional data quality note, and engine footer only.
"""
lines = [
*_render_badge(),
*_render_html_metadata(report),
]
if synthesis_md:
lines.extend(["", synthesis_md.strip()])
# Data quality warnings are NOT rendered into the HTML artifact. The HTML
# is meant to be shared (Slack, email, Notion); recipients haven't asked
# for technical commentary about how the run was produced. Generators see
# the same warnings via collect_html_warnings() routed to stderr by the
# CLI, so they can fix quality issues before sharing.
_append_html_footer(lines, report, save_path)
return "\n".join(lines).strip() + "\n"
def render_for_html_comparison(
entity_reports: list[tuple[str, schema.Report]],
synthesis_md: str | None = None,
*,
save_path: str | None = None,
) -> str:
"""Render comparison markdown intended for shareable HTML conversion.
Same semantics as render_for_html(), but metadata and data quality notes
are aggregated across the compared entities.
"""
if not entity_reports:
raise ValueError("render_for_html_comparison requires at least one report")
entities = [label for label, _ in entity_reports]
main_report = entity_reports[0][1]
meta = (
f"<!-- META: {main_report.range_from} to {main_report.range_to} "
f"· comparing {len(entities)}: {', '.join(entities)} -->"
)
lines = [
*_render_badge(),
meta,
]
if synthesis_md:
lines.extend(["", synthesis_md.strip()])
# Comparison data quality notes also go to stderr, not into the artifact.
_append_html_footer(lines, main_report, save_path)
return "\n".join(lines).strip() + "\n"
def collect_html_warnings(report: schema.Report) -> list[str]:
"""Collect data quality warnings for stderr output (NOT for the HTML artifact).
Returns a list of human-readable warning strings. Empty list if the run
was clean. Used by the CLI to emit diagnostics to stderr after writing
the HTML to stdout/file.
"""
notes: list[str] = []
if _render_degraded_run_warning(report):
notes.append("Run was missing pre-flight resolution. Re-run with `--plan` for richer results.")
elif _render_pre_research_warning(report):
notes.append("Pre-research was skipped, so results may be thinner than a resolved run.")
freshness_warning = _assess_data_freshness(report)
if freshness_warning:
notes.append(freshness_warning)
notes.extend(report.warnings)
return _dedupe_notes(notes)
def collect_html_warnings_comparison(
entity_reports: list[tuple[str, schema.Report]],
) -> list[str]:
"""Collect comparison-mode warnings, prefixed by entity label."""
notes: list[str] = []
for label, report in entity_reports:
for w in collect_html_warnings(report):
notes.append(f"{label}: {w}")
return notes
def _render_html_metadata(report: schema.Report) -> list[str]:
"""Inline metadata as an HTML comment marker.
html_render.py post-processes ``<!-- META: ... -->`` markers into a
``<div class="meta">`` after markdown conversion, so the metadata escapes
the markdown converter's HTML-escaping pass cleanly. Same pattern as the
PASS_THROUGH_FOOTER marker used for the engine tree.
"""
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
if non_empty:
sources = ", ".join(_source_label(s) for s in non_empty)
else:
sources = "no active sources"
return [
f"<!-- META: {report.range_from} to {report.range_to} · {sources} -->",
]
def _render_html_data_quality_note(report: schema.Report) -> str | None:
notes: list[str] = []
degraded_warning = _render_degraded_run_warning(report)
if degraded_warning:
notes.append("This run was missing pre-flight resolution. Re-run with `--plan` for richer results.")
pre_research_warning = _render_pre_research_warning(report)
if pre_research_warning and not degraded_warning:
notes.append("Pre-research was skipped, so results may be thinner than a resolved run.")
freshness_warning = _assess_data_freshness(report)
if freshness_warning:
notes.append(freshness_warning)
notes.extend(report.warnings)
if not notes:
return None
return f"> **Data quality note:** {' '.join(_dedupe_notes(notes))}"
def _render_html_comparison_data_quality_note(
entity_reports: list[tuple[str, schema.Report]],
) -> str | None:
notes: list[str] = []
for label, report in entity_reports:
note = _render_html_data_quality_note(report)
if note:
clean = note.removeprefix("> **Data quality note:** ").strip()
notes.append(f"{label}: {clean}")
if not notes:
return None
return f"> **Data quality note:** {' '.join(_dedupe_notes(notes))}"
def _dedupe_notes(notes: list[str]) -> list[str]:
out: list[str] = []
seen: set[str] = set()
for note in notes:
normalized = " ".join(str(note).split())
if not normalized or normalized in seen:
continue
seen.add(normalized)
out.append(normalized)
return out
def _append_html_footer(lines: list[str], report: schema.Report, save_path: str | None) -> None:
footer = _render_emoji_footer(report, save_path)
lines.append("")
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
lines.extend(footer)
lines.append("<!-- END PASS-THROUGH FOOTER -->")
def _render_canonical_boundary() -> list[str]: def _render_canonical_boundary() -> list[str]:
"""Emit the explicit END-OF-CANONICAL-OUTPUT boundary. """Emit the explicit END-OF-CANONICAL-OUTPUT boundary.
@@ -392,12 +571,226 @@ def _render_comparison_scaffold(topic: str) -> list[str]:
] ]
def render_comparison_multi(
entity_reports: list[tuple[str, schema.Report]],
*,
cluster_limit: int = 4,
fun_level: str = "medium",
save_path: str | None = None,
) -> str:
"""Render N (entity, Report) pairs as a single comparison output.
Reuses _render_comparison_scaffold for the synthesis table and emits
per-entity evidence sections inside one EVIDENCE FOR SYNTHESIS envelope.
The single-Report render_compact path is unchanged.
Args:
entity_reports: Ordered (label, Report) pairs. The first pair is the
user's main topic; the remainder are discovered/explicit competitors.
cluster_limit: Max clusters to surface per entity (kept lower than the
single-entity default to keep N-way comparisons readable).
fun_level: Same fun-level knob as render_compact, applied to each
entity's best-takes block.
save_path: Optional save-path display string for the footer.
"""
if not entity_reports:
raise ValueError("render_comparison_multi requires at least one report")
entities = [label for label, _ in entity_reports]
main_label, main_report = entity_reports[0]
synthesized_topic = " vs ".join(entities)
lines: list[str] = [
*_render_badge(),
f"# last30days v{_skill_version()}: {synthesized_topic}",
"",
*_assistant_safety_lines(),
f"- Comparison mode: {len(entities)} entities ({', '.join(entities)})",
f"- Date range: {main_report.range_from} to {main_report.range_to}",
"",
]
aggregated_warnings: list[str] = []
for label, report in entity_reports:
aggregated_warnings.extend(f"[{label}] {w}" for w in report.warnings)
if aggregated_warnings:
lines.append("## Warnings")
lines.extend(f"- {w}" for w in aggregated_warnings)
lines.append("")
lines.append(
"<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into "
"`What I learned:` prose per LAW 2. Each entity has its own evidence subsection. -->"
)
lines.append("")
resolved_block = _render_resolved_entities_block(entity_reports)
if resolved_block:
lines.extend(resolved_block)
lines.append("")
fun_params = _FUN_LEVELS.get(fun_level, _FUN_LEVELS["medium"])
for label, report in entity_reports:
lines.extend(_render_entity_evidence_block(
label=label,
report=report,
cluster_limit=cluster_limit,
fun_params=fun_params,
))
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
lines.append("")
# Reuse the existing comparison scaffold by feeding it the synthesized
# topic. _parse_comparison_entities splits on " vs " so the scaffold
# picks up all N entities automatically.
scaffold = _render_comparison_scaffold(synthesized_topic)
lines.extend(scaffold)
footer = _render_emoji_footer(main_report, save_path)
if footer:
lines.append("")
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
lines.extend(footer)
lines.append("<!-- END PASS-THROUGH FOOTER -->")
lines.extend(_render_canonical_boundary())
return "\n".join(lines).strip() + "\n"
def _render_resolved_entities_block(
entity_reports: list[tuple[str, schema.Report]],
) -> list[str]:
"""Emit a visible per-entity Step 0.55 resolution summary.
Reads `resolved` dicts from each Report's artifacts. Returns an empty
list when no entity has a resolved payload (mock mode, no web backend,
or artifacts not populated). Missing per-entity fields render as `-`.
Context strings truncate at 120 chars.
"""
any_resolved = any(
isinstance(report.artifacts.get("resolved"), dict)
for _label, report in entity_reports
)
if not any_resolved:
return []
out: list[str] = ["## Resolved Entities", ""]
for label, report in entity_reports:
resolved = report.artifacts.get("resolved") or {}
x_handle = resolved.get("x_handle") or ""
subs = resolved.get("subreddits") or []
gh_user = resolved.get("github_user") or ""
gh_repos = resolved.get("github_repos") or []
context = resolved.get("context") or ""
x_display = f"@{x_handle}" if x_handle else "-"
subs_display = (
", ".join(f"r/{s}" for s in subs[:5]) + (
f" (+{len(subs) - 5})" if len(subs) > 5 else ""
)
) if subs else "-"
gh_display = f"@{gh_user}" if gh_user else "-"
if gh_repos:
gh_display += f" ({', '.join(gh_repos[:3])}" + (
f" +{len(gh_repos) - 3}" if len(gh_repos) > 3 else ""
) + ")"
context_display = _truncate(context, 120) if context else "-"
out.append(
f"- **{label}**: X {x_display} | Subs {subs_display} | "
f"GitHub {gh_display} | Context: {context_display}"
)
return out
def _render_entity_evidence_block(
*,
label: str,
report: schema.Report,
cluster_limit: int,
fun_params: dict,
) -> list[str]:
"""Render one entity's clusters and best-takes inside the evidence envelope."""
candidate_by_id = {c.candidate_id: c for c in report.ranked_candidates}
out: list[str] = [f"## {label}", ""]
if not report.clusters:
out.append("(no significant discussion this month)")
out.append("")
return out
out.append("### Ranked Evidence Clusters")
out.append("")
for index, cluster in enumerate(report.clusters[:cluster_limit], start=1):
out.append(
f"#### {index}. {cluster.title} "
f"(score {cluster.score:.0f}, {len(cluster.candidate_ids)} item"
f"{'s' if len(cluster.candidate_ids) != 1 else ''}, "
f"sources: {', '.join(_source_label(s) for s in cluster.sources)})"
)
if cluster.uncertainty:
out.append(f"- Uncertainty: {cluster.uncertainty}")
for rep_index, candidate_id in enumerate(cluster.representative_ids, start=1):
candidate = candidate_by_id.get(candidate_id)
if not candidate:
continue
out.extend(_render_candidate(candidate, prefix=f"{rep_index}."))
out.append("")
best_takes = _render_best_takes(
report.ranked_candidates,
limit=fun_params["limit"],
threshold=fun_params["threshold"],
)
if best_takes:
out.extend(best_takes)
out.append("")
return out
def render_comparison_multi_context(
entity_reports: list[tuple[str, schema.Report]],
cluster_limit: int = 4,
) -> str:
"""Context-mode rendering for the multi-entity comparison."""
if not entity_reports:
raise ValueError("render_comparison_multi_context requires at least one report")
entities = [label for label, _ in entity_reports]
lines = [
f"Comparison: {' vs '.join(entities)}",
f"Entities: {len(entities)}",
_AI_SAFETY_NOTE,
"",
]
resolved_block = _render_resolved_entities_block(entity_reports)
if resolved_block:
lines.extend(resolved_block)
lines.append("")
for label, report in entity_reports:
lines.append(f"## {label}")
lines.append(f"Intent: {report.query_plan.intent}")
if not report.clusters:
lines.append("- (no significant discussion this month)")
else:
for cluster in report.clusters[:cluster_limit]:
lines.append(
f"- {cluster.title} "
f"[{', '.join(_source_label(s) for s in cluster.sources)}]"
)
lines.append("")
return "\n".join(lines).strip() + "\n"
def render_full(report: schema.Report) -> str: def render_full(report: schema.Report) -> str:
"""Full data dump: ALL clusters + ALL items by source. For saved files and debugging.""" """Full data dump: ALL clusters + ALL items by source. For saved files and debugging."""
# Start with the same header as compact # Start with the same header as compact
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items] non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [ lines = [
f"# last30days v3.0.0: {report.topic}", f"# last30days v{_skill_version()}: {report.topic}",
"", "",
*_assistant_safety_lines(), *_assistant_safety_lines(),
f"- Date range: {report.range_from} to {report.range_to}", f"- Date range: {report.range_from} to {report.range_to}",
@@ -410,6 +803,17 @@ def render_full(report: schema.Report) -> str:
lines.extend(f"- {warning}" for warning in report.warnings) lines.extend(f"- {warning}" for warning in report.warnings)
lines.append("") lines.append("")
# When this Report is a per-entity sub-run from vs-mode / --competitors,
# include the single-row Resolved Entities block so the saved file is
# self-describing. The artifact is populated by last30days.py's
# _competitor_runner and _main_runner closures.
resolved = report.artifacts.get("resolved")
if isinstance(resolved, dict) and resolved.get("entity"):
single_row = _render_resolved_entities_block([(resolved["entity"], report)])
if single_row:
lines.extend(single_row)
lines.append("")
# ALL clusters (no limit) # ALL clusters (no limit)
lines.append("## Ranked Evidence Clusters") lines.append("## Ranked Evidence Clusters")
lines.append("") lines.append("")
@@ -438,7 +842,7 @@ def render_full(report: schema.Report) -> str:
lines.append("## All Items by Source") lines.append("## All Items by Source")
lines.append("") lines.append("")
source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest", source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest",
"hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "perplexity"] "hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "digg", "perplexity"]
for source in source_order: for source in source_order:
items = report.items_by_source.get(source, []) items = report.items_by_source.get(source, [])
if not items: if not items:
@@ -462,7 +866,11 @@ def render_full(report: schema.Report) -> str:
for tc in top_comments[:3]: for tc in top_comments[:3]:
excerpt = tc.get("excerpt", tc.get("text", ""))[:200] excerpt = tc.get("excerpt", tc.get("text", ""))[:200]
tc_score = tc.get("score", "") tc_score = tc.get("score", "")
lines.append(f" Top comment ({tc_score} {vote_label}): {excerpt}") attribution = _comment_attribution(item.source, tc.get("author"))
lines.append(f" Top comment {attribution} ({tc_score} {vote_label}): {excerpt}")
# Digg: inline X-post quotes attached to the cluster.
for post in _digg_posts_for(item, limit=3):
lines.append(f" > {_format_digg_quote(post)}")
# Comment insights for Reddit # Comment insights for Reddit
insights = item.metadata.get("comment_insights", []) insights = item.metadata.get("comment_insights", [])
if insights: if insights:
@@ -581,7 +989,11 @@ def _render_candidate(candidate: schema.Candidate, prefix: str) -> list[str]:
excerpt = tc.get("excerpt") or tc.get("text") or "" excerpt = tc.get("excerpt") or tc.get("text") or ""
score = tc.get("score", "") score = tc.get("score", "")
vote_label = _vote_label_for(primary.source) if primary else "upvotes" vote_label = _vote_label_for(primary.source) if primary else "upvotes"
lines.append(f" - Comment ({score} {vote_label}): {_truncate(excerpt.strip(), 240)}") source = primary.source if primary else None
attribution = _comment_attribution(source, tc.get("author"))
lines.append(f" - {attribution} ({score} {vote_label}): {_truncate(excerpt.strip(), 240)}")
for post in _digg_posts_for(primary):
lines.append(f" - {_format_digg_quote(post)}")
insight = _comment_insight(primary) insight = _comment_insight(primary)
if insight: if insight:
lines.append(f" - Insight: {_truncate(insight, 220)}") lines.append(f" - Insight: {_truncate(insight, 220)}")
@@ -832,6 +1244,7 @@ _FOOTER_SOURCES: list[tuple[str, str, str, str, list[tuple[str, str]]]] = [
("bluesky", "🦋", "Bluesky", "post", [("likes", "likes"), ("reposts", "reposts")]), ("bluesky", "🦋", "Bluesky", "post", [("likes", "likes"), ("reposts", "reposts")]),
("truthsocial", "🇺🇸", "Truth Social", "post", [("likes", "likes"), ("reposts", "reposts")]), ("truthsocial", "🇺🇸", "Truth Social", "post", [("likes", "likes"), ("reposts", "reposts")]),
("github", "🐙", "GitHub", "item", [("reactions", "reactions"), ("comments", "comments")]), ("github", "🐙", "GitHub", "item", [("reactions", "reactions"), ("comments", "comments")]),
("digg", "⛏️", "Digg", "cluster", [("postCount", "posts"), ("uniqueAuthors", "authors")]),
] ]
@@ -1089,6 +1502,7 @@ ENGAGEMENT_DISPLAY: dict[str, list[tuple[str, str]]] = {
"polymarket": [], "polymarket": [],
"github": [("reactions", "react"), ("comments", "cmt")], "github": [("reactions", "react"), ("comments", "cmt")],
"perplexity": [("citations", "cite")], "perplexity": [("citations", "cite")],
"digg": [("postCount", "posts"), ("uniqueAuthors", "auth")],
} }
@@ -1232,6 +1646,33 @@ def _vote_label_for(source: str) -> str:
return _TOP_COMMENT_VOTE_LABEL.get(source, "votes") return _TOP_COMMENT_VOTE_LABEL.get(source, "votes")
# Handle prefixes for commenter attribution. Reddit uses `u/`; everyone else
# uses `@`. Missing source or unknown platform falls back to plain-text so
# we never emit `u/` or `@` with no handle attached.
_HANDLE_PREFIX: dict[str, str] = {
"reddit": "u/",
"tiktok": "@",
"youtube": "@",
"instagram": "@",
"bluesky": "@",
"x": "@",
"threads": "@",
}
def _comment_attribution(source: str | None, author: str | None) -> str:
"""Build the attribution prefix for a top comment line.
Returns a string like ``u/Cyrisaurus`` or ``@moosanoormahomed`` when an
author is captured, or the legacy ``Comment`` marker when the author is
missing, empty, deleted, or removed.
"""
if not author or author in ("[deleted]", "[removed]"):
return "Comment"
prefix = _HANDLE_PREFIX.get(source or "", "")
return f"{prefix}{author}" if prefix else author
def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int | None = None) -> list[dict]: def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int | None = None) -> list[dict]:
"""Return up to `limit` top comments with score at or above the source's minimum. """Return up to `limit` top comments with score at or above the source's minimum.
@@ -1249,16 +1690,6 @@ def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score
return [c for c in comments if (c.get("score") or 0) >= min_score][:limit] return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]
def _top_comment_excerpt(item: schema.SourceItem | None) -> str | None:
if not item:
return None
comments = item.metadata.get("top_comments") or []
if not comments or not isinstance(comments[0], dict):
return None
top = comments[0]
return str(top.get("excerpt") or top.get("text") or "").strip() or None
def _comment_insight(item: schema.SourceItem | None) -> str | None: def _comment_insight(item: schema.SourceItem | None) -> str | None:
if not item: if not item:
return None return None
@@ -1268,6 +1699,39 @@ def _comment_insight(item: schema.SourceItem | None) -> str | None:
return str(insights[0]).strip() or None return str(insights[0]).strip() or None
def _digg_posts_for(item: schema.SourceItem | None, limit: int = 3) -> list[dict]:
"""Return up to `limit` parsed Digg posts attached as enrichment to a cluster.
Returns an empty list for non-digg sources or clusters without enrichment.
"""
if not item or item.source != "digg":
return []
posts = item.metadata.get("posts") or []
if not isinstance(posts, list):
return []
out: list[dict] = []
for entry in posts:
if isinstance(entry, dict) and entry.get("body") and entry.get("username"):
out.append(entry)
if len(out) >= limit:
break
return out
def _format_digg_quote(post: dict, body_limit: int = 200) -> str:
"""Format a Digg-attached X post as an inline 'via Digg' quote line."""
handle = post.get("username") or ""
x_url = post.get("x_url") or ""
body = (post.get("body") or "").replace("\n", " ").strip()
if len(body) > body_limit:
body = body[: body_limit - 1].rstrip() + ""
if x_url and handle:
return f"[@{handle}]({x_url}) via Digg: {body}"
if handle:
return f"@{handle} via Digg: {body}"
return f"via Digg: {body}"
def _transcript_highlights(item: schema.SourceItem | None) -> list[str]: def _transcript_highlights(item: schema.SourceItem | None) -> list[str]:
if not item or item.source != "youtube": if not item or item.source != "youtube":
return [] return []
@@ -11,14 +11,64 @@ import re
import sys import sys
from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Optional
from . import dates, grounding from . import categories, dates, grounding
MAX_SUBS = 10
def _log(msg: str) -> None: def _log(msg: str) -> None:
print(f"[Resolve] {msg}", file=sys.stderr) print(f"[Resolve] {msg}", file=sys.stderr)
def _merge_category_peers(topic: str, subreddits: list[str]) -> tuple[list[str], Optional[str]]:
"""Extend the WebSearch-extracted subreddit list with category peers.
Classifies the topic, fetches the category's peer subs, dedupes
case-insensitively against the existing list, and appends missing
peers in priority order. Caps the final list at MAX_SUBS, preserving
every WebSearch-returned sub (they are the freshest signal) and
trimming from the peer-additions end.
Returns a tuple of (merged_subs, matched_category_id_or_None).
Emits a [Resolve] Matched category log line only when peers were
actually added (not when every peer was already in the WebSearch set).
Classification failures degrade to "no match" the unwidened list
is returned and a warning is logged.
"""
try:
category = categories.detect_category(topic)
except Exception as exc:
_log(f"Category classification failed: {exc}")
return list(subreddits)[:MAX_SUBS], None
if category is None:
return list(subreddits)[:MAX_SUBS], None
peers = categories.peer_subs_for(category)
if not peers:
return list(subreddits)[:MAX_SUBS], category
existing_lower = {s.lower() for s in subreddits}
merged = list(subreddits)
added: list[str] = []
for peer in peers:
if len(merged) >= MAX_SUBS:
break
if peer.lower() in existing_lower:
continue
merged.append(peer)
existing_lower.add(peer.lower())
added.append(peer)
if added:
_log(f"Matched category={category}, adding peers: {', '.join(added)}")
return merged, category
def _has_backend(config: dict) -> bool: def _has_backend(config: dict) -> bool:
"""Check if any web search backend is available.""" """Check if any web search backend is available."""
return bool( return bool(
@@ -134,10 +184,19 @@ def auto_resolve(topic: str, config: dict) -> dict:
config: Dict with API keys (BRAVE_API_KEY, EXA_API_KEY, SERPER_API_KEY). config: Dict with API keys (BRAVE_API_KEY, EXA_API_KEY, SERPER_API_KEY).
Returns: Returns:
Dict with keys: subreddits, x_handle, context, searches_run. Dict with keys: subreddits, x_handle, github_user, github_repos,
Returns empty result if no web search backend is available. context, category, searches_run. Returns empty result if no web
search backend is available.
""" """
empty = {"subreddits": [], "x_handle": "", "context": "", "searches_run": 0} empty = {
"subreddits": [],
"x_handle": "",
"github_user": "",
"github_repos": [],
"context": "",
"category": None,
"searches_run": 0,
}
if not _has_backend(config): if not _has_backend(config):
_log("No web search backend available, skipping resolve") _log("No web search backend available, skipping resolve")
@@ -184,7 +243,9 @@ def auto_resolve(topic: str, config: dict) -> dict:
github_repos = _extract_github_repos(results.get("github", [])) github_repos = _extract_github_repos(results.get("github", []))
context = _build_context_summary(results.get("news", [])) context = _build_context_summary(results.get("news", []))
_log(f"Resolved {len(subreddits)} subreddits, x_handle={x_handle!r}, github_user={github_user!r}, github_repos={github_repos!r}, context_len={len(context)}") subreddits, category = _merge_category_peers(topic, subreddits)
_log(f"Resolved {len(subreddits)} subreddits, x_handle={x_handle!r}, github_user={github_user!r}, github_repos={github_repos!r}, context_len={len(context)}, category={category!r}")
return { return {
"subreddits": subreddits, "subreddits": subreddits,
@@ -192,5 +253,6 @@ def auto_resolve(topic: str, config: dict) -> dict:
"github_user": github_user, "github_user": github_user,
"github_repos": github_repos, "github_repos": github_repos,
"context": context, "context": context,
"category": category,
"searches_run": searches_run, "searches_run": searches_run,
} }
@@ -335,8 +335,9 @@ def poll_device_auth(
""" """
import sys import sys
deadline = time.time() + timeout started_at = time.time()
last_reminder = time.time() deadline = started_at + timeout
last_reminder = started_at
reminder_count = 0 reminder_count = 0
max_reminders = 4 max_reminders = 4
reminder_interval = 30 # seconds between reminders reminder_interval = 30 # seconds between reminders
@@ -12,6 +12,7 @@ SOURCE_QUALITY = {
"xiaohongshu": 0.7, "xiaohongshu": 0.7,
"hackernews": 0.8, "hackernews": 0.8,
"youtube": 0.85, "youtube": 0.85,
"digg": 0.85,
"reddit": 0.6, "reddit": 0.6,
"x": 0.68, "x": 0.68,
"bluesky": 0.66, "bluesky": 0.66,
@@ -26,7 +27,10 @@ def source_quality(source: str) -> float:
return SOURCE_QUALITY.get(source, 0.6) return SOURCE_QUALITY.get(source, 0.6)
def local_relevance(item: schema.SourceItem, ranking_query: str) -> float: def local_relevance(
item: schema.SourceItem,
ranking_query: "str | relevance.PreparedQuery",
) -> float:
text = "\n".join( text = "\n".join(
part part
for part in [item.title, item.body, item.snippet] for part in [item.title, item.body, item.snippet]
@@ -92,6 +96,7 @@ ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
"bluesky": [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)], "bluesky": [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)],
"truthsocial": [("likes", 0.45), ("reposts", 0.30), ("replies", 0.25)], "truthsocial": [("likes", 0.45), ("reposts", 0.30), ("replies", 0.25)],
"polymarket": [("volume", 0.60), ("liquidity", 0.40)], "polymarket": [("volume", 0.60), ("liquidity", 0.40)],
"digg": [("postCount", 0.40), ("uniqueAuthors", 0.30), ("rank_score", 0.30)],
} }
@@ -175,13 +180,14 @@ def normalize(values: list[float | None]) -> list[int | None]:
def annotate_stream( def annotate_stream(
items: list[schema.SourceItem], items: list[schema.SourceItem],
ranking_query: str, ranking_query: "str | relevance.PreparedQuery",
freshness_mode: str, freshness_mode: str,
) -> list[schema.SourceItem]: ) -> list[schema.SourceItem]:
"""Attach local scoring metadata and return items sorted by local_rank_score.""" """Attach local scoring metadata and return items sorted by local_rank_score."""
prepared_query = ranking_query if isinstance(ranking_query, relevance.PreparedQuery) else relevance.PreparedQuery(ranking_query)
engagement_scores = normalize([engagement_raw(item) for item in items]) engagement_scores = normalize([engagement_raw(item) for item in items])
for item, eng_score in zip(items, engagement_scores, strict=True): for item, eng_score in zip(items, engagement_scores, strict=True):
item.local_relevance = local_relevance(item, ranking_query) item.local_relevance = local_relevance(item, prepared_query)
item.freshness = freshness(item, freshness_mode) item.freshness = freshness(item, freshness_mode)
item.engagement_score = eng_score item.engagement_score = eng_score
item.source_quality = source_quality(item.source) item.source_quality = source_quality(item.source)
@@ -0,0 +1,33 @@
"""SKILL.md metadata helpers — single source of truth for parsing skill frontmatter.
Centralizes the version regex that previously lived in render.py and was
duplicated in tests/test_plugin_contract.py and tests/test_version_consistency.py.
"""
import re
from pathlib import Path
# Matches `version: "x.y.z"`, `version: 'x.y.z'`, or `version: x.y.z` in YAML
# frontmatter. Multiline so the pattern can be applied to a full SKILL.md text.
# Three alternation groups — exactly one captures per successful match.
_VERSION_RE = re.compile(
r'''^version:\s*(?:"([^"]+)"|'([^']+)'|(\S+))\s*$''',
re.MULTILINE,
)
def read_skill_version(skill_md_path: Path) -> str | None:
"""Return the version string from a SKILL.md's frontmatter, or None.
Returns None if the file can't be read (missing, permission, decode error)
or if no `version:` line is found. Accepts double-quoted, single-quoted,
or unquoted YAML version scalars.
"""
try:
text = skill_md_path.read_text()
except (OSError, UnicodeDecodeError):
return None
match = _VERSION_RE.search(text)
if not match:
return None
return match.group(1) or match.group(2) or match.group(3)
@@ -26,7 +26,7 @@ def _windows(words: list[str], size: int, overlap: int) -> list[str]:
def extract_best_snippet( def extract_best_snippet(
item: schema.SourceItem, item: schema.SourceItem,
ranking_query: str, ranking_query: "str | relevance.PreparedQuery",
max_words: int = 120, max_words: int = 120,
) -> str: ) -> str:
"""Prefer existing snippets, else extract the best matching evidence window.""" """Prefer existing snippets, else extract the best matching evidence window."""
@@ -43,8 +43,9 @@ def extract_best_snippet(
if not candidates: if not candidates:
return _truncate_words(body, max_words) return _truncate_words(body, max_words)
prepared_query = ranking_query if isinstance(ranking_query, relevance.PreparedQuery) else relevance.PreparedQuery(ranking_query)
best = max( best = max(
candidates, candidates,
key=lambda candidate: relevance.token_overlap_relevance(ranking_query, candidate), key=lambda candidate: relevance.token_overlap_relevance(prepared_query, candidate),
) )
return _truncate_words(best, max_words) return _truncate_words(best, max_words)
+94
View File
@@ -0,0 +1,94 @@
"""Subprocess helpers: safe timeout + process-group cleanup.
Used by bird_x.py (Node.js Bird search) and youtube_yt.py (yt-dlp search
and transcript download). Both need the same os.setsid/killpg cleanup
dance on timeout to avoid orphaning child processes.
"""
from __future__ import annotations
import os
import signal
import subprocess
from dataclasses import dataclass
from typing import Optional, Sequence
class SubprocTimeout(Exception):
"""Raised when a subprocess exceeds its timeout and is killed."""
@dataclass
class SubprocResult:
"""Result of a subprocess run that captured stdout and stderr."""
returncode: int
stdout: str
stderr: str
def run_with_timeout(
cmd: Sequence[str],
*,
timeout: int,
env: Optional[dict] = None,
on_pid: Optional[callable] = None,
) -> SubprocResult:
"""Run a subprocess with process-group cleanup on timeout.
Spawns ``cmd`` inside its own process group via ``os.setsid`` where
available. If ``communicate(timeout=...)`` raises ``TimeoutExpired``,
signals ``SIGTERM`` to the entire group, falls back to ``proc.kill()``
if the signal fails, then waits up to 5 seconds for cleanup, and
raises ``SubprocTimeout``.
Args:
cmd: Command and arguments to spawn.
timeout: Timeout in seconds passed to ``communicate()``.
env: Optional environment dict. If None, inherits parent env.
on_pid: Optional callable invoked with the child PID right after
spawn. Used by bird_x.py to register child PIDs for cleanup
tracking. Exceptions raised by the callback are suppressed.
Returns:
SubprocResult with returncode, stdout, and stderr as strings.
Raises:
SubprocTimeout: If the process exceeded ``timeout``.
FileNotFoundError: If the executable is not found.
OSError: For other spawn failures.
"""
preexec = os.setsid if hasattr(os, "setsid") else None
proc = subprocess.Popen(
list(cmd),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
encoding="utf-8",
errors="replace",
preexec_fn=preexec,
env=env,
)
if on_pid is not None:
try:
on_pid(proc.pid)
except Exception:
pass
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
raise SubprocTimeout(f"Command {cmd[0]} timed out after {timeout}s")
return SubprocResult(
returncode=proc.returncode,
stdout=stdout or "",
stderr=stderr or "",
)
@@ -152,32 +152,13 @@ def search_threads(
_log(f"Searching for '{core_topic}' (depth={depth}, limit={config['results']})") _log(f"Searching for '{core_topic}' (depth={depth}, limit={config['results']})")
try: try:
import requests as _requests data = http.get(
except ImportError:
_requests = None
if not _requests:
_log("requests library not installed, falling back to urllib")
try:
from urllib.parse import urlencode
params = urlencode({"keyword": core_topic})
url = f"{SCRAPECREATORS_BASE}/search?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search", f"{SCRAPECREATORS_BASE}/search",
params={"keyword": core_topic}, params={"keyword": core_topic},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"ScrapeCreators error: {e}") _log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"} return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -11,11 +11,6 @@ import re
import sys import sys
from typing import Any, Dict, List, Optional, Set from typing import Any, Dict, List, Optional, Set
try:
import requests as _requests
except ImportError:
_requests = None
from . import dates, http, log from . import dates, http, log
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/tiktok" SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/tiktok"
@@ -214,27 +209,14 @@ def _hashtag_search(
List of raw TikTok item dicts (aweme_info format). List of raw TikTok item dicts (aweme_info format).
""" """
_log(f"Hashtag search: #{hashtag}") _log(f"Hashtag search: #{hashtag}")
if not _requests:
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"hashtag": hashtag})
url = f"{SCRAPECREATORS_BASE}/search/hashtag?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"Hashtag search error (urllib) for #{hashtag}: {e}")
return []
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search/hashtag", f"{SCRAPECREATORS_BASE}/search/hashtag",
params={"hashtag": hashtag}, params={"hashtag": hashtag},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"Hashtag search error for #{hashtag}: {e}") _log(f"Hashtag search error for #{hashtag}: {e}")
return [] return []
@@ -261,27 +243,14 @@ def _profile_videos(
""" """
_log(f"Profile videos: @{handle}") _log(f"Profile videos: @{handle}")
profile_url = "https://api.scrapecreators.com/v3/tiktok/profile/videos" profile_url = "https://api.scrapecreators.com/v3/tiktok/profile/videos"
if not _requests:
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"handle": handle, "sort_by": "latest"})
url = f"{profile_url}?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"Profile videos error (urllib) for @{handle}: {e}")
return []
else:
try:
resp = _requests.get(
profile_url, profile_url,
params={"handle": handle, "sort_by": "latest"}, params={"handle": handle, "sort_by": "latest"},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"Profile videos error for @{handle}: {e}") _log(f"Profile videos error for @{handle}: {e}")
return [] return []
@@ -318,28 +287,14 @@ def search_tiktok(
_log(f"Searching TikTok for '{core_topic}' (depth={depth}, count={config['results_per_page']})") _log(f"Searching TikTok for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
if not _requests:
_log("requests library not installed, falling back to urllib")
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"query": core_topic, "sort_by": "relevance"})
url = f"{SCRAPECREATORS_BASE}/search/keyword?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as e:
_log(f"ScrapeCreators error (urllib): {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/search/keyword", f"{SCRAPECREATORS_BASE}/search/keyword",
params={"query": core_topic, "sort_by": "relevance"}, params={"query": core_topic, "sort_by": "relevance"},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as e: except Exception as e:
_log(f"ScrapeCreators error: {e}") _log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"} return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -397,7 +352,7 @@ def fetch_captions(
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"]) config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = config["max_captions"] max_captions = config["max_captions"]
if not video_items or not token or not _requests: if not video_items or not token:
return {} return {}
top_items = video_items[:max_captions] top_items = video_items[:max_captions]
@@ -422,14 +377,13 @@ def fetch_captions(
if not url: if not url:
continue continue
try: try:
resp = _requests.get( data = http.get(
f"{SCRAPECREATORS_BASE}/video/transcript", f"{SCRAPECREATORS_BASE}/video/transcript",
params={"url": url}, params={"url": url},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=15, timeout=15,
retries=1,
) )
if resp.status_code == 200:
data = resp.json()
transcript = data.get("transcript") transcript = data.get("transcript")
if transcript: if transcript:
if isinstance(transcript, list): if isinstance(transcript, list):
@@ -620,27 +574,14 @@ def _fetch_post_comments(
List of comment dicts with author, text, digg_count (likes), date. List of comment dicts with author, text, digg_count (likes), date.
Empty list on any error comment failures never crash the pipeline. Empty list on any error comment failures never crash the pipeline.
""" """
if not _requests:
try: try:
from urllib.parse import urlencode data = http.get(
params = urlencode({"url": post_url, "trim": "true"})
url = f"{SCRAPECREATORS_BASE}/video/comments?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as exc:
_log(f"Comment fetch error (urllib) for {post_url}: {exc}")
return []
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/video/comments", f"{SCRAPECREATORS_BASE}/video/comments",
params={"url": post_url, "trim": "true"}, params={"url": post_url, "trim": "true"},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
retries=2,
) )
resp.raise_for_status()
data = resp.json()
except Exception as exc: except Exception as exc:
_log(f"Comment fetch error for {post_url}: {exc}") _log(f"Comment fetch error for {post_url}: {exc}")
return [] return []
@@ -658,7 +599,9 @@ def _fetch_post_comments(
if not text: if not text:
continue continue
user = c.get("user") if isinstance(c.get("user"), dict) else {} user = c.get("user") if isinstance(c.get("user"), dict) else {}
author = user.get("nickname") or user.get("unique_id") or "" # Prefer unique_id (the @handle) over nickname (display name) so
# downstream render can cite @handle consistently across platforms.
author = user.get("unique_id") or user.get("nickname") or ""
create_time = c.get("create_time") create_time = c.get("create_time")
date_str = "" date_str = ""
if create_time: if create_time:
@@ -6,6 +6,8 @@ import threading
import random import random
from typing import Optional from typing import Optional
from .render import _skill_version
# Check if we're in a real terminal (not captured by Claude Code) # Check if we're in a real terminal (not captured by Claude Code)
IS_TTY = sys.stderr.isatty() IS_TTY = sys.stderr.isatty()
@@ -124,6 +126,7 @@ SOURCE_COMPLETION_ORDER = [
"polymarket", "polymarket",
"grounding", "grounding",
"xiaohongshu", "xiaohongshu",
"digg",
] ]
SOURCE_COMPLETION_META = { SOURCE_COMPLETION_META = {
@@ -138,6 +141,7 @@ SOURCE_COMPLETION_META = {
"polymarket": ("Polymarket", "market", "markets", Colors.GREEN), "polymarket": ("Polymarket", "market", "markets", Colors.GREEN),
"grounding": ("Web", "result", "results", Colors.GREEN), "grounding": ("Web", "result", "results", Colors.GREEN),
"xiaohongshu": ("Xiaohongshu", "post", "posts", Colors.RED), "xiaohongshu": ("Xiaohongshu", "post", "posts", Colors.RED),
"digg": ("Digg", "cluster", "clusters", Colors.YELLOW),
} }
@@ -196,7 +200,7 @@ Just start with "last30" and talk to me like normal.
# Shorter promo for single missing key # Shorter promo for single missing key
PROMO_SINGLE_KEY = { PROMO_SINGLE_KEY = {
"reddit": "\n💡 Unlock TikTok and Instagram with SCRAPECREATORS_API_KEY - 10,000 free calls, no CC - scrapecreators.com\n", "reddit": "\n💡 Unlock TikTok and Instagram with SCRAPECREATORS_API_KEY - 100 free credits, no CC - scrapecreators.com\n",
"x": "\n💡 Unlock X: log into x.com in Firefox or Safari, then re-run. Or add AUTH_TOKEN/CT0 or XAI_API_KEY.\n", "x": "\n💡 Unlock X: log into x.com in Firefox or Safari, then re-run. Or add AUTH_TOKEN/CT0 or XAI_API_KEY.\n",
"web": "\n💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY.\n", "web": "\n💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY.\n",
} }
@@ -507,7 +511,8 @@ def show_diagnostic_banner(diag: dict):
if IS_TTY: if IS_TTY:
lines.append(f"{Colors.DIM}┌─────────────────────────────────────────────────────┐{Colors.RESET}") lines.append(f"{Colors.DIM}┌─────────────────────────────────────────────────────┐{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}/last30days v3.0.0 - Source Status{Colors.RESET} {Colors.DIM}{Colors.RESET}") _header = f"/last30days v{_skill_version()} - Source Status"
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}{_header}{Colors.RESET}{' ' * (52 - len(_header))}{Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}") lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
# Reddit # Reddit
@@ -554,7 +559,8 @@ def show_diagnostic_banner(diag: dict):
else: else:
# Plain text for non-TTY (Claude Code / Codex) # Plain text for non-TTY (Claude Code / Codex)
lines.append("┌─────────────────────────────────────────────────────┐") lines.append("┌─────────────────────────────────────────────────────┐")
lines.append("/last30days v3.0.0 - Source Status") _header_plain = f"/last30days v{_skill_version()} - Source Status"
lines.append(f"{_header_plain}{' ' * (52 - len(_header_plain))}")
lines.append("│ │") lines.append("│ │")
if has_reddit and has_scrapecreators: if has_reddit and has_scrapecreators:

Some files were not shown because too many files have changed in this diff Show More