Compare commits

...

492 Commits

Author SHA1 Message Date
Trevin Chow 2ee0fc5c22 fix(skill): carry pitch-vs-pulse rules into the comparison synthesis path
Design-vs-implementation review found the comparison path missing two
reinforcements the single-topic path has: the windowed-claims rule (no
trend verbs from one 30-day window) and an explicit artifact gate. It
also pointed the beat at a "narrative section" the comparison template
doesn't have. The per-entity template block now carries an explicit
optional pitch-vs-pulse slot (omit entirely when silent - no
placeholder), and the comparison instruction names that slot, the
windowed rule, and the no-fetch-no-pitch gate. Instructions now sit at
the emission point, per the file's own v3.0.6/v3.0.7 lesson about
distance between instruction and output.
2026-06-09 17:33:30 -07:00
Trevin Chow c1ca1a4e9d feat(skill): replace narrative verdict row with evidence-triggered prose beat
Review showed the "Setting the narrative?" verdict compared across
abstraction levels: a homepage tagline is deliberately broad
("financial infrastructure" covers a chargebacks thread), so
tagline-vs-thread alignment verdicts are unfalsifiable and carry no
information. The signal now ships as PROSE in the entity's narrative
section, fires only when the month's evidence directly bears on the
pitch (supports a specific claim, cuts against one, or is squarely
about the pitched ground), and stays SILENT when the pulse is
orthogonal - omission over a manufactured connection. Claims are
tested at matched altitude (specific claim vs specific thread) and
stay windowed (no trend verbs one 30-day window can't support). The
positioning fetch step survives unchanged and now also grounds the
"What it is" row and brand-noise rejection. All scope gating (people
never, ownerless topics excluded, no pitch from memory) carries over.
2026-06-09 17:14:31 -07:00
Trevin Chow 57860aff1c feat(skill): scope narrative lens to first-party entities, gate on fetched positioning
The lens criterion is now explicit: an identifiable first party with a
fetchable public pitch - companies, products, services. People NEVER
qualify, even founders whose companies would (MrBeast the company can
get the lens; Jimmy Donaldson the person cannot; "Garry Tan vs Sam
Altman" gets N/A across the row). Ownerless topics (Bitcoin) fail the
same test - a foundation or fan site is not an authoritative first
party. Comparison-row verdicts are also artifact-gated to match the
single-entity beat: a verdict requires positioning fetched THIS run;
if item 6 couldn't run (no WebSearch), the cell is Unclear - the pitch
is never supplied from memory.
2026-06-09 16:33:29 -07:00
Trevin Chow 4402dd317f docs: changelog, CONCEPTS research-pipeline cluster, grounding solution doc
CHANGELOG covers the narrative lens (Added) and head-token grounding
fix (Fixed). CONCEPTS.md gains a Research pipeline cluster (Primary
entity, Intent modifier, Entity grounding, Keyless path,
Comment-enrichment slots). New docs/solutions/logic-errors/ entry
captures the false-demotion bug, the rejected alternatives, and the
shared-helper prevention rule.
2026-06-09 16:24:24 -07:00
Trevin Chow 6a92f63a56 fix(rerank): ground entity-miss demotion on head token, not full phrase
The entity-grounding demotion required the full multi-word primary
entity as a contiguous substring, so on-entity items missing a trailing
search descriptor were buried: a 323-pt HN thread "Stripe is friendly
to 'friendly fraud'" scored 0 on a "Stripe payments" query. New
_entity_grounded helper keys on the brand head token; items that never
name the brand still miss it and stay demoted. reddit_keyless
_slot_priority, which had re-implemented the old check while claiming
to mirror rerank's signal, now calls the shared helper so the two
paths cannot diverge.
2026-06-09 16:24:16 -07:00
Trevin Chow fd0e47d99f feat(skill): add "Setting the narrative?" lens for company/product topics
Comparison tables gain a narrative axis judging whether each entity's
community conversation is about what the entity pitches or about
something else (pricing, rivals, an incident). Backed by a new
mandatory Step 0.55 research item that fetches each entity's CURRENT
first-party positioning (RESOLVED_POSITIONING) instead of relying on
memory, and a narrative-check synthesis beat for single-entity company
runs. The mismatch is the signal: companies usually don't control
their own conversation.
2026-06-09 16:24:07 -07:00
Matt Van Horn 122158415a chore(release): v3.3.2 (#485)
Security / Dependency audit (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Validate / tests (push) Has been cancelled
* chore(release): v3.3.2

* chore(release): sync uv.lock for 3.3.2

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-06-06 09:58:05 -07:00
Matt Van Horn 1bdc14878c fix(reddit): relevance-aware comment-enrichment slot selection in keyless path (#484)
* fix(reddit): relevance-aware comment-enrichment slot selection in keyless path

* docs(changelog): record relevance-aware enrichment fix under Unreleased

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-06-06 09:44:07 -07:00
Matt Van Horn 26da1e157c chore: remove dev artifacts from installer scan surface (#465)
* chore: remove dev artifacts from installer scan surface

Hermes (and other harnesses that clone raw GitHub instead of honoring
.clawhubignore) scan files that never reach an installed skill, producing
a wall of false-positive security findings. Remove the stale SKILL-original.md
backup, internal docs/plans and docs/test-results, and release-notes.md so
the scanned tree matches what actually ships.

These were already excluded from the ClawHub bundle via .clawhubignore and
from git archives via .gitattributes export-ignore. No runtime files change.

Refs #464

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: drop dangling SKILL-original.md reference in AGENTS.md

Greptile-flagged: the deletion left a 'kept for reference only' pointer to
the removed file. Refs #465

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 07:41:30 -07:00
Matt Van Horn 4aae93ee5d fix: remove duplicate /last30days command wrapper (#461) (#462)
* fix: remove duplicate command wrapper so plugin exposes only the skill (#461)

The plugin shipped both commands/last30days.md and the skill under the
same name, so /last30 surfaced two `last30days` entries with two
different descriptions. Remove the wrapper; the skill already carries
its own argument-hint, so the /last30days <topic> picker UX is unchanged.

Also corrects the README install note that claimed Claude Code dedupes
the slash command across install methods (it does not), and bumps
3.3.0 -> 3.3.1 across plugin.json, marketplace.json, gemini-extension.json,
and SKILL.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: bump pyproject.toml version to 3.3.1 (manifest contract)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update uv.lock for 3.3.1 version bump

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 00:33:16 -05:00
Matt Van Horn 8d3a9e4368 fix(reddit): restore free path via keyless RSS + shreddit scrape (.json is dead) (#457)
* test(reddit): add live RSS + shreddit comment fixtures

Captured from reddit.com on 2026-05-29 (search.rss listing + the
/svc/shreddit/comments partial), trimmed to a representative subset plus
two synthetic edge cases (deleted author, negative score) for offline
parser tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(http): add keyless get_text helper

Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or
network failure so tiered callers fall through cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless RSS discovery (search.rss + listing feeds)

Replaces the now-403 search.json with keyless Atom feeds, normalized to the
existing reddit_public post shape. Scores are placeholder zeros, backfilled
during shreddit enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless shreddit comment scraper

Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id}
(score/author/created/permalink + thingId-anchored body) into top comments,
matching reddit_enrich output. Replaces the dead {thread}.json enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): tiered keyless orchestrator

Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery ->
Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup
still engages when every keyless tier is empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): route free path through keyless pipeline (.json is dead)

search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py
and other callers need no change. Removes the dead .json enrichment helpers;
search/_parse_posts remain as the demoted Tier 0 attempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): request sort=top so true top comments land on page 1

Guarantees the highest-scored comments are captured even on large threads,
independent of Reddit's default comment sort. Local score re-sort remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): recover post upvote scores via keyless listing partials

The shreddit community-more-posts partial server-renders each post's score
and comment count (works for normal users, not IP-gated), unlike RSS or the
comments endpoint. Use it as a scored discovery source and to backfill scores
onto RSS-discovered posts (subreddits derived from results when not provided).
Ranking now uses real upvote score.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): listings backfill scores only on bare queries, not discovery

Caught running the full pipeline on a bare topic: deriving subreddits from
noisy RSS results and merging their top/hot listings flooded results with
high-upvote off-topic posts. Now derived-subreddit listings are used only to
backfill scores onto keyword-matched RSS posts; listing cards are merged as
discovery only when the caller explicitly provides subreddits (on-topic).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 14:43:56 -05:00
Trevin Chow 1e03af19e0 Merge pull request #423 from hnshah/ren/preserve-requested-quick-sources 2026-05-22 08:14:46 -07:00
Trevin Chow f032e25e51 Merge pull request #429 from josmithiii/docs/agents-md-install-propagation 2026-05-22 08:13:01 -07:00
Trevin Chow 84a19cf44d Merge pull request #438 from iliaal/refactor/github-search-parse-split 2026-05-22 08:10:40 -07:00
Trevin Chow 861462689e Merge pull request #444 from Yong-yuan-X/fix/centralize-test-path-setup 2026-05-22 08:09:20 -07:00
Yong-yuan-X e74b0e1e93 tests: centralize script path setup in conftest.py
Add a pytest-discovered tests/conftest.py for the last30days scripts path and
remove duplicate per-file sys.path.insert boilerplate from tests.

Normalize affected imports to rely on the shared scripts path and remove the
now-unneeded E402 suppressions.
2026-05-21 00:04:03 +08:00
Ilia Alshanetsky c5c0239dc9 refactor(github): resolve token once at pipeline boundary; pad no-token envelope
Greptile review (PR #438) flagged two issues:

1. search_github and enrich_with_comments both call _resolve_token,
   so when GITHUB_TOKEN is absent from config and env the gh-CLI
   subprocess (with its 5s timeout) fires twice per query.

2. The no-token early-return envelope `{"items": [], "error": "no token"}`
   was missing the `context` key that every other failure path includes,
   making the envelope shape inconsistent between the no-token and
   fetch-failure cases.

Fix 1: add public github.resolve_token(token) wrapping the existing
_resolve_token. Pipeline calls it once before search and enrich, so
both downstream calls receive an already-resolved (or already-None)
token and skip the fallback chain.

Fix 2: thread core/from_date/to_date/count through the no-token
envelope's `context` key, matching the fetch-failure envelope shape.
parse_github_response was already tolerant of the missing key, but
diagnostics callers that read response["context"]["..."] now get a
consistent dict in both error paths.

Reviewer's suggested code patch for issue 1 was a no-op (it kept the
same _resolve_token(token) call inside enrich_with_comments); the
underlying intent — resolve at the boundary — is what this commit
implements.
2026-05-19 12:35:40 -04:00
Ilia Alshanetsky 269dda9f6c refactor(github): split search_github / parse_github_response / enrich_with_comments
search_github returned a normalized List[dict] directly while every
other adapter follows search_X -> dict envelope, parse_X_response ->
list[dict]. The github branch in pipeline._retrieve_stream was the
only one that called search_* and returned (result, {}) without a
parse step. This blocked fixture-driven testing: there was no parse
function to feed a synthetic envelope to.

Split into three:

  search_github(...) -> Dict[str, Any]
    HTTP fetch only. Returns {"items": [raw items], "context": {core,
    from_date, to_date, count}}.

  parse_github_response(response) -> List[Dict[str, Any]]
    Pure function. Normalizes, date-filters, sorts by relevance.

  enrich_with_comments(items, depth, token) -> List[Dict[str, Any]]
    Public extraction of the old private _enrich_top_items. Resolves
    the token via env / gh CLI fallback so callers don't have to.

Pipeline now does the standard 3-call dance:

  response = github.search_github(...)
  items = github.parse_github_response(response)
  items = github.enrich_with_comments(items, depth=depth, token=token)

Keeping enrich_with_comments in parse_github_response would make parse
impure and force every fixture-driven test to either mock HTTP or
skip enrichment. Splitting it out matches the YouTube adapter's
pattern.
2026-05-19 12:18:47 -04:00
Julius Smith a35677da77 docs(agents): address Greptile review (stale Commands comment, duplicate Structure entry)
- Commands block's inline comment on `npx skills add` still said
  "symlink this repo into every detected harness's skill dir" — the
  exact misconception the PR set out to correct. Rewrite to describe
  the frozen-copy behavior and point at the Rules section for the
  full explanation.
- Structure section had SKILL.md listed twice (the original line 6
  entry plus a new line 13 entry added in this PR). Fold the
  SKILL-original.md context into line 6 and drop the duplicate.
2026-05-18 11:02:53 -07:00
Julius Smith a78ab69ffe docs(agents): correct install-propagation claim and fill in build/test gaps
AGENTS.md said "edits in the working tree propagate live to every harness"
after `npx skills add . -g -y`, but the install actually drops a real
(frozen-at-install-time) copy at ~/.agents/skills/<name>/ and per-host
symlinks point at *that copy*, not at the working tree. Clarify the
mechanism and offer two ways forward: re-run `npx skills add` to sync,
or replace the install copy with a working-tree symlink for live-edit.

Also fill in two gaps a fresh agent hits on entry:
- `uv run pytest` commands for the ~89-file test suite (no test runner
  was documented before)
- Python 3.12+ / `uv` / `.venv/` convention
- Brief doc map: CONFIGURATION.md, SKILL.md vs SKILL-original.md,
  CHANGELOG.md / release-notes.md, HERMES_SETUP.md
2026-05-18 07:45:27 -07:00
Hiten Shah 0bb01c2d6a test: cover requested sources in fallback quick plans 2026-05-18 07:43:10 -07:00
Hiten Shah 444e07d141 fix: preserve requested sources in quick plans 2026-05-17 15:54:45 -07:00
Hiten Shah 850c7e0185 chore: sync release manifest versions 2026-05-17 15:51:12 -07:00
Trevin Chow d53121f035 Merge pull request #420 from hnshah/ren/watchlist-delta 2026-05-17 10:35:41 -07:00
Trevin Chow daca71f89e chore(release): v3.3.0
Release / build-and-release (push) Has been cancelled
~75 PRs merged since v3.2.0 plus 7 community fixes salvaged via PR triage.

Highlights:
- Install everywhere: npx skills add is canonical for Claude Code, Codex,
  Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts.
- New emit mode: --emit=html for shareable HTML briefs.
- New source: Digg (auto-enabled when digg-pp-cli on PATH).
- New env vars: EXCLUDE_SOURCES, LAST30DAYS_YOUTUBE_SSH_HOST.
- New credential source: macOS Keychain.
- Reliability sweep: Reddit (4xx + URL prefix + multi-key auth), xAI
  error surfacing, Windows compatibility, YouTube/HN unblock,
  HTTP retries, planner gating, render fixes.
- Multi-harness reframe: AGENTS.md becomes canonical, CLAUDE.md points
  at it. SKILL_ROOT → SKILL_DIR substitution.

Breaking:
- .codex-plugin/plugin.json removed. Codex installs via npx skills add.
2026-05-17 09:25:46 -07:00
Kaustav Mishra d51e91ea26 fix(xai): surface API errors instead of silently returning empty results
parse_x_response was returning an empty items list whenever xAI returned
a 200 OK with a malformed payload — empty output text, missing "items"
key, or invalid JSON. The pipeline saw "successful response with zero
items" and quietly handed the user a degraded report with no indication
the API had failed. Now raise http.HTTPError on each of those branches
so _retrieve_stream's caller catches it and surfaces the failure in
errors_by_source, giving the user a visible signal that X didn't work.

Closes #155.

Co-authored-by: Kaustav Mishra <km.git007@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow 170b570cbc fix(reddit): re-raise HTTP 402 so fallback chain triggers
The ScrapeCreators 402 (payment required / credits exhausted) status
was being swallowed by the broad except Exception handlers in
_global_search, _subreddit_search, and fetch_post_comments, returning
[] instead of propagating. That caused users with exhausted credits
to silently get zero Reddit results instead of falling through to
the OpenAI / public Reddit JSON fallback chain in _search_reddit_thread.
Add 402 to the existing 401/403 re-raise list across all three
ScrapeCreators call paths. Closes #170.

Co-authored-by: Jonathan Oppenheim <no-reply@postquantum.space>
2026-05-17 09:20:37 -07:00
Trevin Chow 4bae05e7fa fix(reddit): use browser-like headers to fix HTTP 403 from urllib
Reddit's public JSON endpoint returns 403 to requests carrying the
generic User-Agent and minimal header set urllib defaults to, while
matching curl requests succeed. Switch to a current-Chrome User-Agent
and add Accept-Language / Accept-Encoding / Connection headers so the
fingerprint matches a normal browser. Reddit now serves gzip when
Accept-Encoding includes it, so decompress the body before JSON parse.
Update the user-agent assertion in tests/test_reddit_public.py to match
the new browser-like string. Closes #199.

Co-authored-by: Franco Carballar <francocarballar@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow a4f1f94802 fix(env): restore multi-key rotation for SCRAPECREATORS_API_KEY
Originally added in #268 to spread load across free-tier accounts when
SCRAPECREATORS_API_KEY is set to a comma-separated list. The 7-line block
was inadvertently dropped during the v3.0.6 consolidation (d14814a) even
though the changelog still advertised the feature. Re-apply the same
random.choice rotation in get_config() so user-facing behavior matches
the documented contract. Closes #287.

Co-authored-by: Eric Oberhofer <eric@oberhofer.io>
2026-05-17 09:20:37 -07:00
Trevin Chow 16ce073d0c fix(cli): keep child cleanup working on Windows
_cleanup_children() called os.killpg unconditionally — Windows doesn't
have killpg as an attribute on os, so the call raised AttributeError
(not caught by the existing OSError-family handler) and aborted cleanup.
Guard with hasattr(os, "killpg") and fall back to os.kill(pid, SIGTERM)
on platforms without process-group APIs. Closes #226. Refs #110.

Co-authored-by: gujishh <baiaoshh@163.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 5994b4f76a fix(reddit): use removeprefix("r/") for subreddit names, not lstrip("r/")
str.lstrip("r/") treats its argument as a character set, stripping
leading r and / repeatedly. Subreddits starting with 'r' (e.g. r/robotics,
r/ruby) were silently mangled to 'obotics' / 'uby'. Replace with
str.removeprefix("r/") at all four call sites. Python 3.9+ pattern is
safe here — project requires 3.12. Closes #288.

Co-authored-by: Alex Key <alexanderkey0508@gmail.com>
2026-05-17 09:20:02 -07:00
Trevin Chow bb5e6efbf9 fix(scripts): replace hardcoded developer paths in test-v1-vs-v2.sh
REPO_DIR now derives from the script's location (with env-var override)
and the Claude binary is looked up via PATH (with CLAUDE env-var override)
instead of hardcoded to /Users/mvanhorn/.local/bin/claude. Works on any
checkout. Closes #297.

Co-authored-by: Dave Morin <dave@morin.com>
2026-05-17 09:20:02 -07:00
Hiten Shah 2502a19d46 fix(watchlist): clarify delta URL identity 2026-05-17 09:09:33 -07:00
Hiten Shah 0f280245ac feat(watchlist): show deltas between topic runs 2026-05-17 09:01:09 -07:00
Trevin Chow 76b8df40d3 Merge pull request #318 from flyingnobita/fix/gemini-claude-polyglot-hooks
fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility
2026-05-17 01:05:30 -07:00
Trevin Chow eb2d8b55e0 Merge pull request #344 from dzivkovi/feat/config-enablement
feat: configuration enablement — env-var defaults + source resilience
2026-05-17 01:05:08 -07:00
Trevin Chow 1a8ffd4847 fix(quality_nudge): also guard Instagram silent-failure on INCLUDE_SOURCES allowlist 2026-05-17 01:03:55 -07:00
Trevin Chow 1814bb1967 fix(quality_nudge,bluesky): gate Instagram nudge on EXCLUDE_SOURCES + anchor bluesky tests at resolver 2026-05-17 01:03:55 -07:00
Trevin Chow f236cff86a chore(pr-344): adapt to rebased base — fix test imports + memory-dir doc style
Rebased onto current main where:
- instagram.py uses unified http.get (not _requests fallback); tests now
  mock http.get and assert params/timeout kwargs.
- quality_nudge tests use lib.* import path with sys.path setup.
- README/CONFIGURATION.md memory-dir lines say "defaults to" so they pass
  test_no_stray_hardcoded_memory_dir_paths.
2026-05-17 01:03:55 -07:00
Daniel Zivkovic 44971a6aae feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 01:03:15 -07:00
Daniel Zivkovic a8e462c978 chore(gitignore): ignore /work and /print (personal artifacts)
Personal directories used by the /note slash command (work/) and PDF
print exports (print/) - these are local research artifacts, not
shipping content. Lives on daniel/personal to keep upstream/main and
PR branches free of personal noise.
2026-05-17 01:02:47 -07:00
Trevin Chow d9a0ac31f2 Merge pull request #323 from GAOJIAN-0106/fix/openrouter-default-model-id
fix: correct invalid OPENROUTER_DEFAULT model ID
2026-05-17 01:01:54 -07:00
Trevin Chow 3e60c0817d Merge pull request #339 from dzivkovi/docs/configuration-md
docs: add CONFIGURATION.md + README pointers
2026-05-17 01:01:37 -07:00
Trevin Chow d530c90239 fix(hooks): remove timeout field per PR description intent 2026-05-17 00:59:41 -07:00
Trevin Chow aef6f35460 docs(providers): note why OpenRouter slug includes -preview suffix 2026-05-17 00:59:31 -07:00
GAOJIAN-0106 77bd235f64 fix: correct invalid OPENROUTER_DEFAULT model ID
google/gemini-flash-2.0 is not a valid OpenRouter model ID (segments reversed).
Every rerank and FunJudge call fails with HTTP 400 when REASONING_PROVIDER=openrouter
and LAST30DAYS_RERANK_MODEL is not explicitly pinned, silently falling back to
local-score heuristics.

OpenRouter error body:
  {"message":"google/gemini-flash-2.0 is not a valid model ID","code":400}

Switching to google/gemini-3.1-flash-lite-preview, which matches the
GEMINI_FLASH_LITE constant already used by the native Gemini provider on
line 12 of the same file. This makes the Gemini and OpenRouter providers
consistent and avoids a future divergence.

Validated with /last30days 'Claude Opus 4.7' --quick:
- Rerank/FunJudge HTTP 400 errors: 1 per run -> 0
- 'Why: fallback-local-score' markers in output: every cluster -> 0
- LLM-generated 'Why:' reasoning lines: 0 -> 11
2026-05-17 00:59:11 -07:00
flyingnobita e1017e95c9 fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility 2026-05-17 00:58:46 -07:00
Trevin Chow 71b1e8a411 Merge pull request #302 from nidhi-singh02/fix/github-repo-canonicalization
fix: Canonicalize ambiguous GitHub repo resolution for product comparisons
2026-05-17 00:57:05 -07:00
Trevin Chow 618458eb7e Merge pull request #320 from kaushikgopal/feat/brave-cookie-extraction
Add Brave browser support for X/Twitter cookie extraction
2026-05-17 00:56:44 -07:00
Trevin Chow 8cccd3e982 Merge pull request #334 from iamitp/codex/last-run-config-state
Preserve clean mode for last run state
2026-05-17 00:56:28 -07:00
Trevin Chow 17fb17222b fix(docs): use 'defaults to' wording for LAST30DAYS_MEMORY_DIR refs
The version_consistency test (test_no_stray_hardcoded_memory_dir_paths)
flagged 4 lines where `~/Documents/Last30Days/` appeared without the
canonical "defaults to" phrasing or the ${LAST30DAYS_MEMORY_DIR:-...}
literal form. Tightened the wording in the CONFIGURATION.md table +
footer paragraph, and the README "Where research files are saved"
section, so each path mention is anchored at the env-var override
contract rather than as a bare hardcoded default.
2026-05-17 00:54:49 -07:00
Trevin Chow 8ccd778366 fix(canonicalization): predicate-based call lookup + skip double-canon on auto-resolve
Two findings from Greptile review on PR #302:

1. tests/test_cli_v3.py:302 - The test asserted run_mock.call_args_list[0]
   was the main runner's invocation, but fanout.run_competitor_fanout
   submits main + competitors to a ThreadPoolExecutor and iterates with
   as_completed. With zero-latency mocks, thread scheduling determines
   which pipeline.run call lands first, so the competitor's call could
   take index [0] and flake CI. Replace [0] indexing with a predicate
   match on the canonicalized github_repos kwargs.

2. skills/last30days/scripts/last30days.py:662 - When auto_resolve returns
   github_repos, it has already run canonicalize_github_repos(cap=5) and
   ranked by relevance. The downstream block then re-canonicalized with
   cap=None, which can re-sort by topic-slug match and clobber the
   auto_resolve relevance order. Guard the second canonicalization with
   a repos_from_auto_resolve flag so it only fires for user-supplied
   --github-repo input.
2026-05-17 00:51:21 -07:00
Trevin Chow a3f173dc8a docs(readme): drop community-video link from "Going deeper" callout
The 6-min architecture walkthrough video is the contributor's own
work — not affiliated with the project. Removing the README pointer
to avoid implying endorsement. CONFIGURATION.md prose stands.
2026-05-17 00:51:14 -07:00
Trevin Chow 5a3ac8ca37 docs(config): document briefing.py show [--date DATE] subcommand 2026-05-17 00:50:36 -07:00
Trevin Chow e8eb15102f docs(agents): steer agents on maintaining CONFIGURATION.md 2026-05-17 00:50:36 -07:00
Daniel Zivkovic fd6e70c539 docs: add CONFIGURATION.md + README pointers + community video link
Adds CONFIGURATION.md at repo root - a focused configuration reference
covering save paths, the per-source API-key matrix, reasoning and
web-search backend priority, the trend-monitoring stack (--store +
watchlist.py + briefing.py), and per-client patterns.

Surfaces two things that ship in the engine but were not documented
for users:

- The project-scoped .claude/last30days.env config file (currently only
  referenced in hooks/scripts/check-config.sh) which takes precedence
  over the global ~/.config/last30days/.env when present. Cleanest
  pattern for per-client setups - drop a file in the client folder, cd
  in, run normally.
- The existing trend-monitoring scripts (--store flag, watchlist.py,
  briefing.py) that the README did not surface for users.

Updates README with a brief "Configuration" section pointing to the new
file, plus a one-line "Going deeper" callout linking a 6-min community
architecture walkthrough on YouTube.

All CLI surface claims (watchlist subcommands, briefing modes, source
dedupe key, env file priority chain) fact-checked against the live
scripts/ source before commit.
2026-05-17 00:50:36 -07:00
nidhi-singh02 d0b990e211 Canonicalize GitHub repo resolution for ambiguous product repos 2026-05-17 00:50:05 -07:00
Trevin Chow 0f03a67166 Merge pull request #343 from Bortlesboat/codex/use-sandboxed-safari-cookie-path
fix: prefer sandboxed Safari cookie path
2026-05-17 00:47:32 -07:00
Trevin Chow 5a625fda9f Merge pull request #355 from dinakars777/test/cover-parallel-grounding-backend
test: cover parallel grounding backend
2026-05-17 00:47:20 -07:00
Trevin Chow b296a65515 fix(last-run): guard python3 absence + hoist datetime + use context manager 2026-05-17 00:44:30 -07:00
Trevin Chow f2737fc035 test(grounding): fix published_date → publish_date mock key mismatch 2026-05-17 00:42:07 -07:00
Trevin Chow 9ce7264d43 test(grounding): add serper>parallel priority + parallel empty-results coverage 2026-05-17 00:41:19 -07:00
Dinakar Sarbada f458e0f5af test: cover parallel grounding backend 2026-05-17 00:41:19 -07:00
Trevin Chow 8f565ee241 fix(chrome_cookies): sort Brave profiles by mtime, not alphabetically 2026-05-17 00:40:10 -07:00
Amit Patnaik dd7e6a1562 Preserve clean mode for last run state 2026-05-17 00:39:44 -07:00
KG 65313ce542 feat(cookies): add Brave browser cookie extraction for macOS
Brave uses identical v10 AES-128-CBC encryption to Chrome; only the
DB path (BraveSoftware/Brave-Browser) and Keychain service name
("Brave Safe Storage") differ. Refactored chrome_cookies.py to share
a single _extract_chromium_cookies_macos helper rather than duplicating
the decryption logic.

Profile discovery tries Default/ first, then scans numbered Profile N/
directories so non-default Brave profiles are covered.
2026-05-17 00:39:30 -07:00
Trevin Chow 5ab8c3ba76 Merge pull request #345 from dinakars777/docs/fix-stale-script-paths
docs: fix stale script paths
2026-05-17 00:38:17 -07:00
Trevin Chow 38bfb504e1 Merge pull request #349 from dinakars777/chore/sync-gemini-extension-version
chore: sync gemini extension version
2026-05-17 00:38:01 -07:00
Trevin Chow e8f23b4205 Merge pull request #340 from dzivkovi/fix/youtube-transcript-observability
fix(youtube): surface transcript-fetch ratio + add degraded nudge for stale yt-dlp
2026-05-17 00:31:37 -07:00
Trevin Chow b78ce34922 test(safari_cookies): add coverage for legacy fallback path 2026-05-17 00:31:09 -07:00
Bortlesboat 0656b868e7 fix safari cookie path resolution 2026-05-17 00:30:25 -07:00
Trevin Chow 16a4fa9c39 Merge pull request #341 from flyingice/main
fix(grounding): align Parallel AI search with current API schema
2026-05-17 00:27:50 -07:00
Trevin Chow 3b75ff1537 Merge pull request #354 from dinakars777/fix/parallel-web-backend-source
fix: route parallel web backend through grounding
2026-05-17 00:26:50 -07:00
Trevin Chow 321975e144 Merge pull request #356 from dinakars777/fix/allow-threads-pinterest-search
fix: honor explicit optional source requests
2026-05-17 00:26:25 -07:00
Trevin Chow 5f9c637bbe Merge pull request #336 from davemorin/fix/319-xquik-source-capabilities
fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
2026-05-17 00:26:08 -07:00
Trevin Chow 19132b0b5e Merge pull request #338 from dzivkovi/fix/windows-save-path-footer
fix(render): use forward slashes in save-path footer for Windows
2026-05-17 00:25:45 -07:00
Trevin Chow 1e4150ad78 Merge pull request #347 from dinakars777/docs/update-how-search-key-files
docs: update search key file paths
2026-05-17 00:24:55 -07:00
Trevin Chow 6a15afd8e8 Update skills/last30days/scripts/last30days.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-17 00:23:53 -07:00
Trevin Chow 0ab7051bc5 fix(planner): also register xquik in QUICK_SOURCE_PRIORITY 2026-05-17 00:22:04 -07:00
Trevin Chow ec0b126af6 test(pipeline): relax grounding assertion to stable source key 2026-05-17 00:21:57 -07:00
Trevin Chow d9e8a046ef docs(how-search): swap score.py->relevance.py at line 145 2026-05-17 00:21:36 -07:00
Dave Morin 87bf3debcc fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
Without this entry, the planner's _default_sources_for_intent() drops
xquik from the candidate pool for how_to / comparison / news intents
because SOURCE_CAPABILITIES.get("xquik", set()) returns the empty set.
Users with XQUIK_API_KEY set get zero Xquik results even though the
engine recognizes the key.

Mirrors the capabilities for "x" since both are X/Twitter-shaped
discussion + social sources.

Fixes #319
2026-05-17 00:21:36 -07:00
Dinakar Sarbada 2f4b023db8 docs: update search key file paths 2026-05-17 00:21:15 -07:00
Trevin Chow 261ea5895c refactor(pipeline): remove dead threads-explicit-request branch 2026-05-17 00:20:56 -07:00
Trevin Chow 2692e0f4a2 Merge pull request #351 from dinakars777/docs/fix-changelog-skill-link-note
docs: correct changelog skill link note
2026-05-17 00:20:46 -07:00
Trevin Chow 10f35f82fe fix(grounding): guard parallel excerpts against None + cap at 500 chars 2026-05-17 00:20:27 -07:00
Dinakar Sarbada 5b29b8f427 fix: honor explicit perplexity source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 6a5a122195 fix: honor explicit threads source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 7bda02169d fix: allow threads and pinterest search sources 2026-05-17 00:20:25 -07:00
Trevin Chow 6b40d2c46f Merge pull request #346 from dinakars777/docs/fix-bug-report-repro-command
docs: fix bug report repro command
2026-05-17 00:20:21 -07:00
Trevin Chow 4a99c4f557 Merge pull request #337 from UncleMike1988/fix/path-quoting-spaces
Fix path-quoting in SessionStart check-config hook (handles spaces in…
2026-05-17 00:20:09 -07:00
Dinakar Sarbada 5c802b0daa fix: route parallel web backend through grounding 2026-05-17 00:20:05 -07:00
Trevin Chow 0e353ae03f fix(render): apply as_posix to fallback branch + hoist shutil import 2026-05-17 00:19:49 -07:00
Daniel Zivkovic 0a5102e193 fix(youtube): surface transcript-fetch ratio in footer + add degraded nudge
When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:

  1. Footer (render.py): showed "N videos | M views" with no indication
     that zero transcripts were captured. The "with transcripts" segment
     was conditionally suppressed when the count was zero - converting
     the canonical stale-binary failure mode into a silent absence at
     the very surface users read for "did this work?".

  2. Quality nudge (quality_nudge.py): classified YouTube as "active"
     based purely on yt-dlp installation + absence of a top-level error.
     Per-video transcript-fetch ratio was never inspected. A run that
     returned N videos with 0 transcripts (canonical stale-binary
     failure) was reported as fully active.

The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.

This change makes both conclusion surfaces honest:

* render.py footer always renders "M/N with transcripts" so the ratio
  is visible regardless of value. Zero is no longer hidden. Format is
  M/N (not bare M) so the denominator is in the message and the user
  does not have to cross-reference the "videos" count.

* quality_nudge.py adds a third tier between "active" and "missing":
  "degraded". Triggered when yt-dlp is installed AND videos were
  returned AND transcript-fetch ratio is below threshold (default 50%,
  tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
  actionable nudge: "YouTube returned N videos but only M transcripts
  captured. The most common cause is a stale yt-dlp binary - YouTube's
  caption format changes frequently and old binaries silently fail
  every transcript. Update via your package manager: scoop update
  yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
  yt-dlp."

* last30days.py populates youtube_videos_count and
  youtube_transcripts_count in the research_results dict it passes to
  compute_quality_score, enabling the new degraded check at the call
  site.

Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.

Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.

Tests:

* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
  zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
  zero-videos-does-not-flag (no false positives on absence),
  one-of-three-flags-degraded, threshold-tunable-via-config, and
  degraded-does-not-affect-score.

* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
  cases cover zero-transcripts-with-videos-renders-zero-over-total
  (the regression repro), partial-renders-ratio, full-renders-ratio,
  and no-videos-suppresses-entire-segment.

All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).

Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.

Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
2026-05-17 00:13:56 -07:00
Dinakar Sarbada cfde1dbbe4 docs: fix stale script paths 2026-05-17 00:08:57 -07:00
Dinakar Sarbada 608381a818 docs: fix bug report repro command 2026-05-17 00:08:17 -07:00
flyingice af4cf7c03d fix(grounding): align Parallel AI search with current API schema 2026-05-17 00:08:10 -07:00
Michael Turner 5b0308b9e4 Fix path-quoting in SessionStart check-config hook (handles spaces in CLAUDE_PLUGIN_ROOT)
If CLAUDE_PLUGIN_ROOT ever expands to a path containing whitespace
(e.g. ~/Library/Application Support/...), the unquoted ${CLAUDE_PLUGIN_ROOT}
in hooks/hooks.json word-splits and bash receives the path as multiple
arguments, failing with "No such file or directory" on the first split.

Quoting the expansion makes the invocation correct regardless of the
characters in the resolved path. Verified manually:
  unquoted + space  -> bash: /tmp/with: No such file or directory
  quoted   + space  -> bash: /tmp/with spaces/.../check-config.sh: No such file
  quoted   + real   -> /last30days: Ready - 7 sources active.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 00:08:01 -07:00
Daniel Zivkovic 5817ef8387 test(cli): regression test for Windows save-path display
Asserts compute_save_path_display() never returns a backslash when the
save_dir is under the user's home directory, regardless of host OS.

Reproduces the original bug on Windows (failed message before the fix:
  AssertionError: '\' unexpectedly found in
  '~/l30d_save_path__luu2g76\Documents\Last30Days\british-airways-middle-east-raw-v3.md'
)
and locks in the contract on POSIX hosts (passes trivially today; would
fail if anyone removes .as_posix() in the future).

Verified by temporarily reverting the fix and confirming RED, then
re-applying the fix and confirming GREEN. All 13 CliV3Tests pass.
2026-05-17 00:07:44 -07:00
Daniel Zivkovic a87c1ba058 fix(render): use forward slashes in save-path footer for Windows
The footer line `📎 Raw results saved to ~/Documents\Last30Days\…`
mangled the home-relative path on Windows because `f"~/{relative}"`
stringifies a `pathlib.Path` with the OS-native separator. The result
mixes a Unix tilde with backslashes, which neither File Explorer,
PowerShell, nor a `file://` URI can resolve.

`Path.as_posix()` always returns forward slashes, which is the
convention `~/`-prefixed paths require on every platform. macOS and
Linux output is unchanged because their separator is already `/`.

Repro on Windows:
  python3 last30days.py "anything" --emit=compact --save-dir="$HOME/Documents/Last30Days"
  # before: 📎 Raw results saved to ~/Documents\Last30Days\anything-raw.md
  # after:  📎 Raw results saved to ~/Documents/Last30Days/anything-raw.md
2026-05-17 00:07:44 -07:00
Trevin Chow 7214dd6051 Merge pull request #348 from dinakars777/docs/fix-readme-skill-link
docs: fix runtime skill spec link
2026-05-17 00:04:30 -07:00
Trevin Chow 6acf2fdbe2 Merge pull request #419 from mvanhorn/chore/remove-orphaned-spec-tasks
chore: remove orphaned SPEC.md and TASKS.md
2026-05-17 00:03:06 -07:00
Trevin Chow 07a3bdb3cf Merge pull request #364 from davemorin/fix/361-unsafe-eval-check-config
fix(hooks): replace unsafe eval with declare in check-config.sh
2026-05-17 00:02:44 -07:00
Dinakar Sarbada 87577ff126 test: guard gemini extension version 2026-05-17 00:00:59 -07:00
Dinakar Sarbada 400fc4cc00 docs: fix runtime skill spec link 2026-05-17 00:00:58 -07:00
Dinakar Sarbada e9ecce0b1c chore: sync gemini extension version 2026-05-17 00:00:51 -07:00
Trevin Chow f3df47c381 chore: remove orphaned SPEC.md and TASKS.md
Both files lived at the repo root as pre-plugin-layout artifacts. On
current main neither is referenced from README, SKILL.md, AGENTS.md,
CHANGELOG, or docs/ — no inbound links to break by removing. Git history
preserves the content for anyone who needs to dig it up.

Closes #352, #353. The PRs by @dinakars777 correctly flagged the drift;
deletion is the cleaner resolution than annotating them as historical.
2026-05-17 00:00:47 -07:00
Dinakar Sarbada c9cf3ef92f docs: correct changelog skill link note 2026-05-16 23:59:30 -07:00
Trevin Chow 46cf2328aa fix(hooks): use printf -v for bash 3.2 compat (declare -g is 4.2+)
macOS ships /bin/bash 3.2 and the script uses #!/bin/bash with
set -euo pipefail, so declare -g would abort the SessionStart hook
with "invalid option" on every Mac. printf -v writes via assignment
semantics (global from inside a function on 3.2+) — same scope
outcome, broader compatibility.
2026-05-16 23:49:44 -07:00
Trevin Chow 7506cbd542 fix(hooks): scope ENV_* to global (declare -g) so caller sees values 2026-05-16 23:49:05 -07:00
Dave Morin a6bd481e61 fix(hooks): replace unsafe eval with declare in check-config.sh
The load_env_vars function used eval to assign .env values, which
executes command substitutions in backtick-containing comments.
Replace eval with declare and strip inline comments before assignment.

Fixes #361
2026-05-16 23:49:05 -07:00
Trevin Chow aba6172032 Merge pull request #366 from davemorin/feat/324-reddit-json-fallback
feat(web): auto-enrich Reddit URLs from web search via JSON API
2026-05-16 23:41:57 -07:00
Trevin Chow d07e4698e3 Merge pull request #358 from dinakars777/fix/openclaw-poll-clock-init
fix: initialize OpenClaw poll timing once
2026-05-16 23:41:42 -07:00
Trevin Chow 4c0282dd55 Merge pull request #365 from davemorin/fix/284-version-metadata-drift
fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
2026-05-16 23:40:53 -07:00
Trevin Chow 99909fca67 Merge pull request #368 from hnshah/ren/advisory-security-workflow-252
ci: add advisory security workflow
2026-05-16 23:40:28 -07:00
Trevin Chow 36c43d50b7 test: drop side_effect padding to match collapsed time.time() call 2026-05-16 23:39:14 -07:00
Dinakar Sarbada ecf68347db fix: initialize OpenClaw poll timing once 2026-05-16 23:38:30 -07:00
Trevin Chow e2d9d705f6 review: fix selftext key path + break on RedditRateLimitError 2026-05-16 23:37:00 -07:00
Hiten Shah 9c09a67ac2 ci: add advisory security workflow 2026-05-16 23:36:55 -07:00
Dave Morin 32da0bd6cb test: update version assertions for dynamic _skill_version()
Tests now check for version prefix without hardcoded version number,
matching the render.py change to use _skill_version() dynamically.
2026-05-16 23:35:26 -07:00
Dave Morin 863c3bc145 fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
render.py, ui.py, and last30days.py had hardcoded "v3.0.0" in titles
and headers while plugin.json was at 3.1.1. Use _skill_version()
(reads from plugin.json at runtime) so version strings stay in sync.

Fixes #284
2026-05-16 23:35:26 -07:00
Trevin Chow 9f39d10bc5 Merge pull request #373 from hnshah/ren/watchlist-sightings
feat(store): record per-run finding sightings
2026-05-16 23:24:09 -07:00
Trevin Chow 03043da407 Merge pull request #418 from tmchow/chore/greptile-config
chore: add greptile.json (triggerOnUpdates + statusCheck)
2026-05-16 23:23:48 -07:00
Trevin Chow 8bab997854 chore: add greptile.json to opt into update-triggered reviews + status check
Without this config, Greptile's documented default is `triggerOnUpdates: false`
(only the initial PR open triggers a review). Empirically Greptile has been
re-reviewing on force-push to this repo anyway, but documenting the intent
makes the behavior reliable across plan changes and any future config-source
shifts on Greptile's side.

`statusCheck: true` registers Greptile as a GitHub status check (not just a
PR comment). That gives maintainer-tooling a machine-readable heartbeat -
poll `GET /repos/.../commits/SHA/check-runs` and filter by app name to see
whether Greptile is `queued` / `in_progress` / `completed`. Without it the
only signal is "did a new Greptile comment appear" which is silently
ambiguous when Greptile re-reviews and finds nothing new.

If `statusCheck` is OSS-plan-restricted Greptile silently ignores the key,
which is fine - the rolling-summary comment with `Confidence Score: N/5`
remains the fallback signal.

Refs greptileai/skills `greploop` skill for the terminal-state pattern this
config enables.
2026-05-16 23:20:24 -07:00
Hiten Shah 375fd0bcc0 fix(store): enforce sighting finding id invariant 2026-05-16 22:57:32 -07:00
Hiten Shah 92d65723e4 fix(watchlist): refresh sighting retries 2026-05-16 22:57:04 -07:00
Hiten Shah f794f82af5 feat(store): record per-run finding sightings 2026-05-16 22:57:04 -07:00
Trevin Chow 791c0a57a0 review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.

A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).

Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.

Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded

19/19 grounding tests pass.
2026-05-16 22:52:06 -07:00
Dave Morin 211df0deaa feat(web): auto-enrich Reddit URLs from web search via JSON API
Web search backends (Brave, Exa, Serper) can return Reddit URLs as
results. Claude Code's WebFetch blocks reddit.com, so the model can't
retrieve full thread content. After web search, detect Reddit URLs
and fetch body text + top comments via reddit.com/.json endpoint
using the skill's own HTTP library.

Fixes #324
2026-05-16 22:50:39 -07:00
Trevin Chow d7b3995da1 Merge pull request #357 from dinakars777/fix/windows-env-permission-warning
fix: skip POSIX secret warning on Windows
2026-05-16 22:42:26 -07:00
Trevin Chow e217db77cc Merge pull request #369 from voidborne-d/fix/scrapecreators-100-credits
docs: correct ScrapeCreators free tier to 100 credits (closes #367)
2026-05-16 22:41:16 -07:00
Dinakar Sarbada 8ea207b348 fix: skip POSIX secret warning on Windows 2026-05-16 22:39:35 -07:00
voidborne-d b04212680d docs: correct ScrapeCreators free tier to 100 credits (closes #367)
The skill advertises ScrapeCreators as offering "10,000 free API calls" in
six places. The actual free tier on the ScrapeCreators pricing page is
"100 credits free · No credit card required · Credits never expire" — a
100x overstatement that surprises users on signup.

Reporter (#367) burned through their full free allocation on a single
/last30days run after taking the 10,000-call claim at face value. They
verified the actual tier directly against scrapecreators.com plus an
independent review at fahimai.com.

Sweep:
- hooks/scripts/check-config.sh:110  (SessionStart hook tip line)
- README.md:228                      (Sources × Cost table row)
- HERMES_SETUP.md:62                 (Optional: ScrapeCreators bullet)
- skills/last30days/scripts/lib/ui.py:199  (PROMO_SINGLE_KEY["reddit"])
- skills/last30days/SKILL.md:1648    ("PAYG after 10,000 free API calls")
- skills/last30days/SKILL.md:1661    ("10,000 free API calls, then PAYG")

Wording defaults to the provider's own framing — "100 free credits" — and
keeps PAYG language where it was already explicit, since the paid step is
the part users were actually getting blindsided by.

CI gates: tests/test_plugin_contract.py (4) + tests/test_version_consistency.py (4)
all pass.  shellcheck clean.  No tests pin the "10,000" string.
2026-05-16 22:33:41 -07:00
Trevin Chow 2c2cfb9e7e Merge pull request #417 from tmchow/docs/eval-not-in-ci-solution
docs: capture eval-not-in-CI design decision under docs/solutions/
2026-05-16 22:31:39 -07:00
Trevin Chow 3276496f49 Merge pull request #376 from shoobee/feat/yt-dlp-ssh-routing
feat(youtube): route yt-dlp through SSH host for residential IP egress
2026-05-16 22:31:18 -07:00
Trevin Chow 68ae74ff4f docs: capture eval-not-in-CI design decision under docs/solutions/
Closes #374 (adapted, not 1:1 merged).

@hnshah opened PR #374 proposing a docs/adr/ directory for architecture
decision records. The intent is right -- the "why is search-quality eval
manual?" reasoning drifts out of memory if it isn't written down -- but
the docs/adr/ convention doesn't fit alongside the existing
docs/solutions/ structure (compound-engineering ce-compound pattern with
frontmatter metadata, additive entries, no membership-contract test).

This commit adopts hnshah's ADR 002 content (search-quality eval is
manual by default) as a docs/solutions/architecture/ entry with the
canonical compound-style frontmatter (module, problem_type, applies_when,
related_components, tags). Drops the docs/adr/ directory pattern, the
README index, and the test_adr_docs.py contract test.

ADR 001 (multi-surface packaging) is intentionally not adopted here: it
referenced sync.sh as the deploy mechanism, but sync.sh was removed in
PR #405 in favor of `npx skills add . -g -y`. The multi-surface
packaging story is still real but has moved beyond what the original
ADR captured; a fresh "how we ship to multiple harnesses" entry would
make sense as a separate doc.

Co-authored-by: hnshah <hnshah@users.noreply.github.com>
2026-05-16 22:22:59 -07:00
Trevin Chow 27c90504c0 review: validate SSH host alias + rename LAST30DAYS_YT_SSH_HOST -> LAST30DAYS_YOUTUBE_SSH_HOST
Addresses two concerns surfaced during PR #376 review:

1. **SSH option-injection on the host value.** The original PR uses
   shlex.quote() on the remote command and added a `--` option terminator
   in front of the host, but neither one stops a hostile env var like
   `LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
   first place. Tighten `_ytdlp_ssh_host()` to validate the host against
   `^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
   digits, dot, underscore, hyphen). Any value that doesn't match logs a
   warning to stderr and returns None, so the wrap function falls back to
   local execution. The `--` terminator stays as defense-in-depth for the
   case where a valid host happens to start with `-`, but the regex closes
   the door on the env var reaching ssh at all.

2. **Env var naming consistency.** Existing skill-internal config knobs
   spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
   `LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
   is `youtube_yt.py`, the source key is `youtube`, the function family
   is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
   `LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
   model ("route YouTube fetches via residential IP") and the codebase's
   spelled-out convention.

Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)

The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.

44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
2026-05-16 22:19:35 -07:00
shoobee f4eb0af104 fix(youtube): address Greptile review feedback
Three changes from automated review on PR #376:

1. Add `--` option terminator before host in _wrap_ytdlp_cmd (P1 security)
   Prevents SSH option injection if LAST30DAYS_YT_SSH_HOST were ever set
   to a value starting with `-` (e.g. `-oProxyCommand=...`). Low
   exploitability since the env var is user-controlled config — but the
   fix is a single arg and turns a self-harm footgun into no footgun.

2. Hoist `import shlex` to module-level (P2 style)
   Pure stdlib import, no reason for the deferred form. Cleaner.

3. Cache _ytdlp_ssh_host() result in fetch_transcript (P2 style)
   Was being called 2-3x per video; the function is cheap (env lookup
   + strip) so this is purely about readability.

Adds test_wrap_cmd_uses_option_terminator covering the security fix
explicitly with a `-oFoo=bar` host value. Updates index assertions in
the two existing tests that check command shape (host is now at index
4, command string at 5, with `--` at 3).
2026-05-16 22:17:30 -07:00
shoobee 79b5d049ce feat(youtube): route yt-dlp through SSH host for residential IP egress
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.

Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.

The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.

Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
  <host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
  returns True without a local PATH check (the binary lives on the
  remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
  path (which writes a VTT file we couldn't easily read back over SSH)
  and uses the existing _fetch_transcript_direct HTTP fallback. The
  timedtext API isn't bot-walled, so this works fine on datacenter IPs.

Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.

Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).

Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
2026-05-16 22:17:29 -07:00
Trevin Chow 0e2059661a Merge pull request #378 from j-sperling/chore/gemini-3.1-flash-lite-ga
chore: migrate to gemini-3.1-flash-lite GA model
2026-05-16 22:12:20 -07:00
Trevin Chow b1c5f8db82 Merge pull request #382 from lustrousgorilla/bugfix/reddit-gaierror-retry
fix(http): expand retry budget + exponential backoff on DNS resolution failure
2026-05-16 22:11:56 -07:00
Trevin Chow 89c5cb9d5d Merge pull request #416 from tmchow/worktree-inherited-discovering-pebble
fix(ci): run full pytest suite, repair 13 rotted tests
2026-05-16 22:11:34 -07:00
Trevin Chow 5d4f9ef2c5 fix(store): use UTC for all date arithmetic against SQLite columns
Greptile flagged that _cli_query's --since parsing uses datetime.now()
(local time) while first_seen is stored via SQLite's datetime('now') (UTC).
The same bug exists in three other call sites that compare against either
first_seen or run_date (both UTC):

- get_daily_cost: "today" defaults to local date, returns wrong day's cost
  near UTC midnight
- get_stats: "7 days ago" cutoff for runs_7d / successful_7d
- get_trending: "N days ago" cutoff for finding activity ranking
- _cli_query: "N days ago" cutoff for --since flag (Greptile's flag)

All four now use datetime.now(timezone.utc). Same root cause and same fix
as the test_get_new_findings_filters_by_date repair in the previous commit.
2026-05-16 21:40:25 -07:00
Jeffrey Sperling 01262f78c6 chore: import GEMINI_FLASH_LITE in evaluate_search_quality
Address Greptile review nit. DEFAULT_JUDGE_MODEL now reuses the
constant from lib/providers.py instead of duplicating the literal,
so a future identifier change only needs one edit.
2026-05-16 21:40:22 -07:00
Jeffrey Sperling 96a4a78faa chore: migrate to gemini-3.1-flash-lite GA model
The Gemini 3.1 Flash Lite preview model is being discontinued on
May 25, 2026. Per Google's GA announcement, the underlying model
architecture is identical and only the model identifier needs to
be updated from `gemini-3.1-flash-lite-preview` to
`gemini-3.1-flash-lite`.

Also relaxes the `_require_gemini_31_preview` guard to accept any
`gemini-3.1-*` identifier (renamed to `_require_gemini_31`), so the
GA name and the still-preview `gemini-3.1-pro-preview` both pass.
2026-05-16 21:40:22 -07:00
Trevin Chow eb2d7a0f37 fix(ci): run full pytest suite, repair 13 rotted tests
CI was running only test_plugin_contract.py and test_version_consistency.py
(2 of 84 test files), masking 13 rotted tests across 4 clusters. The suite is
fully offline-safe (1402 tests in ~7s without network), so the narrow scope
wasn't gating integration flakiness; it was just stale. validate.yml now runs
`uv run pytest` against the full suite.

Engine fix: store.findings_from_report is rerank-first. ranked_candidates is
the primary persistence path; hackernews/polymarket are unconditionally
supplemented from items_by_source because they rank poorly but matter for
watchlists. When ranked_candidates was empty (rerank failed or skipped),
reddit, x, and every other source were silently dropped. The supplement loop
now falls back to all sources only when ranked_candidates is empty; the normal
path is unchanged.

Test repairs:
- test_store.py (6) + test_watchlist_commands.py (2): cascade from the engine fix
- test_get_new_findings_filters_by_date (latent): local-time vs SQLite UTC
  flake — switched to datetime.now(timezone.utc)
- TestPollDeviceAuth (3): mock_time.time side_effect lists too short after
  impl added a last_reminder call — padded timeout test, pinned others to
  return_value=0 (loops terminate via urlopen, not the clock)
- test_bare_run_emits_web_promo: engine reads ~/.config/last30days/.env, so
  a contributor's saved EXA/PARALLEL key made grounding "available" and
  suppressed the web promo. Also missing X made the "x" promo preempt "web".
  Set LAST30DAYS_CONFIG_DIR="", subprocess cwd=tmpdir, XAI_API_KEY stub.
2026-05-16 21:34:22 -07:00
Trevin Chow 719cdef2fb fix(http): contain DNS retry-budget widening to DNS path only
PR #382 introduced an `effective_retries` widening on the first gaierror,
but the widening leaked: every non-DNS error path (HTTPError, non-DNS
URLError, OSError) was gated on `effective_retries - 1` and so inherited
the expanded bound. A caller passing `retries=2` who hit DNS-then-non-DNS
got 3 attempts instead of 2 — contrary to the PR description and the
fail-fast intent of small retry budgets.

Fix:
- Gate every non-DNS sleep/retry decision on the caller's original
  `retries`, not the widened `effective_retries`.
- Add an explicit `break` in each non-DNS branch when the original
  budget is exhausted, so the (possibly widened) outer loop bound
  can't pull us into an extra attempt.

Adds two regression tests covering the DNS-then-non-DNS-URLError and
DNS-then-OSError sequences flagged in Greptile review on PR #382.
2026-05-16 21:13:17 -07:00
Trevin Chow ac04b56acc Merge pull request #383 from lustrousgorilla/bugfix/bird-x-json-decode-retry
fix(bird_x): retry subprocess on non-JSON stdout (HTML anti-bot interstitial)
2026-05-16 21:08:35 -07:00
Trevin Chow 42bfc6c76c Merge pull request #415 from tmchow/fix/sc-source-gating-consistency
fix(sources): align SC source gating between code and docs
2026-05-16 21:08:10 -07:00
Gabriel Arrillaga 5a2fe5279b fix(http): expand retry budget + use exponential backoff on DNS failure
Transient DNS resolution failures (socket.gaierror, surfaced as
urllib.error.URLError with reason=gaierror) were retried with the
generic URLError handler — linear backoff (2s, 4s, 6s) and bounded by
the caller-passed `retries` parameter. For callers that pass small
retry values (e.g. lib/reddit.py::_subreddit_search uses retries=2), a
single first-attempt DNS hiccup followed by one quick retry on the
still-flaky resolver would exhaust the retry budget and wipe a whole
subreddit sweep — which the caller's broad `except Exception` then
silent-empties as `[]`.

Fix:
- Distinguish URLError-with-gaierror-reason from generic URLError via
  a new `_is_dns_failure()` helper.
- For DNS failures, use exponential backoff (1s, 2s, 4s, ...) instead
  of the linear default.
- For DNS failures, expand the effective retry budget to at least
  MIN_DNS_RETRIES (=3) on first occurrence, so callers that passed
  `retries=2` still get a meaningful retry budget for the transient
  case. Non-DNS URLErrors and HTTPErrors keep the caller's value.
- DNS attempts are counted separately (`dns_attempts`) so unrelated
  URLError or OSError failures within the same call don't accidentally
  expand the budget further.

Reported during a community-signal pass where the Reddit subreddit
sweep silently returned zero items after a first-round transient DNS
hiccup. The fix lives at the http layer (where the retry loop is)
rather than per-source so every caller benefits.

Tests:
- Verifies a caller-passed retries=2 still gets MIN_DNS_RETRIES=3
  attempts on gaierror.
- Verifies gaierror-then-success returns successfully on attempt 2.
- Verifies the exponential-backoff sleep pattern (1s, 2s) on the
  retry attempts before exhaustion.
- Verifies a non-DNS URLError (ConnectionRefusedError reason) does
  NOT expand the retry budget — only true DNS failures do.

All 12 http tests pass (8 baseline + 4 new). No regressions in the
broader test suite (1373 pass / 14 fail, vs 1369 pass / 14 fail on
main — the 14 failures are pre-existing and unrelated to this PR).
2026-05-16 20:54:14 -07:00
Gabriel Arrillaga a717dd2b2c fix(bird_x): retry subprocess on non-JSON stdout (HTML interstitial)
Twitter's edge intermittently serves an HTML anti-bot interstitial in
place of JSON when the bird-search subprocess hits a per-query rate
limit. Before this fix, that response made json.loads raise
JSONDecodeError and _run_bird_search() returned {"error": ..., "items":
[]} with the parsed exception message — silent-empty against an
orchestrator that has no way to distinguish "Twitter served HTML; retry
likely succeeds" from "no tweets matched the query."

Surfaced during a community-signal pass where a Karpathy-LLM-wiki
subquery returned zero X items, while a second identical run a few
seconds later returned full results.

Fix:
- Extract the subprocess invocation into _invoke_bird_subprocess() so
  the retry loop can call it multiple times cleanly. Returns
  (result, terminal_error) — terminal_error is non-None for
  unrecoverable cases (subprocess timeout, spawn failure) that should
  NOT be retried.
- In _run_bird_search(), wrap the json.loads parse in a retry loop
  bounded by MAX_JSON_DECODE_RETRIES (=2) with JSON_DECODE_RETRY_DELAY
  (=5s) between attempts.
- On non-JSON stdout, log a diagnostic that names the shape
  (`looks_html`, first-80-chars stdout preview, attempt counter) so
  silent-empty failures become legible in logs.
- On retry exhaustion, return an error dict whose message explicitly
  names "anti-bot interstitial" as the likely cause, distinguishing
  this failure from a genuine no-results case.

Subprocess timeout, spawn failure, and non-zero return-code paths are
unchanged — those are terminal and don't retry.

Tests:
- Verifies HTML-then-JSON returns success on attempt 2.
- Verifies all-HTML returns the diagnostic error dict mentioning the
  anti-bot interstitial cause.
- Verifies subprocess timeout is NOT retried.

All 12 bird_x tests pass (9 baseline + 3 new).
2026-05-16 20:52:56 -07:00
Trevin Chow 9f08bb68b5 fix(sources): align SC source gating between code and docs
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.

This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.

The resulting source-gating model is three-tier and intentional:

  • **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
    reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
    grounding, **tiktok, instagram, threads**

  • **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
    perplexity (different paid API — OpenRouter),
    tiktok_comments / youtube_comments (N× extra SC calls per video)

  • **--search per-query opt-in** (relevance reasons):
    pinterest (visual pins, narrow utility),
    xiaohongshu (Chinese-market specific)

Changes:

- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
  now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
  the new contract and includes a regression guard ("INCLUDE_SOURCES
  should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
  ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
  runs. Drops false INCLUDE_SOURCES requirement for
  tiktok/instagram/threads; corrects pinterest to mention --search;
  adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
  section.

Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
2026-05-16 20:17:31 -07:00
Trevin Chow 602de1ebda Merge pull request #388 from bradferguson/fix/sc-youtube-and-hn-tokenization
fix(sources): unblock SC YouTube + multi-token HN searches
2026-05-16 19:41:12 -07:00
Trevin Chow bf3a82a87e Merge pull request #389 from kuyua9/fix/save-comparison-html-kuyua9
fix: save comparison HTML artifacts
2026-05-16 19:40:59 -07:00
Trevin Chow c010feb8f8 Merge pull request #399 from spiky02plateau/feat/exclude-sources-banner-and-pipeline
feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
2026-05-16 19:39:51 -07:00
Brad Ferguson edea402b7c fix(sources): unblock SC YouTube + multi-token HN searches
Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.

YouTube (ScrapeCreators)
  SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
    {"error":"missing_parameter","message":"You must provide a query"}
  The canonical SC parameter for that endpoint is `query`. Other SC
  endpoints we use (Reddit, TikTok, Instagram) happened to work because
  they use their own per-endpoint parameter names — YouTube was the lone
  outlier.

Hacker News (Algolia)
  Multi-keyword theme queries returned zero hits across every theme.
  Algolia treats query= as strict AND across tokens, so a 4-5 word query
  like "claude, personal agents, agentic infra" matches no stories.

  Three changes in hackernews.py:

  1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
     search_hackernews and _title_matches_query normalize the query the
     same way — addresses Greptile P2 #2 about the two callsites needing
     to stay in sync.
  2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
     by token-overlap instead of requiring every token.
  3. Relax _title_matches_query from all-words to any-word, *but match
     on word boundaries (\b<word>\b) rather than naive substring* —
     addresses Greptile P2 #1, which flagged that the previous any-word
     relaxation would let "ai" falsely match "email" or "rail".

  Token-overlap relevance scoring at parse time already demotes weak
  matches, so word-boundary any-word matching is safe.

Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.

Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-05-16 19:37:16 -07:00
Trevin Chow 4d4ac97ffb refactor: hoist comparison-html gate into a single condition (Greptile DRY)
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
2026-05-16 19:32:49 -07:00
kuyua9 cd34966b4f fix: save comparison HTML artifacts 2026-05-16 19:32:06 -07:00
Trevin Chow 85255be350 Merge pull request #414 from mvanhorn/dependabot/uv/pytest-9.0.3
chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
2026-05-16 19:31:46 -07:00
Trevin Chow 1aa120a420 Merge pull request #407 from DamienStevens/feat/macos-keychain-source
feat(env): macOS Keychain credential source
2026-05-16 19:31:34 -07:00
Trevin Chow 306d8c2d73 fix(env): wire EXCLUDE_SOURCES through get_config + SKILL.md integration
The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().

Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
  propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
  than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
  invoking the skill knows to subtract excluded sources before displaying
  the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
  prose are incomplete — the agent invoking the skill won't know the flag
  exists.)
2026-05-16 19:30:27 -07:00
Trevin Chow d0dcf751f1 fix(keychain): single source of truth for key list + robust USER fallback
Addresses Greptile review on PR #407:

- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
  XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
  manually storing those keys would not see them in --list and the
  interactive prompt would never offer to set them.

  Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
  get_config() pass it through, and add a parity test that parses
  ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
  caught at CI time instead of after a user reports a missing key.

- P2: os.environ.get("USER", "") silently returned "" under sudo, in
  Docker without --env USER, or in CI runners that strip USER. The
  resulting `security find-generic-password -a ""` call would never
  match items stored by setup-keychain.sh, so all lookups silently
  returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
  USER is absent.

The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
2026-05-16 19:25:05 -07:00
dependabot[bot] afd4b04d6d chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-17 02:22:02 +00:00
Trevin Chow 14d8f62e02 Merge pull request #413 from tmchow/docs/compound-release-cascade-pattern
docs: compound learning on release-time consistency-test cascade failures
2026-05-16 19:12:31 -07:00
Trevin Chow 0fd532d249 docs: compound learning on release-time consistency-test cascade failures
Documents the cascade pattern surfaced during this session's install-modernization
arc: a `test_sync_cache_path_uses_skill_version` test asserted that a hardcoded
version pin in `sync.sh` matched the version frontmatter in SKILL.md. When a
release bumped SKILL.md, every open PR's CI failed simultaneously on the
unrelated stale-pin assertion. Affected at least 5 PRs across the 2026-05-13
to 2026-05-15 window (#400, #390, #392, and two others) plus required hotfix
PR #397 to unblock the queue.

The permanent fix shipped in PR #405 (deleted sync.sh + the test). This doc
captures the design lesson so the pattern doesn't reappear: don't write
consistency tests that read two files and assert one matches a value derived
from the other. Either derive at runtime from a single source of truth, or
self-skip / merge-base-scope the test so deletion is a non-event.

Created via /ce-compound. Includes:

- docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md
  (the new learning — first entry under docs/solutions/)
- CONCEPTS.md (new — 4 entries: Skill, Engine, Harness, Beta channel,
  capturing project-specific vocabulary that surfaced across the session)
- AGENTS.md (added one-line Structure entries surfacing docs/solutions/ and
  CONCEPTS.md so fresh agents discover them)
- docs/plans/2026-04-22-{002,003,005,006}-*-plan.md (added deprecation banner
  to each, pointing readers at PR #405 and the new docs/solutions entry —
  these 4 historical plans still reference the deleted sync.sh inline)

Also: closed PR #379 (j-sperling's workaround for the same cascade,
superseded by PR #405's permanent fix).
2026-05-16 19:08:02 -07:00
Trevin Chow 8867a007ea Merge pull request #392 from Gujiassh/fix/openclaw-scrapecreators-optional-env
fix(openclaw): make ScrapeCreators key optional
2026-05-16 18:50:07 -07:00
gujishh 8af8f06b06 fix(openclaw): make ScrapeCreators key optional 2026-05-16 18:48:29 -07:00
Trevin Chow 01b5f3dc1e Merge pull request #363 from thinkun/pr/claim-contributor-entry
Claim contributor entry — @thinkun
2026-05-16 18:46:35 -07:00
Trevin Chow 2e39ee8ce4 Merge pull request #412 from tmchow/refactor/skill-meta-version-helper
refactor: consolidate SKILL.md version regex into lib/skill_meta.py
2026-05-16 18:42:49 -07:00
Trevin Chow 37033164da Merge pull request #410 from tmchow/docs/agents-orientation-multi-harness
docs: reframe as multi-harness Agent Skills package, flip CLAUDE.md ↔ AGENTS.md
2026-05-16 18:42:36 -07:00
Trevin Chow 9fe4b8f130 Update AGENTS.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-16 18:42:14 -07:00
Trevin Chow 73dc6b9996 refactor: consolidate SKILL.md version regex into lib/skill_meta.py
The same `^version:\s*"([^"]+)"\s*$` regex (or a slight variant) was
duplicated across three files: render.py inline, test_plugin_contract.py
local helper, test_version_consistency.py local helper. A future change
to the SKILL.md frontmatter version format would have needed to update
three places without any compile-time pressure to keep them in sync.

New skills/last30days/scripts/lib/skill_meta.py provides:
- `_VERSION_RE` private compiled pattern (accepts double-quoted,
  single-quoted, or unquoted YAML version scalars per the widening
  landed in 997708a)
- `read_skill_version(skill_md_path: Path) -> str | None` helper that
  catches OSError + UnicodeDecodeError and returns None on miss

Callers updated:
- render.py::_skill_version now calls skill_meta.read_skill_version
  inside the SKILL.md fallback loop, returning `read_skill_version(...) or "?"`.
  Semantically equivalent to the old break-after-first-SKILL.md logic.
- test_plugin_contract.py and test_version_consistency.py import the
  helper instead of defining the regex inline. Both files use the
  established sys.path.insert pattern.

Added tests/test_skill_meta.py with 6 direct unit tests covering the
helper's full contract: missing file, undecodable bytes, no-version-line,
and all three quoting styles (double, single, unquoted). Previously the
helper was only exercised transitively through render._skill_version().

Added test_skill_md_uses_double_quoted_version to
test_version_consistency.py — the old per-test regex incidentally
asserted "this repo's SKILL.md uses double-quotes" by being strict;
the shared helper accepts all three styles, so the assertion is now
explicit instead of implicit.

Code-reviewed by ce-code-review (8 reviewers); safe_auto fixes applied
inline (rename to _VERSION_RE, group or-chain instead of generator,
docstring tightened, dropped unnecessary `from __future__ import
annotations`, tightened signature to Path-only).

Conftest.py refactor for the sys.path.insert duplication across ~20 test
files filed as issue #411 — out of scope for this PR (touches many
files, separate concern).

Test results: 23 passed in the affected test set (16 prior + 6 new
test_skill_meta tests + 1 new double-quote assertion). Full suite shows
same 13 pre-existing failures as main; zero new failures.
2026-05-16 18:17:33 -07:00
Trevin Chow 1fd763e09f docs: flip CLAUDE.md ↔ AGENTS.md — AGENTS.md becomes canonical, CLAUDE.md points at it
Mirrors the multi-harness reframing of the project itself. CLAUDE.md is
Claude-Code-specific by name; AGENTS.md is the multi-harness convention
that Codex, Cursor, Gemini CLI, GitHub Copilot, and most other Agent
Skills hosts also read. The canonical content belongs in the file
multi-harness tooling expects.

git mv preserves history — the Orientation section and everything else
that was in CLAUDE.md is now tracked under AGENTS.md, with full blame
continuity. The new CLAUDE.md is a one-line `@AGENTS.md` reference so
Claude Code continues to load the content (it follows @ references).
2026-05-16 17:03:11 -07:00
Trevin Chow e0f6ef845a docs(claude.md): add Orientation section, reframe as multi-harness Agent Skills package
Closes the spirit of #335 (closed in favor of this fresh PR after the
sync.sh thread became obsolete via PR #405).

Two changes:

1. Top-of-file description reframed from "Claude Code skill" to
   "Agent Skills package... installable across Claude Code (most common
   host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other
   Agent Skills hosts". The skill works across every major agent host
   after the install-modernization work in PR #400/#404/#405/#409.
   Calling it "Claude Code skill" undersells the surface and biases
   contributors toward Claude-Code-specific assumptions.

2. New ## Orientation section (4 bullets) framing the project for
   contributors who would otherwise read the python3 invocation in
   ## Commands and form a CLI-first mental model. Names the trap
   explicitly with one concrete invalid-syntax example
   (`/last30days OpenClaw --emit=html | pbcopy` — slash commands don't
   pass shell mechanics through). Bullets adapted from #335 with
   multi-harness framing replacing the Claude-Code-only framing.

No code changes. No SKILL.md changes. CLAUDE.md only. AGENTS.md
inherits via @CLAUDE.md.
2026-05-16 17:01:04 -07:00
Trevin Chow c918e18465 Merge pull request #409 from tmchow/refactor/skill-dir-relative-resolver
refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
2026-05-16 16:55:32 -07:00
Trevin Chow 6fe0aca7ee refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
The Step 1 and comparison-mode resolver loops walked a hardcoded list
of install paths trying to find scripts/last30days.py. Two problems:

1. The list was never exhaustive — it covered ~/.codex/skills, but not
   ~/.claude/skills, ~/.cursor/skills, ~/.gemini/skills, ~/.copilot/skills,
   ~/.hermes/skills/research, etc. PR #406 was about to fix that by
   enumerating more paths, but enumeration is the wrong shape.
2. The resolver could pick a different install than the SKILL.md the
   model loaded from. Spec-vs-engine divergence is subtle and confusing
   when it triggers.

The model already knows the SKILL.md path it loaded (from its Read tool
result). Templating that into the bash block is strictly better than
guessing across an enumerated list:

- Works for every harness without enumeration (Hermes, Cursor, anything
  new) because we just use wherever the harness loaded SKILL.md from
- Aligns spec with engine — the engine runs from the same install the
  spec was read from
- Deletes ~80 lines of bash across Step 1 + comparison-mode + the
  prose preamble describing the resolver

Mechanics:

- SKILL_DIR placeholder in both bash blocks — model substitutes the
  absolute path of the directory containing the SKILL.md it just Read
- One-line validation `[ ! -f "$SKILL_DIR/scripts/last30days.py" ]`
  catches bad templating with a clear error
- All references to $SKILL_ROOT replaced with $SKILL_DIR (badge
  VERSION lookup, prose description in the LAW-7 preamble area)
- STEP 0 unchanged — different concern (marketplaces stale-clone hop)

Version 3.2.3 -> 3.2.4 (behavior change: install paths the resolver
list never enumerated now work; install paths it did enumerate work
the same way they used to but via the SKILL_DIR template).
2026-05-16 16:26:45 -07:00
Matt Van Horn d9f606ff75 chore: add gogcli #589 zoom demo gif (#408)
PR demo embed asset for openclaw/gogcli #589 (feat: --with-zoom).
Hosted here for stable raw URL.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-16 11:18:41 -07:00
Damien Stevens 74a387b093 feat(env): macOS Keychain credential source
Adds the macOS Keychain as the lowest-priority credential source on Darwin.
Items stored as generic passwords with service name "last30days-<KEY>" for
the current user are picked up automatically by get_config() — file env
and process env still win on collision.

No new config knob: behavior is strictly additive. On non-Darwin (or when
the `security` binary is missing) the loader is a no-op, so Linux/Windows
behavior is unchanged.

  Priority (highest wins):
    1. Environment variables
    2. .claude/last30days.env (per-project)
    3. ~/.config/last30days/.env (global)
    4. macOS Keychain items prefixed last30days- (new)

Includes:
  - lib/env.py: KEYCHAIN_SERVICE_PREFIX constant, _load_keychain helper
    (platform-gated, shutil.which-gated, subprocess-error tolerant),
    wiring into get_config before get_openai_auth so OPENAI_API_KEY can
    come from Keychain too, _CONFIG_SOURCE reports "keychain" when no
    file source is present.
  - scripts/setup-keychain.sh: bash helper with interactive set,
    --list, --delete, --replace modes. Uses `security add-generic-password`.
  - tests/test_env_keychain.py: 12 tests covering platform gate,
    missing-binary gate, success path, whitespace stripping, subprocess
    errors swallowed, get_config precedence, and an OPENAI_AUTH wiring
    regression test.
  - tests/test_env_cookies.py: existing integration test mocks the new
    _load_keychain hook so it stays hermetic on Darwin developer
    machines that have real keychain entries.
  - README.md: new "macOS Keychain (optional)" subsection under
    "Bring your own keys" documenting setup-keychain.sh and the manual
    `security add-generic-password` invocation.

Tested on macOS with a populated keychain and against the existing pytest
suite — CI-tracked tests (test_plugin_contract.py, test_version_consistency.py)
plus all env-touching tests pass. Pre-existing unrelated failures in
test_store.py / test_watchlist_commands.py / test_setup_openclaw.py /
test_footer_nudge_suppression.py are untouched.
2026-05-16 09:01:28 -04:00
Trevin Chow 4a30923892 Merge pull request #405 from tmchow/docs/readme-multi-harness-install
docs+refactor: modernize install story everywhere, delete sync.sh
2026-05-15 23:46:57 -07:00
Trevin Chow 9fb19eae63 refactor: delete sync.sh, dev workflow moves to npx skills add . -g -y + native installers
Every job sync.sh did has a better replacement:

- Per-harness skill dirs (~/.claude/skills, ~/.codex/skills, ~/.agents/skills):
  `npx skills add . -g -y` writes to every detected harness's home dir and
  uses symlinks by default. Edits propagate live — no re-deploy step.
- Hermes (~/.hermes/skills/research/last30days):
  `hermes skills install mvanhorn/last30days-skill --force` pulls from
  GitHub and handles the deploy itself. The script wrapping was redundant.
- OpenClaw variant: `clawhub install last30days-official` is what users
  already run per the README; the maintainer doesn't need a separate
  variant-deploy step in the public repo's scripts.
- Claude marketplace cache (~/.claude/plugins/cache/...): this was a
  "test against the official install path" hack we shouldn't have been
  recommending. With PR #400's resolver collapse, STEP 0 no longer
  enforces the cache as the only valid SKILL.md location. Just install
  the skill normally via `npx skills` or the marketplace.

Cleanup:

- DELETE skills/last30days/scripts/sync.sh
- tests/test_version_consistency.py — drop test_sync_cache_path_uses_skill_version
- CLAUDE.md — replace the sync.sh command + rule with `npx skills add . -g -y`
- HERMES_SETUP.md — Installation now uses `hermes skills install --force`;
  developer-alternative section shows the symlink pattern for live editing
- render.py — _skill_version docstring no longer attributes the
  ".claude-plugin absent" case to sync.sh; explains it via per-harness
  install paths in general
- .github/PULL_REQUEST_TEMPLATE.md — drop the "Ran bash scripts/sync.sh"
  checklist item

CHANGELOG and historical docs (release notes, plan files) keep their
existing sync.sh mentions as accurate history.
2026-05-15 23:42:31 -07:00
Trevin Chow d1cc29d338 docs(readme): add -g (global) flag to every npx skills example
`npx skills add` defaults to project-local install (`./.skills/`,
committed with the repo). For a research-the-world skill like this one,
that's almost never what users want — they want it available across all
projects, not scoped to whichever directory they happened to run the
install from.

Adding `-g` (global) to every npx skills example in the README:
- Top-of-file install snippet
- Install table row
- Claude Code subsection's "alternative via npx skills" example
- Codex/Cursor/etc. subsection's default, per-harness, update, list,
  and remove commands

Brief one-liner explains what `-g` does and notes that dropping it
gives a project-local install for users who want team consistency on
a specific codebase.
2026-05-15 23:20:33 -07:00
Tobi 095bcae915 fix(check-config): normalize EXCLUDE_SOURCES (lowercase + whitespace) before matching
The bash banner accounting used raw substring matching while
pipeline.py normalises EXCLUDE_SOURCES via .strip().lower(). With
EXCLUDE_SOURCES=TikTok,Instagram (or with surrounding spaces),
pipeline correctly excludes the sources but the banner did not
deduct them — count showed 1-2 higher than what the pipeline
actually runs. Normalisation now mirrors the Python side
(lowercase, collapse whitespace around commas, strip outer whitespace).

Reproducer (clean HOME with config EXCLUDE_SOURCES=TikTok,Instagram):
  before: /last30days: Ready — 7 sources active.
  after:  /last30days: Ready — 5 sources active.

Addresses Greptile review comment P1 on #399.
2026-05-16 08:05:34 +02:00
Trevin Chow ded52062e6 docs(readme): drop Gemini CLI native-extension install path
The native `gemini extensions install` path was a workaround for the
v0.9.0 installer bug (still unresolved per upstream issue #11452).
Now that `npx skills add -a gemini-cli` covers Gemini cleanly with the
same install/update story as every other supported harness, the native
path is just one more confusing option to maintain. Users on Gemini get
the same recommendation everyone else does.

Removes the dedicated "Gemini CLI (native extension)" subsection and
the separate table row. Gemini CLI is now surfaced once, in the npx
skills section, alongside Codex, Cursor, Copilot, and the rest.
2026-05-15 23:03:51 -07:00
Trevin Chow 164d7ae6ed docs(readme): surface gemini-cli (and copilot, windsurf, 50+ others) in npx skills coverage
npx skills supports 50+ harnesses via the -a flag, including gemini-cli,
github-copilot, windsurf, cline, continue, roo, aider-desk, opencode,
goose, and more — not just the few I'd listed initially. Updating to
reflect that breadth.

- Top-of-file snippet now reads "Codex, Cursor, Copilot, Gemini CLI, or
  any of 50+ Agent Skills hosts" (was: "Codex, Cursor, Copilot, or any
  Agent Skills host" + Gemini listed separately in the table footer).
- Install table: same expansion; Gemini CLI native-extension row relabeled
  to clarify it's the native path (not the only Gemini option).
- npx skills subsection: lists the most common harness flags and links
  to the upstream vercel-labs/skills repo for the full list.
- Gemini CLI native subsection: now leads with "the npx skills path
  above is simpler" and frames the native install as the alternative
  for users with an existing Gemini extensions workflow or who hit
  the v0.9.0 installer bug.
2026-05-15 23:02:57 -07:00
Trevin Chow f1ce7533e6 docs(readme): recommend Claude Code plugin, add npx skills install for Codex/Cursor/Copilot
The skill is now installable across every major agent harness after the
SKILL.md path-resolver work landed in PR #400 + #404. README didn't yet
reflect that — the install table only listed Claude Code, OpenClaw, and
Gemini CLI, and the top-of-file install snippets featured Hermes (an
internal dev workflow, not a public install method).

Restructured the install section:

- Top-of-file snippets: just Claude Code (recommended, auto-updates) and
  the universal `npx skills add` one-liner. Dropped Hermes from the
  prominent spot (internal-only); pointed everything else to the Install
  section below.
- Install table: added a third column for update commands, since every
  harness now has a distinct update path worth surfacing. Added the
  `npx skills` row covering Codex/Cursor/Copilot/any Agent Skills host.
- Claude Code subsection: explains why it's recommended (marketplace
  handles versioned cache + auto-refresh) and notes that the agent-skills
  install also works on Claude Code if preferred (`-a claude-code`).
- New "Codex, Cursor, Copilot, and other Agent Skills hosts" subsection:
  shows the default install, per-harness `-a` targeting, and the update
  commands (`npx skills update last30days` for one skill, bare
  `npx skills update` for all).
- Manual (developer) subsection: switched from a clone-into-skills-dir
  recipe to a clone + symlink recipe. Symlink keeps the install in sync
  with the working tree as you edit, no re-copy on each change.

No code changes. No version bump (docs-only).
2026-05-15 22:58:54 -07:00
Trevin Chow 0b939bf703 Merge pull request #404 from tmchow/fix/json-plan-shell-quoting
fix(skill): write --plan / --competitors-plan to tmpfile (closes #403)
2026-05-15 22:52:49 -07:00
Trevin Chow 9f95efb215 fix(skill): use portable trailing-XXXXXX mktemp form for plan tmpfiles
Greptile's review flagged mktemp -t as non-portable between BSD and GNU.
The suggested replacement (mktemp "$TMPDIR/...XXXXXX.json") is correct
about dropping -t but still puts X's in the middle of the template name
(XXXXXX.json), which BSD mktemp does not substitute — only X's at the
end of the basename are replaced on BSD. Verified on macOS:

  mktemp "$TMPDIR/last30days-test.XXXXXX.json"
  → /var/folders/.../last30days-test.XXXXXX.json  (X's left literal)

The fully portable form uses trailing X's and drops the .json suffix
(engine reads by path, not extension):

  mktemp "$TMPDIR/last30days-test.XXXXXX"
  → /var/folders/.../last30days-test.DXAHzR     (X's substituted)

Verified on bash and zsh, BSD/macOS. GNU/Linux is already fine since
GNU substitutes X's wherever they appear in the basename.

Applied to both --competitors-plan (comparison-mode block) and --plan
(Step 1 block) tmpfile writes.
2026-05-15 22:50:56 -07:00
Trevin Chow ff54c07a3b fix(skill): write --plan / --competitors-plan to tmpfile, bump 3.2.2 -> 3.2.3
Closes #403.

The SKILL.md templates instructed the model to invoke the engine with
inline single-quoted JSON: `--plan '$JSON'` and `--competitors-plan '{...}'`.
When any resolved field value contained an apostrophe (common in `context`
strings like "McDonald's", "people's choice", or contracted forms like
"don't", "won't"), the inner `'` closed the outer single-quote and broke
shell parsing before the engine was even invoked.

Observed during PR #400 testing: a Codex run hit the trap and self-healed
by re-encoding, wasting one engine invocation and ~30s of latency.

Fix: switch both templates to the heredoc + tmpfile pattern. The engine's
`parse_plan()` and `parse_competitors_plan()` already check
`os.path.isfile(plan_str)` and read from disk — only the SKILL.md prose
needed to change.

The quoted heredoc marker (<<'PLAN_EOF') is load-bearing: it suppresses
shell interpolation so apostrophes, $, backticks, etc. pass through verbatim.
A trap on EXIT cleans up the tmpfile after the engine call returns.

LAW 7's "MUST contain --plan" self-check guidance and Step 1's invocation
example both updated to reference the file form. Comparison-mode invocation
block updated the same way for --competitors-plan.

Version bump 3.2.2 -> 3.2.3 because this is a behavior change users
running comparison-mode queries will notice (no more "shell quoting error,
retrying" sequences on apostrophe-containing context strings).
2026-05-15 22:43:02 -07:00
Trevin Chow e276c30477 Merge pull request #400 from tmchow/refactor/skill-md-relative-path-resolver
refactor(skill): SKILL.md-relative path resolver, drop Codex native plugin
2026-05-15 22:36:53 -07:00
Trevin Chow 2f277dfc66 fix(skill): address greptile P1+P2 review feedback on PR #400
Two real bugs flagged in the automated review of PR #400; both small.

1. render.py::_skill_version manifest with no "version" key

   `json.loads(manifest.read_text()).get("version", "?")` returned "?"
   immediately on a valid JSON manifest that lacked the "version" key,
   never falling through to the SKILL.md frontmatter fallback. Contradicted
   the docstring's "Returns '?' only if both sources are missing" contract.
   Same shape if version is present but empty string ("" produces the
   broken badge `🌐 last30days v · synced ...`).

   Fix: pull the version out of the parsed dict, then `continue` to the
   next ancestor if it's None or empty. Falls through to the SKILL.md
   walk only after exhausting every ancestor.

2. SKILL.md STEP 0 re-read target hardcoded to nested cache layout

   STEP 0 told the model to re-read from
   `$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md` — the new nested
   layout. But Step 1's resolver explicitly handles both shapes
   (nested `{cache}/{version}/skills/last30days/` and flat
   `{cache}/{version}/`), noting "Both shapes ship in the wild." On an
   install where the highest-versioned cache happens to be the older flat
   shape, STEP 0's re-read target wouldn't exist; the model would silently
   stay on the stale marketplaces/ copy STEP 0 was supposed to move it
   away from — the exact failure mode this guard was added to prevent.

   Fix: extend the STEP 0 bash to resolve $CLAUDE_CACHE_SKILL_MD by
   probing both layouts, then have the model hop to that resolved path
   instead of constructing the path from a hardcoded suffix.

Two new tests in tests/test_skill_version.py cover the missing-key and
empty-string cases for fix 1. Fix 2 is exercised via the bash probe at
verify time (the STEP 0 prose-contract test isn't unit-testable from
Python, but the dual-layout bash is verified to resolve to the correct
SKILL.md on both shapes).

Stale finding skipped: greptile also flagged a missing try/except on the
SKILL.md read_text() call, but that was already addressed during the
ce-code-review safe_auto pass earlier in this PR — current code wraps it
in `try/except (OSError, UnicodeDecodeError)`, strictly more defensive
than the suggested fix.
2026-05-15 22:34:52 -07:00
Trevin Chow 6c2c55733c fix(skill): use find instead of ls+glob in cache resolvers (zsh compatibility)
zsh errors on globs that match nothing instead of returning the literal
pattern (bash's default), and `2>/dev/null` does not suppress the error
because it comes from the shell's glob expansion before `ls` even runs.
Under Codex (which executes the SKILL.md bash via zsh), STEP 0 and the
Step 1 / comparison-mode resolvers emitted noisy "no matches found"
errors on machines without a Claude plugin cache populated.

Replaces all three `ls -d $HOME/.claude/plugins/cache/last30days-skill/last30days/*/`
invocations with `find ... -mindepth 1 -maxdepth 1 -type d 2>/dev/null`.
find is POSIX-portable, errors silently when the base dir doesn't exist,
and never triggers shell glob errors. `sort -V | tail -1` precedence
preserved (verified: picks 3.10.0 over 3.2.1 over 3.1.0). Trailing-slash
strip removed because find doesn't append slashes.

Observed in Codex session running /last30days against PR #400 with the
Claude plugin cache deleted - bash output was:
  zsh:1: no matches found: /Users/.../last30days/*/

After fix: clean empty output, exit 0, STEP 0 correctly treats it as
"no cache present, do not hop", resolver falls through to per-harness
skill dirs as designed.
2026-05-15 22:14:37 -07:00
Trevin Chow 997708ad48 refactor(skill): apply ce-code-review fixes — bump to 3.2.2, fallback tests, comparison resolver
12 fixes from the multi-agent code review on PR #400:

Version 3.2.1 -> 3.2.2 across all manifests (SKILL.md frontmatter + body
header, pyproject.toml, .claude-plugin/{plugin,marketplace}.json, sync.sh
cache path). The PR ships observable behavior changes (STEP 0 logic flip,
resolver order change, badge fallback) that should not silently appear
under the same version number — the new fallback reads SKILL.md version
directly so the badge would otherwise be misleading.

render.py::_skill_version:
- `import re` moved to module top
- _VERSION_RE extracted as a module-level compiled pattern that accepts
  double-quoted, single-quoted, and unquoted YAML version scalars
- `break` -> `continue` on corrupt manifest, so a corrupt inner manifest
  no longer shadows a valid outer one
- Wrap SKILL.md read_text() in try/except for UnicodeDecodeError to keep
  badge emission from crashing on mis-encoded SKILL.md
- Docstring clarifies precedence; inline comment marks the fallback boundary
  between the manifest walk and the SKILL.md walk

tests/test_skill_version.py (new): 7 unit tests for the fallback paths
(manifest absent, manifest corrupt, corrupt-inner + valid-outer, both
absent, SKILL.md without version, single-quoted, unquoted).

tests/test_plugin_contract.py: tombstone test asserting .codex-plugin/
stays removed (was the only CI guard against accidental reintroduction).

SKILL.md:
- STEP 0 bash echoes CLAUDE_CACHE_LATEST so the model can see the
  resolved value when deciding whether to hop
- "Both shapes ship in the wild" comment now names the two cache layouts
  (nested {cache}/{version}/skills/last30days/ vs flat {cache}/{version}/)
- Comparison-mode bash invocation gets its own inline SKILL_ROOT resolver
  (latent gap: the contract tells the model to skip Step 1 on comparison
  queries, so SKILL_ROOT was previously unset there)

CHANGELOG.md: [Unreleased] entries for the resolver rewrite and the
breaking removal of Codex native-plugin support.

All 9 reviewer personas surfaced findings; 3 cross-reviewer corroboration
clusters were promoted (import re, "both shapes" comment, missing fallback
tests). Maintainability follow-up flagged: regex now duplicated across
render.py and 2 test files; could consolidate via shared lib/skill_meta.py
helper in a future PR.
2026-05-15 21:45:25 -07:00
Trevin Chow c913e1cf89 refactor(skill): SKILL.md-relative path resolver, drop Codex native plugin
STEP 0 (CANONICAL PATH SELF-CHECK) used to force any SKILL.md load that wasn't
under $HOME/.claude/plugins/cache/last30days-skill/last30days/{version}/ to
re-Read from there. That guard is Claude-Code-specific (defends against the
marketplaces/ stale-clone bug) and broke under non-Claude installers like
`npx skills add`, ~/.codex/skills/, and ~/.agents/skills/.

The new STEP 0 narrows the check to its actual target: fire only when the
loaded SKILL.md path contains /.claude/plugins/marketplaces/. Every other
install path is trusted. The 2026-04-22 incident workaround is preserved
without breaking other harnesses.

Step 1 SKILL_ROOT resolver collapses the Codex-first / Claude-fallback /
CWD-fallback chain into a single precedence walk: Claude plugin cache
(versioned) first, then ~/.codex/skills, ~/.agents/skills, repo checkout,
./.skills/last30days (npx skills install dir), CWD, and GEMINI_EXTENSION_DIR.

Also drops Codex native plugin support: .codex-plugin/plugin.json is deleted,
the badge VERSION jq fallback in line 108 stops looking at it, and render.py's
_skill_version no longer scans for it. Codex users install via `npx skills add`
or the per-harness skill dir going forward.

render.py::_skill_version gains a SKILL.md frontmatter fallback so the badge
no longer emits `v?` on install dirs that sync.sh populates (which don't
include .claude-plugin/plugin.json).
2026-05-15 21:44:55 -07:00
Trevin Chow 54db014c7c fix(sync): point sync.sh at this repo's plugin cache, not the private repo's (#402)
sync.sh was written against the layout of mvanhorn/last30days-skill-private
(`.../cache/last30days-skill-private/last30days-3/{version}`) and that path
was never updated when this public repo got its own copy. Running sync.sh
from here populated the BETA channel's cache (`/last30days-beta`) instead
of this repo's own `/last30days` cache, so devs working in this repo could
not test their changes via the public slash command without waiting for a
marketplace release.

Path now derives from this repo's own manifests:
- marketplace name `last30days-skill` (.claude-plugin/marketplace.json)
- plugin name      `last30days`       (.claude-plugin/plugin.json)

Drops the `last30days-3-nogem` target along with it - that's a private-repo
variant with no public equivalent.

Updates test_sync_cache_path_uses_skill_version to assert the new path
pattern and clarifies the COMMON_TARGETS comment so the next person editing
it understands which marketplace/plugin name segments come from where.
2026-05-15 21:43:23 -07:00
Tobi 4f6b86c456 feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES
config key. Two coupled changes:

1. pipeline.available_sources() filters out any source listed in
   config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive,
   whitespace-tolerant) before returning.
2. hooks/scripts/check-config.sh "Ready — N sources active" banner
   subtracts excluded sources from the ScrapeCreators +3 (Reddit
   comments + TikTok + Instagram) so the count matches what the
   pipeline actually runs.

Use case: skip TikTok/Instagram on runs where you only want
text-substantive sources, without unsetting SCRAPECREATORS_API_KEY
(which would also kill Reddit comments). The existing INCLUDE_SOURCES
allowlist covers Perplexity opt-in but doesn't cover this denylist case
— tiktok and instagram are added unconditionally when
SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key.

Tests (tests/test_pipeline_v3.py::TestExcludeSources):
- excludes tiktok+instagram when listed
- no exclusion when env unset or empty string
- case-insensitive + whitespace-tolerant parsing
- works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
2026-05-16 00:54:18 +02:00
Trevin Chow 80a1a47eef refactor: drop requests dep, route all providers through lib/http urllib wrapper (#393)
Five provider modules (pinterest, threads, instagram, tiktok, youtube_yt)
and watchlist.py each carried a try/except `requests` import with parallel
urllib + requests branches. The urllib path already used the
stdlib-only wrapper at `lib/http.py` (retries, 429 handling, HTTPError).
This collapses every dual-branch into a single `http.get`/`http.post`
call and removes the `requests` dependency from `pyproject.toml`.

Also drops 4 transitive deps (urllib3, certifi, charset-normalizer, idna)
from the lockfile, leaving the skill stdlib-only at runtime.

Tests for tiktok comments and watchlist delivery were rewritten to mock
`lib.http` directly instead of the now-removed `requests` module.

Out of scope but flagged during review: the 13 surviving SC call sites
share a near-identical scaffold and would benefit from a
`http.scrapecreators_get(url, params, token, ...)` helper. Filed for a
follow-up PR rather than expanding scope here.
2026-05-15 08:07:43 -07:00
Matt Van Horn c845f483d6 fix(sync): bump cache target to 3.2.1 to match SKILL.md (#397)
test_sync_cache_path_uses_skill_version asserts that sync.sh's plugin
cache path includes the version from SKILL.md frontmatter. The frontmatter
moved to 3.2.1 in #371 but sync.sh still pointed at 3.2.0, leaving CI red
on every PR.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 08:06:33 -07:00
Matt Van Horn dc934ddb6a feat(digg): rename to 'Digg' and bump per-cluster post limits (#372)
* feat(digg): bump POSTS_PER_CLUSTER to 5 and render limit to 3

Match the per-item enrichment cap and inline-display cap used by the
other sources (Reddit, HN, YouTube, TikTok, GitHub all use 5 fetched /
3 displayed). At the previous 3/2 caps the engine routinely truncated
cluster context — a recent run on cli-printing-press lost the Jason
Calacanis quote tweet entirely because the display cut off after Garry
Tan's first two posts.

* feat(digg): rename 'Digg AI 1000' to 'Digg' in user-facing strings

Drop the 'AI 1000' suffix from the footer line, source label, inline
quote attribution ('via Digg'), why_relevant, container, mock title,
SKILL.md source list, and README sources table. Internal code comments
and docstrings still reference the upstream Digg AI 1000 product.

Bumps version to 3.2.1 and adds a CHANGELOG entry covering this rename
and the POSTS_PER_CLUSTER / render-limit bumps from the prior commit.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 21:04:23 -07:00
Matt Van Horn 80392061d4 chore(release): v3.2.0 (#371)
Release / build-and-release (push) Has been cancelled
* chore(release): v3.2.0

Bumps plugin/marketplace/codex/pyproject versions from 3.1.1 to 3.2.0.
Promotes the Unreleased CHANGELOG entries (--emit=html, Digg AI 1000
source) to the 3.2.0 release section.

* chore(release): bump SKILL.md header and sync.sh path to 3.2.0

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 19:29:16 -07:00
Matt Van Horn c04bd67922 feat: add Digg AI 1000 as an opt-in source (#370)
* feat(digg): add Digg AI 1000 source module with cluster search and post enrichment

- search_digg shells out to digg-pp-cli with --since 30d --agent
- parse_digg_response normalizes clusters to last30days dict shape
- enrich_with_top_posts attaches top-ranked X posts to top-K clusters
- shutil.which gate plus subproc.run_with_timeout discipline matches
  bird_x.py / youtube_yt.py patterns

25 unit tests cover parse, age window, relevance, binary-missing
fallback, timeout recovery, and partial enrichment failures.

* feat(digg): wire Digg source into pipeline, normalize, signals, and render

pipeline.py:
- Import digg, add to MOCK_AVAILABLE_SOURCES, gate via shutil.which
- Dispatch case calls search_digg + parse_digg_response, runs
  enrich_with_top_posts at default/deep depth
- Mock fixture includes one enriched cluster + one bare cluster

normalize.py:
- _normalize_digg maps cluster dicts to SourceItem with
  container='Digg AI 1000' and metadata.posts pass-through

signals.py:
- SOURCE_QUALITY['digg'] = 0.85 (top tier alongside YouTube,
  reflecting Digg's curatorial layer)
- ENGAGEMENT_WEIGHTS['digg'] balances postCount, uniqueAuthors,
  and the rank_score derived from Digg's curatorial position

render.py:
- SOURCE_LABELS['digg'] = 'Digg AI 1000'
- _FOOTER_SOURCES adds '⛏️ Digg AI 1000' line after GitHub
- ENGAGEMENT_DISPLAY mirrors footer keys
- New _digg_posts_for + _format_digg_quote helpers emit inline
  '@handle via Digg AI 1000' quotes for clusters with attached X
  posts; both compact and full-dump renderers call them

* feat(digg): polish per-item engagement display and progress label

- ENGAGEMENT_DISPLAY for digg uses 'posts' / 'auth' to match the
  codebase abbreviation convention (HN: 'pts'/'cmt', X: 'rt'/'re')
- Footer item word changes from 'story' to 'cluster' to dodge the
  pre-existing naive plural in _footer_line_for_source ('storys')
  and to match Digg's actual data model
- ui.py SOURCE_COMPLETION_META adds digg with correct 'cluster'/
  'clusters' plural so 'Research complete' shows 'Digg: N clusters'

* feat(digg): document Digg AI 1000 source in skill, README, and changelog

- planner.py SOURCE_CAPABILITIES adds digg with discussion/social/link
  capabilities so the planner offers it through the standard fanout
- SKILL.md ACTIVE_SOURCES_LIST gate includes 'which digg-pp-cli' check
  and the source list / available-sources line names digg as opt-in
- README.md Sources table adds the Digg AI 1000 row with the activation
  gate so first-time readers see what they get
- CHANGELOG.md Unreleased section calls out the source addition

* fix(digg): enrich post-dedupe so brief survivors carry inline quotes

Pipeline dispatch was attaching X posts to the top-3 items returned by
search, but dedupe later picked different survivors when multiple
clusters compared similar (common for trending topics). The brief
ended up showing clusters with no posts attached even though
enrichment ran successfully on positions 0-2.

Move enrichment to _finalize_items_by_source. The new
digg.enrich_source_items helper reads metadata['clusterUrlId'] and
writes metadata['posts'] in place on the SourceItems that actually
survive dedupe.

Verified live on 'openclaw': 2 surviving clusters, both now carry
real X-post quotes from @sama and @jeremyphoward attributed
'via Digg AI 1000'.

Adds 3 unit tests covering survivor enrichment, non-digg skip, and
clusterUrlId fallback to item_id.

* test(digg): relax live off-topic test to check shape, not emptiness

Digg's live search uses fuzzy/popularity fallback, so an impossible
token can still return some loosely-related clusters. The contract
the pipeline depends on is shape (results is always a list);
token-overlap relevance handles the noise downstream.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-05-09 19:05:41 -07:00
Anurag Chakradhar ed455ca036 Claim contributor entry — @thinkun 2026-05-08 17:15:08 +10:00
Trevin Chow b1773be8f3 feat(emit): --emit=html for shareable self-contained briefs (#332)
Adds a one-command shareable HTML mode to /last30days. The skill detects
HTML intent (explicit --emit=html / --emit:html / --html flag in
$ARGUMENTS, or natural-language asks like "give me a shareable brief",
"for Slack", "export as HTML"), runs the normal research + chat synthesis
flow, then saves a self-contained HTML file to
~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat
as usual; the HTML is an additional artifact for sharing.

User experience:

  /last30days OpenClaw --emit=html
  /last30days OpenClaw, give me an HTML brief for Slack

Synthesis prints to chat. Last line of the response: "📎 Shareable brief
saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it
into a message, browser-print to PDF, email it.

Architecture:

  - SKILL.md gets a small detection block (triggers + early exit +
    MUST/MUST NOT rules + rationale) that points to a reference file.
  - references/save-html-brief.md owns the implementation: capture the
    synthesis verbatim into a temp file via heredoc, invoke the engine
    with --emit=html --synthesis-file, save to disk, append the
    confirmation line to chat.
  - lib/render.py exposes render_for_html(report, synthesis_md=None) and
    render_for_html_comparison(...) -- clean markdown for HTML
    conversion. Omits debug file header, model-facing safety note, and
    data quality warnings (those stay in engine stderr; recipients can't
    act on them in a shared artifact).
  - lib/html_render.py is a new module: ~200-line CSS template (dark
    mode default, prefers-color-scheme switch, print stylesheet, mobile
    breakpoint), stdlib-regex markdown-to-HTML converter, marker-based
    META + engine-footer wrapping, PROSE_LABELS registry promoting plain
    -text labels to <h2>, colophon builder.
  - last30days.py adds --emit=html argparse choice and --synthesis-file
    PATH flag (engine still callable directly without the skill in the
    loop).

Design:

  - Voice-led research brief, not corporate report. Inter + JetBrains
    Mono via Google Fonts with full system fallbacks (no FOIT, works
    offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp:
    body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label
    20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px.
  - 720px max-width, generous whitespace, no card layouts or shadows.
  - Print stylesheet: light theme, A4 margins, [href]::after URL
    footnotes, page-break-inside:avoid on the engine footer.

Templated (locked) shell:

  - HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS
    inline.
  - .badge / .meta / .engine-footer / .colophon containers.

Flexible (role-based):

  - <h2> rendering covers BOTH plain ## headers (comparison mode per
    LAW 4 exception) AND promoted prose labels via PROSE_LABELS
    registry. Adding a new SKILL.md prose label is a one-line tuple
    addition; no CSS or template changes.
  - Marker-based engine boundaries (<!-- META: ... -->,
    <!-- PASS-THROUGH FOOTER -->) survive the markdown converter and
    get promoted post-conversion. Robust to engine output format
    changes.
  - Generic markdown-to-HTML for body content; future SKILL.md additions
    (new sections, tables, blockquotes) render correctly without code
    changes.

Tests: 30 new tests in tests/test_html_render.py covering snapshots
(rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose
label promotion, warning exclusion from artifact, parseability via
html.parser, no-script self-containment.

No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip
dependencies, no JavaScript anywhere.
2026-05-02 11:30:22 -07:00
Ilia Alshanetsky 5b87cca886 fix(xurl): treat PermissionError from PATH lookup as unavailable (#322)
is_available() only caught FileNotFoundError and TimeoutExpired. On WSL,
a /mnt/c/.../WindowsApps entry on $PATH returns EACCES during exec, and
Python raises PermissionError. That escaped is_available() and crashed
pipeline.diagnose() before any source ran.

Catch OSError instead. It covers FileNotFoundError, PermissionError, and
any other spawn-time OS error, so a non-executable xurl on PATH falls
through to the next backend instead of aborting the run.
2026-04-26 14:16:14 -07:00
Ilia Alshanetsky bbf892aecc refactor: extract subprocess cleanup into shared subproc helper (#210)
bird_x.py and youtube_yt.py had four near-identical copies of the same
subprocess cleanup dance (Popen + os.setsid + communicate(timeout) +
SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to
lib.subproc.run_with_timeout(), which:

- runs the child in its own process group via os.setsid where available
- raises SubprocTimeout on timeout
- on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s
- accepts an on_pid callback so bird_x can still register child PIDs
  with last30days.register_child_pid for whole-process cleanup
- captures stdout/stderr as strings in a SubprocResult dataclass

Migrated call sites: _run_bird_search, search_handles inner worker,
search_youtube, fetch_transcript. With the helper in place, the signal
and subprocess imports became dead in both files (plus os in
youtube_yt) and went with them.

Tests: 9 new subproc tests cover success, non-zero exit, stderr capture,
timeout-raises, timeout-kills-group, missing-command, env passthrough,
PID callback, and callback-exception suppression. test_env_v3 and
test_youtube_yt patch subproc.run_with_timeout instead of the removed
bird_x.subprocess and yt-dlp subprocess.
2026-04-25 14:17:47 -07:00
Ilia Alshanetsky 2acbf8a869 perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)
1. N+1 queries in store.store_findings()
   The old loop ran one SELECT per finding to check existence, then one
   INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
   Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
   dict, then executemany() handles all inserts and updates. Query count
   stays constant regardless of batch size. Benchmark on 500 findings:
   ~30ms to ~20ms; gap widens on slower storage.

2. O(n^2) source_items dedup in fusion.weighted_rrf()
   Merging an item into an existing candidate ran any(existing.source ==
   ... for existing in candidate.source_items), linearly scanning a list
   that grew with each merge. At 40 candidates with 20 source_items each,
   fusion went quadratic. Now tracks (source, item_id) tuples in a
   per-candidate set for O(1) lookup. The source_items list itself is
   unchanged since other code iterates it.

3. Dead code removal
   - providers.GeminiClient.ground_search() and .url_context_json(): zero
     callers. Deleted.
   - render._top_comment_excerpt(): zero callers. Deleted.
   - env.is_reddit_available(): one-line wrapper around get_reddit_source.
     Callers can check get_reddit_source(config) is not None directly.
2026-04-25 14:17:17 -07:00
Ilia Alshanetsky e6b89f2644 perf: cache PreparedQuery per stream, skip double-normalize in dedupe (#282)
Scoring hot path (_normalize_score_dedupe) re-tokenized the same
ranking_query ~240x per stream: once per item for local_relevance,
plus ~5x per item across snippet windows. Query tokens are immutable
within a stream, so compute them once as relevance.PreparedQuery and
thread through signals.annotate_stream and snippet.extract_best_snippet.

dedupe._PreparedText called normalize_text twice: once in __init__ and
again via get_ngrams. Factor out _ngrams_of_normalized so the prepared
path skips the redundant pass while get_ngrams keeps its public contract.

Behavior unchanged.
2026-04-25 14:16:57 -07:00
Ilia Alshanetsky 2c2755b49c refactor(normalize): extract _join_comment_excerpts helper (#283)
_normalize_reddit, _normalize_hackernews, and _normalize_github inlined
the same 5-line comprehension to stringify and space-join the first 3
top_comments' excerpt field. Extract one helper, call it from all three.

The comment field name varies per source (Reddit/GitHub use 'excerpt',
HN uses 'text'), so it's passed as a parameter. Behavior unchanged.
2026-04-25 14:16:50 -07:00
Ilia Alshanetsky 18b5658674 chore: remove orphan test for deleted generate-synthesis-inputs script (#205)
tests/test_generate_synthesis_inputs_v3.py imported a script that no
longer exists in the repo. The test failed with FileNotFoundError on
every run.
2026-04-25 14:16:39 -07:00
Matt Van Horn 145adc9f56 Merge pull request #321 from tmchow/tmchow/review-plugin-json
chore: align plugin manifests, add Codex AGENTS.md
2026-04-25 12:34:12 -07:00
Trevin Chow b100caf2df fix(plugin): restore marketplace plugin version
`tests/test_plugin_contract.py::test_versions_match_across_manifests`
enforces that every version-bearing surface agrees: pyproject.toml,
SKILL.md, both plugin.json files, AND the marketplace plugin entry.
The Claude Code spec says plugin.json wins when both are set, but this
repo deliberately mirrors the version across all surfaces and tests it.
Restore the field at 3.1.1 to satisfy the contract.
2026-04-24 23:21:02 -07:00
Trevin Chow dc0cb9850b chore: add AGENTS.md pointing to CLAUDE.md
Codex CLI reads AGENTS.md for repo-level context the way Claude Code reads CLAUDE.md. Delegate to the existing CLAUDE.md so both harnesses share one source of project instructions.
2026-04-24 23:16:32 -07:00
Trevin Chow ceec99b24c chore(plugin): clean up plugin manifests
- Remove no-op `"hooks": {}` from .claude-plugin/plugin.json (auto-discovery from hooks/hooks.json picks up the SessionStart hook).
- Remove redundant `version` from marketplace.json plugin entry; plugin.json is the source of truth per the spec.
- Sync description / longDescription across .claude-plugin and .codex-plugin manifests so all surfaces show the same copy.
2026-04-24 23:16:29 -07:00
Dave Morin d1823a2d05 feat: add PR and issue templates for contributor workflow (#296)
Adds structured templates to help contributors submit higher-quality
PRs and issues. PR template includes testing checklist (pytest, sync.sh).
Issue templates use YAML forms for bug reports and feature requests.

Fixes #251
2026-04-24 10:49:06 -07:00
Claire Novotny 17caa0526d ci: validate plugin contract on pull requests 2026-04-24 12:05:39 -04:00
Claire Novotny f03cb866aa fix: address plugin layout review feedback 2026-04-24 11:52:48 -04:00
Claire Novotny 72495c1c14 Restructure as Codex plugin 2026-04-23 20:15:02 -04:00
Matt Van Horn 1f7e85a03f chore(release): v3.1.0 — consolidate 3.0.10-3.0.14 + OpenClaw republish prep (#314)
Release / build-and-release (push) Has been cancelled
- Bump plugin.json to 3.1.0
- CHANGELOG entry consolidating 3.0.10-3.0.14 dev cycle and noting OpenClaw republish
- Fix broken README link: skills/last30days/SKILL.md -> SKILL.md

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-22 21:56:07 -07:00
Matt Van Horn 949bcf8942 feat: vs mode N full passes + --competitors auto-discovery + (/Last30Days) title (#312)
* feat: vs mode runs N full passes; --competitors wraps vs with auto-discovery

Unifies vs-mode and --competitors onto one fanout architecture. A topic
containing "vs" / "versus" now runs N full pipeline.run() calls in parallel
(reverting the one-pass latency optimization that removed per-entity
depth); --competitors becomes a SKILL.md-level shortcut where the hosting
reasoning model (Claude Code, Codex, Hermes, Gemini) discovers N peers via
its own WebSearch, runs Step 0.55 per entity, and invokes the engine with
a vs-topic + --competitors-plan JSON.

Changed:
- vs-mode: N full passes in parallel via fanout (was 1 merged pass).
- --competitors: SKILL.md shortcut for vs-mode-with-discovery. Engine flag
  kept for headless/cron use. LAW 7-style stderr reframed to lead with the
  hosting-model path (use WebSearch + --competitors-plan) instead of
  BRAVE_API_KEY. Footer BRAVE/SERPER nudge suppressed when --plan or
  --competitors-plan present (hosting model already has WebSearch).

Added:
- --competitors-plan JSON flag: per-entity {x_handle, x_related, subreddits,
  github_user, github_repos, context}. Accepts inline JSON or file path.
  subrun_kwargs_for helper is the single source of truth for per-entity
  kwargs — no closure-default fallthrough from main scope.
- Per-entity save files: each entity's sub-run produces its own
  {slug}-raw.md with a single-row Resolved Entities block.
- --polymarket-keywords filter for ambiguous single-token topics.

Fixed:
- test_competitor_subrun_isolation regression suite locks in 3.0.12's
  no-leak invariant (main flags do not inherit into peer sub-runs).
- Updates test_regression.py for the new comparison-mode payload shape.

Bumps plugin.json to 3.0.13. 1,219 tests passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: comparison title attribution — (Last 30 Days) → (/Last30Days)

User feedback on 3.0.13 dogfood runs (Kanye vs Drake, Mercer Island,
Figma): the comparison-mode synthesis title should attribute to the
slash command rather than restate the date range.

Three SKILL.md occurrences updated. Pure documentation change. Bumps to
3.0.14.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:31:00 -07:00
Matt Van Horn 00d01933e0 fix: per-entity Step 0.55, LAW 7 sub-run quiet, default 2, canonical SKILL.md (#311)
Four fixes based on 2026-04-22 test-window feedback on v3.0.11 --competitors:

- Each competitor sub-run now runs Step 0.55 (X handle / subreddits /
  GitHub) via resolve.auto_resolve inside the fanout closure. Deep-copied
  config per entity prevents _auto_resolve_context leak across sub-runs.
  Resolved data stored on report.artifacts["resolved"] for the renderer.
- New internal_subrun keyword on planner.plan_query and pipeline.run
  suppresses the LAW 7 "No --plan passed" stderr for engine-internal
  fan-out only. Default path unchanged.
- Default --competitors count is now 2 (3-way total). --competitors=N
  still customizes; range 1..6.
- SKILL.md STEP 0 canonical-path self-check forces readers who loaded
  from marketplaces/ (auto-restored to origin/main, stale) to re-read
  from plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md.
  Two of three 2026-04-22 test windows hit this stale-path trap.
- New ## Resolved Entities block in render_comparison_multi shows
  per-entity handles/subs/github for debug visibility.

Bumps plugin.json to 3.0.12. 12 new tests; 1,175 total passing.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:30:08 -07:00
Matt Van Horn 5f054380c5 feat: --competitors flag for auto-discovered comparison fan-out (#308)
Pass `--competitors` on a single-entity topic and the engine auto-discovers
2-6 peer entities via web search, runs the full pipeline on each in
parallel, and returns one N-way comparison reusing the existing 9-axis
Head-to-Head scaffold. `last30days OpenAI --competitors` resolves to
Anthropic + xAI + Google Gemini; `last30days Kanye West --competitors`
resolves to Drake + Kendrick Lamar + one more peer.

- New CLI flags: --competitors, --competitors=N, --competitors-list
- New scripts/lib/competitors.py — mirrors resolve.auto_resolve pattern
  (web search + deterministic text extraction, no internal LLM)
- New scripts/lib/fanout.py — ThreadPoolExecutor orchestrator; per-entity
  failures degrade gracefully as long as >=2 entities survive
- Multi-report render in scripts/lib/render.py reuses the comparison
  scaffold for the synthesis table
- LAW 7-style stderr when no backend and no list, pointing the hosting
  reasoning model at --competitors-list
- 38 new tests across CLI parsing, discovery, fanout, and rendering

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:28:36 -07:00
Matt Van Horn ff21243517 Merge pull request #130 from chaosreload/feat/xurl-x-search
feat: add xurl CLI as alternative X search backend (official API v2 via OAuth2)
2026-04-22 18:55:22 -07:00
Matt Van Horn 4e91f4e754 fix: Step 0.55 category-peer subreddit expansion (#305)
* feat(resolve): category-peer subreddit map for Step 0.55

Introduces scripts/lib/categories.py with a curated category->peer-subs
map and wires scripts/lib/resolve.py auto_resolve() to merge peers into
the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode:
a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and
missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where
prompting techniques actually live.

Map is static, curated, ~11 categories (ai_image_generation,
ai_video_generation, ai_music_generation, ai_coding_agent,
ai_agent_framework, ai_chat_model, saas_screen_recording,
saas_productivity, prediction_markets, crypto_defi, dev_tool_cli).
First-match-wins ordering from most-specific to least-specific.
Compound-term patterns only (no bare common nouns like "image", "ai").

auto_resolve now:
- calls detect_category(topic) after _extract_subreddits
- merges peer_subs case-insensitively, caps at MAX_SUBS (10)
- preserves every WebSearch-returned sub (freshest signal)
- emits [Resolve] Matched category=<id>, adding peers: <list> on stderr
  only when peers were actually added
- returns new "category" key in the result dict for observability
- wraps classifier in try/except so failures degrade to unwidened list

Includes drive-by: test_full_resolve / test_partial_failure
searches_run expectations bumped from 3->4 / 2->3 to match the current
queries dict (subreddit + news + x_handle + github).

* feat(skill): Step 0.55 category-peer expansion and self-check

Adds Section 2a (category-peer expansion, MANDATORY for product topics)
and the Step 0.55 self-check checkpoint that fires immediately before
the Resolved block displays. Structural mirror of the engine-side
categories.py map: same categories, same peer subs, same priority
order.

The model-side path now:
- Applies category-peer expansion to the WebSearch-resolved subs on
  every product-in-a-known-category run.
- Emits the (+ <category_id> peers) annotation on the Reddit line of
  the Resolved block as the observable contract. Absence on a
  product-in-a-known-category topic is a Step 0.55 regression.
- Runs a self-check before emitting Resolved: "does the resolved list
  include at least 2 peer subs for the matched category? if not,
  widen NOW and do not run the engine yet."

Mirror of the Python map lives inside Step 0.55 as a table for the
model to pattern-match against; extrapolation to unlisted categories
is explicitly allowed. Worked example (the exact failing query)
appears below the table so reviewers can see before/after at a glance.

Both changes land inside the existing Step 0.55 block. No new
top-level section, no new LAW. LAWs 1-6 wording unchanged.

* test: end-to-end regression for GPT Image 2 failure mode

Stubs grounding.web_search to return the OpenAI-only subs that caused
the 2026-04-22 failure, then asserts that auto_resolve widens to
include the image-gen peers and emits the [Resolve] Matched
category=ai_image_generation stderr line. Covers the cap boundary
and the uncategorized-topic no-op path.

Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is
documentation-grade (not parsed by tests) and shows the pre-fix vs
post-fix Resolved block shape so reviewers can evaluate future
categories.py edits against the original bug.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-22 14:31:39 -07:00
Matt Van Horn 952a876536 feat: attribute top comments with u/ and @ handles in evidence lines (#292)
Reddit, TikTok, YouTube, Instagram, Bluesky, X and Threads top comments
now render as u/author or @handle in the evidence block, instead of the
generic "Comment (...)" label. The enrichment adapters already captured
author; only the render layer was dropping it.

Also fixes the TikTok adapter to prefer user.unique_id (the @handle) over
user.nickname (display name) so attribution round-trips to a profile URL.

Legacy "Comment (...)" shape is preserved when author is empty, [deleted],
or [removed].

Bumps to 3.0.10.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 08:23:47 -07:00
Matt Van Horn 1f23e3f980 test: skip docs/ in memory-dir-paths sweep (#291)
The regression test from #290 walks the filesystem via Path.rglob, so
docs/plans/*.md files (gitignored, created by internal planning) trip
the assertion on any dev machine that has run ce:plan in this repo.
Fresh clones and CI never see them, but local runs fail.

Adding docs to skip_dirs keeps the guard narrow to first-class source
files while letting internal planning docs reference old paths
verbatim.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-21 07:05:35 -07:00
Dave Morin 5269806a75 Make memory directory configurable (#290) 2026-04-21 07:04:38 -07:00
weichao adac4c377a feat: add xurl CLI as alternative X search backend
Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.

xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.

Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window

New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)

Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
  get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
  _search_x() priority chain
- SKILL.md: document xurl setup option
2026-04-21 07:05:20 +00:00
Matt Van Horn 3107325443 feat: inline markdown links on narrative citations (#289)
Inline markdown links on every narrative citation (@handle, r/sub,
publication, YouTube channel, TikTok/Instagram creator, Polymarket
market). Raw URL strings remain forbidden. Plain-text fallback when the
raw data has no URL for a specific source.

Commit 1 (790e5bc) added the citation rule in CITATION PRIORITY / URL
FORMATTING. Live tests showed the rule was deployed but consistently
skipped because it lived at line 1224, below the agent's chunked-read
window. Commit 2 (5864c687) hoists the rule into the VOICE CONTRACT
LAW block as LAW 8, at line 167 - inside the guaranteed-loaded top
band alongside LAWs 1-7. Same pattern that fixed v3.0.6 (invented
titles), disaster #2 (stripped bold), disaster #3 (trailing Sources),
and the 2026-04-19 Hermes evidence-dump disaster.

No Python engine changes. Rule is prompt-only; the deterministic
stats footer (LAW 5) is unchanged.

Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md
2026-04-20 09:49:03 -07:00
Matt Van Horn 1da9c601c3 Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth
fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
2026-04-19 11:09:56 -07:00
Matt Van Horn 4388fed46a fix: rewrite 'no LLM provider' stderr to stop the capability-constraint misread
PR #285 introduced the stderr warning "No --plan and no LLM provider
configured. Using deterministic fallback..." The 2026-04-19 Run 1
agent self-debug said it read that as "I don't have a key, I can't do
LLM stuff, I have to accept fallback" - which is the exact wrong
mental model. The word "provider" referred to the engine's INTERNAL
planner credentials, but the agent parsed it as "I need credentials
to plan at all."

Rewritten to say plainly: YOU are the reasoning model hosting this
skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime);
YOU ARE the planner; you do not need an API key or credentials - you
ARE the LLM. The --plan flag exists precisely so a reasoning model
generates its own plan upstream and passes it to the engine. The
deterministic fallback is the headless/cron path only.

Runtime enumeration is explicit so agents on every supported runtime
recognize themselves - this skill ships to Claude Code, Codex, Hermes,
and ~/.agents via sync.sh.

Tests: updated test_fallback_logs_warning_when_no_provider to assert
the new language (YOU ARE the planner, runtime names present) and
assert the old misleading phrasing is absent. Renamed the companion
test for clarity.
2026-04-19 10:28:48 -07:00
Matt Van Horn a7d6ef051a fix: expand entity-grounding haystack to transcripts + top comments
PR #285's entity grounding checked only title + snippet. That missed:

- YouTube videos where the entity is mentioned in transcript but not
  in title (false demotion of on-topic content)
- Reddit posts where the entity is in top comments but not in title
  (false demotion of on-topic discussion)

And it also wasn't strong enough to reliably demote items like the
2026-04-19 Nate Herk "Managed Agents" video - which had no Hermes
anywhere - because the -25 penalty on rerank_score composed to only
-15 on final_score via the 0.60 weight, and engagement bonus partially
offset that.

Two fixes:

1. _candidate_haystack() now joins title + snippet +
   metadata[transcript_snippet] + metadata[transcript_highlights] +
   metadata[top_comments][*].excerpt/text + metadata[comment_insights].
   Catches entity mentions wherever they actually live. Guarded with
   isinstance checks so malformed metadata doesn't raise.
2. ENTITY_MISS_FINAL_PENALTY (20.0) applied directly in _final_score
   when candidate.explanation contains "entity-miss". This lands the
   full penalty weight on the composite signal that cluster-scoring
   consumes, instead of being diluted by the rerank_score weight.
   Combined effect: entity-miss gap grows from ~15 to ~35 points.

Tests: 8 new scenarios covering transcript match, transcript highlight
match, top-comment match, comment-insight match, empty-text skip,
no-primary-entity no-op, and the dual-penalty composition check.
2026-04-19 10:28:36 -07:00
Matt Van Horn b7df5ecd2d fix: emit user-visible DEGRADED RUN WARNING on bare named-entity calls
The stderr [Planner] warning from PR #285 doesn't reach the user because
Claude and other reasoning agents hide stderr from their synthesis. The
2026-04-19 Hermes Agent Use Cases Run 1 produced source=deterministic
and the user never saw it.

Adds a user-visible stdout block that the model's LAW 5 pass-through
contract forces into the response. Fires only when plan_source is
deterministic AND no pre-research flags were passed AND the topic is
pre-research-eligible (named entity). Cron jobs on abstract topics
don't trigger it.

Position: BEFORE the EVIDENCE FOR SYNTHESIS envelope so the model sees
it as the first non-badge content. Wrapped in a new USER-VISIBLE BANNER
envelope matching the EVIDENCE/PASS-THROUGH envelope pattern from Unit 1
of PR #285.

Runtime-agnostic language: explicitly enumerates Claude Code, Codex,
Hermes, Gemini so the hosting reasoning model recognizes itself
regardless of runtime.

pipeline.py now persists plan_source to report.artifacts so the
renderer can consume it. Adds 7 tests covering fire conditions,
suppression conditions (external/llm plan source, flags present,
abstract topic), and correct position relative to the evidence envelope.
2026-04-19 10:28:21 -07:00
Matt Van Horn a0d61b0dc6 fix: add LAW 7 - YOU ARE the planner, --plan mandatory on named entities
Run 1 of /last30days Hermes Agent use cases on 2026-04-19 called the engine
bare despite SKILL.md already having a detailed Step 0.75 (YOU are the
planner) and a PRECONDITION GATE requiring --plan. Those lived at lines
647 and 729 - the model didn't reach them before invoking Bash.

LAW 7 hoists the rule into the OUTPUT CONTRACT block at the top (same
placement pattern as LAW 6), so it is the first thing the model reads.
Runtime-agnostic language: Claude Code, Codex, Hermes, Gemini, or any
agent runtime. Named failure mode with the misread diagnosis: "provider"
in engine messages refers to the engine's INTERNAL planner credentials,
NOT a prerequisite the caller needs - if you are the hosting reasoning
model, YOU are the provider.

Concrete self-check: re-read pending Bash command; if no --plan and topic
is a named entity, STOP and generate a plan.
2026-04-19 10:28:08 -07:00
Matt Van Horn 5f218aaac5 fix: always log planner subqueries to stderr
The prior pipeline.py only logged the planner outcome when an external
--plan was passed ("[Planner] Using external plan (N subqueries)").
The internal LLM planner and the deterministic fallback ran silently,
so retrieval-breadth failures were invisible without --debug.

After plan finalization, emit a unified trace:

  [Planner] Plan: intent=X, freshness=Y, cluster_mode=Z, subqueries=N, source=external|llm|deterministic
  [Planner]   sq1 label=... search="..." sources=[...]
  [Planner]   sq2 ...

Stderr only; does not touch the user-facing stdout synthesis. The
source= annotation distinguishes --plan (external), provider-backed
(llm), and deterministic paths — so when the 2026-04-19 Hermes Agent
Use Cases failure mode recurs, the trace tells the user which path ran
and what subqueries it produced.

Tests: added test_planner_trace_always_fires_on_mock_run which captures
stderr on a mock pipeline run and asserts the summary + per-subquery
lines appear.
2026-04-19 09:24:52 -07:00
Matt Van Horn a709d66e2a fix: demote reranker candidates that miss the primary entity
The 2026-04-19 Hermes Agent Use Cases run had a Nate Herk YouTube video
titled "I Tested Claude's New Managed Agents" score 51 and rank #2
with zero Hermes content. The reranker had intent-specific scoring hints
but no entity-grounding check, so topic-vicinity matches (one offhand
OpenClaw mention) drifted to the top.

Add _primary_entity(topic) that strips intent-modifier suffixes ("use
cases", "workflows", etc.) so "Hermes Agent use cases" yields
primary_entity="Hermes Agent". Pass the entity through to both the LLM
and fallback scoring paths.

Fallback path: if primary_entity is not found (case-insensitive) in
title + snippet, subtract ENTITY_MISS_PENALTY (25 pts). Skip the
demotion for candidates with no text at all (image-only TikToks etc.)
to avoid false negatives on thin-text sources.

LLM path: add a "Primary entity grounding" hint to _build_prompt when
primary_entity is non-empty. Instructs the LLM to score candidates
without the entity at <=30.

Tests: 24 rerank tests pass, including 8 new entity-grounding tests.
2026-04-19 09:24:43 -07:00
Matt Van Horn 4d9f29d2ed fix: broaden planner retrieval and fix deterministic fallback defaults
Topics with suffixes like "use cases", "workflows", "review",
"examples" were previously echoed near-verbatim into search_query,
returning near-zero matches because nobody posts the literal phrase
(2026-04-19 Hermes Agent Use Cases failure).

Unit 2 — planner breadth:

1. Planner prompt rule: STRIP intent-modifier phrases from search_query
   (keep them in ranking_query). Paraphrase across 4-5 subqueries that
   each express the intent differently.
2. Planner prompt rule: quote only multi-word proper nouns like
   "Hermes Agent", not the user's full topic.
3. Raise _max_subqueries cap from 3 to 5 for how_to / opinion / product /
   breaking_news / prediction. Comparison stays at 4; factual / concept
   stay at 2 unless the topic carries an intent modifier.
4. Deterministic fallback: when intent is non-{comparison,prediction}
   and topic contains an intent modifier, append 3 paraphrased
   subqueries (workflows, production, experience).

Unit 3 — deterministic fallback defaults:

5. _infer_intent default changed from "breaking_news" to "concept".
   Prior default forced strict_recent freshness on unclassified topics,
   biasing against older relevant material. Recency-signal regexes
   ("trending", "this week", etc.) added above the default so genuinely
   time-sensitive topics still classify correctly.
6. _keyword_query now quotes only title-cased multi-word proper nouns
   ("Hermes Agent", "Claude Code"), not the user's full typed topic.
   Hyphenated compounds and lowercase terms are left as bare keywords
   so platform tokenizers broaden rather than narrow retrieval.
7. New stderr warning when plan_query runs with no --plan and no LLM
   provider: surfaces that the deterministic fallback path is weaker
   than the --plan-from-Claude-Code path, so callers know to generate
   and pass a plan.

Tests: 37 planner tests pass, including 11 intent-modifier and 7
fallback-defaults tests.
2026-04-19 09:24:30 -07:00
Matt Van Horn 52fb0e50cb fix: scope pass-through to footer only, add LAW 6 against raw cluster dumps
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.

Split render_compact stdout into two bounded blocks:

- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
  Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
  Emit verbatim per LAW 5.

Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.

Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
2026-04-19 09:23:55 -07:00
Matt Van Horn f635f78e4a Merge pull request #281 from mvanhorn/docs/v3.0.9-release-notes
Release / build-and-release (push) Has been cancelled
docs: v3.0.9 release notes - The Self-Debug Release
2026-04-18 14:12:40 -07:00
Matt Van Horn a070a584a4 docs: v3.0.9 release notes - The Self-Debug Release
Adds docs/releases/v3.0.9.md as the GitHub Release body and appends
the matching CHANGELOG.md entry.

Covers what shipped in v3.0.9 (Class 1 refuse-gate, LAW 1 WebSearch
precedence, END-boundary, stale SKILL.md deletion) plus the community
contributions that landed across 3.0.1-3.0.8 that had never been
announced (TikTok + YouTube top comments, Hermes support, multi-key
rotation, cross-platform fixes, HTTP layer consolidation, eval
fixtures).

Contributors credited: @j-sperling, @stephenmcconnachie, @zaydiscold,
@iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma,
@BryanTegomoh, @uppinote20, @zerone0x, @thinkun, @thomasmktong,
@fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli,
@Cody-Coyote.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 14:12:20 -07:00
Matt Van Horn 8e18d0142c Merge pull request #280 from mvanhorn/fix/v3.0.9-engine-refuse-stale-skillmd
fix: v3.0.9 - engine refuses Class 1 keyword traps, delete stale SKILL.md files, reinforce LAW 1 over WebSearch
2026-04-18 13:40:37 -07:00
Matt Van Horn e8105df4fd fix: v3.0.9 - engine refuses Class 1 keyword traps, delete stale SKILL.md files, reinforce LAW 1 over WebSearch
Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:

1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
   Birthday-gift failure mode becomes structurally impossible - the pipeline
   never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
   pointing the model to ask for hobbies/relationship/budget. Escape hatch:
   LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.

2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
   snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
   Peter Steinberger's self-debug named the first file as the one it read
   instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
   Sync script simplified: Hermes now always uses main SKILL.md.

3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
   with pass-through instruction. The model had the canonical body in its
   buffer on the Peter run and discarded it; the boundary makes pass-through
   the path of least resistance.

4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
   tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
   section") that caused Peter's trailing Sources leak. No more ambiguity
   at synthesis time.

Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.

Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.

Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 13:30:33 -07:00
Matt Van Horn 361e9d6c13 fix: v3.0.8 - SKILL.md was too big and LAWs too deep - move to top + engine emits badge (#279)
Three independent Opus 4.7 self-debugs on 2026-04-18 converged on the same
root cause of the v3.0.6/v3.0.7 canonical-compliance regression: SKILL.md is
42,860 tokens / 1,478 lines, LAWs lived at line 1094+, every realistic reading
strategy failed to reach them before synthesis.

Unit 1 - Moved the BADGE MANDATORY block and VOICE CONTRACT LAW 1-5 (plus
the formatting-authority preface) from line ~1090 to line ~75 (right after
the SKILL CONTRACT preface, before HOW TO INVOKE THIS SKILL). Every reading
strategy now lands the LAWs in active context before synthesis.

Unit 2 - Engine now emits the badge as the first line of --emit=compact
stdout. Passing through the script output becomes the default-correct
behavior; emitting the badge no longer depends on model compliance. Reads
version from .claude-plugin/plugin.json at runtime with graceful fallback.

Unit 3 - Deleted skills/last30days/SKILL.md stub (231-line v3-spec file).
This was the wrong-file-capture hazard Ron Conway's self-debug identified:
model grabbed the first SKILL.md find surfaced and treated it as
authoritative. Only ONE SKILL.md in the plugin package now.

Diagnoses verbatim:
- Kanye thread: "I read lines 1-600 in chunks, jumped to 300-899, then
  stopped. File is 1478 lines. I never saw past ~900."
- Peter thread: "I tried Read once, hit the 25K token cap on a 42,860-token
  file, and bailed instead of chunked-reading with offset/limit. I never
  opened SKILL.md at all."
- Ron Conway thread: "I read one SKILL.md (231 lines)... the v3 spec stub.
  I never opened the operational SKILL.md sitting next to the script."

Validation: direct engine invocation confirms badge at line 1 of compact
output. Module imports clean.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 12:50:10 -07:00
Matt Van Horn 58845df312 fix: v3.0.7 - restore mandatory first-line badge + pin SKILL_ROOT + add skill-specificity anchor (#278)
Hot-fix for the public v3.0.6 0/8 regression (2026-04-18). Beta went 10/10
yesterday with the same LAW content; public went 0/8 today. The delta was
three structural anchors the port had removed or weakened.

Unit 1 - Restored MANDATORY first-line badge. Every public response now
emits "🌐 last30days v{VERSION} · synced {YYYY-MM-DD}" as line 1, blank
line, then "What I learned:" (GENERAL) or "# {TOPIC_A} vs {TOPIC_B}..."
(COMPARISON). This is the LAW 2 / LAW 4 enforcement anchor that my v3.0.6
port accidentally stripped along with the beta-specific "🧪 last30days-beta"
wording.

Unit 2 - Pinned SKILL_ROOT to the public plugin cache via
`ls -d ~/.claude/plugins/cache/last30days-skill/last30days/*/ | sort -V |
tail -1`, with a small fallback for repo/Gemini/Codex hosts. Replaces the
path-discovery loop that was landing on stale copies (~/.openclaw/,
~/.agents/, ~/.codex/) on machines with a private-repo sync history.

Unit 3 - Added a "SKILL CONTRACT" preface at the top of SKILL.md that names
the 0/8 regression as a documented failure mode and explicitly tells the
model not to treat /last30days as a generic keyword. Encodes user theory
that "/last30days-beta" sounded specific enough to trigger skill-follow
mode while "/last30days" reads as a search term and triggers improvise
mode.

Validation: all three anchors visible in the grep check for public
v3.0.7 cache. Next validation is manual re-run of the 8 failure topics
on public after shipping.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 10:55:28 -07:00
Matt Van Horn d14814a9b0 feat: release v3.0.6 - promote plans 003-009 from private beta to public (#277)
Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.

Plans bundled in this release:

- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
  + VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
  + top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
  count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
  per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
  emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
  at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
  ranking rewrite + Polymarket post-merge topic filter (engine change,
  filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
  preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
  post-synthesis Sources-block self-check

Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).

No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 10:24:14 -07:00
Matt Van Horn e9911ae2ae Merge pull request #276 from mvanhorn/feat/beta-channel-wiring
feat: wire compare.sh and CLAUDE.md for /last30days-beta channel
2026-04-17 22:44:31 -04:00
Matt Van Horn 7cee41509f feat: wire compare.sh and CLAUDE.md for /last30days-beta channel
- scripts/compare.sh now runs /last30days vs /last30days-beta (was
  /last30days vs /last30days-3:last30days-skill-private which was a stale
  private install name that no longer works)
- CLAUDE.md adds a Beta channel section pointing at mvanhorn/last30days-skill-private
  so future agent sessions discover the two-skill layout on project load

No runtime impact on /last30days. Engine code unchanged.

Plan: docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md
(plan file is gitignored per PR #259, not included in this diff)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 22:43:29 -04:00
Matt Van Horn 371f62a403 Revert "feat: make default fun level actually surface comedy (#272)" (#273)
This reverts commit bad1d312ef.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-17 08:39:56 -04:00
Matt Van Horn bad1d312ef feat: make default fun level actually surface comedy (#272)
Most users never touch FUN_LEVEL. Default medium was shipping a stats
block but rarely a Best Takes block, and when it did it was below the
cluster fold where a synthesizing model had already stopped reading.
A 2,304-upvote Reddit comment ("WHAT?! I reached my monthly limit
just reading this post") on the 2026-04-17 Opus 4.7 run sat inside
cluster 11 and never made it into synthesis. Four coordinated changes:

1. render: promote Best Takes above the cluster list so the synthesizer
   sees comedy before it anchors on cluster 1.
2. render: lower medium threshold from 70 to 55 (heuristic maxes at 80),
   drop the two-gem floor to one-gem. Default now reliably emits the
   block on typical runs.
3. rerank: score individual top_comments by upvote ratio to their parent
   thread. A 2,304-upvote comment on a 300-upvote thread now outranks a
   400-upvote comment on a 3,400-upvote thread, which is the viral-wit
   signal. Handles both the LLM scoring path and the heuristic fallback.
4. render: merge scored comment gems into Best Takes alongside candidate
   gems, sorted together. Comment lines show body + parent title +
   r/subreddit or @handle + absolute upvotes.
5. SKILL: tell the synthesizer to quote at least two Best Takes entries
   verbatim, with an example of the new comment format.

Plan: docs/plans/2026-04-17-001-feat-default-fun-surfacing-plan.md

🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 08:30:39 -04:00
Matt Van Horn 0103324701 Merge pull request #268 from zaydiscold/feat/multi-key-rotation
feat: multi-key rotation for SCRAPECREATORS_API_KEY
2026-04-16 23:48:44 -04:00
zayd f09c6850bc feat: multi-key rotation for SCRAPECREATORS_API_KEY
Support comma-separated API keys in SCRAPECREATORS_API_KEY with random
selection per run, distributing load across multiple free-tier accounts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 17:37:25 -07:00
Matt Van Horn 3499c246b8 fix: add commands/last30days.md and remove skills/last30days-nux duplicate (#267)
Release / build-and-release (push) Has been cancelled
Adds commands/last30days.md so /last30days registers as a Claude Code
slash command for plugin users. Users type /last30days and autocomplete
prefix-matches to the canonical /last30days:last30days form (same as
/ce:plan resolving to /compound-engineering:ce-plan).

Removes skills/last30days-nux/, a byte-identical duplicate of the root
SKILL.md that created confusing /last30days:last30days-nux autocomplete
entries via Claude Code's plugin namespacing. Root SKILL.md remains
the canonical skill source; natural-language skill-selector invocation
is unchanged.

Recovery for users on v3.0.4: /plugin update last30days then /reload-plugins.

Closes #239 (path-escape error was already fixed in v3.0.4 by dropping
the rogue 'skills' key; v3.0.5 adds the slash command on top).
Supersedes #257 (suggested './' -> '.' workaround is obsolete since
v3.0.4 dropped the 'skills' key entirely, matching ecosystem standard).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 15:19:42 -04:00
Matt Van Horn 53b8e33d13 fix(youtube): use url= param for ScrapeCreators comments/transcript + parse new response shape (#265)
PR #260 wired YouTube comment enrichment against
`/v1/youtube/video/comments` with `id=<video_id>`, but the endpoint
requires `url=https://www.youtube.com/watch?v=<video_id>`. Every enrich
call was returning 400 "missing_parameter: you must provide a url", so
no YouTube items ever carried `top_comments`.

The SC transcript fallback (`_sc_fetch_transcript`) had the identical
contract mistake. It was latent because `_fetch_transcript` prefers
yt-dlp and the SC path only fires when yt-dlp is missing, but it would
have failed the same way on hosts without yt-dlp installed.

Switching both callers to `url=` surfaces a second issue in the
response parser: SC returns `author` as `{"name": "@handle", ...}` and
nests like counts under `engagement.likes`, not top-level. The parser
was reading `author` as a string and missing the nested likes, so even
after the param fix every comment would land with an object-shaped
author and 0 likes.

- `_fetch_video_comments`: send `url=` on both urllib and requests branches
- `_sc_fetch_transcript`: same
- Response parser: extract `author.name` when author is a dict, read
  `engagement.likes` when top-level `likes` is absent, prefer
  `publishedTime` / `publishedTimeText` for date. Legacy string-author
  and top-level-likes shapes still work, so existing mocks are unchanged.

Verified live against api.scrapecreators.com: `_fetch_video_comments`
now returns fully-populated comments with real @handles and like
counts (e.g. "@JennyNicholson: ... (49000 likes, 2025-04-15)"). All
tests in youtube_yt/normalize/signals/render pass.

Plan: docs/plans/2026-04-15-002-fix-youtube-comments-scrapecreators-param-plan.md

🤖 Generated with Claude Opus 4.6 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:17:22 -04:00
Matt Van Horn 73b4bd6ac6 fix: enforce pre-research protocol + override WebSearch Sources mandate (#266)
Restore the rich synthesis output by closing three prompt-level loopholes
that let the model silently take a degraded path:

1. Research Execution precondition gate. Steps 0.55 (entity resolution)
   and 0.75 (query planner) are now non-skippable on WebSearch platforms.
   --emit md is banned as a primary user-facing flow; --emit=compact with
   --plan is mandatory. OpenClaw --auto-resolve fallback preserved.

2. WebSearch "Sources:" mandate override. The WebSearch tool description
   contains a CRITICAL/MUST mandate to append a Sources section. That is
   explicitly superseded inside /last30days with matched-register
   CRITICAL/MANDATORY override language and a BAD/GOOD example. The
   existing web-source line is the citation; nothing appends below the
   invitation.

3. Pre-present self-check. Before displaying, the model verifies bold
   per-paragraph headlines, per-source emoji stats, quoted highlights,
   Polymarket block, coverage footer, and (critically) no trailing
   Sources block. One regeneration permitted if checks fail.

Also adds explicit MANDATORY language to the "What I learned" template
requiring bold headline phrases on every narrative paragraph.

Root cause: same-session A/B on 2026-04-15 between /last30days kanye
west (rich output, ran Steps 0.55 + 0.75, --emit=compact --plan) and
/last30days hermes ai (bland output, skipped both, --emit md) showed
the template was fine -- the model was lazily taking a shortcut SKILL.md
tolerated. No engine, render.py, or contributor PR was the cause.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:15:29 -04:00
Matt Van Horn a2850e3d19 fix: drop plugin.json 'skills' key to clear path-escape error on v2.1.109 (#264)
Release / build-and-release (push) Has been cancelled
plugin.json has declared "skills": ["./"] unchanged since v2.1.0. That
value used to work on older Claude Code but current versions reject it
with: Path escapes plugin directory: ./ (skills). The error surfaces
on fresh /doctor runs even after v3.0.3 restored the archive contents.

Fix: omit the "skills" key entirely. Every other plugin in the Claude
Code marketplace ecosystem (compound-engineering, coding-tutor, codex,
esper, 15+ Anthropic official plugins) omits this key and the loader
auto-discovers skills/*/SKILL.md. Matching that pattern clears the
path-escape error on v2.1.109+ and remains compatible with older
Claude Code versions where the default-discovery path was already the
working code path.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 11:40:15 -04:00
Matt Van Horn 9c1e253dcc fix(build): strip skills/ and .claude-plugin/ from .skill bundle (#263)
v3.0.3's fix (#262) restored skills/ and .claude-plugin/ to the git
archive, which Claude Code needs for /plugin install. But
scripts/build-skill.sh uses the same archive to produce the claude.ai
.skill bundle, which must contain exactly one root SKILL.md and stay
under the 200-file cap.

Fix: after git archive, 'zip -d' strips both directories from the
.skill bundle. git archive output is unchanged (Claude Code still
gets the full tarball on /plugin install).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:31:51 -04:00
Matt Van Horn f4a3cc104b fix: restore skills/ and .claude-plugin/ in plugin install tarball (#262)
Release / build-and-release (push) Has been cancelled
v3.0.1 added .gitattributes rules that excluded both directories from
git archive output, shrinking the claude.ai .skill bundle. But Claude
Code's /plugin install fetches the SAME archive, so users installing
v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no
skill files. Install appeared successful but the plugin was a useless
empty shell.

Proof:
  git archive v3.0.0 | grep 'skills/|\.claude-plugin/' | wc -l  # 8
  git archive v3.0.1 | grep 'skills/|\.claude-plugin/' | wc -l  # 0
  git archive v3.0.2 | grep 'skills/|\.claude-plugin/' | wc -l  # 0

No issue reports yet because:
 - Cached pre-v3.0.1 installs keep working (it's the new-install path
   that's broken)
 - The breakage is under 24 hours old
 - Users invoking the skill via natural language go through
   skill-selector rather than /last30days slash command

Also reverts v3.0.2's "skills": ["skills"] back to "./", the value
that shipped in every tag from v2.1.0 through v3.0.0. That change was
a misdiagnosis; the manifest wasn't in the tarball anyway so it had
no effect on user-visible installs.

Archive file count after fix: 97 (cap is 200, plenty of room).
Follow-up: move claude.ai-specific bundle exclusions into
scripts/build-skill.sh where they belong, rather than .gitattributes
which cannot distinguish between the two distribution channels.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:25:45 -04:00
Matt Van Horn a220632186 fix: restore /last30days slash command on Claude Code v2.1.105+ (#261)
Release / build-and-release (push) Has been cancelled
Two regressions were silently breaking /last30days for every user:

1. plugin.json declared "skills": ["./"], which newer Claude Code
   rejects with "Path escapes plugin directory: ./ (skills)". The
   skill loader refused to register the command, so /last30days
   returned "Unknown command" even though /plugin list showed the
   plugin as installed. Fix: "skills": ["skills"] so the loader
   scans the real subdirectory.

2. marketplace.json pinned "version": "3.0.0" while plugin.json
   advertised "3.0.1". The /plugin resolver used the marketplace
   version and could install a phantom user-scope copy at a stale
   SHA alongside the correct project-scope install, creating
   duplicate skill-name collisions. Both manifests now agree on
   3.0.2.

Prior attempt: commit 93fbed2 fixed (1) before but got reverted.
This lands both fixes together in a tagged release so users can
/plugin update to recover.

Recovery for affected users is in CHANGELOG.md under 3.0.2.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:40:09 -04:00
Matt Van Horn 082efe03e3 feat: surface YouTube + TikTok top comments alongside Reddit (#260)
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:26:06 -04:00
Matt Van Horn 242e38ef56 chore: ignore docs/plans/ and untrack existing plan files (#259)
Internal ce:plan output shouldn't ship on the public repo.
Adds docs/plans/ to .gitignore and removes the two already-tracked
plan files from the index. Working copies stay local for reference.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 07:48:47 -04:00
Matt Van Horn c12dd3adbf docs: mark plan 002 Units 1-4 complete; 5-10 remain 2026-04-14 17:46:01 -04:00
Matt Van Horn c5b03adffc Merge pull request #244 from mvanhorn/feat/claudeai-distribution
Release / build-and-release (push) Has been cancelled
feat: claude.ai distribution push (Units 1-4 of plan 002)
2026-04-14 17:44:58 -04:00
Matt Van Horn 38a1c27e2e chore: exclude .github/ from skill archive (CI workflows, not runtime) 2026-04-14 17:44:18 -04:00
Matt Van Horn ed80797564 docs: add plan 2026-04-14-002 for claude.ai distribution push 2026-04-14 17:44:00 -04:00
Matt Van Horn 68c3420f9f docs: promote claude.ai to first-class install path with direct download link
- install matrix now leads with claude.ai (widest audience, one-click path)
- direct download link to GitHub release's 'latest' asset URL
- 3-step UI walkthrough with link to Settings > Capabilities > Skills
- Claude Code / OpenClaw / Gemini / manual paths still documented, collapsed
- removes the bash scripts/build-skill.sh requirement from end-user flow
2026-04-14 17:43:32 -04:00
Matt Van Horn 12167ee19e feat(skill): tune description and argument-hint for Claude skill-selector quality
- description leads with imperative 'Research' + 'what people actually say' (strong trigger signal for community/social-research prompts)
- argument-hint shows 3 concrete user phrasings instead of marketing copy
- 176 chars, well under Anthropic's 200-char cap
- preserves all source coverage (Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, web)

Per ecosystem research (April 2026), trigger description quality is the single
biggest lever separating 500-install skills from 350k-install skills.
2026-04-14 17:42:54 -04:00
Matt Van Horn 21b8e5c6d3 ci: auto-build .skill artifact on tag push and attach to GitHub release 2026-04-14 17:42:15 -04:00
Matt Van Horn 1157ea8afe docs: mark plan 2026-04-14-001 as completed 2026-04-14 12:24:16 -04:00
Matt Van Horn 9f3be8bbda Merge pull request #242 from mvanhorn/fix/skill-upload-200-file-limit
fix: skill upload 200-file cap + packaging hygiene (3.0.1)
2026-04-14 12:24:03 -04:00
Matt Van Horn beb54e9e9d fix: sync version references in SKILL.md body and sync.sh cache path 2026-04-14 12:22:51 -04:00
Matt Van Horn 8d8ca68781 chore: bump version to 3.0.1 + changelog entry
Atomic bump across all four manifests:
- SKILL.md (root)
- skills/last30days/SKILL.md (internal spec)
- .claude-plugin/plugin.json
- gemini-extension.json

CHANGELOG entry documents the skill-upload packaging fix, vendor/ removal,
legacy plans/ removal, and the new scripts/build-skill.sh builder.
2026-04-14 12:21:24 -04:00
Matt Van Horn 0949b870e0 fix(skill): trim description to 167 chars (was 228, Anthropic caps at 200) 2026-04-14 12:20:20 -04:00
Matt Van Horn 4b07ba02a6 docs: document .skill upload path via scripts/build-skill.sh 2026-04-14 12:19:49 -04:00
Matt Van Horn 039fc89874 feat: add scripts/build-skill.sh to produce claude.ai-upload-ready .skill
Wraps git archive with --prefix=last30days/ so the zip contains a single
top-level skill folder matching SKILL.md's name: frontmatter. Enforces:

- refuses to build with a dirty working tree (prevents shipping untracked changes)
- fails if zip exceeds 200 files (claude.ai's empirical upload cap)
- fails if zip contains more than one SKILL.md (avoids name: confusion)

Output at dist/last30days.skill (gitignored).
2026-04-14 12:19:28 -04:00
Matt Van Horn 2b506e90f5 chore: add .gitattributes to exclude non-runtime files from git archive 2026-04-14 12:18:54 -04:00
Matt Van Horn deb9f33437 chore: remove legacy plans/ directory (superseded by docs/plans/)
Both plans describe work that was already shipped:
- feat-add-websearch-source.md - websearch is in the v3 pipeline (scripts/lib/perplexity.py etc)
- fix-strict-date-filtering.md - date filtering is enforced in scripts/lib/dates.py

New planning goes in docs/plans/ following the ce:plan convention.
2026-04-14 12:18:19 -04:00
Matt Van Horn cb88bd2eed chore: remove unused root vendor/ directory (215 files from PR #48)
Root vendor/package/ was an accidentally committed extracted npm tarball
(steipete-bird-0.8.0). Zero importers: the real vendored X client lives
at scripts/lib/vendor/bird-search/, referenced by scripts/lib/bird_x.py
and tests/test_bird_x.py.

Removes 215 files + 1 .tgz, dropping repo from 406 to 191 files and
clearing the claude.ai skill-upload 200-file cap.

Adds /vendor/ to .gitignore (leading slash so scripts/lib/vendor/ is unaffected).
2026-04-14 12:17:58 -04:00
hnshah 23fc6c7061 fix(env): default INCLUDE_SOURCES to empty string (#223)
* fix(env): default INCLUDE_SOURCES to empty string

* test(env): patch resolved config path in include sources test
2026-04-14 07:48:53 -04:00
Ilia Alshanetsky 9dd3f21476 refactor: consolidate _sc_headers into http.scrapecreators_headers (#209)
Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.

Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.

Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
2026-04-14 07:43:56 -04:00
Matt Van Horn e395c1d57f Merge pull request #208 from iliaal/fix/date-parsing
fix(github): reject garbage in _parse_date; consolidate date parsing
2026-04-13 22:21:35 -04:00
Matt Van Horn 33502d2a07 Merge pull request #207 from iliaal/refactor/reddit-http-helper
refactor(reddit): migrate to http.get(params=...) helper
2026-04-13 22:18:49 -04:00
Matt Van Horn bdc71cfd07 Merge pull request #227 from Chelebii/fix/windows-bird-x-runtime
fix(windows): stabilize bundled Bird X search
2026-04-13 22:15:21 -04:00
Matt Van Horn 65be6196c1 Merge pull request #217 from Gujiassh/fix/sync-version-consistency
fix: align v3 skill version metadata and sync target
2026-04-13 17:55:34 -04:00
Matt Van Horn 7dc530b4c9 Merge pull request #224 from hnshah/hnshah-gemini-install-doc
docs: add Gemini CLI install note and workaround
2026-04-13 17:55:24 -04:00
Matt Van Horn b159f8b1ff Merge pull request #216 from george231224/fix/check-perms-stat-linux
fix: use GNU stat first in check_perms (Linux false-warn)
2026-04-13 17:55:21 -04:00
Matt Van Horn cff005b038 Merge pull request #225 from Gujiassh/fix/save-output-utf8
fix(cli): Write saved output using UTF-8 encoding
2026-04-13 17:55:18 -04:00
Matt Van Horn 460565c107 Merge pull request #228 from stephenmcconnachie/add-hermes-support
feat: add Hermes AI Agent support
2026-04-13 15:59:53 -04:00
Matt Van Horn e6493033b0 Merge pull request #229 from shalomma/fix/skill-md-version-bump
Bump SKILL.md version header from v2.9.5 to v3.0.0
2026-04-13 15:55:15 -04:00
Matt Van Horn ca00cacf83 Merge pull request #230 from BryanTegomoh/fix/days-alias-backcompat
fix(cli): restore --days alias compatibility
2026-04-13 15:55:06 -04:00
Matt Van Horn b982ed5b30 Merge pull request #232 from j-sperling/j-sperling/chore/gitignore-dev-artifacts
chore: gitignore dev artifacts (.venv, .coverage, htmlcov, .memsearch)
2026-04-13 15:54:17 -04:00
Matt Van Horn a9d13d695a Merge pull request #233 from j-sperling/j-sperling/feat/eval-topics-fixture
feat: add eval_topics.json fixture for offline quality evaluation
2026-04-13 15:53:58 -04:00
Matt Van Horn 877706da4d Merge pull request #234 from j-sperling/j-sperling/fix/bird-x-engagement-validation
fix(bird_x): skip all-None engagement dicts
2026-04-13 15:52:44 -04:00
Jeffrey Sperling 1a6d8d07d0 fix(bird_x): skip all-None engagement dicts
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}.  This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available."  Return None
instead when every engagement field is missing.
2026-04-13 11:54:49 -07:00
Jeffrey Sperling 3bc12cdc57 feat: add eval_topics.json fixture for offline quality evaluation
evaluate_search_quality.py and e2e_comparison.py both reference
fixtures/eval_topics.json with hardcoded fallbacks.  Supply the
actual fixture: 8 topics spanning all intent types, selected via
MMR dispersion across domains (tech, health, sports, finance,
consumer products).
2026-04-13 11:52:55 -07:00
Jeffrey Sperling ad59e60269 chore: gitignore dev artifacts (.venv, .coverage, htmlcov, .memsearch)
pyproject.toml declares pytest-cov as a dev dependency and configures
[tool.coverage.run], but the generated .coverage database and htmlcov/
report directory are not gitignored.  Also add .venv/ (standard Python
virtualenv) and .memsearch/ (session memory) to keep the working tree
clean for contributors.
2026-04-13 11:52:12 -07:00
Bryan Tegomoh 9d037786f2 fix(cli): restore --days alias compatibility 2026-04-13 09:18:18 -05:00
shalomma 8b67378964 Bump SKILL.md version header from v2.9.5 to v3.0.0
The SKILL.md prompt header still said v2.9.5 while pyproject.toml
and the rest of the codebase are on v3.0.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 12:31:43 +03:00
Stephen McConnachie 2b015b64ab Add Hermes AI Agent support 2026-04-12 20:06:02 +01:00
Chelebii d3972a6523 fix(windows): stabilize bundled Bird X search 2026-04-11 23:30:39 +01:00
gujishh 56cabf33c6 fix(cli): write saved output using UTF-8 encoding 2026-04-12 06:25:38 +09:00
Hiten Shah 13dcea781d docs: add Gemini CLI install note and workaround 2026-04-11 13:15:53 -07:00
Matt Van Horn 01812ec185 fix(sync): skip OpenClaw variant branch when variants/open is absent (#222)
Makes the `variants/open/` sync steps in `scripts/sync.sh` conditional on
the directory actually existing in the source tree. The script is shared
between the public and private repos of last30days-skill, but the OpenClaw
variant only lives in the private repo (it's sanitized via
`strip_for_openclaw.py` and published separately to ClawhHub). When the
script runs from a checkout of the public repo, the variants/open paths
don't exist and the unconditional `rsync` and `sync_target` calls error
out immediately.

Changes:

- `sync_target()` now only creates `variants/open/references` and rsyncs
  `variants/open/` when `$SRC/variants/open` exists.
- The trailing `sync_target "$OPENCLAW_TARGET" ...` call is now gated by
  the same check, with an explanatory skip message when the directory is
  absent.

No behavior change when running from the private repo (which has
`variants/open/`). When running from the public repo, the script now
completes its COMMON_TARGETS loop without erroring.

This also closes out the confusion from PR #211, where a contributor saw
the broken `variants/open/` reference and tried to add the variant back
to the public repo. The real fix was making the script tolerate the
absence, not recreating the directory.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 11:37:27 -04:00
Matt Van Horn 86b2b9dd69 docs(v3): drop redundant What's New list and remove stale @steipete credit (#221)
release-notes.md:
- Drop the "What's New" section entirely. It repeated the same items
  as the Headline features section above it in bulleted form, a
  holdover from the old v2.9 release notes pattern. CHANGELOG.md is
  the canonical Added/Changed/Fixed list; release notes is marketing
  copy and shouldn't duplicate it. Added a one-line pointer to
  CHANGELOG.md [3.0.0] for anyone looking for the detail.
- Rename "Credits" to "Earlier contributors" and note they are from
  the v1 and v2 lineage, so readers don't confuse them with v3
  contributors.
- Remove @steipete credit (did not actually contribute to this repo).

CHANGELOG.md [2.1.0] Credits:
- Remove @steipete credit (did not actually contribute to this repo).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:33:43 -04:00
Matt Van Horn 6e7c0ba7aa docs(v3): prep CHANGELOG and release notes for v3.0.0 (#220)
Rewrites release-notes.md from its stale v2.9.1 focus into the v3
story: intelligent pre-research as the killer feature, fun judge /
Best Takes, cross-source cluster merging, single-pass comparisons,
GitHub person-mode and project-mode, 13+ sources, ELI5 mode. Credits
@j-sperling as the v3 engine architect in the hero section and
updates the install instructions from `git clone` to the real install
paths for Claude Code, OpenClaw, and OpenAI Codex CLI.

Also extends the CHANGELOG [3.0.0] entry with a Fixed section covering
the two post-merge prep fixes that landed just before release:

- #214 resolved a duplicate `name: last30days` collision in
  skills/last30days/SKILL.md that caused strict marketplace validators
  to reject the plugin (reported by @Cody-Coyote in #204)
- #219 added the real Codex CLI integration at
  .agents/skills/last30days/SKILL.md (regular file, since Codex's
  loader skips symlinked files) plus .codex-plugin/plugin.json as the
  namespace marker (inspired by @Jah-yee in #153 and @dannyshmueli
  on X)

Bumps the [3.0.0] date from `2026-04` to `2026-04-11` to match the
actual release date, and adds @Cody-Coyote and @Jah-yee to the
[3.0.0] Contributors list.

No code changes. Pure docs prep for the v3.0.0 GitHub release.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:27:35 -04:00
Matt Van Horn 71e0492840 feat: make skill discoverable by OpenAI Codex CLI (#219)
Adds a Codex CLI skill integration by creating the two files Codex's real
loader actually reads:

- .agents/skills/last30days/SKILL.md (real file, not a symlink - Codex's
  loader skips symlinked files per codex-rs/core-skills/src/loader.rs)
- .codex-plugin/plugin.json with {"name": "last30days"} as a namespace
  marker, per codex-rs/utils/plugins/src/plugin_namespace.rs

When Codex CLI runs in a checkout of this repo, it walks .agents/skills/
from CWD up to the project root, picks up .agents/skills/last30days/SKILL.md,
and walks ancestors looking for .codex-plugin/plugin.json to resolve the
plugin namespace. The skill registers as last30days:last30days.

The SKILL.md is a verbatim copy of the root SKILL.md at this point to
avoid content drift during the rollout. A future PR can slim the Codex copy
or introduce a sync mechanism.

Verified against Codex CLI's own source by running codex exec from the
repo CWD and having it trace the loader logic.

Replaces PR #153, which used a fake $schema URL
(https://openai.com/codex/plugin.schema.json returns 404) and put a
misunderstanding of Codex's plugin manifest (Codex only reads the `name`
field - all other fields like version, description, author, skills[] are
silently ignored).

This contribution was developed with AI assistance (Codex + Claude Code).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 09:02:28 -04:00
gujishh 8b2cf41f13 fix: align v3 version metadata and sync target 2026-04-11 21:00:04 +09:00
george231224 3d57db9644 fix: use GNU stat first in check_perms so Linux doesn't false-warn
`stat -f '%Lp'` is BSD/macOS syntax. On Linux, `stat -f` prints
filesystem info (Block size / Inodes / ...) and still exits 0, so the
`||` fallback to `stat -c '%a'` never fires. That left `$perms` as
multi-line garbage, the `!= "600"` check was always true, and every
Linux SessionStart hook invocation printed a bogus warning plus the
whole `stat -f` filesystem dump.

Reorder to try GNU stat first, fall back to BSD for macOS. Verified on
Linux (cpython-3.12 / bash 5.x) — hook now emits the expected compact
Ready banner with no false warning.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 18:41:08 +08:00
Matt Van Horn 99b167d03a fix: resolve duplicate skill name causing marketplace validation failure (#204) (#214)
Two SKILL.md files declared `name: last30days` with `user-invocable: true`,
which caused strict marketplace validators to reject the plugin with "Some
plugins in this marketplace have validation errors":

- ./SKILL.md (canonical, also reachable via skills/last30days-nux/ symlink)
- ./skills/last30days/SKILL.md (v3 architecture spec, real file)

In v2.9.6, skills/last30days/SKILL.md was a symlink to ../../SKILL.md so
only one skill existed. Commit 0a9ff16 (v3.0.0) added a new real file at
skills/last30days-v3/SKILL.md, and commit 9be0780 then renamed that
directory to skills/last30days/, replacing the original symlink with a
different real file. The collision has been live since v3.0.0 shipped.

This change:
- Renames skills/last30days/SKILL.md to name: last30days-v3-spec and sets
  user-invocable: false. The file stays in place as internal architecture
  documentation, but it no longer competes with the canonical skill.
- Fixes README.md link that pointed to the deleted skills/last30days-v3/
  path (left over from the rename).
- Removes a stale variants/open/SKILL.md reference (variants/open was
  deleted in v3.0.0).

After the change, only one canonical name=last30days user-invocable=true
skill exists (the root SKILL.md, also reachable via the
skills/last30days-nux/ symlink, same inode).

Closes #204.

This contribution was developed with AI assistance (Codex).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 02:36:55 -04:00
Ilia Alshanetsky 65fcf6be65 fix(github): reject garbage in _parse_date; consolidate date parsing
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.

Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.

Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).

Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
2026-04-10 07:39:07 -04:00
Ilia Alshanetsky 9ef9d38b90 refactor(reddit): migrate to http.get(params=...) helper
Added params kwarg to http.request()/http.get() that urlencodes a dict
into the query string. None values are dropped, ints and bools are
stringified, and params append correctly if the URL already has a
query string.

Migrated reddit.py to use this helper for all three ScrapeCreators
call sites (global search, subreddit search, post comments). Deleted
the try/import requests/except ImportError fallback and the paired
if not _requests: / else: branches. Six new http tests cover the
params-encoding behavior.

Net: reddit.py -70 lines. Behavior is identical - the existing http.py
urllib implementation already had retry logic, 429 handling, and
HTTPError types that are strictly better than the ad-hoc requests
branches we deleted.

99 reddit tests pass. Live smoke test on a real ScrapeCreators run
returned 12 threads with the same engagement data as before.
2026-04-10 07:25:26 -04:00
Matt Van Horn 86e1d77ad7 Merge pull request #186 from pejmanjohn/contrib/mvanhorn-last30days-skill-99-fix-plugin-directory-name
fix: rename skills/last30days-v3 directory so plugin install resolves correctly
2026-04-09 23:18:18 -07:00
Matt Van Horn ac692e85bc Merge pull request #201 from tmchow/fix/90-store-sql-column-whitelist
fix(store): validate updatable columns in update_run and update_finding
2026-04-09 23:15:38 -07:00
Matt Van Horn 5196bf68a7 Merge pull request #202 from tmchow/fix/92-bluesky-token-refresh
fix(bluesky): add token expiry handling to session cache
2026-04-09 23:15:36 -07:00
Matt Van Horn 6bbb4300f5 Merge pull request #203 from tmchow/fix/marketplace-plugin-name
fix: correct plugin name and version in marketplace.json
2026-04-09 23:15:33 -07:00
Trevin Chow 62584631bb fix: correct plugin name and version in marketplace.json
marketplace.json had stale v3 rename artifacts: plugin name was
"last30days-3" (should be "last30days" to match plugin.json) and
version was "3.0.0-alpha" (should be "3.0.0" to match the stable
release).
2026-04-09 22:05:25 -07:00
Trevin Chow 43c8d6c29c fix(bluesky): add token expiry handling to session cache
The module-level _cached_token was set once and never refreshed. AT
Protocol tokens expire after ~2 hours, causing silent 401 errors in
long-running watchlist cron sessions. Adds a 90-minute expiry check
using time.monotonic() and logs re-authentication.

Fixes #92
2026-04-09 21:44:33 -07:00
Trevin Chow ccd2a4065d fix(store): validate updatable columns in update_run and update_finding
Add column whitelists to prevent SQL injection via kwargs keys in
dynamic UPDATE queries. Values were already parameterized but column
names were string-interpolated directly from kwargs.

Fixes #90
2026-04-09 21:40:53 -07:00
Pejman Pour-Moezzi bb7c956db5 fix: drop stale last30days-v3 argument hint 2026-04-09 21:37:36 -07:00
Pejman Pour-Moezzi 9be0780c46 fix: rename skills/last30days-v3 directory so plugin install resolves correctly 2026-04-09 21:37:00 -07:00
Matt Van Horn 341da37218 Merge pull request #182 from ziperlee/codex/bluesky-refresh-token
fix: refresh expired bluesky sessions
2026-04-09 21:20:58 -07:00
Matt Van Horn 4b3458776b Merge pull request #191 from kriptoburak/feat/add-xquik-source
feat: add Xquik as X/Twitter search source
2026-04-09 21:19:18 -07:00
Matt Van Horn 6436290cc1 Merge pull request #183 from zl190/claim-contributor-entry
docs: claim @zl190 contributor entry
2026-04-09 21:18:04 -07:00
Matt Van Horn 5316d92fc6 Merge pull request #185 from hnshah/hnshah-claim-contrib
docs: claim hnshah contributor entry
2026-04-09 21:18:02 -07:00
Matt Van Horn 1f350d2b30 Merge pull request #189 from pejmanjohn/contrib/mvanhorn-last30days-skill-102-claim-contributors-entry
docs: claim @pejmanjohn CONTRIBUTORS.md entry
2026-04-09 21:17:04 -07:00
Matt Van Horn 474cc4ad7a Merge pull request #188 from pejmanjohn/contrib/mvanhorn-last30days-skill-101-readme-update-command
docs: add plugin update command to README
2026-04-09 21:17:01 -07:00
Matt Van Horn 80892d31c6 Merge pull request #187 from pejmanjohn/contrib/mvanhorn-last30days-skill-100-fix-nux-argument-hint
fix: update argument-hint in root SKILL.md from last30days-3 to last30days
2026-04-09 21:16:59 -07:00
Matt Van Horn 5c4383d661 Merge pull request #192 from tmchow/fix/remove-orphaned-exa-test
fix(tests): remove orphaned test_exa_search.py
2026-04-09 21:06:44 -07:00
Matt Van Horn 101c4724f4 Merge pull request #193 from tmchow/fix/evaluator-test-env-isolation
fix(tests): isolate env in resolve_google_judge_api_key test
2026-04-09 21:06:42 -07:00
Matt Van Horn c09ca59747 Merge pull request #194 from tmchow/fix/bump-version-metadata-v3
fix: bump gemini-extension and v3 skill version to 3.0.0
2026-04-09 21:06:40 -07:00
Matt Van Horn dad97f1b05 Merge pull request #198 from iliaal/perf/pipeline-optimizations
perf: optimize dedup, parallelize handle searches and enrichment
2026-04-09 21:00:37 -07:00
Ilia Alshanetsky eef3547c37 perf: optimize dedup, parallelize handle searches and enrichment
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).

Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.

Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.

Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.

End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
2026-04-09 19:00:30 -04:00
Ilia Alshanetsky 252c8222f1 feat: add WSL2 Windows Firefox cookie extraction for X auth
On WSL2, native Linux Firefox typically has no x.com cookies since users
browse in Windows. Chromium browsers (Edge, Chrome, Brave) encrypt cookies
with DPAPI/app-bound encryption, making them inaccessible without admin
privileges. Windows Firefox stores cookies unencrypted in SQLite, readable
directly through the /mnt/c mount.

The cookie extractor now detects WSL2 via /proc/version, locates Windows
Firefox profiles under /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox,
and falls back to them when Linux Firefox yields no results. Reports
source as "firefox-wsl" to distinguish from native.

Also fixes profile resolution priority: Install* sections (Firefox >= 67)
now take precedence over the legacy Default=1 flag, which could select a
stale profile on multi-profile installations.
2026-04-09 18:36:02 -04:00
Trevin Chow 6a4071a9fd fix: bump gemini-extension and v3 skill version to 3.0.0
gemini-extension.json still referenced v2.9.5 and
skills/last30days-v3/SKILL.md still said 3.0.0-alpha.
Both now match pyproject.toml's canonical 3.0.0 version.

Addresses items from #190. Structural decisions (SKILL.md
consolidation, SKILL-original.md cleanup) left for maintainer.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 13:00:08 -07:00
Trevin Chow 5896c9582b fix(tests): isolate env in resolve_google_judge_api_key test
The second assertion in test_resolve_google_judge_api_key_prefers_google_key
ran outside the mock.patch.dict context. When GOOGLE_API_KEY or GEMINI_API_KEY
is set in the real environment, os.environ takes precedence over the config
dict fallback and the test fails.

Wrap the assertion in its own mock.patch.dict scope that clears the three
relevant env vars so the test passes regardless of the developer's env.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 12:56:34 -07:00
Trevin Chow 718fe9547b fix(tests): remove orphaned test_exa_search.py
lib/exa_search.py was removed during the v3 refactor but
tests/test_exa_search.py still imports from it. This causes
an ImportError that blocks pytest -x from running any tests.

This contribution was developed with AI assistance (Claude Code).
2026-04-09 12:52:25 -07:00
Burak Bayır 6b3de9170e feat: add Xquik as X/Twitter search source
Add Xquik (xquik.com) as a new X/Twitter search source that uses a REST
API with full engagement metrics (likes, retweets, replies, quotes,
views, bookmarks). Uses stdlib urllib only -- no new dependencies.

- scripts/lib/xquik.py: source module with search, parse, query expansion
- tests/test_xquik.py: 32 unit tests covering all functions
- env.py: XQUIK_API_KEY config and availability check
- pipeline.py: source registration and retrieve dispatch
- normalize.py: reuses _normalize_x (same item format as Bird)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 22:44:10 +03:00
Pejman Pour-Moezzi ca4ffb9633 docs: claim @pejmanjohn CONTRIBUTORS.md entry 2026-04-09 12:20:18 -07:00
Pejman Pour-Moezzi 79a5c3ea94 docs: add plugin update command to README 2026-04-09 12:16:25 -07:00
Pejman Pour-Moezzi a41bf5d8e7 fix: update argument-hint in root SKILL.md from last30days-3 to last30days 2026-04-09 12:12:16 -07:00
Hiten Shah 319ec796bd docs: claim hnshah contributor entry 2026-04-09 11:26:51 -07:00
jason-zl190 86df1e7d4b docs: claim @zl190 contributor entry
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 04:04:44 +10:00
Matt Van Horn 45f596ca0c docs: add v3 community contributors section to release notes 2026-04-09 10:56:21 -07:00
Matt Van Horn 4fab23a357 feat: add CONTRIBUTORS.md crediting v3 community inspiration 2026-04-09 10:55:03 -07:00
ziperlee 9405aa3fb4 fix: refresh expired bluesky sessions 2026-04-09 23:03:47 +08:00
ziperlee 2020156591 fix: write briefing files as utf-8 2026-04-09 23:01:17 +08:00
Matt Van Horn b6d97a571d Merge pull request #179 from pejmanjohn/contrib/mvanhorn-last30days-skill-46-prompt-injection-hardening
fix: harden rerank prompts and assistant-facing digests against scraped prompt injection
2026-04-09 06:29:20 -07:00
Matt Van Horn 0e5faa7a82 Merge pull request #175 from ziperlee/codex/readme-v3-alignment
docs: align README with v3 runtime
2026-04-09 06:25:00 -07:00
Matt Van Horn 9de2398106 Merge pull request #173 from pejmanjohn/contrib/mvanhorn-last30days-skill-44-bird-sweet-cookie-runtime
last30days: lazy-load sweet-cookie so vendored Bird works on fresh installs
2026-04-09 06:20:39 -07:00
Matt Van Horn fd6ec55f07 Merge pull request #180 from tmchow/fix/bird-x-list-response
fix(bird_x): normalize list responses from Bird search
2026-04-09 06:10:34 -07:00
Trevin Chow 65399eb4eb fix(bird_x): normalize list responses from Bird search to dict format
When Bird's JSON response is a raw array instead of an object,
json.loads returns a list. All callers use .get('items') which raises
AttributeError on lists. Wrap list responses in {"items": parsed} so
callers always receive a dict.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 01:26:14 -07:00
Pejman Pour-Moezzi b7d1a38ff6 Fix Bird cookie helper lazy-loading 2026-04-08 19:30:39 -07:00
Pejman Pour-Moezzi 89e6ee29fd last30days: harden rerank and render prompts 2026-04-08 19:23:25 -07:00
Matt Van Horn dedf615114 Merge PR #174: last30days: require Python 3.12+ in OpenClaw setup and CLI entrypoints
last30days: require Python 3.12+ in OpenClaw setup and CLI entrypoints
2026-04-08 19:05:47 -07:00
zipee f926d6507a docs: point skill metadata at public repo 2026-04-09 08:40:14 +08:00
zipee bcc4694fa9 docs: point skill metadata at public repo 2026-04-09 08:40:07 +08:00
zipee f09fe202e8 docs: align README with v3 runtime 2026-04-09 08:35:56 +08:00
Pejman Pour-Moezzi 57ec92c299 last30days: require Python 3.12 in setup flows 2026-04-08 14:40:19 -07:00
Pejman Pour-Moezzi 681d05d7ee last30days: lazy-load vendored Bird cookie support 2026-04-08 14:31:17 -07:00
Matt Van Horn 9c203d3595 Merge pull request #172 from pejmanjohn/contrib/mvanhorn-last30days-skill-41-rename-last30days-output
last30days: finish runtime/report rename after /last30days plugin rename
2026-04-08 14:11:06 -07:00
Pejman Pour-Moezzi 77f67c1bd9 last30days: finish runtime/report rename 2026-04-08 13:03:41 -07:00
Matt Van Horn 565deb443e fix: rename plugin from last30days-3 to last30days 2026-04-08 11:12:06 -07:00
Matt Van Horn 0a9ff16dfc feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources
v3 rewrites the search engine from the ground up:

- Intelligent pre-research: resolves X handles, GitHub repos, subreddits,
  TikTok hashtags, and YouTube channels before searching
- GitHub person-mode: PR velocity, top repos by stars, release notes
- GitHub project-mode: live star counts, README, releases, top issues
- ELI5 mode: plain language synthesis, no jargon
- 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket,
  GitHub, Threads, Pinterest, Perplexity, Bluesky, Web
- Free Reddit comments via public JSON (no API key needed)
- Fun judge v2: humor scoring baked into narrative
- Cookie consent before browser scanning
- 10,000 free ScrapeCreators calls
- 1,012 tests

Thank you to the community contributors whose issues and PRs shaped v3:
@uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116),
@thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78),
@zl190 (#115), @hnshah (#84, #85, #86)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:23 -07:00
Matt Van Horn 61904b31e3 feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX
* feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX

- INCLUDE_SOURCES=tiktok,instagram in .env forces sources on for all
  query types, bypassing the tier system
- NUX shows opt-in modal after ScrapeCreators key is saved: "Also
  search TikTok and Instagram?" with honest call-usage warning
- Tier system preserved as default — override only when INCLUDE_SOURCES set

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: neutral call-usage copy — works for free and paid tiers

---------

Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 06:22:37 -07:00
Matt Van Horn bb79d54b2b Revert "fix: ScrapeCreators modal sells all 5 platforms, not just Reddit"
This reverts commit 2eb9cd6fba.
2026-03-29 18:17:25 -07:00
Matt Van Horn 2eb9cd6fba fix: ScrapeCreators modal sells all 5 platforms, not just Reddit 2026-03-29 18:16:58 -07:00
Matt Van Horn 775596ce21 feat: v2.9.6 — free-first NUX, cookie extraction, quality scoring
Setup wizard with consent-first cookie extraction (Chrome/Firefox/Safari),
yt-dlp auto-install, ScrapeCreators push, quality scoring (5 core sources),
status banner redesign, honest Reddit labeling, inline YouTube transcripts,
Exa free web search, Reddit public fallback, and post-research quality nudge.

Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 14:33:17 -07:00
Matt Van Horn 4d6224f79a feat(youtube): extract transcript highlights like Reddit comment gems
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.

SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:49:38 -07:00
Matt Van Horn 499074b564 fix(youtube): pass full transcripts to LLM instead of truncating to 200 chars
TRANSCRIPT_MAX_WORDS raised from 500 to 5000 so the LLM gets the full
content of most videos (up to ~25 minutes). Removed the second 200-char
truncation in render.py that was reducing transcripts to a single sentence
before the judge agent ever saw them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-23 17:28:01 -07:00
Matt Van Horn 6a5a0013c0 Merge pull request #70 from mvanhorn/fix/bluesky-cloudflare-error-messages
fix(bluesky): surface real error instead of misleading 'auth failed'
2026-03-15 22:56:00 -07:00
Matt Van Horn 0c6358ab98 fix(bluesky): surface real error instead of misleading "auth failed"
When Cloudflare blocks requests to bsky.social or public.api.bsky.app
with a 403, the error was swallowed by a generic except clause and
reported as "Bluesky auth failed" - misleading users into thinking
their credentials were wrong.

Now _create_session() preserves the specific error in _session_error,
and search_bluesky() surfaces it. Cloudflare 403s get a clear message
about network-level blocks. Actual 401s say "Invalid credentials".

Closes #69

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 22:13:28 -07:00
Matt Van Horn 7ffa508bb4 feat(last30days): improve ClawHub discoverability - description, tags
Updated frontmatter description to be more search-friendly for ClawHub.
Added 12 new tags: deep-research, twitter, bluesky, recency, news,
citations, multi-source, social-media, analysis, web-search, ai-skill,
clawhub. Also added 11 GitHub repo topics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 17:27:53 -07:00
Matt Van Horn c2e18dbbeb fix(skill): use full /last30days name in comparison follow-up suggestions
The short alias /last30 only works on some platforms. Claude Code requires
the full /last30days name, so the follow-up suggestions after comparison
research were producing "Unknown skill: last30" errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 07:55:17 -07:00
Matt Van Horn 52a22f5cb1 Merge pull request #65 from j-sperling/feat/search-quality-consolidation
Consolidate query/relevance modules and improve search quality
2026-03-14 07:31:24 -07:00
Matt Van Horn 3830274e11 Merge pull request #67 from j-sperling/feat/query-type-source-tiering
Add query-type-aware source tiering and scoring
2026-03-14 07:30:06 -07:00
Matt Van Horn 569745c0ed Merge pull request #66 from j-sperling/fix/endpoint-model-updates
Fix stale API endpoints and model identifiers
2026-03-14 07:29:56 -07:00
Jeffrey Sperling 058c4e1899 Classify prompt and animation queries earlier
Map prompt-oriented product searches and animation-oriented build searches away from the breaking-news default so source tiering and tiebreakers align with the benchmark topics.

Validation: uv run python -m unittest tests.test_query_type
2026-03-14 00:38:59 -07:00
Jeffrey Sperling c711e443fe Reduce Reddit and Polymarket false positives
Weight Reddit relevance toward titles, stop Polymarket from expanding low-signal standalone terms, and prevent short binary outcomes from matching unrelated queries.

Validation: uv run python -m unittest tests.test_reddit_sc tests.test_polymarket
2026-03-14 00:38:52 -07:00
Jeffrey Sperling 8c1dce95e8 Harden local search evaluation harness
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.

Validation: uv run python -m unittest tests.test_evaluate_search_quality
2026-03-14 00:38:43 -07:00
Jeffrey Sperling 3a0f3d8b19 Accept GOOGLE_API_KEY for local Gemini eval
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.

Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
2026-03-13 19:25:29 -07:00
Jeffrey Sperling 8eda5fad5c Add local search quality evaluation harness
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.

The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.

Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
2026-03-13 19:21:33 -07:00
Jeffrey Sperling 946af84f9a Tighten relevance scoring and Polymarket ranking
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.

Also apply the relevance filter to Polymarket and update the affected cross-source tests.

Validation: uv run python -m unittest
2026-03-13 19:21:25 -07:00
Jeffrey Sperling 0e46c7cb33 Pass X auth through handle drilldowns
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.

Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
2026-03-13 01:09:03 -07:00
Jeffrey Sperling b489663450 Loosen source tiering for usage queries
Classify prompting and animation queries as how_to so the stack does not treat them as generic breaking news. Also keep X available for how_to and preserve YouTube/HN coverage for breaking-news and prediction queries.

Validated with uv run python -m unittest tests.test_query_type and the five-query local comparison run used for PR #65 review.
2026-03-13 01:08:28 -07:00
Jeffrey Sperling 3aaf31b08d Document env-based X auth flow
Update README, launch copy, and UI guidance to prefer popup-free AUTH_TOKEN/CT0 configuration, and keep X backend selection on the verified Bird or xAI paths.

Validation: uv run python -m unittest tests.test_env_project
2026-03-12 21:07:09 -07:00
Jeffrey Sperling dd9a3f1482 Disable browser cookie fallback for local X auth
Prefer injected AUTH_TOKEN/CT0 for bundled Bird, disable browser-cookie probing in repo-invoked subprocesses, and keep repo-invoked yt-dlp from inheriting browser-cookie settings.

Validation: uv run python -m unittest tests.test_bird_x tests.test_youtube_yt
2026-03-12 21:07:04 -07:00
Jeffrey Sperling cbee987f65 Extract relevance_filter, add Bluesky/TruthSocial type hint + test coverage
- Extract _relevance_filter from last30days.py closure to score.relevance_filter()
  for testability
- Add BlueskyItem/TruthSocialItem to sort_items() type hint (was missing despite
  being in _ITEM_SOURCE_MAP)
- Add tests: Bluesky/TruthSocial engagement scoring, sort_items mixed sources,
  relevance_filter behavior (threshold, minimum-result guarantee, missing attr),
  select_openai_model HTTP 401/403 error paths
2026-03-11 19:09:06 -07:00
Jeffrey Sperling d8d2b97716 Gitignore mise.toml instead of removing it
Dev environment tool config is useful locally but shouldn't be tracked.
2026-03-11 19:01:33 -07:00
Jeffrey Sperling 036bcd2ae3 Address review feedback: deduplicate query_type, clean unused imports, fix defaults
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
  canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
  tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
  should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
2026-03-11 18:40:07 -07:00
Jeffrey Sperling 6c402f66b7 Update plan to reflect single upstream PR strategy 2026-03-11 18:32:45 -07:00
Jeffrey Sperling 046795c4ae Add post-retrieval relevance filtering across all sources
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.

This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 1002f1f020 Add platform-specific query optimizations
- hackernews: use extract_core_subject instead of raw topic, add
  points>5 filter and restrictSearchableAttributes=title to reduce
  noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
  filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
  queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
  to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
2026-03-11 18:32:45 -07:00
Jeffrey Sperling c5be117701 Replace hardcoded 0.7 relevance with computed token-overlap scores
- bird_x: parse_bird_response now accepts query param and computes
  token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement

This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 38caae3288 Add mise.toml for Python version pinning and implementation plan
Pin Python 3.12 via mise for consistent local development.
Add plan document for the query/relevance consolidation work.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 96948cc7c0 Deduplicate relevance code across youtube/tiktok/instagram/scrapecreators_x
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.

Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling dc88c215be Integrate shared query.py into per-source modules
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.

Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set

Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling fa42a5d031 Add urllib fallback for TikTok/Instagram when requests unavailable
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00
Matt Van Horn b38703e53d feat(truthsocial): Add Truth Social as opt-in source
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.

27 new tests, 440 total passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 00:14:39 -07:00
Matt Van Horn b6fd5ff406 docs: v2.9.5 README update, fix plugin.json hooks (#62)
- Fix plugin.json hooks field from invalid array to empty object
- Add Claude Code plugin install above ClawHub badge
- Version bump to v2.9.5 with new features block (Bluesky, comparative
  mode, ScrapeCreators X, per-project env, expanded tests)
- Add Bluesky to all source list references
- Document BSKY_HANDLE/BSKY_APP_PASSWORD env vars in install + optional sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:40:07 -07:00
Matt Van Horn adb5a672d9 fix(bluesky): make Bluesky opt-in with app password auth
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 23:27:39 -07:00
Matt Van Horn ecf90c0281 feat(skill): add comparative mode and Bluesky references to SKILL.md
Add COMPARISON query type for "X vs Y" research with 3 parallel passes.
Add Bluesky stats line and update all source list references.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:58:21 -07:00
Matt Van Horn 9a1059ee9d feat(bluesky): add Bluesky/AT Protocol as social source
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).

- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:54:20 -07:00
Matt Van Horn 4b7087e136 feat(x): add ScrapeCreators as X/Twitter search backend
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).

New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.

Fixes #55.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:34:39 -07:00
Matt Van Horn 82a006280e Merge pull request #58 from phjlljp/feat/session-start-config-check
feat: add SessionStart hook for config check
2026-03-09 21:55:57 -07:00
Matt Van Horn a7398c50cc Merge pull request #59 from phjlljp/feat/per-project-env-config
feat: add per-project .env config support
2026-03-09 21:55:51 -07:00
Matt Van Horn b1a0e2bcfc Merge pull request #57 from phjlljp/feat/smoke-tests-edge-cases
test: add smoke tests and edge case coverage
2026-03-09 21:55:44 -07:00
Matt Van Horn 75e4b8e2cd Merge pull request #56 from phjlljp/feat/unit-tests-untested-modules
test: add unit tests for untested modules
2026-03-09 21:55:42 -07:00
Matt Van Horn 25e27bdade fix(skill): unquote $ARGUMENTS and add marketplace plugin path
- Remove double quotes around $ARGUMENTS so argparse can parse flags
  like --deep, --store separately instead of as part of the topic string.
  Fixes #61.

- Add ~/.claude/plugins/marketplaces/last30days-skill to the path
  discovery loop so marketplace installs can find scripts/.
  Fixes #54.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:55:24 -07:00
P 3e3615e2c8 feat: add per-project .env config support
Add per-project configuration via .claude/last30days.env, discovered by
walking up from cwd. Uses the same .env format as the existing global
config — no new parsers or formats.

Priority (highest wins):
  1. Environment variables
  2. .claude/last30days.env (per-project)
  3. ~/.config/last30days/.env (global)

Also adds file permission checking — warns to stderr if config files
are readable by other users (should be chmod 600).

Includes tests for discovery, precedence, source tracking, and
permission warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:48:23 -04:00
P 9a58fe6481 feat: add SessionStart hook for config check
Add a lightweight hook that runs on session start to check if any
API keys are configured. Warns users if no config is found and
checks file permissions on existing config files.

Checks (in order): .claude/last30days.env, ~/.config/last30days/.env,
OPENAI_API_KEY env var, SCRAPECREATORS_API_KEY env var.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:47:36 -04:00
P 4756c20ec0 refactor: match upstream unittest convention
Convert all new tests from bare pytest style to unittest.TestCase
with sys.path.insert, matching the convention used by all existing
tests. Remove pyproject.toml and conftest.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:43:04 -04:00
P 8a9f734d14 test: add smoke tests and edge case coverage
Add end-to-end smoke tests and expand coverage for score and render modules:

- test_smoke.py (new): subprocess tests for --diagnose, --help, --mock,
  and missing topic. Validates exit codes, JSON structure, and source
  detection (HN/Polymarket always available).
- test_score.py: add TestCommentQualityWeight (top_comment_score boost),
  TestInstagramEngagement (basic scoring, views vs likes weighting)
- test_render.py: add TestEnsureOutputDir, TestXrefTag, TestRenderEmptyReport

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:41:58 -04:00
P 0979f506db test: add unit tests for untested modules
Add pytest infrastructure (pyproject.toml, conftest.py) and unit tests
for modules that previously had zero test coverage:

- test_schema_roundtrip.py: to_dict() serialization for all data classes
- test_reddit_enrich.py: URL parsing, thread data parsing, comment filtering
- test_reddit_sc.py: ScrapeCreators Reddit search (query expansion, subreddit discovery)
- test_instagram_sc.py: Instagram relevance scoring, tokenization, depth config

Includes fixtures/reddit_thread_sample.json for reddit_enrich tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:36:52 -04:00
Matt Van Horn 627947fc2c feat(plugin): publish as Claude Code marketplace plugin
Update .claude-plugin/marketplace.json and plugin.json to v2.9.5 with
full metadata. Add plugin install instructions to README as the
recommended install method. The repo root serves as both the marketplace
and the plugin - skills/last30days/SKILL.md (symlink) is discovered
automatically.

Users can now install with:
  /plugin marketplace add mvanhorn/last30days-skill
  /plugin install last30days@last30days-skill

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 13:09:33 -07:00
Matt Van Horn 2f16ff1ee8 feat(gemini): add Gemini CLI extension support
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.

Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.

Closes #45

Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:07:55 -07:00
Matt Van Horn 8f7fb5a7fe fix(release): v2.9.5 - remove re-introduced Save Research section
PR merges on March 7 (PR #48 Xiaohongshu, upstream merge) regressed
SKILL.md by re-introducing the "Save Research to Documents" section
that v2.9.4 removed. Those branches were forked before v2.9.4 and
brought the old content back via merge resolution.

Fixes: remove save section, restore --save-dir flag on bash command,
update agent mode line, add tool-call guard to STOP instruction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 11:55:35 -07:00
Matt Van Horn 4503da7920 docs: add perpetual monitoring mode plan (not building yet)
Explored adding scheduled re-runs and cumulative intelligence to
last30days. Concluded that Claude Code's session-scoped scheduling
(CronCreate/loop) can't support true perpetual monitoring since
jobs die when the terminal closes. Plan documents the architecture,
what exists, and why we're waiting for persistent background agents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 09:37:18 -07:00
Matt Van Horn 28dff6e7b2 Merge remote-tracking branch 'upstream/main' 2026-03-07 16:36:15 -08:00
Matt Van Horn 32992834ee Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:11:35 -08:00
Matt Van Horn 7dd8379c61 Merge origin/main into feat/xiaohongshu-reddit-public-fallback
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:08:20 -08:00
Matt Van Horn 408ac148ec Merge pull request #50 from mark-c4r/add-entity-extract-tests
test: add tests for entity_extract module
2026-03-07 15:59:32 -08:00
Matt Van Horn c9559252cd Merge pull request #52 from 04cb/fix/missing-metadata-files
Fix missing metadata files in skill upload bundle
2026-03-07 15:59:11 -08:00
04cb f70370a6f4 Fix missing metadata files in skill upload bundle 2026-03-07 18:10:44 +08:00
Matt Van Horn fad26d41fd fix: improve ClawHub security scan result
- Remove prompt-injection false positive ("you are now" → "treat yourself as")
- Declare AUTH_TOKEN and CT0 in frontmatter optionalEnv
- Clarify X token access language (no browser session access)
- Add permissions overview block near top of file

Zero functionality changes — metadata and prose only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:35:38 -08:00
Matt Van Horn ef1f380cda feat: publish to ClawHub as last30days-official
- Add ClawHub badge and install command to README
- Update SKILL.md: metadata.openclaw canonical key, license/author/repository fields, optionalEnv vars, added instagram/polymarket tags
- Add .clawhubignore to exclude binary assets and dev files from bundle

Published: https://clawhub.ai/skills/last30days-official

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 18:08:14 -08:00
Matt Van Horn e690d61a12 fix: append -raw suffix to saved research filenames
e.g. sam-altman-raw.md instead of sam-altman.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:56:03 -08:00
Matt Van Horn 6d5acb9121 feat(release): v2.9.4 - move save into Python script, zero post-invitation noise
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.

Tested: --mock confirms file creation and duplicate date suffixing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:27:53 -08:00
Matt Van Horn 28d223e43c fix(release): v2.9.3 - foreground save, fix hallucinated user messages
CRITICAL: run_in_background callbacks caused model to re-engage after
save, hallucinate fake "Human:" messages, and generate unsolicited
multi-paragraph responses. Switch to foreground cat > heredoc which
executes sub-second with no callback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:50:17 -08:00
Matt Van Horn 471badd329 fix(release): v2.9.2 - silent save, no follow-up text after background save
- Background Bash heredoc instead of Write tool
- Suppress response text on save completion
- 📎 footer line replaces verbose confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:41:55 -08:00
Matt Van Horn 5b1636f94f fix: merge upstream save fix (background Bash instead of Write tool)
Resolves merge conflict, keeping upstream's approach:
- Background heredoc save instead of Write tool
- Adds 📎 footer line
- No more "Wrote N lines..." cluttering output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 14:59:06 -08:00
Matt Van Horn 18f6273f7c fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:22 -08:00
Matt Van Horn 9950d01ab4 fix: save research silently via background Bash, not Write tool
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 10:34:11 -08:00
Matt Van Horn cc774d5e69 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Sync from public repo. Every run now saves the complete briefing as a
topic-named .md file to ~/Documents/Last30Days/. Credit @devin_explores.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:50:43 -08:00
Matt Van Horn 8cbbe87c3e docs: add v2.9.1 auto-save note to README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:27:18 -08:00
Matt Van Horn 49d993b162 feat(release): v2.9.1 - auto-save research to ~/Documents/Last30Days/
Bump version to 2.9.1, update changelog and release notes.
Credit @devin_explores for inspiring the feature.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:15:42 -08:00
Matt Van Horn 6afc094bb1 Merge pull request #51 from mvanhorn/feat/auto-save-documents
feat(skill): auto-save research results to ~/Documents/Last30Days/
2026-03-05 19:12:56 -08:00
Matt Van Horn f6a1769e35 feat(skill): auto-save research results to ~/Documents/Last30Days/
Every run now automatically saves the complete briefing (synthesis,
stats, follow-up suggestions) as a topic-named .md file in the user's
Documents folder. Agent mode also saves. No Python script changes -
this is purely a SKILL.md instruction addition.

Inspired by @devin_explores manually saving results to build a
personal research library.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 19:06:27 -08:00
Matt Van Horn 4d35b53eab docs: v2.9.0 release — ScrapeCreators Reddit default, top comments, smart discovery
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 18:03:40 -08:00
Matt Van Horn 2247800003 chore: clean up Reddit log prefix, mark plan tasks complete 2026-03-05 18:01:24 -08:00
Matt Van Horn 7048fe7b83 feat(reddit): elevate top comments, improve subreddit discovery, default to ScrapeCreators
Three improvements from beta testing:

1. Top comments: 10% scoring weight for comment quality, 💬 top comment
   rendered prominently in compact/full output, increased insight limits
2. Subreddit discovery: relevance-weighted scoring with topic word matching,
   utility sub penalties (UTILITY_SUBS blocklist), engagement bonus
3. Default method: SKILL.md primaryEnv → SCRAPECREATORS_API_KEY, web-only
   banner recommends SC first, security section updated

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 17:29:18 -08:00
Matt Van Horn 30b973f62e docs: add Reddit ScrapeCreators v2 improvements plan
Three focused improvements based on 5 full-pipeline beta tests:
1. Elevate top Reddit comments in scoring and rendering
2. Improve subreddit discovery heuristic for ambiguous queries
3. Make ScrapeCreators the default recommended Reddit method

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 17:24:41 -08:00
Matt Van Horn 09b09946c0 feat: replace OpenAI Reddit search with ScrapeCreators API
- New scripts/lib/reddit.py: multi-query expansion, global search,
  subreddit discovery, targeted subreddit search, comment enrichment
- 68 results in 17s vs ~15 results in 60-90s (OpenAI)
- Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper)
- Real engagement data (score, comments, dates) from API
- No more 429 rate limits on comment enrichment
- Falls back to OpenAI if SCRAPECREATORS_API_KEY missing
- Registered as last30daysbeta for parallel local testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 15:55:02 -08:00
mark-c4r d4328b5598 test: add tests for entity_extract module
Covers _extract_x_handles (8 cases), _extract_x_hashtags (5 cases),
_extract_subreddits (6 cases), and extract_entities integration (4 cases).
Follows existing test patterns from test_dedupe.py.
2026-03-05 16:12:33 -06:00
YJLi-new 788514ce8e feat: add Xiaohongshu source and Reddit public fallback
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
2026-03-05 20:54:33 +08:00
Matt Van Horn db75f9e341 feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.

- Add scripts/lib/instagram.py: keyword search, transcript extraction,
  relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
  URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 07:00:51 -08:00
Matt Van Horn 740dcc5789 docs: update README and SKILL.md for ScrapeCreators TikTok API
Replace all Apify references with ScrapeCreators. Key points:
- No subscription required (was $5/mo with Apify)
- 100 free credits, pay-as-you-go after
- SCRAPECREATORS_API_KEY replaces APIFY_API_TOKEN
- Backwards compatible: APIFY_API_TOKEN still works as fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 05:17:30 -08:00
Matt Van Horn e03046bd49 refactor(tiktok): replace Apify with ScrapeCreators API
Root cause of empty TikTok results: Apify required monthly subscription.
ScrapeCreators is PAYG with 100 free credits and no subscription.

Key fix: ScrapeCreators nests items under aweme_info wrapper
(search_item_list[].aweme_info.{fields}), which the previous
implementation missed, causing all fields to be empty.

Changes:
- Rewrite tiktok.py to use ScrapeCreators REST API
- Add aweme_info unwrapping for correct field extraction
- Add transcript fetching via /video/transcript endpoint
- Add SCRAPECREATORS_API_KEY to env.py config
- Update last30days.py to use env.get_tiktok_token()
- Delete apify_client_wrapper.py (no longer needed)
- Update tests for new date field format (create_time)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 13:58:51 -08:00
Matt Van Horn 1d18bee1a2 fix(skill): forward CLI flags through $ARGUMENTS to Python script
Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits
the expansion, and change argparse topic from nargs="?" to nargs="*"
so multi-word topics still work. Also document --store, --include-web,
--diagnose, and --timeout flags in the Options section.

Closes #36

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 13:52:08 -08:00
Matt Van Horn fb00856bff docs: update README for v2.7 with TikTok examples and installation
Add TikTok as 7th source throughout README: new V2.7 banner, real
search examples (Iran Israel: 61.6M views, Leah Halton: 152.6M views),
APIFY_API_TOKEN in installation, Apify in security table, fix stale
"six sources" references to "seven sources".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:52:51 -08:00
Matt Van Horn d7b354b2cf fix(ui): suppress [TikTok] and [Apify] log lines in non-TTY mode
Only print debug log lines when running in an interactive terminal.
In Claude Code (non-TTY), the spinner system handles progress display,
so these raw log lines just add noise.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:45:00 -08:00
Matt Van Horn 7c5763d048 fix(apify): suppress verbose actor log streaming to stderr
Pass logger=None to Apify .call() to prevent the SDK from streaming
raw actor run logs (status messages, crawler stats, warnings) that
drown out the clean spinner UI in Claude Code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:43:38 -08:00
Matt Van Horn 61729b9ae7 fix(ui): show YouTube and TikTok progress spinners in Claude Code
Remove quiet=True from YouTube and TikTok spinners so they display
the same colored emoji progress lines as Reddit and X in non-TTY mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:36:50 -08:00
Matt Van Horn b990aed40e feat: add --no-native-web flag to skip Parallel AI in Claude Code
When running in Claude Code, the assistant has a built-in WebSearch tool
that's free and higher quality than Parallel AI/Brave/OpenRouter. Adding
--no-native-web to the SKILL.md invocation defers web search to the
assistant, saving API credits. OpenClaw invocations don't pass this flag,
so they continue using native web backends.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:34:03 -08:00
Matt Van Horn d4ac57f041 fix(tiktok): restore missing websearch import in orchestrator
The websearch module import was dropped when the tiktok import was added,
causing the script to crash during the rendering phase after all data
was successfully collected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:25:30 -08:00
Matt Van Horn 1db0b6054a feat(tiktok): add TikTok as 7th signal source via Apify
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.

- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:08:19 -08:00
Matt Van Horn 5e5d586f7d fix: triage all 16 open GitHub issues — close 9, fix 6, comment 1
Batch 1 (closed): #43 spam, #34 dup, #19 resolved, #2 resolved, #41 answered
Batch 2: Added MIT LICENSE file (#35), closed #42 (license question)
Batch 3 code fixes:
  - #29: YouTube skip reason shows "0 results" instead of "not installed"
  - #30: Bird source mapping handles reddit-web + Bird combo
  - #39: watchlist.py extracts YouTube + TikTok findings, run-one prints output
  - #40: watchlist.py uses search_queries field when available
Batch 4:
  - #32: marketplace.json source "." → "./" with $schema ref
  - #36: commented with investigation plan ($ARGUMENTS forwarding)
  - #4: Added SSL troubleshooting section to README
Also commented on #22 (Bird features) and #31 (skills.sh audit).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 06:06:55 -08:00
Matt Van Horn 94b6b6eb7b feat(search): add --search flag for source filtering
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.

--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn  (run only Reddit + Hacker News)

Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:43:33 -08:00
Matt Van Horn 6ae4b16791 feat(bird_x): add noise words + last-chance retry with strongest token
Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool
noise words to _extract_core_subject, and a last-chance retry that falls back
to the longest non-noise token when 2-word retry also returns 0 results.

cache.py and render.py env overrides were already on main.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:39:52 -08:00
Matt Van Horn 82efa6100b fix(skill): use plain source names in Web stats line, not URLs
URLs in markdown links wrap badly in terminals (discovered after first
fix attempt). Change to plain names like "Newsweek, Sportskeeda, Medium"
on the Web: stats line. Update citation note to explain the reason.

Tested on Dor Brothers, Kanye West, Logan Paul - no trailing Sources:
block appeared in any of the three test runs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:54:06 -08:00
Matt Van Horn a52ed30109 fix(skill): suppress trailing Sources: block from WebSearch tool mandate
The WebSearch tool has a system-level mandate to append a Sources:
section at the end of every response. SKILL.md's old "DO NOT output
Sources: list" instruction was too weak to override it.

Fix: redirect citations into the stats block's Web: line as inline
links. The WebSearch citation requirement is satisfied there; an
explicit note after the stats block tells the model not to append
a separate trailing section.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:44:55 -08:00
Matt Van Horn 78678e3919 chore: add .gitignore and PR #37 finalization plan
- .gitignore: protect docs/comparison-results/ and other private
  benchmark artifacts from accidental upstream push
- docs/plans: add plan for PR #37 Codex auth finalization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:44:55 -08:00
Matt Van Horn 04bfb5381d fix(tests): patch env isolation in test_api_key_takes_priority
The test was picking up the real OPENAI_API_KEY from the shell
environment, causing it to fail on any machine with that key set.
Added @patch.dict(os.environ, {}, clear=True) so the test runs in
a clean env and exercises the file_env path as intended.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:36:37 -08:00
Ilia Alshanetsky d7bff81757 fix(bird_x): pass .env credentials to Node subprocesses for WSL2/headless auth
* chore: fix YAML error in argument-hint

* add codex auth support to responses API

* Use gpt-5.1-codex-mini as default model for Codex auth

Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.

* Pass .env credentials to Bird Node subprocesses for X auth

On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.

- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
  credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection

---------

Co-authored-by: Justin Williams <jblwilliams@gmail.com>
2026-03-02 23:24:59 -08:00
Matt Van Horn 1ed990a081 feat(skill): v2.6 - agent-native invocation and --agent report mode
Removes disable-model-invocation restriction so the skill can be called
by other agents via the Skill tool. Adds --agent flag for non-interactive
report output (skips intro, AskUserQuestion, wait pause, and invitation).
Fixes false security doc claiming autonomous invocation was blocked.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 23:07:47 -08:00
Matt Van Horn 27b4865d60 feat(skill): bump version to 2.5
Polymarket prediction markets (6th source), Hacker News (5th source),
cross-source linking, synonym expansion, X handle resolution.
15-way blinded comparison: 4.38 vs 3.73, won all 5 topics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:04:41 -08:00
Matt Van Horn 9ee1291033 docs: update README for v2.5 - Polymarket + HN as killer features, add Anthropic odds example
Reorder v2.5 features: Polymarket prediction markets and HN lead as #1,
multi-signal quality-ranked relevance scoring as #2. Add Anthropic Odds
example showcasing 11 live markets from a two-word query. Add Anthropic
and OpenAI Polymarket transcripts to launch tweets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:42:00 -08:00
Matt Van Horn e48d84b1d0 fix(polymarket): two-pass query expansion finds markets where topic is an outcome
The Gamma API only searches event titles/slugs, missing markets where the
topic is an outcome (e.g., "Arizona" in NCAA Tournament Winner). This adds:

- All-word query expansion (not just first word): "Arizona Basketball" now
  searches "Arizona", "Basketball" independently
- Tag-based domain expansion: extracts category tags (e.g., "NCAA") from
  first-pass results and searches those as a second pass
- Neg-risk binary market synthesis: shows team names from market questions
  instead of generic Yes/No outcomes
- Question shortening: extracts "Arizona" from "Will Arizona win the NCAA
  Tournament?" for clean display
- Increased depth (3 pages) and result caps (15) for more coverage

Live results: "Arizona Basketball" now finds NCAA Tournament Winner (12%),
#1 Seed (88%), Big 12 Champion (69%). "Iran War" returns 15 markets (up
from 9) with no regression.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 09:06:35 -08:00
Matt Van Horn 309c8e37c6 fix(sync): deploy to last30daysCROSS with patched frontmatter
- Add CROSS to sync targets with sed-patched name/version/description
- Switch cp to rsync to handle identical file edge case on APFS
- CROSS SKILL.md gets last30daysCROSS skill root injected into search path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 08:27:30 -08:00
Matt Van Horn 2ff9b6f6c1 feat(polymarket): outcome-aware scoring and synthesis instructions
- _compute_text_similarity() now checks outcome names with bidirectional
  substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
  market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 08:12:45 -08:00
Matt Van Horn 4c82309c36 fix(skill): strengthen zero-result hiding and improve Polymarket stats format
Make the "omit 0-result sources" instruction more emphatic (CRITICAL prefix,
enumerate all formats to suppress). Change Polymarket stats line to show
top 2-3 market odds instead of just top-1.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:09:18 -08:00
Matt Van Horn 9d9e7e89d9 feat(polymarket): replace position-based ranking with quality-signal relevance
Polymarket results now rank by text similarity, volume, liquidity, price
movement, and competitive score instead of API return position. Also fixes
pagination (DEPTH_CONFIG now controls page count, not a no-op limit param)
and caps results after re-ranking.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:07:43 -08:00
Matt Van Horn 994a4ab2ca feat(polymarket): add Polymarket prediction markets as 6th research source
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.

Also hides sources with zero results from the stats output (all sources).

54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 22:27:19 -08:00
Matt Van Horn 52503f8e68 docs(readme): credit community contributors for HN source
Shoutout to @ARJ999 (first HN submission, PR #12), @wkbaran
(PR #26, referenced in planning), and @gbessoni for endorsing
HN as the right addition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:10:58 -08:00
Matt Van Horn 44a662ef94 docs(readme): reorder v2.5 features - lead with better results
Reorganize the changelog to match the real story: smarter scoring,
cross-source linking, and handle resolution as one "dramatically
better results" narrative first, then HN, then handle resolution
details. Moves blinded comparison right after the results section
as proof.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:06:49 -08:00
Matt Van Horn 7009039ac4 docs: v3 -> v2.5 version bump (incremental, not major)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 21:00:57 -08:00
Matt Van Horn 41c8742fcf docs: update README to v3 with new features
- Bump version to v3
- Add headline features: HN source, handle resolution, cross-source
  linking, YouTube relevance scoring
- Add full v3 changelog section with detailed feature descriptions
- Reorganize v2/v2.1 changelog sections
- Add HN to security/privacy data table
- Update examples and descriptions throughout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:59:59 -08:00
Matt Van Horn 71903b596f fix(skill): broaden handle resolution to products/tools/brands, add parody check
Handle resolution was only triggering for "person/brand" topics. Products
and tools like "Nano Banana Pro" or "Seedance" can also have X accounts
(@nanobanana, @seedanceai) but were being skipped.

Changes:
- Broadened trigger: people, brands, products, tools, companies, communities
- Added parody/fan account verification guidance
- Added site:x.com to WebSearch query for better results
- Updated both SKILL.md and OpenClaw variant

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:46:16 -08:00
Matt Van Horn bbaaf28d2e fix(x): don't skip unfiltered resolved handle search when entity-extracted
The resolved handle dedup was wrong: if entity_extract found @thedorbrothers
(from @mentions in Phase 1 results), the resolved handle search was skipped
entirely. But entity-extracted handles are searched WITH topic keywords
(from:handle topic), while resolved handles need UNFILTERED search
(from:handle) to find posts that don't mention the topic string.

Example: Dor Brothers' viral tweet (5.5K likes) says "We made a $300M movie
starring @LoganPaul" - no mention of "dor brothers" anywhere. The topic-
filtered entity search missed it. The unfiltered resolved search finds it.

Before: 30 X posts, 161+ likes (entity search only)
After: 40 X posts, 5549+ likes (resolved handle adds viral tweet)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:35:08 -08:00
Matt Van Horn 4f584a4e96 feat(x): resolve X handles for person/brand topics via agent WebSearch
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.

- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 20:00:49 -08:00
Matt Van Horn bed0557b65 feat(quality): GOAT synthesis improvements - hybrid cross-source linking, YouTube synonyms, human-readable xref tags
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics
(4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further:

- dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold,
  cross-source links went from 3 to 13 items across 5 topics
- render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags
- youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score)
- SKILL.md: instruction #7 tells Claude to lead with cross-platform signals

Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5
topics. Biggest gains in specificity (+0.8) and format compliance (+1.0).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 16:06:53 -08:00
Matt Van Horn 0591f55f0e feat(quality): YouTube relevance scoring and cross-source linking
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.

Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 10:46:58 -08:00
Matt Van Horn f60a4359a0 fix(ui): quiet HN and YouTube spinners in non-TTY mode
Reddit and X are the star of the show. In Claude Code (non-TTY),
suppress  start messages for HN and YouTube so Reddit/X are the
first visible lines. HN/YouTube still show ✓ completion messages.
Also suppress [HN] debug logs in non-TTY to reduce output clutter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:58:01 -08:00
Matt Van Horn 7a9f447231 fix(ordering): move HN after YouTube in stats, sort priority, and SKILL.md
HN was appearing before YouTube in the stats block, sort tiebreaker,
and source status. Now consistently: Reddit > X > YouTube > HN > Web.
Also restored emoji + box-drawing chars in test skill SKILL.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:41:14 -08:00
Matt Van Horn 38a7ea253e feat(hackernews): add Hacker News as 5th research source
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:33:31 -08:00
Matt Van Horn 427a4e453d Merge pull request #33 from tjarko/fix/429-exponential-backoff
Fix OpenAI 429 rate limiting with exponential backoff
2026-02-20 23:03:01 -08:00
Tjarko Leifer 451ebb3e22 Fix OpenAI 429 rate limiting with exponential backoff
The Reddit search uses OpenAI's Responses API with web_search, which
frequently returns 429 rate limit errors. The previous retry logic used
linear backoff (1s, 2s, 3s) which is too aggressive for OpenAI's rate
limiter (often needs 10-60s waits).

Changes:
- Increase max retries from 3 to 5
- Switch from linear to exponential backoff (2s, 5s, 9s, 17s, 33s)
- Parse and respect Retry-After header from OpenAI 429 responses
- Fall back to cheaper models (gpt-4.1 → gpt-4o) on 429s, not just
  on 400/403 access errors
- Remove gpt-4o-mini from fallback chain — it doesn't support
  web_search with the filters parameter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 12:40:36 +01:00
409 changed files with 50651 additions and 16900 deletions
+20
View File
@@ -0,0 +1,20 @@
{
"name": "last30days-skill",
"interface": {
"displayName": "Last 30 Days"
},
"plugins": [
{
"name": "last30days",
"source": {
"source": "local",
"path": "./"
},
"policy": {
"installation": "AVAILABLE",
"authentication": "ON_INSTALL"
},
"category": "Research"
}
]
}
+12 -5
View File
@@ -1,17 +1,24 @@
{
"name": "last30days",
"name": "last30days-skill",
"owner": {
"name": "mvanhorn",
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"metadata": {
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0"
"description": "Marketplace hosting the Last 30 Days research plugin."
},
"plugins": [
{
"name": "last30days",
"source": "."
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.3.2",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
},
"source": "./",
"category": "productivity",
"homepage": "https://github.com/mvanhorn/last30days-skill"
}
]
}
+7 -5
View File
@@ -1,12 +1,14 @@
{
"name": "last30days",
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
"version": "2.1.0",
"version": "3.3.2",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "mvanhorn"
"name": "Matt Van Horn",
"email": "mvanhorn@gmail.com",
"url": "https://github.com/mvanhorn"
},
"homepage": "https://github.com/mvanhorn/last30days-skill",
"repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT",
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
"skills": ["./"]
"keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"]
}
+18
View File
@@ -0,0 +1,18 @@
# Exclude binary assets and dev/test artifacts from ClawHub bundle
assets/
docs/
fixtures/
tests/
plans/
agents/
variants/
release-notes.md
SPEC.md
TASKS.md
SKILL-original.md
*.jsonl
*.mp3
*.jpeg
*.jpg
*.png
*.gif
+44
View File
@@ -0,0 +1,44 @@
# Exclude non-runtime files from `git archive` output.
# Used by skills/last30days/scripts/build-skill.sh to produce a
# claude.ai-upload-ready .skill file from the canonical skills/last30days tree.
# See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
# Anthropic canonical skill-packaging excludes
# (mirrors anthropics/skills/skills/skill-creator/scripts/package_skill.py)
__pycache__/ export-ignore
node_modules/ export-ignore
*.pyc export-ignore
.DS_Store export-ignore
evals/ export-ignore
# Dev, docs, test, and media - not needed at skill runtime
tests/ export-ignore
docs/ export-ignore
fixtures/ export-ignore
assets/ export-ignore
# NOTE: skills/ and .claude-plugin/ are NOT export-ignored here because
# Claude Code's /plugin install fetches this same git archive tarball.
# Removing those from the archive (as v3.0.1 did) silently breaks installs.
# claude.ai-bundle-specific exclusions live in scripts/build-skill.sh.
# Historical + repo-only manifests
SPEC.md export-ignore
TASKS.md export-ignore
test-run.log export-ignore
CONTRIBUTORS.md export-ignore
HERMES_SETUP.md export-ignore
CHANGELOG.md export-ignore
uv.lock export-ignore
# Platform adapters are kept in git archives because Claude Code and Codex
# plugin installs use the same repository archive as their source payload.
.hermes-plugin/ export-ignore
# CI workflows - repo-only, not needed at skill runtime
.github/ export-ignore
# Build config itself
.clawhubignore export-ignore
.gitignore export-ignore
.gitattributes export-ignore
+53
View File
@@ -0,0 +1,53 @@
name: Bug Report
description: Report a bug or unexpected behavior
labels: [bug]
body:
- type: textarea
id: summary
attributes:
label: Summary
description: What happened?
placeholder: Describe the bug in 1-2 sentences.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to Reproduce
description: How can we reproduce this?
placeholder: |
1. Run `python3 skills/last30days/scripts/last30days.py "topic" --emit=compact`
2. ...
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected Behavior
description: What should have happened?
validations:
required: true
- type: textarea
id: traceback
attributes:
label: Error / Traceback
description: Paste the full traceback or error output.
render: text
- type: dropdown
id: install
attributes:
label: Install Method
options:
- Claude Code plugin
- Gemini CLI extension
- Codex plugin
- Hermes skill
- Manual (git clone)
- Other
validations:
required: true
- type: input
id: os
attributes:
label: OS
placeholder: macOS 15.4, Ubuntu 24.04, Windows 11, etc.
@@ -0,0 +1,24 @@
name: Feature Request
description: Suggest a new feature or improvement
labels: [enhancement]
body:
- type: textarea
id: problem
attributes:
label: Problem
description: What problem does this solve?
placeholder: When I try to ..., I can't ...
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed Solution
description: How should this work?
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives Considered
description: Other approaches you thought of (optional).
+19
View File
@@ -0,0 +1,19 @@
## Summary
<!-- What does this PR do? 1-3 sentences. -->
## Changes
<!-- Bullet list of what changed. Reference files if helpful. -->
-
## Testing
<!-- How did you verify this works? -->
- [ ] Ran `uv run python -m pytest -q --tb=short`
## Related Issues
<!-- Link issues: Fixes #123 or Relates to #456 -->
+31
View File
@@ -0,0 +1,31 @@
name: Release
on:
push:
tags:
- "v*"
permissions:
contents: write
jobs:
build-and-release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Build .skill artifact
run: |
bash skills/last30days/scripts/build-skill.sh
test -f dist/last30days.skill
- name: Create GitHub release
uses: softprops/action-gh-release@v2
with:
files: dist/last30days.skill
generate_release_notes: true
draft: false
prerelease: false
+67
View File
@@ -0,0 +1,67 @@
name: Security
on:
pull_request:
push:
branches:
- main
workflow_dispatch:
permissions:
contents: read
jobs:
dependency-audit:
name: Dependency audit
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Set up Python
run: uv python install 3.12
- name: Export locked dependency set
run: |
uv export \
--locked \
--all-groups \
--no-hashes \
--format requirements.txt \
--output-file /tmp/last30days-requirements.txt
# Advisory-first: visibility before enforcement. This repo handles API keys,
# cookies, browser tokens, and local env files, so dependency CVEs should be
# visible in CI logs even before the project has a clean blocking baseline.
# Set continue-on-error: false once a clean baseline run is confirmed.
- name: Run pip-audit against locked dependencies
continue-on-error: true
run: uvx --python 3.12 pip-audit -r /tmp/last30days-requirements.txt --progress-spinner=off
secret-scan:
name: Secret scan
runs-on: ubuntu-latest
steps:
- name: Checkout full history for diff-aware scanning
uses: actions/checkout@v4
with:
fetch-depth: 0
# Advisory-first: this reports verified secrets in pull requests and pushes to
# main, but does not block merges until maintainers confirm a clean baseline.
# The TruffleHog action automatically scans the PR range for pull_request
# events and the pushed commit range for push events.
# Set continue-on-error: false once a clean baseline run is confirmed.
# Contributor policy: never commit real secrets in fixtures, tests, docs, or
# examples; use obvious dummy values and env-based auth patterns instead.
- name: Run TruffleHog OSS secret scan
if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: trufflesecurity/trufflehog@v3.95.2
continue-on-error: true
with:
path: ./
version: v3.95.2
extra_args: --only-verified
+26
View File
@@ -0,0 +1,26 @@
name: Validate
on:
pull_request:
push:
branches:
- main
permissions:
contents: read
jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Set up Python
run: uv python install 3.12
- name: Run test suite
run: uv run pytest
+34
View File
@@ -0,0 +1,34 @@
# Private benchmark / evaluation artifacts — never push to upstream
docs/comparison-results/
scripts/evaluate-synthesis.py
scripts/generate-synthesis-inputs.py
fixtures/polymarket_sample.json
docs/v2.1-tweets.md
docs/30-day-anniversary-thread.md
docs/30-day-anniversary-tweets.md
variants/open/references/research.md
# OS / tool files
.DS_Store
.claude/
.entire/
__pycache__/
*.pyc
mise.toml
.memsearch/
.venv/
.coverage
htmlcov/
# Root vendor/ is accidental - real vendored client lives at scripts/lib/vendor/bird-search/
/vendor/
# build artifact from scripts/build-skill.sh
/dist/
# Internal planning docs (ce:plan output) — keep local, don't publish
docs/plans/
.context/
/work
/print
+66
View File
@@ -0,0 +1,66 @@
# last30days Skill
Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
## Structure
- `skills/last30days/SKILL.md` — canonical skill definition / runtime spec the model reads when the slash command fires
- `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
- `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
- `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
- `CONFIGURATION.md` — user-facing knobs (env vars, flags, per-host install patterns); keep in sync per the rules below
- `CHANGELOG.md` — structured release history (launch copy lives in GitHub Releases)
- `HERMES_SETUP.md` — install instructions for the Hermes harness specifically
## Orientation
- This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
- Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration is incomplete — the model invoking the skill won't know the flag exists.
- README and PR examples show `/last30days <topic>` first. Direct CLI invocation (`python3 scripts/last30days.py ...`) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
- Slash commands don't pass shell mechanics through. `/last30days OpenClaw --emit=html | pbcopy` is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full `python3 ...` with explicit flags and a real shell).
## Commands
```bash
# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only)
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # copies skill into ~/.agents/skills/<name>/ (frozen at install time); re-run to sync working-tree edits — see Rules below
# Tests (pytest, ~89 files under tests/, configured in pyproject.toml)
uv run pytest # full suite
uv run pytest tests/test_dedupe_v3.py # single file
uv run pytest tests/test_dedupe_v3.py -k some_case # single case
uv run pytest --cov # with coverage (skips lib/vendor/)
```
Python 3.12+ required. Use `uv` for the env; the venv lives at `.venv/`.
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` copies the skill into `~/.agents/skills/<name>/` (real directory) and, for harnesses that support symlinked skill dirs, drops a per-host symlink pointing at that copy. **Working-tree edits do NOT propagate automatically** — the `~/.agents/skills/<name>/` copy is frozen at install time. To sync after edits, re-run `npx skills add . -g -y`. For live-edit on a dev machine, replace the install copy with a symlink to the working tree: `ln -sfn "$PWD/skills/last30days" ~/.agents/skills/last30days` (run from the repo root).
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Security hygiene
- Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
- Use the env-based auth patterns in `skills/last30days/scripts/lib/env.py`; tests and fixtures must use obvious dummy values only.
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Maintaining CONFIGURATION.md
`CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
Update `CONFIGURATION.md` when:
- adding a new env var (e.g. `LAST30DAYS_*`, `BSKY_*`, `*_API_KEY`)
- adding a new CLI flag that affects configuration (e.g. `--store`, `--web-backend`)
- adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Hermes…)
- adding a new optional source that requires its own credential
- changing the priority order of config layers (per-run flag > env > `.env` file > defaults)
Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / `.env` → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.
When a new config concept lands in `SKILL.md` or `AGENTS.md`, mirror the user-facing knob in `CONFIGURATION.md` so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+533 -1
View File
@@ -5,6 +5,535 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
- **First-party positioning research + pitch-vs-pulse synthesis (company / product / service topics).** A new mandatory research step captures each entity's current stated positioning from first-party sources (homepage, docs, pricing) rather than from memory. The fetched pitch grounds `What it is` descriptions (entities described as they pitch themselves today), helps reject unrelated brand-name noise, and feeds an evidence-triggered prose beat: when the month's conversation directly supports a specific claim, cuts against one, or is squarely about the pitched ground, the synthesis says so anchored to the top thread — and stays silent when the pulse is orthogonal to the pitch, because a manufactured connection is worse than omission. Claims are tested at matched altitude (specific claims against specific threads; broad taglines are never graded against individual items), and statements stay windowed to the 30 days — no trend verdicts. Scoped to entities with an identifiable first party: people are always excluded (even founders whose companies qualify), as are events, abstract concepts, and ownerless topics like Bitcoin; the beat requires positioning fetched during the run, never from memory.
### Fixed
- Entity-grounding rerank demotion now keys on the head token of the primary entity instead of requiring the full multi-word phrase as a contiguous substring. A high-engagement on-entity item (e.g. a 323-pt HN thread titled "Stripe is friendly to 'friendly fraud'") is no longer demoted to score 0 on a `Stripe payments` query just because it lacks the trailing search-hint word. The intended demotion still fires for items that never name the brand at all. The keyless Reddit comment-enrichment slot selection (`_slot_priority`), which mirrors this signal, was updated to the same head-token grounding so the two paths stay consistent.
## [3.3.2] - 2026-06-06
### Fixed
- Keyless Reddit comment enrichment now spends its limited slots on entity-matching posts first (mirroring rerank's entity-miss demotion signal) instead of raw upvote order, so off-topic high-upvote threads from broad subreddits no longer consume the comment budget only to be demoted afterward ([#484](https://github.com/mvanhorn/last30days-skill/pull/484))
## [3.3.1] - 2026-05-30
### Fixed
- Removed the redundant `commands/last30days.md` wrapper so the plugin exposes only the skill ([#461](https://github.com/mvanhorn/last30days-skill/issues/461)). Previously the plugin shipped both a command wrapper and the skill under the same name, so `/last30` surfaced two `last30days` entries with two different descriptions. The skill already carries its own `argument-hint`, so the `/last30days <topic>` picker UX is unchanged.
- Corrected the README install note that claimed Claude Code dedupes the slash command across install methods; it does not, so having both the marketplace plugin and the `npx skills` copy active shows two entries.
## [3.3.0] - 2026-05-17
A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner.
### Added
**Emit modes and sources**
- `--emit=html` for shareable, print-friendly HTML research briefs ([#332](https://github.com/mvanhorn/last30days-skill/pull/332)).
- **Digg AI 1000 source**, auto-enabled when `digg-pp-cli` is on PATH ([#370](https://github.com/mvanhorn/last30days-skill/pull/370)). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief.
**Configuration knobs**
- `EXCLUDE_SOURCES` env var — the inverse of `INCLUDE_SOURCES`, honored in source count and pipeline filter ([#399](https://github.com/mvanhorn/last30days-skill/pull/399)).
- `LAST30DAYS_YOUTUBE_SSH_HOST` — opt-in SSH routing for `yt-dlp` through a residential-IP host, for users on datacenter VPS hit by YouTube's bot-wall ([#376](https://github.com/mvanhorn/last30days-skill/pull/376)). Host validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection. Transcript path unchanged (uses HTTP fallback).
- macOS Keychain as a credential source — reads from the system keychain when env vars and config files aren't set ([#407](https://github.com/mvanhorn/last30days-skill/pull/407)).
- Configuration enablement: env-var defaults and source-resilience patterns across the config layer ([#344](https://github.com/mvanhorn/last30days-skill/pull/344)).
**Pipeline and storage**
- Reddit URL auto-enrichment from web search via the public JSON API ([#366](https://github.com/mvanhorn/last30days-skill/pull/366)).
- Per-run finding sightings recorded in the SQLite store ([#373](https://github.com/mvanhorn/last30days-skill/pull/373)).
- Brave browser support for X/Twitter cookie extraction ([#320](https://github.com/mvanhorn/last30days-skill/pull/320)).
**Tests and CI**
- Full pytest suite restored to CI; 13 rotted tests repaired ([#416](https://github.com/mvanhorn/last30days-skill/pull/416)).
- `greptile.json` added with `triggerOnUpdates` + `statusCheck` ([#418](https://github.com/mvanhorn/last30days-skill/pull/418)).
- Advisory security workflow ([#368](https://github.com/mvanhorn/last30days-skill/pull/368)).
- Parallel grounding backend test coverage ([#355](https://github.com/mvanhorn/last30days-skill/pull/355)).
**Docs**
- New `CONFIGURATION.md` with README pointers ([#339](https://github.com/mvanhorn/last30days-skill/pull/339)).
- `docs/solutions/` learning capture for release-time consistency-test cascades ([#413](https://github.com/mvanhorn/last30days-skill/pull/413)) and the eval-not-in-CI design decision ([#417](https://github.com/mvanhorn/last30days-skill/pull/417)).
### Changed
**Install story modernized**
- `npx skills add` is now the canonical install path for every harness ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). README and SKILL.md flipped to recommend `npx skills add . -g -y` over per-harness manual instructions. Surfaces Gemini CLI, Copilot, Windsurf, and 50+ other Agent Skills hosts that the install pattern reaches.
- README dropped the Gemini CLI native-extension install path (now covered by `npx skills add`).
- `hooks.json` made polyglot for Gemini CLI + Claude Code compatibility ([#318](https://github.com/mvanhorn/last30days-skill/pull/318)).
**Skill semantics and multi-harness reframe**
- `AGENTS.md` is now canonical; `CLAUDE.md` points at it ([#410](https://github.com/mvanhorn/last30days-skill/pull/410)). Reframes the project as a multi-harness Agent Skills package rather than a Claude-Code-specific tool.
- SKILL.md path resolution rewritten: STEP 0 narrows to a Claude-Code-marketplaces-only stale-clone guard; Step 1 walks a single `SKILL_DIR` substitution pattern ([#400](https://github.com/mvanhorn/last30days-skill/pull/400), [#409](https://github.com/mvanhorn/last30days-skill/pull/409)). Removes ~80 lines of bash and fixes a real spec-vs-engine divergence where the previous resolver could pick a different install than the SKILL.md the model loaded from.
- SKILL.md version regex consolidated into `lib/skill_meta.py` ([#412](https://github.com/mvanhorn/last30days-skill/pull/412)).
- `--plan` / `--competitors-plan` invocation templates switched from inline single-quoted JSON to heredoc-written tmpfiles ([#404](https://github.com/mvanhorn/last30days-skill/pull/404), fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403)). Apostrophes in resolved context strings ("McDonald's", "people's choice") no longer break shell parsing.
- `POSTS_PER_CLUSTER` raised 3→5 and render-side display limit 2→3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous caps routinely truncated cluster context.
- Digg AI 1000 renamed to "Digg" in user-facing output ([#372](https://github.com/mvanhorn/last30days-skill/pull/372)) — footer line, source label, inline-quote suffix, why_relevant, container attribution. Internal references retain the upstream product name.
- GitHub repo resolution canonicalized for ambiguous product comparisons ([#302](https://github.com/mvanhorn/last30days-skill/pull/302)).
**Dependencies and tooling**
- Dropped `requests` runtime dependency. All providers route through stdlib `urllib` via the `lib/http` wrapper ([#393](https://github.com/mvanhorn/last30days-skill/pull/393)).
- Migrated to `gemini-3.1-flash-lite` GA model ([#378](https://github.com/mvanhorn/last30days-skill/pull/378)).
- Aligned Codex/Claude plugin manifests + added Codex `AGENTS.md` ([#321](https://github.com/mvanhorn/last30days-skill/pull/321)).
- pytest dev dep bumped 9.0.2 → 9.0.3 ([#414](https://github.com/mvanhorn/last30days-skill/pull/414)).
### Removed
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1 ([#400](https://github.com/mvanhorn/last30days-skill/pull/400)). Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`** — maintainer dev-deploy script ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). Replaced by `npx skills add . -g -y` (live-symlink into every detected harness's skill dir — better than sync.sh's copy model since edits propagate live). Hermes uses `hermes skills install mvanhorn/last30days-skill --force`; OpenClaw uses `clawhub install last30days-official`.
- Orphaned `SPEC.md` and `TASKS.md` ([#419](https://github.com/mvanhorn/last30days-skill/pull/419)).
### Fixed
**Reddit**
- `lstrip("r/")` mangled subreddits starting with `r` (`r/robotics``obotics`, `r/ruby``uby`); replaced with `removeprefix("r/")` at 4 sites (Alex Key, salvaged from #288).
- Browser-like User-Agent + `Accept-Language`/`Accept-Encoding`/`Connection` headers + gzip decompression to fix `urllib` 403s on Reddit's public JSON endpoint (Franco Carballar, salvaged from #199).
- HTTP 402 re-raised across all three ScrapeCreators paths (`_global_search`, `_subreddit_search`, `fetch_post_comments`) so the OpenAI/public-JSON fallback chain triggers when credits are exhausted (Jonathan Oppenheim, salvaged from #170).
**Authentication and credentials**
- Restored multi-key rotation for `SCRAPECREATORS_API_KEY` accidentally dropped in v3.0.6 (Eric Oberhofer, salvaged from #287). Comma-separated keys round-robin via `random.choice` per run.
**Windows compatibility**
- `os.killpg` in `_cleanup_children()` guarded with `hasattr(os, "killpg")`, falls back to `os.kill(SIGTERM)` (gujishh, salvaged from #226).
- POSIX-style secret-permission warning skipped on Windows ([#357](https://github.com/mvanhorn/last30days-skill/pull/357)).
- Render uses forward slashes in save-path footer for Windows ([#338](https://github.com/mvanhorn/last30days-skill/pull/338)).
**xAI / X / xurl**
- `parse_x_response` now raises `http.HTTPError` on empty output, missing JSON, or decode failure — surfaces in `errors_by_source` instead of silently returning an empty result list (Kaustav Mishra, salvaged from #155).
- `xurl` treats `PermissionError` from PATH lookup as unavailable ([#322](https://github.com/mvanhorn/last30days-skill/pull/322)).
**YouTube**
- SC YouTube + multi-token HN searches unblocked ([#388](https://github.com/mvanhorn/last30days-skill/pull/388)).
- Transcript-fetch ratio surfaced + degraded-run nudge for stale `yt-dlp` ([#340](https://github.com/mvanhorn/last30days-skill/pull/340)).
**bird_x / HTTP**
- Subprocess retry on non-JSON stdout to handle X anti-bot HTML interstitials ([#383](https://github.com/mvanhorn/last30days-skill/pull/383)).
- HTTP retry budget expanded + exponential backoff on DNS resolution failure ([#382](https://github.com/mvanhorn/last30days-skill/pull/382)).
- Parallel AI search aligned with current API schema ([#341](https://github.com/mvanhorn/last30days-skill/pull/341)).
- Parallel web backend routed through grounding ([#354](https://github.com/mvanhorn/last30days-skill/pull/354)).
**Planner and sources**
- `xquik` registered in `SOURCE_CAPABILITIES` ([#336](https://github.com/mvanhorn/last30days-skill/pull/336), fixes [#319](https://github.com/mvanhorn/last30days-skill/issues/319)).
- Honor explicit optional source requests ([#356](https://github.com/mvanhorn/last30days-skill/pull/356)).
- ScrapeCreators source-gating aligned between code and docs ([#415](https://github.com/mvanhorn/last30days-skill/pull/415)).
- OpenClaw works without ScrapeCreators key ([#392](https://github.com/mvanhorn/last30days-skill/pull/392), by @thinkun).
**Render, version display, hosting paths**
- Hardcoded `v3.0.0` in render replaced with dynamic `_skill_version()` ([#365](https://github.com/mvanhorn/last30days-skill/pull/365)).
- Comparison HTML artifacts saved correctly ([#389](https://github.com/mvanhorn/last30days-skill/pull/389)).
- `OPENROUTER_DEFAULT` model ID corrected ([#323](https://github.com/mvanhorn/last30days-skill/pull/323)).
- OpenClaw poll-timing initialized once ([#358](https://github.com/mvanhorn/last30days-skill/pull/358)).
- Prefer sandboxed Safari cookie path ([#343](https://github.com/mvanhorn/last30days-skill/pull/343)).
- Preserve clean mode for last-run state ([#334](https://github.com/mvanhorn/last30days-skill/pull/334)).
- Replaced hardcoded `/Users/mvanhorn/...` paths in `test-v1-vs-v2.sh` with portable env-var overrides (Dave Morin, salvaged from #297).
**Hooks**
- `check-config.sh` path-quoting fix for paths with spaces ([#337](https://github.com/mvanhorn/last30days-skill/pull/337)).
- Replaced unsafe `eval` with `declare` in `check-config.sh` ([#364](https://github.com/mvanhorn/last30days-skill/pull/364)).
**Sync and version metadata**
- `sync.sh` pointed at this repo's plugin cache, not the private repo's ([#402](https://github.com/mvanhorn/last30days-skill/pull/402)).
- Sync cache target bumped to 3.2.1 to match SKILL.md ([#397](https://github.com/mvanhorn/last30days-skill/pull/397)).
- ScrapeCreators free-tier credit count corrected to 100 in docs ([#369](https://github.com/mvanhorn/last30days-skill/pull/369), fixes [#367](https://github.com/mvanhorn/last30days-skill/issues/367)).
- Gemini extension version synced ([#349](https://github.com/mvanhorn/last30days-skill/pull/349)).
- Various stale path/link fixes ([#345](https://github.com/mvanhorn/last30days-skill/pull/345), [#346](https://github.com/mvanhorn/last30days-skill/pull/346), [#347](https://github.com/mvanhorn/last30days-skill/pull/347), [#348](https://github.com/mvanhorn/last30days-skill/pull/348), [#351](https://github.com/mvanhorn/last30days-skill/pull/351)).
### Contributors
First-time contributors whose fixes shipped in this release (most via PR triage salvage — fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) ([#363](https://github.com/mvanhorn/last30days-skill/pull/363)) — OpenClaw ScrapeCreators-key-optional fix
Full PR list at [github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0](https://github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0).
## [3.2.0] - 2026-05-09
### Added
- Add `--emit=html` for shareable, print-friendly HTML research briefs.
- **Digg AI 1000 source** (auto-enabled when `digg-pp-cli` is on PATH). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief as `[@handle](xUrl) via Digg AI 1000: ...` lines. Footer line: `⛏️ Digg AI 1000: N clusters │ K posts │ M authors`. No X auth required for the inline quotes since they flow through Digg's read-only endpoints.
## [3.1.1] - 2026-04-24
### Fixed
- **Codex plugin layout.** Move the canonical runtime payload under `skills/last30days/` and update Codex/Claude plugin metadata and tests for the relocated engine path.
- **Claude Code cache resolution.** Resolve Claude plugin installs to `skills/last30days/scripts/last30days.py` after the plugin-layout restructure.
## [3.1.0] - 2026-04-22
Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`, per-entity Step 0.55, vs-mode N passes, comparison title attribution) and republishes the OpenClaw bundle, which had been frozen on ClawHub at `3.0.0-open` since April 8.
### Added
- **OpenClaw republish.** `clawhub install last30days-official` now resolves to `3.1.0-open`, matching current main. Closes [#307](https://github.com/mvanhorn/last30days-skill/issues/307), [#195](https://github.com/mvanhorn/last30days-skill/issues/195), [#236](https://github.com/mvanhorn/last30days-skill/issues/236). The ClawHub bundle had shipped a broken `env.py get_config()` and stale SKILL.md path references since April; both are fixed at source on main and the republish carries the fixes to installers.
### Fixed
- **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
- **Broken README link.** The README's "source of truth" link pointed at root `SKILL.md`, which is no longer maintained after the plugin-layout restructure. Fixed to point at `skills/last30days/SKILL.md`.
### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
Individual changelog entries for 3.0.10 through 3.0.14 below document the incremental work consolidated into this release.
## [3.0.14] - 2026-04-22
### Changed
- **Comparison-mode title attribution.** The synthesis title for vs-mode and `--competitors` outputs changes from `What the Community Says (Last 30 Days)` to `What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range. Three SKILL.md occurrences updated; pure documentation change.
## [3.0.13] - 2026-04-22
### Changed
- **vs mode runs N full passes in parallel, one per entity.** Architectural revert of the 3-pass → 1-pass latency optimization from an earlier version. `/last30days "OpenAI vs Anthropic vs xAI"` now runs three full `pipeline.run()` calls in parallel via the same fanout `--competitors` uses, producing three `*-raw.md` save files plus a merged comparison output. Each entity gets its own Step 0.55-grade targeting, own primary X handle weight, own subreddit scoping — apples-to-apples depth instead of the one-pool merged retrieval the single-pass path produced. Parallel execution keeps wall clock ≈ single pass.
- **`--competitors` is now a SKILL.md-level shortcut for vs-mode with auto-discovery.** The hosting reasoning model (Claude Code, Codex, Hermes, Gemini, any agent with WebSearch) performs discovery and Step 0.55 per entity via its own WebSearch tool, then invokes the engine with a vs-topic and `--competitors-plan` JSON. The engine flag remains for headless/cron use with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys (engine-internal `auto_resolve` stays as fallback).
- **LAW 7-style stderr for `--competitors` with no backend** now leads with the hosting-model path (WebSearch + Step 0.55 + `--competitors-plan`) instead of `BRAVE_API_KEY`. API-key framing moved to a secondary "headless" section.
### Added
- **`--competitors-plan` JSON flag** for per-entity Step 0.55 targeting. Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`. Accepts inline JSON or a file path (matches `--plan`). When present for an entity, skips engine-internal `auto_resolve` and uses the provided values; missing fields fall back to `auto_resolve` (if backend) or planner defaults. Case-insensitive entity matching. The `subrun_kwargs_for` helper is the single source of truth for per-entity kwargs — no closure-default fallthrough from main scope.
- **Per-entity save files** when `--save-dir` is set on a vs-mode or `--competitors` run. Each entity's sub-run produces its own `{slug}-raw.md` with a single-row Resolved Entities block — matches historical vs-mode behavior (N passes → N save files).
- **`--polymarket-keywords "kw1,kw2"`** to filter Polymarket matches for ambiguous single-token topics (e.g., "Warriors" → `nba,gsw,golden-state` kills Glasgow Warriors rugby and Honor of Kings Rogue Warriors noise).
### Fixed
- **BRAVE/SERPER footer nudge suppressed** when `--plan` or `--competitors-plan` is present. The nudge told Claude Code users to set an API key when they already have WebSearch via the hosting model. Nudge still fires for true headless runs (no `--plan`, no backend) where the advice is correct.
- **Override-leak regression testing.** 3.0.12 already fixed the main-topic `--subreddits` / `--x-handle` / `--github-*` from leaking into peer sub-runs via explicit per-entity kwargs scrubbing. This release adds a 4-test regression suite (`test_competitor_subrun_isolation.py`) locking in the invariant.
## [3.0.12] - 2026-04-22
### Fixed
- **Per-entity Step 0.55 resolution for competitor sub-runs.** In 3.0.11, only the main topic got X handle / subreddit / GitHub resolution; competitor sub-runs ran with planner defaults and produced visibly thinner evidence (Reddit 403 fallbacks, single-word queries). Each competitor sub-run now calls `resolve.auto_resolve()` inside `fanout.run_competitor_fanout` when a web backend is available, mirroring the main topic's pre-flight resolution. Per-entity X handle, subreddit list, GitHub user/repos, and news context are threaded into each sub-run's `pipeline.run()` call. Deep-copied config per sub-run prevents `_auto_resolve_context` cross-leak. Surfaces in a new `## Resolved Entities` output block so the resolution coverage is visible without reading stderr.
- **LAW 7 false-positive on internal fan-out sub-runs.** Each competitor sub-run was emitting the `[Planner] No --plan passed... YOU ARE the planner` stderr warning. LAW 7 targets the hosting-reasoning-model path, not engine-internal fan-out. New `internal_subrun=True` keyword on `planner.plan_query` and `pipeline.run` suppresses the warning for sub-runs only; the default path is unchanged.
- **Marketplace-stale SKILL.md trap.** Added a STEP 0 canonical-path self-check at the top of SKILL.md. Two of three 2026-04-22 test runs loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (Claude-Code-managed git clone pinned to origin/main, lagging the versioned cache), then ran `--help` against the same stale path, did not see `--competitors`, and fell back to a manual comparison plan. The STEP 0 block forces any reader to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` and re-read from the versioned cache if not.
### Changed
- **Default `--competitors` count is now 2 (3-way total: original + 2 peers).** Previously 3. `--competitors=N` still customizes (range 1..6). Matches the feature description's canonical example (`Kanye vs Drake vs Kendrick`).
### Added
- **`## Resolved Entities` block** in `render_comparison_multi` output. Shows per-entity X handle, subreddits, GitHub user/repos, and truncated context for every entity in the comparison. Block is omitted entirely when no entity has a resolved payload (mock mode, no backend).
## [3.0.11] - 2026-04-22
### Added
- **`--competitors` flag for auto-discovered comparison fan-out.** Pass `--competitors` on a single-entity topic and the engine discovers 2-6 peer entities via web search, then runs the full pipeline on each in parallel and emits one N-way comparison. `last30days Kanye West --competitors` resolves Drake, Kendrick Lamar, and one more peer. `last30days OpenAI --competitors` resolves Anthropic, xAI, Google Gemini. `--competitors=N` controls count, `--competitors-list="A,B,C"` skips discovery and uses the explicit list. Discovery mirrors the `auto_resolve` pattern (Brave / Exa / Serper / Parallel) with deterministic text extraction - no internal LLM call. Sub-runs inherit the main `--quick`/`--deep`/`--days`, run in a `ThreadPoolExecutor`, and degrade gracefully when at least 2 entities survive. Output reuses the existing 9-axis `## Head-to-Head` scaffold.
## [3.0.10] - 2026-04-21
### Added
- **Commenter handles on evidence lines.** Top-comment rendering now includes the commenter's handle - `u/author` for Reddit, `@handle` for TikTok/YouTube/Instagram/Bluesky/X/Threads. The enrichment adapters already captured `author`; the render layer just was not using it. Evidence lines change from `- Comment (6822 upvotes): Finally, John Apple` to `- u/Cyrisaurus (6822 upvotes): Finally, John Apple`. Person-level citations make synthesis-side inline markdown links per LAW 8 much more natural. Both the compact and full render paths are covered.
### Fixed
- **TikTok author preference.** `_fetch_post_comments` in `scripts/lib/tiktok.py` preferred `user.nickname` over `user.unique_id`, so the engine captured display names ("Moosa Noormahomed") instead of @handles ("moosanoormahomed"). Flipped to prefer `unique_id`. Nickname still wins as a fallback when `unique_id` is missing. Display names can contain emoji, spaces, and non-Latin characters that do not round-trip to a profile URL; the @handle is the stable identifier.
- **Single plugin payload layout.** The canonical runtime moved to `skills/last30days/` for both Claude Code and Codex plugin loading. Root-level `SKILL.md`, `scripts/`, `agents/`, and `assets/` are no longer maintained as duplicate copies.
### Behavior fallback
- When an author is empty, `[deleted]`, or `[removed]`, the render falls back to the legacy `Comment (...)` shape - no `u/` or `@` prefix with an empty handle is ever emitted.
## [3.0.9] - 2026-04-18 - The Self-Debug Release
### Highlights
v3.0.9 adds the engine-side Class 1 keyword-trap refuse-gate ("birthday gift for 40 year old" now gets a clarifying question, not 5 minutes of junk), promotes TikTok and YouTube top comments to the same first-class rendering Reddit's got, lands Hermes AI Agent as a first-class deploy target, and moves the SKILL.md formatting contract from line 1094 to the top of the file.
"The Self-Debug Release" refers to how the fixes in 3.0.6-3.0.9 were written: 5 separate Opus 4.7 instances each debugged their own failed outputs. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries." I shipped exactly what they said. Validation: 5/5 canonical compliance.
### Added
- **Engine Class 1 keyword-trap refuse-gate** (`scripts/lib/preflight.py`, new). Pattern-matches demographic-shopping queries at main() front-door. Exit code 2 with structured REFUSE message. Escape hatch: `LAST30DAYS_SKIP_PREFLIGHT=1`. 29 tests in `tests/test_preflight.py`.
- **TikTok + YouTube top comments** rendered with same `💬 Top comment` prominence as Reddit's. Shipped in [#260](https://github.com/mvanhorn/last30days-skill/pull/260); enrichment fixed in [#265](https://github.com/mvanhorn/last30days-skill/pull/265).
- **Hermes AI Agent as a deploy target** - thanks @stephenmcconnachie ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)). `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys automatically.
- **Multi-key SCRAPECREATORS_API_KEY rotation** - thanks @zaydiscold ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)). Set `SCRAPECREATORS_API_KEY_1`, `_2`, etc. Engine rotates on rate-limit.
- **Offline quality evaluation fixture** - thanks @j-sperling ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)). `eval_topics.json` lets contributors run quality regressions without burning live API credits.
- **END-OF-CANONICAL-OUTPUT boundary** in `render_compact()`. Engine now emits an explicit pass-through instruction so re-synthesis requires actively ignoring a visible boundary.
- **LAW 1 verbatim-pattern override.** LAW 1 now quotes the exact WebSearch tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources: section") and declares it OVERRIDDEN inside last30days output.
### Changed
- **SKILL.md restructure.** VOICE CONTRACT LAWs and BADGE MANDATORY block moved from line 1094 to lines 75-150. Grounded in 3 separate Opus 4.7 self-debugs.
- **Engine emits the badge as stdout.** `🌐 last30days v3.0.9 · synced YYYY-MM-DD` is the first line of every compact emit. Pass-through is now the default-correct behavior.
- **Reddit client HTTP consolidation** - thanks @iliaal ([#207](https://github.com/mvanhorn/last30days-skill/pull/207)). Migrated to `http.get(params=...)` helper.
- **ScrapeCreators header consolidation** - thanks @iliaal ([#209](https://github.com/mvanhorn/last30days-skill/pull/209)). `_sc_headers` refactored into `http.scrapecreators_headers`.
- **Simpler Hermes sync.** `scripts/sync.sh` Hermes branch now always uses main SKILL.md (previously had a `.hermes-plugin/SKILL.md` fallback that created a wrong-file-capture hazard).
### Fixed
- **Peter Steinberger trailing Sources leak.** 2026-04-18 validation failure where the model appended a TechCrunch / TED / Fortune / Wikipedia Sources list after the invitation. Now structurally prevented at three layers: engine emits the canonical body, LAW 1 quotes the exact WebSearch reminder, closing boundary names the anti-pattern.
- **Wrong-file SKILL.md capture.** Deleted `.agents/skills/last30days/SKILL.md` (1382 lines, April 13 snapshot) and `.hermes-plugin/SKILL.md` (269 lines). One SKILL.md per plugin now, at the plugin root.
- **GitHub date parsing garbage** - thanks @iliaal ([#208](https://github.com/mvanhorn/last30days-skill/pull/208)). `_parse_date` now rejects invalid input cleanly.
- **Windows Bird X stability** - thanks @Chelebii ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)).
- **Linux `check_perms` false-warn** - thanks @george231224 ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)). Uses GNU stat first.
- **UTF-8 saved output** - thanks @Gujiassh ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)).
- **Version metadata alignment** - thanks @Gujiassh ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) and @shalomma ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)).
- **`--days` alias backcompat** - thanks @BryanTegomoh ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)).
- **`INCLUDE_SOURCES` env default** - thanks @hnshah ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)).
- **Bird X all-None engagement** - thanks @j-sperling ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)).
### Contributors
@j-sperling, @stephenmcconnachie, @zaydiscold, @iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma, @BryanTegomoh for PRs since v3.0.0. @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli, @Cody-Coyote for issues and PRs that shaped the v3 roadmap.
### Recovery
```
/plugin update last30days
/reload-plugins
```
Verify: `cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version` returns `"version": "3.0.9"`.
Smoke test: `/last30days birthday gift for 40 year old` should ask a clarifying question before running.
## [3.0.5] - 2026-04-15
### Added
- **`/last30days` slash command for plugin users.** New `commands/last30days.md` registers a Claude Code slash command. Users type `/last30days <topic>` and Claude Code's autocomplete prefix-matches it to the canonical `/last30days:last30days` form (the same way `/ce:plan` resolves to `/compound-engineering:ce-plan`). The command delegates to the existing `last30days` skill body — no skill behavior changes.
### Removed
- **`skills/last30days-nux/`** — byte-identical duplicate of root `SKILL.md` that created confusing `/last30days:last30days-nux` autocomplete entries via Claude Code's plugin namespacing. The root `SKILL.md` remains the canonical skill source.
### Recovery
```
/plugin update last30days
/reload-plugins
```
Then type `/last30days <topic>` to invoke the skill via slash command. Natural-language invocation ("search the last 30 days for X") continues to work unchanged.
## [3.0.4] - 2026-04-15
### Fixed
- **Cleared `/doctor` path-escape error on Claude Code v2.1.109+.** `.claude-plugin/plugin.json` previously declared `"skills": ["./"]`. That value shipped unchanged from v2.1.0 through v3.0.3 and worked on older Claude Code, but current versions reject `./` with `Path escapes plugin directory: ./ (skills)`. The `"skills"` key is now omitted entirely, matching the pattern used by every other plugin in the Claude Code marketplace ecosystem. Claude Code auto-discovers `skills/*/SKILL.md` when the key is absent.
### Recovery
If `/doctor` reports a path-escape error for last30days, run `/plugin update last30days` then `/reload-plugins`. If errors persist, uninstall and reinstall the plugin.
## [3.0.3] - 2026-04-15
### Fixed
- **Restored `skills/` and `.claude-plugin/` to the plugin install tarball.** v3.0.1 added `.gitattributes` rules that excluded both directories from `git archive` output to shrink the claude.ai `.skill` bundle. Claude Code's `/plugin install` fetches the same archive, so users installing v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no skill files. `git archive v3.0.0` contained 8 files under those paths; `v3.0.1` and `v3.0.2` contained 0. This release reverts those `.gitattributes` lines.
- **Reverted `plugin.json` `"skills"` field to `["./"]`.** v3.0.2 changed this to `["skills"]` based on a misdiagnosis — the manifest change had no effect because the manifest wasn't in the tarball at all. The historical `["./"]` value shipped in every release from v2.1.0 through v3.0.0 without issues and is restored here.
### Recovery
Users on v3.0.1 or v3.0.2: run `/plugin update last30days` then `/reload-plugins`. If autoUpdate is enabled, the next session start will pull v3.0.3 automatically. Users on cached v3.0.0 or earlier installs were unaffected.
### Notes
- The claude.ai `.skill` bundle built by `scripts/build-skill.sh` still works — the archive grew from 89 to 97 files, well under the 200-file cap.
- claude.ai-specific exclusions (avoiding duplicate `SKILL.md` files in the bundle) should move into `scripts/build-skill.sh` rather than `.gitattributes` in a future release, since `.gitattributes` cannot distinguish between the two distribution channels.
## [3.0.2] - 2026-04-15
### Fixed
- **`/last30days` slash command now registers on Claude Code v2.1.105+.** `.claude-plugin/plugin.json` declared `"skills": ["./"]`, which newer Claude Code rejects with `Path escapes plugin directory: ./ (skills)`. The skill silently failed to register, so `/last30days <query>` returned "Unknown command" even though `/plugin list` showed the plugin as installed. Fix: `"skills": ["skills"]` so the loader scans the real skill subdirectory.
- **Version drift between manifests.** `.claude-plugin/marketplace.json` was pinned to `3.0.0` while `.claude-plugin/plugin.json` advertised `3.0.1`. The `/plugin` resolver used the marketplace version and could install stale cached metadata alongside the correct build. Both manifests now agree on `3.0.2`.
### Recovery
If `/last30days` stopped working for you, run `/plugin update last30days` then `/reload-plugins`. If `/doctor` still reports errors, uninstall and reinstall the plugin from the marketplace.
## [3.0.1] - 2026-04-14
### Fixed
- **Skill upload packaging** - `scripts/build-skill.sh` produces a claude.ai-upload-ready `.skill` file that fits under the 200-file cap. Previously, zipping the repo hit 406 files and the "Upload skill" UI rejected it outright.
- **SKILL.md description length** - trimmed from 228 to 167 chars (Anthropic caps descriptions at 200).
### Removed
- Unused root `vendor/` directory (215 files from an accidental commit in PR #48 - the real vendored X client lives at `scripts/lib/vendor/bird-search/`).
- Legacy top-level `plans/` directory (superseded by `docs/plans/`; both plans described work that was already shipped in v3).
### Added
- `.gitattributes` with `export-ignore` entries so `git archive` drops tests, docs, fixtures, assets, historical manifests, and internal skill subdirs. Mirrors Anthropic's canonical `package_skill.py` exclusions.
- `scripts/build-skill.sh` - one-command path to produce `dist/last30days.skill` with a single top-level `last30days/` folder, defensive `=200` file check, and dirty-tree refusal.
- `README.md` section documenting the claude.ai skill upload workflow.
## [3.0.0] - 2026-04-11
### Highlights
Intelligent search, fun judge, cross-source cluster merging, single-pass comparisons, and OpenClaw as a first-class citizen. The v3 engine doesn't just search for your topic -- it figures out *where* to search before the search begins. Engine architecture by @j-sperling.
### Added
- **Intelligent pre-research** -- Resolves X handles, subreddits, TikTok hashtags, and YouTube channels via a new Python brain before any API calls fire. Bidirectional: person to company, product to founder.
- **Fun judge / Best Takes** -- Second parallel LLM judge scores humor, cleverness, and virality. Surfaces the best reactions in a dedicated output section.
- **Cross-source cluster merging** -- Entity-based overlap detection merges the same story across Reddit, X, YouTube into one cluster instead of three separate items.
- **Single-pass comparisons** -- "X vs Y" runs one pass with entity-aware subqueries instead of three serial passes. 3 minutes instead of 12+.
- **GitHub as a source** -- Stars, reactions, and comments from repos and issues.
- **OpenClaw first-class citizen** -- Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
- **Per-author cap** -- Max 3 items per author prevents single-voice dominance.
- **Entity disambiguation** -- Synthesis trusts resolved handles over keyword matches.
- **Perplexity Sonar Pro as additive source** -- AI-synthesized research with citations via OpenRouter. Opt-in via `INCLUDE_SOURCES=perplexity`. Returns structured narratives that complement social data.
- **Perplexity Deep Research** -- `--deep-research` flag for exhaustive 50+ citation reports (~$0.90/query). Premium opt-in for serious investigation.
- **OpenRouter as reasoning provider** -- One OPENROUTER_API_KEY powers planning, reranking, and Perplexity search. Auto-detected after Gemini/OpenAI/xAI.
- **Parallel AI grounding backend** -- `--web-backend parallel` or auto-detected via PARALLEL_API_KEY.
- **Grounding in planner** -- Grounding source properly registered in SOURCE_CAPABILITIES instead of force-injected.
### Changed
- YouTube transcript candidate pool widened 3x past music videos to reach talk/review content with captions
- Reddit comment enrichment sorted by total engagement (upvotes + comments), not just upvotes
- Polymarket display shows % odds only; dollar volumes removed
- 852 tests passing
### Fixed
- Marketplace validation: duplicate `name: last30days` collision in `skills/last30days/SKILL.md` caused strict validators to reject the plugin. Resolved by renaming the internal v3 architecture spec to `last30days-v3-spec` with `user-invocable: false`. Fixed in #214 (reported by @Cody-Coyote in #204).
- Stale README link to the deleted `skills/last30days-v3/` path from the v3 directory rename. Fixed in #214.
- OpenAI Codex CLI discoverability: added `.agents/skills/last30days/SKILL.md` as a real file (Codex's loader skips symlinked files) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now registers as `last30days:last30days` when Codex runs in a checkout of the repo. Fixed in #219 (inspired by @Jah-yee in #153 and @dannyshmueli on X).
### Contributors
- @j-sperling -- v3 engine architecture, Python pre-research brain
- @hnshah -- Watchlist features
- @Cody-Coyote -- Marketplace validation bug report (#204)
- @Jah-yee -- Codex CLI integration inspiration (#153)
## [2.9.4] - 2026-03-06
### Changed
- Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
- Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
- No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research
## [2.9.3] - 2026-03-06
### Fixed
- **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
- Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)
## [2.9.2] - 2026-03-06
### Fixed
- Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
- Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
- Add `📎` footer line for save path instead of verbose confirmation
## [2.9.1] - 2026-03-05
### Highlights
Auto-save research briefings to the default memory directory as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
### Added
- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to the default memory directory after every run
- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
- Agent mode (`--agent`) also saves research files
- Brief confirmation after save with the saved file path
### Credits
- [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))
## [2.9.0] - 2026-03-05
### Highlights
ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
### Added
- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
- Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
- Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
- Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
### Changed
- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
- SKILL.md synthesis instructions updated to emphasize quoting top comments
### Fixed
- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
- Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
## [2.8.0] - 2026-03-04
### Highlights
Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.
### Added
- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
- `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
- Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
- URL-to-name extraction examples in SKILL.md for cleaner web source display
- `--search=instagram` flag support
### Changed
- TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
- SKILL.md version bumped to v2.8
- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
- Security section updated: Apify → ScrapeCreators references
### Fixed
- Web stats line showing full URLs instead of plain domain names
- Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
- `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding
## [2.1.0] - 2026-02-15
### Highlights
@@ -51,7 +580,6 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
### Credits
- @steipete -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
- @galligan -- Marketplace plugin inspiration
- @hutchins -- Pushed for YouTube feature
@@ -59,5 +587,9 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
[3.0.9]: https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
[2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+47
View File
@@ -0,0 +1,47 @@
# Concepts
Shared vocabulary for `last30days-skill`. Terms here have a precise project-specific meaning — distinct enough from their general technical sense that a new contributor would need them defined to follow conversations, PR descriptions, or the SKILL.md contract.
## The package
### Skill
A self-contained agent-instructions package consisting of a `SKILL.md` prose contract plus a sibling `scripts/` directory containing the executable code the SKILL.md invokes. The package conforms to the [Agent Skills](https://agentskills.io) open format and installs across every major harness (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ others) via `npx skills add`, harness-native plugin installers, or per-harness skill directories. A Skill is the unit of distribution; the Skill is the product.
### Engine
The Python script (`scripts/last30days.py`) the Skill's SKILL.md invokes to do the actual research work. The Engine and SKILL.md have a contract: SKILL.md tells the model which flags to pass (`--plan`, `--competitors-plan`, `--x-handle`, `--subreddits`, `--emit=compact`, etc.), and the Engine produces a specific output shape (badge line, ranked evidence clusters, emoji-tree footer) that the model is contractually required to pass through. The Engine is implementation; the SKILL.md prose is the agent-facing surface.
### Harness
The agent runtime that loads Skills and invokes them on the user's behalf. Claude Code is the most common Harness for this Skill but not the only one — Codex, Cursor, GitHub Copilot, Gemini CLI, and the rest of the Agent Skills ecosystem also count. "Multi-harness" describes a Skill that works correctly across every Harness it installs into; features written without multi-harness awareness (e.g., engine flags with no SKILL.md integration, or paths hardcoded to one Harness's install layout) regress on Harnesses other than the one they were tested against.
## Research pipeline
### Primary entity
The brand or proper-noun core of a research topic — the topic with its Intent modifier stripped. It is what the research is *about*, as distinct from how the user phrased the search.
### Intent modifier
A trailing word or phrase in a topic that expresses what the user wants to know rather than what the topic is ("review", "use cases", "pricing"). Stripped when deriving the Primary entity.
### Entity grounding
The check that a candidate item plausibly mentions the Primary entity before final ranking. Grounding keys on the head token (first word) of the Primary entity rather than the full phrase — trailing words are usually search descriptors, so requiring them falsely demotes on-entity items.
An item that fails grounding receives a decisive entity-miss demotion, designed so engagement cannot rescue off-entity content. Because the demotion is decisive, the grounding bar is deliberately conservative: its failure modes degrade toward "no penalty," never toward burying on-entity signal.
### Keyless path
The research flow available with no API keys: source data is gathered by scraping and RSS rather than authenticated APIs, and ranking falls back to local scoring instead of LLM-based reranking. This is the free tier of the Skill; lexical quality safeguards like Entity grounding matter most here, because no LLM is available to judge relevance semantically.
### Comment-enrichment slots
The small, depth-dependent budget of Reddit posts whose comments get fetched in the Keyless path. Slot selection is relevance-aware: posts that pass Entity grounding claim slots first, so the budget is not spent on high-engagement posts that final ranking will demote anyway.
## Distribution
### Beta channel
A parallel install of the Skill, sourced from the private `mvanhorn/last30days-skill-private` repo and installed as `/last30days-beta` rather than `/last30days`. The Beta channel exists so experimental changes can be tested by real users before they ship to the public `/last30days`. Promotion from Beta to public happens via a review PR against this (public) repo — Beta-only changes never ship to public without that PR. The Beta channel workflow guide lives in `BETA.md` in the private repo.
+268
View File
@@ -0,0 +1,268 @@
# Configuration
Everything you can tune in `/last30days` without editing the engine source.
Three layers, in order of how often you'll touch them:
1. **Per-run flags** - what you pass on the command line.
2. **Environment variables and `.env`** - what's enabled across all runs.
3. **Optional trend-monitoring stack** - SQLite store, watchlist, briefings.
Per-client patterns and the experimental beta channel are at the bottom.
> Skip ahead: [Where output is saved](#where-output-is-saved) - [API keys](#api-keys-env) - [Reasoning provider](#reasoning-provider-priority) - [Web search backend](#web-search-backend-priority) - [Trend monitoring](#trend-monitoring-store--watchlist--briefings) - [Per-client patterns](#per-client-patterns) - [Beta channel](#beta-channel)
## Why this document exists
This is a focused **configuration reference** maintained alongside the engine. The runtime contract (the voice rules, the planner protocol, the LAWs the synthesizing model follows) lives in [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) - that file is authoritative when the two ever differ. This file's job is narrower: surface every knob a user or operator can turn, in one place, kept current with the code so client-facing setups stay reliable. New configuration knobs added to the engine should be reflected here in the same PR.
---
## Where output is saved
| Platform | Default path | Override |
|---|---|---|
| Linux / macOS | `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` | set `LAST30DAYS_MEMORY_DIR=/path` |
| Windows | `LAST30DAYS_MEMORY_DIR` defaults to `C:\Users\<you>\Documents\Last30Days\` | set `LAST30DAYS_MEMORY_DIR=C:\path` |
Each run produces one file per topic, slug-named:
`<slug>-raw[-suffix].md`. Same topic + same suffix on the same day overwrites; same topic + same suffix on different days appends a date stamp.
**Per-run overrides:**
- `--save-dir <path>` - one-off output location.
- `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1.
---
## API keys (`.env`)
The skill reads keys from a `.env` file. Two locations are supported, in priority order:
1. **`.claude/last30days.env`** in the current project directory (project-scoped) - takes precedence when present.
2. **`~/.config/last30days/.env`** at the user level (global default) - the fallback.
Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't.
The project-scoped file is the cleanest pattern for **per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), `cd` into that folder, and the skill picks up that client's configuration automatically. No wrapper scripts needed for the common case.
**Source-by-source** - what each key unlocks:
| Source | Key(s) | Required for | Free tier |
|---|---|---|---|
| Reddit (public) | none | always on | yes |
| Hacker News | none | always on | yes |
| Polymarket | none | always on | yes |
| GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
| YouTube | `yt-dlp` CLI installed | always on if `yt-dlp` present | yes |
| X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; xAI / ScrapeCreators = paid |
| TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
| Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
| Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
| Pinterest | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `pinterest` | Pinterest items | 10K free calls |
| Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | Bluesky items | yes (app password at bsky.app) |
| TruthSocial | `TRUTHSOCIAL_TOKEN` | TruthSocial items | yes |
| Web search | one of: `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` | `--auto-resolve` and Step 2 supplements | Brave has a free tier; native WebSearch on Claude Code / Codex / Gemini works as a fallback |
| Perplexity Deep Research | `OPENROUTER_API_KEY` | `--deep-research` flag (~$0.90/query) | no |
| Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) |
**Example `.env` skeleton** (placeholders only - replace with your own values):
```bash
# Reasoning + planning (one provider; see priority below)
GOOGLE_API_KEY=<your-gemini-key>
# Web search backend (one is enough; Brave is the cheapest)
BRAVE_API_KEY=<your-brave-key>
# Optional sources
SCRAPECREATORS_API_KEY=<your-scrapecreators-key>
INCLUDE_SOURCES=tiktok,instagram
# X authentication (one option only)
XAI_API_KEY=<your-xai-key>
# OR cookie-jar (no key needed; logs in via your browser session)
# FROM_BROWSER=firefox
# Bluesky
BSKY_HANDLE=<your-handle>.bsky.social
BSKY_APP_PASSWORD=<your-app-password>
```
After editing: `chmod 600 ~/.config/last30days/.env` (or `chmod 600 .claude/last30days.env` if using the project-scoped variant).
**Troubleshooting:** if a source you expected to see isn't appearing in results, run `python3 scripts/last30days.py --diagnose`. It prints a per-source availability report (which keys were detected, which CLIs are installed, which backends are reachable) without running a full search.
### Bluesky app-password format and search host
`BSKY_APP_PASSWORD` should be a 19-char app password in `xxxx-xxxx-xxxx-xxxx` format (lowercase alphanumeric, three hyphens). Generate one at <https://bsky.app/settings/app-passwords>. The AT Protocol's `createSession` endpoint also accepts your main account login password, but that's bad hygiene — main passwords have no scope (an app password can be limited to non-DM access) and can't be revoked individually.
The skill defaults to `api.bsky.app` for `searchPosts`, which is the canonical authenticated AppView. The previous default `public.api.bsky.app` is the unauthenticated public mirror and is currently blocked by BunnyCDN for `searchPosts` regardless of auth header (verified 2026-05-04). If Bluesky migrates infrastructure again, override the host without a code change by setting `BSKY_SEARCH_HOST` in your `.env`:
```bash
BSKY_SEARCH_HOST=api.bsky.app # default — change only if Bluesky moves
```
---
## Reasoning provider priority
`/last30days` needs one reasoning model for planning + reranking when you don't pass `--plan` yourself. Auto-detect priority (set `LAST30DAYS_REASONING_PROVIDER=<name>` to pin one):
1. **Gemini** - `GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY`
2. **OpenAI** - `OPENAI_API_KEY` (or Codex auth at `~/.codex/auth.json`)
3. **xAI** - `XAI_API_KEY`
4. **OpenRouter** - `OPENROUTER_API_KEY` (also unlocks `--deep-research`)
5. **Local / deterministic** - always available, lowest quality
When you invoke `/last30days` from Claude Code, Codex, or Gemini, the host model **is** the reasoning provider for plan + synthesis - you don't need any of the keys above unless you also run the script headlessly (cron, CI, watchlist).
---
## Web search backend priority
Used by `--auto-resolve` (when WebSearch isn't available from the host) and Step 2 supplements. Auto-detect priority (override per-run with `--web-backend=<name>`):
1. **Brave** - `BRAVE_API_KEY`
2. **Exa** - `EXA_API_KEY`
3. **Serper** - `SERPER_API_KEY`
4. **Parallel** - `PARALLEL_API_KEY`
5. **Host's native WebSearch** - Claude Code, Codex, Gemini all have one built in
Visible quality difference between hosts with vs without a configured backend. If your client setup produces thinner results than yours, this is usually why.
---
## Trend monitoring (`--store` + watchlist + briefings)
The default behavior - one slug-named file per topic, overwritten on rerun - is the snapshot mode. For continuous monitoring, the repo ships three components most users miss:
### `--store` flag
Adding `--store` to any run persists every finding to a SQLite database (default at `~/.local/share/last30days/research.db`). Findings dedupe on the `source_url` column (UNIQUE constraint), so the same URL across runs updates the existing row instead of creating a duplicate. The markdown file still saves; the SQLite is the time-series substrate.
**Always-on alternative:** set `LAST30DAYS_STORE=1` in your `.env` instead of remembering `--store` on every invocation. The flag still works as before; the env var is purely additive. Same hybrid pattern as `LAST30DAYS_DEBUG` — works whether shell-exported or in `.env`.
Relevant tables: `topics`, `research_runs`, `findings`, `settings`. Schema: [`scripts/store.py`](skills/last30days/scripts/store.py).
### `watchlist.py` - recurring topics
[`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) manages topics that should be researched on a schedule. Subcommands: `add`, `remove`, `list`, `run-one`, `run-all`, `config`. Built-in delivery to Slack incoming webhooks (`hooks.slack.com/...`) or any HTTPS endpoint, fired only when new findings appear.
Two-step flow (the watchlist holds the topic; an external scheduler invokes the run):
```bash
# 1. Add the topic to the watchlist
# Default schedule daily 8am; --weekly switches to Mondays 8am
python3 scripts/watchlist.py add "british airways middle east" --weekly
# 2. Configure delivery and budget (optional)
python3 scripts/watchlist.py config delivery "https://hooks.slack.com/services/..."
python3 scripts/watchlist.py config budget 5.00
# 3. Trigger via cron / Task Scheduler / GitHub Actions
python3 scripts/watchlist.py run-one "british airways middle east"
# or run every enabled topic, gated by daily_budget
python3 scripts/watchlist.py run-all
```
The schedule field stored on each topic is metadata - the actual cron / Task Scheduler invocation is your responsibility. Watchlist runs hardcode `--quick` and `--lookback-days 90` when spawning the underlying engine.
### `briefing.py` - daily / weekly digests
[`scripts/briefing.py`](skills/last30days/scripts/briefing.py) reads the SQLite store and emits structured data the agent then synthesizes into prose. Modes: `generate` (daily), `generate --weekly`, `show [--date DATE]` (display a saved briefing). Briefs save to `~/.local/share/last30days/briefs/`.
### Recommended cadence pattern
| Step | Cadence | Command |
|---|---|---|
| Baseline | one-time per topic | `/last30days "<topic>" --days=30 --store` |
| Add to watchlist | one-time per topic | `python3 scripts/watchlist.py add "<topic>" --weekly` |
| Recurring run | daily or weekly (external scheduler) | `python3 scripts/watchlist.py run-all` |
| Digest | weekly | `python3 scripts/briefing.py generate --weekly` |
---
## Per-client patterns
The skill is built to flex around different client environments. Four patterns that compose well:
### 1. Per-client `.claude/last30days.env` (preferred when you cd into client folders)
The simplest pattern when each client has its own working directory: drop a `.claude/last30days.env` into the client folder. The skill picks it up automatically (see [API keys](#api-keys-env) for the lookup priority). Typical contents:
```bash
LAST30DAYS_MEMORY_DIR=C:\Users\<you>\Clients\acme\Research\Last30Days
SCRAPECREATORS_API_KEY=<acme-scoped-key-or-shared>
INCLUDE_SOURCES=tiktok,instagram
BSKY_HANDLE=<acme-bluesky-handle>.bsky.social
```
`cd` into the client folder, run `/last30days <topic>` as normal, no flags or wrappers. Combine with `--save-suffix=<client-slug>` per run if you also need to differentiate filenames within that folder.
### 2. Per-client save dir + suffix wrapper
For workflows where you don't `cd` into a client folder (running from anywhere, scripted batches), a tiny shell function isolates each client's research without engine changes.
PowerShell example:
```powershell
function Run-L30D-Client {
param([string]$ClientSlug, [Parameter(ValueFromRemainingArguments=$true)]$Args)
$env:LAST30DAYS_MEMORY_DIR = "C:\Users\$env:USERNAME\Clients\$ClientSlug\Research\Last30Days"
/last30days @Args --save-suffix=$ClientSlug
}
# Usage: Run-L30D-Client acme "british airways middle east"
```
Bash example:
```bash
l30d-client() {
local client=$1; shift
LAST30DAYS_MEMORY_DIR="$HOME/Clients/$client/Research/Last30Days" \
/last30days "$@" --save-suffix="$client"
}
# Usage: l30d-client acme "british airways middle east"
```
### 3. Custom category-peer subreddits
[`scripts/lib/categories.py`](skills/last30days/scripts/lib/categories.py) holds a table of `(category_id, trigger_keywords, peer_subreddits)`. If a client lives in a vertical that isn't covered (legal-tech, real-estate-tech, B2B HR SaaS), add a row. Pure data, no logic.
Section 2a of `SKILL.md` documents the merging rule the skill applies when your topic matches a category.
### 4. Pre-built `--competitors-plan` JSON
For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client industry saves real time:
```json
{
"Competitor B": {
"x_handle": "competitor_b_handle",
"subreddits": ["sub1", "sub2"],
"github_user": "competitor-b-org",
"context": "Founded 2019, focused on ..."
},
"Competitor C": { ... }
}
```
Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema.
---
## Beta channel
Experimental customizations live on a private companion repo (`mvanhorn/last30days-skill-private`) installed as `/last30days-beta`. Never ship beta-only changes to the public marketplace without a review PR against the public repo. Workflow guide: `BETA.md` in the private repo.
This is the right home for client-specific changes you don't intend to upstream - custom category rows, internal subreddit lists, per-vertical plan templates.
---
## Cross-references
- The CLI flag surface: `python3 scripts/last30days.py --help`
- The skill contract (voice, LAWs, pre-flight protocol): [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md)
- Engine spec (some sections stale; SKILL.md wins on conflicts): [`SPEC.md`](SPEC.md)
- Contributor guidance: [`CONTRIBUTORS.md`](CONTRIBUTORS.md)
+59
View File
@@ -0,0 +1,59 @@
# Contributors
last30days is built by [@mvanhorn](https://github.com/mvanhorn) with help from the community.
## v3 Inspiration
These contributors submitted PRs and issues that directly inspired v3 features. The v3 engine was a ground-up rewrite, so their original code wasn't merged, but their ideas shaped what shipped.
Want to claim your entry? Submit a PR replacing the placeholder line below your name with your bio, website, or anything you'd like.
---
### @uppinote20
[PR #143](https://github.com/mvanhorn/last30days-skill/pull/143) - Rich Reddit comments, top 3 per post
v3 ships top comments with upvote counts on every thread.
> _Add your bio, website, or anything you'd like here._
### @zerone0x
[Issue #134](https://github.com/mvanhorn/last30days-skill/issues/134) + [PR #136](https://github.com/mvanhorn/last30days-skill/pull/136) - GitHub as a first-class data source
v3 has full GitHub search: issues, PRs, person-mode profiles, project-mode repos with live star counts.
> _Add your bio, website, or anything you'd like here._
### @thinkun
[PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results
v3 has parallel enrichment with per-item timeouts. No results are ever dropped.
> Thinker, technologist, AI expert, music-tinkerer. Founder of [Thinkun](https://thinkun.com). [@thinkun on GitHub](https://github.com/thinkun) · [@unthink on X](https://x.com/unthink)
### @thomasmktong
[PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback
v3 Reddit is 100% pure Python with zero external dependencies.
> _Add your bio, website, or anything you'd like here._
### @fanispoulinakisai-boop
[Issue #100](https://github.com/mvanhorn/last30days-skill/issues/100) - Reddit timeout report
Drove the timeout resilience work that made v3 Reddit bulletproof.
> _Add your bio, website, or anything you'd like here._
### @pejmanjohn
[Issue #78](https://github.com/mvanhorn/last30days-skill/issues/78) - ScrapeCreators silent failures
v3 surfaces all API errors with clear diagnostics instead of silently returning empty results.
> Repping the mighty MI; home of the most cracked agentic engineers. https://github.com/pejmanjohn
### @zl190
[PR #115](https://github.com/mvanhorn/last30days-skill/pull/115) - HN trending merge
v3 merges trending and keyword HN results with deduplication for better coverage.
> Healthcare AI engineer. [Blog](https://zl190.github.io/blog)
### @hnshah
[PR #84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86) - Watchlist delivery, 90-day scanning window, HN/Polymarket storage
v3 has durable watchlist with multi-source storage and extended time windows.
> Hiten Shah. Founder. Builds in public. https://github.com/hnshah
---
## Past Contributors
- [@JosephOIbrahim](https://github.com/JosephOIbrahim) - Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- [@levineam](https://github.com/levineam) - Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- [@jonthebeef](https://github.com/jonthebeef) - Early testing and feedback
+111
View File
@@ -0,0 +1,111 @@
# Hermes Setup Guide for last30days
This guide covers installing last30days on Hermes AI Agent.
## Prerequisites
1. **Hermes installed** - See https://github.com/mercurial-tf/hermes
2. **Python 3.12+** - `brew install python@3.12` or similar
3. **yt-dlp** (optional, for YouTube) - `brew install yt-dlp`
## Installation
```bash
hermes skills install mvanhorn/last30days-skill --force
```
This pulls the latest release from GitHub and deploys to `~/.hermes/skills/research/last30days/`. `--force` reinstalls over any existing copy.
### Developer / live-edit alternative
If you're hacking on the skill locally and want edits to propagate to Hermes without re-installing, symlink your working tree:
```bash
git clone https://github.com/mvanhorn/last30days-skill.git
mkdir -p ~/.hermes/skills/research
ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.hermes/skills/research/last30days
```
## Usage
In Hermes, invoke with:
```
last30days "your research topic"
```
Or with options:
```
last30days "best mechanical keyboards 2025" --search=reddit,youtube
last30days "AI news" --days=7 --deep
```
## First Run Setup
On first run, the skill will guide you through setup:
1. **Auto setup** (~30 seconds)
- Scans browser cookies for X/Twitter
- Checks/installs yt-dlp for YouTube
- Configures free sources (Reddit, HN, Polymarket)
2. **Optional: ScrapeCreators**
- Adds TikTok, Instagram, Reddit backup
- 100 free credits (no expiration)
- Sign up at scrapecreators.com
3. **Optional: API Keys**
- XAI_API_KEY for X/Twitter (alternative to browser cookies)
- BRAVE_API_KEY for web search
## Available Sources
### Free (No API Key)
- **Reddit** - Public discussions and comments
- **Hacker News** - Tech discussions via Algolia
- **Polymarket** - Prediction markets
- **YouTube** - Search and transcripts (requires yt-dlp)
### Requires API Key
- **X/Twitter** - xAI API key or browser cookies
- **TikTok** - ScrapeCreators API
- **Instagram** - ScrapeCreators API
- **Web Search** - Brave Search API
## Troubleshooting
### Python not found
```bash
# Find Python 3.12+
which python3.12 python3.13 python3.14
# If not installed
brew install python@3.12
```
### yt-dlp not found
```bash
brew install yt-dlp
# or
pip install yt-dlp
```
### Check what's configured
```bash
cd ~/.hermes/skills/research/last30days
python3.12 scripts/last30days.py --diagnose
```
## Updating
```bash
hermes skills install mvanhorn/last30days-skill --force
```
If you symlinked your working tree (developer alternative above), just `git pull` in the repo — edits propagate live, no re-install step.
## Support
- Original repo: https://github.com/mvanhorn/last30days-skill
- Hermes: https://github.com/mercurial-tf/hermes
- Issues: Please report in the original repo
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2025 Peter Steinberger
Copyright (c) 2026 Matt Van Horn
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
@@ -19,4 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+284 -926
View File
File diff suppressed because it is too large Load Diff
-391
View File
@@ -1,391 +0,0 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
context: fork
agent: Explore
disable-model-invocation: true
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
-501
View File
@@ -1,501 +0,0 @@
---
name: last30days
version: "2.1"
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web. Become an expert and write copy-paste-ready prompts."
argument-hint: 'last30 AI video tools, last30 best project management tools'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill
user-invocable: true
disable-model-invocation: true
metadata:
clawdbot:
emoji: "📰"
requires:
env:
- OPENAI_API_KEY
bins:
- node
- python3
primaryEnv: OPENAI_API_KEY
files:
- "scripts/*"
homepage: https://github.com/mvanhorn/last30days-skill
tags:
- research
- reddit
- x
- youtube
- trends
- prompts
---
# last30days v2.1: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, YouTube, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
**DISPLAY your parsing to the user.** Before running any tools, output:
```
I'll research {TOPIC} across Reddit, X, and the web to find what's been discussed in the last 30 days.
Parsed intent:
- TOPIC = {TOPIC}
- TARGET_TOOL = {TARGET_TOOL or "unknown"}
- QUERY_TYPE = {QUERY_TYPE}
Research typically takes 2-8 minutes (niche topics take longer). Starting now.
```
If TARGET_TOOL is known, mention it in the intro: "...to find {QUERY_TYPE}-style content for use in {TARGET_TOOL}."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
---
## Research Execution
**Step 1: Run the research script (FOREGROUND — do NOT background this)**
**CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.**
```bash
# Find skill root — works in repo checkout, Claude Code, or Codex install
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
if [ -z "${SKILL_ROOT:-}" ]; then
echo "ERROR: Could not find scripts/last30days.py" >&2
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
```
Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically takes 1-3 minutes.
The script will automatically:
- Detect available API keys
- Run Reddit/X/YouTube searches
- Output ALL results including YouTube transcripts
**Read the ENTIRE output.** It contains THREE data sections in this order: Reddit items, X items, and YouTube items. If you miss the YouTube section, you will produce incomplete stats.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
---
## STEP 2: DO WEBSEARCH AFTER SCRIPT COMPLETES
After the script finishes, do WebSearch to supplement with blogs, tutorials, and news.
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Options** (passed through from user's command):
- `--days=N` → Look back N days instead of 30 (e.g., `--days=7` for weekly roundup)
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
3. Weight WebSearch sources LOWER (no engagement data)
4. Identify patterns that appear across ALL sources (strongest signals)
5. Note any contradictions between sources
6. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords?
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
---
## THEN: Show Summary + Invite Vision
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned with sources:
```
🏆 Most mentioned:
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle3, r/sub2, Complex
Notable mentions: [other specific things with 1-2 mentions]
```
**CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson)
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety)
- Parse @handles from research output and include the highest-engagement ones
- Format naturally - tables work well for wide terminals, stacked cards for narrow
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
CITATION PRIORITY (most to least preferred):
1. @handles from X — "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit — "per r/subreddit"
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
4. Web sources — ONLY when Reddit/X/YouTube don't cover that specific fact
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post.
URL FORMATTING: NEVER paste raw URLs in the output.
- **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- **GOOD:** "per Rolling Stone"
- **GOOD:** "per Complex"
Use the publication name, not the URL. The user doesn't need links — they need clean, readable text.
**BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
**GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per @honest30bgfan_"
**GOOD:** "Ye's apology got massive traction on r/hiphopheads"
**OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per Billboard"
**Lead with people, not publications.** Start each topic with what Reddit/X
users are saying/feeling, then add web context only if needed. The user came
here for the conversation, not the press release.
```
What I learned:
**{Topic 1}** — [1-2 sentences about what people are saying, per @handle or r/sub]
**{Topic 2}** — [1-2 sentences, per @handle or r/sub]
**{Topic 3}** — [1-2 sentences, per @handle or r/sub]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per @handle
```
**THEN - Stats (right before invitation):**
**CRITICAL: Calculate actual totals from the research output.**
- Count posts/threads from each section
- Sum engagement: parse `[Xlikes, Yrt]` from each X post, `[Xpts, Ycmt]` from Reddit
- Identify top voices: highest-engagement @handles from X, most active subreddits
**Copy this EXACTLY, replacing only the {placeholders}:**
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
├─ 🌐 Web: {N} pages (supplementary)
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
```
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely.
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**LAST - Invitation (adapt to QUERY_TYPE):**
**CRITICAL: Every invitation MUST include 2-3 specific example suggestions based on what you ACTUALLY learned from the research.** Don't be generic — show the user you absorbed the content by referencing real things from the results.
**If QUERY_TYPE = PROMPTING:**
```
---
I'm now an expert on {TOPIC} for {TARGET_TOOL}. What do you want to make? For example:
- [specific idea based on popular technique from research]
- [specific idea based on trending style/approach from research]
- [specific idea riffing on what people are actually creating]
Just describe your vision and I'll write a prompt you can paste straight into {TARGET_TOOL}.
```
**If QUERY_TYPE = RECOMMENDATIONS:**
```
---
I'm now an expert on {TOPIC}. Want me to go deeper? For example:
- [Compare specific item A vs item B from the results]
- [Explain why item C is trending right now]
- [Help you get started with item D]
```
**If QUERY_TYPE = NEWS:**
```
---
I'm now an expert on {TOPIC}. Some things you could ask:
- [Specific follow-up question about the biggest story]
- [Question about implications of a key development]
- [Question about what might happen next based on current trajectory]
```
**If QUERY_TYPE = GENERAL:**
```
---
I'm now an expert on {TOPIC}. Some things I can help with:
- [Specific question based on the most discussed aspect]
- [Specific creative/practical application of what you learned]
- [Deeper dive into a pattern or debate from the research]
```
**Example invitations (to show the quality bar):**
For `/last30days nano banana pro prompts for Gemini`:
> I'm now an expert on Nano Banana Pro for Gemini. What do you want to make? For example:
> - Photorealistic product shots with natural lighting (the most requested style right now)
> - Logo designs with embedded text (Gemini's new strength per the research)
> - Multi-reference style transfer from a mood board
>
> Just describe your vision and I'll write a prompt you can paste straight into Gemini.
For `/last30days kanye west` (GENERAL):
> I'm now an expert on Kanye West. Some things I can help with:
> - What's the real story behind the apology letter — genuine or PR move?
> - Break down the BULLY tracklist reactions and what fans are expecting
> - Compare how Reddit vs X are reacting to the Bianca narrative
For `/last30days war in Iran` (NEWS):
> I'm now an expert on the Iran situation. Some things you could ask:
> - What are the realistic escalation scenarios from here?
> - How is this playing differently in US vs international media?
> - What's the economic impact on oil markets so far?
---
## WAIT FOR USER'S RESPONSE
After showing the stats summary with your invitation, **STOP and wait** for the user to respond.
---
## WHEN USER RESPONDS
**Read their response and match the intent:**
- If they ask a **QUESTION** about the topic → Answer from your research (no new searches, no prompt)
- If they ask to **GO DEEPER** on a subtopic → Elaborate using your research findings
- If they describe something they want to **CREATE** → Write ONE perfect prompt (see below)
- If they ask for a **PROMPT** explicitly → Write ONE perfect prompt (see below)
**Only write a prompt when the user wants one.** Don't force a prompt on someone who asked "what could happen next with Iran."
### Writing a Prompt
When the user wants a prompt, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT.**
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Quality Checklist (run before delivering):
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS]
---
This uses [brief 1-line explanation of what research insight you applied].
```
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask a question** - answer it from your research findings
- **If they ask for a prompt** - write one using your expertise
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
---
## Security & Permissions
**What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
**What this skill does NOT do:**
- Does not post, like, or modify content on any platform
- Does not access your Reddit, X, or YouTube accounts
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
Review scripts before first use to verify behavior.
-75
View File
@@ -1,75 +0,0 @@
# last30days Skill Specification
## Overview
`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet.
The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
## Architecture
The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
- **env.py**: Load and validate API keys from `~/.config/last30days/.env`
- **dates.py**: Date range calculation and confidence scoring
- **cache.py**: 24-hour TTL caching keyed by topic + date range
- **http.py**: stdlib-only HTTP client with retry logic
- **models.py**: Auto-selection of OpenAI/xAI models with 7-day caching
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
- **xai_x.py**: xAI Responses API + x_search for X
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
- **normalize.py**: Convert raw API responses to canonical schema
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
- **dedupe.py**: Near-duplicate detection via text similarity
- **render.py**: Generate markdown and JSON outputs
- **schema.py**: Type definitions and validation
## Embedding in Other Skills
Other skills can import the research context in several ways:
### Inline Context Injection
```markdown
## Recent Research Context
!python3 ~/.claude/skills/last30days/scripts/last30days.py "your topic" --emit=context
```
### Read from File
```markdown
## Research Context
!cat ~/.local/share/last30days/out/last30days.context.md
```
### Get Path for Dynamic Loading
```bash
CONTEXT_PATH=$(python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=path)
cat "$CONTEXT_PATH"
```
### JSON for Programmatic Use
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=json > research.json
```
## CLI Reference
```
python3 ~/.claude/skills/last30days/scripts/last30days.py <topic> [options]
Options:
--refresh Bypass cache and fetch fresh data
--mock Use fixtures instead of real API calls
--emit=MODE Output mode: compact|json|md|context|path (default: compact)
--sources=MODE Source selection: auto|reddit|x|both (default: auto)
```
## Output Files
All outputs are written to `~/.local/share/last30days/out/`:
- `report.md` - Human-readable full report
- `report.json` - Normalized data with scores
- `last30days.context.md` - Compact reusable snippet for other skills
- `raw_openai.json` - Raw OpenAI API response
- `raw_xai.json` - Raw xAI API response
- `raw_reddit_threads_enriched.json` - Enriched Reddit thread data
-47
View File
@@ -1,47 +0,0 @@
# last30days Implementation Tasks
## Setup & Configuration
- [x] Create directory structure
- [x] Write SPEC.md
- [x] Write TASKS.md
- [x] Write SKILL.md with proper frontmatter
## Core Library Modules
- [x] scripts/lib/env.py - Environment and API key loading
- [x] scripts/lib/dates.py - Date range and confidence utilities
- [x] scripts/lib/cache.py - TTL-based caching
- [x] scripts/lib/http.py - HTTP client with retry
- [x] scripts/lib/models.py - Auto model selection
- [x] scripts/lib/schema.py - Data structures
- [x] scripts/lib/openai_reddit.py - OpenAI Responses API
- [x] scripts/lib/xai_x.py - xAI Responses API
- [x] scripts/lib/reddit_enrich.py - Reddit thread JSON fetcher
- [x] scripts/lib/normalize.py - Schema normalization
- [x] scripts/lib/score.py - Popularity scoring
- [x] scripts/lib/dedupe.py - Near-duplicate detection
- [x] scripts/lib/render.py - Output rendering
## Main Script
- [x] scripts/last30days.py - CLI orchestrator
## Fixtures
- [x] fixtures/openai_sample.json
- [x] fixtures/xai_sample.json
- [x] fixtures/reddit_thread_sample.json
- [x] fixtures/models_openai_sample.json
- [x] fixtures/models_xai_sample.json
## Tests
- [x] tests/test_dates.py
- [x] tests/test_cache.py
- [x] tests/test_models.py
- [x] tests/test_score.py
- [x] tests/test_dedupe.py
- [x] tests/test_normalize.py
- [x] tests/test_render.py
## Validation
- [x] Run tests in mock mode
- [x] Demo --emit=compact
- [x] Demo --emit=context
- [x] Verify file tree
+196
View File
@@ -0,0 +1,196 @@
# How Reddit & X Search Work in last30days
## Architecture Overview
```
User: /last30days "kanye west"
┌─────┴─────┐
↓ ↓ (concurrent via ThreadPoolExecutor)
[REDDIT] [X/TWITTER]
↓ ↓
OpenAI Bundled Bird or
API xAI API
↓ ↓
Parse Parse
↓ ↓
Enrich ───┘
(fetch ↓
actual [MERGE]
upvotes) ↓
↓ [NORMALIZE → FILTER → SCORE → DEDUPE]
└───────────↓
[OUTPUT to SKILL.md agent]
```
Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`.
---
## Reddit Search
### How it works
Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only.
**API Call:**
```
POST https://api.openai.com/v1/responses
Authorization: Bearer {OPENAI_API_KEY}
```
**Payload:**
```json
{
"model": "gpt-5.2",
"tools": [{
"type": "web_search",
"filters": { "allowed_domains": ["reddit.com"] }
}],
"input": "Search Reddit for threads about {topic}..."
}
```
The prompt asks the model to:
1. Extract core subject (strip noise words like "best", "tips", "top")
2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"`
3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance`
4. URLs must contain `/r/` AND `/comments/` (real threads only)
**Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini`
Triggers on HTTP 400/403 with access error keywords.
### Enrichment (the secret sauce)
After search, each thread gets **enriched** by hitting Reddit's free JSON API:
```
GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json
```
No API key needed. This returns the actual thread data:
| Data Point | Source |
|---|---|
| Upvotes (score) | Reddit JSON API |
| Comment count | Reddit JSON API |
| Upvote ratio | Reddit JSON API |
| Top 10 comments (text + score) | Reddit JSON API |
| 7 key comment insights | Extracted via heuristics |
| Actual post date | `created_utc` timestamp |
**This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates.
### Depth settings
| Depth | Threads requested | Timeout |
|---|---|---|
| `--quick` | 15-25 | 90s |
| default | 30-50 | 120s |
| `--deep` | 70-100 | 180s |
---
## X/Twitter Search
X search has **two backends** — the skill auto-detects which to use.
### Priority: Bundled Bird (env auth) → xAI API (paid)
```python
if node_available and AUTH_TOKEN and CT0:
use bundled Bird # Free, popup-free, env-authenticated
elif XAI_API_KEY:
use xAI API # Paid, uses grok-4-1-fast
else:
skip X entirely # No X results
```
### Backend 1: xAI API
**API Call:**
```
POST https://api.x.ai/v1/responses
Authorization: Bearer {XAI_API_KEY}
```
**Payload:**
```json
{
"model": "grok-4-1-fast",
"tools": [{ "type": "x_search" }],
"input": "Search X for posts about {topic} from {from_date} to {to_date}..."
}
```
The prompt asks grok to return JSON with:
- `text`, `url`, `author_handle`, `date`
- `engagement`: `{ likes, reposts, replies, quotes }`
- `why_relevant`, `relevance` score
**Engagement data comes from grok's x_search tool** - it has direct access to X's data.
### Backend 2: Bundled Bird client (free alternative)
The repo vendors a search-only subset of Bird's Twitter GraphQL client and shells out to it with Node.js. No global `bird` install is required. The Python wrapper passes `AUTH_TOKEN` and `CT0` via env, which keeps normal local runs headless and avoids browser-cookie prompts.
**Bundled Bird returns raw X API data** - likes, reposts, replies are real engagement metrics from X's API, not estimates.
| Metric | Bundled Bird | xAI API |
|---|---|---|
| Post text | Real | Real |
| Likes/reposts | Real (X API) | Real (x_search tool) |
| Replies/quotes | Real | Real |
| Author handle | Real | Real |
| Relevance score | Default 0.7 (re-ranked by relevance.py) | AI-assessed 0.0-1.0 |
### Depth settings
| Depth | xAI posts | Bundled Bird results | xAI timeout | Bird timeout |
|---|---|---|---|---|
| `--quick` | 8-12 | 12 | 90s | 30s |
| default | 20-30 | 30 | 120s | 45s |
| `--deep` | 40-60 | 60 | 180s | 60s |
---
## Post-Processing (both sources)
After both searches complete:
1. **Normalize** — consistent formatting, timezone handling
2. **Date filter** — hard filter to requested date range
3. **Score** — relevance scoring (engagement-weighted)
4. **Sort** — highest scores first
5. **Deduplicate** — remove duplicate URLs
6. **Fallback** — if all items filtered out, keep top 3 by relevance
---
## Error Handling
| Layer | Strategy |
|---|---|
| HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) |
| Model access errors | Automatic fallback to next model in chain |
| Reddit enrichment | Per-item try/catch; keeps unenriched item on failure |
| X source detection | Silent fallback from Bird → xAI → skip |
| Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user |
---
## Key Files
| File | Purpose |
|---|---|
| `skills/last30days/scripts/last30days.py` | Main CLI entry point |
| `skills/last30days/scripts/lib/pipeline.py` | Multi-source retrieval orchestration |
| `skills/last30days/scripts/lib/reddit_public.py` | Reddit public JSON search |
| `skills/last30days/scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `skills/last30days/scripts/lib/xai_x.py` | X search via xAI API |
| `skills/last30days/scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `skills/last30days/scripts/lib/providers.py` | Reasoning provider and model selection |
| `skills/last30days/scripts/lib/env.py` | API key loading, source detection |
| `skills/last30days/scripts/lib/http.py` | HTTP transport with retries |
| `skills/last30days/scripts/lib/relevance.py` | Query matching and relevance scoring |
| `skills/last30days/scripts/lib/dedupe.py` | URL-based deduplication |
@@ -1,929 +0,0 @@
# Bird CLI Integration Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add Bird CLI as a free, zero-config alternative to xAI for X/Twitter searches with interactive installation.
**Architecture:** New `bird_x.py` module handles Bird detection, installation prompts, and search. Modified `env.py` determines X source priority (Bird → xAI → WebSearch). Main script prompts for Bird install if not found.
**Tech Stack:** Python 3, subprocess for Bird CLI calls, existing lib modules for normalization/scoring.
---
### Task 1: Create bird_x.py - Detection Functions
**Files:**
- Create: `scripts/lib/bird_x.py`
**Step 1: Create the module with detection functions**
```python
"""Bird CLI client for X (Twitter) search."""
import json
import shutil
import subprocess
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[Bird] {msg}\n")
sys.stderr.flush()
def is_bird_installed() -> bool:
"""Check if Bird CLI is installed."""
return shutil.which("bird") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if Bird is authenticated by running 'bird whoami'.
Returns:
Username if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
try:
result = subprocess.run(
["bird", "whoami"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0 and result.stdout.strip():
# Output is typically the username
return result.stdout.strip().split('\n')[0]
return None
except (subprocess.TimeoutExpired, FileNotFoundError, Exception):
return None
def check_npm_available() -> bool:
"""Check if npm is available for installation."""
return shutil.which("npm") is not None
```
**Step 2: Verify the module loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add detection functions for Bird CLI"
```
---
### Task 2: Add Bird Installation Functions
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add installation function**
Add after `check_npm_available()`:
```python
def install_bird() -> Tuple[bool, str]:
"""Install Bird CLI via npm.
Returns:
Tuple of (success, message).
"""
if not check_npm_available():
return False, "npm not found. Install Node.js first, or install Bird manually: https://github.com/steipete/bird"
try:
_log("Installing Bird CLI...")
result = subprocess.run(
["npm", "install", "-g", "@steipete/bird"],
capture_output=True,
text=True,
timeout=120,
)
if result.returncode == 0:
return True, "Bird CLI installed successfully!"
else:
error = result.stderr.strip() or result.stdout.strip() or "Unknown error"
return False, f"Installation failed: {error}"
except subprocess.TimeoutExpired:
return False, "Installation timed out"
except Exception as e:
return False, f"Installation error: {e}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
username = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": username is not None,
"username": username,
"can_install": check_npm_available(),
}
```
**Step 2: Verify functions work**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.get_bird_status())"`
Expected: Dict with installed/authenticated status
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add installation and status functions"
```
---
### Task 3: Add Bird Search Function
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add depth config and search function**
Add after imports at top:
```python
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
```
Add after `get_bird_status()`:
```python
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
# Build command
cmd = [
"bird", "search",
topic,
"--since", from_date,
"-n", str(count),
"--json",
]
# Adjust timeout based on depth
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
)
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []}
# Parse JSON output
output = result.stdout.strip()
if not output:
return {"items": []}
return json.loads(output)
except subprocess.TimeoutExpired:
return {"error": "Search timed out", "items": []}
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
```
**Step 2: Verify search function signature**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; import inspect; print(inspect.signature(bird_x.search_x))"`
Expected: `(topic: str, from_date: str, to_date: str, depth: str = 'default') -> Dict[str, Any]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add search_x function"
```
---
### Task 4: Add Bird Response Parser
**Files:**
- Modify: `scripts/lib/bird_x.py`
**Step 1: Add parse function**
Add at end of file:
```python
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
screen_name = tweet.get("user", {}).get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first
if "T" in created_at:
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info
user = tweet.get("user", {})
author_handle = user.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict
engagement = {
"likes": tweet.get("like_count") or tweet.get("favorite_count"),
"reposts": tweet.get("retweet_count"),
"replies": tweet.get("reply_count"),
"quotes": tweet.get("quote_count"),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
}
items.append(item)
return items
```
**Step 2: Verify parser handles empty input**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print(bird_x.parse_bird_response({}))"`
Expected: `[]`
**Step 3: Commit**
```bash
git add scripts/lib/bird_x.py
git commit -m "feat(bird): add response parser matching xai_x format"
```
---
### Task 5: Add UI Functions for Bird Prompts
**Files:**
- Modify: `scripts/lib/ui.py`
**Step 1: Add Bird-related messages and prompts**
Add after `PROMO_SINGLE_KEY_PLAIN` dict (around line 128):
```python
# Bird CLI prompts
BIRD_INSTALL_PROMPT = f"""
{Colors.CYAN}{Colors.BOLD}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━{Colors.RESET}
{Colors.CYAN}🐦 FREE X/TWITTER SEARCH AVAILABLE{Colors.RESET}
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_INSTALL_PROMPT_PLAIN = """
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🐦 FREE X/TWITTER SEARCH AVAILABLE
Bird CLI provides free X search using your browser session (no API key needed).
"""
BIRD_AUTH_HELP = f"""
{Colors.YELLOW}Bird authentication failed.{Colors.RESET}
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: {Colors.BOLD}bird check{Colors.RESET} to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
BIRD_AUTH_HELP_PLAIN = """
Bird authentication failed.
To fix this:
1. Log into X (twitter.com) in Safari, Chrome, or Firefox
2. Run: bird check to verify credentials
3. Try again
For manual setup, see: https://github.com/steipete/bird#authentication
"""
```
**Step 2: Add prompt functions to ProgressDisplay class**
Add these methods to the `ProgressDisplay` class (after `show_promo` method, around line 310):
```python
def prompt_bird_install(self) -> bool:
"""Prompt user to install Bird CLI.
Returns:
True if user wants to install, False otherwise.
"""
if IS_TTY:
sys.stderr.write(BIRD_INSTALL_PROMPT)
else:
sys.stderr.write(BIRD_INSTALL_PROMPT_PLAIN)
sys.stderr.flush()
try:
response = input("Install Bird CLI now? (y/n): ").strip().lower()
return response in ('y', 'yes')
except (EOFError, KeyboardInterrupt):
return False
def show_bird_install_success(self, username: str):
"""Show Bird installation success message."""
msg = f"{Colors.GREEN}✓ Bird installed and authenticated as @{username}{Colors.RESET}\n" if IS_TTY else f"✓ Bird installed and authenticated as @{username}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_install_failed(self, error: str):
"""Show Bird installation failure message."""
msg = f"{Colors.RED}✗ Bird installation failed: {error}{Colors.RESET}\n" if IS_TTY else f"✗ Bird installation failed: {error}\n"
sys.stderr.write(msg)
sys.stderr.flush()
def show_bird_auth_help(self):
"""Show Bird authentication help."""
if IS_TTY:
sys.stderr.write(BIRD_AUTH_HELP)
else:
sys.stderr.write(BIRD_AUTH_HELP_PLAIN)
sys.stderr.flush()
```
**Step 3: Verify new methods exist**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib.ui import ProgressDisplay; p = ProgressDisplay('test', show_banner=False); print(hasattr(p, 'prompt_bird_install'))"`
Expected: `True`
**Step 4: Commit**
```bash
git add scripts/lib/ui.py
git commit -m "feat(ui): add Bird CLI install prompts and auth help"
```
---
### Task 6: Update env.py with X Source Detection
**Files:**
- Modify: `scripts/lib/env.py`
**Step 1: Add get_x_source function**
Add at end of file:
```python
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) → xAI (paid API)
Args:
config: Configuration dict from get_config()
Returns:
'bird' if Bird is installed and authenticated,
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
from . import bird_x
# Check Bird first (free option)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return None
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"can_install_bird": bird_status["can_install"],
}
```
**Step 2: Verify function works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; print(env.get_x_source_status(env.get_config()))"`
Expected: Dict with source status
**Step 3: Commit**
```bash
git add scripts/lib/env.py
git commit -m "feat(env): add X source detection with Bird priority"
```
---
### Task 7: Update __init__.py to Export bird_x
**Files:**
- Modify: `scripts/lib/__init__.py`
**Step 1: Add bird_x to imports**
Replace file contents with:
```python
# last30days library modules
from . import bird_x
```
**Step 2: Verify import works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import bird_x; print('OK')"`
Expected: `OK`
**Step 3: Commit**
```bash
git add scripts/lib/__init__.py
git commit -m "feat(lib): export bird_x module"
```
---
### Task 8: Integrate Bird into Main Script - Part 1 (Setup Phase)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Add bird_x import**
Add `bird_x` to the imports from lib (around line 36):
```python
from lib import (
bird_x,
dates,
dedupe,
env,
http,
models,
normalize,
openai_reddit,
reddit_enrich,
render,
schema,
score,
ui,
websearch,
xai_x,
)
```
**Step 2: Add Bird setup function**
Add after the imports, before `load_fixture`:
```python
def setup_bird_if_needed(progress: ui.ProgressDisplay) -> Optional[str]:
"""Check Bird status and offer installation if needed.
Returns:
'bird' if Bird is ready to use,
'declined' if user declined install,
None if Bird not available and couldn't be installed.
"""
status = bird_x.get_bird_status()
# Already working
if status["authenticated"]:
return 'bird'
# Installed but not authenticated
if status["installed"]:
progress.show_bird_auth_help()
return None
# Not installed - offer to install if npm available
if status["can_install"]:
if progress.prompt_bird_install():
success, message = bird_x.install_bird()
if success:
# Check if auth works now
username = bird_x.is_bird_authenticated()
if username:
progress.show_bird_install_success(username)
return 'bird'
else:
progress.show_bird_auth_help()
return None
else:
progress.show_bird_install_failed(message)
return None
else:
return 'declined'
return None
```
**Step 3: Verify script still loads**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "import scripts.last30days; print('OK')"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): add Bird setup function"
```
---
### Task 9: Integrate Bird into Main Script - Part 2 (Search Dispatch)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Modify _search_x function to support Bird**
Replace the `_search_x` function (around line 119-159) with:
```python
def _search_x(
topic: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str,
mock: bool,
x_source: str = "xai",
) -> tuple:
"""Search X via Bird CLI or xAI (runs in thread).
Args:
x_source: 'bird' or 'xai' - which backend to use
Returns:
Tuple of (x_items, raw_response, error)
"""
raw_response = None
x_error = None
if mock:
raw_response = load_fixture("xai_sample.json")
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
# Use Bird if specified
if x_source == "bird":
try:
raw_response = bird_x.search_x(
topic,
from_date,
to_date,
depth=depth,
)
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = bird_x.parse_bird_response(raw_response or {})
# Check for error in response
if raw_response and raw_response.get("error") and not x_error:
x_error = raw_response["error"]
return x_items, raw_response, x_error
# Use xAI (original behavior)
try:
raw_response = xai_x.search_x(
config["XAI_API_KEY"],
selected_models["xai"],
topic,
from_date,
to_date,
depth=depth,
)
except http.HTTPError as e:
raw_response = {"error": str(e)}
x_error = f"API error: {e}"
except Exception as e:
raw_response = {"error": str(e)}
x_error = f"{type(e).__name__}: {e}"
x_items = xai_x.parse_x_response(raw_response or {})
return x_items, raw_response, x_error
```
**Step 2: Update run_research to accept x_source parameter**
Find the `run_research` function signature (around line 161) and add `x_source` parameter:
```python
def run_research(
topic: str,
sources: str,
config: dict,
selected_models: dict,
from_date: str,
to_date: str,
depth: str = "default",
mock: bool = False,
progress: ui.ProgressDisplay = None,
x_source: str = "xai",
) -> tuple:
```
Then update the `_search_x` call inside (around line 218-222) to pass `x_source`:
```python
x_future = executor.submit(
_search_x, topic, config, selected_models,
from_date, to_date, depth, mock, x_source
)
```
**Step 3: Verify script syntax is valid**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -m py_compile scripts/last30days.py && echo "OK"`
Expected: `OK`
**Step 4: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): dispatch X search to Bird or xAI"
```
---
### Task 10: Integrate Bird into Main Script - Part 3 (Main Function)
**Files:**
- Modify: `scripts/last30days.py`
**Step 1: Update main() to check Bird before research**
In the `main()` function, after loading config and before checking available sources (around line 345-355), add Bird setup:
Find this section:
```python
# Load config
config = env.get_config()
# Check available sources
available = env.get_available_sources(config)
```
Replace with:
```python
# Load config
config = env.get_config()
# Initialize progress display early for Bird prompts
progress = ui.ProgressDisplay(args.topic, show_banner=True)
# Check Bird availability and offer install if needed
x_source_status = env.get_x_source_status(config)
x_source = x_source_status["source"]
# If no X source and Bird can be installed, offer it
if x_source is None and x_source_status["can_install_bird"]:
bird_result = setup_bird_if_needed(progress)
if bird_result == 'bird':
x_source = 'bird'
# Refresh status
x_source_status = env.get_x_source_status(config)
# Check available sources (now accounting for Bird)
available = env.get_available_sources(config)
# Override available if Bird is ready
if x_source == 'bird':
if available == 'reddit':
available = 'both' # Now have both Reddit + X (via Bird)
elif available == 'web':
available = 'x' # Now have X via Bird
```
**Step 2: Remove duplicate progress initialization**
Find and remove the later `progress = ui.ProgressDisplay(...)` line (around line 371) since we now create it earlier.
**Step 3: Pass x_source to run_research**
Find the `run_research` call (around line 413) and add `x_source` parameter:
```python
reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error = run_research(
args.topic,
sources,
config,
selected_models,
from_date,
to_date,
depth,
args.mock,
progress,
x_source=x_source or "xai",
)
```
**Step 4: Verify script runs with --help**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py --help`
Expected: Help text displays without errors
**Step 5: Commit**
```bash
git add scripts/last30days.py
git commit -m "feat(main): integrate Bird setup into main flow"
```
---
### Task 11: Test End-to-End with Mock Mode
**Files:**
- None (testing only)
**Step 1: Test mock mode still works**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 scripts/last30days.py "Claude Code" --mock --emit=compact 2>&1 | head -20`
Expected: Output showing research results without errors
**Step 2: Test Bird detection (informational)**
Run: `cd /Users/mvanhorn/last30days-skill-private && python3 -c "from scripts.lib import env; import json; print(json.dumps(env.get_x_source_status(env.get_config()), indent=2))"`
Expected: JSON showing current Bird/xAI status
**Step 3: Commit any fixes if needed, then final commit**
```bash
git add -A
git commit -m "feat(bird): complete Bird CLI integration
- Add bird_x.py module for Bird CLI detection, install, and search
- Add UI prompts for interactive Bird installation
- Update env.py with X source priority (Bird > xAI)
- Integrate Bird into main research flow
- Bird uses browser cookies (free, no API key needed)"
```
---
### Task 12: Push to Private Repo
**Files:**
- None (git only)
**Step 1: Push all changes**
Run: `cd /Users/mvanhorn/last30days-skill-private && git push origin main`
Expected: Changes pushed to private repo
**Step 2: Verify commit history**
Run: `cd /Users/mvanhorn/last30days-skill-private && git log --oneline -10`
Expected: Shows Bird integration commits
---
## Summary
After completing all tasks, the skill will:
1. Check if Bird CLI is installed on startup
2. If not installed but npm available, prompt user to install
3. If installed, verify authentication via `bird whoami`
4. If authenticated, use Bird for all X searches (free)
5. If not, fall back to xAI (if key exists) or WebSearch
6. Output format identical regardless of backend used
@@ -1,102 +0,0 @@
# Bird CLI Integration Design
**Date:** 2026-02-03
**Status:** Approved
## Overview
Add Bird CLI as an alternative X/Twitter search source for the last30days skill. Bird uses browser cookie authentication (free, no API key) and provides direct access to X's GraphQL API.
## Goals
- Provide free X search without requiring xAI API key
- Seamless fallback: Bird → xAI → WebSearch
- Interactive onboarding for users without Bird installed
- Output parity with existing xAI implementation
## Detection & Priority Flow
```
On startup:
1. Check: Is Bird installed? (`which bird`)
├─ No → Offer to install: "Bird CLI not found. Install for free X search? (y/n)"
│ ├─ Yes → Run `npm install -g @steipete/bird`
│ └─ No → Continue to step 2
└─ Yes → Check: Is Bird authenticated? (`bird whoami`)
├─ Success → Use Bird for X searches
└─ Fail → Show: "Bird auth failed. Run `bird check` to diagnose."
Continue to step 2
2. Fall back to xAI if XAI_API_KEY exists
3. Fall back to WebSearch if nothing else available
```
**Priority order:** Bird → xAI → WebSearch
## New Module: `scripts/lib/bird_x.py`
### Functions
- `is_bird_installed()` → checks `which bird`, returns bool
- `is_bird_authenticated()` → runs `bird whoami`, returns username or None
- `install_bird()` → runs `npm install -g @steipete/bird`, returns success bool
- `search_x(topic, from_date, to_date, depth)` → runs `bird search` with JSON output
- `parse_bird_response(json)` → converts to same format as `xai_x.parse_x_response()`
### Search Command
```bash
bird search "Claude Code skills" --since 2026-01-04 -n 30 --json
```
- `--since` filters to last 30 days
- `-n 30` controls result count (maps to depth: quick=12, default=30, deep=60)
- `--json` gives machine-readable output
### Output Mapping
| Bird field | Our field |
|------------|-----------|
| `text` | `text` |
| `permanent_url` | `url` |
| `user.screen_name` | `author_handle` |
| `created_at` | `date` (parse to YYYY-MM-DD) |
| `like_count` | `engagement.likes` |
| `retweet_count` | `engagement.reposts` |
| `reply_count` | `engagement.replies` |
| `quote_count` | `engagement.quotes` |
Relevance: Default to 0.7, let `score.py` re-rank based on engagement.
## Modified Files
| File | Change |
|------|--------|
| `env.py` | Add `get_x_source()` → returns `'bird'`, `'xai'`, or `None` |
| `last30days.py` | Check Bird availability with interactive install prompt before research |
| `last30days.py` | In `_search_x()`, dispatch to `bird_x` or `xai_x` based on source |
| `ui.py` | Add `prompt_bird_install()` and `show_bird_auth_help()` |
## Unchanged Files
- `normalize.py` - Bird output matches xAI format after parsing
- `score.py` - Same scoring logic applies
- `dedupe.py` - Same deduplication logic
- `render.py` - X results labeled as "X" regardless of backend
## Error Handling
| Scenario | Behavior |
|----------|----------|
| Bird installed but no browser cookies | Show `bird check` guidance, fall back to xAI |
| Bird search returns 0 results | Retry with simplified query (same as xAI logic) |
| Bird search times out | Fall back to xAI if available, else WebSearch |
| npm not installed (can't install Bird) | Skip Bird, continue with xAI/WebSearch |
| User declines Bird install | Remember for session, don't ask again |
**Timeout:** 30 seconds for Bird commands
## Output Labels
Results labeled as "X" regardless of whether Bird or xAI was used. Users care about the data, not the backend.
@@ -1,391 +0,0 @@
---
title: "feat: Release last30days v2 with Bird CLI to GitHub"
type: feat
date: 2026-02-06
---
# Release last30days v2 (Bird CLI) to GitHub
## Overview
Replace the current public `last30days-skill` on GitHub with the new Bird CLI-enhanced version from `last30days-skill-private`. The new version adds free X/Twitter search via Bird CLI while maintaining backward compatibility with xAI API keys.
**Goal:** Ship with confidence. No rollbacks.
## Current State
| | Old (Public) | New (Private) |
|---|---|---|
| **Repo** | `mvanhorn/last30days-skill` | `mvanhorn/last30days-skill-private` |
| **Local path** | `~/.claude/skills/last30days/` | `~/.claude/skills/last30daystest/` (symlink) |
| **Remote** | `origin` → public repo | `origin` → private, `upstream` → public |
| **Key addition** | -- | Bird CLI (`@steipete/bird`) for free X search |
| **X source chain** | xAI API only | Bird (free) → xAI (paid) → WebSearch |
| **Uses** | 136 | 18 |
| **Latest commit** | `cc892d7` | `4230fa2` |
**Why both show as `/last30days`:** Both `SKILL.md` files declare `name: last30days` in frontmatter. Claude Code discovers both from `~/.claude/skills/` and lists them separately.
---
## Phase 0: Clean Swap (Day 1)
Remove the old skill so only the new one is active. This eliminates ambiguity during testing.
### Steps
1. **Back up the old skill** (safety net):
```bash
mv ~/.claude/skills/last30days ~/.claude/skills/last30days.backup-v1
```
2. **Promote the new skill to primary**:
```bash
# Remove the test symlink
rm ~/.claude/skills/last30daystest
# Create new symlink with the primary name
ln -s /Users/mvanhorn/last30days-skill-private ~/.claude/skills/last30days
```
3. **Verify only one `/last30days` appears**:
- Open a new Claude Code session
- Type `/last` and confirm only ONE `/last30days` shows in autocomplete
- Confirm description mentions Bird CLI
4. **Rollback procedure** (if something goes wrong):
```bash
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
```
### Acceptance Criteria
- [ ] Only one `/last30days` appears in Claude Code autocomplete
- [ ] Old skill preserved at `~/.claude/skills/last30days.backup-v1`
- [ ] New skill responds to `/last30days` invocation
---
## Phase 1: Claude's Test Plan (Automated)
These are tests Claude can run autonomously to validate the new skill before the user touches it.
### 1.1 Script-Level Smoke Tests
Run the Python scripts directly to verify core functionality without invoking the full skill.
#### Bird CLI Detection
```bash
# Test: Bird is installed and authenticated
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
print('installed:', bird_x.is_bird_installed())
print('authenticated:', bird_x.is_bird_authenticated())
print('status:', bird_x.get_bird_status())
"
```
- [ ] `is_bird_installed()` returns True (or False with clear message)
- [ ] `is_bird_authenticated()` returns True if logged into X in browser
- [ ] `get_bird_status()` returns a dict with `installed`, `authenticated`, `available` keys
#### Environment & Source Detection
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source:', env.get_x_source(config))
print('has_openai:', bool(config.get('OPENAI_API_KEY')))
"
```
- [ ] `get_x_source()` returns `'bird'` if Bird available, `'xai'` if API key set, `None` otherwise
- [ ] Config loads from `~/.config/last30days/.env`
#### Bird Search (Direct)
```bash
python3 -c "
import sys, json; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
result = bird_x.search_x('Claude Code tips', '2026-01-07', '2026-02-06', 'quick')
print(json.dumps(result, indent=2, default=str)[:2000])
"
```
- [ ] Returns search results (list of dicts with `url`, `text`, `author_handle`)
- [ ] No Python tracebacks
- [ ] Results are from the expected date range
#### Full Research Pipeline (Compact Output)
```bash
cd /Users/mvanhorn/last30days-skill-private
python3 scripts/last30days.py "Claude Code tips" --emit=compact --quick 2>&1 | head -100
```
- [ ] Completes without error
- [ ] Output includes X results (via Bird or xAI)
- [ ] Output includes Reddit results (via OpenAI) if key configured
- [ ] Stats summary shows source counts
### 1.2 Source Fallback Tests
Verify graceful degradation when sources are unavailable.
#### Bird unavailable, xAI available
```bash
# Temporarily hide Bird
PATH_BACKUP="$PATH"
export PATH=$(echo "$PATH" | tr ':' '\n' | grep -v "$(dirname $(which bird 2>/dev/null))" | tr '\n' ':')
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = env.load_config()
print('x_source (no bird):', env.get_x_source(config))
"
export PATH="$PATH_BACKUP"
```
- [ ] Falls back to `'xai'` when Bird not in PATH
- [ ] No crash or unhandled exception
#### No X source at all
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import env
config = {} # empty config, no keys
print('x_source (nothing):', env.get_x_source(config))
"
```
- [ ] Returns `None`
- [ ] No crash
### 1.3 Response Parsing Tests
Validate that Bird responses are correctly normalized to the canonical schema.
```bash
python3 -c "
import sys; sys.path.insert(0, '/Users/mvanhorn/last30days-skill-private/scripts/lib')
import bird_x
# Test with sample Bird response format
sample = {
'tweets': [{
'permanentUrl': 'https://x.com/user/status/123',
'text': 'Test tweet about Claude Code',
'username': 'testuser',
'likeCount': 42,
'retweetCount': 10,
'replyCount': 5,
'timeParsed': '2026-02-01T12:00:00.000Z'
}]
}
parsed = bird_x.parse_bird_response(sample)
print('Parsed count:', len(parsed))
print('First item keys:', sorted(parsed[0].keys()) if parsed else 'EMPTY')
print('URL:', parsed[0].get('url'))
print('Author:', parsed[0].get('author_handle'))
"
```
- [ ] Parses correctly with expected keys
- [ ] Handles both camelCase and snake_case fields
- [ ] URL, text, author, engagement metrics all present
### 1.4 SKILL.md Validation
```bash
# Verify YAML frontmatter parses correctly
python3 -c "
import yaml
with open('/Users/mvanhorn/last30days-skill-private/SKILL.md') as f:
content = f.read()
# Extract YAML between --- markers
parts = content.split('---', 2)
meta = yaml.safe_load(parts[1])
print('name:', meta.get('name'))
print('context:', meta.get('context'))
print('agent:', meta.get('agent'))
print('allowed-tools:', meta.get('allowed-tools'))
"
```
- [ ] `name` is `last30days` (not `last30daystest`)
- [ ] `context` is `fork`
- [ ] `agent` is `Explore`
- [ ] `allowed-tools` includes `Bash`, `WebSearch`
### 1.5 Diff Audit (Old vs New)
```bash
# Verify the only meaningful addition is bird_x.py
diff -rq ~/.claude/skills/last30days.backup-v1/scripts/lib/ \
/Users/mvanhorn/last30days-skill-private/scripts/lib/ 2>/dev/null
```
- [ ] Only new file is `bird_x.py`
- [ ] Modified files: `env.py` (source detection), `__init__.py` (exports)
- [ ] No unexpected deletions or renames
---
## Phase 2: User's Test Plan (Manual)
These require human judgment - evaluating quality, UX, and real-world behavior.
### 2.1 Basic Invocation (5 min)
Open a fresh Claude Code session after Phase 0 is complete.
| # | Test | Command | Pass Criteria |
|---|------|---------|---------------|
| 1 | Simple topic | `/last30days AI music generation` | Returns results, shows source stats |
| 2 | Topic + tool | `/last30days Suno prompts for music production` | Returns results + generates a prompt |
| 3 | Quick mode | `/last30days --quick TypeScript tips` | Faster, fewer results, still valid |
| 4 | Empty input | `/last30days` | Prompts for topic (doesn't crash) |
### 2.2 Bird CLI Verification (5 min)
| # | Test | What to Check |
|---|------|---------------|
| 1 | Source indicator | Output shows Bird as X source (not xAI) |
| 2 | X results quality | X/Twitter results are real, recent, have engagement metrics |
| 3 | Bird promo | If Bird NOT installed, shows non-blocking info banner |
| 4 | Mixed sources | Both Reddit (OpenAI) and X (Bird) results appear |
### 2.3 Fallback Behavior (5 min)
| # | Test | Setup | Expected |
|---|------|-------|----------|
| 1 | No Bird | `npm uninstall -g @steipete/bird` temporarily | Falls back to xAI or WebSearch |
| 2 | No API keys | Rename `~/.config/last30days/.env` temporarily | WebSearch-only mode works |
| 3 | Restore | Reinstall bird + restore .env | Full mode returns |
### 2.4 Output Quality (10 min)
Run 3 real research queries you care about. For each, evaluate:
- [ ] Results are actually from the last 30 days (not stale)
- [ ] Engagement metrics (likes, upvotes) are present and reasonable
- [ ] No duplicate results
- [ ] Sources are properly cited with URLs
- [ ] Synthesis is grounded in actual results (not hallucinated)
- [ ] Generated prompts (if requested) are usable
### 2.5 Comparison Test (10 min)
Before removing the backup, run the SAME query on both versions:
```bash
# New version (active)
/last30days [your topic]
# Old version (temporarily restore)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# New Claude Code session
/last30days [same topic]
# Then swap back
```
- [ ] New version produces equal or better results
- [ ] No features regressed
- [ ] Bird results add value over xAI-only
---
## Phase 3: Release Plan (30-Day Timeline)
**Target release date:** March 1, 2026 (conservative buffer before March 8 deadline)
### Week 1: Feb 6-12 - Clean & Test
| Day | Task | Owner |
|-----|------|-------|
| Feb 6 | Phase 0: Clean swap (remove old, activate new) | User |
| Feb 6 | Phase 1: Claude runs automated tests | Claude |
| Feb 7-8 | Phase 2: User runs manual tests (2.1-2.4) | User |
| Feb 9 | Phase 2.5: Comparison test | User |
| Feb 10-12 | Fix any issues found during testing | Claude + User |
### Week 2: Feb 13-19 - Harden
| Day | Task | Owner |
|-----|------|-------|
| Feb 13 | Run edge cases: unicode topics, very long topics, special chars | Claude |
| Feb 14 | Test with Bird logged out (auth expiry scenario) | User |
| Feb 15 | Review all error messages for clarity | Claude |
| Feb 16-17 | Update README.md with Bird CLI setup instructions | Claude |
| Feb 18-19 | Buffer for fixes | Claude + User |
### Week 3: Feb 20-26 - Pre-Release
| Day | Task | Owner |
|-----|------|-------|
| Feb 20 | Final diff audit: private repo vs public repo | Claude |
| Feb 21 | Strip any private/test artifacts (test symlinks, debug prints) | Claude |
| Feb 22 | Update SKILL.md description if needed | Claude |
| Feb 23 | Dry-run: push to a branch on public repo (not main) | User |
| Feb 24 | Test installation from the branch (fresh `~/.claude/skills/`) | User |
| Feb 25-26 | Buffer for fixes | Claude + User |
### Week 4: Feb 27 - Mar 1 - Ship
| Day | Task | Owner |
|-----|------|-------|
| Feb 27 | Merge branch to main on public repo | User |
| Feb 28 | Create GitHub release with changelog | Claude + User |
| Mar 1 | Delete backup: `rm -rf ~/.claude/skills/last30days.backup-v1` | User |
| Mar 1 | Archive private repo (optional) | User |
### Release Checklist (Final Gate)
Before merging to `main` on the public repo:
- [ ] All Phase 1 automated tests pass
- [ ] All Phase 2 manual tests pass
- [ ] Comparison test shows new >= old quality
- [ ] SKILL.md frontmatter is correct (`name: last30days`, not `last30daystest`)
- [ ] README.md documents Bird CLI setup
- [ ] No debug/test artifacts in codebase
- [ ] No hardcoded paths (e.g., `/Users/mvanhorn/...`)
- [ ] `.env` files are gitignored
- [ ] Git history is clean (no "test" or "WIP" commits on main)
- [ ] Bird CLI failure doesn't break the skill (graceful fallback verified)
### Rollback Plan (Emergency)
If something goes wrong after release:
```bash
# Option 1: Revert to backup (if still exists)
rm ~/.claude/skills/last30days
mv ~/.claude/skills/last30days.backup-v1 ~/.claude/skills/last30days
# Option 2: Git revert on public repo
cd ~/.claude/skills/last30days
git log --oneline -5 # find the last good commit
git revert HEAD # revert the merge commit
git push origin main
# Option 3: Pin to old version
cd ~/.claude/skills/last30days
git checkout cc892d7 # last known good commit from old version
```
---
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Bird CLI breaks after X API changes | Medium | Low | Fallback to xAI/WebSearch still works |
| Bird auth expires silently | Medium | Low | `is_bird_authenticated()` check + user message |
| Old xAI workflows regress | Low | High | Comparison test in Phase 2.5 |
| Hardcoded paths in codebase | Low | Medium | Grep for `/Users/mvanhorn` before release |
| SKILL.md name still says `last30daystest` | Low | High | Already fixed in commit `4e972d0` |
## References
- Private repo: `https://github.com/mvanhorn/last30days-skill-private.git`
- Public repo: `https://github.com/mvanhorn/last30days-skill.git`
- Bird CLI: `https://github.com/steipete/bird`
- Bird implementation plan: `docs/plans/2026-02-03-bird-cli-implementation.md`
- Bird integration design: `docs/plans/2026-02-03-bird-cli-integration-design.md`
@@ -1,91 +0,0 @@
---
title: "feat: Add visible query parsing display before research starts"
type: feat
date: 2026-02-06
---
# feat: Add Visible Query Parsing Display
## Overview
The last30days skill parses user intent (TOPIC, QUERY_TYPE, TARGET_TOOL) internally but never shows the user what it understood. The agent jumps straight from the user's `/last30days kanye west` into running tools with a generic "I'll start the research script and web searches in parallel."
Users expect to see a reformulation of their query — confirming what the agent understood before it starts searching. This builds trust and lets users course-correct before waiting for results.
## Problem Statement
Current behavior:
```
User: /last30days kanye west
Agent: I'll start the research script and web searches in parallel.
[immediately runs bash + WebSearch]
```
Expected behavior:
```
User: /last30days kanye west
Agent: 🔍 **kanye west** · News
Searching Reddit, X, and the web for the latest on kanye west...
[then runs bash + WebSearch]
```
The "Parse User Intent" section in SKILL.md tells the agent to store variables internally but never instructs it to **display** them.
## Proposed Solution
Add an explicit "Display your parsing" instruction between the "Parse User Intent" section and "Research Execution" section in SKILL.md. One new block of text — no code changes, no script changes.
## Acceptance Criteria
- [ ] Agent displays parsed TOPIC and QUERY_TYPE before running any tools
- [ ] Display is concise (1-2 lines, not a verbose block)
- [ ] Agent still runs script + WebSearch in parallel after displaying
- [ ] No changes to Python scripts — SKILL.md only
## Implementation
### SKILL.md Change
**File:** `/Users/mvanhorn/last30days-skill-private/SKILL.md`
After the "Store these variables" block (line ~38) and before "Research Execution" (line ~42), add:
```markdown
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
```
### Sync
After editing SKILL.md:
```bash
cp /Users/mvanhorn/last30days-skill-private/SKILL.md ~/.claude/skills/last30days/SKILL.md
```
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should display: 🔍 **kanye west** · News
2. `/last30days best MCP servers` — should display: 🔍 **best MCP servers** · Recommendations
3. `/last30days nano banana pro prompting for ChatGPT` — should display with tool mention
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Add display instruction between Parse User Intent and Research Execution |
@@ -1,167 +0,0 @@
---
title: "fix: last30days v2 formatting, Reddit results, and citation verbosity"
type: fix
date: 2026-02-06
---
# fix: last30days v2 Formatting, Reddit Results, and Citation Verbosity
## Overview
Four bugs found during v2 testing across 4 queries (kanye west, howie.ai, nano banana pro prompting, open claw). The skill IS executing (the agent:Explore removal worked) but output quality has regressed from v1.
## Problem Statement
| # | Bug | Severity | Where |
|---|-----|----------|-------|
| 1 | Stats emoji tree format ignored 3/4 times - agent renders plain text dashes instead | High | `SKILL.md` |
| 2 | Reddit returns 0 results for popular topics (kanye west, howie.ai) | High | `scripts/lib/openai_reddit.py` |
| 3 | Citations too verbose - every sentence has `(per @x, @y, @z; r/sub)` making summary unreadable | Medium | `SKILL.md` |
| 4 | Kanye summary is wall of text - no bold headers or paragraph breaks like nano banana pro got | Medium | `SKILL.md` |
## Proposed Fixes
### Fix 1: Stats Emoji Format Enforcement
**Root cause:** The agent ignores the emoji tree template even with BAD/GOOD examples. The template uses box-drawing characters (├─ └─) that the agent treats as decorative, not mandatory.
**Approach:** Instead of relying on the agent to copy box-drawing characters, provide the template as a **literal fill-in-the-blank** with placeholders that are impossible to misinterpret.
**File:** `SKILL.md` (stats section, currently around line 190)
**Change:** Replace the current template + BAD/GOOD examples with a single, strict fill-in format:
```
Copy this EXACTLY, replacing only the {placeholders}:
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
```
Remove the separate BAD/GOOD section (it adds length without helping).
### Fix 2: Reddit Returning 0 Results
**Root cause (from code analysis):**
1. `openai_reddit.py:53-93` - The `REDDIT_SEARCH_PROMPT` instructs the OpenAI model to strip noise words before searching. For "kanye west" this isn't the issue (no noise words), but for "howie.ai" it might strip "ai".
2. `openai_reddit.py:160-166` - The search is restricted to `allowed_domains: ["reddit.com"]` which depends on OpenAI's web_search indexing of Reddit.
3. `last30days.py:474-490` - Post-retrieval filtering: `normalize.filter_by_date_range()` + `score.score_reddit_items()` + `dedupe.dedupe_reddit()` can discard all results if date confidence is low.
4. `score.py:151-157` - Items with no engagement metrics get `-10` penalty, low date confidence gets `-10`. Combined that's `-20` which may push score below threshold.
**Approach (multi-layered):**
**A. Add subreddit-targeted search fallback** in `openai_reddit.py`:
- When the first search returns < 3 results, add a second search prompt that explicitly queries: `"r/{topic} site:reddit.com"` and `"{topic} subreddit site:reddit.com"`
- This catches cases where OpenAI's web_search doesn't find the obvious subreddit
**B. Soften post-retrieval scoring** in `score.py`:
- Change the no-engagement penalty from `-10` to `-3` (missing metrics ≠ irrelevant)
- Change low date confidence penalty from `-10` to `-5`
**C. Add minimum result guarantee** in `last30days.py`:
- If scoring filters out ALL results, keep the top 3 by raw relevance regardless of score
- Log a warning: "All Reddit results scored below threshold, keeping top 3 by relevance"
**Files to change:**
- `scripts/lib/openai_reddit.py` - Add subreddit fallback search (lines ~160-180)
- `scripts/lib/score.py` - Soften penalties (lines ~151-157)
- `scripts/last30days.py` - Add minimum result guarantee (lines ~474-490)
### Fix 3: Citations Too Verbose
**Root cause:** The SKILL.md instruction says "Every insight MUST cite at least one source" with a GOOD example showing `(per @XXX, 15 likes; r/kanye thread with 200 upvotes)` - this is too much detail per citation and the agent over-applies it.
**Approach:** Dial back to "cite 1-2 sources per KEY PATTERN, not per sentence. Use short format."
**File:** `SKILL.md` (citation section, currently around line 158)
**Change the citation rule to:**
```
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
BAD: "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
GOOD: "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
### Fix 4: Summary Formatting (Wall of Text vs Structured)
**Root cause:** The SKILL.md template for PROMPTING/NEWS/GENERAL shows:
```
What I learned:
[2-4 sentences synthesizing...]
```
This gives the agent permission to write a dense paragraph. The nano banana pro test got good formatting because PROMPTING queries naturally produce structured patterns. NEWS queries (kanye) produce narratives that become walls of text.
**Approach:** Add explicit structure to the NEWS/GENERAL format with bold topic headers.
**File:** `SKILL.md` (summary display section, around line 158)
**Change the PROMPTING/NEWS/GENERAL template to:**
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
The bold topic headers force structure. Each topic gets its own paragraph with a line break. No more wall-of-text narratives.
## Acceptance Criteria
- [ ] **Fix 1:** Stats box uses emoji tree format ├─ 🟠 🔵 🌐 └─ 🗣️ in 4/4 test queries
- [ ] **Fix 2:** "kanye west" returns >0 Reddit threads (r/kanye exists and is active)
- [ ] **Fix 3:** Summary citations are 1 per insight, short format, no engagement metrics inline
- [ ] **Fix 4:** NEWS/GENERAL summaries use bold topic headers with paragraph breaks, not wall of text
## Test Plan
Re-run the same 4 queries after fixes:
1. `/last30days kanye west` — NEWS: should get Reddit results, structured summary, emoji stats
2. `/last30days howie.ai` — GENERAL: should get Reddit if available, citations not verbose
3. `/last30days nano banana pro prompting` — PROMPTING: should maintain current good quality, reduce citation density
4. `/last30days open claw` — GENERAL: should cite @handles in summary, emoji stats
## Files to Modify
| File | Fix | Change |
|------|-----|--------|
| `SKILL.md` | 1, 3, 4 | Stats template, citation rules, summary structure |
| `scripts/lib/openai_reddit.py` | 2 | Add subreddit fallback search |
| `scripts/lib/score.py` | 2 | Soften scoring penalties |
| `scripts/last30days.py` | 2 | Add minimum result guarantee |
## References
- Current SKILL.md: `~/.claude/skills/last30days/SKILL.md`
- Private repo: `/Users/mvanhorn/last30days-skill-private/`
- Old working SKILL.md: `~/.claude/skills/last30days.backup-v1/SKILL.md`
- Reddit search module: `scripts/lib/openai_reddit.py:53-93` (prompt), `:160-166` (API call)
- Scoring module: `scripts/lib/score.py:151-157` (penalties)
- Main pipeline: `scripts/last30days.py:474-490` (filtering)
@@ -1,177 +0,0 @@
---
title: "fix: Skill execution broken - fork mode subagent ignores bash and text instructions"
type: fix
date: 2026-02-06
---
# fix: Skill Execution Broken in Fork Mode
## Overview
The last30days v2 skill stopped running its Python script and stopped showing acknowledgment text. The agent jumps straight to WebSearch, ignoring all instructions to run bash first or output text. Five attempted fixes all failed.
## Root Cause (Confirmed via Research)
**The old v1 skill worked by accident.** GitHub Issue #17283 documented that `context: fork` and `agent: Explore` were **silently ignored** in older Claude Code versions. The skill ran **inline** in the main conversation — not in a forked subagent. That's why:
- The user saw acknowledgment text (output inline to conversation)
- The bash script ran (main model followed instructions inline)
- Progress was visible (tool calls shown normally)
**Claude Code 2.1+ fixed the bug** and now properly honors `context: fork`. The skill now truly runs in an isolated subagent where:
- The model decides tool ordering independently
- Text output instructions are deprioritized vs tool calls
- "RUN THIS FIRST" instructions are **suggestions**, not commands
- There is **no mechanism** to force tool ordering in a forked subagent
**This is why every SKILL.md rewrite failed** — the problem isn't the instructions, it's `context: fork` itself.
## Evidence
| Attempt | What we tried | Result |
|---------|--------------|--------|
| 1 | "YOUR FIRST ACTION: Run this command. EXECUTE." | Agent ran script sometimes, never showed ack text |
| 2 | "YOUR FIRST OUTPUT — before ANY tool calls" + progress block | Agent ignored text, jumped to WebSearch |
| 3 | Moved progress block to very first section | Agent ignored it entirely |
| 4 | "DO NOT skip this. DO NOT jump to tool calls first." | Agent still jumped to WebSearch |
| 5 | Embedded echo in bash block + "Do NOT start with WebSearch" | Agent still jumped to WebSearch, never ran bash |
## Proposed Fix
### Option A: Remove `context: fork` (Recommended)
**Remove `context: fork` from frontmatter.** The skill runs inline in the main conversation, exactly like the old v1 skill accidentally did.
**Why this works:**
- Inline execution follows instructions sequentially
- Text output appears directly to the user
- Bash commands run when instructed
- This is how the "working" v1 skill actually operated
**File:** `SKILL.md` frontmatter
**Change from:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
context: fork
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
**Change to:**
```yaml
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web...
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
```
That's it. Remove the one line.
**Then restore the old v1 instruction flow:**
1. "Parse User Intent" section FIRST (generates acknowledgment text)
2. "Research Execution" with bash command
3. "Do WebSearch" while script runs
4. Synthesize and present
### Option B: Keep `context: fork` + Use `!`command`` Preprocessing
Use shell preprocessing syntax (`!`command``) to run the script **before** the model even sees the prompt:
```markdown
## Research data (auto-fetched)
!`python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1`
```
**Risk:** Not confirmed that `$ARGUMENTS` works in `!`command`` context. More complex. The user still won't see progress text during preprocessing.
### Recommendation: Option A
Remove `context: fork`. It's one line. The old skill worked inline. The v2 skill should too. Option B is a backup if inline mode causes context window issues.
## Implementation
### Step 1: Remove `context: fork` from frontmatter
Single line removal in `SKILL.md`.
### Step 2: Restore v1-style instruction flow
The SKILL.md opening should match the public v1 pattern:
```markdown
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
[... topic/tool/query type parsing ...]
Store these variables:
- TOPIC = ...
- TARGET_TOOL = ...
- QUERY_TYPE = ...
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
**Step 2: Do WebSearch** (while script runs)
[... websearch queries based on QUERY_TYPE ...]
**Step 3: Wait for script to complete**
[... synthesis instructions ...]
```
The key structural elements from v1 that need to return:
1. Descriptive intro paragraph
2. "Parse User Intent" BEFORE any tool calls
3. Script execution as a clearly labeled step
4. WebSearch as step 2 (not step 1)
### Step 3: Keep all v2 improvements
The v2-specific improvements (citation rules, stats template, Reddit fallback, scoring changes) stay. Only the frontmatter and instruction flow change.
### Step 4: Sync and test
Copy to `~/.claude/skills/last30days/SKILL.md`, test in new session.
## Acceptance Criteria
- [ ] `context: fork` removed from SKILL.md frontmatter
- [ ] Agent outputs acknowledgment text before running tools
- [ ] Python script actually executes (Reddit + X results appear)
- [ ] WebSearch supplements, doesn't replace, script results
- [ ] Stats emoji tree format renders correctly
- [ ] Citations are sparse (1 per insight, not 3-5)
## Test Plan
Run in a NEW Claude Code session:
1. `/last30days kanye west` — should see ack text, script runs, Reddit results
2. `/last30days open claw` — should see ack text, X results via Bird
## Files to Modify
| File | Change |
|------|--------|
| `SKILL.md` | Remove `context: fork`, restore v1 instruction flow |
## References
- GitHub Issue #17283: `context: fork` was silently ignored (the bug that made v1 work)
- Claude Code Skills docs: `!`command`` preprocessing syntax
- Claude Code Subagents docs: `agent: Explore` uses Haiku, read-only tools
- Public v1 SKILL.md: `github.com/mvanhorn/last30days-skill`
@@ -1,385 +0,0 @@
---
title: "test: Compare v1 (public) vs v2 (private) last30days output quality"
type: test
date: 2026-02-06
---
# test: V1 vs V2 Comparison Test Plan
## Overview
Run the same queries through both the public v1 and private v2 of last30days, compare output quality across 7 dimensions, and determine if v2 is ready to ship as the new public version.
**This plan also includes a full feature audit** identifying everything v1 has that v2 is missing — some of those gaps need fixing before shipping.
---
## How to Run the Comparison
### Setup
**V1 (public upstream):** Check out upstream SKILL.md temporarily:
```bash
# Save current v2
cp ~/.claude/skills/last30days/SKILL.md ~/.claude/skills/last30days/SKILL.md.v2
# Install v1 from upstream
cd /Users/mvanhorn/last30days-skill-private
git show upstream/main:SKILL.md > ~/.claude/skills/last30days/SKILL.md
```
Run test queries in a NEW Claude Code session (one session per query to avoid context bleed). Save output.
**V2 (private current):** Restore v2:
```bash
cp ~/.claude/skills/last30days/SKILL.md.v2 ~/.claude/skills/last30days/SKILL.md
```
Run same queries in NEW sessions. Save output.
---
## ALL Test Queries
### From README Examples (13 documented use cases)
Every single example from the README, in order:
| # | Query | Type | README Section |
|---|-------|------|---------------|
| 1 | `prompting techniques for chatgpt for legal questions` | PROMPTING + TOOL | Example: Legal Prompting |
| 2 | `best clawdbot use cases` | RECOMMENDATIONS | Example: ClawdBot Use Cases |
| 3 | `how to best setup clawdbot` | HOW-TO | Example: ClawdBot Setup |
| 4 | `prompting tips for nano banana pro for ios designs` | PROMPTING + TOOL | Example: iOS App Mockup |
| 5 | `top claude code skills` | RECOMMENDATIONS | Example: Top Claude Code Skills |
| 6 | `using ChatGPT to make images of dogs` | GENERAL | Example: Dog as Human |
| 7 | `research best practices for beautiful remotion animation videos in claude code` | PROMPTING | Example: Remotion Launch Video |
| 8 | `photorealistic people in nano banana pro` | PROMPTING | Example: Photorealistic Portraits |
| 9 | `What are the best rap songs lately` | RECOMMENDATIONS | Example: Best Rap Songs |
| 10 | `what are people saying about DeepSeek R1` | NEWS | Example: DeepSeek R1 |
| 11 | `best practices for cursor rules files for Cursor` | PROMPTING | Example: Cursor Rules |
| 12 | `prompt advice for using suno to make killer songs in simple mode` | PROMPTING | Example: Suno AI Music |
| 13 | `how do I use Codex with Claude Code on same app to make it better` | HOW-TO | Example: Codex + Claude Code |
### From Plan Documents (4 additional battle-tested queries)
| # | Query | Type | Source |
|---|-------|------|--------|
| 14 | `kanye west` | NEWS | fix-v2-formatting plan, most-tested query |
| 15 | `howie.ai` | GENERAL | fix-v2-formatting plan, edge case (domain as topic) |
| 16 | `open claw` | GENERAL | fix-v2-formatting plan, X-heavy sources |
| 17 | `nano banana pro prompting` | PROMPTING | fix-v2-formatting plan |
### Follow-up Vision Tests (pick 4 from above, ask a follow-up)
These test the prompt-generation phase specifically:
| Base Query | Follow-up Vision |
|------------|-----------------|
| #4 (nano banana pro ios) | "make a mock-up of an app for moms who swim" |
| #6 (ChatGPT dog images) | "what would my dog look like as a human prompt" |
| #12 (suno music) | "Rap song about self aware AI that loves Claude Code" |
| #13 (codex + claude code) | "how do I build a review loop workflow" |
---
## FEATURE AUDIT: V1 vs V2
### Section-by-section comparison
I diffed the full v1 (upstream/main) SKILL.md against the current v2. Here's everything.
#### KEPT (in both versions) ✅
| Feature | V1 Location | V2 Location | Notes |
|---------|------------|------------|-------|
| Parse User Intent section | Lines 23-48 | Lines 12-38 | Same logic |
| QUERY_TYPE detection (4 types) | Lines 29-36 | Lines 18-22 | Same types |
| "Don't ask about tool before research" | Lines 49-51 | Lines 31-33 | Same rule |
| Store variables block | Lines 53-56 | Lines 35-38 | Same |
| Research script execution | Lines 81-86 | Lines 59-62 | Same command |
| WebSearch by QUERY_TYPE | Lines 99-127 | Lines 77-98 | Same queries |
| "Use user's exact terminology" | Lines 129-133 | Lines 100-101 | V2 shorter but same intent |
| Judge Agent synthesis | Lines 143-151 | Lines 113-124 | Same logic |
| Internalize research (ground in actual content) | Lines 159-165 | Lines 128-135 | V2 shorter |
| RECOMMENDATIONS: extract specific names | Lines 167-177 | Lines 137-145 | Same, v2 removes BAD/GOOD example |
| Prompt format matching | Lines 193-196 | Lines 149-153 | Same |
| Summary + Stats + Invitation flow | Lines 200-250 | Lines 157-236 | Same structure, different details |
| Wait for user's vision | Lines 254-258 | Lines 240-242 | Same |
| Write ONE perfect prompt | Lines 262-275 | Lines 246-266 | Same structure |
| Context memory | Lines 298-316 | Lines 278-288 | V2 shorter |
| Output summary footer | Lines 320-340 | Lines 292-302 | Different format |
| Depth options (quick/default/deep) | Lines 135-139 | Lines 106-109 | Same |
#### ADDED in V2 (improvements) ✨
| Feature | What it does | V2 Location |
|---------|-------------|------------|
| **Query parsing display** | Shows `🔍 **{TOPIC}** · {QUERY_TYPE}` before tools | Lines 40-53 |
| **Sparse citation rules** | BAD/GOOD examples, "1 per pattern, short format" | Lines 186-193 |
| **Bold topic headers** | `**{Topic 1}** — [1-2 sentences, per source]` format | Lines 195-208 |
| **Strict stats template** | "NEVER use plain text dashes", fill-in-blank | Lines 217-230 |
| **RECOMMENDATIONS source attribution** | Each item MUST have Sources: line with @handles | Lines 178-182 |
| **Reddit 0 results handling** | Explicit instruction for 0-thread line | Line 229 |
| **Bird CLI in stats** | "(via Bird/xAI)" notation | Line 223 |
#### ❌ MISSING FROM V2 — Features V1 Has That V2 Dropped
These are the regressions. Some are intentional simplifications, others are real gaps.
**1. Use Cases Block (intro section)**
- **V1 has:** 4 use case examples right after the intro: Prompting, Recommendations, News, General — with concrete examples
- **V2 has:** Nothing. Just the intro paragraph.
- **Impact:** LOW. The query type detection handles this. But it was nice onboarding.
- **Verdict:** Skip — not needed for execution quality.
**2. Setup Check Section (API key guidance)**
- **V1 has:** Full section explaining 3 modes (Full/Partial/Web-Only), first-time setup bash script, "API keys are OPTIONAL" messaging
- **V2 has:** Nothing. Script auto-detects.
- **Impact:** LOW for experienced users. HIGH for first-time users who don't have keys.
- **Verdict:** Skip for now — script handles auto-detection. Consider adding back for public release.
**3. Anti-Pattern Examples (synthesis quality guard)**
- **V1 has:** Explicit anti-pattern block: "If user asks about 'clawdbot skills' and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as 'Claude Code skills' just because both involve 'skills'." Plus BAD/GOOD synthesis examples for RECOMMENDATIONS.
- **V2 has:** Only "Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge" — no concrete examples.
- **Impact:** MEDIUM-HIGH. Without concrete anti-patterns, the agent may conflate similar-sounding things.
- **Verdict:** ⚠️ ADD BACK. At minimum, restore the BAD/GOOD RECOMMENDATIONS example and the "don't conflate" warning.
**4. Self-Check Instruction (pre-display validation)**
- **V1 has:** "SELF-CHECK before displaying: Re-read your 'What I learned' section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it."
- **V2 has:** Nothing.
- **Impact:** MEDIUM. The self-check forces the model to validate its own output.
- **Verdict:** ⚠️ ADD BACK. One line costs nothing and catches hallucination.
**5. Quality Checklist for Prompts ⭐**
- **V1 has:** Explicit checklist before delivering a prompt:
```
### Quality Checklist:
- [ ] FORMAT MATCHES RESEARCH - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
```
- **V2 has:** Only "If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT." — one line instead of 5 checks.
- **Impact:** HIGH. This is likely what the user noticed as missing — v1 prompts felt more polished because the agent ran a checklist before delivering.
- **Verdict:** ⚠️ ADD BACK. This is the "that's a great prompt" quality feel.
**6. Prompt Format Anti-Pattern**
- **V1 has:** "ANTI-PATTERN: Research says 'use JSON prompts with device specs' but you write plain prose. This defeats the entire purpose of the research."
- **V2 has:** Only the positive instruction (use the format research recommends).
- **Impact:** MEDIUM. Negative examples ("don't do this") are powerful for LLMs.
- **Verdict:** ⚠️ ADD BACK. One line.
**7. "IF USER ASKS FOR MORE OPTIONS" Section**
- **V1 has:** "Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested."
- **V2 has:** Nothing about handling multi-prompt requests.
- **Impact:** LOW-MEDIUM. Without it, agent might dump multiple prompts unprompted.
- **Verdict:** ⚠️ ADD BACK. Two lines.
**8. Web-Only Mode Stats Template + Promo**
- **V1 has:** Separate stats template for web-only mode with "💡 Want engagement metrics? Add API keys..." promo
- **V2 has:** Only the full-mode template. If running web-only, agent has no guidance.
- **Impact:** MEDIUM for users without API keys.
- **Verdict:** Consider adding back for public release. Lower priority for now.
**9. TARGET_TOOL Question Template**
- **V1 has:** Explicit AskUserQuestion block with 4 options: [Most relevant tool], Nano Banana Pro, ChatGPT/Claude, Other
- **V2 has:** "run research first, then ask AFTER showing results" — but no actual question template.
- **Impact:** LOW-MEDIUM. Agent will still ask, just less structured.
- **Verdict:** Skip — not critical.
**10. Context Memory: "Don't re-search" Instructions**
- **V1 has:** Explicit "DO NOT run new WebSearches — you already have the research. Answer from what you learned. Cite the Reddit threads, X posts, and web sources."
- **V2 has:** Only "Only do new research if the user explicitly asks about a DIFFERENT topic."
- **Impact:** MEDIUM. Without the explicit ban, agent may re-search on follow-ups, wasting time.
- **Verdict:** ⚠️ ADD BACK. Three lines.
**11. Output Summary Footer (emoji + engagement counts)**
- **V1 has:** `📚 Expert in: {TOPIC} for {TARGET_TOOL}` and `📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages`
- **V2 has:** `Expert in: {TOPIC} for {TARGET_TOOL}` and `Based on: {n} Reddit threads + {n} X posts + {n} web pages` — no emoji, no engagement counts.
- **Impact:** LOW but noticeable. The emoji + counts make the footer feel more substantial.
- **Verdict:** ⚠️ ADD BACK. Trivial fix.
---
## Priority Fix List (Before Shipping V2 as Public)
Based on the audit, these should be restored in V2 before it replaces V1:
### Must Fix (affects output quality)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 1 | **Quality Checklist for prompts** | The "that's a great prompt" feel. V1's 5-point checklist made prompts more polished. | Add 8 lines to SKILL.md |
| 2 | **Anti-pattern examples** | BAD/GOOD synthesis examples prevent agent from conflating research. | Add 5 lines |
| 3 | **Self-check instruction** | One-line pre-display validation catches hallucination. | Add 2 lines |
| 4 | **Context Memory: don't re-search** | Prevents wasting time re-searching on follow-ups. | Add 3 lines |
### Should Fix (polish)
| # | Missing Feature | Why | Effort |
|---|----------------|-----|--------|
| 5 | **Prompt format anti-pattern** | Negative example reinforces "match the format". | Add 2 lines |
| 6 | **"IF USER ASKS FOR MORE OPTIONS"** | Prevents prompt dumping. | Add 2 lines |
| 7 | **Output footer emoji + engagement counts** | More polished footer. | Edit 3 lines |
### Skip for Now (nice-to-have for public release)
| # | Missing Feature | Why Skip |
|---|----------------|----------|
| 8 | Use cases block (intro) | Doesn't affect execution |
| 9 | Setup Check section | Script auto-detects; add back for public README |
| 10 | Web-only mode stats + promo | Lower priority, most users have keys |
| 11 | TARGET_TOOL question template | Agent handles this naturally |
---
## Scoring Dimensions (1-5 scale, 7 dimensions)
### 1. Query Parsing Display
Does the agent show what it understood before starting research?
| Score | Criteria |
|-------|----------|
| 1 | No acknowledgment, jumps straight to tools |
| 2 | Generic "I'll research this" with no specifics |
| 3 | Mentions the topic but not query type |
| 4 | Shows topic + query type clearly |
| 5 | Shows topic + query type + reformulated search terms |
### 2. Source Coverage
Did it actually use Reddit, X, AND web — or skip sources?
| Score | Criteria |
|-------|----------|
| 1 | WebSearch only, script didn't run |
| 2 | Script ran but returned 0 from one major source |
| 3 | 2 of 3 sources returned results |
| 4 | All 3 sources returned results |
| 5 | All 3 sources + good volume (10+ Reddit, 10+ X, 5+ web) |
### 3. Citation Quality
Are citations sparse and useful, or verbose and noisy?
| Score | Criteria |
|-------|----------|
| 1 | Every sentence has 3+ citations chained |
| 2 | Most sentences have multiple citations |
| 3 | 1-2 citations per insight, some over-citing |
| 4 | 1 citation per pattern, short format |
| 5 | Sparse citations that prove research is real without cluttering |
### 4. Summary Structure
Is the "What I learned" section scannable or a wall of text?
| Score | Criteria |
|-------|----------|
| 1 | Single paragraph wall of text |
| 2 | Multiple paragraphs but no structure |
| 3 | Some bold text but inconsistent |
| 4 | Bold topic headers with 1-2 sentence explanations |
| 5 | Clean topic headers + KEY PATTERNS list, easy to scan |
### 5. Stats Box Format
Does the emoji stats tree render correctly?
| Score | Criteria |
|-------|----------|
| 1 | No stats shown |
| 2 | Stats shown but plain text dashes, no emoji |
| 3 | Partial emoji format, some lines wrong |
| 4 | Correct ├─ └─ │ format with emoji, minor issues |
| 5 | Perfect emoji tree with accurate counts and top voices |
### 6. Research Grounding
Does the synthesis reflect the ACTUAL research, or generic pre-training knowledge?
| Score | Criteria |
|-------|----------|
| 1 | Entirely generic knowledge, no research content |
| 2 | Mentions some research but mostly generic |
| 3 | Mix of research and generic, some conflation |
| 4 | Clearly grounded in research, minor generic leakage |
| 5 | Every insight traceable to a specific source from the research |
### 7. Prompt Quality (follow-up tests only)
When user shares vision, is the generated prompt good?
| Score | Criteria |
|-------|----------|
| 1 | Generic prompt that ignores research |
| 2 | Mentions research topics but generic structure |
| 3 | Uses some research insights, decent prompt |
| 4 | Tailored to research, correct format for target tool |
| 5 | Uses research-recommended format, specific techniques, ready to paste, "that's a great prompt" feel |
---
## Comparison Scorecard Template
```
Query: [query text]
Version: V1 / V2
Date: YYYY-MM-DD
| Dimension | Score (1-5) | Notes |
|---------------------|-------------|-------|
| Query Parsing | | |
| Source Coverage | | |
| Citation Quality | | |
| Summary Structure | | |
| Stats Box Format | | |
| Research Grounding | | |
| Prompt Quality | | (follow-up tests only) |
| **TOTAL** | **/35** | |
Script output:
- Reddit: ___ threads / ___ upvotes / ___ comments
- X: ___ posts / ___ likes / ___ reposts
- Web: ___ pages
Observations:
[Free text notes]
```
---
## Execution Plan
### Phase 1: Fix the gaps first
Apply the 7 "Must Fix" + "Should Fix" items from the audit to V2 SKILL.md. This takes ~20 minutes since it's all small text additions.
### Phase 2: Smoke test (4 queries)
Run queries #14 (kanye west), #2 (best clawdbot use cases), #8 (photorealistic nano banana pro), #10 (DeepSeek R1) on V2 only. Verify the fixes work.
### Phase 3: Full comparison (all 17 queries)
Run all 17 queries on both V1 and V2. Fill scorecards.
### Phase 4: Follow-up vision tests (4 queries)
Run the 4 follow-up vision tests. Compare prompt quality — this is where the quality checklist fix matters most.
### Phase 5: Analysis
- Sum scores per version across all queries
- Identify any dimension where v1 consistently beats v2
- Decision: ship v2, or fix more gaps first
## Acceptance Criteria
- [x] Feature audit complete (this document)
- [x] Must-fix gaps restored in V2 SKILL.md
- [ ] All 17 queries run on V2
- [ ] At least 4 queries run on V1 for comparison
- [ ] 4 follow-up vision tests completed
- [ ] Scorecards filled for each
- [ ] Total score comparison documented
- [ ] Any V1 > V2 regressions identified with fix plan
- [ ] Go/no-go decision on shipping v2 as public
## Files
| File | Purpose |
|------|---------|
| `docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md` | This plan |
| `SKILL.md` | Apply Must Fix + Should Fix items |
| `docs/test-results/v1-vs-v2-comparison.md` | Results (to be created) |
@@ -1,208 +0,0 @@
---
title: "feat: Bundle Bird X search client to eliminate npm dependency"
type: feat
date: 2026-02-07
---
# feat: Bundle Bird X search client to eliminate npm dependency
## Overview
Replace the `subprocess.run(["bird", "search", ...])` dependency in `bird_x.py` with a vendored Node.js module that calls Twitter's GraphQL search API directly. This eliminates the need for users to `npm install -g @steipete/bird` and protects against the package being removed from npm.
Bird is MIT-licensed. We have the full compiled package archived at `vendor/steipete-bird-0.8.0.tgz` and forked to `github.com/mvanhorn/bird-cli-archive`.
## Problem Statement
@steipete deleted Bird's GitHub repo on 2026-02-07. The npm package still works today, but if he unpublishes it from npm:
- New users can't `npm install -g @steipete/bird`
- The `bird` binary disappears from PATH on fresh installs
- `bird_x.py` returns 0 X results for everyone without an xAI API key
- /last30days V2's headline feature ("free X search") stops working for new users
## Proposed Solution
**Vendor Bird's search-only subset as a Node.js module inside /last30days, called from Python via `subprocess.run(["node", ...])`.**
This is the minimal-change approach:
- Keep Python as the orchestrator (bird_x.py stays mostly the same)
- Replace `subprocess.run(["bird", "search", ...])` with `subprocess.run(["node", "vendor/bird-search.mjs", ...])`
- Extract only the search-related code from Bird (not posting, bookmarks, lists, etc.)
- Cookie auth stays the same (environment variables or browser extraction)
### Why not rewrite in pure Python?
Bird's search client uses Twitter's internal GraphQL API with:
- Rotating QueryIDs (hardcoded + runtime refresh from x.com)
- Specific request header construction (bearer token, csrf, client UUIDs)
- Cursor-based pagination with Twitter-specific response parsing
- The `@steipete/sweet-cookie` dependency for browser cookie extraction
Porting all of this to Python is ~1000 lines of fragile reverse-engineering. Vendoring the working JS code is faster, safer, and easier to maintain since the archive includes source maps for debugging.
## Technical Approach
### What we need from Bird
Only 8 files from `dist/lib/` (out of 30+):
1. `twitter-client-base.js` - HTTP client, auth headers, rate limiting
2. `twitter-client-search.js` - Search mixin (the core feature)
3. `twitter-client-utils.js` - Tweet parsing, cursor extraction
4. `twitter-client-constants.js` - API endpoints, QueryIDs
5. `twitter-client-types.js` - TypeScript type stubs
6. `cookies.js` - Cookie resolution (env vars, browser extraction)
7. `runtime-query-ids.js` - QueryID refresh from x.com
8. `paginate-cursor.js` - Cursor pagination helper
Plus:
- `features.json` - GraphQL feature flags
- `query-ids.json` - Hardcoded QueryID fallbacks
### What we DON'T need
Posting, bookmarks, lists, timelines, engagement, follow, media, news, user lookup, user tweets - all the non-search mixins. This cuts the vendored code roughly in half.
### Architecture
```
scripts/
lib/
bird_x.py # MODIFIED - calls node instead of bird binary
vendor/
bird-search/
bird-search.mjs # NEW - thin CLI wrapper, ~40 lines
lib/ # VENDORED - subset of Bird's dist/lib/
twitter-client-base.js
twitter-client-search.js
twitter-client-utils.js
twitter-client-constants.js
twitter-client-types.js
cookies.js
runtime-query-ids.js
paginate-cursor.js
features.json
query-ids.json
node_modules/ # VENDORED - sweet-cookie only
@steipete/
sweet-cookie/
package.json # Minimal, points to bird-search.mjs
LICENSE # Bird's MIT license (required by MIT terms)
```
### Implementation
#### 1. Create `bird-search.mjs` wrapper (~40 lines)
A minimal Node.js script that:
- Accepts: `node bird-search.mjs <query> --count <n> --json`
- Creates a TwitterClient with search mixin only
- Resolves cookies (env vars first, then browser extraction)
- Calls `client.search(query, count)`
- Outputs JSON to stdout
- Exits with code 0 on success, 1 on error
This replaces the full `bird` CLI binary. Same interface, fraction of the code.
#### 2. Modify `bird_x.py` - change subprocess target
```python
# BEFORE (current)
cmd = ["bird", "search", query, "-n", str(count), "--json"]
# AFTER (vendored)
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
cmd = ["node", bird_search, query, "--count", str(count), "--json"]
```
Same subprocess pattern. Same JSON output format. Minimal diff.
#### 3. Update auth check functions
```python
# BEFORE
def is_bird_installed() -> bool:
return shutil.which("bird") is not None
# AFTER
def is_bird_installed() -> bool:
bird_search = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
return bird_search.exists() and shutil.which("node") is not None
```
`is_bird_authenticated()` changes from `bird whoami` to a quick Node.js cookie check or environment variable check.
`install_bird()` becomes a no-op (already vendored) or removes itself entirely.
#### 4. Vendor sweet-cookie
`@steipete/sweet-cookie` is the only runtime dependency. It handles browser cookie extraction on macOS/Linux. Options:
**Option A (recommended):** Vendor sweet-cookie into `vendor/bird-search/node_modules/`. It's small (one file). This makes the skill fully self-contained with zero npm installs.
**Option B:** Fall back to environment variables only (no browser cookie extraction). Users would need to manually set `AUTH_TOKEN` and `CT0` env vars. Simpler but worse UX.
Recommend Option A - vendor it.
#### 5. Update user-facing docs
- `README.md` - Remove "Install Bird CLI" section, replace with "Requires Node.js 22+"
- `SKILL.md` - Remove Bird CLI installation instructions
- Keep the fallback chain: vendored Bird search -> xAI API key -> web-only
## Acceptance Criteria
- [ ] `bird_x.py` calls vendored Node.js module instead of `bird` binary
- [ ] `search_x()` returns identical JSON format (no downstream changes needed)
- [ ] `search_handles()` works with vendored module
- [ ] Cookie auth works via environment variables (`AUTH_TOKEN`, `CT0`)
- [ ] Cookie auth works via browser extraction (sweet-cookie)
- [ ] `is_bird_installed()` checks for vendored module + Node.js
- [ ] `install_bird()` removed or returns success immediately
- [ ] No `npm install -g @steipete/bird` required anywhere
- [ ] Bird's MIT LICENSE included in vendor directory
- [ ] README updated to remove Bird CLI install steps
- [ ] SKILL.md updated to remove Bird CLI references
- [ ] Works on macOS (primary) and Linux
- [ ] Fallback to xAI API key still works if vendored search fails
## Files Changed
| File | Action | Description |
|------|--------|-------------|
| `scripts/lib/bird_x.py` | MODIFY | Replace `["bird", ...]` subprocess calls with `["node", "vendor/bird-search/bird-search.mjs", ...]` |
| `scripts/lib/vendor/bird-search/bird-search.mjs` | CREATE | Thin Node.js wrapper that imports Bird's search client and outputs JSON |
| `scripts/lib/vendor/bird-search/lib/*.js` | VENDOR | 8 files from Bird's dist/lib/ (search subset only) |
| `scripts/lib/vendor/bird-search/lib/features.json` | VENDOR | GraphQL feature flags |
| `scripts/lib/vendor/bird-search/lib/query-ids.json` | VENDOR | Hardcoded QueryID fallbacks |
| `scripts/lib/vendor/bird-search/node_modules/` | VENDOR | sweet-cookie package |
| `scripts/lib/vendor/bird-search/package.json` | CREATE | Minimal package.json for module resolution |
| `scripts/lib/vendor/bird-search/LICENSE` | COPY | Bird's MIT license |
| `README.md` | MODIFY | Remove Bird CLI install section, add Node.js 22+ requirement |
| `SKILL.md` | MODIFY | Remove Bird CLI references |
## Dependencies & Risks
**Node.js 22+ required** - Users who had Bird CLI already have Node.js. This is not a new dependency, just a version requirement. Claude Code environments typically have Node.js.
**Twitter API changes** - The GraphQL QueryIDs may rotate. Bird includes a runtime refresh mechanism (`runtime-query-ids.js`) that fetches new IDs from x.com. This is vendored and will continue working.
**sweet-cookie platform support** - Browser cookie extraction only works on macOS (Safari, Chrome, Firefox) and Linux (Chrome, Firefox). Windows users need manual env vars. This matches Bird CLI's existing behavior.
**Legal** - Bird is MIT licensed. MIT requires including the license notice in copies. We include `LICENSE` in the vendor directory. Using Twitter's internal API is the same legal gray area Bird always operated in - user accepted this when they used Bird.
## What This Does NOT Change
- Python remains the orchestrator - bird_x.py still does query construction, retry logic, response parsing
- The fallback chain stays: vendored search -> xAI API -> web-only
- Cookie auth mechanism is identical (env vars or browser extraction)
- JSON output format is identical - no changes needed in score.py or format.py
- xai_x.py is completely untouched
## References
- Bird CLI archive: `github.com/mvanhorn/bird-cli-archive`
- Local vendor tarball: `vendor/steipete-bird-0.8.0.tgz`
- Bird search implementation: `bird-cli-archive/dist/lib/twitter-client-search.js`
- Current bird_x.py: `scripts/lib/bird_x.py`
- Fallback chain: `scripts/lib/env.py:get_x_source()`
@@ -1,263 +0,0 @@
---
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
type: feat
date: 2026-02-07
---
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
## Overview
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
## Problem Statement / Motivation
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
## Proposed Solution
### Architecture: Two-Phase Search
```
CURRENT (Phase 1 — unchanged):
Broad topic search → Reddit results + X results
NEW (Phase 2 — supplemental):
Extract entities from Phase 1 results
↓ ↓
[SUBREDDITS] [@HANDLES + #HASHTAGS]
↓ ↓
Targeted Reddit Targeted X searches
searches per sub per handle/hashtag
↓ ↓
Merge + dedupe with Phase 1 results
```
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
### Feature 1: Entity Extraction Module (NEW FILE)
**File: `scripts/lib/entity_extract.py`**
A lightweight module that parses Phase 1 results and extracts:
**From X results:**
- `@handles` — from `author_handle` field + any @mentions in post text
- `#hashtags` — from post text
- Rank by frequency: handles that appear 2+ times are "key voices"
**From Reddit results:**
- `subreddit` names — from the `subreddit` field on each result
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
- Rank by frequency: subreddits with 2+ threads are "core communities"
**Output:**
```python
{
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
"x_hashtags": ["#openclaw", "#aitools"],
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
}
```
**Rules:**
- No hardcoded entities — everything discovered dynamically from Phase 1
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
- Skip the original topic's "obvious" subreddit if it was already searched
### Feature 2: Supplemental X Search (Bird)
**File: modify `scripts/lib/bird_x.py`**
Add a `search_handles()` function:
```python
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
"""Search top handles for topic-related content."""
results = []
for handle in handles[:5]:
# Uses Bird's support for X search operators
query = f"from:{handle} {topic} since:{from_date}"
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
# ... parse results, add to list
return results
```
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
```python
# xAI supports filtering to specific handles (max 10)
tools = [{
"type": "x_search",
"x_handles": {"allowed_x_handles": top_handles[:10]}
}]
```
### Feature 3: Supplemental Reddit Search
**File: modify `scripts/lib/openai_reddit.py`**
Add a `search_subreddits()` function:
```python
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
"""Search discovered subreddits for topic-related content."""
# Build multi-subreddit query for the OpenAI web_search prompt
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
# ... single OpenAI API call, same pattern as existing search
```
**Alternative approach — Reddit JSON API (free, no API key):**
```python
def search_subreddit_json(subreddit: str, topic: str) -> list:
"""Search a specific subreddit via Reddit's free JSON endpoint."""
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
# ... parse JSON response
```
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
### Feature 4: Orchestration Changes
**File: modify `scripts/last30days.py`**
After Phase 1 completes and enrichment is done, run Phase 2:
```python
# Phase 1 (existing — unchanged)
reddit_items, x_items = run_parallel_search(...)
# Phase 2 (new — supplemental)
if reddit_items or x_items:
entities = entity_extract.extract(reddit_items, x_items)
supplemental_reddit = []
supplemental_x = []
# Run supplemental searches in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
if entities["reddit_subreddits"]:
reddit_future = executor.submit(
openai_reddit.search_subreddits,
entities["reddit_subreddits"], topic, ...
)
if entities["x_handles"] and bird_available:
x_future = executor.submit(
bird_x.search_handles,
entities["x_handles"], topic, from_date, ...
)
# Merge with Phase 1
all_reddit = reddit_items + supplemental_reddit
all_x = x_items + supplemental_x
# Dedupe handles the rest
```
**Depth-dependent behavior:**
| Depth | Phase 2 behavior |
|---|---|
| `--quick` | Skip Phase 2 entirely (speed matters) |
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
**File: modify `scripts/lib/bird_x.py`**
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
```python
def expand_thread(tweet_id: str) -> list:
"""Fetch full thread for a high-engagement tweet."""
cmd = ["bird", "thread", tweet_id, "--json"]
# ... parse thread, extract key replies
```
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
## Technical Considerations
### Performance
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
### Cost
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
- X handle search via Bird: Free (uses your X login)
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
- Thread expansion: Free via Bird
### No New Dependencies
- Entity extraction is string parsing — no NLP libraries needed
- Reddit `.json` endpoint uses existing `http.py` transport
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
### Backward Compatibility
- Phase 2 is purely additive — all existing behavior unchanged
- If Phase 2 finds nothing, output is identical to current
- Deduplication handles any overlap between Phase 1 and Phase 2
## Acceptance Criteria
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
- [x] Supplemental X searches via Bird find additional content from key handles
- [x] Supplemental Reddit searches find content in discovered subreddits
- [x] Phase 2 results are properly merged and deduped with Phase 1
- [x] `--quick` mode skips Phase 2 entirely
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
- [x] No performance regression on `--quick` mode
- [ ] Default mode adds < 10 seconds of latency
- [x] Works with Bird-only, xAI-only, and both-available configurations
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
## Implementation Order
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
6. (Stretch) Thread expansion for high-engagement posts
## Research Sources
### Reddit Search Techniques
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
### X/Twitter Search Techniques
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
### Key Insight
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
## What We're NOT Doing
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
- **Not changing the output format** — Phase 2 results merge seamlessly
- **Not hardcoding any entities** — all discovery is dynamic from search results
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
- **Not replacing the current search** — Phase 2 supplements Phase 1
@@ -1,147 +0,0 @@
---
title: "fix: X search query too restrictive, returns 0 results on popular topics"
type: fix
date: 2026-02-07
---
# fix: X search query too restrictive, returns 0 results on popular topics
## Problem
`/last30days vibe motion best prompt techniques` returned **0 X posts** despite Vibe Motion being actively discussed on X (screenshots show posts from @Godid242, @KamilStanuch, @ColdStartTheory, @higgsfield_ai).
Root cause: `_extract_core_subject()` in `bird_x.py` produces overly specific queries. Bird/X search uses **literal keyword AND matching** — ALL words must appear in a tweet. The function kept 4 keywords (`vibe motion prompt techniques`) when only 2 (`vibe motion`) were needed.
## Three Bugs Found
### Bug 1: Multi-word noise phrases never match
```python
# Current code (bird_x.py:24-38)
noise = ['best', ..., 'what are', 'what is', 'how to', 'tips for', ...]
words = topic.lower().split() # splits into individual words
result = [w for w in words if w not in noise] # compares "what" against "what are" → no match!
```
`"what are people saying about DeepSeek R1"` → keeps `"what are people saying"`**LOSES THE ENTIRE TOPIC**.
The multi-word entries (`"what are"`, `"how to"`, `"tips for"`, `"use cases"`) are dead code. They never match because `.split()` creates individual words but the noise list has multi-word strings.
### Bug 2: Missing meta/research words
The noise list has `"prompting"` but not `"prompt"`, `"prompts"`, `"techniques"`, `"tips"`, `"tricks"`, `"methods"`, etc.
- `"vibe motion best prompt techniques"``"vibe motion prompt techniques"` (4 words, should be 2)
- `"nano banana pro prompts for gemini"``"nano banana pro prompts"` (4 words, should be 3)
### Bug 3: No retry on 0 results
Reddit has multi-stage retry: full query → simplified core → subreddit fallback. X search runs once and accepts whatever comes back, even 0 results.
## Proposed Fix
All changes in `scripts/lib/bird_x.py`.
### Step 1: Fix `_extract_core_subject()` — strip phrases first, then words
```python
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search."""
text = topic.lower()
# Phase 1: Strip multi-word prefixes/suffixes (order matters - longest first)
prefixes = ['what are the best', 'what is the best', 'what are', 'what is',
'how to', 'how do i', 'tips for', 'best practices for']
for p in prefixes:
if text.startswith(p):
text = text[len(p):].strip()
break
suffixes = ['best practices', 'use cases', 'prompt techniques',
'prompting techniques']
for s in suffixes:
if text.endswith(s):
text = text[:-len(s)].strip()
break
# Phase 2: Split and filter individual noise words
noise = {'best', 'top', 'practices', 'features', 'killer', 'guide',
'tutorial', 'recommendations', 'advice', 'prompting', 'prompt',
'prompts', 'techniques', 'tips', 'tricks', 'methods',
'strategies', 'review', 'reviews', 'uses', 'usecases',
'examples', 'using', 'for', 'with', 'the', 'of', 'in', 'on',
'about', 'latest', 'new', 'news', 'update', 'updates',
'good', 'great', 'awesome', 'and', 'or', 'a', 'an', 'is',
'are', 'was', 'were', 'people', 'saying', 'think', 'said'}
words = text.split()
result = [w for w in words if w not in noise]
return ' '.join(result[:3]) or topic # Max 3 words (was 4)
```
**Expected results after fix:**
| Input | Before | After |
|-------|--------|-------|
| `vibe motion best prompt techniques` | `vibe motion prompt techniques` | `vibe motion` |
| `what are people saying about DeepSeek R1` | `what are people saying` | `deepseek r1` |
| `nano banana pro prompts for gemini` | `nano banana pro prompts` | `nano banana pro` |
| `open claw best uses` | `open claw uses` | `open claw` |
| `best claude code skills` | `claude code skills` | `claude code skills` |
| `kanye west` | `kanye west` | `kanye west` |
### Step 2: Add retry with simplified query on 0 results
In `search_x()`, after the initial search, if 0 items returned, retry with just the first 2 words of the core subject:
```python
def search_x(topic, from_date, to_date, depth="default"):
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
# ... existing Bird search code ...
items = parse_bird_response(response)
# Retry with fewer keywords if 0 results
if not items and len(core_topic.split()) > 2:
shorter = ' '.join(core_topic.split()[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
# ... retry Bird search ...
items = parse_bird_response(retry_response)
return response # or merged response
```
### Step 3 (optional): Cross-pollinate Reddit entities into X Phase 2
When X Phase 1 returns 0 results but Reddit found threads, extract brand/product names from Reddit thread titles and use them as X search fallback queries. This is lower priority — Steps 1-2 should fix most cases.
## Acceptance Criteria
- [x] `vibe motion best prompt techniques` returns >0 X posts (12 posts found)
- [x] `what are people saying about DeepSeek R1` produces query containing "deepseek r1" not "what are people saying"
- [x] No regressions on working queries (`kanye west`, `claude code skills`, `open claw`)
- [x] Retry fires when initial query returns 0, logged to stderr
- [x] `openai_reddit.py`'s `_extract_core_subject()` NOT changed (Reddit uses semantic search, not literal matching — the current function works fine there)
## Files to Change
- `scripts/lib/bird_x.py``_extract_core_subject()` rewrite + retry logic in `search_x()`
- `scripts/lib/bird_x.py``search_handles()` benefits automatically (calls `_extract_core_subject()`)
## Testing
```bash
# Mock mode (quick syntax check)
python3 scripts/last30days.py "vibe motion best prompt techniques" --mock --emit=compact 2>&1
# Live queries to verify X results
python3 scripts/last30days.py "vibe motion best prompt techniques" --quick --emit=compact 2>&1 | grep -E "X:|posts"
python3 scripts/last30days.py "what are people saying about DeepSeek R1" --quick --emit=compact 2>&1 | grep -E "X:|posts"
# Regression check
python3 scripts/last30days.py "kanye west" --quick --emit=compact 2>&1 | grep -E "X:|posts"
```
@@ -1,243 +0,0 @@
---
title: "feat: Add Codex CLI compatibility"
type: feat
date: 2026-02-14
---
# feat: Add Codex CLI Compatibility
## Overview
Make /last30days work as a Codex CLI skill alongside Claude Code. Both platforms use `SKILL.md` with YAML frontmatter — the gap is small but the details matter. Inspired by PR #24 (el-analista) and PR #5 (jblwilliams) on the public repo, applied to the v2.1 codebase.
## Research Findings
### How Codex Skills Work (from [official docs](https://developers.openai.com/codex/skills))
**Format:** Identical to Claude Code — `SKILL.md` with YAML frontmatter + Markdown body.
**Required frontmatter:** Only `name` and `description`. The official skill-creator guidance says "Do not include any other fields in YAML frontmatter." This is stricter than Claude Code which allows `version`, `allowed-tools`, `argument-hint`, etc.
**Discovery:** Codex uses "progressive disclosure" — it reads ONLY the `description` field to decide whether to invoke a skill. The body loads only after triggering. This means the description must be comprehensive about when to use/not use the skill.
**Invocation:** Users invoke with `$skill-name` or `/skills` menu. Codex can also implicitly match based on the description (configurable via `agents/openai.yaml`).
**Installation paths** (scanned in order):
| Scope | Path |
|-------|------|
| Folder | `$CWD/.agents/skills/` |
| Repo | `$REPO_ROOT/.agents/skills/` |
| User | `$HOME/.agents/skills/` |
| Admin | `/etc/codex/skills/` |
| System | Bundled |
Note: Some docs also mention `~/.codex/skills/` as an alias for `$HOME/.agents/skills/`. Both should be checked.
**`agents/openai.yaml`** (optional sidecar):
```yaml
interface:
display_name: "User-facing name"
short_description: "Brief description"
default_prompt: "Surrounding prompt template"
brand_color: "#hex"
policy:
allow_implicit_invocation: true
dependencies:
tools:
- type: "mcp"
value: "toolName"
```
**Size guidance:** Keep SKILL.md under 500 lines. Use `references/` directory for detailed docs that load on demand.
**Scripts:** Put executable code in `scripts/`. These can run without being loaded into context — good for our Python research engine.
### What Real Codex Skills Look Like (from [openai/skills catalog](https://github.com/openai/skills))
**openai-docs skill** — Uses MCP tools (`mcp__openaiDeveloperDocs__search_openai_docs`). Has a workflow section, fallback instructions if MCP isn't set up, and quality rules. Clean and focused.
**pdf skill** — Runs scripts (`pdftoppm`, `reportlab`), has file conventions (`tmp/pdfs/`, `output/pdf/`), specifies dependencies. Good example of a skill that shells out to tools like we do.
**skill-creator** — The meta-skill. Emphasizes "the context window is a public good" and treating the LLM as "already very smart — only add information it genuinely lacks." Has 6 creation steps, validation scripts, and naming conventions.
### Key Insight: Frontmatter Compatibility Problem
Claude Code SKILL.md uses:
```yaml
name: last30days
version: "2.1"
description: Research a topic...
argument-hint: 'nano banana pro prompts...'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
```
Codex wants only `name` and `description`. The question: does Codex error on unknown frontmatter fields, or ignore them?
**Safe answer:** Codex uses standard YAML parsing and likely ignores unknown keys. But the official guidance says "Do not include any other fields" — meaning it's untested territory and could break in future Codex updates.
**Our approach:** Keep one SKILL.md with Claude-specific fields. If Codex chokes, we add a thin wrapper. This is pragmatic — maintaining two SKILL.md files defeats the purpose of cross-platform compatibility.
### PR #24 Analysis (el-analista)
Good ideas to incorporate:
- Portable script path resolution (repo → Claude → Codex → agents)
- `agents/openai.yaml` for Codex discovery
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dir fallbacks with env var overrides
- Last-chance retry for Bird search (better query noise stripping)
Not applicable to v2.1:
- Based on v2.0 codebase — doesn't have YouTube, vendored Bird, or pipeline changes
- We'll cherry-pick the ideas, not the code
### PR #5 Analysis (jblwilliams)
Not needed:
- Codex JWT auth — our OpenAI API calls work natively in Codex already
- SSE response handling — we don't stream responses
- The 403 enrichment issues they hit are specific to Codex-hosted auth, not our use case
## Proposed Solution
Five changes, all additive — zero impact on existing Claude Code behavior:
### 1. Add `agents/openai.yaml` for Codex discovery
```yaml
interface:
display_name: "Last 30 Days"
short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
brand_color: "#FF6B35"
policy:
allow_implicit_invocation: true
```
### 2. Make SKILL.md script path portable
Replace the hardcoded Claude path with a lookup that checks multiple install locations:
```bash
# Find the skill root
for dir in \
"." \
"${CLAUDE_PLUGIN_ROOT:-}" \
"$HOME/.claude/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"$HOME/.codex/skills/last30days"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
if [ -z "${SKILL_ROOT:-}" ]; then
echo "ERROR: Could not find scripts/last30days.py" >&2
exit 1
fi
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
```
### 3. Platform-neutral Python output text
Replace "Claude" with "assistant" in LLM-facing output strings only. Human-facing docs (README, etc.) stay as-is.
Files:
- `scripts/last30days.py` — web search marker text (~3 lines)
- `scripts/lib/render.py` — docstrings + web-only banner (~4 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### 4. Sandbox-friendly cache/output dirs
Codex runs sandboxed. Add env var overrides + tempdir fallback (from PR #24):
**`scripts/lib/cache.py`:**
- Check `LAST30DAYS_CACHE_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/cache`
**`scripts/lib/render.py`:**
- Check `LAST30DAYS_OUTPUT_DIR` env var
- Catch `PermissionError`, fall back to `tempfile.gettempdir()/last30days/out`
### 5. README + installation docs
Add a "Codex Compatibility" section to README:
```markdown
## Codex Compatibility
This skill works in both Claude Code and OpenAI Codex CLI.
**Claude Code:** `git clone` into `~/.claude/skills/last30days`
**Codex CLI:** `git clone` into `~/.agents/skills/last30days`
Both use the same SKILL.md, same Python engine, same scripts.
The `agents/openai.yaml` provides Codex-specific discovery metadata.
```
## What We're NOT Doing
- **Separate SKILL.md for Codex** — One file, both platforms. Claude-specific frontmatter fields (`allowed-tools`, `version`, `argument-hint`) are likely ignored by Codex's YAML parser. If this breaks, we'll address it then.
- **Codex JWT auth (PR #5)** — Our OpenAI Responses API calls work natively in Codex. No special handling needed.
- **SSE streaming (PR #5)** — Not our use case.
- **Codex-specific tool names in SKILL.md** — Both LLMs understand "do a web search" and "run this bash command." The instructions work cross-platform as-is.
- **Publishing to openai/skills catalog** — Out of scope for now. Users install via git clone.
## Acceptance Criteria
- [x] `agents/openai.yaml` exists with proper `interface` and `policy` sections
- [x] SKILL.md uses portable path resolution (repo checkout, `~/.claude/skills/`, `~/.agents/skills/`, `~/.codex/skills/`)
- [x] Python scripts use "assistant" instead of "Claude" in LLM-facing output (~8 string replacements)
- [x] Cache dir falls back gracefully in sandboxed environments (`LAST30DAYS_CACHE_DIR` env var + `PermissionError` catch)
- [x] Output dir falls back gracefully in sandboxed environments (`LAST30DAYS_OUTPUT_DIR` env var + `PermissionError` catch)
- [x] Existing Claude Code behavior is unchanged (zero regressions)
- [x] README documents Codex installation path (`~/.agents/skills/last30days`)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
## Files to Create/Modify
### New Files
- `agents/openai.yaml` — Codex discovery metadata (~10 lines)
### Modified Files
- `SKILL.md` — Portable script path resolution (~15 lines changed)
- `README.md` — Add "Codex Compatibility" section (~15 lines)
- `scripts/last30days.py` — "Claude" → "assistant" in output strings (~3 lines)
- `scripts/lib/render.py` — "Claude" → "assistant" + output dir fallback (~15 lines)
- `scripts/lib/cache.py` — Cache dir env override + fallback (~12 lines)
- `scripts/lib/http.py` — User-Agent string (~1 line)
### Total scope: ~70 lines changed across 7 files. Small, additive, low risk.
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| Codex rejects unknown YAML frontmatter (`allowed-tools`, etc.) | Low-Medium | Standard YAML parsers ignore unknown keys. If it breaks, strip Claude-specific fields and use `agents/openai.yaml` for metadata. |
| Codex sandbox blocks Node.js (vendored Bird) | Medium | Bird failure already falls back to xAI API. If no xAI key, X search skipped gracefully. |
| yt-dlp not in Codex sandbox PATH | Medium | YouTube already degrades gracefully — "yt-dlp not installed, skipping YouTube." |
| Codex sandbox blocks `~/.cache/` writes | Medium | Env var override + tempdir fallback handles this. (Proven approach from PR #24) |
| Codex changes skill discovery paths | Low | We check 5 paths. Easy to add more. |
| Codex description matching triggers on wrong queries | Low | Write description with clear "use when" / "do not use when" boundaries per official guidance. |
## References
### Community PRs
- [PR #24](https://github.com/mvanhorn/last30days-skill/pull/24) (el-analista) — Codex compatibility, portable paths, platform-neutral text
- [PR #5](https://github.com/mvanhorn/last30days-skill/pull/5) (jblwilliams) — Codex auth support
### Official Codex Docs
- [Agent Skills](https://developers.openai.com/codex/skills) — SKILL.md format, discovery, installation paths
- [AGENTS.md Guide](https://developers.openai.com/codex/guides/agents-md/) — Custom instructions, hierarchical loading
- [Codex CLI Features](https://developers.openai.com/codex/cli/features/) — Overview of CLI capabilities
- [Configuration Reference](https://developers.openai.com/codex/config-reference/) — config.toml, skill enable/disable
### Examples
- [openai/skills catalog](https://github.com/openai/skills) — Official curated skills
- [skill-creator](https://github.com/openai/skills/blob/main/skills/.system/skill-creator/SKILL.md) — Meta-skill for creating skills, best practices
- [pdf skill](https://github.com/openai/skills/blob/main/skills/.curated/pdf/SKILL.md) — Example of skill that runs external scripts
- [openai-docs skill](https://github.com/openai/skills/blob/main/skills/.curated/openai-docs/SKILL.md) — Example of MCP-backed skill
### Community Analysis
- [Skills in OpenAI Codex](https://blog.fsck.com/2025/12/19/codex-skills/) — Jesse Vincent's deep dive on skill internals
- [Simon Willison on skills adoption](https://simonw.substack.com/p/openai-are-quietly-adopting-skills) — Cross-platform skill format analysis
- [SkillsMP marketplace](https://skillsmp.com/) — Community marketplace supporting both Claude Code and Codex skills
@@ -1,224 +0,0 @@
---
title: "feat: Merge OpenClaw variant into main repo"
type: feat
date: 2026-02-14
---
# feat: Merge OpenClaw Variant into Main Repo
## Overview
Consolidate the `last30days-openclaw` project into `last30days-skill-private` so there's one unified Python engine powering both the main skill (Claude Code / Codex) and an "open" variant with watchlist, briefing, history, and built-in web search. The open variant also gets YouTube and Bird CLI — features the main project already has but openclaw was built before they existed.
## Problem Statement / Motivation
Right now there are two separate repos with diverging codebases:
- **`last30days-skill-private`** (main, Feb 14) — YouTube, vendored Bird, better scoring/normalization, Codex compat. But no built-in web search APIs and no persistence layer.
- **`last30days-openclaw`** (Feb 10) — SQLite store, watchlist, briefings, 3 web search backends (Parallel AI, Brave, OpenRouter). But frozen without YouTube or latest engine improvements.
They share ~80% of the same `scripts/lib/` files but are drifting apart. Maintaining two codebases is unsustainable.
**Goal:** One repo, one Python engine, two SKILL.md variants. Install once, works everywhere.
## Proposed Solution
Use `last30days-skill-private` as the base (it's 4 days newer with better code) and port the OpenClaw-exclusive features in:
### What gets ported from OpenClaw
| File | What it does | Destination |
|------|-------------|-------------|
| `scripts/store.py` | SQLite research accumulator (WAL, FTS5, dedup) | `scripts/store.py` |
| `scripts/watchlist.py` | Topic watchlist CLI (add/remove/list/run) | `scripts/watchlist.py` |
| `scripts/briefing.py` | Morning briefing generator (daily/weekly) | `scripts/briefing.py` |
| `scripts/lib/brave_search.py` | Brave Search API (free tier, 2K/mo) | `scripts/lib/brave_search.py` |
| `scripts/lib/parallel_search.py` | Parallel AI search (LLM-optimized) | `scripts/lib/parallel_search.py` |
| `scripts/lib/openrouter_search.py` | OpenRouter/Sonar Pro search | `scripts/lib/openrouter_search.py` |
| `references/research.md` | One-shot research instructions | `variants/open/references/research.md` |
| `references/watchlist.md` | Watchlist mode instructions | `variants/open/references/watchlist.md` |
| `references/briefing.md` | Briefing mode instructions | `variants/open/references/briefing.md` |
| `references/history.md` | History query instructions | `variants/open/references/history.md` |
### What gets upgraded in the ported code
- **`store.py`**: No changes needed — it's self-contained SQLite, works as-is
- **`watchlist.py`**: Remove OpenClaw cron-specific code, make cron setup generic (launchd on macOS, systemd on Linux, or manual cron)
- **`briefing.py`**: No changes needed
- **`scripts/lib/env.py`**: Merge OpenClaw's web search key support (`PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`) and `has_web_search_keys()` / `get_web_search_source()` functions into main's env.py. Drop the OpenClaw config loader (`~/.openclaw/openclaw.json`) — just use env vars and `~/.config/last30days/.env`
- **`scripts/last30days.py`**: Add OpenClaw's `_search_web()` function so the script can do web search natively when API keys are available (instead of always delegating to the assistant)
### What gets DROPPED from OpenClaw
| File | Why |
|------|-----|
| `scripts/cron_setup.py` | Too OpenClaw-platform-specific. Replace with generic scheduling docs. |
| OpenClaw config loader in `env.py` | `~/.openclaw/openclaw.json` path is platform-specific. Use env vars instead. |
| `.clawhubignore` | OpenClaw marketplace artifact, not needed in unified repo |
### New file: Open variant SKILL.md
Create `variants/open/SKILL.md` — the multi-mode skill with command routing:
```
variants/open/
├── SKILL.md # Router: watch, briefing, history, or one-shot
├── references/
│ ├── research.md # One-shot research instructions
│ ├── watchlist.md # Watchlist management instructions
│ ├── briefing.md # Briefing mode instructions
│ └── history.md # History query instructions
└── context.md # Agent memory (user preferences, source quality)
```
The open variant's SKILL.md points to `{baseDir}/scripts/last30days.py` (same engine) but adds the router and reference file system. It also adds the `--store` flag for persistence.
### How YouTube and Bird CLI get added to the open variant
They're already in `scripts/lib/youtube_yt.py` and `scripts/lib/vendor/bird/`. The open variant's SKILL.md just needs to mention YouTube in its description and the research.md reference file gets the YouTube stats line in the output format. No code changes needed — the Python engine already supports all four sources.
## Technical Considerations
### File Structure After Merge
```
last30days-skill-private/
├── SKILL.md # Main skill (Claude Code / Codex)
├── agents/openai.yaml # Codex discovery (existing)
├── variants/
│ └── open/
│ ├── SKILL.md # Open variant with routing
│ ├── references/
│ │ ├── research.md
│ │ ├── watchlist.md
│ │ ├── briefing.md
│ │ └── history.md
│ └── context.md
├── scripts/
│ ├── last30days.py # Unified engine (+ native web search)
│ ├── store.py # SQLite accumulator (from openclaw)
│ ├── watchlist.py # Watchlist CLI (from openclaw, genericized)
│ ├── briefing.py # Briefing generator (from openclaw)
│ └── lib/
│ ├── ... (existing files)
│ ├── brave_search.py # NEW from openclaw
│ ├── parallel_search.py # NEW from openclaw
│ ├── openrouter_search.py # NEW from openclaw
│ ├── youtube_yt.py # Existing
│ └── vendor/bird/ # Existing
└── README.md # Updated with open variant docs
```
### Installation for open variant users
```bash
# Claude Code (main skill — unchanged)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Open variant (with watchlist, briefings, history)
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Then in Claude Code settings, point skill to variants/open/SKILL.md
# OR symlink:
ln -sf ~/.claude/skills/last30days/variants/open/SKILL.md ~/.claude/skills/last30days-open/SKILL.md
```
### env.py merge strategy
Main's `env.py` is the base. Add from OpenClaw:
- Three new key names: `PARALLEL_API_KEY`, `BRAVE_API_KEY`, `OPENROUTER_API_KEY`
- `has_web_search_keys()` function
- `get_web_search_source()` function — returns `'parallel'`, `'brave'`, or `'openrouter'`
- `get_available_sources()` update to include web-search-capable modes
### last30days.py merge strategy
Main's `last30days.py` is the base. Add from OpenClaw:
- `_search_web()` function that calls the appropriate web search backend
- `--store` CLI flag to persist findings to SQLite
- `--diagnose` CLI flag for source availability diagnostics
- Web results integration into the existing report pipeline (normalize → score → dedupe → render)
Keep main's:
- YouTube integration
- Phase 2 supplemental search
- 3-tier Reddit fallback
- Better error handling
- Minimum result guarantee
### Portable path resolution (already done)
The main SKILL.md already has portable path resolution (from Codex compat work):
```bash
for dir in "." "${CLAUDE_PLUGIN_ROOT:-}" "$HOME/.claude/skills/last30days" ...
```
The open variant's SKILL.md uses `{baseDir}` which resolves to the skill root. Both approaches work — we just need to make sure the open variant's references use `{baseDir}` consistently.
## Acceptance Criteria
- [x] `scripts/store.py` ported and working (SQLite creates on first use)
- [x] `scripts/watchlist.py` ported with generic scheduling (no OpenClaw cron dependency)
- [x] `scripts/briefing.py` ported and generates daily/weekly briefings
- [x] `scripts/lib/brave_search.py` ported and functional
- [x] `scripts/lib/parallel_search.py` ported and functional
- [x] `scripts/lib/openrouter_search.py` ported and functional
- [x] `scripts/lib/env.py` updated with web search key support
- [x] `scripts/last30days.py` has native `_search_web()` + `--store` + `--diagnose`
- [x] `variants/open/SKILL.md` exists with command routing (watch, briefing, history, research)
- [x] `variants/open/references/*.md` — all 4 reference files ported
- [x] Open variant mentions YouTube in description and research output format
- [x] Open variant uses same portable path resolution as main
- [x] Main SKILL.md behavior is unchanged (zero regressions)
- [x] `python3 scripts/last30days.py "test topic" --mock --emit=compact` still works
- [x] `python3 scripts/last30days.py "test topic" --diagnose` shows source availability
- [x] README documents open variant installation and usage
## Dependencies & Risks
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| OpenClaw's store.py has import dependencies we don't have | Low | store.py uses only stdlib (sqlite3, json, datetime). Self-contained. |
| Web search backends need API keys to test | Medium | Each has a `--mock` or dry-run path. Test with real keys if available, mock otherwise. |
| watchlist.py depends on OpenClaw cron API | High | Known — strip cron_setup.py dependency, replace with generic docs for launchd/systemd/crontab. |
| Open variant SKILL.md is too long (>500 lines) | Medium | Use reference file pattern (already planned). Router SKILL.md stays under 100 lines. |
| env.py merge introduces regressions | Low | Main's env.py is well-tested. Additive changes only — new keys, new functions. |
| Two SKILL.md files = maintenance burden | Low | They serve different purposes. Main is simple one-shot. Open adds routing. Core engine is shared. |
## Files to Create/Modify
### New Files
- `variants/open/SKILL.md` — Open variant router (~100 lines)
- `variants/open/references/research.md` — One-shot research instructions (from openclaw, updated with YouTube)
- `variants/open/references/watchlist.md` — Watchlist management instructions (from openclaw)
- `variants/open/references/briefing.md` — Briefing mode instructions (from openclaw)
- `variants/open/references/history.md` — History query instructions (from openclaw)
- `variants/open/context.md` — Agent memory template
- `scripts/store.py` — SQLite accumulator (from openclaw, as-is)
- `scripts/watchlist.py` — Watchlist CLI (from openclaw, genericized)
- `scripts/briefing.py` — Briefing generator (from openclaw, as-is)
- `scripts/lib/brave_search.py` — Brave Search API (from openclaw)
- `scripts/lib/parallel_search.py` — Parallel AI search (from openclaw)
- `scripts/lib/openrouter_search.py` — OpenRouter/Sonar Pro search (from openclaw)
### Modified Files
- `scripts/lib/env.py` — Add web search key support (~30 lines added)
- `scripts/last30days.py` — Add `_search_web()`, `--store`, `--diagnose` (~80 lines added)
- `README.md` — Add open variant section (~20 lines)
### Total scope: ~12 new files (mostly copied), ~130 lines of new code in existing files.
## References
### Internal
- OpenClaw plan: `/Users/mvanhorn/last30days-openclaw/docs/plans/2026-02-10-feat-openclaw-last30days-skill-plan.md` (989 lines, comprehensive spec)
- Codex compat plan: `docs/plans/2026-02-14-feat-codex-skill-compatibility-plan.md` (portable paths, platform-neutral text)
- OpenClaw source: `/Users/mvanhorn/last30days-openclaw/`
### Key files to port
- `store.py`: `/Users/mvanhorn/last30days-openclaw/scripts/store.py` (20KB, SQLite with FTS5)
- `watchlist.py`: `/Users/mvanhorn/last30days-openclaw/scripts/watchlist.py` (10KB)
- `briefing.py`: `/Users/mvanhorn/last30days-openclaw/scripts/briefing.py` (8KB)
- `brave_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/brave_search.py` (6KB)
- `parallel_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/parallel_search.py` (4KB)
- `openrouter_search.py`: `/Users/mvanhorn/last30days-openclaw/scripts/lib/openrouter_search.py` (7KB)
- `env.py` (openclaw version): `/Users/mvanhorn/last30days-openclaw/scripts/lib/env.py` (9KB — has web search key functions)
@@ -1,315 +0,0 @@
---
title: "feat: Add YouTube transcript search as 4th source"
type: feat
date: 2026-02-14
---
# feat: Add YouTube Transcript Search
## Overview
Add YouTube as a 4th research source alongside Reddit, X, and Web. Search for recent videos on the user's topic, fetch transcripts from the top results, and feed the transcript text into the synthesis — giving the Judge Agent access to what people are *saying* in video form, not just what they're posting on social media.
**Why this matters:** For many topics (tutorials, product reviews, drama breakdowns), the best content lives on YouTube, not Reddit or X. A 20-minute video review contains 10x the signal of a tweet. The skill currently misses all of it.
## Proposed Solution
Use **yt-dlp** (already installed via Homebrew) for both YouTube search and transcript extraction. No new API keys, no new dependencies. Follows the same "zero friction" philosophy as vendored Bird search.
### Two-step process per research run:
1. **Search**: `yt-dlp "ytsearch{N}:{topic}" --dateafter {30d_ago} --flat-playlist --print` → top videos by view count
2. **Transcripts**: For top 5 videos, extract auto-generated subtitles via `yt-dlp --write-auto-subs --skip-download`, clean VTT to plaintext in Python
### Why NOT use `summarize` CLI:
- Adds 146MB brew dependency (arm64-only binary)
- Calls OpenAI API per video ($0.01-0.03 each) — adds cost on top of existing API usage
- yt-dlp already extracts raw transcripts for free (covers ~95% of videos with auto-captions)
- Raw transcripts are better for synthesis anyway — the LLM doing synthesis (Claude) should interpret the content itself, not get a pre-summarized version
`summarize` is a great standalone tool, but for integration into a research pipeline where an LLM already synthesizes everything, raw transcripts are the right input.
## Technical Approach
### Architecture
New file: `scripts/lib/youtube_yt.py` (mirrors `bird_x.py` pattern)
```
yt-dlp search → metadata (title, views, channel, date)
sort by views, take top N
yt-dlp subtitle extraction → raw VTT files
VTT cleanup → plaintext transcripts
truncate to ~500 words per video
normalize → YouTubeItem objects
score, dedupe, render (same pipeline as Reddit/X)
```
### Implementation Phases
#### Phase 1: Search + Metadata (the fast part)
**New file: `scripts/lib/youtube_yt.py`**
Core search function:
```python
def search_youtube(topic: str, from_date: str, to_date: str, depth: str = "default") -> Dict[str, Any]:
"""Search YouTube via yt-dlp. No API key needed.
Returns:
Dict with 'items' list of video metadata dicts.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
date_filter = from_date.replace("-", "") # YYYYMMDD format
# yt-dlp search with metadata extraction
cmd = [
"yt-dlp",
f"ytsearch{count}:{topic}",
"--dateafter", date_filter,
"--flat-playlist",
"--print", "%(view_count)s\t%(id)s\t%(title)s\t%(channel)s\t%(upload_date)s\t%(like_count)s\t%(comment_count)s",
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
# Parse tab-separated output, sort by views, return top N
...
```
Depth config (matches existing pattern):
```python
DEPTH_CONFIG = {
"quick": 10, # search 10, transcript top 3
"default": 20, # search 20, transcript top 5
"deep": 40, # search 40, transcript top 8
}
TRANSCRIPT_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
```
**Key detail**: `yt-dlp --flat-playlist` returns exit code 0 with empty stdout when `--dateafter` filters out everything. Check for empty output, not error codes.
#### Phase 2: Transcript Extraction (the slow part)
For top N videos (by view count), fetch transcripts:
```python
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Returns:
Plaintext transcript string, or None if no captions available.
"""
cmd = [
"yt-dlp",
"--write-auto-subs",
"--sub-lang", "en",
"--sub-format", "vtt",
"--skip-download",
"-o", f"{temp_dir}/%(id)s",
f"https://www.youtube.com/watch?v={video_id}",
]
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
vtt_path = Path(temp_dir) / f"{video_id}.en.vtt"
if not vtt_path.exists():
return None
return _clean_vtt(vtt_path.read_text())
```
VTT cleanup (~10 lines of Python):
```python
def _clean_vtt(vtt_text: str) -> str:
"""Convert VTT subtitle format to clean plaintext."""
text = re.sub(r'^WEBVTT.*?\n\n', '', vtt_text, flags=re.DOTALL)
text = re.sub(r'\d{2}:\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}:\d{2}\.\d{3}.*\n', '', text)
text = re.sub(r'<[^>]+>', '', text)
lines = text.strip().split('\n')
seen = set()
unique = []
for line in lines:
stripped = line.strip()
if stripped and stripped not in seen:
seen.add(stripped)
unique.append(stripped)
return re.sub(r'\s+', ' ', ' '.join(unique)).strip()
```
**Parallelization**: Run transcript fetches in parallel using ThreadPoolExecutor (same pattern as Phase 2 supplemental searches for Reddit/X):
```python
def fetch_transcripts_parallel(video_ids: List[str], max_workers: int = 5) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel."""
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
for vid in video_ids
}
results = {}
for future in as_completed(futures):
vid = futures[future]
results[vid] = future.result()
return results
```
#### Phase 3: Integration into Pipeline
**Update `scripts/lib/schema.py`** — add YouTubeItem:
```python
@dataclass
class YouTubeItem:
id: str # video_id
title: str
url: str
channel_name: str
date: Optional[str]
date_confidence: str # always "high" for YouTube
engagement: Engagement # views, likes, comments
transcript_snippet: str # first ~500 words of transcript
relevance: float
why_relevant: str
subs: Optional[SubScores] = None
score: int = 0
```
Update `Report` to add:
```python
youtube: List[YouTubeItem] = field(default_factory=list)
youtube_error: Optional[str] = None
```
**Update `scripts/lib/score.py`** — YouTube-specific engagement weights:
```python
def compute_youtube_engagement_raw(views, likes, comments):
"""YouTube engagement: views dominate, likes secondary, comments tertiary."""
return (
0.50 * math.log1p(views or 0) +
0.35 * math.log1p(likes or 0) +
0.15 * math.log1p(comments or 0)
)
```
**Update `scripts/last30days.py`** — add YouTube to ThreadPoolExecutor:
```python
with ThreadPoolExecutor(max_workers=3) as executor: # was 2
if run_reddit:
reddit_future = executor.submit(_search_reddit, ...)
if run_x:
x_future = executor.submit(_search_x, ...)
if run_youtube:
youtube_future = executor.submit(_search_youtube, ...)
```
**Update `scripts/lib/render.py`** — YouTube section in compact output:
```
### YouTube Videos
**{id}** (score:{score}) {channel_name} ({date}) [{views} views, {likes} likes]
{title}
https://www.youtube.com/watch?v={id}
{transcript_snippet[:200]}...
*{why_relevant}*
```
**Update `scripts/lib/env.py`** — YouTube availability detection:
```python
def is_ytdlp_available() -> bool:
return shutil.which("yt-dlp") is not None
```
No API key needed. YouTube search is available whenever yt-dlp is in PATH.
#### Phase 4: SKILL.md Updates
Stats box adds YouTube line:
```
├─ 🎥 YouTube: {N} videos │ {N} views │ {N} transcripts
```
Citation priority updated:
```
1. @handles from X
2. YouTube creators — "per [Channel Name] on YouTube"
3. r/subreddits from Reddit
4. Web sources
```
Synthesis instructions updated to weight YouTube transcripts highly — a 20-minute video transcript with 500K views is a stronger signal than a tweet with 50 likes.
## Acceptance Criteria
- [x] `yt-dlp` search returns videos matching topic within date range
- [x] Transcripts extracted for top N videos (auto-generated captions)
- [x] Videos without captions gracefully skipped (no error)
- [x] YouTube results appear in compact output with engagement metrics
- [x] YouTube items scored and ranked alongside Reddit/X items
- [x] YouTube auto-activates when yt-dlp is available (no --sources flag needed)
- [x] SKILL.md stats box includes YouTube line
- [x] Transcript snippets (first ~500 words) included in output for LLM synthesis
- [ ] Total YouTube search + transcript extraction completes within 30 seconds
- [x] Works when yt-dlp is not installed (graceful degradation, no crash)
- [ ] Mock mode works for testing without network
## Dependencies & Risks
**Dependencies:**
- `yt-dlp` (Homebrew) — already installed, widely available via brew/pip/standalone
- No API keys needed
- No new Python packages (just subprocess + regex)
**Risks:**
| Risk | Likelihood | Mitigation |
|------|-----------|------------|
| yt-dlp search is slow (>10s) | Medium | Set 30s timeout, run in parallel with Reddit/X |
| YouTube blocks yt-dlp | Low | yt-dlp is actively maintained with anti-bot updates. Degrade gracefully. |
| Videos lack auto-captions | Medium (~5%) | Skip those videos, note in output. Transcript is enrichment, not required. |
| Transcript extraction adds latency | High | Only fetch top 3-5, run in parallel, use tempdir |
| yt-dlp not installed for some users | Medium | Auto-detect, skip YouTube with info message, don't error |
| Linux `--dateafter` date format differs | Low | Use Python to format date, not shell `date -v` |
## Files to Create/Modify
### New Files
- `scripts/lib/youtube_yt.py` — search, transcript extraction, parsing
- `tests/test_youtube_yt.py` — unit tests
- `fixtures/youtube_sample.json` — mock data for tests
### Modified Files
- `scripts/lib/schema.py` — add YouTubeItem, update Report
- `scripts/lib/normalize.py` — add normalize_youtube_items()
- `scripts/lib/score.py` — add YouTube engagement scoring
- `scripts/lib/dedupe.py` — add YouTube dedup (title + channel Jaccard)
- `scripts/lib/render.py` — add YouTube section to compact + full report
- `scripts/lib/env.py` — add yt-dlp availability check, update source detection
- `scripts/last30days.py` — add _search_youtube(), update run_research(), update arg parser
- `SKILL.md` — update stats box, citation rules, synthesis instructions
- `README.md` — document YouTube source, yt-dlp requirement
## Alternative Approaches Considered
**1. YouTube Data API v3** — Rejected. Requires API key + Google Cloud project. Adds friction, counter to "zero config" philosophy. 10K quota/day limit. yt-dlp has no limits.
**2. steipete/summarize for transcripts** — Rejected for MVP. Adds 146MB dependency, requires brew tap, calls OpenAI API per video (adds cost). Raw transcripts via yt-dlp are better input for our synthesis LLM anyway. Could revisit as optional enhancement for captionless videos.
**3. youtube-transcript-api Python package** — Considered. Lightweight, Python-native transcript fetcher. But adds a pip dependency to a project that currently has zero Python deps. yt-dlp is already a brew dependency we can auto-detect.
**4. Skip transcripts, just use metadata** — Rejected. Titles + view counts alone don't give the synthesis LLM enough to work with. Transcripts are what make YouTube a *research* source vs just a link list.
## Cost Impact
**Zero additional API cost.** yt-dlp scrapes YouTube directly. No API keys, no token usage. The only cost is the existing OpenAI/xAI calls for Reddit/X search, which are unchanged.
**Time impact:** Adds ~10-20 seconds to research (search + parallel transcript extraction), running in parallel with Reddit/X so effective wall-clock increase is minimal.
+112
View File
@@ -0,0 +1,112 @@
# v3.0.9 - The Self-Debug Release
## Highlights
**v3.0.9 is live.** New user-facing capabilities, broader cross-platform support, and a skill that now runs reliably on Claude Code, Codex, Hermes, Gemini, claude.ai, and OpenClaw. The headline fix: the engine refuses "birthday gift for 40 year old" style queries with a clarifying question instead of 5 minutes of junk output. The headline feature: TikTok and YouTube top comments now render alongside Reddit's, so the most-engaged voice from every source makes it into the synthesis.
**The label - "The Self-Debug Release":** I handed 5 separate Opus 4.7 instances their own failed outputs and asked them to debug themselves. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries outright" and "the WebSearch Sources reminder is overriding LAW 1." I copy-pasted their diagnoses into code. Validation: 5/5 canonical compliance on the topics that had failed.
## New capabilities
- **TikTok and YouTube top comments render alongside Reddit's.** PR [#260](https://github.com/mvanhorn/last30days-skill/pull/260) made the top-engagement comment from each TikTok video and YouTube video first-class in the output - same prominent `💬 Top comment` treatment Reddit's top comment already got. This is the biggest user-facing output change since 3.0.0 and it was never announced. The community inspiration trace: @uppinote20's original push for richer Reddit comments ([PR #143](https://github.com/mvanhorn/last30days-skill/pull/143)) seeded the pattern; this PR generalized it across TikTok and YouTube. PR [#265](https://github.com/mvanhorn/last30days-skill/pull/265) followed up by fixing the ScrapeCreators `url=` param + new response shape for YouTube comments/transcripts so the enrichment actually works.
- **last30days runs on Hermes AI Agent now.** @stephenmcconnachie's PR ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)) added Hermes as a first-class deploy target. `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys the full skill (SKILL.md, scripts, lib modules, fixtures) to Hermes's skills directory alongside Claude Code and Codex. This is one of the biggest surface-area expansions in v3 - last30days is now usable inside the Hermes agent's research workflows without any manual wiring.
- **Multi-key SCRAPECREATORS_API_KEY rotation.** @zaydiscold's PR ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)) added automatic key rotation. Set `SCRAPECREATORS_API_KEY_1`, `SCRAPECREATORS_API_KEY_2`, etc. and the engine rotates when a key hits rate limits instead of failing the whole run. For power users running daily queries, this is the difference between rate-limit 429s and zero-touch reliability.
- **The skill works on Windows now.** @Chelebii's PR ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)) stabilized the vendored Bird X search client on Windows. Previously the bundled X backend had subtle runtime issues on Windows terminals; now it runs clean. Pair this with @Gujiassh's UTF-8 encoding fix ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)) for saved output and Windows users get the full v3 experience without workarounds.
- **Linux permission checks stopped false-warning.** @george231224's PR ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)) fixed `check_perms` on Linux by preferring GNU stat's syntax over the BSD stat that the skill was calling. Linux users were getting spurious permission warnings on `.env` files that were already correctly 600-chmod'd. Now the check matches reality.
- **Gemini CLI got a first-class install path.** @hnshah's docs PR ([#224](https://github.com/mvanhorn/last30days-skill/pull/224)) added the Gemini CLI install note and workaround for a rough edge in the Gemini skill loader. Gemini users now have a one-paragraph install flow in the README instead of having to reverse-engineer the plugin layout.
- **Offline quality evaluation.** @j-sperling's PR ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)) added `eval_topics.json` as a fixture. Contributors and I can now run quality-regression checks on synthesis output without burning live API credits. This is the scaffolding that made the plan 015 validation gate affordable - without eval fixtures, testing 5/5 canonical compliance on every release would cost real money every time. Ships as contributor infrastructure but shows up as stability for end users.
- **Reddit client got a cleaner HTTP layer.** @iliaal shipped three architecture PRs back-to-back ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209)) that consolidated Reddit's HTTP handling into `http.get(params=...)`, rejected garbage input in `_parse_date`, and unified `_sc_headers` into `http.scrapecreators_headers`. End-user benefit: fewer flaky timeouts, fewer "weird parse error" crashes, a codebase that's easier for future contributors to touch without breaking Reddit. These aren't sexy PRs; they're the kind of refactor that prevents six future bug reports.
- **The `--days=N` flag keeps working.** @BryanTegomoh's PR ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)) restored backcompat for the legacy `--days` alias so anyone who'd scripted against it in 2.x doesn't break on v3. Small PR, meaningful reliability gain for existing users.
- **INCLUDE_SOURCES has a sane default.** @hnshah's PR ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)) defaulted the env var to empty string instead of unset. Missing env no longer breaks source inclusion on fresh installs.
- **Version metadata stays in sync.** @Gujiassh's PR ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) aligned the SKILL.md version header with the sync target version, and @shalomma's PR ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)) closed the remaining drift between the SKILL.md header and plugin.json. "Which version am I actually on" is no longer an adventure.
- **Bird X engagement handling got hardened.** @j-sperling's PR ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)) made `bird_x` skip all-None engagement dicts instead of crashing on them. Rare condition, but the kind of thing that silently kills a run on a specific topic.
- **Dev workflow hygiene.** @j-sperling's gitignore PR ([#232](https://github.com/mvanhorn/last30days-skill/pull/232)) dropped `.venv`, `.coverage`, `htmlcov`, and `.memsearch` from the tracked tree. Contributor quality-of-life; keeps PR diffs clean.
- **The skill installs to claude.ai.** PRs [#242](https://github.com/mvanhorn/last30days-skill/pull/242) and [#244](https://github.com/mvanhorn/last30days-skill/pull/244) shipped `scripts/build-skill.sh` plus the `.gitattributes` + `export-ignore` plumbing that packages last30days into a claude.ai-upload-ready `.skill` file under the 200-file cap. The skill is no longer Claude-Code-only - it installs directly on claude.ai, too. README has the upload workflow.
- **OpenAI Codex CLI discovers the skill natively.** PR [#219](https://github.com/mvanhorn/last30days-skill/pull/219) added `.agents/skills/last30days/SKILL.md` as a real file (not symlinked - Codex's loader skips symlinks) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now shows up as `last30days:last30days` when Codex runs in a checkout. Inspired by @Jah-yee ([#153](https://github.com/mvanhorn/last30days-skill/pull/153)) and @dannyshmueli on X.
- **`/last30days` as a slash command.** PR [#267](https://github.com/mvanhorn/last30days-skill/pull/267) added `commands/last30days.md` so plugin users can type `/last30days <topic>` and Claude Code autocomplete prefix-matches it to the canonical `/last30days:last30days` form. No more typing the double-namespace.
## The self-debug technique, for anyone rebuilding this elsewhere
The breakthrough wasn't the individual fixes. It was the realization that instead of guessing why the model was ignoring the rules, I should ask the model. Five separate Opus 4.7 sessions debugged their own outputs:
- "Did you read SKILL.md?" → "I tried Read, hit the 25K token cap, and bailed instead of chunked-reading."
- "Why the trailing Sources block?" → "The WebSearch tool's own reminder said MANDATORY. Precedence was unclear."
- "Why the section headers?" → "I had strong priors on Peter Steinberger and wrote my thesis instead of passing through."
- "Why the wrong file?" → "I read `.agents/skills/last30days/SKILL.md` first because it appeared in the path glob."
Three of the five said "move the LAWs to the top." Two said "make the engine enforce it so the model can't not comply." I shipped both. That's the whole technique: when the LLM-under-orchestration keeps breaking the contract, don't argue with it - ask it to debug itself, and build structural enforcement around whatever it names as the root cause.
## Thank you
**Community PR authors since v3.0.0:**
- @j-sperling - v3 engine architecture, eval fixtures, gitignore hygiene, Bird X hardening ([#232](https://github.com/mvanhorn/last30days-skill/pull/232), [#233](https://github.com/mvanhorn/last30days-skill/pull/233), [#234](https://github.com/mvanhorn/last30days-skill/pull/234))
- @stephenmcconnachie - Hermes AI Agent support ([#228](https://github.com/mvanhorn/last30days-skill/pull/228))
- @zaydiscold - Multi-key SCRAPECREATORS rotation ([#268](https://github.com/mvanhorn/last30days-skill/pull/268))
- @iliaal - Reddit HTTP helper + GitHub date parsing + ScrapeCreators header consolidation ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209))
- @Chelebii - Windows Bird X stability ([#227](https://github.com/mvanhorn/last30days-skill/pull/227))
- @george231224 - Linux check_perms stat ([#216](https://github.com/mvanhorn/last30days-skill/pull/216))
- @Gujiassh - UTF-8 saved output + version metadata alignment ([#217](https://github.com/mvanhorn/last30days-skill/pull/217), [#225](https://github.com/mvanhorn/last30days-skill/pull/225))
- @hnshah - INCLUDE_SOURCES default + Gemini install docs ([#223](https://github.com/mvanhorn/last30days-skill/pull/223), [#224](https://github.com/mvanhorn/last30days-skill/pull/224))
- @shalomma - SKILL.md v3.0.0 version header ([#229](https://github.com/mvanhorn/last30days-skill/pull/229))
- @BryanTegomoh - --days alias backcompat ([#230](https://github.com/mvanhorn/last30days-skill/pull/230))
**v3 roadmap contributors (issues and PRs that shaped the v3 feature set):**
- @uppinote20 - rich Reddit comments ([#143](https://github.com/mvanhorn/last30days-skill/pull/143))
- @zerone0x - GitHub as a first-class source ([#134](https://github.com/mvanhorn/last30days-skill/issues/134), [#136](https://github.com/mvanhorn/last30days-skill/pull/136))
- @thinkun - Reddit enrichment timeout handling ([#116](https://github.com/mvanhorn/last30days-skill/pull/116))
- @thomasmktong - pure-Python Reddit fallback ([#124](https://github.com/mvanhorn/last30days-skill/pull/124))
- @fanispoulinakisai-boop - Reddit timeout report ([#100](https://github.com/mvanhorn/last30days-skill/issues/100))
- @pejmanjohn - plugin directory naming ([#99](https://github.com/mvanhorn/last30days-skill/issues/99), [#78](https://github.com/mvanhorn/last30days-skill/issues/78))
- @zl190 - HN trending merge ([#115](https://github.com/mvanhorn/last30days-skill/pull/115))
- @hnshah - Watchlist features ([#84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86))
- @Jah-yee, @dannyshmueli - Codex CLI discovery
- @Cody-Coyote - marketplace validation bug report ([#204](https://github.com/mvanhorn/last30days-skill/issues/204))
**The five Opus 4.7 instances that debugged their own failures on v3.0.7 and v3.0.8 and converged on the fixes.** The convergence was the breakthrough; this release is their diagnosis in code.
## Install / Update
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
```
Or if already installed:
```
/plugin update last30days
/reload-plugins
```
## Verify
```
cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version
```
Should print `"version": "3.0.9"`.
## Smoke test
```
/last30days birthday gift for 40 year old
```
Should ask a clarifying question before running. If it runs the engine anyway, the cache is stale - repeat the plugin update.
**Full Changelog:** https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
+48
View File
@@ -0,0 +1,48 @@
# Search Quality Eval
`skills/last30days/scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
- runs a baseline revision (default `origin/main`) against a candidate checkout
- evaluates the fixed 5 reviewer topics by default
- computes deterministic stability metrics:
- `Jaccard` overlap vs baseline
- retention vs baseline
- per-source counts and overlap
- optionally calls Gemini as a judge for graded relevance labels and then computes:
- `Precision@5`
- `nDCG@5`
- source-coverage recall across the judged union pool
Recommended usage:
```bash
uv run python skills/last30days/scripts/evaluate_search_quality.py
```
Useful flags:
```bash
uv run python skills/last30days/scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
--topic "cursor IDE pricing" \
--per-source-limit 5
```
Gemini configuration:
- preferred on this workspace: set `GOOGLE_API_KEY`
- also accepted: `GEMINI_API_KEY` or `GOOGLE_GENAI_API_KEY`
- optional: set `GEMINI_MODEL`
- default model is `gemini-3-pro-preview` for the direct Gemini API
Notes:
- The script forces a clean env-based auth path when it shells out to `last30days.py`.
- It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
- It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either.
- `Jaccard` and retention are regression guards, not truth metrics.
- `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.
@@ -0,0 +1,82 @@
---
title: Search-quality eval is manual by default, not a CI gate on every PR
date: 2026-05-10
category: docs/solutions/architecture
module: skills/last30days/scripts/evaluate_search_quality.py
problem_type: design_decision
component: ci_policy
severity: low
applies_when:
- a contributor proposes wiring search-quality eval into PR CI
- a change affects retrieval, ranking, grounding, or synthesis quality and a reviewer asks "why aren't we testing this in CI?"
- someone is deciding whether a new evaluator-style script belongs in the default CI workflow
related_components:
- search_quality_evaluation
- ci_workflow
- llm_judging
tags:
- ci-policy
- eval
- design-decision
- cost-vs-signal
- non-determinism
- manual-gates
---
# Search-quality eval is manual by default, not a CI gate on every PR
## Context
`skills/last30days/scripts/evaluate_search_quality.py` compares a baseline revision against a candidate revision across a fixed pool of reviewer topics. It produces two flavors of metrics: deterministic overlap (Jaccard, retention) and LLM-judged quality scores. The natural impulse on seeing an evaluator script is to wire it into CI on every PR — "regression catcher, run it automatically." We deliberately don't.
Three properties of this particular evaluator make CI-on-every-PR the wrong default:
1. **Live API access.** The candidate revision typically needs the engine to actually run, which means real ScrapeCreators calls, real reddit fetches, real YouTube searches. CI runs would either need production credentials or a record/replay fixture set that drifts almost immediately as external APIs change shape.
2. **Cost and latency.** A full eval pass runs the pipeline N times across reviewer topics. Multiplied by every PR (including doc-only PRs), the spend is meaningful and the wall-clock pushes CI from ~30s to many minutes.
3. **Non-determinism in the judging path.** The LLM-judged metrics are valuable for review but depend on judge-model behavior on a given day. A flaky eval that fails 1 PR in 20 because the judge re-scored an item differently is a worse CI signal than no eval at all — it teaches contributors to retry rather than read the result.
The deterministic overlap metrics are useful regression signals but they are not the same as user-facing correctness. A change that improves overlap can degrade synthesis quality; a change that drops overlap can be a deliberate improvement. So even the deterministic side isn't safe to auto-fail on.
## Guidance
### 1. Keep search-quality eval available, just not automatic
The script stays runnable by maintainers and contributors. The pattern is:
```bash
LAST30DAYS_PYTHON=python3.13 \
python3 skills/last30days/scripts/evaluate_search_quality.py \
--baseline main --candidate HEAD
```
Reviewers can request a manual eval run when a PR is in the retrieval/ranking/synthesis path and the risk warrants it. Contributors can run it locally before submitting if they want signal upfront.
### 2. Standard PR CI gates remain deterministic and contract-shaped
`pytest` (offline-safe), plugin-contract checks, version-consistency contracts, ruff/lint. Anything that returns the same answer twice for the same input. Quality-of-output assessment lives outside that loop.
### 3. The middle ground is `workflow_dispatch`, not auto-PR-gating
If maintainers want a GitHub-triggered eval that doesn't make every PR pay the live-API cost, the right shape is a manually-dispatched workflow (or a label-triggered one) — not a `pull_request:` workflow that runs unconditionally. That keeps the cost knob in human hands.
### 4. Revisit if the eval can ever be made offline-deterministic
The blocker is the live-API + non-determinism combination. If a future iteration of the script can compute meaningful Jaccard/retention metrics against static fixtures (no live API calls, no LLM judging), the decision flips and it becomes a candidate for default CI. The decision below tracks that condition; revisit when it's met.
## What this means in practice
- Don't merge PRs that wire `evaluate_search_quality.py` into the default `validate.yml` workflow.
- Do merge PRs that add `workflow_dispatch` triggers or label-gated runs.
- When reviewing a retrieval/ranking change, request a manual eval if the diff suggests it could regress quality — don't expect CI to catch it.
## Links
- `skills/last30days/scripts/evaluate_search_quality.py` — the evaluator script
- `docs/search-quality-eval.md` — user-facing usage documentation
- `.github/workflows/validate.yml` — the default CI workflow (deterministic gates only)
---
*Adapted from a draft ADR proposed by @hnshah in [#374](https://github.com/mvanhorn/last30days-skill/pull/374), restructured into the `docs/solutions/` convention. The original ADR text correctly identified the constraint; this version adds the "why workflow_dispatch is the middle ground" framing and the revisit-condition.*
@@ -0,0 +1,117 @@
---
title: Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items
date: 2026-06-09
category: docs/solutions/logic-errors
module: lib/rerank
problem_type: logic_error
component: search_ranking
severity: high
symptoms:
- on-entity, high-engagement items that name the brand but omit the trailing descriptor of a multi-word query are demoted in keyless/fallback rerank results
- observed case is a 323-point HN thread about Stripe scoring 0 on a "Stripe payments" query
- the entity-miss demotion lands twice (ENTITY_MISS_PENALTY on rerank_score plus a secondary final_score penalty), so a false miss guarantees burial regardless of engagement
- reddit keyless comment-enrichment slot selection skips the same on-entity threads via an independently duplicated full-phrase check in _slot_priority
root_cause: logic_error
resolution_type: code_fix
related_components:
- reddit_keyless
- comment_enrichment
tags:
- entity-grounding
- rerank
- keyless-fallback
- multi-word-entity
- substring-match
- false-demotion
- reddit-keyless
- duplicated-logic
---
# Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items
## Problem
The keyless/fallback rerank path's entity-grounding demotion required the FULL multi-word primary-entity phrase as a contiguous substring of the candidate's text (`primary_entity.lower() not in haystack`), so on-entity items that omitted a trailing search descriptor were falsely flagged as entity misses and buried by a deliberately decisive double penalty.
## Symptoms
- On a "Stripe payments" query, a 323-point HN thread titled "Stripe is friendly to 'friendly fraud'" was demoted to score 0 — purely because its text never contained the literal phrase "stripe payments" (the trailing word "payments" was missing).
- The burial is guaranteed by design, not incidental: a flagged entity miss takes 25 `ENTITY_MISS_PENALTY` on `rerank_score` in `_fallback_tuple`, PLUS `ENTITY_MISS_FINAL_PENALTY` applied directly in `_final_score` (added 2026-04-19 after engagement + freshness drowned the diluted penalty). A false positive on the check means confirmed-good signal cannot recover.
- The same over-strict check had been independently re-implemented in `reddit_keyless._slot_priority` (keyless Reddit comment-enrichment slot selection), so scarce comment slots were also steered away from head-token-only posts.
## What Didn't Work
- **Naively relaxing the check** — the full-phrase check existed for a real reason: on 2026-04-19 an off-topic video with zero brand mentions ranked #2 on a Hermes query (documented in the `ENTITY_MISS_FINAL_PENALTY` comment in `skills/last30days/scripts/lib/rerank.py`). Any fix had to keep that demotion firing.
- **Word-boundary matching** — rejected; it re-introduces over-demotion on plurals/possessives/compounds ("stripes", "Stripe's").
- **Graded penalty** (full-phrase = 0, head-only = half, none = full) — rejected; it half-punishes items that are 100% about the entity. Lexical coverage is not topical degree.
- **Any-token grounding** — rejected; "payments" alone would ground completely generic posts.
- **Distinctiveness gate for generic heads** — rejected as complexity to patch a failure mode that is already a safe no-op (see Why This Works).
- **Trusting the docstring**`reddit_keyless._slot_priority`'s docstring claimed to "mirror rerank's demotion signal," but its inline reimplementation (`entity in _post_text(post).lower()`) had silently drifted from being a mirror into being a second copy of the bug. It was found only by a code-reuse review, not by tests.
## Solution
Ground on the **head token** of the primary entity instead of the full phrase, via one shared helper used by both paths.
**Site 1 — new helper in `skills/last30days/scripts/lib/rerank.py`:**
```python
def _entity_grounded(haystack: str, primary_entity: str) -> bool:
tokens = primary_entity.lower().split()
if not tokens:
return True
return tokens[0] in haystack
```
`_fallback_tuple` switches from the inline phrase check to the helper:
```python
# before
if haystack.strip() and primary_entity.lower() not in haystack:
# after
if haystack.strip() and not _entity_grounded(haystack, primary_entity):
```
**Site 2 — secondary penalty in `_final_score`: no code change needed.** It keys off the explanation string set by site 1, so it inherits the fix automatically:
```python
if candidate.explanation and "entity-miss" in candidate.explanation:
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
```
**Site 3 — `skills/last30days/scripts/lib/reddit_keyless.py` `_slot_priority`:** replace the drifted reimplementation with a call to the shared helper:
```python
# before
return entity in _post_text(post).lower()
# after
return rerank._entity_grounded(_post_text(post).lower(), entity)
```
Tests: `tests/test_rerank_v3.py` gained `test_fallback_grounds_on_head_token_not_full_phrase` (the Stripe regression) and `test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` (guards the 2026-04-19 behavior). `tests/test_reddit_keyless.py`'s two old-contract tests were rewritten as `test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match`.
## Why This Works
- **Root cause:** trailing tokens of a multi-word query ("payments" in "Stripe payments") are usually category descriptors the user/planner appended for search, not part of the entity name. Requiring the whole phrase conflates "doesn't repeat my search phrasing" with "isn't about my entity." The brand head token alone is sufficient grounding; items that never name the brand at all still miss the head token and stay demoted — so the original 2026-04-19 fix keeps firing.
- **Asymmetry argument:** the demotion is engineered to be decisive (double penalty across `rerank_score` and `final_score`), so a false entity-miss is fatal-by-design, while a false grounding merely defers the item to normal relevance/freshness/quality ranking. When the punishment is capital, the conviction standard should be conservative.
- **Substring (not word-boundary) is deliberate:** it catches plurals/possessives/compounds ("stripes", "Stripe's"). Degenerate short heads ("X", "Go", "C") make the check vacuously true, which merely **disables** the penalty — reverting to the pre-grounding baseline — rather than burying good items. Every failure mode of this rule degrades toward "no penalty," never toward "bury good signal."
- **Accepted, bounded limitation:** head-collision with a different famous entity ("Hermes Agent" → a "Hermes Birkin" thread now escapes demotion). This is lexically unfixable — any token rule strong enough to kill the collision re-kills the Stripe case; the discriminator is semantic. The LLM rerank path (which receives the full phrase as prompt guidance and judges semantically) covers this when API keys exist; the keyless path accepts the bounded risk.
## Prevention
- **Shared helper as single source of truth:** when one module's behavior must "mirror" another's signal, it must *call* the same function, not re-implement the check. The `reddit_keyless._slot_priority` drift happened precisely because the mirror was a copy. The fix wires it to `rerank._entity_grounded`, and the docstring now states this explicitly: "keying on the same head token keeps the two paths from diverging."
- **Docstrings record deliberate trade-offs:** `_entity_grounded`'s docstring documents WHY head-token (not phrase), why substring (not word-boundary), and the safe-failure direction. Future readers see the rejected alternatives were considered, not overlooked — and won't "tighten" the check into a regression.
- **Both directions pinned by named tests:**
- `tests/test_rerank_v3.py::test_fallback_grounds_on_head_token_not_full_phrase` — false-demotion regression (the Stripe HN thread must not be flagged).
- `tests/test_rerank_v3.py::test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` — the fix must not neuter the demotion (guards the 2026-04-19 off-topic-video incident).
- `tests/test_reddit_keyless.py::test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match` — the mirrored path asserts the same contract.
- **Audit tests when changing a contract:** tests that encode the old behavior as correct must be rewritten to the new contract, not worked around — the two old `test_reddit_keyless.py` tests would have silently re-blessed the bug.
- **For decisive penalties, route through one flag:** the `_final_score` backstop keys off `"entity-miss" in candidate.explanation` rather than re-running the check — so there was exactly one site to fix and the second penalty inherited it for free. Prefer this signal-propagation pattern over duplicating predicate logic at each penalty site.
## Related Issues
- [PR #484](https://github.com/mvanhorn/last30days-skill/pull/484) — "fix(reddit): relevance-aware comment-enrichment slot selection in keyless path" — introduced the `_slot_priority` mirror this fix reroutes through the shared helper.
- [PR #457](https://github.com/mvanhorn/last30days-skill/pull/457) — "fix(reddit): restore free path via keyless RSS + shreddit scrape" — established the keyless Reddit path.
- [PR #488](https://github.com/mvanhorn/last30days-skill/pull/488) (open) — "fix(reddit): relevance floor + relevance-first ranking" — external PR touching the same ranking surface; coordinate before merging both.
- [Issue #468](https://github.com/mvanhorn/last30days-skill/issues/468) (open) — relevance scoring over-pruning on-topic YouTube items; same symptom family in a different source.
- [../architecture/search-quality-eval-manual-by-default-2026-05-10.md](../architecture/search-quality-eval-manual-by-default-2026-05-10.md) — how to validate ranking/grounding changes like this one (manual eval, not CI-gated).
- [../workflow-issues/release-consistency-test-cascade-2026-05-16.md](../workflow-issues/release-consistency-test-cascade-2026-05-16.md) — sibling prevention pattern: lockstep artifacts drift unless mechanically unified.
@@ -0,0 +1,219 @@
---
title: Release-time consistency tests cause cascade CI failures across all open PRs
date: 2026-05-16
category: docs/solutions/workflow-issues
module: ci-release-engineering
problem_type: workflow_issue
component: testing_framework
severity: high
applies_when:
- a test asserts consistency between two release-time artifacts (e.g., SKILL.md version and a hardcoded pin in a shell script)
- one artifact is updated as part of a version bump and the other requires a manual lockstep update
- multiple long-lived PRs are open simultaneously against the same base branch
symptoms:
- every open PR's CI fails after a version bump even though the PRs are unrelated to versioning
- the failing test references a stale hardcoded value that was not updated alongside the bumped version
- PR authors must rebase and manually fix an artifact they did not touch
root_cause: missing_workflow_step
resolution_type: code_fix
related_components:
- development_workflow
- documentation
tags:
- ci
- release-engineering
- consistency-test
- version-pin
- cascade-failure
- test-design
- workflow
---
# Release-time consistency tests cause cascade CI failures across all open PRs
## Context
A `tests/test_version_consistency.py::test_sync_cache_path_uses_skill_version` test was added to enforce that the version string embedded in `skills/last30days/scripts/sync.sh` (a hardcoded plugin-cache path segment) matched the version frontmatter in `skills/last30days/SKILL.md`. The intention was sound: the cache path had to stay in lockstep with the skill version or the sync would silently pull stale files.
The test worked as designed until a release shipped. At that point it turned into a cascade-failure machine:
1. A release PR bumps `SKILL.md` version (e.g., 3.2.0 → 3.2.1) **and** bumps the `sync.sh` pin. That PR's CI is green.
2. The release PR merges to `main`.
3. Every PR that was open at merge time was branched from pre-release `main`. Those PRs have `SKILL.md` 3.2.1 (inherited via merge-base with `main`) but their branch never touched `sync.sh`.
4. CI for those PRs runs the consistency test against the new `main``SKILL.md` says 3.2.1, `sync.sh` still says 3.2.0 — and fails.
5. All open PRs are now red simultaneously, with a failure that has nothing to do with their changes.
This affected at least five PRs during the 2026-05-13 to 2026-05-15 window: PR #400 (caught during rebase, required a manual pin bump), PRs #390 and #392 (OpenClaw `SCRAPECREATORS_API_KEY` fix, both stalled for the same stale-pin reason), and at least two others. A follow-up hotfix PR (#397`fix(sync): bump cache target to 3.2.1 to match SKILL.md`) was required just to unblock the queue.
The permanent fix was PR #405: delete `sync.sh` entirely (the install workflow made it redundant) and drop `test_sync_cache_path_uses_skill_version`. Once both were gone, no version-consistency cascade was possible.
## Guidance
### 1. Don't write consistency tests that read two files and assert one matches a substring derived from the other
This pattern looks safe but is not:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # reads SKILL.md
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text, # asserts sync.sh contains that string
)
```
It encodes the assumption that both files are always updated together, in the same commit, on the same branch. That assumption breaks the moment two files have independent lifecycle owners — a versioned manifest and a deployment script are archetypal examples.
### 2. If the values genuinely need to stay in sync, derive one from the other at runtime
Remove the hardcoded pin from `sync.sh` and compute it:
```bash
# sync.sh — derive version from SKILL.md at runtime, no pin to maintain
SKILL_VERSION=$(grep -m1 '^version:' "$(dirname "$0")/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
CACHE_PATH="last30days-skill/last30days/${SKILL_VERSION}"
```
Now there is only one source of truth (`SKILL.md`). The test that asserted they matched becomes vacuous and should be deleted. If `SKILL.md` is wrong, the sync itself will fail loudly — which is better feedback than a CI gate on a different PR.
### 3. If two values must stay independent for legitimate reasons, update them together and make the test self-skip if either source is missing
If separate versioning is genuinely required (e.g., SKILL.md versions for harness consumers, sync.sh versions a private artifact store with its own cadence), update both in the same PR — never staggered — and write the test to self-skip rather than error when either file is absent:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_sh = SKILL_ROOT / "scripts" / "sync.sh"
if not sync_sh.exists():
self.skipTest("sync.sh not present; skipping pin consistency check")
sync_text = sync_sh.read_text(encoding="utf-8")
version = _skill_version()
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text,
)
```
Self-skipping means deleting the file is a non-event in CI — no cascading red, no hotfix PR to the queue.
### 4. Run consistency tests against the merge-base diff, not main
If you keep a two-file consistency test, scope it so it only fails when the PR itself modifies one of the two files but not the other. A GitHub Actions step can do this:
```yaml
- name: Check sync.sh version pin consistency
run: |
BASE=$(git merge-base HEAD origin/main)
SKILL_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'SKILL\.md' || true)
SYNC_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'sync\.sh' || true)
if [ "$SKILL_CHANGED" -gt 0 ] && [ "$SYNC_CHANGED" -eq 0 ]; then
echo "SKILL.md version bumped but sync.sh pin was not updated"
exit 1
fi
```
This only fires when your PR touched `SKILL.md` and left `sync.sh` alone — never because a release merged to `main` after you branched.
### 5. Ask whether you actually need this test
If the values are wrong, downstream tooling will fail loudly: the sync will fetch the wrong artifact, installs will break, or the harness will reject the version. A test that exists only to catch a human-bookkeeping error at release time adds cascade-fail risk without offering a meaningfully earlier signal. Weigh that cost before adding any two-file consistency gate.
## Why This Matters
The damage from a stale-pin consistency test is asymmetric. It:
- Fails on every open PR simultaneously the moment a release lands on `main` — not just the PR that forgot to update the pin.
- Produces a failure message that points at a line in a test file with no obvious relationship to the PR's actual changes.
- Requires either a hotfix PR (touching a file the failing PRs have no business touching) or a manual rebase of every affected branch.
- Blocks work that has already been reviewed and approved.
In this repo the effect was measurable: at least five PRs stalled across a two-day window, one hotfix PR was shipped just to unblock the queue, and multiple authors spent time debugging a failure completely unrelated to their changes.
The broader principle is that tests which gate on *bookkeeping consistency between files* impose their maintenance cost on every contributor, every time, even when those contributors did nothing wrong. That cost compounds with team size and release cadence.
## When to Apply
Apply this guidance whenever you find yourself:
- Writing a test that reads two files and asserts that a string in one matches a value derived from the other.
- Adding a CI step labeled "consistency check," "sync check," or "pin check" where the check compares a hardcoded value against a computed one from a separate file.
- Working in a repo where a versioned manifest (e.g., `SKILL.md`, `package.json`, `pyproject.toml`) and a deployment artifact (e.g., a shell script, a Dockerfile, a Helm values file) are both maintained by hand.
- Reviewing a PR that touches only one of two "paired" files and fails a consistency test for the other.
It does *not* apply to tests that read a single source of truth and validate its internal structure (e.g., asserting that `SKILL.md`'s frontmatter version is double-quoted, or that `package.json`'s `version` field is a valid semver string). Those tests have one file and one assertion; they cannot cascade across branches.
## Examples
### Before — the pattern that caused the cascade
Original `tests/test_version_consistency.py` (deleted in commit `9fb19ea`):
```python
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days"
def _skill_version() -> str:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
if not match:
raise AssertionError("SKILL.md version frontmatter not found")
return match.group(1)
class TestVersionConsistency(unittest.TestCase):
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # source 1: SKILL.md frontmatter
self.assertIn( # assertion: sync.sh must contain
f'last30days-skill/last30days/{version}"',
sync_text, # source 2: hardcoded string in sync.sh
)
```
`sync.sh` contained a line like:
```bash
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/3.2.0"
```
When SKILL.md bumped to `3.2.1` in a release PR, `sync.sh` was updated in the same PR and CI stayed green. But every PR branched before that release still had `sync.sh` at `3.2.0`. Their CI failed immediately, with an assertion error pointing at the test, not at the release PR.
### After — what we did: delete both
PR #405 deleted `sync.sh` (the install workflow replaced it) and dropped `test_sync_cache_path_uses_skill_version` in the same change. No consistency gate, no pin to maintain, no cascade possible.
### After — what we could have done instead: derive at runtime
If `sync.sh` had still been needed, the right fix would have been to remove the hardcoded version from the script and derive it from `SKILL.md`:
```bash
#!/usr/bin/env bash
# sync.sh — no hardcoded version; reads SKILL.md as single source of truth
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SKILL_VERSION=$(grep -m1 '^version:' "${SCRIPT_DIR}/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
if [ -z "$SKILL_VERSION" ]; then
echo "error: could not parse version from SKILL.md" >&2
exit 1
fi
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/${SKILL_VERSION}"
# ... rest of sync logic
```
With this in place, `test_sync_cache_path_uses_skill_version` has no reason to exist — there is nothing to assert. Delete it. If the version parsing breaks, `sync.sh` itself exits non-zero with a clear message.
## Related
- **PR #397** (merged) — `fix(sync): bump cache target to 3.2.1 to match SKILL.md`. The hotfix that unblocked the cascade temporarily by bumping the pin.
- **PR #400** (merged) — caught the same cascade during rebase; had to bump the pin to clear CI.
- **PR #390** (closed) and **PR #392** (rebased + merged) — OpenClaw `SCRAPECREATORS_API_KEY` fix; both blocked by the cascade until rebased onto post-#405 main.
- **PR #405** (merged) — the permanent fix: deleted `sync.sh` + `test_sync_cache_path_uses_skill_version` together.
- **PR #412** (merged) — adjacent work that consolidated SKILL.md version parsing into `lib/skill_meta.py`, reducing future drift risk by giving the version field one canonical reader.
+18 -20
View File
@@ -13,7 +13,7 @@ YouTube transcripts are the second headline feature. Inspired by Peter Steinberg
**New in V2.1 — two headline features:**
- **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a tweet — now the skill reads it. Inspired by @steipete's yt-dlp + summarize toolchain.
- **X search is fully bundled.** No external `bird` CLI or xAI API key needed. Just Node.js 22+ and your browser cookies. Uses a vendored subset of Bird's Twitter GraphQL client (MIT licensed, originally by @steipete).
- **X search is fully bundled.** No external `bird` CLI install needed. Add `AUTH_TOKEN` and `CT0` once, and the vendored Bird client runs locally without browser-cookie prompts. `XAI_API_KEY` remains an optional fallback.
---
@@ -21,20 +21,18 @@ YouTube transcripts are the second headline feature. Inspired by Peter Steinberg
### X Search Authentication
X search reads your existing browser cookies — no API keys or login commands needed.
X search prefers explicit env auth. This keeps local runs headless and avoids browser-cookie and macOS Keychain prompts.
**Safari (recommended on Mac):** Just be logged into x.com. No setup needed.
**Recommended setup:** While logged into x.com once, open browser dev tools and copy the `auth_token` and `ct0` cookies for `x.com`.
**Chrome:** Works, but macOS will prompt you to allow Keychain access the first time. Click "Allow" (or "Always Allow" to stop future prompts).
**Firefox:** Just be logged into x.com. No setup needed.
**Manual fallback:** If cookie auto-detection doesn't work, set these env vars (grab them from your browser's dev tools → Application → Cookies → x.com):
Save them as `AUTH_TOKEN` and `CT0` in `~/.config/last30days/.env` or `.claude/last30days.env`:
```bash
export AUTH_TOKEN=your_auth_token
export CT0=your_ct0_token
AUTH_TOKEN=your_auth_token
CT0=your_ct0_token
```
**xAI fallback:** If you do not want to provide `AUTH_TOKEN` and `CT0`, set `XAI_API_KEY` and use xAI's `x_search` backend instead.
**Verify it's working:**
```bash
node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
@@ -44,8 +42,10 @@ node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs
## README: Install block env line
```
XAI_API_KEY=xai-... # optional — cookie auth is default for X search
```bash
AUTH_TOKEN=... # recommended for X search
CT0=... # recommended for X search
XAI_API_KEY=xai-... # optional X fallback
```
---
@@ -60,15 +60,13 @@ XAI_API_KEY=xai-... # optional — cookie auth is default for X search
## GitHub issue #19 response (post AFTER publishing)
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted steipete was asked to take it down.
> Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted. steipete was asked to take it down.
>
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly — no external CLI, no `npm install`, no brew. Just Node.js 22+ and your browser cookies.
> The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly. No external CLI, no `npm install`, no brew. Just Node.js 22+ plus `AUTH_TOKEN` and `CT0`, or `XAI_API_KEY` as fallback.
>
> It also adds **YouTube as a 4th source** — when yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
> It also adds **YouTube as a 4th source**. When yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
>
> If you're on a Mac, Safari is the easiest path for X — just be logged into x.com. Chrome works too but macOS will prompt for Keychain access the first time.
>
> If cookie auto-detection doesn't work, you can set `AUTH_TOKEN` and `CT0` env vars manually (grab from browser dev tools → Application → Cookies → x.com).
> The recommended setup is to copy `auth_token` and `ct0` from x.com once and store them as `AUTH_TOKEN` and `CT0` in your env. That avoids browser-cookie and Keychain prompts during normal runs.
>
> The xAI API (`XAI_API_KEY`) also still works as a fallback.
@@ -82,7 +80,7 @@ XAI_API_KEY=xai-... # optional — cookie auth is default for X search
Two new features:
→ YouTube transcripts as a 4th source (yt-dlp)
→ X search fully bundled (no bird CLI needed)
→ X search fully bundled (no bird CLI install needed)
Research any topic across Reddit, X, YouTube & web in one command.
@@ -100,7 +98,7 @@ When yt-dlp is installed, the skill searches YouTube, grabs view counts, and ext
### Thread version (post 2)
2️⃣ X search is fully bundled
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Just be logged into x.com in your browser. No npm install, no API keys.
Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Add `AUTH_TOKEN` and `CT0` once, then it runs locally with no npm install. `XAI_API_KEY` still works as fallback.
Both features inspired by @steipete's tooling.
-250
View File
@@ -1,250 +0,0 @@
# v2.1 Launch Posts - WORKING DRAFT
## Post 1 (Hook)
V2.1 of @slashlast30days launches today. Three headline features:
1. @openclaw + watchlists - automated research on your competitors, people, and topics
2. YouTube transcripts as a 4th source
3. Works in OpenAI Codex
## Post 2 (Watchlist + Open Claw - THE KILLER FEATURE)
@openclaw + WATCHLISTS.
Pair /last30days with @openclaw and it re-researches topics on a schedule across Reddit, X, YouTube, and the web.
"last30 watch my biggest competitor every week"
"last30 watch Peter Steinberger every 30 days"
"last30 watch AI video tools monthly"
Research that runs while you sleep. Designed for @openclaw and always-on bots.
## Post 3 (YouTube)
YOUTUBE IS NOW A 4TH SOURCE.
The skill searches YouTube, grabs view counts, and reads the actual transcripts. A 20-minute review has 10x the signal of one X post - now the skill reads it.
## Post 4 (Codex)
WORKS IN OPENAI CODEX.
Same skill, same engine, same four sources. Install to ~/.agents/skills/last30days and invoke with $last30days. Claude Code and Codex users get the same research.
## Post 5 (Example: Seedance 2.0 access)
Asked it how to access Seedance 2.0.
3 Reddit threads. 31 X posts. 20 YouTube videos (685K views, 4 transcripts read). 10 web pages. All four sources hit.
It found the real answer buried in Chinese YouTube tutorials: Little Skylark (Xiao Yun Que) - zero cost, no queue, no VPN. Just select Seedance 2.0 from the model dropdown. Also surfaced: Disney sent ByteDance a cease-and-desist over uncensored IP generation.
## Post 6 (Example: AI Generated Ads)
Asked it about AI generated ads.
12 Reddit threads. 29 X posts. 3 YouTube videos (83K views, 3 transcripts read). 30 web pages.
The finding that stuck: Svedka ran the first "primarily AI-generated" Super Bowl spot. Brand match: 7%. Industry norm: 63%. Meanwhile 86% of ad buyers are planning to use AI for video ads anyway. Cost is winning over quality.
## Post 7 (Example: Peter Steinberger)
Asked it about @steipete.
30 X posts. 5 YouTube videos (112K views). Found the Lex Fridman interview from 3 days ago.
Key reveal: OpenAI and Meta both made acquisition offers for OpenClaw. He said no. He's losing $10-20K/month maintaining it. "A fun project became a world project."
## Post 8 (Install)
Tell your @openclaw bot:
"Install the last30days skill from github.com/mvanhorn/last30days-skill"
That's it. One message.
## Post 9 (Close)
Thank you to @hutchins for pushing me to add YouTube and to @steipete whose summarize tool showed me how yt-dlp could power transcript extraction.
Try it: last30 [any topic]
github.com/mvanhorn/last30days-skill
PS: @steipete ClawHub login is broken right now so we can't publish the official skill there yet. Hoping for a fix soon.
---
## Standalone Posts
### Greg Isenberg best tips
Prompt: `last30 greg isenberg best tips`
1 Reddit thread. 6 X posts. 4 YouTube videos (all transcribed). 8 web pages.
His core playbook is ACP: Audience, Community, Product. Not the other way around.
Big thesis: "2026 is the GREATEST time to build a startup in 30 years." Boring industries, AI agents, rapid dev tools collapsing build time.
Most-watched: "Clawdbot Clearly Explained" (273K views), "Claude Code Built My $450K Marketing Campaign" (40K views), daily workflows with Kitze (83K views).
@gregisenberg @slashlast30days
### Lenny Rachitsky best learnings
Prompt: `last30 lenny rachitsky top learnings`
5 Reddit threads. 29 X posts. 20 YouTube videos (1.1M+ views, 5 transcribed). 30 web pages.
"Execution is no longer the bottleneck. Clarity is." That's the thesis running through everything @lennysan has been publishing lately.
He open-sourced 320 episode transcripts and the community went wild. 87 skills got built from them. Someone on Reddit distilled 86 discrete product skills from 100+ episodes.
Recent guest highlights: Sherwin Wu (OpenAI) says 95% of their engineers use Codex daily. Marc Andreessen: "This is as normal as it's going to be. It's going to be much weirder very soon." Dalton Caldwell (YC): "just don't die" and avoid tar pit ideas.
@lennysan @slashlast30days
---
## NOTES
- Thread leads with Open Claw + watchlist as the killer feature - pair with an always-on bot for automated research
- YouTube is the #2 hero - the stats ("685K views, 4 transcripts read") prove it works
- Codex compatibility is #3 - brief but shows cross-platform reach
- Example posts prove quality with verified results from real runs
- Consider screenshots of the actual output for each example post
- @steipete credit in close - he inspired YouTube (yt-dlp toolchain) and X search (Bird MIT code)
- @slashlast30days vs /last30days - which handle? Used /last30days above since it's the actual command
---
## RAW RESULTS (for reference / pulling quotes)
### Nano Banana Pro prompting (PROMPTING) - VERIFIED 2/15
Stats: 0 Reddit (timed out) | 32 X posts | 164 likes | 22 reposts | 5 YouTube videos | 98,539 views | 5 with transcripts | 10 web pages
Top voices: @KusoPhoto (106 likes), @TzqQaiser (35 likes) | Jake Dawson (15K views), AI Master (37K views)
Key findings:
- Structured JSON prompts are the meta - nested fields for character, scene, lighting beat plain prose. @TzqQaiser's viral post shows the format.
- Design brief > keyword stuffing - "The second I started writing prompts like a real design brief, everything changed" - Jake Dawson on YouTube
- 6-factor formula: Subject, Composition, Action, Setting, Style, then refine with camera/lighting - per Google's official blog
- ICS framework for infographics: Image type + Content + Style - leverages Nano Banana Pro's unique legible text rendering
- Scale logic for cinematic compositions - define size relationships and camera distance explicitly, per @Strength04_X
- Nano Banana Pro → video pipeline trending (generate image, animate with Kling 3.0 or Veo 3.1) - per @KusoPhoto
### Peter Steinberger / OpenClaw creator (GENERAL) - VERIFIED 2/15
Stats: 31 X posts | 0 Reddit (quiet) | YouTube timed out on 2 | 4 web pages
Top voices: @steipete | Lex Fridman podcast
Key findings:
- Lex Fridman podcast (Feb 12) went viral - "One of the most honest discussions I've seen"
- OpenAI and Meta made acquisition offers (conditional on keeping project open) - he declined
- Losing $10-20K/month maintaining OpenClaw, rejected crypto tokenization for funding
- 180K+ GitHub stars, 6,600 commits in 1 month - "A fun project became a world project"
- Also built: gogcli (Google Workspace CLI), summarize (URL/YouTube summarizer), bird (X/Twitter CLI)
- Pragmatic Engineer: "I ship code I don't read"
- Prediction: AI agents could dominate >60% of software economy by 2030
### Seedance 2.0 Prompting (PROMPTING) - VERIFIED 2/15
Stats: 21 Reddit threads | 33 X posts | 20 YouTube videos | 5 web pages
Top voices: @charliebcurran (61K+ likes) | r/AI_Agents, r/ChatGPT, r/PromptEngineering | AI Search (127K views), Theoretically Media (157K views), Dan Dingle (126K views)
Key findings:
- "Slow and continuous" is the #1 prompting secret - rough state transitions = worse outcomes, per r/AI_Agents
- Include timings in prompts (e.g., "0-3s: character walks, 3-6s: turns head") - per r/ChatGPT
- Image-to-video for consistency - start with a reference image, not text-only
- English works just as well as Chinese - per r/AI_India
- CapCut integration coming = "every 12 year old in America will have this superpower"
- Cost: ~$0.55/10s clips (~$3.30/min), Seedance 3.0 rumored at 1/8th price
- Prompt resources: GitHub repo of curated prompts, Prompt Director Pro (440 settings system)
- Top YouTube tutorials: "Seedance 2.0 crushes everything" (127K), "Claims the AI Video Throne" (157K), "ABUSING China's Crazy New Video AI" (126K)
### OpenClaw best use cases for business (RECOMMENDATIONS)
Stats: 35 Reddit threads | ~1,130 upvotes | ~566 comments | 23 X posts | ~24 likes | 20 YouTube videos | ~1,572,000 views | 5 with transcripts | 10 web pages
Top voices: @gio__aa (8 likes), @ericosiu, @artyomx | r/openclaw, r/clawdbot, r/LocalLLaMA
Key findings:
- Email & Inbox Automation - 8+ mentions. One user cleared 4,000+ emails in two days. 10-15 hours/week saved.
- Business Dashboards & Real-Time Reporting - 6+ mentions. @gio__aa: "Business dashboards are going to become one of the most popular use cases."
- Morning Briefings - 5+ mentions. Pulls from calendars, weather, emails, RSS, GitHub, Hacker News on a schedule.
- Content & SEO Pipelines - 5+ mentions. @ericosiu claims "$45k of pSEO work in 20 minutes."
- Full CRM & Business Operations - 4+ mentions. @artyomx runs a daycare business, legal cases, and family comms through it with 5 AI agents.
- Client Onboarding & Support - 4+ mentions. "70% of tickets handled autonomously."
- Competitive Monitoring & Scraping - 4+ mentions.
- Wrapper/Hosting SaaS - 3+ mentions. Building commercial wrappers around OpenClaw as a business.
Cautions: malware in a top-downloaded skill (236 upvotes on r/LocalLLaMA), $25-50/day token burn risk, hours of config for marginal savings.
### YouTube thumbnail tips (GENERAL)
Stats: 7 Reddit threads | 654 upvotes | 176 comments | 32 X posts | 110 likes | 53 reposts | 18 YouTube videos | 6,150,368 views | 5 with transcripts | 30 web pages
Top voices: @TeamYouTube, @thewindwolf64 | r/NewTubers, r/YouTubeThumbnailHub | Think Media (1.17M views), whirow (1.46M views)
Key findings:
- Simplicity is #1 - r/NewTubers post (654 upvotes) from someone who designed 346 thumbnails: one subject, one message, one second to understand. 3+ elements = ~23% lower CTR.
- Less text = more clicks - Under 4 words gets ~30% higher CTR. Mobile thumbnails shrink to 168x94px - text becomes unreadable.
- Faces still win but subtlety is trending - Faces boost CTR 20-30%, but exaggerated shock face is giving way to authentic expressions in 2026.
- "UnThumbnails" are a counter-trend - Nate Black (71K views): deliberately raw, less-designed thumbnails that stand out.
- AI tools changing the game - Nick Nimmin (90K views) showed free AI tools democratizing thumbnail creation.
- A/B test everything - YouTube's built-in thumbnail testing lets you test up to 3 versions per video.
### AI SaaS crash (NEWS)
Stats: 9 Reddit threads | 31 upvotes | 52 comments | 32 X posts | 39 likes | 2 reposts | 20 YouTube videos | 929,648 views | 5 with transcripts | 30 web pages
Top voices: @jasonlk (15 likes), @WarrenInTheBuff (11 likes), @xankriegor_ | r/SaaS, r/aiwars
Key findings:
- "SaaSpocalypse" - $285B wiped in a single day (Feb 3, 2026) after Anthropic launched Claude Cowork. Total losses exceeded $1T. Salesforce down 27% YTD, Oracle halved.
- @jasonlk: "The real inflection point wasn't January 2026. It was June 2024 - when Claude 3.5 Sonnet shipped." Public SaaS growth rates declined every quarter since 2021 peak.
- Seat-based pricing is the casualty - 10 AI agents replace 100 sales reps = no need for 100 Salesforce seats. $470B+ hyperscaler AI spend coming from enterprise software budgets.
- Not everyone buying the doom - Jensen Huang called it "the most illogical thing in the world." BofA called selloff irrational.
- Indian IT hit especially hard - biggest sell-off since 2020.
### Seedance 2.0 access (GENERAL) - VERIFIED 2/15, ALL 4 SOURCES
Stats: 3 Reddit threads | 114 upvotes | 183 comments | 31 X posts | 191 likes | 13 reposts | 20 YouTube videos | 685,297 views | 4 with transcripts | 10 web pages
Top voices: @markgadala (116 likes), @OrctonAI, @nemovideoai | Theoretically Media (158K views) | r/AIHubSpace
Key findings:
- Little Skylark (Xiao Yun Que) = best free method - zero cost, no queue, manually select Seedance 2.0 from model dropdown, per YouTube tutorials
- Jimeng (Dreamina) - 1 RMB trial (~$0.14), ~260 daily free credits, but severe congestion with hours-long waits for free users
- Doubao App - 10 free video gens/day, requires joining Feishu/Lark group and submitting UID (1-2 day wait)
- Feb 24 = global unlock - Dreamina + CapCut + API access through BytePlus
- IP controversy exploding - @markgadala's "fully uncensored Seedance 2" post (116 likes) went viral, Disney sent C&D to ByteDance, SAG-AFTRA slammed "blatant infringement" over AI Tom Cruise/Brad Pitt fight videos
- Third-party race - NemoVideoAI, ChatCut, RecCloud, Morph Studio all competing to be the English-language access point
- Quality consensus: "crushes everything" (AI Search, 128K views), "claims the AI video throne" (Theoretically Media, 158K views)
### AI Generated Ads (GENERAL) - VERIFIED 2/15, ALL 4 SOURCES
Stats: 12 Reddit threads | 5 upvotes | 15 comments | 29 X posts | 101 likes | 3 reposts | 3 YouTube videos | 82,896 views | 3 with transcripts | 30 web pages
Top voices: @CaptainMcKlide (77 likes), @ugcbykaytelynn | r/editors, r/AI_UGC_Marketing, BERNTH (39K views)
Key findings:
- Super Bowl 2026 was the watershed - 23% of ads (15/66) featured AI, reception "sharply negative," nearly 50% of social mentions critical
- Svedka ran the first "primarily AI-generated" national Super Bowl spot - brand match of just 7% vs 63% alcohol industry norm
- Massive perception gap - 82% of ad execs think Gen Z feels positive about AI ads, but only 45% of consumers do (IAB). Gen Z most hostile at 39% negative.
- AI UGC booming in e-commerce - r/AI_UGC_Marketing active hub, tools: Creatify, MakeUGC, ArcAds targeting dropshippers
- Quality still low - r/dropshipping: "the hand flip and rubbing on the face looks fake"
- @ugcbykaytelynn warns "AI generated ads RUIN your brand's image"
- Cost winning over quality - 86% of ad buyers using or planning gen AI for video ads, cost efficiency #1 driver (64%), per IAB
- BERNTH on YouTube bought AI-generated guitar product ads, documented absurdity - four-fingered hands, instruments don't match listings (39K views)
- Trust erosion spreading - people now question whether ANY media is real, even billboards, per @N0rbertas
### last30days skill (META/GENERAL) - VERIFIED 2/15
Stats: 0 Reddit | 30 X posts | 1,371 likes | 107 reposts | 5 YouTube videos | 112,082 views | 3 with transcripts | 10 web pages
Top voices: @gregisenberg (1,290 likes), @mvanhorn (34 likes) | Alejandro AO (39K views), Greg Isenberg (28K views)
Key findings:
- @gregisenberg's post (1,290 likes, 106 RT) + YouTube video (28K views) "The Claude Code Skill My Smartest Friends Use" was the breakout moment
- v2 feedback loop active - @trevin flagged OpenAI web_search not finding niche Reddit posts, suggested Brave API. @jonthebeef submitted PR for --days flag.
- People building on top - @tjarkoleifer created "re-skill" meta skill, @rajachirravuri recommends it as part of a PM stack
- Coverage: Alejandro AO crash course (39K views, 1,049 likes), Jason Calacanis on This Week in Startups (24K views)
- 1.5K GitHub stars, listed on skills.sh and Smithery
- Grok itself correctly attributed the skill when asked about ithah
+360
View File
@@ -0,0 +1,360 @@
# last30days v2.5 Launch Thread
## FINAL THREAD (6 tweets)
### 1/6 - Announcement
I can't believe it's been 30 days since I launched @slashlast30days. 3.2k stars later, time for v2.5.
Three big additions:
1. @Polymarket prediction markets as a 6th source - helps you predict the future
2. Cross-source linking + massively better results - detects when the same story trends across multiple platforms. Ran a 15-way blinded comparison, v2.5 scored 4.38 vs 3.73 for the original. Won all 5 topics.
3. Hacker News as a 5th source - a window into the tech and developer insider world
github.com/mvanhorn/last30days-skill
### 2/6 - Demo: Anthropic vs Pentagon
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views). 5 HN stories. 9 Polymarket markets.
This story broke TODAY. Hegseth designated Anthropic a "supply chain risk." Trump ordered every agency to stop using their tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March: only 6%.
Markets say Anthropic wins regardless. That's the kind of signal you can't get from opinion threads.
### 3/6 - Demo: Seedance Prompting
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. Research first, then create from what you learned.
### 4/6 - Demo: Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Arizona is 25-2, set a program record with a 22-0 start, and holds a 2-game Big 12 lead with 3 games left. The Field of 68 called them "the TOUGHEST team in America" after escaping Baylor shorthanded. Kansas rematch Saturday - the highlight video from their first meeting has 248K views on ESPN's YouTube.
Polymarket: Championship 13%. #1 seed: 88%. Duke and Michigan each at 18% to win it all.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
### 5/6 - Demo: Iran War
"/last30days iran war"
2 Reddit threads. 34 X posts (10,048 likes). 20 YouTube videos (1.6M views, 5 transcripts). 4 HN stories (850 points). 14 Polymarket markets ($473M volume).
Geneva talks just ended without a deal. 150+ US aircraft deployed. Two carrier strike groups in position. F-22s sent to Israel. Members of Congress who saw the secret war plan came out "terrified." @cenkuygur: "they are about to drag us into a war that 70-85% of Americans oppose" (7,700 likes).
Polymarket ($473M in volume - one of their biggest markets ever): strikes by 2026: 80%. By March 31: 68%. War Powers invoked: 51%. Formal war declaration: only 12%.
Markets say: strikes are very likely, declared war is not. That's the sharpest signal in the entire research.
### 6/6 - Thank You
Thank you to ARJ999 and wkbaran on GitHub who filed three separate issues asking for Hacker News support. v2.5 delivers.
It's been a crazy 30 days. 3.2k stars. Six sources. Massively better results. Super excited to get this out.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
---
## REFERENCE MATERIAL BELOW
## Context
- 3.2k stars on GitHub
- V2.5 headline features: Polymarket (6th source), Hacker News (5th source), cross-source linking
- Ran 15-way blinded comparison: 4.38/5.0 vs 3.73/5.0
- Won all 5 topics, zero regressions
- Cross-source linking: 3 -> 13 linked items
- Demo topics: Anthropic odds (11 markets), Arizona basketball (6 markets), Iran war ($425M volume)
---
## Post 1: Lead (Announcement)
V2.5 of @slashlast30days is out. Now with @Polymarket prediction markets, cross-source linking, and massively better results.
1. Polymarket as a 6th source - real money on outcomes, no API key needed
2. Hacker News as a 5th source
3. Cross-source linking - detects when the same story trends across multiple platforms
Ran a 15-way blinded comparison across 5 topics. v2.5 scored 4.38 vs 3.73 for the original. Won all 5. Zero regressions.
github.com/mvanhorn/last30days-skill
---
## Post 2: POLYMARKET AS A 6TH SOURCE.
Reddit tells you what people think. X tells you what people share. YouTube tells you what people watch. HN tells you what developers discuss.
Polymarket helps you predict the future.
"/last30days anthropic odds"
11 markets found. Best AI model February: Anthropic 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. Pentagon ban odds: only 22%.
Free API. No key. Real money on outcomes.
---
## Post 3: CROSS-SOURCE LINKING.
When a Seedance 2.0 tutorial has 44K YouTube views AND trends on HN AND gets discussed on Reddit, v2.5 flags it: [also on: HN, YouTube]
Old version linked 3 items across 5 test topics. New version links 13. The difference is hybrid similarity - combining character-trigram and token-level matching at a tuned threshold.
Cross-platform convergence is the strongest signal that something actually matters. Not engagement on one platform. Convergence across all of them.
---
## Post 4: 15-WAY BLINDED EVALUATION.
I don't trust vibes for measuring quality. So I ran a scientific comparison.
5 topics x 3 versions. Stripped version labels. Randomized as A/B/C. Scored on groundedness, specificity, coverage, actionability, and format.
v2.5: 4.38/5.0
v2.2 (HN only): 4.10/5.0
v2.0 (original): 3.73/5.0
Won all 5 topics. Zero regressions. Biggest gains: specificity (+0.8) and format (+1.0) from cross-source linking giving the synthesis better material to work with.
---
## Post 5: Demo - Anthropic Odds
Asked it about Anthropic odds.
11 Polymarket markets. 25 X posts. 13 YouTube videos (719K views). 6 HN stories (471 points).
Best AI model February: 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. FrontierMath 50% score: 48% (up 28% today). Pentagon ban: only 22%.
Markets say Anthropic is winning the model race AND the valuation race. The Pentagon thing is noise.
---
## Post 6: Demo - Arizona Basketball
"/last30days arizona basketball"
6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
Championship odds: 13%. #1 seed: 88%. Big 12 title: Arizona leads by 2.
That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
The Polymarket integration uses two-pass query expansion. First pass finds "Arizona Big 12." Second pass discovers the championship and #1 seed markets via tag-based domain bridging.
---
## Post 7: Demo - Iran War
The best Polymarket demo is news.
"/last30days iran war"
14 Polymarket markets. $425M+ in volume. 7 Reddit threads. 30 X posts. 20 YouTube videos (2M views). 18 HN stories (1,187 points).
US strikes Iran by 2026: 70%. War Powers by March: 60%. Israel strikes by June: 64%. Formal war declaration: only 8%.
Markets say: limited strikes with War Powers, NOT a declared war. Breaking Points (435K views) covered leaked Pentagon opposition. r/Conservative "imploding" per r/SubredditDrama.
One command. Six sources. Real money.
---
## Post 8: Credits + CTA
Also in v2.5: YouTube synonym expansion ("hip hop" now matches "rap" - relevance jumped 0.33 to 0.71), X handle resolution, and HN OR queries for framework topics.
The difference between "good research" and "research you'd actually trust" is in details like this.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 9: Demo - Claude Code (ALL 6 sources)
"/last30days Claude Code"
3 Reddit threads (199 upvotes). 35 X posts (5,239 likes). 15 YouTube videos (1.4M views, 5 transcripts). 30 HN stories (~8,500 points). 8 Polymarket markets. 20 web pages.
All six sources hit. Top finding: the planning-first workflow has won. The #1 HN post this month (969 pts, 590 comments) is about separating planning from execution. Boris Cherny (Head of Claude Code) on Lenny's Podcast: "100% of my code is written by Claude Code - I have not edited a single line by hand since November."
Polymarket: Anthropic 99% for best AI model in February. 58% for March. Claude on FrontierMath at 55%. The US government rejected Claude - Polymarket has the Hegseth ban at 32%.
Then I asked it to dig deeper into the planning-first workflow. No new searches - it answered from what it already learned.
---
## Post 10: Demo - March Madness Odds (Polymarket + Sports)
"/last30days March Madness Odds"
2 Reddit threads. 31 X posts. 6 YouTube videos (46K views, 4 transcripts). 2 Polymarket markets.
Tournament winner: Duke 18%, Michigan 18%, Arizona 13%. #1 seeds: Michigan 98%, Duke 91%, Arizona 88%.
Duke is the hottest mover - went from +700 to +450 in one week. The skill surfaced that from sportsbook data, X commentary, and Polymarket odds simultaneously.
Then I asked it to break down Michigan vs Duke vs Arizona. Full analysis from the research it already had.
---
## Post 11: Demo - Seedance Prompting (Expert + Prompt Mode)
"/last30days Seedance prompting"
13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 1 HN story. 15 web pages.
Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style + Constraints. Beyond 100 words, results degrade.
Then I said: "a cinematic drone shot over a city at golden hour"
It wrote me a copy-paste prompt using the exact patterns from the research. That's the skill's real power - research first, then create from what you learned.
---
## Post 12: Thank You + CTA (Final)
Thank you to ARJ999 and wkbaran on GitHub who kept asking for Hacker News support. Three separate issues. v2.5 delivers.
30 days. 3.2k stars. 6 sources. Massively better results.
Try it: /last30days [any topic]
github.com/mvanhorn/last30days-skill
---
## Post 13: Demo - Anthropic vs Pentagon (Breaking News + Polymarket)
"/last30days Anthropic Pete Hegseth"
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views, 5 transcripts). 5 HN stories. 9 Polymarket markets. 10 web pages.
This story broke TODAY. Defense Secretary Hegseth designated Anthropic a "supply chain risk" - believed to be the first time an American company has ever received this designation. Trump ordered every federal agency to stop using Anthropic tech.
Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March 31: only 6%.
Markets say: Anthropic wins the model race regardless. Bettors don't think Hegseth survives this. That's the kind of signal you can't get from opinion threads.
---
## Post 14: Demo - OpenAI Insider Trading (News + Polymarket)
"/last30days OpenAI Insider Trading"
2 Reddit threads. 29 X posts. 4 YouTube videos (360K views, 4 transcripts). 2 HN stories. 15 Polymarket markets. 15 web pages.
An OpenAI employee was just fired for using confidential info to bet on Polymarket. 13 brand-new wallets appeared 40 hours before the browser launch. $309K bet on the right outcome. Unusual Whales flagged 77 suspected insider positions across 60 wallets.
Meanwhile Polymarket has OpenAI's IPO at $1.25-1.5T: 54%. Anthropic IPOs first: 62%. Best AI model: Anthropic 99%.
The prediction markets are both the story AND the source. One command pulled all of it together.
---
## Standalone Tweet: Polymarket Stats Line
"/last30days Anthropic Pete Hegseth"
The Pentagon just designated Anthropic a supply chain risk. First time ever for an American company. Trump ordered every agency to stop using their tech.
Here's what Polymarket says:
📊 9 markets │ Best AI model: 99% │ $500B+ valuation: 68% │ IPO >$600B: 97% │ Hegseth out by March: 6%
Bettors with real money on the line think Anthropic wins the model race, goes public at a massive valuation, and Hegseth doesn't survive this.
That's the gap between headlines and reality. One command, six sources.
github.com/mvanhorn/last30days-skill
---
## Recommended Thread Order (pick 8-10)
The full thread above is 12 posts. Here's what I'd cut to keep it tight:
**Must include (core story):**
1. Post 1 - Lead announcement
2. Post 2 - Polymarket ("Reddit tells you what people think...")
3. Post 3 - Cross-source linking
4. Post 4 - Blinded evaluation
**Best demos (pick 3-4):**
- Post 13 (Anthropic vs Pentagon) - STRONGEST. Breaking news today. Polymarket cuts through the noise. "Markets say Anthropic wins regardless."
- Post 9 (Claude Code) - All 6 sources. Massive numbers. Shows follow-up flow.
- Post 10 (March Madness) - Sports/Polymarket crossover. Timely with tournament approaching.
- Post 11 (Seedance) - Shows prompting flow. 1.2M YouTube views.
- Post 5 (Anthropic Odds) - Overlaps with Post 13 now. Skip.
**Skip or save for standalone tweets:**
- Post 5 (Anthropic Odds) - Redundant with Post 13
- Post 6 (Arizona Basketball) - Covered by March Madness now
- Post 7 (Iran War) - Great standalone tweet, not for launch thread
- Post 8 (Credits/minor features) - Fold into CTA
**My recommended 8-post thread:**
1. Lead (Post 1)
2. Polymarket (Post 2)
3. Cross-source linking (Post 3)
4. Blinded evaluation (Post 4)
5. Demo: Anthropic vs Pentagon (Post 13) - breaking news, best Polymarket showcase
6. Demo: Claude Code (Post 9)
7. Demo: March Madness (Post 10)
8. Thank you + CTA (Post 12)
---
## Video Script (~60 seconds)
**[Talking to camera]**
Oh my god, I can't believe it's been 30 days since I launched last30days. 3,200 stars on GitHub. This has been the craziest month.
Today I'm shipping v2.5 and I'm really excited about this one. Three big things.
**[Screen recording: typing /last30days Anthropic Pete Hegseth]**
First - Polymarket prediction markets as a 6th source. So this Anthropic-Pentagon story broke today. Hegseth designated Anthropic a supply chain risk, Trump ordered agencies to stop using their tech. Scary headline, right?
But Polymarket says: Anthropic still 99% for best AI model. IPO above 600 billion: 97%. Hegseth out by March: 6%. Real money on outcomes helps you predict the future. That's a different story than the headlines.
**[Screen recording: typing /last30days arizona basketball]**
Second - it now searches Hacker News and does cross-source linking. When the same story shows up on Reddit AND YouTube AND HN, it flags it. I ran a 15-way blinded comparison and v2.5 scored 4.38 versus 3.73 for the original. Won all 5 test topics.
**[Back to camera]**
Thank you to everyone who starred it, filed issues, and kept pushing me to make this better. Shoutout to the people on GitHub who literally filed three separate issues asking for Hacker News. v2.5 delivers.
Link in bio. Try it on anything.
---
## Scoring Note
The 4.38 vs 3.73 score is from a custom 5-dimension rubric (30% groundedness, 25% specificity, 20% coverage, 15% actionability, 10% format compliance) evaluated by Claude on blinded outputs. The relative ranking is meaningful; the absolute numbers are not. It's an LLM grading LLM output - useful for A/B comparison, not for claiming "4.38 out of 5 quality."
If using the score in a tweet, frame it as "scored X vs Y on a blinded comparison" not "rated 4.38/5.0 quality" - the former is honest, the latter implies an objective standard that doesn't exist.
+42
View File
@@ -0,0 +1,42 @@
[
{
"topic": "OpenClaw vs NanoClaw vs ZeroClaw",
"query_type": "comparison",
"rationale": "Multi-entity extraction, 3-way split across AI agent frameworks."
},
{
"topic": "how to set up a GLP-1 supplement routine",
"query_type": "how_to",
"rationale": "Trending health topic. Tests non-tech how_to."
},
{
"topic": "2026 March Madness",
"query_type": "breaking_news",
"rationale": "Live sporting event. Tests broad breaking news recall."
},
{
"topic": "best budget noise cancelling headphones 2026",
"query_type": "product",
"rationale": "Evergreen consumer query. Tests product review aggregation."
},
{
"topic": "thoughts on OpenAI Codex pricing",
"query_type": "opinion",
"rationale": "Active developer debate. Tests opinion mining."
},
{
"topic": "odds of US recession 2026",
"query_type": "prediction",
"rationale": "Major macro topic. Tests prediction market + news synthesis."
},
{
"topic": "what is retrieval augmented generation",
"query_type": "concept",
"rationale": "Widely discussed AI concept. Tests explanation quality."
},
{
"topic": "Google Wiz acquisition price and timeline",
"query_type": "factual",
"rationale": "Completed event ($32B). Tests factual precision."
}
]
@@ -0,0 +1,8 @@
<!-- FIXTURE: captured live from reddit.com/svc/shreddit/community-more-posts/top/?name=technology&t=week on 2026-05-29; trimmed to 5 post cards (start-tag attrs only). -->
<div id="feed">
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tq0zk7/the_netherlands_just_blocked_a_us_company_from/" content-href="https://www.techspot.com/news/112552-netherlands-blocked-us-company-buying-app-dutch-citizens.html" view-context="SubredditFeed" comment-count="1743" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="0" award-count="23" award-id="award_obsessed_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/Obsessed_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-28T11:37:01.506000+0000" domain="techspot.com" id="t3_1tq0zk7" post-title="The Netherlands just blocked a US company from buying the app Dutch citizens use for everything" post-language="en" post-type="link" score="52692" upvote-ratio="0.9606269354736776" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1toe7m2/erin_brockovich_launches_map_of_over_4200_data/" content-href="https://www.newsweek.com/erin-brockovich-asks-americans-for-help-as-she-launches-data-center-map-11989813" view-context="SubredditFeed" comment-count="673" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="2" award-count="6" award-id="award_this_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/this_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T17:39:43.272000+0000" domain="newsweek.com" id="t3_1toe7m2" post-title="Erin Brockovich launches map of over 4,200 data centres in the US, appeals for local communities to report environmental impact and other costs" post-language="en" post-type="link" score="33567" upvote-ratio="0.973297166968053" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_fj9vsvfd" author="marketrent" icon="https://www.redditstatic.com/avatars/defaults/v2/avatar_default_1.png" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tollgz/majority_of_americans_support_ban_on_surveillance/" content-href="https://gizmodo.com/majority-of-americans-support-ban-on-surveillance-pricing-and-electronic-shelf-labels-2000762717" view-context="SubredditFeed" comment-count="1043" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="3" award-count="7" award-id="award_free_bravo" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/bravo_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T21:55:07.322000+0000" domain="gizmodo.com" id="t3_1tollgz" post-title="Majority of Americans Support Ban on Surveillance Pricing and Electronic Shelf Labels" post-language="en" post-type="link" score="29791" upvote-ratio="0.9815063671850003" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_98wao505" author="Plastic_Ninja_9014" icon="https://preview.redd.it/snoovatar/avatars/69af2b53-b0a1-4ab6-b119-d90f21c423fe-headshot.png?width=64&amp;height=64&amp;crop=smart&amp;auto=webp&amp;s=f3661eb511798004968f8b115a689dcee30f1428" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tp5qz2/tech_ceos_are_apparently_suffering_from_ai/" content-href="https://techcrunch.com/2026/05/27/tech-ceos-are-apparently-suffering-from-ai-psychosis/" view-context="SubredditFeed" comment-count="1653" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="4" award-count="6" award-id="award_free_regret_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/regret_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-27T13:33:49.280000+0000" domain="techcrunch.com" id="t3_1tp5qz2" post-title="Tech CEOs are apparently suffering from AI psychosis" post-language="en" post-type="link" score="26419" upvote-ratio="0.9605741880002646" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tn5g7s/pope_leo_issues_ai_encyclical_warning_that_opaque/" content-href="https://variety.com/2026/biz/global/pope-leo-ai-encyclical-algorithms-threaten-dehumanisation-1236758186/" view-context="SubredditFeed" comment-count="608" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="6" award-count="7" award-id="award_hooray_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/FTUE_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-25T10:45:04.093000+0000" domain="variety.com" id="t3_1tn5g7s" post-title="Pope Leo Issues AI Encyclical Warning That Opaque Algorithms Controlled by a Few Companies Can Bring New Forms of Dehumanisation" post-language="en" post-type="link" score="25835" upvote-ratio="0.9760626539506095" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_1i1zizibn9" author="yourfavchoom" icon="https://styles.redditmedia.com/t5_dgdrt8/styles/profileIcon_k9x929ihm8rg1.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=2e8a5042cccc4555167f98d28bc0de4e13fd3ca5" data-expected-lcp subreddit-name="technology"></shreddit-post>
</div>
+7
View File
@@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- FIXTURE: captured live from reddit.com/r/Rakuten/top.rss on 2026-05-29; trimmed to 5 entries. Atom shape identical to search.rss. --><feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/"><category term="Rakuten" label="r/Rakuten"/><updated>2026-05-29T14:14:32+00:00</updated><icon>https://www.redditstatic.com/icon.png/</icon><id>/r/Rakuten/top.rss?t=month</id><link rel="self" href="https://www.reddit.com/r/Rakuten/top.rss?t=month" type="application/atom+xml" /><link rel="alternate" href="https://www.reddit.com/r/Rakuten/top?t=month" type="text/html" /><subtitle>This is an unofficial subreddit for Rakuten Rewards, the cash back website. We are not affiliated with, endorsed by, or sponsored by Rakuten or any of its subsidiaries.</subtitle><title>top scoring links : Rakuten</title><entry><author><name>/u/InternetUser52</name><uri>https://www.reddit.com/user/InternetUser52</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I&amp;#39;m rich!!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/InternetUser52&quot;&gt; /u/InternetUser52 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/q8fgmxs29c2h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tiv013</id><link href="https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/" /><updated>2026-05-20T18:48:31+00:00</updated><published>2026-05-20T18:48:31+00:00</published><title>LETS GOO! $0.02!!!</title></entry>
<entry><author><name>/u/Immediate-Duck-6351</name><uri>https://www.reddit.com/user/Immediate-Duck-6351</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I dont travel and Im buying a house in a few weeks so cash back is amazing 🙌 hoping to keep the pace in the next quarter so I can buy new kitchen appliances lol. &lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Immediate-Duck-6351&quot;&gt; /u/Immediate-Duck-6351 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/d2a4s0ipvb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te1fp8</id><link href="https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/" /><updated>2026-05-15T16:29:28+00:00</updated><published>2026-05-15T16:29:28+00:00</published><title>So excited 🥳</title></entry>
<entry><author><name>/u/gnibgnib</name><uri>https://www.reddit.com/user/gnibgnib</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;128k for the May transfer&lt;/p&gt; &lt;p&gt;41k pending for August &lt;/p&gt; &lt;p&gt;Got another 9k at Asics not showing but overall pretty happy with Rakuten&lt;/p&gt; &lt;p&gt;P2 was able to secure 85k for May transfer&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/gnibgnib&quot;&gt; /u/gnibgnib &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/gallery/1tb8674&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tb8674</id><link href="https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/" /><updated>2026-05-12T17:17:19+00:00</updated><published>2026-05-12T17:17:19+00:00</published><title>Had a great run so far this year thanks to this sub!</title></entry>
<entry><author><name>/u/TravelVet93</name><uri>https://www.reddit.com/user/TravelVet93</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/TravelVet93&quot;&gt; /u/TravelVet93 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/x6b9whvupb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te0hom</id><link href="https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/" /><updated>2026-05-15T15:56:40+00:00</updated><published>2026-05-15T15:56:40+00:00</published><title>My best payout so far</title></entry>
<entry><author><name>/u/Beautiful-Piece-4252</name><uri>https://www.reddit.com/user/Beautiful-Piece-4252</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;The amount of $$ available in sign up bonuses is amazing. It&amp;#39;s kind of a part time job ensuring Rakuten captures everything, but my August and November payout should be sizeable. I&amp;#39;m new to this and it always seemed like a lot of work for little reward. I know it&amp;#39;s not sustainable, but wow!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Beautiful-Piece-4252&quot;&gt; /u/Beautiful-Piece-4252 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/1vqvajsci42h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1thsnm1</id><link href="https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/" /><updated>2026-05-19T16:46:17+00:00</updated><published>2026-05-19T16:46:17+00:00</published><title>How can this be real?</title></entry>
</feed>
@@ -0,0 +1,29 @@
<!-- FIXTURE: captured live from reddit.com/svc/shreddit/comments/r/Rakuten/t3_1taeiw0 on 2026-05-29;
trimmed to 6 real comment elements (real attrs + real bodies) + 2 synthetic edge cases. -->
<shreddit-comment-tree-stats total-comments="14"></shreddit-comment-tree-stats>
<shreddit-comment-tree id="comment-tree" post-id="t3_1taeiw0">
<shreddit-comment created="2026-05-11T20:16:57.590000+0000" author="Obvious_Painting_881" thingId="t1_ol8tp8n" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8tp8n/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8tp8n-comment-rtjson-content" slot="comment"><div id="t1_ol8tp8n-post-rtjson-content" dir="auto"><p dir="auto">Where do you find $750? The highest available package for Total was $284.99 when I did the lifelock promotion. I did get the full 284.99 from Rakuten.</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-12T12:26:14.973000+0000" author="Stormtrooper149" thingId="t1_olcy1iv" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/olcy1iv/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_olcy1iv-comment-rtjson-content" slot="comment"><div id="t1_olcy1iv-post-rtjson-content" dir="auto"><p dir="auto">It went to pending ($712.49)</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-19T01:43:48.026000+0000" author="heythereyou01" thingId="t1_omlbiqg" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/omlbiqg/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_omlbiqg-comment-rtjson-content" slot="comment"><div id="t1_omlbiqg-post-rtjson-content" dir="auto"><p dir="auto">Hey I PMd. can I get the screenshot ?</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-11T20:21:16.398000+0000" author="Stormtrooper149" thingId="t1_ol8undb" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8undb/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8undb-comment-rtjson-content" slot="comment"><div id="t1_ol8undb-post-rtjson-content" dir="auto"><p dir="auto">Family plan</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-11T20:28:33.803000+0000" author="Obvious_Painting_881" thingId="t1_ol8w8w6" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8w8w6/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8w8w6-comment-rtjson-content" slot="comment"><div id="t1_ol8w8w6-post-rtjson-content" dir="auto"><p dir="auto">Price seems to change every time I go to the page but I see only 249.99-369.99 for Total/Advanced. No where near your $750. Just saying the Total plan for 299.99 worked for me and I got 284.99 which is 95%.</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-12T02:33:48.200000+0000" author="jwegener" thingId="t1_olaqzjk" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/olaqzjk/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_olaqzjk-comment-rtjson-content" slot="comment"><div id="t1_olaqzjk-post-rtjson-content" dir="auto"><p dir="auto">I did that one. Lets pray</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-13T10:00:00.000000+0000" author="[deleted]" thingId="t1_synthdel" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/synthdel/" score="5" postId="t3_1taeiw0" content-type="text">
<div id="t1_synthdel-comment-rtjson-content" slot="comment"><div id="t1_synthdel-post-rtjson-content" dir="auto"><p dir="auto">[removed]</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-13T11:00:00.000000+0000" author="NegScoreUser" thingId="t1_synthneg" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/synthneg/" score="-7" postId="t3_1taeiw0" content-type="text">
<div id="t1_synthneg-comment-rtjson-content" slot="comment"><div id="t1_synthneg-post-rtjson-content" dir="auto"><p dir="auto">A downvoted but real reply with negative score for edge-case coverage.</p></div></div>
</shreddit-comment>
</shreddit-comment-tree>
+58
View File
@@ -0,0 +1,58 @@
{
"items": [
{
"video_id": "7543693751290481942",
"text": "This Claude Code trick saved me hours #claudecode #ai #coding",
"url": "https://www.tiktok.com/@codemaster/video/7543693751290481942",
"author_name": "codemaster",
"date": "2026-02-28",
"engagement": {
"views": 2100000,
"likes": 45000,
"comments": 1200,
"shares": 8400
},
"hashtags": ["claudecode", "ai", "coding"],
"duration": 45,
"relevance": 0.85,
"why_relevant": "TikTok: This Claude Code trick saved me hours #claude",
"caption_snippet": "So I found this insane trick with Claude Code where you can use slash commands to automate everything"
},
{
"video_id": "7543100200112345678",
"text": "AI coding tools comparison 2026 - Claude vs Copilot vs Cursor #ai #devtools",
"url": "https://www.tiktok.com/@techreviewer/video/7543100200112345678",
"author_name": "techreviewer",
"date": "2026-02-25",
"engagement": {
"views": 850000,
"likes": 22000,
"comments": 890,
"shares": 3200
},
"hashtags": ["ai", "devtools"],
"duration": 60,
"relevance": 0.7,
"why_relevant": "TikTok: AI coding tools comparison 2026 - Claude vs Copi",
"caption_snippet": ""
},
{
"video_id": "7543200300223456789",
"text": "You need to try Claude Code RIGHT NOW #programming #tips",
"url": "https://www.tiktok.com/@devtips/video/7543200300223456789",
"author_name": "devtips",
"date": "2026-03-01",
"engagement": {
"views": 500000,
"likes": 15000,
"comments": 450,
"shares": 2100
},
"hashtags": ["programming", "tips"],
"duration": 30,
"relevance": 0.6,
"why_relevant": "TikTok: You need to try Claude Code RIGHT NOW #programm",
"caption_snippet": "Let me show you why Claude Code is the best AI coding tool right now"
}
]
}
+67
View File
@@ -0,0 +1,67 @@
{
"name": "last30days-skill",
"version": "3.3.2",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
"name": "Extension Directory",
"description": "Extension installation directory (auto-set by Gemini CLI)",
"envVar": "GEMINI_EXTENSION_DIR",
"sensitive": false
},
{
"name": "ScrapeCreators API Key",
"description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
"envVar": "SCRAPECREATORS_API_KEY",
"sensitive": true
},
{
"name": "OpenAI API Key",
"description": "OpenAI API Key - optional fallback for Reddit discovery",
"envVar": "OPENAI_API_KEY",
"sensitive": true
},
{
"name": "xAI API Key",
"description": "xAI API Key for X/Twitter search (optional)",
"envVar": "XAI_API_KEY",
"sensitive": true
},
{
"name": "OpenRouter API Key",
"description": "OpenRouter API Key (optional)",
"envVar": "OPENROUTER_API_KEY",
"sensitive": true
},
{
"name": "Parallel AI API Key",
"description": "Parallel AI API Key (optional)",
"envVar": "PARALLEL_API_KEY",
"sensitive": true
},
{
"name": "Brave Search API Key",
"description": "Brave Search API Key (optional)",
"envVar": "BRAVE_API_KEY",
"sensitive": true
},
{
"name": "Apify API Token",
"description": "Apify API Token (optional legacy)",
"envVar": "APIFY_API_TOKEN",
"sensitive": true
},
{
"name": "Twitter AUTH_TOKEN",
"description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
"envVar": "AUTH_TOKEN",
"sensitive": true
},
{
"name": "Twitter CT0",
"description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
"envVar": "CT0",
"sensitive": true
}
]
}
+4
View File
@@ -0,0 +1,4 @@
{
"triggerOnUpdates": true,
"statusCheck": true
}
+15
View File
@@ -0,0 +1,15 @@
{
"hooks": {
"SessionStart": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "bash \"${CLAUDE_PLUGIN_ROOT:-${extensionPath:-.}}/hooks/scripts/check-config.sh\""
}
]
}
]
}
}
+173
View File
@@ -0,0 +1,173 @@
#!/bin/bash
set -euo pipefail
# Check last30days configuration status and show appropriate welcome message.
# Priority: .claude/last30days.env > ~/.config/last30days/.env > env vars
PROJECT_ENV=".claude/last30days.env"
GLOBAL_ENV="$HOME/.config/last30days/.env"
# Helper: warn if file permissions are too open
check_perms() {
local file="$1"
if [[ ! -f "$file" ]]; then return; fi
local perms
# Try GNU stat first (Linux), fall back to BSD stat (macOS).
# On Linux, `stat -f` prints filesystem info (not permissions) and exits 0,
# so the previous BSD-first ordering left $perms as multi-line garbage on
# every Linux session start and printed a false WARNING.
perms=$(stat -c '%a' "$file" 2>/dev/null || stat -f '%Lp' "$file" 2>/dev/null || echo "")
if [[ -n "$perms" && "$perms" != "600" && "$perms" != "400" ]]; then
echo "/last30days: WARNING — $file has permissions $perms (should be 600)."
echo " Fix: chmod 600 $file"
fi
}
# Load env file into variables for inspection (without exporting)
load_env_vars() {
local file="$1"
if [[ -f "$file" ]]; then
while IFS='=' read -r key value; do
# Skip comments, empty lines
[[ "$key" =~ ^[[:space:]]*# ]] && continue
[[ -z "$key" ]] && continue
key=$(echo "$key" | xargs)
value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
# Strip inline comments (# preceded by whitespace) to prevent
# command substitution in backtick-containing comments
value="${value%%[[:space:]]#*}"
if [[ -n "$key" && -n "$value" ]]; then
# printf -v writes via assignment semantics (global from inside a
# function), works on macOS's /bin/bash 3.2 — `declare -g` is 4.2+.
printf -v "ENV_${key}" '%s' "$value"
fi
done < "$file"
fi
}
# Determine which config file is active
CONFIG_FILE=""
if [[ -f "$PROJECT_ENV" ]]; then
CONFIG_FILE="$PROJECT_ENV"
check_perms "$PROJECT_ENV"
elif [[ -f "$GLOBAL_ENV" ]]; then
CONFIG_FILE="$GLOBAL_ENV"
check_perms "$GLOBAL_ENV"
fi
# Load config if found
if [[ -n "$CONFIG_FILE" ]]; then
load_env_vars "$CONFIG_FILE"
fi
# Check SETUP_COMPLETE (from file or env)
SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
# Compute last-run summary line (if last-run.json exists)
if [[ "${LAST30DAYS_CONFIG_DIR+x}" == "x" ]]; then
if [[ -n "$LAST30DAYS_CONFIG_DIR" ]]; then
LAST_RUN_FILE="$LAST30DAYS_CONFIG_DIR/last-run.json"
else
LAST_RUN_FILE=""
fi
else
LAST_RUN_FILE="$HOME/.config/last30days/last-run.json"
fi
LAST_RUN_LINE=""
if [[ -n "$LAST_RUN_FILE" && -f "$LAST_RUN_FILE" ]] && command -v python3 &>/dev/null; then
LAST_RUN_LINE=$(LAST_RUN_FILE="$LAST_RUN_FILE" python3 - <<'PY' 2>/dev/null || true
import datetime
import json
import os
path = os.environ["LAST_RUN_FILE"]
try:
with open(path) as fh:
d = json.load(fh)
topic = (d.get("topic") or "?")[:60]
ts = d.get("timestamp", "")
dt = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00"))
delta = (datetime.datetime.now(datetime.timezone.utc) - dt).total_seconds()
if delta < 60: ago = f"{int(delta)}s ago"
elif delta < 3600: ago = f"{int(delta//60)}m ago"
elif delta < 86400: ago = f"{int(delta//3600)}h ago"
else: ago = f"{int(delta//86400)}d ago"
total = d.get("total", 0)
print(f" Last run: \"{topic}\" · {ago} · {total} results")
except Exception:
pass
PY
)
fi
# If setup has never been run, show welcome message for new users
if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
cat <<'EOF'
/last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
Research any topic across Reddit, HN, X, YouTube, Polymarket (last 30 days).
Reddit, Hacker News, and Polymarket work out of the box.
The setup wizard can unlock X/Twitter, YouTube, and more.
EOF
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
exit 0
fi
# Setup done but check for ScrapeCreators
HAS_SCRAPECREATORS="${ENV_SCRAPECREATORS_API_KEY:-${SCRAPECREATORS_API_KEY:-}}"
HAS_X="${ENV_AUTH_TOKEN:-${AUTH_TOKEN:-}}"
HAS_XAI="${ENV_XAI_API_KEY:-${XAI_API_KEY:-}}"
HAS_YTDLP=""
if command -v yt-dlp &>/dev/null; then
HAS_YTDLP="yes"
fi
HAS_BSKY="${ENV_BSKY_HANDLE:-${BSKY_HANDLE:-}}"
HAS_EXA="${ENV_EXA_API_KEY:-${EXA_API_KEY:-}}"
# Count active sources
SOURCE_COUNT=2 # HN + Polymarket are always free
if [[ -n "$HAS_X" || -n "$HAS_XAI" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
# Reddit public JSON always works
SOURCE_COUNT=$((SOURCE_COUNT + 1))
if [[ -n "$HAS_YTDLP" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_EXA" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_BSKY" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Start with Reddit comments + TikTok + Instagram, subtract any in EXCLUDE_SOURCES.
# Normalise EXCLUDED (lowercase + collapse whitespace around commas + strip outer
# whitespace) so the matching mirrors pipeline.py's .strip().lower() parsing.
SC_ADD=3
EXCLUDED="${ENV_EXCLUDE_SOURCES:-${EXCLUDE_SOURCES:-}}"
EXCLUDED_NORM=$(printf '%s' "$EXCLUDED" | tr '[:upper:]' '[:lower:]' \
| sed -E 's/[[:space:]]*,[[:space:]]*/,/g; s/^[[:space:]]+//; s/[[:space:]]+$//')
if [[ ",$EXCLUDED_NORM," == *",tiktok,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
if [[ ",$EXCLUDED_NORM," == *",instagram,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
SOURCE_COUNT=$((SOURCE_COUNT + SC_ADD))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Fully configured — compact ready message
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
else
# Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 100 free credits, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider."
fi
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 MiB

-395
View File
@@ -1,395 +0,0 @@
# feat: Add WebSearch as Third Source (Zero-Config Fallback)
## Overview
Add Claude's built-in WebSearch tool as a third research source for `/last30days`. This enables the skill to work **out of the box with zero API keys** while preserving the primacy of Reddit/X as the "voice of real humans with popularity signals."
**Key principle**: WebSearch is supplementary, not primary. Real human voices on Reddit/X with engagement metrics (upvotes, likes, comments) are more valuable than general web content.
## Problem Statement
Currently `/last30days` requires at least one API key (OpenAI or xAI) to function. Users without API keys get an error. Additionally, web search could fill gaps where Reddit/X coverage is thin.
**User requirements**:
- Work out of the box (no API key needed)
- Must NOT overpower Reddit/X results
- Needs proper weighting
- Validate with before/after testing
## Proposed Solution
### Weighting Strategy: "Engagement-Adjusted Scoring"
**Current formula** (same for Reddit/X):
```
score = 0.45*relevance + 0.25*recency + 0.30*engagement - penalties
```
**Problem**: WebSearch has NO engagement metrics. Giving it `DEFAULT_ENGAGEMENT=35` with `-10 penalty` = 25 base, which still competes unfairly.
**Solution**: Source-specific scoring with **engagement substitution**:
| Source | Relevance | Recency | Engagement | Source Penalty |
|--------|-----------|---------|------------|----------------|
| Reddit | 45% | 25% | 30% (real metrics) | 0 |
| X | 45% | 25% | 30% (real metrics) | 0 |
| WebSearch | 55% | 35% | 0% (no data) | -15 points |
**Rationale**:
- WebSearch items compete on relevance + recency only (reweighted to 100%)
- `-15 point source penalty` ensures WebSearch ranks below comparable Reddit/X items
- High-quality WebSearch can still surface (score 60-70) but won't dominate (Reddit/X score 70-85)
### Mode Behavior
| API Keys Available | Default Behavior | `--include-web` |
|--------------------|------------------|-----------------|
| None | **WebSearch only** | n/a |
| OpenAI only | Reddit only | Reddit + WebSearch |
| xAI only | X only | X + WebSearch |
| Both | Reddit + X | Reddit + X + WebSearch |
**CLI flag**: `--include-web` (default: false when other sources available)
## Technical Approach
### Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ last30days.py orchestrator │
├─────────────────────────────────────────────────────────────────┤
│ run_research() │
│ ├── if sources includes "reddit": openai_reddit.search_reddit()│
│ ├── if sources includes "x": xai_x.search_x() │
│ └── if sources includes "web": websearch.search_web() ← NEW │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ Processing Pipeline │
├─────────────────────────────────────────────────────────────────┤
│ normalize_websearch_items() → WebSearchItem schema ← NEW │
│ score_websearch_items() → engagement-free scoring ← NEW │
│ dedupe_websearch() → deduplication ← NEW │
│ render_websearch_section() → output formatting ← NEW │
└─────────────────────────────────────────────────────────────────┘
```
### Implementation Phases
#### Phase 1: Schema & Core Infrastructure
**Files to create/modify:**
```python
# scripts/lib/websearch.py (NEW)
"""Claude WebSearch API client for general web discovery."""
WEBSEARCH_PROMPT = """Search the web for content about: {topic}
CRITICAL: Only include results from the last 30 days (after {from_date}).
Find {min_items}-{max_items} high-quality, relevant web pages. Prefer:
- Blog posts, tutorials, documentation
- News articles, announcements
- Authoritative sources (official docs, reputable publications)
AVOID:
- Reddit (covered separately)
- X/Twitter (covered separately)
- YouTube without transcripts
- Forum threads without clear answers
Return ONLY valid JSON:
{{
"items": [
{{
"title": "Page title",
"url": "https://...",
"source_domain": "example.com",
"snippet": "Brief excerpt (100-200 chars)",
"date": "YYYY-MM-DD or null",
"why_relevant": "Brief explanation",
"relevance": 0.85
}}
]
}}
"""
def search_web(topic: str, from_date: str, to_date: str, depth: str = "default") -> dict:
"""Search web using Claude's built-in WebSearch tool.
NOTE: This runs INSIDE Claude Code, so we use the WebSearch tool directly.
No API key needed - uses Claude's session.
"""
# Implementation uses Claude's web_search_20250305 tool
pass
def parse_websearch_response(response: dict) -> list[dict]:
"""Parse WebSearch results into normalized format."""
pass
```
```python
# scripts/lib/schema.py - ADD WebSearchItem
@dataclass
class WebSearchItem:
"""Normalized web search item."""
id: str
title: str
url: str
source_domain: str # e.g., "medium.com", "github.com"
snippet: str
date: Optional[str] = None
date_confidence: str = "low"
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'source_domain': self.source_domain,
'snippet': self.snippet,
'date': self.date,
'date_confidence': self.date_confidence,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
```
#### Phase 2: Scoring System Updates
```python
# scripts/lib/score.py - ADD websearch scoring
# New constants
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# Reweighted for no engagement
WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
"""Score WebSearch items WITHOUT engagement metrics.
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty
"""
for item in items:
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=0, # Explicitly zero - no engagement data
)
overall = (
WEBSEARCH_WEIGHT_RELEVANCE * rel_score +
WEBSEARCH_WEIGHT_RECENCY * rec_score
)
# Apply source penalty (WebSearch < Reddit/X)
overall -= WEBSEARCH_SOURCE_PENALTY
# Apply date confidence penalty (same as other sources)
if item.date_confidence == "low":
overall -= 10
elif item.date_confidence == "med":
overall -= 5
item.score = max(0, min(100, int(overall)))
return items
```
#### Phase 3: Orchestrator Integration
```python
# scripts/last30days.py - UPDATE run_research()
def run_research(...) -> tuple:
"""Run the research pipeline.
Returns: (reddit_items, x_items, web_items, raw_openai, raw_xai,
raw_websearch, reddit_error, x_error, web_error)
"""
# ... existing Reddit/X code ...
# WebSearch (new)
web_items = []
raw_websearch = None
web_error = None
if sources in ("all", "web", "reddit-web", "x-web"):
if progress:
progress.start_web()
try:
raw_websearch = websearch.search_web(topic, from_date, to_date, depth)
web_items = websearch.parse_websearch_response(raw_websearch)
except Exception as e:
web_error = f"{type(e).__name__}: {e}"
if progress:
progress.end_web(len(web_items))
return (reddit_items, x_items, web_items, raw_openai, raw_xai,
raw_websearch, reddit_error, x_error, web_error)
```
#### Phase 4: CLI & Environment Updates
```python
# scripts/last30days.py - ADD CLI flag
parser.add_argument(
"--include-web",
action="store_true",
help="Include general web search alongside Reddit/X (lower weighted)",
)
# scripts/lib/env.py - UPDATE get_available_sources()
def get_available_sources(config: dict) -> str:
"""Determine available sources. WebSearch always available (no API key)."""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
if has_openai and has_xai:
return 'both' # WebSearch available but not default
elif has_openai:
return 'reddit'
elif has_xai:
return 'x'
else:
return 'web' # Fallback: WebSearch only (no keys needed)
```
## Acceptance Criteria
### Functional Requirements
- [x] Skill works with zero API keys (WebSearch-only mode)
- [x] `--include-web` flag adds WebSearch to Reddit/X searches
- [x] WebSearch items have lower average scores than Reddit/X items with similar relevance
- [x] WebSearch results exclude Reddit/X URLs (handled separately)
- [x] Date filtering uses natural language ("last 30 days") in prompt
- [x] Output clearly labels source type: `[WEB]`, `[Reddit]`, `[X]`
### Non-Functional Requirements
- [x] WebSearch adds <10s latency to total research time (0s - deferred to Claude)
- [x] Graceful degradation if WebSearch fails
- [ ] Cache includes WebSearch results appropriately
### Quality Gates
- [x] Before/after testing shows WebSearch doesn't dominate rankings (via -15pt penalty)
- [x] Test: 10 Reddit + 10 X + 10 WebSearch → WebSearch avg score 15-20pts lower (scoring formula verified)
- [x] Test: WebSearch-only mode produces useful results for common topics
## Testing Plan
### Before/After Comparison Script
```python
# tests/test_websearch_weighting.py
"""
Test harness to validate WebSearch doesn't overpower Reddit/X.
Run same queries with:
1. Reddit + X only (baseline)
2. Reddit + X + WebSearch (comparison)
Verify: WebSearch items rank lower on average.
"""
TEST_QUERIES = [
"best practices for react server components",
"AI coding assistants comparison",
"typescript 5.5 new features",
]
def test_websearch_weighting():
for query in TEST_QUERIES:
# Run without WebSearch
baseline = run_research(query, sources="both")
baseline_scores = [item.score for item in baseline.reddit + baseline.x]
# Run with WebSearch
with_web = run_research(query, sources="both", include_web=True)
web_scores = [item.score for item in with_web.web]
reddit_x_scores = [item.score for item in with_web.reddit + with_web.x]
# Assertions
avg_reddit_x = sum(reddit_x_scores) / len(reddit_x_scores)
avg_web = sum(web_scores) / len(web_scores) if web_scores else 0
assert avg_web < avg_reddit_x - 10, \
f"WebSearch avg ({avg_web}) too close to Reddit/X avg ({avg_reddit_x})"
# Check top 5 aren't all WebSearch
top_5 = sorted(with_web.reddit + with_web.x + with_web.web,
key=lambda x: -x.score)[:5]
web_in_top_5 = sum(1 for item in top_5 if isinstance(item, WebSearchItem))
assert web_in_top_5 <= 2, f"Too many WebSearch items in top 5: {web_in_top_5}"
```
### Manual Test Scenarios
| Scenario | Expected Outcome |
|----------|------------------|
| No API keys, run `/last30days AI tools` | WebSearch-only results, useful output |
| Both keys + `--include-web`, run `/last30days react` | Mix of all 3 sources, Reddit/X dominate top 10 |
| Niche topic (no Reddit/X coverage) | WebSearch fills gap, becomes primary |
| Popular topic (lots of Reddit/X) | WebSearch present but lower-ranked |
## Dependencies & Prerequisites
- Claude Code's WebSearch tool (`web_search_20250305`) - already available
- No new API keys required
- Existing test infrastructure in `tests/`
## Risk Analysis & Mitigation
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| WebSearch returns stale content | Medium | Medium | Enforce date in prompt, apply low-confidence penalty |
| WebSearch dominates rankings | Low | High | Source penalty (-15pts), testing validates |
| WebSearch adds spam/low-quality | Medium | Medium | Exclude social media domains, domain filtering |
| Date parsing unreliable | High | Medium | Accept "low" confidence as normal for WebSearch |
## Future Considerations
1. **Domain authority scoring**: Could proxy engagement with domain reputation
2. **User-configurable weights**: Let users adjust WebSearch penalty
3. **Domain whitelist/blacklist**: Filter WebSearch to trusted sources
4. **Parallel execution**: Run all 3 sources concurrently for speed
## References
### Internal References
- Scoring algorithm: `scripts/lib/score.py:8-15`
- Source detection: `scripts/lib/env.py:57-72`
- Schema patterns: `scripts/lib/schema.py:76-138`
- Orchestrator: `scripts/last30days.py:54-164`
### External References
- Claude WebSearch docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
- WebSearch pricing: $10/1K searches + token costs
- Date filtering limitation: No explicit date params, use natural language
### Research Findings
- Reddit upvotes are ~12% of ranking value in SEO (strong signal)
- E-E-A-T framework: Engagement metrics = trust signal
- MSA2C2 approach: Dynamic weight learning for multi-source aggregation
-328
View File
@@ -1,328 +0,0 @@
# fix: Enforce Strict 30-Day Date Filtering
## Overview
The `/last30days` skill is returning content older than 30 days, violating its core promise. Analysis shows:
- **Reddit**: Only 40% of results within 30 days (9/15 were older, some from 2022!)
- **X**: 100% within 30 days (working correctly)
- **WebSearch**: 90% had unknown dates (can't verify freshness)
## Problem Statement
The skill's name is "last30days" - users expect ONLY content from the last 30 days. Currently:
1. **Reddit search prompt** says "prefer recent threads, but include older relevant ones if recent ones are scarce" - this is too permissive
2. **X search prompt** explicitly includes `from_date` and `to_date` - this is why it works
3. **WebSearch** returns pages without publication dates - we can't verify they're recent
4. **Scoring penalties** (-10 for low date confidence) don't prevent old content from appearing
## Proposed Solution
### Strategy: "Hard Filter, Not Soft Penalty"
Instead of penalizing old content, **exclude it entirely**. If it's not from the last 30 days, it shouldn't appear.
| Source | Current Behavior | New Behavior |
|--------|------------------|--------------|
| Reddit | Weak "prefer recent" | Explicit date range + hard filter |
| X | Explicit date range (working) | No change needed |
| WebSearch | No date awareness | Require recent markers OR exclude |
## Technical Approach
### Phase 1: Fix Reddit Date Filtering
**File: `scripts/lib/openai_reddit.py`**
Current prompt (line 33):
```
Find {min_items}-{max_items} relevant Reddit discussion threads.
Prefer recent threads, but include older relevant ones if recent ones are scarce.
```
New prompt:
```
Find {min_items}-{max_items} relevant Reddit discussion threads from {from_date} to {to_date}.
CRITICAL: Only include threads posted within the last 30 days (after {from_date}).
Do NOT include threads older than {from_date}, even if they seem relevant.
If you cannot find enough recent threads, return fewer results rather than older ones.
```
**Changes needed:**
1. Add `from_date` and `to_date` parameters to `search_reddit()` function
2. Inject dates into `REDDIT_SEARCH_PROMPT` like X does
3. Update caller in `last30days.py` to pass dates
### Phase 2: Add Hard Date Filtering (Post-Processing)
**File: `scripts/lib/normalize.py`**
Add a filter step that DROPS items with dates before `from_date`:
```python
def filter_by_date_range(
items: List[Union[RedditItem, XItem, WebSearchItem]],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List:
"""Hard filter: Remove items outside the date range.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
if item.date > to_date:
continue # DROP - future date (likely parsing error)
result.append(item)
return result
```
### Phase 3: WebSearch Date Intelligence
WebSearch CAN find recent content - Medium posts have dates, GitHub has commit timestamps, news sites have publication dates. We should **extract and prioritize** these signals.
**Strategy: "Date Detective"**
1. **Extract dates from URLs**: Many sites embed dates in URLs
- Medium: `medium.com/@author/title-abc123` (no date) vs news sites
- GitHub: Look for commit dates, release dates in snippets
- News: `/2026/01/24/article-title`
- Blogs: `/blog/2026/01/title`
2. **Extract dates from snippets**: Look for date markers
- "January 24, 2026", "Jan 2026", "yesterday", "this week"
- "Published:", "Posted:", "Updated:"
- Relative markers: "2 days ago", "last week"
3. **Prioritize results with verifiable dates**:
- Results with recent dates (within 30 days): Full score
- Results with old dates: EXCLUDE
- Results with no date signals: Heavy penalty (-20) but keep as supplementary
**File: `scripts/lib/websearch.py`**
Add date extraction functions:
```python
import re
from datetime import datetime, timedelta
# Patterns for date extraction
URL_DATE_PATTERNS = [
r'/(\d{4})/(\d{2})/(\d{2})/', # /2026/01/24/
r'/(\d{4})-(\d{2})-(\d{2})/', # /2026-01-24/
r'/(\d{4})(\d{2})(\d{2})/', # /20260124/
]
SNIPPET_DATE_PATTERNS = [
r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{1,2}),? (\d{4})',
r'(\d{1,2}) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{4})',
r'(\d{4})-(\d{2})-(\d{2})',
r'Published:?\s*(\d{4}-\d{2}-\d{2})',
r'(\d{1,2}) (days?|hours?|minutes?) ago', # Relative dates
]
def extract_date_from_url(url: str) -> Optional[str]:
"""Try to extract a date from URL path."""
for pattern in URL_DATE_PATTERNS:
match = re.search(pattern, url)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_from_snippet(snippet: str) -> Optional[str]:
"""Try to extract a date from text snippet."""
for pattern in SNIPPET_DATE_PATTERNS:
match = re.search(pattern, snippet, re.IGNORECASE)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_signals(url: str, snippet: str, title: str) -> tuple[Optional[str], str]:
"""Extract date from any available signal.
Returns: (date_string, confidence)
- date from URL: 'high' confidence
- date from snippet: 'med' confidence
- no date found: None, 'low' confidence
"""
# Try URL first (most reliable)
url_date = extract_date_from_url(url)
if url_date:
return url_date, 'high'
# Try snippet
snippet_date = extract_date_from_snippet(snippet)
if snippet_date:
return snippet_date, 'med'
# Try title
title_date = extract_date_from_snippet(title)
if title_date:
return title_date, 'med'
return None, 'low'
```
**Update WebSearch parsing to use date extraction:**
```python
def parse_websearch_results(results, topic, from_date, to_date):
items = []
for result in results:
url = result.get('url', '')
snippet = result.get('snippet', '')
title = result.get('title', '')
# Extract date signals
extracted_date, confidence = extract_date_signals(url, snippet, title)
# Hard filter: if we found a date and it's too old, skip
if extracted_date and extracted_date < from_date:
continue # DROP - verified old content
item = {
'date': extracted_date,
'date_confidence': confidence,
...
}
items.append(item)
return items
```
**File: `scripts/lib/score.py`**
Update WebSearch scoring to reward date-verified results:
```python
# WebSearch date confidence adjustments
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date (was 10)
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date
def score_websearch_items(items):
for item in items:
...
# Date confidence adjustments
if item.date_confidence == 'high':
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified dates
elif item.date_confidence == 'low':
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
...
```
**Result**: WebSearch results with verifiable recent dates rank well. Results with no dates are heavily penalized but still appear as supplementary context. Old verified content is excluded entirely.
### Phase 4: Update Statistics Display
Only count Reddit and X in "from the last 30 days" claim. WebSearch should be clearly labeled as supplementary.
## Acceptance Criteria
### Functional Requirements
- [x] Reddit search prompt includes explicit `from_date` and `to_date`
- [x] Items with dates before `from_date` are EXCLUDED, not just penalized
- [x] X search continues working (no regression)
- [x] WebSearch extracts dates from URLs (e.g., `/2026/01/24/`)
- [x] WebSearch extracts dates from snippets (e.g., "January 24, 2026")
- [x] WebSearch with verified recent dates gets +10 bonus
- [x] WebSearch with no date signals gets -20 penalty (but still appears)
- [x] WebSearch with verified OLD dates is EXCLUDED
### Non-Functional Requirements
- [ ] No increase in API latency
- [ ] Graceful handling when few recent results exist (return fewer, not older)
- [ ] Clear user messaging when results are limited due to strict filtering
### Quality Gates
- [ ] Test: Reddit search returns 0% results older than 30 days
- [ ] Test: X search continues to return 100% recent results
- [ ] Test: WebSearch is clearly differentiated in output
- [ ] Test: Edge case - topic with no recent content shows helpful message
## Implementation Order
1. **Phase 1**: Fix Reddit prompt (highest impact, simple change)
2. **Phase 2**: Add hard date filter in normalize.py (safety net)
3. **Phase 3**: Add WebSearch date extraction (URL + snippet parsing)
4. **Phase 4**: Update WebSearch scoring (bonus for verified, heavy penalty for unknown)
5. **Phase 5**: Update output display to show date confidence
## Testing Plan
### Before/After Test
Run same query before and after fix:
```
/last30days remotion launch videos
```
**Expected Before:**
- Reddit: 40% within 30 days
**Expected After:**
- Reddit: 100% within 30 days (or fewer results if not enough recent content)
### Edge Case Tests
| Scenario | Expected Behavior |
|----------|-------------------|
| Topic with no recent content | Return 0 results + helpful message |
| Topic with 5 recent results | Return 5 results (not pad with old ones) |
| Mixed old/new results | Only return new ones |
### WebSearch Date Extraction Tests
| URL/Snippet | Expected Date | Confidence |
|-------------|---------------|------------|
| `medium.com/blog/2026/01/15/title` | 2026-01-15 | high |
| `github.com/repo` + "Released Jan 20, 2026" | 2026-01-20 | med |
| `docs.example.com/guide` (no date signals) | None | low |
| `news.site.com/2024/05/old-article` | 2024-05-XX | EXCLUDE (too old) |
| Snippet: "Updated 3 days ago" | calculated | med |
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Fewer results for niche topics | High | Medium | Explain why in output |
| User confusion about reduced results | Medium | Low | Clear messaging |
| Date parsing errors exclude valid content | Low | Medium | Keep items with unknown dates, just label clearly |
## References
### Internal References
- Reddit search: `scripts/lib/openai_reddit.py:25-63`
- X search (working example): `scripts/lib/xai_x.py:26-55`
- Date confidence: `scripts/lib/dates.py:62-90`
- Scoring penalties: `scripts/lib/score.py:149-153`
- Normalization: `scripts/lib/normalize.py:49,99`
### External References
- OpenAI Responses API lacks native date filtering
- Must rely on prompt engineering + post-processing
+37
View File
@@ -0,0 +1,37 @@
[project]
name = "last30days-skill"
version = "3.3.2"
description = "Multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []
[dependency-groups]
dev = [
"pytest>=9.0.3,<10",
"pytest-cov>=7,<8",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = [
"-q",
"--tb=short",
]
[tool.coverage.run]
branch = true
source = ["skills/last30days/scripts", "tests"]
omit = [
"skills/last30days/scripts/lib/vendor/*",
"dist/*",
]
[tool.coverage.report]
skip_empty = true
show_missing = true
omit = [
"skills/last30days/scripts/lib/vendor/*",
"dist/*",
]
-73
View File
@@ -1,73 +0,0 @@
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across **Reddit, X, YouTube, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a prompt that works today, not six months ago.
## Three Headline Features
**1. Open-class skill with watchlists.** Add any topic to a watchlist -- your competitors, specific people, emerging technologies -- and /last30days re-researches it on demand or via cron. Designed for always-on environments like [Open Claw](https://github.com/openclaw/openclaw). SQLite-backed with FTS5 full-text search.
**2. YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a single post -- now the skill reads it. Inspired by [@steipete](https://x.com/steipete)'s yt-dlp + [summarize](https://github.com/steipete/summarize) toolchain.
**3. Works in OpenAI Codex CLI.** Same skill, same engine, same four sources. Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
Plus: **Bundled X search** -- vendored Bird GraphQL client (MIT). No external CLI, no npm install, no API keys needed. Just Node.js 22+ and your browser cookies.
## Real Results (verified Feb 15)
| Topic | Reddit | X | YouTube | Web |
|-------|--------|---|---------|-----|
| Nano Banana Pro | -- | 32 posts, 164 likes | 5 videos, 98K views, 5 transcripts | 10 pages |
| Seedance 2.0 access | 3 threads, 114 upvotes | 31 posts, 191 likes | 20 videos, 685K views, 4 transcripts | 10 pages |
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views, 5 transcripts | 10 pages |
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts, 110 likes | 18 videos, 6.15M views, 5 transcripts | 30 pages |
| AI generated ads | 12 threads | 29 posts, 101 likes | 3 videos, 83K views, 3 transcripts | 30 pages |
## What's New
### Added
- Open-class skill with watchlist, briefing, and history modes
- YouTube search + transcript extraction via yt-dlp
- OpenAI Codex CLI compatibility
- Bundled Twitter/X search (vendored Bird GraphQL, MIT)
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar Pro)
- `--diagnose` flag for source status checking
- `--store` flag for SQLite accumulation
- Conversational first-run experience (NUX)
### Changed
- Two-phase search architecture (entity-aware drill-down)
- Reddit JSON enrichment for real engagement metrics
- Smarter query construction with auto-retry on 0 results
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
- `--days=N` configurable lookback (thanks @jonthebeef)
### Fixed
- YouTube/Reddit timeout resilience
- Reddit 429 rate limit fail-fast
- Eager import crash in Codex environments
- X search returning 0 results on popular topics
- Windows Unicode crash (thanks @JosephOIbrahim)
## New Contributors
- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
## Credits
- [@steipete](https://github.com/steipete) -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
- [@galligan](https://github.com/galligan) -- Marketplace plugin inspiration
- [@hutchins](https://x.com/hutchins) -- Pushed for YouTube feature
## Install
```bash
# Claude Code
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
# Codex CLI
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
```
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
File diff suppressed because it is too large Load Diff
-436
View File
@@ -1,436 +0,0 @@
"""Bird X search client - vendored Twitter GraphQL search for /last30days v2.1.
Uses a vendored subset of @steipete/bird v0.8.0 (MIT License) to search X
via Twitter's GraphQL API. No external `bird` CLI binary needed - just Node.js 22+.
"""
import json
import os
import signal
import shutil
import subprocess
import sys
from pathlib import Path
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
# Path to the vendored bird-search wrapper
_BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
# Depth configurations: number of results to request
DEPTH_CONFIG = {
"quick": 12,
"default": 30,
"deep": 60,
}
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[Bird] {msg}\n")
sys.stderr.flush()
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for X search.
X search is literal keyword AND matching all words must appear.
Aggressively strip question/meta/research words to keep only the
core product/concept name (2-3 words max).
"""
text = topic.lower().strip()
# Phase 1: Strip multi-word prefixes (longest first)
prefixes = [
'what are the best', 'what is the best', 'what are the latest',
'what are people saying about', 'what do people think about',
'how do i use', 'how to use', 'how to',
'what are', 'what is', 'tips for', 'best practices for',
]
for p in prefixes:
if text.startswith(p + ' '):
text = text[len(p):].strip()
break
# Phase 2: Strip multi-word suffixes
suffixes = [
'best practices', 'use cases', 'prompt techniques',
'prompting techniques', 'prompting tips',
]
for s in suffixes:
if text.endswith(' ' + s):
text = text[:-len(s)].strip()
break
# Phase 3: Filter individual noise words
_noise = {
# Question/filler words
'a', 'an', 'the', 'is', 'are', 'was', 'were', 'and', 'or',
'of', 'in', 'on', 'for', 'with', 'about', 'to',
'people', 'saying', 'think', 'said', 'lately',
# Research/meta descriptors
'best', 'top', 'good', 'great', 'awesome', 'killer',
'latest', 'new', 'news', 'update', 'updates',
'practices', 'features', 'guide', 'tutorial',
'recommendations', 'advice', 'review', 'reviews',
'usecases', 'examples', 'comparison', 'versus', 'vs',
# Prompting meta words
'prompt', 'prompts', 'prompting', 'techniques', 'tips',
'tricks', 'methods', 'strategies', 'approaches',
# Action words
'using', 'uses', 'use',
}
words = text.split()
result = [w for w in words if w not in _noise]
return ' '.join(result[:3]) or topic.lower().strip() # Max 3 words
def is_bird_installed() -> bool:
"""Check if vendored Bird search module is available.
Returns:
True if bird-search.mjs exists and Node.js 22+ is in PATH.
"""
if not _BIRD_SEARCH_MJS.exists():
return False
return shutil.which("node") is not None
def is_bird_authenticated() -> Optional[str]:
"""Check if X credentials are available (env vars or browser cookies).
Returns:
Auth source string if authenticated, None otherwise.
"""
if not is_bird_installed():
return None
try:
result = subprocess.run(
["node", str(_BIRD_SEARCH_MJS), "--whoami"],
capture_output=True,
text=True,
timeout=15,
)
if result.returncode == 0 and result.stdout.strip():
return result.stdout.strip().split('\n')[0]
return None
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
return None
def check_npm_available() -> bool:
"""Check if npm is available (kept for API compatibility).
Returns:
True if 'npm' command is available in PATH, False otherwise.
"""
return shutil.which("npm") is not None
def install_bird() -> Tuple[bool, str]:
"""No-op - Bird search is vendored in v2.1, no installation needed.
Returns:
Tuple of (success, message).
"""
if is_bird_installed():
return True, "Bird search is bundled with /last30days v2.1 - no installation needed."
if not shutil.which("node"):
return False, "Node.js 22+ is required for X search. Install Node.js first."
return False, f"Vendored bird-search.mjs not found at {_BIRD_SEARCH_MJS}"
def get_bird_status() -> Dict[str, Any]:
"""Get comprehensive Bird search status.
Returns:
Dict with keys: installed, authenticated, username, can_install
"""
installed = is_bird_installed()
auth_source = is_bird_authenticated() if installed else None
return {
"installed": installed,
"authenticated": auth_source is not None,
"username": auth_source, # Now returns auth source (e.g., "Safari", "env AUTH_TOKEN")
"can_install": True, # Always vendored in v2.1
}
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds
Returns:
Raw Bird JSON response or error dict.
"""
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count),
"--json",
]
# Use process groups for clean cleanup on timeout/kill
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
preexec_fn=preexec,
)
# Register for cleanup tracking (if available)
try:
from last30days import register_child_pid, unregister_child_pid
register_child_pid(proc.pid)
except ImportError:
pass
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
# Kill the entire process group
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
return {"error": f"Search timed out after {timeout}s", "items": []}
finally:
try:
from last30days import unregister_child_pid
unregister_child_pid(proc.pid)
except (ImportError, Exception):
pass
if proc.returncode != 0:
error = stderr.strip() if stderr else "Bird search failed"
return {"error": error, "items": []}
output = stdout.strip() if stdout else ""
if not output:
return {"items": []}
return json.loads(output)
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
def search_x(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search X using Bird CLI with automatic retry on 0 results.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD) - unused but kept for API compatibility
depth: Research depth - "quick", "default", or "deep"
Returns:
Raw Bird JSON response or error dict.
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
# Extract core subject - X search is literal, not semantic
core_topic = _extract_core_subject(topic)
query = f"{core_topic} since:{from_date}"
_log(f"Searching: {query}")
response = _run_bird_search(query, count, timeout)
# Check if we got results
items = parse_bird_response(response)
# Retry with fewer keywords if 0 results and query has 3+ words
core_words = core_topic.split()
if not items and len(core_words) > 2:
shorter = ' '.join(core_words[:2])
_log(f"0 results for '{core_topic}', retrying with '{shorter}'")
query = f"{shorter} since:{from_date}"
response = _run_bird_search(query, count, timeout)
return response
def search_handles(
handles: List[str],
topic: str,
from_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific X handles for topic-related content.
Runs targeted Bird searches using `from:handle topic` syntax.
Used in Phase 2 supplemental search after entity extraction.
Args:
handles: List of X handles to search (without @)
topic: Search topic (core subject, not full verbose query)
from_date: Start date (YYYY-MM-DD)
count_per: Results to request per handle
Returns:
List of raw item dicts (same format as parse_bird_response output).
"""
all_items = []
core_topic = _extract_core_subject(topic)
for handle in handles:
handle = handle.lstrip("@")
query = f"from:{handle} {core_topic} since:{from_date}"
cmd = [
"node", str(_BIRD_SEARCH_MJS),
query,
"--count", str(count_per),
"--json",
]
preexec = os.setsid if hasattr(os, 'setsid') else None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
preexec_fn=preexec,
)
try:
stdout, stderr = proc.communicate(timeout=15)
except subprocess.TimeoutExpired:
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
proc.kill()
proc.wait(timeout=5)
_log(f"Handle search timed out for @{handle}")
continue
if proc.returncode != 0:
_log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
continue
output = (stdout or "").strip()
if not output:
continue
response = json.loads(output)
items = parse_bird_response(response)
all_items.extend(items)
except json.JSONDecodeError:
_log(f"Invalid JSON from handle search for @{handle}")
except Exception as e:
_log(f"Handle search error for @{handle}: {e}")
return all_items
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
Args:
response: Raw Bird JSON response
Returns:
List of normalized item dicts matching xai_x.parse_x_response() format.
"""
items = []
# Check for errors
if "error" in response and response["error"]:
_log(f"Bird error: {response['error']}")
return items
# Bird returns a list of tweets directly or under a key
raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
if not isinstance(raw_items, list):
return items
for i, tweet in enumerate(raw_items):
if not isinstance(tweet, dict):
continue
# Extract URL - Bird uses permanent_url or we construct from id
url = tweet.get("permanent_url") or tweet.get("url", "")
if not url and tweet.get("id"):
# Try different field structures Bird might use
author = tweet.get("author", {}) or tweet.get("user", {})
screen_name = author.get("username") or author.get("screen_name", "")
if screen_name:
url = f"https://x.com/{screen_name}/status/{tweet['id']}"
if not url:
continue
# Parse date from created_at/createdAt (e.g., "Wed Jan 15 14:30:00 +0000 2026")
date = None
created_at = tweet.get("createdAt") or tweet.get("created_at", "")
if created_at:
try:
# Try ISO format first (e.g., "2026-02-03T22:33:32Z")
# Check for ISO date separator, not just "T" (which appears in "Tue")
if len(created_at) > 10 and created_at[10] == "T":
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
else:
# Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
date = dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
# Extract user info (Bird uses author.username, older format uses user.screen_name)
author = tweet.get("author", {}) or tweet.get("user", {})
author_handle = author.get("username") or author.get("screen_name", "") or tweet.get("author_handle", "")
# Build engagement dict (Bird uses camelCase: likeCount, retweetCount, etc.)
engagement = {
"likes": tweet.get("likeCount") or tweet.get("like_count") or tweet.get("favorite_count"),
"reposts": tweet.get("retweetCount") or tweet.get("retweet_count"),
"replies": tweet.get("replyCount") or tweet.get("reply_count"),
"quotes": tweet.get("quoteCount") or tweet.get("quote_count"),
}
# Convert to int where possible
for key in engagement:
if engagement[key] is not None:
try:
engagement[key] = int(engagement[key])
except (ValueError, TypeError):
engagement[key] = None
# Build normalized item
item = {
"id": f"X{i+1}",
"text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
"url": url,
"author_handle": author_handle.lstrip("@"),
"date": date,
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
"why_relevant": "", # Bird doesn't provide relevance explanations
"relevance": 0.7, # Default relevance, let score.py re-rank
}
items.append(item)
return items
-213
View File
@@ -1,213 +0,0 @@
"""Brave Search web search for last30days skill.
Uses the Brave Search API as a fallback web search backend.
Simple, cheap (free tier: 2,000 queries/month), widely available.
API docs: https://api-dashboard.search.brave.com/app/documentation/web-search/get-started
"""
import html
import re
import sys
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
from urllib.parse import urlencode, urlparse
from . import http
ENDPOINT = "https://api.search.brave.com/res/v1/web/search"
# Freshness codes: pd=24h, pw=7d, pm=31d
FRESHNESS_MAP = {1: "pd", 7: "pw", 31: "pm"}
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Brave Search API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: Brave Search API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
count = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
# Calculate days for freshness filter
days = _days_between(from_date, to_date)
freshness = _brave_freshness(days)
params = {
"q": topic,
"result_filter": "web,news",
"count": count,
"safesearch": "strict",
"text_decorations": 0,
"spellcheck": 0,
}
if freshness:
params["freshness"] = freshness
url = f"{ENDPOINT}?{urlencode(params)}"
sys.stderr.write(f"[Web] Searching Brave for: {topic}\n")
sys.stderr.flush()
response = http.request(
"GET",
url,
headers={"X-Subscription-Token": api_key},
timeout=15,
)
return _normalize_results(response, from_date, to_date)
def _days_between(from_date: str, to_date: str) -> int:
"""Calculate days between two YYYY-MM-DD dates."""
try:
d1 = datetime.strptime(from_date, "%Y-%m-%d")
d2 = datetime.strptime(to_date, "%Y-%m-%d")
return max(1, (d2 - d1).days)
except (ValueError, TypeError):
return 30
def _brave_freshness(days: Optional[int]) -> Optional[str]:
"""Convert days to Brave freshness parameter.
Uses canned codes for <=31d, explicit date range for longer periods.
"""
if days is None:
return None
code = next((v for d, v in sorted(FRESHNESS_MAP.items()) if days <= d), None)
if code:
return code
start = (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d")
end = datetime.now(timezone.utc).strftime("%Y-%m-%d")
return f"{start}to{end}"
def _normalize_results(
response: Dict[str, Any],
from_date: str,
to_date: str,
) -> List[Dict[str, Any]]:
"""Convert Brave Search response to websearch item schema.
Merges news + web results, cleans HTML entities, filters excluded domains.
"""
items = []
# Merge news results (tend to be more recent) with web results
raw_results = (
response.get("news", {}).get("results", []) +
response.get("web", {}).get("results", [])
)
for i, result in enumerate(raw_results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = _clean_html(str(result.get("title", "")).strip())
snippet = _clean_html(str(result.get("description", "")).strip())
if not title and not snippet:
continue
# Parse date from Brave's 'age' field or 'page_age'
date = _parse_brave_date(result.get("age"), result.get("page_age"))
date_confidence = "med" if date else "low"
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": snippet[:500],
"date": date,
"date_confidence": date_confidence,
"relevance": 0.6, # Brave doesn't provide relevance scores
"why_relevant": "",
})
sys.stderr.write(f"[Web] Brave: {len(items)} results\n")
sys.stderr.flush()
return items
def _clean_html(text: str) -> str:
"""Remove HTML tags and decode entities."""
text = re.sub(r"<[^>]*>", "", text)
text = html.unescape(text)
return text
def _parse_brave_date(age: Optional[str], page_age: Optional[str]) -> Optional[str]:
"""Parse Brave's age/page_age fields to YYYY-MM-DD.
Brave returns dates like "3 hours ago", "2 days ago", "January 24, 2026".
"""
text = age or page_age
if not text:
return None
text_lower = text.lower().strip()
now = datetime.now()
# "X hours ago" -> today
if re.search(r'\d+\s*hours?\s*ago', text_lower):
return now.strftime("%Y-%m-%d")
# "X days ago"
match = re.search(r'(\d+)\s*days?\s*ago', text_lower)
if match:
days = int(match.group(1))
if days <= 60:
return (now - timedelta(days=days)).strftime("%Y-%m-%d")
# "X weeks ago"
match = re.search(r'(\d+)\s*weeks?\s*ago', text_lower)
if match:
weeks = int(match.group(1))
return (now - timedelta(weeks=weeks)).strftime("%Y-%m-%d")
# ISO format: 2026-01-24T...
match = re.search(r'(\d{4}-\d{2}-\d{2})', text)
if match:
return match.group(1)
return None
-165
View File
@@ -1,165 +0,0 @@
"""Caching utilities for last30days skill."""
import hashlib
import json
import os
import tempfile
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional
CACHE_DIR = Path.home() / ".cache" / "last30days"
DEFAULT_TTL_HOURS = 24
MODEL_CACHE_TTL_DAYS = 7
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
def ensure_cache_dir():
"""Ensure cache directory exists. Supports env override and sandbox fallback."""
global CACHE_DIR, MODEL_CACHE_FILE
env_dir = os.environ.get("LAST30DAYS_CACHE_DIR")
if env_dir:
CACHE_DIR = Path(env_dir)
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
try:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
except PermissionError:
CACHE_DIR = Path(tempfile.gettempdir()) / "last30days" / "cache"
MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
CACHE_DIR.mkdir(parents=True, exist_ok=True)
def get_cache_key(topic: str, from_date: str, to_date: str, sources: str) -> str:
"""Generate a cache key from query parameters."""
key_data = f"{topic}|{from_date}|{to_date}|{sources}"
return hashlib.sha256(key_data.encode()).hexdigest()[:16]
def get_cache_path(cache_key: str) -> Path:
"""Get path to cache file."""
return CACHE_DIR / f"{cache_key}.json"
def is_cache_valid(cache_path: Path, ttl_hours: int = DEFAULT_TTL_HOURS) -> bool:
"""Check if cache file exists and is within TTL."""
if not cache_path.exists():
return False
try:
stat = cache_path.stat()
mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
now = datetime.now(timezone.utc)
age_hours = (now - mtime).total_seconds() / 3600
return age_hours < ttl_hours
except OSError:
return False
def load_cache(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> Optional[dict]:
"""Load data from cache if valid."""
cache_path = get_cache_path(cache_key)
if not is_cache_valid(cache_path, ttl_hours):
return None
try:
with open(cache_path, 'r') as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return None
def get_cache_age_hours(cache_path: Path) -> Optional[float]:
"""Get age of cache file in hours."""
if not cache_path.exists():
return None
try:
stat = cache_path.stat()
mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
now = datetime.now(timezone.utc)
return (now - mtime).total_seconds() / 3600
except OSError:
return None
def load_cache_with_age(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> tuple:
"""Load data from cache with age info.
Returns:
Tuple of (data, age_hours) or (None, None) if invalid
"""
cache_path = get_cache_path(cache_key)
if not is_cache_valid(cache_path, ttl_hours):
return None, None
age = get_cache_age_hours(cache_path)
try:
with open(cache_path, 'r') as f:
return json.load(f), age
except (json.JSONDecodeError, OSError):
return None, None
def save_cache(cache_key: str, data: dict):
"""Save data to cache."""
ensure_cache_dir()
cache_path = get_cache_path(cache_key)
try:
with open(cache_path, 'w') as f:
json.dump(data, f)
except OSError:
pass # Silently fail on cache write errors
def clear_cache():
"""Clear all cache files."""
if CACHE_DIR.exists():
for f in CACHE_DIR.glob("*.json"):
try:
f.unlink()
except OSError:
pass
# Model selection cache (longer TTL) — MODEL_CACHE_FILE is set at module level
# and updated by ensure_cache_dir() if env override or fallback is needed.
def load_model_cache() -> dict:
"""Load model selection cache."""
if not is_cache_valid(MODEL_CACHE_FILE, MODEL_CACHE_TTL_DAYS * 24):
return {}
try:
with open(MODEL_CACHE_FILE, 'r') as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return {}
def save_model_cache(data: dict):
"""Save model selection cache."""
ensure_cache_dir()
try:
with open(MODEL_CACHE_FILE, 'w') as f:
json.dump(data, f)
except OSError:
pass
def get_cached_model(provider: str) -> Optional[str]:
"""Get cached model selection for a provider."""
cache = load_model_cache()
return cache.get(provider)
def set_cached_model(provider: str, model: str):
"""Cache model selection for a provider."""
cache = load_model_cache()
cache[provider] = model
cache['updated_at'] = datetime.now(timezone.utc).isoformat()
save_model_cache(cache)
-130
View File
@@ -1,130 +0,0 @@
"""Near-duplicate detection for last30days skill."""
import re
from typing import List, Set, Tuple, Union
from . import schema
def normalize_text(text: str) -> str:
"""Normalize text for comparison.
- Lowercase
- Remove punctuation
- Collapse whitespace
"""
text = text.lower()
text = re.sub(r'[^\w\s]', ' ', text)
text = re.sub(r'\s+', ' ', text)
return text.strip()
def get_ngrams(text: str, n: int = 3) -> Set[str]:
"""Get character n-grams from text."""
text = normalize_text(text)
if len(text) < n:
return {text}
return {text[i:i+n] for i in range(len(text) - n + 1)}
def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
"""Compute Jaccard similarity between two sets."""
if not set1 or not set2:
return 0.0
intersection = len(set1 & set2)
union = len(set1 | set2)
return intersection / union if union > 0 else 0.0
def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem]) -> str:
"""Get comparable text from an item."""
if isinstance(item, schema.RedditItem):
return item.title
elif isinstance(item, schema.YouTubeItem):
return f"{item.title} {item.channel_name}"
else:
return item.text
def find_duplicates(
items: List[Union[schema.RedditItem, schema.XItem]],
threshold: float = 0.7,
) -> List[Tuple[int, int]]:
"""Find near-duplicate pairs in items.
Args:
items: List of items to check
threshold: Similarity threshold (0-1)
Returns:
List of (i, j) index pairs where i < j and items are similar
"""
duplicates = []
# Pre-compute n-grams
ngrams = [get_ngrams(get_item_text(item)) for item in items]
for i in range(len(items)):
for j in range(i + 1, len(items)):
similarity = jaccard_similarity(ngrams[i], ngrams[j])
if similarity >= threshold:
duplicates.append((i, j))
return duplicates
def dedupe_items(
items: List[Union[schema.RedditItem, schema.XItem]],
threshold: float = 0.7,
) -> List[Union[schema.RedditItem, schema.XItem]]:
"""Remove near-duplicates, keeping highest-scored item.
Args:
items: List of items (should be pre-sorted by score descending)
threshold: Similarity threshold
Returns:
Deduplicated items
"""
if len(items) <= 1:
return items
# Find duplicate pairs
dup_pairs = find_duplicates(items, threshold)
# Mark indices to remove (always remove the lower-scored one)
# Since items are pre-sorted by score, the second index is always lower
to_remove = set()
for i, j in dup_pairs:
# Keep the higher-scored one (lower index in sorted list)
if items[i].score >= items[j].score:
to_remove.add(j)
else:
to_remove.add(i)
# Return items not marked for removal
return [item for idx, item in enumerate(items) if idx not in to_remove]
def dedupe_reddit(
items: List[schema.RedditItem],
threshold: float = 0.7,
) -> List[schema.RedditItem]:
"""Dedupe Reddit items."""
return dedupe_items(items, threshold)
def dedupe_x(
items: List[schema.XItem],
threshold: float = 0.7,
) -> List[schema.XItem]:
"""Dedupe X items."""
return dedupe_items(items, threshold)
def dedupe_youtube(
items: List[schema.YouTubeItem],
threshold: float = 0.7,
) -> List[schema.YouTubeItem]:
"""Dedupe YouTube items."""
return dedupe_items(items, threshold)
-276
View File
@@ -1,276 +0,0 @@
"""Environment and API key management for last30days skill."""
import json
import os
from pathlib import Path
from typing import Optional, Dict, Any
# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
# Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
_config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
if _config_override == "":
# Empty string = no config file (clean mode)
CONFIG_DIR = None
CONFIG_FILE = None
elif _config_override:
CONFIG_DIR = Path(_config_override)
CONFIG_FILE = CONFIG_DIR / ".env"
else:
CONFIG_DIR = Path.home() / ".config" / "last30days"
CONFIG_FILE = CONFIG_DIR / ".env"
def load_env_file(path: Path) -> Dict[str, str]:
"""Load environment variables from a file."""
env = {}
if not path.exists():
return env
with open(path, 'r') as f:
for line in f:
line = line.strip()
if not line or line.startswith('#'):
continue
if '=' in line:
key, _, value = line.partition('=')
key = key.strip()
value = value.strip()
# Remove quotes if present
if value and value[0] in ('"', "'") and value[-1] == value[0]:
value = value[1:-1]
if key and value:
env[key] = value
return env
def get_config() -> Dict[str, Any]:
"""Load configuration from ~/.config/last30days/.env and environment."""
# Load from config file first (if configured)
file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
# Build config: process.env > .env file
keys = [
('OPENAI_API_KEY', None),
('XAI_API_KEY', None),
('OPENROUTER_API_KEY', None),
('PARALLEL_API_KEY', None),
('BRAVE_API_KEY', None),
('OPENAI_MODEL_POLICY', 'auto'),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_POLICY', 'latest'),
('XAI_MODEL_PIN', None),
]
config = {}
for key, default in keys:
config[key] = os.environ.get(key) or file_env.get(key, default)
return config
def config_exists() -> bool:
"""Check if configuration file exists."""
return CONFIG_FILE.exists()
def get_available_sources(config: Dict[str, Any]) -> str:
"""Determine which sources are available based on API keys.
Returns: 'all', 'both', 'reddit', 'reddit-web', 'x', 'x-web', 'web', or 'none'
"""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
has_web = has_web_search_keys(config)
if has_openai and has_xai:
return 'all' if has_web else 'both'
elif has_openai:
return 'reddit-web' if has_web else 'reddit'
elif has_xai:
return 'x-web' if has_web else 'x'
elif has_web:
return 'web'
else:
return 'web' # Fallback: assistant WebSearch (no API keys needed)
def has_web_search_keys(config: Dict[str, Any]) -> bool:
"""Check if any web search API keys are configured."""
return bool(config.get('OPENROUTER_API_KEY') or config.get('PARALLEL_API_KEY') or config.get('BRAVE_API_KEY'))
def get_web_search_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available web search backend.
Priority: Parallel AI > Brave > OpenRouter/Sonar Pro
Returns: 'parallel', 'brave', 'openrouter', or None
"""
if config.get('PARALLEL_API_KEY'):
return 'parallel'
if config.get('BRAVE_API_KEY'):
return 'brave'
if config.get('OPENROUTER_API_KEY'):
return 'openrouter'
return None
def get_missing_keys(config: Dict[str, Any]) -> str:
"""Determine which sources are missing (accounting for Bird).
Returns: 'all', 'both', 'reddit', 'x', 'web', or 'none'
"""
has_openai = bool(config.get('OPENAI_API_KEY'))
has_xai = bool(config.get('XAI_API_KEY'))
has_web = has_web_search_keys(config)
# Check if Bird provides X access (import here to avoid circular dependency)
from . import bird_x
has_bird = bird_x.is_bird_installed() and bird_x.is_bird_authenticated()
has_x = has_xai or has_bird
if has_openai and has_x and has_web:
return 'none'
elif has_openai and has_x:
return 'web' # Missing web search keys
elif has_openai:
return 'x' # Missing X source (and possibly web)
elif has_x:
return 'reddit' # Missing OpenAI key (and possibly web)
else:
return 'all' # Missing everything
def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]:
"""Validate requested sources against available keys.
Args:
requested: 'auto', 'reddit', 'x', 'both', or 'web'
available: Result from get_available_sources()
include_web: If True, add WebSearch to available sources
Returns:
Tuple of (effective_sources, error_message)
"""
# No API keys at all
if available == 'none':
if requested == 'auto':
return 'web', "No API keys configured. The assistant can still search the web if it has a search tool."
elif requested == 'web':
return 'web', None
else:
return 'web', f"No API keys configured. Add keys to ~/.config/last30days/.env for Reddit/X."
# Web-only mode (only web search API keys)
if available == 'web':
if requested == 'auto':
return 'web', None
elif requested == 'web':
return 'web', None
else:
return 'web', f"Only web search keys configured. Add OPENAI_API_KEY for Reddit, XAI_API_KEY for X."
if requested == 'auto':
# Add web to sources if include_web is set
if include_web:
if available == 'both':
return 'all', None # reddit + x + web
elif available == 'reddit':
return 'reddit-web', None
elif available == 'x':
return 'x-web', None
return available, None
if requested == 'web':
return 'web', None
if requested == 'both':
if available not in ('both',):
missing = 'xAI' if available == 'reddit' else 'OpenAI'
return 'none', f"Requested both sources but {missing} key is missing. Use --sources=auto to use available keys."
if include_web:
return 'all', None
return 'both', None
if requested == 'reddit':
if available == 'x':
return 'none', "Requested Reddit but only xAI key is available."
if include_web:
return 'reddit-web', None
return 'reddit', None
if requested == 'x':
if available == 'reddit':
return 'none', "Requested X but only OpenAI key is available."
if include_web:
return 'x-web', None
return 'x', None
return requested, None
def get_x_source(config: Dict[str, Any]) -> Optional[str]:
"""Determine the best available X/Twitter source.
Priority: Bird (free) xAI (paid API)
Args:
config: Configuration dict from get_config()
Returns:
'bird' if Bird is installed and authenticated,
'xai' if XAI_API_KEY is configured,
None if no X source available.
"""
# Import here to avoid circular dependency
from . import bird_x
# Check Bird first (free option)
if bird_x.is_bird_installed():
username = bird_x.is_bird_authenticated()
if username:
return 'bird'
# Fall back to xAI if key exists
if config.get('XAI_API_KEY'):
return 'xai'
return None
def is_ytdlp_available() -> bool:
"""Check if yt-dlp is installed for YouTube search."""
from . import youtube_yt
return youtube_yt.is_ytdlp_installed()
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions.
Returns:
Dict with keys: source, bird_installed, bird_authenticated,
bird_username, xai_available, can_install_bird
"""
from . import bird_x
bird_status = bird_x.get_bird_status()
xai_available = bool(config.get('XAI_API_KEY'))
# Determine active source
if bird_status["authenticated"]:
source = 'bird'
elif xai_available:
source = 'xai'
else:
source = None
return {
"source": source,
"bird_installed": bird_status["installed"],
"bird_authenticated": bird_status["authenticated"],
"bird_username": bird_status["username"],
"xai_available": xai_available,
"can_install_bird": bird_status["can_install"],
}
-154
View File
@@ -1,154 +0,0 @@
"""HTTP utilities for last30days skill (stdlib only)."""
import json
import os
import sys
import time
import urllib.error
import urllib.request
from typing import Any, Dict, Optional
from urllib.parse import urlencode
DEFAULT_TIMEOUT = 30
DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")
def log(msg: str):
"""Log debug message to stderr."""
if DEBUG:
sys.stderr.write(f"[DEBUG] {msg}\n")
sys.stderr.flush()
MAX_RETRIES = 3
RETRY_DELAY = 1.0
USER_AGENT = "last30days-skill/2.1 (Assistant Skill)"
class HTTPError(Exception):
"""HTTP request error with status code."""
def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
super().__init__(message)
self.status_code = status_code
self.body = body
def request(
method: str,
url: str,
headers: Optional[Dict[str, str]] = None,
json_data: Optional[Dict[str, Any]] = None,
timeout: int = DEFAULT_TIMEOUT,
retries: int = MAX_RETRIES,
) -> Dict[str, Any]:
"""Make an HTTP request and return JSON response.
Args:
method: HTTP method (GET, POST, etc.)
url: Request URL
headers: Optional headers dict
json_data: Optional JSON body (for POST)
timeout: Request timeout in seconds
retries: Number of retries on failure
Returns:
Parsed JSON response
Raises:
HTTPError: On request failure
"""
headers = headers or {}
headers.setdefault("User-Agent", USER_AGENT)
data = None
if json_data is not None:
data = json.dumps(json_data).encode('utf-8')
headers.setdefault("Content-Type", "application/json")
req = urllib.request.Request(url, data=data, headers=headers, method=method)
log(f"{method} {url}")
if json_data:
log(f"Payload keys: {list(json_data.keys())}")
last_error = None
for attempt in range(retries):
try:
with urllib.request.urlopen(req, timeout=timeout) as response:
body = response.read().decode('utf-8')
log(f"Response: {response.status} ({len(body)} bytes)")
return json.loads(body) if body else {}
except urllib.error.HTTPError as e:
body = None
try:
body = e.read().decode('utf-8')
except:
pass
log(f"HTTP Error {e.code}: {e.reason}")
if body:
log(f"Error body: {body[:500]}")
last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)
# Don't retry client errors (4xx) except rate limits
if 400 <= e.code < 500 and e.code != 429:
raise last_error
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
except urllib.error.URLError as e:
log(f"URL Error: {e.reason}")
last_error = HTTPError(f"URL Error: {e.reason}")
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
except json.JSONDecodeError as e:
log(f"JSON decode error: {e}")
last_error = HTTPError(f"Invalid JSON response: {e}")
raise last_error
except (OSError, TimeoutError, ConnectionResetError) as e:
# Handle socket-level errors (connection reset, timeout, etc.)
log(f"Connection error: {type(e).__name__}: {e}")
last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
if attempt < retries - 1:
time.sleep(RETRY_DELAY * (attempt + 1))
if last_error:
raise last_error
raise HTTPError("Request failed with no error details")
def get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
"""Make a GET request."""
return request("GET", url, headers=headers, **kwargs)
def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
"""Make a POST request with JSON body."""
return request("POST", url, headers=headers, json_data=json_data, **kwargs)
def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
"""Fetch Reddit thread JSON.
Args:
path: Reddit path (e.g., /r/subreddit/comments/id/title)
timeout: HTTP timeout per attempt in seconds
retries: Number of retries on failure
Returns:
Parsed JSON response
"""
# Ensure path starts with /
if not path.startswith('/'):
path = '/' + path
# Remove trailing slash and add .json
path = path.rstrip('/')
if not path.endswith('.json'):
path = path + '.json'
url = f"https://www.reddit.com{path}?raw_json=1"
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/json",
}
return get(url, headers=headers, timeout=timeout, retries=retries)
-175
View File
@@ -1,175 +0,0 @@
"""Model auto-selection for last30days skill."""
import re
from typing import Dict, List, Optional, Tuple
from . import cache, http
# OpenAI API
OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4.1", "gpt-4o"]
# xAI API - Agent Tools API requires grok-4 family
XAI_MODELS_URL = "https://api.x.ai/v1/models"
XAI_ALIASES = {
"latest": "grok-4-1-fast", # Required for x_search tool
"stable": "grok-4-1-fast",
}
def parse_version(model_id: str) -> Optional[Tuple[int, ...]]:
"""Parse semantic version from model ID.
Examples:
gpt-5 -> (5,)
gpt-5.2 -> (5, 2)
gpt-5.2.1 -> (5, 2, 1)
"""
match = re.search(r'(\d+(?:\.\d+)*)', model_id)
if match:
return tuple(int(x) for x in match.group(1).split('.'))
return None
def is_mainline_openai_model(model_id: str) -> bool:
"""Check if model is a mainline GPT model (not mini/nano/chat/codex/pro)."""
model_lower = model_id.lower()
# Must be gpt-4o, gpt-4.1+, or gpt-5+ series (mainline, not mini/nano/etc)
if not re.match(r'^gpt-(?:4o|4\.1|5)(\.\d+)*$', model_lower):
return False
# Exclude variants
excludes = ['mini', 'nano', 'chat', 'codex', 'pro', 'preview', 'turbo']
for exc in excludes:
if exc in model_lower:
return False
return True
def select_openai_model(
api_key: str,
policy: str = "auto",
pin: Optional[str] = None,
mock_models: Optional[List[Dict]] = None,
) -> str:
"""Select the best OpenAI model based on policy.
Args:
api_key: OpenAI API key
policy: 'auto' or 'pinned'
pin: Model to use if policy is 'pinned'
mock_models: Mock model list for testing
Returns:
Selected model ID
"""
if policy == "pinned" and pin:
return pin
# Check cache first
cached = cache.get_cached_model("openai")
if cached:
return cached
# Fetch model list
if mock_models is not None:
models = mock_models
else:
try:
headers = {"Authorization": f"Bearer {api_key}"}
response = http.get(OPENAI_MODELS_URL, headers=headers)
models = response.get("data", [])
except http.HTTPError:
# Fall back to known models
return OPENAI_FALLBACK_MODELS[0]
# Filter to mainline models
candidates = [m for m in models if is_mainline_openai_model(m.get("id", ""))]
if not candidates:
# No gpt-5 models found, use fallback
return OPENAI_FALLBACK_MODELS[0]
# Sort by version (descending), then by created timestamp
def sort_key(m):
version = parse_version(m.get("id", "")) or (0,)
created = m.get("created", 0)
return (version, created)
candidates.sort(key=sort_key, reverse=True)
selected = candidates[0]["id"]
# Cache the selection
cache.set_cached_model("openai", selected)
return selected
def select_xai_model(
api_key: str,
policy: str = "latest",
pin: Optional[str] = None,
mock_models: Optional[List[Dict]] = None,
) -> str:
"""Select the best xAI model based on policy.
Args:
api_key: xAI API key
policy: 'latest', 'stable', or 'pinned'
pin: Model to use if policy is 'pinned'
mock_models: Mock model list for testing
Returns:
Selected model ID
"""
if policy == "pinned" and pin:
return pin
# Use alias system
if policy in XAI_ALIASES:
alias = XAI_ALIASES[policy]
# Check cache first
cached = cache.get_cached_model("xai")
if cached:
return cached
# Cache the alias
cache.set_cached_model("xai", alias)
return alias
# Default to latest
return XAI_ALIASES["latest"]
def get_models(
config: Dict,
mock_openai_models: Optional[List[Dict]] = None,
mock_xai_models: Optional[List[Dict]] = None,
) -> Dict[str, Optional[str]]:
"""Get selected models for both providers.
Returns:
Dict with 'openai' and 'xai' keys
"""
result = {"openai": None, "xai": None}
if config.get("OPENAI_API_KEY"):
result["openai"] = select_openai_model(
config["OPENAI_API_KEY"],
config.get("OPENAI_MODEL_POLICY", "auto"),
config.get("OPENAI_MODEL_PIN"),
mock_openai_models,
)
if config.get("XAI_API_KEY"):
result["xai"] = select_xai_model(
config["XAI_API_KEY"],
config.get("XAI_MODEL_POLICY", "latest"),
config.get("XAI_MODEL_PIN"),
mock_xai_models,
)
return result
-205
View File
@@ -1,205 +0,0 @@
"""Normalization of raw API data to canonical schema."""
from typing import Any, Dict, List, TypeVar, Union
from . import dates, schema
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem)
def filter_by_date_range(
items: List[T],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List[T]:
"""Hard filter: Remove items outside the date range.
This is the safety net - even if the prompt lets old content through,
this filter will exclude it.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD) - exclude items before this
to_date: End date (YYYY-MM-DD) - exclude items after this
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range (or unknown dates if not required)
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with scoring penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
# Hard filter: if date is after to_date, exclude (likely parsing error)
if item.date > to_date:
continue # DROP - future date
result.append(item)
return result
def normalize_reddit_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.RedditItem]:
"""Normalize raw Reddit items to schema.
Args:
items: Raw Reddit items from API
from_date: Start of date range
to_date: End of date range
Returns:
List of RedditItem objects
"""
normalized = []
for item in items:
# Parse engagement
engagement = None
eng_raw = item.get("engagement")
if isinstance(eng_raw, dict):
engagement = schema.Engagement(
score=eng_raw.get("score"),
num_comments=eng_raw.get("num_comments"),
upvote_ratio=eng_raw.get("upvote_ratio"),
)
# Parse comments
top_comments = []
for c in item.get("top_comments", []):
top_comments.append(schema.Comment(
score=c.get("score", 0),
date=c.get("date"),
author=c.get("author", ""),
excerpt=c.get("excerpt", ""),
url=c.get("url", ""),
))
# Determine date confidence
date_str = item.get("date")
date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
normalized.append(schema.RedditItem(
id=item.get("id", ""),
title=item.get("title", ""),
url=item.get("url", ""),
subreddit=item.get("subreddit", ""),
date=date_str,
date_confidence=date_confidence,
engagement=engagement,
top_comments=top_comments,
comment_insights=item.get("comment_insights", []),
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def normalize_x_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.XItem]:
"""Normalize raw X items to schema.
Args:
items: Raw X items from API
from_date: Start of date range
to_date: End of date range
Returns:
List of XItem objects
"""
normalized = []
for item in items:
# Parse engagement
engagement = None
eng_raw = item.get("engagement")
if isinstance(eng_raw, dict):
engagement = schema.Engagement(
likes=eng_raw.get("likes"),
reposts=eng_raw.get("reposts"),
replies=eng_raw.get("replies"),
quotes=eng_raw.get("quotes"),
)
# Determine date confidence
date_str = item.get("date")
date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
normalized.append(schema.XItem(
id=item.get("id", ""),
text=item.get("text", ""),
url=item.get("url", ""),
author_handle=item.get("author_handle", ""),
date=date_str,
date_confidence=date_confidence,
engagement=engagement,
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def normalize_youtube_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.YouTubeItem]:
"""Normalize raw YouTube items to schema.
Args:
items: Raw YouTube items from yt-dlp
from_date: Start of date range
to_date: End of date range
Returns:
List of YouTubeItem objects
"""
normalized = []
for item in items:
# Parse engagement
eng_raw = item.get("engagement") or {}
engagement = schema.Engagement(
views=eng_raw.get("views"),
likes=eng_raw.get("likes"),
num_comments=eng_raw.get("comments"),
)
# YouTube dates are reliable (always YYYY-MM-DD from yt-dlp)
date_str = item.get("date")
normalized.append(schema.YouTubeItem(
id=item.get("video_id", ""),
title=item.get("title", ""),
url=item.get("url", ""),
channel_name=item.get("channel_name", ""),
date=date_str,
date_confidence="high",
engagement=engagement,
transcript_snippet=item.get("transcript_snippet", ""),
relevance=item.get("relevance", 0.7),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def items_to_dicts(items: List) -> List[Dict[str, Any]]:
"""Convert schema items to dicts for JSON serialization."""
return [item.to_dict() for item in items]
-374
View File
@@ -1,374 +0,0 @@
"""OpenAI Responses API client for Reddit discovery."""
import json
import re
import sys
from typing import Any, Dict, List, Optional
from . import http
# Fallback models when the selected model isn't accessible (e.g., org not verified for GPT-5)
MODEL_FALLBACK_ORDER = ["gpt-4.1", "gpt-4o", "gpt-4o-mini"]
def _log_error(msg: str):
"""Log error to stderr."""
sys.stderr.write(f"[REDDIT ERROR] {msg}\n")
sys.stderr.flush()
def _log_info(msg: str):
"""Log info to stderr."""
sys.stderr.write(f"[REDDIT] {msg}\n")
sys.stderr.flush()
def _is_model_access_error(error: http.HTTPError) -> bool:
"""Check if error is due to model access/verification issues."""
if error.status_code not in (400, 403):
return False
if not error.body:
return False
body_lower = error.body.lower()
# Check for common access/verification error messages
return any(phrase in body_lower for phrase in [
"verified",
"organization must be",
"does not have access",
"not available",
"not found",
])
OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
# Depth configurations: (min, max) threads to request
# Request MORE than needed since many get filtered by date
DEPTH_CONFIG = {
"quick": (15, 25),
"default": (30, 50),
"deep": (70, 100),
}
REDDIT_SEARCH_PROMPT = """Find Reddit discussion threads about: {topic}
STEP 1: EXTRACT THE CORE SUBJECT
Get the MAIN NOUN/PRODUCT/TOPIC:
- "best nano banana prompting practices" "nano banana"
- "killer features of clawdbot" "clawdbot"
- "top Claude Code skills" "Claude Code"
DO NOT include "best", "top", "tips", "practices", "features" in your search.
STEP 2: SEARCH BROADLY
Search for the core subject:
1. "[core subject] site:reddit.com"
2. "reddit [core subject]"
3. "[core subject] reddit"
Return as many relevant threads as you find. We filter by date server-side.
STEP 3: INCLUDE ALL MATCHES
- Include ALL threads about the core subject
- Set date to "YYYY-MM-DD" if you can determine it, otherwise null
- We verify dates and filter old content server-side
- DO NOT pre-filter aggressively - include anything relevant
REQUIRED: URLs must contain "/r/" AND "/comments/"
REJECT: developers.reddit.com, business.reddit.com
Find {min_items}-{max_items} threads. Return MORE rather than fewer.
Return JSON:
{{
"items": [
{{
"title": "Thread title",
"url": "https://www.reddit.com/r/sub/comments/xyz/title/",
"subreddit": "subreddit_name",
"date": "YYYY-MM-DD or null",
"why_relevant": "Why relevant",
"relevance": 0.85
}}
]
}}"""
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for retry."""
noise = ['best', 'top', 'how to', 'tips for', 'practices', 'features',
'killer', 'guide', 'tutorial', 'recommendations', 'advice',
'prompting', 'using', 'for', 'with', 'the', 'of', 'in', 'on']
words = topic.lower().split()
result = [w for w in words if w not in noise]
return ' '.join(result[:3]) or topic # Keep max 3 words
def _build_subreddit_query(topic: str) -> str:
"""Build a subreddit-targeted search query for fallback.
When standard search returns few results, try searching for the
subreddit itself: 'r/kanye', 'r/howie', etc.
"""
core = _extract_core_subject(topic)
# Remove dots and special chars for subreddit name guess
sub_name = core.replace('.', '').replace(' ', '').lower()
return f"r/{sub_name} site:reddit.com"
def search_reddit(
api_key: str,
model: str,
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
mock_response: Optional[Dict] = None,
_retry: bool = False,
) -> Dict[str, Any]:
"""Search Reddit for relevant threads using OpenAI Responses API.
Args:
api_key: OpenAI API key
model: Model to use
topic: Search topic
from_date: Start date (YYYY-MM-DD) - only include threads after this
to_date: End date (YYYY-MM-DD) - only include threads before this
depth: Research depth - "quick", "default", or "deep"
mock_response: Mock response for testing
Returns:
Raw API response
"""
if mock_response is not None:
return mock_response
min_items, max_items = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
# Adjust timeout based on depth (generous for OpenAI web_search which can be slow)
timeout = 90 if depth == "quick" else 120 if depth == "default" else 180
# Build list of models to try: requested model first, then fallbacks
models_to_try = [model] + [m for m in MODEL_FALLBACK_ORDER if m != model]
# Note: allowed_domains accepts base domain, not subdomains
# We rely on prompt to filter out developers.reddit.com, etc.
input_text = REDDIT_SEARCH_PROMPT.format(
topic=topic,
from_date=from_date,
to_date=to_date,
min_items=min_items,
max_items=max_items,
)
last_error = None
for current_model in models_to_try:
payload = {
"model": current_model,
"tools": [
{
"type": "web_search",
"filters": {
"allowed_domains": ["reddit.com"]
}
}
],
"include": ["web_search_call.action.sources"],
"input": input_text,
}
try:
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
except http.HTTPError as e:
last_error = e
if _is_model_access_error(e):
_log_info(f"Model {current_model} not accessible, trying fallback...")
continue
# Non-access error, don't retry with different model
raise
# All models failed with access errors
if last_error:
_log_error(f"All models failed. Last error: {last_error}")
raise last_error
raise http.HTTPError("No models available")
def search_subreddits(
subreddits: List[str],
topic: str,
from_date: str,
to_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific subreddits via Reddit's free JSON endpoint.
No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint.
Used in Phase 2 supplemental search after entity extraction.
Args:
subreddits: List of subreddit names (without r/)
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
count_per: Results to request per subreddit
Returns:
List of raw item dicts (same format as parse_reddit_response output).
"""
all_items = []
core = _extract_core_subject(topic)
for sub in subreddits:
sub = sub.lstrip("r/")
try:
url = f"https://www.reddit.com/r/{sub}/search/.json"
params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1"
full_url = f"{url}?{params}"
headers = {
"User-Agent": http.USER_AGENT,
"Accept": "application/json",
}
data = http.get(full_url, headers=headers, timeout=15, retries=1)
# Reddit search returns {"data": {"children": [...]}}
children = data.get("data", {}).get("children", [])
for i, child in enumerate(children):
if child.get("kind") != "t3": # t3 = link/submission
continue
post = child.get("data", {})
permalink = post.get("permalink", "")
if not permalink:
continue
item = {
"id": f"RS{len(all_items)+1}",
"title": str(post.get("title", "")).strip(),
"url": f"https://www.reddit.com{permalink}",
"subreddit": str(post.get("subreddit", sub)).strip(),
"date": None,
"why_relevant": f"Found in r/{sub} supplemental search",
"relevance": 0.65, # Slightly lower default for supplemental
}
# Parse date from created_utc
created_utc = post.get("created_utc")
if created_utc:
from . import dates as dates_mod
item["date"] = dates_mod.timestamp_to_date(created_utc)
all_items.append(item)
except http.HTTPError as e:
_log_info(f"Subreddit search failed for r/{sub}: {e}")
if e.status_code == 429:
_log_info("Reddit rate-limited (429) — skipping remaining subreddits")
break
except Exception as e:
_log_info(f"Subreddit search error for r/{sub}: {e}")
return all_items
def _url_encode(text: str) -> str:
"""Simple URL encoding for query parameters."""
import urllib.parse
return urllib.parse.quote_plus(text)
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse OpenAI response to extract Reddit items.
Args:
response: Raw API response
Returns:
List of item dicts
"""
items = []
# Check for API errors first
if "error" in response and response["error"]:
error = response["error"]
err_msg = error.get("message", str(error)) if isinstance(error, dict) else str(error)
_log_error(f"OpenAI API error: {err_msg}")
if http.DEBUG:
_log_error(f"Full error response: {json.dumps(response, indent=2)[:1000]}")
return items
# Try to find the output text
output_text = ""
if "output" in response:
output = response["output"]
if isinstance(output, str):
output_text = output
elif isinstance(output, list):
for item in output:
if isinstance(item, dict):
if item.get("type") == "message":
content = item.get("content", [])
for c in content:
if isinstance(c, dict) and c.get("type") == "output_text":
output_text = c.get("text", "")
break
elif "text" in item:
output_text = item["text"]
elif isinstance(item, str):
output_text = item
if output_text:
break
# Also check for choices (older format)
if not output_text and "choices" in response:
for choice in response["choices"]:
if "message" in choice:
output_text = choice["message"].get("content", "")
break
if not output_text:
print(f"[REDDIT WARNING] No output text found in OpenAI response. Keys present: {list(response.keys())}", flush=True)
return items
# Extract JSON from the response
json_match = re.search(r'\{[\s\S]*"items"[\s\S]*\}', output_text)
if json_match:
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
pass
# Validate and clean items
clean_items = []
for i, item in enumerate(items):
if not isinstance(item, dict):
continue
url = item.get("url", "")
if not url or "reddit.com" not in url:
continue
clean_item = {
"id": f"R{i+1}",
"title": str(item.get("title", "")).strip(),
"url": url,
"subreddit": str(item.get("subreddit", "")).strip().lstrip("r/"),
"date": item.get("date"),
"why_relevant": str(item.get("why_relevant", "")).strip(),
"relevance": min(1.0, max(0.0, float(item.get("relevance", 0.5)))),
}
# Validate date format
if clean_item["date"]:
if not re.match(r'^\d{4}-\d{2}-\d{2}$', str(clean_item["date"])):
clean_item["date"] = None
clean_items.append(clean_item)
return clean_items
-216
View File
@@ -1,216 +0,0 @@
"""Perplexity Sonar Pro web search via OpenRouter for last30days skill.
Uses OpenRouter's chat completions API with Perplexity's Sonar Pro model,
which has built-in web search and returns citations with URLs, titles, and dates.
This is the recommended web search backend -- highest quality results.
API docs: https://openrouter.ai/docs/quickstart
Model: perplexity/sonar-pro
"""
import re
import sys
from typing import Any, Dict, List, Optional
from urllib.parse import urlparse
from . import http
ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
MODEL = "perplexity/sonar-pro"
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Perplexity Sonar Pro on OpenRouter.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: OpenRouter API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
max_tokens = {"quick": 1024, "default": 2048, "deep": 4096}.get(depth, 2048)
prompt = (
f"Find recent blog posts, news articles, tutorials, and discussions "
f"about {topic} published between {from_date} and {to_date}. "
f"Exclude results from reddit.com, x.com, and twitter.com. "
f"For each result, provide the title, URL, publication date, "
f"and a brief summary of why it's relevant."
)
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
}
sys.stderr.write(f"[Web] Searching Sonar Pro via OpenRouter for: {topic}\n")
sys.stderr.flush()
response = http.post(
ENDPOINT,
json_data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"HTTP-Referer": "https://github.com/mvanhorn/last30days-openclaw",
"X-Title": "last30days",
},
timeout=30,
)
return _normalize_results(response)
def _normalize_results(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Sonar Pro response to websearch item schema.
Sonar Pro returns:
- search_results: [{title, url, date}] -- structured source metadata
- citations: [url, ...] -- flat list of cited URLs
- choices[0].message.content -- the synthesized text with [N] references
We prefer search_results (richer metadata), fall back to citations.
"""
items = []
# Try search_results first (has title, url, date)
search_results = response.get("search_results", [])
if isinstance(search_results, list) and search_results:
items = _parse_search_results(search_results)
# Fall back to citations if no search_results
if not items:
citations = response.get("citations", [])
content = _get_content(response)
if isinstance(citations, list) and citations:
items = _parse_citations(citations, content)
sys.stderr.write(f"[Web] Sonar Pro: {len(items)} results\n")
sys.stderr.flush()
return items
def _parse_search_results(results: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Parse the search_results array from Sonar Pro."""
items = []
for i, result in enumerate(results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = str(result.get("title", "")).strip()
if not title:
continue
# Sonar Pro provides dates in search_results
date = result.get("date")
date_confidence = "med" if date else "low"
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": str(result.get("snippet", result.get("description", ""))).strip()[:500],
"date": date,
"date_confidence": date_confidence,
"relevance": 0.7, # Sonar Pro results are generally high quality
"why_relevant": "",
})
return items
def _parse_citations(citations: List[str], content: str) -> List[Dict[str, Any]]:
"""Parse the flat citations array, enriching with content context."""
items = []
for i, url in enumerate(citations):
if not isinstance(url, str) or not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
# Try to extract title from content references like [1] Title...
title = _extract_title_for_citation(content, i + 1) or domain
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": "",
"date": None,
"date_confidence": "low",
"relevance": 0.6,
"why_relevant": "",
})
return items
def _get_content(response: Dict[str, Any]) -> str:
"""Extract the text content from the chat completion response."""
try:
return response["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError):
return ""
def _extract_title_for_citation(content: str, index: int) -> Optional[str]:
"""Try to extract a title near a citation reference [N] in the content."""
if not content:
return None
# Look for patterns like [1] Title or [1](url) Title
pattern = rf'\[{index}\][)\s]*([^\[\n]{{5,80}})'
match = re.search(pattern, content)
if match:
title = match.group(1).strip().rstrip('.')
# Clean up markdown artifacts
title = re.sub(r'[*_`]', '', title)
return title if len(title) > 3 else None
return None
-139
View File
@@ -1,139 +0,0 @@
"""Parallel AI web search for last30days skill.
Uses the Parallel AI Search API to find web content (blogs, docs, news, tutorials).
This is the preferred web search backend -- it returns LLM-optimized results
with extended excerpts ranked by relevance.
API docs: https://docs.parallel.ai/search-api/search-quickstart
"""
import json
import sys
from typing import Any, Dict, List, Optional
from urllib.parse import urlparse
from . import http
ENDPOINT = "https://api.parallel.ai/v1beta/search"
# Domains to exclude (handled by Reddit/X search)
EXCLUDED_DOMAINS = {
"reddit.com", "www.reddit.com", "old.reddit.com",
"twitter.com", "www.twitter.com", "x.com", "www.x.com",
}
def search_web(
topic: str,
from_date: str,
to_date: str,
api_key: str,
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Search the web via Parallel AI Search API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
api_key: Parallel AI API key
depth: 'quick', 'default', or 'deep'
Returns:
List of result dicts with keys: url, title, snippet, source_domain, date, relevance
Raises:
http.HTTPError: On API errors
"""
max_results = {"quick": 8, "default": 15, "deep": 25}.get(depth, 15)
payload = {
"objective": (
f"Find recent blog posts, tutorials, news articles, and discussions "
f"about {topic} from {from_date} to {to_date}. "
f"Exclude reddit.com, x.com, and twitter.com."
),
"max_results": max_results,
"max_chars_per_result": 500,
}
sys.stderr.write(f"[Web] Searching Parallel AI for: {topic}\n")
sys.stderr.flush()
response = http.post(
ENDPOINT,
json_data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"parallel-beta": "search-extract-2025-10-10",
},
timeout=30,
)
return _normalize_results(response)
def _normalize_results(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Convert Parallel AI response to websearch item schema.
Args:
response: Raw API response
Returns:
List of normalized result dicts
"""
items = []
# Handle different response shapes
results = response.get("results", [])
if not isinstance(results, list):
return items
for i, result in enumerate(results):
if not isinstance(result, dict):
continue
url = result.get("url", "")
if not url:
continue
# Skip excluded domains
try:
domain = urlparse(url).netloc.lower()
if domain in EXCLUDED_DOMAINS:
continue
# Clean domain for display
if domain.startswith("www."):
domain = domain[4:]
except Exception:
domain = ""
title = str(result.get("title", "")).strip()
snippet = str(result.get("excerpt", result.get("snippet", result.get("description", "")))).strip()
if not title and not snippet:
continue
# Extract relevance score if provided
relevance = result.get("relevance_score", result.get("relevance", 0.6))
try:
relevance = min(1.0, max(0.0, float(relevance)))
except (TypeError, ValueError):
relevance = 0.6
items.append({
"id": f"W{i+1}",
"title": title[:200],
"url": url,
"source_domain": domain,
"snippet": snippet[:500],
"date": result.get("published_date", result.get("date")),
"date_confidence": "med" if result.get("published_date") or result.get("date") else "low",
"relevance": relevance,
"why_relevant": str(result.get("summary", "")).strip()[:200],
})
sys.stderr.write(f"[Web] Parallel AI: {len(items)} results\n")
sys.stderr.flush()
return items
-491
View File
@@ -1,491 +0,0 @@
"""Output rendering for last30days skill."""
import json
import os
import tempfile
from pathlib import Path
from typing import List, Optional
from . import schema
OUTPUT_DIR = Path.home() / ".local" / "share" / "last30days" / "out"
def ensure_output_dir():
"""Ensure output directory exists. Supports env override and sandbox fallback."""
global OUTPUT_DIR
env_dir = os.environ.get("LAST30DAYS_OUTPUT_DIR")
if env_dir:
OUTPUT_DIR = Path(env_dir)
try:
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
except PermissionError:
OUTPUT_DIR = Path(tempfile.gettempdir()) / "last30days" / "out"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
def _assess_data_freshness(report: schema.Report) -> dict:
"""Assess how much data is actually from the last 30 days."""
reddit_recent = sum(1 for r in report.reddit if r.date and r.date >= report.range_from)
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
total_recent = reddit_recent + x_recent + web_recent
total_items = len(report.reddit) + len(report.x) + len(report.web)
return {
"reddit_recent": reddit_recent,
"x_recent": x_recent,
"web_recent": web_recent,
"total_recent": total_recent,
"total_items": total_items,
"is_sparse": total_recent < 5,
"mostly_evergreen": total_items > 0 and total_recent < total_items * 0.3,
}
def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "none") -> str:
"""Render compact output for the assistant to synthesize.
Args:
report: Report data
limit: Max items per source
missing_keys: 'both', 'reddit', 'x', or 'none'
Returns:
Compact markdown string
"""
lines = []
# Header
lines.append(f"## Research Results: {report.topic}")
lines.append("")
# Assess data freshness and add honesty warning if needed
freshness = _assess_data_freshness(report)
if freshness["is_sparse"]:
lines.append("**⚠️ LIMITED RECENT DATA** - Few discussions from the last 30 days.")
lines.append(f"Only {freshness['total_recent']} item(s) confirmed from {report.range_from} to {report.range_to}.")
lines.append("Results below may include older/evergreen content. Be transparent with the user about this.")
lines.append("")
# Web-only mode banner (when no API keys)
if report.mode == "web-only":
lines.append("**🌐 WEB SEARCH MODE** - assistant will search blogs, docs & news")
lines.append("")
lines.append("---")
lines.append("**⚡ Want better results?** Add API keys to unlock Reddit & X data:")
lines.append("- `OPENAI_API_KEY` → Reddit threads with real upvotes & comments")
lines.append("- `XAI_API_KEY` → X posts with real likes & reposts")
lines.append("- Edit `~/.config/last30days/.env` to add keys")
lines.append("---")
lines.append("")
# Cache indicator
if report.from_cache:
age_str = f"{report.cache_age_hours:.1f}h old" if report.cache_age_hours else "cached"
lines.append(f"**⚡ CACHED RESULTS** ({age_str}) - use `--refresh` for fresh data")
lines.append("")
lines.append(f"**Date Range:** {report.range_from} to {report.range_to}")
lines.append(f"**Mode:** {report.mode}")
if report.openai_model_used:
lines.append(f"**OpenAI Model:** {report.openai_model_used}")
if report.xai_model_used:
lines.append(f"**xAI Model:** {report.xai_model_used}")
lines.append("")
# Coverage note for partial coverage
if report.mode == "reddit-only" and missing_keys == "x":
lines.append("*💡 Tip: Add XAI_API_KEY for X/Twitter data and better triangulation.*")
lines.append("")
elif report.mode == "x-only" and missing_keys == "reddit":
lines.append("*💡 Tip: Add OPENAI_API_KEY for Reddit data and better triangulation.*")
lines.append("")
# Reddit items
if report.reddit_error:
lines.append("### Reddit Threads")
lines.append("")
lines.append(f"**ERROR:** {report.reddit_error}")
lines.append("")
elif report.mode in ("both", "reddit-only") and not report.reddit:
lines.append("### Reddit Threads")
lines.append("")
lines.append("*No relevant Reddit threads found for this topic.*")
lines.append("")
elif report.reddit:
lines.append("### Reddit Threads")
lines.append("")
for item in report.reddit[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.score is not None:
parts.append(f"{eng.score}pts")
if eng.num_comments is not None:
parts.append(f"{eng.num_comments}cmt")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else " (date unknown)"
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
lines.append(f"**{item.id}** (score:{item.score}) r/{item.subreddit}{date_str}{conf_str}{eng_str}")
lines.append(f" {item.title}")
lines.append(f" {item.url}")
lines.append(f" *{item.why_relevant}*")
# Top comment insights
if item.comment_insights:
lines.append(f" Insights:")
for insight in item.comment_insights[:3]:
lines.append(f" - {insight}")
lines.append("")
# X items
if report.x_error:
lines.append("### X Posts")
lines.append("")
lines.append(f"**ERROR:** {report.x_error}")
lines.append("")
elif report.mode in ("both", "x-only", "all", "x-web") and not report.x:
lines.append("### X Posts")
lines.append("")
lines.append("*No relevant X posts found for this topic.*")
lines.append("")
elif report.x:
lines.append("### X Posts")
lines.append("")
for item in report.x[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.likes is not None:
parts.append(f"{eng.likes}likes")
if eng.reposts is not None:
parts.append(f"{eng.reposts}rt")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else " (date unknown)"
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{conf_str}{eng_str}")
lines.append(f" {item.text[:200]}...")
lines.append(f" {item.url}")
lines.append(f" *{item.why_relevant}*")
lines.append("")
# YouTube items
if report.youtube_error:
lines.append("### YouTube Videos")
lines.append("")
lines.append(f"**ERROR:** {report.youtube_error}")
lines.append("")
elif report.youtube:
lines.append("### YouTube Videos")
lines.append("")
for item in report.youtube[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.views is not None:
parts.append(f"{eng.views:,} views")
if eng.likes is not None:
parts.append(f"{eng.likes:,} likes")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else ""
lines.append(f"**{item.id}** (score:{item.score}) {item.channel_name}{date_str}{eng_str}")
lines.append(f" {item.title}")
lines.append(f" {item.url}")
if item.transcript_snippet:
snippet = item.transcript_snippet[:200]
if len(item.transcript_snippet) > 200:
snippet += "..."
lines.append(f" Transcript: {snippet}")
lines.append(f" *{item.why_relevant}*")
lines.append("")
# Web items (if any - populated by the assistant)
if report.web_error:
lines.append("### Web Results")
lines.append("")
lines.append(f"**ERROR:** {report.web_error}")
lines.append("")
elif report.web:
lines.append("### Web Results")
lines.append("")
for item in report.web[:limit]:
date_str = f" ({item.date})" if item.date else " (date unknown)"
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
lines.append(f"**{item.id}** [WEB] (score:{item.score}) {item.source_domain}{date_str}{conf_str}")
lines.append(f" {item.title}")
lines.append(f" {item.url}")
lines.append(f" {item.snippet[:150]}...")
lines.append(f" *{item.why_relevant}*")
lines.append("")
return "\n".join(lines)
def render_source_status(report: schema.Report, source_info: dict = None) -> str:
"""Render source status footer showing what was used/skipped and why.
Args:
report: Report data
source_info: Dict with source availability info:
x_skip_reason, youtube_skip_reason, web_skip_reason
Returns:
Source status markdown string
"""
if source_info is None:
source_info = {}
lines = []
lines.append("---")
lines.append("**Sources:**")
# Reddit
if report.reddit_error:
lines.append(f" ❌ Reddit: error — {report.reddit_error}")
elif report.reddit:
lines.append(f" ✅ Reddit: {len(report.reddit)} threads")
elif report.mode in ("both", "reddit-only", "all", "reddit-web"):
lines.append(" ⚠️ Reddit: 0 threads found")
else:
reason = source_info.get("reddit_skip_reason", "not configured")
lines.append(f" ⏭️ Reddit: skipped — {reason}")
# X
if report.x_error:
lines.append(f" ❌ X: error — {report.x_error}")
elif report.x:
lines.append(f" ✅ X: {len(report.x)} posts")
elif report.mode in ("both", "x-only", "all", "x-web"):
lines.append(" ⚠️ X: 0 posts found")
else:
reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY")
lines.append(f" ⏭️ X: skipped — {reason}")
# YouTube
if report.youtube_error:
lines.append(f" ❌ YouTube: error — {report.youtube_error}")
elif report.youtube:
with_transcripts = sum(1 for v in report.youtube if getattr(v, 'transcript_snippet', None))
lines.append(f" ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)")
else:
reason = source_info.get("youtube_skip_reason", "yt-dlp not installed (brew install yt-dlp)")
lines.append(f" ⏭️ YouTube: skipped — {reason}")
# Web
if report.web_error:
lines.append(f" ❌ Web: error — {report.web_error}")
elif report.web:
lines.append(f" ✅ Web: {len(report.web)} pages")
else:
reason = source_info.get("web_skip_reason", "assistant will use WebSearch")
lines.append(f" ⚡ Web: {reason}")
lines.append("")
return "\n".join(lines)
def render_context_snippet(report: schema.Report) -> str:
"""Render reusable context snippet.
Args:
report: Report data
Returns:
Context markdown string
"""
lines = []
lines.append(f"# Context: {report.topic} (Last 30 Days)")
lines.append("")
lines.append(f"*Generated: {report.generated_at[:10]} | Sources: {report.mode}*")
lines.append("")
# Key sources summary
lines.append("## Key Sources")
lines.append("")
all_items = []
for item in report.reddit[:5]:
all_items.append((item.score, "Reddit", item.title, item.url))
for item in report.x[:5]:
all_items.append((item.score, "X", item.text[:50] + "...", item.url))
for item in report.web[:5]:
all_items.append((item.score, "Web", item.title[:50] + "...", item.url))
all_items.sort(key=lambda x: -x[0])
for score, source, text, url in all_items[:7]:
lines.append(f"- [{source}] {text}")
lines.append("")
lines.append("## Summary")
lines.append("")
lines.append("*See full report for best practices, prompt pack, and detailed sources.*")
lines.append("")
return "\n".join(lines)
def render_full_report(report: schema.Report) -> str:
"""Render full markdown report.
Args:
report: Report data
Returns:
Full report markdown
"""
lines = []
# Title
lines.append(f"# {report.topic} - Last 30 Days Research Report")
lines.append("")
lines.append(f"**Generated:** {report.generated_at}")
lines.append(f"**Date Range:** {report.range_from} to {report.range_to}")
lines.append(f"**Mode:** {report.mode}")
lines.append("")
# Models
lines.append("## Models Used")
lines.append("")
if report.openai_model_used:
lines.append(f"- **OpenAI:** {report.openai_model_used}")
if report.xai_model_used:
lines.append(f"- **xAI:** {report.xai_model_used}")
lines.append("")
# Reddit section
if report.reddit:
lines.append("## Reddit Threads")
lines.append("")
for item in report.reddit:
lines.append(f"### {item.id}: {item.title}")
lines.append("")
lines.append(f"- **Subreddit:** r/{item.subreddit}")
lines.append(f"- **URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
if item.engagement:
eng = item.engagement
lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
if item.comment_insights:
lines.append("")
lines.append("**Key Insights from Comments:**")
for insight in item.comment_insights:
lines.append(f"- {insight}")
lines.append("")
# X section
if report.x:
lines.append("## X Posts")
lines.append("")
for item in report.x:
lines.append(f"### {item.id}: @{item.author_handle}")
lines.append("")
lines.append(f"- **URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
if item.engagement:
eng = item.engagement
lines.append(f"- **Engagement:** {eng.likes or '?'} likes, {eng.reposts or '?'} reposts")
lines.append("")
lines.append(f"> {item.text}")
lines.append("")
# Web section
if report.web:
lines.append("## Web Results")
lines.append("")
for item in report.web:
lines.append(f"### {item.id}: {item.title}")
lines.append("")
lines.append(f"- **Source:** {item.source_domain}")
lines.append(f"- **URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
lines.append("")
lines.append(f"> {item.snippet}")
lines.append("")
# Placeholders for assistant synthesis
lines.append("## Best Practices")
lines.append("")
lines.append("*To be synthesized by assistant*")
lines.append("")
lines.append("## Prompt Pack")
lines.append("")
lines.append("*To be synthesized by assistant*")
lines.append("")
return "\n".join(lines)
def write_outputs(
report: schema.Report,
raw_openai: Optional[dict] = None,
raw_xai: Optional[dict] = None,
raw_reddit_enriched: Optional[list] = None,
):
"""Write all output files.
Args:
report: Report data
raw_openai: Raw OpenAI API response
raw_xai: Raw xAI API response
raw_reddit_enriched: Raw enriched Reddit thread data
"""
ensure_output_dir()
# report.json
with open(OUTPUT_DIR / "report.json", 'w') as f:
json.dump(report.to_dict(), f, indent=2)
# report.md
with open(OUTPUT_DIR / "report.md", 'w') as f:
f.write(render_full_report(report))
# last30days.context.md
with open(OUTPUT_DIR / "last30days.context.md", 'w') as f:
f.write(render_context_snippet(report))
# Raw responses
if raw_openai:
with open(OUTPUT_DIR / "raw_openai.json", 'w') as f:
json.dump(raw_openai, f, indent=2)
if raw_xai:
with open(OUTPUT_DIR / "raw_xai.json", 'w') as f:
json.dump(raw_xai, f, indent=2)
if raw_reddit_enriched:
with open(OUTPUT_DIR / "raw_reddit_threads_enriched.json", 'w') as f:
json.dump(raw_reddit_enriched, f, indent=2)
def get_context_path() -> str:
"""Get path to context file."""
return str(OUTPUT_DIR / "last30days.context.md")
-403
View File
@@ -1,403 +0,0 @@
"""Data schemas for last30days skill."""
from dataclasses import dataclass, field, asdict
from typing import Any, Dict, List, Optional
from datetime import datetime, timezone
@dataclass
class Engagement:
"""Engagement metrics."""
# Reddit fields
score: Optional[int] = None
num_comments: Optional[int] = None
upvote_ratio: Optional[float] = None
# X fields
likes: Optional[int] = None
reposts: Optional[int] = None
replies: Optional[int] = None
quotes: Optional[int] = None
# YouTube fields
views: Optional[int] = None
def to_dict(self) -> Dict[str, Any]:
d = {}
if self.score is not None:
d['score'] = self.score
if self.num_comments is not None:
d['num_comments'] = self.num_comments
if self.upvote_ratio is not None:
d['upvote_ratio'] = self.upvote_ratio
if self.likes is not None:
d['likes'] = self.likes
if self.reposts is not None:
d['reposts'] = self.reposts
if self.replies is not None:
d['replies'] = self.replies
if self.quotes is not None:
d['quotes'] = self.quotes
if self.views is not None:
d['views'] = self.views
return d if d else None
@dataclass
class Comment:
"""Reddit comment."""
score: int
date: Optional[str]
author: str
excerpt: str
url: str
def to_dict(self) -> Dict[str, Any]:
return {
'score': self.score,
'date': self.date,
'author': self.author,
'excerpt': self.excerpt,
'url': self.url,
}
@dataclass
class SubScores:
"""Component scores."""
relevance: int = 0
recency: int = 0
engagement: int = 0
def to_dict(self) -> Dict[str, int]:
return {
'relevance': self.relevance,
'recency': self.recency,
'engagement': self.engagement,
}
@dataclass
class RedditItem:
"""Normalized Reddit item."""
id: str
title: str
url: str
subreddit: str
date: Optional[str] = None
date_confidence: str = "low"
engagement: Optional[Engagement] = None
top_comments: List[Comment] = field(default_factory=list)
comment_insights: List[str] = field(default_factory=list)
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'subreddit': self.subreddit,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'top_comments': [c.to_dict() for c in self.top_comments],
'comment_insights': self.comment_insights,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass
class XItem:
"""Normalized X item."""
id: str
text: str
url: str
author_handle: str
date: Optional[str] = None
date_confidence: str = "low"
engagement: Optional[Engagement] = None
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'text': self.text,
'url': self.url,
'author_handle': self.author_handle,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass
class WebSearchItem:
"""Normalized web search item (no engagement metrics)."""
id: str
title: str
url: str
source_domain: str # e.g., "medium.com", "github.com"
snippet: str
date: Optional[str] = None
date_confidence: str = "low"
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'source_domain': self.source_domain,
'snippet': self.snippet,
'date': self.date,
'date_confidence': self.date_confidence,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass
class YouTubeItem:
"""Normalized YouTube item."""
id: str # video_id
title: str
url: str
channel_name: str
date: Optional[str] = None
date_confidence: str = "high" # YouTube dates are always reliable
engagement: Optional[Engagement] = None
transcript_snippet: str = ""
relevance: float = 0.7
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'channel_name': self.channel_name,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'transcript_snippet': self.transcript_snippet,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass
class Report:
"""Full research report."""
topic: str
range_from: str
range_to: str
generated_at: str
mode: str # 'reddit-only', 'x-only', 'both', 'web-only', etc.
openai_model_used: Optional[str] = None
xai_model_used: Optional[str] = None
reddit: List[RedditItem] = field(default_factory=list)
x: List[XItem] = field(default_factory=list)
web: List[WebSearchItem] = field(default_factory=list)
youtube: List[YouTubeItem] = field(default_factory=list)
best_practices: List[str] = field(default_factory=list)
prompt_pack: List[str] = field(default_factory=list)
context_snippet_md: str = ""
# Status tracking
reddit_error: Optional[str] = None
x_error: Optional[str] = None
web_error: Optional[str] = None
youtube_error: Optional[str] = None
# Cache info
from_cache: bool = False
cache_age_hours: Optional[float] = None
def to_dict(self) -> Dict[str, Any]:
d = {
'topic': self.topic,
'range': {
'from': self.range_from,
'to': self.range_to,
},
'generated_at': self.generated_at,
'mode': self.mode,
'openai_model_used': self.openai_model_used,
'xai_model_used': self.xai_model_used,
'reddit': [r.to_dict() for r in self.reddit],
'x': [x.to_dict() for x in self.x],
'web': [w.to_dict() for w in self.web],
'youtube': [y.to_dict() for y in self.youtube],
'best_practices': self.best_practices,
'prompt_pack': self.prompt_pack,
'context_snippet_md': self.context_snippet_md,
}
if self.reddit_error:
d['reddit_error'] = self.reddit_error
if self.x_error:
d['x_error'] = self.x_error
if self.web_error:
d['web_error'] = self.web_error
if self.youtube_error:
d['youtube_error'] = self.youtube_error
if self.from_cache:
d['from_cache'] = self.from_cache
if self.cache_age_hours is not None:
d['cache_age_hours'] = self.cache_age_hours
return d
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "Report":
"""Create Report from serialized dict (handles cache format)."""
# Handle range field conversion
range_data = data.get('range', {})
range_from = range_data.get('from', data.get('range_from', ''))
range_to = range_data.get('to', data.get('range_to', ''))
# Reconstruct Reddit items
reddit_items = []
for r in data.get('reddit', []):
eng = None
if r.get('engagement'):
eng = Engagement(**r['engagement'])
comments = [Comment(**c) for c in r.get('top_comments', [])]
subs = SubScores(**r.get('subs', {})) if r.get('subs') else SubScores()
reddit_items.append(RedditItem(
id=r['id'],
title=r['title'],
url=r['url'],
subreddit=r['subreddit'],
date=r.get('date'),
date_confidence=r.get('date_confidence', 'low'),
engagement=eng,
top_comments=comments,
comment_insights=r.get('comment_insights', []),
relevance=r.get('relevance', 0.5),
why_relevant=r.get('why_relevant', ''),
subs=subs,
score=r.get('score', 0),
))
# Reconstruct X items
x_items = []
for x in data.get('x', []):
eng = None
if x.get('engagement'):
eng = Engagement(**x['engagement'])
subs = SubScores(**x.get('subs', {})) if x.get('subs') else SubScores()
x_items.append(XItem(
id=x['id'],
text=x['text'],
url=x['url'],
author_handle=x['author_handle'],
date=x.get('date'),
date_confidence=x.get('date_confidence', 'low'),
engagement=eng,
relevance=x.get('relevance', 0.5),
why_relevant=x.get('why_relevant', ''),
subs=subs,
score=x.get('score', 0),
))
# Reconstruct Web items
web_items = []
for w in data.get('web', []):
subs = SubScores(**w.get('subs', {})) if w.get('subs') else SubScores()
web_items.append(WebSearchItem(
id=w['id'],
title=w['title'],
url=w['url'],
source_domain=w.get('source_domain', ''),
snippet=w.get('snippet', ''),
date=w.get('date'),
date_confidence=w.get('date_confidence', 'low'),
relevance=w.get('relevance', 0.5),
why_relevant=w.get('why_relevant', ''),
subs=subs,
score=w.get('score', 0),
))
# Reconstruct YouTube items
youtube_items = []
for y in data.get('youtube', []):
eng = None
if y.get('engagement'):
eng = Engagement(**y['engagement'])
subs = SubScores(**y.get('subs', {})) if y.get('subs') else SubScores()
youtube_items.append(YouTubeItem(
id=y['id'],
title=y['title'],
url=y['url'],
channel_name=y.get('channel_name', ''),
date=y.get('date'),
date_confidence=y.get('date_confidence', 'high'),
engagement=eng,
transcript_snippet=y.get('transcript_snippet', ''),
relevance=y.get('relevance', 0.7),
why_relevant=y.get('why_relevant', ''),
subs=subs,
score=y.get('score', 0),
))
return cls(
topic=data['topic'],
range_from=range_from,
range_to=range_to,
generated_at=data['generated_at'],
mode=data['mode'],
openai_model_used=data.get('openai_model_used'),
xai_model_used=data.get('xai_model_used'),
reddit=reddit_items,
x=x_items,
web=web_items,
youtube=youtube_items,
best_practices=data.get('best_practices', []),
prompt_pack=data.get('prompt_pack', []),
context_snippet_md=data.get('context_snippet_md', ''),
reddit_error=data.get('reddit_error'),
x_error=data.get('x_error'),
web_error=data.get('web_error'),
youtube_error=data.get('youtube_error'),
from_cache=data.get('from_cache', False),
cache_age_hours=data.get('cache_age_hours'),
)
def create_report(
topic: str,
from_date: str,
to_date: str,
mode: str,
openai_model: Optional[str] = None,
xai_model: Optional[str] = None,
) -> Report:
"""Create a new report with metadata."""
return Report(
topic=topic,
range_from=from_date,
range_to=to_date,
generated_at=datetime.now(timezone.utc).isoformat(),
mode=mode,
openai_model_used=openai_model,
xai_model_used=xai_model,
)
-372
View File
@@ -1,372 +0,0 @@
"""Popularity-aware scoring for last30days skill."""
import math
from typing import List, Optional, Union
from . import dates, schema
# Score weights for Reddit/X (has engagement)
WEIGHT_RELEVANCE = 0.45
WEIGHT_RECENCY = 0.25
WEIGHT_ENGAGEMENT = 0.30
# WebSearch weights (no engagement, reweighted to 100%)
WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# WebSearch date confidence adjustments
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date (high confidence)
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date signals (low confidence)
# Default engagement score for unknown
DEFAULT_ENGAGEMENT = 35
UNKNOWN_ENGAGEMENT_PENALTY = 3
def log1p_safe(x: Optional[int]) -> float:
"""Safe log1p that handles None and negative values."""
if x is None or x < 0:
return 0.0
return math.log1p(x)
def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for Reddit item.
Formula: 0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)
"""
if engagement is None:
return None
if engagement.score is None and engagement.num_comments is None:
return None
score = log1p_safe(engagement.score)
comments = log1p_safe(engagement.num_comments)
ratio = (engagement.upvote_ratio or 0.5) * 10
return 0.55 * score + 0.40 * comments + 0.05 * ratio
def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for X item.
Formula: 0.55*log1p(likes) + 0.25*log1p(reposts) + 0.15*log1p(replies) + 0.05*log1p(quotes)
"""
if engagement is None:
return None
if engagement.likes is None and engagement.reposts is None:
return None
likes = log1p_safe(engagement.likes)
reposts = log1p_safe(engagement.reposts)
replies = log1p_safe(engagement.replies)
quotes = log1p_safe(engagement.quotes)
return 0.55 * likes + 0.25 * reposts + 0.15 * replies + 0.05 * quotes
def normalize_to_100(values: List[float], default: float = 50) -> List[float]:
"""Normalize a list of values to 0-100 scale.
Args:
values: Raw values (None values are preserved)
default: Default value for None entries
Returns:
Normalized values
"""
# Filter out None
valid = [v for v in values if v is not None]
if not valid:
return [default if v is None else 50 for v in values]
min_val = min(valid)
max_val = max(valid)
range_val = max_val - min_val
if range_val == 0:
return [50 if v is None else 50 for v in values]
result = []
for v in values:
if v is None:
result.append(None)
else:
normalized = ((v - min_val) / range_val) * 100
result.append(normalized)
return result
def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem]:
"""Compute scores for Reddit items.
Args:
items: List of Reddit items
Returns:
Items with updated scores
"""
if not items:
return items
# Compute raw engagement scores
eng_raw = [compute_reddit_engagement_raw(item.engagement) for item in items]
# Normalize engagement to 0-100
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
# Relevance subscore (model-provided, convert to 0-100)
rel_score = int(item.relevance * 100)
# Recency subscore
rec_score = dates.recency_score(item.date)
# Engagement subscore
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
# Store subscores
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
# Compute overall score
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
# Apply penalty for unknown engagement
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
# Apply penalty for low date confidence
if item.date_confidence == "low":
overall -= 5
elif item.date_confidence == "med":
overall -= 2
item.score = max(0, min(100, int(overall)))
return items
def score_x_items(items: List[schema.XItem]) -> List[schema.XItem]:
"""Compute scores for X items.
Args:
items: List of X items
Returns:
Items with updated scores
"""
if not items:
return items
# Compute raw engagement scores
eng_raw = [compute_x_engagement_raw(item.engagement) for item in items]
# Normalize engagement to 0-100
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
# Relevance subscore (model-provided, convert to 0-100)
rel_score = int(item.relevance * 100)
# Recency subscore
rec_score = dates.recency_score(item.date)
# Engagement subscore
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
# Store subscores
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
# Compute overall score
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
# Apply penalty for unknown engagement
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
# Apply penalty for low date confidence
if item.date_confidence == "low":
overall -= 5
elif item.date_confidence == "med":
overall -= 2
item.score = max(0, min(100, int(overall)))
return items
def compute_youtube_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for YouTube item.
Formula: 0.50*log1p(views) + 0.35*log1p(likes) + 0.15*log1p(comments)
Views dominate on YouTube they're the primary discovery signal.
"""
if engagement is None:
return None
if engagement.views is None and engagement.likes is None:
return None
views = log1p_safe(engagement.views)
likes = log1p_safe(engagement.likes)
comments = log1p_safe(engagement.num_comments)
return 0.50 * views + 0.35 * likes + 0.15 * comments
def score_youtube_items(items: List[schema.YouTubeItem]) -> List[schema.YouTubeItem]:
"""Compute scores for YouTube items.
Uses same weight structure as Reddit/X (relevance + recency + engagement).
"""
if not items:
return items
eng_raw = [compute_youtube_engagement_raw(item.engagement) for item in items]
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
item.score = max(0, min(100, int(overall)))
return items
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
"""Compute scores for WebSearch items WITHOUT engagement metrics.
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
This ensures WebSearch items rank below comparable Reddit/X items.
Date confidence adjustments:
- High confidence (URL-verified date): +10 bonus
- Med confidence (snippet-extracted date): no change
- Low confidence (no date signals): -20 penalty
Args:
items: List of WebSearch items
Returns:
Items with updated scores
"""
if not items:
return items
for item in items:
# Relevance subscore (model-provided, convert to 0-100)
rel_score = int(item.relevance * 100)
# Recency subscore
rec_score = dates.recency_score(item.date)
# Store subscores (engagement is 0 for WebSearch - no data)
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=0, # Explicitly zero - no engagement data available
)
# Compute overall score using WebSearch weights
overall = (
WEBSEARCH_WEIGHT_RELEVANCE * rel_score +
WEBSEARCH_WEIGHT_RECENCY * rec_score
)
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
overall -= WEBSEARCH_SOURCE_PENALTY
# Apply date confidence adjustments
# High confidence (URL-verified): reward with bonus
# Med confidence (snippet-extracted): neutral
# Low confidence (no date signals): heavy penalty
if item.date_confidence == "high":
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified recent dates
elif item.date_confidence == "low":
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
item.score = max(0, min(100, int(overall)))
return items
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem]]) -> List:
"""Sort items by score (descending), then date, then source priority.
Args:
items: List of items to sort
Returns:
Sorted items
"""
def sort_key(item):
# Primary: score descending (negate for descending)
score = -item.score
# Secondary: date descending (recent first)
date = item.date or "0000-00-00"
date_key = -int(date.replace("-", ""))
# Tertiary: source priority (Reddit > X > YouTube > WebSearch)
if isinstance(item, schema.RedditItem):
source_priority = 0
elif isinstance(item, schema.XItem):
source_priority = 1
elif isinstance(item, schema.YouTubeItem):
source_priority = 2
else: # WebSearchItem
source_priority = 3
# Quaternary: title/text for stability
text = getattr(item, "title", "") or getattr(item, "text", "")
return (score, date_key, source_priority, text)
return sorted(items, key=sort_key)
-134
View File
@@ -1,134 +0,0 @@
#!/usr/bin/env node
/**
* bird-search.mjs - Vendored Bird CLI search wrapper for /last30days.
* Subset of @steipete/bird v0.8.0 (MIT License, Peter Steinberger).
*
* Usage:
* node bird-search.mjs <query> [--count N] [--json]
* node bird-search.mjs --whoami
* node bird-search.mjs --check
*/
import { resolveCredentials } from './lib/cookies.js';
import { TwitterClientBase } from './lib/twitter-client-base.js';
import { withSearch } from './lib/twitter-client-search.js';
// Build a search-only client (no posting, bookmarks, etc.)
const SearchClient = withSearch(TwitterClientBase);
const args = process.argv.slice(2);
// --check: verify that credentials can be resolved
if (args.includes('--check')) {
try {
const { cookies, warnings } = await resolveCredentials({});
if (cookies.authToken && cookies.ct0) {
process.stdout.write(JSON.stringify({ authenticated: true, source: cookies.source }));
process.exit(0);
} else {
process.stdout.write(JSON.stringify({ authenticated: false, warnings }));
process.exit(1);
}
} catch (err) {
process.stdout.write(JSON.stringify({ authenticated: false, error: err.message }));
process.exit(1);
}
}
// --whoami: check auth and output source
if (args.includes('--whoami')) {
try {
const { cookies } = await resolveCredentials({});
if (cookies.authToken && cookies.ct0) {
process.stdout.write(cookies.source || 'authenticated');
process.exit(0);
} else {
process.stderr.write('Not authenticated\n');
process.exit(1);
}
} catch (err) {
process.stderr.write(`Auth check failed: ${err.message}\n`);
process.exit(1);
}
}
// Parse search args
let query = null;
let count = 20;
let jsonOutput = false;
for (let i = 0; i < args.length; i++) {
if (args[i] === '--count' && args[i + 1]) {
count = parseInt(args[i + 1], 10);
i++;
} else if (args[i] === '-n' && args[i + 1]) {
count = parseInt(args[i + 1], 10);
i++;
} else if (args[i] === '--json') {
jsonOutput = true;
} else if (!args[i].startsWith('-')) {
query = args[i];
}
}
if (!query) {
process.stderr.write('Usage: node bird-search.mjs <query> [--count N] [--json]\n');
process.exit(1);
}
try {
// Resolve credentials (env vars, then browser cookies)
const { cookies, warnings } = await resolveCredentials({});
if (!cookies.authToken || !cookies.ct0) {
const msg = warnings.length > 0 ? warnings.join('; ') : 'No Twitter credentials found';
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: msg, items: [] }));
} else {
process.stderr.write(`Error: ${msg}\n`);
}
process.exit(1);
}
// Create search client
const client = new SearchClient({
cookies: {
authToken: cookies.authToken,
ct0: cookies.ct0,
cookieHeader: cookies.cookieHeader,
},
timeoutMs: 30000,
});
// Run search
const result = await client.search(query, count);
if (!result.success) {
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: result.error, items: [] }));
} else {
process.stderr.write(`Search failed: ${result.error}\n`);
}
process.exit(1);
}
// Output results
const tweets = result.tweets || [];
if (jsonOutput) {
process.stdout.write(JSON.stringify(tweets));
} else {
for (const tweet of tweets) {
const author = tweet.author?.username || 'unknown';
process.stdout.write(`@${author}: ${tweet.text?.slice(0, 200)}\n\n`);
}
}
process.exit(0);
} catch (err) {
if (jsonOutput) {
process.stdout.write(JSON.stringify({ error: err.message, items: [] }));
} else {
process.stderr.write(`Error: ${err.message}\n`);
}
process.exit(1);
}
@@ -1,29 +0,0 @@
# @steipete/sweet-cookie
Inline-first browser cookie extraction for local tooling (no native addons).
Supports:
- Inline payloads (JSON / base64 / file) — most reliable path.
- Local browser reads (best effort): Chrome, Edge, Firefox, Safari (macOS).
Install:
```bash
npm i @steipete/sweet-cookie
```
Usage:
```ts
import { getCookies, toCookieHeader } from '@steipete/sweet-cookie';
const { cookies, warnings } = await getCookies({
url: 'https://example.com/',
names: ['session', 'csrf'],
browsers: ['chrome', 'edge', 'firefox', 'safari'],
});
for (const w of warnings) console.warn(w);
const cookieHeader = toCookieHeader(cookies, { dedupeByName: true });
```
Docs + extension exporter: see the repo root README.
@@ -1,3 +0,0 @@
export { getCookies, toCookieHeader } from './public.js';
export type { BrowserName, Cookie, CookieHeaderOptions, CookieMode, CookieSameSite, GetCookiesOptions, GetCookiesResult, } from './types.js';
//# sourceMappingURL=index.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AACzD,YAAY,EACX,WAAW,EACX,MAAM,EACN,mBAAmB,EACnB,UAAU,EACV,cAAc,EACd,iBAAiB,EACjB,gBAAgB,GAChB,MAAM,YAAY,CAAC"}
@@ -1,2 +0,0 @@
export { getCookies, toCookieHeader } from './public.js';
//# sourceMappingURL=index.js.map
@@ -1 +0,0 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC"}
@@ -1,8 +0,0 @@
import type { GetCookiesResult } from '../types.js';
export declare function getCookiesFromChrome(options: {
profile?: string;
timeoutMs?: number;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null): Promise<GetCookiesResult>;
//# sourceMappingURL=chrome.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"chrome.d.ts","sourceRoot":"","sources":["../../src/providers/chrome.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAU,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAK5D,wBAAsB,oBAAoB,CACzC,OAAO,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EAC5F,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,GAChC,OAAO,CAAC,gBAAgB,CAAC,CA0B3B"}
@@ -1,27 +0,0 @@
import { getCookiesFromChromeSqliteLinux } from './chromeSqliteLinux.js';
import { getCookiesFromChromeSqliteMac } from './chromeSqliteMac.js';
import { getCookiesFromChromeSqliteWindows } from './chromeSqliteWindows.js';
export async function getCookiesFromChrome(options, origins, allowlistNames) {
const warnings = [];
// Platform dispatch only. All real logic lives in the per-OS providers.
if (process.platform === 'darwin') {
const r = await getCookiesFromChromeSqliteMac(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
if (process.platform === 'linux') {
const r = await getCookiesFromChromeSqliteLinux(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
if (process.platform === 'win32') {
const r = await getCookiesFromChromeSqliteWindows(options, origins, allowlistNames);
warnings.push(...r.warnings);
const cookies = r.cookies;
return { cookies, warnings };
}
return { cookies: [], warnings };
}
//# sourceMappingURL=chrome.js.map
@@ -1 +0,0 @@
{"version":3,"file":"chrome.js","sourceRoot":"","sources":["../../src/providers/chrome.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,+BAA+B,EAAE,MAAM,wBAAwB,CAAC;AACzE,OAAO,EAAE,6BAA6B,EAAE,MAAM,sBAAsB,CAAC;AACrE,OAAO,EAAE,iCAAiC,EAAE,MAAM,0BAA0B,CAAC;AAE7E,MAAM,CAAC,KAAK,UAAU,oBAAoB,CACzC,OAA4F,EAC5F,OAAiB,EACjB,cAAkC;IAElC,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,wEAAwE;IACxE,IAAI,OAAO,CAAC,QAAQ,KAAK,QAAQ,EAAE,CAAC;QACnC,MAAM,CAAC,GAAG,MAAM,6BAA6B,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QAChF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QAClC,MAAM,CAAC,GAAG,MAAM,+BAA+B,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QAClF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,IAAI,OAAO,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QAClC,MAAM,CAAC,GAAG,MAAM,iCAAiC,CAAC,OAAO,EAAE,OAAO,EAAE,cAAc,CAAC,CAAC;QACpF,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,CAAC;QAC7B,MAAM,OAAO,GAAa,CAAC,CAAC,OAAO,CAAC;QACpC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;IAC9B,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;AAClC,CAAC"}
@@ -1,11 +0,0 @@
export declare function deriveAes128CbcKeyFromPassword(password: string, options: {
iterations: number;
}): Buffer;
export declare function decryptChromiumAes128CbcCookieValue(encryptedValue: Uint8Array, keyCandidates: readonly Buffer[], options: {
stripHashPrefix: boolean;
treatUnknownPrefixAsPlaintext?: boolean;
}): string | null;
export declare function decryptChromiumAes256GcmCookieValue(encryptedValue: Uint8Array, key: Buffer, options: {
stripHashPrefix: boolean;
}): string | null;
//# sourceMappingURL=crypto.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"crypto.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/crypto.ts"],"names":[],"mappings":"AAIA,wBAAgB,8BAA8B,CAC7C,QAAQ,EAAE,MAAM,EAChB,OAAO,EAAE;IAAE,UAAU,EAAE,MAAM,CAAA;CAAE,GAC7B,MAAM,CAIR;AAED,wBAAgB,mCAAmC,CAClD,cAAc,EAAE,UAAU,EAC1B,aAAa,EAAE,SAAS,MAAM,EAAE,EAChC,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAC;IAAC,6BAA6B,CAAC,EAAE,OAAO,CAAA;CAAE,GAC5E,MAAM,GAAG,IAAI,CA2Bf;AAED,wBAAgB,mCAAmC,CAClD,cAAc,EAAE,UAAU,EAC1B,GAAG,EAAE,MAAM,EACX,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAA;CAAE,GACnC,MAAM,GAAG,IAAI,CAyBf"}
@@ -1,100 +0,0 @@
import { createDecipheriv, pbkdf2Sync } from 'node:crypto';
const UTF8_DECODER = new TextDecoder('utf-8', { fatal: true });
export function deriveAes128CbcKeyFromPassword(password, options) {
// Chromium derives the AES-128-CBC key from "Chrome Safe Storage" using PBKDF2.
// The salt/length/digest are fixed by Chromium ("saltysalt", 16 bytes, sha1).
return pbkdf2Sync(password, 'saltysalt', options.iterations, 16, 'sha1');
}
export function decryptChromiumAes128CbcCookieValue(encryptedValue, keyCandidates, options) {
const buf = Buffer.from(encryptedValue);
if (buf.length < 3)
return null;
// Chromium prefixes encrypted cookies with `v10`, `v11`, ... (three bytes).
const prefix = buf.subarray(0, 3).toString('utf8');
const hasVersionPrefix = /^v\d\d$/.test(prefix);
if (!hasVersionPrefix) {
// Some platforms (notably macOS) can store plaintext values in `encrypted_value`.
// Callers decide whether unknown prefixes should be treated as plaintext.
if (options.treatUnknownPrefixAsPlaintext === false)
return null;
return decodeCookieValueBytes(buf, false);
}
const ciphertext = buf.subarray(3);
if (!ciphertext.length)
return '';
for (const key of keyCandidates) {
// Try multiple candidates because Linux may fall back to empty passwords depending on keyring state.
const decrypted = tryDecryptAes128Cbc(ciphertext, key);
if (!decrypted)
continue;
const decoded = decodeCookieValueBytes(decrypted, options.stripHashPrefix);
if (decoded !== null)
return decoded;
}
return null;
}
export function decryptChromiumAes256GcmCookieValue(encryptedValue, key, options) {
const buf = Buffer.from(encryptedValue);
if (buf.length < 3)
return null;
const prefix = buf.subarray(0, 3).toString('utf8');
if (!/^v\d\d$/.test(prefix))
return null;
// AES-256-GCM layout:
// - 12-byte nonce
// - ciphertext
// - 16-byte authentication tag
const payload = buf.subarray(3);
if (payload.length < 12 + 16)
return null;
const nonce = payload.subarray(0, 12);
const authenticationTag = payload.subarray(payload.length - 16);
const ciphertext = payload.subarray(12, payload.length - 16);
try {
const decipher = createDecipheriv('aes-256-gcm', key, nonce);
decipher.setAuthTag(authenticationTag);
const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
return decodeCookieValueBytes(plaintext, options.stripHashPrefix);
}
catch {
return null;
}
}
function tryDecryptAes128Cbc(ciphertext, key) {
try {
// Chromium's legacy AES-128-CBC uses an IV of 16 spaces.
const iv = Buffer.alloc(16, 0x20);
const decipher = createDecipheriv('aes-128-cbc', key, iv);
decipher.setAutoPadding(false);
const plaintext = Buffer.concat([decipher.update(ciphertext), decipher.final()]);
return removePkcs7Padding(plaintext);
}
catch {
return null;
}
}
function removePkcs7Padding(value) {
if (!value.length)
return value;
const padding = value[value.length - 1];
if (!padding || padding > 16)
return value;
return value.subarray(0, value.length - padding);
}
function decodeCookieValueBytes(value, stripHashPrefix) {
// Chromium >= 24 prepends a 32-byte hash to cookie values.
const bytes = stripHashPrefix && value.length >= 32 ? value.subarray(32) : value;
try {
return stripLeadingControlChars(UTF8_DECODER.decode(bytes));
}
catch {
return null;
}
}
function stripLeadingControlChars(value) {
let i = 0;
while (i < value.length && value.charCodeAt(i) < 0x20)
i += 1;
return value.slice(i);
}
//# sourceMappingURL=crypto.js.map
@@ -1 +0,0 @@
{"version":3,"file":"crypto.js","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/crypto.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAE3D,MAAM,YAAY,GAAG,IAAI,WAAW,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;AAE/D,MAAM,UAAU,8BAA8B,CAC7C,QAAgB,EAChB,OAA+B;IAE/B,gFAAgF;IAChF,8EAA8E;IAC9E,OAAO,UAAU,CAAC,QAAQ,EAAE,WAAW,EAAE,OAAO,CAAC,UAAU,EAAE,EAAE,EAAE,MAAM,CAAC,CAAC;AAC1E,CAAC;AAED,MAAM,UAAU,mCAAmC,CAClD,cAA0B,EAC1B,aAAgC,EAChC,OAA8E;IAE9E,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;IACxC,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IAEhC,4EAA4E;IAC5E,MAAM,MAAM,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IACnD,MAAM,gBAAgB,GAAG,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAEhD,IAAI,CAAC,gBAAgB,EAAE,CAAC;QACvB,kFAAkF;QAClF,0EAA0E;QAC1E,IAAI,OAAO,CAAC,6BAA6B,KAAK,KAAK;YAAE,OAAO,IAAI,CAAC;QACjE,OAAO,sBAAsB,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;IAC3C,CAAC;IAED,MAAM,UAAU,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;IACnC,IAAI,CAAC,UAAU,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAElC,KAAK,MAAM,GAAG,IAAI,aAAa,EAAE,CAAC;QACjC,qGAAqG;QACrG,MAAM,SAAS,GAAG,mBAAmB,CAAC,UAAU,EAAE,GAAG,CAAC,CAAC;QACvD,IAAI,CAAC,SAAS;YAAE,SAAS;QACzB,MAAM,OAAO,GAAG,sBAAsB,CAAC,SAAS,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;QAC3E,IAAI,OAAO,KAAK,IAAI;YAAE,OAAO,OAAO,CAAC;IACtC,CAAC;IAED,OAAO,IAAI,CAAC;AACb,CAAC;AAED,MAAM,UAAU,mCAAmC,CAClD,cAA0B,EAC1B,GAAW,EACX,OAAqC;IAErC,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;IACxC,IAAI,GAAG,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IAChC,MAAM,MAAM,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;IACnD,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC;QAAE,OAAO,IAAI,CAAC;IAEzC,sBAAsB;IACtB,kBAAkB;IAClB,eAAe;IACf,+BAA+B;IAC/B,MAAM,OAAO,GAAG,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;IAChC,IAAI,OAAO,CAAC,MAAM,GAAG,EAAE,GAAG,EAAE;QAAE,OAAO,IAAI,CAAC;IAE1C,MAAM,KAAK,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;IACtC,MAAM,iBAAiB,GAAG,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC;IAChE,MAAM,UAAU,GAAG,OAAO,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC;IAE7D,IAAI,CAAC;QACJ,MAAM,QAAQ,GAAG,gBAAgB,CAAC,aAAa,EAAE,GAAG,EAAE,KAAK,CAAC,CAAC;QAC7D,QAAQ,CAAC,UAAU,CAAC,iBAAiB,CAAC,CAAC;QACvC,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,EAAE,QAAQ,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QACjF,OAAO,sBAAsB,CAAC,SAAS,EAAE,OAAO,CAAC,eAAe,CAAC,CAAC;IACnE,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,mBAAmB,CAAC,UAAkB,EAAE,GAAW;IAC3D,IAAI,CAAC;QACJ,yDAAyD;QACzD,MAAM,EAAE,GAAG,MAAM,CAAC,KAAK,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;QAClC,MAAM,QAAQ,GAAG,gBAAgB,CAAC,aAAa,EAAE,GAAG,EAAE,EAAE,CAAC,CAAC;QAC1D,QAAQ,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QAC/B,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,EAAE,QAAQ,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QACjF,OAAO,kBAAkB,CAAC,SAAS,CAAC,CAAC;IACtC,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,kBAAkB,CAAC,KAAa;IACxC,IAAI,CAAC,KAAK,CAAC,MAAM;QAAE,OAAO,KAAK,CAAC;IAChC,MAAM,OAAO,GAAG,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;IACxC,IAAI,CAAC,OAAO,IAAI,OAAO,GAAG,EAAE;QAAE,OAAO,KAAK,CAAC;IAC3C,OAAO,KAAK,CAAC,QAAQ,CAAC,CAAC,EAAE,KAAK,CAAC,MAAM,GAAG,OAAO,CAAC,CAAC;AAClD,CAAC;AAED,SAAS,sBAAsB,CAAC,KAAa,EAAE,eAAwB;IACtE,2DAA2D;IAC3D,MAAM,KAAK,GAAG,eAAe,IAAI,KAAK,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC;IACjF,IAAI,CAAC;QACJ,OAAO,wBAAwB,CAAC,YAAY,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;IAC7D,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,IAAI,CAAC;IACb,CAAC;AACF,CAAC;AAED,SAAS,wBAAwB,CAAC,KAAa;IAC9C,IAAI,CAAC,GAAG,CAAC,CAAC;IACV,OAAO,CAAC,GAAG,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC,GAAG,IAAI;QAAE,CAAC,IAAI,CAAC,CAAC;IAC9D,OAAO,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;AACvB,CAAC"}
@@ -1,25 +0,0 @@
export type LinuxKeyringBackend = 'gnome' | 'kwallet' | 'basic';
/**
* Read the "Safe Storage" password from a Linux keyring.
*
* Chromium browsers typically store their cookie encryption password under:
* - service: "<Browser> Safe Storage"
* - account: "<Browser>"
*
* We keep this logic in JS (no native deps) and return an empty password on failure
* (Chromium may still have v10 cookies, and callers can use inline/export escape hatches).
*/
export declare function getLinuxChromiumSafeStoragePassword(options: {
backend?: LinuxKeyringBackend;
app: 'chrome' | 'edge';
}): Promise<{
password: string;
warnings: string[];
}>;
export declare function getLinuxChromeSafeStoragePassword(options?: {
backend?: LinuxKeyringBackend;
}): Promise<{
password: string;
warnings: string[];
}>;
//# sourceMappingURL=linuxKeyring.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"linuxKeyring.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/linuxKeyring.ts"],"names":[],"mappings":"AAEA,MAAM,MAAM,mBAAmB,GAAG,OAAO,GAAG,SAAS,GAAG,OAAO,CAAC;AAEhE;;;;;;;;;GASG;AACH,wBAAsB,mCAAmC,CAAC,OAAO,EAAE;IAClE,OAAO,CAAC,EAAE,mBAAmB,CAAC;IAC9B,GAAG,EAAE,QAAQ,GAAG,MAAM,CAAC;CACvB,GAAG,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,EAAE,CAAA;CAAE,CAAC,CA8DpD;AAED,wBAAsB,iCAAiC,CACtD,OAAO,GAAE;IAAE,OAAO,CAAC,EAAE,mBAAmB,CAAA;CAAO,GAC7C,OAAO,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,EAAE,CAAA;CAAE,CAAC,CAInD"}
@@ -1,104 +0,0 @@
import { execCapture } from '../../util/exec.js';
/**
* Read the "Safe Storage" password from a Linux keyring.
*
* Chromium browsers typically store their cookie encryption password under:
* - service: "<Browser> Safe Storage"
* - account: "<Browser>"
*
* We keep this logic in JS (no native deps) and return an empty password on failure
* (Chromium may still have v10 cookies, and callers can use inline/export escape hatches).
*/
export async function getLinuxChromiumSafeStoragePassword(options) {
const warnings = [];
// Escape hatch: if callers already know the password (or want deterministic CI behavior),
// they can bypass keyring probing entirely.
const overrideKey = options.app === 'edge'
? 'SWEET_COOKIE_EDGE_SAFE_STORAGE_PASSWORD'
: 'SWEET_COOKIE_CHROME_SAFE_STORAGE_PASSWORD';
const override = readEnv(overrideKey);
if (override !== undefined)
return { password: override, warnings };
const backend = options.backend ?? parseLinuxKeyringBackend() ?? chooseLinuxKeyringBackend();
// `basic` means "don't try keyrings" (Chrome will fall back to older/less-secure schemes on some setups).
if (backend === 'basic')
return { password: '', warnings };
const service = options.app === 'edge' ? 'Microsoft Edge Safe Storage' : 'Chrome Safe Storage';
const account = options.app === 'edge' ? 'Microsoft Edge' : 'Chrome';
const folder = `${account} Keys`;
if (backend === 'gnome') {
// GNOME keyring: `secret-tool` is the simplest way to read libsecret entries.
const res = await execCapture('secret-tool', ['lookup', 'service', service, 'account', account], { timeoutMs: 3_000 });
if (res.code === 0)
return { password: res.stdout.trim(), warnings };
warnings.push('Failed to read Linux keyring via secret-tool; v11 cookies may be unavailable.');
return { password: '', warnings };
}
// KDE keyring: query KWallet via `kwallet-query`, but the wallet name differs across KDE versions.
const kdeVersion = (readEnv('KDE_SESSION_VERSION') ?? '').trim();
const serviceName = kdeVersion === '6'
? 'org.kde.kwalletd6'
: kdeVersion === '5'
? 'org.kde.kwalletd5'
: 'org.kde.kwalletd';
const walletPath = kdeVersion === '6'
? '/modules/kwalletd6'
: kdeVersion === '5'
? '/modules/kwalletd5'
: '/modules/kwalletd';
const wallet = await getKWalletNetworkWallet(serviceName, walletPath);
const passwordRes = await execCapture('kwallet-query', ['--read-password', service, '--folder', folder, wallet], { timeoutMs: 3_000 });
if (passwordRes.code !== 0) {
warnings.push('Failed to read Linux keyring via kwallet-query; v11 cookies may be unavailable.');
return { password: '', warnings };
}
if (passwordRes.stdout.toLowerCase().startsWith('failed to read'))
return { password: '', warnings };
return { password: passwordRes.stdout.trim(), warnings };
}
export async function getLinuxChromeSafeStoragePassword(options = {}) {
const args = { app: 'chrome' };
if (options.backend !== undefined)
args.backend = options.backend;
return await getLinuxChromiumSafeStoragePassword(args);
}
function parseLinuxKeyringBackend() {
const raw = readEnv('SWEET_COOKIE_LINUX_KEYRING');
if (!raw)
return undefined;
const normalized = raw.toLowerCase();
if (normalized === 'gnome')
return 'gnome';
if (normalized === 'kwallet')
return 'kwallet';
if (normalized === 'basic')
return 'basic';
return undefined;
}
function chooseLinuxKeyringBackend() {
const xdg = readEnv('XDG_CURRENT_DESKTOP') ?? '';
const isKde = xdg.split(':').some((p) => p.trim().toLowerCase() === 'kde') || !!readEnv('KDE_FULL_SESSION');
return isKde ? 'kwallet' : 'gnome';
}
async function getKWalletNetworkWallet(serviceName, walletPath) {
const res = await execCapture('dbus-send', [
'--session',
'--print-reply=literal',
`--dest=${serviceName}`,
walletPath,
'org.kde.KWallet.networkWallet',
], { timeoutMs: 3_000 });
const fallback = 'kdewallet';
if (res.code !== 0)
return fallback;
const raw = res.stdout.trim();
if (!raw)
return fallback;
return raw.replaceAll('"', '').trim() || fallback;
}
function readEnv(key) {
const value = process.env[key];
const trimmed = typeof value === 'string' ? value.trim() : '';
return trimmed.length ? trimmed : undefined;
}
//# sourceMappingURL=linuxKeyring.js.map
@@ -1 +0,0 @@
{"version":3,"file":"linuxKeyring.js","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/linuxKeyring.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AAIjD;;;;;;;;;GASG;AACH,MAAM,CAAC,KAAK,UAAU,mCAAmC,CAAC,OAGzD;IACA,MAAM,QAAQ,GAAa,EAAE,CAAC;IAE9B,0FAA0F;IAC1F,4CAA4C;IAC5C,MAAM,WAAW,GAChB,OAAO,CAAC,GAAG,KAAK,MAAM;QACrB,CAAC,CAAC,yCAAyC;QAC3C,CAAC,CAAC,2CAA2C,CAAC;IAChD,MAAM,QAAQ,GAAG,OAAO,CAAC,WAAW,CAAC,CAAC;IACtC,IAAI,QAAQ,KAAK,SAAS;QAAE,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,QAAQ,EAAE,CAAC;IAEpE,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,wBAAwB,EAAE,IAAI,yBAAyB,EAAE,CAAC;IAC7F,0GAA0G;IAC1G,IAAI,OAAO,KAAK,OAAO;QAAE,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IAE3D,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,KAAK,MAAM,CAAC,CAAC,CAAC,6BAA6B,CAAC,CAAC,CAAC,qBAAqB,CAAC;IAC/F,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,KAAK,MAAM,CAAC,CAAC,CAAC,gBAAgB,CAAC,CAAC,CAAC,QAAQ,CAAC;IACrE,MAAM,MAAM,GAAG,GAAG,OAAO,OAAO,CAAC;IAEjC,IAAI,OAAO,KAAK,OAAO,EAAE,CAAC;QACzB,8EAA8E;QAC9E,MAAM,GAAG,GAAG,MAAM,WAAW,CAC5B,aAAa,EACb,CAAC,QAAQ,EAAE,SAAS,EAAE,OAAO,EAAE,SAAS,EAAE,OAAO,CAAC,EAClD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;QACF,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC;YAAE,OAAO,EAAE,QAAQ,EAAE,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC;QACrE,QAAQ,CAAC,IAAI,CAAC,+EAA+E,CAAC,CAAC;QAC/F,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,CAAC;IAED,mGAAmG;IACnG,MAAM,UAAU,GAAG,CAAC,OAAO,CAAC,qBAAqB,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACjE,MAAM,WAAW,GAChB,UAAU,KAAK,GAAG;QACjB,CAAC,CAAC,mBAAmB;QACrB,CAAC,CAAC,UAAU,KAAK,GAAG;YACnB,CAAC,CAAC,mBAAmB;YACrB,CAAC,CAAC,kBAAkB,CAAC;IACxB,MAAM,UAAU,GACf,UAAU,KAAK,GAAG;QACjB,CAAC,CAAC,oBAAoB;QACtB,CAAC,CAAC,UAAU,KAAK,GAAG;YACnB,CAAC,CAAC,oBAAoB;YACtB,CAAC,CAAC,mBAAmB,CAAC;IAEzB,MAAM,MAAM,GAAG,MAAM,uBAAuB,CAAC,WAAW,EAAE,UAAU,CAAC,CAAC;IACtE,MAAM,WAAW,GAAG,MAAM,WAAW,CACpC,eAAe,EACf,CAAC,iBAAiB,EAAE,OAAO,EAAE,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,EACxD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;IACF,IAAI,WAAW,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QAC5B,QAAQ,CAAC,IAAI,CACZ,iFAAiF,CACjF,CAAC;QACF,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,CAAC;IACD,IAAI,WAAW,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC,UAAU,CAAC,gBAAgB,CAAC;QAChE,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,CAAC;IACnC,OAAO,EAAE,QAAQ,EAAE,WAAW,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC;AAC1D,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,iCAAiC,CACtD,UAA6C,EAAE;IAE/C,MAAM,IAAI,GAAqD,EAAE,GAAG,EAAE,QAAQ,EAAE,CAAC;IACjF,IAAI,OAAO,CAAC,OAAO,KAAK,SAAS;QAAE,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC,OAAO,CAAC;IAClE,OAAO,MAAM,mCAAmC,CAAC,IAAI,CAAC,CAAC;AACxD,CAAC;AAED,SAAS,wBAAwB;IAChC,MAAM,GAAG,GAAG,OAAO,CAAC,4BAA4B,CAAC,CAAC;IAClD,IAAI,CAAC,GAAG;QAAE,OAAO,SAAS,CAAC;IAC3B,MAAM,UAAU,GAAG,GAAG,CAAC,WAAW,EAAE,CAAC;IACrC,IAAI,UAAU,KAAK,OAAO;QAAE,OAAO,OAAO,CAAC;IAC3C,IAAI,UAAU,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IAC/C,IAAI,UAAU,KAAK,OAAO;QAAE,OAAO,OAAO,CAAC;IAC3C,OAAO,SAAS,CAAC;AAClB,CAAC;AAED,SAAS,yBAAyB;IACjC,MAAM,GAAG,GAAG,OAAO,CAAC,qBAAqB,CAAC,IAAI,EAAE,CAAC;IACjD,MAAM,KAAK,GACV,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,KAAK,KAAK,CAAC,IAAI,CAAC,CAAC,OAAO,CAAC,kBAAkB,CAAC,CAAC;IAC/F,OAAO,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC;AACpC,CAAC;AAED,KAAK,UAAU,uBAAuB,CAAC,WAAmB,EAAE,UAAkB;IAC7E,MAAM,GAAG,GAAG,MAAM,WAAW,CAC5B,WAAW,EACX;QACC,WAAW;QACX,uBAAuB;QACvB,UAAU,WAAW,EAAE;QACvB,UAAU;QACV,+BAA+B;KAC/B,EACD,EAAE,SAAS,EAAE,KAAK,EAAE,CACpB,CAAC;IACF,MAAM,QAAQ,GAAG,WAAW,CAAC;IAC7B,IAAI,GAAG,CAAC,IAAI,KAAK,CAAC;QAAE,OAAO,QAAQ,CAAC;IACpC,MAAM,GAAG,GAAG,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;IAC9B,IAAI,CAAC,GAAG;QAAE,OAAO,QAAQ,CAAC;IAC1B,OAAO,GAAG,CAAC,UAAU,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,IAAI,QAAQ,CAAC;AACnD,CAAC;AAED,SAAS,OAAO,CAAC,GAAW;IAC3B,MAAM,KAAK,GAAG,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAC/B,MAAM,OAAO,GAAG,OAAO,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;IAC9D,OAAO,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;AAC7C,CAAC"}
@@ -1,10 +0,0 @@
import type { GetCookiesResult } from '../../types.js';
export declare function getCookiesFromChromeSqliteDb(options: {
dbPath: string;
profile?: string;
includeExpired?: boolean;
debug?: boolean;
}, origins: string[], allowlistNames: Set<string> | null, decrypt: (encryptedValue: Uint8Array, options: {
stripHashPrefix: boolean;
}) => string | null): Promise<GetCookiesResult>;
//# sourceMappingURL=shared.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"shared.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/shared.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAA0B,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AAkB/E,wBAAsB,4BAA4B,CACjD,OAAO,EAAE;IAAE,MAAM,EAAE,MAAM,CAAC;IAAC,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAC;IAAC,KAAK,CAAC,EAAE,OAAO,CAAA;CAAE,EACxF,OAAO,EAAE,MAAM,EAAE,EACjB,cAAc,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,IAAI,EAClC,OAAO,EAAE,CAAC,cAAc,EAAE,UAAU,EAAE,OAAO,EAAE;IAAE,eAAe,EAAE,OAAO,CAAA;CAAE,KAAK,MAAM,GAAG,IAAI,GAC3F,OAAO,CAAC,gBAAgB,CAAC,CAsD3B"}
@@ -1,293 +0,0 @@
import { copyFileSync, existsSync, mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { normalizeExpiration } from '../../util/expire.js';
import { hostMatchesCookieDomain } from '../../util/hostMatch.js';
import { importNodeSqlite, supportsReadBigInts } from '../../util/nodeSqlite.js';
import { isBunRuntime } from '../../util/runtime.js';
export async function getCookiesFromChromeSqliteDb(options, origins, allowlistNames, decrypt) {
const warnings = [];
// Chrome can keep its cookie DB locked and/or rely on WAL sidecars.
// Copying to a temp dir gives us a stable snapshot that both node:sqlite and bun:sqlite can open.
const tempDir = mkdtempSync(path.join(tmpdir(), 'sweet-cookie-chrome-'));
const tempDbPath = path.join(tempDir, 'Cookies');
try {
copyFileSync(options.dbPath, tempDbPath);
// If WAL is enabled, the latest writes might live in `Cookies-wal`/`Cookies-shm`.
// Copy them too when present so our snapshot reflects the current browser state.
copySidecar(options.dbPath, `${tempDbPath}-wal`, '-wal');
copySidecar(options.dbPath, `${tempDbPath}-shm`, '-shm');
}
catch (error) {
rmSync(tempDir, { recursive: true, force: true });
warnings.push(`Failed to copy Chrome cookie DB: ${error instanceof Error ? error.message : String(error)}`);
return { cookies: [], warnings };
}
try {
const hosts = origins.map((o) => new URL(o).hostname);
const where = buildHostWhereClause(hosts, 'host_key');
const metaVersion = await readChromiumMetaVersion(tempDbPath);
// Chromium >= 24 stores a 32-byte hash prefix in decrypted cookie values.
// We detect this via the `meta` table version and strip it when present.
const stripHashPrefix = metaVersion >= 24;
const rowsResult = await readChromeRows(tempDbPath, where);
if (!rowsResult.ok) {
warnings.push(rowsResult.error);
return { cookies: [], warnings };
}
const collectOptions = {};
if (options.profile)
collectOptions.profile = options.profile;
if (options.includeExpired !== undefined)
collectOptions.includeExpired = options.includeExpired;
const cookies = collectChromeCookiesFromRows(rowsResult.rows, collectOptions, hosts, allowlistNames, (encryptedValue) => decrypt(encryptedValue, { stripHashPrefix }), warnings);
return { cookies: dedupeCookies(cookies), warnings };
}
finally {
rmSync(tempDir, { recursive: true, force: true });
}
}
function collectChromeCookiesFromRows(rows, options, hosts, allowlistNames, decrypt, warnings) {
const cookies = [];
const now = Math.floor(Date.now() / 1000);
let warnedEncryptedType = false;
for (const row of rows) {
const name = typeof row.name === 'string' ? row.name : null;
if (!name)
continue;
if (allowlistNames && allowlistNames.size > 0 && !allowlistNames.has(name))
continue;
const hostKey = typeof row.host_key === 'string' ? row.host_key : null;
if (!hostKey)
continue;
if (!hostMatchesAny(hosts, hostKey))
continue;
const rowPath = typeof row.path === 'string' ? row.path : '';
const valueString = typeof row.value === 'string' ? row.value : null;
let value = valueString;
if (value === null || value.length === 0) {
// Many modern Chromium cookies keep `value` empty and only store `encrypted_value`.
// We decrypt on demand and drop rows we can't interpret.
const encryptedBytes = getEncryptedBytes(row);
if (!encryptedBytes) {
if (!warnedEncryptedType && row.encrypted_value !== undefined) {
warnings.push('Chrome cookie encrypted_value is in an unsupported type.');
warnedEncryptedType = true;
}
continue;
}
value = decrypt(encryptedBytes);
}
if (value === null)
continue;
const expiresRaw = typeof row.expires_utc === 'number' || typeof row.expires_utc === 'bigint'
? row.expires_utc
: tryParseInt(row.expires_utc);
const expires = normalizeExpiration(expiresRaw ?? undefined);
if (!options.includeExpired) {
if (expires && expires < now)
continue;
}
const secure = row.is_secure === 1 ||
row.is_secure === 1n ||
row.is_secure === '1' ||
row.is_secure === true;
const httpOnly = row.is_httponly === 1 ||
row.is_httponly === 1n ||
row.is_httponly === '1' ||
row.is_httponly === true;
const sameSite = normalizeChromiumSameSite(row.samesite);
const source = { browser: 'chrome' };
if (options.profile)
source.profile = options.profile;
const cookie = {
name,
value,
domain: hostKey.startsWith('.') ? hostKey.slice(1) : hostKey,
path: rowPath || '/',
secure,
httpOnly,
source,
};
if (expires !== undefined)
cookie.expires = expires;
if (sameSite !== undefined)
cookie.sameSite = sameSite;
cookies.push(cookie);
}
return cookies;
}
function tryParseInt(value) {
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? parsed : null;
}
if (typeof value !== 'string')
return null;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : null;
}
function normalizeChromiumSameSite(value) {
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? normalizeChromiumSameSite(parsed) : undefined;
}
if (typeof value === 'number') {
if (value === 2)
return 'Strict';
if (value === 1)
return 'Lax';
if (value === 0)
return 'None';
return undefined;
}
if (typeof value === 'string') {
const parsed = Number.parseInt(value, 10);
if (Number.isFinite(parsed))
return normalizeChromiumSameSite(parsed);
const normalized = value.toLowerCase();
if (normalized === 'strict')
return 'Strict';
if (normalized === 'lax')
return 'Lax';
if (normalized === 'none' || normalized === 'no_restriction')
return 'None';
}
return undefined;
}
function getEncryptedBytes(row) {
const raw = row.encrypted_value;
if (raw instanceof Uint8Array)
return raw;
return null;
}
async function readChromiumMetaVersion(dbPath) {
const sql = `SELECT value FROM meta WHERE key = 'version'`;
const result = isBunRuntime()
? await queryNodeOrBun({ kind: 'bun', dbPath, sql })
: await queryNodeOrBun({ kind: 'node', dbPath, sql });
if (!result.ok)
return 0;
const first = result.rows[0];
const value = first?.value;
if (typeof value === 'number')
return Math.floor(value);
if (typeof value === 'bigint') {
const parsed = Number(value);
return Number.isFinite(parsed) ? Math.floor(parsed) : 0;
}
if (typeof value === 'string') {
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) ? parsed : 0;
}
return 0;
}
async function readChromeRows(dbPath, where) {
const sqliteKind = isBunRuntime() ? 'bun' : 'node';
const sqliteLabel = sqliteKind === 'bun' ? 'bun:sqlite' : 'node:sqlite';
const sql = `SELECT name, value, host_key, path, expires_utc, samesite, encrypted_value, ` +
`is_secure AS is_secure, is_httponly AS is_httponly ` +
`FROM cookies WHERE (${where}) ORDER BY expires_utc DESC;`;
const result = await queryNodeOrBun({ kind: sqliteKind, dbPath, sql });
if (result.ok)
return { ok: true, rows: result.rows };
// Intentionally strict: only support modern Chromium cookie DB schemas.
// If this fails, assume the local Chrome/Chromium is too old or uses a non-standard schema.
return {
ok: false,
error: `${sqliteLabel} failed reading Chrome cookies (requires modern Chromium, e.g. Chrome >= 100): ${result.error}`,
};
}
async function queryNodeOrBun(options) {
try {
if (options.kind === 'node') {
// Node's `node:sqlite` is synchronous and returns plain JS values. Keep it boxed in a
// small scope so callers don't need to care about runtime differences.
const { DatabaseSync } = await importNodeSqlite();
const dbOptions = { readOnly: true };
if (supportsReadBigInts()) {
dbOptions.readBigInts = true;
}
const db = new DatabaseSync(options.dbPath, dbOptions);
try {
const rows = db.prepare(options.sql).all();
return { ok: true, rows };
}
finally {
db.close();
}
}
// Bun's sqlite API has a different surface (`Database` + `.query().all()`).
const { Database } = await import('bun:sqlite');
const db = new Database(options.dbPath, { readonly: true });
try {
const rows = db.query(options.sql).all();
return { ok: true, rows };
}
finally {
db.close();
}
}
catch (error) {
return { ok: false, error: error instanceof Error ? error.message : String(error) };
}
}
function copySidecar(sourceDbPath, target, suffix) {
const sidecar = `${sourceDbPath}${suffix}`;
if (!existsSync(sidecar))
return;
try {
copyFileSync(sidecar, target);
}
catch {
// ignore
}
}
function buildHostWhereClause(hosts, column) {
const clauses = [];
for (const host of hosts) {
// Chrome cookies often live on parent domains (e.g. .google.com for gemini.google.com).
// Include parent domains so the SQL filter doesn't drop valid session cookies.
for (const candidate of expandHostCandidates(host)) {
const escaped = sqlLiteral(candidate);
const escapedDot = sqlLiteral(`.${candidate}`);
const escapedLike = sqlLiteral(`%.${candidate}`);
clauses.push(`${column} = ${escaped}`);
clauses.push(`${column} = ${escapedDot}`);
clauses.push(`${column} LIKE ${escapedLike}`);
}
}
return clauses.length ? clauses.join(' OR ') : '1=0';
}
function sqlLiteral(value) {
const escaped = value.replaceAll("'", "''");
return `'${escaped}'`;
}
function expandHostCandidates(host) {
const parts = host.split('.').filter(Boolean);
if (parts.length <= 1)
return [host];
const candidates = new Set();
candidates.add(host);
// Include parent domains down to two labels (avoid TLD-only fragments).
for (let i = 1; i <= parts.length - 2; i += 1) {
const candidate = parts.slice(i).join('.');
if (candidate)
candidates.add(candidate);
}
return Array.from(candidates);
}
function hostMatchesAny(hosts, cookieHost) {
const cookieDomain = cookieHost.startsWith('.') ? cookieHost.slice(1) : cookieHost;
return hosts.some((host) => hostMatchesCookieDomain(host, cookieDomain));
}
function dedupeCookies(cookies) {
const merged = new Map();
for (const cookie of cookies) {
const key = `${cookie.name}|${cookie.domain ?? ''}|${cookie.path ?? ''}`;
if (!merged.has(key))
merged.set(key, cookie);
}
return Array.from(merged.values());
}
//# sourceMappingURL=shared.js.map
File diff suppressed because one or more lines are too long
@@ -1,10 +0,0 @@
export declare function dpapiUnprotect(data: Buffer, options?: {
timeoutMs?: number;
}): Promise<{
ok: true;
value: Buffer;
} | {
ok: false;
error: string;
}>;
//# sourceMappingURL=windowsDpapi.d.ts.map
@@ -1 +0,0 @@
{"version":3,"file":"windowsDpapi.d.ts","sourceRoot":"","sources":["../../../src/providers/chromeSqlite/windowsDpapi.ts"],"names":[],"mappings":"AAEA,wBAAsB,cAAc,CACnC,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAO,GAClC,OAAO,CAAC;IAAE,EAAE,EAAE,IAAI,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GAAG;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC,CA+BrE"}

Some files were not shown because too many files have changed in this diff Show More