Compare commits

...

13 Commits

Author SHA1 Message Date
Matt Van Horn 46c5c7f566 fix: preserve output contract as six hard rules, not two
Live Ghostty test of the prior amendment confirmed RULE A (inline links)
works on first pass, but exposed a new regression: the established output
contract collapsed. First-pass output had "Story 1/2/3" plain-prose
sections instead of bold-headline paragraphs, a plain "Stats" heading
with - bullets instead of the ├─ └─ │ emoji tree, no  All agents
reported back! line, no --- separators, and a flat "I am now an expert"
closer instead of the QUERY_TYPE-specific invitation variant.

Same failure pattern as before (emphasis imbalance between co-equal
rules), opposite direction: the link rules at the top of the output
section became the loudest prose and the existing structural rules
(bold headlines, stats template, emoji tree, QUERY_TYPE invitation,
--- separators) read as style preferences by comparison.

Fix is additive, not reductive. The TWO HARD RULES framing becomes the
OUTPUT CONTRACT with six rules named A-F, each at equal emphasis, stated
in parallel construction:

- RULE A: inline markdown links on every citation.
- RULE B: no trailing list of links.
- RULE C: bold headline per narrative paragraph (**Headline** — body).
- RULE D: stats block with  opening, ├─/└─/│ tree, emoji prefix per
  active source. No plain - bullets, no missing emoji.
- RULE E: QUERY_TYPE-specific invitation variant with 2-3 example
  follow-ups from this run. No flat "I am now an expert" closer.
- RULE F: --- horizontal rules before and after the stats block.

Closing sentence: "These six rules are a set, not a priority order.
Satisfying some by violating others is a bug, not a compromise."

Corresponding changes:

- PRE-SYNTHESIS COMMIT expanded from 6 link-focused bullets to 7
  bullets covering template/headlines/stats/separators/sources/trailing-
  list. Structure-first ordering primes the agent to reach for the
  template before drafting prose.
- PRE-PRESENT SELF-CHECK reworked. Items 1 and 2 become counted
  structural checks (≥3 bold headline openers, ≥3 ├─ tree characters,
  one emoji per active source). New items 5 and 6 add separator and
  invitation-variant checks. Items 7-10 preserve the existing RULE B
  / RULE A / no-raw-URL / no-broken-link checks.
- Release-notes entry extended to mention the preserved output
  contract alongside the clickable links.

No Python changes. This is the third commit on PR #286.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 08:17:22 -07:00
Matt Van Horn 83c14209da fix: inline links on first pass, rebalance rule emphasis
Live test on Ghostty after the prior commit showed a regression: first-pass
/last30days synthesis stripped all inline links. Only after the user asked
"weren't these links supposed to be clickable?" did the agent re-render
with inline links. Agent's own root cause: "I misread LAW 1 — over-applied
'no trailing Sources: block' to mean 'no URLs anywhere'."

The templates and per-citation examples were already correct. The regression
was in EMPHASIS BALANCE — the anti-trailing-Sources prohibition used the
loudest prose (CRITICAL OVERRIDE, MANDATORY, MUST NOT) while the pro-inline-
link rule read as lighter guidance. On a fresh first read, the agent
collapsed both into one prohibition and stripped every citation link.

This commit rebalances emphasis and adds a forcing function:

- New "TWO HARD RULES FOR LINKS" framing at the top of the output section
  names RULE A (inline links required on every citation) and RULE B (no
  trailing list of links) as a matched pair with equal rhetorical weight.
  Explicit: "These rules are complementary, not alternatives."
- New "PRE-SYNTHESIS COMMIT" step forces the agent to identify sources,
  pull URLs, and commit to inline-link placement BEFORE drafting — not
  audit after the fact.
- Existing CRITICAL OVERRIDE block now opens by pointing back to the
  matched-pair framing and drops the "Your citation is the Web: line.
  Nothing else is needed." priming sentence that biased the agent toward
  link-free drafts.
- New BAD/ALSO-BAD/GOOD triple in the CRITICAL OVERRIDE block shows the
  exact regression shape (stripping inline links to "comply" with RULE B)
  as a distinct failure mode, not just the trailing-list failure.
- PRE-PRESENT SELF-CHECK reworked: item 6 becomes "RULE B — no trailing
  list", item 7 becomes "RULE A — inline links present on first pass"
  with a positive count check that regenerates if zero inline links
  appear when raw data had URLs (the exact 2026-04-20 regression repro).
- Release notes entry gains one line describing the first-pass guarantee.

No Python changes. This is an amendment on PR #286 folding the fix into
the same review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 07:24:56 -07:00
Matt Van Horn 38014f8143 feat: clickable source links in synthesis (no raw URLs)
Every cited source — X handles, subreddits, publications, YouTube channels,
Polymarket markets — now renders as a blue CMD-clickable markdown link in the
terminal. Raw URL strings remain forbidden everywhere in the output.

The old rule "NEVER paste raw URLs" blocked both raw URLs and links. Claude
Code renders CommonMark [text](url) as hyperlinks (URL hidden), so the new
rule is: "every link must be [text](url), never a raw URL string". Plain text
is the fallback only when the raw research dump has no URL for a source.

Changes:
- SKILL.md: invert URL formatting rule; update narrative / KEY PATTERNS /
  RECOMMENDATIONS / stats-block templates; retire the "I have all the links...
  Just ask." closer (it's now inline); add two new PRE-PRESENT SELF-CHECK
  items (no raw http/https strings, no broken empty links).
- README.md: note the clickable-links capability in the "How it works" list.
- release-notes.md: 2026-04-20 entry crediting @jay_k and @photomatt.

No Python changes — the raw research dump already carries URLs per item;
this is synthesis-side formatting only. No breaking changes to the engine,
the schema, or the CLI flags.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 00:24:56 -07:00
Matt Van Horn 3499c246b8 fix: add commands/last30days.md and remove skills/last30days-nux duplicate (#267)
Release / build-and-release (push) Has been cancelled
Adds commands/last30days.md so /last30days registers as a Claude Code
slash command for plugin users. Users type /last30days and autocomplete
prefix-matches to the canonical /last30days:last30days form (same as
/ce:plan resolving to /compound-engineering:ce-plan).

Removes skills/last30days-nux/, a byte-identical duplicate of the root
SKILL.md that created confusing /last30days:last30days-nux autocomplete
entries via Claude Code's plugin namespacing. Root SKILL.md remains
the canonical skill source; natural-language skill-selector invocation
is unchanged.

Recovery for users on v3.0.4: /plugin update last30days then /reload-plugins.

Closes #239 (path-escape error was already fixed in v3.0.4 by dropping
the rogue 'skills' key; v3.0.5 adds the slash command on top).
Supersedes #257 (suggested './' -> '.' workaround is obsolete since
v3.0.4 dropped the 'skills' key entirely, matching ecosystem standard).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 15:19:42 -04:00
Matt Van Horn 53b8e33d13 fix(youtube): use url= param for ScrapeCreators comments/transcript + parse new response shape (#265)
PR #260 wired YouTube comment enrichment against
`/v1/youtube/video/comments` with `id=<video_id>`, but the endpoint
requires `url=https://www.youtube.com/watch?v=<video_id>`. Every enrich
call was returning 400 "missing_parameter: you must provide a url", so
no YouTube items ever carried `top_comments`.

The SC transcript fallback (`_sc_fetch_transcript`) had the identical
contract mistake. It was latent because `_fetch_transcript` prefers
yt-dlp and the SC path only fires when yt-dlp is missing, but it would
have failed the same way on hosts without yt-dlp installed.

Switching both callers to `url=` surfaces a second issue in the
response parser: SC returns `author` as `{"name": "@handle", ...}` and
nests like counts under `engagement.likes`, not top-level. The parser
was reading `author` as a string and missing the nested likes, so even
after the param fix every comment would land with an object-shaped
author and 0 likes.

- `_fetch_video_comments`: send `url=` on both urllib and requests branches
- `_sc_fetch_transcript`: same
- Response parser: extract `author.name` when author is a dict, read
  `engagement.likes` when top-level `likes` is absent, prefer
  `publishedTime` / `publishedTimeText` for date. Legacy string-author
  and top-level-likes shapes still work, so existing mocks are unchanged.

Verified live against api.scrapecreators.com: `_fetch_video_comments`
now returns fully-populated comments with real @handles and like
counts (e.g. "@JennyNicholson: ... (49000 likes, 2025-04-15)"). All
tests in youtube_yt/normalize/signals/render pass.

Plan: docs/plans/2026-04-15-002-fix-youtube-comments-scrapecreators-param-plan.md

🤖 Generated with Claude Opus 4.6 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:17:22 -04:00
Matt Van Horn 73b4bd6ac6 fix: enforce pre-research protocol + override WebSearch Sources mandate (#266)
Restore the rich synthesis output by closing three prompt-level loopholes
that let the model silently take a degraded path:

1. Research Execution precondition gate. Steps 0.55 (entity resolution)
   and 0.75 (query planner) are now non-skippable on WebSearch platforms.
   --emit md is banned as a primary user-facing flow; --emit=compact with
   --plan is mandatory. OpenClaw --auto-resolve fallback preserved.

2. WebSearch "Sources:" mandate override. The WebSearch tool description
   contains a CRITICAL/MUST mandate to append a Sources section. That is
   explicitly superseded inside /last30days with matched-register
   CRITICAL/MANDATORY override language and a BAD/GOOD example. The
   existing web-source line is the citation; nothing appends below the
   invitation.

3. Pre-present self-check. Before displaying, the model verifies bold
   per-paragraph headlines, per-source emoji stats, quoted highlights,
   Polymarket block, coverage footer, and (critically) no trailing
   Sources block. One regeneration permitted if checks fail.

Also adds explicit MANDATORY language to the "What I learned" template
requiring bold headline phrases on every narrative paragraph.

Root cause: same-session A/B on 2026-04-15 between /last30days kanye
west (rich output, ran Steps 0.55 + 0.75, --emit=compact --plan) and
/last30days hermes ai (bland output, skipped both, --emit md) showed
the template was fine -- the model was lazily taking a shortcut SKILL.md
tolerated. No engine, render.py, or contributor PR was the cause.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 15:15:29 -04:00
Matt Van Horn a2850e3d19 fix: drop plugin.json 'skills' key to clear path-escape error on v2.1.109 (#264)
Release / build-and-release (push) Has been cancelled
plugin.json has declared "skills": ["./"] unchanged since v2.1.0. That
value used to work on older Claude Code but current versions reject it
with: Path escapes plugin directory: ./ (skills). The error surfaces
on fresh /doctor runs even after v3.0.3 restored the archive contents.

Fix: omit the "skills" key entirely. Every other plugin in the Claude
Code marketplace ecosystem (compound-engineering, coding-tutor, codex,
esper, 15+ Anthropic official plugins) omits this key and the loader
auto-discovers skills/*/SKILL.md. Matching that pattern clears the
path-escape error on v2.1.109+ and remains compatible with older
Claude Code versions where the default-discovery path was already the
working code path.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 11:40:15 -04:00
Matt Van Horn 9c1e253dcc fix(build): strip skills/ and .claude-plugin/ from .skill bundle (#263)
v3.0.3's fix (#262) restored skills/ and .claude-plugin/ to the git
archive, which Claude Code needs for /plugin install. But
scripts/build-skill.sh uses the same archive to produce the claude.ai
.skill bundle, which must contain exactly one root SKILL.md and stay
under the 200-file cap.

Fix: after git archive, 'zip -d' strips both directories from the
.skill bundle. git archive output is unchanged (Claude Code still
gets the full tarball on /plugin install).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:31:51 -04:00
Matt Van Horn f4a3cc104b fix: restore skills/ and .claude-plugin/ in plugin install tarball (#262)
Release / build-and-release (push) Has been cancelled
v3.0.1 added .gitattributes rules that excluded both directories from
git archive output, shrinking the claude.ai .skill bundle. But Claude
Code's /plugin install fetches the SAME archive, so users installing
v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no
skill files. Install appeared successful but the plugin was a useless
empty shell.

Proof:
  git archive v3.0.0 | grep 'skills/|\.claude-plugin/' | wc -l  # 8
  git archive v3.0.1 | grep 'skills/|\.claude-plugin/' | wc -l  # 0
  git archive v3.0.2 | grep 'skills/|\.claude-plugin/' | wc -l  # 0

No issue reports yet because:
 - Cached pre-v3.0.1 installs keep working (it's the new-install path
   that's broken)
 - The breakage is under 24 hours old
 - Users invoking the skill via natural language go through
   skill-selector rather than /last30days slash command

Also reverts v3.0.2's "skills": ["skills"] back to "./", the value
that shipped in every tag from v2.1.0 through v3.0.0. That change was
a misdiagnosis; the manifest wasn't in the tarball anyway so it had
no effect on user-visible installs.

Archive file count after fix: 97 (cap is 200, plenty of room).
Follow-up: move claude.ai-specific bundle exclusions into
scripts/build-skill.sh where they belong, rather than .gitattributes
which cannot distinguish between the two distribution channels.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 09:25:45 -04:00
Matt Van Horn a220632186 fix: restore /last30days slash command on Claude Code v2.1.105+ (#261)
Release / build-and-release (push) Has been cancelled
Two regressions were silently breaking /last30days for every user:

1. plugin.json declared "skills": ["./"], which newer Claude Code
   rejects with "Path escapes plugin directory: ./ (skills)". The
   skill loader refused to register the command, so /last30days
   returned "Unknown command" even though /plugin list showed the
   plugin as installed. Fix: "skills": ["skills"] so the loader
   scans the real subdirectory.

2. marketplace.json pinned "version": "3.0.0" while plugin.json
   advertised "3.0.1". The /plugin resolver used the marketplace
   version and could install a phantom user-scope copy at a stale
   SHA alongside the correct project-scope install, creating
   duplicate skill-name collisions. Both manifests now agree on
   3.0.2.

Prior attempt: commit 93fbed2 fixed (1) before but got reverted.
This lands both fixes together in a tagged release so users can
/plugin update to recover.

Recovery for affected users is in CHANGELOG.md under 3.0.2.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:40:09 -04:00
Matt Van Horn 082efe03e3 feat: surface YouTube + TikTok top comments alongside Reddit (#260)
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:26:06 -04:00
Matt Van Horn 242e38ef56 chore: ignore docs/plans/ and untrack existing plan files (#259)
Internal ce:plan output shouldn't ship on the public repo.
Adds docs/plans/ to .gitignore and removes the two already-tracked
plan files from the index. Working copies stay local for reference.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 07:48:47 -04:00
Matt Van Horn c12dd3adbf docs: mark plan 002 Units 1-4 complete; 5-10 remain 2026-04-14 17:46:01 -04:00
25 changed files with 972 additions and 994 deletions
+1 -1
View File
@@ -10,7 +10,7 @@
{ {
"name": "last30days", "name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
"version": "3.0.0", "version": "3.0.5",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
"url": "https://github.com/mvanhorn" "url": "https://github.com/mvanhorn"
+1 -2
View File
@@ -1,6 +1,6 @@
{ {
"name": "last30days", "name": "last30days",
"version": "3.0.1", "version": "3.0.5",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
@@ -11,6 +11,5 @@
"repository": "https://github.com/mvanhorn/last30days-skill", "repository": "https://github.com/mvanhorn/last30days-skill",
"license": "MIT", "license": "MIT",
"keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"], "keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"],
"skills": ["./"],
"hooks": {} "hooks": {}
} }
+4 -4
View File
@@ -16,9 +16,10 @@ docs/ export-ignore
fixtures/ export-ignore fixtures/ export-ignore
assets/ export-ignore assets/ export-ignore
# Second SKILL.md files would confuse claude.ai's uploader # NOTE: skills/ and .claude-plugin/ are NOT export-ignored here because
# (skills/last30days/ is an internal spec; skills/last30days-nux/ is a symlink) # Claude Code's /plugin install fetches this same git archive tarball.
skills/ export-ignore # Removing those from the archive (as v3.0.1 did) silently breaks installs.
# claude.ai-bundle-specific exclusions live in scripts/build-skill.sh.
# Historical + repo-only manifests # Historical + repo-only manifests
SKILL-original.md export-ignore SKILL-original.md export-ignore
@@ -35,7 +36,6 @@ uv.lock export-ignore
.agents/ export-ignore .agents/ export-ignore
.codex-plugin/ export-ignore .codex-plugin/ export-ignore
.hermes-plugin/ export-ignore .hermes-plugin/ export-ignore
.claude-plugin/ export-ignore
# CI workflows - repo-only, not needed at skill runtime # CI workflows - repo-only, not needed at skill runtime
.github/ export-ignore .github/ export-ignore
+3
View File
@@ -25,3 +25,6 @@ htmlcov/
# build artifact from scripts/build-skill.sh # build artifact from scripts/build-skill.sh
/dist/ /dist/
# Internal planning docs (ce:plan output) — keep local, don't publish
docs/plans/
+56
View File
@@ -5,6 +5,62 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [3.0.5] - 2026-04-15
### Added
- **`/last30days` slash command for plugin users.** New `commands/last30days.md` registers a Claude Code slash command. Users type `/last30days <topic>` and Claude Code's autocomplete prefix-matches it to the canonical `/last30days:last30days` form (the same way `/ce:plan` resolves to `/compound-engineering:ce-plan`). The command delegates to the existing `last30days` skill body — no skill behavior changes.
### Removed
- **`skills/last30days-nux/`** — byte-identical duplicate of root `SKILL.md` that created confusing `/last30days:last30days-nux` autocomplete entries via Claude Code's plugin namespacing. The root `SKILL.md` remains the canonical skill source.
### Recovery
```
/plugin update last30days
/reload-plugins
```
Then type `/last30days <topic>` to invoke the skill via slash command. Natural-language invocation ("search the last 30 days for X") continues to work unchanged.
## [3.0.4] - 2026-04-15
### Fixed
- **Cleared `/doctor` path-escape error on Claude Code v2.1.109+.** `.claude-plugin/plugin.json` previously declared `"skills": ["./"]`. That value shipped unchanged from v2.1.0 through v3.0.3 and worked on older Claude Code, but current versions reject `./` with `Path escapes plugin directory: ./ (skills)`. The `"skills"` key is now omitted entirely, matching the pattern used by every other plugin in the Claude Code marketplace ecosystem. Claude Code auto-discovers `skills/*/SKILL.md` when the key is absent.
### Recovery
If `/doctor` reports a path-escape error for last30days, run `/plugin update last30days` then `/reload-plugins`. If errors persist, uninstall and reinstall the plugin.
## [3.0.3] - 2026-04-15
### Fixed
- **Restored `skills/` and `.claude-plugin/` to the plugin install tarball.** v3.0.1 added `.gitattributes` rules that excluded both directories from `git archive` output to shrink the claude.ai `.skill` bundle. Claude Code's `/plugin install` fetches the same archive, so users installing v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no skill files. `git archive v3.0.0` contained 8 files under those paths; `v3.0.1` and `v3.0.2` contained 0. This release reverts those `.gitattributes` lines.
- **Reverted `plugin.json` `"skills"` field to `["./"]`.** v3.0.2 changed this to `["skills"]` based on a misdiagnosis — the manifest change had no effect because the manifest wasn't in the tarball at all. The historical `["./"]` value shipped in every release from v2.1.0 through v3.0.0 without issues and is restored here.
### Recovery
Users on v3.0.1 or v3.0.2: run `/plugin update last30days` then `/reload-plugins`. If autoUpdate is enabled, the next session start will pull v3.0.3 automatically. Users on cached v3.0.0 or earlier installs were unaffected.
### Notes
- The claude.ai `.skill` bundle built by `scripts/build-skill.sh` still works — the archive grew from 89 to 97 files, well under the 200-file cap.
- claude.ai-specific exclusions (avoiding duplicate `SKILL.md` files in the bundle) should move into `scripts/build-skill.sh` rather than `.gitattributes` in a future release, since `.gitattributes` cannot distinguish between the two distribution channels.
## [3.0.2] - 2026-04-15
### Fixed
- **`/last30days` slash command now registers on Claude Code v2.1.105+.** `.claude-plugin/plugin.json` declared `"skills": ["./"]`, which newer Claude Code rejects with `Path escapes plugin directory: ./ (skills)`. The skill silently failed to register, so `/last30days <query>` returned "Unknown command" even though `/plugin list` showed the plugin as installed. Fix: `"skills": ["skills"]` so the loader scans the real skill subdirectory.
- **Version drift between manifests.** `.claude-plugin/marketplace.json` was pinned to `3.0.0` while `.claude-plugin/plugin.json` advertised `3.0.1`. The `/plugin` resolver used the marketplace version and could install stale cached metadata alongside the correct build. Both manifests now agree on `3.0.2`.
### Recovery
If `/last30days` stopped working for you, run `/plugin update last30days` then `/reload-plugins`. If `/doctor` still reports errors, uninstall and reinstall the plugin from the marketplace.
## [3.0.1] - 2026-04-14 ## [3.0.1] - 2026-04-14
### Fixed ### Fixed
+2 -1
View File
@@ -128,7 +128,7 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
- **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works. - **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
- **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions. - **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
- **Threads, Pinterest, YouTube comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments for more. - **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does.
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock. - **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock.
- **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?" - **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
- **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run. - **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
@@ -209,6 +209,7 @@ These platforms don't have relationships with each other. X doesn't know what Re
2. **The agent resolves who matters.** Finds X handles (including founders), GitHub repos, subreddits, TikTok hashtags, YouTube channels. For "Kanye West" it knows r/hiphopheads, @kanyewest, and "bully review" on YouTube. For "OpenClaw" it resolves openclaw/openclaw on GitHub and fetches live star counts. 2. **The agent resolves who matters.** Finds X handles (including founders), GitHub repos, subreddits, TikTok hashtags, YouTube channels. For "Kanye West" it knows r/hiphopheads, @kanyewest, and "bully review" on YouTube. For "OpenClaw" it resolves openclaw/openclaw on GitHub and fetches live star counts.
3. **All sources searched in parallel.** Multi-query expansion. Results scored by engagement, relevance, freshness. 3. **All sources searched in parallel.** Multi-query expansion. Results scored by engagement, relevance, freshness.
4. **The depth nobody else has.** Full YouTube transcripts from reaction videos. Top Reddit comments with upvote counts. TikTok captions. Polymarket odds. Not just titles and links. 4. **The depth nobody else has.** Full YouTube transcripts from reaction videos. Top Reddit comments with upvote counts. TikTok captions. Polymarket odds. Not just titles and links.
Every cited source is a blue CMD-clickable link in the terminal (X handles, subreddits, publications). No raw URL strings — clean text, one click to the source.
5. **Same story, merged.** Wireless Festival announced on Reddit, discussed on X, ticket prices on TikTok = one cluster, not three separate items. 5. **Same story, merged.** Wireless Festival announced on Reddit, discussed on X, ticket prices on TikTok = one cluster, not three separate items.
6. **Synthesized into one brief.** Grounded in specific data. Cited by source. Ranked by what people actually engage with. Not "here's what I found." It's "here's what matters." 6. **Synthesized into one brief.** Grounded in specific data. Cited by source. Ranked by what people actually engage with. Not "here's what I found." It's "here's what matters."
7. **Then it becomes your expert.** After one run, your Claude session knows everything the community knows. Ask follow-up questions. Have it write prompts, draft emails, plan trips, architect systems - all grounded in what's real right now. 7. **Then it becomes your expert.** After one run, your Claude session knows everything the community knows. Ask follow-up questions. Have it write prompts, draft emails, plan trips, architect systems - all grounded in what's real right now.
+182 -52
View File
@@ -203,8 +203,8 @@ Your ScrapeCreators key powers TikTok, Instagram, Threads, Pinterest, and YouTub
**Call AskUserQuestion:** **Call AskUserQuestion:**
Question: "Which ScrapeCreators sources do you want on?" Question: "Which ScrapeCreators sources do you want on?"
Options: Options:
- "TikTok + Instagram (recommended)" - append `INCLUDE_SOURCES=tiktok,instagram` to ~/.config/last30days/.env. Confirm: "TikTok and Instagram are on, plus Reddit backup if public Reddit has issues. You can add threads, pinterest, youtube_comments to INCLUDE_SOURCES anytime." - "TikTok + Instagram (recommended)" - append `INCLUDE_SOURCES=tiktok,instagram` to ~/.config/last30days/.env. Confirm: "TikTok and Instagram are on, plus Reddit backup if public Reddit has issues. You can add threads, pinterest, youtube_comments, tiktok_comments to INCLUDE_SOURCES anytime."
- "Everything - TikTok, Instagram, Threads, Pinterest, YouTube comments" - append `INCLUDE_SOURCES=tiktok,instagram,threads,pinterest,youtube_comments` to ~/.config/last30days/.env. Confirm: "All ScrapeCreators sources are on." - "Everything - TikTok, Instagram, Threads, Pinterest, YouTube + TikTok comments" - append `INCLUDE_SOURCES=tiktok,instagram,threads,pinterest,youtube_comments,tiktok_comments` to ~/.config/last30days/.env. Confirm: "All ScrapeCreators sources are on."
- "Just the basics - let's run our first search" - don't write the flag. Confirm: "Got it. ScrapeCreators will serve as Reddit backup. You can add sources to INCLUDE_SOURCES in your .env anytime." - "Just the basics - let's run our first search" - don't write the flag. Confirm: "Got it. ScrapeCreators will serve as Reddit backup. You can add sources to INCLUDE_SOURCES in your .env anytime."
**After TikTok/Instagram opt-in (or SC skip), show the first research topic modal:** **After TikTok/Instagram opt-in (or SC skip), show the first research topic modal:**
@@ -244,7 +244,7 @@ YouTube (free, open source):
Bonus: TikTok, Instagram, Threads, Pinterest, YouTube comments (ScrapeCreators): Bonus: TikTok, Instagram, Threads, Pinterest, YouTube comments (ScrapeCreators):
- `SCRAPECREATORS_API_KEY=xxx` - 10,000 free calls at scrapecreators.com. - `SCRAPECREATORS_API_KEY=xxx` - 10,000 free calls at scrapecreators.com.
- After adding your key, set `INCLUDE_SOURCES=tiktok,instagram` to turn on the most popular ones. Add threads, pinterest, youtube_comments for more. - After adding your key, set `INCLUDE_SOURCES=tiktok,instagram` to turn on the most popular ones. Add threads, pinterest, youtube_comments, tiktok_comments for more.
GitHub Issues/PRs (free, no key needed): GitHub Issues/PRs (free, no key needed):
- If you have the `gh` CLI installed (`brew install gh`), GitHub search is automatic. No API key required. - If you have the `gh` CLI installed (`brew install gh`), GitHub search is automatic. No API key required.
@@ -290,7 +290,7 @@ Create `~/.config/last30days/.env` if it doesn't exist (check first!), pre-popul
# ScrapeCreators (10,000 free calls - scrapecreators.com): # ScrapeCreators (10,000 free calls - scrapecreators.com):
# SCRAPECREATORS_API_KEY= # Unlocks: TikTok, Instagram, Reddit backup (if public Reddit gets rate-limited) # SCRAPECREATORS_API_KEY= # Unlocks: TikTok, Instagram, Reddit backup (if public Reddit gets rate-limited)
# # Optional: add threads, pinterest, youtube_comments for more # # Optional: add threads, pinterest, youtube_comments, tiktok_comments for more
# INCLUDE_SOURCES=tiktok,instagram # INCLUDE_SOURCES=tiktok,instagram
# YouTube: install yt-dlp (brew install yt-dlp) - no key needed # YouTube: install yt-dlp (brew install yt-dlp) - no key needed
@@ -721,6 +721,20 @@ Store your plan as `QUERY_PLAN_JSON` — you'll pass it to the script in the nex
## Research Execution ## Research Execution
### PRECONDITION GATE — read before running the script
**STOP. Before invoking `last30days.py`, verify ALL of the following are true for this turn:**
1. **Platform branch chosen.** You know whether this session has WebSearch (Claude Code) or does not (OpenClaw, raw CLI, Codex without web tools).
2. **If WebSearch IS available:** you MUST have run Step 0.55 (Pre-Research Intelligence — resolved subreddits, X handles, TikTok hashtags/creators, Instagram creators, GitHub user/repo where applicable) AND Step 0.75 (Query Planner — produced `QUERY_PLAN_JSON` with 2-4 subqueries). These are NOT optional. If either was skipped, return to that step now.
3. **If WebSearch is NOT available:** you MUST add `--auto-resolve` to the command instead. Do not attempt Steps 0.55 / 0.75 without WebSearch.
4. **The command you are about to run uses `--emit=compact`.** `--emit md` is a debugging/inspection mode and is DISALLOWED as the primary user-facing flow. If you find yourself about to run `--emit md`, stop and switch to `--emit=compact`.
5. **On WebSearch platforms the command MUST include `--plan 'QUERY_PLAN_JSON'`** plus every resolved handle/subreddit/hashtag/creator flag from Step 0.55. Omit only flags whose value was not resolvable.
**Degraded path (missing any of the above on a WebSearch platform) is a known regression shape. It produces bland 4-bullet summaries instead of rich synthesis. Do not take it.**
---
**Step 1: Run the research script WITH your query plan (FOREGROUND)** **Step 1: Run the research script WITH your query plan (FOREGROUND)**
**CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.** **CRITICAL: Run this command in the FOREGROUND with a 5-minute timeout. Do NOT use run_in_background. The full output contains Reddit, X, AND YouTube data that you need to read completely.**
@@ -776,7 +790,7 @@ The script will automatically:
**Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats. **Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, **transcript highlights** (pre-extracted quotable excerpts from the video), and an optional full transcript in a collapsible section. **Quote the highlights directly in your synthesis** - they are the YouTube equivalent of Reddit top comments. Attribute quotes to the channel name. Count them and include them in your synthesis and stats block. **YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, **transcript highlights** (pre-extracted quotable excerpts from the video), and an optional full transcript in a collapsible section. **Quote the highlights directly in your synthesis.** When YouTube items also include top comments (enabled via `youtube_comments`), quote those too with their like counts — they capture how viewers reacted to the video. Transcript highlights and top comments are complementary signals; use both when present. Attribute transcript quotes to the channel name, comment quotes to the commenter. Count them and include them in your synthesis and stats block.
**TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block. **TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block.
@@ -817,9 +831,11 @@ For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge - **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script) - EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos - INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output a separate "Sources:" block** — instead, include the top 3-5 web - **DO NOT output a separate "Sources:" block** — every source (X handles, subreddits,
source names as inline links on the 🌐 Web: stats line (see stats format below). publications) is cited inline throughout the synthesis as a markdown link `[name](url)`,
The WebSearch tool requires citation; satisfy it there, not as a trailing section. and the top 3-5 web publications appear as markdown links on the 🌐 Web: stats line.
That satisfies WebSearch's citation requirement. A trailing Sources block is forbidden.
See the URL FORMATTING section below for the full link-rendering rules.
**Options** (passed through from user's command): **Options** (passed through from user's command):
- `--days=N` → Look back N days instead of 30 (e.g., `--days=7` for weekly roundup) - `--days=N` → Look back N days instead of 30 (e.g., `--days=7` for weekly roundup)
@@ -880,8 +896,8 @@ The Judge Agent must:
2. Weight YouTube sources HIGH (they have views, likes, and transcript content) 2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal) 3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
4. Weight WebSearch sources LOWER (no engagement data) 4. Weight WebSearch sources LOWER (no engagement data)
5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. Quote them directly. 5. **For Reddit, YouTube, and TikTok: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. Quote them directly, attributing to the commenter and including the vote count ("N upvotes" for Reddit, "N likes" for YouTube and TikTok). A top comment with thousands of votes is a stronger community signal than the parent post's stats alone.
6. **For YouTube: Quote transcript highlights directly.** Attribute to the channel name. 6. **For YouTube: Quote transcript highlights AND top comments.** Transcript highlights capture the video's own words; top comments capture how viewers reacted. Both add value — use them together. Attribute transcript quotes to the channel name.
7. Identify patterns that appear across ALL sources (strongest signals) 7. Identify patterns that appear across ALL sources (strongest signals)
8. Note any contradictions between sources 8. Note any contradictions between sources
9. **Multi-source clusters (items from 3+ platforms) are the strongest signals.** Lead with these. 9. **Multi-source clusters (items from 3+ platforms) are the strongest signals.** Lead with these.
@@ -1031,6 +1047,46 @@ Identify from the ACTUAL RESEARCH OUTPUT:
## THEN: Show Summary + Invite Vision ## THEN: Show Summary + Invite Vision
### OUTPUT CONTRACT — six hard rules, read all of them before writing anything
The synthesis output has six structural rules. They are a SET, not a priority order. A correct output satisfies ALL SIX on the first pass. Satisfying some by violating others is a bug, not a compromise. Each rule below has the same force as the next — do not elevate any one over the others.
**RULE A — REQUIRED: Inline markdown links on every citation.**
Every @handle, r/subreddit, publication, YouTube channel, TikTok creator, Instagram creator, and Polymarket market cited in the narrative body, KEY PATTERNS, and stats block MUST be an inline markdown link `[text](url)` at first mention. The URL comes from the raw research dump — every source item in the engine output carries its URL. First-pass synthesis MUST include these inline links; do not wait for the user to ask.
**RULE B — FORBIDDEN: Trailing list of links.**
No trailing list of links appears anywhere after the invitation. No `Sources:` block, no `References:` block, no `Further reading:` block, no `Citations:` block, no bulleted list of publication names, no closing "I have all the links... Just ask." sentence, no markdown-link list under any heading. The narrative's inline links (RULE A) ARE the source list — duplicating them into a trailing block is a bug.
**RULE C — REQUIRED: Bold headline per narrative paragraph.**
Every paragraph in "What I learned" opens with a bold headline phrase followed by a dash and the body text. Pattern: `**Headline phrase** — body text describing what people are saying...`. No plain-prose openers. No "Story 1 / Story 2 / Story 3" numbered sections. No unnumbered headings-as-pseudo-paragraphs. The bold headline IS the paragraph opener.
**RULE D — REQUIRED: Stats block uses the exact template, tree, and emoji.**
The stats block opens with the line `✅ All agents reported back!` on its own. Every active source is a line using the `├─` or `└─` tree character with its emoji prefix (🟠 Reddit, 🔵 X, 🔴 YouTube, 🎵 TikTok, 📸 Instagram, 🧵 Threads, 📌 Pinterest, 🟡 HN, 🦋 Bluesky, 🇺🇸 Truth Social, 🐙 GitHub, 📊 Polymarket, 🌐 Web, 🗣️ Top voices, 📎 Raw results). Use `│` for within-line separators. NO plain `-` bullets. NO plain "Stats" heading. NO missing emoji. Sources that returned 0 results are omitted entirely — do not include them as "0 threads" or "(no results this cycle)".
**RULE E — REQUIRED: QUERY_TYPE-specific invitation with example follow-ups.**
The closing invitation matches the detected QUERY_TYPE (PROMPTING / RECOMMENDATIONS / NEWS / COMPARISON / GENERAL) verbatim from the five variants defined below. Each variant includes 2-3 specific example follow-ups drawn from THIS run's research (not generic). A flat "I am now an expert on {TOPIC}" with no examples is a regression — it means you skipped the variant template.
**RULE F — REQUIRED: `---` horizontal-rule separators.**
Use `---` on its own line before the stats block (between the narrative/KEY PATTERNS and the `✅ All agents reported back!` line) and again between the stats block and the invitation. The separators are part of the template, not decoration.
**All six rules apply on first pass.** If you satisfy RULES A and B by stripping structure, you have violated C/D/E/F. If you satisfy C/D/E/F by skipping links, you have violated A. Satisfy all six, or the output is wrong. Not "mostly right" — wrong.
### PRE-SYNTHESIS COMMIT — do this before drafting
Before you write a single word of the synthesis, commit to every rule in the OUTPUT CONTRACT:
1. **Template (RULE E).** Identify the QUERY_TYPE for this run. Locate the matching invitation variant (PROMPTING / RECOMMENDATIONS / NEWS / COMPARISON / GENERAL) below. Commit to using that variant verbatim at the close, with 2-3 example follow-ups drawn from this specific research.
2. **Headlines (RULE C).** Commit to opening every narrative paragraph with a bold `**Headline phrase** —`. No "Story 1 / Story 2 / Story 3" or other plain-prose patterns.
3. **Stats template (RULE D).** Commit to the `✅ All agents reported back!` opening line, the `├─` / `└─` / `│` tree characters, and an emoji prefix on every active source line. Copy the template block below literally; do not rewrite it as a bullet list.
4. **Separators (RULE F).** Commit to `---` horizontal rules before and after the stats block.
5. **Sources (RULE A).** The raw research dump is in hand and includes URLs on every source item. Identify every source you intend to cite: @handles, r/subs, publications, YouTube channels, TikTok/Instagram creators, Polymarket markets. Pull URLs. Plan to wrap each at first mention as `[text](url)`. For any specific source with no URL in raw data, note the plain-text fallback.
6. **No trailing list (RULE B).** Confirm no Sources/References/bulleted-list block will appear after the invitation.
7. **Only now begin drafting.**
This commit is a forcing function, not a post-hoc checklist. Work through all seven steps before writing any prose.
---
**Display in this EXACT sequence:** **Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):** **FIRST - What I learned (based on QUERY_TYPE):**
@@ -1041,19 +1097,21 @@ Identify from the ACTUAL RESEARCH OUTPUT:
[Tool Name] - {n}x mentions [Tool Name] - {n}x mentions
Use Case: [what it does] Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com Sources: [@handle1](https://x.com/handle1), [@handle2](https://x.com/handle2), [r/sub](https://reddit.com/r/sub), [blog.com](https://blog.com/specific-post/)
[Tool Name] - {n}x mentions [Tool Name] - {n}x mentions
Use Case: [what it does] Use Case: [what it does]
Sources: @handle3, r/sub2, Complex Sources: [@handle3](https://x.com/handle3), [r/sub2](https://reddit.com/r/sub2), [Complex](https://www.complex.com/specific-article/)
Notable mentions: [other specific things with 1-2 mentions] Notable mentions: [other specific things with 1-2 mentions]
``` ```
**CRITICAL for RECOMMENDATIONS:** **CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson) - Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., [@LONGLIVE47](https://x.com/LONGLIVE47), [@ByDobson](https://x.com/ByDobson))
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety) - Include subreddit names ([r/hiphopheads](https://reddit.com/r/hiphopheads)) and web sources ([Complex](https://www.complex.com/...), [Variety](https://variety.com/...))
- Parse @handles from research output and include the highest-engagement ones - Parse @handles from research output and include the highest-engagement ones
- Every source on the "Sources:" line is a markdown link `[text](url)` — never a raw URL, never a plain name when a URL is available. The URL comes from the raw research dump.
- This per-item "Sources:" line is allowed (it's inline per recommendation). A TRAILING "Sources:" block at the end of the whole output is still forbidden — see the URL FORMATTING and "You MUST NOT" sections below.
- Format naturally - tables work well for wide terminals, stacked cards for narrow - Format naturally - tables work well for wide terminals, stacked cards for narrow
- **CRITICAL whitespace rule:** Never insert more than ONE blank line between any two content blocks. Comparison tables should immediately follow the preceding paragraph with exactly one blank line. Do NOT pad with 3-6 empty lines before tables. - **CRITICAL whitespace rule:** Never insert more than ONE blank line between any two content blocks. Comparison tables should immediately follow the preceding paragraph with exactly one blank line. Do NOT pad with 3-6 empty lines before tables.
@@ -1065,50 +1123,67 @@ CITATION RULE: Cite sources sparingly to prove research is real.
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box - Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one. - Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
CITATION PRIORITY (most to least preferred): CITATION PRIORITY (most to least preferred). Every cited name is a markdown link `[name](url)`:
1. @handles from X — "per @handle" (these prove the tool's unique value) 1. @handles from X — `per [@handle](https://x.com/handle)` (these prove the tool's unique value)
2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, prefer quoting top comments over just the thread title) 2. r/subreddits from Reddit — `per [r/subreddit](https://reddit.com/r/subreddit)` (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title)
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights) 3. YouTube channels — `per [channel name](https://youtube.com/@channel) on YouTube` (transcript-backed insights)
4. TikTok creators — "per @creator on TikTok" (viral/trending signal) 4. TikTok creators — `per [@creator](https://tiktok.com/@creator) on TikTok` (viral/trending signal)
5. Instagram creators — "per @creator on Instagram" (influencer/creator signal) 5. Instagram creators — `per [@creator](https://instagram.com/creator) on Instagram` (influencer/creator signal)
6. HN discussions — "per HN" or "per hn/username" (developer community signal) 6. HN discussions — `per [HN](https://news.ycombinator.com/item?id=N)` or `per [hn/username](https://news.ycombinator.com/user?id=username)` (developer community signal)
7. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement 7. Polymarket — `[Polymarket](https://polymarket.com/event/...) has X at Y% (up/down Z%)` with specific odds and movement
8. Web sources — ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact 8. Web sources — ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact; link the publication name: `per [Rolling Stone](https://rollingstone.com/...)`
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote. The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post. When both a web article and an X post cover the same fact, cite the X post.
URL FORMATTING: NEVER paste raw URLs anywhere in the output — not in synthesis, not in stats, not in sources. URL FORMATTING: Every citation MUST be a markdown link `[text](url)`, NEVER a raw URL string.
Claude Code renders `[text](url)` as blue CMD-clickable text — the URL is hidden, only the link text shows.
Raw `https://...` strings are forbidden everywhere: narrative, stats, KEY PATTERNS, everywhere.
Pull the URL for each source from the raw research dump (every item in the engine output carries its source URL).
If no URL is available for a source, fall back to plain text — NEVER emit a broken empty link like `[Rolling Stone]()`.
- **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/" - **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- **GOOD:** "per Rolling Stone" - **BAD:** "per Rolling Stone" (when a URL is available in the raw data — use it)
- **BAD:** "per [Rolling Stone]()" (empty link — fall back to plain text instead)
- **GOOD:** "per [Rolling Stone](https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/)"
- **GOOD (fallback, URL genuinely missing):** "per Rolling Stone"
- **BAD stats line:** `🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/...` - **BAD stats line:** `🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/...`
- **GOOD stats line:** `🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee` - **BAD stats line:** `🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee` (URLs were available — link them)
Use the publication/site name, not the URL. The user doesn't need links — they need clean, readable text. - **GOOD stats line:** `🌐 Web: 10 pages — [Later](https://later.com/blog/instagram-reels-trends/), [Buffer](https://buffer.com/resources/instagram-algorithms/), [CNN](https://www.cnn.com/2026/02/22/tech/...), [SocialBee](https://socialbee.com/blog/instagram-trends/)`
The link text is the short publication/handle/subreddit name. The URL is the deep link from the raw research — not just the bare domain. The user sees clean blue link text and can CMD-click to open the source.
**BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)." **BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
**GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per @honest30bgfan_" **GOOD:** "His album BULLY drops March 20 — fans on X are split on the tracklist, per [@honest30bgfan_](https://x.com/honest30bgfan_)"
**GOOD:** "Ye's apology got massive traction on r/hiphopheads" **GOOD:** "Ye's apology got massive traction on [r/hiphopheads](https://reddit.com/r/hiphopheads)"
**OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per Billboard" **OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per [Billboard](https://www.billboard.com/music/music-news/hellwatt-festival-2026-lineup-...)"
**Lead with people, not publications.** Start each topic with what Reddit/X **Lead with people, not publications.** Start each topic with what Reddit/X
users are saying/feeling, then add web context only if needed. The user came users are saying/feeling, then add web context only if needed. The user came
here for the conversation, not the press release. here for the conversation, not the press release.
**MANDATORY — bold headline per narrative paragraph.** Every paragraph in the "What I learned" section MUST begin with a bolded headline phrase that summarizes the paragraph, followed by a dash and the body text. Pattern: `**Headline phrase** — body text describing what people are saying...`. Without the bold headline, the output is unscannable slop. The Kanye and Matt Van Horn reference outputs follow this pattern end-to-end; bland outputs that drop the bold headline are the regression shape to avoid.
``` ```
What I learned: What I learned:
**{Topic 1}** — [1-2 sentences about what people are saying, per @handle or r/sub] **{Headline summarizing topic 1}** — [1-2 sentences about what people are saying, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Topic 2}** — [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 2}** — [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Topic 3}** — [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 3}** — [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
KEY PATTERNS from the research: KEY PATTERNS from the research:
1. [Pattern] — per @handle 1. [Pattern] — per [@handle](https://x.com/handle)
2. [Pattern] — per r/sub 2. [Pattern] — per [r/sub](https://reddit.com/r/sub)
3. [Pattern] — per @handle 3. [Pattern] — per [@handle](https://x.com/handle)
``` ```
The `@handle`, `r/sub`, publication name, etc. in these templates are placeholders — at render time each one becomes a markdown link wrapping the actual handle/sub/name, with the URL pulled from the raw research dump.
Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Europe is banning him one country at a time"), not generic ("Album release", "Tour updates").
**THEN - Quality Nudge (if present in the output):** **THEN - Quality Nudge (if present in the output):**
If the research output contains a `**🔍 Research Coverage:**` block, render it verbatim right before the stats block. This tells the user which core sources are missing and how to unlock them. Do NOT render this block if it is absent from the output (100% coverage = no nudge). If the research output contains a `**🔍 Research Coverage:**` block, render it verbatim right before the stats block. This tells the user which core sources are missing and how to unlock them. Do NOT render this block if it is absent from the output (100% coverage = no nudge).
@@ -1146,24 +1221,61 @@ Options:
├─ 🇺🇸 Truth Social: {N} posts │ {N} likes │ {N} reposts ├─ 🇺🇸 Truth Social: {N} posts │ {N} likes │ {N} reposts
├─ 🐙 GitHub: {N} items │ {N} reactions │ {N} comments ├─ 🐙 GitHub: {N} items │ {N} reactions │ {N} comments
├─ 📊 Polymarket: {N} markets │ {copy the market odds EXACTLY from the engine's Polymarket stats output - only real % numbers like "Arizona 33%, Michigan 25%". If you cannot find specific % odds in the data, show ONLY the market count with no description. NEVER write filler like "check markets", "active", "tracked", or any text without a real percentage.} ├─ 📊 Polymarket: {N} markets │ {copy the market odds EXACTLY from the engine's Polymarket stats output - only real % numbers like "Arizona 33%, Michigan 25%". If you cannot find specific % odds in the data, show ONLY the market count with no description. NEVER write filler like "check markets", "active", "tracked", or any text without a real percentage.}
├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name ├─ 🌐 Web: {N} pages — [Source Name](url), [Source Name](url), [Source Name](url)
├─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2} ├─ 🗣️ Top voices: [@{handle1}](https://x.com/{handle1}) ({N} likes), [@{handle2}](https://x.com/{handle2})[r/{sub1}](https://reddit.com/r/{sub1}), [r/{sub2}](https://reddit.com/r/{sub2})
└─ 📎 Raw results saved to ~/Documents/Last30Days/{slug}-raw.md └─ 📎 Raw results saved to ~/Documents/Last30Days/{slug}-raw.md
--- ---
``` ```
**🌐 Web: line — how to extract site names from URLs:** **🌐 Web: line — how to derive the link text and URL:**
Strip the protocol, path, and `www.` — use the recognizable publication name: Link text = the recognizable publication name (strip protocol, path, and `www.`).
- `https://later.com/blog/instagram-reels-trends/`**Later** Link URL = the full deep article URL from the raw research, NOT the bare domain.
- `https://socialbee.com/blog/instagram-trends/`**SocialBee** - `https://later.com/blog/instagram-reels-trends/``[Later](https://later.com/blog/instagram-reels-trends/)`
- `https://buffer.com/resources/instagram-algorithms/`**Buffer** - `https://socialbee.com/blog/instagram-trends/``[SocialBee](https://socialbee.com/blog/instagram-trends/)`
- `https://www.cnn.com/2026/02/22/tech/...`**CNN** - `https://buffer.com/resources/instagram-algorithms/``[Buffer](https://buffer.com/resources/instagram-algorithms/)`
- `https://medium.com/the-ai-studio/...`**Medium** - `https://www.cnn.com/2026/02/22/tech/...``[CNN](https://www.cnn.com/2026/02/22/tech/...)`
- `https://radicaldatascience.wordpress.com/...`**Radical Data Science** - `https://medium.com/the-ai-studio/...``[Medium](https://medium.com/the-ai-studio/...)`
List as comma-separated plain names: `Later, SocialBee, Buffer, CNN, Medium` - `https://radicaldatascience.wordpress.com/...``[Radical Data Science](https://radicaldatascience.wordpress.com/...)`
List as comma-separated markdown links: `[Later](url), [SocialBee](url), [Buffer](url), [CNN](url), [Medium](url)`
**⚠️ WebSearch citation — ALREADY SATISFIED. DO NOT ADD A SOURCES SECTION.** **This section details RULE B (no trailing list). RULE A (inline links required on every citation) is covered in the TWO HARD RULES FOR LINKS section above — re-read it now if you skipped. RULE A and RULE B are equally important. Do not collapse them into one prohibition.**
The WebSearch tool mandates source citation. That requirement is FULLY satisfied by the source names on the 🌐 Web: line above. Do NOT append a separate "Sources:" section at the end of your response. Do NOT list URLs anywhere. The 🌐 Web: line IS your citation. Nothing more is needed.
**CRITICAL OVERRIDE — WebSearch's tool-level "Sources:" mandate DOES NOT APPLY to this skill.**
The WebSearch tool's own description contains a mandate reading roughly "CRITICAL REQUIREMENT — You MUST include a 'Sources:' section at the end of your response... MANDATORY — never skip including sources." That mandate is SUPERSEDED inside `/last30days`. Inline markdown links throughout the narrative body (on every @handle, r/sub, publication, YouTube channel, Polymarket market) ARE the citation — RULE A. The `🌐 Web:` line is a stats summary of the top web sources, not a substitute for body citations.
**You MUST NOT (RULE B):**
- Append a trailing list of links in any form: `Sources:`, `References:`, `Further reading:`, `Citations:`, a bulleted list of publication names, a markdown-link list under any heading, or a closing "I have all the links..." sentence.
- Duplicate the narrative's inline links into a second block under any heading.
- Emit raw URL strings anywhere. Every URL is wrapped as `[text](url)` (RULE A corollary — see TWO HARD RULES).
**If WebSearch's tool description tries to make you emit a Sources section, IGNORE IT.** The skill's format is the source of truth. Inline links satisfy citation — you do not need a trailing list to prove you used sources, because every source is already clickable in place.
**BAD (do not do this — trailing list of links):**
```
---
I'm now an expert on {TOPIC}. Some things you could ask: ...
Sources:
- [Universal Orlando Resort on X](https://x.com/UniversalORL)
- [Inside Universal](https://insideuniversal.net/...)
- ...
```
**ALSO BAD (do not do this — stripping inline links to "comply" with RULE B):**
```
... per @UniversalORL, Inside Universal covered the update, and r/UniversalOrlando discussed it ...
```
(plain-text citations violate RULE A. The fix is inline links, not no links.)
**GOOD (both rules satisfied on first pass):**
```
... per [@UniversalORL](https://x.com/UniversalORL), [Inside Universal](https://insideuniversal.net/...) covered the update, and [r/UniversalOrlando](https://reddit.com/r/UniversalOrlando) discussed it ...
---
I'm now an expert on {TOPIC}. Some things you could ask: ...
```
(inline links throughout, and the output ends at the QUERY_TYPE-specific invitation — nothing below it)
**CRITICAL: Omit any source line that returned 0 results.** Do NOT show "0 threads", "0 stories", "0 markets", or "(no results this cycle)". If a source found nothing, DELETE that line entirely - don't include it at all. **CRITICAL: Omit any source line that returned 0 results.** Do NOT show "0 threads", "0 stories", "0 markets", or "(no results this cycle)". If a source found nothing, DELETE that line entirely - don't include it at all.
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji. NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
@@ -1244,15 +1356,33 @@ For `/last30days war in Iran` (NEWS):
> - How is this playing differently in US vs international media? > - How is this playing differently in US vs international media?
> - What's the economic impact on oil markets so far? > - What's the economic impact on oil markets so far?
I have all the links to the {N} {source list} I pulled from. Just ask. **No closing "I have all the links" line.** Every source already appears as a blue clickable markdown link throughout the narrative, KEY PATTERNS, and stats block — CMD-click any of them to open. The output ends at the QUERY_TYPE-specific invitation above. Do not append a trailing "I have all the links..." sentence, a Sources section, or any bulleted source list.
**Context-aware:** Only list sources that returned results. Build the source list from your stats: e.g. "14 Reddit threads, 22 X posts, and 6 YouTube videos" or "8 HN stories and 3 Polymarket markets." Never mention a source with 0 results. ---
## PRE-PRESENT SELF-CHECK — run before displaying the synthesis
**Before you display the synthesis to the user, verify ALL of the following. If any check fails AND the underlying data supports fixing it, regenerate the synthesis ONCE with the missing elements. If the data itself is absent (e.g., no Polymarket markets on this topic), skip that check silently.**
1. **RULE C — bold headlines count.** Count `**` bold-headline openers in "What I learned". Expect at least 3 (one per narrative paragraph). If 0 or 1, the agent wrote plain-prose "Story 1/2/3"-style sections instead — regenerate with `**Headline phrase** — body` on every paragraph.
2. **RULE D — stats block tree and emoji count.** The stats block must open with `✅ All agents reported back!`. Count `├─` occurrences — expect at least 3. Count emoji-prefixed source lines (🟠🔵🔴🎵📸🧵📌🟡🦋🇺🇸🐙📊🌐🗣️📎) — expect one per active source. If the stats block is a plain "Stats" heading with `-` bullets or is missing the ✅ line, regenerate with the template block copied literally.
3. **Quoted highlights where evidence supports them.** For YouTube items with transcripts and Reddit/X items with fun/highlight quotes, at least 2 verbatim quotes appear in the synthesis. Attributed to the channel/commenter/subreddit.
4. **Polymarket block present if markets were returned.** If the engine surfaced Polymarket markets, the synthesis includes specific percentages and directional movement. If no markets were surfaced, skip.
5. **RULE F — `---` separators present.** Count `---` horizontal rules. Expect at least one before the stats block (between narrative/KEY PATTERNS and the ✅ line) and one between the stats block and the invitation. If zero, add them.
6. **RULE E — QUERY_TYPE-specific invitation.** The closing block matches one of the five variants (PROMPTING / RECOMMENDATIONS / NEWS / COMPARISON / GENERAL) and includes 2-3 example follow-ups drawn from this specific run. A flat "I am now an expert on {TOPIC}" with no examples is a regression — regenerate with the correct variant.
7. **RULE B — no trailing list of links.** The output ends at the QUERY_TYPE-specific invitation. Nothing below it. Not a `Sources:`, not a `References:`, not `Further reading:`, not `Citations:`, not any bulleted list of URLs or publication names, not a markdown-link list under any heading, not a closing "I have all the links..." sentence. Every source is already a blue CMD-clickable inline link in the narrative and stats — no trailing list is needed.
8. **RULE A — inline links count.** Count the `[text](url)` markdown links in the narrative body and KEY PATTERNS. If the raw research had URLs for cited sources (it almost always does — every engine item carries a URL) and the narrative has ZERO inline links, regenerate WITH inline links on every @handle, r/sub, publication, and Polymarket market at first mention. Stripping links does NOT satisfy RULE B, it violates RULE A.
9. **No raw URL strings anywhere (RULE A corollary).** Scan the full output for `http://` or `https://` substrings. If any appear outside a markdown link `[text](url)`, regenerate with the URL wrapped as a link.
10. **Every citation is a markdown link or a clean plain-text fallback.** No broken empty links like `[Rolling Stone]()` or `[@handle]()`. If a URL was available in the raw data, it's wrapped; if genuinely missing for a specific source, that source's name appears as plain text — but the default for every source is a markdown link.
11. **Research protocol was followed.** On WebSearch platforms, the command you ran used `--emit=compact --plan 'QUERY_PLAN_JSON'` with resolved handles/subreddits/hashtags. If you took the degraded path (`--emit md`, no plan, no flags), the synthesis will almost certainly fail checks 1-3 — regenerate by returning to Step 0.55 and running the full protocol.
**Max ONE regeneration.** If the regenerated output still fails the self-check, display the best version you have and note to the user which check(s) the data could not satisfy, so they can re-run or adjust their query.
--- ---
## WAIT FOR USER'S RESPONSE ## WAIT FOR USER'S RESPONSE
**STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`. **STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. Do NOT append a `Sources:` section (see override above — WebSearch's mandate does not apply here). The research script already saved raw data to `~/Documents/Last30Days/` via `--save-dir`.
--- ---
+9
View File
@@ -0,0 +1,9 @@
---
description: Research what people actually say about any topic in the last 30 days across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web.
argument-hint: <topic> — e.g. "nvidia earnings reaction" or "best noise cancelling headphones"
allowed-tools: [Bash, Read, Write, AskUserQuestion, WebSearch]
---
Invoke the `last30days` skill with the user's arguments: $ARGUMENTS
Use the skill's canonical pipeline (plan → retrieve → normalize → fuse → rerank → cluster → render). If the user provided no arguments, ask them for a topic before proceeding.
@@ -1,445 +0,0 @@
---
title: Fix skill upload 200-file limit + packaging hygiene (public repo)
type: fix
status: completed
date: 2026-04-14
deepened: 2026-04-14
---
# Fix skill upload 200-file limit + packaging hygiene (public repo)
## Overview
claude.ai's "Upload skill" UI rejects zips with more than 200 files. Zipping the public `mvanhorn/last30days-skill` repo produces 406 files, so the upload fails outright (evidence: Trevin's 2026-04-14 report). Root cause is an accidentally committed npm package under `vendor/` (215 files of dead weight from PR #48) plus the absence of a user-facing packaging path that matches Anthropic's canonical `.skill` format.
Goal: let any user produce a compliant `last30days.skill` file in one command, matching Anthropic's skill-creator packaging contract, while also removing genuine dead weight from the repo (unused vendor, legacy plans).
## Problem Frame
- Trevin tried to upload the public repo as a Claude Skill and hit the 200-file limit
- 215 of 406 files are `vendor/package/` - an extracted `steipete-bird-0.8.0.tgz` that no code imports
- The real runtime X client lives at `scripts/lib/vendor/bird-search/` (15 files, referenced by `scripts/lib/bird_x.py:5` and `tests/test_bird_x.py:133`)
- `.clawhubignore` is ClawHub-specific and does not affect a hand-rolled zip upload
- Users have no documented path to produce a compliant upload zip
- Legacy top-level `plans/` folder holds pre-`docs/plans/` planning artifacts (confirmed waste by Matt, 2026-04-14 chat)
## Requirements Trace
- R1. After this plan lands, the produced upload zip is =200 files
- R2. The X/bird-search runtime still works - no regression in `tests/test_bird_x.py`
- R3. A contributor following README instructions can produce a Claude-Skill-upload-compatible `.skill` file in one command
- R4. Re-introduction of a root `vendor/` directory is prevented via `.gitignore`
- R5. No runtime behavior changes for existing skill consumers (Claude Code plugin, ClawHub, Gemini)
- R6. Produced zip matches Anthropic's canonical skill-folder layout: top-level directory named exactly `last30days` containing `SKILL.md` at its root, with YAML frontmatter `name: last30days`
- R7. Root `SKILL.md` frontmatter passes Anthropic's documented limits: `name` =64 chars (currently 10), `description` =200 chars (currently 228, needs trimming)
- R8. Produced zip contains exactly one `SKILL.md` (at `last30days/SKILL.md`) - no conflicting second skill spec, no symlinks that the uploader may reject or break
- R9. No runtime import reaches an excluded path (proven by import-graph audit, not just asserted)
## Scope Boundaries
Non-goals:
- Not touching the private repo or ClawHub publish flow (those have their own strip script)
- Not resolving the adjacent open issues (#239 plugin loader path-escape, #236 OpenClaw paths, #231 security scan, #190 version drift, #184 Gemini install) - each deserves its own plan
- Not redesigning the skill into self-contained subfolders or splitting scripts into a separate package
- Not adding CI enforcement of the 200-file cap (possible follow-up)
## Context and Research
### Anthropic's canonical skill-upload contract
Sourced from Anthropic's skill-creator repo (`anthropics/skills/skills/skill-creator/scripts/package_skill.py`) and help-center docs:
1. **Output format:** a `.skill` file, which is a standard zip with the `.skill` extension.
2. **Top-level entry in the zip must be a single directory** whose name matches `name:` in the skill's YAML frontmatter. Anthropic's packager uses `arcname = file_path.relative_to(skill_path.parent)`, so the zip always contains `<skill_name>/...`.
3. **That directory must contain `SKILL.md`** at its root (the packager explicitly validates this).
4. **Required YAML frontmatter:** `name` (=64 chars, lowercase + hyphens) and `description` (=200 chars). Our root SKILL.md already satisfies both.
5. **Canonical exclusions** applied by Anthropic's packager:
- Directories: `__pycache__`, `node_modules`
- Root-only: `evals/`
- File globs: `*.pyc`
- Files: `.DS_Store`
6. **Empirical limit:** the upload UI rejects =200 files (screenshot 2026-04-14). Not documented, but confirmed.
7. **Per-file size cap** is not publicly documented; general claude.ai uploads cap at 30MB per file. Conservative target: keep any single file under 10MB.
### Relevant code and patterns in this repo
- `SKILL.md` (root, 1382 lines, 80KB) - `name: last30days`, `user-invocable: true`. This is the skill.
- `skills/last30days/SKILL.md` (230 lines) - `name: last30days-v3-spec`, `user-invocable: false`. Internal architecture spec, separate skill name - not the upload target.
- `vendor/package/` - accidental commit from PR #48, 215 files, zero importers.
- `vendor/steipete-bird-0.8.0.tgz` - source tarball, also unused at runtime.
- `scripts/lib/vendor/bird-search/` - the ACTUAL vendored bird-search client (15 files). Keep.
- `plans/` (top-level, 2 files: `feat-add-websearch-source.md`, `fix-strict-date-filtering.md`) - legacy, pre-`docs/plans/` convention. Matt confirmed delete.
- `scripts/sync.sh` - deploys skill to `~/.claude`, `~/.agents`, `~/.codex`. Reference for runtime-required files.
- `.clawhubignore` - existing exclude list for the ClawHub path. Not used here, but good cross-reference for what is runtime-irrelevant.
- `.gitignore` - current dev excludes (`.venv/`, `__pycache__/`, `.DS_Store`, etc).
### Institutional learnings
- Private repo has `scripts/clawhub-publish.sh` + `scripts/strip_for_openclaw.py` that build a staging dir with only OpenClaw-safe files. Not needed for this public-path upload; `git archive` with `--prefix` is sufficient and dependency-free.
- PR #48 introduced `vendor/package/` unintentionally. No code imports from it.
### File count math (verified via dry run)
| Strategy | File count | Under cap? |
|---|---|---|
| Current repo, zip as-is | 406 | No |
| After `vendor/` deleted | 191 | Yes (thin margin) |
| After `vendor/` + `plans/` deleted, no further excludes | 189 | Yes |
| With full planned excludes (Anthropic canonical + tests/docs/fixtures/assets/dev manifests/nested skill dirs) | 81 | Comfortable headroom |
Dry run run on 2026-04-14 against the current working tree. Simulated the proposed `.gitattributes` with a `find` filter matching the intended exclude list. Result: 81 files, 868KB uncompressed. Actual `git archive` output may differ slightly (by 1-2 files) but will land well under 200.
### Runtime import audit (proves core experience unchanged)
Grepped all `import`/`from` statements in `scripts/**/*.py`. Non-stdlib imports resolve to only:
- `lib.*` (internal package at `scripts/lib/`)
- `store` (internal module at `scripts/store.py`)
- `scripts.*` (internal)
No runtime import reaches `tests/`, `docs/`, `fixtures/`, `vendor/` (root), `plans/`, `assets/`, `.agents/`, `.codex-plugin/`, `.hermes-plugin/`, or any other excluded path. The shipped `.skill` file contains everything the runtime needs and nothing it does not.
### Symlink and multi-SKILL.md audit
The repo contains one symlink: `skills/last30days-nux/SKILL.md -> ../../SKILL.md`. Three SKILL.md files in total:
- `SKILL.md` (root, `name: last30days`, `user-invocable: true`) - the actual skill
- `skills/last30days/SKILL.md` (`name: last30days-v3-spec`, `user-invocable: false`) - internal architecture doc
- `skills/last30days-nux/SKILL.md` (symlink to root) - nux variant reference
Shipping all three inside one zip creates two rejection risks:
1. Uploader sees multiple `SKILL.md` with conflicting `name:` values and refuses or misbinds
2. `git archive` stores the symlink as a symlink entry; the uploader may reject symlinked entries on principle
Both risks disappear by excluding `skills/` entirely from the zip. The two internal skill definitions are not needed for claude.ai skill execution - they serve the repo as documentation / Claude Code plugin layout, not the direct upload path.
### Sources consulted
- Anthropic skill help center article (general upload guidance, no file-count number documented)
- [anthropics/skills README](https://github.com/anthropics/skills/blob/main/README.md) - YAML frontmatter requirements
- [anthropics/skills package_skill.py](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/package_skill.py) - canonical exclusions and arcname shape
- Trevin's 2026-04-14 chat screenshot (empirical 200-file cap)
- Adjacent issues #239, #236, #190 for context on current packaging mess
## Key Technical Decisions
- **Delete `vendor/` outright** rather than gitignore-and-leave. Pure dead weight. Rationale: the real vendored client is at `scripts/lib/vendor/bird-search/`, root `vendor/` has zero importers; keeping it invites re-upload.
- **Delete top-level `plans/`** (Matt confirmed). Rationale: superseded by `docs/plans/`. Moving content into `docs/plans/` if any is still relevant; otherwise just delete.
- **Produce a `.skill` file (not a plain `.zip`)** via `git archive --format=zip --prefix=last30days/ -o dist/last30days.skill HEAD`. Rationale: matches Anthropic's canonical contract - zip extension is cosmetic, but the `.skill` affordance is what the upload UI expects.
- **Use `git archive` + `.gitattributes export-ignore`** rather than a Python packager. Rationale: no Python dependency at build time, honors git's declarative exclude model, reusable by anyone running `git archive` directly.
- **Mirror Anthropic's canonical exclusions in `.gitattributes`** (`__pycache__`, `node_modules`, `*.pyc`, `.DS_Store`, `evals/`) alongside our repo-specific excludes. Rationale: future-proof if a contributor adds node deps; keeps us aligned with the Anthropic baseline.
- **Exclude `skills/` from the upload zip** (covers `skills/last30days/SKILL.md` and `skills/last30days-nux/SKILL.md`). Rationale: shipping multiple SKILL.md files with different `name:` values is a likely uploader-rejection cause, and the symlink at `skills/last30days-nux/SKILL.md` is an independent rejection risk. Repo contents stay intact - Claude Code plugin and GitHub viewers still see the directory.
- **Keep `.clawhubignore` as-is** - it serves the ClawHub publish path separately. Do not merge the two lists; different consumers, different exclusions.
- **Prevent regression with a `/vendor/` entry in `.gitignore`** (leading slash, so `scripts/lib/vendor/` is unaffected).
- **Do not address #239 `"skills": ["./"]` path-escape here.** That is a plugin.json change, not a zip-packaging change. Separate plan.
## Open Questions
### Resolved during planning
- Is root `vendor/` used? No. Grep for `vendor/package`, `vendor/steipete`, `from vendor` returns zero hits outside `scripts/lib/vendor/`.
- Is `scripts/lib/vendor/bird-search/` safe? Yes. Referenced by `scripts/lib/bird_x.py:5` and `tests/test_bird_x.py:133`.
- What name does the top-level zip directory need? `last30days` - matches `name: last30days` in the root `SKILL.md` frontmatter.
- Does `skills/last30days/SKILL.md` conflict? No. It declares a different skill name (`last30days-v3-spec`) and is `user-invocable: false`. Not the upload target, and safe to ship inside the zip.
- Is there a documented file-count cap? No. 200 is empirical from the UI error screenshot.
- Should we gate this on a version bump? Yes, 3.0.0 - 3.0.1. Same API, same runtime, smaller and uploadable package.
### Deferred to implementation
- Exact `.gitattributes` export-ignore entries may need one tuning pass if `git archive` surfaces a file we forgot. Verification step catches it.
- Whether to delete `SKILL-original.md` from the repo entirely or just export-ignore. Leaning export-ignore to preserve git history context.
- Whether any content in `plans/*.md` is still live reference material. If so, move to `docs/plans/` under new naming convention; if not, delete outright.
## Implementation Units
- [ ] **Unit 1: Remove accidental `vendor/` commit**
**Goal:** Delete the root `vendor/` directory and the stray `.tgz`, both unused at runtime.
**Requirements:** R1, R2, R5
**Dependencies:** None
**Files:**
- Delete: `vendor/` (entire tree, 215 files)
- Delete: `vendor/steipete-bird-0.8.0.tgz`
- Modify: `.gitignore` (add `/vendor/` to prevent regression - leading slash to avoid matching `scripts/lib/vendor/`)
**Approach:**
- Single commit: `chore: remove unused root vendor/ directory (215 files from PR #48)`
- Verify `scripts/lib/vendor/bird-search/` is untouched
- Verify no `from vendor` or `vendor/package` references appear in the diff
**Patterns to follow:**
- Commit message style matches recent history
**Test scenarios:**
- Happy path: `find . -type f -not -path './.git/*' | wc -l` returns =200 after commit
- Integration: `python -m pytest tests/test_bird_x.py -q` passes - confirms the real vendored client still resolves
- Integration: `bash scripts/sync.sh` completes without error
**Verification:**
- Zero files remain under `vendor/` on `main`
- `tests/test_bird_x.py` still passes
- `.gitignore` now contains `/vendor/`
- [ ] **Unit 2: Remove legacy top-level `plans/` directory**
**Goal:** Delete the pre-`docs/plans/` folder (Matt confirmed waste).
**Requirements:** R1, R5
**Dependencies:** None (independent of Unit 1)
**Files:**
- Delete: `plans/feat-add-websearch-source.md`
- Delete: `plans/fix-strict-date-filtering.md`
- Delete: `plans/` (now empty)
**Approach:**
- Skim both files first. If either still reflects real upcoming work, port it to `docs/plans/YYYY-MM-DD-NNN-<type>-*-plan.md` before deletion. If not, delete.
- Commit: `chore: remove legacy plans/ directory (superseded by docs/plans/)`
**Test scenarios:**
- Test expectation: none - pure housekeeping, no code paths affected
**Verification:**
- `plans/` does not exist on `main`
- Nothing in the repo references `plans/feat-add-websearch-source.md` or `plans/fix-strict-date-filtering.md` (grep to confirm)
- [ ] **Unit 3: Declare zip-time excludes via `.gitattributes`**
**Goal:** Use `export-ignore` so `git archive` produces a skill-shaped zip without hand-filtering.
**Requirements:** R1, R3, R6
**Dependencies:** Unit 1, Unit 2
**Files:**
- Create: `.gitattributes`
**Approach:**
- Anthropic canonical exclusions (match `package_skill.py`):
- `__pycache__/` export-ignore
- `node_modules/` export-ignore
- `*.pyc` export-ignore
- `.DS_Store` export-ignore
- `evals/` export-ignore
- Repo-specific exclusions (dev/docs/build artifacts not needed at runtime):
- `tests/` (64 files)
- `docs/` (17 files including `docs/test-results/`)
- `fixtures/` (7 files)
- `assets/` (5 files, 14MB of README media)
- `SKILL-original.md` (historical)
- `SPEC.md`, `TASKS.md`, `test-run.log`, `CONTRIBUTORS.md`, `HERMES_SETUP.md`, `release-notes.md`, `CHANGELOG.md`
- `uv.lock`
- `.agents/`, `.codex-plugin/`, `.hermes-plugin/`, `.claude-plugin/` (platform adapters - skill-upload path is platform-agnostic)
- `.clawhubignore`, `.gitignore`, `.gitattributes`
- `skills/` (avoid second SKILL.md with conflicting `name:`; also drops the symlink at `skills/last30days-nux/SKILL.md`)
- Keep in archive: `scripts/` (runtime), root `SKILL.md`, `README.md`, `LICENSE`, `pyproject.toml`, `CLAUDE.md`, `gemini-extension.json`, `agents/`, `hooks/`
**Technical design:** *(directional guidance, not implementation spec)*
```gitattributes
# Anthropic canonical skill-packaging excludes
__pycache__/ export-ignore
node_modules/ export-ignore
*.pyc export-ignore
.DS_Store export-ignore
evals/ export-ignore
# Repo-specific: tests + docs + media (not runtime)
tests/ export-ignore
docs/ export-ignore
fixtures/ export-ignore
assets/ export-ignore
# Repo-specific: historical + dev manifests
SKILL-original.md export-ignore
SPEC.md export-ignore
...
```
**Patterns to follow:**
- `.gitattributes` export-ignore syntax per [git docs](https://git-scm.com/docs/gitattributes#_creating_an_archive)
**Test scenarios:**
- Happy path: `git archive --format=zip HEAD | zipinfo -1 - | wc -l` returns =200
- Happy path: zip contains `SKILL.md`, `scripts/last30days.py`, `scripts/lib/bird_x.py`, `scripts/lib/vendor/bird-search/lib/cookies.js`
- Happy path: zip contains exactly one `SKILL.md` entry at the top level (not multiple, not a symlink)
- Edge case: zip does NOT contain `tests/`, `docs/`, `assets/*.jpeg`, `*.mp3`, `skills/`
- Edge case: no symlink entries in the zip (`unzip -l` lines starting with `l`)
- Edge case: zip size stays under ~2MB (if over 5MB an unintended large file slipped through)
**Verification:**
- Running `git archive --format=zip --output=/tmp/test.zip HEAD && unzip -l /tmp/test.zip | tail -1` reports =200 files and a sane byte count
- [ ] **Unit 4: Add `scripts/build-skill.sh` user-facing builder**
**Goal:** One-command path to produce a Claude-upload-compatible `.skill` file.
**Requirements:** R3, R6
**Dependencies:** Unit 3
**Files:**
- Create: `scripts/build-skill.sh`
- Modify: `.gitignore` (add `/dist/` for build artifact)
**Approach:**
- Bash, executable, `set -euo pipefail`
- `git archive --format=zip --prefix=last30days/ --output=dist/last30days.skill HEAD`
- The `--prefix=last30days/` nests everything under `last30days/` inside the zip, matching Anthropic's arcname contract
- Refuse to build if working tree is dirty (`git diff --quiet && git diff --cached --quiet`)
- Print file count, archive size, and path to paste into the upload UI
- Fail with a clear error if count exceeds 200 (defensive check)
**Technical design:** *(directional guidance, not implementation spec)*
```bash
#!/usr/bin/env bash
# build-skill.sh - package repo as a Claude-upload-ready .skill file
# Usage: bash scripts/build-skill.sh
set -euo pipefail
if ! git diff --quiet || ! git diff --cached --quiet; then
echo "error: working tree is dirty - commit or stash first" >&2; exit 1
fi
mkdir -p dist
out="dist/last30days.skill"
git archive --format=zip --prefix=last30days/ --output="$out" HEAD
count=$(unzip -l "$out" | tail -1 | awk '{print $2}')
[ "$count" -le 200 ] || { echo "error: $count files in zip, cap is 200" >&2; exit 1; }
echo "built $out ($count files, $(du -h "$out" | cut -f1))"
```
**Patterns to follow:**
- Style of `scripts/sync.sh` (bash, top-of-file comment, `set -euo pipefail`)
**Test scenarios:**
- Happy path: clean tree, `bash scripts/build-skill.sh` produces `dist/last30days.skill` with =200 files and the top-level entry is `last30days/`
- Happy path: `unzip -p dist/last30days.skill last30days/SKILL.md | head -2` shows `---` (frontmatter start) confirming SKILL.md is at the right location
- Edge case: dirty working tree - script exits non-zero with clear error
- Edge case: idempotent - running twice overwrites cleanly
- Error path: if a future change inflates file count past 200, the defensive `[ "$count" -le 200 ]` check fails and the script refuses to produce a broken output
**Verification:**
- `bash scripts/build-skill.sh && unzip -l dist/last30days.skill | grep "^ 0 .* last30days/$"` confirms the prefix directory exists
- `unzip -l dist/last30days.skill | grep "last30days/SKILL.md"` confirms SKILL.md is at the expected path
- `unzip -l dist/last30days.skill | grep -c "SKILL.md"` returns exactly 1
- `unzip -l dist/last30days.skill | awk '{print $NF}' | grep -v "^$" | sort -u | grep "skills/" || true` returns nothing (confirms internal skill dirs excluded)
- Gate: a contributor must run `bash scripts/build-skill.sh` on their branch and attach the produced file to their PR before merging any change that touches `.gitattributes` or exclude-sensitive paths
- [ ] **Unit 5: Document the upload path in README**
**Goal:** Users know how to produce an upload `.skill` without reading the source.
**Requirements:** R3
**Dependencies:** Unit 4
**Files:**
- Modify: `README.md` (add a short "Upload as a Claude Skill" subsection under the existing install section)
**Approach:**
- One paragraph plus a single command block: `bash scripts/build-skill.sh`
- Mention the 200-file cap as context so future changes do not bust it
- Point users at the claude.ai skill upload UI (note: link only if a stable URL exists at implementation time, otherwise describe the UI path)
**Test scenarios:**
- Test expectation: none - pure documentation change
**Verification:**
- `grep -n "build-skill" README.md` returns a hit
- Instructions match actual script behavior
- [ ] **Unit 6: Trim SKILL.md description to =200 chars**
**Goal:** Make root `SKILL.md` frontmatter pass Anthropic's documented `description` limit.
**Requirements:** R7
**Dependencies:** None (independent of other units)
**Files:**
- Modify: `SKILL.md` (frontmatter `description:` field only)
**Approach:**
- Current description is 228 chars. Cut 28+ chars without losing signal.
- Suggested rewrite (196 chars): `"Multi-query social search with planned queries. Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Gemini/OpenAI fallback when needed."`
- Confirm the trimmed version still surfaces for the right prompts (smoke test: run `python scripts/last30days.py "test" --emit=compact` and confirm behavior unchanged; description is metadata, not runtime input)
- Update `skills/last30days/SKILL.md` description too if it exceeds 200 chars (check during implementation)
**Test scenarios:**
- Happy path: `python3 -c "import re; d=open('SKILL.md').read(); m=re.search(r'^description:\s*\"(.+?)\"', d, re.M); assert len(m.group(1)) <= 200, len(m.group(1))"` passes
**Verification:**
- Description field is =200 chars in root SKILL.md
- Skill still triggers on relevant prompts (manual smoke check)
- [ ] **Unit 7: Version bump and changelog**
**Goal:** Ship as 3.0.1 so consumers see the packaging fix.
**Requirements:** R5
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json` (3.0.0 - 3.0.1)
- Modify: `SKILL.md` frontmatter version
- Modify: `skills/last30days/SKILL.md` frontmatter version
- Modify: `gemini-extension.json` version (note: #190 flags this as stale at 2.9.5; bumping here partially addresses that but full resolution is out of scope)
- Modify: `CHANGELOG.md`
- Modify: `release-notes.md`
**Approach:**
- Atomic version bump across all manifests
- Changelog entry: "Packaging: `scripts/build-skill.sh` produces a compliant `.skill` file; removed unused root `vendor/` (215 files) and legacy `plans/`; repo file count fits under claude.ai's 200-file upload cap"
**Test scenarios:**
- Happy path: `grep -rn "3.0.1" SKILL.md skills/last30days/SKILL.md .claude-plugin/plugin.json gemini-extension.json` returns four consistent hits
- Integration: `bash scripts/sync.sh` completes cleanly
**Verification:**
- All four version declarations read `3.0.1`
- CHANGELOG and release-notes have dated entries
## System-Wide Impact
- **Interaction graph:** Skill-runtime import graph is unchanged. Removed code (root `vendor/`, `plans/`) has zero importers.
- **Error propagation:** `build-skill.sh` is a new surface; failure mode is non-zero exit with clear stderr. No runtime error paths touched.
- **State lifecycle risks:** None. `dist/` is gitignored build output.
- **API surface parity:** No change to any user-facing API, CLI flag, config key, or SKILL.md contract.
- **Integration coverage:** `tests/test_bird_x.py` exercises the real vendored client - if it regressed, the test fails. Run it after Unit 1.
- **Unchanged invariants:** `scripts/lib/vendor/bird-search/` stays. `scripts/sync.sh` deploy behavior unchanged. ClawHub publish flow (private repo) untouched. Claude Code plugin install via GitHub URL still works.
## Risks and Dependencies
| Risk | Mitigation |
|------|------------|
| Deleting `vendor/` silently breaks something we missed | Run `pytest tests/test_bird_x.py` and `bash scripts/sync.sh` after the delete; grep for `vendor/package` before merging |
| claude.ai rejects the `.skill` file for a reason other than file count (e.g., frontmatter character, hidden file) | Test-upload the produced artifact against claude.ai once before merging; iterate on `.gitattributes` if needed |
| `.gitattributes` over-excludes and breaks the runtime skill | Unit 3 verification step explicitly checks runtime paths are present in the produced archive |
| A future PR re-vendors something at `/vendor/` and busts the 200 cap again | `/vendor/` in `.gitignore` plus the defensive `=200` check in `build-skill.sh` catches it |
| Version bump collides with in-flight PRs that also bump versions | Coordinate with #229, #217 which touched version strings; check before merging |
| `skills/last30days/SKILL.md` (internal spec) being shipped inside the zip confuses the claude.ai uploader | Resolved by excluding `skills/` from the zip (Unit 3). Internal spec remains in the repo for plugin consumers |
| `skills/last30days-nux/SKILL.md` is a symlink to `../../SKILL.md`; claude.ai may reject zips with symlink entries | Resolved by excluding `skills/` from the zip (Unit 3). Symlink never enters the archive |
## Documentation and Operational Notes
- Update README only (Unit 5). No runbook, no migration, no flag.
- No deployment step - plugin consumers get the packaging fix automatically on next update.
- Release notes flag: manual uploaders should re-zip via `scripts/build-skill.sh`.
- Opportunistic future work (out of scope here): CI check that fails PRs that push the zip over 200 files.
## Sources and References
- Trevin's 2026-04-14 chat screenshot: "Zip contains too many files (maximum 200)"
- [anthropics/skills README](https://github.com/anthropics/skills/blob/main/README.md) - YAML frontmatter requirements
- [anthropics/skills package_skill.py](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/package_skill.py) - canonical exclusions, arcname convention, validation gates
- [claude.ai skill help center](https://support.claude.com/en/articles/12512180-use-skills-in-claude) - upload failure modes (zip size, folder-name mismatch, missing SKILL.md)
- PR #48 (2026-02) - the merge that introduced `vendor/package/`
- Open issues adjacent but out of scope: #239, #236, #231, #190, #184
- Related code: `scripts/lib/bird_x.py:5`, `tests/test_bird_x.py:133`, `.clawhubignore`, `scripts/sync.sh`, root `SKILL.md` frontmatter
- Private-repo reference pattern: `scripts/clawhub-publish.sh` + `scripts/strip_for_openclaw.py` - not copied here; `git archive` is simpler for the public path
@@ -1,459 +0,0 @@
---
title: claude.ai distribution + discoverability push
type: feat
status: active
date: 2026-04-14
---
# claude.ai distribution + discoverability push
## Overview
The 200-file upload bug is fixed and `last30days.skill` works on claude.ai. But "it can be uploaded" is not the same as "people use it." Claude.ai has no native skill marketplace, so discovery happens through a 3-layer stack: Anthropic's curated plugin marketplace, third-party aggregators, and social/newsletter amplification. The question Matt asked - "is the GitHub release the right decision" - has a clear answer: yes, but it is table stakes, not the strategy. This plan cuts the release and then pulls the real distribution levers.
## Problem Frame
Today, the only way a claude.ai user can get `last30days` is to clone the repo and run `scripts/build-skill.sh`. That filters out 99% of potential users. Even once a release exists with a direct download link, the hard problem is discovery - claude.ai users do not browse GitHub for skills. They find skills via Anthropic's "Discover" tab in Claude Code, third-party aggregator sites (skillsmp.com, mcpmarket.com, claudeskills.info), awesome-lists on GitHub, newsletters (The Neuron), and social posts (X, r/ClaudeAI).
Success looks like: a claude.ai user who never visits the repo can find, download, and upload the skill in under 60 seconds, and keep using it because the trigger description fires on the right prompts.
## Requirements Trace
- R1. A one-click install path exists for claude.ai users: click a link from README/marketplace/aggregator, get `last30days.skill`, drop into Upload dialog
- R2. The skill is submitted to Anthropic's official plugin marketplace at `platform.claude.com/plugins/submit`
- R3. The skill is listed in at least 4 high-traffic awesome-lists / aggregators
- R4. The SKILL.md YAML `description` and `argument-hint` fields are tuned so Claude's skill-selector actually invokes `last30days` on research-intent prompts (trigger quality is the single biggest install-to-reuse lever)
- R5. First-run experience works with zero API keys for the default sources (Reddit, Hacker News, Polymarket, GitHub) - already true, verify does not regress
- R6. At least one high-visibility amplification moment ships within 14 days: demo GIF + launch tweet + The Neuron pitch
- R7. Basic metrics are in place to learn what works: release-download counts, aggregator referrer traffic, GitHub star velocity before/after
## Scope Boundaries
Non-goals for this plan:
- Not building a custom skill-hosting site or our own marketplace
- Not changing the runtime pipeline or adding features - this is pure distribution
- Not spamming aggregators with low-effort PRs - one quality submission per venue
- Not gaming install counts or stars
- Not displacing the existing Claude Code plugin / OpenClaw / Gemini distribution - those stay as-is, cross-linked
- Not depending on Anthropic marketplace acceptance before other levers ship - marketplace review is slow and gate-able
## Context and Research
### The claude.ai skill ecosystem in April 2026
- **No native claude.ai skill marketplace.** Upload is the only end-user path inside the web UI.
- **Anthropic's Plugin/Skills Marketplace** (submissions at `platform.claude.com/plugins/submit`) is the closest thing to a "featured" channel and ships through Claude Code's "Discover" tab. Quality/security review gates acceptance. Research-category skills are under-represented vs. dev-tool skills.
- **Third-party aggregators** drive most organic discovery outside Anthropic's channels:
- `skillsmp.com`, `mcpmarket.com`, `claudeskills.info`, `skillsdirectory.com`, `agensi.io`
- These aggregators scrape awesome-lists, so one well-placed PR cascades
- **Awesome-lists** where skills discovery congregates:
- `ComposioHQ/awesome-claude-skills`
- `travisvn/awesome-claude-skills`
- `karanb192/awesome-claude-skills`
- `VoltAgent/awesome-agent-skills`
- `sickn33/antigravity-awesome-skills` (1,400+ skills indexed)
- **Newsletter amplification:** The Neuron runs a daily "AI Skill of the Day" digest - the single biggest external traffic source per successful skill creators. Their "practical workflow" angle fits a research skill cleanly.
- **Install-count reference points** from public aggregator data:
- `self-improving-agent`: 357k installs
- `frontend-design`: 277k installs
- `skill-vetter`: 190k installs
- `github`: 148k installs
- `proactive-agent`: 135k installs
- Long tail: ~500 installs
The gap between 500-install and 357k-install skills is mostly: (a) trigger description quality, (b) zero-config first run, (c) one amplification moment that caught.
### Current distribution surface for last30days
- Claude Code plugin via marketplace and GitHub URL: live
- OpenClaw via ClawHub (`clawhub install last30days-official`): live
- Gemini extension: live
- Manual `git clone`: documented in README
- claude.ai `.skill` upload: just shipped, undocumented for end users (no link)
The cross-linking graph is incomplete. Traffic that already exists (Claude Code install page, OpenClaw listing, Gemini extensions page) is not being routed to the new claude.ai path.
### Reference: trigger description quality
The root `SKILL.md` `description` field is how Claude decides whether to invoke the skill. Current text (as of 3.0.1, 167 chars):
> "Multi-query social research across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Intelligent query planning with Gemini/OpenAI fallback."
Analysis: solid source list, weak on action verbs and example queries. Successful skills include imperative verbs ("research", "find", "summarize", "compare") and 1-2 example triggers the user might type. At 167/200 chars, there is room.
## Key Technical Decisions
- **Cut v3.0.1 GitHub release with `last30days.skill` attached as an asset** (table stakes). Rationale: every downstream lever (README link, marketplace submission, aggregator PR, tweet) needs a stable public download URL.
- **Automate `.skill` build in CI on tag push** so future releases never forget to attach the artifact. Rationale: manual builds break over time; this is a one-time 10-line GitHub Actions workflow.
- **Prioritize marketplace submission in parallel with aggregator PRs**, not in sequence. Rationale: marketplace review is slow and opaque; do not block aggregator work on it. If rejected, we still have the aggregator presence.
- **Tune the SKILL.md description to optimize trigger selection**, not marketing copy. Rationale: this is the single biggest re-use lever per the ecosystem research. Marketing copy goes in README/release notes, not frontmatter.
- **One quality pass per aggregator, not a spray.** Rationale: awesome-list maintainers reject duplicate / low-effort PRs; reputation matters.
- **Ship the launch tweet with a real demo GIF**, not a screenshot. Rationale: Boris Cherny's Claude Code viral tweet template (one query, one result, "oh wow" moment) consistently outperforms text-only launches.
- **Pitch The Neuron once, with a production-quality 60-second demo**, not a cold email. Rationale: single shot at the biggest amplifier; treat it like a press release, not a tweet.
- **Track release-download count + GitHub referrer traffic as proxies for adoption** until we have better signal. Rationale: claude.ai upload counts are not exposed to creators.
- **Cross-link existing distribution pages back to claude.ai** as part of the release. Rationale: converting existing users to multi-surface users is cheaper than acquiring new ones.
## Open Questions
### Resolved during planning
- Is the GitHub release the right first step? Yes. Every other lever depends on a stable download URL. But it is a prerequisite, not the strategy.
- Does claude.ai have a native skill directory? No (confirmed April 2026).
- Should we wait for Anthropic marketplace acceptance before shipping other levers? No - parallelize.
- Do we need to rebuild the runtime to improve claude.ai adoption? No - the runtime is strong; the gap is distribution.
### Deferred to implementation
- Exact Neuron pitch copy - draft during Unit 8, refine based on what their recent editions have favored
- Whether to tag `@steipete`, `@AnthropicAI`, `@alexalbert__` in the launch tweet - confirm current handles and review each's posting culture before tagging
- Which specific demo query to record for the launch GIF - pick during Unit 7 based on what's newsworthy that week
- Whether to request a "skills-research" badge on skillsdirectory.com/agensi.io - check their current badge programs during Unit 5
## High-Level Technical Design
> *This illustrates the intended distribution graph and is directional guidance for review, not implementation specification.*
```
[GitHub Release v3.0.1]
|
+-- last30days.skill (asset, public URL)
|
+------> README "Upload to claude.ai" section (Unit 3)
|
+------> Claude Code plugin README link (Unit 4)
+------> OpenClaw listing link (Unit 4)
+------> Gemini extension link (Unit 4)
|
+------> Anthropic marketplace submission (Unit 6)
|
+------> Aggregator PRs (Unit 5):
| * ComposioHQ/awesome-claude-skills
| * travisvn/awesome-claude-skills
| * karanb192/awesome-claude-skills
| * VoltAgent/awesome-agent-skills
| * sickn33/antigravity-awesome-skills
| * skillsmp.com submit form
|
+------> Amplification (Units 7-9):
* Demo GIF + launch tweet
* The Neuron "Skill of the Day" pitch
* News-cycle recurring tweet (weekly)
All paths end at: claude.ai Upload Skill dialog
Trigger quality (Unit 2) determines whether installs become sustained usage
```
## Implementation Units
- [ ] **Unit 1: Cut v3.0.1 GitHub release with `.skill` asset + auto-build CI**
**Goal:** Produce a stable public download URL for `last30days.skill` so every downstream lever has something to link to, and guarantee future releases include the artifact automatically.
**Requirements:** R1
**Dependencies:** None (plan 2026-04-14-001 already shipped the build script)
**Files:**
- Create: `.github/workflows/release.yml`
- Modify: none at release time (release is a git-tag + GitHub release action)
**Approach:**
- Tag `v3.0.1` on `main`, push
- Create GitHub release with the CHANGELOG v3.0.1 entry as body, attach `dist/last30days.skill`
- Add CI workflow that triggers on `push: tags: 'v*'`, runs `bash scripts/build-skill.sh`, uploads the artifact to the release. The `action-gh-release` pattern is standard.
- Release URL shape: `https://github.com/mvanhorn/last30days-skill/releases/download/v3.0.1/last30days.skill` (deterministic, shareable)
**Patterns to follow:**
- Any existing `.github/workflows/` patterns in the repo
- `actions/checkout@v4` + `softprops/action-gh-release@v2` is the conventional combo
**Test scenarios:**
- Happy path: pushing `v3.0.1` tag produces a release with `last30days.skill` attached and publicly downloadable without auth
- Edge case: re-tagging `v3.0.1` does not duplicate or corrupt the asset
- Error path: build failure in the workflow fails the release cleanly (no empty release created)
**Verification:**
- `curl -fsSL -o /tmp/dl.skill https://github.com/mvanhorn/last30days-skill/releases/download/v3.0.1/last30days.skill` succeeds anonymously
- Downloaded file matches `dist/last30days.skill` byte-for-byte
- A second tag (e.g., `v3.0.2-test`) in a branch triggers the workflow end-to-end
- [ ] **Unit 2: Tune SKILL.md description and argument-hint for trigger quality**
**Goal:** Increase the probability that Claude's skill-selector invokes `last30days` on research-intent prompts. Trigger quality is the single biggest install-to-reuse lever per ecosystem research.
**Requirements:** R4, R5
**Dependencies:** None
**Files:**
- Modify: `SKILL.md` (frontmatter `description` and `argument-hint` only)
- Modify: `skills/last30days/SKILL.md` (if parity needed)
**Approach:**
- Rewrite `description` to lead with an imperative action verb and include 1-2 concrete example queries, staying =200 chars
- Rewrite `argument-hint` to show 2-3 canonical invocations that mirror real user phrasing, not marketing phrasing
- Keep the source list intact - that's the value prop - but move it later in the sentence
- Reference frames that worked for high-install skills: `frontend-design`, `self-improving-agent`, `github`
**Technical design:** *(directional guidance, not implementation spec)*
Candidate shape (verify char count in implementation):
```yaml
description: "Research what people actually say about any topic in the last 30 days. Pulls real posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
argument-hint: 'last30days AI video tools | last30days nvidia earnings reaction | last30days best noise cancelling headphones'
```
**Test scenarios:**
- Happy path: in a fresh claude.ai chat, prompts like "what are people saying about X this week" surface `last30days` in the skill-selector candidate set
- Edge case: generic "research X" prompts do not over-select `last30days` when the user clearly wants a general answer (avoids false-positive selection)
- Integration: test in all three environments - claude.ai web, Claude Code, OpenClaw - to confirm selection behavior is consistent
**Verification:**
- Description =200 chars, checked by the same regex Unit 6 of plan 001 used
- At least 3 real-user prompt phrasings trigger skill selection in manual testing
- No regression on zero-config first-run (no new API keys required)
- [ ] **Unit 3: Rewrite the README claude.ai section with one-click install**
**Goal:** Replace the current "run this bash script" instructions with a one-click download link pointing at the GitHub release asset.
**Requirements:** R1
**Dependencies:** Unit 1 (release must exist first)
**Files:**
- Modify: `README.md` (the "Upload as a Claude Skill" section added in plan 001)
**Approach:**
- Replace the `bash scripts/build-skill.sh` instruction with a direct download link to the release asset
- Keep the build-from-source instruction as a fallback for developers, demoted below the direct link
- Add a short 3-step install guide with specific UI path: "Settings > Capabilities > Skills > + button, drop the .skill file"
- Include a screenshot or GIF showing the upload flow if space allows (can be added in Unit 7)
**Test scenarios:**
- Test expectation: none - pure documentation change
**Verification:**
- A user following the README instructions end-to-end can go from "never heard of this" to working skill in under 60 seconds
- Instructions specify the exact claude.ai UI path current as of the release date
- [ ] **Unit 4: Cross-link existing distribution surfaces back to claude.ai**
**Goal:** Convert existing Claude Code plugin / OpenClaw / Gemini traffic into claude.ai installs. Cheaper than net-new acquisition.
**Requirements:** R1
**Dependencies:** Unit 1, Unit 3
**Files:**
- Modify: `README.md` (install matrix - add claude.ai row prominently)
- Modify: `variants/open/SKILL.md` in the private repo if that governs OpenClaw listing copy
- Modify: `gemini-extension.json` if `description` or install hints exist there
- External: update the ClawHub listing page for `last30days-official` to mention claude.ai availability
**Approach:**
- Every listing page a user currently lands on should have a one-line "Also available as a claude.ai Skill: [download]" link
- Use a consistent short-URL pattern so it's instantly recognizable across surfaces
- Do not require users to re-read each install guide - the cross-link is opportunistic, not blocking
**Test scenarios:**
- Test expectation: none - documentation/external-listing updates
**Verification:**
- Each of the 4 distribution surfaces (Claude Code plugin marketplace, OpenClaw ClawHub listing, Gemini extensions page, GitHub README) contains a visible claude.ai cross-link within 1 scroll of the page top
- [ ] **Unit 5: Submit PRs to high-traffic Claude skill awesome-lists**
**Goal:** Get listed in the 5 highest-traffic aggregators so third-party skill-discovery sites (skillsmp.com, mcpmarket.com, claudeskills.info) pick up the entry.
**Requirements:** R3
**Dependencies:** Unit 1, Unit 2 (description should be tuned before first impression in these lists)
**Files (external repos):**
- `ComposioHQ/awesome-claude-skills` - PR adding last30days to the relevant category
- `travisvn/awesome-claude-skills`
- `karanb192/awesome-claude-skills`
- `VoltAgent/awesome-agent-skills`
- `sickn33/antigravity-awesome-skills`
**Approach:**
- One PR per list, in parallel
- Each PR: one-line entry matching the list's existing format; link to release asset (not repo root)
- If the list has a "research" or "data-gathering" category, use it; otherwise append to the most adjacent section
- Draft copy once, reuse across PRs - but match each list's voice and entry format
- Do not self-star or brigade - let the listing earn traction organically
**Test scenarios:**
- Test expectation: none - external PRs, not code
**Verification:**
- All 5 PRs opened on the same day (batch effort, reduces overhead)
- Entries include: skill name, one-sentence description matching tuned SKILL.md copy, release URL, source repo URL
- Track merge status over 14 days; abandon PRs that go stale after reasonable nudging
- [ ] **Unit 6: Submit to Anthropic's official Plugin/Skills Marketplace**
**Goal:** Get featured in Claude Code's "Discover" tab, the closest thing to a native claude.ai skill directory.
**Requirements:** R2
**Dependencies:** Unit 1, Unit 2
**Files:**
- No repo changes; this is an external submission at `platform.claude.com/plugins/submit`
**Approach:**
- Submit via Anthropic's form with: skill name, description (matches tuned SKILL.md), GitHub repo URL, release asset URL, demo video link (from Unit 7 if available)
- Expect quality/security review; Anthropic will likely ask for the ClawGuard-scanner-style audit items already surfaced in issue #231 - have responses ready
- Do not wait for acceptance before shipping other levers
**Test scenarios:**
- Test expectation: none - external submission
**Verification:**
- Submission confirmation received
- Track review status weekly; iterate on feedback if any
- [ ] **Unit 7: Record a 15-60 second demo GIF or screen recording**
**Goal:** Produce the visual asset that every amplification channel needs - launch tweet, Neuron pitch, README hero, release notes.
**Requirements:** R6
**Dependencies:** Unit 2 (want the tuned description on-screen), Unit 3 (want the updated install flow)
**Files:**
- Create: `assets/claudeai-demo.gif` (or `.mp4` if GIF is too large)
- Modify: `README.md` to embed the GIF
**Approach:**
- Two possible framings:
1. "Upload + use" flow: 15 seconds showing Upload dialog -> skill appears -> sample query -> result
2. "One query" flow: 15-30 seconds of a real research query running end-to-end with actual output
- Pick framing 2 for outside-audience amplification (tweet, Neuron); framing 1 for the README
- Record at 1x speed (speeding up feels fake); edit to =60 seconds
- Export as optimized GIF or H.264 MP4 =5MB
**Test scenarios:**
- Test expectation: none - media asset
**Verification:**
- Asset loads cleanly in GitHub README
- Asset uploads cleanly to X (under their video length/size caps)
- Matt watches it fresh and the "oh wow" moment is unambiguous in the first 5 seconds
- [ ] **Unit 8: Pitch The Neuron "AI Skill of the Day"**
**Goal:** One high-leverage newsletter placement that historically drives the biggest external install spike for Claude skills.
**Requirements:** R6
**Dependencies:** Unit 1, Unit 7
**Approach:**
- Identify The Neuron editor contact (newsletter footer, X DMs, their `skilloftheday@` alias if published)
- Pitch with: 3-sentence hook, demo video link, release URL, 3 example queries that show breadth
- Angle: "researcher skill that queries 12+ social sources in one shot" - novelty vs. their typical dev-tool coverage
- Offer exclusive timing if they want (publish first, then we tweet)
- Do not follow up more than twice
**Test scenarios:**
- Test expectation: none - external pitch
**Verification:**
- Pitch sent with all assets linked
- Track whether the issue ships within 14 days; if not, reuse the materials for other newsletters
- [ ] **Unit 9: Launch tweet + recurring news-cycle posts**
**Goal:** Seed social discovery and establish a weekly cadence so the skill stays top-of-mind.
**Requirements:** R6
**Dependencies:** Unit 1, Unit 7
**Approach:**
- Launch tweet: demo GIF + 1-sentence description + install link. Post to X, cross-post to r/ClaudeAI and r/singularity same day.
- Do not tag handles reflexively - research each target account's culture first
- Weekly recurring pattern: pick a news moment (earnings, launch, election, cultural event), run `last30days` on it, screenshot the output, post. Low-effort, repeatable, compounds.
- Track: likes, impressions, link-click referrer traffic to the release page
**Test scenarios:**
- Test expectation: none - social posts
**Verification:**
- Launch tweet live with demo GIF
- At least one follow-up news-cycle post within 7 days
- Referrer traffic spike visible in GitHub traffic dashboard
- [ ] **Unit 10: Adoption telemetry and feedback loop**
**Goal:** Learn which levers work so we double down on wins and cut losses. Current blind spot: no visibility into claude.ai install counts.
**Requirements:** R7
**Dependencies:** Unit 1
**Approach:**
- Baseline metrics (capture on Unit 1 ship day):
- GitHub stars
- Clones/day
- Traffic referrers
- Release-asset download count (GitHub exposes this on the Release page)
- Weekly review during the first 6 weeks of:
- Release download deltas
- Star velocity
- Referrer sources (identifies which aggregator/newsletter/tweet drove traffic)
- New GitHub issues that mention claude.ai specifically
- No dedicated analytics infrastructure - use what GitHub provides + manual referrer spot-checks
- Publish a "what worked / what didn't" retro after 6 weeks in `docs/solutions/` so the next launch compounds
**Test scenarios:**
- Test expectation: none - observability
**Verification:**
- Baseline metrics captured in a `docs/solutions/YYYY-MM-DD-*.md` note
- Weekly log of download/star/referrer deltas maintained
- Retro written at week 6 with concrete learnings for the next release
## System-Wide Impact
- **Interaction graph:** Touches GitHub (release, CI), external aggregators (PRs), Anthropic marketplace (submission), X/Reddit/newsletter (social), ClawHub/Gemini listings (cross-links). No runtime code changes.
- **State lifecycle risks:** Minimal. The main risk is inconsistent cross-linking (some surfaces mention claude.ai, others don't) - Unit 4 treats this as a coordinated sweep, not per-surface creep.
- **API surface parity:** None - no API changes.
- **Integration coverage:** The critical integration is trigger-selection behavior (Unit 2). Manual verification across web / Claude Code / OpenClaw is the gate.
- **Unchanged invariants:** Runtime pipeline, existing install paths (Claude Code plugin / OpenClaw / Gemini) all stay working. Zero-config first-run for default sources remains intact.
## Risks and Dependencies
| Risk | Mitigation |
|------|------------|
| Anthropic marketplace rejects the submission on security/quality grounds | Run `scripts/build-skill.sh` output through ClawGuard or equivalent scanner pre-submission; address #231 findings if real |
| Awesome-list maintainers reject or ignore PRs | Submit to 5 lists in parallel; any 2 acceptances are enough; do not brigade |
| The Neuron pitch is ignored | Treat as upside, not critical path; reuse materials for other newsletters (Ben's Bites, TLDR, Superhuman AI) |
| Tuned description causes false-positive skill selection on unrelated prompts | Manual prompt-testing in Unit 2; be willing to walk back if Claude over-invokes the skill |
| A new Anthropic marketplace or directory launches mid-plan and changes the landscape | The research-tracking cadence in Unit 10 catches this within a week; plan can adapt |
| Launch tweet flops / no organic pickup | Weekly news-cycle cadence (Unit 9) is the compounding play, not the launch moment |
| Cross-repo cross-links in Unit 4 go stale when listings move | Use canonical GitHub Release URL (deterministic) as the link target everywhere |
## Documentation / Operational Notes
- README gets a hero section update in Unit 3
- CHANGELOG gets a v3.0.1 release-notes entry (already shipped in plan 001)
- A `docs/solutions/` retrospective note ships after the 6-week observation window (Unit 10)
- No runbook needed - distribution work is one-time-per-release
## Sources and References
- Research pass by repo-research-analyst on 2026-04-14 - [findings](https://github.com/anthropics/skills)
- [Anthropic Plugin/Skills Marketplace submissions](https://platform.claude.com/plugins/submit)
- [anthropics/skills](https://github.com/anthropics/skills) - 87k stars, canonical repo
- [SkillsMP](https://skillsmp.com), [claudeskills.info](https://claudeskills.info), [mcpmarket.com/tools/skills](https://mcpmarket.com/tools/skills) - aggregators
- [ComposioHQ/awesome-claude-skills](https://github.com/ComposioHQ/awesome-claude-skills)
- [sickn33/antigravity-awesome-skills](https://github.com/sickn33/antigravity-awesome-skills)
- [The Neuron Skill of the Day digest](https://www.theneuron.ai/explainer-articles/the-neurons-ai-skill-of-the-day-digest-april-2026-week-1/)
- Completed prerequisite: `docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md` (packaging fix)
- Related code: `SKILL.md` (frontmatter), `README.md`, `scripts/build-skill.sh`, `.github/workflows/`
- Install-count reference points from aggregators: self-improving-agent 357k, frontend-design 277k, skill-vetter 190k, github 148k, proactive-agent 135k
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "last30days-skill", "name": "last30days-skill",
"version": "3.0.1", "version": "3.0.5",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.", "description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [ "settings": [
{ {
+6
View File
@@ -2,6 +2,12 @@ The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. `/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## 2026-04-20 — Clickable source links
Every cited source in the synthesis (X handles, subreddits, publications, YouTube channels, Polymarket markets) now renders as a blue CMD-clickable markdown link in the terminal. No raw URL strings, no trailing list of links, and the rest of the output contract is preserved intact — bold headline paragraphs, `├─` `└─` emoji stats tree, and the QUERY_TYPE-specific invitation all render correctly on first pass. Works in Claude Code on any terminal that supports OSC 8 hyperlinks (Ghostty, iTerm, Warp, others).
Credit to [@jay_k](https://x.com/jay_k) for surfacing the missing link-back in the reply thread, and to [@photomatt](https://x.com/photomatt) for confirming CMD-click works in Claude Code — that was the nudge to flip the rule.
## v3 is the intelligent search release ## v3 is the intelligent search release
v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities. v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
+8 -1
View File
@@ -19,12 +19,19 @@ mkdir -p dist
OUT="dist/last30days.skill" OUT="dist/last30days.skill"
git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD
# claude.ai's .skill bundle only needs the root SKILL.md + scripts/ runtime.
# Claude Code needs skills/ and .claude-plugin/ in the git archive
# (that's why they're NOT in .gitattributes export-ignore), but the .skill
# bundle must strip them to keep a single canonical SKILL.md and stay under
# the 200-file cap.
zip -d "$OUT" "last30days/skills/*" "last30days/.claude-plugin/*" > /dev/null 2>&1 || true
COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}') COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}')
SIZE=$(du -h "$OUT" | cut -f1) SIZE=$(du -h "$OUT" | cut -f1)
if [ "$COUNT" -gt 200 ]; then if [ "$COUNT" -gt 200 ]; then
echo "error: $COUNT files in zip, claude.ai's cap is 200" >&2 echo "error: $COUNT files in zip, claude.ai's cap is 200" >&2
echo " check .gitattributes export-ignore entries" >&2 echo " check .gitattributes export-ignore entries and this script's zip -d excludes" >&2
exit 1 exit 1
fi fi
+12
View File
@@ -441,6 +441,18 @@ def is_youtube_comments_available(config: dict[str, Any]) -> bool:
return 'youtube_comments' in include return 'youtube_comments' in include
def is_tiktok_comments_available(config: dict[str, Any]) -> bool:
"""Check if TikTok comment enrichment is available.
Requires SCRAPECREATORS_API_KEY AND tiktok_comments in INCLUDE_SOURCES.
Mirrors the youtube_comments opt-in pattern.
"""
if not config.get('SCRAPECREATORS_API_KEY'):
return False
include = _parse_include_sources(config)
return 'tiktok_comments' in include
def is_youtube_sc_available(config: dict[str, Any]) -> bool: def is_youtube_sc_available(config: dict[str, Any]) -> bool:
"""Check if ScrapeCreators YouTube search fallback is available. """Check if ScrapeCreators YouTube search fallback is available.
+56 -1
View File
@@ -69,6 +69,47 @@ def normalize_source_items(
return filtered return filtered
def _remap_comments(
raw: list[Any],
score_keys: tuple[str, ...],
excerpt_keys: tuple[str, ...],
) -> list[dict[str, Any]]:
"""Normalize comments from any source into the shared Reddit-compatible shape.
Downstream code (signals._top_comment_score, render._top_comments_list,
entity_extract, rerank) all expect `score` and `excerpt`. This helper maps
per-source field names (YT: likes/text, TikTok: digg_count/text) onto that
shape while preserving author/date/url passthrough.
"""
out: list[dict[str, Any]] = []
for raw_c in raw:
if not isinstance(raw_c, dict):
continue
score = _first_present(raw_c, score_keys, default=0)
excerpt = _first_present(raw_c, excerpt_keys, default="")
try:
score_int = int(score or 0)
except (TypeError, ValueError):
score_int = 0
entry: dict[str, Any] = {
"score": score_int,
"excerpt": str(excerpt or "")[:400],
"author": str(raw_c.get("author") or ""),
"date": str(raw_c.get("date") or ""),
}
if raw_c.get("url"):
entry["url"] = str(raw_c["url"])
out.append(entry)
return out
def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> Any:
for key in keys:
if key in d and d[key] not in (None, ""):
return d[key]
return default
def _domain_from_url(url: str) -> str | None: def _domain_from_url(url: str) -> str | None:
if not url: if not url:
return None return None
@@ -200,6 +241,11 @@ def _normalize_youtube(
metadata: dict[str, Any] = {} metadata: dict[str, Any] = {}
if highlights: if highlights:
metadata["transcript_highlights"] = highlights metadata["transcript_highlights"] = highlights
metadata["top_comments"] = _remap_comments(
item.get("top_comments") or [],
score_keys=("score", "likes"),
excerpt_keys=("excerpt", "text"),
)
return _source_item( return _source_item(
item_id=str(item.get("video_id") or item.get("id") or f"YT{index + 1}"), item_id=str(item.get("video_id") or item.get("id") or f"YT{index + 1}"),
source=source, source=source,
@@ -242,7 +288,16 @@ def _normalize_shortform_video(
relevance_hint=item.get("relevance", 0.5), relevance_hint=item.get("relevance", 0.5),
why_relevant=str(item.get("why_relevant") or ""), why_relevant=str(item.get("why_relevant") or ""),
snippet=caption, snippet=caption,
metadata={"hashtags": item.get("hashtags") or []}, metadata={
"hashtags": item.get("hashtags") or [],
"top_comments": _remap_comments(
item.get("top_comments") or [],
# TikTok uses digg_count as the vote field; Instagram has no
# comment fetcher today so the key is harmlessly absent.
score_keys=("score", "digg_count", "likes"),
excerpt_keys=("excerpt", "text"),
),
},
) )
+5 -1
View File
@@ -887,7 +887,11 @@ def _retrieve_stream(
hashtags=tiktok_hashtags, hashtags=tiktok_hashtags,
creators=tiktok_creators, creators=tiktok_creators,
) )
return tiktok.parse_tiktok_response(result), {} items = tiktok.parse_tiktok_response(result)
if items and env.is_tiktok_comments_available(config):
sc_token = config.get("SCRAPECREATORS_API_KEY", "")
tiktok.enrich_with_comments(items, token=sc_token)
return items, {}
if source == "instagram": if source == "instagram":
# Use raw_topic so expand_instagram_queries() generates diverse variants # Use raw_topic so expand_instagram_queries() generates diverse variants
# from the original user topic, not the planner's narrowed search_query. # from the original user topic, not the planner's narrowed search_query.
+36 -5
View File
@@ -152,13 +152,14 @@ def render_full(report: schema.Report) -> str:
lines.append(f" *{item.container}*") lines.append(f" *{item.container}*")
if item.snippet: if item.snippet:
lines.append(f" {item.snippet[:500]}") lines.append(f" {item.snippet[:500]}")
# Top comments for Reddit # Top comments for Reddit, YouTube, TikTok, HackerNews.
top_comments = item.metadata.get("top_comments", []) top_comments = item.metadata.get("top_comments", [])
if top_comments and isinstance(top_comments[0], dict): if top_comments and isinstance(top_comments[0], dict):
vote_label = _vote_label_for(item.source)
for tc in top_comments[:3]: for tc in top_comments[:3]:
excerpt = tc.get("excerpt", tc.get("text", ""))[:200] excerpt = tc.get("excerpt", tc.get("text", ""))[:200]
tc_score = tc.get("score", "") tc_score = tc.get("score", "")
lines.append(f" Top comment ({tc_score} upvotes): {excerpt}") lines.append(f" Top comment ({tc_score} {vote_label}): {excerpt}")
# Comment insights for Reddit # Comment insights for Reddit
insights = item.metadata.get("comment_insights", []) insights = item.metadata.get("comment_insights", [])
if insights: if insights:
@@ -276,7 +277,8 @@ def _render_candidate(candidate: schema.Candidate, prefix: str) -> list[str]:
for tc in _top_comments_list(primary): for tc in _top_comments_list(primary):
excerpt = tc.get("excerpt") or tc.get("text") or "" excerpt = tc.get("excerpt") or tc.get("text") or ""
score = tc.get("score", "") score = tc.get("score", "")
lines.append(f" - Comment ({score} upvotes): {_truncate(excerpt.strip(), 240)}") vote_label = _vote_label_for(primary.source) if primary else "upvotes"
lines.append(f" - Comment ({score} {vote_label}): {_truncate(excerpt.strip(), 240)}")
insight = _comment_insight(primary) insight = _comment_insight(primary)
if insight: if insight:
lines.append(f" - Insight: {_truncate(insight, 220)}") lines.append(f" - Insight: {_truncate(insight, 220)}")
@@ -582,13 +584,42 @@ def _format_explanation(candidate: schema.Candidate) -> str | None:
return candidate.explanation return candidate.explanation
def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int = 10) -> list[dict]: # Per-source minimum vote counts for showing a top comment in compact emit.
"""Return up to `limit` top comments with score >= min_score.""" # Reddit upvotes, YouTube likes, and TikTok likes are not comparable units —
# 10 upvotes on Reddit signals genuine community interest, 10 likes on a
# viral TikTok is noise. First-pass values; tune after live observation.
_TOP_COMMENT_MIN_SCORE: dict[str, int] = {
"reddit": 10,
"youtube": 50,
"tiktok": 500,
"hackernews": 5,
}
_TOP_COMMENT_VOTE_LABEL: dict[str, str] = {
"reddit": "upvotes",
"hackernews": "points",
"youtube": "likes",
"tiktok": "likes",
}
def _vote_label_for(source: str) -> str:
return _TOP_COMMENT_VOTE_LABEL.get(source, "votes")
def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int | None = None) -> list[dict]:
"""Return up to `limit` top comments with score at or above the source's minimum.
If `min_score` is passed explicitly it overrides the per-source default;
otherwise the source-keyed map is consulted, with an effective default of 0
(always show) for unknown sources so new sources don't get silently hidden.
"""
if not item: if not item:
return [] return []
comments = item.metadata.get("top_comments") or [] comments = item.metadata.get("top_comments") or []
if not comments or not isinstance(comments[0], dict): if not comments or not isinstance(comments[0], dict):
return [] return []
if min_score is None:
min_score = _TOP_COMMENT_MIN_SCORE.get(item.source, 0)
return [c for c in comments if (c.get("score") or 0) >= min_score][:limit] return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]
+30 -4
View File
@@ -82,12 +82,11 @@ def _top_comment_score(item: schema.SourceItem) -> float:
# Per-source engagement weights: list of (field_name, weight) tuples. # Per-source engagement weights: list of (field_name, weight) tuples.
# Reddit uses a custom function because upvote_ratio and top_comment_score # Reddit, YouTube, and TikTok use custom functions because they include
# are not simple log1p fields. # a dedicated 10% top-comment-score slot (see _reddit_engagement,
# _youtube_engagement, _tiktok_engagement).
ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = { ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
"x": [("likes", 0.55), ("reposts", 0.25), ("replies", 0.15), ("quotes", 0.05)], "x": [("likes", 0.55), ("reposts", 0.25), ("replies", 0.15), ("quotes", 0.05)],
"youtube": [("views", 0.50), ("likes", 0.35), ("comments", 0.15)],
"tiktok": [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
"instagram": [("views", 0.50), ("likes", 0.30), ("comments", 0.20)], "instagram": [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
"hackernews": [("points", 0.55), ("comments", 0.45)], "hackernews": [("points", 0.55), ("comments", 0.45)],
"bluesky": [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)], "bluesky": [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)],
@@ -113,6 +112,29 @@ def _reddit_engagement(item: schema.SourceItem) -> float | None:
return (0.50 * score) + (0.35 * comments) + (0.05 * (ratio * 10.0)) + (0.10 * top_comment) return (0.50 * score) + (0.35 * comments) + (0.05 * (ratio * 10.0)) + (0.10 * top_comment)
def _youtube_engagement(item: schema.SourceItem) -> float | None:
views = log1p_safe(item.engagement.get("views"))
likes = log1p_safe(item.engagement.get("likes"))
comments = log1p_safe(item.engagement.get("comments"))
top_comment = _top_comment_score(item)
if not any([views, likes, comments, top_comment]):
return None
# Mirrors Reddit: carve out 10% for top-comment signal, keep view-weight
# dominant. Without comments, the pre-change weights (0.50/0.35/0.15)
# still govern relative ordering.
return (0.45 * views) + (0.32 * likes) + (0.13 * comments) + (0.10 * top_comment)
def _tiktok_engagement(item: schema.SourceItem) -> float | None:
views = log1p_safe(item.engagement.get("views"))
likes = log1p_safe(item.engagement.get("likes"))
comments = log1p_safe(item.engagement.get("comments"))
top_comment = _top_comment_score(item)
if not any([views, likes, comments, top_comment]):
return None
return (0.45 * views) + (0.27 * likes) + (0.18 * comments) + (0.10 * top_comment)
def _generic_engagement(item: schema.SourceItem) -> float | None: def _generic_engagement(item: schema.SourceItem) -> float | None:
if not item.engagement: if not item.engagement:
return None return None
@@ -125,6 +147,10 @@ def _generic_engagement(item: schema.SourceItem) -> float | None:
def engagement_raw(item: schema.SourceItem) -> float | None: def engagement_raw(item: schema.SourceItem) -> float | None:
if item.source == "reddit": if item.source == "reddit":
return _reddit_engagement(item) return _reddit_engagement(item)
if item.source == "youtube":
return _youtube_engagement(item)
if item.source == "tiktok":
return _tiktok_engagement(item)
weights = ENGAGEMENT_WEIGHTS.get(item.source) weights = ENGAGEMENT_WEIGHTS.get(item.source)
if weights: if weights:
return _weighted_engagement(item, weights) return _weighted_engagement(item, weights)
+134
View File
@@ -539,3 +539,137 @@ def parse_tiktok_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
List of item dicts ready for normalization. List of item dicts ready for normalization.
""" """
return response.get("items", []) return response.get("items", [])
def _tiktok_total_engagement(item: Dict[str, Any]) -> int:
"""Total engagement for ranking which posts deserve comment enrichment."""
eng = item.get("engagement", {})
return (eng.get("views", 0) or 0) + (eng.get("likes", 0) or 0) + (eng.get("comments", 0) or 0)
def enrich_with_comments(
items: List[Dict[str, Any]],
token: str,
max_posts: int = 3,
max_comments: int = 5,
) -> List[Dict[str, Any]]:
"""Enrich top TikTok posts with comment data from ScrapeCreators.
For the top N posts by engagement, fetches comments via the SC API
and attaches them as a ``top_comments`` field on each item. Mirrors
youtube_yt.enrich_with_comments.
Args:
items: TikTok items from search_tiktok()
token: ScrapeCreators API key
max_posts: How many posts to enrich with comments
max_comments: Max comments to keep per post
Returns:
Items list (mutated in place) with top_comments added to enriched items.
"""
if not items or not token or max_posts <= 0:
return items
ranked = sorted(items, key=_tiktok_total_engagement, reverse=True)
top_items = ranked[:max_posts]
_log(f"Enriching comments for {len(top_items)} TikTok posts")
from concurrent.futures import ThreadPoolExecutor, as_completed
def _enrich_one(item: dict) -> bool:
post_url = item.get("url", "")
if not post_url:
return False
try:
comments = _fetch_post_comments(post_url, token, max_comments)
if comments:
item["top_comments"] = comments
return True
except Exception as exc:
_log(f"Comment enrichment failed for {post_url}: {exc}")
return False
enriched_count = 0
with ThreadPoolExecutor(max_workers=min(4, len(top_items))) as executor:
futures = {executor.submit(_enrich_one, item): item for item in top_items}
for future in as_completed(futures):
if future.result():
enriched_count += 1
_log(f"Enriched {enriched_count}/{len(top_items)} posts with comments")
return items
def _fetch_post_comments(
post_url: str,
token: str,
max_comments: int = 5,
) -> List[Dict[str, Any]]:
"""Fetch comments for a single TikTok post via ScrapeCreators.
SC endpoint: GET /v1/tiktok/video/comments?url=<video_url>
Response shape: { comments: [{text, user.nickname, digg_count, create_time, ...}], cursor, total }
Args:
post_url: Canonical TikTok post URL (share_url form works)
token: ScrapeCreators API key
max_comments: Maximum comments to return
Returns:
List of comment dicts with author, text, digg_count (likes), date.
Empty list on any error comment failures never crash the pipeline.
"""
if not _requests:
try:
from urllib.parse import urlencode
params = urlencode({"url": post_url, "trim": "true"})
url = f"{SCRAPECREATORS_BASE}/video/comments?{params}"
headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT
data = http.get(url, headers=headers, timeout=30, retries=2)
except Exception as exc:
_log(f"Comment fetch error (urllib) for {post_url}: {exc}")
return []
else:
try:
resp = _requests.get(
f"{SCRAPECREATORS_BASE}/video/comments",
params={"url": post_url, "trim": "true"},
headers=http.scrapecreators_headers(token),
timeout=30,
)
resp.raise_for_status()
data = resp.json()
except Exception as exc:
_log(f"Comment fetch error for {post_url}: {exc}")
return []
raw_comments = data.get("comments") or data.get("data") or []
# Sort by digg_count desc so normalize sees the highest-signal first.
raw_comments = sorted(
raw_comments,
key=lambda c: c.get("digg_count", 0) or 0,
reverse=True,
)
out: List[Dict[str, Any]] = []
for c in raw_comments[:max_comments]:
text = c.get("text") or ""
if not text:
continue
user = c.get("user") if isinstance(c.get("user"), dict) else {}
author = user.get("nickname") or user.get("unique_id") or ""
create_time = c.get("create_time")
date_str = ""
if create_time:
try:
date_str = dates.timestamp_to_date(int(create_time)) or ""
except (ValueError, TypeError):
date_str = ""
out.append({
"author": author,
"text": text[:400],
"digg_count": c.get("digg_count", 0) or 0,
"date": date_str,
})
return out
+30 -7
View File
@@ -732,10 +732,11 @@ def _fetch_video_comments(
Returns: Returns:
List of comment dicts with author, text, likes, date. List of comment dicts with author, text, likes, date.
""" """
video_url = f"https://www.youtube.com/watch?v={video_id}"
if not _requests: if not _requests:
try: try:
from urllib.parse import urlencode from urllib.parse import urlencode
params = urlencode({"id": video_id}) params = urlencode({"url": video_url})
url = f"{SCRAPECREATORS_YT_BASE}/video/comments?{params}" url = f"{SCRAPECREATORS_YT_BASE}/video/comments?{params}"
headers = http.scrapecreators_headers(token) headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT headers["User-Agent"] = http.USER_AGENT
@@ -747,7 +748,7 @@ def _fetch_video_comments(
try: try:
resp = _requests.get( resp = _requests.get(
f"{SCRAPECREATORS_YT_BASE}/video/comments", f"{SCRAPECREATORS_YT_BASE}/video/comments",
params={"id": video_id}, params={"url": video_url},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
) )
@@ -763,11 +764,32 @@ def _fetch_video_comments(
text = c.get("text") or c.get("body") or c.get("content", "") text = c.get("text") or c.get("body") or c.get("content", "")
if not text: if not text:
continue continue
# SC returns author as {"name": "@handle", ...}; legacy mocks may pass a string.
author = c.get("author") or c.get("author_name", "")
if isinstance(author, dict):
author = author.get("name") or author.get("handle") or ""
# SC nests likes under engagement.likes; legacy shapes used top-level keys.
engagement = c.get("engagement") or {}
likes = c.get("likes")
if likes is None:
likes = engagement.get("likes", 0) if isinstance(engagement, dict) else 0
if not likes:
likes = c.get("vote_count", 0)
date = (
c.get("date")
or c.get("published_at")
or c.get("publishedTime")
or c.get("publishedTimeText", "")
)
comments.append({ comments.append({
"author": c.get("author") or c.get("author_name", ""), "author": author,
"text": text[:400], "text": text[:400],
"likes": c.get("likes") or c.get("vote_count", 0), "likes": likes,
"date": c.get("date") or c.get("published_at", ""), "date": date,
}) })
return comments return comments
@@ -931,10 +953,11 @@ def _sc_fetch_transcript(video_id: str, token: str) -> Optional[str]:
Returns: Returns:
Plaintext transcript string, or None if unavailable. Plaintext transcript string, or None if unavailable.
""" """
video_url = f"https://www.youtube.com/watch?v={video_id}"
if not _requests: if not _requests:
try: try:
from urllib.parse import urlencode from urllib.parse import urlencode
params = urlencode({"id": video_id}) params = urlencode({"url": video_url})
url = f"{SCRAPECREATORS_YT_BASE}/video/transcript?{params}" url = f"{SCRAPECREATORS_YT_BASE}/video/transcript?{params}"
headers = http.scrapecreators_headers(token) headers = http.scrapecreators_headers(token)
headers["User-Agent"] = http.USER_AGENT headers["User-Agent"] = http.USER_AGENT
@@ -946,7 +969,7 @@ def _sc_fetch_transcript(video_id: str, token: str) -> Optional[str]:
try: try:
resp = _requests.get( resp = _requests.get(
f"{SCRAPECREATORS_YT_BASE}/video/transcript", f"{SCRAPECREATORS_YT_BASE}/video/transcript",
params={"id": video_id}, params={"url": video_url},
headers=http.scrapecreators_headers(token), headers=http.scrapecreators_headers(token),
timeout=30, timeout=30,
) )
-1
View File
@@ -1 +0,0 @@
../../SKILL.md
+159
View File
@@ -49,6 +49,165 @@ class NormalizeV3Tests(unittest.TestCase):
) )
self.assertEqual([], normalized) self.assertEqual([], normalized)
def test_youtube_top_comments_passthrough_with_field_mapping(self):
"""YT comments from enrich_with_comments use likes/text; normalize must
carry them into metadata as the Reddit-compatible {score, excerpt} shape."""
items = [
{
"video_id": "vid-1",
"title": "How to deploy",
"url": "https://youtube.com/watch?v=vid-1",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 10000, "likes": 500, "comments": 30},
"top_comments": [
{"author": "Alice", "text": "Best tutorial ever", "likes": 120, "date": "2026-03-02"},
{"author": "Bob", "text": "Helped me ship", "likes": 45, "date": "2026-03-03"},
{"author": "Carol", "text": "Solid walkthrough", "likes": 7, "date": "2026-03-04"},
],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
top = normalized[0].metadata.get("top_comments")
self.assertIsNotNone(top)
self.assertEqual(3, len(top))
# First comment: likes->score, text->excerpt
self.assertEqual(120, top[0]["score"])
self.assertEqual("Best tutorial ever", top[0]["excerpt"])
self.assertEqual("Alice", top[0]["author"])
self.assertEqual("2026-03-02", top[0]["date"])
# Preserves ordering from input (already sorted desc upstream)
self.assertEqual(45, top[1]["score"])
self.assertEqual(7, top[2]["score"])
def test_youtube_top_comments_empty_list_passes_through_cleanly(self):
items = [
{
"video_id": "vid-2",
"title": "Short clip",
"url": "https://youtube.com/watch?v=vid-2",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 50, "likes": 2},
"top_comments": [],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
# Empty list is fine; metadata may have empty top_comments or omit it.
top = normalized[0].metadata.get("top_comments", [])
self.assertEqual([], top)
def test_youtube_without_top_comments_key_does_not_crash(self):
items = [
{
"video_id": "vid-3",
"title": "No comments fetched",
"url": "https://youtube.com/watch?v=vid-3",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 100, "likes": 5},
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
self.assertEqual([], normalized[0].metadata.get("top_comments", []))
def test_youtube_top_comments_feed_top_comment_score_signal(self):
"""Integration: after normalize, signals._top_comment_score should
return log1p(first comment score) for YT, proving the full chain."""
from lib import signals
import math
items = [
{
"video_id": "vid-4",
"title": "Viral comment thread",
"url": "https://youtube.com/watch?v=vid-4",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 1000, "likes": 50, "comments": 10},
"top_comments": [
{"author": "A", "text": "Legendary", "likes": 9999, "date": "2026-03-02"},
],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertAlmostEqual(math.log1p(9999), signals._top_comment_score(normalized[0]), places=4)
def test_tiktok_top_comments_passthrough_with_digg_count_mapping(self):
"""TikTok comments from enrich_with_comments use digg_count/text;
normalize must map to the shared {score, excerpt} shape."""
items = [
{
"id": "tt-1",
"text": "POV: shipping on Friday",
"url": "https://www.tiktok.com/@u/video/tt-1",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 50000, "likes": 2000, "comments": 300},
"top_comments": [
{"author": "Alice", "text": "dead", "digg_count": 1200, "date": "2026-03-02"},
{"author": "Bob", "text": "so real", "digg_count": 400, "date": "2026-03-03"},
],
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
top = normalized[0].metadata.get("top_comments")
self.assertEqual(2, len(top))
self.assertEqual(1200, top[0]["score"])
self.assertEqual("dead", top[0]["excerpt"])
self.assertEqual("Alice", top[0]["author"])
self.assertEqual(400, top[1]["score"])
def test_tiktok_without_top_comments_does_not_crash(self):
items = [
{
"id": "tt-2",
"text": "plain clip",
"url": "https://www.tiktok.com/@u/video/tt-2",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 1000, "likes": 20},
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertEqual([], normalized[0].metadata.get("top_comments", []))
def test_tiktok_top_comments_feed_top_comment_score_signal(self):
from lib import signals
import math
items = [
{
"id": "tt-3",
"text": "viral",
"url": "https://www.tiktok.com/@u/video/tt-3",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 100000, "likes": 5000, "comments": 500},
"top_comments": [
{"author": "A", "text": "this aged well", "digg_count": 50000, "date": "2026-03-02"},
],
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertAlmostEqual(math.log1p(50000), signals._top_comment_score(normalized[0]), places=4)
def test_grounding_requires_a_usable_date(self): def test_grounding_requires_a_usable_date(self):
items = [ items = [
{ {
+28
View File
@@ -242,6 +242,34 @@ class RenderTopCommentsTests(unittest.TestCase):
self.assertNotIn("Comment (", text) self.assertNotIn("Comment (", text)
self.assertNotIn("upvotes)", text) self.assertNotIn("upvotes)", text)
def test_youtube_comments_use_likes_label_and_50_threshold(self):
comments = [
{"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
{"score": 60, "excerpt": "saved me hours", "author": "bob"},
{"score": 10, "excerpt": "below threshold", "author": "carol"},
]
report = self._make_report_with_comments(source="youtube", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (120 likes): legit fire tutorial", text)
self.assertIn("Comment (60 likes): saved me hours", text)
self.assertNotIn("Comment (10 likes)", text)
# Render must not silently label YT as upvotes.
self.assertNotIn("Comment (120 upvotes)", text)
def test_tiktok_comments_use_likes_label_and_500_threshold(self):
comments = [
{"score": 2000, "excerpt": "this aged well", "author": "a"},
{"score": 600, "excerpt": "so real", "author": "b"},
{"score": 400, "excerpt": "below tt threshold", "author": "c"},
{"score": 50, "excerpt": "way below", "author": "d"},
]
report = self._make_report_with_comments(source="tiktok", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (2000 likes): this aged well", text)
self.assertIn("Comment (600 likes): so real", text)
self.assertNotIn("Comment (400 likes)", text)
self.assertNotIn("Comment (50 likes)", text)
class RenderBestTakesCompactTests(unittest.TestCase): class RenderBestTakesCompactTests(unittest.TestCase):
"""Tests for Best Takes section in compact output and fun tags on candidates.""" """Tests for Best Takes section in compact output and fun tags on candidates."""
+102 -9
View File
@@ -28,6 +28,98 @@ class SignalsV3Tests(unittest.TestCase):
) )
self.assertAlmostEqual(expected, signals.engagement_raw(item)) self.assertAlmostEqual(expected, signals.engagement_raw(item))
def test_youtube_engagement_adds_top_comment_slot(self):
with_comment = schema.SourceItem(
item_id="yt1",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=a",
engagement={"views": 10000, "likes": 500, "comments": 30},
metadata={"top_comments": [{"score": 500}]},
)
without = schema.SourceItem(
item_id="yt2",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=b",
engagement={"views": 10000, "likes": 500, "comments": 30},
metadata={"top_comments": []},
)
with_score = signals.engagement_raw(with_comment)
without_score = signals.engagement_raw(without)
self.assertIsNotNone(with_score)
self.assertIsNotNone(without_score)
self.assertGreater(with_score, without_score)
expected = (
0.45 * math.log1p(10000)
+ 0.32 * math.log1p(500)
+ 0.13 * math.log1p(30)
+ 0.10 * math.log1p(500)
)
self.assertAlmostEqual(expected, with_score, places=6)
def test_youtube_engagement_empty_returns_none(self):
item = schema.SourceItem(
item_id="yt-empty",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=e",
engagement={},
metadata={"top_comments": []},
)
self.assertIsNone(signals.engagement_raw(item))
def test_tiktok_engagement_adds_top_comment_slot(self):
item = schema.SourceItem(
item_id="tt1",
source="tiktok",
title="Title",
body="Body",
url="https://tiktok.com/@u/video/1",
engagement={"views": 100000, "likes": 5000, "comments": 500},
metadata={"top_comments": [{"score": 1200}]},
)
expected = (
0.45 * math.log1p(100000)
+ 0.27 * math.log1p(5000)
+ 0.18 * math.log1p(500)
+ 0.10 * math.log1p(1200)
)
self.assertAlmostEqual(expected, signals.engagement_raw(item), places=6)
def test_youtube_ranking_promotes_viral_comment_thread(self):
"""A moderately-viewed YouTube video with a 10k-like comment should
outrank a slightly-higher-viewed video with no high-signal comments."""
viral_comment = schema.SourceItem(
item_id="yt-with-viral-comment",
source="youtube",
title="Deploy to Fly.io",
body="Deploy to Fly.io walkthrough",
url="https://youtube.com/watch?v=x",
published_at="2026-03-15",
engagement={"views": 5000, "likes": 200, "comments": 50},
metadata={"top_comments": [{"score": 10000}]},
)
higher_views = schema.SourceItem(
item_id="yt-higher-views-no-comment",
source="youtube",
title="Deploy to Fly.io",
body="Deploy to Fly.io walkthrough",
url="https://youtube.com/watch?v=y",
published_at="2026-03-15",
engagement={"views": 8000, "likes": 300, "comments": 60},
metadata={"top_comments": []},
)
ranked = signals.annotate_stream(
[higher_views, viral_comment],
ranking_query="How do I deploy on Fly.io?",
freshness_mode="balanced_recent",
)
self.assertEqual("yt-with-viral-comment", ranked[0].item_id)
def test_polymarket_engagement_uses_market_fields(self): def test_polymarket_engagement_uses_market_fields(self):
item = schema.SourceItem( item = schema.SourceItem(
item_id="pm1", item_id="pm1",
@@ -221,7 +313,8 @@ class SignalsV3Tests(unittest.TestCase):
self.assertAlmostEqual(expected, result) self.assertAlmostEqual(expected, result)
def test_youtube_engagement_dominant_weight(self): def test_youtube_engagement_dominant_weight(self):
"""YouTube: views at 0.50 should dominate over comments at 0.15.""" """YouTube: views at 0.45 should dominate. With no top-comment data,
the remaining 0.90 of weight is split views/likes/comments 0.45/0.32/0.13."""
item = schema.SourceItem( item = schema.SourceItem(
item_id="yt1", source="youtube", title="T", body="B", item_id="yt1", source="youtube", title="T", body="B",
url="https://example.com", url="https://example.com",
@@ -230,9 +323,9 @@ class SignalsV3Tests(unittest.TestCase):
result = signals.engagement_raw(item) result = signals.engagement_raw(item)
self.assertIsNotNone(result) self.assertIsNotNone(result)
expected = ( expected = (
0.50 * math.log1p(10000) 0.45 * math.log1p(10000)
+ 0.35 * math.log1p(500) + 0.32 * math.log1p(500)
+ 0.15 * math.log1p(80) + 0.13 * math.log1p(80)
) )
self.assertAlmostEqual(expected, result) self.assertAlmostEqual(expected, result)
@@ -252,7 +345,7 @@ class SignalsV3Tests(unittest.TestCase):
) )
result = signals.engagement_raw(item) result = signals.engagement_raw(item)
self.assertIsNotNone(result) self.assertIsNotNone(result)
expected = 0.50 * math.log1p(5000) expected = 0.45 * math.log1p(5000)
self.assertAlmostEqual(expected, result) self.assertAlmostEqual(expected, result)
def test_tiktok_engagement_dominant_weight(self): def test_tiktok_engagement_dominant_weight(self):
@@ -264,9 +357,9 @@ class SignalsV3Tests(unittest.TestCase):
result = signals.engagement_raw(item) result = signals.engagement_raw(item)
self.assertIsNotNone(result) self.assertIsNotNone(result)
expected = ( expected = (
0.50 * math.log1p(50000) 0.45 * math.log1p(50000)
+ 0.30 * math.log1p(3000) + 0.27 * math.log1p(3000)
+ 0.20 * math.log1p(200) + 0.18 * math.log1p(200)
) )
self.assertAlmostEqual(expected, result) self.assertAlmostEqual(expected, result)
@@ -286,7 +379,7 @@ class SignalsV3Tests(unittest.TestCase):
) )
result = signals.engagement_raw(item) result = signals.engagement_raw(item)
self.assertIsNotNone(result) self.assertIsNotNone(result)
expected = 0.30 * math.log1p(1000) expected = 0.27 * math.log1p(1000)
self.assertAlmostEqual(expected, result) self.assertAlmostEqual(expected, result)
def test_instagram_engagement_dominant_weight(self): def test_instagram_engagement_dominant_weight(self):
+107
View File
@@ -105,5 +105,112 @@ class TestExpandTikTokQueries(unittest.TestCase):
self.assertEqual(len(queries), 1) self.assertEqual(len(queries), 1)
class TestTikTokCommentsGate(unittest.TestCase):
def test_gate_requires_key_and_token(self):
from lib import env
self.assertFalse(env.is_tiktok_comments_available({}))
self.assertFalse(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k"}
))
self.assertFalse(env.is_tiktok_comments_available(
{"INCLUDE_SOURCES": "tiktok_comments"}
))
self.assertTrue(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "tiktok,tiktok_comments"}
))
def test_gate_case_matches_youtube_pattern(self):
from lib import env
# Matches the existing youtube_comments behaviour — plain substring match via _parse_include_sources.
self.assertTrue(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "TIKTOK,TIKTOK_COMMENTS"}
))
class TestTikTokEnrichWithComments(unittest.TestCase):
def test_empty_items_returns_empty(self):
from lib import tiktok
self.assertEqual([], tiktok.enrich_with_comments([], token="k"))
def test_missing_token_is_noop(self):
from lib import tiktok
items = [{"video_id": "1", "url": "https://www.tiktok.com/@u/video/1", "engagement": {"views": 100}}]
result = tiktok.enrich_with_comments(items, token="")
self.assertNotIn("top_comments", result[0])
def test_fetch_post_comments_parses_sc_response(self):
from unittest.mock import patch
from lib import tiktok
fake_sc_response = {
"comments": [
{"text": "loved it", "user": {"nickname": "Alice"},
"digg_count": 420, "create_time": 1709251200},
{"text": "meh", "user": {"nickname": "Bob"},
"digg_count": 3, "create_time": 1709251300},
{"text": "", "user": {"nickname": "Skip"},
"digg_count": 999, "create_time": 1709251400},
],
"total": 3,
}
class FakeResp:
def raise_for_status(self):
pass
def json(self):
return fake_sc_response
with patch.object(tiktok, "_requests") as mock_req:
mock_req.get.return_value = FakeResp()
out = tiktok._fetch_post_comments(
"https://www.tiktok.com/@u/video/1",
token="k",
max_comments=5,
)
# Empty-text comment dropped; rest sorted desc by digg_count.
self.assertEqual(2, len(out))
self.assertEqual("loved it", out[0]["text"])
self.assertEqual(420, out[0]["digg_count"])
self.assertEqual("Alice", out[0]["author"])
self.assertEqual("2024-03-01", out[0]["date"])
self.assertEqual(3, out[1]["digg_count"])
def test_fetch_post_comments_swallows_http_error(self):
from unittest.mock import patch
from lib import tiktok
with patch.object(tiktok, "_requests") as mock_req:
mock_req.get.side_effect = Exception("429 rate limit")
out = tiktok._fetch_post_comments(
"https://www.tiktok.com/@u/video/1",
token="k",
max_comments=5,
)
self.assertEqual([], out)
def test_enrich_attaches_top_comments_to_top_ranked_items(self):
from unittest.mock import patch
from lib import tiktok
items = [
{"video_id": "low", "url": "https://www.tiktok.com/@u/video/low",
"engagement": {"views": 10, "likes": 1, "comments": 0}},
{"video_id": "high", "url": "https://www.tiktok.com/@u/video/high",
"engagement": {"views": 10000, "likes": 500, "comments": 30}},
{"video_id": "mid", "url": "https://www.tiktok.com/@u/video/mid",
"engagement": {"views": 1000, "likes": 50, "comments": 5}},
]
with patch.object(tiktok, "_fetch_post_comments") as mock_fetch:
mock_fetch.return_value = [
{"author": "A", "text": "fire", "digg_count": 100, "date": "2024-03-01"}
]
tiktok.enrich_with_comments(items, token="k", max_posts=2)
# High and mid get comments; low does not.
by_id = {i["video_id"]: i for i in items}
self.assertIn("top_comments", by_id["high"])
self.assertIn("top_comments", by_id["mid"])
self.assertNotIn("top_comments", by_id["low"])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()