Support comma-separated API keys in SCRAPECREATORS_API_KEY with random
selection per run, distributing load across multiple free-tier accounts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds commands/last30days.md so /last30days registers as a Claude Code
slash command for plugin users. Users type /last30days and autocomplete
prefix-matches to the canonical /last30days:last30days form (same as
/ce:plan resolving to /compound-engineering:ce-plan).
Removes skills/last30days-nux/, a byte-identical duplicate of the root
SKILL.md that created confusing /last30days:last30days-nux autocomplete
entries via Claude Code's plugin namespacing. Root SKILL.md remains
the canonical skill source; natural-language skill-selector invocation
is unchanged.
Recovery for users on v3.0.4: /plugin update last30days then /reload-plugins.
Closes#239 (path-escape error was already fixed in v3.0.4 by dropping
the rogue 'skills' key; v3.0.5 adds the slash command on top).
Supersedes #257 (suggested './' -> '.' workaround is obsolete since
v3.0.4 dropped the 'skills' key entirely, matching ecosystem standard).
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
PR #260 wired YouTube comment enrichment against
`/v1/youtube/video/comments` with `id=<video_id>`, but the endpoint
requires `url=https://www.youtube.com/watch?v=<video_id>`. Every enrich
call was returning 400 "missing_parameter: you must provide a url", so
no YouTube items ever carried `top_comments`.
The SC transcript fallback (`_sc_fetch_transcript`) had the identical
contract mistake. It was latent because `_fetch_transcript` prefers
yt-dlp and the SC path only fires when yt-dlp is missing, but it would
have failed the same way on hosts without yt-dlp installed.
Switching both callers to `url=` surfaces a second issue in the
response parser: SC returns `author` as `{"name": "@handle", ...}` and
nests like counts under `engagement.likes`, not top-level. The parser
was reading `author` as a string and missing the nested likes, so even
after the param fix every comment would land with an object-shaped
author and 0 likes.
- `_fetch_video_comments`: send `url=` on both urllib and requests branches
- `_sc_fetch_transcript`: same
- Response parser: extract `author.name` when author is a dict, read
`engagement.likes` when top-level `likes` is absent, prefer
`publishedTime` / `publishedTimeText` for date. Legacy string-author
and top-level-likes shapes still work, so existing mocks are unchanged.
Verified live against api.scrapecreators.com: `_fetch_video_comments`
now returns fully-populated comments with real @handles and like
counts (e.g. "@JennyNicholson: ... (49000 likes, 2025-04-15)"). All
tests in youtube_yt/normalize/signals/render pass.
Plan: docs/plans/2026-04-15-002-fix-youtube-comments-scrapecreators-param-plan.md
🤖 Generated with Claude Opus 4.6 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restore the rich synthesis output by closing three prompt-level loopholes
that let the model silently take a degraded path:
1. Research Execution precondition gate. Steps 0.55 (entity resolution)
and 0.75 (query planner) are now non-skippable on WebSearch platforms.
--emit md is banned as a primary user-facing flow; --emit=compact with
--plan is mandatory. OpenClaw --auto-resolve fallback preserved.
2. WebSearch "Sources:" mandate override. The WebSearch tool description
contains a CRITICAL/MUST mandate to append a Sources section. That is
explicitly superseded inside /last30days with matched-register
CRITICAL/MANDATORY override language and a BAD/GOOD example. The
existing web-source line is the citation; nothing appends below the
invitation.
3. Pre-present self-check. Before displaying, the model verifies bold
per-paragraph headlines, per-source emoji stats, quoted highlights,
Polymarket block, coverage footer, and (critically) no trailing
Sources block. One regeneration permitted if checks fail.
Also adds explicit MANDATORY language to the "What I learned" template
requiring bold headline phrases on every narrative paragraph.
Root cause: same-session A/B on 2026-04-15 between /last30days kanye
west (rich output, ran Steps 0.55 + 0.75, --emit=compact --plan) and
/last30days hermes ai (bland output, skipped both, --emit md) showed
the template was fine -- the model was lazily taking a shortcut SKILL.md
tolerated. No engine, render.py, or contributor PR was the cause.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
plugin.json has declared "skills": ["./"] unchanged since v2.1.0. That
value used to work on older Claude Code but current versions reject it
with: Path escapes plugin directory: ./ (skills). The error surfaces
on fresh /doctor runs even after v3.0.3 restored the archive contents.
Fix: omit the "skills" key entirely. Every other plugin in the Claude
Code marketplace ecosystem (compound-engineering, coding-tutor, codex,
esper, 15+ Anthropic official plugins) omits this key and the loader
auto-discovers skills/*/SKILL.md. Matching that pattern clears the
path-escape error on v2.1.109+ and remains compatible with older
Claude Code versions where the default-discovery path was already the
working code path.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
v3.0.3's fix (#262) restored skills/ and .claude-plugin/ to the git
archive, which Claude Code needs for /plugin install. But
scripts/build-skill.sh uses the same archive to produce the claude.ai
.skill bundle, which must contain exactly one root SKILL.md and stay
under the 200-file cap.
Fix: after git archive, 'zip -d' strips both directories from the
.skill bundle. git archive output is unchanged (Claude Code still
gets the full tarball on /plugin install).
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
v3.0.1 added .gitattributes rules that excluded both directories from
git archive output, shrinking the claude.ai .skill bundle. But Claude
Code's /plugin install fetches the SAME archive, so users installing
v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no
skill files. Install appeared successful but the plugin was a useless
empty shell.
Proof:
git archive v3.0.0 | grep 'skills/|\.claude-plugin/' | wc -l # 8
git archive v3.0.1 | grep 'skills/|\.claude-plugin/' | wc -l # 0
git archive v3.0.2 | grep 'skills/|\.claude-plugin/' | wc -l # 0
No issue reports yet because:
- Cached pre-v3.0.1 installs keep working (it's the new-install path
that's broken)
- The breakage is under 24 hours old
- Users invoking the skill via natural language go through
skill-selector rather than /last30days slash command
Also reverts v3.0.2's "skills": ["skills"] back to "./", the value
that shipped in every tag from v2.1.0 through v3.0.0. That change was
a misdiagnosis; the manifest wasn't in the tarball anyway so it had
no effect on user-visible installs.
Archive file count after fix: 97 (cap is 200, plenty of room).
Follow-up: move claude.ai-specific bundle exclusions into
scripts/build-skill.sh where they belong, rather than .gitattributes
which cannot distinguish between the two distribution channels.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Two regressions were silently breaking /last30days for every user:
1. plugin.json declared "skills": ["./"], which newer Claude Code
rejects with "Path escapes plugin directory: ./ (skills)". The
skill loader refused to register the command, so /last30days
returned "Unknown command" even though /plugin list showed the
plugin as installed. Fix: "skills": ["skills"] so the loader
scans the real subdirectory.
2. marketplace.json pinned "version": "3.0.0" while plugin.json
advertised "3.0.1". The /plugin resolver used the marketplace
version and could install a phantom user-scope copy at a stale
SHA alongside the correct project-scope install, creating
duplicate skill-name collisions. Both manifests now agree on
3.0.2.
Prior attempt: commit 93fbed2 fixed (1) before but got reverted.
This lands both fixes together in a tagged release so users can
/plugin update to recover.
Recovery for affected users is in CHANGELOG.md under 3.0.2.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Internal ce:plan output shouldn't ship on the public repo.
Adds docs/plans/ to .gitignore and removes the two already-tracked
plan files from the index. Working copies stay local for reference.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- install matrix now leads with claude.ai (widest audience, one-click path)
- direct download link to GitHub release's 'latest' asset URL
- 3-step UI walkthrough with link to Settings > Capabilities > Skills
- Claude Code / OpenClaw / Gemini / manual paths still documented, collapsed
- removes the bash scripts/build-skill.sh requirement from end-user flow
- description leads with imperative 'Research' + 'what people actually say' (strong trigger signal for community/social-research prompts)
- argument-hint shows 3 concrete user phrasings instead of marketing copy
- 176 chars, well under Anthropic's 200-char cap
- preserves all source coverage (Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, web)
Per ecosystem research (April 2026), trigger description quality is the single
biggest lever separating 500-install skills from 350k-install skills.
Atomic bump across all four manifests:
- SKILL.md (root)
- skills/last30days/SKILL.md (internal spec)
- .claude-plugin/plugin.json
- gemini-extension.json
CHANGELOG entry documents the skill-upload packaging fix, vendor/ removal,
legacy plans/ removal, and the new scripts/build-skill.sh builder.
Wraps git archive with --prefix=last30days/ so the zip contains a single
top-level skill folder matching SKILL.md's name: frontmatter. Enforces:
- refuses to build with a dirty working tree (prevents shipping untracked changes)
- fails if zip exceeds 200 files (claude.ai's empirical upload cap)
- fails if zip contains more than one SKILL.md (avoids name: confusion)
Output at dist/last30days.skill (gitignored).
Both plans describe work that was already shipped:
- feat-add-websearch-source.md - websearch is in the v3 pipeline (scripts/lib/perplexity.py etc)
- fix-strict-date-filtering.md - date filtering is enforced in scripts/lib/dates.py
New planning goes in docs/plans/ following the ce:plan convention.
Root vendor/package/ was an accidentally committed extracted npm tarball
(steipete-bird-0.8.0). Zero importers: the real vendored X client lives
at scripts/lib/vendor/bird-search/, referenced by scripts/lib/bird_x.py
and tests/test_bird_x.py.
Removes 215 files + 1 .tgz, dropping repo from 406 to 191 files and
clearing the claude.ai skill-upload 200-file cap.
Adds /vendor/ to .gitignore (leading slash so scripts/lib/vendor/ is unaffected).
Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.
Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.
Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}. This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available." Return None
instead when every engagement field is missing.
evaluate_search_quality.py and e2e_comparison.py both reference
fixtures/eval_topics.json with hardcoded fallbacks. Supply the
actual fixture: 8 topics spanning all intent types, selected via
MMR dispersion across domains (tech, health, sports, finance,
consumer products).
pyproject.toml declares pytest-cov as a dev dependency and configures
[tool.coverage.run], but the generated .coverage database and htmlcov/
report directory are not gitignored. Also add .venv/ (standard Python
virtualenv) and .memsearch/ (session memory) to keep the working tree
clean for contributors.
The SKILL.md prompt header still said v2.9.5 while pyproject.toml
and the rest of the codebase are on v3.0.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Matt Van Horn