Transient DNS resolution failures (socket.gaierror, surfaced as
urllib.error.URLError with reason=gaierror) were retried with the
generic URLError handler — linear backoff (2s, 4s, 6s) and bounded by
the caller-passed `retries` parameter. For callers that pass small
retry values (e.g. lib/reddit.py::_subreddit_search uses retries=2), a
single first-attempt DNS hiccup followed by one quick retry on the
still-flaky resolver would exhaust the retry budget and wipe a whole
subreddit sweep — which the caller's broad `except Exception` then
silent-empties as `[]`.
Fix:
- Distinguish URLError-with-gaierror-reason from generic URLError via
a new `_is_dns_failure()` helper.
- For DNS failures, use exponential backoff (1s, 2s, 4s, ...) instead
of the linear default.
- For DNS failures, expand the effective retry budget to at least
MIN_DNS_RETRIES (=3) on first occurrence, so callers that passed
`retries=2` still get a meaningful retry budget for the transient
case. Non-DNS URLErrors and HTTPErrors keep the caller's value.
- DNS attempts are counted separately (`dns_attempts`) so unrelated
URLError or OSError failures within the same call don't accidentally
expand the budget further.
Reported during a community-signal pass where the Reddit subreddit
sweep silently returned zero items after a first-round transient DNS
hiccup. The fix lives at the http layer (where the retry loop is)
rather than per-source so every caller benefits.
Tests:
- Verifies a caller-passed retries=2 still gets MIN_DNS_RETRIES=3
attempts on gaierror.
- Verifies gaierror-then-success returns successfully on attempt 2.
- Verifies the exponential-backoff sleep pattern (1s, 2s) on the
retry attempts before exhaustion.
- Verifies a non-DNS URLError (ConnectionRefusedError reason) does
NOT expand the retry budget — only true DNS failures do.
All 12 http tests pass (8 baseline + 4 new). No regressions in the
broader test suite (1373 pass / 14 fail, vs 1369 pass / 14 fail on
main — the 14 failures are pre-existing and unrelated to this PR).
Twitter's edge intermittently serves an HTML anti-bot interstitial in
place of JSON when the bird-search subprocess hits a per-query rate
limit. Before this fix, that response made json.loads raise
JSONDecodeError and _run_bird_search() returned {"error": ..., "items":
[]} with the parsed exception message — silent-empty against an
orchestrator that has no way to distinguish "Twitter served HTML; retry
likely succeeds" from "no tweets matched the query."
Surfaced during a community-signal pass where a Karpathy-LLM-wiki
subquery returned zero X items, while a second identical run a few
seconds later returned full results.
Fix:
- Extract the subprocess invocation into _invoke_bird_subprocess() so
the retry loop can call it multiple times cleanly. Returns
(result, terminal_error) — terminal_error is non-None for
unrecoverable cases (subprocess timeout, spawn failure) that should
NOT be retried.
- In _run_bird_search(), wrap the json.loads parse in a retry loop
bounded by MAX_JSON_DECODE_RETRIES (=2) with JSON_DECODE_RETRY_DELAY
(=5s) between attempts.
- On non-JSON stdout, log a diagnostic that names the shape
(`looks_html`, first-80-chars stdout preview, attempt counter) so
silent-empty failures become legible in logs.
- On retry exhaustion, return an error dict whose message explicitly
names "anti-bot interstitial" as the likely cause, distinguishing
this failure from a genuine no-results case.
Subprocess timeout, spawn failure, and non-zero return-code paths are
unchanged — those are terminal and don't retry.
Tests:
- Verifies HTML-then-JSON returns success on attempt 2.
- Verifies all-HTML returns the diagnostic error dict mentioning the
anti-bot interstitial cause.
- Verifies subprocess timeout is NOT retried.
All 12 bird_x tests pass (9 baseline + 3 new).
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.
This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.
The resulting source-gating model is three-tier and intentional:
• **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
grounding, **tiktok, instagram, threads**
• **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
perplexity (different paid API — OpenRouter),
tiktok_comments / youtube_comments (N× extra SC calls per video)
• **--search per-query opt-in** (relevance reasons):
pinterest (visual pins, narrow utility),
xiaohongshu (Chinese-market specific)
Changes:
- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
the new contract and includes a regression guard ("INCLUDE_SOURCES
should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
runs. Drops false INCLUDE_SOURCES requirement for
tiktok/instagram/threads; corrects pinterest to mention --search;
adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
section.
Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.
YouTube (ScrapeCreators)
SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
{"error":"missing_parameter","message":"You must provide a query"}
The canonical SC parameter for that endpoint is `query`. Other SC
endpoints we use (Reddit, TikTok, Instagram) happened to work because
they use their own per-endpoint parameter names — YouTube was the lone
outlier.
Hacker News (Algolia)
Multi-keyword theme queries returned zero hits across every theme.
Algolia treats query= as strict AND across tokens, so a 4-5 word query
like "claude, personal agents, agentic infra" matches no stories.
Three changes in hackernews.py:
1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
search_hackernews and _title_matches_query normalize the query the
same way — addresses Greptile P2 #2 about the two callsites needing
to stay in sync.
2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
by token-overlap instead of requiring every token.
3. Relax _title_matches_query from all-words to any-word, *but match
on word boundaries (\b<word>\b) rather than naive substring* —
addresses Greptile P2 #1, which flagged that the previous any-word
relaxation would let "ai" falsely match "email" or "rail".
Token-overlap relevance scoring at parse time already demotes weak
matches, so word-boundary any-word matching is safe.
Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().
Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
invoking the skill knows to subtract excluded sources before displaying
the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
prose are incomplete — the agent invoking the skill won't know the flag
exists.)
Addresses Greptile review on PR #407:
- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
manually storing those keys would not see them in --list and the
interactive prompt would never offer to set them.
Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
get_config() pass it through, and add a parity test that parses
ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
caught at CI time instead of after a user reports a missing key.
- P2: os.environ.get("USER", "") silently returned "" under sudo, in
Docker without --env USER, or in CI runners that strip USER. The
resulting `security find-generic-password -a ""` call would never
match items stored by setup-keychain.sh, so all lookups silently
returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
USER is absent.
The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
Documents the cascade pattern surfaced during this session's install-modernization
arc: a `test_sync_cache_path_uses_skill_version` test asserted that a hardcoded
version pin in `sync.sh` matched the version frontmatter in SKILL.md. When a
release bumped SKILL.md, every open PR's CI failed simultaneously on the
unrelated stale-pin assertion. Affected at least 5 PRs across the 2026-05-13
to 2026-05-15 window (#400, #390, #392, and two others) plus required hotfix
PR #397 to unblock the queue.
The permanent fix shipped in PR #405 (deleted sync.sh + the test). This doc
captures the design lesson so the pattern doesn't reappear: don't write
consistency tests that read two files and assert one matches a value derived
from the other. Either derive at runtime from a single source of truth, or
self-skip / merge-base-scope the test so deletion is a non-event.
Created via /ce-compound. Includes:
- docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md
(the new learning — first entry under docs/solutions/)
- CONCEPTS.md (new — 4 entries: Skill, Engine, Harness, Beta channel,
capturing project-specific vocabulary that surfaced across the session)
- AGENTS.md (added one-line Structure entries surfacing docs/solutions/ and
CONCEPTS.md so fresh agents discover them)
- docs/plans/2026-04-22-{002,003,005,006}-*-plan.md (added deprecation banner
to each, pointing readers at PR #405 and the new docs/solutions entry —
these 4 historical plans still reference the deleted sync.sh inline)
Also: closed PR #379 (j-sperling's workaround for the same cascade,
superseded by PR #405's permanent fix).
The same `^version:\s*"([^"]+)"\s*$` regex (or a slight variant) was
duplicated across three files: render.py inline, test_plugin_contract.py
local helper, test_version_consistency.py local helper. A future change
to the SKILL.md frontmatter version format would have needed to update
three places without any compile-time pressure to keep them in sync.
New skills/last30days/scripts/lib/skill_meta.py provides:
- `_VERSION_RE` private compiled pattern (accepts double-quoted,
single-quoted, or unquoted YAML version scalars per the widening
landed in 997708a)
- `read_skill_version(skill_md_path: Path) -> str | None` helper that
catches OSError + UnicodeDecodeError and returns None on miss
Callers updated:
- render.py::_skill_version now calls skill_meta.read_skill_version
inside the SKILL.md fallback loop, returning `read_skill_version(...) or "?"`.
Semantically equivalent to the old break-after-first-SKILL.md logic.
- test_plugin_contract.py and test_version_consistency.py import the
helper instead of defining the regex inline. Both files use the
established sys.path.insert pattern.
Added tests/test_skill_meta.py with 6 direct unit tests covering the
helper's full contract: missing file, undecodable bytes, no-version-line,
and all three quoting styles (double, single, unquoted). Previously the
helper was only exercised transitively through render._skill_version().
Added test_skill_md_uses_double_quoted_version to
test_version_consistency.py — the old per-test regex incidentally
asserted "this repo's SKILL.md uses double-quotes" by being strict;
the shared helper accepts all three styles, so the assertion is now
explicit instead of implicit.
Code-reviewed by ce-code-review (8 reviewers); safe_auto fixes applied
inline (rename to _VERSION_RE, group or-chain instead of generator,
docstring tightened, dropped unnecessary `from __future__ import
annotations`, tightened signature to Path-only).
Conftest.py refactor for the sys.path.insert duplication across ~20 test
files filed as issue #411 — out of scope for this PR (touches many
files, separate concern).
Test results: 23 passed in the affected test set (16 prior + 6 new
test_skill_meta tests + 1 new double-quote assertion). Full suite shows
same 13 pre-existing failures as main; zero new failures.
Mirrors the multi-harness reframing of the project itself. CLAUDE.md is
Claude-Code-specific by name; AGENTS.md is the multi-harness convention
that Codex, Cursor, Gemini CLI, GitHub Copilot, and most other Agent
Skills hosts also read. The canonical content belongs in the file
multi-harness tooling expects.
git mv preserves history — the Orientation section and everything else
that was in CLAUDE.md is now tracked under AGENTS.md, with full blame
continuity. The new CLAUDE.md is a one-line `@AGENTS.md` reference so
Claude Code continues to load the content (it follows @ references).
Closes the spirit of #335 (closed in favor of this fresh PR after the
sync.sh thread became obsolete via PR #405).
Two changes:
1. Top-of-file description reframed from "Claude Code skill" to
"Agent Skills package... installable across Claude Code (most common
host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other
Agent Skills hosts". The skill works across every major agent host
after the install-modernization work in PR #400/#404/#405/#409.
Calling it "Claude Code skill" undersells the surface and biases
contributors toward Claude-Code-specific assumptions.
2. New ## Orientation section (4 bullets) framing the project for
contributors who would otherwise read the python3 invocation in
## Commands and form a CLI-first mental model. Names the trap
explicitly with one concrete invalid-syntax example
(`/last30days OpenClaw --emit=html | pbcopy` — slash commands don't
pass shell mechanics through). Bullets adapted from #335 with
multi-harness framing replacing the Claude-Code-only framing.
No code changes. No SKILL.md changes. CLAUDE.md only. AGENTS.md
inherits via @CLAUDE.md.
The Step 1 and comparison-mode resolver loops walked a hardcoded list
of install paths trying to find scripts/last30days.py. Two problems:
1. The list was never exhaustive — it covered ~/.codex/skills, but not
~/.claude/skills, ~/.cursor/skills, ~/.gemini/skills, ~/.copilot/skills,
~/.hermes/skills/research, etc. PR #406 was about to fix that by
enumerating more paths, but enumeration is the wrong shape.
2. The resolver could pick a different install than the SKILL.md the
model loaded from. Spec-vs-engine divergence is subtle and confusing
when it triggers.
The model already knows the SKILL.md path it loaded (from its Read tool
result). Templating that into the bash block is strictly better than
guessing across an enumerated list:
- Works for every harness without enumeration (Hermes, Cursor, anything
new) because we just use wherever the harness loaded SKILL.md from
- Aligns spec with engine — the engine runs from the same install the
spec was read from
- Deletes ~80 lines of bash across Step 1 + comparison-mode + the
prose preamble describing the resolver
Mechanics:
- SKILL_DIR placeholder in both bash blocks — model substitutes the
absolute path of the directory containing the SKILL.md it just Read
- One-line validation `[ ! -f "$SKILL_DIR/scripts/last30days.py" ]`
catches bad templating with a clear error
- All references to $SKILL_ROOT replaced with $SKILL_DIR (badge
VERSION lookup, prose description in the LAW-7 preamble area)
- STEP 0 unchanged — different concern (marketplaces stale-clone hop)
Version 3.2.3 -> 3.2.4 (behavior change: install paths the resolver
list never enumerated now work; install paths it did enumerate work
the same way they used to but via the SKILL_DIR template).
Adds the macOS Keychain as the lowest-priority credential source on Darwin.
Items stored as generic passwords with service name "last30days-<KEY>" for
the current user are picked up automatically by get_config() — file env
and process env still win on collision.
No new config knob: behavior is strictly additive. On non-Darwin (or when
the `security` binary is missing) the loader is a no-op, so Linux/Windows
behavior is unchanged.
Priority (highest wins):
1. Environment variables
2. .claude/last30days.env (per-project)
3. ~/.config/last30days/.env (global)
4. macOS Keychain items prefixed last30days- (new)
Includes:
- lib/env.py: KEYCHAIN_SERVICE_PREFIX constant, _load_keychain helper
(platform-gated, shutil.which-gated, subprocess-error tolerant),
wiring into get_config before get_openai_auth so OPENAI_API_KEY can
come from Keychain too, _CONFIG_SOURCE reports "keychain" when no
file source is present.
- scripts/setup-keychain.sh: bash helper with interactive set,
--list, --delete, --replace modes. Uses `security add-generic-password`.
- tests/test_env_keychain.py: 12 tests covering platform gate,
missing-binary gate, success path, whitespace stripping, subprocess
errors swallowed, get_config precedence, and an OPENAI_AUTH wiring
regression test.
- tests/test_env_cookies.py: existing integration test mocks the new
_load_keychain hook so it stays hermetic on Darwin developer
machines that have real keychain entries.
- README.md: new "macOS Keychain (optional)" subsection under
"Bring your own keys" documenting setup-keychain.sh and the manual
`security add-generic-password` invocation.
Tested on macOS with a populated keychain and against the existing pytest
suite — CI-tracked tests (test_plugin_contract.py, test_version_consistency.py)
plus all env-touching tests pass. Pre-existing unrelated failures in
test_store.py / test_watchlist_commands.py / test_setup_openclaw.py /
test_footer_nudge_suppression.py are untouched.
Every job sync.sh did has a better replacement:
- Per-harness skill dirs (~/.claude/skills, ~/.codex/skills, ~/.agents/skills):
`npx skills add . -g -y` writes to every detected harness's home dir and
uses symlinks by default. Edits propagate live — no re-deploy step.
- Hermes (~/.hermes/skills/research/last30days):
`hermes skills install mvanhorn/last30days-skill --force` pulls from
GitHub and handles the deploy itself. The script wrapping was redundant.
- OpenClaw variant: `clawhub install last30days-official` is what users
already run per the README; the maintainer doesn't need a separate
variant-deploy step in the public repo's scripts.
- Claude marketplace cache (~/.claude/plugins/cache/...): this was a
"test against the official install path" hack we shouldn't have been
recommending. With PR #400's resolver collapse, STEP 0 no longer
enforces the cache as the only valid SKILL.md location. Just install
the skill normally via `npx skills` or the marketplace.
Cleanup:
- DELETE skills/last30days/scripts/sync.sh
- tests/test_version_consistency.py — drop test_sync_cache_path_uses_skill_version
- CLAUDE.md — replace the sync.sh command + rule with `npx skills add . -g -y`
- HERMES_SETUP.md — Installation now uses `hermes skills install --force`;
developer-alternative section shows the symlink pattern for live editing
- render.py — _skill_version docstring no longer attributes the
".claude-plugin absent" case to sync.sh; explains it via per-harness
install paths in general
- .github/PULL_REQUEST_TEMPLATE.md — drop the "Ran bash scripts/sync.sh"
checklist item
CHANGELOG and historical docs (release notes, plan files) keep their
existing sync.sh mentions as accurate history.
`npx skills add` defaults to project-local install (`./.skills/`,
committed with the repo). For a research-the-world skill like this one,
that's almost never what users want — they want it available across all
projects, not scoped to whichever directory they happened to run the
install from.
Adding `-g` (global) to every npx skills example in the README:
- Top-of-file install snippet
- Install table row
- Claude Code subsection's "alternative via npx skills" example
- Codex/Cursor/etc. subsection's default, per-harness, update, list,
and remove commands
Brief one-liner explains what `-g` does and notes that dropping it
gives a project-local install for users who want team consistency on
a specific codebase.
The bash banner accounting used raw substring matching while
pipeline.py normalises EXCLUDE_SOURCES via .strip().lower(). With
EXCLUDE_SOURCES=TikTok,Instagram (or with surrounding spaces),
pipeline correctly excludes the sources but the banner did not
deduct them — count showed 1-2 higher than what the pipeline
actually runs. Normalisation now mirrors the Python side
(lowercase, collapse whitespace around commas, strip outer whitespace).
Reproducer (clean HOME with config EXCLUDE_SOURCES=TikTok,Instagram):
before: /last30days: Ready — 7 sources active.
after: /last30days: Ready — 5 sources active.
Addresses Greptile review comment P1 on #399.
The native `gemini extensions install` path was a workaround for the
v0.9.0 installer bug (still unresolved per upstream issue #11452).
Now that `npx skills add -a gemini-cli` covers Gemini cleanly with the
same install/update story as every other supported harness, the native
path is just one more confusing option to maintain. Users on Gemini get
the same recommendation everyone else does.
Removes the dedicated "Gemini CLI (native extension)" subsection and
the separate table row. Gemini CLI is now surfaced once, in the npx
skills section, alongside Codex, Cursor, Copilot, and the rest.
npx skills supports 50+ harnesses via the -a flag, including gemini-cli,
github-copilot, windsurf, cline, continue, roo, aider-desk, opencode,
goose, and more — not just the few I'd listed initially. Updating to
reflect that breadth.
- Top-of-file snippet now reads "Codex, Cursor, Copilot, Gemini CLI, or
any of 50+ Agent Skills hosts" (was: "Codex, Cursor, Copilot, or any
Agent Skills host" + Gemini listed separately in the table footer).
- Install table: same expansion; Gemini CLI native-extension row relabeled
to clarify it's the native path (not the only Gemini option).
- npx skills subsection: lists the most common harness flags and links
to the upstream vercel-labs/skills repo for the full list.
- Gemini CLI native subsection: now leads with "the npx skills path
above is simpler" and frames the native install as the alternative
for users with an existing Gemini extensions workflow or who hit
the v0.9.0 installer bug.
The skill is now installable across every major agent harness after the
SKILL.md path-resolver work landed in PR #400 + #404. README didn't yet
reflect that — the install table only listed Claude Code, OpenClaw, and
Gemini CLI, and the top-of-file install snippets featured Hermes (an
internal dev workflow, not a public install method).
Restructured the install section:
- Top-of-file snippets: just Claude Code (recommended, auto-updates) and
the universal `npx skills add` one-liner. Dropped Hermes from the
prominent spot (internal-only); pointed everything else to the Install
section below.
- Install table: added a third column for update commands, since every
harness now has a distinct update path worth surfacing. Added the
`npx skills` row covering Codex/Cursor/Copilot/any Agent Skills host.
- Claude Code subsection: explains why it's recommended (marketplace
handles versioned cache + auto-refresh) and notes that the agent-skills
install also works on Claude Code if preferred (`-a claude-code`).
- New "Codex, Cursor, Copilot, and other Agent Skills hosts" subsection:
shows the default install, per-harness `-a` targeting, and the update
commands (`npx skills update last30days` for one skill, bare
`npx skills update` for all).
- Manual (developer) subsection: switched from a clone-into-skills-dir
recipe to a clone + symlink recipe. Symlink keeps the install in sync
with the working tree as you edit, no re-copy on each change.
No code changes. No version bump (docs-only).
Greptile's review flagged mktemp -t as non-portable between BSD and GNU.
The suggested replacement (mktemp "$TMPDIR/...XXXXXX.json") is correct
about dropping -t but still puts X's in the middle of the template name
(XXXXXX.json), which BSD mktemp does not substitute — only X's at the
end of the basename are replaced on BSD. Verified on macOS:
mktemp "$TMPDIR/last30days-test.XXXXXX.json"
→ /var/folders/.../last30days-test.XXXXXX.json (X's left literal)
The fully portable form uses trailing X's and drops the .json suffix
(engine reads by path, not extension):
mktemp "$TMPDIR/last30days-test.XXXXXX"
→ /var/folders/.../last30days-test.DXAHzR (X's substituted)
Verified on bash and zsh, BSD/macOS. GNU/Linux is already fine since
GNU substitutes X's wherever they appear in the basename.
Applied to both --competitors-plan (comparison-mode block) and --plan
(Step 1 block) tmpfile writes.
Closes#403.
The SKILL.md templates instructed the model to invoke the engine with
inline single-quoted JSON: `--plan '$JSON'` and `--competitors-plan '{...}'`.
When any resolved field value contained an apostrophe (common in `context`
strings like "McDonald's", "people's choice", or contracted forms like
"don't", "won't"), the inner `'` closed the outer single-quote and broke
shell parsing before the engine was even invoked.
Observed during PR #400 testing: a Codex run hit the trap and self-healed
by re-encoding, wasting one engine invocation and ~30s of latency.
Fix: switch both templates to the heredoc + tmpfile pattern. The engine's
`parse_plan()` and `parse_competitors_plan()` already check
`os.path.isfile(plan_str)` and read from disk — only the SKILL.md prose
needed to change.
The quoted heredoc marker (<<'PLAN_EOF') is load-bearing: it suppresses
shell interpolation so apostrophes, $, backticks, etc. pass through verbatim.
A trap on EXIT cleans up the tmpfile after the engine call returns.
LAW 7's "MUST contain --plan" self-check guidance and Step 1's invocation
example both updated to reference the file form. Comparison-mode invocation
block updated the same way for --competitors-plan.
Version bump 3.2.2 -> 3.2.3 because this is a behavior change users
running comparison-mode queries will notice (no more "shell quoting error,
retrying" sequences on apostrophe-containing context strings).
Two real bugs flagged in the automated review of PR #400; both small.
1. render.py::_skill_version manifest with no "version" key
`json.loads(manifest.read_text()).get("version", "?")` returned "?"
immediately on a valid JSON manifest that lacked the "version" key,
never falling through to the SKILL.md frontmatter fallback. Contradicted
the docstring's "Returns '?' only if both sources are missing" contract.
Same shape if version is present but empty string ("" produces the
broken badge `🌐 last30days v · synced ...`).
Fix: pull the version out of the parsed dict, then `continue` to the
next ancestor if it's None or empty. Falls through to the SKILL.md
walk only after exhausting every ancestor.
2. SKILL.md STEP 0 re-read target hardcoded to nested cache layout
STEP 0 told the model to re-read from
`$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md` — the new nested
layout. But Step 1's resolver explicitly handles both shapes
(nested `{cache}/{version}/skills/last30days/` and flat
`{cache}/{version}/`), noting "Both shapes ship in the wild." On an
install where the highest-versioned cache happens to be the older flat
shape, STEP 0's re-read target wouldn't exist; the model would silently
stay on the stale marketplaces/ copy STEP 0 was supposed to move it
away from — the exact failure mode this guard was added to prevent.
Fix: extend the STEP 0 bash to resolve $CLAUDE_CACHE_SKILL_MD by
probing both layouts, then have the model hop to that resolved path
instead of constructing the path from a hardcoded suffix.
Two new tests in tests/test_skill_version.py cover the missing-key and
empty-string cases for fix 1. Fix 2 is exercised via the bash probe at
verify time (the STEP 0 prose-contract test isn't unit-testable from
Python, but the dual-layout bash is verified to resolve to the correct
SKILL.md on both shapes).
Stale finding skipped: greptile also flagged a missing try/except on the
SKILL.md read_text() call, but that was already addressed during the
ce-code-review safe_auto pass earlier in this PR — current code wraps it
in `try/except (OSError, UnicodeDecodeError)`, strictly more defensive
than the suggested fix.
zsh errors on globs that match nothing instead of returning the literal
pattern (bash's default), and `2>/dev/null` does not suppress the error
because it comes from the shell's glob expansion before `ls` even runs.
Under Codex (which executes the SKILL.md bash via zsh), STEP 0 and the
Step 1 / comparison-mode resolvers emitted noisy "no matches found"
errors on machines without a Claude plugin cache populated.
Replaces all three `ls -d $HOME/.claude/plugins/cache/last30days-skill/last30days/*/`
invocations with `find ... -mindepth 1 -maxdepth 1 -type d 2>/dev/null`.
find is POSIX-portable, errors silently when the base dir doesn't exist,
and never triggers shell glob errors. `sort -V | tail -1` precedence
preserved (verified: picks 3.10.0 over 3.2.1 over 3.1.0). Trailing-slash
strip removed because find doesn't append slashes.
Observed in Codex session running /last30days against PR #400 with the
Claude plugin cache deleted - bash output was:
zsh:1: no matches found: /Users/.../last30days/*/
After fix: clean empty output, exit 0, STEP 0 correctly treats it as
"no cache present, do not hop", resolver falls through to per-harness
skill dirs as designed.
12 fixes from the multi-agent code review on PR #400:
Version 3.2.1 -> 3.2.2 across all manifests (SKILL.md frontmatter + body
header, pyproject.toml, .claude-plugin/{plugin,marketplace}.json, sync.sh
cache path). The PR ships observable behavior changes (STEP 0 logic flip,
resolver order change, badge fallback) that should not silently appear
under the same version number — the new fallback reads SKILL.md version
directly so the badge would otherwise be misleading.
render.py::_skill_version:
- `import re` moved to module top
- _VERSION_RE extracted as a module-level compiled pattern that accepts
double-quoted, single-quoted, and unquoted YAML version scalars
- `break` -> `continue` on corrupt manifest, so a corrupt inner manifest
no longer shadows a valid outer one
- Wrap SKILL.md read_text() in try/except for UnicodeDecodeError to keep
badge emission from crashing on mis-encoded SKILL.md
- Docstring clarifies precedence; inline comment marks the fallback boundary
between the manifest walk and the SKILL.md walk
tests/test_skill_version.py (new): 7 unit tests for the fallback paths
(manifest absent, manifest corrupt, corrupt-inner + valid-outer, both
absent, SKILL.md without version, single-quoted, unquoted).
tests/test_plugin_contract.py: tombstone test asserting .codex-plugin/
stays removed (was the only CI guard against accidental reintroduction).
SKILL.md:
- STEP 0 bash echoes CLAUDE_CACHE_LATEST so the model can see the
resolved value when deciding whether to hop
- "Both shapes ship in the wild" comment now names the two cache layouts
(nested {cache}/{version}/skills/last30days/ vs flat {cache}/{version}/)
- Comparison-mode bash invocation gets its own inline SKILL_ROOT resolver
(latent gap: the contract tells the model to skip Step 1 on comparison
queries, so SKILL_ROOT was previously unset there)
CHANGELOG.md: [Unreleased] entries for the resolver rewrite and the
breaking removal of Codex native-plugin support.
All 9 reviewer personas surfaced findings; 3 cross-reviewer corroboration
clusters were promoted (import re, "both shapes" comment, missing fallback
tests). Maintainability follow-up flagged: regex now duplicated across
render.py and 2 test files; could consolidate via shared lib/skill_meta.py
helper in a future PR.
STEP 0 (CANONICAL PATH SELF-CHECK) used to force any SKILL.md load that wasn't
under $HOME/.claude/plugins/cache/last30days-skill/last30days/{version}/ to
re-Read from there. That guard is Claude-Code-specific (defends against the
marketplaces/ stale-clone bug) and broke under non-Claude installers like
`npx skills add`, ~/.codex/skills/, and ~/.agents/skills/.
The new STEP 0 narrows the check to its actual target: fire only when the
loaded SKILL.md path contains /.claude/plugins/marketplaces/. Every other
install path is trusted. The 2026-04-22 incident workaround is preserved
without breaking other harnesses.
Step 1 SKILL_ROOT resolver collapses the Codex-first / Claude-fallback /
CWD-fallback chain into a single precedence walk: Claude plugin cache
(versioned) first, then ~/.codex/skills, ~/.agents/skills, repo checkout,
./.skills/last30days (npx skills install dir), CWD, and GEMINI_EXTENSION_DIR.
Also drops Codex native plugin support: .codex-plugin/plugin.json is deleted,
the badge VERSION jq fallback in line 108 stops looking at it, and render.py's
_skill_version no longer scans for it. Codex users install via `npx skills add`
or the per-harness skill dir going forward.
render.py::_skill_version gains a SKILL.md frontmatter fallback so the badge
no longer emits `v?` on install dirs that sync.sh populates (which don't
include .claude-plugin/plugin.json).
sync.sh was written against the layout of mvanhorn/last30days-skill-private
(`.../cache/last30days-skill-private/last30days-3/{version}`) and that path
was never updated when this public repo got its own copy. Running sync.sh
from here populated the BETA channel's cache (`/last30days-beta`) instead
of this repo's own `/last30days` cache, so devs working in this repo could
not test their changes via the public slash command without waiting for a
marketplace release.
Path now derives from this repo's own manifests:
- marketplace name `last30days-skill` (.claude-plugin/marketplace.json)
- plugin name `last30days` (.claude-plugin/plugin.json)
Drops the `last30days-3-nogem` target along with it - that's a private-repo
variant with no public equivalent.
Updates test_sync_cache_path_uses_skill_version to assert the new path
pattern and clarifies the COMMON_TARGETS comment so the next person editing
it understands which marketplace/plugin name segments come from where.
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES
config key. Two coupled changes:
1. pipeline.available_sources() filters out any source listed in
config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive,
whitespace-tolerant) before returning.
2. hooks/scripts/check-config.sh "Ready — N sources active" banner
subtracts excluded sources from the ScrapeCreators +3 (Reddit
comments + TikTok + Instagram) so the count matches what the
pipeline actually runs.
Use case: skip TikTok/Instagram on runs where you only want
text-substantive sources, without unsetting SCRAPECREATORS_API_KEY
(which would also kill Reddit comments). The existing INCLUDE_SOURCES
allowlist covers Perplexity opt-in but doesn't cover this denylist case
— tiktok and instagram are added unconditionally when
SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key.
Tests (tests/test_pipeline_v3.py::TestExcludeSources):
- excludes tiktok+instagram when listed
- no exclusion when env unset or empty string
- case-insensitive + whitespace-tolerant parsing
- works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
Five provider modules (pinterest, threads, instagram, tiktok, youtube_yt)
and watchlist.py each carried a try/except `requests` import with parallel
urllib + requests branches. The urllib path already used the
stdlib-only wrapper at `lib/http.py` (retries, 429 handling, HTTPError).
This collapses every dual-branch into a single `http.get`/`http.post`
call and removes the `requests` dependency from `pyproject.toml`.
Also drops 4 transitive deps (urllib3, certifi, charset-normalizer, idna)
from the lockfile, leaving the skill stdlib-only at runtime.
Tests for tiktok comments and watchlist delivery were rewritten to mock
`lib.http` directly instead of the now-removed `requests` module.
Out of scope but flagged during review: the 13 surviving SC call sites
share a near-identical scaffold and would benefit from a
`http.scrapecreators_get(url, params, token, ...)` helper. Filed for a
follow-up PR rather than expanding scope here.
test_sync_cache_path_uses_skill_version asserts that sync.sh's plugin
cache path includes the version from SKILL.md frontmatter. The frontmatter
moved to 3.2.1 in #371 but sync.sh still pointed at 3.2.0, leaving CI red
on every PR.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(digg): bump POSTS_PER_CLUSTER to 5 and render limit to 3
Match the per-item enrichment cap and inline-display cap used by the
other sources (Reddit, HN, YouTube, TikTok, GitHub all use 5 fetched /
3 displayed). At the previous 3/2 caps the engine routinely truncated
cluster context — a recent run on cli-printing-press lost the Jason
Calacanis quote tweet entirely because the display cut off after Garry
Tan's first two posts.
* feat(digg): rename 'Digg AI 1000' to 'Digg' in user-facing strings
Drop the 'AI 1000' suffix from the footer line, source label, inline
quote attribution ('via Digg'), why_relevant, container, mock title,
SKILL.md source list, and README sources table. Internal code comments
and docstrings still reference the upstream Digg AI 1000 product.
Bumps version to 3.2.1 and adds a CHANGELOG entry covering this rename
and the POSTS_PER_CLUSTER / render-limit bumps from the prior commit.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Trevin Chow