Addresses Greptile review on PR #407:
- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
manually storing those keys would not see them in --list and the
interactive prompt would never offer to set them.
Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
get_config() pass it through, and add a parity test that parses
ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
caught at CI time instead of after a user reports a missing key.
- P2: os.environ.get("USER", "") silently returned "" under sudo, in
Docker without --env USER, or in CI runners that strip USER. The
resulting `security find-generic-password -a ""` call would never
match items stored by setup-keychain.sh, so all lookups silently
returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
USER is absent.
The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
Adds the macOS Keychain as the lowest-priority credential source on Darwin.
Items stored as generic passwords with service name "last30days-<KEY>" for
the current user are picked up automatically by get_config() — file env
and process env still win on collision.
No new config knob: behavior is strictly additive. On non-Darwin (or when
the `security` binary is missing) the loader is a no-op, so Linux/Windows
behavior is unchanged.
Priority (highest wins):
1. Environment variables
2. .claude/last30days.env (per-project)
3. ~/.config/last30days/.env (global)
4. macOS Keychain items prefixed last30days- (new)
Includes:
- lib/env.py: KEYCHAIN_SERVICE_PREFIX constant, _load_keychain helper
(platform-gated, shutil.which-gated, subprocess-error tolerant),
wiring into get_config before get_openai_auth so OPENAI_API_KEY can
come from Keychain too, _CONFIG_SOURCE reports "keychain" when no
file source is present.
- scripts/setup-keychain.sh: bash helper with interactive set,
--list, --delete, --replace modes. Uses `security add-generic-password`.
- tests/test_env_keychain.py: 12 tests covering platform gate,
missing-binary gate, success path, whitespace stripping, subprocess
errors swallowed, get_config precedence, and an OPENAI_AUTH wiring
regression test.
- tests/test_env_cookies.py: existing integration test mocks the new
_load_keychain hook so it stays hermetic on Darwin developer
machines that have real keychain entries.
- README.md: new "macOS Keychain (optional)" subsection under
"Bring your own keys" documenting setup-keychain.sh and the manual
`security add-generic-password` invocation.
Tested on macOS with a populated keychain and against the existing pytest
suite — CI-tracked tests (test_plugin_contract.py, test_version_consistency.py)
plus all env-touching tests pass. Pre-existing unrelated failures in
test_store.py / test_watchlist_commands.py / test_setup_openclaw.py /
test_footer_nudge_suppression.py are untouched.
Every job sync.sh did has a better replacement:
- Per-harness skill dirs (~/.claude/skills, ~/.codex/skills, ~/.agents/skills):
`npx skills add . -g -y` writes to every detected harness's home dir and
uses symlinks by default. Edits propagate live — no re-deploy step.
- Hermes (~/.hermes/skills/research/last30days):
`hermes skills install mvanhorn/last30days-skill --force` pulls from
GitHub and handles the deploy itself. The script wrapping was redundant.
- OpenClaw variant: `clawhub install last30days-official` is what users
already run per the README; the maintainer doesn't need a separate
variant-deploy step in the public repo's scripts.
- Claude marketplace cache (~/.claude/plugins/cache/...): this was a
"test against the official install path" hack we shouldn't have been
recommending. With PR #400's resolver collapse, STEP 0 no longer
enforces the cache as the only valid SKILL.md location. Just install
the skill normally via `npx skills` or the marketplace.
Cleanup:
- DELETE skills/last30days/scripts/sync.sh
- tests/test_version_consistency.py — drop test_sync_cache_path_uses_skill_version
- CLAUDE.md — replace the sync.sh command + rule with `npx skills add . -g -y`
- HERMES_SETUP.md — Installation now uses `hermes skills install --force`;
developer-alternative section shows the symlink pattern for live editing
- render.py — _skill_version docstring no longer attributes the
".claude-plugin absent" case to sync.sh; explains it via per-harness
install paths in general
- .github/PULL_REQUEST_TEMPLATE.md — drop the "Ran bash scripts/sync.sh"
checklist item
CHANGELOG and historical docs (release notes, plan files) keep their
existing sync.sh mentions as accurate history.
Two real bugs flagged in the automated review of PR #400; both small.
1. render.py::_skill_version manifest with no "version" key
`json.loads(manifest.read_text()).get("version", "?")` returned "?"
immediately on a valid JSON manifest that lacked the "version" key,
never falling through to the SKILL.md frontmatter fallback. Contradicted
the docstring's "Returns '?' only if both sources are missing" contract.
Same shape if version is present but empty string ("" produces the
broken badge `🌐 last30days v · synced ...`).
Fix: pull the version out of the parsed dict, then `continue` to the
next ancestor if it's None or empty. Falls through to the SKILL.md
walk only after exhausting every ancestor.
2. SKILL.md STEP 0 re-read target hardcoded to nested cache layout
STEP 0 told the model to re-read from
`$CLAUDE_CACHE_LATEST/skills/last30days/SKILL.md` — the new nested
layout. But Step 1's resolver explicitly handles both shapes
(nested `{cache}/{version}/skills/last30days/` and flat
`{cache}/{version}/`), noting "Both shapes ship in the wild." On an
install where the highest-versioned cache happens to be the older flat
shape, STEP 0's re-read target wouldn't exist; the model would silently
stay on the stale marketplaces/ copy STEP 0 was supposed to move it
away from — the exact failure mode this guard was added to prevent.
Fix: extend the STEP 0 bash to resolve $CLAUDE_CACHE_SKILL_MD by
probing both layouts, then have the model hop to that resolved path
instead of constructing the path from a hardcoded suffix.
Two new tests in tests/test_skill_version.py cover the missing-key and
empty-string cases for fix 1. Fix 2 is exercised via the bash probe at
verify time (the STEP 0 prose-contract test isn't unit-testable from
Python, but the dual-layout bash is verified to resolve to the correct
SKILL.md on both shapes).
Stale finding skipped: greptile also flagged a missing try/except on the
SKILL.md read_text() call, but that was already addressed during the
ce-code-review safe_auto pass earlier in this PR — current code wraps it
in `try/except (OSError, UnicodeDecodeError)`, strictly more defensive
than the suggested fix.
12 fixes from the multi-agent code review on PR #400:
Version 3.2.1 -> 3.2.2 across all manifests (SKILL.md frontmatter + body
header, pyproject.toml, .claude-plugin/{plugin,marketplace}.json, sync.sh
cache path). The PR ships observable behavior changes (STEP 0 logic flip,
resolver order change, badge fallback) that should not silently appear
under the same version number — the new fallback reads SKILL.md version
directly so the badge would otherwise be misleading.
render.py::_skill_version:
- `import re` moved to module top
- _VERSION_RE extracted as a module-level compiled pattern that accepts
double-quoted, single-quoted, and unquoted YAML version scalars
- `break` -> `continue` on corrupt manifest, so a corrupt inner manifest
no longer shadows a valid outer one
- Wrap SKILL.md read_text() in try/except for UnicodeDecodeError to keep
badge emission from crashing on mis-encoded SKILL.md
- Docstring clarifies precedence; inline comment marks the fallback boundary
between the manifest walk and the SKILL.md walk
tests/test_skill_version.py (new): 7 unit tests for the fallback paths
(manifest absent, manifest corrupt, corrupt-inner + valid-outer, both
absent, SKILL.md without version, single-quoted, unquoted).
tests/test_plugin_contract.py: tombstone test asserting .codex-plugin/
stays removed (was the only CI guard against accidental reintroduction).
SKILL.md:
- STEP 0 bash echoes CLAUDE_CACHE_LATEST so the model can see the
resolved value when deciding whether to hop
- "Both shapes ship in the wild" comment now names the two cache layouts
(nested {cache}/{version}/skills/last30days/ vs flat {cache}/{version}/)
- Comparison-mode bash invocation gets its own inline SKILL_ROOT resolver
(latent gap: the contract tells the model to skip Step 1 on comparison
queries, so SKILL_ROOT was previously unset there)
CHANGELOG.md: [Unreleased] entries for the resolver rewrite and the
breaking removal of Codex native-plugin support.
All 9 reviewer personas surfaced findings; 3 cross-reviewer corroboration
clusters were promoted (import re, "both shapes" comment, missing fallback
tests). Maintainability follow-up flagged: regex now duplicated across
render.py and 2 test files; could consolidate via shared lib/skill_meta.py
helper in a future PR.
STEP 0 (CANONICAL PATH SELF-CHECK) used to force any SKILL.md load that wasn't
under $HOME/.claude/plugins/cache/last30days-skill/last30days/{version}/ to
re-Read from there. That guard is Claude-Code-specific (defends against the
marketplaces/ stale-clone bug) and broke under non-Claude installers like
`npx skills add`, ~/.codex/skills/, and ~/.agents/skills/.
The new STEP 0 narrows the check to its actual target: fire only when the
loaded SKILL.md path contains /.claude/plugins/marketplaces/. Every other
install path is trusted. The 2026-04-22 incident workaround is preserved
without breaking other harnesses.
Step 1 SKILL_ROOT resolver collapses the Codex-first / Claude-fallback /
CWD-fallback chain into a single precedence walk: Claude plugin cache
(versioned) first, then ~/.codex/skills, ~/.agents/skills, repo checkout,
./.skills/last30days (npx skills install dir), CWD, and GEMINI_EXTENSION_DIR.
Also drops Codex native plugin support: .codex-plugin/plugin.json is deleted,
the badge VERSION jq fallback in line 108 stops looking at it, and render.py's
_skill_version no longer scans for it. Codex users install via `npx skills add`
or the per-harness skill dir going forward.
render.py::_skill_version gains a SKILL.md frontmatter fallback so the badge
no longer emits `v?` on install dirs that sync.sh populates (which don't
include .claude-plugin/plugin.json).
sync.sh was written against the layout of mvanhorn/last30days-skill-private
(`.../cache/last30days-skill-private/last30days-3/{version}`) and that path
was never updated when this public repo got its own copy. Running sync.sh
from here populated the BETA channel's cache (`/last30days-beta`) instead
of this repo's own `/last30days` cache, so devs working in this repo could
not test their changes via the public slash command without waiting for a
marketplace release.
Path now derives from this repo's own manifests:
- marketplace name `last30days-skill` (.claude-plugin/marketplace.json)
- plugin name `last30days` (.claude-plugin/plugin.json)
Drops the `last30days-3-nogem` target along with it - that's a private-repo
variant with no public equivalent.
Updates test_sync_cache_path_uses_skill_version to assert the new path
pattern and clarifies the COMMON_TARGETS comment so the next person editing
it understands which marketplace/plugin name segments come from where.
Five provider modules (pinterest, threads, instagram, tiktok, youtube_yt)
and watchlist.py each carried a try/except `requests` import with parallel
urllib + requests branches. The urllib path already used the
stdlib-only wrapper at `lib/http.py` (retries, 429 handling, HTTPError).
This collapses every dual-branch into a single `http.get`/`http.post`
call and removes the `requests` dependency from `pyproject.toml`.
Also drops 4 transitive deps (urllib3, certifi, charset-normalizer, idna)
from the lockfile, leaving the skill stdlib-only at runtime.
Tests for tiktok comments and watchlist delivery were rewritten to mock
`lib.http` directly instead of the now-removed `requests` module.
Out of scope but flagged during review: the 13 surviving SC call sites
share a near-identical scaffold and would benefit from a
`http.scrapecreators_get(url, params, token, ...)` helper. Filed for a
follow-up PR rather than expanding scope here.
* feat(digg): add Digg AI 1000 source module with cluster search and post enrichment
- search_digg shells out to digg-pp-cli with --since 30d --agent
- parse_digg_response normalizes clusters to last30days dict shape
- enrich_with_top_posts attaches top-ranked X posts to top-K clusters
- shutil.which gate plus subproc.run_with_timeout discipline matches
bird_x.py / youtube_yt.py patterns
25 unit tests cover parse, age window, relevance, binary-missing
fallback, timeout recovery, and partial enrichment failures.
* feat(digg): wire Digg source into pipeline, normalize, signals, and render
pipeline.py:
- Import digg, add to MOCK_AVAILABLE_SOURCES, gate via shutil.which
- Dispatch case calls search_digg + parse_digg_response, runs
enrich_with_top_posts at default/deep depth
- Mock fixture includes one enriched cluster + one bare cluster
normalize.py:
- _normalize_digg maps cluster dicts to SourceItem with
container='Digg AI 1000' and metadata.posts pass-through
signals.py:
- SOURCE_QUALITY['digg'] = 0.85 (top tier alongside YouTube,
reflecting Digg's curatorial layer)
- ENGAGEMENT_WEIGHTS['digg'] balances postCount, uniqueAuthors,
and the rank_score derived from Digg's curatorial position
render.py:
- SOURCE_LABELS['digg'] = 'Digg AI 1000'
- _FOOTER_SOURCES adds '⛏️ Digg AI 1000' line after GitHub
- ENGAGEMENT_DISPLAY mirrors footer keys
- New _digg_posts_for + _format_digg_quote helpers emit inline
'@handle via Digg AI 1000' quotes for clusters with attached X
posts; both compact and full-dump renderers call them
* feat(digg): polish per-item engagement display and progress label
- ENGAGEMENT_DISPLAY for digg uses 'posts' / 'auth' to match the
codebase abbreviation convention (HN: 'pts'/'cmt', X: 'rt'/'re')
- Footer item word changes from 'story' to 'cluster' to dodge the
pre-existing naive plural in _footer_line_for_source ('storys')
and to match Digg's actual data model
- ui.py SOURCE_COMPLETION_META adds digg with correct 'cluster'/
'clusters' plural so 'Research complete' shows 'Digg: N clusters'
* feat(digg): document Digg AI 1000 source in skill, README, and changelog
- planner.py SOURCE_CAPABILITIES adds digg with discussion/social/link
capabilities so the planner offers it through the standard fanout
- SKILL.md ACTIVE_SOURCES_LIST gate includes 'which digg-pp-cli' check
and the source list / available-sources line names digg as opt-in
- README.md Sources table adds the Digg AI 1000 row with the activation
gate so first-time readers see what they get
- CHANGELOG.md Unreleased section calls out the source addition
* fix(digg): enrich post-dedupe so brief survivors carry inline quotes
Pipeline dispatch was attaching X posts to the top-3 items returned by
search, but dedupe later picked different survivors when multiple
clusters compared similar (common for trending topics). The brief
ended up showing clusters with no posts attached even though
enrichment ran successfully on positions 0-2.
Move enrichment to _finalize_items_by_source. The new
digg.enrich_source_items helper reads metadata['clusterUrlId'] and
writes metadata['posts'] in place on the SourceItems that actually
survive dedupe.
Verified live on 'openclaw': 2 surviving clusters, both now carry
real X-post quotes from @sama and @jeremyphoward attributed
'via Digg AI 1000'.
Adds 3 unit tests covering survivor enrichment, non-digg skip, and
clusterUrlId fallback to item_id.
* test(digg): relax live off-topic test to check shape, not emptiness
Digg's live search uses fuzzy/popularity fallback, so an impossible
token can still return some loosely-related clusters. The contract
the pipeline depends on is shape (results is always a list);
token-overlap relevance handles the noise downstream.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Adds a one-command shareable HTML mode to /last30days. The skill detects
HTML intent (explicit --emit=html / --emit:html / --html flag in
$ARGUMENTS, or natural-language asks like "give me a shareable brief",
"for Slack", "export as HTML"), runs the normal research + chat synthesis
flow, then saves a self-contained HTML file to
~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat
as usual; the HTML is an additional artifact for sharing.
User experience:
/last30days OpenClaw --emit=html
/last30days OpenClaw, give me an HTML brief for Slack
Synthesis prints to chat. Last line of the response: "📎 Shareable brief
saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it
into a message, browser-print to PDF, email it.
Architecture:
- SKILL.md gets a small detection block (triggers + early exit +
MUST/MUST NOT rules + rationale) that points to a reference file.
- references/save-html-brief.md owns the implementation: capture the
synthesis verbatim into a temp file via heredoc, invoke the engine
with --emit=html --synthesis-file, save to disk, append the
confirmation line to chat.
- lib/render.py exposes render_for_html(report, synthesis_md=None) and
render_for_html_comparison(...) -- clean markdown for HTML
conversion. Omits debug file header, model-facing safety note, and
data quality warnings (those stay in engine stderr; recipients can't
act on them in a shared artifact).
- lib/html_render.py is a new module: ~200-line CSS template (dark
mode default, prefers-color-scheme switch, print stylesheet, mobile
breakpoint), stdlib-regex markdown-to-HTML converter, marker-based
META + engine-footer wrapping, PROSE_LABELS registry promoting plain
-text labels to <h2>, colophon builder.
- last30days.py adds --emit=html argparse choice and --synthesis-file
PATH flag (engine still callable directly without the skill in the
loop).
Design:
- Voice-led research brief, not corporate report. Inter + JetBrains
Mono via Google Fonts with full system fallbacks (no FOIT, works
offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp:
body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label
20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px.
- 720px max-width, generous whitespace, no card layouts or shadows.
- Print stylesheet: light theme, A4 margins, [href]::after URL
footnotes, page-break-inside:avoid on the engine footer.
Templated (locked) shell:
- HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS
inline.
- .badge / .meta / .engine-footer / .colophon containers.
Flexible (role-based):
- <h2> rendering covers BOTH plain ## headers (comparison mode per
LAW 4 exception) AND promoted prose labels via PROSE_LABELS
registry. Adding a new SKILL.md prose label is a one-line tuple
addition; no CSS or template changes.
- Marker-based engine boundaries (<!-- META: ... -->,
<!-- PASS-THROUGH FOOTER -->) survive the markdown converter and
get promoted post-conversion. Robust to engine output format
changes.
- Generic markdown-to-HTML for body content; future SKILL.md additions
(new sections, tables, blockquotes) render correctly without code
changes.
Tests: 30 new tests in tests/test_html_render.py covering snapshots
(rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose
label promotion, warning exclusion from artifact, parseability via
html.parser, no-script self-containment.
No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip
dependencies, no JavaScript anywhere.
is_available() only caught FileNotFoundError and TimeoutExpired. On WSL,
a /mnt/c/.../WindowsApps entry on $PATH returns EACCES during exec, and
Python raises PermissionError. That escaped is_available() and crashed
pipeline.diagnose() before any source ran.
Catch OSError instead. It covers FileNotFoundError, PermissionError, and
any other spawn-time OS error, so a non-executable xurl on PATH falls
through to the next backend instead of aborting the run.
bird_x.py and youtube_yt.py had four near-identical copies of the same
subprocess cleanup dance (Popen + os.setsid + communicate(timeout) +
SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to
lib.subproc.run_with_timeout(), which:
- runs the child in its own process group via os.setsid where available
- raises SubprocTimeout on timeout
- on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s
- accepts an on_pid callback so bird_x can still register child PIDs
with last30days.register_child_pid for whole-process cleanup
- captures stdout/stderr as strings in a SubprocResult dataclass
Migrated call sites: _run_bird_search, search_handles inner worker,
search_youtube, fetch_transcript. With the helper in place, the signal
and subprocess imports became dead in both files (plus os in
youtube_yt) and went with them.
Tests: 9 new subproc tests cover success, non-zero exit, stderr capture,
timeout-raises, timeout-kills-group, missing-command, env passthrough,
PID callback, and callback-exception suppression. test_env_v3 and
test_youtube_yt patch subproc.run_with_timeout instead of the removed
bird_x.subprocess and yt-dlp subprocess.
* feat: vs mode runs N full passes; --competitors wraps vs with auto-discovery
Unifies vs-mode and --competitors onto one fanout architecture. A topic
containing "vs" / "versus" now runs N full pipeline.run() calls in parallel
(reverting the one-pass latency optimization that removed per-entity
depth); --competitors becomes a SKILL.md-level shortcut where the hosting
reasoning model (Claude Code, Codex, Hermes, Gemini) discovers N peers via
its own WebSearch, runs Step 0.55 per entity, and invokes the engine with
a vs-topic + --competitors-plan JSON.
Changed:
- vs-mode: N full passes in parallel via fanout (was 1 merged pass).
- --competitors: SKILL.md shortcut for vs-mode-with-discovery. Engine flag
kept for headless/cron use. LAW 7-style stderr reframed to lead with the
hosting-model path (use WebSearch + --competitors-plan) instead of
BRAVE_API_KEY. Footer BRAVE/SERPER nudge suppressed when --plan or
--competitors-plan present (hosting model already has WebSearch).
Added:
- --competitors-plan JSON flag: per-entity {x_handle, x_related, subreddits,
github_user, github_repos, context}. Accepts inline JSON or file path.
subrun_kwargs_for helper is the single source of truth for per-entity
kwargs — no closure-default fallthrough from main scope.
- Per-entity save files: each entity's sub-run produces its own
{slug}-raw.md with a single-row Resolved Entities block.
- --polymarket-keywords filter for ambiguous single-token topics.
Fixed:
- test_competitor_subrun_isolation regression suite locks in 3.0.12's
no-leak invariant (main flags do not inherit into peer sub-runs).
- Updates test_regression.py for the new comparison-mode payload shape.
Bumps plugin.json to 3.0.13. 1,219 tests passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: comparison title attribution — (Last 30 Days) → (/Last30Days)
User feedback on 3.0.13 dogfood runs (Kanye vs Drake, Mercer Island,
Figma): the comparison-mode synthesis title should attribute to the
slash command rather than restate the date range.
Three SKILL.md occurrences updated. Pure documentation change. Bumps to
3.0.14.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four fixes based on 2026-04-22 test-window feedback on v3.0.11 --competitors:
- Each competitor sub-run now runs Step 0.55 (X handle / subreddits /
GitHub) via resolve.auto_resolve inside the fanout closure. Deep-copied
config per entity prevents _auto_resolve_context leak across sub-runs.
Resolved data stored on report.artifacts["resolved"] for the renderer.
- New internal_subrun keyword on planner.plan_query and pipeline.run
suppresses the LAW 7 "No --plan passed" stderr for engine-internal
fan-out only. Default path unchanged.
- Default --competitors count is now 2 (3-way total). --competitors=N
still customizes; range 1..6.
- SKILL.md STEP 0 canonical-path self-check forces readers who loaded
from marketplaces/ (auto-restored to origin/main, stale) to re-read
from plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md.
Two of three 2026-04-22 test windows hit this stale-path trap.
- New ## Resolved Entities block in render_comparison_multi shows
per-entity handles/subs/github for debug visibility.
Bumps plugin.json to 3.0.12. 12 new tests; 1,175 total passing.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pass `--competitors` on a single-entity topic and the engine auto-discovers
2-6 peer entities via web search, runs the full pipeline on each in
parallel, and returns one N-way comparison reusing the existing 9-axis
Head-to-Head scaffold. `last30days OpenAI --competitors` resolves to
Anthropic + xAI + Google Gemini; `last30days Kanye West --competitors`
resolves to Drake + Kendrick Lamar + one more peer.
- New CLI flags: --competitors, --competitors=N, --competitors-list
- New scripts/lib/competitors.py — mirrors resolve.auto_resolve pattern
(web search + deterministic text extraction, no internal LLM)
- New scripts/lib/fanout.py — ThreadPoolExecutor orchestrator; per-entity
failures degrade gracefully as long as >=2 entities survive
- Multi-report render in scripts/lib/render.py reuses the comparison
scaffold for the synthesis table
- LAW 7-style stderr when no backend and no list, pointing the hosting
reasoning model at --competitors-list
- 38 new tests across CLI parsing, discovery, fanout, and rendering
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolve): category-peer subreddit map for Step 0.55
Introduces scripts/lib/categories.py with a curated category->peer-subs
map and wires scripts/lib/resolve.py auto_resolve() to merge peers into
the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode:
a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and
missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where
prompting techniques actually live.
Map is static, curated, ~11 categories (ai_image_generation,
ai_video_generation, ai_music_generation, ai_coding_agent,
ai_agent_framework, ai_chat_model, saas_screen_recording,
saas_productivity, prediction_markets, crypto_defi, dev_tool_cli).
First-match-wins ordering from most-specific to least-specific.
Compound-term patterns only (no bare common nouns like "image", "ai").
auto_resolve now:
- calls detect_category(topic) after _extract_subreddits
- merges peer_subs case-insensitively, caps at MAX_SUBS (10)
- preserves every WebSearch-returned sub (freshest signal)
- emits [Resolve] Matched category=<id>, adding peers: <list> on stderr
only when peers were actually added
- returns new "category" key in the result dict for observability
- wraps classifier in try/except so failures degrade to unwidened list
Includes drive-by: test_full_resolve / test_partial_failure
searches_run expectations bumped from 3->4 / 2->3 to match the current
queries dict (subreddit + news + x_handle + github).
* feat(skill): Step 0.55 category-peer expansion and self-check
Adds Section 2a (category-peer expansion, MANDATORY for product topics)
and the Step 0.55 self-check checkpoint that fires immediately before
the Resolved block displays. Structural mirror of the engine-side
categories.py map: same categories, same peer subs, same priority
order.
The model-side path now:
- Applies category-peer expansion to the WebSearch-resolved subs on
every product-in-a-known-category run.
- Emits the (+ <category_id> peers) annotation on the Reddit line of
the Resolved block as the observable contract. Absence on a
product-in-a-known-category topic is a Step 0.55 regression.
- Runs a self-check before emitting Resolved: "does the resolved list
include at least 2 peer subs for the matched category? if not,
widen NOW and do not run the engine yet."
Mirror of the Python map lives inside Step 0.55 as a table for the
model to pattern-match against; extrapolation to unlisted categories
is explicitly allowed. Worked example (the exact failing query)
appears below the table so reviewers can see before/after at a glance.
Both changes land inside the existing Step 0.55 block. No new
top-level section, no new LAW. LAWs 1-6 wording unchanged.
* test: end-to-end regression for GPT Image 2 failure mode
Stubs grounding.web_search to return the OpenAI-only subs that caused
the 2026-04-22 failure, then asserts that auto_resolve widens to
include the image-gen peers and emits the [Resolve] Matched
category=ai_image_generation stderr line. Covers the cap boundary
and the uncategorized-topic no-op path.
Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is
documentation-grade (not parsed by tests) and shows the pre-fix vs
post-fix Resolved block shape so reviewers can evaluate future
categories.py edits against the original bug.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Reddit, TikTok, YouTube, Instagram, Bluesky, X and Threads top comments
now render as u/author or @handle in the evidence block, instead of the
generic "Comment (...)" label. The enrichment adapters already captured
author; only the render layer was dropping it.
Also fixes the TikTok adapter to prefer user.unique_id (the @handle) over
user.nickname (display name) so attribution round-trips to a profile URL.
Legacy "Comment (...)" shape is preserved when author is empty, [deleted],
or [removed].
Bumps to 3.0.10.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
The regression test from #290 walks the filesystem via Path.rglob, so
docs/plans/*.md files (gitignored, created by internal planning) trip
the assertion on any dev machine that has run ce:plan in this repo.
Fresh clones and CI never see them, but local runs fail.
Adding docs to skip_dirs keeps the guard narrow to first-class source
files while letting internal planning docs reference old paths
verbatim.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.
xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.
Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window
New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)
Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
_search_x() priority chain
- SKILL.md: document xurl setup option
PR #285 introduced the stderr warning "No --plan and no LLM provider
configured. Using deterministic fallback..." The 2026-04-19 Run 1
agent self-debug said it read that as "I don't have a key, I can't do
LLM stuff, I have to accept fallback" - which is the exact wrong
mental model. The word "provider" referred to the engine's INTERNAL
planner credentials, but the agent parsed it as "I need credentials
to plan at all."
Rewritten to say plainly: YOU are the reasoning model hosting this
skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime);
YOU ARE the planner; you do not need an API key or credentials - you
ARE the LLM. The --plan flag exists precisely so a reasoning model
generates its own plan upstream and passes it to the engine. The
deterministic fallback is the headless/cron path only.
Runtime enumeration is explicit so agents on every supported runtime
recognize themselves - this skill ships to Claude Code, Codex, Hermes,
and ~/.agents via sync.sh.
Tests: updated test_fallback_logs_warning_when_no_provider to assert
the new language (YOU ARE the planner, runtime names present) and
assert the old misleading phrasing is absent. Renamed the companion
test for clarity.
PR #285's entity grounding checked only title + snippet. That missed:
- YouTube videos where the entity is mentioned in transcript but not
in title (false demotion of on-topic content)
- Reddit posts where the entity is in top comments but not in title
(false demotion of on-topic discussion)
And it also wasn't strong enough to reliably demote items like the
2026-04-19 Nate Herk "Managed Agents" video - which had no Hermes
anywhere - because the -25 penalty on rerank_score composed to only
-15 on final_score via the 0.60 weight, and engagement bonus partially
offset that.
Two fixes:
1. _candidate_haystack() now joins title + snippet +
metadata[transcript_snippet] + metadata[transcript_highlights] +
metadata[top_comments][*].excerpt/text + metadata[comment_insights].
Catches entity mentions wherever they actually live. Guarded with
isinstance checks so malformed metadata doesn't raise.
2. ENTITY_MISS_FINAL_PENALTY (20.0) applied directly in _final_score
when candidate.explanation contains "entity-miss". This lands the
full penalty weight on the composite signal that cluster-scoring
consumes, instead of being diluted by the rerank_score weight.
Combined effect: entity-miss gap grows from ~15 to ~35 points.
Tests: 8 new scenarios covering transcript match, transcript highlight
match, top-comment match, comment-insight match, empty-text skip,
no-primary-entity no-op, and the dual-penalty composition check.
The stderr [Planner] warning from PR #285 doesn't reach the user because
Claude and other reasoning agents hide stderr from their synthesis. The
2026-04-19 Hermes Agent Use Cases Run 1 produced source=deterministic
and the user never saw it.
Adds a user-visible stdout block that the model's LAW 5 pass-through
contract forces into the response. Fires only when plan_source is
deterministic AND no pre-research flags were passed AND the topic is
pre-research-eligible (named entity). Cron jobs on abstract topics
don't trigger it.
Position: BEFORE the EVIDENCE FOR SYNTHESIS envelope so the model sees
it as the first non-badge content. Wrapped in a new USER-VISIBLE BANNER
envelope matching the EVIDENCE/PASS-THROUGH envelope pattern from Unit 1
of PR #285.
Runtime-agnostic language: explicitly enumerates Claude Code, Codex,
Hermes, Gemini so the hosting reasoning model recognizes itself
regardless of runtime.
pipeline.py now persists plan_source to report.artifacts so the
renderer can consume it. Adds 7 tests covering fire conditions,
suppression conditions (external/llm plan source, flags present,
abstract topic), and correct position relative to the evidence envelope.
The prior pipeline.py only logged the planner outcome when an external
--plan was passed ("[Planner] Using external plan (N subqueries)").
The internal LLM planner and the deterministic fallback ran silently,
so retrieval-breadth failures were invisible without --debug.
After plan finalization, emit a unified trace:
[Planner] Plan: intent=X, freshness=Y, cluster_mode=Z, subqueries=N, source=external|llm|deterministic
[Planner] sq1 label=... search="..." sources=[...]
[Planner] sq2 ...
Stderr only; does not touch the user-facing stdout synthesis. The
source= annotation distinguishes --plan (external), provider-backed
(llm), and deterministic paths — so when the 2026-04-19 Hermes Agent
Use Cases failure mode recurs, the trace tells the user which path ran
and what subqueries it produced.
Tests: added test_planner_trace_always_fires_on_mock_run which captures
stderr on a mock pipeline run and asserts the summary + per-subquery
lines appear.
The 2026-04-19 Hermes Agent Use Cases run had a Nate Herk YouTube video
titled "I Tested Claude's New Managed Agents" score 51 and rank #2
with zero Hermes content. The reranker had intent-specific scoring hints
but no entity-grounding check, so topic-vicinity matches (one offhand
OpenClaw mention) drifted to the top.
Add _primary_entity(topic) that strips intent-modifier suffixes ("use
cases", "workflows", etc.) so "Hermes Agent use cases" yields
primary_entity="Hermes Agent". Pass the entity through to both the LLM
and fallback scoring paths.
Fallback path: if primary_entity is not found (case-insensitive) in
title + snippet, subtract ENTITY_MISS_PENALTY (25 pts). Skip the
demotion for candidates with no text at all (image-only TikToks etc.)
to avoid false negatives on thin-text sources.
LLM path: add a "Primary entity grounding" hint to _build_prompt when
primary_entity is non-empty. Instructs the LLM to score candidates
without the entity at <=30.
Tests: 24 rerank tests pass, including 8 new entity-grounding tests.
Topics with suffixes like "use cases", "workflows", "review",
"examples" were previously echoed near-verbatim into search_query,
returning near-zero matches because nobody posts the literal phrase
(2026-04-19 Hermes Agent Use Cases failure).
Unit 2 — planner breadth:
1. Planner prompt rule: STRIP intent-modifier phrases from search_query
(keep them in ranking_query). Paraphrase across 4-5 subqueries that
each express the intent differently.
2. Planner prompt rule: quote only multi-word proper nouns like
"Hermes Agent", not the user's full topic.
3. Raise _max_subqueries cap from 3 to 5 for how_to / opinion / product /
breaking_news / prediction. Comparison stays at 4; factual / concept
stay at 2 unless the topic carries an intent modifier.
4. Deterministic fallback: when intent is non-{comparison,prediction}
and topic contains an intent modifier, append 3 paraphrased
subqueries (workflows, production, experience).
Unit 3 — deterministic fallback defaults:
5. _infer_intent default changed from "breaking_news" to "concept".
Prior default forced strict_recent freshness on unclassified topics,
biasing against older relevant material. Recency-signal regexes
("trending", "this week", etc.) added above the default so genuinely
time-sensitive topics still classify correctly.
6. _keyword_query now quotes only title-cased multi-word proper nouns
("Hermes Agent", "Claude Code"), not the user's full typed topic.
Hyphenated compounds and lowercase terms are left as bare keywords
so platform tokenizers broaden rather than narrow retrieval.
7. New stderr warning when plan_query runs with no --plan and no LLM
provider: surfaces that the deterministic fallback path is weaker
than the --plan-from-Claude-Code path, so callers know to generate
and pass a plan.
Tests: 37 planner tests pass, including 11 intent-modifier and 7
fallback-defaults tests.
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.
Split render_compact stdout into two bounded blocks:
- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
Emit verbatim per LAW 5.
Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.
Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:
1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
Birthday-gift failure mode becomes structurally impossible - the pipeline
never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
pointing the model to ask for hobbies/relationship/budget. Escape hatch:
LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.
2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
Peter Steinberger's self-debug named the first file as the one it read
instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
Sync script simplified: Hermes now always uses main SKILL.md.
3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
with pass-through instruction. The model had the canonical body in its
buffer on the Peter run and discarded it; the boundary makes pass-through
the path of least resistance.
4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
section") that caused Peter's trailing Sources leak. No more ambiguity
at synthesis time.
Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.
Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.
Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Most users never touch FUN_LEVEL. Default medium was shipping a stats
block but rarely a Best Takes block, and when it did it was below the
cluster fold where a synthesizing model had already stopped reading.
A 2,304-upvote Reddit comment ("WHAT?! I reached my monthly limit
just reading this post") on the 2026-04-17 Opus 4.7 run sat inside
cluster 11 and never made it into synthesis. Four coordinated changes:
1. render: promote Best Takes above the cluster list so the synthesizer
sees comedy before it anchors on cluster 1.
2. render: lower medium threshold from 70 to 55 (heuristic maxes at 80),
drop the two-gem floor to one-gem. Default now reliably emits the
block on typical runs.
3. rerank: score individual top_comments by upvote ratio to their parent
thread. A 2,304-upvote comment on a 300-upvote thread now outranks a
400-upvote comment on a 3,400-upvote thread, which is the viral-wit
signal. Handles both the LLM scoring path and the heuristic fallback.
4. render: merge scored comment gems into Best Takes alongside candidate
gems, sorted together. Comment lines show body + parent title +
r/subreddit or @handle + absolute upvotes.
5. SKILL: tell the synthesizer to quote at least two Best Takes entries
verbatim, with an example of the new comment format.
Plan: docs/plans/2026-04-17-001-feat-default-fun-surfacing-plan.md
🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}. This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available." Return None
instead when every engagement field is missing.
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.
Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.
Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).
Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
Added params kwarg to http.request()/http.get() that urlencodes a dict
into the query string. None values are dropped, ints and bools are
stringified, and params append correctly if the URL already has a
query string.
Migrated reddit.py to use this helper for all three ScrapeCreators
call sites (global search, subreddit search, post comments). Deleted
the try/import requests/except ImportError fallback and the paired
if not _requests: / else: branches. Six new http tests cover the
params-encoding behavior.
Net: reddit.py -70 lines. Behavior is identical - the existing http.py
urllib implementation already had retry logic, 429 handling, and
HTTPError types that are strictly better than the ad-hoc requests
branches we deleted.
99 reddit tests pass. Live smoke test on a real ScrapeCreators run
returned 12 threads with the same engagement data as before.
Add column whitelists to prevent SQL injection via kwargs keys in
dynamic UPDATE queries. Values were already parameterized but column
names were string-interpolated directly from kwargs.
Fixes#90
Trevin Chow