The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().
Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
invoking the skill knows to subtract excluded sources before displaying
the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
prose are incomplete — the agent invoking the skill won't know the flag
exists.)
The bash banner accounting used raw substring matching while
pipeline.py normalises EXCLUDE_SOURCES via .strip().lower(). With
EXCLUDE_SOURCES=TikTok,Instagram (or with surrounding spaces),
pipeline correctly excludes the sources but the banner did not
deduct them — count showed 1-2 higher than what the pipeline
actually runs. Normalisation now mirrors the Python side
(lowercase, collapse whitespace around commas, strip outer whitespace).
Reproducer (clean HOME with config EXCLUDE_SOURCES=TikTok,Instagram):
before: /last30days: Ready — 7 sources active.
after: /last30days: Ready — 5 sources active.
Addresses Greptile review comment P1 on #399.
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES
config key. Two coupled changes:
1. pipeline.available_sources() filters out any source listed in
config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive,
whitespace-tolerant) before returning.
2. hooks/scripts/check-config.sh "Ready — N sources active" banner
subtracts excluded sources from the ScrapeCreators +3 (Reddit
comments + TikTok + Instagram) so the count matches what the
pipeline actually runs.
Use case: skip TikTok/Instagram on runs where you only want
text-substantive sources, without unsetting SCRAPECREATORS_API_KEY
(which would also kill Reddit comments). The existing INCLUDE_SOURCES
allowlist covers Perplexity opt-in but doesn't cover this denylist case
— tiktok and instagram are added unconditionally when
SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key.
Tests (tests/test_pipeline_v3.py::TestExcludeSources):
- excludes tiktok+instagram when listed
- no exclusion when env unset or empty string
- case-insensitive + whitespace-tolerant parsing
- works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
Five provider modules (pinterest, threads, instagram, tiktok, youtube_yt)
and watchlist.py each carried a try/except `requests` import with parallel
urllib + requests branches. The urllib path already used the
stdlib-only wrapper at `lib/http.py` (retries, 429 handling, HTTPError).
This collapses every dual-branch into a single `http.get`/`http.post`
call and removes the `requests` dependency from `pyproject.toml`.
Also drops 4 transitive deps (urllib3, certifi, charset-normalizer, idna)
from the lockfile, leaving the skill stdlib-only at runtime.
Tests for tiktok comments and watchlist delivery were rewritten to mock
`lib.http` directly instead of the now-removed `requests` module.
Out of scope but flagged during review: the 13 surviving SC call sites
share a near-identical scaffold and would benefit from a
`http.scrapecreators_get(url, params, token, ...)` helper. Filed for a
follow-up PR rather than expanding scope here.
test_sync_cache_path_uses_skill_version asserts that sync.sh's plugin
cache path includes the version from SKILL.md frontmatter. The frontmatter
moved to 3.2.1 in #371 but sync.sh still pointed at 3.2.0, leaving CI red
on every PR.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(digg): bump POSTS_PER_CLUSTER to 5 and render limit to 3
Match the per-item enrichment cap and inline-display cap used by the
other sources (Reddit, HN, YouTube, TikTok, GitHub all use 5 fetched /
3 displayed). At the previous 3/2 caps the engine routinely truncated
cluster context — a recent run on cli-printing-press lost the Jason
Calacanis quote tweet entirely because the display cut off after Garry
Tan's first two posts.
* feat(digg): rename 'Digg AI 1000' to 'Digg' in user-facing strings
Drop the 'AI 1000' suffix from the footer line, source label, inline
quote attribution ('via Digg'), why_relevant, container, mock title,
SKILL.md source list, and README sources table. Internal code comments
and docstrings still reference the upstream Digg AI 1000 product.
Bumps version to 3.2.1 and adds a CHANGELOG entry covering this rename
and the POSTS_PER_CLUSTER / render-limit bumps from the prior commit.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* chore(release): v3.2.0
Bumps plugin/marketplace/codex/pyproject versions from 3.1.1 to 3.2.0.
Promotes the Unreleased CHANGELOG entries (--emit=html, Digg AI 1000
source) to the 3.2.0 release section.
* chore(release): bump SKILL.md header and sync.sh path to 3.2.0
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(digg): add Digg AI 1000 source module with cluster search and post enrichment
- search_digg shells out to digg-pp-cli with --since 30d --agent
- parse_digg_response normalizes clusters to last30days dict shape
- enrich_with_top_posts attaches top-ranked X posts to top-K clusters
- shutil.which gate plus subproc.run_with_timeout discipline matches
bird_x.py / youtube_yt.py patterns
25 unit tests cover parse, age window, relevance, binary-missing
fallback, timeout recovery, and partial enrichment failures.
* feat(digg): wire Digg source into pipeline, normalize, signals, and render
pipeline.py:
- Import digg, add to MOCK_AVAILABLE_SOURCES, gate via shutil.which
- Dispatch case calls search_digg + parse_digg_response, runs
enrich_with_top_posts at default/deep depth
- Mock fixture includes one enriched cluster + one bare cluster
normalize.py:
- _normalize_digg maps cluster dicts to SourceItem with
container='Digg AI 1000' and metadata.posts pass-through
signals.py:
- SOURCE_QUALITY['digg'] = 0.85 (top tier alongside YouTube,
reflecting Digg's curatorial layer)
- ENGAGEMENT_WEIGHTS['digg'] balances postCount, uniqueAuthors,
and the rank_score derived from Digg's curatorial position
render.py:
- SOURCE_LABELS['digg'] = 'Digg AI 1000'
- _FOOTER_SOURCES adds '⛏️ Digg AI 1000' line after GitHub
- ENGAGEMENT_DISPLAY mirrors footer keys
- New _digg_posts_for + _format_digg_quote helpers emit inline
'@handle via Digg AI 1000' quotes for clusters with attached X
posts; both compact and full-dump renderers call them
* feat(digg): polish per-item engagement display and progress label
- ENGAGEMENT_DISPLAY for digg uses 'posts' / 'auth' to match the
codebase abbreviation convention (HN: 'pts'/'cmt', X: 'rt'/'re')
- Footer item word changes from 'story' to 'cluster' to dodge the
pre-existing naive plural in _footer_line_for_source ('storys')
and to match Digg's actual data model
- ui.py SOURCE_COMPLETION_META adds digg with correct 'cluster'/
'clusters' plural so 'Research complete' shows 'Digg: N clusters'
* feat(digg): document Digg AI 1000 source in skill, README, and changelog
- planner.py SOURCE_CAPABILITIES adds digg with discussion/social/link
capabilities so the planner offers it through the standard fanout
- SKILL.md ACTIVE_SOURCES_LIST gate includes 'which digg-pp-cli' check
and the source list / available-sources line names digg as opt-in
- README.md Sources table adds the Digg AI 1000 row with the activation
gate so first-time readers see what they get
- CHANGELOG.md Unreleased section calls out the source addition
* fix(digg): enrich post-dedupe so brief survivors carry inline quotes
Pipeline dispatch was attaching X posts to the top-3 items returned by
search, but dedupe later picked different survivors when multiple
clusters compared similar (common for trending topics). The brief
ended up showing clusters with no posts attached even though
enrichment ran successfully on positions 0-2.
Move enrichment to _finalize_items_by_source. The new
digg.enrich_source_items helper reads metadata['clusterUrlId'] and
writes metadata['posts'] in place on the SourceItems that actually
survive dedupe.
Verified live on 'openclaw': 2 surviving clusters, both now carry
real X-post quotes from @sama and @jeremyphoward attributed
'via Digg AI 1000'.
Adds 3 unit tests covering survivor enrichment, non-digg skip, and
clusterUrlId fallback to item_id.
* test(digg): relax live off-topic test to check shape, not emptiness
Digg's live search uses fuzzy/popularity fallback, so an impossible
token can still return some loosely-related clusters. The contract
the pipeline depends on is shape (results is always a list);
token-overlap relevance handles the noise downstream.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Adds a one-command shareable HTML mode to /last30days. The skill detects
HTML intent (explicit --emit=html / --emit:html / --html flag in
$ARGUMENTS, or natural-language asks like "give me a shareable brief",
"for Slack", "export as HTML"), runs the normal research + chat synthesis
flow, then saves a self-contained HTML file to
~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat
as usual; the HTML is an additional artifact for sharing.
User experience:
/last30days OpenClaw --emit=html
/last30days OpenClaw, give me an HTML brief for Slack
Synthesis prints to chat. Last line of the response: "📎 Shareable brief
saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it
into a message, browser-print to PDF, email it.
Architecture:
- SKILL.md gets a small detection block (triggers + early exit +
MUST/MUST NOT rules + rationale) that points to a reference file.
- references/save-html-brief.md owns the implementation: capture the
synthesis verbatim into a temp file via heredoc, invoke the engine
with --emit=html --synthesis-file, save to disk, append the
confirmation line to chat.
- lib/render.py exposes render_for_html(report, synthesis_md=None) and
render_for_html_comparison(...) -- clean markdown for HTML
conversion. Omits debug file header, model-facing safety note, and
data quality warnings (those stay in engine stderr; recipients can't
act on them in a shared artifact).
- lib/html_render.py is a new module: ~200-line CSS template (dark
mode default, prefers-color-scheme switch, print stylesheet, mobile
breakpoint), stdlib-regex markdown-to-HTML converter, marker-based
META + engine-footer wrapping, PROSE_LABELS registry promoting plain
-text labels to <h2>, colophon builder.
- last30days.py adds --emit=html argparse choice and --synthesis-file
PATH flag (engine still callable directly without the skill in the
loop).
Design:
- Voice-led research brief, not corporate report. Inter + JetBrains
Mono via Google Fonts with full system fallbacks (no FOIT, works
offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp:
body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label
20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px.
- 720px max-width, generous whitespace, no card layouts or shadows.
- Print stylesheet: light theme, A4 margins, [href]::after URL
footnotes, page-break-inside:avoid on the engine footer.
Templated (locked) shell:
- HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS
inline.
- .badge / .meta / .engine-footer / .colophon containers.
Flexible (role-based):
- <h2> rendering covers BOTH plain ## headers (comparison mode per
LAW 4 exception) AND promoted prose labels via PROSE_LABELS
registry. Adding a new SKILL.md prose label is a one-line tuple
addition; no CSS or template changes.
- Marker-based engine boundaries (<!-- META: ... -->,
<!-- PASS-THROUGH FOOTER -->) survive the markdown converter and
get promoted post-conversion. Robust to engine output format
changes.
- Generic markdown-to-HTML for body content; future SKILL.md additions
(new sections, tables, blockquotes) render correctly without code
changes.
Tests: 30 new tests in tests/test_html_render.py covering snapshots
(rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose
label promotion, warning exclusion from artifact, parseability via
html.parser, no-script self-containment.
No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip
dependencies, no JavaScript anywhere.
is_available() only caught FileNotFoundError and TimeoutExpired. On WSL,
a /mnt/c/.../WindowsApps entry on $PATH returns EACCES during exec, and
Python raises PermissionError. That escaped is_available() and crashed
pipeline.diagnose() before any source ran.
Catch OSError instead. It covers FileNotFoundError, PermissionError, and
any other spawn-time OS error, so a non-executable xurl on PATH falls
through to the next backend instead of aborting the run.
bird_x.py and youtube_yt.py had four near-identical copies of the same
subprocess cleanup dance (Popen + os.setsid + communicate(timeout) +
SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to
lib.subproc.run_with_timeout(), which:
- runs the child in its own process group via os.setsid where available
- raises SubprocTimeout on timeout
- on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s
- accepts an on_pid callback so bird_x can still register child PIDs
with last30days.register_child_pid for whole-process cleanup
- captures stdout/stderr as strings in a SubprocResult dataclass
Migrated call sites: _run_bird_search, search_handles inner worker,
search_youtube, fetch_transcript. With the helper in place, the signal
and subprocess imports became dead in both files (plus os in
youtube_yt) and went with them.
Tests: 9 new subproc tests cover success, non-zero exit, stderr capture,
timeout-raises, timeout-kills-group, missing-command, env passthrough,
PID callback, and callback-exception suppression. test_env_v3 and
test_youtube_yt patch subproc.run_with_timeout instead of the removed
bird_x.subprocess and yt-dlp subprocess.
1. N+1 queries in store.store_findings()
The old loop ran one SELECT per finding to check existence, then one
INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
dict, then executemany() handles all inserts and updates. Query count
stays constant regardless of batch size. Benchmark on 500 findings:
~30ms to ~20ms; gap widens on slower storage.
2. O(n^2) source_items dedup in fusion.weighted_rrf()
Merging an item into an existing candidate ran any(existing.source ==
... for existing in candidate.source_items), linearly scanning a list
that grew with each merge. At 40 candidates with 20 source_items each,
fusion went quadratic. Now tracks (source, item_id) tuples in a
per-candidate set for O(1) lookup. The source_items list itself is
unchanged since other code iterates it.
3. Dead code removal
- providers.GeminiClient.ground_search() and .url_context_json(): zero
callers. Deleted.
- render._top_comment_excerpt(): zero callers. Deleted.
- env.is_reddit_available(): one-line wrapper around get_reddit_source.
Callers can check get_reddit_source(config) is not None directly.
Scoring hot path (_normalize_score_dedupe) re-tokenized the same
ranking_query ~240x per stream: once per item for local_relevance,
plus ~5x per item across snippet windows. Query tokens are immutable
within a stream, so compute them once as relevance.PreparedQuery and
thread through signals.annotate_stream and snippet.extract_best_snippet.
dedupe._PreparedText called normalize_text twice: once in __init__ and
again via get_ngrams. Factor out _ngrams_of_normalized so the prepared
path skips the redundant pass while get_ngrams keeps its public contract.
Behavior unchanged.
_normalize_reddit, _normalize_hackernews, and _normalize_github inlined
the same 5-line comprehension to stringify and space-join the first 3
top_comments' excerpt field. Extract one helper, call it from all three.
The comment field name varies per source (Reddit/GitHub use 'excerpt',
HN uses 'text'), so it's passed as a parameter. Behavior unchanged.
`tests/test_plugin_contract.py::test_versions_match_across_manifests`
enforces that every version-bearing surface agrees: pyproject.toml,
SKILL.md, both plugin.json files, AND the marketplace plugin entry.
The Claude Code spec says plugin.json wins when both are set, but this
repo deliberately mirrors the version across all surfaces and tests it.
Restore the field at 3.1.1 to satisfy the contract.
Codex CLI reads AGENTS.md for repo-level context the way Claude Code reads CLAUDE.md. Delegate to the existing CLAUDE.md so both harnesses share one source of project instructions.
- Remove no-op `"hooks": {}` from .claude-plugin/plugin.json (auto-discovery from hooks/hooks.json picks up the SessionStart hook).
- Remove redundant `version` from marketplace.json plugin entry; plugin.json is the source of truth per the spec.
- Sync description / longDescription across .claude-plugin and .codex-plugin manifests so all surfaces show the same copy.
Adds structured templates to help contributors submit higher-quality
PRs and issues. PR template includes testing checklist (pytest, sync.sh).
Issue templates use YAML forms for bug reports and feature requests.
Fixes#251
* feat: vs mode runs N full passes; --competitors wraps vs with auto-discovery
Unifies vs-mode and --competitors onto one fanout architecture. A topic
containing "vs" / "versus" now runs N full pipeline.run() calls in parallel
(reverting the one-pass latency optimization that removed per-entity
depth); --competitors becomes a SKILL.md-level shortcut where the hosting
reasoning model (Claude Code, Codex, Hermes, Gemini) discovers N peers via
its own WebSearch, runs Step 0.55 per entity, and invokes the engine with
a vs-topic + --competitors-plan JSON.
Changed:
- vs-mode: N full passes in parallel via fanout (was 1 merged pass).
- --competitors: SKILL.md shortcut for vs-mode-with-discovery. Engine flag
kept for headless/cron use. LAW 7-style stderr reframed to lead with the
hosting-model path (use WebSearch + --competitors-plan) instead of
BRAVE_API_KEY. Footer BRAVE/SERPER nudge suppressed when --plan or
--competitors-plan present (hosting model already has WebSearch).
Added:
- --competitors-plan JSON flag: per-entity {x_handle, x_related, subreddits,
github_user, github_repos, context}. Accepts inline JSON or file path.
subrun_kwargs_for helper is the single source of truth for per-entity
kwargs — no closure-default fallthrough from main scope.
- Per-entity save files: each entity's sub-run produces its own
{slug}-raw.md with a single-row Resolved Entities block.
- --polymarket-keywords filter for ambiguous single-token topics.
Fixed:
- test_competitor_subrun_isolation regression suite locks in 3.0.12's
no-leak invariant (main flags do not inherit into peer sub-runs).
- Updates test_regression.py for the new comparison-mode payload shape.
Bumps plugin.json to 3.0.13. 1,219 tests passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: comparison title attribution — (Last 30 Days) → (/Last30Days)
User feedback on 3.0.13 dogfood runs (Kanye vs Drake, Mercer Island,
Figma): the comparison-mode synthesis title should attribute to the
slash command rather than restate the date range.
Three SKILL.md occurrences updated. Pure documentation change. Bumps to
3.0.14.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four fixes based on 2026-04-22 test-window feedback on v3.0.11 --competitors:
- Each competitor sub-run now runs Step 0.55 (X handle / subreddits /
GitHub) via resolve.auto_resolve inside the fanout closure. Deep-copied
config per entity prevents _auto_resolve_context leak across sub-runs.
Resolved data stored on report.artifacts["resolved"] for the renderer.
- New internal_subrun keyword on planner.plan_query and pipeline.run
suppresses the LAW 7 "No --plan passed" stderr for engine-internal
fan-out only. Default path unchanged.
- Default --competitors count is now 2 (3-way total). --competitors=N
still customizes; range 1..6.
- SKILL.md STEP 0 canonical-path self-check forces readers who loaded
from marketplaces/ (auto-restored to origin/main, stale) to re-read
from plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md.
Two of three 2026-04-22 test windows hit this stale-path trap.
- New ## Resolved Entities block in render_comparison_multi shows
per-entity handles/subs/github for debug visibility.
Bumps plugin.json to 3.0.12. 12 new tests; 1,175 total passing.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pass `--competitors` on a single-entity topic and the engine auto-discovers
2-6 peer entities via web search, runs the full pipeline on each in
parallel, and returns one N-way comparison reusing the existing 9-axis
Head-to-Head scaffold. `last30days OpenAI --competitors` resolves to
Anthropic + xAI + Google Gemini; `last30days Kanye West --competitors`
resolves to Drake + Kendrick Lamar + one more peer.
- New CLI flags: --competitors, --competitors=N, --competitors-list
- New scripts/lib/competitors.py — mirrors resolve.auto_resolve pattern
(web search + deterministic text extraction, no internal LLM)
- New scripts/lib/fanout.py — ThreadPoolExecutor orchestrator; per-entity
failures degrade gracefully as long as >=2 entities survive
- Multi-report render in scripts/lib/render.py reuses the comparison
scaffold for the synthesis table
- LAW 7-style stderr when no backend and no list, pointing the hosting
reasoning model at --competitors-list
- 38 new tests across CLI parsing, discovery, fanout, and rendering
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolve): category-peer subreddit map for Step 0.55
Introduces scripts/lib/categories.py with a curated category->peer-subs
map and wires scripts/lib/resolve.py auto_resolve() to merge peers into
the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode:
a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and
missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where
prompting techniques actually live.
Map is static, curated, ~11 categories (ai_image_generation,
ai_video_generation, ai_music_generation, ai_coding_agent,
ai_agent_framework, ai_chat_model, saas_screen_recording,
saas_productivity, prediction_markets, crypto_defi, dev_tool_cli).
First-match-wins ordering from most-specific to least-specific.
Compound-term patterns only (no bare common nouns like "image", "ai").
auto_resolve now:
- calls detect_category(topic) after _extract_subreddits
- merges peer_subs case-insensitively, caps at MAX_SUBS (10)
- preserves every WebSearch-returned sub (freshest signal)
- emits [Resolve] Matched category=<id>, adding peers: <list> on stderr
only when peers were actually added
- returns new "category" key in the result dict for observability
- wraps classifier in try/except so failures degrade to unwidened list
Includes drive-by: test_full_resolve / test_partial_failure
searches_run expectations bumped from 3->4 / 2->3 to match the current
queries dict (subreddit + news + x_handle + github).
* feat(skill): Step 0.55 category-peer expansion and self-check
Adds Section 2a (category-peer expansion, MANDATORY for product topics)
and the Step 0.55 self-check checkpoint that fires immediately before
the Resolved block displays. Structural mirror of the engine-side
categories.py map: same categories, same peer subs, same priority
order.
The model-side path now:
- Applies category-peer expansion to the WebSearch-resolved subs on
every product-in-a-known-category run.
- Emits the (+ <category_id> peers) annotation on the Reddit line of
the Resolved block as the observable contract. Absence on a
product-in-a-known-category topic is a Step 0.55 regression.
- Runs a self-check before emitting Resolved: "does the resolved list
include at least 2 peer subs for the matched category? if not,
widen NOW and do not run the engine yet."
Mirror of the Python map lives inside Step 0.55 as a table for the
model to pattern-match against; extrapolation to unlisted categories
is explicitly allowed. Worked example (the exact failing query)
appears below the table so reviewers can see before/after at a glance.
Both changes land inside the existing Step 0.55 block. No new
top-level section, no new LAW. LAWs 1-6 wording unchanged.
* test: end-to-end regression for GPT Image 2 failure mode
Stubs grounding.web_search to return the OpenAI-only subs that caused
the 2026-04-22 failure, then asserts that auto_resolve widens to
include the image-gen peers and emits the [Resolve] Matched
category=ai_image_generation stderr line. Covers the cap boundary
and the uncategorized-topic no-op path.
Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is
documentation-grade (not parsed by tests) and shows the pre-fix vs
post-fix Resolved block shape so reviewers can evaluate future
categories.py edits against the original bug.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Reddit, TikTok, YouTube, Instagram, Bluesky, X and Threads top comments
now render as u/author or @handle in the evidence block, instead of the
generic "Comment (...)" label. The enrichment adapters already captured
author; only the render layer was dropping it.
Also fixes the TikTok adapter to prefer user.unique_id (the @handle) over
user.nickname (display name) so attribution round-trips to a profile URL.
Legacy "Comment (...)" shape is preserved when author is empty, [deleted],
or [removed].
Bumps to 3.0.10.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
The regression test from #290 walks the filesystem via Path.rglob, so
docs/plans/*.md files (gitignored, created by internal planning) trip
the assertion on any dev machine that has run ce:plan in this repo.
Fresh clones and CI never see them, but local runs fail.
Adding docs to skip_dirs keeps the guard narrow to first-class source
files while letting internal planning docs reference old paths
verbatim.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.
xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.
Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window
New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)
Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
_search_x() priority chain
- SKILL.md: document xurl setup option
Inline markdown links on every narrative citation (@handle, r/sub,
publication, YouTube channel, TikTok/Instagram creator, Polymarket
market). Raw URL strings remain forbidden. Plain-text fallback when the
raw data has no URL for a specific source.
Commit 1 (790e5bc) added the citation rule in CITATION PRIORITY / URL
FORMATTING. Live tests showed the rule was deployed but consistently
skipped because it lived at line 1224, below the agent's chunked-read
window. Commit 2 (5864c687) hoists the rule into the VOICE CONTRACT
LAW block as LAW 8, at line 167 - inside the guaranteed-loaded top
band alongside LAWs 1-7. Same pattern that fixed v3.0.6 (invented
titles), disaster #2 (stripped bold), disaster #3 (trailing Sources),
and the 2026-04-19 Hermes evidence-dump disaster.
No Python engine changes. Rule is prompt-only; the deterministic
stats footer (LAW 5) is unchanged.
Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md
PR #285 introduced the stderr warning "No --plan and no LLM provider
configured. Using deterministic fallback..." The 2026-04-19 Run 1
agent self-debug said it read that as "I don't have a key, I can't do
LLM stuff, I have to accept fallback" - which is the exact wrong
mental model. The word "provider" referred to the engine's INTERNAL
planner credentials, but the agent parsed it as "I need credentials
to plan at all."
Rewritten to say plainly: YOU are the reasoning model hosting this
skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime);
YOU ARE the planner; you do not need an API key or credentials - you
ARE the LLM. The --plan flag exists precisely so a reasoning model
generates its own plan upstream and passes it to the engine. The
deterministic fallback is the headless/cron path only.
Runtime enumeration is explicit so agents on every supported runtime
recognize themselves - this skill ships to Claude Code, Codex, Hermes,
and ~/.agents via sync.sh.
Tests: updated test_fallback_logs_warning_when_no_provider to assert
the new language (YOU ARE the planner, runtime names present) and
assert the old misleading phrasing is absent. Renamed the companion
test for clarity.
PR #285's entity grounding checked only title + snippet. That missed:
- YouTube videos where the entity is mentioned in transcript but not
in title (false demotion of on-topic content)
- Reddit posts where the entity is in top comments but not in title
(false demotion of on-topic discussion)
And it also wasn't strong enough to reliably demote items like the
2026-04-19 Nate Herk "Managed Agents" video - which had no Hermes
anywhere - because the -25 penalty on rerank_score composed to only
-15 on final_score via the 0.60 weight, and engagement bonus partially
offset that.
Two fixes:
1. _candidate_haystack() now joins title + snippet +
metadata[transcript_snippet] + metadata[transcript_highlights] +
metadata[top_comments][*].excerpt/text + metadata[comment_insights].
Catches entity mentions wherever they actually live. Guarded with
isinstance checks so malformed metadata doesn't raise.
2. ENTITY_MISS_FINAL_PENALTY (20.0) applied directly in _final_score
when candidate.explanation contains "entity-miss". This lands the
full penalty weight on the composite signal that cluster-scoring
consumes, instead of being diluted by the rerank_score weight.
Combined effect: entity-miss gap grows from ~15 to ~35 points.
Tests: 8 new scenarios covering transcript match, transcript highlight
match, top-comment match, comment-insight match, empty-text skip,
no-primary-entity no-op, and the dual-penalty composition check.
The stderr [Planner] warning from PR #285 doesn't reach the user because
Claude and other reasoning agents hide stderr from their synthesis. The
2026-04-19 Hermes Agent Use Cases Run 1 produced source=deterministic
and the user never saw it.
Adds a user-visible stdout block that the model's LAW 5 pass-through
contract forces into the response. Fires only when plan_source is
deterministic AND no pre-research flags were passed AND the topic is
pre-research-eligible (named entity). Cron jobs on abstract topics
don't trigger it.
Position: BEFORE the EVIDENCE FOR SYNTHESIS envelope so the model sees
it as the first non-badge content. Wrapped in a new USER-VISIBLE BANNER
envelope matching the EVIDENCE/PASS-THROUGH envelope pattern from Unit 1
of PR #285.
Runtime-agnostic language: explicitly enumerates Claude Code, Codex,
Hermes, Gemini so the hosting reasoning model recognizes itself
regardless of runtime.
pipeline.py now persists plan_source to report.artifacts so the
renderer can consume it. Adds 7 tests covering fire conditions,
suppression conditions (external/llm plan source, flags present,
abstract topic), and correct position relative to the evidence envelope.
Run 1 of /last30days Hermes Agent use cases on 2026-04-19 called the engine
bare despite SKILL.md already having a detailed Step 0.75 (YOU are the
planner) and a PRECONDITION GATE requiring --plan. Those lived at lines
647 and 729 - the model didn't reach them before invoking Bash.
LAW 7 hoists the rule into the OUTPUT CONTRACT block at the top (same
placement pattern as LAW 6), so it is the first thing the model reads.
Runtime-agnostic language: Claude Code, Codex, Hermes, Gemini, or any
agent runtime. Named failure mode with the misread diagnosis: "provider"
in engine messages refers to the engine's INTERNAL planner credentials,
NOT a prerequisite the caller needs - if you are the hosting reasoning
model, YOU are the provider.
Concrete self-check: re-read pending Bash command; if no --plan and topic
is a named entity, STOP and generate a plan.
The prior pipeline.py only logged the planner outcome when an external
--plan was passed ("[Planner] Using external plan (N subqueries)").
The internal LLM planner and the deterministic fallback ran silently,
so retrieval-breadth failures were invisible without --debug.
After plan finalization, emit a unified trace:
[Planner] Plan: intent=X, freshness=Y, cluster_mode=Z, subqueries=N, source=external|llm|deterministic
[Planner] sq1 label=... search="..." sources=[...]
[Planner] sq2 ...
Stderr only; does not touch the user-facing stdout synthesis. The
source= annotation distinguishes --plan (external), provider-backed
(llm), and deterministic paths — so when the 2026-04-19 Hermes Agent
Use Cases failure mode recurs, the trace tells the user which path ran
and what subqueries it produced.
Tests: added test_planner_trace_always_fires_on_mock_run which captures
stderr on a mock pipeline run and asserts the summary + per-subquery
lines appear.
The 2026-04-19 Hermes Agent Use Cases run had a Nate Herk YouTube video
titled "I Tested Claude's New Managed Agents" score 51 and rank #2
with zero Hermes content. The reranker had intent-specific scoring hints
but no entity-grounding check, so topic-vicinity matches (one offhand
OpenClaw mention) drifted to the top.
Add _primary_entity(topic) that strips intent-modifier suffixes ("use
cases", "workflows", etc.) so "Hermes Agent use cases" yields
primary_entity="Hermes Agent". Pass the entity through to both the LLM
and fallback scoring paths.
Fallback path: if primary_entity is not found (case-insensitive) in
title + snippet, subtract ENTITY_MISS_PENALTY (25 pts). Skip the
demotion for candidates with no text at all (image-only TikToks etc.)
to avoid false negatives on thin-text sources.
LLM path: add a "Primary entity grounding" hint to _build_prompt when
primary_entity is non-empty. Instructs the LLM to score candidates
without the entity at <=30.
Tests: 24 rerank tests pass, including 8 new entity-grounding tests.
Topics with suffixes like "use cases", "workflows", "review",
"examples" were previously echoed near-verbatim into search_query,
returning near-zero matches because nobody posts the literal phrase
(2026-04-19 Hermes Agent Use Cases failure).
Unit 2 — planner breadth:
1. Planner prompt rule: STRIP intent-modifier phrases from search_query
(keep them in ranking_query). Paraphrase across 4-5 subqueries that
each express the intent differently.
2. Planner prompt rule: quote only multi-word proper nouns like
"Hermes Agent", not the user's full topic.
3. Raise _max_subqueries cap from 3 to 5 for how_to / opinion / product /
breaking_news / prediction. Comparison stays at 4; factual / concept
stay at 2 unless the topic carries an intent modifier.
4. Deterministic fallback: when intent is non-{comparison,prediction}
and topic contains an intent modifier, append 3 paraphrased
subqueries (workflows, production, experience).
Unit 3 — deterministic fallback defaults:
5. _infer_intent default changed from "breaking_news" to "concept".
Prior default forced strict_recent freshness on unclassified topics,
biasing against older relevant material. Recency-signal regexes
("trending", "this week", etc.) added above the default so genuinely
time-sensitive topics still classify correctly.
6. _keyword_query now quotes only title-cased multi-word proper nouns
("Hermes Agent", "Claude Code"), not the user's full typed topic.
Hyphenated compounds and lowercase terms are left as bare keywords
so platform tokenizers broaden rather than narrow retrieval.
7. New stderr warning when plan_query runs with no --plan and no LLM
provider: surfaces that the deterministic fallback path is weaker
than the --plan-from-Claude-Code path, so callers know to generate
and pass a plan.
Tests: 37 planner tests pass, including 11 intent-modifier and 7
fallback-defaults tests.
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.
Split render_compact stdout into two bounded blocks:
- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
Emit verbatim per LAW 5.
Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.
Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
Adds docs/releases/v3.0.9.md as the GitHub Release body and appends
the matching CHANGELOG.md entry.
Covers what shipped in v3.0.9 (Class 1 refuse-gate, LAW 1 WebSearch
precedence, END-boundary, stale SKILL.md deletion) plus the community
contributions that landed across 3.0.1-3.0.8 that had never been
announced (TikTok + YouTube top comments, Hermes support, multi-key
rotation, cross-platform fixes, HTTP layer consolidation, eval
fixtures).
Contributors credited: @j-sperling, @stephenmcconnachie, @zaydiscold,
@iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma,
@BryanTegomoh, @uppinote20, @zerone0x, @thinkun, @thomasmktong,
@fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli,
@Cody-Coyote.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:
1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
Birthday-gift failure mode becomes structurally impossible - the pipeline
never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
pointing the model to ask for hobbies/relationship/budget. Escape hatch:
LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.
2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
Peter Steinberger's self-debug named the first file as the one it read
instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
Sync script simplified: Hermes now always uses main SKILL.md.
3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
with pass-through instruction. The model had the canonical body in its
buffer on the Peter run and discarded it; the boundary makes pass-through
the path of least resistance.
4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
section") that caused Peter's trailing Sources leak. No more ambiguity
at synthesis time.
Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.
Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.
Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent Opus 4.7 self-debugs on 2026-04-18 converged on the same
root cause of the v3.0.6/v3.0.7 canonical-compliance regression: SKILL.md is
42,860 tokens / 1,478 lines, LAWs lived at line 1094+, every realistic reading
strategy failed to reach them before synthesis.
Unit 1 - Moved the BADGE MANDATORY block and VOICE CONTRACT LAW 1-5 (plus
the formatting-authority preface) from line ~1090 to line ~75 (right after
the SKILL CONTRACT preface, before HOW TO INVOKE THIS SKILL). Every reading
strategy now lands the LAWs in active context before synthesis.
Unit 2 - Engine now emits the badge as the first line of --emit=compact
stdout. Passing through the script output becomes the default-correct
behavior; emitting the badge no longer depends on model compliance. Reads
version from .claude-plugin/plugin.json at runtime with graceful fallback.
Unit 3 - Deleted skills/last30days/SKILL.md stub (231-line v3-spec file).
This was the wrong-file-capture hazard Ron Conway's self-debug identified:
model grabbed the first SKILL.md find surfaced and treated it as
authoritative. Only ONE SKILL.md in the plugin package now.
Diagnoses verbatim:
- Kanye thread: "I read lines 1-600 in chunks, jumped to 300-899, then
stopped. File is 1478 lines. I never saw past ~900."
- Peter thread: "I tried Read once, hit the 25K token cap on a 42,860-token
file, and bailed instead of chunked-reading with offset/limit. I never
opened SKILL.md at all."
- Ron Conway thread: "I read one SKILL.md (231 lines)... the v3 spec stub.
I never opened the operational SKILL.md sitting next to the script."
Validation: direct engine invocation confirms badge at line 1 of compact
output. Module imports clean.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hot-fix for the public v3.0.6 0/8 regression (2026-04-18). Beta went 10/10
yesterday with the same LAW content; public went 0/8 today. The delta was
three structural anchors the port had removed or weakened.
Unit 1 - Restored MANDATORY first-line badge. Every public response now
emits "🌐 last30days v{VERSION} · synced {YYYY-MM-DD}" as line 1, blank
line, then "What I learned:" (GENERAL) or "# {TOPIC_A} vs {TOPIC_B}..."
(COMPARISON). This is the LAW 2 / LAW 4 enforcement anchor that my v3.0.6
port accidentally stripped along with the beta-specific "🧪 last30days-beta"
wording.
Unit 2 - Pinned SKILL_ROOT to the public plugin cache via
`ls -d ~/.claude/plugins/cache/last30days-skill/last30days/*/ | sort -V |
tail -1`, with a small fallback for repo/Gemini/Codex hosts. Replaces the
path-discovery loop that was landing on stale copies (~/.openclaw/,
~/.agents/, ~/.codex/) on machines with a private-repo sync history.
Unit 3 - Added a "SKILL CONTRACT" preface at the top of SKILL.md that names
the 0/8 regression as a documented failure mode and explicitly tells the
model not to treat /last30days as a generic keyword. Encodes user theory
that "/last30days-beta" sounded specific enough to trigger skill-follow
mode while "/last30days" reads as a search term and triggers improvise
mode.
Validation: all three anchors visible in the grep check for public
v3.0.7 cache. Next validation is manual re-run of the 8 failure topics
on public after shipping.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.
Plans bundled in this release:
- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
+ VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
+ top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
ranking rewrite + Polymarket post-merge topic filter (engine change,
filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
post-synthesis Sources-block self-check
Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).
No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Trevin Chow