Adds xurl (https://github.com/openclaw/xurl) as a third X search
backend, sitting after xAI API and Bird/GraphQL in the priority chain.
xurl uses the official X API v2 with OAuth2+PKCE authentication,
requiring only a free X Developer App. It auto-refreshes tokens and
works reliably as a stable fallback when xAI API key or browser
cookies are not available.
Limitations:
- X API search/recent returns last 7 days only (vs Bird's full archive)
- No AI-powered relevance scoring (uses token_overlap_relevance instead)
- Free tier: 180 requests per 15-minute window
New files:
- scripts/lib/xurl_x.py: xurl CLI wrapper with search + parse
- tests/test_xurl_x.py: 30 unit tests (all passing)
Modified files:
- scripts/lib/env.py: detect xurl in get_x_source_with_method(),
get_missing_keys(), and get_x_source_status()
- scripts/last30days.py: add xurl_x import and xurl branch in
_search_x() priority chain
- SKILL.md: document xurl setup option
Inline markdown links on every narrative citation (@handle, r/sub,
publication, YouTube channel, TikTok/Instagram creator, Polymarket
market). Raw URL strings remain forbidden. Plain-text fallback when the
raw data has no URL for a specific source.
Commit 1 (790e5bc) added the citation rule in CITATION PRIORITY / URL
FORMATTING. Live tests showed the rule was deployed but consistently
skipped because it lived at line 1224, below the agent's chunked-read
window. Commit 2 (5864c687) hoists the rule into the VOICE CONTRACT
LAW block as LAW 8, at line 167 - inside the guaranteed-loaded top
band alongside LAWs 1-7. Same pattern that fixed v3.0.6 (invented
titles), disaster #2 (stripped bold), disaster #3 (trailing Sources),
and the 2026-04-19 Hermes evidence-dump disaster.
No Python engine changes. Rule is prompt-only; the deterministic
stats footer (LAW 5) is unchanged.
Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md
Run 1 of /last30days Hermes Agent use cases on 2026-04-19 called the engine
bare despite SKILL.md already having a detailed Step 0.75 (YOU are the
planner) and a PRECONDITION GATE requiring --plan. Those lived at lines
647 and 729 - the model didn't reach them before invoking Bash.
LAW 7 hoists the rule into the OUTPUT CONTRACT block at the top (same
placement pattern as LAW 6), so it is the first thing the model reads.
Runtime-agnostic language: Claude Code, Codex, Hermes, Gemini, or any
agent runtime. Named failure mode with the misread diagnosis: "provider"
in engine messages refers to the engine's INTERNAL planner credentials,
NOT a prerequisite the caller needs - if you are the hosting reasoning
model, YOU are the provider.
Concrete self-check: re-read pending Bash command; if no --plan and topic
is a named entity, STOP and generate a plan.
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.
Split render_compact stdout into two bounded blocks:
- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
Emit verbatim per LAW 5.
Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.
Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:
1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
Birthday-gift failure mode becomes structurally impossible - the pipeline
never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
pointing the model to ask for hobbies/relationship/budget. Escape hatch:
LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.
2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
Peter Steinberger's self-debug named the first file as the one it read
instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
Sync script simplified: Hermes now always uses main SKILL.md.
3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
with pass-through instruction. The model had the canonical body in its
buffer on the Peter run and discarded it; the boundary makes pass-through
the path of least resistance.
4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
section") that caused Peter's trailing Sources leak. No more ambiguity
at synthesis time.
Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.
Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.
Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three independent Opus 4.7 self-debugs on 2026-04-18 converged on the same
root cause of the v3.0.6/v3.0.7 canonical-compliance regression: SKILL.md is
42,860 tokens / 1,478 lines, LAWs lived at line 1094+, every realistic reading
strategy failed to reach them before synthesis.
Unit 1 - Moved the BADGE MANDATORY block and VOICE CONTRACT LAW 1-5 (plus
the formatting-authority preface) from line ~1090 to line ~75 (right after
the SKILL CONTRACT preface, before HOW TO INVOKE THIS SKILL). Every reading
strategy now lands the LAWs in active context before synthesis.
Unit 2 - Engine now emits the badge as the first line of --emit=compact
stdout. Passing through the script output becomes the default-correct
behavior; emitting the badge no longer depends on model compliance. Reads
version from .claude-plugin/plugin.json at runtime with graceful fallback.
Unit 3 - Deleted skills/last30days/SKILL.md stub (231-line v3-spec file).
This was the wrong-file-capture hazard Ron Conway's self-debug identified:
model grabbed the first SKILL.md find surfaced and treated it as
authoritative. Only ONE SKILL.md in the plugin package now.
Diagnoses verbatim:
- Kanye thread: "I read lines 1-600 in chunks, jumped to 300-899, then
stopped. File is 1478 lines. I never saw past ~900."
- Peter thread: "I tried Read once, hit the 25K token cap on a 42,860-token
file, and bailed instead of chunked-reading with offset/limit. I never
opened SKILL.md at all."
- Ron Conway thread: "I read one SKILL.md (231 lines)... the v3 spec stub.
I never opened the operational SKILL.md sitting next to the script."
Validation: direct engine invocation confirms badge at line 1 of compact
output. Module imports clean.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hot-fix for the public v3.0.6 0/8 regression (2026-04-18). Beta went 10/10
yesterday with the same LAW content; public went 0/8 today. The delta was
three structural anchors the port had removed or weakened.
Unit 1 - Restored MANDATORY first-line badge. Every public response now
emits "🌐 last30days v{VERSION} · synced {YYYY-MM-DD}" as line 1, blank
line, then "What I learned:" (GENERAL) or "# {TOPIC_A} vs {TOPIC_B}..."
(COMPARISON). This is the LAW 2 / LAW 4 enforcement anchor that my v3.0.6
port accidentally stripped along with the beta-specific "🧪 last30days-beta"
wording.
Unit 2 - Pinned SKILL_ROOT to the public plugin cache via
`ls -d ~/.claude/plugins/cache/last30days-skill/last30days/*/ | sort -V |
tail -1`, with a small fallback for repo/Gemini/Codex hosts. Replaces the
path-discovery loop that was landing on stale copies (~/.openclaw/,
~/.agents/, ~/.codex/) on machines with a private-repo sync history.
Unit 3 - Added a "SKILL CONTRACT" preface at the top of SKILL.md that names
the 0/8 regression as a documented failure mode and explicitly tells the
model not to treat /last30days as a generic keyword. Encodes user theory
that "/last30days-beta" sounded specific enough to trigger skill-follow
mode while "/last30days" reads as a search term and triggers improvise
mode.
Validation: all three anchors visible in the grep check for public
v3.0.7 cache. Next validation is manual re-run of the 8 failure topics
on public after shipping.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.
Plans bundled in this release:
- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
+ VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
+ top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
ranking rewrite + Polymarket post-merge topic filter (engine change,
filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
post-synthesis Sources-block self-check
Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).
No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Most users never touch FUN_LEVEL. Default medium was shipping a stats
block but rarely a Best Takes block, and when it did it was below the
cluster fold where a synthesizing model had already stopped reading.
A 2,304-upvote Reddit comment ("WHAT?! I reached my monthly limit
just reading this post") on the 2026-04-17 Opus 4.7 run sat inside
cluster 11 and never made it into synthesis. Four coordinated changes:
1. render: promote Best Takes above the cluster list so the synthesizer
sees comedy before it anchors on cluster 1.
2. render: lower medium threshold from 70 to 55 (heuristic maxes at 80),
drop the two-gem floor to one-gem. Default now reliably emits the
block on typical runs.
3. rerank: score individual top_comments by upvote ratio to their parent
thread. A 2,304-upvote comment on a 300-upvote thread now outranks a
400-upvote comment on a 3,400-upvote thread, which is the viral-wit
signal. Handles both the LLM scoring path and the heuristic fallback.
4. render: merge scored comment gems into Best Takes alongside candidate
gems, sorted together. Comment lines show body + parent title +
r/subreddit or @handle + absolute upvotes.
5. SKILL: tell the synthesizer to quote at least two Best Takes entries
verbatim, with an example of the new comment format.
Plan: docs/plans/2026-04-17-001-feat-default-fun-surfacing-plan.md
🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restore the rich synthesis output by closing three prompt-level loopholes
that let the model silently take a degraded path:
1. Research Execution precondition gate. Steps 0.55 (entity resolution)
and 0.75 (query planner) are now non-skippable on WebSearch platforms.
--emit md is banned as a primary user-facing flow; --emit=compact with
--plan is mandatory. OpenClaw --auto-resolve fallback preserved.
2. WebSearch "Sources:" mandate override. The WebSearch tool description
contains a CRITICAL/MUST mandate to append a Sources section. That is
explicitly superseded inside /last30days with matched-register
CRITICAL/MANDATORY override language and a BAD/GOOD example. The
existing web-source line is the citation; nothing appends below the
invitation.
3. Pre-present self-check. Before displaying, the model verifies bold
per-paragraph headlines, per-source emoji stats, quoted highlights,
Polymarket block, coverage footer, and (critically) no trailing
Sources block. One regeneration permitted if checks fail.
Also adds explicit MANDATORY language to the "What I learned" template
requiring bold headline phrases on every narrative paragraph.
Root cause: same-session A/B on 2026-04-15 between /last30days kanye
west (rich output, ran Steps 0.55 + 0.75, --emit=compact --plan) and
/last30days hermes ai (bland output, skipped both, --emit md) showed
the template was fine -- the model was lazily taking a shortcut SKILL.md
tolerated. No engine, render.py, or contributor PR was the cause.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
- description leads with imperative 'Research' + 'what people actually say' (strong trigger signal for community/social-research prompts)
- argument-hint shows 3 concrete user phrasings instead of marketing copy
- 176 chars, well under Anthropic's 200-char cap
- preserves all source coverage (Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, web)
Per ecosystem research (April 2026), trigger description quality is the single
biggest lever separating 500-install skills from 350k-install skills.
Atomic bump across all four manifests:
- SKILL.md (root)
- skills/last30days/SKILL.md (internal spec)
- .claude-plugin/plugin.json
- gemini-extension.json
CHANGELOG entry documents the skill-upload packaging fix, vendor/ removal,
legacy plans/ removal, and the new scripts/build-skill.sh builder.
The SKILL.md prompt header still said v2.9.5 while pyproject.toml
and the rest of the codebase are on v3.0.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX
- INCLUDE_SOURCES=tiktok,instagram in .env forces sources on for all
query types, bypassing the tier system
- NUX shows opt-in modal after ScrapeCreators key is saved: "Also
search TikTok and Instagram?" with honest call-usage warning
- Tier system preserved as default — override only when INCLUDE_SOURCES set
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: neutral call-usage copy — works for free and paid tiers
---------
Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.
SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated frontmatter description to be more search-friendly for ClawHub.
Added 12 new tags: deep-research, twitter, bluesky, recency, news,
citations, multi-source, social-media, analysis, web-search, ai-skill,
clawhub. Also added 11 GitHub repo topics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The short alias /last30 only works on some platforms. Claude Code requires
the full /last30days name, so the follow-up suggestions after comparison
research were producing "Unknown skill: last30" errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.
27 new tests, 440 total passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add COMPARISON query type for "X vs Y" research with 3 parallel passes.
Add Bluesky stats line and update all source list references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove double quotes around $ARGUMENTS so argparse can parse flags
like --deep, --store separately instead of as part of the topic string.
Fixes#61.
- Add ~/.claude/plugins/marketplaces/last30days-skill to the path
discovery loop so marketplace installs can find scripts/.
Fixes#54.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.
Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.
Closes#45
Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PR merges on March 7 (PR #48 Xiaohongshu, upstream merge) regressed
SKILL.md by re-introducing the "Save Research to Documents" section
that v2.9.4 removed. Those branches were forked before v2.9.4 and
brought the old content back via merge resolution.
Fixes: remove save section, restore --save-dir flag on bash command,
update agent mode line, add tool-call guard to STOP instruction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove prompt-injection false positive ("you are now" → "treat yourself as")
- Declare AUTH_TOKEN and CT0 in frontmatter optionalEnv
- Clarify X token access language (no browser session access)
- Add permissions overview block near top of file
Zero functionality changes — metadata and prose only.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.
Tested: --mock confirms file creation and duplicate date suffixing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CRITICAL: run_in_background callbacks caused model to re-engage after
save, hallucinate fake "Human:" messages, and generate unsolicited
multi-paragraph responses. Switch to foreground cat > heredoc which
executes sub-second with no callback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Background Bash heredoc instead of Write tool
- Suppress response text on save completion
- 📎 footer line replaces verbose confirmation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Write tool displays "Wrote N lines..." after the invitation,
ruining the end-of-run experience. Now saves via background Bash
with a subtle 📎 footer line in the invitation text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sync from public repo. Every run now saves the complete briefing as a
topic-named .md file to ~/Documents/Last30Days/. Credit @devin_explores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bump version to 2.9.1, update changelog and release notes.
Credit @devin_explores for inspiring the feature.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Every run now automatically saves the complete briefing (synthesis,
stats, follow-up suggestions) as a topic-named .md file in the user's
Documents folder. Agent mode also saves. No Python script changes -
this is purely a SKILL.md instruction addition.
Inspired by @devin_explores manually saving results to build a
personal research library.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
weichao