Five Opus 4.7 self-debugs on v3.0.8 (3 passing, 2 failing runs) converged
on four fixes:
1. Engine refuses Class 1 demographic-shopping queries at main() front-door.
Birthday-gift failure mode becomes structurally impossible - the pipeline
never runs on a doomed query. Exit code 2 with a REFUSE message on stderr
pointing the model to ask for hobbies/relationship/budget. Escape hatch:
LAST30DAYS_SKIP_PREFLIGHT=1 for "just run it" overrides.
2. Delete stale `.agents/skills/last30days/SKILL.md` (1382 lines, April 13
snapshot) and `.hermes-plugin/SKILL.md` (269 lines, April 13 snapshot).
Peter Steinberger's self-debug named the first file as the one it read
instead of the real SKILL.md. One SKILL.md per plugin, at the plugin root.
Sync script simplified: Hermes now always uses main SKILL.md.
3. render_compact() appends an explicit END-OF-CANONICAL-OUTPUT boundary
with pass-through instruction. The model had the canonical body in its
buffer on the Peter run and discarded it; the boundary makes pass-through
the path of least resistance.
4. LAW 1 gains a verbatim-pattern override clause naming the exact WebSearch
tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources:
section") that caused Peter's trailing Sources leak. No more ambiguity
at synthesis time.
Tests: tests/test_preflight.py, 29 scenarios covering Class 1 matches
(birthday gift, best-for-demographic, what-to-buy-relationship), qualifier
skips (budget, hobbies, activity after year-old), and the REFUSE message
shape.
Validation gate before merging to main: re-run the 5 debug topics
(Peter Steinberger, birthday gift for 40 year old, Kanye West, Garry Tan,
OpenClaw vs Paperclip vs Hermes) on v3.0.9 and confirm 5/5 canonical
compliance. Rollback to v3.0.8 if any previously-passing topic regresses.
Plan: docs/plans/2026-04-18-015-fix-engine-refuse-keyword-traps-delete-stale-skillmd-files-plan.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.
Plans bundled in this release:
- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
+ VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
+ top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
ranking rewrite + Polymarket post-merge topic filter (engine change,
filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
post-synthesis Sources-block self-check
Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).
No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX
- INCLUDE_SOURCES=tiktok,instagram in .env forces sources on for all
query types, bypassing the tier system
- NUX shows opt-in modal after ScrapeCreators key is saved: "Also
search TikTok and Instagram?" with honest call-usage warning
- Tier system preserved as default — override only when INCLUDE_SOURCES set
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: neutral call-usage copy — works for free and paid tiers
---------
Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.
Also apply the relevance filter to Polymarket and update the affected cross-source tests.
Validation: uv run python -m unittest
Update README, launch copy, and UI guidance to prefer popup-free AUTH_TOKEN/CT0 configuration, and keep X backend selection on the verified Bird or xAI paths.
Validation: uv run python -m unittest tests.test_env_project
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.
This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
- bird_x: parse_bird_response now accepts query param and computes
token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement
This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.
- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:
1. Source selection: each query type has tier-1 (always run) and tier-2
(run if available) sources. Unlisted sources are opt-in only.
Truth Social is always opt-in regardless of query type.
2. WebSearch penalty: varies by query type instead of flat -15pt.
Concept queries get 0 penalty (web docs are authoritative),
how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.
3. Tiebreaker ordering: source priority varies by query type.
YouTube ranks first for how_to, Polymarket for prediction,
HN for concept queries, X for breaking news.
All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.
27 new tests, 440 total passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).
- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).
New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.
Fixes#55.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.
Tested: --mock confirms file creation and duplicate date suffixing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New scripts/lib/reddit.py: multi-query expansion, global search,
subreddit discovery, targeted subreddit search, comment enrichment
- 68 results in 17s vs ~15 results in 60-90s (OpenAI)
- Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper)
- Real engagement data (score, comments, dates) from API
- No more 429 rate limits on comment enrichment
- Falls back to OpenAI if SCRAPECREATORS_API_KEY missing
- Registered as last30daysbeta for parallel local testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.
- Add scripts/lib/instagram.py: keyword search, transcript extraction,
relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause of empty TikTok results: Apify required monthly subscription.
ScrapeCreators is PAYG with 100 free credits and no subscription.
Key fix: ScrapeCreators nests items under aweme_info wrapper
(search_item_list[].aweme_info.{fields}), which the previous
implementation missed, causing all fields to be empty.
Changes:
- Rewrite tiktok.py to use ScrapeCreators REST API
- Add aweme_info unwrapping for correct field extraction
- Add transcript fetching via /video/transcript endpoint
- Add SCRAPECREATORS_API_KEY to env.py config
- Update last30days.py to use env.get_tiktok_token()
- Delete apify_client_wrapper.py (no longer needed)
- Update tests for new date field format (create_time)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits
the expansion, and change argparse topic from nargs="?" to nargs="*"
so multi-word topics still work. Also document --store, --include-web,
--diagnose, and --timeout flags in the Options section.
Closes#36
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When running in Claude Code, the assistant has a built-in WebSearch tool
that's free and higher quality than Parallel AI/Brave/OpenRouter. Adding
--no-native-web to the SKILL.md invocation defers web search to the
assistant, saving API credits. OpenClaw invocations don't pass this flag,
so they continue using native web backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The websearch module import was dropped when the tiktok import was added,
causing the script to crash during the rendering phase after all data
was successfully collected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.
- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.
--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn (run only Reddit + Hacker News)
Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: fix YAML error in argument-hint
* add codex auth support to responses API
* Use gpt-5.1-codex-mini as default model for Codex auth
Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.
* Pass .env credentials to Bird Node subprocesses for X auth
On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.
- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection
---------
Co-authored-by: Justin Williams <jblwilliams@gmail.com>
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The resolved handle dedup was wrong: if entity_extract found @thedorbrothers
(from @mentions in Phase 1 results), the resolved handle search was skipped
entirely. But entity-extracted handles are searched WITH topic keywords
(from:handle topic), while resolved handles need UNFILTERED search
(from:handle) to find posts that don't mention the topic string.
Example: Dor Brothers' viral tweet (5.5K likes) says "We made a $300M movie
starring @LoganPaul" - no mention of "dor brothers" anywhere. The topic-
filtered entity search missed it. The unfiltered resolved search finds it.
Before: 30 X posts, 161+ likes (entity search only)
After: 40 X posts, 5549+ likes (resolved handle adds viral tweet)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.
- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.
Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OpenAI Responses API web_search takes 60-90s but the generic
future timeout was killing it at 30s (quick) / 60s (default).
Added reddit_future key to TIMEOUT_PROFILES (60/90/120s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube: Add youtube_future timeout key (60/90/120s for quick/default/deep)
separate from the shared future timeout. YouTube needs more time because
it does search + parallel transcript fetching. Previously, 20 videos +
5 transcripts exceeded the 60s budget and all results were discarded.
Reddit 429: Propagate rate-limit errors instead of swallowing them.
Enrichment now uses 10s timeout / 1 retry (was 30s / 3 retries).
On first 429, cancel remaining enrichment and skip Phase 2 Reddit.
Total time wasted on 429 drops from ~75s to ~12s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hard date filter in the main script was double-filtering YouTube
results. youtube_yt.py already applies a soft date filter that prefers
recent videos but keeps older ones when < 3 are within range (for
evergreen topics like "youtube thumbnails"). The hard filter then
removed all of them, resulting in 0 YouTube items.
YouTube content has a longer shelf life than tweets/posts, so the
soft filter's fallback behavior is correct.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dave Morin