Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.
Plans bundled in this release:
- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
+ VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
+ top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
ranking rewrite + Polymarket post-merge topic filter (engine change,
filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
post-synthesis Sources-block self-check
Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).
No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Most users never touch FUN_LEVEL. Default medium was shipping a stats
block but rarely a Best Takes block, and when it did it was below the
cluster fold where a synthesizing model had already stopped reading.
A 2,304-upvote Reddit comment ("WHAT?! I reached my monthly limit
just reading this post") on the 2026-04-17 Opus 4.7 run sat inside
cluster 11 and never made it into synthesis. Four coordinated changes:
1. render: promote Best Takes above the cluster list so the synthesizer
sees comedy before it anchors on cluster 1.
2. render: lower medium threshold from 70 to 55 (heuristic maxes at 80),
drop the two-gem floor to one-gem. Default now reliably emits the
block on typical runs.
3. rerank: score individual top_comments by upvote ratio to their parent
thread. A 2,304-upvote comment on a 300-upvote thread now outranks a
400-upvote comment on a 3,400-upvote thread, which is the viral-wit
signal. Handles both the LLM scoring path and the heuristic fallback.
4. render: merge scored comment gems into Best Takes alongside candidate
gems, sorted together. Comment lines show body + parent title +
r/subreddit or @handle + absolute upvotes.
5. SKILL: tell the synthesizer to quote at least two Best Takes entries
verbatim, with an example of the new comment format.
Plan: docs/plans/2026-04-17-001-feat-default-fun-surfacing-plan.md
🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.56.1
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.
SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TRANSCRIPT_MAX_WORDS raised from 500 to 5000 so the LLM gets the full
content of most videos (up to ~25 minutes). Removed the second 200-char
truncation in render.py that was reducing transcripts to a single sentence
before the judge agent ever saw them.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.
27 new tests, 440 total passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).
- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.
- Add scripts/lib/instagram.py: keyword search, transcript extraction,
relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.
- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.
--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn (run only Reddit + Hacker News)
Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: fix YAML error in argument-hint
* add codex auth support to responses API
* Use gpt-5.1-codex-mini as default model for Codex auth
Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.
* Pass .env credentials to Bird Node subprocesses for X auth
On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.
- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection
---------
Co-authored-by: Justin Williams <jblwilliams@gmail.com>
- _compute_text_similarity() now checks outcome names with bidirectional
substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.
- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics
(4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further:
- dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold,
cross-source links went from 3 to 13 items across 5 topics
- render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags
- youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score)
- SKILL.md: instruction #7 tells Claude to lead with cross-platform signals
Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5
topics. Biggest gains in specificity (+0.8) and format compliance (+1.0).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.
Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HN was appearing before YouTube in the stats block, sort tiebreaker,
and source status. Now consistently: Reddit > X > YouTube > HN > Web.
Also restored emoji + box-drawing chars in test skill SKILL.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:
- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override
Also fixes the UX gap where missing sources were silently skipped:
- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add agents/openai.yaml for Codex skill discovery
- Make SKILL.md script path portable (repo, Claude, Codex, agents dirs)
- Platform-neutral output text ("assistant" instead of "Claude")
- Sandbox-friendly cache/output dirs with env var overrides and tempdir fallback
- Add Codex installation docs to README
Inspired by PR #24 (el-analista) and PR #5 (jblwilliams).
Zero impact on existing Claude Code behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube search and transcript extraction runs automatically when yt-dlp
is installed. Searches for topic videos from the last N days, fetches
auto-generated transcripts for top results, and feeds them through the
same scoring pipeline (relevance + recency + engagement) as Reddit/X.
New files:
- youtube_yt.py: search, transcript extraction, VTT cleanup
Modified files:
- schema.py: YouTubeItem dataclass, updated Report
- normalize.py: normalize_youtube_items()
- score.py: YouTube engagement scoring (views-dominated)
- dedupe.py: YouTube deduplication
- render.py: YouTube section in compact output
- env.py: is_ytdlp_available() check
- ui.py: YouTube progress messages
- last30days.py: _search_youtube(), parallel execution with Reddit/X
- SKILL.md: YouTube in stats box, citation priority
- README.md: YouTube docs, yt-dlp requirement, Peter shoutout
Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ⚠️ LIMITED RECENT DATA warning when:
- Fewer than 5 items are confirmed from the date range
- Tells Claude to be transparent with user about data freshness
Example output for obscure topic (June Oven):
"Only 4 item(s) confirmed from 2025-12-26 to 2026-01-25.
Results below may include older/evergreen content."
Popular topics (clawdbot, nano banana) don't show the warning.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Skill now works without any API keys using WebSearch fallback
- Shows promo banner marketing Reddit/X data when keys are missing
- Partial mode (one key) shows shorter tip for the missing source
- Updated SKILL.md to document three modes: Full, Partial, Web-Only
- Added get_missing_keys() to env.py for promo logic
- Added show_promo(), start_web_only(), show_web_only_complete() to ui.py
- Updated render_compact() to include inline promo for web-only mode
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Claude's built-in WebSearch tool as a third research source for
/last30days. This enables the skill to work out of the box with zero
API keys while preserving Reddit/X as the primary sources.
Key changes:
- Add WebSearchItem schema for web results (no engagement metrics)
- Add score_websearch_items() with 55/45 relevance/recency weighting
- Apply -15pt source penalty so WebSearch ranks below Reddit/X
- Add --include-web CLI flag to opt-in to WebSearch
- Return 'web' mode when no API keys configured (zero-config)
- Update render.py with [WEB] source label formatting
When WebSearch is enabled, the script outputs instructions for Claude
to use its built-in WebSearch tool, then synthesize results together.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Show "⚡ CACHED RESULTS (Xh old)" in compact output header
- Add "use --refresh for fresh data" hint
- Track from_cache and cache_age_hours in Report schema
- Update UI to show cache age in stderr message
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- SKILL.md: Move "What I learned" BEFORE "Research Complete" stats
- Add error tracking to Report schema (reddit_error, x_error fields)
- Wrap OpenAI API calls in try/catch with clear error messages
- Show explicit error or "no results" messages in compact output
- Fix false positive error detection for null error fields
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Research topics across Reddit + X from the last 30 days using
OpenAI and xAI APIs. Features:
- Auto model selection (GPT-5.x, Grok-3)
- Popularity-aware scoring (relevance + recency + engagement)
- Reddit thread enrichment with real metrics
- Near-duplicate detection
- Multiple emit modes (compact, json, context, path)
- 24h caching with --refresh bypass
- NUX for API key setup
- 87 passing unit tests
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Matt Van Horn