- Remove duplicate detect_query_type from query.py (divergent 5-type version);
canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.
This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
- hackernews: use extract_core_subject instead of raw topic, add
points>5 filter and restrictSearchableAttributes=title to reduce
noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
- bird_x: parse_bird_response now accepts query param and computes
token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement
This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.
Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.
Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set
Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.
Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.
Integration into per-module imports follows in next commits.
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.
OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering
xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
grok-4-1-fast-non-reasoning — same token price, faster response,
no wasted reasoning tokens for structured extraction
Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.
- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:
1. Source selection: each query type has tier-1 (always run) and tier-2
(run if available) sources. Unlisted sources are opt-in only.
Truth Social is always opt-in regardless of query type.
2. WebSearch penalty: varies by query type instead of flat -15pt.
Concept queries get 0 penalty (web docs are authoritative),
how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.
3. Tiebreaker ordering: source priority varies by query type.
YouTube ranks first for how_to, Polymarket for prediction,
HN for concept queries, X for breaking news.
All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.
Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.
27 new tests, 440 total passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).
- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).
New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.
Fixes#55.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add per-project configuration via .claude/last30days.env, discovered by
walking up from cwd. Uses the same .env format as the existing global
config — no new parsers or formats.
Priority (highest wins):
1. Environment variables
2. .claude/last30days.env (per-project)
3. ~/.config/last30days/.env (global)
Also adds file permission checking — warns to stderr if config files
are readable by other users (should be chmod 600).
Includes tests for discovery, precedence, source tracking, and
permission warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts between ScrapeCreators Reddit (main) and
public Reddit fallback (PR #48). Priority: ScrapeCreators ->
OpenAI -> public Reddit fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --save-dir flag to last30days.py that saves raw research output
during the existing script run. Remove entire "Save Research to
Documents" section from SKILL.md (~45 lines). No more extra tool
calls, no (No output), no multi-minute cogitation after invitation.
Tested: --mock confirms file creation and duplicate date suffixing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New scripts/lib/reddit.py: multi-query expansion, global search,
subreddit discovery, targeted subreddit search, comment enrichment
- 68 results in 17s vs ~15 results in 60-90s (OpenAI)
- Cost: ~$0.02/search vs $0.03-0.10 (15-50x cheaper)
- Real engagement data (score, comments, dates) from API
- No more 429 rate limits on comment enrichment
- Falls back to OpenAI if SCRAPECREATORS_API_KEY missing
- Registered as last30daysbeta for parallel local testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- add xiaohongshu/xhs source path via xiaohongshu-mcp HTTP API\n- add Reddit public JSON fallback when OpenAI auth is unavailable\n- update diagnostics/UI rendering for new source availability states\n- harden Xiaohongshu availability probe to reduce false negatives\n- include source status reporting for Xiaohongshu
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.
- Add scripts/lib/instagram.py: keyword search, transcript extraction,
relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause of empty TikTok results: Apify required monthly subscription.
ScrapeCreators is PAYG with 100 free credits and no subscription.
Key fix: ScrapeCreators nests items under aweme_info wrapper
(search_item_list[].aweme_info.{fields}), which the previous
implementation missed, causing all fields to be empty.
Changes:
- Rewrite tiktok.py to use ScrapeCreators REST API
- Add aweme_info unwrapping for correct field extraction
- Add transcript fetching via /video/transcript endpoint
- Add SCRAPECREATORS_API_KEY to env.py config
- Update last30days.py to use env.get_tiktok_token()
- Delete apify_client_wrapper.py (no longer needed)
- Update tests for new date field format (create_time)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove double quotes around $ARGUMENTS in SKILL.md so bash word-splits
the expansion, and change argparse topic from nargs="?" to nargs="*"
so multi-word topics still work. Also document --store, --include-web,
--diagnose, and --timeout flags in the Options section.
Closes#36
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only print debug log lines when running in an interactive terminal.
In Claude Code (non-TTY), the spinner system handles progress display,
so these raw log lines just add noise.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass logger=None to Apify .call() to prevent the SDK from streaming
raw actor run logs (status messages, crawler stats, warnings) that
drown out the clean spinner UI in Claude Code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove quiet=True from YouTube and TikTok spinners so they display
the same colored emoji progress lines as Reddit and X in non-TTY mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When running in Claude Code, the assistant has a built-in WebSearch tool
that's free and higher quality than Parallel AI/Brave/OpenRouter. Adding
--no-native-web to the SKILL.md invocation defers web search to the
assistant, saving API credits. OpenClaw invocations don't pass this flag,
so they continue using native web backends.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The websearch module import was dropped when the tiktok import was added,
causing the script to crash during the rendering phase after all data
was successfully collected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TikTok search, scoring, and rendering using the Apify platform
(clockworks/tiktok-scraper actor). Users bring their own APIFY_API_TOKEN
($5/month free credits, no CC required). The shared apify_client_wrapper
module is designed for reuse by future Facebook/Instagram sources.
- New modules: tiktok.py (search + caption extraction), apify_client_wrapper.py
- Schema: TikTokItem dataclass, shares field on Engagement, Report.tiktok
- Pipeline: normalize → filter → score → sort → dedupe → cross-link → render
- Scoring: 0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)
- SKILL.md bumped to v2.7 with TikTok stats, citations, and security docs
- 26 unit tests covering relevance, normalize, score, dedupe, render, round-trip
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inspired by PR #26 (wkbaran), whose early work on HN/YouTube sources helped
shape what we built in v2.5. Cherry-picks the source-filtering concept as a
clean implementation against our existing architecture.
--search=SOURCES accepts comma-separated: reddit, x, hn, youtube, polymarket, web
Example: --search reddit,hn (run only Reddit + Hacker News)
Also:
- bird_x: add noise words (trending, viral, plugin, skills) + last-chance retry
- render: show xAI tip for reddit-only mode regardless of missing_keys value
- tests: new test_bird_x.py (5 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool
noise words to _extract_core_subject, and a last-chance retry that falls back
to the longest non-noise token when 2-word retry also returns 0 results.
cache.py and render.py env overrides were already on main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: fix YAML error in argument-hint
* add codex auth support to responses API
* Use gpt-5.1-codex-mini as default model for Codex auth
Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.
* Pass .env credentials to Bird Node subprocesses for X auth
On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.
- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection
---------
Co-authored-by: Justin Williams <jblwilliams@gmail.com>
The Gamma API only searches event titles/slugs, missing markets where the
topic is an outcome (e.g., "Arizona" in NCAA Tournament Winner). This adds:
- All-word query expansion (not just first word): "Arizona Basketball" now
searches "Arizona", "Basketball" independently
- Tag-based domain expansion: extracts category tags (e.g., "NCAA") from
first-pass results and searches those as a second pass
- Neg-risk binary market synthesis: shows team names from market questions
instead of generic Yes/No outcomes
- Question shortening: extracts "Arizona" from "Will Arizona win the NCAA
Tournament?" for clean display
- Increased depth (3 pages) and result caps (15) for more coverage
Live results: "Arizona Basketball" now finds NCAA Tournament Winner (12%),
#1 Seed (88%), Big 12 Champion (69%). "Iran War" returns 15 markets (up
from 9) with no regression.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CROSS to sync targets with sed-patched name/version/description
- Switch cp to rsync to handle identical file edge case on APFS
- CROSS SKILL.md gets last30daysCROSS skill root injected into search path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _compute_text_similarity() now checks outcome names with bidirectional
substring matching (0.85) and token overlap (0.7), not just event titles
- Collect outcomes from ALL active markets per event, filter to >1% price
- Reorder outcome_prices to surface topic-matching outcome before top-3 truncation
- Add SKILL.md "Prediction Markets" synthesis section with structural/long-term
market preference, domain examples, citation format, and narrative weaving
- Add Polymarket to citation priority list between HN and Web
- Update stats box template to show up to 5 market odds
- Fix render.py "vol24h" label to "volume"
- Add NCAA seed fixture event for outcome-only matching tests
- 82 polymarket tests pass (14 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polymarket results now rank by text similarity, volume, liquidity, price
movement, and competitive score instead of API return position. Also fixes
pagination (DEPTH_CONFIG now controls page count, not a no-op limit param)
and caps results after re-ranking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The resolved handle dedup was wrong: if entity_extract found @thedorbrothers
(from @mentions in Phase 1 results), the resolved handle search was skipped
entirely. But entity-extracted handles are searched WITH topic keywords
(from:handle topic), while resolved handles need UNFILTERED search
(from:handle) to find posts that don't mention the topic string.
Example: Dor Brothers' viral tweet (5.5K likes) says "We made a $300M movie
starring @LoganPaul" - no mention of "dor brothers" anywhere. The topic-
filtered entity search missed it. The unfiltered resolved search finds it.
Before: 30 X posts, 161+ likes (entity search only)
After: 40 X posts, 5549+ likes (resolved handle adds viral tweet)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.
- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics
(4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further:
- dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold,
cross-source links went from 3 to 13 items across 5 topics
- render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags
- youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score)
- SKILL.md: instruction #7 tells Claude to lead with cross-platform signals
Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5
topics. Biggest gains in specificity (+0.8) and format compliance (+1.0).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YouTube videos now get real relevance scores based on token overlap
between the search query and video title (was hardcoded at 0.7).
Uses ratio overlap with stopword removal, floored at 0.1.
Cross-source linking annotates items that discuss the same story
across different platforms (e.g., Reddit + HN + X). Items get
bidirectional cross_refs displayed as [xref: R3, HN5] in compact
output so Claude can triangulate multi-platform coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Jeffrey Sperling