Map prompt-oriented product searches and animation-oriented build searches away from the breaking-news default so source tiering and tiebreakers align with the benchmark topics.
Validation: uv run python -m unittest tests.test_query_type
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.
Validation: uv run python -m unittest tests.test_evaluate_search_quality
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.
Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.
The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.
Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.
Also apply the relevance filter to Polymarket and update the affected cross-source tests.
Validation: uv run python -m unittest
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.
Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
Classify prompting and animation queries as how_to so the stack does not treat them as generic breaking news. Also keep X available for how_to and preserve YouTube/HN coverage for breaking-news and prediction queries.
Validated with uv run python -m unittest tests.test_query_type and the five-query local comparison run used for PR #65 review.
Update README, launch copy, and UI guidance to prefer popup-free AUTH_TOKEN/CT0 configuration, and keep X backend selection on the verified Bird or xAI paths.
Validation: uv run python -m unittest tests.test_env_project
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.
This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
- hackernews: use extract_core_subject instead of raw topic, add
points>5 filter and restrictSearchableAttributes=title to reduce
noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
- bird_x: parse_bird_response now accepts query param and computes
token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement
This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.
Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.
Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set
Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.
Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.
Integration into per-module imports follows in next commits.
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.
OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering
xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
grok-4-1-fast-non-reasoning — same token price, faster response,
no wasted reasoning tokens for structured extraction
Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.
- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:
1. Source selection: each query type has tier-1 (always run) and tier-2
(run if available) sources. Unlisted sources are opt-in only.
Truth Social is always opt-in regardless of query type.
2. WebSearch penalty: varies by query type instead of flat -15pt.
Concept queries get 0 penalty (web docs are authoritative),
how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.
3. Tiebreaker ordering: source priority varies by query type.
YouTube ranks first for how_to, Polymarket for prediction,
HN for concept queries, X for breaking news.
All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.
Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
- test_models: update xAI model expectations to grok-4-1-fast (matching
current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
fix import path to use sys.path.insert pattern (matching all other tests)
Mastodon-compatible API at truthsocial.com/api/v2/search.
Opt-in via TRUTHSOCIAL_TOKEN env var (bearer token from browser).
Silent when unconfigured. Full pipeline: search, parse, normalize,
score, dedupe, render across all 10 pipeline files.
27 new tests, 440 total passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix plugin.json hooks field from invalid array to empty object
- Add Claude Code plugin install above ClawHub badge
- Version bump to v2.9.5 with new features block (Bluesky, comparative
mode, ScrapeCreators X, per-project env, expanded tests)
- Add Bluesky to all source list references
- Document BSKY_HANDLE/BSKY_APP_PASSWORD env vars in install + optional sections
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
searchPosts endpoint now returns 403 for unauthenticated requests.
Add session auth via createSession, gate on BSKY_HANDLE + BSKY_APP_PASSWORD
env vars. When unconfigured, Bluesky is completely invisible (no error).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add COMPARISON query type for "X vs Y" research with 3 parallel passes.
Add Bluesky stats line and update all source list references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free, no-auth-required search via public.api.bsky.app.
Always-on like HN and Polymarket (no API key needed).
- New scripts/lib/bluesky.py: search + parse via AT Protocol
- BlueskyItem schema, normalization, scoring, deduplication
- Wired into orchestrator ThreadPoolExecutor with timeout config
- Rendering in compact, full, and JSON output modes
- 14 unit tests covering parsing, dates, relevance, edge cases
- --search=bluesky / --search=bsky for bluesky-only mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
One SCRAPECREATORS_API_KEY now covers Reddit, TikTok, Instagram, AND X.
Priority: Bird (free) > xAI API > ScrapeCreators (shared key).
New module scrapecreators_x.py follows the same pattern as tiktok.py.
Updated env.py source routing and last30days.py orchestrator dispatch.
Includes 20 unit tests.
Fixes#55.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove double quotes around $ARGUMENTS so argparse can parse flags
like --deep, --store separately instead of as part of the topic string.
Fixes#61.
- Add ~/.claude/plugins/marketplaces/last30days-skill to the path
discovery loop so marketplace installs can find scripts/.
Fixes#54.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add per-project configuration via .claude/last30days.env, discovered by
walking up from cwd. Uses the same .env format as the existing global
config — no new parsers or formats.
Priority (highest wins):
1. Environment variables
2. .claude/last30days.env (per-project)
3. ~/.config/last30days/.env (global)
Also adds file permission checking — warns to stderr if config files
are readable by other users (should be chmod 600).
Includes tests for discovery, precedence, source tracking, and
permission warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a lightweight hook that runs on session start to check if any
API keys are configured. Warns users if no config is found and
checks file permissions on existing config files.
Checks (in order): .claude/last30days.env, ~/.config/last30days/.env,
OPENAI_API_KEY env var, SCRAPECREATORS_API_KEY env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert all new tests from bare pytest style to unittest.TestCase
with sys.path.insert, matching the convention used by all existing
tests. Remove pyproject.toml and conftest.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add pytest infrastructure (pyproject.toml, conftest.py) and unit tests
for modules that previously had zero test coverage:
- test_schema_roundtrip.py: to_dict() serialization for all data classes
- test_reddit_enrich.py: URL parsing, thread data parsing, comment filtering
- test_reddit_sc.py: ScrapeCreators Reddit search (query expansion, subreddit discovery)
- test_instagram_sc.py: Instagram relevance scoring, tokenization, depth config
Includes fixtures/reddit_thread_sample.json for reddit_enrich tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update .claude-plugin/marketplace.json and plugin.json to v2.9.5 with
full metadata. Add plugin install instructions to README as the
recommended install method. The repo root serves as both the marketplace
and the plugin - skills/last30days/SKILL.md (symlink) is discovered
automatically.
Users can now install with:
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days@last30days-skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.
Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.
Closes#45
Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PR merges on March 7 (PR #48 Xiaohongshu, upstream merge) regressed
SKILL.md by re-introducing the "Save Research to Documents" section
that v2.9.4 removed. Those branches were forked before v2.9.4 and
brought the old content back via merge resolution.
Fixes: remove save section, restore --save-dir flag on bash command,
update agent mode line, add tool-call guard to STOP instruction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Explored adding scheduled re-runs and cumulative intelligence to
last30days. Concluded that Claude Code's session-scoped scheduling
(CronCreate/loop) can't support true perpetual monitoring since
jobs die when the terminal closes. Plan documents the architecture,
what exists, and why we're waiting for persistent background agents.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Jeffrey Sperling