When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}. This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available." Return None
instead when every engagement field is missing.
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).
Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.
Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.
Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.
End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
When Bird's JSON response is a raw array instead of an object,
json.loads returns a list. All callers use .get('items') which raises
AttributeError on lists. Wrap list responses in {"items": parsed} so
callers always receive a dict.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.
Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
- hackernews: use extract_core_subject instead of raw topic, add
points>5 filter and restrictSearchableAttributes=title to reduce
noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
- bird_x: parse_bird_response now accepts query param and computes
token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement
This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.
Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set
Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool
noise words to _extract_core_subject, and a last-chance retry that falls back
to the longest non-noise token when 2-word retry also returns 0 results.
cache.py and render.py env overrides were already on main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: fix YAML error in argument-hint
* add codex auth support to responses API
* Use gpt-5.1-codex-mini as default model for Codex auth
Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.
* Pass .env credentials to Bird Node subprocesses for X auth
On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.
- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection
---------
Co-authored-by: Justin Williams <jblwilliams@gmail.com>
When a topic is a person/brand (e.g. "Dor Brothers", "Jason Calacanis"),
the agent now resolves their X handle via WebSearch before running the
script, then passes --x-handle to search their posts unfiltered (no
topic keywords required). This finds posts the entity made without
mentioning their own name.
- SKILL.md + OpenClaw variant: Step 0.5 handle resolution instructions
- last30days.py: --x-handle CLI arg, passed through to _run_supplemental()
- bird_x.search_handles(): topic is now Optional[str] for unfiltered mode
- schema.py: resolved_x_handle field on Report
- render.py: show resolved handle in stats output
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Script was hanging indefinitely when API sources were slow or
unresponsive. Now enforces bounded execution:
- Global timeout watchdog (180s default, 90s --quick, 300s --deep)
- Per-source future.result() timeouts (60s/30s/90s by depth)
- Parallel Reddit enrichment capped at 15 items / 45s total
- Subprocess process-group isolation (os.setsid + killpg)
- atexit cleanup kills all tracked child processes
- --timeout=N flag for user override
Also fixes the UX gap where missing sources were silently skipped:
- Pre-flight diagnostic banner shows source status before research
- Source status footer in compact output shows used/skipped/why
- Actionable fix commands for each missing source
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Vendor Bird's Twitter GraphQL search client directly into /last30days,
eliminating the dependency on `npm install -g @steipete/bird`. X search
now works out of the box with just Node.js 22+ and browser cookies.
- Add vendored bird-search.mjs wrapper (search-only subset of Bird v0.8.0)
- Vendor @steipete/sweet-cookie for browser cookie extraction
- Update bird_x.py to call vendored Node.js module instead of `bird` binary
- Update README.md and SKILL.md for v2.1 (remove Bird CLI install steps)
- Include Bird's MIT LICENSE in vendor directory
The fallback chain is: vendored search -> xAI API key -> web-only mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs in _extract_core_subject():
1. Multi-word noise phrases ("what are", "how to") never matched
because code compared individual words against multi-word strings.
"what are people saying about DeepSeek R1" became "what are people
saying" — losing the entire topic.
2. Missing meta words — "prompt", "techniques", "tips" weren't
filtered (only "prompting" was). "vibe motion best prompt
techniques" kept 4 keywords instead of 2.
3. No retry on 0 results — Reddit retries with simplified queries
but X accepted 0 and moved on.
Fix: Two-phase extraction (strip multi-word prefixes/suffixes first,
then individual noise words), expanded noise set, max 3 words (was 4),
and automatic retry with first 2 words when Bird returns 0 results.
Before → After:
- "vibe motion best prompt techniques" → "vibe motion" (was 4 words, 0 results)
- "what are people saying about DeepSeek R1" → "deepseek r1" (was "what are people saying")
- "nano banana pro prompts for gemini" → "nano banana pro" (was 4 words)
Tested: vibe motion (12 X posts, was 0), DeepSeek R1 (12 posts),
kanye west (12 posts, no regression).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the initial broad search (Phase 1), extract key entities from results
and run targeted secondary searches to surface content the broad pass missed:
- New entity_extract.py: parses @handles, #hashtags, subreddits from results
- bird_x.py: search_handles() does targeted from:handle searches via Bird CLI
- openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint
- last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes
Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X).
Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Add _extract_core_subject() to strip noise words before X search
- X search is literal keyword matching, not semantic
- "best open claw usecases" → "open claw" (27 results vs 0)
2. Fix ISO date detection bug
- Was: if "T" in created_at (matched "Tue" in "Tue Feb 03...")
- Now: if created_at[10] == "T" (proper ISO separator check)
- Dates now parse correctly: (2026-02-03) instead of (date unknown)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1. last30days.py: Check isinstance(dict) before .get() - Bird returns
list on success, dict on error
2. bird_x.py: Update field mappings for Bird's actual response format:
- author.username not user.screen_name
- createdAt not created_at (camelCase)
- likeCount, retweetCount, etc. (camelCase)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Bird CLI doesn't support --since as a CLI flag.
Use X's search syntax: 'topic since:YYYY-MM-DD'
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- is_bird_installed() checks if bird command is in PATH
- is_bird_authenticated() runs bird whoami to verify auth
- check_npm_available() checks if npm can install bird
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Matt Van Horn