Add a pytest-discovered tests/conftest.py for the last30days scripts path and
remove duplicate per-file sys.path.insert boilerplate from tests.
Normalize affected imports to rely on the shared scripts path and remove the
now-unneeded E402 suppressions.
Twitter's edge intermittently serves an HTML anti-bot interstitial in
place of JSON when the bird-search subprocess hits a per-query rate
limit. Before this fix, that response made json.loads raise
JSONDecodeError and _run_bird_search() returned {"error": ..., "items":
[]} with the parsed exception message — silent-empty against an
orchestrator that has no way to distinguish "Twitter served HTML; retry
likely succeeds" from "no tweets matched the query."
Surfaced during a community-signal pass where a Karpathy-LLM-wiki
subquery returned zero X items, while a second identical run a few
seconds later returned full results.
Fix:
- Extract the subprocess invocation into _invoke_bird_subprocess() so
the retry loop can call it multiple times cleanly. Returns
(result, terminal_error) — terminal_error is non-None for
unrecoverable cases (subprocess timeout, spawn failure) that should
NOT be retried.
- In _run_bird_search(), wrap the json.loads parse in a retry loop
bounded by MAX_JSON_DECODE_RETRIES (=2) with JSON_DECODE_RETRY_DELAY
(=5s) between attempts.
- On non-JSON stdout, log a diagnostic that names the shape
(`looks_html`, first-80-chars stdout preview, attempt counter) so
silent-empty failures become legible in logs.
- On retry exhaustion, return an error dict whose message explicitly
names "anti-bot interstitial" as the likely cause, distinguishing
this failure from a genuine no-results case.
Subprocess timeout, spawn failure, and non-zero return-code paths are
unchanged — those are terminal and don't retry.
Tests:
- Verifies HTML-then-JSON returns success on attempt 2.
- Verifies all-HTML returns the diagnostic error dict mentioning the
anti-bot interstitial cause.
- Verifies subprocess timeout is NOT retried.
All 12 bird_x tests pass (9 baseline + 3 new).
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}. This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available." Return None
instead when every engagement field is missing.
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.
Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
Cherry-picked from PR #24 (el-analista). Adds trending/viral/plugin/skill/tool
noise words to _extract_core_subject, and a last-chance retry that falls back
to the longest non-noise token when 2-word retry also returns 0 results.
cache.py and render.py env overrides were already on main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Yong-yuan-X