Add a pytest-discovered tests/conftest.py for the last30days scripts path and
remove duplicate per-file sys.path.insert boilerplate from tests.
Normalize affected imports to rely on the shared scripts path and remove the
now-unneeded E402 suppressions.
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.
This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.
The resulting source-gating model is three-tier and intentional:
• **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
grounding, **tiktok, instagram, threads**
• **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
perplexity (different paid API — OpenRouter),
tiktok_comments / youtube_comments (N× extra SC calls per video)
• **--search per-query opt-in** (relevance reasons):
pinterest (visual pins, narrow utility),
xiaohongshu (Chinese-market specific)
Changes:
- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
the new contract and includes a regression guard ("INCLUDE_SOURCES
should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
runs. Drops false INCLUDE_SOURCES requirement for
tiktok/instagram/threads; corrects pinterest to mention --search;
adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
section.
Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
bird_x.py and youtube_yt.py had four near-identical copies of the same
subprocess cleanup dance (Popen + os.setsid + communicate(timeout) +
SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to
lib.subproc.run_with_timeout(), which:
- runs the child in its own process group via os.setsid where available
- raises SubprocTimeout on timeout
- on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s
- accepts an on_pid callback so bird_x can still register child PIDs
with last30days.register_child_pid for whole-process cleanup
- captures stdout/stderr as strings in a SubprocResult dataclass
Migrated call sites: _run_bird_search, search_handles inner worker,
search_youtube, fetch_transcript. With the helper in place, the signal
and subprocess imports became dead in both files (plus os in
youtube_yt) and went with them.
Tests: 9 new subproc tests cover success, non-zero exit, stderr capture,
timeout-raises, timeout-kills-group, missing-command, env passthrough,
PID callback, and callback-exception suppression. test_env_v3 and
test_youtube_yt patch subproc.run_with_timeout instead of the removed
bird_x.subprocess and yt-dlp subprocess.
Yong-yuan-X