fix(sources): align SC source gating between code and docs
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) — docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in that the code didn't actually enforce, and threads was inconsistently gated relative to its same-key siblings. This commit picks the "code as source of truth + EXCLUDE_SOURCES as suppression knob" model and aligns docs to match. It also promotes threads to the same auto-on tier as tiktok and instagram, since all three share the SC key and per-call cost shape — there was no real product reason for threads being opt-in while the other two weren't. The resulting source-gating model is three-tier and intentional: • **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES): reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial, grounding, **tiktok, instagram, threads** • **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons): perplexity (different paid API — OpenRouter), tiktok_comments / youtube_comments (N× extra SC calls per video) • **--search per-query opt-in** (relevance reasons): pinterest (visual pins, narrow utility), xiaohongshu (Chinese-market specific) Changes: - env.py: `is_threads_available()` drops the INCLUDE_SOURCES check, now mirrors tiktok/instagram (SC key → True). Docstring updated. - tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in the new contract and includes a regression guard ("INCLUDE_SOURCES should not be needed"). - SKILL.md: lines 333-338 rewritten so the model's "Build ACTIVE_SOURCES_LIST" checklist reflects what the engine actually runs. Drops false INCLUDE_SOURCES requirement for tiktok/instagram/threads; corrects pinterest to mention --search; adds missing INCLUDE_SOURCES=perplexity requirement. - README: same alignment for the user-facing "Everything else in v3" section. Note on EXCLUDE_SOURCES references in the new docs: the suppression flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES as the opt-out path; that prose is forward-looking until #399 lands. The behavior changes in this PR (threads auto-on) are self-contained and don't require #399 to function — but for users who want to suppress the newly-auto-on threads source, #399 needs to land first.
This commit is contained in:
@@ -152,8 +152,10 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
|
||||
|
||||
- **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
|
||||
- **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
|
||||
- **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does.
|
||||
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock.
|
||||
- **TikTok, Instagram, Threads.** All three activate automatically once `SCRAPECREATORS_API_KEY` is set — same key, same per-call cost. Suppress any of them with `EXCLUDE_SOURCES=tiktok,instagram,threads` (any comma-separated subset).
|
||||
- **Pinterest.** Per-query opt-in (visual pins, narrow utility): the model passes `--search=pinterest` for the runs that need it. Requires `SCRAPECREATORS_API_KEY`.
|
||||
- **YouTube + TikTok comments.** Persistent opt-in via `INCLUDE_SOURCES=youtube_comments,tiktok_comments` because each video pulls N extra ScrapeCreators calls on top of the base search. Surface top comments with vote counts the same way Reddit does.
|
||||
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` and `INCLUDE_SOURCES=perplexity` (it's a separate paid API — opt-in keeps you from being surprise-billed).
|
||||
- **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
|
||||
- **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
|
||||
- **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section.
|
||||
|
||||
Reference in New Issue
Block a user