feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES config key. Two coupled changes: 1. pipeline.available_sources() filters out any source listed in config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive, whitespace-tolerant) before returning. 2. hooks/scripts/check-config.sh "Ready — N sources active" banner subtracts excluded sources from the ScrapeCreators +3 (Reddit comments + TikTok + Instagram) so the count matches what the pipeline actually runs. Use case: skip TikTok/Instagram on runs where you only want text-substantive sources, without unsetting SCRAPECREATORS_API_KEY (which would also kill Reddit comments). The existing INCLUDE_SOURCES allowlist covers Perplexity opt-in but doesn't cover this denylist case — tiktok and instagram are added unconditionally when SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key. Tests (tests/test_pipeline_v3.py::TestExcludeSources): - excludes tiktok+instagram when listed - no exclusion when env unset or empty string - case-insensitive + whitespace-tolerant parsing - works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
This commit is contained in:
@@ -97,7 +97,16 @@ if [[ -n "$HAS_BSKY" ]]; then
|
||||
SOURCE_COUNT=$((SOURCE_COUNT + 1))
|
||||
fi
|
||||
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
|
||||
SOURCE_COUNT=$((SOURCE_COUNT + 3)) # Reddit comments + TikTok + Instagram
|
||||
# Start with Reddit comments + TikTok + Instagram, subtract any in EXCLUDE_SOURCES
|
||||
SC_ADD=3
|
||||
EXCLUDED="${ENV_EXCLUDE_SOURCES:-${EXCLUDE_SOURCES:-}}"
|
||||
if [[ ",$EXCLUDED," == *",tiktok,"* ]]; then
|
||||
SC_ADD=$((SC_ADD - 1))
|
||||
fi
|
||||
if [[ ",$EXCLUDED," == *",instagram,"* ]]; then
|
||||
SC_ADD=$((SC_ADD - 1))
|
||||
fi
|
||||
SOURCE_COUNT=$((SOURCE_COUNT + SC_ADD))
|
||||
fi
|
||||
|
||||
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
|
||||
|
||||
Reference in New Issue
Block a user