review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public Reddit JSON API to recover thread body + top comments (the Claude Code WebFetch tool blocks reddit.com directly). That bypass is sound and the fixed problem is real - but the always-on shape ignores user intent on source gating. A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would still get Reddit content smuggled back in via web-search URLs that happen to point at reddit.com threads. This contradicts the suppression contract that EXCLUDE_SOURCES is supposed to provide (see lib/pipeline.available_sources where the same env var gates the top-level Reddit source). Add a _reddit_excluded(config) check in web_search() that mirrors the parsing pattern from lib/pipeline (comma-separated, case-insensitive, whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the enrichment pass entirely - the web results themselves still flow through, but they're not augmented with Reddit body/comments. Four new tests in test_grounding_v3.py cover: - EXCLUDE_SOURCES=reddit skips enrichment - case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv - Other sources in EXCLUDE_SOURCES don't trigger the gate - Enrichment runs normally when reddit isn't excluded 19/19 grounding tests pass.
This commit is contained in:
@@ -232,17 +232,32 @@ def web_search(
|
||||
raise ValueError(f"Unsupported web backend: {backend!r}")
|
||||
else:
|
||||
return [], {}
|
||||
if items:
|
||||
if items and not _reddit_excluded(config):
|
||||
items = _enrich_reddit_items(items)
|
||||
return items, artifact
|
||||
|
||||
|
||||
def _reddit_excluded(config: dict) -> bool:
|
||||
"""Return True when EXCLUDE_SOURCES contains 'reddit'.
|
||||
|
||||
Respects the same suppression knob the pipeline uses for source gating,
|
||||
so a user who set EXCLUDE_SOURCES=reddit doesn't get Reddit content
|
||||
smuggled back in via web-search URLs.
|
||||
"""
|
||||
raw = (config.get("EXCLUDE_SOURCES") or "").split(",")
|
||||
return any(s.strip().lower() == "reddit" for s in raw)
|
||||
|
||||
|
||||
def _enrich_reddit_items(items: list[dict]) -> list[dict]:
|
||||
"""Enrich web search results that are Reddit URLs with thread body and comments.
|
||||
|
||||
Claude Code's WebFetch blocks reddit.com, so the model can't retrieve
|
||||
Reddit content from web search results. This fetches it via the public
|
||||
JSON API (reddit.com/.../.json) which bypasses that restriction.
|
||||
|
||||
Callers should gate this with EXCLUDE_SOURCES=reddit handling (see
|
||||
`_reddit_excluded`) so a user who explicitly excluded Reddit doesn't
|
||||
get Reddit content via web-search URLs.
|
||||
"""
|
||||
from . import reddit_enrich
|
||||
|
||||
|
||||
Reference in New Issue
Block a user