refactor(github): split search_github / parse_github_response / enrich_with_comments

search_github returned a normalized List[dict] directly while every
other adapter follows search_X -> dict envelope, parse_X_response ->
list[dict]. The github branch in pipeline._retrieve_stream was the
only one that called search_* and returned (result, {}) without a
parse step. This blocked fixture-driven testing: there was no parse
function to feed a synthetic envelope to.

Split into three:

  search_github(...) -> Dict[str, Any]
    HTTP fetch only. Returns {"items": [raw items], "context": {core,
    from_date, to_date, count}}.

  parse_github_response(response) -> List[Dict[str, Any]]
    Pure function. Normalizes, date-filters, sorts by relevance.

  enrich_with_comments(items, depth, token) -> List[Dict[str, Any]]
    Public extraction of the old private _enrich_top_items. Resolves
    the token via env / gh CLI fallback so callers don't have to.

Pipeline now does the standard 3-call dance:

  response = github.search_github(...)
  items = github.parse_github_response(response)
  items = github.enrich_with_comments(items, depth=depth, token=token)

Keeping enrich_with_comments in parse_github_response would make parse
impure and force every fixture-driven test to either mock HTTP or
skip enrichment. Splitting it out matches the YouTube adapter's
pattern.
This commit is contained in:
Ilia Alshanetsky
2026-05-19 12:18:47 -04:00
parent 850c7e0185
commit 269dda9f6c
3 changed files with 190 additions and 32 deletions
+5 -2
View File
@@ -1007,8 +1007,11 @@ def _retrieve_stream(
result = polymarket.search_polymarket(subquery.search_query, from_date, to_date, depth=depth)
return polymarket.parse_polymarket_response(result, topic=subquery.search_query), {}
if source == "github":
result = github.search_github(subquery.search_query, from_date, to_date, depth=depth, token=config.get("GITHUB_TOKEN"))
return result, {}
token = config.get("GITHUB_TOKEN")
response = github.search_github(subquery.search_query, from_date, to_date, depth=depth, token=token)
items = github.parse_github_response(response)
items = github.enrich_with_comments(items, depth=depth, token=token)
return items, {}
if source == "pinterest":
result = pinterest.search_pinterest(
subquery.search_query, from_date, to_date,