fix(sources): unblock SC YouTube + multi-token HN searches

Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.

YouTube (ScrapeCreators)
  SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
    {"error":"missing_parameter","message":"You must provide a query"}
  The canonical SC parameter for that endpoint is `query`. Other SC
  endpoints we use (Reddit, TikTok, Instagram) happened to work because
  they use their own per-endpoint parameter names — YouTube was the lone
  outlier.

Hacker News (Algolia)
  Multi-keyword theme queries returned zero hits across every theme.
  Algolia treats query= as strict AND across tokens, so a 4-5 word query
  like "claude, personal agents, agentic infra" matches no stories.

  Three changes in hackernews.py:

  1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
     search_hackernews and _title_matches_query normalize the query the
     same way — addresses Greptile P2 #2 about the two callsites needing
     to stay in sync.
  2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
     by token-overlap instead of requiring every token.
  3. Relax _title_matches_query from all-words to any-word, *but match
     on word boundaries (\b<word>\b) rather than naive substring* —
     addresses Greptile P2 #1, which flagged that the previous any-word
     relaxation would let "ai" falsely match "email" or "rail".

  Token-overlap relevance scoring at parse time already demotes weak
  matches, so word-boundary any-word matching is safe.

Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.

Co-authored-by: Trevin Chow <trevin@trevinchow.com>
This commit is contained in:
Brad Ferguson
2026-05-16 19:37:16 -07:00
committed by Trevin Chow
parent 2e39ee8ce4
commit edea402b7c
3 changed files with 94 additions and 22 deletions
+36 -4
View File
@@ -160,12 +160,44 @@ def test_title_matches_query_empty_query():
def test_title_matches_query_partial_match():
"""Test that all query words must match."""
"""Any-word matching: at least one query token in title is enough.
Previously required *all* tokens, which killed every hit on multi-keyword
theme queries like 'claude, personal agents, agentic infra' since no real
HN title contains all 5 tokens verbatim. Token-overlap relevance at parse
time still demotes weak matches, so the loosened gate is safe.
"""
title = "New AI framework"
query = "AI blockchain"
# "blockchain" is not in title, so should fail
assert hackernews._title_matches_query(title, query) is False
# "AI" matches as a whole word, even though "blockchain" doesn't appear
assert hackernews._title_matches_query(title, query) is True
def test_title_matches_query_no_token_in_title():
"""If no query token appears in the title at all, reject."""
assert hackernews._title_matches_query("New rust compiler", "AI blockchain") is False
def test_title_matches_query_word_boundary_not_substring():
"""Short tokens must match on word boundaries, not as substrings.
Without word-boundary matching, 'ai' would falsely match 'email',
'rail', 'artists', etc.
"""
# 'ai' as a substring of 'email' must not match
assert hackernews._title_matches_query("New email service", "ai blockchain") is False
# 'ai' as a whole word does match
assert hackernews._title_matches_query("Cool AI tool launched", "ai blockchain") is True
def test_title_matches_query_flattens_hyphens_and_commas():
"""Query tokens split on hyphens/commas the same way search_hackernews
flattens them, so the post-filter stays aligned with what Algolia saw."""
# query 'ts-bun-node' flattens to ['ts', 'bun', 'node']; title contains 'bun'
assert hackernews._title_matches_query("Bun 1.2 released", "ts-bun-node") is True
# query 'rust, go, zig' flattens; title contains 'go'
assert hackernews._title_matches_query("Go 1.24 generics update", "rust, go, zig") is True
# === Tests for search_hackernews() ===