fix(sources): unblock SC YouTube + multi-token HN searches
Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.
YouTube (ScrapeCreators)
SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
{"error":"missing_parameter","message":"You must provide a query"}
The canonical SC parameter for that endpoint is `query`. Other SC
endpoints we use (Reddit, TikTok, Instagram) happened to work because
they use their own per-endpoint parameter names — YouTube was the lone
outlier.
Hacker News (Algolia)
Multi-keyword theme queries returned zero hits across every theme.
Algolia treats query= as strict AND across tokens, so a 4-5 word query
like "claude, personal agents, agentic infra" matches no stories.
Three changes in hackernews.py:
1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
search_hackernews and _title_matches_query normalize the query the
same way — addresses Greptile P2 #2 about the two callsites needing
to stay in sync.
2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
by token-overlap instead of requiring every token.
3. Relax _title_matches_query from all-words to any-word, *but match
on word boundaries (\b<word>\b) rather than naive substring* —
addresses Greptile P2 #1, which flagged that the previous any-word
relaxation would let "ai" falsely match "email" or "rail".
Token-overlap relevance scoring at parse time already demotes weak
matches, so word-boundary any-word matching is safe.
Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.
Co-authored-by: Trevin Chow <trevin@trevinchow.com>
This commit is contained in:
committed by
Trevin Chow
parent
2e39ee8ce4
commit
edea402b7c
@@ -160,12 +160,44 @@ def test_title_matches_query_empty_query():
|
||||
|
||||
|
||||
def test_title_matches_query_partial_match():
|
||||
"""Test that all query words must match."""
|
||||
"""Any-word matching: at least one query token in title is enough.
|
||||
|
||||
Previously required *all* tokens, which killed every hit on multi-keyword
|
||||
theme queries like 'claude, personal agents, agentic infra' since no real
|
||||
HN title contains all 5 tokens verbatim. Token-overlap relevance at parse
|
||||
time still demotes weak matches, so the loosened gate is safe.
|
||||
"""
|
||||
title = "New AI framework"
|
||||
query = "AI blockchain"
|
||||
|
||||
# "blockchain" is not in title, so should fail
|
||||
assert hackernews._title_matches_query(title, query) is False
|
||||
|
||||
# "AI" matches as a whole word, even though "blockchain" doesn't appear
|
||||
assert hackernews._title_matches_query(title, query) is True
|
||||
|
||||
|
||||
def test_title_matches_query_no_token_in_title():
|
||||
"""If no query token appears in the title at all, reject."""
|
||||
assert hackernews._title_matches_query("New rust compiler", "AI blockchain") is False
|
||||
|
||||
|
||||
def test_title_matches_query_word_boundary_not_substring():
|
||||
"""Short tokens must match on word boundaries, not as substrings.
|
||||
|
||||
Without word-boundary matching, 'ai' would falsely match 'email',
|
||||
'rail', 'artists', etc.
|
||||
"""
|
||||
# 'ai' as a substring of 'email' must not match
|
||||
assert hackernews._title_matches_query("New email service", "ai blockchain") is False
|
||||
# 'ai' as a whole word does match
|
||||
assert hackernews._title_matches_query("Cool AI tool launched", "ai blockchain") is True
|
||||
|
||||
|
||||
def test_title_matches_query_flattens_hyphens_and_commas():
|
||||
"""Query tokens split on hyphens/commas the same way search_hackernews
|
||||
flattens them, so the post-filter stays aligned with what Algolia saw."""
|
||||
# query 'ts-bun-node' flattens to ['ts', 'bun', 'node']; title contains 'bun'
|
||||
assert hackernews._title_matches_query("Bun 1.2 released", "ts-bun-node") is True
|
||||
# query 'rust, go, zig' flattens; title contains 'go'
|
||||
assert hackernews._title_matches_query("Go 1.24 generics update", "rust, go, zig") is True
|
||||
|
||||
|
||||
# === Tests for search_hackernews() ===
|
||||
|
||||
Reference in New Issue
Block a user