1. N+1 queries in store.store_findings()
The old loop ran one SELECT per finding to check existence, then one
INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
dict, then executemany() handles all inserts and updates. Query count
stays constant regardless of batch size. Benchmark on 500 findings:
~30ms to ~20ms; gap widens on slower storage.
2. O(n^2) source_items dedup in fusion.weighted_rrf()
Merging an item into an existing candidate ran any(existing.source ==
... for existing in candidate.source_items), linearly scanning a list
that grew with each merge. At 40 candidates with 20 source_items each,
fusion went quadratic. Now tracks (source, item_id) tuples in a
per-candidate set for O(1) lookup. The source_items list itself is
unchanged since other code iterates it.
3. Dead code removal
- providers.GeminiClient.ground_search() and .url_context_json(): zero
callers. Deleted.
- render._top_comment_excerpt(): zero callers. Deleted.
- env.is_reddit_available(): one-line wrapper around get_reddit_source.
Callers can check get_reddit_source(config) is not None directly.
Scoring hot path (_normalize_score_dedupe) re-tokenized the same
ranking_query ~240x per stream: once per item for local_relevance,
plus ~5x per item across snippet windows. Query tokens are immutable
within a stream, so compute them once as relevance.PreparedQuery and
thread through signals.annotate_stream and snippet.extract_best_snippet.
dedupe._PreparedText called normalize_text twice: once in __init__ and
again via get_ngrams. Factor out _ngrams_of_normalized so the prepared
path skips the redundant pass while get_ngrams keeps its public contract.
Behavior unchanged.
_normalize_reddit, _normalize_hackernews, and _normalize_github inlined
the same 5-line comprehension to stringify and space-join the first 3
top_comments' excerpt field. Extract one helper, call it from all three.
The comment field name varies per source (Reddit/GitHub use 'excerpt',
HN uses 'text'), so it's passed as a parameter. Behavior unchanged.
Ilia Alshanetsky