perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)

1. N+1 queries in store.store_findings()
   The old loop ran one SELECT per finding to check existence, then one
   INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
   Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
   dict, then executemany() handles all inserts and updates. Query count
   stays constant regardless of batch size. Benchmark on 500 findings:
   ~30ms to ~20ms; gap widens on slower storage.

2. O(n^2) source_items dedup in fusion.weighted_rrf()
   Merging an item into an existing candidate ran any(existing.source ==
   ... for existing in candidate.source_items), linearly scanning a list
   that grew with each merge. At 40 candidates with 20 source_items each,
   fusion went quadratic. Now tracks (source, item_id) tuples in a
   per-candidate set for O(1) lookup. The source_items list itself is
   unchanged since other code iterates it.

3. Dead code removal
   - providers.GeminiClient.ground_search() and .url_context_json(): zero
     callers. Deleted.
   - render._top_comment_excerpt(): zero callers. Deleted.
   - env.is_reddit_available(): one-line wrapper around get_reddit_source.
     Callers can check get_reddit_source(config) is not None directly.
This commit is contained in:
Ilia Alshanetsky
2026-04-25 17:17:17 -04:00
committed by GitHub
parent e6b89f2644
commit 2acbf8a869
5 changed files with 76 additions and 74 deletions
-10
View File
@@ -1505,16 +1505,6 @@ def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score
return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]
def _top_comment_excerpt(item: schema.SourceItem | None) -> str | None:
if not item:
return None
comments = item.metadata.get("top_comments") or []
if not comments or not isinstance(comments[0], dict):
return None
top = comments[0]
return str(top.get("excerpt") or top.get("text") or "").strip() or None
def _comment_insight(item: schema.SourceItem | None) -> str | None:
if not item:
return None