perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)

1. N+1 queries in store.store_findings()
   The old loop ran one SELECT per finding to check existence, then one
   INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
   Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
   dict, then executemany() handles all inserts and updates. Query count
   stays constant regardless of batch size. Benchmark on 500 findings:
   ~30ms to ~20ms; gap widens on slower storage.

2. O(n^2) source_items dedup in fusion.weighted_rrf()
   Merging an item into an existing candidate ran any(existing.source ==
   ... for existing in candidate.source_items), linearly scanning a list
   that grew with each merge. At 40 candidates with 20 source_items each,
   fusion went quadratic. Now tracks (source, item_id) tuples in a
   per-candidate set for O(1) lookup. The source_items list itself is
   unchanged since other code iterates it.

3. Dead code removal
   - providers.GeminiClient.ground_search() and .url_context_json(): zero
     callers. Deleted.
   - render._top_comment_excerpt(): zero callers. Deleted.
   - env.is_reddit_available(): one-line wrapper around get_reddit_source.
     Callers can check get_reddit_source(config) is not None directly.
This commit is contained in:
Ilia Alshanetsky
2026-04-25 17:17:17 -04:00
committed by GitHub
parent e6b89f2644
commit 2acbf8a869
5 changed files with 76 additions and 74 deletions
+6 -1
View File
@@ -116,6 +116,8 @@ def weighted_rrf(
"""Fuse ranked lists into a single candidate pool."""
subqueries = {subquery.label: subquery for subquery in plan.subqueries}
candidates: dict[str, schema.Candidate] = {}
# Track (source, item_id) pairs already attached to each candidate for O(1) dedup.
seen_source_items: dict[str, set[tuple[str, str]]] = {}
for (label, source), items in streams.items():
subquery = subqueries[label]
@@ -154,6 +156,7 @@ def weighted_rrf(
]
},
)
seen_source_items[key] = {(item.source, item.item_id)}
continue
candidate = candidates[key]
@@ -179,7 +182,9 @@ def weighted_rrf(
candidate.subquery_labels.append(label)
if item.source not in candidate.sources:
candidate.sources.append(item.source)
if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items):
source_item_key = (item.source, item.item_id)
if source_item_key not in seen_source_items[key]:
seen_source_items[key].add(source_item_key)
candidate.source_items.append(item)
candidate.metadata.setdefault("provenance", []).append(
{