perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)
1. N+1 queries in store.store_findings()
The old loop ran one SELECT per finding to check existence, then one
INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
dict, then executemany() handles all inserts and updates. Query count
stays constant regardless of batch size. Benchmark on 500 findings:
~30ms to ~20ms; gap widens on slower storage.
2. O(n^2) source_items dedup in fusion.weighted_rrf()
Merging an item into an existing candidate ran any(existing.source ==
... for existing in candidate.source_items), linearly scanning a list
that grew with each merge. At 40 candidates with 20 source_items each,
fusion went quadratic. Now tracks (source, item_id) tuples in a
per-candidate set for O(1) lookup. The source_items list itself is
unchanged since other code iterates it.
3. Dead code removal
- providers.GeminiClient.ground_search() and .url_context_json(): zero
callers. Deleted.
- render._top_comment_excerpt(): zero callers. Deleted.
- env.is_reddit_available(): one-line wrapper around get_reddit_source.
Callers can check get_reddit_source(config) is not None directly.
This commit is contained in:
@@ -1505,16 +1505,6 @@ def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score
|
||||
return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]
|
||||
|
||||
|
||||
def _top_comment_excerpt(item: schema.SourceItem | None) -> str | None:
|
||||
if not item:
|
||||
return None
|
||||
comments = item.metadata.get("top_comments") or []
|
||||
if not comments or not isinstance(comments[0], dict):
|
||||
return None
|
||||
top = comments[0]
|
||||
return str(top.get("excerpt") or top.get("text") or "").strip() or None
|
||||
|
||||
|
||||
def _comment_insight(item: schema.SourceItem | None) -> str | None:
|
||||
if not item:
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user