perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)
1. N+1 queries in store.store_findings()
The old loop ran one SELECT per finding to check existence, then one
INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
dict, then executemany() handles all inserts and updates. Query count
stays constant regardless of batch size. Benchmark on 500 findings:
~30ms to ~20ms; gap widens on slower storage.
2. O(n^2) source_items dedup in fusion.weighted_rrf()
Merging an item into an existing candidate ran any(existing.source ==
... for existing in candidate.source_items), linearly scanning a list
that grew with each merge. At 40 candidates with 20 source_items each,
fusion went quadratic. Now tracks (source, item_id) tuples in a
per-candidate set for O(1) lookup. The source_items list itself is
unchanged since other code iterates it.
3. Dead code removal
- providers.GeminiClient.ground_search() and .url_context_json(): zero
callers. Deleted.
- render._top_comment_excerpt(): zero callers. Deleted.
- env.is_reddit_available(): one-line wrapper around get_reddit_source.
Callers can check get_reddit_source(config) is not None directly.
This commit is contained in:
@@ -93,13 +93,6 @@ class GeminiClient(ReasoningClient):
|
||||
)
|
||||
return extract_gemini_text(payload)
|
||||
|
||||
def ground_search(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
return self._generate_content(model, prompt, tools=[{"google_search": {}}])
|
||||
|
||||
def url_context_json(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
return self.generate_json(model, prompt, tools=[{"url_context": {}}])
|
||||
|
||||
|
||||
class OpenAIClient(ReasoningClient):
|
||||
name = "openai"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user