perf: optimize dedup, parallelize handle searches and enrichment
The dedup hot path recomputed normalize_text() 4 times per comparison and recomputed item_text() on every inner-loop iteration. Pre-computing n-gram sets and token sets into a _PreparedText cache cuts dedup time by 6x (2.16s to 0.39s on 300 unique items). Bird handle searches spawned one Node process per handle sequentially. Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern applied to YouTube comment enrichment (was serial, Reddit was already parallel) and the retry-thin-sources phase in the pipeline. Clustering now pre-computes candidate text and uses prepared_similarity for the O(n^2) grouping and MMR representative selection loops. Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses extend() instead of list concatenation. End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
This commit is contained in:
@@ -699,18 +699,27 @@ def enrich_with_comments(
|
||||
top_items = ranked[:max_videos]
|
||||
_log(f"Enriching comments for {len(top_items)} YouTube videos")
|
||||
|
||||
enriched_count = 0
|
||||
for item in top_items:
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
def _enrich_one(item: dict) -> bool:
|
||||
video_id = item.get("video_id", "")
|
||||
if not video_id:
|
||||
continue
|
||||
return False
|
||||
try:
|
||||
comments = _fetch_video_comments(video_id, token, max_comments)
|
||||
if comments:
|
||||
item["top_comments"] = comments
|
||||
enriched_count += 1
|
||||
return True
|
||||
except Exception as exc:
|
||||
_log(f"Comment enrichment failed for {video_id}: {exc}")
|
||||
return False
|
||||
|
||||
enriched_count = 0
|
||||
with ThreadPoolExecutor(max_workers=min(4, len(top_items))) as executor:
|
||||
futures = {executor.submit(_enrich_one, item): item for item in top_items}
|
||||
for future in as_completed(futures):
|
||||
if future.result():
|
||||
enriched_count += 1
|
||||
|
||||
_log(f"Enriched {enriched_count}/{len(top_items)} videos with comments")
|
||||
return items
|
||||
|
||||
Reference in New Issue
Block a user