feat: Add YouTube as 4th research source via yt-dlp
YouTube search and transcript extraction runs automatically when yt-dlp is installed. Searches for topic videos from the last N days, fetches auto-generated transcripts for top results, and feeds them through the same scoring pipeline (relevance + recency + engagement) as Reddit/X. New files: - youtube_yt.py: search, transcript extraction, VTT cleanup Modified files: - schema.py: YouTubeItem dataclass, updated Report - normalize.py: normalize_youtube_items() - score.py: YouTube engagement scoring (views-dominated) - dedupe.py: YouTube deduplication - render.py: YouTube section in compact output - env.py: is_ytdlp_available() check - ui.py: YouTube progress messages - last30days.py: _search_youtube(), parallel execution with Reddit/X - SKILL.md: YouTube in stats box, citation priority - README.md: YouTube docs, yt-dlp requirement, Peter shoutout Inspired by Peter Steinberger's yt-dlp + summarize toolchain approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+11
-1
@@ -36,10 +36,12 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
|
||||
return intersection / union if union > 0 else 0.0
|
||||
|
||||
|
||||
def get_item_text(item: Union[schema.RedditItem, schema.XItem]) -> str:
|
||||
def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem]) -> str:
|
||||
"""Get comparable text from an item."""
|
||||
if isinstance(item, schema.RedditItem):
|
||||
return item.title
|
||||
elif isinstance(item, schema.YouTubeItem):
|
||||
return f"{item.title} {item.channel_name}"
|
||||
else:
|
||||
return item.text
|
||||
|
||||
@@ -118,3 +120,11 @@ def dedupe_x(
|
||||
) -> List[schema.XItem]:
|
||||
"""Dedupe X items."""
|
||||
return dedupe_items(items, threshold)
|
||||
|
||||
|
||||
def dedupe_youtube(
|
||||
items: List[schema.YouTubeItem],
|
||||
threshold: float = 0.7,
|
||||
) -> List[schema.YouTubeItem]:
|
||||
"""Dedupe YouTube items."""
|
||||
return dedupe_items(items, threshold)
|
||||
|
||||
Reference in New Issue
Block a user