feat(quality): YouTube relevance scoring and cross-source linking
YouTube videos now get real relevance scores based on token overlap between the search query and video title (was hardcoded at 0.7). Uses ratio overlap with stopword removal, floored at 0.1. Cross-source linking annotates items that discuss the same story across different platforms (e.g., Reddit + HN + X). Items get bidirectional cross_refs displayed as [xref: R3, HN5] in compact output so Claude can triangulate multi-platform coverage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -0,0 +1,226 @@
|
||||
---
|
||||
title: "feat: YouTube relevance scoring and cross-source linking"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-02-25
|
||||
---
|
||||
|
||||
# feat: YouTube Relevance Scoring and Cross-Source Linking
|
||||
|
||||
## Overview
|
||||
|
||||
Two output quality improvements for the last30days monthiversary:
|
||||
|
||||
1. **YouTube relevance scoring** - Replace the hardcoded `relevance: 0.7` with real token-overlap scoring so relevant niche videos beat viral off-topic ones.
|
||||
2. **Cross-source linking** - When the same story appears on Reddit + HN + X, annotate items with `[xref: R3, HN5]` so Claude can synthesize cross-platform discussion.
|
||||
|
||||
## Problem Statement / Motivation
|
||||
|
||||
**YouTube scoring is broken.** Every YouTube video starts with `relevance: 0.7` (70/100 subscore). Since the scoring formula is `45% relevance + 25% recency + 30% engagement`, all YouTube items share the same 31.5pt relevance floor. Ranking is purely engagement-driven - a viral off-topic video beats a niche relevant one.
|
||||
|
||||
**Cross-source coverage is invisible.** The same story often appears across Reddit, HN, and X (e.g., a product launch). Currently each source is deduped independently, but there's no signal telling Claude "these items are about the same thing." Claude has to manually notice the overlap, and often doesn't.
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Feature 1: YouTube Relevance Scoring
|
||||
|
||||
**Where:** `youtube_yt.py` (compute), `normalize.py` (pass through - already works)
|
||||
|
||||
**Algorithm:** Token ratio overlap between `core_topic` and video title.
|
||||
|
||||
```python
|
||||
# youtube_yt.py - new function
|
||||
STOPWORDS = {'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
|
||||
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
|
||||
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can'}
|
||||
|
||||
def _compute_relevance(query: str, title: str) -> float:
|
||||
"""Compute relevance as ratio of query tokens found in title.
|
||||
|
||||
Uses ratio overlap (intersection / query_length) so short queries
|
||||
score higher when fully represented in the title. Floors at 0.1.
|
||||
"""
|
||||
q_tokens = {w for w in query.lower().split() if w not in STOPWORDS and len(w) > 1}
|
||||
t_tokens = {w for w in re.sub(r'[^\w\s]', ' ', title.lower()).split()
|
||||
if w not in STOPWORDS and len(w) > 1}
|
||||
|
||||
if not q_tokens:
|
||||
return 0.5 # Neutral fallback
|
||||
|
||||
overlap = len(q_tokens & t_tokens)
|
||||
ratio = overlap / len(q_tokens)
|
||||
return max(0.1, min(1.0, ratio))
|
||||
```
|
||||
|
||||
**Key decisions:**
|
||||
- **Use `core_topic`** (already stripped of noise words by `_extract_core_subject()`) - not the raw verbose query
|
||||
- **Ratio overlap** (intersection/query_length), not strict Jaccard - so "Claude Code" (2 tokens) vs "Claude Code Tutorial" (3 tokens) = 2/2 = 1.0, not 2/3 = 0.67
|
||||
- **Stopword removal** on both sides - prevents "How To Use Claude Code For Beginners" from diluting the match
|
||||
- **Floor at 0.1** - even zero-match videos get `rel_score=10` not 0, consistent with other sources' defaults
|
||||
- **No LLM call** - this is pure string matching, zero latency/cost
|
||||
|
||||
**Files to modify:**
|
||||
|
||||
#### `scripts/lib/youtube_yt.py`
|
||||
|
||||
- [x] Add `STOPWORDS` set and `_compute_relevance(query, title)` function
|
||||
- [x] In `search_youtube()`, replace `"relevance": 0.7` (line 186) with `"relevance": _compute_relevance(core_topic, video.get("title", ""))`
|
||||
- [x] Update `"why_relevant"` to be more descriptive: `f"YouTube: {title_excerpt}"`
|
||||
|
||||
#### `scripts/lib/normalize.py`
|
||||
|
||||
- No changes needed - already passes through `item.get("relevance", 0.7)` at line 196
|
||||
|
||||
#### `scripts/lib/score.py`
|
||||
|
||||
- No changes needed - `score_youtube_items()` already reads `item.relevance` correctly
|
||||
|
||||
### Feature 2: Cross-Source Linking
|
||||
|
||||
**Where:** `dedupe.py` (new function), `schema.py` (new field), `render.py` (display), `last30days.py` (call site)
|
||||
|
||||
**Algorithm:** Reuse existing Jaccard char-trigram similarity from `dedupe.py`. Compare items across sources at threshold 0.5 (lower than within-source 0.7 since titles differ more across platforms). Bidirectional - both items get the cross-ref.
|
||||
|
||||
**Key decisions:**
|
||||
- **Link, don't merge** - keep items separate with `cross_refs: ["R3", "HN5"]`. Let Claude handle the narrative.
|
||||
- **Bidirectional** - if R3 links to HN5, HN5 also links to R3. Claude reads source-by-source and needs to see the link from either direction.
|
||||
- **Threshold 0.5** for cross-source (vs 0.7 within-source). Lower because the same story has different titles across platforms. Still high enough to avoid false positives.
|
||||
- **Truncate X text to 100 chars** for comparison - full tweets dilute Jaccard against short Reddit/HN titles
|
||||
- **Just IDs in cross_refs** - no similarity scores (noise for Claude's synthesis)
|
||||
- **Include WebSearchItem** - add to `get_item_text()` using `item.title`
|
||||
- **Don't persist to SQLite** - cross_refs are cheap to recompute, not worth a schema migration
|
||||
|
||||
**Performance:** With 45 items total (typical) and 10 source pairs, ~1000 char-trigram comparisons. Sub-millisecond. Even --deep mode (~150 items) is <12K comparisons - trivial.
|
||||
|
||||
**Files to modify:**
|
||||
|
||||
#### `scripts/lib/schema.py`
|
||||
|
||||
- [x] Add `cross_refs: List[str] = field(default_factory=list)` to all 5 item types (RedditItem, XItem, YouTubeItem, HackerNewsItem, WebSearchItem)
|
||||
- [x] Update `to_dict()` on each item type to include `cross_refs` (only when non-empty)
|
||||
- [x] Update `Report.from_dict()` to deserialize `cross_refs` for each item type
|
||||
|
||||
#### `scripts/lib/dedupe.py`
|
||||
|
||||
- [x] Update `get_item_text()` type hints to include `WebSearchItem` and handle it (use `item.title`)
|
||||
- [x] Add `get_cross_source_text()` function - same as `get_item_text()` but truncates X text to 100 chars
|
||||
- [x] Add `cross_source_link()` function:
|
||||
|
||||
```python
|
||||
def cross_source_link(
|
||||
*source_lists: List[Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
|
||||
schema.HackerNewsItem, schema.WebSearchItem]],
|
||||
threshold: float = 0.5,
|
||||
) -> None:
|
||||
"""Annotate items with cross-source references.
|
||||
|
||||
Compares items across different source types. When similarity exceeds
|
||||
threshold, adds bidirectional cross_refs with the related item's ID.
|
||||
Modifies items in-place.
|
||||
"""
|
||||
all_items = []
|
||||
for source_list in source_lists:
|
||||
all_items.extend(source_list)
|
||||
|
||||
if len(all_items) <= 1:
|
||||
return
|
||||
|
||||
# Pre-compute trigrams using cross-source text extraction
|
||||
ngrams = [get_ngrams(get_cross_source_text(item)) for item in all_items]
|
||||
|
||||
for i in range(len(all_items)):
|
||||
for j in range(i + 1, len(all_items)):
|
||||
# Skip same-source comparisons (already handled by per-source dedupe)
|
||||
if type(all_items[i]) == type(all_items[j]):
|
||||
continue
|
||||
|
||||
similarity = jaccard_similarity(ngrams[i], ngrams[j])
|
||||
if similarity >= threshold:
|
||||
# Bidirectional cross-reference
|
||||
if all_items[j].id not in all_items[i].cross_refs:
|
||||
all_items[i].cross_refs.append(all_items[j].id)
|
||||
if all_items[i].id not in all_items[j].cross_refs:
|
||||
all_items[j].cross_refs.append(all_items[i].id)
|
||||
```
|
||||
|
||||
#### `scripts/last30days.py`
|
||||
|
||||
- [x] After the dedupe step (after line ~1107), call `dedupe.cross_source_link()`:
|
||||
|
||||
```python
|
||||
# Cross-source linking
|
||||
dedupe.cross_source_link(
|
||||
deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_web,
|
||||
)
|
||||
```
|
||||
|
||||
#### `scripts/lib/render.py`
|
||||
|
||||
- [x] In `render_compact()`, append `[xref: ...]` to items that have cross_refs:
|
||||
|
||||
```python
|
||||
# After the existing item line
|
||||
if hasattr(item, 'cross_refs') and item.cross_refs:
|
||||
xref_str = ', '.join(item.cross_refs)
|
||||
line += f" [xref: {xref_str}]"
|
||||
```
|
||||
|
||||
#### `SKILL.md`
|
||||
|
||||
- [x] Add a note in the synthesis instructions about cross-refs:
|
||||
"Items tagged `[xref: ...]` reference the same story on another platform. Use these to triangulate: 'This was widely discussed - Reddit thread (R3) with 142 comments, HN discussion (HN5) with 89 points, and several X posts (X12).'"
|
||||
|
||||
### Tests
|
||||
|
||||
#### `tests/test_youtube_relevance.py` (new)
|
||||
|
||||
- [x] Test exact match: query "Claude Code" vs title "Claude Code" = 1.0
|
||||
- [x] Test partial match: query "Claude Code" vs title "Claude Code Tutorial" = 1.0 (ratio)
|
||||
- [x] Test low match: query "Claude Code" vs title "Python Web Scraping" = 0.1 (floor)
|
||||
- [x] Test empty query: returns 0.5
|
||||
- [x] Test empty title: returns 0.1
|
||||
- [x] Test stopword handling: query "how to use Claude" vs title "Using Claude" = high match
|
||||
- [x] Test integration: search_youtube returns varied relevance scores (mock yt-dlp)
|
||||
|
||||
#### `tests/test_cross_source.py` (new)
|
||||
|
||||
- [x] Test no cross-refs: unrelated items across sources
|
||||
- [x] Test bidirectional: matching Reddit + HN items both get cross_refs
|
||||
- [x] Test multi-source: same story on 3+ sources
|
||||
- [x] Test same-source skip: items from same source type are not cross-linked
|
||||
- [x] Test X text truncation: long tweet vs short Reddit title
|
||||
- [x] Test empty lists: no crash on empty source lists
|
||||
- [x] Test schema round-trip: cross_refs survive to_dict() / from_dict()
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] YouTube items have varied relevance scores (not all 0.7)
|
||||
- [x] A video with title matching the query scores higher than one without
|
||||
- [x] Cross-source items about the same story have `cross_refs` pointing to each other
|
||||
- [x] Cross-refs are bidirectional
|
||||
- [x] Compact output shows `[xref: ...]` tags on linked items
|
||||
- [x] SKILL.md tells Claude how to use cross-refs in synthesis
|
||||
- [x] All existing tests still pass
|
||||
- [x] New tests cover YouTube relevance edge cases
|
||||
- [x] New tests cover cross-source linking edge cases
|
||||
- [x] Run sync.sh to deploy after changes
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
**Low risk:**
|
||||
- Both features add fields/functions without changing existing logic
|
||||
- YouTube relevance is a drop-in replacement for a hardcoded value
|
||||
- Cross-source linking only adds metadata (cross_refs) without modifying item order or scores
|
||||
- All existing tests should pass unchanged
|
||||
|
||||
**Edge cases:**
|
||||
- CJK/non-English titles: token splitting works but relevance may be less accurate. Acceptable for v1.
|
||||
- Very short queries (1 token): ratio overlap = 0 or 1. The 0.1 floor handles the 0 case.
|
||||
- Clickbait YouTube titles with no query overlap: these correctly get low relevance now (improvement over blindly getting 0.7)
|
||||
|
||||
## Sources & References
|
||||
|
||||
- YouTube relevance: Currently hardcoded at `youtube_yt.py:186`
|
||||
- Dedupe infrastructure: `dedupe.py` - Jaccard similarity on char trigrams
|
||||
- Score weights: `score.py:8-10` - 45% relevance + 25% recency + 30% engagement
|
||||
- Existing HN plan as template: `docs/plans/2026-02-24-fix-hn-ordering-and-emoji-plan.md`
|
||||
@@ -1113,6 +1113,11 @@ def main():
|
||||
by_relevance = sorted(normalized_reddit, key=lambda item: item.relevance, reverse=True)
|
||||
deduped_reddit = by_relevance[:3]
|
||||
|
||||
# Cross-source linking: annotate items that discuss the same story
|
||||
dedupe.cross_source_link(
|
||||
deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_web,
|
||||
)
|
||||
|
||||
progress.end_processing()
|
||||
|
||||
# Create report
|
||||
|
||||
+57
-1
@@ -36,7 +36,11 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
|
||||
return intersection / union if union > 0 else 0.0
|
||||
|
||||
|
||||
def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem, schema.HackerNewsItem]) -> str:
|
||||
AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
|
||||
schema.HackerNewsItem, schema.WebSearchItem]
|
||||
|
||||
|
||||
def get_item_text(item: AnyItem) -> str:
|
||||
"""Get comparable text from an item."""
|
||||
if isinstance(item, schema.RedditItem):
|
||||
return item.title
|
||||
@@ -44,10 +48,23 @@ def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeIte
|
||||
return item.title
|
||||
elif isinstance(item, schema.YouTubeItem):
|
||||
return f"{item.title} {item.channel_name}"
|
||||
elif isinstance(item, schema.WebSearchItem):
|
||||
return item.title
|
||||
else:
|
||||
return item.text
|
||||
|
||||
|
||||
def _get_cross_source_text(item: AnyItem) -> str:
|
||||
"""Get text for cross-source comparison.
|
||||
|
||||
Same as get_item_text() but truncates X posts to 100 chars
|
||||
to level the playing field against short Reddit/HN titles.
|
||||
"""
|
||||
if isinstance(item, schema.XItem):
|
||||
return item.text[:100]
|
||||
return get_item_text(item)
|
||||
|
||||
|
||||
def find_duplicates(
|
||||
items: List[Union[schema.RedditItem, schema.XItem]],
|
||||
threshold: float = 0.7,
|
||||
@@ -138,3 +155,42 @@ def dedupe_hackernews(
|
||||
) -> List[schema.HackerNewsItem]:
|
||||
"""Dedupe Hacker News items."""
|
||||
return dedupe_items(items, threshold)
|
||||
|
||||
|
||||
def cross_source_link(
|
||||
*source_lists: List[AnyItem],
|
||||
threshold: float = 0.5,
|
||||
) -> None:
|
||||
"""Annotate items with cross-source references.
|
||||
|
||||
Compares items across different source types using Jaccard similarity
|
||||
on char trigrams. When similarity exceeds threshold, adds bidirectional
|
||||
cross_refs with the related item's ID. Modifies items in-place.
|
||||
|
||||
Args:
|
||||
*source_lists: Variable number of per-source item lists
|
||||
threshold: Similarity threshold for cross-linking (default 0.5)
|
||||
"""
|
||||
all_items = []
|
||||
for source_list in source_lists:
|
||||
all_items.extend(source_list)
|
||||
|
||||
if len(all_items) <= 1:
|
||||
return
|
||||
|
||||
# Pre-compute trigrams using cross-source text extraction
|
||||
ngrams = [get_ngrams(_get_cross_source_text(item)) for item in all_items]
|
||||
|
||||
for i in range(len(all_items)):
|
||||
for j in range(i + 1, len(all_items)):
|
||||
# Skip same-source comparisons (handled by per-source dedupe)
|
||||
if type(all_items[i]) is type(all_items[j]):
|
||||
continue
|
||||
|
||||
similarity = jaccard_similarity(ngrams[i], ngrams[j])
|
||||
if similarity >= threshold:
|
||||
# Bidirectional cross-reference
|
||||
if all_items[j].id not in all_items[i].cross_refs:
|
||||
all_items[i].cross_refs.append(all_items[j].id)
|
||||
if all_items[i].id not in all_items[j].cross_refs:
|
||||
all_items[j].cross_refs.append(all_items[i].id)
|
||||
|
||||
+13
-5
@@ -11,6 +11,14 @@ from . import schema
|
||||
OUTPUT_DIR = Path.home() / ".local" / "share" / "last30days" / "out"
|
||||
|
||||
|
||||
def _xref_tag(item) -> str:
|
||||
"""Return ' [xref: X1, HN3]' string if item has cross_refs, else ''."""
|
||||
refs = getattr(item, 'cross_refs', None)
|
||||
if refs:
|
||||
return f" [xref: {', '.join(refs)}]"
|
||||
return ""
|
||||
|
||||
|
||||
def ensure_output_dir():
|
||||
"""Ensure output directory exists. Supports env override and sandbox fallback."""
|
||||
global OUTPUT_DIR
|
||||
@@ -134,7 +142,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
date_str = f" ({item.date})" if item.date else " (date unknown)"
|
||||
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
|
||||
|
||||
lines.append(f"**{item.id}** (score:{item.score}) r/{item.subreddit}{date_str}{conf_str}{eng_str}")
|
||||
lines.append(f"**{item.id}** (score:{item.score}) r/{item.subreddit}{date_str}{conf_str}{eng_str}{_xref_tag(item)}")
|
||||
lines.append(f" {item.title}")
|
||||
lines.append(f" {item.url}")
|
||||
lines.append(f" *{item.why_relevant}*")
|
||||
@@ -176,7 +184,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
date_str = f" ({item.date})" if item.date else " (date unknown)"
|
||||
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
|
||||
|
||||
lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{conf_str}{eng_str}")
|
||||
lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{conf_str}{eng_str}{_xref_tag(item)}")
|
||||
lines.append(f" {item.text[:200]}...")
|
||||
lines.append(f" {item.url}")
|
||||
lines.append(f" *{item.why_relevant}*")
|
||||
@@ -205,7 +213,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
|
||||
date_str = f" ({item.date})" if item.date else ""
|
||||
|
||||
lines.append(f"**{item.id}** (score:{item.score}) {item.channel_name}{date_str}{eng_str}")
|
||||
lines.append(f"**{item.id}** (score:{item.score}) {item.channel_name}{date_str}{eng_str}{_xref_tag(item)}")
|
||||
lines.append(f" {item.title}")
|
||||
lines.append(f" {item.url}")
|
||||
if item.transcript_snippet:
|
||||
@@ -239,7 +247,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
|
||||
date_str = f" ({item.date})" if item.date else ""
|
||||
|
||||
lines.append(f"**{item.id}** (score:{item.score}) hn/{item.author}{date_str}{eng_str}")
|
||||
lines.append(f"**{item.id}** (score:{item.score}) hn/{item.author}{date_str}{eng_str}{_xref_tag(item)}")
|
||||
lines.append(f" {item.title}")
|
||||
lines.append(f" {item.hn_url}")
|
||||
lines.append(f" *{item.why_relevant}*")
|
||||
@@ -265,7 +273,7 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
date_str = f" ({item.date})" if item.date else " (date unknown)"
|
||||
conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
|
||||
|
||||
lines.append(f"**{item.id}** [WEB] (score:{item.score}) {item.source_domain}{date_str}{conf_str}")
|
||||
lines.append(f"**{item.id}** [WEB] (score:{item.score}) {item.source_domain}{date_str}{conf_str}{_xref_tag(item)}")
|
||||
lines.append(f" {item.title}")
|
||||
lines.append(f" {item.url}")
|
||||
lines.append(f" {item.snippet[:150]}...")
|
||||
|
||||
+30
-5
@@ -93,9 +93,10 @@ class RedditItem:
|
||||
why_relevant: str = ""
|
||||
subs: SubScores = field(default_factory=SubScores)
|
||||
score: int = 0
|
||||
cross_refs: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
d = {
|
||||
'id': self.id,
|
||||
'title': self.title,
|
||||
'url': self.url,
|
||||
@@ -110,6 +111,9 @@ class RedditItem:
|
||||
'subs': self.subs.to_dict(),
|
||||
'score': self.score,
|
||||
}
|
||||
if self.cross_refs:
|
||||
d['cross_refs'] = self.cross_refs
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -126,9 +130,10 @@ class XItem:
|
||||
why_relevant: str = ""
|
||||
subs: SubScores = field(default_factory=SubScores)
|
||||
score: int = 0
|
||||
cross_refs: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
d = {
|
||||
'id': self.id,
|
||||
'text': self.text,
|
||||
'url': self.url,
|
||||
@@ -141,6 +146,9 @@ class XItem:
|
||||
'subs': self.subs.to_dict(),
|
||||
'score': self.score,
|
||||
}
|
||||
if self.cross_refs:
|
||||
d['cross_refs'] = self.cross_refs
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -157,9 +165,10 @@ class WebSearchItem:
|
||||
why_relevant: str = ""
|
||||
subs: SubScores = field(default_factory=SubScores)
|
||||
score: int = 0
|
||||
cross_refs: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
d = {
|
||||
'id': self.id,
|
||||
'title': self.title,
|
||||
'url': self.url,
|
||||
@@ -172,6 +181,9 @@ class WebSearchItem:
|
||||
'subs': self.subs.to_dict(),
|
||||
'score': self.score,
|
||||
}
|
||||
if self.cross_refs:
|
||||
d['cross_refs'] = self.cross_refs
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -189,9 +201,10 @@ class YouTubeItem:
|
||||
why_relevant: str = ""
|
||||
subs: SubScores = field(default_factory=SubScores)
|
||||
score: int = 0
|
||||
cross_refs: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
d = {
|
||||
'id': self.id,
|
||||
'title': self.title,
|
||||
'url': self.url,
|
||||
@@ -205,6 +218,9 @@ class YouTubeItem:
|
||||
'subs': self.subs.to_dict(),
|
||||
'score': self.score,
|
||||
}
|
||||
if self.cross_refs:
|
||||
d['cross_refs'] = self.cross_refs
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -224,9 +240,10 @@ class HackerNewsItem:
|
||||
why_relevant: str = ""
|
||||
subs: SubScores = field(default_factory=SubScores)
|
||||
score: int = 0
|
||||
cross_refs: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
d = {
|
||||
'id': self.id,
|
||||
'title': self.title,
|
||||
'url': self.url,
|
||||
@@ -242,6 +259,9 @@ class HackerNewsItem:
|
||||
'subs': self.subs.to_dict(),
|
||||
'score': self.score,
|
||||
}
|
||||
if self.cross_refs:
|
||||
d['cross_refs'] = self.cross_refs
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -338,6 +358,7 @@ class Report:
|
||||
why_relevant=r.get('why_relevant', ''),
|
||||
subs=subs,
|
||||
score=r.get('score', 0),
|
||||
cross_refs=r.get('cross_refs', []),
|
||||
))
|
||||
|
||||
# Reconstruct X items
|
||||
@@ -359,6 +380,7 @@ class Report:
|
||||
why_relevant=x.get('why_relevant', ''),
|
||||
subs=subs,
|
||||
score=x.get('score', 0),
|
||||
cross_refs=x.get('cross_refs', []),
|
||||
))
|
||||
|
||||
# Reconstruct Web items
|
||||
@@ -377,6 +399,7 @@ class Report:
|
||||
why_relevant=w.get('why_relevant', ''),
|
||||
subs=subs,
|
||||
score=w.get('score', 0),
|
||||
cross_refs=w.get('cross_refs', []),
|
||||
))
|
||||
|
||||
# Reconstruct YouTube items
|
||||
@@ -399,6 +422,7 @@ class Report:
|
||||
why_relevant=y.get('why_relevant', ''),
|
||||
subs=subs,
|
||||
score=y.get('score', 0),
|
||||
cross_refs=y.get('cross_refs', []),
|
||||
))
|
||||
|
||||
# Reconstruct HackerNews items
|
||||
@@ -424,6 +448,7 @@ class Report:
|
||||
why_relevant=h.get('why_relevant', ''),
|
||||
subs=subs,
|
||||
score=h.get('score', 0),
|
||||
cross_refs=h.get('cross_refs', []),
|
||||
))
|
||||
|
||||
return cls(
|
||||
|
||||
@@ -17,7 +17,7 @@ import sys
|
||||
import tempfile
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
# Depth configurations: how many videos to search / transcribe
|
||||
DEPTH_CONFIG = {
|
||||
@@ -35,6 +35,38 @@ TRANSCRIPT_LIMITS = {
|
||||
# Max words to keep from each transcript
|
||||
TRANSCRIPT_MAX_WORDS = 500
|
||||
|
||||
# Stopwords for relevance computation (common English words that dilute token overlap)
|
||||
STOPWORDS = frozenset({
|
||||
'the', 'a', 'an', 'to', 'for', 'how', 'is', 'in', 'of', 'on',
|
||||
'and', 'with', 'from', 'by', 'at', 'this', 'that', 'it', 'my',
|
||||
'your', 'i', 'me', 'we', 'you', 'what', 'are', 'do', 'can',
|
||||
'its', 'be', 'or', 'not', 'no', 'so', 'if', 'but', 'about',
|
||||
'all', 'just', 'get', 'has', 'have', 'was', 'will',
|
||||
})
|
||||
|
||||
|
||||
def _tokenize(text: str) -> Set[str]:
|
||||
"""Lowercase, strip punctuation, remove stopwords, drop single-char tokens."""
|
||||
words = re.sub(r'[^\w\s]', ' ', text.lower()).split()
|
||||
return {w for w in words if w not in STOPWORDS and len(w) > 1}
|
||||
|
||||
|
||||
def _compute_relevance(query: str, title: str) -> float:
|
||||
"""Compute relevance as ratio of query tokens found in title.
|
||||
|
||||
Uses ratio overlap (intersection / query_length) so short queries
|
||||
score higher when fully represented in the title. Floors at 0.1.
|
||||
"""
|
||||
q_tokens = _tokenize(query)
|
||||
t_tokens = _tokenize(title)
|
||||
|
||||
if not q_tokens:
|
||||
return 0.5 # Neutral fallback for empty/stopword-only queries
|
||||
|
||||
overlap = len(q_tokens & t_tokens)
|
||||
ratio = overlap / len(q_tokens)
|
||||
return max(0.1, min(1.0, ratio))
|
||||
|
||||
|
||||
def _log(msg: str):
|
||||
"""Log to stderr."""
|
||||
@@ -183,8 +215,8 @@ def search_youtube(
|
||||
"comments": comment_count,
|
||||
},
|
||||
"duration": video.get("duration"),
|
||||
"relevance": 0.7, # Default; no LLM relevance scoring for YouTube
|
||||
"why_relevant": f"YouTube video about {core_topic}",
|
||||
"relevance": _compute_relevance(core_topic, video.get("title", "")),
|
||||
"why_relevant": f"YouTube: {video.get('title', core_topic)[:60]}",
|
||||
})
|
||||
|
||||
# Soft date filter: prefer recent items but fall back to all if too few
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
"""Tests for cross-source linking."""
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
# Add lib to path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||
|
||||
from lib import dedupe, schema
|
||||
|
||||
|
||||
class TestCrossSourceLink(unittest.TestCase):
|
||||
def _make_reddit(self, id, title, score=50):
|
||||
item = schema.RedditItem(id=id, title=title, url="", subreddit="test")
|
||||
item.score = score
|
||||
return item
|
||||
|
||||
def _make_hn(self, id, title, score=50):
|
||||
item = schema.HackerNewsItem(id=id, title=title, url="", hn_url="", author="user")
|
||||
item.score = score
|
||||
return item
|
||||
|
||||
def _make_x(self, id, text, score=50):
|
||||
item = schema.XItem(id=id, text=text, url="", author_handle="user")
|
||||
item.score = score
|
||||
return item
|
||||
|
||||
def _make_yt(self, id, title, score=50):
|
||||
item = schema.YouTubeItem(id=id, title=title, url="", channel_name="ch")
|
||||
item.score = score
|
||||
return item
|
||||
|
||||
def _make_web(self, id, title, score=50):
|
||||
item = schema.WebSearchItem(id=id, title=title, url="", source_domain="example.com", snippet="")
|
||||
item.score = score
|
||||
return item
|
||||
|
||||
def test_no_crossrefs_for_unrelated(self):
|
||||
reddit = [self._make_reddit("R1", "Best Claude Code Tips")]
|
||||
hn = [self._make_hn("HN1", "Python Django Release Notes")]
|
||||
dedupe.cross_source_link(reddit, hn)
|
||||
self.assertEqual(reddit[0].cross_refs, [])
|
||||
self.assertEqual(hn[0].cross_refs, [])
|
||||
|
||||
def test_bidirectional_link(self):
|
||||
reddit = [self._make_reddit("R1", "OpenAI launches GPT-5 with new features")]
|
||||
hn = [self._make_hn("HN1", "OpenAI launches GPT-5 with new features")]
|
||||
dedupe.cross_source_link(reddit, hn)
|
||||
self.assertIn("HN1", reddit[0].cross_refs)
|
||||
self.assertIn("R1", hn[0].cross_refs)
|
||||
|
||||
def test_multi_source_link(self):
|
||||
reddit = [self._make_reddit("R1", "Claude Code gets new skill system")]
|
||||
hn = [self._make_hn("HN1", "Claude Code gets new skill system")]
|
||||
yt = [self._make_yt("YT1", "Claude Code gets new skill system")]
|
||||
dedupe.cross_source_link(reddit, hn, yt)
|
||||
# All three should reference each other
|
||||
self.assertEqual(len(reddit[0].cross_refs), 2)
|
||||
self.assertEqual(len(hn[0].cross_refs), 2)
|
||||
self.assertEqual(len(yt[0].cross_refs), 2)
|
||||
|
||||
def test_same_source_not_linked(self):
|
||||
reddit = [
|
||||
self._make_reddit("R1", "OpenAI GPT-5 launch details"),
|
||||
self._make_reddit("R2", "OpenAI GPT-5 launch details"),
|
||||
]
|
||||
dedupe.cross_source_link(reddit)
|
||||
self.assertEqual(reddit[0].cross_refs, [])
|
||||
self.assertEqual(reddit[1].cross_refs, [])
|
||||
|
||||
def test_x_text_truncation_helps(self):
|
||||
# Truncation increases similarity vs full tweet.
|
||||
# A near-identical short tweet should match a Reddit title.
|
||||
reddit = [self._make_reddit("R1", "Anthropic releases Claude 4 model")]
|
||||
x_short = [self._make_x("X1", "Anthropic releases Claude 4 model today!")]
|
||||
dedupe.cross_source_link(reddit, x_short)
|
||||
self.assertIn("X1", reddit[0].cross_refs)
|
||||
self.assertIn("R1", x_short[0].cross_refs)
|
||||
|
||||
def test_long_x_text_may_not_match(self):
|
||||
# When a tweet diverges significantly after the shared prefix,
|
||||
# Jaccard drops below 0.5 even with truncation. This is expected.
|
||||
reddit = [self._make_reddit("R1", "Anthropic releases Claude 4 model")]
|
||||
x_long = [self._make_x("X1",
|
||||
"Anthropic releases Claude 4 model and it's incredible. "
|
||||
"The reasoning capabilities are next level. Just tested it "
|
||||
"on my entire codebase and it understood everything."
|
||||
)]
|
||||
dedupe.cross_source_link(reddit, x_long)
|
||||
# May or may not match depending on trigram overlap - just verify no crash
|
||||
self.assertIsInstance(reddit[0].cross_refs, list)
|
||||
|
||||
def test_empty_lists(self):
|
||||
# Should not crash
|
||||
dedupe.cross_source_link([], [], [])
|
||||
|
||||
def test_single_item(self):
|
||||
reddit = [self._make_reddit("R1", "Test item")]
|
||||
dedupe.cross_source_link(reddit)
|
||||
self.assertEqual(reddit[0].cross_refs, [])
|
||||
|
||||
def test_no_duplicate_refs(self):
|
||||
reddit = [self._make_reddit("R1", "Same exact title repeated")]
|
||||
hn = [self._make_hn("HN1", "Same exact title repeated")]
|
||||
# Call twice - should not duplicate refs
|
||||
dedupe.cross_source_link(reddit, hn)
|
||||
dedupe.cross_source_link(reddit, hn)
|
||||
self.assertEqual(reddit[0].cross_refs.count("HN1"), 1)
|
||||
self.assertEqual(hn[0].cross_refs.count("R1"), 1)
|
||||
|
||||
def test_web_items_linked(self):
|
||||
web = [self._make_web("W1", "Claude Code skill system overview")]
|
||||
hn = [self._make_hn("HN1", "Claude Code skill system overview")]
|
||||
dedupe.cross_source_link(web, hn)
|
||||
self.assertIn("HN1", web[0].cross_refs)
|
||||
self.assertIn("W1", hn[0].cross_refs)
|
||||
|
||||
|
||||
class TestCrossRefsSchemaRoundTrip(unittest.TestCase):
|
||||
def test_reddit_roundtrip(self):
|
||||
item = schema.RedditItem(id="R1", title="Test", url="", subreddit="test",
|
||||
cross_refs=["HN1", "X2"])
|
||||
d = item.to_dict()
|
||||
self.assertEqual(d['cross_refs'], ["HN1", "X2"])
|
||||
|
||||
def test_reddit_empty_crossrefs_omitted(self):
|
||||
item = schema.RedditItem(id="R1", title="Test", url="", subreddit="test")
|
||||
d = item.to_dict()
|
||||
self.assertNotIn('cross_refs', d)
|
||||
|
||||
def test_report_roundtrip(self):
|
||||
report = schema.Report(
|
||||
topic="test", range_from="2026-01-01", range_to="2026-02-01",
|
||||
generated_at="2026-02-01T00:00:00Z", mode="both",
|
||||
reddit=[schema.RedditItem(id="R1", title="T", url="", subreddit="s",
|
||||
cross_refs=["HN1"])],
|
||||
hackernews=[schema.HackerNewsItem(id="HN1", title="T", url="", hn_url="",
|
||||
author="u", cross_refs=["R1"])],
|
||||
)
|
||||
d = report.to_dict()
|
||||
restored = schema.Report.from_dict(d)
|
||||
self.assertEqual(restored.reddit[0].cross_refs, ["HN1"])
|
||||
self.assertEqual(restored.hackernews[0].cross_refs, ["R1"])
|
||||
|
||||
|
||||
class TestGetCrossSourceText(unittest.TestCase):
|
||||
def test_x_truncated(self):
|
||||
item = schema.XItem(id="X1", text="A" * 200, url="", author_handle="u")
|
||||
result = dedupe._get_cross_source_text(item)
|
||||
self.assertEqual(len(result), 100)
|
||||
|
||||
def test_reddit_uses_title(self):
|
||||
item = schema.RedditItem(id="R1", title="My Title", url="", subreddit="s")
|
||||
result = dedupe._get_cross_source_text(item)
|
||||
self.assertEqual(result, "My Title")
|
||||
|
||||
def test_web_uses_title(self):
|
||||
item = schema.WebSearchItem(id="W1", title="Web Title", url="",
|
||||
source_domain="example.com", snippet="snip")
|
||||
result = dedupe._get_cross_source_text(item)
|
||||
self.assertEqual(result, "Web Title")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Tests for YouTube relevance scoring."""
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
# Add lib to path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||
|
||||
from lib.youtube_yt import _compute_relevance, _tokenize
|
||||
|
||||
|
||||
class TestTokenize(unittest.TestCase):
|
||||
def test_basic(self):
|
||||
tokens = _tokenize("Claude Code Tutorial")
|
||||
self.assertIn("claude", tokens)
|
||||
self.assertIn("code", tokens)
|
||||
self.assertIn("tutorial", tokens)
|
||||
|
||||
def test_removes_stopwords(self):
|
||||
tokens = _tokenize("how to use Claude Code for beginners")
|
||||
self.assertNotIn("how", tokens)
|
||||
self.assertNotIn("to", tokens)
|
||||
self.assertNotIn("for", tokens)
|
||||
self.assertIn("claude", tokens)
|
||||
self.assertIn("code", tokens)
|
||||
self.assertIn("beginners", tokens)
|
||||
|
||||
def test_strips_punctuation(self):
|
||||
tokens = _tokenize("What's new in Claude?")
|
||||
self.assertIn("claude", tokens)
|
||||
self.assertNotIn("what's", tokens)
|
||||
|
||||
def test_drops_single_char(self):
|
||||
tokens = _tokenize("a b c Claude")
|
||||
self.assertNotIn("a", tokens)
|
||||
self.assertNotIn("b", tokens)
|
||||
self.assertIn("claude", tokens)
|
||||
|
||||
def test_empty(self):
|
||||
tokens = _tokenize("")
|
||||
self.assertEqual(tokens, set())
|
||||
|
||||
def test_all_stopwords(self):
|
||||
tokens = _tokenize("the a an to for how")
|
||||
self.assertEqual(tokens, set())
|
||||
|
||||
|
||||
class TestComputeRelevance(unittest.TestCase):
|
||||
def test_exact_match(self):
|
||||
result = _compute_relevance("Claude Code", "Claude Code")
|
||||
self.assertEqual(result, 1.0)
|
||||
|
||||
def test_full_match_in_longer_title(self):
|
||||
result = _compute_relevance("Claude Code", "Claude Code Tutorial")
|
||||
self.assertEqual(result, 1.0)
|
||||
|
||||
def test_partial_match(self):
|
||||
result = _compute_relevance("Claude Code Tips", "Claude Tips for Beginners")
|
||||
self.assertGreater(result, 0.5)
|
||||
self.assertLess(result, 1.0)
|
||||
|
||||
def test_no_match(self):
|
||||
result = _compute_relevance("Claude Code", "Python Web Scraping")
|
||||
self.assertEqual(result, 0.1) # Floor
|
||||
|
||||
def test_empty_query_returns_neutral(self):
|
||||
result = _compute_relevance("", "Some Video Title")
|
||||
self.assertEqual(result, 0.5)
|
||||
|
||||
def test_stopword_only_query(self):
|
||||
result = _compute_relevance("how to the", "Some Video Title")
|
||||
self.assertEqual(result, 0.5)
|
||||
|
||||
def test_empty_title(self):
|
||||
result = _compute_relevance("Claude Code", "")
|
||||
self.assertEqual(result, 0.1) # Floor
|
||||
|
||||
def test_case_insensitive(self):
|
||||
result = _compute_relevance("claude code", "CLAUDE CODE Tutorial")
|
||||
self.assertEqual(result, 1.0)
|
||||
|
||||
def test_stopwords_in_title_dont_inflate(self):
|
||||
# "Claude Code" query against a title with lots of stopwords
|
||||
# Should still match well since Claude and Code are present
|
||||
result = _compute_relevance(
|
||||
"Claude Code",
|
||||
"How To Use Claude Code For Complete Beginners"
|
||||
)
|
||||
self.assertEqual(result, 1.0)
|
||||
|
||||
def test_floor_at_0_1(self):
|
||||
result = _compute_relevance("quantum computing", "cat videos compilation")
|
||||
self.assertEqual(result, 0.1)
|
||||
|
||||
def test_cap_at_1_0(self):
|
||||
result = _compute_relevance("AI", "AI AI AI AI AI")
|
||||
self.assertLessEqual(result, 1.0)
|
||||
|
||||
def test_single_word_query(self):
|
||||
result = _compute_relevance("Seedance", "Seedance AI Video Generator Review")
|
||||
self.assertEqual(result, 1.0)
|
||||
|
||||
def test_single_word_no_match(self):
|
||||
result = _compute_relevance("Seedance", "Random cooking video")
|
||||
self.assertEqual(result, 0.1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user