feat(podcasts): make podcasts always available + smarter mention matching
Changes: - Podcasts source is now always available when yt-dlp is installed (same as YouTube). Previously required explicit opt-in via INCLUDE_SOURCES or --search=podcasts. - Smarter mention matching: extract key terms from multi-word topics and use max count across terms. "Kanye West Bully album" now matches episodes mentioning "Kanye" 85 times (previously 0 due to exact phrase). - SKILL.md: add podcast channel resolution to Step 0.55, include --podcast-channels in execution command, update ACTIVE_SOURCES_LIST. Tested: Kanye West query now finds 5 podcast hits including hidden mentions in off-topic episodes (Lost Civilizations, Mike WiLL Made-It). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -375,7 +375,7 @@ Common patterns:
|
||||
- Always active: Reddit, Hacker News, Polymarket
|
||||
- If gh CLI is installed (check `which gh`): add GitHub
|
||||
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set: add X
|
||||
- If yt-dlp is installed (check `which yt-dlp`): add YouTube
|
||||
- If yt-dlp is installed (check `which yt-dlp`): add YouTube AND Podcasts
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads
|
||||
@@ -615,6 +615,27 @@ Store as `RESOLVED_IG_CREATORS`.
|
||||
|
||||
Store as `RESOLVED_YT_QUERIES`.
|
||||
|
||||
**6. Podcast channels** — **INFER 6-12 YouTube podcast channel @handles from topic knowledge.** Think in two dimensions:
|
||||
|
||||
1. **Domain podcasts** — What YouTube podcasts focus on this topic's domain?
|
||||
- Hip-hop/music → `DrinkChamps,JoeBuddenTV,BreakfastClubPower1051FM,OfficialFlagrant`
|
||||
- Tech/AI/startups → `lexfridman,DwarkeshPatel,AllInPod,MyFirstMillionPod,LennysPodcast`
|
||||
- Business/finance → `AcquiredFM,InvestLikeTheBest,PatrickBoyleOnFinance,PropGPod`
|
||||
- Sports → `PatMcAfeeShowOfficial,ShannonSharpe,ClubShayShay`
|
||||
- Culture/celebs → `joerogan,CallHerDaddy,ClubShayShay`
|
||||
- Knitting/crafts → `FruityKnitting,VeryPinkKnits,GroceryGirlsKnit`
|
||||
|
||||
2. **Cross-domain podcasts** — What popular interview/deep-dive podcasts might cover this topic even if it's not their main focus?
|
||||
- Business-adjacent topics → `AcquiredFM,InvestLikeTheBest` (company deep dives)
|
||||
- Tech-adjacent topics → `lexfridman,AllInPod` (broad tech interviews)
|
||||
- Culture-adjacent topics → `joerogan,OfficialFlagrant` (celebrity interviews)
|
||||
|
||||
**Rationale:** The engine uses these channels for transcript-first discovery. Even if the topic isn't in an episode title, it may be discussed within the episode. Acquired's "The NFL" episode mentions Taylor Swift 18 times, ESPN 117 times — invisible to YouTube search but found by transcript scanning.
|
||||
|
||||
**Handle accuracy:** Return your best guess at the exact @handle. If wrong, the engine falls back to a search-based lookup. Don't stress the exact spelling — `@AcquiredFM`, `@lexfridman`, `@joerogan` work; `@FLAGRANT` fails but falls back to find `@OfficialFlagrant`.
|
||||
|
||||
Store as `RESOLVED_PODCAST_CHANNELS` (comma-separated, no @ prefix).
|
||||
|
||||
**Concrete examples:**
|
||||
|
||||
| Topic | WebSearches needed | Reddit subs | TikTok hashtags | TikTok creators | IG creators | YT queries |
|
||||
@@ -635,6 +656,7 @@ Resolved:
|
||||
- Reddit: r/{sub1}, r/{sub2}, r/{sub3}
|
||||
- TikTok: #{hashtag1}, #{hashtag2}
|
||||
- YouTube: {query1}, {query2}
|
||||
- Podcasts: @{channel1}, @{channel2}, @{channel3}
|
||||
```
|
||||
|
||||
Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful.
|
||||
@@ -760,6 +782,7 @@ fi
|
||||
- `--ig-creators={RESOLVED_IG_CREATORS}` (from Step 0.55)
|
||||
- `--github-user={RESOLVED_GITHUB_USER}` (from Step 0.5b, person topics only)
|
||||
- `--github-repo={RESOLVED_GITHUB_REPOS}` (from Step 0.5c, product/project topics only)
|
||||
- `--podcast-channels={RESOLVED_PODCAST_CHANNELS}` (from Step 0.55, 6-12 @handles)
|
||||
- Omit any flag where the value was not resolved (empty).
|
||||
|
||||
**If you skipped Steps 0.55 and 0.75 (no WebSearch -- OpenClaw, Codex, etc.), add:**
|
||||
|
||||
@@ -124,7 +124,10 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
|
||||
available.append("pinterest")
|
||||
if env.is_xquik_available(config):
|
||||
available.append("xquik")
|
||||
if podcast_yt.is_available() and ("podcasts" in include_sources or (requested_sources and "podcasts" in requested_sources)):
|
||||
# Podcasts: available whenever yt-dlp is installed (same as YouTube).
|
||||
# Opt-out only. The source returns empty when no channels are resolved,
|
||||
# so there's no cost to having it available.
|
||||
if podcast_yt.is_available():
|
||||
available.append("podcasts")
|
||||
return available
|
||||
|
||||
|
||||
@@ -214,12 +214,52 @@ def _fetch_captions(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
_NOISE_WORDS = frozenset({
|
||||
"the", "a", "an", "of", "and", "or", "for", "to", "in", "on", "at",
|
||||
"best", "top", "new", "latest", "review", "news", "vs", "versus",
|
||||
"album", "song", "episode", "podcast", "interview", "this", "that",
|
||||
"what", "how", "why", "where", "when", "who",
|
||||
})
|
||||
|
||||
|
||||
def _extract_key_terms(topic: str) -> List[str]:
|
||||
"""Extract meaningful terms from topic for matching.
|
||||
|
||||
For "Kanye West Bully album" -> ["Kanye West", "Bully"] or similar.
|
||||
For single words, just returns the word.
|
||||
"""
|
||||
words = [w.strip() for w in topic.split() if w.strip()]
|
||||
# Remove noise words
|
||||
meaningful = [w for w in words if w.lower() not in _NOISE_WORDS and len(w) > 2]
|
||||
if not meaningful:
|
||||
return [topic.strip()]
|
||||
|
||||
# If the topic has 2+ meaningful words, also include the full phrase
|
||||
# and the first 2 words as a potential entity name
|
||||
terms = []
|
||||
if len(meaningful) >= 2:
|
||||
# Full phrase first (for exact entity matches like "Taylor Swift")
|
||||
terms.append(" ".join(meaningful[:2]))
|
||||
terms.extend(meaningful)
|
||||
return terms
|
||||
|
||||
|
||||
def _count_mentions(text: str, topic: str) -> int:
|
||||
"""Count case-insensitive topic mentions in text."""
|
||||
# Build a regex pattern from the topic words
|
||||
# For multi-word topics like "Taylor Swift", search for the full phrase
|
||||
pattern = re.escape(topic.strip())
|
||||
return len(re.findall(pattern, text, re.IGNORECASE))
|
||||
"""Count case-insensitive topic mentions in text.
|
||||
|
||||
Uses the maximum mention count across key terms extracted from the topic.
|
||||
"Kanye West Bully album" -> max mentions of ["Kanye West", "Kanye", "West", "Bully"].
|
||||
This way, an episode mentioning "Kanye" 85 times counts as 85, not 0.
|
||||
"""
|
||||
text_lower = text.lower()
|
||||
terms = _extract_key_terms(topic)
|
||||
max_count = 0
|
||||
for term in terms:
|
||||
pattern = re.escape(term.lower())
|
||||
count = len(re.findall(pattern, text_lower))
|
||||
if count > max_count:
|
||||
max_count = count
|
||||
return max_count
|
||||
|
||||
|
||||
def _extract_mention_context(text: str, topic: str, max_excerpts: int = 3) -> List[str]:
|
||||
|
||||
Reference in New Issue
Block a user