feat(podcasts): make podcasts always available + smarter mention matching

Changes:
- Podcasts source is now always available when yt-dlp is installed (same as
  YouTube). Previously required explicit opt-in via INCLUDE_SOURCES or
  --search=podcasts.
- Smarter mention matching: extract key terms from multi-word topics and
  use max count across terms. "Kanye West Bully album" now matches
  episodes mentioning "Kanye" 85 times (previously 0 due to exact phrase).
- SKILL.md: add podcast channel resolution to Step 0.55, include
  --podcast-channels in execution command, update ACTIVE_SOURCES_LIST.

Tested: Kanye West query now finds 5 podcast hits including hidden
mentions in off-topic episodes (Lost Civilizations, Mike WiLL Made-It).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-04-10 21:04:12 -04:00
parent 49d45c2b42
commit 09ed497804
3 changed files with 73 additions and 7 deletions
+24 -1
View File
@@ -375,7 +375,7 @@ Common patterns:
- Always active: Reddit, Hacker News, Polymarket - Always active: Reddit, Hacker News, Polymarket
- If gh CLI is installed (check `which gh`): add GitHub - If gh CLI is installed (check `which gh`): add GitHub
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set: add X - If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set: add X
- If yt-dlp is installed (check `which yt-dlp`): add YouTube - If yt-dlp is installed (check `which yt-dlp`): add YouTube AND Podcasts
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok - If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram - If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads - If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads
@@ -615,6 +615,27 @@ Store as `RESOLVED_IG_CREATORS`.
Store as `RESOLVED_YT_QUERIES`. Store as `RESOLVED_YT_QUERIES`.
**6. Podcast channels****INFER 6-12 YouTube podcast channel @handles from topic knowledge.** Think in two dimensions:
1. **Domain podcasts** — What YouTube podcasts focus on this topic's domain?
- Hip-hop/music → `DrinkChamps,JoeBuddenTV,BreakfastClubPower1051FM,OfficialFlagrant`
- Tech/AI/startups → `lexfridman,DwarkeshPatel,AllInPod,MyFirstMillionPod,LennysPodcast`
- Business/finance → `AcquiredFM,InvestLikeTheBest,PatrickBoyleOnFinance,PropGPod`
- Sports → `PatMcAfeeShowOfficial,ShannonSharpe,ClubShayShay`
- Culture/celebs → `joerogan,CallHerDaddy,ClubShayShay`
- Knitting/crafts → `FruityKnitting,VeryPinkKnits,GroceryGirlsKnit`
2. **Cross-domain podcasts** — What popular interview/deep-dive podcasts might cover this topic even if it's not their main focus?
- Business-adjacent topics → `AcquiredFM,InvestLikeTheBest` (company deep dives)
- Tech-adjacent topics → `lexfridman,AllInPod` (broad tech interviews)
- Culture-adjacent topics → `joerogan,OfficialFlagrant` (celebrity interviews)
**Rationale:** The engine uses these channels for transcript-first discovery. Even if the topic isn't in an episode title, it may be discussed within the episode. Acquired's "The NFL" episode mentions Taylor Swift 18 times, ESPN 117 times — invisible to YouTube search but found by transcript scanning.
**Handle accuracy:** Return your best guess at the exact @handle. If wrong, the engine falls back to a search-based lookup. Don't stress the exact spelling — `@AcquiredFM`, `@lexfridman`, `@joerogan` work; `@FLAGRANT` fails but falls back to find `@OfficialFlagrant`.
Store as `RESOLVED_PODCAST_CHANNELS` (comma-separated, no @ prefix).
**Concrete examples:** **Concrete examples:**
| Topic | WebSearches needed | Reddit subs | TikTok hashtags | TikTok creators | IG creators | YT queries | | Topic | WebSearches needed | Reddit subs | TikTok hashtags | TikTok creators | IG creators | YT queries |
@@ -635,6 +656,7 @@ Resolved:
- Reddit: r/{sub1}, r/{sub2}, r/{sub3} - Reddit: r/{sub1}, r/{sub2}, r/{sub3}
- TikTok: #{hashtag1}, #{hashtag2} - TikTok: #{hashtag1}, #{hashtag2}
- YouTube: {query1}, {query2} - YouTube: {query1}, {query2}
- Podcasts: @{channel1}, @{channel2}, @{channel3}
``` ```
Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful. Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful.
@@ -760,6 +782,7 @@ fi
- `--ig-creators={RESOLVED_IG_CREATORS}` (from Step 0.55) - `--ig-creators={RESOLVED_IG_CREATORS}` (from Step 0.55)
- `--github-user={RESOLVED_GITHUB_USER}` (from Step 0.5b, person topics only) - `--github-user={RESOLVED_GITHUB_USER}` (from Step 0.5b, person topics only)
- `--github-repo={RESOLVED_GITHUB_REPOS}` (from Step 0.5c, product/project topics only) - `--github-repo={RESOLVED_GITHUB_REPOS}` (from Step 0.5c, product/project topics only)
- `--podcast-channels={RESOLVED_PODCAST_CHANNELS}` (from Step 0.55, 6-12 @handles)
- Omit any flag where the value was not resolved (empty). - Omit any flag where the value was not resolved (empty).
**If you skipped Steps 0.55 and 0.75 (no WebSearch -- OpenClaw, Codex, etc.), add:** **If you skipped Steps 0.55 and 0.75 (no WebSearch -- OpenClaw, Codex, etc.), add:**
+4 -1
View File
@@ -124,7 +124,10 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.append("pinterest") available.append("pinterest")
if env.is_xquik_available(config): if env.is_xquik_available(config):
available.append("xquik") available.append("xquik")
if podcast_yt.is_available() and ("podcasts" in include_sources or (requested_sources and "podcasts" in requested_sources)): # Podcasts: available whenever yt-dlp is installed (same as YouTube).
# Opt-out only. The source returns empty when no channels are resolved,
# so there's no cost to having it available.
if podcast_yt.is_available():
available.append("podcasts") available.append("podcasts")
return available return available
+45 -5
View File
@@ -214,12 +214,52 @@ def _fetch_captions(video_id: str, temp_dir: str) -> Optional[str]:
return None return None
_NOISE_WORDS = frozenset({
"the", "a", "an", "of", "and", "or", "for", "to", "in", "on", "at",
"best", "top", "new", "latest", "review", "news", "vs", "versus",
"album", "song", "episode", "podcast", "interview", "this", "that",
"what", "how", "why", "where", "when", "who",
})
def _extract_key_terms(topic: str) -> List[str]:
"""Extract meaningful terms from topic for matching.
For "Kanye West Bully album" -> ["Kanye West", "Bully"] or similar.
For single words, just returns the word.
"""
words = [w.strip() for w in topic.split() if w.strip()]
# Remove noise words
meaningful = [w for w in words if w.lower() not in _NOISE_WORDS and len(w) > 2]
if not meaningful:
return [topic.strip()]
# If the topic has 2+ meaningful words, also include the full phrase
# and the first 2 words as a potential entity name
terms = []
if len(meaningful) >= 2:
# Full phrase first (for exact entity matches like "Taylor Swift")
terms.append(" ".join(meaningful[:2]))
terms.extend(meaningful)
return terms
def _count_mentions(text: str, topic: str) -> int: def _count_mentions(text: str, topic: str) -> int:
"""Count case-insensitive topic mentions in text.""" """Count case-insensitive topic mentions in text.
# Build a regex pattern from the topic words
# For multi-word topics like "Taylor Swift", search for the full phrase Uses the maximum mention count across key terms extracted from the topic.
pattern = re.escape(topic.strip()) "Kanye West Bully album" -> max mentions of ["Kanye West", "Kanye", "West", "Bully"].
return len(re.findall(pattern, text, re.IGNORECASE)) This way, an episode mentioning "Kanye" 85 times counts as 85, not 0.
"""
text_lower = text.lower()
terms = _extract_key_terms(topic)
max_count = 0
for term in terms:
pattern = re.escape(term.lower())
count = len(re.findall(pattern, text_lower))
if count > max_count:
max_count = count
return max_count
def _extract_mention_context(text: str, topic: str, max_excerpts: int = 3) -> List[str]: def _extract_mention_context(text: str, topic: str, max_excerpts: int = 3) -> List[str]: