Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Matt Van Horn
|
09ed497804 | ||
|
Matt Van Horn
|
49d45c2b42 |
File diff suppressed because it is too large
Load Diff
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"name": "last30days"
|
||||
}
|
||||
@@ -15,7 +15,3 @@ variants/open/references/research.md
|
||||
__pycache__/
|
||||
*.pyc
|
||||
mise.toml
|
||||
.memsearch/
|
||||
.venv/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
@@ -1,269 +0,0 @@
|
||||
---
|
||||
name: last30days
|
||||
version: "3.0.0"
|
||||
description: "Multi-query social search with intelligent planning. Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web."
|
||||
argument-hint: 'last30days AI video tools, last30days best noise cancelling headphones'
|
||||
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
||||
homepage: https://github.com/mvanhorn/last30days-skill
|
||||
repository: https://github.com/mvanhorn/last30days-skill
|
||||
author: mvanhorn
|
||||
license: MIT
|
||||
user-invocable: true
|
||||
metadata:
|
||||
hermes:
|
||||
emoji: "📰"
|
||||
tags:
|
||||
- research
|
||||
- deep-research
|
||||
- reddit
|
||||
- x
|
||||
- twitter
|
||||
- youtube
|
||||
- tiktok
|
||||
- instagram
|
||||
- hackernews
|
||||
- polymarket
|
||||
- trends
|
||||
- recency
|
||||
- news
|
||||
- citations
|
||||
- multi-source
|
||||
- social-media
|
||||
- analysis
|
||||
- web-search
|
||||
requires:
|
||||
env:
|
||||
- SCRAPECREATORS_API_KEY
|
||||
optionalEnv:
|
||||
- OPENAI_API_KEY
|
||||
- XAI_API_KEY
|
||||
- OPENROUTER_API_KEY
|
||||
- PARALLEL_API_KEY
|
||||
- BRAVE_API_KEY
|
||||
- APIFY_API_TOKEN
|
||||
- AUTH_TOKEN
|
||||
- CT0
|
||||
- BSKY_HANDLE
|
||||
- BSKY_APP_PASSWORD
|
||||
- TRUTHSOCIAL_TOKEN
|
||||
bins:
|
||||
- node
|
||||
- python3
|
||||
primaryEnv: SCRAPECREATORS_API_KEY
|
||||
files:
|
||||
- "scripts/*"
|
||||
homepage: https://github.com/mvanhorn/last30days-skill
|
||||
---
|
||||
|
||||
# last30days v3.0.0: Research Any Topic from the Last 30 Days
|
||||
|
||||
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `~/Documents/Last30Days/`. X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
|
||||
|
||||
Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now.
|
||||
|
||||
## Runtime Preflight
|
||||
|
||||
Before running any `last30days.py` command in this skill, resolve a Python 3.12+ interpreter once and keep it in `LAST30DAYS_PYTHON`:
|
||||
|
||||
```bash
|
||||
for py in python3.14 python3.13 python3.12 python3; do
|
||||
command -v "$py" >/dev/null 2>&1 || continue
|
||||
"$py" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)' || continue
|
||||
LAST30DAYS_PYTHON="$py"
|
||||
break
|
||||
done
|
||||
|
||||
if [ -z "${LAST30DAYS_PYTHON:-}" ]; then
|
||||
echo "ERROR: last30days v3 requires Python 3.12+. Install python3.12 or python3.13 and rerun." >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
## Step 0: First-Run Setup Wizard
|
||||
|
||||
**CRITICAL: ALWAYS execute Step 0 BEFORE Step 1, even if the user provided a topic.** If the user typed `last30days Mercer Island`, you MUST check for FIRST_RUN and present the wizard BEFORE running research. The topic "Mercer Island" is preserved — research runs immediately after the wizard completes. Do NOT skip the wizard because a topic was provided. The wizard takes 10 seconds and only runs once ever.
|
||||
|
||||
To detect first run: check if `~/.config/last30days/.env` exists. If it does NOT exist, this is a first run. **Do NOT run any Bash commands or show any command output to detect this — just check the file existence silently.** If the file exists and contains `SETUP_COMPLETE=true`, skip this section **silently** and proceed to Step 1. **Do NOT say "Setup is complete" or any other status message — just move on.** The user doesn't need to be told setup is done every time they run the skill.
|
||||
|
||||
**When first run is detected, detect your platform first:**
|
||||
|
||||
**If you do NOT have WebSearch capability (raw CLI):** Run the terminal-only setup flow below.
|
||||
**If you DO have WebSearch (Hermes):** Run the standard setup flow below.
|
||||
|
||||
---
|
||||
|
||||
### Terminal-Only / Non-WebSearch Setup Flow
|
||||
|
||||
Run environment detection first:
|
||||
```bash
|
||||
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" setup --terminal
|
||||
```
|
||||
|
||||
Read the JSON output. It tells you what's already configured. Display a status summary:
|
||||
|
||||
```
|
||||
👋 Welcome to last30days!
|
||||
|
||||
Detected:
|
||||
{✅ or ❌} yt-dlp (YouTube search)
|
||||
{✅ or ❌} X/Twitter ({method} configured)
|
||||
{✅ or ❌} ScrapeCreators (TikTok, Instagram, Reddit backup)
|
||||
{✅ or ❌} Web search ({backend} configured)
|
||||
```
|
||||
|
||||
Then for each missing item, offer setup in priority order:
|
||||
|
||||
1. **ScrapeCreators** (if not configured): "ScrapeCreators adds TikTok and Instagram search (plus a Reddit backup if public Reddit gets rate-limited). 10,000 free calls, no credit card. (No referrals, no kickbacks - we don't get a cut.)"
|
||||
- Option A: "ScrapeCreators via GitHub (recommended)" — Check if `gh` CLI was detected in the environment detection output above. If gh IS detected: description should say "Registers directly via GitHub CLI in ~2 seconds - no browser needed". Before running the command, display: "Registering via GitHub CLI..." If gh is NOT detected: description should say "Copies a one-time code to your clipboard and opens GitHub to authorize". Then run `"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" setup --github`, parse JSON output. Tries PAT first (if `gh` is installed), falls back to device flow which copies a one-time code to your clipboard and opens your browser. If `status` is `success`, write `SCRAPECREATORS_API_KEY=*** to .env.
|
||||
- Option B: "I have a key" — accept paste, write to .env
|
||||
- Option C: "Skip for now"
|
||||
|
||||
2. **X/Twitter** (if not configured): "X search finds tweets and conversations. To unlock X: add FROM_BROWSER=auto (reads browser cookies, free), XAI_API_KEY (no browser access, api.x.ai), or AUTH_TOKEN+CT0 (manual cookies)."
|
||||
- Option A: "I have an xAI API key" (recommended for servers — persistent, no expiry). Write XAI_API_KEY to .env.
|
||||
- Option B: "I have AUTH_TOKEN + CT0 from my browser" — accept both, write to .env
|
||||
- Option C: "Skip for now"
|
||||
|
||||
3. **YouTube** (if yt-dlp not found): "YouTube search needs yt-dlp. Run: `pip install yt-dlp`"
|
||||
|
||||
4. **Web search** (if no Brave/Exa/Serper key): "A web search key enables smarter results. Brave Search is free for 2,000 queries/month at brave.com/search/api"
|
||||
|
||||
After setup, write `SETUP_COMPLETE=true` to .env and proceed to research.
|
||||
|
||||
**Skip to "END OF FIRST-RUN WIZARD" below after completing the terminal-only flow.**
|
||||
|
||||
---
|
||||
|
||||
### Hermes Setup Flow (Standard)
|
||||
|
||||
**You MUST follow these steps IN ORDER. Do NOT skip ahead to the topic picker or research. The sequence is: (1) welcome text -> (2) setup modal -> (3) run setup if chosen -> (4) optional ScrapeCreators modal -> (5) topic picker. You MUST start at step 1.**
|
||||
|
||||
**Step 1: Display the following welcome text ONCE as a normal message (not blockquoted). Then IMMEDIATELY call AskUserQuestion - do NOT repeat any of the welcome text inside the AskUserQuestion call.**
|
||||
|
||||
Welcome to last30days!
|
||||
|
||||
I research any topic across Reddit, X, YouTube, and other sources - synthesizing what people are actually saying right now.
|
||||
|
||||
Auto setup gives you 5 core sources for free in 30 seconds:
|
||||
- X/Twitter - reads your x.com browser cookies to authenticate (not saved to disk). Chrome on macOS will prompt for Keychain access.
|
||||
- Reddit with comments - public JSON, no API key needed
|
||||
- YouTube search + transcripts - installs yt-dlp (open source, 190K+ GitHub stars)
|
||||
- Hacker News + Polymarket + GitHub (if `gh` CLI installed) - always on, zero config
|
||||
|
||||
Want TikTok and Instagram too? ScrapeCreators adds those (10,000 free calls, scrapecreators.com). No kickbacks, no affiliation.
|
||||
|
||||
**Then call AskUserQuestion with ONLY this question and these options - no additional text:**
|
||||
|
||||
Question: "How would you like to set up?"
|
||||
Options:
|
||||
- "Auto setup (~30 seconds) - scans browser cookies for X + installs yt-dlp for YouTube"
|
||||
- "Manual setup - show me what to configure"
|
||||
- "Skip for now - Reddit (with comments), HN, Polymarket, GitHub (if gh installed), Web"
|
||||
|
||||
**If the user picks 1 (Auto setup):**
|
||||
|
||||
**Before running the setup command, get cookie consent:**
|
||||
|
||||
Check if `BROWSER_CONSENT=true` already exists in `~/.config/last30days/.env`. If it does, skip the consent prompt and run setup directly.
|
||||
|
||||
If `BROWSER_CONSENT=true` is NOT present, **call AskUserQuestion:**
|
||||
Question: "Auto setup will scan your browser for x.com cookies to authenticate X search. Cookies are read live, not saved to disk. Chrome on macOS will prompt for Keychain access. OK to proceed?"
|
||||
Options:
|
||||
- "Yes, scan my cookies for X" - Run setup as normal. Append `BROWSER_CONSENT=true` to .env after setup completes.
|
||||
- "Skip X, just set up YouTube" - Run setup with YouTube only (install yt-dlp). Do not scan cookies.
|
||||
- "I have an xAI API key instead" - Ask them to paste it, write XAI_API_KEY to .env. Then install yt-dlp.
|
||||
|
||||
Run the setup subcommand:
|
||||
```bash
|
||||
cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py setup
|
||||
```
|
||||
Show the user the results (what cookies were found, whether yt-dlp was installed).
|
||||
|
||||
**Then show the optional ScrapeCreators offer (plain text, then modal):**
|
||||
|
||||
Want TikTok and Instagram too? ScrapeCreators adds those platforms - 10,000 free calls, no credit card. It also serves as a Reddit backup if public Reddit ever gets rate-limited.
|
||||
|
||||
**Before showing the ScrapeCreators modal, check for `gh` CLI:** Run `which gh` via Bash silently. Store the result as gh_available (true if found, false if not).
|
||||
|
||||
**Call AskUserQuestion:**
|
||||
Question: "Want to add TikTok, Instagram, and Reddit backup via ScrapeCreators? (We don't get a cut.)"
|
||||
Options:
|
||||
- "ScrapeCreators via GitHub (fastest, recommended)" - If gh_available: description should say "Registers directly via GitHub CLI in ~2 seconds - no browser needed". If NOT gh_available: description should say "Copies a one-time code to your clipboard and opens GitHub to authorize". After the user selects this option: If gh_available, display "Registering via GitHub CLI..." before running the command. If NOT gh_available, display "I'll copy a one-time code to your clipboard and open GitHub. When GitHub asks for a device code, just paste (Cmd+V on Mac, Ctrl+V on Windows/Linux)." Then run `cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py setup --github` via Bash with a 5-minute timeout. This tries PAT auth first (if `gh` CLI is installed, zero browser needed), then falls back to GitHub device flow which copies a one-time code to your clipboard and opens GitHub in your browser. Parse the JSON stdout. If `status` is `success`, write `SCRAPECREATORS_API_KEY=*** to `~/.config/last30days/.env`. If `method` is `pat`, show: "You're in! Registered via GitHub CLI - zero browser needed. 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `method` is `device` and `clipboard_ok` is true, show: "You're in! (The authorization code was copied to your clipboard automatically.) 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `method` is `device` and `clipboard_ok` is false, show: "You're in! 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `status` is `timeout` or `error`, show: "GitHub auth didn't complete. No worries - you can sign up at scrapecreators.com instead or try again later." Then offer the web signup option.
|
||||
- "Open scrapecreators.com (Google sign-in)" - run `open https://scrapecreators.com` via Bash to open in the user's browser. Then ask them to paste the API key they get. When they paste it, write SCRAPECREATORS_API_KEY=*** to ~/.config/last30days/.env
|
||||
- "I have a key" - accept the key, write to .env
|
||||
- "Skip for now" - proceed without ScrapeCreators
|
||||
|
||||
**After SC key is saved (not if skipped), show the TikTok/Instagram opt-in:**
|
||||
|
||||
**Call AskUserQuestion:**
|
||||
Question: "Enable TikTok and Instagram search?"
|
||||
Options:
|
||||
- "Yes, enable TikTok + Instagram" - Write `TIKTOK_ENABLED=true` and `INSTAGRAM_ENABLED=true` to .env. Then show: "TikTok and Instagram are now enabled. You can disable them later by editing ~/.config/last30days/.env."
|
||||
- "No, skip for now" - proceed without enabling
|
||||
|
||||
**After setup completes, write `SETUP_COMPLETE=true` to .env.**
|
||||
|
||||
---
|
||||
|
||||
## END OF FIRST-RUN WIZARD
|
||||
|
||||
Proceed to Step 1.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Parse Topic
|
||||
|
||||
The user invoked: `last30days {QUERY}`
|
||||
|
||||
Extract the topic. If the query is empty or ambiguous, ask for clarification.
|
||||
|
||||
## Step 2: Execute Research
|
||||
|
||||
Run the research engine:
|
||||
|
||||
```bash
|
||||
cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py "{TOPIC}" --emit=compact --lookback-days=30
|
||||
```
|
||||
|
||||
Optional flags based on user request:
|
||||
- `--search=reddit,youtube,hackernews` - Specific sources only
|
||||
- `--days=7` - Shorter time range
|
||||
- `--deep` - Higher recall mode
|
||||
- `--save` - Save to ~/Documents/Last30Days/
|
||||
|
||||
## Step 3: Display Results
|
||||
|
||||
Show the research output to the user. The compact output includes:
|
||||
- Executive summary
|
||||
- Ranked evidence clusters with scores
|
||||
- Source statistics (upvotes, views, engagement)
|
||||
- Citations with URLs
|
||||
- Confidence levels and uncertainty notes
|
||||
|
||||
## Security & Permissions
|
||||
|
||||
**What this skill does:**
|
||||
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, and as a Reddit backup when public Reddit is unavailable (requires SCRAPECREATORS_API_KEY)
|
||||
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY)
|
||||
- Sends search queries to Twitter's GraphQL API (via optional user-provided AUTH_TOKEN/CT0 env vars — no browser session access) or xAI's API (`api.x.ai`) for X search
|
||||
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
|
||||
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
|
||||
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
||||
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 10,000 free API calls)
|
||||
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
||||
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
||||
- Stores research findings in local SQLite database (watchlist mode only)
|
||||
- Saves research briefings as .md files to ~/Documents/Last30Days/
|
||||
|
||||
**What this skill does NOT do:**
|
||||
- Does not post, like, or modify content on any platform
|
||||
- Does not access your Reddit, X, or YouTube accounts
|
||||
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
|
||||
- Does not log, cache, or write API keys to output files
|
||||
- Does not send data to any endpoint not listed above
|
||||
- Hacker News and Polymarket sources are always available (no API key, no binary dependency)
|
||||
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (10,000 free API calls, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable.
|
||||
- Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output
|
||||
|
||||
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
|
||||
|
||||
Review scripts before first use to verify behavior.
|
||||
+2
-9
@@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [3.0.0] - 2026-04-11
|
||||
## [3.0.0] - 2026-04
|
||||
|
||||
### Highlights
|
||||
|
||||
@@ -34,18 +34,10 @@ Intelligent search, fun judge, cross-source cluster merging, single-pass compari
|
||||
- Polymarket display shows % odds only; dollar volumes removed
|
||||
- 852 tests passing
|
||||
|
||||
### Fixed
|
||||
|
||||
- Marketplace validation: duplicate `name: last30days` collision in `skills/last30days/SKILL.md` caused strict validators to reject the plugin. Resolved by renaming the internal v3 architecture spec to `last30days-v3-spec` with `user-invocable: false`. Fixed in #214 (reported by @Cody-Coyote in #204).
|
||||
- Stale README link to the deleted `skills/last30days-v3/` path from the v3 directory rename. Fixed in #214.
|
||||
- OpenAI Codex CLI discoverability: added `.agents/skills/last30days/SKILL.md` as a real file (Codex's loader skips symlinked files) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now registers as `last30days:last30days` when Codex runs in a checkout of the repo. Fixed in #219 (inspired by @Jah-yee in #153 and @dannyshmueli on X).
|
||||
|
||||
### Contributors
|
||||
|
||||
- @j-sperling -- v3 engine architecture, Python pre-research brain
|
||||
- @hnshah -- Watchlist features
|
||||
- @Cody-Coyote -- Marketplace validation bug report (#204)
|
||||
- @Jah-yee -- Codex CLI integration inspiration (#153)
|
||||
|
||||
## [2.9.4] - 2026-03-06
|
||||
|
||||
@@ -189,6 +181,7 @@ Three headline features: watchlists for always-on bots, YouTube transcripts as a
|
||||
|
||||
### Credits
|
||||
|
||||
- @steipete -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
|
||||
- @galligan -- Marketplace plugin inspiration
|
||||
- @hutchins -- Pushed for YouTube feature
|
||||
|
||||
|
||||
-121
@@ -1,121 +0,0 @@
|
||||
# Hermes Setup Guide for last30days
|
||||
|
||||
This guide covers installing last30days on Hermes AI Agent.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Hermes installed** - See https://github.com/mercurial-tf/hermes
|
||||
2. **Python 3.12+** - `brew install python@3.12` or similar
|
||||
3. **yt-dlp** (optional, for YouTube) - `brew install yt-dlp`
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: Via sync.sh (Recommended)
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/mvanhorn/last30days-skill.git
|
||||
cd last30days-skill
|
||||
|
||||
# Run the sync script
|
||||
bash scripts/sync.sh
|
||||
```
|
||||
|
||||
This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days/`
|
||||
|
||||
### Option 2: Manual Copy
|
||||
|
||||
```bash
|
||||
# Create directory
|
||||
mkdir -p ~/.hermes/skills/research/last30days
|
||||
|
||||
# Copy files
|
||||
cp -r scripts ~/.hermes/skills/research/last30days/
|
||||
cp .hermes-plugin/SKILL.md ~/.hermes/skills/research/last30days/
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
In Hermes, invoke with:
|
||||
|
||||
```
|
||||
last30days "your research topic"
|
||||
```
|
||||
|
||||
Or with options:
|
||||
```
|
||||
last30days "best mechanical keyboards 2025" --search=reddit,youtube
|
||||
last30days "AI news" --days=7 --deep
|
||||
```
|
||||
|
||||
## First Run Setup
|
||||
|
||||
On first run, the skill will guide you through setup:
|
||||
|
||||
1. **Auto setup** (~30 seconds)
|
||||
- Scans browser cookies for X/Twitter
|
||||
- Checks/installs yt-dlp for YouTube
|
||||
- Configures free sources (Reddit, HN, Polymarket)
|
||||
|
||||
2. **Optional: ScrapeCreators**
|
||||
- Adds TikTok, Instagram, Reddit backup
|
||||
- 10,000 free API calls
|
||||
- Sign up at scrapecreators.com
|
||||
|
||||
3. **Optional: API Keys**
|
||||
- XAI_API_KEY for X/Twitter (alternative to browser cookies)
|
||||
- BRAVE_API_KEY for web search
|
||||
|
||||
## Available Sources
|
||||
|
||||
### Free (No API Key)
|
||||
- **Reddit** - Public discussions and comments
|
||||
- **Hacker News** - Tech discussions via Algolia
|
||||
- **Polymarket** - Prediction markets
|
||||
- **YouTube** - Search and transcripts (requires yt-dlp)
|
||||
|
||||
### Requires API Key
|
||||
- **X/Twitter** - xAI API key or browser cookies
|
||||
- **TikTok** - ScrapeCreators API
|
||||
- **Instagram** - ScrapeCreators API
|
||||
- **Web Search** - Brave Search API
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Python not found
|
||||
```bash
|
||||
# Find Python 3.12+
|
||||
which python3.12 python3.13 python3.14
|
||||
|
||||
# If not installed
|
||||
brew install python@3.12
|
||||
```
|
||||
|
||||
### yt-dlp not found
|
||||
```bash
|
||||
brew install yt-dlp
|
||||
# or
|
||||
pip install yt-dlp
|
||||
```
|
||||
|
||||
### Check what's configured
|
||||
```bash
|
||||
cd ~/.hermes/skills/research/last30days
|
||||
python3.12 scripts/last30days.py --diagnose
|
||||
```
|
||||
|
||||
## Updating
|
||||
|
||||
To update to the latest version:
|
||||
|
||||
```bash
|
||||
cd last30days-skill
|
||||
git pull
|
||||
bash scripts/sync.sh
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
- Original repo: https://github.com/mvanhorn/last30days-skill
|
||||
- Hermes: https://github.com/mercurial-tf/hermes
|
||||
- Issues: Please report in the original repo
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
|
||||
|
||||
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
|
||||
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days-v3/SKILL.md](skills/last30days-v3/SKILL.md), which is the source of truth for the latest command and setup behavior.
|
||||
|
||||
Claude Code:
|
||||
```
|
||||
@@ -24,12 +24,6 @@ OpenClaw:
|
||||
clawhub install last30days-official
|
||||
```
|
||||
|
||||
Hermes:
|
||||
```
|
||||
# The skill auto-deploys when you run sync.sh
|
||||
# Or manually copy to ~/.hermes/skills/research/last30days/
|
||||
```
|
||||
|
||||
Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
|
||||
|
||||
---
|
||||
|
||||
@@ -59,7 +59,7 @@ metadata:
|
||||
- clawhub
|
||||
---
|
||||
|
||||
# last30days v3.0.0: Research Any Topic from the Last 30 Days
|
||||
# last30days v2.9.5: Research Any Topic from the Last 30 Days
|
||||
|
||||
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `~/Documents/Last30Days/`. X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
|
||||
|
||||
@@ -375,7 +375,7 @@ Common patterns:
|
||||
- Always active: Reddit, Hacker News, Polymarket
|
||||
- If gh CLI is installed (check `which gh`): add GitHub
|
||||
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set: add X
|
||||
- If yt-dlp is installed (check `which yt-dlp`): add YouTube
|
||||
- If yt-dlp is installed (check `which yt-dlp`): add YouTube AND Podcasts
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram
|
||||
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads
|
||||
@@ -615,6 +615,27 @@ Store as `RESOLVED_IG_CREATORS`.
|
||||
|
||||
Store as `RESOLVED_YT_QUERIES`.
|
||||
|
||||
**6. Podcast channels** — **INFER 6-12 YouTube podcast channel @handles from topic knowledge.** Think in two dimensions:
|
||||
|
||||
1. **Domain podcasts** — What YouTube podcasts focus on this topic's domain?
|
||||
- Hip-hop/music → `DrinkChamps,JoeBuddenTV,BreakfastClubPower1051FM,OfficialFlagrant`
|
||||
- Tech/AI/startups → `lexfridman,DwarkeshPatel,AllInPod,MyFirstMillionPod,LennysPodcast`
|
||||
- Business/finance → `AcquiredFM,InvestLikeTheBest,PatrickBoyleOnFinance,PropGPod`
|
||||
- Sports → `PatMcAfeeShowOfficial,ShannonSharpe,ClubShayShay`
|
||||
- Culture/celebs → `joerogan,CallHerDaddy,ClubShayShay`
|
||||
- Knitting/crafts → `FruityKnitting,VeryPinkKnits,GroceryGirlsKnit`
|
||||
|
||||
2. **Cross-domain podcasts** — What popular interview/deep-dive podcasts might cover this topic even if it's not their main focus?
|
||||
- Business-adjacent topics → `AcquiredFM,InvestLikeTheBest` (company deep dives)
|
||||
- Tech-adjacent topics → `lexfridman,AllInPod` (broad tech interviews)
|
||||
- Culture-adjacent topics → `joerogan,OfficialFlagrant` (celebrity interviews)
|
||||
|
||||
**Rationale:** The engine uses these channels for transcript-first discovery. Even if the topic isn't in an episode title, it may be discussed within the episode. Acquired's "The NFL" episode mentions Taylor Swift 18 times, ESPN 117 times — invisible to YouTube search but found by transcript scanning.
|
||||
|
||||
**Handle accuracy:** Return your best guess at the exact @handle. If wrong, the engine falls back to a search-based lookup. Don't stress the exact spelling — `@AcquiredFM`, `@lexfridman`, `@joerogan` work; `@FLAGRANT` fails but falls back to find `@OfficialFlagrant`.
|
||||
|
||||
Store as `RESOLVED_PODCAST_CHANNELS` (comma-separated, no @ prefix).
|
||||
|
||||
**Concrete examples:**
|
||||
|
||||
| Topic | WebSearches needed | Reddit subs | TikTok hashtags | TikTok creators | IG creators | YT queries |
|
||||
@@ -635,6 +656,7 @@ Resolved:
|
||||
- Reddit: r/{sub1}, r/{sub2}, r/{sub3}
|
||||
- TikTok: #{hashtag1}, #{hashtag2}
|
||||
- YouTube: {query1}, {query2}
|
||||
- Podcasts: @{channel1}, @{channel2}, @{channel3}
|
||||
```
|
||||
|
||||
Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful.
|
||||
@@ -760,6 +782,7 @@ fi
|
||||
- `--ig-creators={RESOLVED_IG_CREATORS}` (from Step 0.55)
|
||||
- `--github-user={RESOLVED_GITHUB_USER}` (from Step 0.5b, person topics only)
|
||||
- `--github-repo={RESOLVED_GITHUB_REPOS}` (from Step 0.5c, product/project topics only)
|
||||
- `--podcast-channels={RESOLVED_PODCAST_CHANNELS}` (from Step 0.55, 6-12 @handles)
|
||||
- Omit any flag where the value was not resolved (empty).
|
||||
|
||||
**If you skipped Steps 0.55 and 0.75 (no WebSearch -- OpenClaw, Codex, etc.), add:**
|
||||
|
||||
@@ -0,0 +1,319 @@
|
||||
---
|
||||
title: "feat: YouTube podcast source with transcript-first discovery"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-04-10
|
||||
---
|
||||
|
||||
# feat: YouTube podcast source with transcript-first discovery
|
||||
|
||||
## Overview
|
||||
|
||||
Add a "podcasts" source to last30days that discovers podcast content on YouTube by scanning transcripts, not searching titles. The LLM planner resolves topic-relevant podcast channels (e.g., "NVIDIA" -> Acquired, Lex Fridman, Dwarkesh Patel, All-In). The engine fetches recent episodes from those channels, downloads their auto-captions (no video download), and greps for the search topic. Episodes with 5+ topic mentions become podcast results with transcript highlights.
|
||||
|
||||
This finds content invisible to any search engine. Acquired's "The NFL" episode mentions Taylor Swift 18 times, ESPN 117 times, Netflix 102 times - none in the title. A Dwarkesh Patel episode titled "The single biggest bottleneck to scaling AI compute" contains 156 mentions of NVIDIA. No YouTube search finds these. Transcript scanning does.
|
||||
|
||||
Zero new API keys. Zero new dependencies. Reuses existing yt-dlp + transcript pipeline. Podcasts get their own identity in stats and synthesis.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
YouTube captures a lot of podcast content, but it's mixed with news clips, reaction videos, and shorts. The general YouTube search treats a 2:24:55 Drink Champs interview the same as a 0:30 TMZ clip. Worse, the highest-value podcast content is often invisible to search entirely because the topic is discussed within an episode titled something else.
|
||||
|
||||
Two insights make this solvable:
|
||||
1. Podcast episodes are identifiable by duration (>20 minutes) and channel.
|
||||
2. YouTube auto-captions are free, downloadable without the video (~7 seconds per episode via yt-dlp), and searchable. Transcript scanning discovers content that title-based search cannot.
|
||||
|
||||
The LLM already resolves subreddits and X handles per topic. Podcast channels are the same pattern.
|
||||
|
||||
## Requirements Trace
|
||||
|
||||
- R1. LLM resolves topic-relevant podcast YouTube channels dynamically (no hardcoded list)
|
||||
- R2. Engine scans recent episode transcripts for the search topic, not just titles
|
||||
- R3. Podcast results get their own source identity with own stats line and synthesis treatment
|
||||
- R4. Reuses existing yt-dlp transcript pipeline (no new dependencies)
|
||||
- R5. Does not duplicate regular YouTube results (dedup by video ID in fusion)
|
||||
- R6. Channel resolution works in both the agent layer (SKILL.md) and the Python planner
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
- Not building a new API integration (reuses yt-dlp entirely)
|
||||
- Not adding PodcastIndex, AssemblyAI, or any podcast-specific API
|
||||
- Not changing how the regular YouTube source works
|
||||
- Not building a podcast channel database
|
||||
- Channels that can't be resolved are skipped silently (graceful degradation)
|
||||
|
||||
## Context & Research
|
||||
|
||||
### Relevant Code and Patterns
|
||||
|
||||
- `scripts/lib/youtube_yt.py` - YouTube search + transcript pipeline. Key functions: `search_youtube()`, `fetch_transcripts()`, `extract_transcript_highlights()`
|
||||
- `scripts/lib/youtube_yt.py` - `--write-auto-sub --skip-download` fetches captions without downloading video
|
||||
- Step 0.55 in `SKILL.md` - subreddit resolution pattern (WebSearch + LLM knowledge -> `--subreddits=`)
|
||||
- `scripts/lib/pipeline.py` - source dispatch via if/elif chain in `_retrieve_stream()`, 4-point registration pattern
|
||||
- `scripts/lib/normalize.py` - `_normalize_youtube()` handles transcript data, reusable for podcasts
|
||||
- `scripts/lib/signals.py` - `SOURCE_QUALITY` dict (YouTube is 0.85)
|
||||
- `scripts/lib/planner.py` - `QueryPlan` schema, `SOURCE_CAPABILITIES` dict
|
||||
|
||||
### Proof of Concept Results (2026-04-10)
|
||||
|
||||
**Transcript-first discovery test:** Fetched auto-captions for 5 recent Acquired episodes (35 seconds total, no video download). Grepped for topics not in any episode title:
|
||||
|
||||
| Topic | Mentions | Episode title | Discoverable by search? |
|
||||
|-------|----------|---------------|------------------------|
|
||||
| ESPN | 117 | The NFL | No |
|
||||
| Super Bowl | 108 | The NFL | No |
|
||||
| Netflix | 102 | The NFL | No |
|
||||
| Amazon | 87 | The NFL | No |
|
||||
| Costco | 63 | The NFL / others | No |
|
||||
| Disney | 48 | The NFL | No |
|
||||
| LVMH | 27 | Formula 1 / others | No |
|
||||
| Taylor Swift | 18 | The NFL | No |
|
||||
|
||||
**Full E2E test (topic: NVIDIA, 4 channels):** LLM resolved Acquired, Lex Fridman, Dwarkesh Patel, All-In. Scanned 14 episodes. Results:
|
||||
|
||||
| Podcast | Episode | NVIDIA mentions | Title mentions NVIDIA? |
|
||||
|---------|---------|----------------|----------------------|
|
||||
| Lex Fridman | Jensen Huang interview | 159 | Yes |
|
||||
| Dwarkesh Patel | Dylan Patel: AI compute bottleneck | 156 | No |
|
||||
| Acquired | 10 Years (w/ Michael Lewis) | 24 | No |
|
||||
| All-In | SpaceX IPO, Iran, Quantum... | 6 | No |
|
||||
|
||||
3 of 4 hits are invisible to YouTube search. The Dylan Patel episode (156 mentions!) is entirely about NVIDIA's GPU supply chain but the title never says "NVIDIA."
|
||||
|
||||
**Channel handle resolution test:** LLM resolves podcast name + @handle guess. Engine tries @handle first (fast), falls back to `ytsearch1:` if wrong. Tested across 12 channels (tech, hip-hop, knitting): 11/12 resolved on first @handle attempt, 12/12 with fallback. Even niche channels (Fruity Knitting, Grocery Girls Knit, Roxanne Richardson) resolved correctly.
|
||||
|
||||
**Rate limit test:** 4 channels x 3-4 episodes = 14 caption fetches took ~2 minutes sequential. Parallelized with 4 workers: ~30-40 seconds. No YouTube throttling observed. Runs concurrently with Reddit/X/everything else in a 3-minute research run.
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Transcript-first discovery, not title/search-based:** The core innovation. Instead of searching YouTube for `{topic} {podcast_name}` (which only finds episodes titled about the topic), we fetch captions from recent episodes and grep for the topic. This discovers hidden mentions. The approach is validated by POC data showing 3/4 NVIDIA hits were invisible to search.
|
||||
|
||||
- **LLM-resolved channels, not hardcoded:** The LLM planner (agent layer or Python Gemini/OpenAI) resolves 6-12 channels per topic using two-dimensional reasoning: (1) domain podcasts that focus on the topic's area, (2) cross-domain podcasts that might cover it. Tested: the LLM correctly resolved channels for NVIDIA (tech), Kanye (hip-hop), and knitting (craft) - including niche channels like Fruity Knitting and Grocery Girls Knit. Three resolution paths mirror the existing planner architecture:
|
||||
- Path 1: Agent layer (SKILL.md with WebSearch) resolves channels in Step 0.55
|
||||
- Path 2: Python planner (Gemini/OpenAI) generates channels as a `podcast_channels` field in the QueryPlan
|
||||
- Path 3: Fallback (no LLM) uses a small default list of ~5 broad-appeal channels
|
||||
|
||||
- **Handle-first channel resolution with search fallback:** The LLM returns both the podcast name and its best guess at the @handle. The engine tries the @handle first (instant, 92% success rate in testing). If the handle fails, it falls back to `ytsearch1:"{podcast name}" podcast full episode` to find the channel URL. Channels that can't be resolved either way are skipped silently.
|
||||
|
||||
- **New source module wrapping YouTube functions:** `podcast_yt.py` imports `fetch_transcripts()` and `extract_transcript_highlights()` from `youtube_yt.py`. It adds the channel-fetching, caption-scanning, and mention-counting logic. This keeps the regular YouTube source untouched and gives podcasts their own pipeline identity.
|
||||
|
||||
- **Duration filter >= 1200 seconds (20 minutes):** Eliminates clips, shorts, and news segments. Tested empirically - only full podcast episodes survive this filter.
|
||||
|
||||
- **SOURCE_QUALITY: 0.88 (above YouTube's 0.85):** Podcast episodes contain long-form expert discussion with full context. The quality bonus ensures podcast results rank above equivalent YouTube clips when both exist.
|
||||
|
||||
- **Mention count threshold: 5+:** Episodes with fewer than 5 topic mentions are noise (passing references). 5+ indicates substantive discussion. Tested: Taylor Swift at 18 mentions in the NFL episode is substantive discussion of her impact on viewership. "Apple" at 3 mentions in a random episode is just name-dropping.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Resolved During Planning
|
||||
|
||||
- **Can yt-dlp fetch captions without downloading video?** Yes. `yt-dlp --write-auto-sub --sub-lang en --skip-download --sub-format vtt` fetches only the subtitle file. ~7 seconds per episode, ~2MB per 4-hour episode.
|
||||
- **Will this double-count YouTube content?** No. Fusion deduplicates by item ID. Both sources use `yt_{video_id}` format.
|
||||
- **Can LLMs resolve niche podcast channels?** Yes. Tested with knitting: Fruity Knitting, VeryPink Knits, Grocery Girls Knit, Roxanne Richardson all resolved correctly via @handle.
|
||||
- **What about rate limits?** 14 caption fetches across 4 channels showed no throttling. Running in parallel with 4 workers keeps total time under 40 seconds. yt-dlp doesn't use the YouTube Data API (no quota).
|
||||
- **How does the LLM know which podcasts to pick?** Two-dimensional prompt: (1) "What YouTube podcasts focus on {topic's domain}?" and (2) "What popular interview/deep-dive podcasts have likely discussed {topic}?" The LLM returns channel names + @handle guesses.
|
||||
|
||||
### Deferred to Implementation
|
||||
|
||||
- **Exact duration threshold:** Starting with 1200s (20 min). May tune to 900s (15 min) if testing shows missed content.
|
||||
- **Mention count threshold tuning:** Starting with 5. May need per-source calibration (a 30-minute podcast with 5 mentions is denser than a 4-hour one with 5 mentions).
|
||||
- **Caption language handling:** Starting with English (`--sub-lang en`). Multilingual support deferred.
|
||||
- **Parallel worker count:** Starting with 4 workers. May tune based on YouTube throttling behavior at scale.
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
> *This illustrates the intended approach and is directional guidance for review, not implementation specification.*
|
||||
|
||||
```
|
||||
PODCAST DISCOVERY FLOW:
|
||||
|
||||
User query: "NVIDIA"
|
||||
|
|
||||
LLM planner resolves podcast channels:
|
||||
"NVIDIA is a tech/AI company. Domain podcasts: none specific.
|
||||
Cross-domain: Acquired (@AcquiredFM), Lex Fridman (@lexfridman),
|
||||
Dwarkesh Patel (@DwarkeshPatel), All-In (@AllInPod)"
|
||||
|
|
||||
Engine receives: --podcast-channels=AcquiredFM,lexfridman,DwarkeshPatel,AllInPod
|
||||
|
|
||||
For each channel (parallel, 4 workers):
|
||||
|
|
||||
[1] Resolve @handle -> channel URL
|
||||
Try: https://youtube.com/@AcquiredFM/videos
|
||||
If fail: ytsearch1:"Acquired podcast full episode" -> extract channel_url
|
||||
If fail: skip channel
|
||||
|
|
||||
[2] Fetch last 3 episode IDs + metadata (duration, date, title)
|
||||
yt-dlp --flat-playlist --playlist-end 3
|
||||
|
|
||||
[3] Filter: duration >= 1200s AND upload_date in date range
|
||||
|
|
||||
[4] For each surviving episode:
|
||||
Fetch auto-captions: yt-dlp --write-auto-sub --skip-download
|
||||
Grep captions for "nvidia" (case-insensitive)
|
||||
If mentions >= 5: HIT - extract transcript highlights around mentions
|
||||
|
|
||||
Merge all hits, deduplicate by video_id
|
||||
Score: mention_count * log(views)
|
||||
Return as source="podcasts" items with transcript_snippet + mention_count
|
||||
```
|
||||
|
||||
## Implementation Units
|
||||
|
||||
- [ ] **Unit 1: Podcast transcript-scan module**
|
||||
|
||||
**Goal:** Create `scripts/lib/podcast_yt.py` with the channel-fetching, caption-scanning, mention-counting pipeline. Returns podcast episodes discovered via transcript scanning.
|
||||
|
||||
**Requirements:** R2, R3, R4
|
||||
|
||||
**Dependencies:** None (youtube_yt.py already exists)
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/lib/podcast_yt.py`
|
||||
- Test: `tests/test_podcast_yt.py`
|
||||
|
||||
**Approach:**
|
||||
- `search_podcast_youtube(topic, from_date, to_date, depth, channels)`:
|
||||
- For each channel handle (in parallel via ThreadPoolExecutor, max 4 workers):
|
||||
1. Resolve handle to channel URL (try @handle first, search fallback)
|
||||
2. Fetch last N episode IDs + metadata via `yt-dlp --flat-playlist --playlist-end N`
|
||||
3. Filter: `duration >= 1200` and `upload_date` within date range
|
||||
4. Fetch auto-captions via `yt-dlp --write-auto-sub --skip-download --sub-lang en`
|
||||
5. Grep captions for topic keywords (case-insensitive). Count mentions.
|
||||
6. If mentions >= MENTION_THRESHOLD: include as hit. Extract transcript highlights around mentions using `extract_transcript_highlights()` from `youtube_yt`.
|
||||
- Merge results, deduplicate by video_id
|
||||
- Score: `mention_count * log(views + 1)`
|
||||
- Skip channels that can't be resolved or have no recent episodes
|
||||
|
||||
- `resolve_channel(handle)`: Try `@{handle}` URL first. If 404, search `ytsearch1:"{handle}" podcast full episode`, extract channel_url. Return channel_url or None.
|
||||
|
||||
- EPISODES_PER_CHANNEL: quick=2, default=3, deep=4
|
||||
- MENTION_THRESHOLD: 5
|
||||
- RESULTS_CAP: quick=4, default=8, deep=20
|
||||
|
||||
**Patterns to follow:**
|
||||
- `scripts/lib/youtube_yt.py` `search_and_transcribe()` for search-then-enrich flow
|
||||
- `scripts/lib/youtube_yt.py` `extract_transcript_highlights()` for highlight extraction
|
||||
- `scripts/lib/hackernews.py` for clean module structure with `_log()`, `DEPTH_CONFIG`
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path (hidden mention): topic "Taylor Swift", channels=["AcquiredFM"] -> scans NFL episode, finds 18 mentions, returns episode with highlights about Taylor Swift's NFL viewership impact
|
||||
- Happy path (title match): topic "kanye west", channels=["RevoltTV"] -> scans Kanye interview, finds 500+ mentions, returns with highlights
|
||||
- Happy path (scoring): episode with 156 mentions and 205K views scores higher than one with 6 mentions and 145K views
|
||||
- Happy path (handle resolution): @AcquiredFM resolves directly. @SomeWrongHandle fails, search fallback finds correct channel.
|
||||
- Edge case: topic "quantum computing" has <5 mentions in all episodes -> returns empty (threshold not met)
|
||||
- Edge case: @handle doesn't exist AND search fallback fails -> channel skipped silently, other channels still scanned
|
||||
- Edge case: channel has no episodes in date range -> skipped
|
||||
- Edge case: episode has no auto-captions available -> skipped with log warning
|
||||
- Error path: yt-dlp not installed -> returns empty items with log warning
|
||||
- Error path: caption download times out -> skip that episode, continue
|
||||
|
||||
**Verification:**
|
||||
- Discovers episodes where topic is discussed but not in the title (Acquired/NFL/Taylor Swift)
|
||||
- Also discovers episodes where topic IS the subject (via same transcript scan)
|
||||
- All returned items have duration >= 1200
|
||||
- Each item has: video_id, title, channel, url, date, duration, engagement, transcript_snippet, mention_count
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 2: Pipeline integration**
|
||||
|
||||
**Goal:** Register "podcasts" as a new source in pipeline, normalizer, signals, planner, env, and render.
|
||||
|
||||
**Requirements:** R3, R5, R6
|
||||
|
||||
**Dependencies:** Unit 1
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/lib/pipeline.py` (import, MOCK_AVAILABLE_SOURCES, available_sources, _retrieve_stream)
|
||||
- Modify: `scripts/lib/normalize.py` (add normalizer - reuse `_normalize_youtube` with source override)
|
||||
- Modify: `scripts/lib/signals.py` (add SOURCE_QUALITY: 0.88)
|
||||
- Modify: `scripts/lib/planner.py` (add SOURCE_CAPABILITIES, extend QueryPlan schema with `podcast_channels` field, add prompt guidance for LLM channel resolution)
|
||||
- Modify: `scripts/lib/env.py` (add is_podcast_yt_available - checks yt-dlp installed + "podcasts" in INCLUDE_SOURCES)
|
||||
- Modify: `scripts/lib/render.py` (add SOURCE_LABELS: "podcasts" -> "Podcasts")
|
||||
- Test: `tests/test_podcast_yt.py` (pipeline dispatch test)
|
||||
|
||||
**Approach:**
|
||||
- Availability: yt-dlp installed + "podcasts" in INCLUDE_SOURCES. No API key needed.
|
||||
- SOURCE_CAPABILITIES: `{"podcasts": {"discussion", "longform", "expert", "interview"}}`
|
||||
- Normalizer: reuse `_normalize_youtube` via lambda wrapper, override source to "podcasts". Add `mention_count` to metadata.
|
||||
- CLI flag: `--podcast-channels=handle1,handle2,...` parsed from args
|
||||
- Planner: extend QueryPlan with `podcast_channels: list[str]`. Prompt guidance for LLM: "List 6-12 YouTube podcast channel @handles that would discuss this topic. Think in two dimensions: (1) domain podcasts that focus on this area, (2) popular cross-domain interview/deep-dive podcasts that might cover it. Return @handles. If unsure of exact handle, return your best guess."
|
||||
- Planner: include "podcasts" source for general/opinion/comparison intents
|
||||
- Dedup: podcast items use `yt_{video_id}` ID format (same as YouTube). Fusion dedup handles collisions.
|
||||
|
||||
**Patterns to follow:**
|
||||
- 4-point pipeline registration (same as all sources)
|
||||
- `_normalize_youtube` reuse via lambda (like tiktok/instagram share `_normalize_shortform_video`)
|
||||
- `scripts/lib/env.py` INCLUDE_SOURCES opt-in pattern
|
||||
|
||||
**Test scenarios:**
|
||||
- Happy path: "podcasts" in available_sources when yt-dlp installed + INCLUDE_SOURCES contains "podcasts"
|
||||
- Happy path: pipeline dispatches to podcast_yt.search_podcast_youtube when source="podcasts"
|
||||
- Edge case: yt-dlp not installed -> podcasts not available
|
||||
- Edge case: "podcasts" not in INCLUDE_SOURCES -> not available even with yt-dlp
|
||||
- Integration: podcast video_id collides with YouTube result -> fusion deduplicates, keeps higher score
|
||||
|
||||
**Verification:**
|
||||
- `python3 scripts/last30days.py "NVIDIA" --podcast-channels=AcquiredFM,lexfridman` returns podcast results
|
||||
- Stats output shows "Podcasts" line separate from "YouTube"
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 3: SKILL.md podcast channel resolution + synthesis**
|
||||
|
||||
**Goal:** Add podcast channel resolution to Step 0.55 and podcast-specific synthesis guidance to the Judge Agent section.
|
||||
|
||||
**Requirements:** R1, R3, R6
|
||||
|
||||
**Dependencies:** Unit 2
|
||||
|
||||
**Files:**
|
||||
- Modify: `SKILL.md`
|
||||
|
||||
**Approach:**
|
||||
- **Step 0.55 addition:** Add "Resolve podcast channels" alongside subreddit, X handle, and TikTok resolution. The agent resolves 6-12 @handles using two-dimensional reasoning (domain + cross-domain). For niche topics, supplement with `WebSearch("{TOPIC} podcast YouTube channel")`. Display resolved channels: "Podcasts: @AcquiredFM, @lexfridman, @DrinkChamps". Pass as `--podcast-channels=AcquiredFM,lexfridman,DrinkChamps`.
|
||||
|
||||
- **Step 0.75 addition:** Add "podcasts" to available sources list. Include in primary subquery sources.
|
||||
|
||||
- **Synthesis guidance addition:** "For podcasts: lead with the guest's name and the podcast name. Quote transcript highlights as direct quotes with speaker attribution. Podcast content represents considered opinion, not hot takes - a 2-hour interview has more nuance than a tweet. When both a podcast and a YouTube clip cover the same topic, prefer the podcast's longer-form analysis."
|
||||
|
||||
- **Stats format:** `├─ 🎙️ Podcasts: {N} episodes │ {N} views │ {N} with transcripts`
|
||||
|
||||
- **INCLUDE_SOURCES:** Add "podcasts" as an option. Note in setup: "Requires yt-dlp (already installed if YouTube works). No API key needed."
|
||||
|
||||
- **Invitation section:** Reference podcast episodes in follow-up suggestions ("Want me to pull more from that Lex Fridman episode?")
|
||||
|
||||
**Patterns to follow:**
|
||||
- Step 0.55 subreddit resolution pattern
|
||||
- Source-specific synthesis guidance (YouTube highlights, Reddit top comments)
|
||||
|
||||
**Test scenarios:**
|
||||
- Test expectation: none - SKILL.md is an instruction document. Verification is manual E2E.
|
||||
|
||||
**Verification:**
|
||||
- `/last30days NVIDIA` resolves tech podcast channels and passes them to engine
|
||||
- `/last30days Kanye West` resolves hip-hop podcast channels
|
||||
- `/last30days knitting` resolves craft podcast channels (Fruity Knitting, etc.)
|
||||
- Stats show 🎙️ Podcasts line. Synthesis quotes podcast content with speaker attribution.
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| LLM guesses wrong @handle | Handle-first resolution with search fallback. 92% first-attempt success in testing, 100% with fallback. Wrong handles fail fast and skip silently. |
|
||||
| Transcript scanning adds latency | Runs in parallel with all other sources. 4 channels x 3 episodes = ~30-40s parallelized. Invisible in a 3-minute research run. |
|
||||
| Topic mentions below threshold (lots of misses) | LLM picks channels likely to discuss the topic. When it picks well, hit rate is high (4/14 episodes in NVIDIA test). Misses cost ~7s per episode in wasted caption download - acceptable. |
|
||||
| YouTube throttles caption downloads | 14 sequential downloads showed no throttling. Capping at 4 parallel workers adds safety margin. If throttled, degrade gracefully (fewer episodes scanned). |
|
||||
| Niche topics have no relevant podcast channels | LLM returns fewer channels (3-4 instead of 10-12). If none can be resolved, podcast source returns empty. Other sources (Reddit, X, YouTube) still run. |
|
||||
| Same video in both YouTube and podcast results | Fusion deduplicates by `yt_{video_id}`. Podcast version gets 0.88 quality score vs YouTube's 0.85, so podcast version wins dedup. |
|
||||
|
||||
## Sources & References
|
||||
|
||||
- POC: transcript scan of 5 Acquired episodes found ESPN (117), Netflix (102), Taylor Swift (18), LVMH (27) - all invisible to search
|
||||
- POC: E2E NVIDIA test across 4 channels found 5 hits, 3 invisible to search (including 156-mention Dwarkesh Patel episode)
|
||||
- POC: handle resolution tested 12 channels (tech, hip-hop, knitting) - 11/12 first-attempt, 12/12 with fallback
|
||||
- Related code: `scripts/lib/youtube_yt.py`, `scripts/lib/pipeline.py`, `scripts/lib/hackernews.py`
|
||||
- Pattern: SKILL.md Step 0.55 subreddit resolution
|
||||
- yt-dlp docs: https://github.com/yt-dlp/yt-dlp
|
||||
- Acquired FM: https://www.youtube.com/@AcquiredFM
|
||||
@@ -1,42 +0,0 @@
|
||||
[
|
||||
{
|
||||
"topic": "OpenClaw vs NanoClaw vs ZeroClaw",
|
||||
"query_type": "comparison",
|
||||
"rationale": "Multi-entity extraction, 3-way split across AI agent frameworks."
|
||||
},
|
||||
{
|
||||
"topic": "how to set up a GLP-1 supplement routine",
|
||||
"query_type": "how_to",
|
||||
"rationale": "Trending health topic. Tests non-tech how_to."
|
||||
},
|
||||
{
|
||||
"topic": "2026 March Madness",
|
||||
"query_type": "breaking_news",
|
||||
"rationale": "Live sporting event. Tests broad breaking news recall."
|
||||
},
|
||||
{
|
||||
"topic": "best budget noise cancelling headphones 2026",
|
||||
"query_type": "product",
|
||||
"rationale": "Evergreen consumer query. Tests product review aggregation."
|
||||
},
|
||||
{
|
||||
"topic": "thoughts on OpenAI Codex pricing",
|
||||
"query_type": "opinion",
|
||||
"rationale": "Active developer debate. Tests opinion mining."
|
||||
},
|
||||
{
|
||||
"topic": "odds of US recession 2026",
|
||||
"query_type": "prediction",
|
||||
"rationale": "Major macro topic. Tests prediction market + news synthesis."
|
||||
},
|
||||
{
|
||||
"topic": "what is retrieval augmented generation",
|
||||
"query_type": "concept",
|
||||
"rationale": "Widely discussed AI concept. Tests explanation quality."
|
||||
},
|
||||
{
|
||||
"topic": "Google Wiz acquisition price and timeline",
|
||||
"query_type": "factual",
|
||||
"rationale": "Completed event ($32B). Tests factual precision."
|
||||
}
|
||||
]
|
||||
+62
-73
@@ -1,86 +1,75 @@
|
||||
The AI world reinvents itself every month. This skill keeps you current.
|
||||
|
||||
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
|
||||
|
||||
## v3 is the intelligent search release
|
||||
|
||||
v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
|
||||
|
||||
Type "OpenClaw" and v3 resolves @steipete, r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags before a single API call fires. Type "Peter Steinberger" and it resolves his X handle and GitHub profile, switches to person mode, and shows what he shipped this month at 85% merge rate across 22 PRs. None of that was on Google.
|
||||
|
||||
## Headline features
|
||||
|
||||
### Intelligent pre-research
|
||||
|
||||
The killer feature. A new Python pre-research brain resolves X handles, GitHub repos, subreddits, TikTok hashtags, and YouTube channels before searching. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags, all resolved before a single API call.
|
||||
|
||||
### Best Takes
|
||||
|
||||
A second LLM judge scores every result for humor, wit, and virality alongside relevance. Every brief now ends with a Best Takes section surfacing the cleverest one-liners and most viral quotes. The Reddit and X people are funny, and the old engine buried their best stuff.
|
||||
|
||||
### Cross-source cluster merging
|
||||
|
||||
When the same story hits Reddit, X, and YouTube, v3 merges them into one cluster instead of three duplicates. Entity-based overlap detection catches matches even when the titles use different words.
|
||||
|
||||
### Single-pass comparisons
|
||||
|
||||
"X vs Y" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides at once. Same depth, 3 minutes.
|
||||
|
||||
### GitHub person-mode and project-mode
|
||||
|
||||
When the topic is a person, the engine switches from keyword search to author-scoped queries. PR velocity, top repos by stars, release notes for what shipped this month, woven into the narrative alongside X posts and Reddit threads.
|
||||
|
||||
When the topic is a project, it pulls live star counts, READMEs, releases, and top issues from the GitHub API. No stale blog posts.
|
||||
|
||||
### ELI5 mode
|
||||
|
||||
Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations, just clearer. Say "eli5 off" to go back.
|
||||
|
||||
### 13+ sources
|
||||
|
||||
v3 adds Threads, Pinterest, Perplexity, Bluesky, and Parallel AI grounding to the existing Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and Web lineup. Perplexity Deep Research (`--deep-research`) gives you 50+ citation reports for serious investigation.
|
||||
|
||||
### Per-author cap and entity disambiguation
|
||||
|
||||
Max 3 items per author prevents single-voice dominance. Synthesis trusts resolved handles over fuzzy keyword matches.
|
||||
|
||||
## Install
|
||||
|
||||
Claude Code:
|
||||
|
||||
```
|
||||
/plugin marketplace add mvanhorn/last30days-skill
|
||||
```
|
||||
|
||||
OpenClaw:
|
||||
|
||||
```
|
||||
clawhub install last30days-official
|
||||
```
|
||||
|
||||
OpenAI Codex CLI: run `codex` from a checkout of this repo and v3's skill at `.agents/skills/last30days/SKILL.md` will be discovered automatically. Or copy `SKILL.md` to `~/.agents/skills/last30days/SKILL.md` for a global install.
|
||||
|
||||
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
|
||||
`/last30days` researches your topic across **Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web** from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
|
||||
|
||||
## v3 Community
|
||||
|
||||
v3 was shaped by community contributors whose PRs and issues inspired core features. Their code wasn't merged directly (v3 was a ground-up rewrite), but their ideas drove what shipped.
|
||||
v3 was shaped by community contributors whose PRs and issues inspired core features. Their code wasn't merged directly (v3 was a ground-up rewrite), but their ideas drove what shipped. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the full list.
|
||||
|
||||
Thanks to @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, and @hnshah. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the full list.
|
||||
Thanks to @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, and @hnshah.
|
||||
|
||||
Contributors who shaped the release itself:
|
||||
## What's New in v2.9.1
|
||||
|
||||
- @Jah-yee (#153) surfaced the need for a real Codex CLI integration, which shipped in #219
|
||||
- @Cody-Coyote (#204) reported the marketplace validation bug that needed fixing before v3 could ship cleanly
|
||||
- @dannyshmueli pushed for v3 and Codex family support publicly on X
|
||||
**Auto-save to ~/Documents/Last30Days/.** Every run now saves the complete research briefing - synthesis, stats, and follow-up suggestions - as a topic-named `.md` file to your Documents folder. Build a personal research library without lifting a finger. Inspired by [@devin_explores](https://x.com/devin_explores) who was already doing this manually.
|
||||
|
||||
Full Added / Changed / Fixed detail lives in [CHANGELOG.md](CHANGELOG.md) under `[3.0.0]`.
|
||||
## Three Headline Features in v2.9
|
||||
|
||||
## Earlier contributors
|
||||
**1. ScrapeCreators Reddit as default.** One `SCRAPECREATORS_API_KEY` now covers Reddit, TikTok, and Instagram - three sources, one key. No more `OPENAI_API_KEY` required for Reddit search. Faster, more reliable, and simpler to configure.
|
||||
|
||||
From the v1 and v2 lineage:
|
||||
**2. Smart subreddit discovery.** Relevance-weighted scoring replaces pure frequency count. Each candidate subreddit is scored by `frequency x recency x topic-word match`, and a `UTILITY_SUBS` blocklist filters noise subs like r/tipofmytongue. Search "Claude Code skills" and get r/ClaudeAI, r/ClaudeCode, r/openclaw - not generic programming subs.
|
||||
|
||||
- [@galligan](https://github.com/galligan) for marketplace plugin inspiration
|
||||
- [@hutchins](https://x.com/hutchins) for pushing the YouTube feature
|
||||
**3. Top comments elevated.** The best comment on each Reddit thread now carries a 10% weight in engagement scoring and displays prominently with upvote counts. Reddit's value is in the comments - now the skill surfaces them.
|
||||
|
||||
30 days of research. 30 seconds of work. Thirteen sources. Zero stale prompts.
|
||||
Plus: **Instagram Reels** (v2.8), **Polymarket prediction markets** (v2.5), **YouTube transcripts** (v2.1), **bundled X search** - no external CLI needed.
|
||||
|
||||
## Beta Test Results (v2.9)
|
||||
|
||||
| Topic | Time | Threads | Discovered Subreddits |
|
||||
|-------|------|---------|----------------------|
|
||||
| Claude Code skills | 77.1s | 99 | r/ClaudeAI, r/ClaudeCode, r/openclaw |
|
||||
| Kanye West | 71.7s | 84 | r/hiphopheads, r/NFCWestMemeWar, r/Kanye |
|
||||
| Anthropic odds | 68.0s | 65 | r/Anthropic, r/ClaudeAI, r/OpenAI |
|
||||
| Best rap songs lately | 68.9s | 114 | r/BestofRedditorUpdates, r/rap, r/TeenageRapFans |
|
||||
| Nano Banana Pro | 66.6s | 99 | r/GeminiAI, r/nanobanana2pro, r/macbookpro |
|
||||
|
||||
## What's New
|
||||
|
||||
### Added
|
||||
- ScrapeCreators Reddit backend with keyword search and subreddit discovery
|
||||
- Smart subreddit discovery with relevance-weighted scoring
|
||||
- Utility subreddit blocklist (`UTILITY_SUBS`)
|
||||
- Top comment scoring (10% engagement weight) and prominent rendering
|
||||
- Comment excerpts increased to 400 chars, insights raised to 10
|
||||
|
||||
### Changed
|
||||
- `primaryEnv` → `SCRAPECREATORS_API_KEY` (one key for Reddit, TikTok, Instagram)
|
||||
- Reddit engagement scoring: `0.55/0.40/0.05` → `0.50/0.35/0.05/0.10`
|
||||
- SKILL.md synthesis instructions emphasize quoting top comments
|
||||
|
||||
### Fixed
|
||||
- Utility sub noise in subreddit discovery
|
||||
- Reddit no longer requires `OPENAI_API_KEY`
|
||||
|
||||
## New Contributors
|
||||
|
||||
- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
|
||||
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
|
||||
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
|
||||
|
||||
## Credits
|
||||
|
||||
- [@steipete](https://github.com/steipete) -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
|
||||
- [@galligan](https://github.com/galligan) -- Marketplace plugin inspiration
|
||||
- [@hutchins](https://x.com/hutchins) -- Pushed for YouTube feature
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Claude Code
|
||||
git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
|
||||
|
||||
# Codex CLI
|
||||
git clone https://github.com/mvanhorn/last30days-skill.git ~/.agents/skills/last30days
|
||||
```
|
||||
|
||||
30 days of research. 30 seconds of work. Eight sources. Zero stale prompts.
|
||||
|
||||
@@ -165,18 +165,12 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
|
||||
parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
|
||||
parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
|
||||
parser.add_argument(
|
||||
"--days",
|
||||
"--lookback-days",
|
||||
dest="lookback_days",
|
||||
type=int,
|
||||
default=30,
|
||||
help="Number of days to look back for research (default: 30, watchlist uses 90)",
|
||||
)
|
||||
parser.add_argument("--lookback-days", type=int, default=30, help="Number of days to look back for research (default: 30, watchlist uses 90)")
|
||||
parser.add_argument("--auto-resolve", action="store_true",
|
||||
help="Use web search to discover subreddits/handles before planning (for platforms without WebSearch)")
|
||||
parser.add_argument("--github-user", help="GitHub username for person-mode search (e.g., steipete)")
|
||||
parser.add_argument("--github-repo", help="Comma-separated owner/repo for project-mode search (e.g., openclaw/openclaw,paperclipai/paperclip)")
|
||||
parser.add_argument("--podcast-channels", help="Comma-separated YouTube @handles for podcast transcript scanning (e.g., AcquiredFM,lexfridman,DwarkeshPatel)")
|
||||
return parser
|
||||
|
||||
|
||||
@@ -315,6 +309,7 @@ def main() -> int:
|
||||
|
||||
github_user = args.github_user.lstrip("@").lower() if args.github_user else None
|
||||
github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None
|
||||
podcast_channels = [c.strip().lstrip("@") for c in args.podcast_channels.split(",") if c.strip()] if args.podcast_channels else None
|
||||
|
||||
# --deep-research: auto-enable perplexity source and set deep flag
|
||||
if args.deep_research:
|
||||
@@ -344,6 +339,7 @@ def main() -> int:
|
||||
lookback_days=args.lookback_days,
|
||||
github_user=github_user,
|
||||
github_repos=github_repos,
|
||||
podcast_channels=podcast_channels,
|
||||
)
|
||||
except Exception as exc:
|
||||
progress.end_processing()
|
||||
|
||||
@@ -460,7 +460,7 @@ def parse_bird_response(response: Dict[str, Any], query: str = "") -> List[Dict[
|
||||
"url": url,
|
||||
"author_handle": author_handle.lstrip("@"),
|
||||
"date": date,
|
||||
"engagement": engagement if any(v is not None for v in engagement.values()) else None,
|
||||
"engagement": engagement,
|
||||
"why_relevant": "", # Bird doesn't provide relevance explanations
|
||||
"relevance": _compute_relevance(query, str(tweet.get("text", ""))) if query else 0.7,
|
||||
}
|
||||
|
||||
@@ -53,6 +53,7 @@ def normalize_source_items(
|
||||
"xiaohongshu": _normalize_grounding,
|
||||
"github": _normalize_github,
|
||||
"perplexity": _normalize_grounding,
|
||||
"podcasts": lambda s, i, idx, fd, td: _normalize_youtube(s, i, idx, fd, td),
|
||||
}
|
||||
normalizer = normalizers.get(source)
|
||||
if normalizer is None:
|
||||
|
||||
@@ -40,6 +40,7 @@ from . import (
|
||||
xai_x,
|
||||
xiaohongshu_api,
|
||||
xquik,
|
||||
podcast_yt,
|
||||
youtube_yt,
|
||||
)
|
||||
from .cluster import cluster_candidates
|
||||
@@ -77,6 +78,7 @@ MOCK_AVAILABLE_SOURCES = [
|
||||
"github",
|
||||
"perplexity",
|
||||
"xquik",
|
||||
"podcasts",
|
||||
]
|
||||
|
||||
|
||||
@@ -122,6 +124,11 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
|
||||
available.append("pinterest")
|
||||
if env.is_xquik_available(config):
|
||||
available.append("xquik")
|
||||
# Podcasts: available whenever yt-dlp is installed (same as YouTube).
|
||||
# Opt-out only. The source returns empty when no channels are resolved,
|
||||
# so there's no cost to having it available.
|
||||
if podcast_yt.is_available():
|
||||
available.append("podcasts")
|
||||
return available
|
||||
|
||||
|
||||
@@ -177,6 +184,7 @@ def run(
|
||||
lookback_days: int = 30,
|
||||
github_user: str | None = None,
|
||||
github_repos: list[str] | None = None,
|
||||
podcast_channels: list[str] | None = None,
|
||||
) -> schema.Report:
|
||||
settings = DEPTH_SETTINGS[depth]
|
||||
requested_sources = normalize_requested_sources(requested_sources)
|
||||
@@ -318,6 +326,7 @@ def run(
|
||||
tiktok_hashtags=tiktok_hashtags,
|
||||
tiktok_creators=tiktok_creators,
|
||||
ig_creators=ig_creators,
|
||||
podcast_channels=podcast_channels,
|
||||
)
|
||||
] = (subquery, source)
|
||||
|
||||
@@ -348,6 +357,7 @@ def run(
|
||||
tiktok_hashtags=tiktok_hashtags,
|
||||
tiktok_creators=tiktok_creators,
|
||||
ig_creators=ig_creators,
|
||||
podcast_channels=podcast_channels,
|
||||
)
|
||||
except Exception as retry_exc:
|
||||
bundle.errors_by_source[source] = f"{exc} (retried once, still failed: {retry_exc})"
|
||||
@@ -787,6 +797,7 @@ def _retrieve_stream(
|
||||
tiktok_hashtags: list[str] | None = None,
|
||||
tiktok_creators: list[str] | None = None,
|
||||
ig_creators: list[str] | None = None,
|
||||
podcast_channels: list[str] | None = None,
|
||||
) -> tuple[list[dict], dict]:
|
||||
# Early exit if source was rate-limited by a sibling future
|
||||
if rate_limited_sources is not None and source in rate_limited_sources:
|
||||
@@ -874,6 +885,13 @@ def _retrieve_stream(
|
||||
sc_token = config.get("SCRAPECREATORS_API_KEY", "")
|
||||
youtube_yt.enrich_with_comments(items, token=sc_token)
|
||||
return items, {}
|
||||
if source == "podcasts":
|
||||
podcast_query = raw_topic or subquery.search_query
|
||||
result = podcast_yt.search_podcast_youtube(
|
||||
podcast_query, from_date, to_date,
|
||||
depth=depth, channels=podcast_channels,
|
||||
)
|
||||
return result.get("items", []), {}
|
||||
if source == "tiktok":
|
||||
# Use raw_topic so expand_tiktok_queries() generates diverse variants
|
||||
# from the original user topic, not the planner's narrowed search_query.
|
||||
|
||||
@@ -71,6 +71,7 @@ SOURCE_CAPABILITIES = {
|
||||
"github": {"discussion", "link"},
|
||||
"grounding": {"web", "reference", "link"},
|
||||
"perplexity": {"web", "reference", "analysis"},
|
||||
"podcasts": {"discussion", "video_longform", "expert"},
|
||||
}
|
||||
DEFAULT_INTENT_CAPABILITIES = {
|
||||
"comparison": {"discussion", "video", "web", "reference", "social", "link", "market"},
|
||||
|
||||
@@ -0,0 +1,430 @@
|
||||
"""YouTube podcast discovery via transcript scanning.
|
||||
|
||||
Discovers podcast content by fetching auto-captions from LLM-resolved
|
||||
YouTube podcast channels and grepping for the search topic. Finds content
|
||||
invisible to title-based search — e.g., Acquired's "The NFL" episode
|
||||
mentions Taylor Swift 18 times, ESPN 117 times, Netflix 102 times.
|
||||
|
||||
Uses yt-dlp for channel playlist fetch + caption download. No API keys.
|
||||
Reuses transcript highlight extraction from youtube_yt.
|
||||
"""
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from . import log
|
||||
|
||||
# How many recent episodes to scan per channel, by depth
|
||||
EPISODES_PER_CHANNEL = {
|
||||
"quick": 2,
|
||||
"default": 3,
|
||||
"deep": 4,
|
||||
}
|
||||
|
||||
# Minimum topic mentions in captions to count as a hit
|
||||
MENTION_THRESHOLD = 5
|
||||
|
||||
# Max total results to return
|
||||
RESULTS_CAP = {
|
||||
"quick": 4,
|
||||
"default": 8,
|
||||
"deep": 20,
|
||||
}
|
||||
|
||||
# Min duration in seconds to qualify as a podcast episode
|
||||
MIN_DURATION = 1200 # 20 minutes
|
||||
|
||||
|
||||
def _log(msg: str):
|
||||
log.source_log("Podcasts", msg, tty_only=False)
|
||||
|
||||
|
||||
def is_available() -> bool:
|
||||
"""Podcast source is available when yt-dlp is installed."""
|
||||
return shutil.which("yt-dlp") is not None
|
||||
|
||||
|
||||
def resolve_channel(handle: str) -> Optional[str]:
|
||||
"""Resolve a YouTube @handle to a channel URL.
|
||||
|
||||
Tries the @handle directly first (fast, ~92% success rate).
|
||||
Falls back to ytsearch1 if the handle doesn't resolve.
|
||||
|
||||
Returns the channel URL (https://www.youtube.com/channel/...) or None.
|
||||
"""
|
||||
# Try @handle directly - use the channel/videos URL format
|
||||
# yt-dlp can fetch from @handle URLs directly for playlist operations
|
||||
direct_url = f"https://www.youtube.com/@{handle}/videos"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["yt-dlp", "--playlist-end", "1",
|
||||
"--print", "%(channel_url)s",
|
||||
"--no-download", "--no-warnings", "--ignore-config", "--no-cookies-from-browser",
|
||||
direct_url],
|
||||
capture_output=True, text=True, timeout=20,
|
||||
)
|
||||
channel_url = result.stdout.strip().split("\n")[0].strip()
|
||||
if channel_url and channel_url.startswith("http"):
|
||||
_log(f"Resolved @{handle} -> {channel_url}")
|
||||
return channel_url
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# Fallback: search for the podcast
|
||||
_log(f"@{handle} not found, trying search fallback")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["yt-dlp", "--flat-playlist", "--playlist-end", "1",
|
||||
"--print", "%(channel_url)s",
|
||||
f'ytsearch1:"{handle}" podcast full episode'],
|
||||
capture_output=True, text=True, timeout=20,
|
||||
)
|
||||
channel_url = result.stdout.strip()
|
||||
if channel_url and channel_url.startswith("http"):
|
||||
_log(f"Search fallback resolved {handle} -> {channel_url}")
|
||||
return channel_url
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
_log(f"Could not resolve channel: {handle}")
|
||||
return None
|
||||
|
||||
|
||||
def _fetch_recent_episodes(
|
||||
channel_url: str,
|
||||
limit: int,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Fetch recent long-form episodes from a channel.
|
||||
|
||||
Returns list of dicts with video_id, title, channel, duration, date, views, likes.
|
||||
Filters to episodes with duration >= MIN_DURATION.
|
||||
"""
|
||||
import json as _json
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["yt-dlp", f"--playlist-end={limit + 2}",
|
||||
"--dump-json", "--no-download", "--no-warnings", "--ignore-config", "--no-cookies-from-browser",
|
||||
f"{channel_url}/videos"],
|
||||
capture_output=True, text=True, timeout=60,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
return []
|
||||
|
||||
episodes = []
|
||||
for line in result.stdout.strip().split("\n"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
video = _json.loads(line)
|
||||
except _json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
video_id = video.get("id", "")
|
||||
title = video.get("title", "")
|
||||
channel = video.get("channel", video.get("uploader", ""))
|
||||
duration = video.get("duration") or 0
|
||||
upload_date_raw = video.get("upload_date", "")
|
||||
views = video.get("view_count") or 0
|
||||
likes = video.get("like_count") or 0
|
||||
|
||||
# Convert YYYYMMDD to YYYY-MM-DD
|
||||
date_str = None
|
||||
if upload_date_raw and len(upload_date_raw) >= 8:
|
||||
date_str = f"{upload_date_raw[:4]}-{upload_date_raw[4:6]}-{upload_date_raw[6:8]}"
|
||||
|
||||
# Filter: duration >= MIN_DURATION
|
||||
if duration < MIN_DURATION:
|
||||
continue
|
||||
|
||||
# Filter: within date range (soft - keep if no date available)
|
||||
if date_str and (date_str < from_date or date_str > to_date):
|
||||
continue
|
||||
|
||||
episodes.append({
|
||||
"video_id": video_id,
|
||||
"title": title,
|
||||
"channel_name": channel,
|
||||
"duration": duration,
|
||||
"date": date_str,
|
||||
"views": views,
|
||||
"likes": likes,
|
||||
"url": f"https://www.youtube.com/watch?v={video_id}",
|
||||
})
|
||||
|
||||
return episodes[:limit]
|
||||
|
||||
|
||||
def _fetch_captions(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
"""Fetch auto-captions for a video. Returns caption text or None."""
|
||||
out_template = os.path.join(temp_dir, f"cap_{video_id}")
|
||||
try:
|
||||
subprocess.run(
|
||||
["yt-dlp", "--write-auto-sub", "--sub-lang", "en",
|
||||
"--skip-download", "--sub-format", "vtt",
|
||||
"-o", out_template,
|
||||
f"https://www.youtube.com/watch?v={video_id}"],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
return None
|
||||
|
||||
vtt_path = f"{out_template}.en.vtt"
|
||||
if not os.path.exists(vtt_path):
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(vtt_path, "r", encoding="utf-8") as f:
|
||||
text = f.read()
|
||||
os.remove(vtt_path)
|
||||
# Strip VTT formatting: timestamps, alignment, tags, duplicate lines
|
||||
# VTT auto-captions repeat lines as they scroll, so deduplicate
|
||||
lines = []
|
||||
prev_line = ""
|
||||
for line in text.split("\n"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith("WEBVTT") or line.startswith("Kind:") or line.startswith("Language:"):
|
||||
continue
|
||||
if re.match(r"^\d{2}:\d{2}:", line):
|
||||
continue
|
||||
if re.match(r"^NOTE\b", line):
|
||||
continue
|
||||
if "align:" in line or "position:" in line:
|
||||
continue
|
||||
# Strip inline VTT tags like <c>, </c>, timestamps
|
||||
cleaned = re.sub(r"<[^>]+>", "", line)
|
||||
cleaned = cleaned.strip()
|
||||
if cleaned and not re.match(r"^\d+$", cleaned) and cleaned != prev_line:
|
||||
lines.append(cleaned)
|
||||
prev_line = cleaned
|
||||
return " ".join(lines)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
_NOISE_WORDS = frozenset({
|
||||
"the", "a", "an", "of", "and", "or", "for", "to", "in", "on", "at",
|
||||
"best", "top", "new", "latest", "review", "news", "vs", "versus",
|
||||
"album", "song", "episode", "podcast", "interview", "this", "that",
|
||||
"what", "how", "why", "where", "when", "who",
|
||||
})
|
||||
|
||||
|
||||
def _extract_key_terms(topic: str) -> List[str]:
|
||||
"""Extract meaningful terms from topic for matching.
|
||||
|
||||
For "Kanye West Bully album" -> ["Kanye West", "Bully"] or similar.
|
||||
For single words, just returns the word.
|
||||
"""
|
||||
words = [w.strip() for w in topic.split() if w.strip()]
|
||||
# Remove noise words
|
||||
meaningful = [w for w in words if w.lower() not in _NOISE_WORDS and len(w) > 2]
|
||||
if not meaningful:
|
||||
return [topic.strip()]
|
||||
|
||||
# If the topic has 2+ meaningful words, also include the full phrase
|
||||
# and the first 2 words as a potential entity name
|
||||
terms = []
|
||||
if len(meaningful) >= 2:
|
||||
# Full phrase first (for exact entity matches like "Taylor Swift")
|
||||
terms.append(" ".join(meaningful[:2]))
|
||||
terms.extend(meaningful)
|
||||
return terms
|
||||
|
||||
|
||||
def _count_mentions(text: str, topic: str) -> int:
|
||||
"""Count case-insensitive topic mentions in text.
|
||||
|
||||
Uses the maximum mention count across key terms extracted from the topic.
|
||||
"Kanye West Bully album" -> max mentions of ["Kanye West", "Kanye", "West", "Bully"].
|
||||
This way, an episode mentioning "Kanye" 85 times counts as 85, not 0.
|
||||
"""
|
||||
text_lower = text.lower()
|
||||
terms = _extract_key_terms(topic)
|
||||
max_count = 0
|
||||
for term in terms:
|
||||
pattern = re.escape(term.lower())
|
||||
count = len(re.findall(pattern, text_lower))
|
||||
if count > max_count:
|
||||
max_count = count
|
||||
return max_count
|
||||
|
||||
|
||||
def _extract_mention_context(text: str, topic: str, max_excerpts: int = 3) -> List[str]:
|
||||
"""Extract text snippets around topic mentions for highlights."""
|
||||
words = text.split()
|
||||
topic_lower = topic.lower()
|
||||
excerpts = []
|
||||
|
||||
for i, word in enumerate(words):
|
||||
# Check if we're near a mention
|
||||
window = " ".join(words[max(0, i - 5):i + 15]).lower()
|
||||
if topic_lower in window and len(excerpts) < max_excerpts:
|
||||
start = max(0, i - 10)
|
||||
end = min(len(words), i + 30)
|
||||
excerpt = " ".join(words[start:end])
|
||||
# Avoid duplicate excerpts
|
||||
if not any(excerpt[:50] in e for e in excerpts):
|
||||
excerpts.append(excerpt)
|
||||
|
||||
return excerpts
|
||||
|
||||
|
||||
def _scan_channel(
|
||||
handle: str,
|
||||
topic: str,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
episodes_limit: int,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Scan a single channel's recent episodes for topic mentions.
|
||||
|
||||
Returns list of hit items with mention_count and transcript data.
|
||||
"""
|
||||
# Step 1: Resolve channel handle to URL
|
||||
channel_url = resolve_channel(handle)
|
||||
if not channel_url:
|
||||
return []
|
||||
|
||||
# Step 2: Fetch recent long-form episodes
|
||||
episodes = _fetch_recent_episodes(channel_url, episodes_limit, from_date, to_date)
|
||||
if not episodes:
|
||||
_log(f"No recent long-form episodes from {handle}")
|
||||
return []
|
||||
|
||||
_log(f"Scanning {len(episodes)} episodes from {handle}")
|
||||
|
||||
# Step 3: Fetch captions and grep for topic
|
||||
hits = []
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
for ep in episodes:
|
||||
caption_text = _fetch_captions(ep["video_id"], temp_dir)
|
||||
if not caption_text:
|
||||
continue
|
||||
|
||||
mention_count = _count_mentions(caption_text, topic)
|
||||
if mention_count < MENTION_THRESHOLD:
|
||||
continue
|
||||
|
||||
# Extract highlights around the mentions
|
||||
from .youtube_yt import extract_transcript_highlights
|
||||
highlights = extract_transcript_highlights(caption_text, topic, limit=5)
|
||||
mention_excerpts = _extract_mention_context(caption_text, topic)
|
||||
|
||||
# Cap transcript for storage
|
||||
words = caption_text.split()
|
||||
transcript_snippet = " ".join(words[:5000]) if len(words) > 5000 else caption_text
|
||||
|
||||
hits.append({
|
||||
"video_id": ep["video_id"],
|
||||
"title": ep["title"],
|
||||
"channel_name": ep["channel_name"],
|
||||
"url": ep["url"],
|
||||
"date": ep["date"],
|
||||
"duration": ep["duration"],
|
||||
"engagement": {
|
||||
"views": ep["views"],
|
||||
"likes": ep["likes"],
|
||||
},
|
||||
"mention_count": mention_count,
|
||||
"transcript_snippet": transcript_snippet,
|
||||
"transcript_highlights": highlights,
|
||||
"mention_excerpts": mention_excerpts,
|
||||
"relevance": min(1.0, mention_count / 50),
|
||||
"why_relevant": f"Podcast: {ep['channel_name']} - {ep['title'][:60]} ({mention_count} mentions)",
|
||||
})
|
||||
|
||||
_log(f" HIT: {ep['title'][:60]} ({mention_count} mentions)")
|
||||
|
||||
return hits
|
||||
|
||||
|
||||
def search_podcast_youtube(
|
||||
topic: str,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
depth: str = "default",
|
||||
channels: Optional[List[str]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Discover podcast content by scanning transcripts of resolved channels.
|
||||
|
||||
Args:
|
||||
topic: Search topic
|
||||
from_date: Start date (YYYY-MM-DD)
|
||||
to_date: End date (YYYY-MM-DD)
|
||||
depth: 'quick', 'default', or 'deep'
|
||||
channels: List of YouTube @handles to scan
|
||||
|
||||
Returns:
|
||||
Dict with 'items' list. Each item has transcript and mention data.
|
||||
"""
|
||||
if not is_available():
|
||||
_log("yt-dlp not installed")
|
||||
return {"items": [], "error": "yt-dlp not installed"}
|
||||
|
||||
if not channels:
|
||||
_log("No podcast channels provided")
|
||||
return {"items": []}
|
||||
|
||||
episodes_limit = EPISODES_PER_CHANNEL.get(depth, EPISODES_PER_CHANNEL["default"])
|
||||
results_cap = RESULTS_CAP.get(depth, RESULTS_CAP["default"])
|
||||
|
||||
_log(f"Scanning {len(channels)} podcast channels for '{topic}' (depth={depth}, {episodes_limit} eps/channel)")
|
||||
|
||||
# Scan channels in parallel
|
||||
all_hits: List[Dict[str, Any]] = []
|
||||
max_workers = min(4, len(channels))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
futures = {
|
||||
executor.submit(
|
||||
_scan_channel, handle, topic, from_date, to_date, episodes_limit,
|
||||
): handle
|
||||
for handle in channels
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
handle = futures[future]
|
||||
try:
|
||||
hits = future.result()
|
||||
all_hits.extend(hits)
|
||||
except Exception as exc:
|
||||
_log(f"Error scanning {handle}: {type(exc).__name__}: {exc}")
|
||||
|
||||
# Deduplicate by video_id
|
||||
seen = set()
|
||||
unique_hits = []
|
||||
for hit in all_hits:
|
||||
vid = hit["video_id"]
|
||||
if vid not in seen:
|
||||
seen.add(vid)
|
||||
unique_hits.append(hit)
|
||||
|
||||
# Score: mention_count * log(views + 1)
|
||||
for hit in unique_hits:
|
||||
views = hit["engagement"].get("views", 0)
|
||||
hit["_score"] = hit["mention_count"] * math.log(views + 1)
|
||||
|
||||
# Sort by score descending
|
||||
unique_hits.sort(key=lambda x: x["_score"], reverse=True)
|
||||
|
||||
# Cap results
|
||||
results = unique_hits[:results_cap]
|
||||
|
||||
# Clean up internal scoring field
|
||||
for hit in results:
|
||||
hit.pop("_score", None)
|
||||
|
||||
_log(f"Found {len(results)} podcast hits across {len(channels)} channels")
|
||||
return {"items": results}
|
||||
@@ -14,6 +14,7 @@ SOURCE_LABELS = {
|
||||
"x": "X",
|
||||
"github": "GitHub",
|
||||
"perplexity": "Perplexity",
|
||||
"podcasts": "Podcasts",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ SOURCE_QUALITY = {
|
||||
"polymarket": 0.5,
|
||||
"instagram": 0.58,
|
||||
"tiktok": 0.58,
|
||||
"podcasts": 0.88,
|
||||
}
|
||||
|
||||
|
||||
|
||||
+3
-62
@@ -24,7 +24,7 @@ sync_target() {
|
||||
|
||||
echo ""
|
||||
echo "--- Syncing to $target ---"
|
||||
mkdir -p "$target/scripts/lib"
|
||||
mkdir -p "$target/scripts/lib" "$target/variants/open/references"
|
||||
|
||||
cp "$skill_md" "$target/SKILL.md"
|
||||
|
||||
@@ -35,13 +35,7 @@ sync_target() {
|
||||
"$SRC/scripts/store.py" \
|
||||
"$target/scripts/"
|
||||
rsync -a "$SRC/scripts/lib/"*.py "$target/scripts/lib/"
|
||||
|
||||
# The OpenClaw variant lives in the private repo only. Skip cleanly when
|
||||
# running this script from the public repo where variants/open does not exist.
|
||||
if [ -d "$SRC/variants/open" ]; then
|
||||
mkdir -p "$target/variants/open/references"
|
||||
rsync -a "$SRC/variants/open/" "$target/variants/open/"
|
||||
fi
|
||||
rsync -a "$SRC/variants/open/" "$target/variants/open/"
|
||||
|
||||
if [ -d "$SRC/scripts/lib/vendor" ]; then
|
||||
rsync -a "$SRC/scripts/lib/vendor" "$target/scripts/lib/"
|
||||
@@ -69,60 +63,7 @@ for t in "${COMMON_TARGETS[@]}"; do
|
||||
sync_target "$t" "$SRC/SKILL.md"
|
||||
done
|
||||
|
||||
# Hermes sync: deploy to Hermes skills directory if it exists
|
||||
HERMES_TARGET="$HOME/.hermes/skills/research/last30days"
|
||||
if [ -d "$HOME/.hermes/skills/research" ]; then
|
||||
echo ""
|
||||
echo "--- Syncing to Hermes ---"
|
||||
mkdir -p "$HERMES_TARGET/scripts/lib"
|
||||
|
||||
# Use Hermes-specific SKILL.md if available, fallback to main
|
||||
if [ -f "$SRC/.hermes-plugin/SKILL.md" ]; then
|
||||
cp "$SRC/.hermes-plugin/SKILL.md" "$HERMES_TARGET/SKILL.md"
|
||||
else
|
||||
cp "$SRC/SKILL.md" "$HERMES_TARGET/SKILL.md"
|
||||
fi
|
||||
|
||||
rsync -a \
|
||||
"$SRC/scripts/last30days.py" \
|
||||
"$SRC/scripts/watchlist.py" \
|
||||
"$SRC/scripts/briefing.py" \
|
||||
"$SRC/scripts/store.py" \
|
||||
"$HERMES_TARGET/scripts/"
|
||||
rsync -a "$SRC/scripts/lib/"*.py "$HERMES_TARGET/scripts/lib/"
|
||||
|
||||
if [ -d "$SRC/scripts/lib/vendor" ]; then
|
||||
rsync -a "$SRC/scripts/lib/vendor" "$HERMES_TARGET/scripts/lib/"
|
||||
fi
|
||||
|
||||
if [ -d "$SRC/fixtures" ]; then
|
||||
mkdir -p "$HERMES_TARGET/fixtures"
|
||||
rsync -a "$SRC/fixtures/" "$HERMES_TARGET/fixtures/"
|
||||
fi
|
||||
|
||||
mod_count=$(ls "$HERMES_TARGET/scripts/lib/"*.py 2>/dev/null | wc -l | tr -d ' ')
|
||||
echo " Copied $mod_count modules to Hermes"
|
||||
|
||||
if (
|
||||
cd "$HERMES_TARGET/scripts" &&
|
||||
python3 -c "import briefing, store, watchlist; from lib import youtube_yt, bird_x, render, ui; print(' Import check: OK')"
|
||||
); then
|
||||
true
|
||||
else
|
||||
echo " Import check FAILED"
|
||||
fi
|
||||
fi
|
||||
|
||||
# OpenClaw sync only runs when the private-repo OpenClaw variant is present
|
||||
# in the source tree. The public repo does not ship variants/open (the variant
|
||||
# is sanitized via strip_for_openclaw.py and published separately from
|
||||
# last30days-skill-private).
|
||||
if [ -d "$SRC/variants/open" ]; then
|
||||
sync_target "$OPENCLAW_TARGET" "$SRC/variants/open/SKILL.md"
|
||||
else
|
||||
echo ""
|
||||
echo "Skipping OpenClaw target (no variants/open in this repo)"
|
||||
fi
|
||||
sync_target "$OPENCLAW_TARGET" "$SRC/variants/open/SKILL.md"
|
||||
|
||||
echo ""
|
||||
echo "Sync complete."
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
---
|
||||
name: last30days-v3-spec
|
||||
name: last30days
|
||||
version: "3.0.0"
|
||||
description: "Internal architecture spec for the v3 last30days runtime pipeline. Not user-invocable."
|
||||
description: "Multi-query social search with intelligent planning. Agent plans queries when possible, falls back to Gemini/OpenAI when not. Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web."
|
||||
argument-hint: "last30days codex vs claude code"
|
||||
allowed-tools: Bash, Read, Write, WebSearch
|
||||
homepage: https://github.com/mvanhorn/last30days-skill
|
||||
repository: https://github.com/mvanhorn/last30days-skill
|
||||
author: mvanhorn
|
||||
license: MIT
|
||||
user-invocable: false
|
||||
user-invocable: true
|
||||
---
|
||||
|
||||
# last30days v3.0.0
|
||||
@@ -86,6 +86,7 @@ fi
|
||||
- `yt-dlp` enables YouTube.
|
||||
- Planning and reranking fall back gracefully: Gemini -> OpenAI -> xAI -> deterministic/local.
|
||||
- Web retrieval stays within Brave/Serper dated results. Undated web hits are dropped.
|
||||
- For OpenClaw-specific watchlist, briefing, and history workflows, use `variants/open/SKILL.md`.
|
||||
|
||||
## Output model
|
||||
|
||||
|
||||
+1
-27
@@ -175,7 +175,7 @@ class TestVendoredBirdRuntime(unittest.TestCase):
|
||||
}
|
||||
]
|
||||
items = parse_bird_response(tweets, "test query")
|
||||
self.assertIsNone(items[0]["engagement"])
|
||||
self.assertIsNone(items[0]["engagement"]["likes"])
|
||||
|
||||
def test_fallback_to_second_key(self):
|
||||
tweets = [
|
||||
@@ -203,32 +203,6 @@ class TestVendoredBirdRuntime(unittest.TestCase):
|
||||
items = parse_bird_response(tweets, "test query")
|
||||
self.assertEqual(0, items[0]["engagement"]["likes"])
|
||||
|
||||
def test_engagement_none_when_all_fields_missing(self):
|
||||
"""All-None engagement dict should become None, not propagate."""
|
||||
tweets = [
|
||||
{
|
||||
"id": "1",
|
||||
"text": "test",
|
||||
"permanent_url": "https://x.com/u/status/1",
|
||||
}
|
||||
]
|
||||
items = parse_bird_response(tweets, "test query")
|
||||
self.assertIsNone(items[0]["engagement"])
|
||||
|
||||
def test_engagement_preserved_when_any_field_present(self):
|
||||
"""Engagement dict kept when at least one metric exists."""
|
||||
tweets = [
|
||||
{
|
||||
"id": "1",
|
||||
"text": "test",
|
||||
"permanent_url": "https://x.com/u/status/1",
|
||||
"likeCount": 5,
|
||||
}
|
||||
]
|
||||
items = parse_bird_response(tweets, "test query")
|
||||
self.assertIsNotNone(items[0]["engagement"])
|
||||
self.assertEqual(5, items[0]["engagement"]["likes"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -77,13 +77,6 @@ class CliV3Tests(unittest.TestCase):
|
||||
with self.assertRaises(SystemExit):
|
||||
cli.parse_search_flag(" , ")
|
||||
|
||||
def test_build_parser_accepts_days_alias_and_preserves_topic_tokens(self):
|
||||
parser = cli.build_parser()
|
||||
args, extra = parser.parse_known_args(["--days", "7", "biosecurity", "ai", "agents"])
|
||||
self.assertEqual(7, args.lookback_days)
|
||||
self.assertEqual(["biosecurity", "ai", "agents"], args.topic)
|
||||
self.assertEqual([], extra)
|
||||
|
||||
def test_ensure_supported_python_rejects_old_interpreter_with_actionable_error(self):
|
||||
stderr = io.StringIO()
|
||||
with redirect_stderr(stderr):
|
||||
|
||||
Reference in New Issue
Block a user