diff --git a/.hermes-plugin/SKILL.md b/.hermes-plugin/SKILL.md new file mode 100644 index 0000000..ca89da3 --- /dev/null +++ b/.hermes-plugin/SKILL.md @@ -0,0 +1,269 @@ +--- +name: last30days +version: "3.0.0" +description: "Multi-query social search with intelligent planning. Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web." +argument-hint: 'last30days AI video tools, last30days best noise cancelling headphones' +allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch +homepage: https://github.com/mvanhorn/last30days-skill +repository: https://github.com/mvanhorn/last30days-skill +author: mvanhorn +license: MIT +user-invocable: true +metadata: + hermes: + emoji: "📰" + tags: + - research + - deep-research + - reddit + - x + - twitter + - youtube + - tiktok + - instagram + - hackernews + - polymarket + - trends + - recency + - news + - citations + - multi-source + - social-media + - analysis + - web-search + requires: + env: + - SCRAPECREATORS_API_KEY + optionalEnv: + - OPENAI_API_KEY + - XAI_API_KEY + - OPENROUTER_API_KEY + - PARALLEL_API_KEY + - BRAVE_API_KEY + - APIFY_API_TOKEN + - AUTH_TOKEN + - CT0 + - BSKY_HANDLE + - BSKY_APP_PASSWORD + - TRUTHSOCIAL_TOKEN + bins: + - node + - python3 + primaryEnv: SCRAPECREATORS_API_KEY + files: + - "scripts/*" + homepage: https://github.com/mvanhorn/last30days-skill +--- + +# last30days v3.0.0: Research Any Topic from the Last 30 Days + +> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `~/Documents/Last30Days/`. X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section. + +Research ANY topic across Reddit, X, YouTube, and other sources. Surface what people are actually discussing, recommending, betting on, and debating right now. + +## Runtime Preflight + +Before running any `last30days.py` command in this skill, resolve a Python 3.12+ interpreter once and keep it in `LAST30DAYS_PYTHON`: + +```bash +for py in python3.14 python3.13 python3.12 python3; do + command -v "$py" >/dev/null 2>&1 || continue + "$py" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)' || continue + LAST30DAYS_PYTHON="$py" + break +done + +if [ -z "${LAST30DAYS_PYTHON:-}" ]; then + echo "ERROR: last30days v3 requires Python 3.12+. Install python3.12 or python3.13 and rerun." >&2 + exit 1 +fi +``` + +## Step 0: First-Run Setup Wizard + +**CRITICAL: ALWAYS execute Step 0 BEFORE Step 1, even if the user provided a topic.** If the user typed `last30days Mercer Island`, you MUST check for FIRST_RUN and present the wizard BEFORE running research. The topic "Mercer Island" is preserved — research runs immediately after the wizard completes. Do NOT skip the wizard because a topic was provided. The wizard takes 10 seconds and only runs once ever. + +To detect first run: check if `~/.config/last30days/.env` exists. If it does NOT exist, this is a first run. **Do NOT run any Bash commands or show any command output to detect this — just check the file existence silently.** If the file exists and contains `SETUP_COMPLETE=true`, skip this section **silently** and proceed to Step 1. **Do NOT say "Setup is complete" or any other status message — just move on.** The user doesn't need to be told setup is done every time they run the skill. + +**When first run is detected, detect your platform first:** + +**If you do NOT have WebSearch capability (raw CLI):** Run the terminal-only setup flow below. +**If you DO have WebSearch (Hermes):** Run the standard setup flow below. + +--- + +### Terminal-Only / Non-WebSearch Setup Flow + +Run environment detection first: +```bash +"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" setup --terminal +``` + +Read the JSON output. It tells you what's already configured. Display a status summary: + +``` +👋 Welcome to last30days! + +Detected: +{✅ or ❌} yt-dlp (YouTube search) +{✅ or ❌} X/Twitter ({method} configured) +{✅ or ❌} ScrapeCreators (TikTok, Instagram, Reddit backup) +{✅ or ❌} Web search ({backend} configured) +``` + +Then for each missing item, offer setup in priority order: + +1. **ScrapeCreators** (if not configured): "ScrapeCreators adds TikTok and Instagram search (plus a Reddit backup if public Reddit gets rate-limited). 10,000 free calls, no credit card. (No referrals, no kickbacks - we don't get a cut.)" + - Option A: "ScrapeCreators via GitHub (recommended)" — Check if `gh` CLI was detected in the environment detection output above. If gh IS detected: description should say "Registers directly via GitHub CLI in ~2 seconds - no browser needed". Before running the command, display: "Registering via GitHub CLI..." If gh is NOT detected: description should say "Copies a one-time code to your clipboard and opens GitHub to authorize". Then run `"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" setup --github`, parse JSON output. Tries PAT first (if `gh` is installed), falls back to device flow which copies a one-time code to your clipboard and opens your browser. If `status` is `success`, write `SCRAPECREATORS_API_KEY=*** to .env. + - Option B: "I have a key" — accept paste, write to .env + - Option C: "Skip for now" + +2. **X/Twitter** (if not configured): "X search finds tweets and conversations. To unlock X: add FROM_BROWSER=auto (reads browser cookies, free), XAI_API_KEY (no browser access, api.x.ai), or AUTH_TOKEN+CT0 (manual cookies)." + - Option A: "I have an xAI API key" (recommended for servers — persistent, no expiry). Write XAI_API_KEY to .env. + - Option B: "I have AUTH_TOKEN + CT0 from my browser" — accept both, write to .env + - Option C: "Skip for now" + +3. **YouTube** (if yt-dlp not found): "YouTube search needs yt-dlp. Run: `pip install yt-dlp`" + +4. **Web search** (if no Brave/Exa/Serper key): "A web search key enables smarter results. Brave Search is free for 2,000 queries/month at brave.com/search/api" + +After setup, write `SETUP_COMPLETE=true` to .env and proceed to research. + +**Skip to "END OF FIRST-RUN WIZARD" below after completing the terminal-only flow.** + +--- + +### Hermes Setup Flow (Standard) + +**You MUST follow these steps IN ORDER. Do NOT skip ahead to the topic picker or research. The sequence is: (1) welcome text -> (2) setup modal -> (3) run setup if chosen -> (4) optional ScrapeCreators modal -> (5) topic picker. You MUST start at step 1.** + +**Step 1: Display the following welcome text ONCE as a normal message (not blockquoted). Then IMMEDIATELY call AskUserQuestion - do NOT repeat any of the welcome text inside the AskUserQuestion call.** + +Welcome to last30days! + +I research any topic across Reddit, X, YouTube, and other sources - synthesizing what people are actually saying right now. + +Auto setup gives you 5 core sources for free in 30 seconds: +- X/Twitter - reads your x.com browser cookies to authenticate (not saved to disk). Chrome on macOS will prompt for Keychain access. +- Reddit with comments - public JSON, no API key needed +- YouTube search + transcripts - installs yt-dlp (open source, 190K+ GitHub stars) +- Hacker News + Polymarket + GitHub (if `gh` CLI installed) - always on, zero config + +Want TikTok and Instagram too? ScrapeCreators adds those (10,000 free calls, scrapecreators.com). No kickbacks, no affiliation. + +**Then call AskUserQuestion with ONLY this question and these options - no additional text:** + +Question: "How would you like to set up?" +Options: +- "Auto setup (~30 seconds) - scans browser cookies for X + installs yt-dlp for YouTube" +- "Manual setup - show me what to configure" +- "Skip for now - Reddit (with comments), HN, Polymarket, GitHub (if gh installed), Web" + +**If the user picks 1 (Auto setup):** + +**Before running the setup command, get cookie consent:** + +Check if `BROWSER_CONSENT=true` already exists in `~/.config/last30days/.env`. If it does, skip the consent prompt and run setup directly. + +If `BROWSER_CONSENT=true` is NOT present, **call AskUserQuestion:** +Question: "Auto setup will scan your browser for x.com cookies to authenticate X search. Cookies are read live, not saved to disk. Chrome on macOS will prompt for Keychain access. OK to proceed?" +Options: +- "Yes, scan my cookies for X" - Run setup as normal. Append `BROWSER_CONSENT=true` to .env after setup completes. +- "Skip X, just set up YouTube" - Run setup with YouTube only (install yt-dlp). Do not scan cookies. +- "I have an xAI API key instead" - Ask them to paste it, write XAI_API_KEY to .env. Then install yt-dlp. + +Run the setup subcommand: +```bash +cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py setup +``` +Show the user the results (what cookies were found, whether yt-dlp was installed). + +**Then show the optional ScrapeCreators offer (plain text, then modal):** + +Want TikTok and Instagram too? ScrapeCreators adds those platforms - 10,000 free calls, no credit card. It also serves as a Reddit backup if public Reddit ever gets rate-limited. + +**Before showing the ScrapeCreators modal, check for `gh` CLI:** Run `which gh` via Bash silently. Store the result as gh_available (true if found, false if not). + +**Call AskUserQuestion:** +Question: "Want to add TikTok, Instagram, and Reddit backup via ScrapeCreators? (We don't get a cut.)" +Options: +- "ScrapeCreators via GitHub (fastest, recommended)" - If gh_available: description should say "Registers directly via GitHub CLI in ~2 seconds - no browser needed". If NOT gh_available: description should say "Copies a one-time code to your clipboard and opens GitHub to authorize". After the user selects this option: If gh_available, display "Registering via GitHub CLI..." before running the command. If NOT gh_available, display "I'll copy a one-time code to your clipboard and open GitHub. When GitHub asks for a device code, just paste (Cmd+V on Mac, Ctrl+V on Windows/Linux)." Then run `cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py setup --github` via Bash with a 5-minute timeout. This tries PAT auth first (if `gh` CLI is installed, zero browser needed), then falls back to GitHub device flow which copies a one-time code to your clipboard and opens GitHub in your browser. Parse the JSON stdout. If `status` is `success`, write `SCRAPECREATORS_API_KEY=*** to `~/.config/last30days/.env`. If `method` is `pat`, show: "You're in! Registered via GitHub CLI - zero browser needed. 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `method` is `device` and `clipboard_ok` is true, show: "You're in! (The authorization code was copied to your clipboard automatically.) 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `method` is `device` and `clipboard_ok` is false, show: "You're in! 10,000 free calls. TikTok, Instagram, and Reddit backup are now active." If `status` is `timeout` or `error`, show: "GitHub auth didn't complete. No worries - you can sign up at scrapecreators.com instead or try again later." Then offer the web signup option. +- "Open scrapecreators.com (Google sign-in)" - run `open https://scrapecreators.com` via Bash to open in the user's browser. Then ask them to paste the API key they get. When they paste it, write SCRAPECREATORS_API_KEY=*** to ~/.config/last30days/.env +- "I have a key" - accept the key, write to .env +- "Skip for now" - proceed without ScrapeCreators + +**After SC key is saved (not if skipped), show the TikTok/Instagram opt-in:** + +**Call AskUserQuestion:** +Question: "Enable TikTok and Instagram search?" +Options: +- "Yes, enable TikTok + Instagram" - Write `TIKTOK_ENABLED=true` and `INSTAGRAM_ENABLED=true` to .env. Then show: "TikTok and Instagram are now enabled. You can disable them later by editing ~/.config/last30days/.env." +- "No, skip for now" - proceed without enabling + +**After setup completes, write `SETUP_COMPLETE=true` to .env.** + +--- + +## END OF FIRST-RUN WIZARD + +Proceed to Step 1. + +--- + +## Step 1: Parse Topic + +The user invoked: `last30days {QUERY}` + +Extract the topic. If the query is empty or ambiguous, ask for clarification. + +## Step 2: Execute Research + +Run the research engine: + +```bash +cd {SKILL_DIR} && "${LAST30DAYS_PYTHON}" scripts/last30days.py "{TOPIC}" --emit=compact --lookback-days=30 +``` + +Optional flags based on user request: +- `--search=reddit,youtube,hackernews` - Specific sources only +- `--days=7` - Shorter time range +- `--deep` - Higher recall mode +- `--save` - Save to ~/Documents/Last30Days/ + +## Step 3: Display Results + +Show the research output to the user. The compact output includes: +- Executive summary +- Ranked evidence clusters with scores +- Source statistics (upvotes, views, engagement) +- Citations with URLs +- Confidence levels and uncertainty notes + +## Security & Permissions + +**What this skill does:** +- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, and as a Reddit backup when public Reddit is unavailable (requires SCRAPECREATORS_API_KEY) +- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery (fallback if no SCRAPECREATORS_API_KEY) +- Sends search queries to Twitter's GraphQL API (via optional user-provided AUTH_TOKEN/CT0 env vars — no browser session access) or xAI's API (`api.x.ai`) for X search +- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth) +- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth) +- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data) +- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 10,000 free API calls) +- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search +- Fetches public Reddit thread data from `reddit.com` for engagement metrics +- Stores research findings in local SQLite database (watchlist mode only) +- Saves research briefings as .md files to ~/Documents/Last30Days/ + +**What this skill does NOT do:** +- Does not post, like, or modify content on any platform +- Does not access your Reddit, X, or YouTube accounts +- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.) +- Does not log, cache, or write API keys to output files +- Does not send data to any endpoint not listed above +- Hacker News and Polymarket sources are always available (no API key, no binary dependency) +- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (10,000 free API calls, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable. +- Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output + +**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed) + +Review scripts before first use to verify behavior. diff --git a/HERMES_SETUP.md b/HERMES_SETUP.md new file mode 100644 index 0000000..0e0e503 --- /dev/null +++ b/HERMES_SETUP.md @@ -0,0 +1,121 @@ +# Hermes Setup Guide for last30days + +This guide covers installing last30days on Hermes AI Agent. + +## Prerequisites + +1. **Hermes installed** - See https://github.com/mercurial-tf/hermes +2. **Python 3.12+** - `brew install python@3.12` or similar +3. **yt-dlp** (optional, for YouTube) - `brew install yt-dlp` + +## Installation + +### Option 1: Via sync.sh (Recommended) + +```bash +# Clone the repo +git clone https://github.com/mvanhorn/last30days-skill.git +cd last30days-skill + +# Run the sync script +bash scripts/sync.sh +``` + +This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days/` + +### Option 2: Manual Copy + +```bash +# Create directory +mkdir -p ~/.hermes/skills/research/last30days + +# Copy files +cp -r scripts ~/.hermes/skills/research/last30days/ +cp .hermes-plugin/SKILL.md ~/.hermes/skills/research/last30days/ +``` + +## Usage + +In Hermes, invoke with: + +``` +last30days "your research topic" +``` + +Or with options: +``` +last30days "best mechanical keyboards 2025" --search=reddit,youtube +last30days "AI news" --days=7 --deep +``` + +## First Run Setup + +On first run, the skill will guide you through setup: + +1. **Auto setup** (~30 seconds) + - Scans browser cookies for X/Twitter + - Checks/installs yt-dlp for YouTube + - Configures free sources (Reddit, HN, Polymarket) + +2. **Optional: ScrapeCreators** + - Adds TikTok, Instagram, Reddit backup + - 10,000 free API calls + - Sign up at scrapecreators.com + +3. **Optional: API Keys** + - XAI_API_KEY for X/Twitter (alternative to browser cookies) + - BRAVE_API_KEY for web search + +## Available Sources + +### Free (No API Key) +- **Reddit** - Public discussions and comments +- **Hacker News** - Tech discussions via Algolia +- **Polymarket** - Prediction markets +- **YouTube** - Search and transcripts (requires yt-dlp) + +### Requires API Key +- **X/Twitter** - xAI API key or browser cookies +- **TikTok** - ScrapeCreators API +- **Instagram** - ScrapeCreators API +- **Web Search** - Brave Search API + +## Troubleshooting + +### Python not found +```bash +# Find Python 3.12+ +which python3.12 python3.13 python3.14 + +# If not installed +brew install python@3.12 +``` + +### yt-dlp not found +```bash +brew install yt-dlp +# or +pip install yt-dlp +``` + +### Check what's configured +```bash +cd ~/.hermes/skills/research/last30days +python3.12 scripts/last30days.py --diagnose +``` + +## Updating + +To update to the latest version: + +```bash +cd last30days-skill +git pull +bash scripts/sync.sh +``` + +## Support + +- Original repo: https://github.com/mvanhorn/last30days-skill +- Hermes: https://github.com/mercurial-tf/hermes +- Issues: Please report in the original repo diff --git a/README.md b/README.md index 27ec62c..a21120a 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,12 @@ OpenClaw: clawhub install last30days-official ``` +Hermes: +``` +# The skill auto-deploys when you run sync.sh +# Or manually copy to ~/.hermes/skills/research/last30days/ +``` + Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds. --- diff --git a/scripts/sync.sh b/scripts/sync.sh index 6199d30..e182605 100755 --- a/scripts/sync.sh +++ b/scripts/sync.sh @@ -69,6 +69,50 @@ for t in "${COMMON_TARGETS[@]}"; do sync_target "$t" "$SRC/SKILL.md" done +# Hermes sync: deploy to Hermes skills directory if it exists +HERMES_TARGET="$HOME/.hermes/skills/research/last30days" +if [ -d "$HOME/.hermes/skills/research" ]; then + echo "" + echo "--- Syncing to Hermes ---" + mkdir -p "$HERMES_TARGET/scripts/lib" + + # Use Hermes-specific SKILL.md if available, fallback to main + if [ -f "$SRC/.hermes-plugin/SKILL.md" ]; then + cp "$SRC/.hermes-plugin/SKILL.md" "$HERMES_TARGET/SKILL.md" + else + cp "$SRC/SKILL.md" "$HERMES_TARGET/SKILL.md" + fi + + rsync -a \ + "$SRC/scripts/last30days.py" \ + "$SRC/scripts/watchlist.py" \ + "$SRC/scripts/briefing.py" \ + "$SRC/scripts/store.py" \ + "$HERMES_TARGET/scripts/" + rsync -a "$SRC/scripts/lib/"*.py "$HERMES_TARGET/scripts/lib/" + + if [ -d "$SRC/scripts/lib/vendor" ]; then + rsync -a "$SRC/scripts/lib/vendor" "$HERMES_TARGET/scripts/lib/" + fi + + if [ -d "$SRC/fixtures" ]; then + mkdir -p "$HERMES_TARGET/fixtures" + rsync -a "$SRC/fixtures/" "$HERMES_TARGET/fixtures/" + fi + + mod_count=$(ls "$HERMES_TARGET/scripts/lib/"*.py 2>/dev/null | wc -l | tr -d ' ') + echo " Copied $mod_count modules to Hermes" + + if ( + cd "$HERMES_TARGET/scripts" && + python3 -c "import briefing, store, watchlist; from lib import youtube_yt, bird_x, render, ui; print(' Import check: OK')" + ); then + true + else + echo " Import check FAILED" + fi +fi + # OpenClaw sync only runs when the private-repo OpenClaw variant is present # in the source tree. The public repo does not ship variants/open (the variant # is sanitized via strip_for_openclaw.py and published separately from