chore(release): v3.2.0 (#371 )

* chore(release): v3.2.0 Bumps plugin/marketplace/codex/pyproject versions from 3.1.1 to 3.2.0. Promotes the Unreleased CHANGELOG entries (--emit=html, Digg AI 1000 source) to the 3.2.0 release section. * chore(release): bump SKILL.md header and sync.sh path to 3.2.0 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
feat: add Digg AI 1000 as an opt-in source (#370 )
2026-05-09 19:29:16 -07:00 · 2026-05-09 19:05:41 -07:00 · 2026-05-02 11:30:22 -07:00 · 2026-04-26 14:16:14 -07:00 · 2026-04-25 14:17:47 -07:00 · 2026-04-25 14:17:17 -07:00
188 changed files with 3253 additions and 543 deletions
@@ -0,0 +1,20 @@
 {
  "name": "last30days-skill",
  "interface": {
    "displayName": "Last 30 Days"
  },
  "plugins": [
    {
      "name": "last30days",
      "source": {
        "source": "local",
        "path": "./"
      },
      "policy": {
        "installation": "AVAILABLE",
        "authentication": "ON_INSTALL"
      },
      "category": "Research"
    }
  ]
 }
@@ -1,16 +1,17 @@
 {
  "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
  "name": "last30days-skill",
  "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
  "owner": {
    "name": "Matt Van Horn",
    "url": "https://github.com/mvanhorn"
  },
  "metadata": {
    "description": "Marketplace hosting the Last 30 Days research plugin."
  },
  "plugins": [
    {
      "name": "last30days",
-      "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, and 5+ more sources.",
+      "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
-      "version": "3.0.9",
+      "version": "3.2.0",
      "author": {
        "name": "Matt Van Horn",
        "url": "https://github.com/mvanhorn"
@@ -1,6 +1,6 @@
 {
  "name": "last30days",
-  "version": "3.0.14",
+  "version": "3.2.0",
  "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
  "author": {
    "name": "Matt Van Horn",
@@ -10,6 +10,5 @@
  "homepage": "https://github.com/mvanhorn/last30days-skill",
  "repository": "https://github.com/mvanhorn/last30days-skill",
  "license": "MIT",
-  "keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"],
+  "keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"]
  "hooks": {}
 }
@@ -1,3 +1,43 @@
 {
-  "name": "last30days"
+  "name": "last30days",
  "version": "3.2.0",
  "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
  "author": {
    "name": "Matt Van Horn",
    "email": "mvanhorn@gmail.com",
    "url": "https://github.com/mvanhorn"
  },
  "homepage": "https://github.com/mvanhorn/last30days-skill",
  "repository": "https://github.com/mvanhorn/last30days-skill",
  "license": "MIT",
  "keywords": [
    "research",
    "reddit",
    "twitter",
    "youtube",
    "tiktok",
    "instagram",
    "trends",
    "polymarket",
    "github",
    "hacker-news"
  ],
  "skills": "./skills/",
  "interface": {
    "displayName": "Last 30 Days",
    "shortDescription": "Research recent discussion across social and web sources",
    "longDescription": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
    "developerName": "Matt Van Horn",
    "category": "Research",
    "capabilities": [
      "Interactive",
      "Read",
      "Write"
    ],
    "websiteURL": "https://github.com/mvanhorn/last30days-skill",
    "privacyPolicyURL": "https://docs.github.com/en/site-policy/privacy-policies/github-general-privacy-statement",
    "termsOfServiceURL": "https://docs.github.com/en/site-policy/github-terms/github-terms-of-service",
    "defaultPrompt": "Use Last 30 Days to research this topic from the last 30 days across Reddit, X, YouTube, and web.",
    "brandColor": "#FF6B35"
  }
 }
@@ -1,5 +1,6 @@
 # Exclude non-runtime files from `git archive` output.
-# Used by scripts/build-skill.sh to produce a claude.ai-upload-ready .skill file.
+# Used by skills/last30days/scripts/build-skill.sh to produce a
 # claude.ai-upload-ready .skill file from the canonical skills/last30days tree.
 # See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
 # Anthropic canonical skill-packaging excludes
@@ -32,9 +33,8 @@ release-notes.md   export-ignore
 CHANGELOG.md       export-ignore
 uv.lock            export-ignore
-# Platform adapters - skill-upload path is platform-agnostic
+# Platform adapters are kept in git archives because Claude Code and Codex
-.agents/         export-ignore
+# plugin installs use the same repository archive as their source payload.
 .codex-plugin/   export-ignore
 .hermes-plugin/  export-ignore
 # CI workflows - repo-only, not needed at skill runtime
@@ -0,0 +1,53 @@
 name: Bug Report
 description: Report a bug or unexpected behavior
 labels: [bug]
 body:
  - type: textarea
    id: summary
    attributes:
      label: Summary
      description: What happened?
      placeholder: Describe the bug in 1-2 sentences.
    validations:
      required: true
  - type: textarea
    id: repro
    attributes:
      label: Steps to Reproduce
      description: How can we reproduce this?
      placeholder: |
        1. Run `python3 scripts/last30days.py "topic" --emit compact`
        2. ...
    validations:
      required: true
  - type: textarea
    id: expected
    attributes:
      label: Expected Behavior
      description: What should have happened?
    validations:
      required: true
  - type: textarea
    id: traceback
    attributes:
      label: Error / Traceback
      description: Paste the full traceback or error output.
      render: text
  - type: dropdown
    id: install
    attributes:
      label: Install Method
      options:
        - Claude Code plugin
        - Gemini CLI extension
        - Codex plugin
        - Hermes skill
        - Manual (git clone)
        - Other
    validations:
      required: true
  - type: input
    id: os
    attributes:
      label: OS
      placeholder: macOS 15.4, Ubuntu 24.04, Windows 11, etc.
@@ -0,0 +1,24 @@
 name: Feature Request
 description: Suggest a new feature or improvement
 labels: [enhancement]
 body:
  - type: textarea
    id: problem
    attributes:
      label: Problem
      description: What problem does this solve?
      placeholder: When I try to ..., I can't ...
    validations:
      required: true
  - type: textarea
    id: solution
    attributes:
      label: Proposed Solution
      description: How should this work?
    validations:
      required: true
  - type: textarea
    id: alternatives
    attributes:
      label: Alternatives Considered
      description: Other approaches you thought of (optional).
@@ -0,0 +1,20 @@
 ## Summary
 <!-- What does this PR do? 1-3 sentences. -->
 ## Changes
 <!-- Bullet list of what changed. Reference files if helpful. -->
 -
 ## Testing
 <!-- How did you verify this works? -->
 - [ ] Ran `uv run python -m pytest -q --tb=short`
 - [ ] Ran `bash scripts/sync.sh` (if scripts/ changed)
 ## Related Issues
 <!-- Link issues: Fixes #123 or Relates to #456 -->
@@ -19,7 +19,7 @@ jobs:
      - name: Build .skill artifact
        run: |
-          bash scripts/build-skill.sh
+          bash skills/last30days/scripts/build-skill.sh
          test -f dist/last30days.skill
      - name: Create GitHub release
@@ -0,0 +1,26 @@
 name: Validate
 on:
  pull_request:
  push:
    branches:
      - main
 permissions:
  contents: read
 jobs:
  plugin-contract:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Install uv
        uses: astral-sh/setup-uv@v5
      - name: Set up Python
        run: uv python install 3.12
      - name: Run plugin contract tests
        run: uv run pytest tests/test_plugin_contract.py tests/test_version_consistency.py
@@ -0,0 +1 @@
@CLAUDE.md
@@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [Unreleased]
 ## [3.2.0] - 2026-05-09
 ### Added
 - Add `--emit=html` for shareable, print-friendly HTML research briefs.
 - **Digg AI 1000 source** (auto-enabled when `digg-pp-cli` is on PATH). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief as `[@handle](xUrl) via Digg AI 1000: ...` lines. Footer line: `⛏️ Digg AI 1000: N clusters │ K posts │ M authors`. No X auth required for the inline quotes since they flow through Digg's read-only endpoints.
 ## [3.1.1] - 2026-04-24
 ### Fixed
 - **Codex plugin layout.** Move the canonical runtime payload under `skills/last30days/` and update Codex/Claude plugin metadata and tests for the relocated engine path.
 - **Claude Code cache resolution.** Resolve Claude plugin installs to `skills/last30days/scripts/last30days.py` after the plugin-layout restructure.
 ## [3.1.0] - 2026-04-22
 Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`, per-entity Step 0.55, vs-mode N passes, comparison title attribution) and republishes the OpenClaw bundle, which had been frozen on ClawHub at `3.0.0-open` since April 8.
 ### Added
 - **OpenClaw republish.** `clawhub install last30days-official` now resolves to `3.1.0-open`, matching current main. Closes [#307](https://github.com/mvanhorn/last30days-skill/issues/307), [#195](https://github.com/mvanhorn/last30days-skill/issues/195), [#236](https://github.com/mvanhorn/last30days-skill/issues/236). The ClawHub bundle had shipped a broken `env.py get_config()` and stale SKILL.md path references since April; both are fixed at source on main and the republish carries the fixes to installers.
 ### Fixed
 - **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
 - **Broken README link.** The README's "source of truth" link pointed at `skills/last30days/SKILL.md`, a path that does not exist. Fixed to point at root `SKILL.md`.
 ### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
 Individual changelog entries for 3.0.10 through 3.0.14 below document the incremental work consolidated into this release.
 ## [3.0.14] - 2026-04-22
 ### Changed
@@ -61,6 +94,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 - **TikTok author preference.** `_fetch_post_comments` in `scripts/lib/tiktok.py` preferred `user.nickname` over `user.unique_id`, so the engine captured display names ("Moosa Noormahomed") instead of @handles ("moosanoormahomed"). Flipped to prefer `unique_id`. Nickname still wins as a fallback when `unique_id` is missing. Display names can contain emoji, spaces, and non-Latin characters that do not round-trip to a profile URL; the @handle is the stable identifier.
 - **Single plugin payload layout.** The canonical runtime moved to `skills/last30days/` for both Claude Code and Codex plugin loading. Root-level `SKILL.md`, `scripts/`, `agents/`, and `assets/` are no longer maintained as duplicate copies.
 ### Behavior fallback
@@ -4,20 +4,20 @@ Claude Code skill for researching any topic across Reddit, X, YouTube, and web.
 Python scripts with multi-source search aggregation.
 ## Structure
- `scripts/last30days.py` — main research engine
+- `skills/last30days/SKILL.md` — canonical skill definition
- `scripts/lib/` — search, enrichment, rendering modules
+- `skills/last30days/scripts/last30days.py` — main research engine
- `scripts/lib/vendor/bird-search/` — vendored X search client
+- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `SKILL.md` — skill definition (deployed to ~/.claude/skills/last30days/)
+- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
 ## Commands
 ```bash
-python3 scripts/last30days.py "test query" --emit=compact  # Run research
+python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
-bash scripts/sync.sh                                        # Deploy to ~/.claude, ~/.agents, ~/.codex
+bash skills/last30days/scripts/sync.sh
 ```
 ## Rules
 - `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- After edits: run `bash scripts/sync.sh` to deploy
+- After edits: run `bash skills/last30days/scripts/sync.sh` to deploy
 - Git remote: origin = public (`mvanhorn/last30days-skill`)
 ## Beta channel
@@ -18,7 +18,7 @@ git clone https://github.com/mvanhorn/last30days-skill.git
 cd last30days-skill
 # Run the sync script
-bash scripts/sync.sh
+bash skills/last30days/scripts/sync.sh
 ```
 This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days/`
@@ -30,8 +30,8 @@ This will auto-detect Hermes and deploy to `~/.hermes/skills/research/last30days
 mkdir -p ~/.hermes/skills/research/last30days
 # Copy files
-cp -r scripts ~/.hermes/skills/research/last30days/
+cp skills/last30days/SKILL.md ~/.hermes/skills/research/last30days/
-cp .hermes-plugin/SKILL.md ~/.hermes/skills/research/last30days/
+cp -r skills/last30days/scripts ~/.hermes/skills/research/last30days/
 ```
 ## Usage
@@ -111,7 +111,7 @@ To update to the latest version:
 ```bash
 cd last30days-skill
 git pull
-bash scripts/sync.sh
+bash skills/last30days/scripts/sync.sh
 ```
 ## Support
@@ -12,7 +12,7 @@
 **An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
-This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
+This README tracks the current v3 pipeline. The runtime skill spec lives in [SKILL.md](SKILL.md), which is the source of truth for the latest command and setup behavior.
 Claude Code:
 ```
@@ -68,6 +68,7 @@ If you're meeting with a CEO, have you read all their tweets and YouTube transcr
 | **Hacker News** | The developer consensus. 825 points, 899 comments. Where technical people actually argue. |
 | **Polymarket** | Not opinions. Odds. Backed by real money. 96% confidence on album sales. 4% on an acquisition. |
 | **GitHub** | For people: PR velocity, top repos by stars, release notes. For topics: issues and discussions. |
 | **Digg AI 1000** | Curated story clusters from ~1000 high-signal AI accounts on X, with attributable inline quotes (no X auth required). Auto-enabled when `digg-pp-cli` is on PATH. |
 | **Threads** | The post-Twitter text layer. Conversations from creators and brands. |
 | **Pinterest** | Visual discovery. Pins, saves, and comments on products and ideas. |
 | **Bluesky** | The decentralized social layer. AT Protocol posts from the post-Twitter migration. |
@@ -96,6 +97,28 @@ The synthesis ranks by what real people actually engaged with. Social relevancy,
 ## What v3 Changed
 ### Shareable HTML briefs
 Ask for an HTML brief and the skill saves a self-contained, dark-mode, print-friendly file you can drop into Slack, email, or Notion. No raw markdown leaks. Inline CSS, system-font fallbacks behind Inter and JetBrains Mono. No JavaScript. Works offline.
 ```
 /last30days OpenClaw --emit=html
 ```
 or just ask in plain language:
 ```
 /last30days OpenClaw, give me a shareable HTML brief
 /last30days Cursor IDE for slack
 /last30days Anthropic earnings export as html
 ```
 The skill emits the synthesis in chat as usual AND saves a brief to `${LAST30DAYS_MEMORY_DIR}/{topic}-brief.html` (defaults to `~/Documents/Last30Days/`). The chat response ends with the file path so you can `open` it or drag it into a message.
 What's in the file: badge, inline metadata line, the model's synthesis verbatim with all citations, the engine footer (✅ All agents reported back! tree), and a colophon noting the topic + how to re-run. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs; they never leak into the shareable artifact.
 For direct CLI use without the model in the loop, the engine also accepts `--synthesis-file PATH` to convert any markdown synthesis to HTML.
 ### Intelligent search: the killer feature
 The v3 engine doesn't just search for your topic. It figures out *where* to search before the search begins. Type "OpenClaw" and the engine resolves @steipete (Peter Steinberger, the creator), r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags - all via a new Python pre-research brain built by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
@@ -189,7 +212,7 @@ gemini extensions install ./last30days-skill
 git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
 ```
-Or build the claude.ai `.skill` file from source: `bash scripts/build-skill.sh` produces `dist/last30days.skill`.
+Or build the claude.ai `.skill` file from source: `bash skills/last30days/scripts/build-skill.sh` produces `dist/last30days.skill`.
 Reddit (with comments), Hacker News, Polymarket, and GitHub work immediately. Zero configuration. Run `/last30days` once and the setup wizard unlocks more sources in 30 seconds.
@@ -1,6 +1,6 @@
 [project]
 name = "last30days-skill"
-version = "3.0.0"
+version = "3.2.0"
 description = "Multi-source last-30-days research skill"
 readme = "README.md"
 requires-python = ">=3.12"
@@ -24,9 +24,9 @@ addopts = [
 [tool.coverage.run]
 branch = true
-source = ["scripts", "tests"]
+source = ["skills/last30days/scripts", "tests"]
 omit = [
-  "scripts/lib/vendor/*",
+  "skills/last30days/scripts/lib/vendor/*",
  "dist/*",
 ]
@@ -34,7 +34,6 @@ omit = [
 skip_empty = true
 show_missing = true
 omit = [
-  "scripts/lib/vendor/*",
+  "skills/last30days/scripts/lib/vendor/*",
  "dist/*",
 ]
@@ -58,7 +58,7 @@ OpenClaw:
 clawhub install last30days-official
 ```
-OpenAI Codex CLI: run `codex` from a checkout of this repo and v3's skill at `.agents/skills/last30days/SKILL.md` will be discovered automatically. Or copy `SKILL.md` to `~/.agents/skills/last30days/SKILL.md` for a global install.
+OpenAI Codex CLI: install the repo as a local Codex marketplace/plugin. The plugin manifest lives at `.codex-plugin/plugin.json`, and the canonical skill payload is `skills/last30days/SKILL.md`.
 Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
@@ -1,6 +1,6 @@
 ---
 name: last30days
-version: "3.0.1"
+version: "3.2.0"
 description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
 argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
 allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -45,6 +45,7 @@ metadata:
      - instagram
      - hackernews
      - polymarket
      - digg
      - bluesky
      - truthsocial
      - trends
@@ -104,7 +105,7 @@ These anchors used to live at line 1094 of this file. Three independent Opus 4.7
 🌐 last30days v{VERSION} · synced {YYYY-MM-DD}
 ```
-Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_ROOT/.claude-plugin/plugin.json"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins.
+Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_ROOT/../../.codex-plugin/plugin.json" 2>/dev/null || jq -r '.version' "$SKILL_ROOT/.claude-plugin/plugin.json"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins.
 **Why the badge is MANDATORY:** it is the structural anchor for the canonical output shape. Without it the model drifts into blog-post narrative format with `##` section headers and invented titles, violating LAW 2 and LAW 4. The 2026-04-18 public v3.0.6 0/8 regression produced outputs with section headers like "The headline", "Why he is everywhere", "1. gstack dominates", "The 'Homecoming' peak". Direct cause: this anchor was absent. Do NOT skip the badge. Do NOT describe it. Do NOT paraphrase it. Emit it verbatim as line 1.
@@ -233,7 +234,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
 ---
-# last30days v3.0.1: Research Any Topic from the Last 30 Days
+# last30days v3.2.0: Research Any Topic from the Last 30 Days
 > **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
@@ -317,6 +318,7 @@ Common patterns:
 - Always active: Reddit, Hacker News, Polymarket
 - If gh CLI is installed (check `which gh`): add GitHub
 - If digg-pp-cli is installed (check `which digg-pp-cli`): add Digg AI 1000
 - If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set, or xurl CLI is installed and authenticated: add X
 - If yt-dlp is installed (check `which yt-dlp`): add YouTube
 - If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok
@@ -824,7 +826,7 @@ Only show lines for platforms where something was resolved. Skip empty lines. On
 - For how_to: prioritize YouTube (tutorials) and Reddit (guides)
 - Primary subquery weight = 1.0, secondary = 0.6-0.8, peripheral = 0.3-0.5
-**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key)
+**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg AI 1000 clusters - only if `digg-pp-cli` is on PATH)
 **Intent → freshness_mode mapping:**
 - breaking_news, prediction → `strict_recent`
@@ -867,26 +869,36 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
 **IMPORTANT: Include `--x-handle={RESOLVED_HANDLE}` in the command. For comparison mode: Pass `--x-handle={TOPIC_A_HANDLE}` to the first pass, `--x-handle={TOPIC_B_HANDLE}` to the second pass, and both to the head-to-head pass. Also include `--subreddits={RESOLVED_SUBREDDITS}`, `--tiktok-hashtags={RESOLVED_HASHTAGS}`, `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}`, and `--ig-creators={RESOLVED_IG_CREATORS}` from Step 0.55. Omit any flag where the value was not resolved (empty).**
 ```bash
-# PIN SKILL_ROOT to the public plugin cache (highest-version dir wins on upgrade).
+# PIN SKILL_ROOT to an installed plugin cache first (highest-version dir wins on upgrade).
-# DO NOT write your own path-discovery loop. The 2026-04-18 Peter Steinberger run 1
+# Prefer Codex's skill package path when installed as a Codex plugin. Keep the Claude
-# regression was caused by a custom discovery loop landing on ~/.openclaw/skills/last30days/
+# plugin-root fallback for other hosts, then fall back to a repo checkout.
-# (a stale copy from a private-repo sync pattern). That path contains a pre-plan-007
+SKILL_ROOT="$(ls -d "$HOME/.codex/plugins/cache/"*/last30days/*/skills/last30days/ 2>/dev/null | sort -V | tail -1)"
 # engine and produces non-canonical output. This pinned resolution ignores every stale
 # copy (~/.openclaw/, ~/.agents/, ~/.codex/) and picks the plugin cache exclusively.
 SKILL_ROOT="$(ls -d "$HOME/.claude/plugins/cache/last30days-skill/last30days/"*/ 2>/dev/null | sort -V | tail -1)"
 SKILL_ROOT="${SKILL_ROOT%/}"
-# Fallback for repo checkout / Gemini / Codex hosts where the plugin cache does not exist.
+# Fallback for Claude plugin cache.
 # Only runs if the public plugin cache is missing entirely.
 if [ -z "$SKILL_ROOT" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
-  for dir in "." "${CLAUDE_PLUGIN_ROOT:-}" "${GEMINI_EXTENSION_DIR:-}"; do
+  CLAUDE_PLUGIN_ROOT="$(ls -d "$HOME/.claude/plugins/cache/last30days-skill/last30days/"*/ 2>/dev/null | sort -V | tail -1)"
  CLAUDE_PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT%/}"
  if [ -n "$CLAUDE_PLUGIN_ROOT" ]; then
    if [ -f "$CLAUDE_PLUGIN_ROOT/skills/last30days/scripts/last30days.py" ]; then
      SKILL_ROOT="$CLAUDE_PLUGIN_ROOT/skills/last30days"
    elif [ -f "$CLAUDE_PLUGIN_ROOT/scripts/last30days.py" ]; then
      SKILL_ROOT="$CLAUDE_PLUGIN_ROOT"
    fi
  fi
 fi
 # Fallback for repo checkout / Gemini / local development hosts where the plugin cache does not exist.
 if [ -z "$SKILL_ROOT" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
  for dir in "." "./skills/last30days" "${CLAUDE_PLUGIN_ROOT:-}" "${GEMINI_EXTENSION_DIR:-}"; do
    [ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
  done
 fi
 if [ -z "${SKILL_ROOT:-}" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
-  echo "ERROR: Could not find scripts/last30days.py in public plugin cache or repo checkout" >&2
+  echo "ERROR: Could not find scripts/last30days.py in Codex/Claude plugin cache or repo checkout" >&2
-  echo "Expected: $HOME/.claude/plugins/cache/last30days-skill/last30days/{VERSION}/scripts/last30days.py" >&2
+  echo "Expected Codex: $HOME/.codex/plugins/cache/{MARKETPLACE}/last30days/{VERSION}/skills/last30days/scripts/last30days.py" >&2
  echo "Expected Claude: $HOME/.claude/plugins/cache/last30days-skill/last30days/{VERSION}/skills/last30days/scripts/last30days.py" >&2
  exit 1
 fi
@@ -1497,6 +1509,33 @@ Close with `I have all the links to the {N} {source list} I pulled from. Just as
 ---
 ## SHAREABLE HTML BRIEF (when the user asked for one)
 **This section fires if EITHER trigger is true:**
 - `$ARGUMENTS` contains `--emit=html`, `--emit:html`, or `--html` as a flag
 - The user's natural-language request asks for an HTML brief, shareable doc, or file for sharing (Slack, email, Notion, "export as HTML", etc). Use your judgment for phrasing variants.
 **If neither trigger fires, skip this entire section and proceed to WAIT FOR USER'S RESPONSE.** No HTML save flow, no reference read needed.
 **When triggered, you MUST:**
 - Read `references/save-html-brief.md` BEFORE proceeding to WAIT FOR USER'S RESPONSE
 - Follow that file's instructions exactly - it is the canonical source for the save flow
 - Append the confirmation line (`📎 Shareable brief saved to <path>`) to your already-emitted chat response
 **You MUST NOT:**
 - Improvise the HTML save flow from memory or from instructions you've seen before
 - Skip the reference read because the steps "look familiar"
 - Save to a different path than the reference specifies
 - Add data quality warnings, debug headers, or safety notes to the saved HTML
 - Re-research the topic for the HTML render - the engine cache covers the second invocation
 **Why the directive is forceful:** the reference file is the only source of truth for the save flow. Skipping it produces broken artifacts - wrong path conventions, missing synthesis content, leaked engine debug output, or warnings that don't belong in shareable docs.
 ---
 ## WAIT FOR USER'S RESPONSE
 **STOP and wait** for the user to respond. Do NOT call any tools after displaying the invitation. Do NOT append a `Sources:` section (see override above - WebSearch's mandate does not apply here). The research script already saved raw data to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`) via `--save-dir`.
@@ -0,0 +1,90 @@
 # Save shareable HTML brief
 This reference file is loaded by the main `SKILL.md` when the user asked for an HTML brief (either explicitly via `--emit=html` / `--emit:html` / `--html`, or in natural language - "give me a shareable HTML brief", "for Slack", "for Notion", "export as HTML", etc.). The detection happens in `SKILL.md` so that the common no-HTML path stays short; the implementation lives here.
 The contract: the synthesis still appears in chat as the primary output. The HTML is an additional artifact saved to disk for sharing. Both happen in the same turn.
 ## When to fire this flow
 - After you have already emitted the full chat response: badge, "What I learned:" (or comparison title), bold-lead-in paragraphs with citations, KEY PATTERNS list, engine footer pass-through, invitation block.
 - BEFORE the WAIT FOR USER'S RESPONSE pause.
 - ONLY if the user asked. Do NOT save HTML when the user didn't ask for it.
 ## How to fire it
 ```bash
 # 1. Write your synthesis prose VERBATIM to a temp file. The synthesis is the
 #    "What I learned:" prose label, the bold-lead-in paragraphs with their
 #    inline citations as you wrote them in chat, and the "KEY PATTERNS from
 #    the research:" numbered list. Do NOT include the badge or the engine
 #    footer in the temp file - the engine adds those when it renders the HTML.
 #    Use the EXACT text you just wrote in chat. Do not paraphrase, do not
 #    summarize, do not reorder. The HTML must read identically to the chat
 #    response in voice and citations.
 SYNTHESIS_FILE="/tmp/last30days-synthesis-${CLAUDE_SESSION_ID}.md"
 cat > "$SYNTHESIS_FILE" <<'SYNTHESIS_EOF'
 What I learned:
 **{First headline}** - {body with [name](url) inline citations}
 **{Second headline}** - {body}
 **{Third headline}** - {body}
 KEY PATTERNS from the research:
 1. {pattern} - per [@handle](url)
 2. {pattern} - per [r/sub](url)
 3. {pattern} - per [@handle](url)
 SYNTHESIS_EOF
 # 2. Convert the synthesis to a self-contained HTML file via the engine.
 #    The engine reuses the cache from your earlier engine run (same topic
 #    + plan), so this second invocation is typically <1s on cache hit.
 SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//')
 HTML_PATH="${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html"
 "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "${TOPIC}" \
  --emit=html \
  --synthesis-file "$SYNTHESIS_FILE" \
  > "$HTML_PATH"
 # 3. Append ONE line to your already-emitted chat response, after the
 #    invitation block. Use a paperclip emoji as a visible signal that an
 #    artifact was produced:
 echo "📎 Shareable brief saved to $HTML_PATH"
 ```
 ## What ends up in the HTML file
 The engine's `--emit=html` renderer combines:
 - The badge (`🌐 last30days vX.Y.Z · synced YYYY-MM-DD`) at the top
 - A single inline metadata line (`{date range} · {active sources}`) below the badge
 - Your synthesis verbatim, with prose labels promoted to `<h2>` and bold lead-ins preserved
 - All `[name](url)` citations rendered as `<a>` tags
 - The engine footer (`✅ All agents reported back!` tree) preserved verbatim in monospace
 - A colophon with the topic and a re-run hint
 The renderer strips engine-internal noise that doesn't belong in a shareable artifact: the `# last30days vX.Y.Z: TOPIC` debug file header, the model-facing `> Safety note:` blockquote, and the `I'm now an expert on X` invitation block. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs - they never leak into the share-ready file.
 ## Comparison mode
 Same flow when the topic is `X vs Y` (or `X vs Y vs Z`). The engine routes through `render_for_html_comparison` internally; you don't need to do anything special. The synthesis temp file should still contain the comparison-shaped synthesis you wrote in chat (`## Quick Verdict`, `## {Entity}` per entity, `## Head-to-Head` table, `## The Bottom Line`, `## The emerging stack` per LAW 4 comparison exception).
 ## Follow-up turn
 If the user runs `/last30days OpenClaw` normally, sees the synthesis in chat, and THEN says "save that as HTML" or "give me a shareable version" in a follow-up turn, do the same save flow on the synthesis you wrote in the previous turn. Do not re-research; the synthesis is already in the conversation history. Just write it to the temp file and call the engine with `--emit=html --synthesis-file`.
 ## What NOT to do
 - Do NOT save HTML if the user didn't ask. The sparse mode (no synthesis) produces a thin file; not useful as a shareable.
 - Do NOT add content to the temp file beyond your synthesis prose. The badge / footer / colophon come from the engine.
 - Do NOT change the file path convention. `${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html` is the canonical location.
 - Do NOT silently overwrite an existing file without telling the user. If `$HTML_PATH` already exists from a prior run, the engine will pick a date-suffixed name (`{slug}-brief-YYYY-MM-DD.html`) automatically; just print whichever path the redirect produced.
 - Do NOT include the data quality warning text in the temp file or in your final chat line. Warnings are an engine-stderr concern, not an artifact concern.
 ## Edge cases
 - **Topic with shell-special characters** (quotes, ampersands): the temp filename uses a slugified version, but the engine receives the raw topic. The `cat <<'SYNTHESIS_EOF'` quoted heredoc form handles arbitrary content without expansion. Your synthesis text can include any character.
 - **Very long synthesis**: no upper bound. The engine handles long markdown bodies. Just paste verbatim.
 - **Synthesis with images or non-ASCII**: emoji and Unicode pass through. Image tags pass through as raw HTML; the renderer doesn't transform them. If you didn't include images in chat, don't add them here.
 - **No `${LAST30DAYS_MEMORY_DIR}` set**: defaults to `~/Documents/Last30Days/` per the SKILL.md `Configuration` section.
@@ -1,13 +1,14 @@
 #!/usr/bin/env bash
 # build-skill.sh - package this repo as a claude.ai-upload-ready .skill file
-# Usage: bash scripts/build-skill.sh  (run from repo root)
+# Usage: bash skills/last30days/scripts/build-skill.sh  (run from repo root)
 #
 # Produces dist/last30days.skill, a zip with a single top-level `last30days/`
-# directory containing SKILL.md and the scripts/ runtime. See
+# directory containing SKILL.md and the scripts/ runtime from skills/last30days.
 # See
 # docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
 set -euo pipefail
-REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
 cd "$REPO_ROOT"
 if ! git diff --quiet || ! git diff --cached --quiet; then
@@ -17,14 +18,7 @@ fi
 mkdir -p dist
 OUT="dist/last30days.skill"
-git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD
+git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD:skills/last30days
 # claude.ai's .skill bundle only needs the root SKILL.md + scripts/ runtime.
 # Claude Code needs skills/ and .claude-plugin/ in the git archive
 # (that's why they're NOT in .gitattributes export-ignore), but the .skill
 # bundle must strip them to keep a single canonical SKILL.md and stay under
 # the 200-file cap.
 zip -d "$OUT" "last30days/skills/*" "last30days/.claude-plugin/*" > /dev/null 2>&1 || true
 COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}')
 SIZE=$(du -h "$OUT" | cut -f1)
@@ -1,6 +1,6 @@
 #!/bin/bash
 # A/B test runner: public release vs private beta
-# Usage: bash scripts/compare.sh "Kanye West"
+# Usage: bash skills/last30days/scripts/compare.sh "Kanye West"
 #
 # Runs /last30days (public release) and /last30days-beta (private beta)
 # sequentially with a 30s gap, saves raw results with distinct suffixes,
@@ -9,8 +9,8 @@
 set -e
 if [ $# -eq 0 ]; then
-  echo "Usage: bash scripts/compare.sh <topic>"
+  echo "Usage: bash skills/last30days/scripts/compare.sh <topic>"
-  echo "  Example: bash scripts/compare.sh Kevin Rose"
+  echo "  Example: bash skills/last30days/scripts/compare.sh Kevin Rose"
  exit 1
 fi
 TOPIC="$*"
@@ -22,7 +22,8 @@ from lib import env as envlib
 from lib import schema
-REPO_ROOT = Path(__file__).resolve().parent.parent
+SKILL_ROOT = Path(__file__).resolve().parents[1]
 REPO_ROOT = Path(__file__).resolve().parents[3]
 EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json"
@@ -307,7 +308,10 @@ def create_eval_env() -> dict[str, str]:
 def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]:
-    cmd = [sys.executable, "scripts/last30days.py", topic, "--emit=json"]
+    engine = repo_dir / "skills" / "last30days" / "scripts" / "last30days.py"
    if not engine.exists():
        engine = repo_dir / "scripts" / "last30days.py"
    cmd = [sys.executable, str(engine), topic, "--emit=json"]
    if search:
        cmd.extend(["--search", search])
    if quick:
@@ -41,7 +41,7 @@ if os.name == "nt":
 SCRIPT_DIR = Path(__file__).parent.resolve()
 sys.path.insert(0, str(SCRIPT_DIR))
-from lib import env, pipeline, render, schema, ui
+from lib import env, html_render, pipeline, render, schema, ui
 _child_pids: set[int] = set()
 _child_pids_lock = threading.Lock()
@@ -91,30 +91,46 @@ def slugify(value: str) -> str:
    return slug or "last30days"
-def save_output(report: schema.Report, emit: str, save_dir: str, suffix: str = "") -> Path:
+def save_output(
    report: schema.Report,
    emit: str,
    save_dir: str,
    suffix: str = "",
    synthesis_md: str | None = None,
 ) -> Path:
    from datetime import datetime
    path = Path(save_dir).expanduser().resolve()
    path.mkdir(parents=True, exist_ok=True)
    slug = slugify(report.topic)
-    extension = "json" if emit == "json" else "md"
+    extension = "json" if emit == "json" else "html" if emit == "html" else "md"
    raw_label = "raw-html" if emit == "html" else "raw"
    suffix_part = f"-{suffix}" if suffix else ""
-    out_path = path / f"{slug}-raw{suffix_part}.{extension}"
+    out_path = path / f"{slug}-{raw_label}{suffix_part}.{extension}"
    if out_path.exists():
-        out_path = path / f"{slug}-raw{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
+        out_path = path / f"{slug}-{raw_label}{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
-    # Always save the FULL dump to disk (all items, all sources, transcripts).
+    # Markdown saves keep the complete debug artifact. JSON and HTML preserve
-    # Claude sees compact clusters via --emit=compact on stdout.
+    # their requested wire format so file extensions match their content.
-    # The saved file is the complete debug artifact.
+    if emit in {"json", "html"}:
-    if emit == "json":
+        content = emit_output(report, emit, synthesis_md=synthesis_md)
        content = emit_output(report, emit)
    else:
        content = render.render_full(report)
    out_path.write_text(content, encoding="utf-8")
    return out_path
-def emit_output(report: schema.Report, emit: str, fun_level: str = "medium", save_path: str | None = None) -> str:
+def emit_output(
    report: schema.Report,
    emit: str,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    if emit == "json":
        return json.dumps(schema.to_dict(report), indent=2, sort_keys=True)
    if emit == "html":
        return html_render.render_html(
            report, fun_level=fun_level, save_path=save_path, synthesis_md=synthesis_md,
        )
    if emit in {"compact", "md"}:
        return render.render_compact(report, fun_level=fun_level, save_path=save_path)
    if emit == "context":
@@ -127,6 +143,7 @@ def emit_comparison_output(
    emit: str,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    if emit == "json":
        payload = {
@@ -138,6 +155,13 @@ def emit_comparison_output(
            ],
        }
        return json.dumps(payload, indent=2, sort_keys=True)
    if emit == "html":
        return html_render.render_html_comparison(
            entity_reports,
            fun_level=fun_level,
            save_path=save_path,
            synthesis_md=synthesis_md,
        )
    if emit in {"compact", "md"}:
        return render.render_comparison_multi(
            entity_reports, fun_level=fun_level, save_path=save_path,
@@ -156,9 +180,10 @@ def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str)
    from pathlib import Path as _Path
    path = _Path(save_dir).expanduser().resolve()
    slug = slugify(topic)
-    extension = "json" if emit == "json" else "md"
+    extension = "json" if emit == "json" else "html" if emit == "html" else "md"
    raw_label = "raw-html" if emit == "html" else "raw"
    suffix_part = f"-{suffix}" if suffix else ""
-    raw = path / f"{slug}-raw{suffix_part}.{extension}"
+    raw = path / f"{slug}-{raw_label}{suffix_part}.{extension}"
    try:
        home = _Path.home().resolve()
        relative = raw.relative_to(home)
@@ -167,6 +192,14 @@ def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str)
        return str(raw)
 def read_synthesis_file(path: str) -> str:
    try:
        return Path(path).expanduser().read_text(encoding="utf-8")
    except OSError as exc:
        sys.stderr.write(f"[last30days] Cannot read --synthesis-file: {exc}\n")
        raise SystemExit(2)
 def persist_report(report: schema.Report) -> dict[str, int]:
    import store
@@ -193,7 +226,7 @@ def persist_report(report: schema.Report) -> dict[str, int]:
 def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Research a topic across live social, market, and grounded web sources.")
    parser.add_argument("topic", nargs="*", help="Research topic")
-    parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md"])
+    parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md", "html"])
    parser.add_argument("--search", help="Comma-separated source list")
    parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
    parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
@@ -201,6 +234,7 @@ def build_parser() -> argparse.ArgumentParser:
    parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
    parser.add_argument("--diagnose", action="store_true", help="Print provider and source availability")
    parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
    parser.add_argument("--synthesis-file", help="Markdown synthesis to embed in --emit=html output")
    parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
    parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
    parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
@@ -537,6 +571,13 @@ def main() -> int:
        parser.print_usage(sys.stderr)
        return 2
    synthesis_md = None
    if args.synthesis_file:
        if args.emit == "html":
            synthesis_md = read_synthesis_file(args.synthesis_file)
        else:
            sys.stderr.write("[last30days] Warning: --synthesis-file is only used with --emit=html; ignoring.\n")
    if not os.environ.get("LAST30DAYS_SKIP_PREFLIGHT"):
        from lib import preflight
        refuse_msg = preflight.check_class_1_trap(topic)
@@ -854,15 +895,29 @@ def main() -> int:
    if entity_reports:
        rendered = emit_comparison_output(
-            entity_reports, args.emit, fun_level=fun_level, save_path=footer_save_path,
+            entity_reports,
            args.emit,
            fun_level=fun_level,
            save_path=footer_save_path,
            synthesis_md=synthesis_md,
        )
    else:
        rendered = emit_output(
-            report, args.emit, fun_level=fun_level, save_path=footer_save_path,
+            report,
            args.emit,
            fun_level=fun_level,
            save_path=footer_save_path,
            synthesis_md=synthesis_md,
        )
    if args.save_dir:
        # Save the main topic's raw file (single-entity or comparison main).
-        save_path = save_output(report, args.emit, args.save_dir, suffix=args.save_suffix or "")
+        save_path = save_output(
            report,
            args.emit,
            args.save_dir,
            suffix=args.save_suffix or "",
            synthesis_md=synthesis_md,
        )
        sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
        # Competitor / vs-mode: also save a per-entity raw file for each peer.
        # Matches historical vs-mode behavior (N passes → N save files).
@@ -871,6 +926,7 @@ def main() -> int:
                peer_path = save_output(
                    entity_report, args.emit, args.save_dir,
                    suffix=args.save_suffix or "",
                    synthesis_md=synthesis_md,
                )
                sys.stderr.write(f"[last30days] Saved output to {peer_path}\n")
        sys.stderr.flush()
@@ -7,13 +7,11 @@ See scripts/lib/vendor/bird-search/package.json for authoritative version.
 import json
 import os
 import signal
 import shutil
 import subprocess
 import sys
 from pathlib import Path
-from . import http, log
+from . import http, log, subproc
 from datetime import datetime
 from typing import Any, Dict, List, Optional, Tuple
@@ -168,62 +166,51 @@ def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
        "--json",
    ]
-    # Use process groups for clean cleanup on timeout/kill
+    pid_holder: list[int] = []
    preexec = os.setsid if hasattr(os, 'setsid') else None
-    try:
+    def _register(pid: int) -> None:
-        proc = subprocess.Popen(
+        pid_holder.append(pid)
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            encoding="utf-8",
            errors="replace",
            preexec_fn=preexec,
            env=_subprocess_env(),
        )
        # Register for cleanup tracking (if available)
        try:
-            from last30days import register_child_pid, unregister_child_pid
+            from last30days import register_child_pid
-            register_child_pid(proc.pid)
+            register_child_pid(pid)
        except ImportError:
            pass
-        try:
+    try:
-            stdout, stderr = proc.communicate(timeout=timeout)
+        result = subproc.run_with_timeout(
-        except subprocess.TimeoutExpired:
+            cmd,
-            # Kill the entire process group
+            timeout=timeout,
-            try:
+            env=_subprocess_env(),
-                os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
+            on_pid=_register,
-            except (ProcessLookupError, PermissionError, OSError):
+        )
-                proc.kill()
+    except subproc.SubprocTimeout:
-            proc.wait(timeout=5)
+        return {"error": f"Search timed out after {timeout}s", "items": []}
-            return {"error": f"Search timed out after {timeout}s", "items": []}
+    except Exception as e:
-        finally:
+        return {"error": str(e), "items": []}
    finally:
        if pid_holder:
            try:
                from last30days import unregister_child_pid
-                unregister_child_pid(proc.pid)
+                unregister_child_pid(pid_holder[0])
            except Exception:
                pass
-        if proc.returncode != 0:
+    if result.returncode != 0:
-            error = stderr.strip() if stderr else "Bird search failed"
+        error = result.stderr.strip() or "Bird search failed"
-            return {"error": error, "items": []}
+        return {"error": error, "items": []}
-        output = stdout.strip() if stdout else ""
+    output = result.stdout.strip()
-        if not output:
+    if not output:
-            return {"items": []}
+        return {"items": []}
    try:
        parsed = json.loads(output)
        if isinstance(parsed, list):
            return {"items": parsed}
        return parsed
    except json.JSONDecodeError as e:
        return {"error": f"Invalid JSON response: {e}", "items": []}
-    except Exception as e:
+
-        return {"error": str(e), "items": []}
+    if isinstance(parsed, list):
        return {"items": parsed}
    return parsed
 def search_x(
@@ -330,47 +317,29 @@ def search_handles(
            "--json",
        ]
-        preexec = os.setsid if hasattr(os, 'setsid') else None
+        try:
            result = subproc.run_with_timeout(cmd, timeout=15, env=_subprocess_env())
        except subproc.SubprocTimeout:
            _log(f"Handle search timed out for @{handle}")
            return []
        except OSError as e:
            _log(f"Handle search error for @{handle}: {e}")
            return []
        if result.returncode != 0:
            _log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
            return []
        output = result.stdout.strip()
        if not output:
            return []
        try:
            proc = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                text=True,
                encoding="utf-8",
                errors="replace",
                preexec_fn=preexec,
                env=_subprocess_env(),
            )
            try:
                stdout, stderr = proc.communicate(timeout=15)
            except subprocess.TimeoutExpired:
                try:
                    os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
                except (ProcessLookupError, PermissionError, OSError):
                    proc.kill()
                proc.wait(timeout=5)
                _log(f"Handle search timed out for @{handle}")
                return []
            if proc.returncode != 0:
                _log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
                return []
            output = (stdout or "").strip()
            if not output:
                return []
            response = json.loads(output)
            return parse_bird_response(response, query=core_topic)
        except json.JSONDecodeError:
            _log(f"Invalid JSON from handle search for @{handle}")
-        except (OSError, subprocess.SubprocessError) as e:
+            return []
-            _log(f"Handle search error for @{handle}: {e}")
+        return parse_bird_response(response, query=core_topic)
        return []
    from concurrent.futures import ThreadPoolExecutor, as_completed
@@ -39,11 +39,14 @@ def normalize_text(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()
 def _ngrams_of_normalized(norm: str, n: int = 3) -> set[str]:
    if len(norm) < n:
        return {norm} if norm else set()
    return {norm[index:index + n] for index in range(len(norm) - n + 1)}
 def get_ngrams(text: str, n: int = 3) -> set[str]:
-    text = normalize_text(text)
+    return _ngrams_of_normalized(normalize_text(text), n)
    if len(text) < n:
        return {text} if text else set()
    return {text[index:index + n] for index in range(len(text) - n + 1)}
 def jaccard_similarity(left: set[str], right: set[str]) -> float:
@@ -90,7 +93,7 @@ class _PreparedText:
    def __init__(self, raw: str) -> None:
        norm = normalize_text(raw)
-        self.ngrams = get_ngrams(norm) if norm else set()
+        self.ngrams = _ngrams_of_normalized(norm)
        self.tokens = _tokenize(norm)
@@ -0,0 +1,413 @@
 """Digg AI 1000 source for last30days.
 Shells out to ``digg-pp-cli`` (read-only, no auth required) to surface
 clustered stories curated from ~1000 high-signal AI accounts on X. Each
 cluster carries a published TLDR, a curatorial rank, and a list of X
 posts that can be fetched as inline quotes.
 Activation gate: this source is only available when ``digg-pp-cli`` is
 on PATH. ``pipeline.available_sources`` checks ``shutil.which`` before
 including ``digg`` in the source list. The functions below also detect
 the missing-binary case as a defensive fallback.
 Primary path: ``digg-pp-cli search <topic> --since 30d --agent --limit N``.
 Optional enrichment: ``digg-pp-cli posts <clusterUrlId> --agent --by rank
 --limit M`` for the top K clusters in default/deep depth, attaching the
 top-ranked X posts to each cluster's ``posts`` field.
 """
 from __future__ import annotations
 import json
 import shutil
 from datetime import datetime, timedelta, timezone
 from typing import Any, Dict, List, Optional
 from . import log, subproc
 from .relevance import token_overlap_relevance
 CLI_BIN = "digg-pp-cli"
 # Per-depth knobs.
 DEPTH_CONFIG = {
    "quick": 8,
    "default": 20,
    "deep": 40,
 }
 # How many top-ranked clusters get post enrichment, per depth. Quick mode
 # skips enrichment to keep latency low (clusters already carry a TLDR).
 ENRICH_CONFIG = {
    "quick": 0,
    "default": 3,
    "deep": 5,
 }
 # X posts pulled per enriched cluster.
 POSTS_PER_CLUSTER = 3
 SEARCH_TIMEOUT = 30
 POSTS_TIMEOUT = 15
 def _log(msg: str) -> None:
    log.source_log("Digg", msg)
 def _is_available() -> bool:
    """True when the digg-pp-cli binary is on PATH."""
    return shutil.which(CLI_BIN) is not None
 def _today() -> datetime:
    return datetime.now(timezone.utc)
 def _parse_first_post_age(age: Optional[str], today: Optional[datetime] = None) -> Optional[str]:
    """Convert a digg firstPostAge token (e.g. '5d', '17d', '5h', '1w', '1m')
    into a YYYY-MM-DD string. Returns None when the value is outside the
    last-30-day window or cannot be parsed.
    Digg uses minutes-symbol-collision for 'months' (per agent-context:
    'Nh, Nd, Nw, Nm (e.g. 30d, 1w, 12h, 1m)'), so 'Nm' is months ~30 days.
    """
    if not age or not isinstance(age, str):
        return None
    age = age.strip().lower()
    if len(age) < 2:
        return None
    unit = age[-1]
    try:
        amount = int(age[:-1])
    except (ValueError, TypeError):
        return None
    if amount < 0:
        return None
    base = today or _today()
    if unit == "h":
        delta = timedelta(hours=amount)
    elif unit == "d":
        delta = timedelta(days=amount)
    elif unit == "w":
        delta = timedelta(weeks=amount)
    elif unit == "m":
        delta = timedelta(days=amount * 30)
    else:
        return None
    if delta > timedelta(days=30):
        return None
    point = base - delta
    return point.date().isoformat()
 def _build_search_args(query: str, limit: int) -> List[str]:
    return [
        CLI_BIN,
        "search",
        query,
        "--since",
        "30d",
        "--agent",
        "--limit",
        str(limit),
    ]
 def _build_posts_args(cluster_url_id: str, posts_per: int) -> List[str]:
    return [
        CLI_BIN,
        "posts",
        cluster_url_id,
        "--agent",
        "--by",
        "rank",
        "--limit",
        str(posts_per),
    ]
 def _run_cli(cmd: List[str], timeout: int) -> Dict[str, Any]:
    """Invoke digg-pp-cli and parse the JSON envelope.
    Returns ``{"results": [...]}`` on success, ``{"results": [], "error": "..."}``
    on failure. Never raises; the pipeline relies on shape consistency.
    """
    if not _is_available():
        return {"results": [], "error": f"{CLI_BIN} not on PATH"}
    try:
        result = subproc.run_with_timeout(cmd, timeout=timeout)
    except subproc.SubprocTimeout as exc:
        _log(f"Timeout: {exc}")
        return {"results": [], "error": str(exc)}
    except FileNotFoundError as exc:
        _log(f"Binary missing: {exc}")
        return {"results": [], "error": str(exc)}
    except OSError as exc:
        _log(f"Spawn failed: {exc}")
        return {"results": [], "error": str(exc)}
    if result.returncode != 0:
        snippet = (result.stderr or "").strip().splitlines()[:1]
        first = snippet[0] if snippet else f"exit {result.returncode}"
        _log(f"CLI exit {result.returncode}: {first}")
        return {"results": [], "error": first}
    stdout = result.stdout or ""
    if not stdout.strip():
        return {"results": []}
    try:
        data = json.loads(stdout)
    except json.JSONDecodeError as exc:
        _log(f"JSON decode failed: {exc}")
        return {"results": [], "error": f"json decode: {exc}"}
    if not isinstance(data, dict):
        return {"results": []}
    results = data.get("results")
    if not isinstance(results, list):
        return {"results": []}
    return data
 def search_digg(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
 ) -> Dict[str, Any]:
    """Search Digg AI 1000 clusters via digg-pp-cli.
    Args:
        topic: search query.
        from_date: YYYY-MM-DD start (advisory; --since 30d is the actual filter).
        to_date: YYYY-MM-DD end (advisory; same).
        depth: 'quick' | 'default' | 'deep'.
    Returns:
        Dict with ``results`` list. On failure, ``results`` is empty and an
        ``error`` key carries a one-line description.
    """
    limit = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    if not topic or not topic.strip():
        return {"results": []}
    cmd = _build_search_args(topic, limit)
    _log(f"search '{topic}' (limit={limit}, since=30d)")
    response = _run_cli(cmd, timeout=SEARCH_TIMEOUT)
    n = len(response.get("results") or [])
    _log(f"found {n} clusters")
    return response
 def _build_url(cluster_url_id: str) -> str:
    return f"https://di.gg/ai/{cluster_url_id}"
 def _rank_score(rank: Optional[int]) -> float:
    """Convert Digg rank (lower is better, top 50 are notable) into a
    positive engagement-style signal in [0, 50]. Anything off the top-50
    leaderboard contributes 0.
    """
    if rank is None:
        return 0.0
    try:
        r = int(rank)
    except (TypeError, ValueError):
        return 0.0
    if r < 1 or r > 50:
        return 0.0
    return float(51 - r)
 def parse_digg_response(
    response: Dict[str, Any],
    query: str = "",
 ) -> List[Dict[str, Any]]:
    """Parse a digg search envelope into normalized item dicts.
    Args:
        response: payload from ``search_digg``.
        query: original search query, used for token-overlap relevance.
    Returns:
        List of dicts ready for ``normalize._normalize_digg``.
    """
    raw = response.get("results") if isinstance(response, dict) else None
    if not isinstance(raw, list):
        return []
    items: List[Dict[str, Any]] = []
    for i, cluster in enumerate(raw):
        if not isinstance(cluster, dict):
            continue
        cluster_url_id = cluster.get("clusterUrlId")
        if not cluster_url_id:
            continue
        title = str(cluster.get("title") or "").strip()
        tldr = str(cluster.get("tldr") or "").strip()
        rank = cluster.get("rank")
        post_count = cluster.get("postCount") or 0
        unique_authors = cluster.get("uniqueAuthors") or 0
        first_post_age = cluster.get("firstPostAge")
        date_str = _parse_first_post_age(first_post_age)
        if date_str is None and first_post_age:
            # firstPostAge present but outside 30d -> drop; last30days contract.
            continue
        rank_decay = max(0.3, 1.0 - (i * 0.02))
        if query:
            content_score = token_overlap_relevance(query, f"{title} {tldr}".strip())
        else:
            content_score = 0.5
        rank_boost = min(0.2, _rank_score(rank) / 250.0)
        relevance = min(1.0, 0.55 * rank_decay + 0.35 * content_score + rank_boost)
        items.append(
            {
                "id": str(cluster_url_id),
                "title": title or f"Digg cluster {i + 1}",
                "url": _build_url(str(cluster_url_id)),
                "tldr": tldr,
                "author": "",
                "date": date_str,
                "engagement": {
                    "postCount": int(post_count) if isinstance(post_count, (int, float)) else 0,
                    "uniqueAuthors": int(unique_authors) if isinstance(unique_authors, (int, float)) else 0,
                    "rank": int(rank) if isinstance(rank, (int, float)) else None,
                    "rank_score": _rank_score(rank),
                },
                "first_post_age": first_post_age,
                "posts": [],
                "relevance": round(relevance, 2),
                "why_relevant": (
                    f"Digg AI 1000 cluster (rank {rank}, {post_count} posts, {unique_authors} authors)"
                    if rank is not None
                    else f"Digg AI 1000 cluster ({post_count} posts, {unique_authors} authors)"
                ),
            }
        )
    return items
 def _parse_post(raw_post: Dict[str, Any]) -> Optional[Dict[str, Any]]:
    """Reduce a digg post payload into the small dict render uses.
    We deliberately keep this minimal: an inline quote needs the author
    handle, the body, the post type, and the X URL.
    """
    if not isinstance(raw_post, dict):
        return None
    body = str(raw_post.get("body") or "").strip()
    if not body:
        return None
    author = raw_post.get("author") or {}
    if not isinstance(author, dict):
        author = {}
    username = str(author.get("username") or "").strip()
    if not username:
        return None
    x_url = str(raw_post.get("xUrl") or "").strip()
    if not x_url:
        return None
    return {
        "username": username,
        "display_name": str(author.get("display_name") or "").strip() or username,
        "category": str(author.get("category") or "").strip(),
        "rank": author.get("rank"),
        "body": body,
        "post_type": str(raw_post.get("post_type") or "tweet").strip(),
        "x_url": x_url,
        "posted_at": raw_post.get("posted_at"),
    }
 def fetch_top_posts(cluster_url_id: str, posts_per: int = POSTS_PER_CLUSTER) -> List[Dict[str, Any]]:
    """Fetch top-ranked X posts attached to a cluster.
    Returns an empty list on any failure (timeout, missing cluster, JSON
    error). Never raises.
    """
    if posts_per <= 0:
        return []
    cmd = _build_posts_args(cluster_url_id, posts_per)
    response = _run_cli(cmd, timeout=POSTS_TIMEOUT)
    raw = response.get("results") or []
    out: List[Dict[str, Any]] = []
    for entry in raw:
        post = _parse_post(entry)
        if post is not None:
            out.append(post)
    return out
 def enrich_with_top_posts(
    items: List[Dict[str, Any]],
    top_k: int = 3,
    posts_per: int = POSTS_PER_CLUSTER,
 ) -> List[Dict[str, Any]]:
    """Attach top X posts to the first ``top_k`` clusters by Digg rank order.
    Mutates and returns the same list. Items that already have posts, or
    whose ``postCount`` is 0, are skipped.
    """
    if top_k <= 0 or posts_per <= 0:
        return items
    enriched = 0
    for item in items:
        if enriched >= top_k:
            break
        if item.get("posts"):
            continue
        engagement = item.get("engagement") or {}
        if not engagement.get("postCount"):
            continue
        cluster_url_id = item.get("id")
        if not cluster_url_id:
            continue
        posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
        item["posts"] = posts
        enriched += 1
    if enriched:
        _log(f"enriched {enriched} clusters with X posts")
    return items
 def enrich_source_items(items: list, top_k: int = 3, posts_per: int = POSTS_PER_CLUSTER) -> list:
    """Attach top X posts to the first ``top_k`` SourceItems that survived dedupe.
    Reads ``metadata['clusterUrlId']`` and writes ``metadata['posts']`` in
    place. Skips items that already carry a non-empty ``metadata['posts']``,
    items whose engagement ``postCount`` is 0, and items whose source is not
    'digg'. Designed to run from `_finalize_items_by_source` so enrichment
    is spent on the items the brief actually shows.
    """
    if top_k <= 0 or posts_per <= 0:
        return items
    enriched = 0
    for item in items:
        if enriched >= top_k:
            break
        if getattr(item, "source", None) != "digg":
            continue
        metadata = getattr(item, "metadata", None) or {}
        if metadata.get("posts"):
            continue
        engagement = getattr(item, "engagement", None) or {}
        if not engagement.get("postCount"):
            continue
        cluster_url_id = metadata.get("clusterUrlId") or item.item_id
        if not cluster_url_id:
            continue
        posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
        if posts:
            metadata["posts"] = posts
            enriched += 1
    if enriched:
        _log(f"post-dedupe enriched {enriched} clusters with X posts")
    return items
@@ -372,14 +372,6 @@ def config_exists() -> bool:
    return False
 def is_reddit_available(config: dict[str, Any]) -> bool:
    """Check if Reddit search is available.
    v3 uses ScrapeCreators only.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def get_reddit_source(config: dict[str, Any]) -> str | None:
    """Determine which Reddit backend to use.
@@ -116,6 +116,8 @@ def weighted_rrf(
    """Fuse ranked lists into a single candidate pool."""
    subqueries = {subquery.label: subquery for subquery in plan.subqueries}
    candidates: dict[str, schema.Candidate] = {}
    # Track (source, item_id) pairs already attached to each candidate for O(1) dedup.
    seen_source_items: dict[str, set[tuple[str, str]]] = {}
    for (label, source), items in streams.items():
        subquery = subqueries[label]
@@ -154,6 +156,7 @@ def weighted_rrf(
                        ]
                    },
                )
                seen_source_items[key] = {(item.source, item.item_id)}
                continue
            candidate = candidates[key]
@@ -179,7 +182,9 @@ def weighted_rrf(
                candidate.subquery_labels.append(label)
            if item.source not in candidate.sources:
                candidate.sources.append(item.source)
-            if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items):
+            source_item_key = (item.source, item.item_id)
            if source_item_key not in seen_source_items[key]:
                seen_source_items[key].add(source_item_key)
                candidate.source_items.append(item)
            candidate.metadata.setdefault("provenance", []).append(
                {
@@ -0,0 +1,674 @@
 """HTML rendering for shareable last30days reports."""
 from __future__ import annotations
 import html
 import re
 from datetime import date
 from . import render, schema
 PROSE_LABELS = [
    ("What I learned:", "What I learned"),
    ("KEY PATTERNS from the research:", "Key patterns from the research"),
 ]
 INVITATION_PATTERN = re.compile(r"^---\nI'm now an expert.*?Just ask\.$", re.MULTILINE | re.DOTALL)
 EVIDENCE_BLOCK_PATTERN = re.compile(r"<!-- EVIDENCE FOR SYNTHESIS.*?<!-- END EVIDENCE FOR SYNTHESIS -->", re.DOTALL)
 PASS_THROUGH_FOOTER_PATTERN = re.compile(r"<!-- PASS-THROUGH FOOTER.*?-->\n(.*?)<!-- END PASS-THROUGH FOOTER -->", re.DOTALL)
 CANONICAL_BOUNDARY_PATTERN = re.compile(r"\n?---\n# END OF last30days CANONICAL OUTPUT.*$", re.DOTALL)
 # render_for_html emits metadata as <!-- META: ... --> so it survives the
 # markdown converter (which escapes raw HTML inside paragraphs). Promoted to
 # a styled <div class="meta"> after conversion.
 META_MARKER_PATTERN = re.compile(r"<!--\s*META:\s*(.*?)\s*-->")
 CSS = """
 :root {
  --bg: #0e0e10;
  --bg-elev: #18181b;
  --fg: #fafafa;
  --fg-muted: #a1a1aa;
  --fg-subtle: #71717a;
  --accent: #a855f7;
  --accent-soft: #c4b5fd;
  --border: #27272a;
  --code-bg: #1a1a1d;
  --max-w: 720px;
 }
@media (prefers-color-scheme: light) {
  :root {
    --bg: #ffffff;
    --bg-elev: #fafafa;
    --fg: #18181b;
    --fg-muted: #52525b;
    --fg-subtle: #71717a;
    --accent: #7c3aed;
    --accent-soft: #6d28d9;
    --border: #e4e4e7;
    --code-bg: #f4f4f5;
  }
 }
 * { box-sizing: border-box; }
 html, body {
  margin: 0;
  padding: 0;
  background: var(--bg);
  color: var(--fg);
  font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif;
  font-size: 17px;
  line-height: 1.65;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  text-rendering: optimizeLegibility;
 }
 body {
  max-width: var(--max-w);
  margin: 0 auto;
  padding: 4rem 1.5rem 6rem;
 }
 .badge {
  display: inline-block;
  padding: 0.4rem 0.85rem;
  margin-bottom: 2.5rem;
  background: var(--bg-elev);
  border: 1px solid var(--border);
  border-radius: 999px;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13px;
  font-weight: 500;
  color: var(--fg-muted);
  letter-spacing: 0;
 }
 .badge .accent { color: var(--accent); }
 .meta {
  margin: -1.5rem 0 2.5rem;
  color: var(--fg-subtle);
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13px;
  letter-spacing: 0.01em;
 }
 h1 {
  margin: 0 0 1.5rem;
  color: var(--fg);
  font-size: 30px;
  font-weight: 700;
  line-height: 1.2;
  letter-spacing: 0;
 }
 h2,
 .prose-label {
  margin: 2.75rem 0 1.25rem;
  color: var(--fg);
  font-size: 20px;
  font-weight: 600;
  line-height: 1.35;
  letter-spacing: 0;
 }
 .badge + h2,
 .badge + .prose-label { margin-top: 0.5rem; }
 h3 {
  margin: 2rem 0 0.85rem;
  color: var(--fg);
  font-size: 17px;
  font-weight: 600;
  line-height: 1.4;
  letter-spacing: 0;
 }
 p {
  margin: 0 0 1.4rem;
  color: var(--fg-muted);
 }
 p strong,
 li strong,
 td strong {
  color: var(--fg);
  font-weight: 600;
 }
 a {
  color: var(--accent);
  text-decoration: none;
  border-bottom: 1px solid transparent;
  transition: border-color 0.15s ease;
 }
 a:hover { border-bottom-color: var(--accent); }
 ul,
 ol {
  margin: 0 0 1.6rem;
  padding-left: 1.5rem;
  color: var(--fg-muted);
 }
 li {
  margin: 0.6rem 0;
  padding-left: 0.4rem;
 }
 li::marker {
  color: var(--accent);
  font-weight: 600;
 }
 blockquote {
  margin: 1.5rem 0;
  padding-left: 1rem;
  border-left: 3px solid var(--accent);
  color: var(--fg-muted);
 }
 hr {
  margin: 2.5rem 0;
  border: 0;
  border-top: 1px solid var(--border);
 }
 code {
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 0.92em;
  background: var(--code-bg);
  padding: 0.15rem 0.4rem;
  border-radius: 4px;
  color: var(--accent-soft);
 }
 pre {
  margin: 1.4rem 0;
  background: var(--code-bg);
  border: 1px solid var(--border);
  border-radius: 8px;
  padding: 1rem 1.25rem;
  overflow-x: auto;
  font-size: 14px;
  line-height: 1.6;
 }
 pre code {
  background: none;
  padding: 0;
  color: var(--fg);
 }
 table {
  width: 100%;
  border-collapse: collapse;
  margin: 1.5rem 0;
  font-size: 15px;
 }
 th,
 td {
  text-align: left;
  padding: 0.75rem 1rem;
  border-bottom: 1px solid var(--border);
  vertical-align: top;
 }
 th {
  color: var(--fg-muted);
  font-weight: 600;
  font-size: 13px;
  letter-spacing: 0;
  text-transform: uppercase;
 }
 td { color: var(--fg-muted); }
 td:first-child { color: var(--fg); font-weight: 500; }
 .engine-footer {
  margin: 3rem 0 2.5rem;
  padding: 1.25rem 1.5rem;
  background: var(--bg-elev);
  border: 1px solid var(--border);
  border-radius: 8px;
  color: var(--fg-muted);
 }
 .engine-footer pre {
  margin: 0;
  padding: 0;
  background: transparent;
  border: 0;
  border-radius: 0;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13.5px;
  font-weight: 400;
  line-height: 1.75;
  color: inherit;
  white-space: pre-wrap;
  word-break: break-word;
 }
 .colophon {
  margin-top: 4rem;
  padding-top: 2rem;
  border-top: 1px solid var(--border);
  color: var(--fg-subtle);
  font-size: 13px;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  line-height: 1.7;
 }
 .colophon .rerun {
  display: inline-block;
  padding: 0.15rem 0.5rem;
  margin-left: 0.25rem;
  background: var(--code-bg);
  border-radius: 4px;
  color: var(--accent-soft);
  font-size: 0.95em;
 }
@media print {
  :root {
    --bg: #ffffff;
    --bg-elev: #f5f5f5;
    --fg: #000000;
    --fg-muted: #1f2937;
    --fg-subtle: #4b5563;
    --accent: #6d28d9;
    --accent-soft: #6d28d9;
    --border: #d4d4d8;
    --code-bg: #f4f4f5;
  }
  @page { size: A4; margin: 1.5cm 2cm; }
  body {
    max-width: none;
    padding: 0;
    font-size: 11pt;
  }
  a {
    color: inherit;
    border-bottom: 0;
    text-decoration: underline;
  }
  a[href]::after {
    content: " (" attr(href) ")";
    font-size: 0.85em;
    color: var(--fg-subtle);
  }
  .engine-footer { page-break-inside: avoid; }
 }
@media (max-width: 600px) {
  body {
    padding: 2.5rem 1.25rem 4rem;
    font-size: 16px;
  }
  h1 { font-size: 25px; }
  .badge { font-size: 12px; }
  th, td { padding: 0.65rem 0.5rem; }
 }
 """.strip()
 HTML_TEMPLATE = """<!DOCTYPE html>
 <html lang="en">
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <title>last30days · __TITLE__</title>
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&amp;family=JetBrains+Mono:wght@400;500&amp;display=swap" rel="stylesheet">
 <style>
 __CSS__
 </style>
 </head>
 <body>
 __BODY__
 __COLOPHON__
 </body>
 </html>
 """
 def render_html(
    report: schema.Report,
    *,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    _ = fun_level
    md = render.render_for_html(report, synthesis_md=synthesis_md, save_path=save_path)
    md = _strip_evidence_block(md)
    md = _strip_invitation(md)
    md = _strip_canonical_boundary(md)
    md = _promote_prose_labels(md)
    body = _markdown_to_html(md)
    body = _wrap_engine_footer(body)
    body = _promote_meta_marker(body)
    colophon = _build_colophon(report)
    return _wrap_in_template(body, colophon, report.topic)
 def render_html_comparison(
    entity_reports: list[tuple[str, schema.Report]],
    *,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    _ = fun_level
    md = render.render_for_html_comparison(
        entity_reports, synthesis_md=synthesis_md, save_path=save_path,
    )
    md = _strip_evidence_block(md)
    md = _strip_invitation(md)
    md = _strip_canonical_boundary(md)
    md = _promote_prose_labels(md)
    body = _markdown_to_html(md)
    body = _wrap_engine_footer(body)
    body = _promote_meta_marker(body)
    topic = " vs ".join(label for label, _ in entity_reports)
    colophon = _build_colophon(entity_reports[0][1], topic=topic)
    return _wrap_in_template(body, colophon, topic)
 def _strip_evidence_block(md: str) -> str:
    return EVIDENCE_BLOCK_PATTERN.sub("", md)
 def _strip_invitation(md: str) -> str:
    return INVITATION_PATTERN.sub("", md)
 def _strip_canonical_boundary(md: str) -> str:
    return CANONICAL_BOUNDARY_PATTERN.sub("", md)
 def _promote_prose_labels(md: str) -> str:
    for source, normalized in PROSE_LABELS:
        md = re.sub(
            rf"^{re.escape(source)}$",
            f"## {normalized}",
            md,
            flags=re.MULTILINE,
        )
    return md
 def _markdown_to_html(md: str) -> str:
    md, footers = _protect_engine_footers(md)
    global _ENGINE_FOOTER_STORE
    _ENGINE_FOOTER_STORE = footers
    # Strip HTML comments EXCEPT preserved markers used for post-processing
    # (META is promoted to <div class="meta"> after markdown conversion).
    md = re.sub(r"<!--(?!\s*META:).*?-->", "", md, flags=re.DOTALL)
    lines = md.splitlines()
    out: list[str] = []
    paragraph: list[str] = []
    list_type: str | None = None
    in_code = False
    code_lines: list[str] = []
    index = 0
    def flush_paragraph() -> None:
        nonlocal paragraph
        if paragraph:
            text = " ".join(part.strip() for part in paragraph).strip()
            if text:
                out.append(f"<p>{_inline_markdown(text)}</p>")
            paragraph = []
    def close_list() -> None:
        nonlocal list_type
        if list_type:
            out.append(f"</{list_type}>")
            list_type = None
    while index < len(lines):
        line = lines[index]
        stripped = line.strip()
        if in_code:
            if stripped.startswith("```"):
                out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
                code_lines = []
                in_code = False
            else:
                code_lines.append(line)
            index += 1
            continue
        if stripped.startswith("```"):
            flush_paragraph()
            close_list()
            in_code = True
            code_lines = []
            index += 1
            continue
        if stripped in footers:
            flush_paragraph()
            close_list()
            out.append(stripped)
            index += 1
            continue
        if not stripped:
            flush_paragraph()
            close_list()
            index += 1
            continue
        if stripped == "---":
            flush_paragraph()
            close_list()
            out.append("<hr>")
            index += 1
            continue
        if index + 1 < len(lines) and _is_table_row(stripped) and _is_table_separator(lines[index + 1].strip()):
            flush_paragraph()
            close_list()
            table_lines = [stripped]
            index += 2
            while index < len(lines) and _is_table_row(lines[index].strip()):
                table_lines.append(lines[index].strip())
                index += 1
            out.append(_render_table(table_lines))
            continue
        heading = re.match(r"^(#{1,4})\s+(.+)$", stripped)
        if heading:
            flush_paragraph()
            close_list()
            level = min(len(heading.group(1)), 3)
            out.append(f"<h{level}>{_inline_markdown(heading.group(2))}</h{level}>")
            index += 1
            continue
        if stripped.startswith(">"):
            flush_paragraph()
            close_list()
            quote_lines = []
            while index < len(lines) and lines[index].strip().startswith(">"):
                quote_lines.append(lines[index].strip().lstrip(">").strip())
                index += 1
            out.append(f"<blockquote>{_inline_markdown(' '.join(quote_lines))}</blockquote>")
            continue
        unordered = re.match(r"^[-*]\s+(.+)$", stripped)
        ordered = re.match(r"^\d+[.)]\s+(.+)$", stripped)
        if unordered or ordered:
            flush_paragraph()
            next_type = "ul" if unordered else "ol"
            if list_type != next_type:
                close_list()
                out.append(f"<{next_type}>")
                list_type = next_type
            item = unordered.group(1) if unordered else ordered.group(1)
            out.append(f"<li>{_inline_markdown(item)}</li>")
            index += 1
            continue
        if stripped.startswith("🌐 last30days"):
            flush_paragraph()
            close_list()
            badge_text = _inline_markdown(stripped.removeprefix("🌐").strip())
            out.append(f'<div class="badge"><span class="accent">🌐</span> {badge_text}</div>')
            index += 1
            continue
        paragraph.append(line)
        index += 1
    if in_code:
        out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
    flush_paragraph()
    close_list()
    return "\n".join(out).strip()
 def _protect_engine_footers(md: str) -> tuple[str, dict[str, str]]:
    footers: dict[str, str] = {}
    def replace(match: re.Match[str]) -> str:
        token = f"__LAST30DAYS_ENGINE_FOOTER_{len(footers)}__"
        footers[token] = match.group(1).strip("\n")
        return f"\n{token}\n"
    return PASS_THROUGH_FOOTER_PATTERN.sub(replace, md), footers
 def _wrap_engine_footer(body: str) -> str:
    def replace(match: re.Match[str]) -> str:
        footer = html.escape(_ENGINE_FOOTER_STORE.get(match.group(0), ""), quote=False)
        return f'<div class="engine-footer"><pre>{footer}</pre></div>'
    return re.sub(
        r"__LAST30DAYS_ENGINE_FOOTER_\d+__",
        replace,
        body,
    )
 def _promote_meta_marker(body: str) -> str:
    """Promote ``<!-- META: ... -->`` markers into a styled ``<div class="meta">``.
    The marker is preserved through the comment-strip pass (see
    _markdown_to_html exemption) but the markdown converter wraps it in
    ``<p>`` and HTML-escapes the angle brackets. After conversion the body
    contains shapes like:
      <p>&lt;!-- META: TEXT --&gt;</p>
      <p><!-- META: TEXT --></p>     (when not escaped)
    Both collapse to ``<div class="meta">TEXT</div>``.
    """
    def replace(match: re.Match[str]) -> str:
        text = match.group(1).strip()
        return f'<div class="meta">{text}</div>'
    # Escaped form (most common after markdown conversion)
    body = re.sub(
        r"<p>\s*&lt;!--\s*META:\s*(.*?)\s*--&gt;\s*</p>",
        replace,
        body,
    )
    body = re.sub(r"&lt;!--\s*META:\s*(.*?)\s*--&gt;", replace, body)
    # Unescaped form (paranoid fallback)
    body = re.sub(r"<p>\s*<!--\s*META:\s*(.*?)\s*-->\s*</p>", replace, body)
    body = re.sub(r"<!--\s*META:\s*(.*?)\s*-->", replace, body)
    return body
 _ENGINE_FOOTER_STORE: dict[str, str] = {}
 def _inline_markdown(text: str) -> str:
    escaped = html.escape(text, quote=True)
    code_tokens: dict[str, str] = {}
    def code_replace(match: re.Match[str]) -> str:
        token = f"__CODE_{len(code_tokens)}__"
        code_tokens[token] = f"<code>{match.group(1)}</code>"
        return token
    escaped = re.sub(r"`([^`]+)`", code_replace, escaped)
    escaped = re.sub(r"\*\*([^*]+)\*\*", r"<strong>\1</strong>", escaped)
    escaped = re.sub(
        r"\[([^\]]+)\]\(([^)\s]+)\)",
        r'<a href="\2">\1</a>',
        escaped,
    )
    for token, value in code_tokens.items():
        escaped = escaped.replace(token, value)
    return escaped
 def _is_table_row(line: str) -> bool:
    return "|" in line and len(_split_table_cells(line)) >= 2
 def _is_table_separator(line: str) -> bool:
    cells = _split_table_cells(line)
    return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell.strip()) for cell in cells)
 def _split_table_cells(line: str) -> list[str]:
    return [cell.strip() for cell in line.strip().strip("|").split("|")]
 def _render_table(rows: list[str]) -> str:
    header = _split_table_cells(rows[0])
    body_rows = [_split_table_cells(row) for row in rows[1:]]
    out = ["<table>", "<thead>", "<tr>"]
    out.extend(f"<th>{_inline_markdown(cell)}</th>" for cell in header)
    out.extend(["</tr>", "</thead>", "<tbody>"])
    for row in body_rows:
        out.append("<tr>")
        out.extend(f"<td>{_inline_markdown(cell)}</td>" for cell in row)
        out.append("</tr>")
    out.extend(["</tbody>", "</table>"])
    return "\n".join(out)
 def _build_colophon(report: schema.Report, *, topic: str | None = None) -> str:
    display_topic = topic or report.topic
    generated = _generated_date(report)
    version = render._skill_version()
    escaped_topic = html.escape(display_topic)
    rerun = html.escape(f"/last30days {display_topic}")
    return (
        '<div class="colophon">\n'
        f"  Generated {generated} by /last30days v{html.escape(version)} · topic: {escaped_topic}<br>\n"
        f'  Re-run for fresh data: <span class="rerun">{rerun}</span>\n'
        "</div>"
    )
 def _generated_date(report: schema.Report) -> str:
    if report.generated_at:
        return report.generated_at[:10]
    return date.today().strftime("%Y-%m-%d")
 def _wrap_in_template(body: str, colophon: str, title: str) -> str:
    return (
        HTML_TEMPLATE
        .replace("__TITLE__", html.escape(title))
        .replace("__CSS__", CSS)
        .replace("__BODY__", body)
        .replace("__COLOPHON__", colophon)
    )
@@ -49,6 +49,7 @@ def normalize_source_items(
        "xquik": _normalize_x,
        "pinterest": _normalize_pinterest,
        "polymarket": _normalize_polymarket,
        "digg": _normalize_digg,
        "grounding": _normalize_grounding,
        "xiaohongshu": _normalize_grounding,
        "github": _normalize_github,
@@ -110,6 +111,19 @@ def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> An
    return default
 def _join_comment_excerpts(
    top_comments: list[Any],
    key: str,
    limit: int = 3,
 ) -> str:
    """Space-join the `key` field from the first `limit` dict-shaped comments."""
    return " ".join(
        str(comment.get(key) or "").strip()
        for comment in top_comments[:limit]
        if isinstance(comment, dict)
    )
 def _domain_from_url(url: str) -> str | None:
    if not url:
        return None
@@ -169,11 +183,7 @@ def _normalize_reddit(
    to_date: str,
 ) -> schema.SourceItem:
    top_comments = item.get("top_comments") or []
-    comment_text = " ".join(
+    comment_text = _join_comment_excerpts(top_comments, "excerpt")
        str(comment.get("excerpt") or "").strip()
        for comment in top_comments[:3]
        if isinstance(comment, dict)
    )
    body = "\n".join(
        part
        for part in [
@@ -338,11 +348,7 @@ def _normalize_hackernews(
    to_date: str,
 ) -> schema.SourceItem:
    top_comments = item.get("top_comments") or []
-    comment_text = " ".join(
+    comment_text = _join_comment_excerpts(top_comments, "text")
        str(comment.get("text") or "").strip()
        for comment in top_comments[:3]
        if isinstance(comment, dict)
    )
    title = str(item.get("title") or "").strip()
    body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part)
    return _source_item(
@@ -394,6 +400,53 @@ def _normalize_microblog(
    )
 def _normalize_digg(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    """Normalizer for Digg AI 1000 clusters.
    Each cluster is one item. The TLDR carries the most useful body for
    rerank and synthesis. Top-ranked X posts attached at search time are
    passed through under metadata['posts'] so render can emit them as
    inline 'via Digg AI 1000' quotes.
    """
    title = str(item.get("title") or "").strip()
    tldr = str(item.get("tldr") or "").strip()
    body = "\n\n".join(part for part in [title, tldr] if part)
    posts = item.get("posts") or []
    if not isinstance(posts, list):
        posts = []
    cluster_url_id = str(item.get("id") or f"DG{index + 1}")
    return _source_item(
        item_id=cluster_url_id,
        source=source,
        title=title or f"Digg cluster {index + 1}",
        body=body,
        url=str(item.get("url") or f"https://di.gg/ai/{cluster_url_id}"),
        author="",
        container="Digg AI 1000",
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=tldr[:400],
        metadata={
            "clusterUrlId": cluster_url_id,
            "tldr": tldr,
            "rank": (item.get("engagement") or {}).get("rank"),
            "uniqueAuthors": (item.get("engagement") or {}).get("uniqueAuthors"),
            "postCount": (item.get("engagement") or {}).get("postCount"),
            "firstPostAge": item.get("first_post_age"),
            "posts": posts,
        },
    )
 def _normalize_polymarket(
    source: str,
    item: dict[str, Any],
@@ -441,11 +494,7 @@ def _normalize_github(
    title = str(item.get("title") or "").strip()
    snippet_text = str(item.get("snippet") or "").strip()
    top_comments = item.get("metadata", {}).get("top_comments") or []
-    comment_text = " ".join(
+    comment_text = _join_comment_excerpts(top_comments, "excerpt")
        str(comment.get("excerpt") or "").strip()
        for comment in top_comments[:3]
        if isinstance(comment, dict)
    )
    body = "\n".join(part for part in [title, snippet_text, comment_text] if part)
    metadata = item.get("metadata") or {}
    return _source_item(
@@ -15,6 +15,7 @@ from . import (
    bluesky,
    dates,
    dedupe,
    digg,
    entity_extract,
    env,
    github,
@@ -30,6 +31,7 @@ from . import (
    query,
    reddit,
    reddit_public,
    relevance,
    rerank,
    schema,
    signals,
@@ -78,6 +80,7 @@ MOCK_AVAILABLE_SOURCES = [
    "github",
    "perplexity",
    "xquik",
    "digg",
 ]
@@ -105,6 +108,8 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
    available.extend(["hackernews", "polymarket"])
    if config.get("GITHUB_TOKEN") or which("gh"):
        available.append("github")
    if which("digg-pp-cli"):
        available.append("digg")
    if env.is_bluesky_available(config):
        available.append("bluesky")
    if env.is_truthsocial_available(config):
@@ -500,11 +505,12 @@ def _normalize_score_dedupe(
        source, raw_items, from_date, to_date,
        freshness_mode=freshness_mode,
    )
-    normalized = signals.annotate_stream(normalized, ranking_query, freshness_mode)
+    prepared_query = relevance.PreparedQuery(ranking_query)
    normalized = signals.annotate_stream(normalized, prepared_query, freshness_mode)
    normalized = signals.prune_low_relevance(normalized)
    normalized = dedupe.dedupe_items(normalized)
    for item in normalized:
-        item.snippet = snippet.extract_best_snippet(item, ranking_query)
+        item.snippet = snippet.extract_best_snippet(item, prepared_query)
    return normalized
@@ -529,6 +535,12 @@ def _finalize_items_by_source(
            keywords = config.get("_polymarket_keywords") if isinstance(config, dict) else None
            if keywords:
                items = polymarket.filter_items_against_keywords(items, keywords)
        if source == "digg" and items:
            # Pull top-ranked X posts only for the survivors that will appear
            # in the brief. Spending the enrichment budget here (rather than
            # at retrieval time) keeps the inline 'via Digg AI 1000' quotes
            # paired with the clusters dedupe actually kept.
            digg.enrich_source_items(items, top_k=3)
        finalized[source] = items
    return finalized
@@ -964,6 +976,13 @@ def _retrieve_stream(
    if source == "hackernews":
        result = hackernews.search_hackernews(subquery.search_query, from_date, to_date, depth=depth)
        return hackernews.parse_hackernews_response(result, query=subquery.search_query), {}
    if source == "digg":
        result = digg.search_digg(subquery.search_query, from_date, to_date, depth=depth)
        items = digg.parse_digg_response(result, query=subquery.search_query)
        # Enrichment with attached X posts is deferred to
        # _finalize_items_by_source so it runs on the items that actually
        # survive dedupe rather than on top-K of the raw fanout.
        return items, {}
    if source == "bluesky":
        result = bluesky.search_bluesky(subquery.search_query, from_date, to_date, depth=depth, config=config)
        return bluesky.parse_bluesky_response(result), {}
@@ -1056,6 +1075,45 @@ def _mock_stream_results(source: str, subquery: schema.SubQuery) -> tuple[list[d
                "why_relevant": "Brave web search",
            }
        ],
        "digg": [
            {
                "id": "mock1abc",
                "title": f"Digg AI 1000 cluster about {subquery.search_query}",
                "url": "https://di.gg/ai/mock1abc",
                "tldr": f"Curated cluster summarizing recent {subquery.search_query} discussion across the AI 1000.",
                "author": "",
                "date": dates.get_date_range(3)[0],
                "engagement": {"postCount": 8, "uniqueAuthors": 5, "rank": 2, "rank_score": 49.0},
                "first_post_age": "3d",
                "posts": [
                    {
                        "username": "exampledev",
                        "display_name": "Example Dev",
                        "category": "Engineer",
                        "rank": 142,
                        "body": f"Quote from the AI 1000 about {subquery.search_query}.",
                        "post_type": "tweet",
                        "x_url": "https://x.com/exampledev/status/1",
                        "posted_at": dates.get_date_range(3)[0],
                    },
                ],
                "relevance": 0.84,
                "why_relevant": "Mock Digg cluster",
            },
            {
                "id": "mock2def",
                "title": f"Second Digg cluster on {subquery.search_query}",
                "url": "https://di.gg/ai/mock2def",
                "tldr": f"Another angle on {subquery.search_query}.",
                "author": "",
                "date": dates.get_date_range(8)[0],
                "engagement": {"postCount": 3, "uniqueAuthors": 2, "rank": 18, "rank_score": 33.0},
                "first_post_age": "8d",
                "posts": [],
                "relevance": 0.71,
                "why_relevant": "Mock Digg cluster",
            },
        ],
    }
    if source == "grounding":
        return payloads.get(source, []), {
@@ -67,6 +67,7 @@ SOURCE_CAPABILITIES = {
    "bluesky": {"discussion", "social"},
    "truthsocial": {"discussion", "social"},
    "polymarket": {"market"},
    "digg": {"discussion", "social", "link"},
    "xiaohongshu": {"video", "video_shortform", "social"},
    "github": {"discussion", "link"},
    "grounding": {"web", "reference", "link"},
@@ -93,13 +93,6 @@ class GeminiClient(ReasoningClient):
        )
        return extract_gemini_text(payload)
    def ground_search(self, model: str, prompt: str) -> dict[str, Any]:
        return self._generate_content(model, prompt, tools=[{"google_search": {}}])
    def url_context_json(self, model: str, prompt: str) -> dict[str, Any]:
        return self.generate_json(model, prompt, tools=[{"url_context": {}}])
 class OpenAIClient(ReasoningClient):
    name = "openai"
@@ -71,8 +71,29 @@ def _normalize_phrase(text: str) -> str:
    return ' '.join(re.sub(r'[^\w\s]', ' ', text.lower()).split())
 class PreparedQuery:
    """Precomputed query shape reused across items in a stream.
    Built once per ranking_query; reused by token_overlap_relevance so the
    per-item normalize/score loops don't re-tokenize the same query N times.
    """
    __slots__ = ("raw", "q_tokens", "informative_q_tokens", "normalized_phrase")
    def __init__(self, query: str) -> None:
        self.raw = query
        self.q_tokens = tokenize(query)
        informative = {t for t in self.q_tokens if t not in LOW_SIGNAL_QUERY_TOKENS}
        self.informative_q_tokens = informative or self.q_tokens
        self.normalized_phrase = _normalize_phrase(query)
 def _as_prepared(query: "str | PreparedQuery") -> PreparedQuery:
    return query if isinstance(query, PreparedQuery) else PreparedQuery(query)
 def token_overlap_relevance(
-    query: str,
+    query: "str | PreparedQuery",
    text: str,
    hashtags: Optional[List[str]] = None,
 ) -> float:
@@ -95,7 +116,8 @@ def token_overlap_relevance(
    Returns:
        Float between 0.0 and 1.0 (0.5 for empty queries)
    """
-    q_tokens = tokenize(query)
+    prepared = _as_prepared(query)
    q_tokens = prepared.q_tokens
    # Combine text and hashtags for matching
    combined = text
@@ -119,9 +141,7 @@ def token_overlap_relevance(
    if overlap == 0:
        return 0.0
-    informative_q_tokens = {t for t in q_tokens if t not in LOW_SIGNAL_QUERY_TOKENS}
+    informative_q_tokens = prepared.informative_q_tokens
    if not informative_q_tokens:
        informative_q_tokens = q_tokens
    coverage = overlap / len(q_tokens)
    informative_overlap = len(informative_q_tokens & t_tokens) / len(informative_q_tokens)
@@ -129,7 +149,7 @@ def token_overlap_relevance(
    precision = overlap / precision_denominator
    phrase_bonus = 0.0
-    normalized_query = _normalize_phrase(query)
+    normalized_query = prepared.normalized_phrase
    normalized_text = _normalize_phrase(combined)
    if normalized_query and normalized_query in normalized_text:
        phrase_bonus = 0.12 if len(normalized_query.split()) > 1 else 0.16
@@ -12,7 +12,7 @@ from . import dates, schema
 def _skill_version() -> str:
-    """Read plugin version from .claude-plugin/plugin.json if available.
+    """Read plugin version from a plugin manifest if available.
    Tries nearest plugin.json by walking up from render.py's own location.
    Falls back to "?" if not found. This keeps the badge emission from
@@ -20,12 +20,13 @@ def _skill_version() -> str:
    """
    here = pathlib.Path(__file__).resolve()
    for parent in [here.parent, *here.parents]:
-        candidate = parent / ".claude-plugin" / "plugin.json"
+        for manifest_dir in (".codex-plugin", ".claude-plugin"):
-        if candidate.is_file():
+            candidate = parent / manifest_dir / "plugin.json"
-            try:
+            if candidate.is_file():
-                return json.loads(candidate.read_text()).get("version", "?")
+                try:
-            except (json.JSONDecodeError, OSError):
+                    return json.loads(candidate.read_text()).get("version", "?")
-                return "?"
+                except (json.JSONDecodeError, OSError):
                    return "?"
    return "?"
@@ -52,6 +53,7 @@ SOURCE_LABELS = {
    "xiaohongshu": "Xiaohongshu",
    "x": "X",
    "github": "GitHub",
    "digg": "Digg AI 1000",
    "perplexity": "Perplexity",
 }
@@ -170,6 +172,168 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
    return "\n".join(lines).strip() + "\n"
 def render_for_html(
    report: schema.Report,
    synthesis_md: str | None = None,
    *,
    save_path: str | None = None,
 ) -> str:
    """Render markdown intended for shareable HTML conversion.
    This output keeps the public badge, compact source/date metadata, an
    optional one-line data quality note, optional synthesized brief markdown,
    and the engine footer. It deliberately omits the debug file header,
    model-facing safety note, and evidence scratchpad emitted by
    render_compact().
    When synthesis_md is None, the body is intentionally sparse: badge,
    metadata, optional data quality note, and engine footer only.
    """
    lines = [
        *_render_badge(),
        *_render_html_metadata(report),
    ]
    if synthesis_md:
        lines.extend(["", synthesis_md.strip()])
    # Data quality warnings are NOT rendered into the HTML artifact. The HTML
    # is meant to be shared (Slack, email, Notion); recipients haven't asked
    # for technical commentary about how the run was produced. Generators see
    # the same warnings via collect_html_warnings() routed to stderr by the
    # CLI, so they can fix quality issues before sharing.
    _append_html_footer(lines, report, save_path)
    return "\n".join(lines).strip() + "\n"
 def render_for_html_comparison(
    entity_reports: list[tuple[str, schema.Report]],
    synthesis_md: str | None = None,
    *,
    save_path: str | None = None,
 ) -> str:
    """Render comparison markdown intended for shareable HTML conversion.
    Same semantics as render_for_html(), but metadata and data quality notes
    are aggregated across the compared entities.
    """
    if not entity_reports:
        raise ValueError("render_for_html_comparison requires at least one report")
    entities = [label for label, _ in entity_reports]
    main_report = entity_reports[0][1]
    meta = (
        f"<!-- META: {main_report.range_from} to {main_report.range_to} "
        f"· comparing {len(entities)}: {', '.join(entities)} -->"
    )
    lines = [
        *_render_badge(),
        meta,
    ]
    if synthesis_md:
        lines.extend(["", synthesis_md.strip()])
    # Comparison data quality notes also go to stderr, not into the artifact.
    _append_html_footer(lines, main_report, save_path)
    return "\n".join(lines).strip() + "\n"
 def collect_html_warnings(report: schema.Report) -> list[str]:
    """Collect data quality warnings for stderr output (NOT for the HTML artifact).
    Returns a list of human-readable warning strings. Empty list if the run
    was clean. Used by the CLI to emit diagnostics to stderr after writing
    the HTML to stdout/file.
    """
    notes: list[str] = []
    if _render_degraded_run_warning(report):
        notes.append("Run was missing pre-flight resolution. Re-run with `--plan` for richer results.")
    elif _render_pre_research_warning(report):
        notes.append("Pre-research was skipped, so results may be thinner than a resolved run.")
    freshness_warning = _assess_data_freshness(report)
    if freshness_warning:
        notes.append(freshness_warning)
    notes.extend(report.warnings)
    return _dedupe_notes(notes)
 def collect_html_warnings_comparison(
    entity_reports: list[tuple[str, schema.Report]],
 ) -> list[str]:
    """Collect comparison-mode warnings, prefixed by entity label."""
    notes: list[str] = []
    for label, report in entity_reports:
        for w in collect_html_warnings(report):
            notes.append(f"{label}: {w}")
    return notes
 def _render_html_metadata(report: schema.Report) -> list[str]:
    """Inline metadata as an HTML comment marker.
    html_render.py post-processes ``<!-- META: ... -->`` markers into a
    ``<div class="meta">`` after markdown conversion, so the metadata escapes
    the markdown converter's HTML-escaping pass cleanly. Same pattern as the
    PASS_THROUGH_FOOTER marker used for the engine tree.
    """
    non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
    if non_empty:
        sources = ", ".join(_source_label(s) for s in non_empty)
    else:
        sources = "no active sources"
    return [
        f"<!-- META: {report.range_from} to {report.range_to} · {sources} -->",
    ]
 def _render_html_data_quality_note(report: schema.Report) -> str | None:
    notes: list[str] = []
    degraded_warning = _render_degraded_run_warning(report)
    if degraded_warning:
        notes.append("This run was missing pre-flight resolution. Re-run with `--plan` for richer results.")
    pre_research_warning = _render_pre_research_warning(report)
    if pre_research_warning and not degraded_warning:
        notes.append("Pre-research was skipped, so results may be thinner than a resolved run.")
    freshness_warning = _assess_data_freshness(report)
    if freshness_warning:
        notes.append(freshness_warning)
    notes.extend(report.warnings)
    if not notes:
        return None
    return f"> **Data quality note:** {' '.join(_dedupe_notes(notes))}"
 def _render_html_comparison_data_quality_note(
    entity_reports: list[tuple[str, schema.Report]],
 ) -> str | None:
    notes: list[str] = []
    for label, report in entity_reports:
        note = _render_html_data_quality_note(report)
        if note:
            clean = note.removeprefix("> **Data quality note:** ").strip()
            notes.append(f"{label}: {clean}")
    if not notes:
        return None
    return f"> **Data quality note:** {' '.join(_dedupe_notes(notes))}"
 def _dedupe_notes(notes: list[str]) -> list[str]:
    out: list[str] = []
    seen: set[str] = set()
    for note in notes:
        normalized = " ".join(str(note).split())
        if not normalized or normalized in seen:
            continue
        seen.add(normalized)
        out.append(normalized)
    return out
 def _append_html_footer(lines: list[str], report: schema.Report, save_path: str | None) -> None:
    footer = _render_emoji_footer(report, save_path)
    lines.append("")
    lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
    lines.extend(footer)
    lines.append("<!-- END PASS-THROUGH FOOTER -->")
 def _render_canonical_boundary() -> list[str]:
    """Emit the explicit END-OF-CANONICAL-OUTPUT boundary.
@@ -663,7 +827,7 @@ def render_full(report: schema.Report) -> str:
    lines.append("## All Items by Source")
    lines.append("")
    source_order = ["reddit", "x", "youtube", "tiktok", "instagram", "threads", "pinterest",
-                    "hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "perplexity"]
+                    "hackernews", "bluesky", "truthsocial", "polymarket", "grounding", "xiaohongshu", "github", "digg", "perplexity"]
    for source in source_order:
        items = report.items_by_source.get(source, [])
        if not items:
@@ -689,6 +853,9 @@ def render_full(report: schema.Report) -> str:
                    tc_score = tc.get("score", "")
                    attribution = _comment_attribution(item.source, tc.get("author"))
                    lines.append(f"  Top comment {attribution} ({tc_score} {vote_label}): {excerpt}")
            # Digg AI 1000: inline X-post quotes attached to the cluster.
            for post in _digg_posts_for(item, limit=3):
                lines.append(f"  > {_format_digg_quote(post)}")
            # Comment insights for Reddit
            insights = item.metadata.get("comment_insights", [])
            if insights:
@@ -810,6 +977,8 @@ def _render_candidate(candidate: schema.Candidate, prefix: str) -> list[str]:
        source = primary.source if primary else None
        attribution = _comment_attribution(source, tc.get("author"))
        lines.append(f"   - {attribution} ({score} {vote_label}): {_truncate(excerpt.strip(), 240)}")
    for post in _digg_posts_for(primary):
        lines.append(f"   - {_format_digg_quote(post)}")
    insight = _comment_insight(primary)
    if insight:
        lines.append(f"   - Insight: {_truncate(insight, 220)}")
@@ -1060,6 +1229,7 @@ _FOOTER_SOURCES: list[tuple[str, str, str, str, list[tuple[str, str]]]] = [
    ("bluesky",     "🦋", "Bluesky",      "post",     [("likes", "likes"), ("reposts", "reposts")]),
    ("truthsocial", "🇺🇸", "Truth Social", "post",     [("likes", "likes"), ("reposts", "reposts")]),
    ("github",      "🐙", "GitHub",       "item",     [("reactions", "reactions"), ("comments", "comments")]),
    ("digg",        "⛏️", "Digg AI 1000", "cluster",  [("postCount", "posts"), ("uniqueAuthors", "authors")]),
 ]
@@ -1317,6 +1487,7 @@ ENGAGEMENT_DISPLAY: dict[str, list[tuple[str, str]]] = {
    "polymarket":   [],
    "github":       [("reactions", "react"), ("comments", "cmt")],
    "perplexity":   [("citations", "cite")],
    "digg":         [("postCount", "posts"), ("uniqueAuthors", "auth")],
 }
@@ -1504,16 +1675,6 @@ def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score
    return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]
 def _top_comment_excerpt(item: schema.SourceItem | None) -> str | None:
    if not item:
        return None
    comments = item.metadata.get("top_comments") or []
    if not comments or not isinstance(comments[0], dict):
        return None
    top = comments[0]
    return str(top.get("excerpt") or top.get("text") or "").strip() or None
 def _comment_insight(item: schema.SourceItem | None) -> str | None:
    if not item:
        return None
@@ -1523,6 +1684,39 @@ def _comment_insight(item: schema.SourceItem | None) -> str | None:
    return str(insights[0]).strip() or None
 def _digg_posts_for(item: schema.SourceItem | None, limit: int = 2) -> list[dict]:
    """Return up to `limit` parsed Digg posts attached as enrichment to a cluster.
    Returns an empty list for non-digg sources or clusters without enrichment.
    """
    if not item or item.source != "digg":
        return []
    posts = item.metadata.get("posts") or []
    if not isinstance(posts, list):
        return []
    out: list[dict] = []
    for entry in posts:
        if isinstance(entry, dict) and entry.get("body") and entry.get("username"):
            out.append(entry)
        if len(out) >= limit:
            break
    return out
 def _format_digg_quote(post: dict, body_limit: int = 200) -> str:
    """Format a Digg-attached X post as an inline 'via Digg AI 1000' quote line."""
    handle = post.get("username") or ""
    x_url = post.get("x_url") or ""
    body = (post.get("body") or "").replace("\n", " ").strip()
    if len(body) > body_limit:
        body = body[: body_limit - 1].rstrip() + "…"
    if x_url and handle:
        return f"[@{handle}]({x_url}) via Digg AI 1000: {body}"
    if handle:
        return f"@{handle} via Digg AI 1000: {body}"
    return f"via Digg AI 1000: {body}"
 def _transcript_highlights(item: schema.SourceItem | None) -> list[str]:
    if not item or item.source != "youtube":
        return []
@@ -12,6 +12,7 @@ SOURCE_QUALITY = {
    "xiaohongshu": 0.7,
    "hackernews": 0.8,
    "youtube": 0.85,
    "digg": 0.85,
    "reddit": 0.6,
    "x": 0.68,
    "bluesky": 0.66,
@@ -26,7 +27,10 @@ def source_quality(source: str) -> float:
    return SOURCE_QUALITY.get(source, 0.6)
-def local_relevance(item: schema.SourceItem, ranking_query: str) -> float:
+def local_relevance(
    item: schema.SourceItem,
    ranking_query: "str | relevance.PreparedQuery",
 ) -> float:
    text = "\n".join(
        part
        for part in [item.title, item.body, item.snippet]
@@ -92,6 +96,7 @@ ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
    "bluesky":      [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)],
    "truthsocial":  [("likes", 0.45), ("reposts", 0.30), ("replies", 0.25)],
    "polymarket":   [("volume", 0.60), ("liquidity", 0.40)],
    "digg":         [("postCount", 0.40), ("uniqueAuthors", 0.30), ("rank_score", 0.30)],
 }
@@ -175,13 +180,14 @@ def normalize(values: list[float | None]) -> list[int | None]:
 def annotate_stream(
    items: list[schema.SourceItem],
-    ranking_query: str,
+    ranking_query: "str | relevance.PreparedQuery",
    freshness_mode: str,
 ) -> list[schema.SourceItem]:
    """Attach local scoring metadata and return items sorted by local_rank_score."""
    prepared_query = ranking_query if isinstance(ranking_query, relevance.PreparedQuery) else relevance.PreparedQuery(ranking_query)
    engagement_scores = normalize([engagement_raw(item) for item in items])
    for item, eng_score in zip(items, engagement_scores, strict=True):
-        item.local_relevance = local_relevance(item, ranking_query)
+        item.local_relevance = local_relevance(item, prepared_query)
        item.freshness = freshness(item, freshness_mode)
        item.engagement_score = eng_score
        item.source_quality = source_quality(item.source)
@@ -26,7 +26,7 @@ def _windows(words: list[str], size: int, overlap: int) -> list[str]:
 def extract_best_snippet(
    item: schema.SourceItem,
-    ranking_query: str,
+    ranking_query: "str | relevance.PreparedQuery",
    max_words: int = 120,
 ) -> str:
    """Prefer existing snippets, else extract the best matching evidence window."""
@@ -43,8 +43,9 @@ def extract_best_snippet(
    if not candidates:
        return _truncate_words(body, max_words)
    prepared_query = ranking_query if isinstance(ranking_query, relevance.PreparedQuery) else relevance.PreparedQuery(ranking_query)
    best = max(
        candidates,
-        key=lambda candidate: relevance.token_overlap_relevance(ranking_query, candidate),
+        key=lambda candidate: relevance.token_overlap_relevance(prepared_query, candidate),
    )
    return _truncate_words(best, max_words)
@@ -0,0 +1,94 @@
 """Subprocess helpers: safe timeout + process-group cleanup.
 Used by bird_x.py (Node.js Bird search) and youtube_yt.py (yt-dlp search
 and transcript download). Both need the same os.setsid/killpg cleanup
 dance on timeout to avoid orphaning child processes.
 """
 from __future__ import annotations
 import os
 import signal
 import subprocess
 from dataclasses import dataclass
 from typing import Optional, Sequence
 class SubprocTimeout(Exception):
    """Raised when a subprocess exceeds its timeout and is killed."""
@dataclass
 class SubprocResult:
    """Result of a subprocess run that captured stdout and stderr."""
    returncode: int
    stdout: str
    stderr: str
 def run_with_timeout(
    cmd: Sequence[str],
    *,
    timeout: int,
    env: Optional[dict] = None,
    on_pid: Optional[callable] = None,
 ) -> SubprocResult:
    """Run a subprocess with process-group cleanup on timeout.
    Spawns ``cmd`` inside its own process group via ``os.setsid`` where
    available. If ``communicate(timeout=...)`` raises ``TimeoutExpired``,
    signals ``SIGTERM`` to the entire group, falls back to ``proc.kill()``
    if the signal fails, then waits up to 5 seconds for cleanup, and
    raises ``SubprocTimeout``.
    Args:
        cmd: Command and arguments to spawn.
        timeout: Timeout in seconds passed to ``communicate()``.
        env: Optional environment dict. If None, inherits parent env.
        on_pid: Optional callable invoked with the child PID right after
            spawn. Used by bird_x.py to register child PIDs for cleanup
            tracking. Exceptions raised by the callback are suppressed.
    Returns:
        SubprocResult with returncode, stdout, and stderr as strings.
    Raises:
        SubprocTimeout: If the process exceeded ``timeout``.
        FileNotFoundError: If the executable is not found.
        OSError: For other spawn failures.
    """
    preexec = os.setsid if hasattr(os, "setsid") else None
    proc = subprocess.Popen(
        list(cmd),
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
        encoding="utf-8",
        errors="replace",
        preexec_fn=preexec,
        env=env,
    )
    if on_pid is not None:
        try:
            on_pid(proc.pid)
        except Exception:
            pass
    try:
        stdout, stderr = proc.communicate(timeout=timeout)
    except subprocess.TimeoutExpired:
        try:
            os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
        except (ProcessLookupError, PermissionError, OSError):
            proc.kill()
        proc.wait(timeout=5)
        raise SubprocTimeout(f"Command {cmd[0]} timed out after {timeout}s")
    return SubprocResult(
        returncode=proc.returncode,
        stdout=stdout or "",
        stderr=stderr or "",
    )
@@ -124,6 +124,7 @@ SOURCE_COMPLETION_ORDER = [
    "polymarket",
    "grounding",
    "xiaohongshu",
    "digg",
 ]
 SOURCE_COMPLETION_META = {
@@ -138,6 +139,7 @@ SOURCE_COMPLETION_META = {
    "polymarket": ("Polymarket", "market", "markets", Colors.GREEN),
    "grounding": ("Web", "result", "results", Colors.GREEN),
    "xiaohongshu": ("Xiaohongshu", "post", "posts", Colors.RED),
    "digg": ("Digg", "cluster", "clusters", Colors.YELLOW),
 }
@@ -46,9 +46,10 @@ def is_available() -> bool:
            timeout=10,
        )
        return result.returncode == 0 and '"username"' in result.stdout
-    except FileNotFoundError:
+    except (OSError, subprocess.TimeoutExpired):
-        return False
+        # OSError covers FileNotFoundError (no xurl on PATH) and
-    except subprocess.TimeoutExpired:
+        # PermissionError (a non-executable match on PATH, e.g. WSL's
        # /mnt/c/.../WindowsApps shim returning EACCES on exec).
        return False
@@ -8,11 +8,8 @@ Inspired by Peter Steinberger's toolchain approach (yt-dlp + summarize CLI).
 import json
 import math
 import os
 import re
 import signal
 import shutil
 import subprocess
 import sys
 import tempfile
 import urllib.error
@@ -37,7 +34,7 @@ TRANSCRIPT_LIMITS = {
 # Max words to keep from each transcript
 TRANSCRIPT_MAX_WORDS = 5000
-from . import http, log
+from . import http, log, subproc
 from .relevance import token_overlap_relevance as _compute_relevance
@@ -227,30 +224,16 @@ def search_youtube(
        "--no-download",
    ]
    preexec = os.setsid if hasattr(os, 'setsid') else None
    try:
-        proc = subprocess.Popen(
+        result = subproc.run_with_timeout(cmd, timeout=120)
-            cmd,
+    except subproc.SubprocTimeout:
-            stdout=subprocess.PIPE,
+        _log("YouTube search timed out (120s)")
-            stderr=subprocess.PIPE,
+        return {"items": [], "error": "Search timed out"}
            text=True,
            preexec_fn=preexec,
        )
        try:
            stdout, stderr = proc.communicate(timeout=120)
        except subprocess.TimeoutExpired:
            try:
                os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
            except (ProcessLookupError, PermissionError, OSError):
                proc.kill()
            proc.wait(timeout=5)
            _log("YouTube search timed out (120s)")
            return {"items": [], "error": "Search timed out"}
    except FileNotFoundError:
        return {"items": [], "error": "yt-dlp not found"}
-    if not (stdout or "").strip():
+    stdout = result.stdout
    if not stdout.strip():
        _log("YouTube search returned 0 results")
        return {"items": []}
@@ -452,25 +435,10 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
        f"https://www.youtube.com/watch?v={video_id}",
    ]
    preexec = os.setsid if hasattr(os, 'setsid') else None
    try:
-        proc = subprocess.Popen(
+        subproc.run_with_timeout(cmd, timeout=30)
-            cmd,
+    except subproc.SubprocTimeout:
-            stdout=subprocess.PIPE,
+        return None
            stderr=subprocess.PIPE,
            text=True,
            preexec_fn=preexec,
        )
        try:
            proc.communicate(timeout=30)
        except subprocess.TimeoutExpired:
            try:
                os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
            except (ProcessLookupError, PermissionError, OSError):
                proc.kill()
            proc.wait(timeout=5)
            return None
    except FileNotFoundError:
        return None
@@ -556,7 +524,7 @@ def fetch_transcripts_parallel(
                vid = futures[future]
                try:
                    results[vid] = future.result()
-                except (OSError, subprocess.SubprocessError) as exc:
+                except OSError as exc:
                    _log(f"Transcript fetch error for {vid}: {exc}")
                    results[vid] = None
                except Exception as exc:
@@ -346,61 +346,83 @@ def store_findings(
    findings: List[Dict[str, Any]],
 ) -> Dict[str, int]:
    """Store findings with URL-based dedup. Returns counts of new/updated."""
    # Collect findings that have a URL, preserving order.
    with_urls: List[tuple[str, Dict[str, Any]]] = []
    for f in findings:
        url = f.get("source_url") or f.get("url")
        if url:
            with_urls.append((url, f))
    if not with_urls:
        conn = _connect()
        try:
            conn.execute(
                "UPDATE research_runs SET findings_new = 0, findings_updated = 0 WHERE id = ?",
                (run_id,),
            )
            conn.commit()
        finally:
            conn.close()
        return {"new": 0, "updated": 0}
    conn = _connect()
    new_count = 0
    updated_count = 0
    try:
-        for f in findings:
+        # Single batch SELECT to find existing findings by URL.
-            url = f.get("source_url") or f.get("url")
+        urls = [url for url, _ in with_urls]
-            if not url:
+        placeholders = ",".join("?" for _ in urls)
-                continue
+        rows = conn.execute(
            f"SELECT id, source_url, engagement_score FROM findings WHERE source_url IN ({placeholders})",
            urls,
        ).fetchall()
        existing_by_url = {row["source_url"]: row for row in rows}
-            existing = conn.execute(
+        update_rows: List[tuple] = []
-                "SELECT id, engagement_score, sighting_count FROM findings WHERE source_url = ?",
+        insert_rows: List[tuple] = []
                (url,),
            ).fetchone()
        for url, f in with_urls:
            existing = existing_by_url.get(url)
            new_engagement = f.get("engagement_score", 0)
            if existing:
-                # Update engagement and re-sighting info
+                update_rows.append((
-                new_engagement = f.get("engagement_score", 0)
+                    max(new_engagement, existing["engagement_score"] or 0),
-                conn.execute(
+                    run_id,
-                    """UPDATE findings SET
+                    existing["id"],
-                           last_seen = datetime('now'),
+                ))
                           sighting_count = sighting_count + 1,
                           engagement_score = ?,
                           run_id = ?
                       WHERE id = ?""",
                    (
                        max(new_engagement, existing["engagement_score"] or 0),
                        run_id,
                        existing["id"],
                    ),
                )
                updated_count += 1
            else:
-                # New finding
+                insert_rows.append((
-                conn.execute(
+                    run_id,
-                    """INSERT INTO findings
+                    topic_id,
-                       (run_id, topic_id, source, source_url, source_title,
+                    f.get("source", "unknown"),
-                        author, content, summary, engagement_score, relevance_score)
+                    url,
-                       VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+                    f.get("source_title") or f.get("title", ""),
-                    (
+                    f.get("author", ""),
-                        run_id,
+                    f.get("content") or f.get("text", ""),
-                        topic_id,
+                    f.get("summary", ""),
-                        f.get("source", "unknown"),
+                    new_engagement,
-                        url,
+                    f.get("relevance_score", 0),
-                        f.get("source_title") or f.get("title", ""),
+                ))
                        f.get("author", ""),
                        f.get("content") or f.get("text", ""),
                        f.get("summary", ""),
                        f.get("engagement_score", 0),
                        f.get("relevance_score", 0),
                    ),
                )
                new_count += 1
-        # Update run stats
+        if update_rows:
            conn.executemany(
                """UPDATE findings SET
                       last_seen = datetime('now'),
                       sighting_count = sighting_count + 1,
                       engagement_score = ?,
                       run_id = ?
                   WHERE id = ?""",
                update_rows,
            )
        if insert_rows:
            conn.executemany(
                """INSERT INTO findings
                   (run_id, topic_id, source, source_url, source_title,
                    author, content, summary, engagement_score, relevance_score)
                   VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
                insert_rows,
            )
        new_count = len(insert_rows)
        updated_count = len(update_rows)
        conn.execute(
            "UPDATE research_runs SET findings_new = ?, findings_updated = ? WHERE id = ?",
            (new_count, updated_count, run_id),
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Matt Van Horn	80392061d4	chore(release): v3.2.0 (#371 ) Release / build-and-release (push) Has been cancelled Details * chore(release): v3.2.0 Bumps plugin/marketplace/codex/pyproject versions from 3.1.1 to 3.2.0. Promotes the Unreleased CHANGELOG entries (--emit=html, Digg AI 1000 source) to the 3.2.0 release section. * chore(release): bump SKILL.md header and sync.sh path to 3.2.0 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>	2026-05-09 19:29:16 -07:00
Matt Van Horn	c04bd67922	feat: add Digg AI 1000 as an opt-in source (#370 ) * feat(digg): add Digg AI 1000 source module with cluster search and post enrichment - search_digg shells out to digg-pp-cli with --since 30d --agent - parse_digg_response normalizes clusters to last30days dict shape - enrich_with_top_posts attaches top-ranked X posts to top-K clusters - shutil.which gate plus subproc.run_with_timeout discipline matches bird_x.py / youtube_yt.py patterns 25 unit tests cover parse, age window, relevance, binary-missing fallback, timeout recovery, and partial enrichment failures. * feat(digg): wire Digg source into pipeline, normalize, signals, and render pipeline.py: - Import digg, add to MOCK_AVAILABLE_SOURCES, gate via shutil.which - Dispatch case calls search_digg + parse_digg_response, runs enrich_with_top_posts at default/deep depth - Mock fixture includes one enriched cluster + one bare cluster normalize.py: - _normalize_digg maps cluster dicts to SourceItem with container='Digg AI 1000' and metadata.posts pass-through signals.py: - SOURCE_QUALITY['digg'] = 0.85 (top tier alongside YouTube, reflecting Digg's curatorial layer) - ENGAGEMENT_WEIGHTS['digg'] balances postCount, uniqueAuthors, and the rank_score derived from Digg's curatorial position render.py: - SOURCE_LABELS['digg'] = 'Digg AI 1000' - _FOOTER_SOURCES adds '⛏️ Digg AI 1000' line after GitHub - ENGAGEMENT_DISPLAY mirrors footer keys - New _digg_posts_for + _format_digg_quote helpers emit inline '@handle via Digg AI 1000' quotes for clusters with attached X posts; both compact and full-dump renderers call them * feat(digg): polish per-item engagement display and progress label - ENGAGEMENT_DISPLAY for digg uses 'posts' / 'auth' to match the codebase abbreviation convention (HN: 'pts'/'cmt', X: 'rt'/'re') - Footer item word changes from 'story' to 'cluster' to dodge the pre-existing naive plural in _footer_line_for_source ('storys') and to match Digg's actual data model - ui.py SOURCE_COMPLETION_META adds digg with correct 'cluster'/ 'clusters' plural so 'Research complete' shows 'Digg: N clusters' * feat(digg): document Digg AI 1000 source in skill, README, and changelog - planner.py SOURCE_CAPABILITIES adds digg with discussion/social/link capabilities so the planner offers it through the standard fanout - SKILL.md ACTIVE_SOURCES_LIST gate includes 'which digg-pp-cli' check and the source list / available-sources line names digg as opt-in - README.md Sources table adds the Digg AI 1000 row with the activation gate so first-time readers see what they get - CHANGELOG.md Unreleased section calls out the source addition * fix(digg): enrich post-dedupe so brief survivors carry inline quotes Pipeline dispatch was attaching X posts to the top-3 items returned by search, but dedupe later picked different survivors when multiple clusters compared similar (common for trending topics). The brief ended up showing clusters with no posts attached even though enrichment ran successfully on positions 0-2. Move enrichment to _finalize_items_by_source. The new digg.enrich_source_items helper reads metadata['clusterUrlId'] and writes metadata['posts'] in place on the SourceItems that actually survive dedupe. Verified live on 'openclaw': 2 surviving clusters, both now carry real X-post quotes from @sama and @jeremyphoward attributed 'via Digg AI 1000'. Adds 3 unit tests covering survivor enrichment, non-digg skip, and clusterUrlId fallback to item_id. * test(digg): relax live off-topic test to check shape, not emptiness Digg's live search uses fuzzy/popularity fallback, so an impossible token can still return some loosely-related clusters. The contract the pipeline depends on is shape (results is always a list); token-overlap relevance handles the noise downstream. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>	2026-05-09 19:05:41 -07:00
Trevin Chow	b1773be8f3	feat(emit): --emit=html for shareable self-contained briefs (#332 ) Adds a one-command shareable HTML mode to /last30days. The skill detects HTML intent (explicit --emit=html / --emit:html / --html flag in $ARGUMENTS, or natural-language asks like "give me a shareable brief", "for Slack", "export as HTML"), runs the normal research + chat synthesis flow, then saves a self-contained HTML file to ~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat as usual; the HTML is an additional artifact for sharing. User experience: /last30days OpenClaw --emit=html /last30days OpenClaw, give me an HTML brief for Slack Synthesis prints to chat. Last line of the response: "📎 Shareable brief saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it into a message, browser-print to PDF, email it. Architecture: - SKILL.md gets a small detection block (triggers + early exit + MUST/MUST NOT rules + rationale) that points to a reference file. - references/save-html-brief.md owns the implementation: capture the synthesis verbatim into a temp file via heredoc, invoke the engine with --emit=html --synthesis-file, save to disk, append the confirmation line to chat. - lib/render.py exposes render_for_html(report, synthesis_md=None) and render_for_html_comparison(...) -- clean markdown for HTML conversion. Omits debug file header, model-facing safety note, and data quality warnings (those stay in engine stderr; recipients can't act on them in a shared artifact). - lib/html_render.py is a new module: ~200-line CSS template (dark mode default, prefers-color-scheme switch, print stylesheet, mobile breakpoint), stdlib-regex markdown-to-HTML converter, marker-based META + engine-footer wrapping, PROSE_LABELS registry promoting plain -text labels to <h2>, colophon builder. - last30days.py adds --emit=html argparse choice and --synthesis-file PATH flag (engine still callable directly without the skill in the loop). Design: - Voice-led research brief, not corporate report. Inter + JetBrains Mono via Google Fonts with full system fallbacks (no FOIT, works offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp: body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label 20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px. - 720px max-width, generous whitespace, no card layouts or shadows. - Print stylesheet: light theme, A4 margins, [href]::after URL footnotes, page-break-inside:avoid on the engine footer. Templated (locked) shell: - HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS inline. - .badge / .meta / .engine-footer / .colophon containers. Flexible (role-based): - <h2> rendering covers BOTH plain ## headers (comparison mode per LAW 4 exception) AND promoted prose labels via PROSE_LABELS registry. Adding a new SKILL.md prose label is a one-line tuple addition; no CSS or template changes. - Marker-based engine boundaries (<!-- META: ... -->, <!-- PASS-THROUGH FOOTER -->) survive the markdown converter and get promoted post-conversion. Robust to engine output format changes. - Generic markdown-to-HTML for body content; future SKILL.md additions (new sections, tables, blockquotes) render correctly without code changes. Tests: 30 new tests in tests/test_html_render.py covering snapshots (rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose label promotion, warning exclusion from artifact, parseability via html.parser, no-script self-containment. No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip dependencies, no JavaScript anywhere.	2026-05-02 11:30:22 -07:00
Ilia Alshanetsky	5b87cca886	fix(xurl): treat PermissionError from PATH lookup as unavailable (#322 ) is_available() only caught FileNotFoundError and TimeoutExpired. On WSL, a /mnt/c/.../WindowsApps entry on $PATH returns EACCES during exec, and Python raises PermissionError. That escaped is_available() and crashed pipeline.diagnose() before any source ran. Catch OSError instead. It covers FileNotFoundError, PermissionError, and any other spawn-time OS error, so a non-executable xurl on PATH falls through to the next backend instead of aborting the run.	2026-04-26 14:16:14 -07:00
Ilia Alshanetsky	bbf892aecc	refactor: extract subprocess cleanup into shared subproc helper (#210 ) bird_x.py and youtube_yt.py had four near-identical copies of the same subprocess cleanup dance (Popen + os.setsid + communicate(timeout) + SIGTERM via killpg + proc.kill() fallback + wait(5)). Extract to lib.subproc.run_with_timeout(), which: - runs the child in its own process group via os.setsid where available - raises SubprocTimeout on timeout - on timeout: SIGTERM the group, fall back to proc.kill(), wait up to 5s - accepts an on_pid callback so bird_x can still register child PIDs with last30days.register_child_pid for whole-process cleanup - captures stdout/stderr as strings in a SubprocResult dataclass Migrated call sites: _run_bird_search, search_handles inner worker, search_youtube, fetch_transcript. With the helper in place, the signal and subprocess imports became dead in both files (plus os in youtube_yt) and went with them. Tests: 9 new subproc tests cover success, non-zero exit, stderr capture, timeout-raises, timeout-kills-group, missing-command, env passthrough, PID callback, and callback-exception suppression. test_env_v3 and test_youtube_yt patch subproc.run_with_timeout instead of the removed bird_x.subprocess and yt-dlp subprocess.	2026-04-25 14:17:47 -07:00
Ilia Alshanetsky	2acbf8a869	perf: batch store_findings, dedup source_items in O(1), remove dead code (#206 ) 1. N+1 queries in store.store_findings() The old loop ran one SELECT per finding to check existence, then one INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips. Now: one batch SELECT with WHERE source_url IN (...) builds a lookup dict, then executemany() handles all inserts and updates. Query count stays constant regardless of batch size. Benchmark on 500 findings: ~30ms to ~20ms; gap widens on slower storage. 2. O(n^2) source_items dedup in fusion.weighted_rrf() Merging an item into an existing candidate ran any(existing.source == ... for existing in candidate.source_items), linearly scanning a list that grew with each merge. At 40 candidates with 20 source_items each, fusion went quadratic. Now tracks (source, item_id) tuples in a per-candidate set for O(1) lookup. The source_items list itself is unchanged since other code iterates it. 3. Dead code removal - providers.GeminiClient.ground_search() and .url_context_json(): zero callers. Deleted. - render._top_comment_excerpt(): zero callers. Deleted. - env.is_reddit_available(): one-line wrapper around get_reddit_source. Callers can check get_reddit_source(config) is not None directly.	2026-04-25 14:17:17 -07:00
Ilia Alshanetsky	e6b89f2644	perf: cache PreparedQuery per stream, skip double-normalize in dedupe (#282 ) Scoring hot path (_normalize_score_dedupe) re-tokenized the same ranking_query ~240x per stream: once per item for local_relevance, plus ~5x per item across snippet windows. Query tokens are immutable within a stream, so compute them once as relevance.PreparedQuery and thread through signals.annotate_stream and snippet.extract_best_snippet. dedupe._PreparedText called normalize_text twice: once in __init__ and again via get_ngrams. Factor out _ngrams_of_normalized so the prepared path skips the redundant pass while get_ngrams keeps its public contract. Behavior unchanged.	2026-04-25 14:16:57 -07:00
Ilia Alshanetsky	2c2755b49c	refactor(normalize): extract _join_comment_excerpts helper (#283 ) _normalize_reddit, _normalize_hackernews, and _normalize_github inlined the same 5-line comprehension to stringify and space-join the first 3 top_comments' excerpt field. Extract one helper, call it from all three. The comment field name varies per source (Reddit/GitHub use 'excerpt', HN uses 'text'), so it's passed as a parameter. Behavior unchanged.	2026-04-25 14:16:50 -07:00
Ilia Alshanetsky	18b5658674	chore: remove orphan test for deleted generate-synthesis-inputs script (#205 ) tests/test_generate_synthesis_inputs_v3.py imported a script that no longer exists in the repo. The test failed with FileNotFoundError on every run.	2026-04-25 14:16:39 -07:00
Matt Van Horn	145adc9f56	Merge pull request #321 from tmchow/tmchow/review-plugin-json chore: align plugin manifests, add Codex AGENTS.md	2026-04-25 12:34:12 -07:00
Trevin Chow	b100caf2df	fix(plugin): restore marketplace plugin version `tests/test_plugin_contract.py::test_versions_match_across_manifests` enforces that every version-bearing surface agrees: pyproject.toml, SKILL.md, both plugin.json files, AND the marketplace plugin entry. The Claude Code spec says plugin.json wins when both are set, but this repo deliberately mirrors the version across all surfaces and tests it. Restore the field at 3.1.1 to satisfy the contract.	2026-04-24 23:21:02 -07:00
Trevin Chow	dc0cb9850b	chore: add AGENTS.md pointing to CLAUDE.md Codex CLI reads AGENTS.md for repo-level context the way Claude Code reads CLAUDE.md. Delegate to the existing CLAUDE.md so both harnesses share one source of project instructions.	2026-04-24 23:16:32 -07:00
Trevin Chow	ceec99b24c	chore(plugin): clean up plugin manifests - Remove no-op `"hooks": {}` from .claude-plugin/plugin.json (auto-discovery from hooks/hooks.json picks up the SessionStart hook). - Remove redundant `version` from marketplace.json plugin entry; plugin.json is the source of truth per the spec. - Sync description / longDescription across .claude-plugin and .codex-plugin manifests so all surfaces show the same copy.	2026-04-24 23:16:29 -07:00
Dave Morin	d1823a2d05	feat: add PR and issue templates for contributor workflow (#296 ) Adds structured templates to help contributors submit higher-quality PRs and issues. PR template includes testing checklist (pytest, sync.sh). Issue templates use YAML forms for bug reports and feature requests. Fixes #251	2026-04-24 10:49:06 -07:00
Claire Novotny	17caa0526d	ci: validate plugin contract on pull requests	2026-04-24 12:05:39 -04:00
Claire Novotny	f03cb866aa	fix: address plugin layout review feedback	2026-04-24 11:52:48 -04:00
Claire Novotny	72495c1c14	Restructure as Codex plugin	2026-04-23 20:15:02 -04:00
Matt Van Horn	1f7e85a03f	chore(release): v3.1.0 — consolidate 3.0.10-3.0.14 + OpenClaw republish prep (#314 ) Release / build-and-release (push) Has been cancelled Details - Bump plugin.json to 3.1.0 - CHANGELOG entry consolidating 3.0.10-3.0.14 dev cycle and noting OpenClaw republish - Fix broken README link: skills/last30days/SKILL.md -> SKILL.md Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>	2026-04-22 21:56:07 -07:00