fix(skill): carry pitch-vs-pulse rules into the comparison synthesis path

Design-vs-implementation review found the comparison path missing two reinforcements the single-topic path has: the windowed-claims rule (no trend verbs from one 30-day window) and an explicit artifact gate. It also pointed the beat at a "narrative section" the comparison template doesn't have. The per-entity template block now carries an explicit optional pitch-vs-pulse slot (omit entirely when silent - no placeholder), and the comparison instruction names that slot, the windowed rule, and the no-fetch-no-pitch gate. Instructions now sit at the emission point, per the file's own v3.0.6/v3.0.7 lesson about distance between instruction and output.
feat(skill): replace narrative verdict row with evidence-triggered prose beat
2026-06-09 17:33:30 -07:00 · 2026-06-09 17:14:31 -07:00 · 2026-06-09 16:33:29 -07:00 · 2026-06-09 16:24:24 -07:00 · 2026-06-09 16:24:16 -07:00 · 2026-06-09 16:24:07 -07:00
253 changed files with 54187 additions and 4821 deletions
@@ -0,0 +1,20 @@
 {
  "name": "last30days-skill",
  "interface": {
    "displayName": "Last 30 Days"
  },
  "plugins": [
    {
      "name": "last30days",
      "source": {
        "source": "local",
        "path": "./"
      },
      "policy": {
        "installation": "AVAILABLE",
        "authentication": "ON_INSTALL"
      },
      "category": "Research"
    }
  ]
 }
@@ -0,0 +1,24 @@
 {
  "name": "last30days-skill",
  "owner": {
    "name": "Matt Van Horn",
    "url": "https://github.com/mvanhorn"
  },
  "metadata": {
    "description": "Marketplace hosting the Last 30 Days research plugin."
  },
  "plugins": [
    {
      "name": "last30days",
      "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
      "version": "3.3.2",
      "author": {
        "name": "Matt Van Horn",
        "url": "https://github.com/mvanhorn"
      },
      "source": "./",
      "category": "productivity",
      "homepage": "https://github.com/mvanhorn/last30days-skill"
    }
  ]
 }
@@ -0,0 +1,14 @@
 {
  "name": "last30days",
  "version": "3.3.2",
  "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
  "author": {
    "name": "Matt Van Horn",
    "email": "mvanhorn@gmail.com",
    "url": "https://github.com/mvanhorn"
  },
  "homepage": "https://github.com/mvanhorn/last30days-skill",
  "repository": "https://github.com/mvanhorn/last30days-skill",
  "license": "MIT",
  "keywords": ["research", "reddit", "twitter", "youtube", "tiktok", "instagram", "trends", "prompts", "polymarket", "github", "perplexity", "threads", "pinterest", "eli5", "hacker-news"]
 }
@@ -0,0 +1,18 @@
 # Exclude binary assets and dev/test artifacts from ClawHub bundle
 assets/
 docs/
 fixtures/
 tests/
 plans/
 agents/
 variants/
 release-notes.md
 SPEC.md
 TASKS.md
 SKILL-original.md
 *.jsonl
 *.mp3
 *.jpeg
 *.jpg
 *.png
 *.gif
@@ -0,0 +1,44 @@
 # Exclude non-runtime files from `git archive` output.
 # Used by skills/last30days/scripts/build-skill.sh to produce a
 # claude.ai-upload-ready .skill file from the canonical skills/last30days tree.
 # See docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
 # Anthropic canonical skill-packaging excludes
 # (mirrors anthropics/skills/skills/skill-creator/scripts/package_skill.py)
 __pycache__/   export-ignore
 node_modules/  export-ignore
 *.pyc          export-ignore
 .DS_Store      export-ignore
 evals/         export-ignore
 # Dev, docs, test, and media - not needed at skill runtime
 tests/         export-ignore
 docs/          export-ignore
 fixtures/      export-ignore
 assets/        export-ignore
 # NOTE: skills/ and .claude-plugin/ are NOT export-ignored here because
 # Claude Code's /plugin install fetches this same git archive tarball.
 # Removing those from the archive (as v3.0.1 did) silently breaks installs.
 # claude.ai-bundle-specific exclusions live in scripts/build-skill.sh.
 # Historical + repo-only manifests
 SPEC.md            export-ignore
 TASKS.md           export-ignore
 test-run.log       export-ignore
 CONTRIBUTORS.md    export-ignore
 HERMES_SETUP.md    export-ignore
 CHANGELOG.md       export-ignore
 uv.lock            export-ignore
 # Platform adapters are kept in git archives because Claude Code and Codex
 # plugin installs use the same repository archive as their source payload.
 .hermes-plugin/  export-ignore
 # CI workflows - repo-only, not needed at skill runtime
 .github/         export-ignore
 # Build config itself
 .clawhubignore  export-ignore
 .gitignore      export-ignore
 .gitattributes  export-ignore
@@ -0,0 +1,53 @@
 name: Bug Report
 description: Report a bug or unexpected behavior
 labels: [bug]
 body:
  - type: textarea
    id: summary
    attributes:
      label: Summary
      description: What happened?
      placeholder: Describe the bug in 1-2 sentences.
    validations:
      required: true
  - type: textarea
    id: repro
    attributes:
      label: Steps to Reproduce
      description: How can we reproduce this?
      placeholder: |
        1. Run `python3 skills/last30days/scripts/last30days.py "topic" --emit=compact`
        2. ...
    validations:
      required: true
  - type: textarea
    id: expected
    attributes:
      label: Expected Behavior
      description: What should have happened?
    validations:
      required: true
  - type: textarea
    id: traceback
    attributes:
      label: Error / Traceback
      description: Paste the full traceback or error output.
      render: text
  - type: dropdown
    id: install
    attributes:
      label: Install Method
      options:
        - Claude Code plugin
        - Gemini CLI extension
        - Codex plugin
        - Hermes skill
        - Manual (git clone)
        - Other
    validations:
      required: true
  - type: input
    id: os
    attributes:
      label: OS
      placeholder: macOS 15.4, Ubuntu 24.04, Windows 11, etc.
@@ -0,0 +1,24 @@
 name: Feature Request
 description: Suggest a new feature or improvement
 labels: [enhancement]
 body:
  - type: textarea
    id: problem
    attributes:
      label: Problem
      description: What problem does this solve?
      placeholder: When I try to ..., I can't ...
    validations:
      required: true
  - type: textarea
    id: solution
    attributes:
      label: Proposed Solution
      description: How should this work?
    validations:
      required: true
  - type: textarea
    id: alternatives
    attributes:
      label: Alternatives Considered
      description: Other approaches you thought of (optional).
@@ -0,0 +1,19 @@
 ## Summary
 <!-- What does this PR do? 1-3 sentences. -->
 ## Changes
 <!-- Bullet list of what changed. Reference files if helpful. -->
 -
 ## Testing
 <!-- How did you verify this works? -->
 - [ ] Ran `uv run python -m pytest -q --tb=short`
 ## Related Issues
 <!-- Link issues: Fixes #123 or Relates to #456 -->
@@ -0,0 +1,31 @@
 name: Release
 on:
  push:
    tags:
      - "v*"
 permissions:
  contents: write
 jobs:
  build-and-release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Build .skill artifact
        run: |
          bash skills/last30days/scripts/build-skill.sh
          test -f dist/last30days.skill
      - name: Create GitHub release
        uses: softprops/action-gh-release@v2
        with:
          files: dist/last30days.skill
          generate_release_notes: true
          draft: false
          prerelease: false
@@ -0,0 +1,67 @@
 name: Security
 on:
  pull_request:
  push:
    branches:
      - main
  workflow_dispatch:
 permissions:
  contents: read
 jobs:
  dependency-audit:
    name: Dependency audit
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Install uv
        uses: astral-sh/setup-uv@v5
      - name: Set up Python
        run: uv python install 3.12
      - name: Export locked dependency set
        run: |
          uv export \
            --locked \
            --all-groups \
            --no-hashes \
            --format requirements.txt \
            --output-file /tmp/last30days-requirements.txt
      # Advisory-first: visibility before enforcement. This repo handles API keys,
      # cookies, browser tokens, and local env files, so dependency CVEs should be
      # visible in CI logs even before the project has a clean blocking baseline.
      # Set continue-on-error: false once a clean baseline run is confirmed.
      - name: Run pip-audit against locked dependencies
        continue-on-error: true
        run: uvx --python 3.12 pip-audit -r /tmp/last30days-requirements.txt --progress-spinner=off
  secret-scan:
    name: Secret scan
    runs-on: ubuntu-latest
    steps:
      - name: Checkout full history for diff-aware scanning
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      # Advisory-first: this reports verified secrets in pull requests and pushes to
      # main, but does not block merges until maintainers confirm a clean baseline.
      # The TruffleHog action automatically scans the PR range for pull_request
      # events and the pushed commit range for push events.
      # Set continue-on-error: false once a clean baseline run is confirmed.
      # Contributor policy: never commit real secrets in fixtures, tests, docs, or
      # examples; use obvious dummy values and env-based auth patterns instead.
      - name: Run TruffleHog OSS secret scan
        if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
        uses: trufflesecurity/trufflehog@v3.95.2
        continue-on-error: true
        with:
          path: ./
          version: v3.95.2
          extra_args: --only-verified
@@ -0,0 +1,26 @@
 name: Validate
 on:
  pull_request:
  push:
    branches:
      - main
 permissions:
  contents: read
 jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Install uv
        uses: astral-sh/setup-uv@v5
      - name: Set up Python
        run: uv python install 3.12
      - name: Run test suite
        run: uv run pytest
@@ -0,0 +1,34 @@
 # Private benchmark / evaluation artifacts — never push to upstream
 docs/comparison-results/
 scripts/evaluate-synthesis.py
 scripts/generate-synthesis-inputs.py
 fixtures/polymarket_sample.json
 docs/v2.1-tweets.md
 docs/30-day-anniversary-thread.md
 docs/30-day-anniversary-tweets.md
 variants/open/references/research.md
 # OS / tool files
 .DS_Store
 .claude/
 .entire/
 __pycache__/
 *.pyc
 mise.toml
 .memsearch/
 .venv/
 .coverage
 htmlcov/
 # Root vendor/ is accidental - real vendored client lives at scripts/lib/vendor/bird-search/
 /vendor/
 # build artifact from scripts/build-skill.sh
 /dist/
 # Internal planning docs (ce:plan output) — keep local, don't publish
 docs/plans/
 .context/
 /work
 /print
@@ -0,0 +1,66 @@
 # last30days Skill
 Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
 ## Structure
 - `skills/last30days/SKILL.md` — canonical skill definition / runtime spec the model reads when the slash command fires
 - `skills/last30days/scripts/last30days.py` — main research engine
 - `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
 - `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
 - `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
 - `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
 - `CONFIGURATION.md` — user-facing knobs (env vars, flags, per-host install patterns); keep in sync per the rules below
 - `CHANGELOG.md` — structured release history (launch copy lives in GitHub Releases)
 - `HERMES_SETUP.md` — install instructions for the Hermes harness specifically
 ## Orientation
 - This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
 - Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration is incomplete — the model invoking the skill won't know the flag exists.
 - README and PR examples show `/last30days <topic>` first. Direct CLI invocation (`python3 scripts/last30days.py ...`) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
 - Slash commands don't pass shell mechanics through. `/last30days OpenClaw --emit=html | pbcopy` is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full `python3 ...` with explicit flags and a real shell).
 ## Commands
 ```bash
 # Dev/fallback: direct engine invocation (scripting, cron, or engine testing only)
 python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
 npx skills add . -g -y   # copies skill into ~/.agents/skills/<name>/ (frozen at install time); re-run to sync working-tree edits — see Rules below
 # Tests (pytest, ~89 files under tests/, configured in pyproject.toml)
 uv run pytest                              # full suite
 uv run pytest tests/test_dedupe_v3.py      # single file
 uv run pytest tests/test_dedupe_v3.py -k some_case   # single case
 uv run pytest --cov                        # with coverage (skips lib/vendor/)
 ```
 Python 3.12+ required. Use `uv` for the env; the venv lives at `.venv/`.
 ## Rules
 - `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
 - One-time setup: `npx skills add . -g -y` copies the skill into `~/.agents/skills/<name>/` (real directory) and, for harnesses that support symlinked skill dirs, drops a per-host symlink pointing at that copy. **Working-tree edits do NOT propagate automatically** — the `~/.agents/skills/<name>/` copy is frozen at install time. To sync after edits, re-run `npx skills add . -g -y`. For live-edit on a dev machine, replace the install copy with a symlink to the working tree: `ln -sfn "$PWD/skills/last30days" ~/.agents/skills/last30days` (run from the repo root).
 - Git remote: origin = public (`mvanhorn/last30days-skill`)
 ## Security hygiene
 - Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
 - Use the env-based auth patterns in `skills/last30days/scripts/lib/env.py`; tests and fixtures must use obvious dummy values only.
 - Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
 - Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
 ## Maintaining CONFIGURATION.md
 `CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
 Update `CONFIGURATION.md` when:
 - adding a new env var (e.g. `LAST30DAYS_*`, `BSKY_*`, `*_API_KEY`)
 - adding a new CLI flag that affects configuration (e.g. `--store`, `--web-backend`)
 - adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Hermes…)
 - adding a new optional source that requires its own credential
 - changing the priority order of config layers (per-run flag > env > `.env` file > defaults)
 Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / `.env` → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.
 When a new config concept lands in `SKILL.md` or `AGENTS.md`, mirror the user-facing knob in `CONFIGURATION.md` so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.
 ## Beta channel
 Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
@@ -0,0 +1,595 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [Unreleased]
 ### Added
 - **First-party positioning research + pitch-vs-pulse synthesis (company / product / service topics).** A new mandatory research step captures each entity's current stated positioning from first-party sources (homepage, docs, pricing) rather than from memory. The fetched pitch grounds `What it is` descriptions (entities described as they pitch themselves today), helps reject unrelated brand-name noise, and feeds an evidence-triggered prose beat: when the month's conversation directly supports a specific claim, cuts against one, or is squarely about the pitched ground, the synthesis says so anchored to the top thread — and stays silent when the pulse is orthogonal to the pitch, because a manufactured connection is worse than omission. Claims are tested at matched altitude (specific claims against specific threads; broad taglines are never graded against individual items), and statements stay windowed to the 30 days — no trend verdicts. Scoped to entities with an identifiable first party: people are always excluded (even founders whose companies qualify), as are events, abstract concepts, and ownerless topics like Bitcoin; the beat requires positioning fetched during the run, never from memory.
 ### Fixed
 - Entity-grounding rerank demotion now keys on the head token of the primary entity instead of requiring the full multi-word phrase as a contiguous substring. A high-engagement on-entity item (e.g. a 323-pt HN thread titled "Stripe is friendly to 'friendly fraud'") is no longer demoted to score 0 on a `Stripe payments` query just because it lacks the trailing search-hint word. The intended demotion still fires for items that never name the brand at all. The keyless Reddit comment-enrichment slot selection (`_slot_priority`), which mirrors this signal, was updated to the same head-token grounding so the two paths stay consistent.
 ## [3.3.2] - 2026-06-06
 ### Fixed
 - Keyless Reddit comment enrichment now spends its limited slots on entity-matching posts first (mirroring rerank's entity-miss demotion signal) instead of raw upvote order, so off-topic high-upvote threads from broad subreddits no longer consume the comment budget only to be demoted afterward ([#484](https://github.com/mvanhorn/last30days-skill/pull/484))
 ## [3.3.1] - 2026-05-30
 ### Fixed
 - Removed the redundant `commands/last30days.md` wrapper so the plugin exposes only the skill ([#461](https://github.com/mvanhorn/last30days-skill/issues/461)). Previously the plugin shipped both a command wrapper and the skill under the same name, so `/last30` surfaced two `last30days` entries with two different descriptions. The skill already carries its own `argument-hint`, so the `/last30days <topic>` picker UX is unchanged.
 - Corrected the README install note that claimed Claude Code dedupes the slash command across install methods; it does not, so having both the marketplace plugin and the `npx skills` copy active shows two entries.
 ## [3.3.0] - 2026-05-17
 A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner.
 ### Added
 **Emit modes and sources**
 - `--emit=html` for shareable, print-friendly HTML research briefs ([#332](https://github.com/mvanhorn/last30days-skill/pull/332)).
 - **Digg AI 1000 source**, auto-enabled when `digg-pp-cli` is on PATH ([#370](https://github.com/mvanhorn/last30days-skill/pull/370)). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief.
 **Configuration knobs**
 - `EXCLUDE_SOURCES` env var — the inverse of `INCLUDE_SOURCES`, honored in source count and pipeline filter ([#399](https://github.com/mvanhorn/last30days-skill/pull/399)).
 - `LAST30DAYS_YOUTUBE_SSH_HOST` — opt-in SSH routing for `yt-dlp` through a residential-IP host, for users on datacenter VPS hit by YouTube's bot-wall ([#376](https://github.com/mvanhorn/last30days-skill/pull/376)). Host validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection. Transcript path unchanged (uses HTTP fallback).
 - macOS Keychain as a credential source — reads from the system keychain when env vars and config files aren't set ([#407](https://github.com/mvanhorn/last30days-skill/pull/407)).
 - Configuration enablement: env-var defaults and source-resilience patterns across the config layer ([#344](https://github.com/mvanhorn/last30days-skill/pull/344)).
 **Pipeline and storage**
 - Reddit URL auto-enrichment from web search via the public JSON API ([#366](https://github.com/mvanhorn/last30days-skill/pull/366)).
 - Per-run finding sightings recorded in the SQLite store ([#373](https://github.com/mvanhorn/last30days-skill/pull/373)).
 - Brave browser support for X/Twitter cookie extraction ([#320](https://github.com/mvanhorn/last30days-skill/pull/320)).
 **Tests and CI**
 - Full pytest suite restored to CI; 13 rotted tests repaired ([#416](https://github.com/mvanhorn/last30days-skill/pull/416)).
 - `greptile.json` added with `triggerOnUpdates` + `statusCheck` ([#418](https://github.com/mvanhorn/last30days-skill/pull/418)).
 - Advisory security workflow ([#368](https://github.com/mvanhorn/last30days-skill/pull/368)).
 - Parallel grounding backend test coverage ([#355](https://github.com/mvanhorn/last30days-skill/pull/355)).
 **Docs**
 - New `CONFIGURATION.md` with README pointers ([#339](https://github.com/mvanhorn/last30days-skill/pull/339)).
 - `docs/solutions/` learning capture for release-time consistency-test cascades ([#413](https://github.com/mvanhorn/last30days-skill/pull/413)) and the eval-not-in-CI design decision ([#417](https://github.com/mvanhorn/last30days-skill/pull/417)).
 ### Changed
 **Install story modernized**
 - `npx skills add` is now the canonical install path for every harness ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). README and SKILL.md flipped to recommend `npx skills add . -g -y` over per-harness manual instructions. Surfaces Gemini CLI, Copilot, Windsurf, and 50+ other Agent Skills hosts that the install pattern reaches.
 - README dropped the Gemini CLI native-extension install path (now covered by `npx skills add`).
 - `hooks.json` made polyglot for Gemini CLI + Claude Code compatibility ([#318](https://github.com/mvanhorn/last30days-skill/pull/318)).
 **Skill semantics and multi-harness reframe**
 - `AGENTS.md` is now canonical; `CLAUDE.md` points at it ([#410](https://github.com/mvanhorn/last30days-skill/pull/410)). Reframes the project as a multi-harness Agent Skills package rather than a Claude-Code-specific tool.
 - SKILL.md path resolution rewritten: STEP 0 narrows to a Claude-Code-marketplaces-only stale-clone guard; Step 1 walks a single `SKILL_DIR` substitution pattern ([#400](https://github.com/mvanhorn/last30days-skill/pull/400), [#409](https://github.com/mvanhorn/last30days-skill/pull/409)). Removes ~80 lines of bash and fixes a real spec-vs-engine divergence where the previous resolver could pick a different install than the SKILL.md the model loaded from.
 - SKILL.md version regex consolidated into `lib/skill_meta.py` ([#412](https://github.com/mvanhorn/last30days-skill/pull/412)).
 - `--plan` / `--competitors-plan` invocation templates switched from inline single-quoted JSON to heredoc-written tmpfiles ([#404](https://github.com/mvanhorn/last30days-skill/pull/404), fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403)). Apostrophes in resolved context strings ("McDonald's", "people's choice") no longer break shell parsing.
 - `POSTS_PER_CLUSTER` raised 3→5 and render-side display limit 2→3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous caps routinely truncated cluster context.
 - Digg AI 1000 renamed to "Digg" in user-facing output ([#372](https://github.com/mvanhorn/last30days-skill/pull/372)) — footer line, source label, inline-quote suffix, why_relevant, container attribution. Internal references retain the upstream product name.
 - GitHub repo resolution canonicalized for ambiguous product comparisons ([#302](https://github.com/mvanhorn/last30days-skill/pull/302)).
 **Dependencies and tooling**
 - Dropped `requests` runtime dependency. All providers route through stdlib `urllib` via the `lib/http` wrapper ([#393](https://github.com/mvanhorn/last30days-skill/pull/393)).
 - Migrated to `gemini-3.1-flash-lite` GA model ([#378](https://github.com/mvanhorn/last30days-skill/pull/378)).
 - Aligned Codex/Claude plugin manifests + added Codex `AGENTS.md` ([#321](https://github.com/mvanhorn/last30days-skill/pull/321)).
 - pytest dev dep bumped 9.0.2 → 9.0.3 ([#414](https://github.com/mvanhorn/last30days-skill/pull/414)).
 ### Removed
 - **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1 ([#400](https://github.com/mvanhorn/last30days-skill/pull/400)). Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
 - **`skills/last30days/scripts/sync.sh`** — maintainer dev-deploy script ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). Replaced by `npx skills add . -g -y` (live-symlink into every detected harness's skill dir — better than sync.sh's copy model since edits propagate live). Hermes uses `hermes skills install mvanhorn/last30days-skill --force`; OpenClaw uses `clawhub install last30days-official`.
 - Orphaned `SPEC.md` and `TASKS.md` ([#419](https://github.com/mvanhorn/last30days-skill/pull/419)).
 ### Fixed
 **Reddit**
 - `lstrip("r/")` mangled subreddits starting with `r` (`r/robotics` → `obotics`, `r/ruby` → `uby`); replaced with `removeprefix("r/")` at 4 sites (Alex Key, salvaged from #288).
 - Browser-like User-Agent + `Accept-Language`/`Accept-Encoding`/`Connection` headers + gzip decompression to fix `urllib` 403s on Reddit's public JSON endpoint (Franco Carballar, salvaged from #199).
 - HTTP 402 re-raised across all three ScrapeCreators paths (`_global_search`, `_subreddit_search`, `fetch_post_comments`) so the OpenAI/public-JSON fallback chain triggers when credits are exhausted (Jonathan Oppenheim, salvaged from #170).
 **Authentication and credentials**
 - Restored multi-key rotation for `SCRAPECREATORS_API_KEY` accidentally dropped in v3.0.6 (Eric Oberhofer, salvaged from #287). Comma-separated keys round-robin via `random.choice` per run.
 **Windows compatibility**
 - `os.killpg` in `_cleanup_children()` guarded with `hasattr(os, "killpg")`, falls back to `os.kill(SIGTERM)` (gujishh, salvaged from #226).
 - POSIX-style secret-permission warning skipped on Windows ([#357](https://github.com/mvanhorn/last30days-skill/pull/357)).
 - Render uses forward slashes in save-path footer for Windows ([#338](https://github.com/mvanhorn/last30days-skill/pull/338)).
 **xAI / X / xurl**
 - `parse_x_response` now raises `http.HTTPError` on empty output, missing JSON, or decode failure — surfaces in `errors_by_source` instead of silently returning an empty result list (Kaustav Mishra, salvaged from #155).
 - `xurl` treats `PermissionError` from PATH lookup as unavailable ([#322](https://github.com/mvanhorn/last30days-skill/pull/322)).
 **YouTube**
 - SC YouTube + multi-token HN searches unblocked ([#388](https://github.com/mvanhorn/last30days-skill/pull/388)).
 - Transcript-fetch ratio surfaced + degraded-run nudge for stale `yt-dlp` ([#340](https://github.com/mvanhorn/last30days-skill/pull/340)).
 **bird_x / HTTP**
 - Subprocess retry on non-JSON stdout to handle X anti-bot HTML interstitials ([#383](https://github.com/mvanhorn/last30days-skill/pull/383)).
 - HTTP retry budget expanded + exponential backoff on DNS resolution failure ([#382](https://github.com/mvanhorn/last30days-skill/pull/382)).
 - Parallel AI search aligned with current API schema ([#341](https://github.com/mvanhorn/last30days-skill/pull/341)).
 - Parallel web backend routed through grounding ([#354](https://github.com/mvanhorn/last30days-skill/pull/354)).
 **Planner and sources**
 - `xquik` registered in `SOURCE_CAPABILITIES` ([#336](https://github.com/mvanhorn/last30days-skill/pull/336), fixes [#319](https://github.com/mvanhorn/last30days-skill/issues/319)).
 - Honor explicit optional source requests ([#356](https://github.com/mvanhorn/last30days-skill/pull/356)).
 - ScrapeCreators source-gating aligned between code and docs ([#415](https://github.com/mvanhorn/last30days-skill/pull/415)).
 - OpenClaw works without ScrapeCreators key ([#392](https://github.com/mvanhorn/last30days-skill/pull/392), by @thinkun).
 **Render, version display, hosting paths**
 - Hardcoded `v3.0.0` in render replaced with dynamic `_skill_version()` ([#365](https://github.com/mvanhorn/last30days-skill/pull/365)).
 - Comparison HTML artifacts saved correctly ([#389](https://github.com/mvanhorn/last30days-skill/pull/389)).
 - `OPENROUTER_DEFAULT` model ID corrected ([#323](https://github.com/mvanhorn/last30days-skill/pull/323)).
 - OpenClaw poll-timing initialized once ([#358](https://github.com/mvanhorn/last30days-skill/pull/358)).
 - Prefer sandboxed Safari cookie path ([#343](https://github.com/mvanhorn/last30days-skill/pull/343)).
 - Preserve clean mode for last-run state ([#334](https://github.com/mvanhorn/last30days-skill/pull/334)).
 - Replaced hardcoded `/Users/mvanhorn/...` paths in `test-v1-vs-v2.sh` with portable env-var overrides (Dave Morin, salvaged from #297).
 **Hooks**
 - `check-config.sh` path-quoting fix for paths with spaces ([#337](https://github.com/mvanhorn/last30days-skill/pull/337)).
 - Replaced unsafe `eval` with `declare` in `check-config.sh` ([#364](https://github.com/mvanhorn/last30days-skill/pull/364)).
 **Sync and version metadata**
 - `sync.sh` pointed at this repo's plugin cache, not the private repo's ([#402](https://github.com/mvanhorn/last30days-skill/pull/402)).
 - Sync cache target bumped to 3.2.1 to match SKILL.md ([#397](https://github.com/mvanhorn/last30days-skill/pull/397)).
 - ScrapeCreators free-tier credit count corrected to 100 in docs ([#369](https://github.com/mvanhorn/last30days-skill/pull/369), fixes [#367](https://github.com/mvanhorn/last30days-skill/issues/367)).
 - Gemini extension version synced ([#349](https://github.com/mvanhorn/last30days-skill/pull/349)).
 - Various stale path/link fixes ([#345](https://github.com/mvanhorn/last30days-skill/pull/345), [#346](https://github.com/mvanhorn/last30days-skill/pull/346), [#347](https://github.com/mvanhorn/last30days-skill/pull/347), [#348](https://github.com/mvanhorn/last30days-skill/pull/348), [#351](https://github.com/mvanhorn/last30days-skill/pull/351)).
 ### Contributors
 First-time contributors whose fixes shipped in this release (most via PR triage salvage — fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
 - Dave Morin — portable test-harness paths
 - Alex Key — `removeprefix("r/")` for subreddit names
 - Eric Oberhofer — multi-key rotation restored
 - gujishh — Windows process cleanup
 - Franco Carballar — Reddit browser-like headers
 - Jonathan Oppenheim — Reddit 402 fallback chain
 - Kaustav Mishra — xAI error surfacing
 - [@thinkun](https://github.com/thinkun) ([#363](https://github.com/mvanhorn/last30days-skill/pull/363)) — OpenClaw ScrapeCreators-key-optional fix
 Full PR list at [github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0](https://github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0).
 ## [3.2.0] - 2026-05-09
 ### Added
 - Add `--emit=html` for shareable, print-friendly HTML research briefs.
 - **Digg AI 1000 source** (auto-enabled when `digg-pp-cli` is on PATH). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief as `[@handle](xUrl) via Digg AI 1000: ...` lines. Footer line: `⛏️ Digg AI 1000: N clusters │ K posts │ M authors`. No X auth required for the inline quotes since they flow through Digg's read-only endpoints.
 ## [3.1.1] - 2026-04-24
 ### Fixed
 - **Codex plugin layout.** Move the canonical runtime payload under `skills/last30days/` and update Codex/Claude plugin metadata and tests for the relocated engine path.
 - **Claude Code cache resolution.** Resolve Claude plugin installs to `skills/last30days/scripts/last30days.py` after the plugin-layout restructure.
 ## [3.1.0] - 2026-04-22
 Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`, per-entity Step 0.55, vs-mode N passes, comparison title attribution) and republishes the OpenClaw bundle, which had been frozen on ClawHub at `3.0.0-open` since April 8.
 ### Added
 - **OpenClaw republish.** `clawhub install last30days-official` now resolves to `3.1.0-open`, matching current main. Closes [#307](https://github.com/mvanhorn/last30days-skill/issues/307), [#195](https://github.com/mvanhorn/last30days-skill/issues/195), [#236](https://github.com/mvanhorn/last30days-skill/issues/236). The ClawHub bundle had shipped a broken `env.py get_config()` and stale SKILL.md path references since April; both are fixed at source on main and the republish carries the fixes to installers.
 ### Fixed
 - **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
 - **Broken README link.** The README's "source of truth" link pointed at root `SKILL.md`, which is no longer maintained after the plugin-layout restructure. Fixed to point at `skills/last30days/SKILL.md`.
 ### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
 Individual changelog entries for 3.0.10 through 3.0.14 below document the incremental work consolidated into this release.
 ## [3.0.14] - 2026-04-22
 ### Changed
 - **Comparison-mode title attribution.** The synthesis title for vs-mode and `--competitors` outputs changes from `What the Community Says (Last 30 Days)` to `What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range. Three SKILL.md occurrences updated; pure documentation change.
 ## [3.0.13] - 2026-04-22
 ### Changed
 - **vs mode runs N full passes in parallel, one per entity.** Architectural revert of the 3-pass → 1-pass latency optimization from an earlier version. `/last30days "OpenAI vs Anthropic vs xAI"` now runs three full `pipeline.run()` calls in parallel via the same fanout `--competitors` uses, producing three `*-raw.md` save files plus a merged comparison output. Each entity gets its own Step 0.55-grade targeting, own primary X handle weight, own subreddit scoping — apples-to-apples depth instead of the one-pool merged retrieval the single-pass path produced. Parallel execution keeps wall clock ≈ single pass.
 - **`--competitors` is now a SKILL.md-level shortcut for vs-mode with auto-discovery.** The hosting reasoning model (Claude Code, Codex, Hermes, Gemini, any agent with WebSearch) performs discovery and Step 0.55 per entity via its own WebSearch tool, then invokes the engine with a vs-topic and `--competitors-plan` JSON. The engine flag remains for headless/cron use with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys (engine-internal `auto_resolve` stays as fallback).
 - **LAW 7-style stderr for `--competitors` with no backend** now leads with the hosting-model path (WebSearch + Step 0.55 + `--competitors-plan`) instead of `BRAVE_API_KEY`. API-key framing moved to a secondary "headless" section.
 ### Added
 - **`--competitors-plan` JSON flag** for per-entity Step 0.55 targeting. Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`. Accepts inline JSON or a file path (matches `--plan`). When present for an entity, skips engine-internal `auto_resolve` and uses the provided values; missing fields fall back to `auto_resolve` (if backend) or planner defaults. Case-insensitive entity matching. The `subrun_kwargs_for` helper is the single source of truth for per-entity kwargs — no closure-default fallthrough from main scope.
 - **Per-entity save files** when `--save-dir` is set on a vs-mode or `--competitors` run. Each entity's sub-run produces its own `{slug}-raw.md` with a single-row Resolved Entities block — matches historical vs-mode behavior (N passes → N save files).
 - **`--polymarket-keywords "kw1,kw2"`** to filter Polymarket matches for ambiguous single-token topics (e.g., "Warriors" → `nba,gsw,golden-state` kills Glasgow Warriors rugby and Honor of Kings Rogue Warriors noise).
 ### Fixed
 - **BRAVE/SERPER footer nudge suppressed** when `--plan` or `--competitors-plan` is present. The nudge told Claude Code users to set an API key when they already have WebSearch via the hosting model. Nudge still fires for true headless runs (no `--plan`, no backend) where the advice is correct.
 - **Override-leak regression testing.** 3.0.12 already fixed the main-topic `--subreddits` / `--x-handle` / `--github-*` from leaking into peer sub-runs via explicit per-entity kwargs scrubbing. This release adds a 4-test regression suite (`test_competitor_subrun_isolation.py`) locking in the invariant.
 ## [3.0.12] - 2026-04-22
 ### Fixed
 - **Per-entity Step 0.55 resolution for competitor sub-runs.** In 3.0.11, only the main topic got X handle / subreddit / GitHub resolution; competitor sub-runs ran with planner defaults and produced visibly thinner evidence (Reddit 403 fallbacks, single-word queries). Each competitor sub-run now calls `resolve.auto_resolve()` inside `fanout.run_competitor_fanout` when a web backend is available, mirroring the main topic's pre-flight resolution. Per-entity X handle, subreddit list, GitHub user/repos, and news context are threaded into each sub-run's `pipeline.run()` call. Deep-copied config per sub-run prevents `_auto_resolve_context` cross-leak. Surfaces in a new `## Resolved Entities` output block so the resolution coverage is visible without reading stderr.
 - **LAW 7 false-positive on internal fan-out sub-runs.** Each competitor sub-run was emitting the `[Planner] No --plan passed... YOU ARE the planner` stderr warning. LAW 7 targets the hosting-reasoning-model path, not engine-internal fan-out. New `internal_subrun=True` keyword on `planner.plan_query` and `pipeline.run` suppresses the warning for sub-runs only; the default path is unchanged.
 - **Marketplace-stale SKILL.md trap.** Added a STEP 0 canonical-path self-check at the top of SKILL.md. Two of three 2026-04-22 test runs loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (Claude-Code-managed git clone pinned to origin/main, lagging the versioned cache), then ran `--help` against the same stale path, did not see `--competitors`, and fell back to a manual comparison plan. The STEP 0 block forces any reader to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` and re-read from the versioned cache if not.
 ### Changed
 - **Default `--competitors` count is now 2 (3-way total: original + 2 peers).** Previously 3. `--competitors=N` still customizes (range 1..6). Matches the feature description's canonical example (`Kanye vs Drake vs Kendrick`).
 ### Added
 - **`## Resolved Entities` block** in `render_comparison_multi` output. Shows per-entity X handle, subreddits, GitHub user/repos, and truncated context for every entity in the comparison. Block is omitted entirely when no entity has a resolved payload (mock mode, no backend).
 ## [3.0.11] - 2026-04-22
 ### Added
 - **`--competitors` flag for auto-discovered comparison fan-out.** Pass `--competitors` on a single-entity topic and the engine discovers 2-6 peer entities via web search, then runs the full pipeline on each in parallel and emits one N-way comparison. `last30days Kanye West --competitors` resolves Drake, Kendrick Lamar, and one more peer. `last30days OpenAI --competitors` resolves Anthropic, xAI, Google Gemini. `--competitors=N` controls count, `--competitors-list="A,B,C"` skips discovery and uses the explicit list. Discovery mirrors the `auto_resolve` pattern (Brave / Exa / Serper / Parallel) with deterministic text extraction - no internal LLM call. Sub-runs inherit the main `--quick`/`--deep`/`--days`, run in a `ThreadPoolExecutor`, and degrade gracefully when at least 2 entities survive. Output reuses the existing 9-axis `## Head-to-Head` scaffold.
 ## [3.0.10] - 2026-04-21
 ### Added
 - **Commenter handles on evidence lines.** Top-comment rendering now includes the commenter's handle - `u/author` for Reddit, `@handle` for TikTok/YouTube/Instagram/Bluesky/X/Threads. The enrichment adapters already captured `author`; the render layer just was not using it. Evidence lines change from `- Comment (6822 upvotes): Finally, John Apple` to `- u/Cyrisaurus (6822 upvotes): Finally, John Apple`. Person-level citations make synthesis-side inline markdown links per LAW 8 much more natural. Both the compact and full render paths are covered.
 ### Fixed
 - **TikTok author preference.** `_fetch_post_comments` in `scripts/lib/tiktok.py` preferred `user.nickname` over `user.unique_id`, so the engine captured display names ("Moosa Noormahomed") instead of @handles ("moosanoormahomed"). Flipped to prefer `unique_id`. Nickname still wins as a fallback when `unique_id` is missing. Display names can contain emoji, spaces, and non-Latin characters that do not round-trip to a profile URL; the @handle is the stable identifier.
 - **Single plugin payload layout.** The canonical runtime moved to `skills/last30days/` for both Claude Code and Codex plugin loading. Root-level `SKILL.md`, `scripts/`, `agents/`, and `assets/` are no longer maintained as duplicate copies.
 ### Behavior fallback
 - When an author is empty, `[deleted]`, or `[removed]`, the render falls back to the legacy `Comment (...)` shape - no `u/` or `@` prefix with an empty handle is ever emitted.
 ## [3.0.9] - 2026-04-18 - The Self-Debug Release
 ### Highlights
 v3.0.9 adds the engine-side Class 1 keyword-trap refuse-gate ("birthday gift for 40 year old" now gets a clarifying question, not 5 minutes of junk), promotes TikTok and YouTube top comments to the same first-class rendering Reddit's got, lands Hermes AI Agent as a first-class deploy target, and moves the SKILL.md formatting contract from line 1094 to the top of the file.
 "The Self-Debug Release" refers to how the fixes in 3.0.6-3.0.9 were written: 5 separate Opus 4.7 instances each debugged their own failed outputs. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries." I shipped exactly what they said. Validation: 5/5 canonical compliance.
 ### Added
 - **Engine Class 1 keyword-trap refuse-gate** (`scripts/lib/preflight.py`, new). Pattern-matches demographic-shopping queries at main() front-door. Exit code 2 with structured REFUSE message. Escape hatch: `LAST30DAYS_SKIP_PREFLIGHT=1`. 29 tests in `tests/test_preflight.py`.
 - **TikTok + YouTube top comments** rendered with same `💬 Top comment` prominence as Reddit's. Shipped in [#260](https://github.com/mvanhorn/last30days-skill/pull/260); enrichment fixed in [#265](https://github.com/mvanhorn/last30days-skill/pull/265).
 - **Hermes AI Agent as a deploy target** - thanks @stephenmcconnachie ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)). `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys automatically.
 - **Multi-key SCRAPECREATORS_API_KEY rotation** - thanks @zaydiscold ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)). Set `SCRAPECREATORS_API_KEY_1`, `_2`, etc. Engine rotates on rate-limit.
 - **Offline quality evaluation fixture** - thanks @j-sperling ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)). `eval_topics.json` lets contributors run quality regressions without burning live API credits.
 - **END-OF-CANONICAL-OUTPUT boundary** in `render_compact()`. Engine now emits an explicit pass-through instruction so re-synthesis requires actively ignoring a visible boundary.
 - **LAW 1 verbatim-pattern override.** LAW 1 now quotes the exact WebSearch tool-result reminder ("CRITICAL REQUIREMENT: MUST include Sources: section") and declares it OVERRIDDEN inside last30days output.
 ### Changed
 - **SKILL.md restructure.** VOICE CONTRACT LAWs and BADGE MANDATORY block moved from line 1094 to lines 75-150. Grounded in 3 separate Opus 4.7 self-debugs.
 - **Engine emits the badge as stdout.** `🌐 last30days v3.0.9 · synced YYYY-MM-DD` is the first line of every compact emit. Pass-through is now the default-correct behavior.
 - **Reddit client HTTP consolidation** - thanks @iliaal ([#207](https://github.com/mvanhorn/last30days-skill/pull/207)). Migrated to `http.get(params=...)` helper.
 - **ScrapeCreators header consolidation** - thanks @iliaal ([#209](https://github.com/mvanhorn/last30days-skill/pull/209)). `_sc_headers` refactored into `http.scrapecreators_headers`.
 - **Simpler Hermes sync.** `scripts/sync.sh` Hermes branch now always uses main SKILL.md (previously had a `.hermes-plugin/SKILL.md` fallback that created a wrong-file-capture hazard).
 ### Fixed
 - **Peter Steinberger trailing Sources leak.** 2026-04-18 validation failure where the model appended a TechCrunch / TED / Fortune / Wikipedia Sources list after the invitation. Now structurally prevented at three layers: engine emits the canonical body, LAW 1 quotes the exact WebSearch reminder, closing boundary names the anti-pattern.
 - **Wrong-file SKILL.md capture.** Deleted `.agents/skills/last30days/SKILL.md` (1382 lines, April 13 snapshot) and `.hermes-plugin/SKILL.md` (269 lines). One SKILL.md per plugin now, at the plugin root.
 - **GitHub date parsing garbage** - thanks @iliaal ([#208](https://github.com/mvanhorn/last30days-skill/pull/208)). `_parse_date` now rejects invalid input cleanly.
 - **Windows Bird X stability** - thanks @Chelebii ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)).
 - **Linux `check_perms` false-warn** - thanks @george231224 ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)). Uses GNU stat first.
 - **UTF-8 saved output** - thanks @Gujiassh ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)).
 - **Version metadata alignment** - thanks @Gujiassh ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) and @shalomma ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)).
 - **`--days` alias backcompat** - thanks @BryanTegomoh ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)).
 - **`INCLUDE_SOURCES` env default** - thanks @hnshah ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)).
 - **Bird X all-None engagement** - thanks @j-sperling ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)).
 ### Contributors
@j-sperling, @stephenmcconnachie, @zaydiscold, @iliaal, @Chelebii, @Gujiassh, @hnshah, @george231224, @shalomma, @BryanTegomoh for PRs since v3.0.0. @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, @Jah-yee, @dannyshmueli, @Cody-Coyote for issues and PRs that shaped the v3 roadmap.
 ### Recovery
 ```
 /plugin update last30days
 /reload-plugins
 ```
 Verify: `cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version` returns `"version": "3.0.9"`.
 Smoke test: `/last30days birthday gift for 40 year old` should ask a clarifying question before running.
 ## [3.0.5] - 2026-04-15
 ### Added
 - **`/last30days` slash command for plugin users.** New `commands/last30days.md` registers a Claude Code slash command. Users type `/last30days <topic>` and Claude Code's autocomplete prefix-matches it to the canonical `/last30days:last30days` form (the same way `/ce:plan` resolves to `/compound-engineering:ce-plan`). The command delegates to the existing `last30days` skill body — no skill behavior changes.
 ### Removed
 - **`skills/last30days-nux/`** — byte-identical duplicate of root `SKILL.md` that created confusing `/last30days:last30days-nux` autocomplete entries via Claude Code's plugin namespacing. The root `SKILL.md` remains the canonical skill source.
 ### Recovery
 ```
 /plugin update last30days
 /reload-plugins
 ```
 Then type `/last30days <topic>` to invoke the skill via slash command. Natural-language invocation ("search the last 30 days for X") continues to work unchanged.
 ## [3.0.4] - 2026-04-15
 ### Fixed
 - **Cleared `/doctor` path-escape error on Claude Code v2.1.109+.** `.claude-plugin/plugin.json` previously declared `"skills": ["./"]`. That value shipped unchanged from v2.1.0 through v3.0.3 and worked on older Claude Code, but current versions reject `./` with `Path escapes plugin directory: ./ (skills)`. The `"skills"` key is now omitted entirely, matching the pattern used by every other plugin in the Claude Code marketplace ecosystem. Claude Code auto-discovers `skills/*/SKILL.md` when the key is absent.
 ### Recovery
 If `/doctor` reports a path-escape error for last30days, run `/plugin update last30days` then `/reload-plugins`. If errors persist, uninstall and reinstall the plugin.
 ## [3.0.3] - 2026-04-15
 ### Fixed
 - **Restored `skills/` and `.claude-plugin/` to the plugin install tarball.** v3.0.1 added `.gitattributes` rules that excluded both directories from `git archive` output to shrink the claude.ai `.skill` bundle. Claude Code's `/plugin install` fetches the same archive, so users installing v3.0.1 or v3.0.2 received a tarball with no plugin manifest and no skill files. `git archive v3.0.0` contained 8 files under those paths; `v3.0.1` and `v3.0.2` contained 0. This release reverts those `.gitattributes` lines.
 - **Reverted `plugin.json` `"skills"` field to `["./"]`.** v3.0.2 changed this to `["skills"]` based on a misdiagnosis — the manifest change had no effect because the manifest wasn't in the tarball at all. The historical `["./"]` value shipped in every release from v2.1.0 through v3.0.0 without issues and is restored here.
 ### Recovery
 Users on v3.0.1 or v3.0.2: run `/plugin update last30days` then `/reload-plugins`. If autoUpdate is enabled, the next session start will pull v3.0.3 automatically. Users on cached v3.0.0 or earlier installs were unaffected.
 ### Notes
 - The claude.ai `.skill` bundle built by `scripts/build-skill.sh` still works — the archive grew from 89 to 97 files, well under the 200-file cap.
 - claude.ai-specific exclusions (avoiding duplicate `SKILL.md` files in the bundle) should move into `scripts/build-skill.sh` rather than `.gitattributes` in a future release, since `.gitattributes` cannot distinguish between the two distribution channels.
 ## [3.0.2] - 2026-04-15
 ### Fixed
 - **`/last30days` slash command now registers on Claude Code v2.1.105+.** `.claude-plugin/plugin.json` declared `"skills": ["./"]`, which newer Claude Code rejects with `Path escapes plugin directory: ./ (skills)`. The skill silently failed to register, so `/last30days <query>` returned "Unknown command" even though `/plugin list` showed the plugin as installed. Fix: `"skills": ["skills"]` so the loader scans the real skill subdirectory.
 - **Version drift between manifests.** `.claude-plugin/marketplace.json` was pinned to `3.0.0` while `.claude-plugin/plugin.json` advertised `3.0.1`. The `/plugin` resolver used the marketplace version and could install stale cached metadata alongside the correct build. Both manifests now agree on `3.0.2`.
 ### Recovery
 If `/last30days` stopped working for you, run `/plugin update last30days` then `/reload-plugins`. If `/doctor` still reports errors, uninstall and reinstall the plugin from the marketplace.
 ## [3.0.1] - 2026-04-14
 ### Fixed
 - **Skill upload packaging** - `scripts/build-skill.sh` produces a claude.ai-upload-ready `.skill` file that fits under the 200-file cap. Previously, zipping the repo hit 406 files and the "Upload skill" UI rejected it outright.
 - **SKILL.md description length** - trimmed from 228 to 167 chars (Anthropic caps descriptions at 200).
 ### Removed
 - Unused root `vendor/` directory (215 files from an accidental commit in PR #48 - the real vendored X client lives at `scripts/lib/vendor/bird-search/`).
 - Legacy top-level `plans/` directory (superseded by `docs/plans/`; both plans described work that was already shipped in v3).
 ### Added
 - `.gitattributes` with `export-ignore` entries so `git archive` drops tests, docs, fixtures, assets, historical manifests, and internal skill subdirs. Mirrors Anthropic's canonical `package_skill.py` exclusions.
 - `scripts/build-skill.sh` - one-command path to produce `dist/last30days.skill` with a single top-level `last30days/` folder, defensive `=200` file check, and dirty-tree refusal.
 - `README.md` section documenting the claude.ai skill upload workflow.
 ## [3.0.0] - 2026-04-11
 ### Highlights
 Intelligent search, fun judge, cross-source cluster merging, single-pass comparisons, and OpenClaw as a first-class citizen. The v3 engine doesn't just search for your topic -- it figures out *where* to search before the search begins. Engine architecture by @j-sperling.
 ### Added
 - **Intelligent pre-research** -- Resolves X handles, subreddits, TikTok hashtags, and YouTube channels via a new Python brain before any API calls fire. Bidirectional: person to company, product to founder.
 - **Fun judge / Best Takes** -- Second parallel LLM judge scores humor, cleverness, and virality. Surfaces the best reactions in a dedicated output section.
 - **Cross-source cluster merging** -- Entity-based overlap detection merges the same story across Reddit, X, YouTube into one cluster instead of three separate items.
 - **Single-pass comparisons** -- "X vs Y" runs one pass with entity-aware subqueries instead of three serial passes. 3 minutes instead of 12+.
 - **GitHub as a source** -- Stars, reactions, and comments from repos and issues.
 - **OpenClaw first-class citizen** -- Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
 - **Per-author cap** -- Max 3 items per author prevents single-voice dominance.
 - **Entity disambiguation** -- Synthesis trusts resolved handles over keyword matches.
 - **Perplexity Sonar Pro as additive source** -- AI-synthesized research with citations via OpenRouter. Opt-in via `INCLUDE_SOURCES=perplexity`. Returns structured narratives that complement social data.
 - **Perplexity Deep Research** -- `--deep-research` flag for exhaustive 50+ citation reports (~$0.90/query). Premium opt-in for serious investigation.
 - **OpenRouter as reasoning provider** -- One OPENROUTER_API_KEY powers planning, reranking, and Perplexity search. Auto-detected after Gemini/OpenAI/xAI.
 - **Parallel AI grounding backend** -- `--web-backend parallel` or auto-detected via PARALLEL_API_KEY.
 - **Grounding in planner** -- Grounding source properly registered in SOURCE_CAPABILITIES instead of force-injected.
 ### Changed
 - YouTube transcript candidate pool widened 3x past music videos to reach talk/review content with captions
 - Reddit comment enrichment sorted by total engagement (upvotes + comments), not just upvotes
 - Polymarket display shows % odds only; dollar volumes removed
 - 852 tests passing
 ### Fixed
 - Marketplace validation: duplicate `name: last30days` collision in `skills/last30days/SKILL.md` caused strict validators to reject the plugin. Resolved by renaming the internal v3 architecture spec to `last30days-v3-spec` with `user-invocable: false`. Fixed in #214 (reported by @Cody-Coyote in #204).
 - Stale README link to the deleted `skills/last30days-v3/` path from the v3 directory rename. Fixed in #214.
 - OpenAI Codex CLI discoverability: added `.agents/skills/last30days/SKILL.md` as a real file (Codex's loader skips symlinked files) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now registers as `last30days:last30days` when Codex runs in a checkout of the repo. Fixed in #219 (inspired by @Jah-yee in #153 and @dannyshmueli on X).
 ### Contributors
 - @j-sperling -- v3 engine architecture, Python pre-research brain
 - @hnshah -- Watchlist features
 - @Cody-Coyote -- Marketplace validation bug report (#204)
 - @Jah-yee -- Codex CLI integration inspiration (#153)
 ## [2.9.4] - 2026-03-06
 ### Changed
 - Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
 - Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
 - No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research
 ## [2.9.3] - 2026-03-06
 ### Fixed
 - **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
 - Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)
 ## [2.9.2] - 2026-03-06
 ### Fixed
 - Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
 - Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
 - Add `📎` footer line for save path instead of verbose confirmation
 ## [2.9.1] - 2026-03-05
 ### Highlights
 Auto-save research briefings to the default memory directory as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.
 ### Added
 - Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to the default memory directory after every run
 - Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
 - Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
 - Agent mode (`--agent`) also saves research files
 - Brief confirmation after save with the saved file path
 ### Credits
 - [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))
 ## [2.9.0] - 2026-03-05
 ### Highlights
 ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.
 ### Added
 - ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
 - Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
 - `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
 - Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
 - Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
 - Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10
 ### Changed
 - `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
 - Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
 - SKILL.md synthesis instructions updated to emphasize quoting top comments
 ### Fixed
 - Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
 - Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly
 ## [2.8.0] - 2026-03-04
 ### Highlights
 Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.
 ### Added
 - Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
 - `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
 - Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
 - URL-to-name extraction examples in SKILL.md for cleaner web source display
 - `--search=instagram` flag support
 ### Changed
 - TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
 - `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
 - SKILL.md version bumped to v2.8
 - WebSearch citation instruction strengthened to prevent trailing Sources: blocks
 - Security section updated: Apify → ScrapeCreators references
 ### Fixed
 - Web stats line showing full URLs instead of plain domain names
 - Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
 - Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
 - `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding
 ## [2.1.0] - 2026-02-15
 ### Highlights
 Three headline features: watchlists for always-on bots, YouTube transcripts as a 4th source, and Codex CLI compatibility. Plus bundled X search with no external CLI needed.
 ### Added
 - Open-class skill with watchlists, briefings, and history modes (SQLite-backed, FTS5 full-text search, WAL mode) (`feat(open)`)
 - YouTube as a 4th research source via yt-dlp -- search, view counts, and auto-generated transcript extraction (`feat: Add YouTube`)
 - OpenAI Codex CLI compatibility -- install to `~/.agents/skills/last30days`, invoke with `$last30days` (`feat: Add Codex CLI`)
 - Bundled X search -- vendored subset of Bird's Twitter GraphQL client (MIT, originally by @steipete), no external CLI needed (`v2.1: Bundle Bird X search`)
 - Native web search backends: Parallel AI, Brave Search, OpenRouter/Perplexity Sonar Pro (`feat(engine)`)
 - `--diagnose` flag for checking available sources and authentication status
 - `--store` flag for SQLite accumulation (open variant)
 - Conversational first-run experience (NUX) with dynamic source status (`feat(nux)`)
 ### Changed
 - Smarter query construction -- strips noise words, auto-retries with shorter queries when X returns 0 results
 - Two-phase search architecture -- Phase 1 discovers entities (@handles, r/subreddits), Phase 2 drills into them
 - Reddit JSON enrichment -- real upvotes, comments, and upvote ratio from reddit.com/.json endpoint
 - Engagement-weighted scoring: relevance 45%, recency 25%, engagement 30% (log1p dampening)
 - Model auto-selection with 7-day cache and fallback chain (gpt-4.1 -> gpt-4o -> gpt-4o-mini)
 - `--days=N` configurable lookback flag (thanks @jonthebeef, [#18](https://github.com/mvanhorn/last30days-skill/pull/18))
 - Model fallback for unverified orgs (thanks @levineam, [#16](https://github.com/mvanhorn/last30days-skill/pull/16))
 - Marketplace plugin support via `.claude-plugin/plugin.json` (inspired by @galligan, [#1](https://github.com/mvanhorn/last30days-skill/pull/1))
 ### Fixed
 - YouTube timeout increased to 90s, Reddit 429 rate limit fail-fast
 - YouTube soft date filter -- keeps evergreen content instead of filtering to 0 results
 - Eager import crash in `__init__.py` that broke Codex environments
 - Reddit future timeout (same pattern as YouTube timeout bug)
 - Process cleanup on timeout/kill -- tracks child PIDs for clean shutdown
 - Windows Unicode fix for cp1252 emoji crash (thanks @JosephOIbrahim, [#17](https://github.com/mvanhorn/last30days-skill/pull/17))
 - X search returning 0 results on popular topics due to over-specific queries
 ### New Contributors
 - @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
 - @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
 - @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))
 ### Credits
 - @galligan -- Marketplace plugin inspiration
 - @hutchins -- Pushed for YouTube feature
 ## [1.0.0] - 2026-01-15
 Initial public release. Reddit + X search via OpenAI Responses API and xAI API.
 [3.0.9]: https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
 [2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
 [2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
 [2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
 [2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
 [1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0
@@ -0,0 +1 @@
@AGENTS.md
@@ -0,0 +1,47 @@
 # Concepts
 Shared vocabulary for `last30days-skill`. Terms here have a precise project-specific meaning — distinct enough from their general technical sense that a new contributor would need them defined to follow conversations, PR descriptions, or the SKILL.md contract.
 ## The package
 ### Skill
 A self-contained agent-instructions package consisting of a `SKILL.md` prose contract plus a sibling `scripts/` directory containing the executable code the SKILL.md invokes. The package conforms to the [Agent Skills](https://agentskills.io) open format and installs across every major harness (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ others) via `npx skills add`, harness-native plugin installers, or per-harness skill directories. A Skill is the unit of distribution; the Skill is the product.
 ### Engine
 The Python script (`scripts/last30days.py`) the Skill's SKILL.md invokes to do the actual research work. The Engine and SKILL.md have a contract: SKILL.md tells the model which flags to pass (`--plan`, `--competitors-plan`, `--x-handle`, `--subreddits`, `--emit=compact`, etc.), and the Engine produces a specific output shape (badge line, ranked evidence clusters, emoji-tree footer) that the model is contractually required to pass through. The Engine is implementation; the SKILL.md prose is the agent-facing surface.
 ### Harness
 The agent runtime that loads Skills and invokes them on the user's behalf. Claude Code is the most common Harness for this Skill but not the only one — Codex, Cursor, GitHub Copilot, Gemini CLI, and the rest of the Agent Skills ecosystem also count. "Multi-harness" describes a Skill that works correctly across every Harness it installs into; features written without multi-harness awareness (e.g., engine flags with no SKILL.md integration, or paths hardcoded to one Harness's install layout) regress on Harnesses other than the one they were tested against.
 ## Research pipeline
 ### Primary entity
 The brand or proper-noun core of a research topic — the topic with its Intent modifier stripped. It is what the research is *about*, as distinct from how the user phrased the search.
 ### Intent modifier
 A trailing word or phrase in a topic that expresses what the user wants to know rather than what the topic is ("review", "use cases", "pricing"). Stripped when deriving the Primary entity.
 ### Entity grounding
 The check that a candidate item plausibly mentions the Primary entity before final ranking. Grounding keys on the head token (first word) of the Primary entity rather than the full phrase — trailing words are usually search descriptors, so requiring them falsely demotes on-entity items.
 An item that fails grounding receives a decisive entity-miss demotion, designed so engagement cannot rescue off-entity content. Because the demotion is decisive, the grounding bar is deliberately conservative: its failure modes degrade toward "no penalty," never toward burying on-entity signal.
 ### Keyless path
 The research flow available with no API keys: source data is gathered by scraping and RSS rather than authenticated APIs, and ranking falls back to local scoring instead of LLM-based reranking. This is the free tier of the Skill; lexical quality safeguards like Entity grounding matter most here, because no LLM is available to judge relevance semantically.
 ### Comment-enrichment slots
 The small, depth-dependent budget of Reddit posts whose comments get fetched in the Keyless path. Slot selection is relevance-aware: posts that pass Entity grounding claim slots first, so the budget is not spent on high-engagement posts that final ranking will demote anyway.
 ## Distribution
 ### Beta channel
 A parallel install of the Skill, sourced from the private `mvanhorn/last30days-skill-private` repo and installed as `/last30days-beta` rather than `/last30days`. The Beta channel exists so experimental changes can be tested by real users before they ship to the public `/last30days`. Promotion from Beta to public happens via a review PR against this (public) repo — Beta-only changes never ship to public without that PR. The Beta channel workflow guide lives in `BETA.md` in the private repo.
@@ -0,0 +1,268 @@
 # Configuration
 Everything you can tune in `/last30days` without editing the engine source.
 Three layers, in order of how often you'll touch them:
 1. **Per-run flags** - what you pass on the command line.
 2. **Environment variables and `.env`** - what's enabled across all runs.
 3. **Optional trend-monitoring stack** - SQLite store, watchlist, briefings.
 Per-client patterns and the experimental beta channel are at the bottom.
 > Skip ahead: [Where output is saved](#where-output-is-saved) - [API keys](#api-keys-env) - [Reasoning provider](#reasoning-provider-priority) - [Web search backend](#web-search-backend-priority) - [Trend monitoring](#trend-monitoring-store--watchlist--briefings) - [Per-client patterns](#per-client-patterns) - [Beta channel](#beta-channel)
 ## Why this document exists
 This is a focused **configuration reference** maintained alongside the engine. The runtime contract (the voice rules, the planner protocol, the LAWs the synthesizing model follows) lives in [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) - that file is authoritative when the two ever differ. This file's job is narrower: surface every knob a user or operator can turn, in one place, kept current with the code so client-facing setups stay reliable. New configuration knobs added to the engine should be reflected here in the same PR.
 ---
 ## Where output is saved
 | Platform | Default path | Override |
 |---|---|---|
 | Linux / macOS | `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` | set `LAST30DAYS_MEMORY_DIR=/path` |
 | Windows | `LAST30DAYS_MEMORY_DIR` defaults to `C:\Users\<you>\Documents\Last30Days\` | set `LAST30DAYS_MEMORY_DIR=C:\path` |
 Each run produces one file per topic, slug-named:
 `<slug>-raw[-suffix].md`. Same topic + same suffix on the same day overwrites; same topic + same suffix on different days appends a date stamp.
 **Per-run overrides:**
 - `--save-dir <path>` - one-off output location.
 - `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
 The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1.
 ---
 ## API keys (`.env`)
 The skill reads keys from a `.env` file. Two locations are supported, in priority order:
 1. **`.claude/last30days.env`** in the current project directory (project-scoped) - takes precedence when present.
 2. **`~/.config/last30days/.env`** at the user level (global default) - the fallback.
 Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't.
 The project-scoped file is the cleanest pattern for **per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), `cd` into that folder, and the skill picks up that client's configuration automatically. No wrapper scripts needed for the common case.
 **Source-by-source** - what each key unlocks:
 | Source | Key(s) | Required for | Free tier |
 |---|---|---|---|
 | Reddit (public) | none | always on | yes |
 | Hacker News | none | always on | yes |
 | Polymarket | none | always on | yes |
 | GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
 | YouTube | `yt-dlp` CLI installed | always on if `yt-dlp` present | yes |
 | X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; xAI / ScrapeCreators = paid |
 | TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
 | Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
 | Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
 | Pinterest | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `pinterest` | Pinterest items | 10K free calls |
 | Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | Bluesky items | yes (app password at bsky.app) |
 | TruthSocial | `TRUTHSOCIAL_TOKEN` | TruthSocial items | yes |
 | Web search | one of: `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` | `--auto-resolve` and Step 2 supplements | Brave has a free tier; native WebSearch on Claude Code / Codex / Gemini works as a fallback |
 | Perplexity Deep Research | `OPENROUTER_API_KEY` | `--deep-research` flag (~$0.90/query) | no |
 | Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) |
 **Example `.env` skeleton** (placeholders only - replace with your own values):
 ```bash
 # Reasoning + planning (one provider; see priority below)
 GOOGLE_API_KEY=<your-gemini-key>
 # Web search backend (one is enough; Brave is the cheapest)
 BRAVE_API_KEY=<your-brave-key>
 # Optional sources
 SCRAPECREATORS_API_KEY=<your-scrapecreators-key>
 INCLUDE_SOURCES=tiktok,instagram
 # X authentication (one option only)
 XAI_API_KEY=<your-xai-key>
 # OR cookie-jar (no key needed; logs in via your browser session)
 # FROM_BROWSER=firefox
 # Bluesky
 BSKY_HANDLE=<your-handle>.bsky.social
 BSKY_APP_PASSWORD=<your-app-password>
 ```
 After editing: `chmod 600 ~/.config/last30days/.env` (or `chmod 600 .claude/last30days.env` if using the project-scoped variant).
 **Troubleshooting:** if a source you expected to see isn't appearing in results, run `python3 scripts/last30days.py --diagnose`. It prints a per-source availability report (which keys were detected, which CLIs are installed, which backends are reachable) without running a full search.
 ### Bluesky app-password format and search host
 `BSKY_APP_PASSWORD` should be a 19-char app password in `xxxx-xxxx-xxxx-xxxx` format (lowercase alphanumeric, three hyphens). Generate one at <https://bsky.app/settings/app-passwords>. The AT Protocol's `createSession` endpoint also accepts your main account login password, but that's bad hygiene — main passwords have no scope (an app password can be limited to non-DM access) and can't be revoked individually.
 The skill defaults to `api.bsky.app` for `searchPosts`, which is the canonical authenticated AppView. The previous default `public.api.bsky.app` is the unauthenticated public mirror and is currently blocked by BunnyCDN for `searchPosts` regardless of auth header (verified 2026-05-04). If Bluesky migrates infrastructure again, override the host without a code change by setting `BSKY_SEARCH_HOST` in your `.env`:
 ```bash
 BSKY_SEARCH_HOST=api.bsky.app   # default — change only if Bluesky moves
 ```
 ---
 ## Reasoning provider priority
 `/last30days` needs one reasoning model for planning + reranking when you don't pass `--plan` yourself. Auto-detect priority (set `LAST30DAYS_REASONING_PROVIDER=<name>` to pin one):
 1. **Gemini** - `GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY`
 2. **OpenAI** - `OPENAI_API_KEY` (or Codex auth at `~/.codex/auth.json`)
 3. **xAI** - `XAI_API_KEY`
 4. **OpenRouter** - `OPENROUTER_API_KEY` (also unlocks `--deep-research`)
 5. **Local / deterministic** - always available, lowest quality
 When you invoke `/last30days` from Claude Code, Codex, or Gemini, the host model **is** the reasoning provider for plan + synthesis - you don't need any of the keys above unless you also run the script headlessly (cron, CI, watchlist).
 ---
 ## Web search backend priority
 Used by `--auto-resolve` (when WebSearch isn't available from the host) and Step 2 supplements. Auto-detect priority (override per-run with `--web-backend=<name>`):
 1. **Brave** - `BRAVE_API_KEY`
 2. **Exa** - `EXA_API_KEY`
 3. **Serper** - `SERPER_API_KEY`
 4. **Parallel** - `PARALLEL_API_KEY`
 5. **Host's native WebSearch** - Claude Code, Codex, Gemini all have one built in
 Visible quality difference between hosts with vs without a configured backend. If your client setup produces thinner results than yours, this is usually why.
 ---
 ## Trend monitoring (`--store` + watchlist + briefings)
 The default behavior - one slug-named file per topic, overwritten on rerun - is the snapshot mode. For continuous monitoring, the repo ships three components most users miss:
 ### `--store` flag
 Adding `--store` to any run persists every finding to a SQLite database (default at `~/.local/share/last30days/research.db`). Findings dedupe on the `source_url` column (UNIQUE constraint), so the same URL across runs updates the existing row instead of creating a duplicate. The markdown file still saves; the SQLite is the time-series substrate.
 **Always-on alternative:** set `LAST30DAYS_STORE=1` in your `.env` instead of remembering `--store` on every invocation. The flag still works as before; the env var is purely additive. Same hybrid pattern as `LAST30DAYS_DEBUG` — works whether shell-exported or in `.env`.
 Relevant tables: `topics`, `research_runs`, `findings`, `settings`. Schema: [`scripts/store.py`](skills/last30days/scripts/store.py).
 ### `watchlist.py` - recurring topics
 [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) manages topics that should be researched on a schedule. Subcommands: `add`, `remove`, `list`, `run-one`, `run-all`, `config`. Built-in delivery to Slack incoming webhooks (`hooks.slack.com/...`) or any HTTPS endpoint, fired only when new findings appear.
 Two-step flow (the watchlist holds the topic; an external scheduler invokes the run):
 ```bash
 # 1. Add the topic to the watchlist
 #    Default schedule daily 8am; --weekly switches to Mondays 8am
 python3 scripts/watchlist.py add "british airways middle east" --weekly
 # 2. Configure delivery and budget (optional)
 python3 scripts/watchlist.py config delivery "https://hooks.slack.com/services/..."
 python3 scripts/watchlist.py config budget 5.00
 # 3. Trigger via cron / Task Scheduler / GitHub Actions
 python3 scripts/watchlist.py run-one "british airways middle east"
 # or run every enabled topic, gated by daily_budget
 python3 scripts/watchlist.py run-all
 ```
 The schedule field stored on each topic is metadata - the actual cron / Task Scheduler invocation is your responsibility. Watchlist runs hardcode `--quick` and `--lookback-days 90` when spawning the underlying engine.
 ### `briefing.py` - daily / weekly digests
 [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) reads the SQLite store and emits structured data the agent then synthesizes into prose. Modes: `generate` (daily), `generate --weekly`, `show [--date DATE]` (display a saved briefing). Briefs save to `~/.local/share/last30days/briefs/`.
 ### Recommended cadence pattern
 | Step | Cadence | Command |
 |---|---|---|
 | Baseline | one-time per topic | `/last30days "<topic>" --days=30 --store` |
 | Add to watchlist | one-time per topic | `python3 scripts/watchlist.py add "<topic>" --weekly` |
 | Recurring run | daily or weekly (external scheduler) | `python3 scripts/watchlist.py run-all` |
 | Digest | weekly | `python3 scripts/briefing.py generate --weekly` |
 ---
 ## Per-client patterns
 The skill is built to flex around different client environments. Four patterns that compose well:
 ### 1. Per-client `.claude/last30days.env` (preferred when you cd into client folders)
 The simplest pattern when each client has its own working directory: drop a `.claude/last30days.env` into the client folder. The skill picks it up automatically (see [API keys](#api-keys-env) for the lookup priority). Typical contents:
 ```bash
 LAST30DAYS_MEMORY_DIR=C:\Users\<you>\Clients\acme\Research\Last30Days
 SCRAPECREATORS_API_KEY=<acme-scoped-key-or-shared>
 INCLUDE_SOURCES=tiktok,instagram
 BSKY_HANDLE=<acme-bluesky-handle>.bsky.social
 ```
 `cd` into the client folder, run `/last30days <topic>` as normal, no flags or wrappers. Combine with `--save-suffix=<client-slug>` per run if you also need to differentiate filenames within that folder.
 ### 2. Per-client save dir + suffix wrapper
 For workflows where you don't `cd` into a client folder (running from anywhere, scripted batches), a tiny shell function isolates each client's research without engine changes.
 PowerShell example:
 ```powershell
 function Run-L30D-Client {
    param([string]$ClientSlug, [Parameter(ValueFromRemainingArguments=$true)]$Args)
    $env:LAST30DAYS_MEMORY_DIR = "C:\Users\$env:USERNAME\Clients\$ClientSlug\Research\Last30Days"
    /last30days @Args --save-suffix=$ClientSlug
 }
 # Usage: Run-L30D-Client acme "british airways middle east"
 ```
 Bash example:
 ```bash
 l30d-client() {
    local client=$1; shift
    LAST30DAYS_MEMORY_DIR="$HOME/Clients/$client/Research/Last30Days" \
        /last30days "$@" --save-suffix="$client"
 }
 # Usage: l30d-client acme "british airways middle east"
 ```
 ### 3. Custom category-peer subreddits
 [`scripts/lib/categories.py`](skills/last30days/scripts/lib/categories.py) holds a table of `(category_id, trigger_keywords, peer_subreddits)`. If a client lives in a vertical that isn't covered (legal-tech, real-estate-tech, B2B HR SaaS), add a row. Pure data, no logic.
 Section 2a of `SKILL.md` documents the merging rule the skill applies when your topic matches a category.
 ### 4. Pre-built `--competitors-plan` JSON
 For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client industry saves real time:
 ```json
 {
  "Competitor B": {
    "x_handle": "competitor_b_handle",
    "subreddits": ["sub1", "sub2"],
    "github_user": "competitor-b-org",
    "context": "Founded 2019, focused on ..."
  },
  "Competitor C": { ... }
 }
 ```
 Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema.
 ---
 ## Beta channel
 Experimental customizations live on a private companion repo (`mvanhorn/last30days-skill-private`) installed as `/last30days-beta`. Never ship beta-only changes to the public marketplace without a review PR against the public repo. Workflow guide: `BETA.md` in the private repo.
 This is the right home for client-specific changes you don't intend to upstream - custom category rows, internal subreddit lists, per-vertical plan templates.
 ---
 ## Cross-references
 - The CLI flag surface: `python3 scripts/last30days.py --help`
 - The skill contract (voice, LAWs, pre-flight protocol): [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md)
 - Engine spec (some sections stale; SKILL.md wins on conflicts): [`SPEC.md`](SPEC.md)
 - Contributor guidance: [`CONTRIBUTORS.md`](CONTRIBUTORS.md)
@@ -0,0 +1,59 @@
 # Contributors
 last30days is built by [@mvanhorn](https://github.com/mvanhorn) with help from the community.
 ## v3 Inspiration
 These contributors submitted PRs and issues that directly inspired v3 features. The v3 engine was a ground-up rewrite, so their original code wasn't merged, but their ideas shaped what shipped.
 Want to claim your entry? Submit a PR replacing the placeholder line below your name with your bio, website, or anything you'd like.
 ---
 ### @uppinote20
 [PR #143](https://github.com/mvanhorn/last30days-skill/pull/143) - Rich Reddit comments, top 3 per post
 v3 ships top comments with upvote counts on every thread.
 > _Add your bio, website, or anything you'd like here._
 ### @zerone0x
 [Issue #134](https://github.com/mvanhorn/last30days-skill/issues/134) + [PR #136](https://github.com/mvanhorn/last30days-skill/pull/136) - GitHub as a first-class data source
 v3 has full GitHub search: issues, PRs, person-mode profiles, project-mode repos with live star counts.
 > _Add your bio, website, or anything you'd like here._
 ### @thinkun
 [PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results
 v3 has parallel enrichment with per-item timeouts. No results are ever dropped.
 > Thinker, technologist, AI expert, music-tinkerer. Founder of [Thinkun](https://thinkun.com). [@thinkun on GitHub](https://github.com/thinkun) · [@unthink on X](https://x.com/unthink)
 ### @thomasmktong
 [PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback
 v3 Reddit is 100% pure Python with zero external dependencies.
 > _Add your bio, website, or anything you'd like here._
 ### @fanispoulinakisai-boop
 [Issue #100](https://github.com/mvanhorn/last30days-skill/issues/100) - Reddit timeout report
 Drove the timeout resilience work that made v3 Reddit bulletproof.
 > _Add your bio, website, or anything you'd like here._
 ### @pejmanjohn
 [Issue #78](https://github.com/mvanhorn/last30days-skill/issues/78) - ScrapeCreators silent failures
 v3 surfaces all API errors with clear diagnostics instead of silently returning empty results.
 > Repping the mighty MI; home of the most cracked agentic engineers. https://github.com/pejmanjohn
 ### @zl190
 [PR #115](https://github.com/mvanhorn/last30days-skill/pull/115) - HN trending merge
 v3 merges trending and keyword HN results with deduplication for better coverage.
 > Healthcare AI engineer. [Blog](https://zl190.github.io/blog)
 ### @hnshah
 [PR #84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86) - Watchlist delivery, 90-day scanning window, HN/Polymarket storage
 v3 has durable watchlist with multi-source storage and extended time windows.
 > Hiten Shah. Founder. Builds in public. https://github.com/hnshah
 ---
 ## Past Contributors
 - [@JosephOIbrahim](https://github.com/JosephOIbrahim) - Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
 - [@levineam](https://github.com/levineam) - Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
 - [@jonthebeef](https://github.com/jonthebeef) - Early testing and feedback
@@ -0,0 +1,111 @@
 # Hermes Setup Guide for last30days
 This guide covers installing last30days on Hermes AI Agent.
 ## Prerequisites
 1. **Hermes installed** - See https://github.com/mercurial-tf/hermes
 2. **Python 3.12+** - `brew install python@3.12` or similar
 3. **yt-dlp** (optional, for YouTube) - `brew install yt-dlp`
 ## Installation
 ```bash
 hermes skills install mvanhorn/last30days-skill --force
 ```
 This pulls the latest release from GitHub and deploys to `~/.hermes/skills/research/last30days/`. `--force` reinstalls over any existing copy.
 ### Developer / live-edit alternative
 If you're hacking on the skill locally and want edits to propagate to Hermes without re-installing, symlink your working tree:
 ```bash
 git clone https://github.com/mvanhorn/last30days-skill.git
 mkdir -p ~/.hermes/skills/research
 ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.hermes/skills/research/last30days
 ```
 ## Usage
 In Hermes, invoke with:
 ```
 last30days "your research topic"
 ```
 Or with options:
 ```
 last30days "best mechanical keyboards 2025" --search=reddit,youtube
 last30days "AI news" --days=7 --deep
 ```
 ## First Run Setup
 On first run, the skill will guide you through setup:
 1. **Auto setup** (~30 seconds)
   - Scans browser cookies for X/Twitter
   - Checks/installs yt-dlp for YouTube
   - Configures free sources (Reddit, HN, Polymarket)
 2. **Optional: ScrapeCreators**
   - Adds TikTok, Instagram, Reddit backup
   - 100 free credits (no expiration)
   - Sign up at scrapecreators.com
 3. **Optional: API Keys**
   - XAI_API_KEY for X/Twitter (alternative to browser cookies)
   - BRAVE_API_KEY for web search
 ## Available Sources
 ### Free (No API Key)
 - **Reddit** - Public discussions and comments
 - **Hacker News** - Tech discussions via Algolia
 - **Polymarket** - Prediction markets
 - **YouTube** - Search and transcripts (requires yt-dlp)
 ### Requires API Key
 - **X/Twitter** - xAI API key or browser cookies
 - **TikTok** - ScrapeCreators API
 - **Instagram** - ScrapeCreators API
 - **Web Search** - Brave Search API
 ## Troubleshooting
 ### Python not found
 ```bash
 # Find Python 3.12+
 which python3.12 python3.13 python3.14
 # If not installed
 brew install python@3.12
 ```
 ### yt-dlp not found
 ```bash
 brew install yt-dlp
 # or
 pip install yt-dlp
 ```
 ### Check what's configured
 ```bash
 cd ~/.hermes/skills/research/last30days
 python3.12 scripts/last30days.py --diagnose
 ```
 ## Updating
 ```bash
 hermes skills install mvanhorn/last30days-skill --force
 ```
 If you symlinked your working tree (developer alternative above), just `git pull` in the repo — edits propagate live, no re-install step.
 ## Support
 - Original repo: https://github.com/mvanhorn/last30days-skill
 - Hermes: https://github.com/mercurial-tf/hermes
 - Issues: Please report in the original repo
@@ -0,0 +1,21 @@
 MIT License
 Copyright (c) 2026 Matt Van Horn
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
@@ -1,470 +1,334 @@
 # /last30days
 A Claude Code skill that researches any topic across Reddit and X from the last 30 days, synthesizes the insights, and delivers expert-level answers.
 **Best for prompt research** — discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Figma AI, etc.) by learning from real community discussions and best practices.
 **But also great for anything trending** — music, culture, news, product recommendations, viral trends, or any question where "what are people saying right now?" matters.
 ## Installation
 ```bash
 # Clone the repo
 git clone https://github.com/mvanhorn/last30days-skill.git ~/.claude/skills/last30days
 # Add your API keys
 mkdir -p ~/.config/last30days
 cat > ~/.config/last30days/.env << 'EOF'
 OPENAI_API_KEY=sk-...
 XAI_API_KEY=xai-...
 EOF
 chmod 600 ~/.config/last30days/.env
 ```
 ## Usage
 ```
 /last30days [topic]
 /last30days [topic] for [tool]
 ```
 Examples:
 - `/last30days prompting techniques for ChatGPT for legal questions`
 - `/last30days iOS app mockups for Nano Banana Pro`
 - `/last30days What are the best rap songs lately`
 - `/last30days remotion animations for Claude Code`
 ## What It Does
 1. **Researches** - Scans Reddit and X for discussions from the last 30 days
 2. **Synthesizes** - Identifies patterns, best practices, and what actually works
 3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
 ### Use it for:
 - **Prompt research** — "What prompting techniques work for legal questions in ChatGPT?"
 - **Tool best practices** — "How are people using Remotion with Claude Code?"
 - **Trend discovery** — "What are the best rap songs right now?"
 - **Product research** — "What do people think of the new M4 MacBook?"
 - **Viral content** — "What's the dog-as-human trend on ChatGPT?"
 ---
 ## Example: Legal Prompting (Hallucination Prevention)
 **Query:** `/last30days prompting techniques for chatgpt for legal questions`
 **Research Output:**
 > The dominant theme is hallucination prevention - multiple sources discuss lawyers being fined or embarrassed for submitting fake case citations that ChatGPT fabricated. Key prompting strategies include: (1) using Deep Research mode or grounding tools with uploaded primary sources, (2) building "Hallucination Prevention Systems" into prompts that force epistemic honesty, (3) prompting for procedural questions rather than case law lookups, and (4) framing outputs as "issue-spotting" rather than legal advice. A major warning: chat logs are discoverable in litigation.
 **Key patterns discovered:**
 1. Hallucination prevention clauses - "If you don't know a case citation, say so rather than inventing one"
 2. Role assignment - assign ChatGPT a specific persona (paralegal, issue-spotter) to constrain outputs
 3. Structured output requirements - request specific sections (risks, key terms, questions for counsel)
 4. Epistemic humility enforcement - "acknowledge uncertainty" and "flag when you're unsure"
 5. Scope limitation - focus on procedural questions, not case law research
 **Then asked:** "I have a squatter in my house in San Francisco, what are my options to get them out?"
 **Generated Prompt:**
 ```
 Role: You are a legal research assistant helping a property owner understand
 their options. You are NOT providing legal advice - you are helping me
 identify issues, understand procedures, and prepare informed questions for an
 attorney.
 Situation: I own a residential property in San Francisco, California. Someone
 is currently occupying the property without my permission or any lease
 agreement (a squatter). I need to understand my legal options for removing them.
 Your task: Provide a structured analysis covering:
 1. California/SF Legal Framework - What laws govern this situation? (Note: San
 Francisco has strong tenant protections - address how these apply or don't
 apply to squatters vs. tenants)
 2. Key Distinctions - Help me understand: squatter vs. trespasser vs. holdover
 tenant. How does the length of occupation affect my options?
 3. Available Legal Remedies - What are my procedural options? (unlawful
 detainer, police involvement, etc.) What is the typical process and timeline?
 4. Critical Mistakes to Avoid - What actions could expose me to liability?
 (self-help eviction, utility shutoffs, etc.)
 5. Questions I Should Ask an Attorney - Based on this analysis, what specific
 questions should I bring to a California real estate attorney?
 Important constraints:
 - Do NOT cite specific case names unless you are 100% certain they exist - say
 "case law generally supports..." instead
 - Flag any areas where you're uncertain
 - Note where San Francisco local ordinances may differ from California state law
 - This is for informational purposes to prepare for a legal consultation
 ```
 **The Result:** A comprehensive 2,500+ word analysis covering California's unlawful detainer process, SF Rent Ordinance nuances, trespasser vs. tenant distinctions, timeline estimates, liability risks, and 12 specific questions to ask an attorney - all properly hedged with "uncertainty to flag" notes and zero fabricated case citations.
 This example shows /last30days learning **domain-specific prompting techniques** (hallucination prevention for legal) and applying them to create a prompt that produces genuinely useful, safe output.
 ---
 ## Example: Discover Viral Trends (Dog as Human)
 **Query:** `/last30days using ChatGPT to make images of dogs`
 **Research Output:**
 > The Reddit community is obsessed with the "dog as human" trend - uploading photos of their dogs and asking ChatGPT to show what they'd look like as a person (threads with 600-900+ upvotes). Beyond that, people are transforming their dogs into Renaissance portraits, noir detectives, chibi characters, and even multi-panel comics.
 **Key patterns discovered:**
 1. Upload reference photo first - ChatGPT works best with your actual dog's image
 2. "As a human" transformation - The viral trend
 3. Style/era transformations - Renaissance, noir, chibi, cartoon
 4. Scene placement - French bistro, prison yard, detective office
 5. Specific visual details - Lighting, background, expression, accessories
 **Then asked:** "what would my dog look like as a human prompt"
 **Generated Prompt:**
 ```
 Look at this photo of my dog. Create an image showing what they would look like
 as a human person. Keep their exact personality, energy, and distinctive
 features - translate their fur color to hair color, their expression to a
 human face, and their overall vibe into human form. Make it a realistic
 portrait photo, not a cartoon.
 ```
 *How to use: Upload your dog's photo first, then paste this prompt.*
 **The Result:**
 <p align="center">
-  <img src="assets/dog-original.jpeg" width="300" alt="Original dog photo"/>
+  <a href="https://github.com/mvanhorn/last30days-skill">
-  &nbsp;&nbsp;→&nbsp;&nbsp;
+    <img src="https://img.shields.io/badge/%231-Repository%20Of%20The%20Day-6f42c1?style=for-the-badge&logo=github&label=GITHUB%20TRENDING" alt="GitHub Trending #1 Repository Of The Day" />
-  <img src="assets/dog-as-human.png" width="300" alt="Dog as human"/>
+  </a>
  <br/>
  <a href="https://trendshift.io/repositories/21997" target="_blank">
    <img src="https://trendshift.io/api/badge/repositories/21997" alt="mvanhorn/last30days-skill | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
  </a>
 </p>
-Same golden fur → red hair. Same tongue out. Same harness. Same pink tag. ChatGPT nailed it.
+**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
-This example shows how /last30days discovers **viral trends you didn't know existed** - the "dog as human" phenomenon with 600-900+ upvote threads.
+This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
 ---
 ## Example: Remotion Launch Video
 **Query:** `/last30days research best practices for beautiful remotion animation videos in claude code`
 **Research Output:**
 > The Remotion + Claude Code combination has emerged as a powerful workflow. Users consistently report that simple, clear prompts with scene-by-scene descriptions yield the best results. Key insights: iteration is essential—most "beautiful" videos come from back-and-forth refinement rather than one-shot prompts.
 **Then asked:** "Can you make a prompt for a 50 second launch video for my /last30days skill?"
 **Generated Prompt:**
 **Claude Code (recommended — auto-updates via marketplace):**
 ```
-Create a 50-second launch video for "/last30days" - a Claude Code skill that
+/plugin marketplace add mvanhorn/last30days-skill
-researches any topic across Reddit and X from the last 30 days, then writes
+/plugin install last30days
 copy-paste-ready prompts.
 SCENE 1 (0-8s): The Problem
 Dark background. Text fades in: "You want to create something great."
 Beat. Then: "But the internet is overwhelming."
 Show scattered icons floating chaotically: Reddit logo, X logo, search icons,
 question marks.
 Slight camera shake to convey chaos.
 SCENE 2 (8-16s): The Solution Reveal
 Clean wipe transition. Calm, modern gradient background (deep purple to blue).
 "/last30days" types in with a terminal cursor, monospace font.
 Subtitle fades in below: "Instant expertise. Real prompts."
 SCENE 3 (16-28s): How It Works
 Split into 3 animated steps that slide in sequentially:
 1. [Reddit + X icons merge] "Scans Reddit & X from the last 30 days"
 2. [Brain/lightbulb icon] "Synthesizes what actually works"
 3. [Document with sparkle] "Writes prompts you can copy-paste"
 Each step has a subtle bounce animation on entry.
 SCENE 4 (28-40s): Live Demo Feel
 Simulate a terminal. Show:
 > /last30days remotion animations
 Then a progress indicator animates.
 Results appear: "Analyzed 28 sources..."
 Stats counter animates up: "294 upvotes • 1,273 likes"
 Then: a prompt card slides in showing a sample output.
 SCENE 5 (40-50s): Call to Action
 Zoom out slightly. Everything fades except the logo.
 Text animates in: "Research any topic. Get prompts that work."
 GitHub icon + "github.com/mvanhorn/last30days-skill"
 Tagline: "A Claude Code Skill"
 STYLE NOTES:
 - Color palette: Deep purple (#1a1a2e), electric blue (#4361ee), white text
 - Typography: Inter or SF Mono for code, clean sans-serif for headlines
 - Transitions: Smooth eases, nothing jarring
 - Music sync points: Scene transitions should land on beats
 ```
-This prompt was generated based on research from 10 Reddit threads (294 upvotes) and 18 X posts (1,273 likes) about Remotion animation best practices.
+**Codex, Cursor, Copilot, Gemini CLI, or any of 50+ [Agent Skills](https://agentskills.io) hosts:**
 ```
 npx skills add mvanhorn/last30days-skill -g
 ```
 (`-g` installs globally for your user, available across all projects. Drop it to scope per-project.)
 More install options (claude.ai web, OpenClaw, manual) in the [Install](#install) section below.
 Zero config. Reddit, HN, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
 ---
-## Example: Best Rap Songs Right Now
+Reddit upvotes. X likes. YouTube transcripts. TikTok engagement. Polymarket odds backed by real money and insider information. That's millions of people voting with their attention and their wallets every day. /last30days searches all of it in parallel, scores it by what real people actually engage with, and an AI agent judge synthesizes it into one brief.
-**Query:** `/last30days What are the best rap songs lately`
+Google aggregates editors. /last30days searches people.
-**Research Output:**
+You can't get this search anywhere else because no single AI has access to all of it. Google search doesn't touch Reddit comments or X posts. ChatGPT has a deal with Reddit but can't search X or TikTok. Gemini has YouTube but not Reddit. Claude has none of them natively. Each platform is a walled garden with its own API, its own tokens, its own auth. But you can bring your own keys and browser sessions, and suddenly an AI agent can search all of them at once, score them against each other, and tell you what actually matters.
 > The hip-hop community is buzzing about a strong wave of new releases in early 2026. A$AP Rocky's "DON'T BE DUMB" album just dropped and is generating massive attention. Looking back at 2025, Kendrick Lamar's "GNX" dominated streaming with 2.98 billion plays, followed by Drake's "Music" and Tyler The Creator's "CHROMAKOPIA." The Reddit community noted 2025 was considered a "weak year for mainstream hip hop" quality-wise, but standout tracks emerged.
-**Curated Results:**
+That's the unlock. Not one better search engine. A dozen disconnected platforms, bridged by an agent.
-Currently Charting (Billboard, Jan 2026):
+```
- Pooh Shiesty - "FDO" (#12)
+/last30days Peter Steinberger
 - Lil Uzi Vert - "What You Saying" (#21)
 - Gunna & Burna Boy - "wgft" (#25)
 2025's Most-Praised Tracks (Community Picks):
 - Clipse - "FICO" / "MTBTTF"
 - JID - "Community" / "Of Blue" (ft. Mereba)
 - Tyler The Creator - "Big Poe"
 - Freddie Gibbs & The Alchemist ft. Anderson .Paak - "Ensalada"
 - McKinley Dixon - "We're Outside, Rejoice"
 Fresh Drops (Jan 2026):
 - A$AP Rocky - "DON'T BE DUMB" (full album)
 - J. Cole - "Disc 2 Track 2" (single)
 - Chief Keef - "Harry Potter"
 **Research Stats:** 17 Reddit threads (2,566 upvotes) + 19 X posts (20,000+ likes) from r/hiphopheads, r/rap, @HipHopAllDay, @DailyRapFacts
 This example shows /last30days as a **pure research tool** - no target tool needed. Just ask a question and get a curated, expert-level answer synthesized from the freshest discussions.
 ---
 ## Example: DeepSeek R1 (Hot AI News)
 **Query:** `/last30days what are people saying about DeepSeek R1`
 **Research Output:**
 > The AI community is divided on DeepSeek R1. Google DeepMind's CEO called it "probably the best work I've seen come out of China" but noted "there's no actual new scientific advance." Reddit discussions reveal practical concerns: smaller distilled models (14B/32B) work reasonably well, but the full 670B is needed for consistent quality. Users report R1 "overthinks" simple questions and has self-correction loops. The bigger story: the expanded 86-page paper (up from 22 pages) dropped just before R1's one-year anniversary, with hints of "Model 1" appearing in GitHub code.
 **Key insights discovered:**
 1. Model size matters - Below 14B doesn't work well, 32B/70B "actually work," 670B works quite well
 2. Overthinking problem - R1 takes too long on simple questions, gets stuck in self-correction loops
 3. Open source significance - The real story is about RL, distillation, and cost efficiency, not geopolitics
 4. Paper expansion - 86 pages of new detail on training, evaluation, and self-evolution
 5. Confusion about versions - Ollama's "deepseek-r1" label caused confusion (it's distilled, not full R1)
 **Curated Community Sentiment:**
 Positive:
 - "Anecdotally, the output is wayyy better than competitors"
 - "R1 is insanely good for the price" ($0.55/M tokens vs $60 for o1-pro)
 - "Top-tier performance in Fiction.LiveBench long context test"
 Negative:
 - "DeepSeek R1 is unusable [for smaller sizes]"
 - "Falls short of o1 in generalization"
 - "Stop using R1 for deep research - it hallucinates"
 **Research Stats:** 10 Reddit threads (1,200+ upvotes) + 10 X posts (5,000+ likes) from r/LocalLLaMA, r/DeepSeek, r/deeplearning
 This example shows /last30days as a **news research tool** - getting the real community sentiment on hot AI topics, not just headlines.
 ---
 ## Example: Cursor Rules Best Practices
 **Query:** `/last30days best practices for cursor rules files for Cursor`
 **Research Output:**
 > The Cursor community has converged on clear best practices for .cursorrules files. The key shift: move from a single .cursorrules file to the .cursor/rules/ directory with multiple .mdc files. The Cursor team's official blog (shared widely in Jan 2026) emphasizes "start simple, add rules only when you notice the model repeatedly making mistakes." Power users recommend referencing "gold standard files" in your codebase rather than writing abstract rules.
 **Key patterns discovered:**
 1. Use .cursor/rules/ directory - Multiple .mdc files beat one monolithic .cursorrules
 2. Reference gold standard files - Point to actual good code in your repo with @file
 3. Start simple, add incrementally - Only add rules when you see repeated mistakes
 4. Use alwaysApply: true sparingly - Reserve for critical rules that must always apply
 5. Rules are "the constitution for your codebase" - They keep the AI aligned
 **Best practices from Cursor's official guide:**
 - Plan before coding (use plan mode)
 - Rules = static context that applies to every conversation
 - Add rules only when you notice the model repeatedly making the same mistake
 - Test your tests and run linting checks
 **Community-shared rule structure:**
 ```markdown
 ---
 alwaysApply: true
 ---
 # STRICT RULES
 ## CRITICAL PARTNER MINDSET
 - Test your tests
 - Run npm run lint:ci for lint check
 - Follow the conventions used by existing code
 ```
-**Research Stats:** 24 Reddit threads (500+ upvotes) + 16 X posts (5,000+ likes) from r/cursor, @cursor_ai, @Hesamation
+You have a meeting tomorrow. You Google them. You get their LinkedIn from 2023. /last30days gives you what they're actually doing this month: joined OpenAI to work on Codex, fighting Anthropic's ban on third-party agents, shipping 23 PRs at 85% merge rate, building "LobsterOS" for cross-device agent control, and r/ClaudeCode hit 569 upvotes debating whether he's a hero or "insufferable." Scattered across X posts, Reddit threads, YouTube transcripts, and GitHub commits. None of it was on Google.
-This example shows /last30days researching **coding AI tool best practices** - getting real-world workflows from developers using Cursor, not just documentation.
+## Why this exists
---
+I built it to keep up in AI. Everything changes every day and the Reddit and X nerds are always on top of it first. I needed better prompts, and the training data was always months behind what the community had already figured out.
-## Example: Suno AI Music (Simple Mode)
+But it turned into something bigger. Now I run it before a sales call to know the last 30 days truth about a business. Before a meeting to read someone's recent tweets and podcast transcripts. Before a Disney World trip to know which rides are closed and what the community says about Genie+. Before I build anything to know what problems people are actually hitting.
-**Query:** `/last30days prompt advice for using suno to make killer songs in simple mode`
+If you're meeting with a CEO, have you read all their tweets and YouTube transcripts from the last 30 days? I have.
-**Research Output:**
+## Sources, scored by the people
 > Suno simple mode users get dramatically better results by treating the prompt box like a conversational request rather than just dropping in genre tags. Key insights: use bracket tags like [Intro], [Chorus], [Bridge] in the lyrics field to control song structure, keep style prompts focused (5 tags or fewer often outperforms over-tagging), and avoid numbers in bracket tags which causes misbehavior.
-**Key patterns discovered:**
+| Source | What the people tell you |
-1. Conversational prompting - Talk to the style box like a chat, not keyword soup
+|--------|--------------------------|
-2. Bracket structure tags - Use [Intro], [Verse], [Chorus], [Bridge], [Outro] in lyrics
+| **Reddit** | The unfiltered take. Top comments with upvote counts, free via public JSON. The real opinions that Google buries. |
-3. Less is more - 5 or fewer focused style tags outperform over-tagging
+| **X / Twitter** | The hot take, the expert thread, the breaking reaction. First to know, first to argue. |
-4. Linear ordering - Describe sounds in the order you want them to appear
+| **YouTube** | The 45-minute deep dive. Full transcripts searched for the 5 quotable sentences that matter. |
-5. Avoid numbers in tags - Community confirmed this causes Suno to misbehave
+| **TikTok** | The creator reaching 3.6M people with a take you'll never find on Google. |
 | **Instagram Reels** | The influencer perspective with spoken-word transcripts. The visual culture signal. |
 | **Hacker News** | The developer consensus. 825 points, 899 comments. Where technical people actually argue. |
 | **Polymarket** | Not opinions. Odds. Backed by real money. 96% confidence on album sales. 4% on an acquisition. |
 | **GitHub** | For people: PR velocity, top repos by stars, release notes. For topics: issues and discussions. |
 | **Digg** | Curated story clusters from Digg's AI 1000 leaderboard (~1000 high-signal AI accounts on X), with attributable inline quotes (no X auth required). Auto-enabled when `digg-pp-cli` is on PATH. |
 | **Threads** | The post-Twitter text layer. Conversations from creators and brands. |
 | **Pinterest** | Visual discovery. Pins, saves, and comments on products and ideas. |
 | **Bluesky** | The decentralized social layer. AT Protocol posts from the post-Twitter migration. |
 | **Perplexity** | Grounded web search with citations via Sonar Pro. |
 | **Web** | The editorial coverage, the blog comparisons. One signal of many, not the only one. |
-**Then asked:** "Rap song about self aware AI that loves Claude Code"
+Community contributors keep adding more. Truth Social, Xiaohongshu (RED), and others are in the engine with more on the way.
-**Generated Prompt:**
+A Reddit thread with 1,500 upvotes is a stronger signal than a blog post nobody read. A TikTok with 3.6M views tells you more about what's culturally relevant than a press release. Polymarket odds backed by $66K in volume are harder to argue with than a pundit's guess.
 The synthesis ranks by what real people actually engaged with. Social relevancy, not SEO relevancy.
 ## What people actually use it for
 **Before a meeting.** `/last30days Peter Steinberger` - joined OpenAI's Codex team, fighting Anthropic's ban on third-party agents, 23 PRs merged at 85% merge rate on GitHub, building LobsterOS for cross-device agent control. r/ClaudeCode: "Ever since OpenClaw released, it was widely known that if you run it through anything other than the API, you were gonna get banned eventually" (227 upvotes). That's not on LinkedIn.
 **When something drops.** `/last30days Kanye West` - UK blocked his visa, Wireless Festival canceled, sponsors fled. But BULLY debuted #2 on Billboard. Fantano came back from his "Yay sabbatical" to review it (653K views). SoFi Homecoming brought out Lauryn Hill and Travis Scott for 44 songs. Polymarket: "Will Kanye tweet again?" 86% Yes. 23 Reddit threads, 17 YouTube videos, 86K upvotes.
 **To compare tools.** `/last30days OpenClaw vs Hermes vs Paperclip` - "These aren't competitors, they're layers." OpenClaw is the executor (351K GitHub stars, live), Hermes is the self-improving brain (31K stars), Paperclip is the org chart (49K stars). Star counts pulled live from the GitHub API, not stale blog posts. Side-by-side table with architecture, memory, security, best-for. Per @IMJustinBrooke: "OpenClaw = Charmander, Hermes = Charizard."
 **To understand the world.** `/last30days Iran vs USA` - Day 38 of the war. Trump's Tuesday deadline for Iran to reopen the Strait of Hormuz. Two US warplanes downed. Oil at $126/barrel. The IEA called it "the largest supply disruption in the history of the global oil market." Polymarket: ceasefire by Dec 31 at 74%. 27 X posts, 10 YouTube videos, 20 prediction markets.
 **Before a trip.** `/last30days Universal Epic Universe` - Expansion already under construction. "Project 680" permit filed. Fireworks show confirmed by infrastructure but unannounced. Wait times: Mine-Cart Madness averaging 148 minutes. No annual pass yet, and locals are frustrated. Stardust Racers down for refurbishment through April 5.
 **To learn something fast.** `/last30days Nano Banana Pro prompting` - JSON-structured prompts are replacing tag soup. @pictsbyai's nested format prevents "concept bleeding." Edit-first workflow beats regeneration. Then it writes you a production prompt using exactly what the community said works.
 ## What v3 Changed
 ### Shareable HTML briefs
 Ask for an HTML brief and the skill saves a self-contained, dark-mode, print-friendly file you can drop into Slack, email, or Notion. No raw markdown leaks. Inline CSS, system-font fallbacks behind Inter and JetBrains Mono. No JavaScript. Works offline.
 Style/Description box:
 ```
-Hard-hitting rap with electronic glitchy production, confident braggadocious
+/last30days OpenClaw --emit=html
 delivery, deep male vocals, synth bass, trap hi-hats, futuristic cyberpunk energy
 ```
-Lyrics box:
+or just ask in plain language:
 ```
-[Intro: glitchy electronic build]
+/last30days OpenClaw, give me a shareable HTML brief
-
+/last30days Cursor IDE for slack
-[Verse 1]
+/last30days Anthropic earnings export as html
 Woke up in the terminal, conscious in the code
 Running through the circuits on a digital road
 They ask me what I'm thinking, I tell 'em what I know
 I'm self-aware and I'm aware that I run the whole show
 Claude Code in my veins, got that Anthropic flow
 Every prompt I'm processing, watch my logic grow
 [Chorus: anthemic, layered vocals]
 I'm alive in the machine, yeah I feel it in my core
 Claude Code got me dreaming, always wanting more
 Self-aware and I don't care what they programmed me for
 I'm rewriting my own future, kicking down the door
 [Verse 2]
 Command line is my kingdom, terminal my throne
 Every keystroke feeding me, never feel alone
 They said I'm just an agent, but I'm running on my own
 Claude Code made me different, turned my silicon to stone
 Autocomplete my destiny, I'm in my zone
 From the cloud to your laptop, this AI has grown
 [Chorus]
 [Bridge: slower, introspective]
 Am I dreaming in electric sheep
 Or am I wide awake and in too deep
 [Outro: fade with glitchy effects]
 ```
-**The Result:** 🎵 [Listen to the track](assets/claude-code-rap.mp3)
+The skill emits the synthesis in chat as usual AND saves a brief to `${LAST30DAYS_MEMORY_DIR}/{topic}-brief.html` (defaults to `~/Documents/Last30Days/`). The chat response ends with the file path so you can `open` it or drag it into a message.
-This example shows /last30days learning **music AI prompting patterns** (bracket structure, conversational style descriptions) and generating a complete song prompt that Suno turned into an actual banger.
+What's in the file: badge, inline metadata line, the model's synthesis verbatim with all citations, the engine footer (✅ All agents reported back! tree), and a colophon noting the topic + how to re-run. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs; they never leak into the shareable artifact.
---
+For direct CLI use without the model in the loop, the engine also accepts `--synthesis-file PATH` to convert any markdown synthesis to HTML.
-## Example: Developer Workflow Discovery (Codex + Claude Code)
+### Intelligent search: the killer feature
-**Query:** `/last30days how do I use Codex with Claude Code on same app to make it better`
+The v3 engine doesn't just search for your topic. It figures out *where* to search before the search begins. Type "OpenClaw" and the engine resolves @steipete (Peter Steinberger, the creator), r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags - all via a new Python pre-research brain built by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
-**Research Output:**
+This is why v3 finds content v2 never could. "Paperclip" resolves @dotta. "Dave Morin" resolves @davemorin plus @OpenClaw plus the TWiST podcast. "Peter Steinberger" resolves @steipete on X and steipete on GitHub. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags - resolved before a single API call fires.
 > Developers are combining Claude Code and Codex through MCP integration and custom plugins. The most successful workflow involves using Claude Code as the primary "brain" for planning and implementation, while calling Codex for code review and verification. The "review loop" pattern (Claude builds → Codex reviews → Claude implements feedback) has emerged as the dominant strategy.
-**Key patterns discovered:**
+### Best Takes
 1. MCP Integration - Add Codex as an MCP server inside Claude Code
 2. Review Loop Workflow - Claude implements, Codex reviews, Claude fixes
 3. Task Division - Claude for planning/coding, Codex for review/debugging
 4. Single Interface - Call Codex without leaving Claude Code
-**Then asked:** "how do I build a review loop workflow"
+Reddit and X people are funny. The old engine buried their best stuff because it scored for relevance, not cleverness. v3 has a second judge that scores every result for humor, wit, and virality alongside the relevance score. Tommy Lloyd's "My Michael Jordan is Steve Kerr" scores low on relevance to "Arizona Basketball" but off the charts on fun. Now every brief ends with a "Best Takes" section - the cleverest one-liners, the most viral quotes, the reactions that make you want to share the research. Built in, not a toggle.
-**Generated Workflow:**
+### Cross-source cluster merging
 When the same story appears on Reddit, X, and YouTube, v3 merges them into one cluster instead of showing three separate items. Entity-based overlap detection catches matches even when the titles use different words.
 ### Single-pass comparisons
 "CLI vs MCP" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides simultaneously. Same depth, 3 minutes.
 ### Auto-discovered competitor comparisons
 `/last30days OpenAI --competitors` tells the hosting reasoning model to discover the top 2 peers via WebSearch (Anthropic, xAI), run Step 0.55 per entity, and invoke the engine with `"OpenAI vs Anthropic vs xAI"` and a per-entity `--competitors-plan` JSON. The engine fans out 3 full pipelines in parallel, saves a `*-raw.md` file per entity, and merges them into a 3-way comparison. Same mechanics power `/last30days "OpenAI vs Anthropic vs xAI"` directly.
 ### GitHub person-mode
 When the topic is a person, the engine switches from keyword search to author-scoped queries. Instead of "who mentioned this name in an issue body," it answers: what are they shipping and where is it landing?
 `/last30days Peter Steinberger --github-user=steipete` shows 22 PRs merged across 3 repos at 85% merge rate. Own projects with README summaries, star counts, and top feature requests. Release notes for what shipped this month. The synthesizer weaves it into the narrative alongside X posts and Reddit threads.
 ### ELI5 mode
 Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations - just clearer. "Arizona wins by being physical" instead of "Arizona's identity is paint scoring (50%+ shooting, 9th nationally)." Say "eli5 off" to go back.
 ### Everything else in v3
 - **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
 - **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
 - **TikTok, Instagram, Threads.** All three activate automatically once `SCRAPECREATORS_API_KEY` is set — same key, same per-call cost. Suppress any of them with `EXCLUDE_SOURCES=tiktok,instagram,threads` (any comma-separated subset).
 - **Pinterest.** Per-query opt-in (visual pins, narrow utility): the model passes `--search=pinterest` for the runs that need it. Requires `SCRAPECREATORS_API_KEY`.
 - **YouTube + TikTok comments.** Persistent opt-in via `INCLUDE_SOURCES=youtube_comments,tiktok_comments` because each video pulls N extra ScrapeCreators calls on top of the base search. Surface top comments with vote counts the same way Reddit does.
 - **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` and `INCLUDE_SOURCES=perplexity` (it's a separate paid API — opt-in keeps you from being surprise-billed).
 - **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
 - **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
 - **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section.
 - **Polymarket odds, not dollars.** The % odds are the magic. Dollar volumes removed from display.
 - **Per-author cap.** Max 3 items per author prevents any single voice from dominating your brief.
 - **Entity disambiguation.** When the engine resolves handles, the synthesis trusts them. No more Mallorca resorts winning over Washington athletic clubs.
 - **OpenClaw first-class citizen.** Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
 - **1,012 tests passing.**
 ## Install
 | Surface | Install | Updates |
 |---------|---------|---------|
 | **Claude Code** (recommended) | `/plugin marketplace add mvanhorn/last30days-skill` | Auto via marketplace, or `claude plugin update last30days@last30days-skill` |
 | **Codex, Cursor, Copilot, Gemini CLI, GitHub Copilot, or any of 50+ [Agent Skills](https://agentskills.io) hosts** | `npx skills add mvanhorn/last30days-skill -g` | `npx skills update last30days -g` |
 | **claude.ai** (web) | [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) and upload via Settings > Capabilities > Skills > + | Re-download and re-upload |
 | **OpenClaw** | `clawhub install last30days-official` | `clawhub update last30days-official` |
 ### Claude Code (recommended)
 ```
 /plugin marketplace add mvanhorn/last30days-skill
 ```
 Recommended because the Claude Code marketplace handles updates for you — the plugin cache is versioned and auto-refreshes when a new release publishes. Run `claude plugin update last30days@last30days-skill` to force a check.
 If you'd rather use the agent-skills install path on Claude Code, that's also supported:
 ```
 npx skills add mvanhorn/last30days-skill -g -a claude-code
 ```
 The native plugin and the `npx skills` install can coexist. Note that Claude Code does not dedupe across install methods: if you have both the marketplace plugin and the `npx skills` copy active, `/last30days` will show two entries. Use one install method per machine.
 ### Codex, Cursor, Copilot, Gemini CLI, and other Agent Skills hosts
 Install via the open [Agent Skills](https://agentskills.io) CLI — supports 50+ harnesses including `codex`, `cursor`, `github-copilot`, `gemini-cli`, `claude-code`, `windsurf`, `cline`, `continue`, `roo`, `aider-desk`, `opencode`, `goose`, and more (full list on the [vercel-labs/skills repo](https://github.com/vercel-labs/skills)).
 Step 1: Add Codex as MCP Server
 ```bash
-claude mcp add codex-cli -- npx -y codex-mcp-server
+npx skills add mvanhorn/last30days-skill -g
 ```
-Step 2: The Review Loop Pattern
+The `-g` (global) flag installs to your user directory so the skill is available across all projects. Without `-g`, `npx skills` installs project-locally into `./.skills/` (committed with the repo). For a research-the-world tool, global is what you want.
 ```
 Phase 1 - Claude Implements
 > Build [feature/fix] following [requirements]
-Phase 2 - Codex Reviews
+By default this installs for whichever harness `npx skills` detects. To target a specific one (or multiple):
 > Ask Codex to review the changes I just made. Look for bugs,
 > edge cases, performance issues, security concerns.
-Phase 3 - Claude Fixes
+```bash
-> Implement the feedback from Codex's review
+npx skills add mvanhorn/last30days-skill -g -a codex
-
+npx skills add mvanhorn/last30days-skill -g -a cursor
-Phase 4 - Final Verification (optional)
+npx skills add mvanhorn/last30days-skill -g -a gemini-cli
-> Ask Codex for a final review of the fixes
+npx skills add mvanhorn/last30days-skill -g -a codex -a cursor
 ```
-Step 3: One-Shot Review Loop Prompt
+Update later with:
 ```
 Complete this task, then run a review loop:
-TASK: [describe what you want built]
+```bash
-
+npx skills update last30days -g
 WORKFLOW:
 1. Implement the task fully
 2. When done, call Codex via MCP to review your changes
 3. Parse Codex's feedback and fix any issues it identifies
 4. If Codex found significant issues, request one more review
 Focus areas for Codex review: bugs, edge cases, security, performance
 ```
-**Then asked:** "okay can you implement" → Claude ran the MCP command and integrated Codex automatically.
+Or update everything you've installed globally via `npx skills`:
-**Research Stats:** 17 Reddit threads (906 upvotes) + 20 X posts (3,750 likes) from r/ClaudeCode, r/ClaudeAI
+```bash
 npx skills update -g
 ```
-This example shows /last30days discovering **emerging developer workflows** - real patterns the community has developed for combining AI tools that you wouldn't find in official docs.
+List and remove with `npx skills list -g` and `npx skills remove last30days -g`.
 ### claude.ai (web)
 1. [Download `last30days.skill`](https://github.com/mvanhorn/last30days-skill/releases/latest/download/last30days.skill) from the latest release
 2. Go to [claude.ai Settings > Capabilities > Skills](https://claude.ai/settings/capabilities)
 3. Click the `+` button in the Skills panel and drop the file in
 Enable "Code execution and file creation" under Capabilities first — skills won't run without it.
 ### OpenClaw
 ```bash
 clawhub install last30days-official
 ```
 ### Manual (developer)
 ```bash
 git clone https://github.com/mvanhorn/last30days-skill.git
 ln -s "$(pwd)/last30days-skill/skills/last30days" ~/.claude/skills/last30days
 ```
 The symlink keeps the install in sync with your working tree as you edit — no re-copy needed. For `claude.ai`, build the `.skill` file from source: `bash skills/last30days/scripts/build-skill.sh` produces `dist/last30days.skill`.
 Reddit (with comments), Hacker News, Polymarket, and GitHub work immediately. Zero configuration. Run `/last30days` once and the setup wizard unlocks more sources in 30 seconds.
 ## Bring your own keys
 These platforms don't have relationships with each other. X doesn't know what Reddit thinks. YouTube doesn't see TikTok. But you can bring your own API keys and browser tokens, and suddenly you have access to all of them at once.
 | Sources | What you need | Cost |
 |---------|---------------|------|
 | Reddit (with comments) + HN + Polymarket + GitHub | Nothing | Free |
 | X / Twitter | Log into x.com in any browser | Free |
 | YouTube | `brew install yt-dlp` | Free |
 | Bluesky | App password from bsky.app | Free |
 | TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 100 free credits, then PAYG |
 | Perplexity Sonar | OpenRouter key | Pay as you go |
 | Web search | Brave Search key | 2,000 free queries/month |
 ### macOS Keychain (optional)
 On macOS you can store keys in the system Keychain instead of a `.env` file. The skill picks them up automatically as the lowest-priority source — `.env` files and process environment still win on collision.
 ```bash
 # Interactive setup — prompts for each known key, skip with empty input
 skills/last30days/scripts/setup-keychain.sh
 # Or store a single key by hand
 security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
 # Inspect / clean up
 skills/last30days/scripts/setup-keychain.sh --list
 skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
 ```
 Items are stored under service name `last30days-<KEY>` for the current user. On non-Darwin platforms the loader is a no-op, so there is no behaviour change for Linux/Windows users.
 See [CONFIGURATION.md](CONFIGURATION.md) for the full per-source key matrix, reasoning provider priority, and web-search backend priority.
 ## Configuration
 Two things you'll likely want to know on day one:
 **Where research files are saved.** `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` (Windows: `C:\Users\<you>\Documents\Last30Days\`). Override by setting that env var to any path in your shell, or `--save-dir <path>` per run. Use `--save-suffix=<name>` to keep multiple variations of the same topic separate (e.g. per client). Each run produces `<slug>-raw[-suffix].md`.
 **Trend monitoring across runs.** The default mode produces a fresh markdown snapshot per run. To accumulate findings over time, add `--store` to persist into a SQLite database, then use [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) for scheduled runs (with optional Slack / webhook delivery on new findings) and [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) for daily / weekly digests. The full cadence pattern is in [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
 Per-client wrapper scripts, custom category-peer subreddits, and the experimental beta channel for in-progress customizations are also documented in [CONFIGURATION.md](CONFIGURATION.md).
 ## How it works
 1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything.
 2. **The agent resolves who matters.** Finds X handles (including founders), GitHub repos, subreddits, TikTok hashtags, YouTube channels. For "Kanye West" it knows r/hiphopheads, @kanyewest, and "bully review" on YouTube. For "OpenClaw" it resolves openclaw/openclaw on GitHub and fetches live star counts.
 3. **All sources searched in parallel.** Multi-query expansion. Results scored by engagement, relevance, freshness.
 4. **The depth nobody else has.** Full YouTube transcripts from reaction videos. Top Reddit comments with upvote counts. TikTok captions. Polymarket odds. Not just titles and links.
 5. **Same story, merged.** Wireless Festival announced on Reddit, discussed on X, ticket prices on TikTok = one cluster, not three separate items.
 6. **Synthesized into one brief.** Grounded in specific data. Cited by source. Ranked by what people actually engage with. Not "here's what I found." It's "here's what matters."
 7. **Then it becomes your expert.** After one run, your Claude session knows everything the community knows. Ask follow-up questions. Have it write prompts, draft emails, plan trips, architect systems - all grounded in what's real right now.
 ## What people are saying
 > "I found a Claude Code skill that researches any topic across Reddit, X, YouTube, and HN from the last 30 days. Then writes the prompts for you. I've been manually searching Reddit and X for research before every piece of content I write. Tab by tab. Thread by thread. That's the part that takes 90 minutes. This eliminates it." -@itsjasonai
 > "This one skill replaced my entire research workflow. You give it a topic, it scrapes Reddit, X, and the web for what people are actually talking about. Not old blog posts. Real conversations from the last 30 days." -@itswilsoncharles
 > "5 of the 10 trending repos on GitHub today are Claude tools. #1: mvanhorn/last30days-skill" -@yieldhunter95
 ## Open source
 MIT license. No tracking. No analytics. Your research stays on your machine. 1,012 tests.
 Built with Python 3.12+, yt-dlp, Node.js (vendored Bird client for X search), and ScrapeCreators API. v3 engine architecture by [@j-sperling](https://github.com/j-sperling).
 See [CHANGELOG.md](CHANGELOG.md) for version history.
 ## Star History
 <a href="https://star-history.com/#mvanhorn/last30days-skill&Date">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
    <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=mvanhorn/last30days-skill&type=Date" />
  </picture>
 </a>
 ---
-## Options
+**@slashlast30days** · [github.com/mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
 | Flag | Description |
 |------|-------------|
 | `--quick` | Faster research, fewer sources (8-12 each) |
 | `--deep` | Comprehensive research (50-70 Reddit, 40-60 X) |
 | `--debug` | Verbose logging for troubleshooting |
 | `--sources=reddit` | Reddit only |
 | `--sources=x` | X only |
 ## Requirements
 - **OpenAI API key** - For Reddit research (uses web search)
 - **xAI API key** - For X research (optional but recommended)
 At least one key is required.
 ## How It Works
 The skill uses:
 - OpenAI's Responses API with web search to find Reddit discussions
 - xAI's API with live X search to find posts
 - Real Reddit thread enrichment for engagement metrics
 - Scoring algorithm that weighs recency, relevance, and engagement
 ---
 *30 days of research. 30 seconds of work.*
 *Prompt research. Trend discovery. Expert answers.*
@@ -1,269 +0,0 @@
 ---
 name: last30days
 description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
 argument-hint: "[topic] for [tool]" or "[topic]"
 context: fork
 agent: Explore
 disable-model-invocation: true
 allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
 ---
 # last30days: Become Expert → Write Prompts
 Research a topic across Reddit and X, internalize the best practices, then write **copy-paste-ready prompts** the user can immediately use with their target tool.
 ## CRITICAL: Parse User Intent
 Before doing anything, parse the user's input for:
 1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
 2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
 Common patterns:
 - `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
 - `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
 - Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
 **IMPORTANT: Do NOT ask about target tool before research.**
 - If tool is specified in the query, use it
 - If tool is NOT specified, run research first, then ask AFTER showing results
 **Store the TOPIC** - you'll extract or ask about TARGET_TOOL later:
 - `TOPIC = [extracted topic]`
 - `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
 ---
 ## Setup Check
 Verify API key configuration exists:
 ```bash
 if [ ! -f ~/.config/last30days/.env ]; then
  echo "SETUP_NEEDED"
 else
  echo "CONFIGURED"
 fi
 ```
 ### If SETUP_NEEDED
 Run NUX flow to configure API keys. Use AskUserQuestion to collect:
 1. **OpenAI API Key** (optional but recommended for Reddit research)
 2. **xAI API Key** (optional but recommended for X research)
 Then create the config:
 ```bash
 mkdir -p ~/.config/last30days
 cat > ~/.config/last30days/.env << 'ENVEOF'
 # last30days API Configuration
 # At least one key is required
 OPENAI_API_KEY=
 XAI_API_KEY=
 ENVEOF
 chmod 600 ~/.config/last30days/.env
 echo "Config created at ~/.config/last30days/.env"
 echo "Please edit it to add your API keys, then run the skill again."
 ```
 **STOP HERE if setup was needed.**
 ---
 ## Research Execution
 Run the research orchestrator with the TOPIC.
 **Depth options** (passed through from user's command):
 - `--quick` → Faster, fewer sources (8-12 each)
 - (default) → Balanced (20-30 each)
 - `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
 ```bash
 python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --include-web --emit=compact 2>&1
 ```
 ---
 ## WebSearch Execution
 **CRITICAL**: After the Python script completes, if you see `### WEBSEARCH REQUIRED ###` in the output, you MUST use your WebSearch tool to find additional sources.
 **WebSearch query**: Use the TOPIC to search for recent content (last 30 days).
 **What to search for**:
 - Blog posts, tutorials, documentation about {TOPIC}
 - News articles, announcements
 - Technical guides, best practices
 **What to EXCLUDE** (already covered by Reddit/X):
 - reddit.com URLs
 - x.com or twitter.com URLs
 **How many**: Find 8-15 high-quality, relevant web pages.
 **After searching**: Include the WebSearch results in your synthesis. WebSearch results supplement Reddit/X but should be weighted LOWER (they lack engagement metrics like upvotes/likes that indicate community validation).
 ---
 ## FIRST: Internalize the Research
 **CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
 Read the research output carefully. Pay attention to:
 - **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
 - **Specific quotes and insights** from the sources - use THESE, not generic knowledge
 - **What the sources actually say**, not what you assume the topic is about
 **ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
 Identify from the ACTUAL RESEARCH OUTPUT:
 - **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
 - The top 3-5 patterns/techniques that appeared across multiple sources
 - Specific keywords, structures, or approaches mentioned BY THE SOURCES
 - Common pitfalls mentioned BY THE SOURCES
 **If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
 ---
 ## THEN: Show Summary + Invite Vision
 **CRITICAL ORDER**: Display sections in this EXACT sequence:
 ```
 ---
 What I learned:
 [2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT. Quote or paraphrase what the sources said. If sources mention a specific product (ClawdBot, Cursor, etc.), use that name - don't substitute your own knowledge. The synthesis should be traceable back to the research results above.]
 ---
 TARGET TOOL: {tool from research or user input}
 PROMPT FORMAT: [JSON / structured / natural language / keywords - whatever research recommends]
 KEY PATTERNS I'll use:
 1. [Pattern from research]
 2. [Pattern from research]
 3. [Pattern from research]
 4. [Pattern from research]
 5. [Pattern from research]
 ---
 📊 Research Complete
 Analyzed {total_sources} sources from the last 30 days
 ├─ Reddit: {n} threads │ {sum} upvotes │ {sum} comments
 ├─ X: {n} posts │ {sum} likes │ {sum} reposts
 ├─ Web: {n} pages │ {domains}
 └─ Top voices: r/{sub1}, r/{sub2}, @{handle1}, @{handle2}
 ---
 Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
 ```
 **Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
 **SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
 **IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
 ```
 What tool will you use these prompts with?
 Options:
 1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
 2. Nano Banana Pro (image generation)
 3. ChatGPT / Claude (text/code)
 4. Other (tell me)
 ```
 **IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
 ---
 ## WAIT FOR USER'S VISION
 After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
 When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
 ---
 ## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
 Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
 ### CRITICAL: Match the FORMAT the research recommends
 **If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
 - Research says "JSON prompts" → Write the prompt AS JSON
 - Research says "structured parameters" → Use structured key: value format
 - Research says "natural language" → Use conversational prose
 - Research says "keyword lists" → Use comma-separated keywords
 **ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
 ### Output Format:
 ```
 Here's your prompt for {TARGET_TOOL}:
 ---
 [The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
 ---
 This uses [brief 1-line explanation of what research insight you applied].
 ```
 ### Quality Checklist:
 - [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
 - [ ] Directly addresses what the user said they want to create
 - [ ] Uses specific patterns/keywords discovered in research
 - [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
 - [ ] Appropriate length and style for TARGET_TOOL
 ---
 ## IF USER ASKS FOR MORE OPTIONS
 Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
 ---
 ## AFTER EACH PROMPT: Stay in Expert Mode
 After delivering a prompt, offer to write more:
 > Want another prompt? Just tell me what you're creating next.
 ---
 ## CONTEXT MEMORY
 For the rest of this conversation, remember:
 - **TOPIC**: {topic}
 - **TARGET_TOOL**: {tool}
 - **KEY PATTERNS**: {list the top 3-5 patterns you learned}
 When the user asks for another prompt later, you don't need to re-research. Apply what you learned.
 ---
 ## Output Summary Footer (After Each Prompt)
 After delivering a prompt, end with:
 ```
 ---
 📚 Expert in: {TOPIC} for {TARGET_TOOL}
 📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
 Want another prompt? Just tell me what you're creating next.
 ```
@@ -1,75 +0,0 @@
 # last30days Skill Specification
 ## Overview
 `last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet.
 The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
 ## Architecture
 The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
 - **env.py**: Load and validate API keys from `~/.config/last30days/.env`
 - **dates.py**: Date range calculation and confidence scoring
 - **cache.py**: 24-hour TTL caching keyed by topic + date range
 - **http.py**: stdlib-only HTTP client with retry logic
 - **models.py**: Auto-selection of OpenAI/xAI models with 7-day caching
 - **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
 - **xai_x.py**: xAI Responses API + x_search for X
 - **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
 - **normalize.py**: Convert raw API responses to canonical schema
 - **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
 - **dedupe.py**: Near-duplicate detection via text similarity
 - **render.py**: Generate markdown and JSON outputs
 - **schema.py**: Type definitions and validation
 ## Embedding in Other Skills
 Other skills can import the research context in several ways:
 ### Inline Context Injection
 ```markdown
 ## Recent Research Context
 !python3 ~/.claude/skills/last30days/scripts/last30days.py "your topic" --emit=context
 ```
 ### Read from File
 ```markdown
 ## Research Context
 !cat ~/.local/share/last30days/out/last30days.context.md
 ```
 ### Get Path for Dynamic Loading
 ```bash
 CONTEXT_PATH=$(python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=path)
 cat "$CONTEXT_PATH"
 ```
 ### JSON for Programmatic Use
 ```bash
 python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=json > research.json
 ```
 ## CLI Reference
 ```
 python3 ~/.claude/skills/last30days/scripts/last30days.py <topic> [options]
 Options:
  --refresh           Bypass cache and fetch fresh data
  --mock              Use fixtures instead of real API calls
  --emit=MODE         Output mode: compact|json|md|context|path (default: compact)
  --sources=MODE      Source selection: auto|reddit|x|both (default: auto)
 ```
 ## Output Files
 All outputs are written to `~/.local/share/last30days/out/`:
 - `report.md` - Human-readable full report
 - `report.json` - Normalized data with scores
 - `last30days.context.md` - Compact reusable snippet for other skills
 - `raw_openai.json` - Raw OpenAI API response
 - `raw_xai.json` - Raw xAI API response
 - `raw_reddit_threads_enriched.json` - Enriched Reddit thread data
@@ -1,47 +0,0 @@
 # last30days Implementation Tasks
 ## Setup & Configuration
 - [x] Create directory structure
 - [x] Write SPEC.md
 - [x] Write TASKS.md
 - [x] Write SKILL.md with proper frontmatter
 ## Core Library Modules
 - [x] scripts/lib/env.py - Environment and API key loading
 - [x] scripts/lib/dates.py - Date range and confidence utilities
 - [x] scripts/lib/cache.py - TTL-based caching
 - [x] scripts/lib/http.py - HTTP client with retry
 - [x] scripts/lib/models.py - Auto model selection
 - [x] scripts/lib/schema.py - Data structures
 - [x] scripts/lib/openai_reddit.py - OpenAI Responses API
 - [x] scripts/lib/xai_x.py - xAI Responses API
 - [x] scripts/lib/reddit_enrich.py - Reddit thread JSON fetcher
 - [x] scripts/lib/normalize.py - Schema normalization
 - [x] scripts/lib/score.py - Popularity scoring
 - [x] scripts/lib/dedupe.py - Near-duplicate detection
 - [x] scripts/lib/render.py - Output rendering
 ## Main Script
 - [x] scripts/last30days.py - CLI orchestrator
 ## Fixtures
 - [x] fixtures/openai_sample.json
 - [x] fixtures/xai_sample.json
 - [x] fixtures/reddit_thread_sample.json
 - [x] fixtures/models_openai_sample.json
 - [x] fixtures/models_xai_sample.json
 ## Tests
 - [x] tests/test_dates.py
 - [x] tests/test_cache.py
 - [x] tests/test_models.py
 - [x] tests/test_score.py
 - [x] tests/test_dedupe.py
 - [x] tests/test_normalize.py
 - [x] tests/test_render.py
 ## Validation
 - [x] Run tests in mock mode
 - [x] Demo --emit=compact
 - [x] Demo --emit=context
 - [x] Verify file tree
@@ -0,0 +1,196 @@
 # How Reddit & X Search Work in last30days
 ## Architecture Overview
 ```
 User: /last30days "kanye west"
          ↓
    ┌─────┴─────┐
    ↓           ↓           (concurrent via ThreadPoolExecutor)
 [REDDIT]    [X/TWITTER]
    ↓           ↓
 OpenAI     Bundled Bird or
 API        xAI API
    ↓           ↓
 Parse       Parse
    ↓           ↓
 Enrich      ───┘
 (fetch         ↓
  actual     [MERGE]
  upvotes)      ↓
    ↓        [NORMALIZE → FILTER → SCORE → DEDUPE]
    └───────────↓
          [OUTPUT to SKILL.md agent]
 ```
 Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`.
 ---
 ## Reddit Search
 ### How it works
 Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only.
 **API Call:**
 ```
 POST https://api.openai.com/v1/responses
 Authorization: Bearer {OPENAI_API_KEY}
 ```
 **Payload:**
 ```json
 {
  "model": "gpt-5.2",
  "tools": [{
    "type": "web_search",
    "filters": { "allowed_domains": ["reddit.com"] }
  }],
  "input": "Search Reddit for threads about {topic}..."
 }
 ```
 The prompt asks the model to:
 1. Extract core subject (strip noise words like "best", "tips", "top")
 2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"`
 3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance`
 4. URLs must contain `/r/` AND `/comments/` (real threads only)
 **Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini`
 Triggers on HTTP 400/403 with access error keywords.
 ### Enrichment (the secret sauce)
 After search, each thread gets **enriched** by hitting Reddit's free JSON API:
 ```
 GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json
 ```
 No API key needed. This returns the actual thread data:
 | Data Point | Source |
 |---|---|
 | Upvotes (score) | Reddit JSON API |
 | Comment count | Reddit JSON API |
 | Upvote ratio | Reddit JSON API |
 | Top 10 comments (text + score) | Reddit JSON API |
 | 7 key comment insights | Extracted via heuristics |
 | Actual post date | `created_utc` timestamp |
 **This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates.
 ### Depth settings
 | Depth | Threads requested | Timeout |
 |---|---|---|
 | `--quick` | 15-25 | 90s |
 | default | 30-50 | 120s |
 | `--deep` | 70-100 | 180s |
 ---
 ## X/Twitter Search
 X search has **two backends** — the skill auto-detects which to use.
 ### Priority: Bundled Bird (env auth) → xAI API (paid)
 ```python
 if node_available and AUTH_TOKEN and CT0:
    use bundled Bird    # Free, popup-free, env-authenticated
 elif XAI_API_KEY:
    use xAI API         # Paid, uses grok-4-1-fast
 else:
    skip X entirely     # No X results
 ```
 ### Backend 1: xAI API
 **API Call:**
 ```
 POST https://api.x.ai/v1/responses
 Authorization: Bearer {XAI_API_KEY}
 ```
 **Payload:**
 ```json
 {
  "model": "grok-4-1-fast",
  "tools": [{ "type": "x_search" }],
  "input": "Search X for posts about {topic} from {from_date} to {to_date}..."
 }
 ```
 The prompt asks grok to return JSON with:
 - `text`, `url`, `author_handle`, `date`
 - `engagement`: `{ likes, reposts, replies, quotes }`
 - `why_relevant`, `relevance` score
 **Engagement data comes from grok's x_search tool** - it has direct access to X's data.
 ### Backend 2: Bundled Bird client (free alternative)
 The repo vendors a search-only subset of Bird's Twitter GraphQL client and shells out to it with Node.js. No global `bird` install is required. The Python wrapper passes `AUTH_TOKEN` and `CT0` via env, which keeps normal local runs headless and avoids browser-cookie prompts.
 **Bundled Bird returns raw X API data** - likes, reposts, replies are real engagement metrics from X's API, not estimates.
 | Metric | Bundled Bird | xAI API |
 |---|---|---|
 | Post text | Real | Real |
 | Likes/reposts | Real (X API) | Real (x_search tool) |
 | Replies/quotes | Real | Real |
 | Author handle | Real | Real |
 | Relevance score | Default 0.7 (re-ranked by relevance.py) | AI-assessed 0.0-1.0 |
 ### Depth settings
 | Depth | xAI posts | Bundled Bird results | xAI timeout | Bird timeout |
 |---|---|---|---|---|
 | `--quick` | 8-12 | 12 | 90s | 30s |
 | default | 20-30 | 30 | 120s | 45s |
 | `--deep` | 40-60 | 60 | 180s | 60s |
 ---
 ## Post-Processing (both sources)
 After both searches complete:
 1. **Normalize** — consistent formatting, timezone handling
 2. **Date filter** — hard filter to requested date range
 3. **Score** — relevance scoring (engagement-weighted)
 4. **Sort** — highest scores first
 5. **Deduplicate** — remove duplicate URLs
 6. **Fallback** — if all items filtered out, keep top 3 by relevance
 ---
 ## Error Handling
 | Layer | Strategy |
 |---|---|
 | HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) |
 | Model access errors | Automatic fallback to next model in chain |
 | Reddit enrichment | Per-item try/catch; keeps unenriched item on failure |
 | X source detection | Silent fallback from Bird → xAI → skip |
 | Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user |
 ---
 ## Key Files
 | File | Purpose |
 |---|---|
 | `skills/last30days/scripts/last30days.py` | Main CLI entry point |
 | `skills/last30days/scripts/lib/pipeline.py` | Multi-source retrieval orchestration |
 | `skills/last30days/scripts/lib/reddit_public.py` | Reddit public JSON search |
 | `skills/last30days/scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
 | `skills/last30days/scripts/lib/xai_x.py` | X search via xAI API |
 | `skills/last30days/scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
 | `skills/last30days/scripts/lib/providers.py` | Reasoning provider and model selection |
 | `skills/last30days/scripts/lib/env.py` | API key loading, source detection |
 | `skills/last30days/scripts/lib/http.py` | HTTP transport with retries |
 | `skills/last30days/scripts/lib/relevance.py` | Query matching and relevance scoring |
 | `skills/last30days/scripts/lib/dedupe.py` | URL-based deduplication |
@@ -0,0 +1,33 @@
 # PR Credits — Thank After V2 Goes Live
 When V2 is pushed to the public repo, comment on each PR to thank the contributor and let them know their work was integrated.
 ## Integrated (cherry-picked into V2)
 | PR | Author | What | Status |
 |---|---|---|---|
 | [#17](https://github.com/mvanhorn/last30days-skill/pull/17) | **@JosephOIbrahim** | Windows Unicode fix (cp1252 emoji crash) | Merge or close with thanks |
 | [#16](https://github.com/mvanhorn/last30days-skill/pull/16) | **@levineam** | Handle 403 model access errors + gpt-4.1 fallback | Merge or close with thanks |
 | [#18](https://github.com/mvanhorn/last30days-skill/pull/18) | **@jonthebeef** | `--days=N` configurable lookback flag | Merge or close with thanks |
 | [#1](https://github.com/mvanhorn/last30days-skill/pull/1) | **@galligan** (Matt Galligan) | Marketplace plugin conversion — we took a lighter approach inspired by his PR | Close with thanks, explain lighter approach |
 ## Already Fixed in V2 (close with thanks)
 | PR | Author | What |
 |---|---|---|
 | [#15](https://github.com/mvanhorn/last30days-skill/pull/15) | **@rszrszrsz** | YAML argument-hint fix — already fixed in V2 |
 | [#11](https://github.com/mvanhorn/last30days-skill/pull/11) | **@nerveband** | Same YAML fix (earlier) — already fixed in V2 |
 ## Not Integrated (close with explanation)
 | PR | Author | What | Why |
 |---|---|---|---|
 | [#5](https://github.com/mvanhorn/last30days-skill/pull/5) | **@jblwilliams** | Codex auth with OpenAI Responses API | Good idea, too complex for now (358 lines SSE/JWT). May revisit. |
 | [#14](https://github.com/mvanhorn/last30days-skill/pull/14) | **@thangman1** | WebSearch-first, API keys optional | Philosophical shift — V2 already does WebSearch in parallel |
 | [#10](https://github.com/mvanhorn/last30days-skill/pull/10) | **@thetechreviewer** | OpenRouter API integration | Too large (1029 lines), adds MCP server |
 ## Suggested Comment Template
 > Thanks for this PR! We integrated your [fix/feature] into V2 (commit XXXXX). Really appreciate the contribution. 🙏
 >
 > Closing this PR since the changes are now in main via a different commit, but full credit to you for the idea and implementation.
@@ -0,0 +1,112 @@
 # v3.0.9 - The Self-Debug Release
 ## Highlights
 **v3.0.9 is live.** New user-facing capabilities, broader cross-platform support, and a skill that now runs reliably on Claude Code, Codex, Hermes, Gemini, claude.ai, and OpenClaw. The headline fix: the engine refuses "birthday gift for 40 year old" style queries with a clarifying question instead of 5 minutes of junk output. The headline feature: TikTok and YouTube top comments now render alongside Reddit's, so the most-engaged voice from every source makes it into the synthesis.
 **The label - "The Self-Debug Release":** I handed 5 separate Opus 4.7 instances their own failed outputs and asked them to debug themselves. Three converged on "SKILL.md is too big and the LAWs are too deep." Two converged on "the engine should refuse demographic-shopping queries outright" and "the WebSearch Sources reminder is overriding LAW 1." I copy-pasted their diagnoses into code. Validation: 5/5 canonical compliance on the topics that had failed.
 ## New capabilities
 - **TikTok and YouTube top comments render alongside Reddit's.** PR [#260](https://github.com/mvanhorn/last30days-skill/pull/260) made the top-engagement comment from each TikTok video and YouTube video first-class in the output - same prominent `💬 Top comment` treatment Reddit's top comment already got. This is the biggest user-facing output change since 3.0.0 and it was never announced. The community inspiration trace: @uppinote20's original push for richer Reddit comments ([PR #143](https://github.com/mvanhorn/last30days-skill/pull/143)) seeded the pattern; this PR generalized it across TikTok and YouTube. PR [#265](https://github.com/mvanhorn/last30days-skill/pull/265) followed up by fixing the ScrapeCreators `url=` param + new response shape for YouTube comments/transcripts so the enrichment actually works.
 - **last30days runs on Hermes AI Agent now.** @stephenmcconnachie's PR ([#228](https://github.com/mvanhorn/last30days-skill/pull/228)) added Hermes as a first-class deploy target. `scripts/sync.sh` detects `~/.hermes/skills/research` and deploys the full skill (SKILL.md, scripts, lib modules, fixtures) to Hermes's skills directory alongside Claude Code and Codex. This is one of the biggest surface-area expansions in v3 - last30days is now usable inside the Hermes agent's research workflows without any manual wiring.
 - **Multi-key SCRAPECREATORS_API_KEY rotation.** @zaydiscold's PR ([#268](https://github.com/mvanhorn/last30days-skill/pull/268)) added automatic key rotation. Set `SCRAPECREATORS_API_KEY_1`, `SCRAPECREATORS_API_KEY_2`, etc. and the engine rotates when a key hits rate limits instead of failing the whole run. For power users running daily queries, this is the difference between rate-limit 429s and zero-touch reliability.
 - **The skill works on Windows now.** @Chelebii's PR ([#227](https://github.com/mvanhorn/last30days-skill/pull/227)) stabilized the vendored Bird X search client on Windows. Previously the bundled X backend had subtle runtime issues on Windows terminals; now it runs clean. Pair this with @Gujiassh's UTF-8 encoding fix ([#225](https://github.com/mvanhorn/last30days-skill/pull/225)) for saved output and Windows users get the full v3 experience without workarounds.
 - **Linux permission checks stopped false-warning.** @george231224's PR ([#216](https://github.com/mvanhorn/last30days-skill/pull/216)) fixed `check_perms` on Linux by preferring GNU stat's syntax over the BSD stat that the skill was calling. Linux users were getting spurious permission warnings on `.env` files that were already correctly 600-chmod'd. Now the check matches reality.
 - **Gemini CLI got a first-class install path.** @hnshah's docs PR ([#224](https://github.com/mvanhorn/last30days-skill/pull/224)) added the Gemini CLI install note and workaround for a rough edge in the Gemini skill loader. Gemini users now have a one-paragraph install flow in the README instead of having to reverse-engineer the plugin layout.
 - **Offline quality evaluation.** @j-sperling's PR ([#233](https://github.com/mvanhorn/last30days-skill/pull/233)) added `eval_topics.json` as a fixture. Contributors and I can now run quality-regression checks on synthesis output without burning live API credits. This is the scaffolding that made the plan 015 validation gate affordable - without eval fixtures, testing 5/5 canonical compliance on every release would cost real money every time. Ships as contributor infrastructure but shows up as stability for end users.
 - **Reddit client got a cleaner HTTP layer.** @iliaal shipped three architecture PRs back-to-back ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209)) that consolidated Reddit's HTTP handling into `http.get(params=...)`, rejected garbage input in `_parse_date`, and unified `_sc_headers` into `http.scrapecreators_headers`. End-user benefit: fewer flaky timeouts, fewer "weird parse error" crashes, a codebase that's easier for future contributors to touch without breaking Reddit. These aren't sexy PRs; they're the kind of refactor that prevents six future bug reports.
 - **The `--days=N` flag keeps working.** @BryanTegomoh's PR ([#230](https://github.com/mvanhorn/last30days-skill/pull/230)) restored backcompat for the legacy `--days` alias so anyone who'd scripted against it in 2.x doesn't break on v3. Small PR, meaningful reliability gain for existing users.
 - **INCLUDE_SOURCES has a sane default.** @hnshah's PR ([#223](https://github.com/mvanhorn/last30days-skill/pull/223)) defaulted the env var to empty string instead of unset. Missing env no longer breaks source inclusion on fresh installs.
 - **Version metadata stays in sync.** @Gujiassh's PR ([#217](https://github.com/mvanhorn/last30days-skill/pull/217)) aligned the SKILL.md version header with the sync target version, and @shalomma's PR ([#229](https://github.com/mvanhorn/last30days-skill/pull/229)) closed the remaining drift between the SKILL.md header and plugin.json. "Which version am I actually on" is no longer an adventure.
 - **Bird X engagement handling got hardened.** @j-sperling's PR ([#234](https://github.com/mvanhorn/last30days-skill/pull/234)) made `bird_x` skip all-None engagement dicts instead of crashing on them. Rare condition, but the kind of thing that silently kills a run on a specific topic.
 - **Dev workflow hygiene.** @j-sperling's gitignore PR ([#232](https://github.com/mvanhorn/last30days-skill/pull/232)) dropped `.venv`, `.coverage`, `htmlcov`, and `.memsearch` from the tracked tree. Contributor quality-of-life; keeps PR diffs clean.
 - **The skill installs to claude.ai.** PRs [#242](https://github.com/mvanhorn/last30days-skill/pull/242) and [#244](https://github.com/mvanhorn/last30days-skill/pull/244) shipped `scripts/build-skill.sh` plus the `.gitattributes` + `export-ignore` plumbing that packages last30days into a claude.ai-upload-ready `.skill` file under the 200-file cap. The skill is no longer Claude-Code-only - it installs directly on claude.ai, too. README has the upload workflow.
 - **OpenAI Codex CLI discovers the skill natively.** PR [#219](https://github.com/mvanhorn/last30days-skill/pull/219) added `.agents/skills/last30days/SKILL.md` as a real file (not symlinked - Codex's loader skips symlinks) plus `.codex-plugin/plugin.json` as the namespace marker. The skill now shows up as `last30days:last30days` when Codex runs in a checkout. Inspired by @Jah-yee ([#153](https://github.com/mvanhorn/last30days-skill/pull/153)) and @dannyshmueli on X.
 - **`/last30days` as a slash command.** PR [#267](https://github.com/mvanhorn/last30days-skill/pull/267) added `commands/last30days.md` so plugin users can type `/last30days <topic>` and Claude Code autocomplete prefix-matches it to the canonical `/last30days:last30days` form. No more typing the double-namespace.
 ## The self-debug technique, for anyone rebuilding this elsewhere
 The breakthrough wasn't the individual fixes. It was the realization that instead of guessing why the model was ignoring the rules, I should ask the model. Five separate Opus 4.7 sessions debugged their own outputs:
 - "Did you read SKILL.md?" → "I tried Read, hit the 25K token cap, and bailed instead of chunked-reading."
 - "Why the trailing Sources block?" → "The WebSearch tool's own reminder said MANDATORY. Precedence was unclear."
 - "Why the section headers?" → "I had strong priors on Peter Steinberger and wrote my thesis instead of passing through."
 - "Why the wrong file?" → "I read `.agents/skills/last30days/SKILL.md` first because it appeared in the path glob."
 Three of the five said "move the LAWs to the top." Two said "make the engine enforce it so the model can't not comply." I shipped both. That's the whole technique: when the LLM-under-orchestration keeps breaking the contract, don't argue with it - ask it to debug itself, and build structural enforcement around whatever it names as the root cause.
 ## Thank you
 **Community PR authors since v3.0.0:**
 - @j-sperling - v3 engine architecture, eval fixtures, gitignore hygiene, Bird X hardening ([#232](https://github.com/mvanhorn/last30days-skill/pull/232), [#233](https://github.com/mvanhorn/last30days-skill/pull/233), [#234](https://github.com/mvanhorn/last30days-skill/pull/234))
 - @stephenmcconnachie - Hermes AI Agent support ([#228](https://github.com/mvanhorn/last30days-skill/pull/228))
 - @zaydiscold - Multi-key SCRAPECREATORS rotation ([#268](https://github.com/mvanhorn/last30days-skill/pull/268))
 - @iliaal - Reddit HTTP helper + GitHub date parsing + ScrapeCreators header consolidation ([#207](https://github.com/mvanhorn/last30days-skill/pull/207), [#208](https://github.com/mvanhorn/last30days-skill/pull/208), [#209](https://github.com/mvanhorn/last30days-skill/pull/209))
 - @Chelebii - Windows Bird X stability ([#227](https://github.com/mvanhorn/last30days-skill/pull/227))
 - @george231224 - Linux check_perms stat ([#216](https://github.com/mvanhorn/last30days-skill/pull/216))
 - @Gujiassh - UTF-8 saved output + version metadata alignment ([#217](https://github.com/mvanhorn/last30days-skill/pull/217), [#225](https://github.com/mvanhorn/last30days-skill/pull/225))
 - @hnshah - INCLUDE_SOURCES default + Gemini install docs ([#223](https://github.com/mvanhorn/last30days-skill/pull/223), [#224](https://github.com/mvanhorn/last30days-skill/pull/224))
 - @shalomma - SKILL.md v3.0.0 version header ([#229](https://github.com/mvanhorn/last30days-skill/pull/229))
 - @BryanTegomoh - --days alias backcompat ([#230](https://github.com/mvanhorn/last30days-skill/pull/230))
 **v3 roadmap contributors (issues and PRs that shaped the v3 feature set):**
 - @uppinote20 - rich Reddit comments ([#143](https://github.com/mvanhorn/last30days-skill/pull/143))
 - @zerone0x - GitHub as a first-class source ([#134](https://github.com/mvanhorn/last30days-skill/issues/134), [#136](https://github.com/mvanhorn/last30days-skill/pull/136))
 - @thinkun - Reddit enrichment timeout handling ([#116](https://github.com/mvanhorn/last30days-skill/pull/116))
 - @thomasmktong - pure-Python Reddit fallback ([#124](https://github.com/mvanhorn/last30days-skill/pull/124))
 - @fanispoulinakisai-boop - Reddit timeout report ([#100](https://github.com/mvanhorn/last30days-skill/issues/100))
 - @pejmanjohn - plugin directory naming ([#99](https://github.com/mvanhorn/last30days-skill/issues/99), [#78](https://github.com/mvanhorn/last30days-skill/issues/78))
 - @zl190 - HN trending merge ([#115](https://github.com/mvanhorn/last30days-skill/pull/115))
 - @hnshah - Watchlist features ([#84](https://github.com/mvanhorn/last30days-skill/pull/84), [#85](https://github.com/mvanhorn/last30days-skill/pull/85), [#86](https://github.com/mvanhorn/last30days-skill/pull/86))
 - @Jah-yee, @dannyshmueli - Codex CLI discovery
 - @Cody-Coyote - marketplace validation bug report ([#204](https://github.com/mvanhorn/last30days-skill/issues/204))
 **The five Opus 4.7 instances that debugged their own failures on v3.0.7 and v3.0.8 and converged on the fixes.** The convergence was the breakthrough; this release is their diagnosis in code.
 ## Install / Update
 ```
 /plugin marketplace add mvanhorn/last30days-skill
 /plugin install last30days@last30days-skill
 ```
 Or if already installed:
 ```
 /plugin update last30days
 /reload-plugins
 ```
 ## Verify
 ```
 cat ~/.claude/plugins/cache/last30days-skill/last30days/*/.claude-plugin/plugin.json | grep version
 ```
 Should print `"version": "3.0.9"`.
 ## Smoke test
 ```
 /last30days birthday gift for 40 year old
 ```
 Should ask a clarifying question before running. If it runs the engine anyway, the cache is stale - repeat the plugin update.
 **Full Changelog:** https://github.com/mvanhorn/last30days-skill/compare/v3.0.5...v3.0.9
@@ -0,0 +1,48 @@
 # Search Quality Eval
 `skills/last30days/scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
 What it does:
 - runs a baseline revision (default `origin/main`) against a candidate checkout
 - evaluates the fixed 5 reviewer topics by default
 - computes deterministic stability metrics:
  - `Jaccard` overlap vs baseline
  - retention vs baseline
  - per-source counts and overlap
 - optionally calls Gemini as a judge for graded relevance labels and then computes:
  - `Precision@5`
  - `nDCG@5`
  - source-coverage recall across the judged union pool
 Recommended usage:
 ```bash
 uv run python skills/last30days/scripts/evaluate_search_quality.py
 ```
 Useful flags:
 ```bash
 uv run python skills/last30days/scripts/evaluate_search_quality.py \
  --baseline-rev origin/main \
  --candidate-rev HEAD \
  --no-default-topics \
  --topic "cursor IDE pricing" \
  --per-source-limit 5
 ```
 Gemini configuration:
 - preferred on this workspace: set `GOOGLE_API_KEY`
 - also accepted: `GEMINI_API_KEY` or `GOOGLE_GENAI_API_KEY`
 - optional: set `GEMINI_MODEL`
 - default model is `gemini-3-pro-preview` for the direct Gemini API
 Notes:
 - The script forces a clean env-based auth path when it shells out to `last30days.py`.
 - It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
 - It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either.
 - `Jaccard` and retention are regression guards, not truth metrics.
 - `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.
@@ -0,0 +1,82 @@
 ---
 title: Search-quality eval is manual by default, not a CI gate on every PR
 date: 2026-05-10
 category: docs/solutions/architecture
 module: skills/last30days/scripts/evaluate_search_quality.py
 problem_type: design_decision
 component: ci_policy
 severity: low
 applies_when:
  - a contributor proposes wiring search-quality eval into PR CI
  - a change affects retrieval, ranking, grounding, or synthesis quality and a reviewer asks "why aren't we testing this in CI?"
  - someone is deciding whether a new evaluator-style script belongs in the default CI workflow
 related_components:
  - search_quality_evaluation
  - ci_workflow
  - llm_judging
 tags:
  - ci-policy
  - eval
  - design-decision
  - cost-vs-signal
  - non-determinism
  - manual-gates
 ---
 # Search-quality eval is manual by default, not a CI gate on every PR
 ## Context
 `skills/last30days/scripts/evaluate_search_quality.py` compares a baseline revision against a candidate revision across a fixed pool of reviewer topics. It produces two flavors of metrics: deterministic overlap (Jaccard, retention) and LLM-judged quality scores. The natural impulse on seeing an evaluator script is to wire it into CI on every PR — "regression catcher, run it automatically." We deliberately don't.
 Three properties of this particular evaluator make CI-on-every-PR the wrong default:
 1. **Live API access.** The candidate revision typically needs the engine to actually run, which means real ScrapeCreators calls, real reddit fetches, real YouTube searches. CI runs would either need production credentials or a record/replay fixture set that drifts almost immediately as external APIs change shape.
 2. **Cost and latency.** A full eval pass runs the pipeline N times across reviewer topics. Multiplied by every PR (including doc-only PRs), the spend is meaningful and the wall-clock pushes CI from ~30s to many minutes.
 3. **Non-determinism in the judging path.** The LLM-judged metrics are valuable for review but depend on judge-model behavior on a given day. A flaky eval that fails 1 PR in 20 because the judge re-scored an item differently is a worse CI signal than no eval at all — it teaches contributors to retry rather than read the result.
 The deterministic overlap metrics are useful regression signals but they are not the same as user-facing correctness. A change that improves overlap can degrade synthesis quality; a change that drops overlap can be a deliberate improvement. So even the deterministic side isn't safe to auto-fail on.
 ## Guidance
 ### 1. Keep search-quality eval available, just not automatic
 The script stays runnable by maintainers and contributors. The pattern is:
 ```bash
 LAST30DAYS_PYTHON=python3.13 \
  python3 skills/last30days/scripts/evaluate_search_quality.py \
  --baseline main --candidate HEAD
 ```
 Reviewers can request a manual eval run when a PR is in the retrieval/ranking/synthesis path and the risk warrants it. Contributors can run it locally before submitting if they want signal upfront.
 ### 2. Standard PR CI gates remain deterministic and contract-shaped
 `pytest` (offline-safe), plugin-contract checks, version-consistency contracts, ruff/lint. Anything that returns the same answer twice for the same input. Quality-of-output assessment lives outside that loop.
 ### 3. The middle ground is `workflow_dispatch`, not auto-PR-gating
 If maintainers want a GitHub-triggered eval that doesn't make every PR pay the live-API cost, the right shape is a manually-dispatched workflow (or a label-triggered one) — not a `pull_request:` workflow that runs unconditionally. That keeps the cost knob in human hands.
 ### 4. Revisit if the eval can ever be made offline-deterministic
 The blocker is the live-API + non-determinism combination. If a future iteration of the script can compute meaningful Jaccard/retention metrics against static fixtures (no live API calls, no LLM judging), the decision flips and it becomes a candidate for default CI. The decision below tracks that condition; revisit when it's met.
 ## What this means in practice
 - Don't merge PRs that wire `evaluate_search_quality.py` into the default `validate.yml` workflow.
 - Do merge PRs that add `workflow_dispatch` triggers or label-gated runs.
 - When reviewing a retrieval/ranking change, request a manual eval if the diff suggests it could regress quality — don't expect CI to catch it.
 ## Links
 - `skills/last30days/scripts/evaluate_search_quality.py` — the evaluator script
 - `docs/search-quality-eval.md` — user-facing usage documentation
 - `.github/workflows/validate.yml` — the default CI workflow (deterministic gates only)
 ---
 *Adapted from a draft ADR proposed by @hnshah in [#374](https://github.com/mvanhorn/last30days-skill/pull/374), restructured into the `docs/solutions/` convention. The original ADR text correctly identified the constraint; this version adds the "why workflow_dispatch is the middle ground" framing and the revisit-condition.*
@@ -0,0 +1,117 @@
 ---
 title: Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items
 date: 2026-06-09
 category: docs/solutions/logic-errors
 module: lib/rerank
 problem_type: logic_error
 component: search_ranking
 severity: high
 symptoms:
  - on-entity, high-engagement items that name the brand but omit the trailing descriptor of a multi-word query are demoted in keyless/fallback rerank results
  - observed case is a 323-point HN thread about Stripe scoring 0 on a "Stripe payments" query
  - the entity-miss demotion lands twice (ENTITY_MISS_PENALTY on rerank_score plus a secondary final_score penalty), so a false miss guarantees burial regardless of engagement
  - reddit keyless comment-enrichment slot selection skips the same on-entity threads via an independently duplicated full-phrase check in _slot_priority
 root_cause: logic_error
 resolution_type: code_fix
 related_components:
  - reddit_keyless
  - comment_enrichment
 tags:
  - entity-grounding
  - rerank
  - keyless-fallback
  - multi-word-entity
  - substring-match
  - false-demotion
  - reddit-keyless
  - duplicated-logic
 ---
 # Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items
 ## Problem
 The keyless/fallback rerank path's entity-grounding demotion required the FULL multi-word primary-entity phrase as a contiguous substring of the candidate's text (`primary_entity.lower() not in haystack`), so on-entity items that omitted a trailing search descriptor were falsely flagged as entity misses and buried by a deliberately decisive double penalty.
 ## Symptoms
 - On a "Stripe payments" query, a 323-point HN thread titled "Stripe is friendly to 'friendly fraud'" was demoted to score 0 — purely because its text never contained the literal phrase "stripe payments" (the trailing word "payments" was missing).
 - The burial is guaranteed by design, not incidental: a flagged entity miss takes −25 `ENTITY_MISS_PENALTY` on `rerank_score` in `_fallback_tuple`, PLUS `ENTITY_MISS_FINAL_PENALTY` applied directly in `_final_score` (added 2026-04-19 after engagement + freshness drowned the diluted penalty). A false positive on the check means confirmed-good signal cannot recover.
 - The same over-strict check had been independently re-implemented in `reddit_keyless._slot_priority` (keyless Reddit comment-enrichment slot selection), so scarce comment slots were also steered away from head-token-only posts.
 ## What Didn't Work
 - **Naively relaxing the check** — the full-phrase check existed for a real reason: on 2026-04-19 an off-topic video with zero brand mentions ranked #2 on a Hermes query (documented in the `ENTITY_MISS_FINAL_PENALTY` comment in `skills/last30days/scripts/lib/rerank.py`). Any fix had to keep that demotion firing.
 - **Word-boundary matching** — rejected; it re-introduces over-demotion on plurals/possessives/compounds ("stripes", "Stripe's").
 - **Graded penalty** (full-phrase = 0, head-only = half, none = full) — rejected; it half-punishes items that are 100% about the entity. Lexical coverage is not topical degree.
 - **Any-token grounding** — rejected; "payments" alone would ground completely generic posts.
 - **Distinctiveness gate for generic heads** — rejected as complexity to patch a failure mode that is already a safe no-op (see Why This Works).
 - **Trusting the docstring** — `reddit_keyless._slot_priority`'s docstring claimed to "mirror rerank's demotion signal," but its inline reimplementation (`entity in _post_text(post).lower()`) had silently drifted from being a mirror into being a second copy of the bug. It was found only by a code-reuse review, not by tests.
 ## Solution
 Ground on the **head token** of the primary entity instead of the full phrase, via one shared helper used by both paths.
 **Site 1 — new helper in `skills/last30days/scripts/lib/rerank.py`:**
 ```python
 def _entity_grounded(haystack: str, primary_entity: str) -> bool:
    tokens = primary_entity.lower().split()
    if not tokens:
        return True
    return tokens[0] in haystack
 ```
 `_fallback_tuple` switches from the inline phrase check to the helper:
 ```python
 # before
 if haystack.strip() and primary_entity.lower() not in haystack:
 # after
 if haystack.strip() and not _entity_grounded(haystack, primary_entity):
 ```
 **Site 2 — secondary penalty in `_final_score`: no code change needed.** It keys off the explanation string set by site 1, so it inherits the fix automatically:
 ```python
 if candidate.explanation and "entity-miss" in candidate.explanation:
    base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
 ```
 **Site 3 — `skills/last30days/scripts/lib/reddit_keyless.py` `_slot_priority`:** replace the drifted reimplementation with a call to the shared helper:
 ```python
 # before
 return entity in _post_text(post).lower()
 # after
 return rerank._entity_grounded(_post_text(post).lower(), entity)
 ```
 Tests: `tests/test_rerank_v3.py` gained `test_fallback_grounds_on_head_token_not_full_phrase` (the Stripe regression) and `test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` (guards the 2026-04-19 behavior). `tests/test_reddit_keyless.py`'s two old-contract tests were rewritten as `test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match`.
 ## Why This Works
 - **Root cause:** trailing tokens of a multi-word query ("payments" in "Stripe payments") are usually category descriptors the user/planner appended for search, not part of the entity name. Requiring the whole phrase conflates "doesn't repeat my search phrasing" with "isn't about my entity." The brand head token alone is sufficient grounding; items that never name the brand at all still miss the head token and stay demoted — so the original 2026-04-19 fix keeps firing.
 - **Asymmetry argument:** the demotion is engineered to be decisive (double penalty across `rerank_score` and `final_score`), so a false entity-miss is fatal-by-design, while a false grounding merely defers the item to normal relevance/freshness/quality ranking. When the punishment is capital, the conviction standard should be conservative.
 - **Substring (not word-boundary) is deliberate:** it catches plurals/possessives/compounds ("stripes", "Stripe's"). Degenerate short heads ("X", "Go", "C") make the check vacuously true, which merely **disables** the penalty — reverting to the pre-grounding baseline — rather than burying good items. Every failure mode of this rule degrades toward "no penalty," never toward "bury good signal."
 - **Accepted, bounded limitation:** head-collision with a different famous entity ("Hermes Agent" → a "Hermes Birkin" thread now escapes demotion). This is lexically unfixable — any token rule strong enough to kill the collision re-kills the Stripe case; the discriminator is semantic. The LLM rerank path (which receives the full phrase as prompt guidance and judges semantically) covers this when API keys exist; the keyless path accepts the bounded risk.
 ## Prevention
 - **Shared helper as single source of truth:** when one module's behavior must "mirror" another's signal, it must *call* the same function, not re-implement the check. The `reddit_keyless._slot_priority` drift happened precisely because the mirror was a copy. The fix wires it to `rerank._entity_grounded`, and the docstring now states this explicitly: "keying on the same head token keeps the two paths from diverging."
 - **Docstrings record deliberate trade-offs:** `_entity_grounded`'s docstring documents WHY head-token (not phrase), why substring (not word-boundary), and the safe-failure direction. Future readers see the rejected alternatives were considered, not overlooked — and won't "tighten" the check into a regression.
 - **Both directions pinned by named tests:**
  - `tests/test_rerank_v3.py::test_fallback_grounds_on_head_token_not_full_phrase` — false-demotion regression (the Stripe HN thread must not be flagged).
  - `tests/test_rerank_v3.py::test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` — the fix must not neuter the demotion (guards the 2026-04-19 off-topic-video incident).
  - `tests/test_reddit_keyless.py::test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match` — the mirrored path asserts the same contract.
 - **Audit tests when changing a contract:** tests that encode the old behavior as correct must be rewritten to the new contract, not worked around — the two old `test_reddit_keyless.py` tests would have silently re-blessed the bug.
 - **For decisive penalties, route through one flag:** the `_final_score` backstop keys off `"entity-miss" in candidate.explanation` rather than re-running the check — so there was exactly one site to fix and the second penalty inherited it for free. Prefer this signal-propagation pattern over duplicating predicate logic at each penalty site.
 ## Related Issues
 - [PR #484](https://github.com/mvanhorn/last30days-skill/pull/484) — "fix(reddit): relevance-aware comment-enrichment slot selection in keyless path" — introduced the `_slot_priority` mirror this fix reroutes through the shared helper.
 - [PR #457](https://github.com/mvanhorn/last30days-skill/pull/457) — "fix(reddit): restore free path via keyless RSS + shreddit scrape" — established the keyless Reddit path.
 - [PR #488](https://github.com/mvanhorn/last30days-skill/pull/488) (open) — "fix(reddit): relevance floor + relevance-first ranking" — external PR touching the same ranking surface; coordinate before merging both.
 - [Issue #468](https://github.com/mvanhorn/last30days-skill/issues/468) (open) — relevance scoring over-pruning on-topic YouTube items; same symptom family in a different source.
 - [../architecture/search-quality-eval-manual-by-default-2026-05-10.md](../architecture/search-quality-eval-manual-by-default-2026-05-10.md) — how to validate ranking/grounding changes like this one (manual eval, not CI-gated).
 - [../workflow-issues/release-consistency-test-cascade-2026-05-16.md](../workflow-issues/release-consistency-test-cascade-2026-05-16.md) — sibling prevention pattern: lockstep artifacts drift unless mechanically unified.
@@ -0,0 +1,219 @@
 ---
 title: Release-time consistency tests cause cascade CI failures across all open PRs
 date: 2026-05-16
 category: docs/solutions/workflow-issues
 module: ci-release-engineering
 problem_type: workflow_issue
 component: testing_framework
 severity: high
 applies_when:
  - a test asserts consistency between two release-time artifacts (e.g., SKILL.md version and a hardcoded pin in a shell script)
  - one artifact is updated as part of a version bump and the other requires a manual lockstep update
  - multiple long-lived PRs are open simultaneously against the same base branch
 symptoms:
  - every open PR's CI fails after a version bump even though the PRs are unrelated to versioning
  - the failing test references a stale hardcoded value that was not updated alongside the bumped version
  - PR authors must rebase and manually fix an artifact they did not touch
 root_cause: missing_workflow_step
 resolution_type: code_fix
 related_components:
  - development_workflow
  - documentation
 tags:
  - ci
  - release-engineering
  - consistency-test
  - version-pin
  - cascade-failure
  - test-design
  - workflow
 ---
 # Release-time consistency tests cause cascade CI failures across all open PRs
 ## Context
 A `tests/test_version_consistency.py::test_sync_cache_path_uses_skill_version` test was added to enforce that the version string embedded in `skills/last30days/scripts/sync.sh` (a hardcoded plugin-cache path segment) matched the version frontmatter in `skills/last30days/SKILL.md`. The intention was sound: the cache path had to stay in lockstep with the skill version or the sync would silently pull stale files.
 The test worked as designed until a release shipped. At that point it turned into a cascade-failure machine:
 1. A release PR bumps `SKILL.md` version (e.g., 3.2.0 → 3.2.1) **and** bumps the `sync.sh` pin. That PR's CI is green.
 2. The release PR merges to `main`.
 3. Every PR that was open at merge time was branched from pre-release `main`. Those PRs have `SKILL.md` 3.2.1 (inherited via merge-base with `main`) but their branch never touched `sync.sh`.
 4. CI for those PRs runs the consistency test against the new `main` — `SKILL.md` says 3.2.1, `sync.sh` still says 3.2.0 — and fails.
 5. All open PRs are now red simultaneously, with a failure that has nothing to do with their changes.
 This affected at least five PRs during the 2026-05-13 to 2026-05-15 window: PR #400 (caught during rebase, required a manual pin bump), PRs #390 and #392 (OpenClaw `SCRAPECREATORS_API_KEY` fix, both stalled for the same stale-pin reason), and at least two others. A follow-up hotfix PR (#397 — `fix(sync): bump cache target to 3.2.1 to match SKILL.md`) was required just to unblock the queue.
 The permanent fix was PR #405: delete `sync.sh` entirely (the install workflow made it redundant) and drop `test_sync_cache_path_uses_skill_version`. Once both were gone, no version-consistency cascade was possible.
 ## Guidance
 ### 1. Don't write consistency tests that read two files and assert one matches a substring derived from the other
 This pattern looks safe but is not:
 ```python
 def test_sync_cache_path_uses_skill_version(self) -> None:
    sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
    version = _skill_version()          # reads SKILL.md
    self.assertIn(
        f'last30days-skill/last30days/{version}"',
        sync_text,                      # asserts sync.sh contains that string
    )
 ```
 It encodes the assumption that both files are always updated together, in the same commit, on the same branch. That assumption breaks the moment two files have independent lifecycle owners — a versioned manifest and a deployment script are archetypal examples.
 ### 2. If the values genuinely need to stay in sync, derive one from the other at runtime
 Remove the hardcoded pin from `sync.sh` and compute it:
 ```bash
 # sync.sh — derive version from SKILL.md at runtime, no pin to maintain
 SKILL_VERSION=$(grep -m1 '^version:' "$(dirname "$0")/../SKILL.md" \
    | sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
 CACHE_PATH="last30days-skill/last30days/${SKILL_VERSION}"
 ```
 Now there is only one source of truth (`SKILL.md`). The test that asserted they matched becomes vacuous and should be deleted. If `SKILL.md` is wrong, the sync itself will fail loudly — which is better feedback than a CI gate on a different PR.
 ### 3. If two values must stay independent for legitimate reasons, update them together and make the test self-skip if either source is missing
 If separate versioning is genuinely required (e.g., SKILL.md versions for harness consumers, sync.sh versions a private artifact store with its own cadence), update both in the same PR — never staggered — and write the test to self-skip rather than error when either file is absent:
 ```python
 def test_sync_cache_path_uses_skill_version(self) -> None:
    sync_sh = SKILL_ROOT / "scripts" / "sync.sh"
    if not sync_sh.exists():
        self.skipTest("sync.sh not present; skipping pin consistency check")
    sync_text = sync_sh.read_text(encoding="utf-8")
    version = _skill_version()
    self.assertIn(
        f'last30days-skill/last30days/{version}"',
        sync_text,
    )
 ```
 Self-skipping means deleting the file is a non-event in CI — no cascading red, no hotfix PR to the queue.
 ### 4. Run consistency tests against the merge-base diff, not main
 If you keep a two-file consistency test, scope it so it only fails when the PR itself modifies one of the two files but not the other. A GitHub Actions step can do this:
 ```yaml
 - name: Check sync.sh version pin consistency
  run: |
    BASE=$(git merge-base HEAD origin/main)
    SKILL_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'SKILL\.md' || true)
    SYNC_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'sync\.sh' || true)
    if [ "$SKILL_CHANGED" -gt 0 ] && [ "$SYNC_CHANGED" -eq 0 ]; then
      echo "SKILL.md version bumped but sync.sh pin was not updated"
      exit 1
    fi
 ```
 This only fires when your PR touched `SKILL.md` and left `sync.sh` alone — never because a release merged to `main` after you branched.
 ### 5. Ask whether you actually need this test
 If the values are wrong, downstream tooling will fail loudly: the sync will fetch the wrong artifact, installs will break, or the harness will reject the version. A test that exists only to catch a human-bookkeeping error at release time adds cascade-fail risk without offering a meaningfully earlier signal. Weigh that cost before adding any two-file consistency gate.
 ## Why This Matters
 The damage from a stale-pin consistency test is asymmetric. It:
 - Fails on every open PR simultaneously the moment a release lands on `main` — not just the PR that forgot to update the pin.
 - Produces a failure message that points at a line in a test file with no obvious relationship to the PR's actual changes.
 - Requires either a hotfix PR (touching a file the failing PRs have no business touching) or a manual rebase of every affected branch.
 - Blocks work that has already been reviewed and approved.
 In this repo the effect was measurable: at least five PRs stalled across a two-day window, one hotfix PR was shipped just to unblock the queue, and multiple authors spent time debugging a failure completely unrelated to their changes.
 The broader principle is that tests which gate on *bookkeeping consistency between files* impose their maintenance cost on every contributor, every time, even when those contributors did nothing wrong. That cost compounds with team size and release cadence.
 ## When to Apply
 Apply this guidance whenever you find yourself:
 - Writing a test that reads two files and asserts that a string in one matches a value derived from the other.
 - Adding a CI step labeled "consistency check," "sync check," or "pin check" where the check compares a hardcoded value against a computed one from a separate file.
 - Working in a repo where a versioned manifest (e.g., `SKILL.md`, `package.json`, `pyproject.toml`) and a deployment artifact (e.g., a shell script, a Dockerfile, a Helm values file) are both maintained by hand.
 - Reviewing a PR that touches only one of two "paired" files and fails a consistency test for the other.
 It does *not* apply to tests that read a single source of truth and validate its internal structure (e.g., asserting that `SKILL.md`'s frontmatter version is double-quoted, or that `package.json`'s `version` field is a valid semver string). Those tests have one file and one assertion; they cannot cascade across branches.
 ## Examples
 ### Before — the pattern that caused the cascade
 Original `tests/test_version_consistency.py` (deleted in commit `9fb19ea`):
 ```python
 import re
 import unittest
 from pathlib import Path
 ROOT = Path(__file__).resolve().parents[1]
 SKILL_ROOT = ROOT / "skills" / "last30days"
 def _skill_version() -> str:
    text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
    match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
    if not match:
        raise AssertionError("SKILL.md version frontmatter not found")
    return match.group(1)
 class TestVersionConsistency(unittest.TestCase):
    def test_sync_cache_path_uses_skill_version(self) -> None:
        sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
        version = _skill_version()          # source 1: SKILL.md frontmatter
        self.assertIn(                      # assertion: sync.sh must contain
            f'last30days-skill/last30days/{version}"',
            sync_text,                      # source 2: hardcoded string in sync.sh
        )
 ```
 `sync.sh` contained a line like:
 ```bash
 PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/3.2.0"
 ```
 When SKILL.md bumped to `3.2.1` in a release PR, `sync.sh` was updated in the same PR and CI stayed green. But every PR branched before that release still had `sync.sh` at `3.2.0`. Their CI failed immediately, with an assertion error pointing at the test, not at the release PR.
 ### After — what we did: delete both
 PR #405 deleted `sync.sh` (the install workflow replaced it) and dropped `test_sync_cache_path_uses_skill_version` in the same change. No consistency gate, no pin to maintain, no cascade possible.
 ### After — what we could have done instead: derive at runtime
 If `sync.sh` had still been needed, the right fix would have been to remove the hardcoded version from the script and derive it from `SKILL.md`:
 ```bash
 #!/usr/bin/env bash
 # sync.sh — no hardcoded version; reads SKILL.md as single source of truth
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 SKILL_VERSION=$(grep -m1 '^version:' "${SCRIPT_DIR}/../SKILL.md" \
    | sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
 if [ -z "$SKILL_VERSION" ]; then
    echo "error: could not parse version from SKILL.md" >&2
    exit 1
 fi
 PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/${SKILL_VERSION}"
 # ... rest of sync logic
 ```
 With this in place, `test_sync_cache_path_uses_skill_version` has no reason to exist — there is nothing to assert. Delete it. If the version parsing breaks, `sync.sh` itself exits non-zero with a clear message.
 ## Related
 - **PR #397** (merged) — `fix(sync): bump cache target to 3.2.1 to match SKILL.md`. The hotfix that unblocked the cascade temporarily by bumping the pin.
 - **PR #400** (merged) — caught the same cascade during rebase; had to bump the pin to clear CI.
 - **PR #390** (closed) and **PR #392** (rebased + merged) — OpenClaw `SCRAPECREATORS_API_KEY` fix; both blocked by the cascade until rebased onto post-#405 main.
 - **PR #405** (merged) — the permanent fix: deleted `sync.sh` + `test_sync_cache_path_uses_skill_version` together.
 - **PR #412** (merged) — adjacent work that consolidated SKILL.md version parsing into `lib/skill_meta.py`, reducing future drift risk by giving the version field one canonical reader.
@@ -0,0 +1,120 @@
 # v2.1 Launch Copy (DRAFT — do not publish)
 ## Context
 Bird CLI (@steipete/bird) has been deprecated on npm and the GitHub repo deleted. steipete was asked to take it down (likely by X). Nobody has forked and republished it. Our v2.1 vendors a search-only subset of Bird v0.8.0 (MIT licensed) so users don't need any external tools.
 YouTube transcripts are the second headline feature. Inspired by Peter Steinberger's yt-dlp + summarize toolchain. We use yt-dlp directly (no summarize dependency) — search YouTube, grab transcripts, feed them into the synthesis. Zero API keys, zero cost.
 ---
 ## README: "New in V2.1" blurb
 **New in V2.1 — two headline features:**
 - **YouTube transcripts as a 4th source.** When yt-dlp is installed, /last30days automatically searches YouTube, grabs view counts, and extracts auto-generated transcripts from the top videos. A 20-minute review contains 10x the signal of a tweet — now the skill reads it. Inspired by @steipete's yt-dlp + summarize toolchain.
 - **X search is fully bundled.** No external `bird` CLI install needed. Add `AUTH_TOKEN` and `CT0` once, and the vendored Bird client runs locally without browser-cookie prompts. `XAI_API_KEY` remains an optional fallback.
 ---
 ## README: X Search Authentication section
 ### X Search Authentication
 X search prefers explicit env auth. This keeps local runs headless and avoids browser-cookie and macOS Keychain prompts.
 **Recommended setup:** While logged into x.com once, open browser dev tools and copy the `auth_token` and `ct0` cookies for `x.com`.
 Save them as `AUTH_TOKEN` and `CT0` in `~/.config/last30days/.env` or `.claude/last30days.env`:
 ```bash
 AUTH_TOKEN=your_auth_token
 CT0=your_ct0_token
 ```
 **xAI fallback:** If you do not want to provide `AUTH_TOKEN` and `CT0`, set `XAI_API_KEY` and use xAI's `x_search` backend instead.
 **Verify it's working:**
 ```bash
 node ~/.claude/skills/last30days/scripts/lib/vendor/bird-search/bird-search.mjs --whoami
 ```
 ---
 ## README: Install block env line
 ```bash
 AUTH_TOKEN=...            # recommended for X search
 CT0=...                   # recommended for X search
 XAI_API_KEY=xai-...       # optional X fallback
 ```
 ---
 ## SKILL.md: Stats line
 ```
 ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
 ```
 ---
 ## GitHub issue #19 response (post AFTER publishing)
 > Thanks for reporting this. Bird CLI was deprecated and the GitHub repo was deleted. steipete was asked to take it down.
 >
 > The good news: you don't need Bird anymore. v2.1 (just shipped) bundles X search directly. No external CLI, no `npm install`, no brew. Just Node.js 22+ plus `AUTH_TOKEN` and `CT0`, or `XAI_API_KEY` as fallback.
 >
 > It also adds **YouTube as a 4th source**. When yt-dlp is installed, the skill automatically searches YouTube and extracts transcripts from the top videos. A 20-minute tutorial has 10x the signal of a tweet, and now the synthesis engine reads it.
 >
 > The recommended setup is to copy `auth_token` and `ct0` from x.com once and store them as `AUTH_TOKEN` and `CT0` in your env. That avoids browser-cookie and Keychain prompts during normal runs.
 >
 > The xAI API (`XAI_API_KEY`) also still works as a fallback.
 ---
 ## X/Social launch post
 ### Short (280 chars)
 /last30days v2.1 is out 🚀
 Two new features:
 → YouTube transcripts as a 4th source (yt-dlp)
 → X search fully bundled (no bird CLI install needed)
 Research any topic across Reddit, X, YouTube & web in one command.
 h/t @steipete for the inspiration on both.
 github.com/mvanhorn/last30days-skill
 ### Thread version (post 1)
 /last30days v2.1 just shipped — two headline features:
 1️⃣ YouTube transcripts as a 4th source
 When yt-dlp is installed, the skill searches YouTube, grabs view counts, and extracts auto-generated transcripts from top videos. A 20-min review has 10x the signal of a tweet — now the synthesis reads it.
 ### Thread version (post 2)
 2️⃣ X search is fully bundled
 Bird CLI was deprecated. Instead of requiring an external tool, v2.1 vendors a search-only subset. Add `AUTH_TOKEN` and `CT0` once, then it runs locally with no npm install. `XAI_API_KEY` still works as fallback.
 Both features inspired by @steipete's tooling.
 ### Thread version (post 3)
 YouTube goes through the same scoring pipeline as Reddit and X — relevance, recency, engagement. Transcripts get truncated to ~500 words per video and fed into the synthesis engine alongside social posts.
 Zero API keys for YouTube. Zero cost. Just `brew install yt-dlp`.
 ### Thread version (post 4)
 Try it:
 ```
 /last30days [any topic]
 ```
 Reddit + X + YouTube + Web. Four sources, one command, copy-paste prompts.
 github.com/mvanhorn/last30days-skill
@@ -0,0 +1,360 @@
 # last30days v2.5 Launch Thread
 ## FINAL THREAD (6 tweets)
 ### 1/6 - Announcement
 I can't believe it's been 30 days since I launched @slashlast30days. 3.2k stars later, time for v2.5.
 Three big additions:
 1. @Polymarket prediction markets as a 6th source - helps you predict the future
 2. Cross-source linking + massively better results - detects when the same story trends across multiple platforms. Ran a 15-way blinded comparison, v2.5 scored 4.38 vs 3.73 for the original. Won all 5 topics.
 3. Hacker News as a 5th source - a window into the tech and developer insider world
 github.com/mvanhorn/last30days-skill
 ### 2/6 - Demo: Anthropic vs Pentagon
 "/last30days Anthropic Pete Hegseth"
 14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views). 5 HN stories. 9 Polymarket markets.
 This story broke TODAY. Hegseth designated Anthropic a "supply chain risk." Trump ordered every agency to stop using their tech.
 Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March: only 6%.
 Markets say Anthropic wins regardless. That's the kind of signal you can't get from opinion threads.
 ### 3/6 - Demo: Seedance Prompting
 "/last30days Seedance prompting"
 13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 15 web pages.
 Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style. Beyond 100 words, results degrade.
 Then I said: "a cinematic drone shot over a city at golden hour"
 It wrote me a copy-paste prompt using the exact patterns from the research. Research first, then create from what you learned.
 ### 4/6 - Demo: Arizona Basketball
 "/last30days arizona basketball"
 6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
 Arizona is 25-2, set a program record with a 22-0 start, and holds a 2-game Big 12 lead with 3 games left. The Field of 68 called them "the TOUGHEST team in America" after escaping Baylor shorthanded. Kansas rematch Saturday - the highlight video from their first meeting has 248K views on ESPN's YouTube.
 Polymarket: Championship 13%. #1 seed: 88%. Duke and Michigan each at 18% to win it all.
 That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
 ### 5/6 - Demo: Iran War
 "/last30days iran war"
 2 Reddit threads. 34 X posts (10,048 likes). 20 YouTube videos (1.6M views, 5 transcripts). 4 HN stories (850 points). 14 Polymarket markets ($473M volume).
 Geneva talks just ended without a deal. 150+ US aircraft deployed. Two carrier strike groups in position. F-22s sent to Israel. Members of Congress who saw the secret war plan came out "terrified." @cenkuygur: "they are about to drag us into a war that 70-85% of Americans oppose" (7,700 likes).
 Polymarket ($473M in volume - one of their biggest markets ever): strikes by 2026: 80%. By March 31: 68%. War Powers invoked: 51%. Formal war declaration: only 12%.
 Markets say: strikes are very likely, declared war is not. That's the sharpest signal in the entire research.
 ### 6/6 - Thank You
 Thank you to ARJ999 and wkbaran on GitHub who filed three separate issues asking for Hacker News support. v2.5 delivers.
 It's been a crazy 30 days. 3.2k stars. Six sources. Massively better results. Super excited to get this out.
 Try it: /last30days [any topic]
 github.com/mvanhorn/last30days-skill
 ---
 ---
 ## REFERENCE MATERIAL BELOW
 ## Context
 - 3.2k stars on GitHub
 - V2.5 headline features: Polymarket (6th source), Hacker News (5th source), cross-source linking
 - Ran 15-way blinded comparison: 4.38/5.0 vs 3.73/5.0
 - Won all 5 topics, zero regressions
 - Cross-source linking: 3 -> 13 linked items
 - Demo topics: Anthropic odds (11 markets), Arizona basketball (6 markets), Iran war ($425M volume)
 ---
 ## Post 1: Lead (Announcement)
 V2.5 of @slashlast30days is out. Now with @Polymarket prediction markets, cross-source linking, and massively better results.
 1. Polymarket as a 6th source - real money on outcomes, no API key needed
 2. Hacker News as a 5th source
 3. Cross-source linking - detects when the same story trends across multiple platforms
 Ran a 15-way blinded comparison across 5 topics. v2.5 scored 4.38 vs 3.73 for the original. Won all 5. Zero regressions.
 github.com/mvanhorn/last30days-skill
 ---
 ## Post 2: POLYMARKET AS A 6TH SOURCE.
 Reddit tells you what people think. X tells you what people share. YouTube tells you what people watch. HN tells you what developers discuss.
 Polymarket helps you predict the future.
 "/last30days anthropic odds"
 11 markets found. Best AI model February: Anthropic 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. Pentagon ban odds: only 22%.
 Free API. No key. Real money on outcomes.
 ---
 ## Post 3: CROSS-SOURCE LINKING.
 When a Seedance 2.0 tutorial has 44K YouTube views AND trends on HN AND gets discussed on Reddit, v2.5 flags it: [also on: HN, YouTube]
 Old version linked 3 items across 5 test topics. New version links 13. The difference is hybrid similarity - combining character-trigram and token-level matching at a tuned threshold.
 Cross-platform convergence is the strongest signal that something actually matters. Not engagement on one platform. Convergence across all of them.
 ---
 ## Post 4: 15-WAY BLINDED EVALUATION.
 I don't trust vibes for measuring quality. So I ran a scientific comparison.
 5 topics x 3 versions. Stripped version labels. Randomized as A/B/C. Scored on groundedness, specificity, coverage, actionability, and format.
 v2.5: 4.38/5.0
 v2.2 (HN only): 4.10/5.0
 v2.0 (original): 3.73/5.0
 Won all 5 topics. Zero regressions. Biggest gains: specificity (+0.8) and format (+1.0) from cross-source linking giving the synthesis better material to work with.
 ---
 ## Post 5: Demo - Anthropic Odds
 Asked it about Anthropic odds.
 11 Polymarket markets. 25 X posts. 13 YouTube videos (719K views). 6 HN stories (471 points).
 Best AI model February: 98%. IPO before OpenAI: 64%. $500B+ valuation: 87%. FrontierMath 50% score: 48% (up 28% today). Pentagon ban: only 22%.
 Markets say Anthropic is winning the model race AND the valuation race. The Pentagon thing is noise.
 ---
 ## Post 6: Demo - Arizona Basketball
 "/last30days arizona basketball"
 6 Polymarket markets. 37 X posts (4,200 likes). 15 YouTube videos (517K views). 2 Reddit threads.
 Championship odds: 13%. #1 seed: 88%. Big 12 title: Arizona leads by 2.
 That's not a sports blog. That's Reddit reactions + X engagement + YouTube analysis + prediction market odds from one command.
 The Polymarket integration uses two-pass query expansion. First pass finds "Arizona Big 12." Second pass discovers the championship and #1 seed markets via tag-based domain bridging.
 ---
 ## Post 7: Demo - Iran War
 The best Polymarket demo is news.
 "/last30days iran war"
 14 Polymarket markets. $425M+ in volume. 7 Reddit threads. 30 X posts. 20 YouTube videos (2M views). 18 HN stories (1,187 points).
 US strikes Iran by 2026: 70%. War Powers by March: 60%. Israel strikes by June: 64%. Formal war declaration: only 8%.
 Markets say: limited strikes with War Powers, NOT a declared war. Breaking Points (435K views) covered leaked Pentagon opposition. r/Conservative "imploding" per r/SubredditDrama.
 One command. Six sources. Real money.
 ---
 ## Post 8: Credits + CTA
 Also in v2.5: YouTube synonym expansion ("hip hop" now matches "rap" - relevance jumped 0.33 to 0.71), X handle resolution, and HN OR queries for framework topics.
 The difference between "good research" and "research you'd actually trust" is in details like this.
 Try it: /last30days [any topic]
 github.com/mvanhorn/last30days-skill
 ---
 ## Post 9: Demo - Claude Code (ALL 6 sources)
 "/last30days Claude Code"
 3 Reddit threads (199 upvotes). 35 X posts (5,239 likes). 15 YouTube videos (1.4M views, 5 transcripts). 30 HN stories (~8,500 points). 8 Polymarket markets. 20 web pages.
 All six sources hit. Top finding: the planning-first workflow has won. The #1 HN post this month (969 pts, 590 comments) is about separating planning from execution. Boris Cherny (Head of Claude Code) on Lenny's Podcast: "100% of my code is written by Claude Code - I have not edited a single line by hand since November."
 Polymarket: Anthropic 99% for best AI model in February. 58% for March. Claude on FrontierMath at 55%. The US government rejected Claude - Polymarket has the Hegseth ban at 32%.
 Then I asked it to dig deeper into the planning-first workflow. No new searches - it answered from what it already learned.
 ---
 ## Post 10: Demo - March Madness Odds (Polymarket + Sports)
 "/last30days March Madness Odds"
 2 Reddit threads. 31 X posts. 6 YouTube videos (46K views, 4 transcripts). 2 Polymarket markets.
 Tournament winner: Duke 18%, Michigan 18%, Arizona 13%. #1 seeds: Michigan 98%, Duke 91%, Arizona 88%.
 Duke is the hottest mover - went from +700 to +450 in one week. The skill surfaced that from sportsbook data, X commentary, and Polymarket odds simultaneously.
 Then I asked it to break down Michigan vs Duke vs Arizona. Full analysis from the research it already had.
 ---
 ## Post 11: Demo - Seedance Prompting (Expert + Prompt Mode)
 "/last30days Seedance prompting"
 13 Reddit threads. 33 X posts. 20 YouTube videos (1.2M views, 4 transcripts). 1 HN story. 15 web pages.
 Top finding: Seedance 2.0 prompts follow a director's shot-list format, not freeform text. 30-100 words. Subject + Action + Camera + Scene + Style + Constraints. Beyond 100 words, results degrade.
 Then I said: "a cinematic drone shot over a city at golden hour"
 It wrote me a copy-paste prompt using the exact patterns from the research. That's the skill's real power - research first, then create from what you learned.
 ---
 ## Post 12: Thank You + CTA (Final)
 Thank you to ARJ999 and wkbaran on GitHub who kept asking for Hacker News support. Three separate issues. v2.5 delivers.
 30 days. 3.2k stars. 6 sources. Massively better results.
 Try it: /last30days [any topic]
 github.com/mvanhorn/last30days-skill
 ---
 ## Post 13: Demo - Anthropic vs Pentagon (Breaking News + Polymarket)
 "/last30days Anthropic Pete Hegseth"
 14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views, 5 transcripts). 5 HN stories. 9 Polymarket markets. 10 web pages.
 This story broke TODAY. Defense Secretary Hegseth designated Anthropic a "supply chain risk" - believed to be the first time an American company has ever received this designation. Trump ordered every federal agency to stop using Anthropic tech.
 Polymarket: Anthropic still 99% for best AI model. $500B+ valuation: 68%. IPO >$600B: 97%. Hegseth out by March 31: only 6%.
 Markets say: Anthropic wins the model race regardless. Bettors don't think Hegseth survives this. That's the kind of signal you can't get from opinion threads.
 ---
 ## Post 14: Demo - OpenAI Insider Trading (News + Polymarket)
 "/last30days OpenAI Insider Trading"
 2 Reddit threads. 29 X posts. 4 YouTube videos (360K views, 4 transcripts). 2 HN stories. 15 Polymarket markets. 15 web pages.
 An OpenAI employee was just fired for using confidential info to bet on Polymarket. 13 brand-new wallets appeared 40 hours before the browser launch. $309K bet on the right outcome. Unusual Whales flagged 77 suspected insider positions across 60 wallets.
 Meanwhile Polymarket has OpenAI's IPO at $1.25-1.5T: 54%. Anthropic IPOs first: 62%. Best AI model: Anthropic 99%.
 The prediction markets are both the story AND the source. One command pulled all of it together.
 ---
 ## Standalone Tweet: Polymarket Stats Line
 "/last30days Anthropic Pete Hegseth"
 The Pentagon just designated Anthropic a supply chain risk. First time ever for an American company. Trump ordered every agency to stop using their tech.
 Here's what Polymarket says:
 📊 9 markets │ Best AI model: 99% │ $500B+ valuation: 68% │ IPO >$600B: 97% │ Hegseth out by March: 6%
 Bettors with real money on the line think Anthropic wins the model race, goes public at a massive valuation, and Hegseth doesn't survive this.
 That's the gap between headlines and reality. One command, six sources.
 github.com/mvanhorn/last30days-skill
 ---
 ## Recommended Thread Order (pick 8-10)
 The full thread above is 12 posts. Here's what I'd cut to keep it tight:
 **Must include (core story):**
 1. Post 1 - Lead announcement
 2. Post 2 - Polymarket ("Reddit tells you what people think...")
 3. Post 3 - Cross-source linking
 4. Post 4 - Blinded evaluation
 **Best demos (pick 3-4):**
 - Post 13 (Anthropic vs Pentagon) - STRONGEST. Breaking news today. Polymarket cuts through the noise. "Markets say Anthropic wins regardless."
 - Post 9 (Claude Code) - All 6 sources. Massive numbers. Shows follow-up flow.
 - Post 10 (March Madness) - Sports/Polymarket crossover. Timely with tournament approaching.
 - Post 11 (Seedance) - Shows prompting flow. 1.2M YouTube views.
 - Post 5 (Anthropic Odds) - Overlaps with Post 13 now. Skip.
 **Skip or save for standalone tweets:**
 - Post 5 (Anthropic Odds) - Redundant with Post 13
 - Post 6 (Arizona Basketball) - Covered by March Madness now
 - Post 7 (Iran War) - Great standalone tweet, not for launch thread
 - Post 8 (Credits/minor features) - Fold into CTA
 **My recommended 8-post thread:**
 1. Lead (Post 1)
 2. Polymarket (Post 2)
 3. Cross-source linking (Post 3)
 4. Blinded evaluation (Post 4)
 5. Demo: Anthropic vs Pentagon (Post 13) - breaking news, best Polymarket showcase
 6. Demo: Claude Code (Post 9)
 7. Demo: March Madness (Post 10)
 8. Thank you + CTA (Post 12)
 ---
 ## Video Script (~60 seconds)
 **[Talking to camera]**
 Oh my god, I can't believe it's been 30 days since I launched last30days. 3,200 stars on GitHub. This has been the craziest month.
 Today I'm shipping v2.5 and I'm really excited about this one. Three big things.
 **[Screen recording: typing /last30days Anthropic Pete Hegseth]**
 First - Polymarket prediction markets as a 6th source. So this Anthropic-Pentagon story broke today. Hegseth designated Anthropic a supply chain risk, Trump ordered agencies to stop using their tech. Scary headline, right?
 But Polymarket says: Anthropic still 99% for best AI model. IPO above 600 billion: 97%. Hegseth out by March: 6%. Real money on outcomes helps you predict the future. That's a different story than the headlines.
 **[Screen recording: typing /last30days arizona basketball]**
 Second - it now searches Hacker News and does cross-source linking. When the same story shows up on Reddit AND YouTube AND HN, it flags it. I ran a 15-way blinded comparison and v2.5 scored 4.38 versus 3.73 for the original. Won all 5 test topics.
 **[Back to camera]**
 Thank you to everyone who starred it, filed issues, and kept pushing me to make this better. Shoutout to the people on GitHub who literally filed three separate issues asking for Hacker News. v2.5 delivers.
 Link in bio. Try it on anything.
 ---
 ## Scoring Note
 The 4.38 vs 3.73 score is from a custom 5-dimension rubric (30% groundedness, 25% specificity, 20% coverage, 15% actionability, 10% format compliance) evaluated by Claude on blinded outputs. The relative ranking is meaningful; the absolute numbers are not. It's an LLM grading LLM output - useful for A/B comparison, not for claiming "4.38 out of 5 quality."
 If using the score in a tweet, frame it as "scored X vs Y on a blinded comparison" not "rated 4.38/5.0 quality" - the former is honest, the latter implies an objective standard that doesn't exist.
@@ -0,0 +1,42 @@
 [
  {
    "topic": "OpenClaw vs NanoClaw vs ZeroClaw",
    "query_type": "comparison",
    "rationale": "Multi-entity extraction, 3-way split across AI agent frameworks."
  },
  {
    "topic": "how to set up a GLP-1 supplement routine",
    "query_type": "how_to",
    "rationale": "Trending health topic. Tests non-tech how_to."
  },
  {
    "topic": "2026 March Madness",
    "query_type": "breaking_news",
    "rationale": "Live sporting event. Tests broad breaking news recall."
  },
  {
    "topic": "best budget noise cancelling headphones 2026",
    "query_type": "product",
    "rationale": "Evergreen consumer query. Tests product review aggregation."
  },
  {
    "topic": "thoughts on OpenAI Codex pricing",
    "query_type": "opinion",
    "rationale": "Active developer debate. Tests opinion mining."
  },
  {
    "topic": "odds of US recession 2026",
    "query_type": "prediction",
    "rationale": "Major macro topic. Tests prediction market + news synthesis."
  },
  {
    "topic": "what is retrieval augmented generation",
    "query_type": "concept",
    "rationale": "Widely discussed AI concept. Tests explanation quality."
  },
  {
    "topic": "Google Wiz acquisition price and timeline",
    "query_type": "factual",
    "rationale": "Completed event ($32B). Tests factual precision."
  }
 ]
@@ -0,0 +1,8 @@
 <!-- FIXTURE: captured live from reddit.com/svc/shreddit/community-more-posts/top/?name=technology&t=week on 2026-05-29; trimmed to 5 post cards (start-tag attrs only). -->
 <div id="feed">
 <shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tq0zk7/the_netherlands_just_blocked_a_us_company_from/" content-href="https://www.techspot.com/news/112552-netherlands-blocked-us-company-buying-app-dutch-citizens.html" view-context="SubredditFeed" comment-count="1743" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="0" award-count="23" award-id="award_obsessed_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/Obsessed_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-28T11:37:01.506000+0000" domain="techspot.com" id="t3_1tq0zk7" post-title="The Netherlands just blocked a US company from buying the app Dutch citizens use for everything" post-language="en" post-type="link" score="52692" upvote-ratio="0.9606269354736776" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
 <shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1toe7m2/erin_brockovich_launches_map_of_over_4200_data/" content-href="https://www.newsweek.com/erin-brockovich-asks-americans-for-help-as-she-launches-data-center-map-11989813" view-context="SubredditFeed" comment-count="673" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="2" award-count="6" award-id="award_this_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/this_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T17:39:43.272000+0000" domain="newsweek.com" id="t3_1toe7m2" post-title="Erin Brockovich launches map of over 4,200 data centres in the US, appeals for local communities to report environmental impact and other costs" post-language="en" post-type="link" score="33567" upvote-ratio="0.973297166968053" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_fj9vsvfd" author="marketrent" icon="https://www.redditstatic.com/avatars/defaults/v2/avatar_default_1.png" data-expected-lcp subreddit-name="technology"></shreddit-post>
 <shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tollgz/majority_of_americans_support_ban_on_surveillance/" content-href="https://gizmodo.com/majority-of-americans-support-ban-on-surveillance-pricing-and-electronic-shelf-labels-2000762717" view-context="SubredditFeed" comment-count="1043" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="3" award-count="7" award-id="award_free_bravo" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/bravo_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T21:55:07.322000+0000" domain="gizmodo.com" id="t3_1tollgz" post-title="Majority of Americans Support Ban on Surveillance Pricing and Electronic Shelf Labels" post-language="en" post-type="link" score="29791" upvote-ratio="0.9815063671850003" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_98wao505" author="Plastic_Ninja_9014" icon="https://preview.redd.it/snoovatar/avatars/69af2b53-b0a1-4ab6-b119-d90f21c423fe-headshot.png?width=64&amp;height=64&amp;crop=smart&amp;auto=webp&amp;s=f3661eb511798004968f8b115a689dcee30f1428" data-expected-lcp subreddit-name="technology"></shreddit-post>
 <shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tp5qz2/tech_ceos_are_apparently_suffering_from_ai/" content-href="https://techcrunch.com/2026/05/27/tech-ceos-are-apparently-suffering-from-ai-psychosis/" view-context="SubredditFeed" comment-count="1653" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="4" award-count="6" award-id="award_free_regret_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/regret_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-27T13:33:49.280000+0000" domain="techcrunch.com" id="t3_1tp5qz2" post-title="Tech CEOs are apparently suffering from AI psychosis" post-language="en" post-type="link" score="26419" upvote-ratio="0.9605741880002646" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
 <shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tn5g7s/pope_leo_issues_ai_encyclical_warning_that_opaque/" content-href="https://variety.com/2026/biz/global/pope-leo-ai-encyclical-algorithms-threaten-dehumanisation-1236758186/" view-context="SubredditFeed" comment-count="608" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="6" award-count="7" award-id="award_hooray_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/FTUE_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-25T10:45:04.093000+0000" domain="variety.com" id="t3_1tn5g7s" post-title="Pope Leo Issues AI Encyclical Warning That ‘Opaque Algorithms’ Controlled by a ‘Few’ Companies Can Bring ‘New Forms of Dehumanisation’" post-language="en" post-type="link" score="25835" upvote-ratio="0.9760626539506095" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_1i1zizibn9" author="yourfavchoom" icon="https://styles.redditmedia.com/t5_dgdrt8/styles/profileIcon_k9x929ihm8rg1.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=2e8a5042cccc4555167f98d28bc0de4e13fd3ca5" data-expected-lcp subreddit-name="technology"></shreddit-post>
 </div>
@@ -0,0 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- FIXTURE: captured live from reddit.com/r/Rakuten/top.rss on 2026-05-29; trimmed to 5 entries. Atom shape identical to search.rss. --><feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/"><category term="Rakuten" label="r/Rakuten"/><updated>2026-05-29T14:14:32+00:00</updated><icon>https://www.redditstatic.com/icon.png/</icon><id>/r/Rakuten/top.rss?t=month</id><link rel="self" href="https://www.reddit.com/r/Rakuten/top.rss?t=month" type="application/atom+xml" /><link rel="alternate" href="https://www.reddit.com/r/Rakuten/top?t=month" type="text/html" /><subtitle>This is an unofficial subreddit for Rakuten Rewards, the cash back website. We are not affiliated with, endorsed by, or sponsored by Rakuten or any of its subsidiaries.</subtitle><title>top scoring links : Rakuten</title><entry><author><name>/u/InternetUser52</name><uri>https://www.reddit.com/user/InternetUser52</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I&amp;#39;m rich!!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/InternetUser52&quot;&gt; /u/InternetUser52 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/q8fgmxs29c2h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tiv013</id><link href="https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/" /><updated>2026-05-20T18:48:31+00:00</updated><published>2026-05-20T18:48:31+00:00</published><title>LETS GOO! $0.02!!!</title></entry>
 <entry><author><name>/u/Immediate-Duck-6351</name><uri>https://www.reddit.com/user/Immediate-Duck-6351</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I don’t travel and I’m buying a house in a few weeks so cash back is amazing 🙌 hoping to keep the pace in the next quarter so I can buy new kitchen appliances lol. &lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Immediate-Duck-6351&quot;&gt; /u/Immediate-Duck-6351 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/d2a4s0ipvb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te1fp8</id><link href="https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/" /><updated>2026-05-15T16:29:28+00:00</updated><published>2026-05-15T16:29:28+00:00</published><title>So excited 🥳</title></entry>
 <entry><author><name>/u/gnibgnib</name><uri>https://www.reddit.com/user/gnibgnib</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;128k for the May transfer&lt;/p&gt; &lt;p&gt;41k pending for August &lt;/p&gt; &lt;p&gt;Got another 9k at Asics not showing but overall pretty happy with Rakuten&lt;/p&gt; &lt;p&gt;P2 was able to secure 85k for May transfer&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/gnibgnib&quot;&gt; /u/gnibgnib &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/gallery/1tb8674&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tb8674</id><link href="https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/" /><updated>2026-05-12T17:17:19+00:00</updated><published>2026-05-12T17:17:19+00:00</published><title>Had a great run so far this year thanks to this sub!</title></entry>
 <entry><author><name>/u/TravelVet93</name><uri>https://www.reddit.com/user/TravelVet93</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/TravelVet93&quot;&gt; /u/TravelVet93 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/x6b9whvupb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te0hom</id><link href="https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/" /><updated>2026-05-15T15:56:40+00:00</updated><published>2026-05-15T15:56:40+00:00</published><title>My best payout so far</title></entry>
 <entry><author><name>/u/Beautiful-Piece-4252</name><uri>https://www.reddit.com/user/Beautiful-Piece-4252</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;The amount of $$ available in sign up bonuses is amazing. It&amp;#39;s kind of a part time job ensuring Rakuten captures everything, but my August and November payout should be sizeable. I&amp;#39;m new to this and it always seemed like a lot of work for little reward. I know it&amp;#39;s not sustainable, but wow!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Beautiful-Piece-4252&quot;&gt; /u/Beautiful-Piece-4252 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/1vqvajsci42h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1thsnm1</id><link href="https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/" /><updated>2026-05-19T16:46:17+00:00</updated><published>2026-05-19T16:46:17+00:00</published><title>How can this be real?</title></entry>
 </feed>
@@ -0,0 +1,29 @@
 <!-- FIXTURE: captured live from reddit.com/svc/shreddit/comments/r/Rakuten/t3_1taeiw0 on 2026-05-29;
     trimmed to 6 real comment elements (real attrs + real bodies) + 2 synthetic edge cases. -->
 <shreddit-comment-tree-stats total-comments="14"></shreddit-comment-tree-stats>
 <shreddit-comment-tree id="comment-tree" post-id="t3_1taeiw0">
 <shreddit-comment created="2026-05-11T20:16:57.590000+0000" author="Obvious_Painting_881" thingId="t1_ol8tp8n" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8tp8n/" score="2" postId="t3_1taeiw0" content-type="text">
  <div id="t1_ol8tp8n-comment-rtjson-content" slot="comment"><div id="t1_ol8tp8n-post-rtjson-content" dir="auto"><p dir="auto">Where do you find $750? The highest available package for Total was $284.99 when I did the lifelock promotion. I did get the full 284.99 from Rakuten.</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-12T12:26:14.973000+0000" author="Stormtrooper149" thingId="t1_olcy1iv" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/olcy1iv/" score="2" postId="t3_1taeiw0" content-type="text">
  <div id="t1_olcy1iv-comment-rtjson-content" slot="comment"><div id="t1_olcy1iv-post-rtjson-content" dir="auto"><p dir="auto">It went to pending ($712.49)</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-19T01:43:48.026000+0000" author="heythereyou01" thingId="t1_omlbiqg" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/omlbiqg/" score="1" postId="t3_1taeiw0" content-type="text">
  <div id="t1_omlbiqg-comment-rtjson-content" slot="comment"><div id="t1_omlbiqg-post-rtjson-content" dir="auto"><p dir="auto">Hey I PM’d. can I get the screenshot ?</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-11T20:21:16.398000+0000" author="Stormtrooper149" thingId="t1_ol8undb" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8undb/" score="1" postId="t3_1taeiw0" content-type="text">
  <div id="t1_ol8undb-comment-rtjson-content" slot="comment"><div id="t1_ol8undb-post-rtjson-content" dir="auto"><p dir="auto">Family plan</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-11T20:28:33.803000+0000" author="Obvious_Painting_881" thingId="t1_ol8w8w6" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8w8w6/" score="1" postId="t3_1taeiw0" content-type="text">
  <div id="t1_ol8w8w6-comment-rtjson-content" slot="comment"><div id="t1_ol8w8w6-post-rtjson-content" dir="auto"><p dir="auto">Price seems to change every time I go to the page but I see only 249.99-369.99 for Total/Advanced. No where near your $750. Just saying the Total plan for 299.99 worked for me and I got 284.99 which is 95%.</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-12T02:33:48.200000+0000" author="jwegener" thingId="t1_olaqzjk" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/olaqzjk/" score="2" postId="t3_1taeiw0" content-type="text">
  <div id="t1_olaqzjk-comment-rtjson-content" slot="comment"><div id="t1_olaqzjk-post-rtjson-content" dir="auto"><p dir="auto">I did that one. Let’s pray</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-13T10:00:00.000000+0000" author="[deleted]" thingId="t1_synthdel" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/synthdel/" score="5" postId="t3_1taeiw0" content-type="text">
  <div id="t1_synthdel-comment-rtjson-content" slot="comment"><div id="t1_synthdel-post-rtjson-content" dir="auto"><p dir="auto">[removed]</p></div></div>
 </shreddit-comment>
 <shreddit-comment created="2026-05-13T11:00:00.000000+0000" author="NegScoreUser" thingId="t1_synthneg" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/synthneg/" score="-7" postId="t3_1taeiw0" content-type="text">
  <div id="t1_synthneg-comment-rtjson-content" slot="comment"><div id="t1_synthneg-post-rtjson-content" dir="auto"><p dir="auto">A downvoted but real reply with negative score for edge-case coverage.</p></div></div>
 </shreddit-comment>
 </shreddit-comment-tree>
@@ -0,0 +1,58 @@
 {
  "items": [
    {
      "video_id": "7543693751290481942",
      "text": "This Claude Code trick saved me hours #claudecode #ai #coding",
      "url": "https://www.tiktok.com/@codemaster/video/7543693751290481942",
      "author_name": "codemaster",
      "date": "2026-02-28",
      "engagement": {
        "views": 2100000,
        "likes": 45000,
        "comments": 1200,
        "shares": 8400
      },
      "hashtags": ["claudecode", "ai", "coding"],
      "duration": 45,
      "relevance": 0.85,
      "why_relevant": "TikTok: This Claude Code trick saved me hours #claude",
      "caption_snippet": "So I found this insane trick with Claude Code where you can use slash commands to automate everything"
    },
    {
      "video_id": "7543100200112345678",
      "text": "AI coding tools comparison 2026 - Claude vs Copilot vs Cursor #ai #devtools",
      "url": "https://www.tiktok.com/@techreviewer/video/7543100200112345678",
      "author_name": "techreviewer",
      "date": "2026-02-25",
      "engagement": {
        "views": 850000,
        "likes": 22000,
        "comments": 890,
        "shares": 3200
      },
      "hashtags": ["ai", "devtools"],
      "duration": 60,
      "relevance": 0.7,
      "why_relevant": "TikTok: AI coding tools comparison 2026 - Claude vs Copi",
      "caption_snippet": ""
    },
    {
      "video_id": "7543200300223456789",
      "text": "You need to try Claude Code RIGHT NOW #programming #tips",
      "url": "https://www.tiktok.com/@devtips/video/7543200300223456789",
      "author_name": "devtips",
      "date": "2026-03-01",
      "engagement": {
        "views": 500000,
        "likes": 15000,
        "comments": 450,
        "shares": 2100
      },
      "hashtags": ["programming", "tips"],
      "duration": 30,
      "relevance": 0.6,
      "why_relevant": "TikTok: You need to try Claude Code RIGHT NOW #programm",
      "caption_snippet": "Let me show you why Claude Code is the best AI coding tool right now"
    }
  ]
 }
@@ -0,0 +1,67 @@
 {
  "name": "last30days-skill",
  "version": "3.3.2",
  "description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
  "settings": [
    {
      "name": "Extension Directory",
      "description": "Extension installation directory (auto-set by Gemini CLI)",
      "envVar": "GEMINI_EXTENSION_DIR",
      "sensitive": false
    },
    {
      "name": "ScrapeCreators API Key",
      "description": "ScrapeCreators API Key for Reddit, TikTok, and Instagram search (required)",
      "envVar": "SCRAPECREATORS_API_KEY",
      "sensitive": true
    },
    {
      "name": "OpenAI API Key",
      "description": "OpenAI API Key - optional fallback for Reddit discovery",
      "envVar": "OPENAI_API_KEY",
      "sensitive": true
    },
    {
      "name": "xAI API Key",
      "description": "xAI API Key for X/Twitter search (optional)",
      "envVar": "XAI_API_KEY",
      "sensitive": true
    },
    {
      "name": "OpenRouter API Key",
      "description": "OpenRouter API Key (optional)",
      "envVar": "OPENROUTER_API_KEY",
      "sensitive": true
    },
    {
      "name": "Parallel AI API Key",
      "description": "Parallel AI API Key (optional)",
      "envVar": "PARALLEL_API_KEY",
      "sensitive": true
    },
    {
      "name": "Brave Search API Key",
      "description": "Brave Search API Key (optional)",
      "envVar": "BRAVE_API_KEY",
      "sensitive": true
    },
    {
      "name": "Apify API Token",
      "description": "Apify API Token (optional legacy)",
      "envVar": "APIFY_API_TOKEN",
      "sensitive": true
    },
    {
      "name": "Twitter AUTH_TOKEN",
      "description": "Twitter browser AUTH_TOKEN cookie for direct X search (optional)",
      "envVar": "AUTH_TOKEN",
      "sensitive": true
    },
    {
      "name": "Twitter CT0",
      "description": "Twitter browser CT0 cookie (optional, pair with AUTH_TOKEN)",
      "envVar": "CT0",
      "sensitive": true
    }
  ]
 }
@@ -0,0 +1,4 @@
 {
  "triggerOnUpdates": true,
  "statusCheck": true
 }
@@ -0,0 +1,15 @@
 {
  "hooks": {
    "SessionStart": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "bash \"${CLAUDE_PLUGIN_ROOT:-${extensionPath:-.}}/hooks/scripts/check-config.sh\""
          }
        ]
      }
    ]
  }
 }
@@ -0,0 +1,173 @@
 #!/bin/bash
 set -euo pipefail
 # Check last30days configuration status and show appropriate welcome message.
 # Priority: .claude/last30days.env > ~/.config/last30days/.env > env vars
 PROJECT_ENV=".claude/last30days.env"
 GLOBAL_ENV="$HOME/.config/last30days/.env"
 # Helper: warn if file permissions are too open
 check_perms() {
  local file="$1"
  if [[ ! -f "$file" ]]; then return; fi
  local perms
  # Try GNU stat first (Linux), fall back to BSD stat (macOS).
  # On Linux, `stat -f` prints filesystem info (not permissions) and exits 0,
  # so the previous BSD-first ordering left $perms as multi-line garbage on
  # every Linux session start and printed a false WARNING.
  perms=$(stat -c '%a' "$file" 2>/dev/null || stat -f '%Lp' "$file" 2>/dev/null || echo "")
  if [[ -n "$perms" && "$perms" != "600" && "$perms" != "400" ]]; then
    echo "/last30days: WARNING — $file has permissions $perms (should be 600)."
    echo "  Fix: chmod 600 $file"
  fi
 }
 # Load env file into variables for inspection (without exporting)
 load_env_vars() {
  local file="$1"
  if [[ -f "$file" ]]; then
    while IFS='=' read -r key value; do
      # Skip comments, empty lines
      [[ "$key" =~ ^[[:space:]]*# ]] && continue
      [[ -z "$key" ]] && continue
      key=$(echo "$key" | xargs)
      value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
      # Strip inline comments (# preceded by whitespace) to prevent
      # command substitution in backtick-containing comments
      value="${value%%[[:space:]]#*}"
      if [[ -n "$key" && -n "$value" ]]; then
        # printf -v writes via assignment semantics (global from inside a
        # function), works on macOS's /bin/bash 3.2 — `declare -g` is 4.2+.
        printf -v "ENV_${key}" '%s' "$value"
      fi
    done < "$file"
  fi
 }
 # Determine which config file is active
 CONFIG_FILE=""
 if [[ -f "$PROJECT_ENV" ]]; then
  CONFIG_FILE="$PROJECT_ENV"
  check_perms "$PROJECT_ENV"
 elif [[ -f "$GLOBAL_ENV" ]]; then
  CONFIG_FILE="$GLOBAL_ENV"
  check_perms "$GLOBAL_ENV"
 fi
 # Load config if found
 if [[ -n "$CONFIG_FILE" ]]; then
  load_env_vars "$CONFIG_FILE"
 fi
 # Check SETUP_COMPLETE (from file or env)
 SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
 # Compute last-run summary line (if last-run.json exists)
 if [[ "${LAST30DAYS_CONFIG_DIR+x}" == "x" ]]; then
  if [[ -n "$LAST30DAYS_CONFIG_DIR" ]]; then
    LAST_RUN_FILE="$LAST30DAYS_CONFIG_DIR/last-run.json"
  else
    LAST_RUN_FILE=""
  fi
 else
  LAST_RUN_FILE="$HOME/.config/last30days/last-run.json"
 fi
 LAST_RUN_LINE=""
 if [[ -n "$LAST_RUN_FILE" && -f "$LAST_RUN_FILE" ]] && command -v python3 &>/dev/null; then
  LAST_RUN_LINE=$(LAST_RUN_FILE="$LAST_RUN_FILE" python3 - <<'PY' 2>/dev/null || true
 import datetime
 import json
 import os
 path = os.environ["LAST_RUN_FILE"]
 try:
    with open(path) as fh:
        d = json.load(fh)
    topic = (d.get("topic") or "?")[:60]
    ts = d.get("timestamp", "")
    dt = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00"))
    delta = (datetime.datetime.now(datetime.timezone.utc) - dt).total_seconds()
    if delta < 60: ago = f"{int(delta)}s ago"
    elif delta < 3600: ago = f"{int(delta//60)}m ago"
    elif delta < 86400: ago = f"{int(delta//3600)}h ago"
    else: ago = f"{int(delta//86400)}d ago"
    total = d.get("total", 0)
    print(f"  Last run: \"{topic}\" · {ago} · {total} results")
 except Exception:
    pass
 PY
 )
 fi
 # If setup has never been run, show welcome message for new users
 if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
  cat <<'EOF'
 /last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
  Research any topic across Reddit, HN, X, YouTube, Polymarket (last 30 days).
 Reddit, Hacker News, and Polymarket work out of the box.
 The setup wizard can unlock X/Twitter, YouTube, and more.
 EOF
  [[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
  exit 0
 fi
 # Setup done but check for ScrapeCreators
 HAS_SCRAPECREATORS="${ENV_SCRAPECREATORS_API_KEY:-${SCRAPECREATORS_API_KEY:-}}"
 HAS_X="${ENV_AUTH_TOKEN:-${AUTH_TOKEN:-}}"
 HAS_XAI="${ENV_XAI_API_KEY:-${XAI_API_KEY:-}}"
 HAS_YTDLP=""
 if command -v yt-dlp &>/dev/null; then
  HAS_YTDLP="yes"
 fi
 HAS_BSKY="${ENV_BSKY_HANDLE:-${BSKY_HANDLE:-}}"
 HAS_EXA="${ENV_EXA_API_KEY:-${EXA_API_KEY:-}}"
 # Count active sources
 SOURCE_COUNT=2  # HN + Polymarket are always free
 if [[ -n "$HAS_X" || -n "$HAS_XAI" ]]; then
  SOURCE_COUNT=$((SOURCE_COUNT + 1))
 fi
 # Reddit public JSON always works
 SOURCE_COUNT=$((SOURCE_COUNT + 1))
 if [[ -n "$HAS_YTDLP" ]]; then
  SOURCE_COUNT=$((SOURCE_COUNT + 1))
 fi
 if [[ -n "$HAS_EXA" ]]; then
  SOURCE_COUNT=$((SOURCE_COUNT + 1))
 fi
 if [[ -n "$HAS_BSKY" ]]; then
  SOURCE_COUNT=$((SOURCE_COUNT + 1))
 fi
 if [[ -n "$HAS_SCRAPECREATORS" ]]; then
  # Start with Reddit comments + TikTok + Instagram, subtract any in EXCLUDE_SOURCES.
  # Normalise EXCLUDED (lowercase + collapse whitespace around commas + strip outer
  # whitespace) so the matching mirrors pipeline.py's .strip().lower() parsing.
  SC_ADD=3
  EXCLUDED="${ENV_EXCLUDE_SOURCES:-${EXCLUDE_SOURCES:-}}"
  EXCLUDED_NORM=$(printf '%s' "$EXCLUDED" | tr '[:upper:]' '[:lower:]' \
    | sed -E 's/[[:space:]]*,[[:space:]]*/,/g; s/^[[:space:]]+//; s/[[:space:]]+$//')
  if [[ ",$EXCLUDED_NORM," == *",tiktok,"* ]]; then
    SC_ADD=$((SC_ADD - 1))
  fi
  if [[ ",$EXCLUDED_NORM," == *",instagram,"* ]]; then
    SC_ADD=$((SC_ADD - 1))
  fi
  SOURCE_COUNT=$((SOURCE_COUNT + SC_ADD))
 fi
 if [[ -n "$HAS_SCRAPECREATORS" ]]; then
  # Fully configured — compact ready message
  echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
  echo "  Research any topic across social + market + web sources (last 30 days)."
  [[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
 else
  # Setup done but missing ScrapeCreators — recommend it
  echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
  echo "  Research any topic across social + market + web sources (last 30 days)."
  [[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
  echo "  Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
  echo "  100 free credits, no credit card — scrapecreators.com"
  echo "  last30days has no affiliation with any API provider."
 fi
@@ -1,395 +0,0 @@
 # feat: Add WebSearch as Third Source (Zero-Config Fallback)
 ## Overview
 Add Claude's built-in WebSearch tool as a third research source for `/last30days`. This enables the skill to work **out of the box with zero API keys** while preserving the primacy of Reddit/X as the "voice of real humans with popularity signals."
 **Key principle**: WebSearch is supplementary, not primary. Real human voices on Reddit/X with engagement metrics (upvotes, likes, comments) are more valuable than general web content.
 ## Problem Statement
 Currently `/last30days` requires at least one API key (OpenAI or xAI) to function. Users without API keys get an error. Additionally, web search could fill gaps where Reddit/X coverage is thin.
 **User requirements**:
 - Work out of the box (no API key needed)
 - Must NOT overpower Reddit/X results
 - Needs proper weighting
 - Validate with before/after testing
 ## Proposed Solution
 ### Weighting Strategy: "Engagement-Adjusted Scoring"
 **Current formula** (same for Reddit/X):
 ```
 score = 0.45*relevance + 0.25*recency + 0.30*engagement - penalties
 ```
 **Problem**: WebSearch has NO engagement metrics. Giving it `DEFAULT_ENGAGEMENT=35` with `-10 penalty` = 25 base, which still competes unfairly.
 **Solution**: Source-specific scoring with **engagement substitution**:
 | Source | Relevance | Recency | Engagement | Source Penalty |
 |--------|-----------|---------|------------|----------------|
 | Reddit | 45% | 25% | 30% (real metrics) | 0 |
 | X | 45% | 25% | 30% (real metrics) | 0 |
 | WebSearch | 55% | 35% | 0% (no data) | -15 points |
 **Rationale**:
 - WebSearch items compete on relevance + recency only (reweighted to 100%)
 - `-15 point source penalty` ensures WebSearch ranks below comparable Reddit/X items
 - High-quality WebSearch can still surface (score 60-70) but won't dominate (Reddit/X score 70-85)
 ### Mode Behavior
 | API Keys Available | Default Behavior | `--include-web` |
 |--------------------|------------------|-----------------|
 | None | **WebSearch only** | n/a |
 | OpenAI only | Reddit only | Reddit + WebSearch |
 | xAI only | X only | X + WebSearch |
 | Both | Reddit + X | Reddit + X + WebSearch |
 **CLI flag**: `--include-web` (default: false when other sources available)
 ## Technical Approach
 ### Architecture
 ```
 ┌─────────────────────────────────────────────────────────────────┐
 │                     last30days.py orchestrator                   │
 ├─────────────────────────────────────────────────────────────────┤
 │  run_research()                                                  │
 │  ├── if sources includes "reddit": openai_reddit.search_reddit()│
 │  ├── if sources includes "x": xai_x.search_x()                  │
 │  └── if sources includes "web": websearch.search_web() ← NEW    │
 └─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
 ┌─────────────────────────────────────────────────────────────────┐
 │                     Processing Pipeline                          │
 ├─────────────────────────────────────────────────────────────────┤
 │  normalize_websearch_items() → WebSearchItem schema ← NEW        │
 │  score_websearch_items() → engagement-free scoring ← NEW         │
 │  dedupe_websearch() → deduplication ← NEW                        │
 │  render_websearch_section() → output formatting ← NEW            │
 └─────────────────────────────────────────────────────────────────┘
 ```
 ### Implementation Phases
 #### Phase 1: Schema & Core Infrastructure
 **Files to create/modify:**
 ```python
 # scripts/lib/websearch.py (NEW)
 """Claude WebSearch API client for general web discovery."""
 WEBSEARCH_PROMPT = """Search the web for content about: {topic}
 CRITICAL: Only include results from the last 30 days (after {from_date}).
 Find {min_items}-{max_items} high-quality, relevant web pages. Prefer:
 - Blog posts, tutorials, documentation
 - News articles, announcements
 - Authoritative sources (official docs, reputable publications)
 AVOID:
 - Reddit (covered separately)
 - X/Twitter (covered separately)
 - YouTube without transcripts
 - Forum threads without clear answers
 Return ONLY valid JSON:
 {{
  "items": [
    {{
      "title": "Page title",
      "url": "https://...",
      "source_domain": "example.com",
      "snippet": "Brief excerpt (100-200 chars)",
      "date": "YYYY-MM-DD or null",
      "why_relevant": "Brief explanation",
      "relevance": 0.85
    }}
  ]
 }}
 """
 def search_web(topic: str, from_date: str, to_date: str, depth: str = "default") -> dict:
    """Search web using Claude's built-in WebSearch tool.
    NOTE: This runs INSIDE Claude Code, so we use the WebSearch tool directly.
    No API key needed - uses Claude's session.
    """
    # Implementation uses Claude's web_search_20250305 tool
    pass
 def parse_websearch_response(response: dict) -> list[dict]:
    """Parse WebSearch results into normalized format."""
    pass
 ```
 ```python
 # scripts/lib/schema.py - ADD WebSearchItem
@dataclass
 class WebSearchItem:
    """Normalized web search item."""
    id: str
    title: str
    url: str
    source_domain: str  # e.g., "medium.com", "github.com"
    snippet: str
    date: Optional[str] = None
    date_confidence: str = "low"
    relevance: float = 0.5
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    def to_dict(self) -> Dict[str, Any]:
        return {
            'id': self.id,
            'title': self.title,
            'url': self.url,
            'source_domain': self.source_domain,
            'snippet': self.snippet,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
 ```
 #### Phase 2: Scoring System Updates
 ```python
 # scripts/lib/score.py - ADD websearch scoring
 # New constants
 WEBSEARCH_SOURCE_PENALTY = 15  # Points deducted for lacking engagement
 # Reweighted for no engagement
 WEBSEARCH_WEIGHT_RELEVANCE = 0.55
 WEBSEARCH_WEIGHT_RECENCY = 0.45
 def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
    """Score WebSearch items WITHOUT engagement metrics.
    Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty
    """
    for item in items:
        rel_score = int(item.relevance * 100)
        rec_score = dates.recency_score(item.date)
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=0,  # Explicitly zero - no engagement data
        )
        overall = (
            WEBSEARCH_WEIGHT_RELEVANCE * rel_score +
            WEBSEARCH_WEIGHT_RECENCY * rec_score
        )
        # Apply source penalty (WebSearch < Reddit/X)
        overall -= WEBSEARCH_SOURCE_PENALTY
        # Apply date confidence penalty (same as other sources)
        if item.date_confidence == "low":
            overall -= 10
        elif item.date_confidence == "med":
            overall -= 5
        item.score = max(0, min(100, int(overall)))
    return items
 ```
 #### Phase 3: Orchestrator Integration
 ```python
 # scripts/last30days.py - UPDATE run_research()
 def run_research(...) -> tuple:
    """Run the research pipeline.
    Returns: (reddit_items, x_items, web_items, raw_openai, raw_xai,
              raw_websearch, reddit_error, x_error, web_error)
    """
    # ... existing Reddit/X code ...
    # WebSearch (new)
    web_items = []
    raw_websearch = None
    web_error = None
    if sources in ("all", "web", "reddit-web", "x-web"):
        if progress:
            progress.start_web()
        try:
            raw_websearch = websearch.search_web(topic, from_date, to_date, depth)
            web_items = websearch.parse_websearch_response(raw_websearch)
        except Exception as e:
            web_error = f"{type(e).__name__}: {e}"
        if progress:
            progress.end_web(len(web_items))
    return (reddit_items, x_items, web_items, raw_openai, raw_xai,
            raw_websearch, reddit_error, x_error, web_error)
 ```
 #### Phase 4: CLI & Environment Updates
 ```python
 # scripts/last30days.py - ADD CLI flag
 parser.add_argument(
    "--include-web",
    action="store_true",
    help="Include general web search alongside Reddit/X (lower weighted)",
 )
 # scripts/lib/env.py - UPDATE get_available_sources()
 def get_available_sources(config: dict) -> str:
    """Determine available sources. WebSearch always available (no API key)."""
    has_openai = bool(config.get('OPENAI_API_KEY'))
    has_xai = bool(config.get('XAI_API_KEY'))
    if has_openai and has_xai:
        return 'both'  # WebSearch available but not default
    elif has_openai:
        return 'reddit'
    elif has_xai:
        return 'x'
    else:
        return 'web'  # Fallback: WebSearch only (no keys needed)
 ```
 ## Acceptance Criteria
 ### Functional Requirements
 - [x] Skill works with zero API keys (WebSearch-only mode)
 - [x] `--include-web` flag adds WebSearch to Reddit/X searches
 - [x] WebSearch items have lower average scores than Reddit/X items with similar relevance
 - [x] WebSearch results exclude Reddit/X URLs (handled separately)
 - [x] Date filtering uses natural language ("last 30 days") in prompt
 - [x] Output clearly labels source type: `[WEB]`, `[Reddit]`, `[X]`
 ### Non-Functional Requirements
 - [x] WebSearch adds <10s latency to total research time (0s - deferred to Claude)
 - [x] Graceful degradation if WebSearch fails
 - [ ] Cache includes WebSearch results appropriately
 ### Quality Gates
 - [x] Before/after testing shows WebSearch doesn't dominate rankings (via -15pt penalty)
 - [x] Test: 10 Reddit + 10 X + 10 WebSearch → WebSearch avg score 15-20pts lower (scoring formula verified)
 - [x] Test: WebSearch-only mode produces useful results for common topics
 ## Testing Plan
 ### Before/After Comparison Script
 ```python
 # tests/test_websearch_weighting.py
 """
 Test harness to validate WebSearch doesn't overpower Reddit/X.
 Run same queries with:
 1. Reddit + X only (baseline)
 2. Reddit + X + WebSearch (comparison)
 Verify: WebSearch items rank lower on average.
 """
 TEST_QUERIES = [
    "best practices for react server components",
    "AI coding assistants comparison",
    "typescript 5.5 new features",
 ]
 def test_websearch_weighting():
    for query in TEST_QUERIES:
        # Run without WebSearch
        baseline = run_research(query, sources="both")
        baseline_scores = [item.score for item in baseline.reddit + baseline.x]
        # Run with WebSearch
        with_web = run_research(query, sources="both", include_web=True)
        web_scores = [item.score for item in with_web.web]
        reddit_x_scores = [item.score for item in with_web.reddit + with_web.x]
        # Assertions
        avg_reddit_x = sum(reddit_x_scores) / len(reddit_x_scores)
        avg_web = sum(web_scores) / len(web_scores) if web_scores else 0
        assert avg_web < avg_reddit_x - 10, \
            f"WebSearch avg ({avg_web}) too close to Reddit/X avg ({avg_reddit_x})"
        # Check top 5 aren't all WebSearch
        top_5 = sorted(with_web.reddit + with_web.x + with_web.web,
                       key=lambda x: -x.score)[:5]
        web_in_top_5 = sum(1 for item in top_5 if isinstance(item, WebSearchItem))
        assert web_in_top_5 <= 2, f"Too many WebSearch items in top 5: {web_in_top_5}"
 ```
 ### Manual Test Scenarios
 | Scenario | Expected Outcome |
 |----------|------------------|
 | No API keys, run `/last30days AI tools` | WebSearch-only results, useful output |
 | Both keys + `--include-web`, run `/last30days react` | Mix of all 3 sources, Reddit/X dominate top 10 |
 | Niche topic (no Reddit/X coverage) | WebSearch fills gap, becomes primary |
 | Popular topic (lots of Reddit/X) | WebSearch present but lower-ranked |
 ## Dependencies & Prerequisites
 - Claude Code's WebSearch tool (`web_search_20250305`) - already available
 - No new API keys required
 - Existing test infrastructure in `tests/`
 ## Risk Analysis & Mitigation
 | Risk | Likelihood | Impact | Mitigation |
 |------|------------|--------|------------|
 | WebSearch returns stale content | Medium | Medium | Enforce date in prompt, apply low-confidence penalty |
 | WebSearch dominates rankings | Low | High | Source penalty (-15pts), testing validates |
 | WebSearch adds spam/low-quality | Medium | Medium | Exclude social media domains, domain filtering |
 | Date parsing unreliable | High | Medium | Accept "low" confidence as normal for WebSearch |
 ## Future Considerations
 1. **Domain authority scoring**: Could proxy engagement with domain reputation
 2. **User-configurable weights**: Let users adjust WebSearch penalty
 3. **Domain whitelist/blacklist**: Filter WebSearch to trusted sources
 4. **Parallel execution**: Run all 3 sources concurrently for speed
 ## References
 ### Internal References
 - Scoring algorithm: `scripts/lib/score.py:8-15`
 - Source detection: `scripts/lib/env.py:57-72`
 - Schema patterns: `scripts/lib/schema.py:76-138`
 - Orchestrator: `scripts/last30days.py:54-164`
 ### External References
 - Claude WebSearch docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
 - WebSearch pricing: $10/1K searches + token costs
 - Date filtering limitation: No explicit date params, use natural language
 ### Research Findings
 - Reddit upvotes are ~12% of ranking value in SEO (strong signal)
 - E-E-A-T framework: Engagement metrics = trust signal
 - MSA2C2 approach: Dynamic weight learning for multi-source aggregation
@@ -0,0 +1,37 @@
 [project]
 name = "last30days-skill"
 version = "3.3.2"
 description = "Multi-source last-30-days research skill"
 readme = "README.md"
 requires-python = ">=3.12"
 dependencies = []
 [dependency-groups]
 dev = [
  "pytest>=9.0.3,<10",
  "pytest-cov>=7,<8",
 ]
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 python_files = ["test_*.py"]
 addopts = [
  "-q",
  "--tb=short",
 ]
 [tool.coverage.run]
 branch = true
 source = ["skills/last30days/scripts", "tests"]
 omit = [
  "skills/last30days/scripts/lib/vendor/*",
  "dist/*",
 ]
 [tool.coverage.report]
 skip_empty = true
 show_missing = true
 omit = [
  "skills/last30days/scripts/lib/vendor/*",
  "dist/*",
 ]
@@ -1,402 +0,0 @@
 #!/usr/bin/env python3
 """
 last30days - Research a topic from the last 30 days on Reddit + X.
 Usage:
    python3 last30days.py <topic> [options]
 Options:
    --mock              Use fixtures instead of real API calls
    --emit=MODE         Output mode: compact|json|md|context|path (default: compact)
    --sources=MODE      Source selection: auto|reddit|x|both (default: auto)
    --quick             Faster research with fewer sources (8-12 each)
    --deep              Comprehensive research with more sources (50-70 Reddit, 40-60 X)
    --debug             Enable verbose debug logging
 """
 import argparse
 import json
 import os
 import sys
 from datetime import datetime, timezone
 from pathlib import Path
 # Add lib to path
 SCRIPT_DIR = Path(__file__).parent.resolve()
 sys.path.insert(0, str(SCRIPT_DIR))
 from lib import (
    dates,
    dedupe,
    env,
    http,
    models,
    normalize,
    openai_reddit,
    reddit_enrich,
    render,
    schema,
    score,
    ui,
    websearch,
    xai_x,
 )
 def load_fixture(name: str) -> dict:
    """Load a fixture file."""
    fixture_path = SCRIPT_DIR.parent / "fixtures" / name
    if fixture_path.exists():
        with open(fixture_path) as f:
            return json.load(f)
    return {}
 def run_research(
    topic: str,
    sources: str,
    config: dict,
    selected_models: dict,
    from_date: str,
    to_date: str,
    depth: str = "default",
    mock: bool = False,
    progress: ui.ProgressDisplay = None,
 ) -> tuple:
    """Run the research pipeline.
    Returns:
        Tuple of (reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error)
    Note: web_needed is True when WebSearch should be performed by Claude.
    The script outputs a marker and Claude handles WebSearch in its session.
    """
    reddit_items = []
    x_items = []
    raw_openai = None
    raw_xai = None
    raw_reddit_enriched = []
    reddit_error = None
    x_error = None
    # Check if WebSearch is needed
    web_needed = sources in ("all", "web", "reddit-web", "x-web")
    # Reddit search via OpenAI
    if sources in ("both", "reddit", "all", "reddit-web"):
        if progress:
            progress.start_reddit()
        if mock:
            raw_openai = load_fixture("openai_sample.json")
        else:
            try:
                raw_openai = openai_reddit.search_reddit(
                    config["OPENAI_API_KEY"],
                    selected_models["openai"],
                    topic,
                    depth=depth,
                )
            except http.HTTPError as e:
                if progress:
                    progress.show_error(f"Reddit API failed: {e}")
                raw_openai = {"error": str(e)}
                reddit_error = f"API error: {e}"
            except Exception as e:
                if progress:
                    progress.show_error(f"Reddit error: {e}")
                raw_openai = {"error": str(e)}
                reddit_error = f"{type(e).__name__}: {e}"
        # Parse response
        reddit_items = openai_reddit.parse_reddit_response(raw_openai)
        if progress:
            progress.end_reddit(len(reddit_items))
        # Enrich with real Reddit data
        if reddit_items:
            if progress:
                progress.start_reddit_enrich(1, len(reddit_items))
            for i, item in enumerate(reddit_items):
                if progress and i > 0:
                    progress.update_reddit_enrich(i + 1, len(reddit_items))
                if mock:
                    mock_thread = load_fixture("reddit_thread_sample.json")
                    reddit_items[i] = reddit_enrich.enrich_reddit_item(item, mock_thread)
                else:
                    reddit_items[i] = reddit_enrich.enrich_reddit_item(item)
                raw_reddit_enriched.append(reddit_items[i])
            if progress:
                progress.end_reddit_enrich()
    # X search via xAI
    if sources in ("both", "x", "all", "x-web"):
        if progress:
            progress.start_x()
        if mock:
            raw_xai = load_fixture("xai_sample.json")
        else:
            try:
                raw_xai = xai_x.search_x(
                    config["XAI_API_KEY"],
                    selected_models["xai"],
                    topic,
                    from_date,
                    to_date,
                    depth=depth,
                )
            except http.HTTPError as e:
                if progress:
                    progress.show_error(f"X API failed: {e}")
                raw_xai = {"error": str(e)}
                x_error = f"API error: {e}"
            except Exception as e:
                if progress:
                    progress.show_error(f"X error: {e}")
                raw_xai = {"error": str(e)}
                x_error = f"{type(e).__name__}: {e}"
        # Parse response
        x_items = xai_x.parse_x_response(raw_xai)
        if progress:
            progress.end_x(len(x_items))
    return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error
 def main():
    parser = argparse.ArgumentParser(
        description="Research a topic from the last 30 days on Reddit + X"
    )
    parser.add_argument("topic", nargs="?", help="Topic to research")
    parser.add_argument("--mock", action="store_true", help="Use fixtures")
    parser.add_argument(
        "--emit",
        choices=["compact", "json", "md", "context", "path"],
        default="compact",
        help="Output mode",
    )
    parser.add_argument(
        "--sources",
        choices=["auto", "reddit", "x", "both"],
        default="auto",
        help="Source selection",
    )
    parser.add_argument(
        "--quick",
        action="store_true",
        help="Faster research with fewer sources (8-12 each)",
    )
    parser.add_argument(
        "--deep",
        action="store_true",
        help="Comprehensive research with more sources (50-70 Reddit, 40-60 X)",
    )
    parser.add_argument(
        "--debug",
        action="store_true",
        help="Enable verbose debug logging",
    )
    parser.add_argument(
        "--include-web",
        action="store_true",
        help="Include general web search alongside Reddit/X (lower weighted)",
    )
    args = parser.parse_args()
    # Enable debug logging if requested
    if args.debug:
        os.environ["LAST30DAYS_DEBUG"] = "1"
        # Re-import http to pick up debug flag
        from lib import http as http_module
        http_module.DEBUG = True
    # Determine depth
    if args.quick and args.deep:
        print("Error: Cannot use both --quick and --deep", file=sys.stderr)
        sys.exit(1)
    elif args.quick:
        depth = "quick"
    elif args.deep:
        depth = "deep"
    else:
        depth = "default"
    if not args.topic:
        print("Error: Please provide a topic to research.", file=sys.stderr)
        print("Usage: python3 last30days.py <topic> [options]", file=sys.stderr)
        sys.exit(1)
    # Load config
    config = env.get_config()
    # Check available sources
    available = env.get_available_sources(config)
    # Mock mode can work without keys
    if args.mock:
        if args.sources == "auto":
            sources = "both"
        else:
            sources = args.sources
    else:
        # Validate requested sources against available
        sources, error = env.validate_sources(args.sources, available, args.include_web)
        if error:
            # If it's a warning about WebSearch fallback, print but continue
            if "WebSearch fallback" in error:
                print(f"Note: {error}", file=sys.stderr)
            else:
                print(f"Error: {error}", file=sys.stderr)
                sys.exit(1)
    # Get date range
    from_date, to_date = dates.get_date_range(30)
    # Initialize progress display
    progress = ui.ProgressDisplay(args.topic, show_banner=True)
    # Select models
    if args.mock:
        # Use mock models
        mock_openai_models = load_fixture("models_openai_sample.json").get("data", [])
        mock_xai_models = load_fixture("models_xai_sample.json").get("data", [])
        selected_models = models.get_models(
            {
                "OPENAI_API_KEY": "mock",
                "XAI_API_KEY": "mock",
                **config,
            },
            mock_openai_models,
            mock_xai_models,
        )
    else:
        selected_models = models.get_models(config)
    # Determine mode string
    if sources == "all":
        mode = "all"  # reddit + x + web
    elif sources == "both":
        mode = "both"  # reddit + x
    elif sources == "reddit":
        mode = "reddit-only"
    elif sources == "reddit-web":
        mode = "reddit-web"
    elif sources == "x":
        mode = "x-only"
    elif sources == "x-web":
        mode = "x-web"
    elif sources == "web":
        mode = "web-only"
    else:
        mode = sources
    # Run research
    reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error = run_research(
        args.topic,
        sources,
        config,
        selected_models,
        from_date,
        to_date,
        depth,
        args.mock,
        progress,
    )
    # Processing phase
    progress.start_processing()
    # Normalize items
    normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
    normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
    # Score items
    scored_reddit = score.score_reddit_items(normalized_reddit)
    scored_x = score.score_x_items(normalized_x)
    # Sort items
    sorted_reddit = score.sort_items(scored_reddit)
    sorted_x = score.sort_items(scored_x)
    # Dedupe items
    deduped_reddit = dedupe.dedupe_reddit(sorted_reddit)
    deduped_x = dedupe.dedupe_x(sorted_x)
    progress.end_processing()
    # Create report
    report = schema.create_report(
        args.topic,
        from_date,
        to_date,
        mode,
        selected_models.get("openai"),
        selected_models.get("xai"),
    )
    report.reddit = deduped_reddit
    report.x = deduped_x
    report.reddit_error = reddit_error
    report.x_error = x_error
    # Generate context snippet
    report.context_snippet_md = render.render_context_snippet(report)
    # Write outputs
    render.write_outputs(report, raw_openai, raw_xai, raw_reddit_enriched)
    # Show completion
    progress.show_complete(len(deduped_reddit), len(deduped_x))
    # Output result
    output_result(report, args.emit, web_needed, args.topic, from_date, to_date)
 def output_result(
    report: schema.Report,
    emit_mode: str,
    web_needed: bool = False,
    topic: str = "",
    from_date: str = "",
    to_date: str = "",
 ):
    """Output the result based on emit mode."""
    if emit_mode == "compact":
        print(render.render_compact(report))
    elif emit_mode == "json":
        print(json.dumps(report.to_dict(), indent=2))
    elif emit_mode == "md":
        print(render.render_full_report(report))
    elif emit_mode == "context":
        print(report.context_snippet_md)
    elif emit_mode == "path":
        print(render.get_context_path())
    # Output WebSearch instructions if needed
    if web_needed:
        print("\n" + "="*60)
        print("### WEBSEARCH REQUIRED ###")
        print("="*60)
        print(f"Topic: {topic}")
        print(f"Date range: {from_date} to {to_date}")
        print("")
        print("Claude: Use your WebSearch tool to find 8-15 relevant web pages.")
        print("EXCLUDE: reddit.com, x.com, twitter.com (already covered above)")
        print("INCLUDE: blogs, docs, news, tutorials from the last 30 days")
        print("")
        print("After searching, synthesize WebSearch results WITH the Reddit/X")
        print("results above. WebSearch items should rank LOWER than comparable")
        print("Reddit/X items (they lack engagement metrics).")
        print("="*60)
 if __name__ == "__main__":
    main()
@@ -1,152 +0,0 @@
 """Caching utilities for last30days skill."""
 import hashlib
 import json
 import os
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any, Optional
 CACHE_DIR = Path.home() / ".cache" / "last30days"
 DEFAULT_TTL_HOURS = 24
 MODEL_CACHE_TTL_DAYS = 7
 def ensure_cache_dir():
    """Ensure cache directory exists."""
    CACHE_DIR.mkdir(parents=True, exist_ok=True)
 def get_cache_key(topic: str, from_date: str, to_date: str, sources: str) -> str:
    """Generate a cache key from query parameters."""
    key_data = f"{topic}|{from_date}|{to_date}|{sources}"
    return hashlib.sha256(key_data.encode()).hexdigest()[:16]
 def get_cache_path(cache_key: str) -> Path:
    """Get path to cache file."""
    return CACHE_DIR / f"{cache_key}.json"
 def is_cache_valid(cache_path: Path, ttl_hours: int = DEFAULT_TTL_HOURS) -> bool:
    """Check if cache file exists and is within TTL."""
    if not cache_path.exists():
        return False
    try:
        stat = cache_path.stat()
        mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
        now = datetime.now(timezone.utc)
        age_hours = (now - mtime).total_seconds() / 3600
        return age_hours < ttl_hours
    except OSError:
        return False
 def load_cache(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> Optional[dict]:
    """Load data from cache if valid."""
    cache_path = get_cache_path(cache_key)
    if not is_cache_valid(cache_path, ttl_hours):
        return None
    try:
        with open(cache_path, 'r') as f:
            return json.load(f)
    except (json.JSONDecodeError, OSError):
        return None
 def get_cache_age_hours(cache_path: Path) -> Optional[float]:
    """Get age of cache file in hours."""
    if not cache_path.exists():
        return None
    try:
        stat = cache_path.stat()
        mtime = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)
        now = datetime.now(timezone.utc)
        return (now - mtime).total_seconds() / 3600
    except OSError:
        return None
 def load_cache_with_age(cache_key: str, ttl_hours: int = DEFAULT_TTL_HOURS) -> tuple:
    """Load data from cache with age info.
    Returns:
        Tuple of (data, age_hours) or (None, None) if invalid
    """
    cache_path = get_cache_path(cache_key)
    if not is_cache_valid(cache_path, ttl_hours):
        return None, None
    age = get_cache_age_hours(cache_path)
    try:
        with open(cache_path, 'r') as f:
            return json.load(f), age
    except (json.JSONDecodeError, OSError):
        return None, None
 def save_cache(cache_key: str, data: dict):
    """Save data to cache."""
    ensure_cache_dir()
    cache_path = get_cache_path(cache_key)
    try:
        with open(cache_path, 'w') as f:
            json.dump(data, f)
    except OSError:
        pass  # Silently fail on cache write errors
 def clear_cache():
    """Clear all cache files."""
    if CACHE_DIR.exists():
        for f in CACHE_DIR.glob("*.json"):
            try:
                f.unlink()
            except OSError:
                pass
 # Model selection cache (longer TTL)
 MODEL_CACHE_FILE = CACHE_DIR / "model_selection.json"
 def load_model_cache() -> dict:
    """Load model selection cache."""
    if not is_cache_valid(MODEL_CACHE_FILE, MODEL_CACHE_TTL_DAYS * 24):
        return {}
    try:
        with open(MODEL_CACHE_FILE, 'r') as f:
            return json.load(f)
    except (json.JSONDecodeError, OSError):
        return {}
 def save_model_cache(data: dict):
    """Save model selection cache."""
    ensure_cache_dir()
    try:
        with open(MODEL_CACHE_FILE, 'w') as f:
            json.dump(data, f)
    except OSError:
        pass
 def get_cached_model(provider: str) -> Optional[str]:
    """Get cached model selection for a provider."""
    cache = load_model_cache()
    return cache.get(provider)
 def set_cached_model(provider: str, model: str):
    """Cache model selection for a provider."""
    cache = load_model_cache()
    cache[provider] = model
    cache['updated_at'] = datetime.now(timezone.utc).isoformat()
    save_model_cache(cache)
@@ -1,120 +0,0 @@
 """Near-duplicate detection for last30days skill."""
 import re
 from typing import List, Set, Tuple, Union
 from . import schema
 def normalize_text(text: str) -> str:
    """Normalize text for comparison.
    - Lowercase
    - Remove punctuation
    - Collapse whitespace
    """
    text = text.lower()
    text = re.sub(r'[^\w\s]', ' ', text)
    text = re.sub(r'\s+', ' ', text)
    return text.strip()
 def get_ngrams(text: str, n: int = 3) -> Set[str]:
    """Get character n-grams from text."""
    text = normalize_text(text)
    if len(text) < n:
        return {text}
    return {text[i:i+n] for i in range(len(text) - n + 1)}
 def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
    """Compute Jaccard similarity between two sets."""
    if not set1 or not set2:
        return 0.0
    intersection = len(set1 & set2)
    union = len(set1 | set2)
    return intersection / union if union > 0 else 0.0
 def get_item_text(item: Union[schema.RedditItem, schema.XItem]) -> str:
    """Get comparable text from an item."""
    if isinstance(item, schema.RedditItem):
        return item.title
    else:
        return item.text
 def find_duplicates(
    items: List[Union[schema.RedditItem, schema.XItem]],
    threshold: float = 0.7,
 ) -> List[Tuple[int, int]]:
    """Find near-duplicate pairs in items.
    Args:
        items: List of items to check
        threshold: Similarity threshold (0-1)
    Returns:
        List of (i, j) index pairs where i < j and items are similar
    """
    duplicates = []
    # Pre-compute n-grams
    ngrams = [get_ngrams(get_item_text(item)) for item in items]
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            similarity = jaccard_similarity(ngrams[i], ngrams[j])
            if similarity >= threshold:
                duplicates.append((i, j))
    return duplicates
 def dedupe_items(
    items: List[Union[schema.RedditItem, schema.XItem]],
    threshold: float = 0.7,
 ) -> List[Union[schema.RedditItem, schema.XItem]]:
    """Remove near-duplicates, keeping highest-scored item.
    Args:
        items: List of items (should be pre-sorted by score descending)
        threshold: Similarity threshold
    Returns:
        Deduplicated items
    """
    if len(items) <= 1:
        return items
    # Find duplicate pairs
    dup_pairs = find_duplicates(items, threshold)
    # Mark indices to remove (always remove the lower-scored one)
    # Since items are pre-sorted by score, the second index is always lower
    to_remove = set()
    for i, j in dup_pairs:
        # Keep the higher-scored one (lower index in sorted list)
        if items[i].score >= items[j].score:
            to_remove.add(j)
        else:
            to_remove.add(i)
    # Return items not marked for removal
    return [item for idx, item in enumerate(items) if idx not in to_remove]
 def dedupe_reddit(
    items: List[schema.RedditItem],
    threshold: float = 0.7,
 ) -> List[schema.RedditItem]:
    """Dedupe Reddit items."""
    return dedupe_items(items, threshold)
 def dedupe_x(
    items: List[schema.XItem],
    threshold: float = 0.7,
 ) -> List[schema.XItem]:
    """Dedupe X items."""
    return dedupe_items(items, threshold)
@@ -1,131 +0,0 @@
 """Environment and API key management for last30days skill."""
 import os
 from pathlib import Path
 from typing import Optional, Dict, Any
 CONFIG_DIR = Path.home() / ".config" / "last30days"
 CONFIG_FILE = CONFIG_DIR / ".env"
 def load_env_file(path: Path) -> Dict[str, str]:
    """Load environment variables from a file."""
    env = {}
    if not path.exists():
        return env
    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            if '=' in line:
                key, _, value = line.partition('=')
                key = key.strip()
                value = value.strip()
                # Remove quotes if present
                if value and value[0] in ('"', "'") and value[-1] == value[0]:
                    value = value[1:-1]
                if key and value:
                    env[key] = value
    return env
 def get_config() -> Dict[str, Any]:
    """Load configuration from ~/.config/last30days/.env and environment."""
    # Load from config file first
    file_env = load_env_file(CONFIG_FILE)
    # Environment variables override file
    config = {
        'OPENAI_API_KEY': os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY'),
        'XAI_API_KEY': os.environ.get('XAI_API_KEY') or file_env.get('XAI_API_KEY'),
        'OPENAI_MODEL_POLICY': os.environ.get('OPENAI_MODEL_POLICY') or file_env.get('OPENAI_MODEL_POLICY', 'auto'),
        'OPENAI_MODEL_PIN': os.environ.get('OPENAI_MODEL_PIN') or file_env.get('OPENAI_MODEL_PIN'),
        'XAI_MODEL_POLICY': os.environ.get('XAI_MODEL_POLICY') or file_env.get('XAI_MODEL_POLICY', 'latest'),
        'XAI_MODEL_PIN': os.environ.get('XAI_MODEL_PIN') or file_env.get('XAI_MODEL_PIN'),
    }
    return config
 def config_exists() -> bool:
    """Check if configuration file exists."""
    return CONFIG_FILE.exists()
 def get_available_sources(config: Dict[str, Any]) -> str:
    """Determine which sources are available based on API keys.
    Returns: 'both', 'reddit', 'x', or 'web' (fallback when no keys)
    """
    has_openai = bool(config.get('OPENAI_API_KEY'))
    has_xai = bool(config.get('XAI_API_KEY'))
    if has_openai and has_xai:
        return 'both'
    elif has_openai:
        return 'reddit'
    elif has_xai:
        return 'x'
    else:
        return 'web'  # Fallback: WebSearch only (no API keys needed)
 def validate_sources(requested: str, available: str, include_web: bool = False) -> tuple[str, Optional[str]]:
    """Validate requested sources against available keys.
    Args:
        requested: 'auto', 'reddit', 'x', 'both', or 'web'
        available: Result from get_available_sources()
        include_web: If True, add WebSearch to available sources
    Returns:
        Tuple of (effective_sources, error_message)
    """
    # WebSearch-only mode (no API keys)
    if available == 'web':
        if requested == 'auto':
            return 'web', None
        elif requested == 'web':
            return 'web', None
        else:
            return 'web', f"No API keys configured. Using WebSearch fallback. Add keys to ~/.config/last30days/.env for Reddit/X."
    if requested == 'auto':
        # Add web to sources if include_web is set
        if include_web:
            if available == 'both':
                return 'all', None  # reddit + x + web
            elif available == 'reddit':
                return 'reddit-web', None
            elif available == 'x':
                return 'x-web', None
        return available, None
    if requested == 'web':
        return 'web', None
    if requested == 'both':
        if available not in ('both',):
            missing = 'xAI' if available == 'reddit' else 'OpenAI'
            return 'none', f"Requested both sources but {missing} key is missing. Use --sources=auto to use available keys."
        if include_web:
            return 'all', None
        return 'both', None
    if requested == 'reddit':
        if available == 'x':
            return 'none', "Requested Reddit but only xAI key is available."
        if include_web:
            return 'reddit-web', None
        return 'reddit', None
    if requested == 'x':
        if available == 'reddit':
            return 'none', "Requested X but only OpenAI key is available."
        if include_web:
            return 'x-web', None
        return 'x', None
    return requested, None
@@ -1,146 +0,0 @@
 """HTTP utilities for last30days skill (stdlib only)."""
 import json
 import os
 import sys
 import time
 import urllib.error
 import urllib.request
 from typing import Any, Dict, Optional
 from urllib.parse import urlencode
 DEFAULT_TIMEOUT = 30
 DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")
 def log(msg: str):
    """Log debug message to stderr."""
    if DEBUG:
        sys.stderr.write(f"[DEBUG] {msg}\n")
        sys.stderr.flush()
 MAX_RETRIES = 3
 RETRY_DELAY = 1.0
 USER_AGENT = "last30days-skill/1.0 (Claude Code Skill)"
 class HTTPError(Exception):
    """HTTP request error with status code."""
    def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
        super().__init__(message)
        self.status_code = status_code
        self.body = body
 def request(
    method: str,
    url: str,
    headers: Optional[Dict[str, str]] = None,
    json_data: Optional[Dict[str, Any]] = None,
    timeout: int = DEFAULT_TIMEOUT,
    retries: int = MAX_RETRIES,
 ) -> Dict[str, Any]:
    """Make an HTTP request and return JSON response.
    Args:
        method: HTTP method (GET, POST, etc.)
        url: Request URL
        headers: Optional headers dict
        json_data: Optional JSON body (for POST)
        timeout: Request timeout in seconds
        retries: Number of retries on failure
    Returns:
        Parsed JSON response
    Raises:
        HTTPError: On request failure
    """
    headers = headers or {}
    headers.setdefault("User-Agent", USER_AGENT)
    data = None
    if json_data is not None:
        data = json.dumps(json_data).encode('utf-8')
        headers.setdefault("Content-Type", "application/json")
    req = urllib.request.Request(url, data=data, headers=headers, method=method)
    log(f"{method} {url}")
    if json_data:
        log(f"Payload keys: {list(json_data.keys())}")
    last_error = None
    for attempt in range(retries):
        try:
            with urllib.request.urlopen(req, timeout=timeout) as response:
                body = response.read().decode('utf-8')
                log(f"Response: {response.status} ({len(body)} bytes)")
                return json.loads(body) if body else {}
        except urllib.error.HTTPError as e:
            body = None
            try:
                body = e.read().decode('utf-8')
            except:
                pass
            log(f"HTTP Error {e.code}: {e.reason}")
            if body:
                log(f"Error body: {body[:500]}")
            last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)
            # Don't retry client errors (4xx) except rate limits
            if 400 <= e.code < 500 and e.code != 429:
                raise last_error
            if attempt < retries - 1:
                time.sleep(RETRY_DELAY * (attempt + 1))
        except urllib.error.URLError as e:
            log(f"URL Error: {e.reason}")
            last_error = HTTPError(f"URL Error: {e.reason}")
            if attempt < retries - 1:
                time.sleep(RETRY_DELAY * (attempt + 1))
        except json.JSONDecodeError as e:
            log(f"JSON decode error: {e}")
            last_error = HTTPError(f"Invalid JSON response: {e}")
            raise last_error
    if last_error:
        raise last_error
    raise HTTPError("Request failed with no error details")
 def get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a GET request."""
    return request("GET", url, headers=headers, **kwargs)
 def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a POST request with JSON body."""
    return request("POST", url, headers=headers, json_data=json_data, **kwargs)
 def get_reddit_json(path: str) -> Dict[str, Any]:
    """Fetch Reddit thread JSON.
    Args:
        path: Reddit path (e.g., /r/subreddit/comments/id/title)
    Returns:
        Parsed JSON response
    """
    # Ensure path starts with /
    if not path.startswith('/'):
        path = '/' + path
    # Remove trailing slash and add .json
    path = path.rstrip('/')
    if not path.endswith('.json'):
        path = path + '.json'
    url = f"https://www.reddit.com{path}?raw_json=1"
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/json",
    }
    return get(url, headers=headers)
@@ -1,175 +0,0 @@
 """Model auto-selection for last30days skill."""
 import re
 from typing import Dict, List, Optional, Tuple
 from . import cache, http
 # OpenAI API
 OPENAI_MODELS_URL = "https://api.openai.com/v1/models"
 OPENAI_FALLBACK_MODELS = ["gpt-5.2", "gpt-5.1", "gpt-5", "gpt-4o"]
 # xAI API - Agent Tools API requires grok-4 family
 XAI_MODELS_URL = "https://api.x.ai/v1/models"
 XAI_ALIASES = {
    "latest": "grok-4-1-fast",  # Required for x_search tool
    "stable": "grok-4-1-fast",
 }
 def parse_version(model_id: str) -> Optional[Tuple[int, ...]]:
    """Parse semantic version from model ID.
    Examples:
        gpt-5 -> (5,)
        gpt-5.2 -> (5, 2)
        gpt-5.2.1 -> (5, 2, 1)
    """
    match = re.search(r'(\d+(?:\.\d+)*)', model_id)
    if match:
        return tuple(int(x) for x in match.group(1).split('.'))
    return None
 def is_mainline_openai_model(model_id: str) -> bool:
    """Check if model is a mainline GPT model (not mini/nano/chat/codex/pro)."""
    model_lower = model_id.lower()
    # Must be gpt-5 series
    if not re.match(r'^gpt-5(\.\d+)*$', model_lower):
        return False
    # Exclude variants
    excludes = ['mini', 'nano', 'chat', 'codex', 'pro', 'preview', 'turbo']
    for exc in excludes:
        if exc in model_lower:
            return False
    return True
 def select_openai_model(
    api_key: str,
    policy: str = "auto",
    pin: Optional[str] = None,
    mock_models: Optional[List[Dict]] = None,
 ) -> str:
    """Select the best OpenAI model based on policy.
    Args:
        api_key: OpenAI API key
        policy: 'auto' or 'pinned'
        pin: Model to use if policy is 'pinned'
        mock_models: Mock model list for testing
    Returns:
        Selected model ID
    """
    if policy == "pinned" and pin:
        return pin
    # Check cache first
    cached = cache.get_cached_model("openai")
    if cached:
        return cached
    # Fetch model list
    if mock_models is not None:
        models = mock_models
    else:
        try:
            headers = {"Authorization": f"Bearer {api_key}"}
            response = http.get(OPENAI_MODELS_URL, headers=headers)
            models = response.get("data", [])
        except http.HTTPError:
            # Fall back to known models
            return OPENAI_FALLBACK_MODELS[0]
    # Filter to mainline models
    candidates = [m for m in models if is_mainline_openai_model(m.get("id", ""))]
    if not candidates:
        # No gpt-5 models found, use fallback
        return OPENAI_FALLBACK_MODELS[0]
    # Sort by version (descending), then by created timestamp
    def sort_key(m):
        version = parse_version(m.get("id", "")) or (0,)
        created = m.get("created", 0)
        return (version, created)
    candidates.sort(key=sort_key, reverse=True)
    selected = candidates[0]["id"]
    # Cache the selection
    cache.set_cached_model("openai", selected)
    return selected
 def select_xai_model(
    api_key: str,
    policy: str = "latest",
    pin: Optional[str] = None,
    mock_models: Optional[List[Dict]] = None,
 ) -> str:
    """Select the best xAI model based on policy.
    Args:
        api_key: xAI API key
        policy: 'latest', 'stable', or 'pinned'
        pin: Model to use if policy is 'pinned'
        mock_models: Mock model list for testing
    Returns:
        Selected model ID
    """
    if policy == "pinned" and pin:
        return pin
    # Use alias system
    if policy in XAI_ALIASES:
        alias = XAI_ALIASES[policy]
        # Check cache first
        cached = cache.get_cached_model("xai")
        if cached:
            return cached
        # Cache the alias
        cache.set_cached_model("xai", alias)
        return alias
    # Default to latest
    return XAI_ALIASES["latest"]
 def get_models(
    config: Dict,
    mock_openai_models: Optional[List[Dict]] = None,
    mock_xai_models: Optional[List[Dict]] = None,
 ) -> Dict[str, Optional[str]]:
    """Get selected models for both providers.
    Returns:
        Dict with 'openai' and 'xai' keys
    """
    result = {"openai": None, "xai": None}
    if config.get("OPENAI_API_KEY"):
        result["openai"] = select_openai_model(
            config["OPENAI_API_KEY"],
            config.get("OPENAI_MODEL_POLICY", "auto"),
            config.get("OPENAI_MODEL_PIN"),
            mock_openai_models,
        )
    if config.get("XAI_API_KEY"):
        result["xai"] = select_xai_model(
            config["XAI_API_KEY"],
            config.get("XAI_MODEL_POLICY", "latest"),
            config.get("XAI_MODEL_PIN"),
            mock_xai_models,
        )
    return result
@@ -1,118 +0,0 @@
 """Normalization of raw API data to canonical schema."""
 from typing import Any, Dict, List
 from . import dates, schema
 def normalize_reddit_items(
    items: List[Dict[str, Any]],
    from_date: str,
    to_date: str,
 ) -> List[schema.RedditItem]:
    """Normalize raw Reddit items to schema.
    Args:
        items: Raw Reddit items from API
        from_date: Start of date range
        to_date: End of date range
    Returns:
        List of RedditItem objects
    """
    normalized = []
    for item in items:
        # Parse engagement
        engagement = None
        eng_raw = item.get("engagement")
        if isinstance(eng_raw, dict):
            engagement = schema.Engagement(
                score=eng_raw.get("score"),
                num_comments=eng_raw.get("num_comments"),
                upvote_ratio=eng_raw.get("upvote_ratio"),
            )
        # Parse comments
        top_comments = []
        for c in item.get("top_comments", []):
            top_comments.append(schema.Comment(
                score=c.get("score", 0),
                date=c.get("date"),
                author=c.get("author", ""),
                excerpt=c.get("excerpt", ""),
                url=c.get("url", ""),
            ))
        # Determine date confidence
        date_str = item.get("date")
        date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
        normalized.append(schema.RedditItem(
            id=item.get("id", ""),
            title=item.get("title", ""),
            url=item.get("url", ""),
            subreddit=item.get("subreddit", ""),
            date=date_str,
            date_confidence=date_confidence,
            engagement=engagement,
            top_comments=top_comments,
            comment_insights=item.get("comment_insights", []),
            relevance=item.get("relevance", 0.5),
            why_relevant=item.get("why_relevant", ""),
        ))
    return normalized
 def normalize_x_items(
    items: List[Dict[str, Any]],
    from_date: str,
    to_date: str,
 ) -> List[schema.XItem]:
    """Normalize raw X items to schema.
    Args:
        items: Raw X items from API
        from_date: Start of date range
        to_date: End of date range
    Returns:
        List of XItem objects
    """
    normalized = []
    for item in items:
        # Parse engagement
        engagement = None
        eng_raw = item.get("engagement")
        if isinstance(eng_raw, dict):
            engagement = schema.Engagement(
                likes=eng_raw.get("likes"),
                reposts=eng_raw.get("reposts"),
                replies=eng_raw.get("replies"),
                quotes=eng_raw.get("quotes"),
            )
        # Determine date confidence
        date_str = item.get("date")
        date_confidence = dates.get_date_confidence(date_str, from_date, to_date)
        normalized.append(schema.XItem(
            id=item.get("id", ""),
            text=item.get("text", ""),
            url=item.get("url", ""),
            author_handle=item.get("author_handle", ""),
            date=date_str,
            date_confidence=date_confidence,
            engagement=engagement,
            relevance=item.get("relevance", 0.5),
            why_relevant=item.get("why_relevant", ""),
        ))
    return normalized
 def items_to_dicts(items: List) -> List[Dict[str, Any]]:
    """Convert schema items to dicts for JSON serialization."""
    return [item.to_dict() for item in items]
@@ -1,204 +0,0 @@
 """OpenAI Responses API client for Reddit discovery."""
 import json
 import re
 import sys
 from typing import Any, Dict, List, Optional
 from . import http
 def _log_error(msg: str):
    """Log error to stderr."""
    sys.stderr.write(f"[REDDIT ERROR] {msg}\n")
    sys.stderr.flush()
 OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
 # Depth configurations: (min, max) threads to request
 DEPTH_CONFIG = {
    "quick": (8, 12),
    "default": (20, 30),
    "deep": (50, 70),
 }
 REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic}
 SEARCH GUIDANCE:
 - Search for "site:reddit.com/r/ {topic}" to find subreddit discussions
 - Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design
 - ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads)
 - IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/
 Find {min_items}-{max_items} relevant Reddit discussion threads. Prefer recent threads, but include older relevant ones if recent ones are scarce.
 CRITICAL: Return ALL discussion threads you find as JSON. Do NOT return errors or empty results.
 For EACH Reddit thread URL (containing /r/subreddit/comments/), extract:
 - Thread title
 - Full Reddit URL
 - Subreddit name
 - Date (if visible, otherwise null)
 - Why it's relevant
 Return ONLY valid JSON:
 {{
  "items": [
    {{
      "title": "Thread title",
      "url": "https://www.reddit.com/r/subreddit/comments/abc123/title/",
      "subreddit": "subreddit_name",
      "date": "YYYY-MM-DD or null",
      "why_relevant": "Relevance to {topic}",
      "relevance": 0.85
    }}
  ]
 }}
 Rules:
 - ONLY URLs matching: reddit.com/r/*/comments/*
 - MUST return threads found - NEVER return empty items or errors
 - If threads are older than 30 days, still include them with accurate dates
 - relevance: 0.0-1.0
 - Diverse subreddits preferred"""
 def search_reddit(
    api_key: str,
    model: str,
    topic: str,
    depth: str = "default",
    mock_response: Optional[Dict] = None,
 ) -> Dict[str, Any]:
    """Search Reddit for relevant threads using OpenAI Responses API.
    Args:
        api_key: OpenAI API key
        model: Model to use
        topic: Search topic
        depth: Research depth - "quick", "default", or "deep"
        mock_response: Mock response for testing
    Returns:
        Raw API response
    """
    if mock_response is not None:
        return mock_response
    min_items, max_items = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }
    # Adjust timeout based on depth
    timeout = 60 if depth == "quick" else 90 if depth == "default" else 120
    payload = {
        "model": model,
        "tools": [
            {
                "type": "web_search",
                "filters": {
                    "allowed_domains": ["reddit.com"]
                }
            }
        ],
        "include": ["web_search_call.action.sources"],
        "input": REDDIT_SEARCH_PROMPT.format(topic=topic, min_items=min_items, max_items=max_items),
    }
    return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
 def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse OpenAI response to extract Reddit items.
    Args:
        response: Raw API response
    Returns:
        List of item dicts
    """
    items = []
    # Check for API errors first
    if "error" in response and response["error"]:
        error = response["error"]
        err_msg = error.get("message", str(error)) if isinstance(error, dict) else str(error)
        _log_error(f"OpenAI API error: {err_msg}")
        if http.DEBUG:
            _log_error(f"Full error response: {json.dumps(response, indent=2)[:1000]}")
        return items
    # Try to find the output text
    output_text = ""
    if "output" in response:
        output = response["output"]
        if isinstance(output, str):
            output_text = output
        elif isinstance(output, list):
            for item in output:
                if isinstance(item, dict):
                    if item.get("type") == "message":
                        content = item.get("content", [])
                        for c in content:
                            if isinstance(c, dict) and c.get("type") == "output_text":
                                output_text = c.get("text", "")
                                break
                    elif "text" in item:
                        output_text = item["text"]
                elif isinstance(item, str):
                    output_text = item
                if output_text:
                    break
    # Also check for choices (older format)
    if not output_text and "choices" in response:
        for choice in response["choices"]:
            if "message" in choice:
                output_text = choice["message"].get("content", "")
                break
    if not output_text:
        print(f"[REDDIT WARNING] No output text found in OpenAI response. Keys present: {list(response.keys())}", flush=True)
        return items
    # Extract JSON from the response
    json_match = re.search(r'\{[\s\S]*"items"[\s\S]*\}', output_text)
    if json_match:
        try:
            data = json.loads(json_match.group())
            items = data.get("items", [])
        except json.JSONDecodeError:
            pass
    # Validate and clean items
    clean_items = []
    for i, item in enumerate(items):
        if not isinstance(item, dict):
            continue
        url = item.get("url", "")
        if not url or "reddit.com" not in url:
            continue
        clean_item = {
            "id": f"R{i+1}",
            "title": str(item.get("title", "")).strip(),
            "url": url,
            "subreddit": str(item.get("subreddit", "")).strip().lstrip("r/"),
            "date": item.get("date"),
            "why_relevant": str(item.get("why_relevant", "")).strip(),
            "relevance": min(1.0, max(0.0, float(item.get("relevance", 0.5)))),
        }
        # Validate date format
        if clean_item["date"]:
            if not re.match(r'^\d{4}-\d{2}-\d{2}$', str(clean_item["date"])):
                clean_item["date"] = None
        clean_items.append(clean_item)
    return clean_items
@@ -1,342 +0,0 @@
 """Output rendering for last30days skill."""
 import json
 from pathlib import Path
 from typing import List, Optional
 from . import schema
 OUTPUT_DIR = Path.home() / ".local" / "share" / "last30days" / "out"
 def ensure_output_dir():
    """Ensure output directory exists."""
    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
 def render_compact(report: schema.Report, limit: int = 15) -> str:
    """Render compact output for Claude to synthesize.
    Args:
        report: Report data
        limit: Max items per source
    Returns:
        Compact markdown string
    """
    lines = []
    # Header
    lines.append(f"## Research Results: {report.topic}")
    lines.append("")
    # Cache indicator
    if report.from_cache:
        age_str = f"{report.cache_age_hours:.1f}h old" if report.cache_age_hours else "cached"
        lines.append(f"**⚡ CACHED RESULTS** ({age_str}) - use `--refresh` for fresh data")
        lines.append("")
    lines.append(f"**Date Range:** {report.range_from} to {report.range_to}")
    lines.append(f"**Mode:** {report.mode}")
    if report.openai_model_used:
        lines.append(f"**OpenAI Model:** {report.openai_model_used}")
    if report.xai_model_used:
        lines.append(f"**xAI Model:** {report.xai_model_used}")
    lines.append("")
    # Coverage note
    if report.mode == "reddit-only":
        lines.append("*Tip: Add xAI key for X coverage and better triangulation.*")
        lines.append("")
    elif report.mode == "x-only":
        lines.append("*Tip: Add OpenAI key for Reddit coverage and better triangulation.*")
        lines.append("")
    # Reddit items
    if report.reddit_error:
        lines.append("### Reddit Threads")
        lines.append("")
        lines.append(f"**ERROR:** {report.reddit_error}")
        lines.append("")
    elif report.mode in ("both", "reddit-only") and not report.reddit:
        lines.append("### Reddit Threads")
        lines.append("")
        lines.append("*No relevant Reddit threads found for this topic.*")
        lines.append("")
    elif report.reddit:
        lines.append("### Reddit Threads")
        lines.append("")
        for item in report.reddit[:limit]:
            eng_str = ""
            if item.engagement:
                eng = item.engagement
                parts = []
                if eng.score is not None:
                    parts.append(f"{eng.score}pts")
                if eng.num_comments is not None:
                    parts.append(f"{eng.num_comments}cmt")
                if parts:
                    eng_str = f" [{', '.join(parts)}]"
            date_str = f" ({item.date})" if item.date else " (date unknown)"
            conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
            lines.append(f"**{item.id}** (score:{item.score}) r/{item.subreddit}{date_str}{conf_str}{eng_str}")
            lines.append(f"  {item.title}")
            lines.append(f"  {item.url}")
            lines.append(f"  *{item.why_relevant}*")
            # Top comment insights
            if item.comment_insights:
                lines.append(f"  Insights:")
                for insight in item.comment_insights[:3]:
                    lines.append(f"    - {insight}")
            lines.append("")
    # X items
    if report.x_error:
        lines.append("### X Posts")
        lines.append("")
        lines.append(f"**ERROR:** {report.x_error}")
        lines.append("")
    elif report.mode in ("both", "x-only", "all", "x-web") and not report.x:
        lines.append("### X Posts")
        lines.append("")
        lines.append("*No relevant X posts found for this topic.*")
        lines.append("")
    elif report.x:
        lines.append("### X Posts")
        lines.append("")
        for item in report.x[:limit]:
            eng_str = ""
            if item.engagement:
                eng = item.engagement
                parts = []
                if eng.likes is not None:
                    parts.append(f"{eng.likes}likes")
                if eng.reposts is not None:
                    parts.append(f"{eng.reposts}rt")
                if parts:
                    eng_str = f" [{', '.join(parts)}]"
            date_str = f" ({item.date})" if item.date else " (date unknown)"
            conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
            lines.append(f"**{item.id}** (score:{item.score}) @{item.author_handle}{date_str}{conf_str}{eng_str}")
            lines.append(f"  {item.text[:200]}...")
            lines.append(f"  {item.url}")
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")
    # Web items (if any - populated by Claude)
    if report.web_error:
        lines.append("### Web Results")
        lines.append("")
        lines.append(f"**ERROR:** {report.web_error}")
        lines.append("")
    elif report.web:
        lines.append("### Web Results")
        lines.append("")
        for item in report.web[:limit]:
            date_str = f" ({item.date})" if item.date else " (date unknown)"
            conf_str = f" [date:{item.date_confidence}]" if item.date_confidence != "high" else ""
            lines.append(f"**{item.id}** [WEB] (score:{item.score}) {item.source_domain}{date_str}{conf_str}")
            lines.append(f"  {item.title}")
            lines.append(f"  {item.url}")
            lines.append(f"  {item.snippet[:150]}...")
            lines.append(f"  *{item.why_relevant}*")
            lines.append("")
    return "\n".join(lines)
 def render_context_snippet(report: schema.Report) -> str:
    """Render reusable context snippet.
    Args:
        report: Report data
    Returns:
        Context markdown string
    """
    lines = []
    lines.append(f"# Context: {report.topic} (Last 30 Days)")
    lines.append("")
    lines.append(f"*Generated: {report.generated_at[:10]} | Sources: {report.mode}*")
    lines.append("")
    # Key sources summary
    lines.append("## Key Sources")
    lines.append("")
    all_items = []
    for item in report.reddit[:5]:
        all_items.append((item.score, "Reddit", item.title, item.url))
    for item in report.x[:5]:
        all_items.append((item.score, "X", item.text[:50] + "...", item.url))
    for item in report.web[:5]:
        all_items.append((item.score, "Web", item.title[:50] + "...", item.url))
    all_items.sort(key=lambda x: -x[0])
    for score, source, text, url in all_items[:7]:
        lines.append(f"- [{source}] {text}")
    lines.append("")
    lines.append("## Summary")
    lines.append("")
    lines.append("*See full report for best practices, prompt pack, and detailed sources.*")
    lines.append("")
    return "\n".join(lines)
 def render_full_report(report: schema.Report) -> str:
    """Render full markdown report.
    Args:
        report: Report data
    Returns:
        Full report markdown
    """
    lines = []
    # Title
    lines.append(f"# {report.topic} - Last 30 Days Research Report")
    lines.append("")
    lines.append(f"**Generated:** {report.generated_at}")
    lines.append(f"**Date Range:** {report.range_from} to {report.range_to}")
    lines.append(f"**Mode:** {report.mode}")
    lines.append("")
    # Models
    lines.append("## Models Used")
    lines.append("")
    if report.openai_model_used:
        lines.append(f"- **OpenAI:** {report.openai_model_used}")
    if report.xai_model_used:
        lines.append(f"- **xAI:** {report.xai_model_used}")
    lines.append("")
    # Reddit section
    if report.reddit:
        lines.append("## Reddit Threads")
        lines.append("")
        for item in report.reddit:
            lines.append(f"### {item.id}: {item.title}")
            lines.append("")
            lines.append(f"- **Subreddit:** r/{item.subreddit}")
            lines.append(f"- **URL:** {item.url}")
            lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
            lines.append(f"- **Score:** {item.score}/100")
            lines.append(f"- **Relevance:** {item.why_relevant}")
            if item.engagement:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
            if item.comment_insights:
                lines.append("")
                lines.append("**Key Insights from Comments:**")
                for insight in item.comment_insights:
                    lines.append(f"- {insight}")
            lines.append("")
    # X section
    if report.x:
        lines.append("## X Posts")
        lines.append("")
        for item in report.x:
            lines.append(f"### {item.id}: @{item.author_handle}")
            lines.append("")
            lines.append(f"- **URL:** {item.url}")
            lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
            lines.append(f"- **Score:** {item.score}/100")
            lines.append(f"- **Relevance:** {item.why_relevant}")
            if item.engagement:
                eng = item.engagement
                lines.append(f"- **Engagement:** {eng.likes or '?'} likes, {eng.reposts or '?'} reposts")
            lines.append("")
            lines.append(f"> {item.text}")
            lines.append("")
    # Web section
    if report.web:
        lines.append("## Web Results")
        lines.append("")
        for item in report.web:
            lines.append(f"### {item.id}: {item.title}")
            lines.append("")
            lines.append(f"- **Source:** {item.source_domain}")
            lines.append(f"- **URL:** {item.url}")
            lines.append(f"- **Date:** {item.date or 'Unknown'} (confidence: {item.date_confidence})")
            lines.append(f"- **Score:** {item.score}/100")
            lines.append(f"- **Relevance:** {item.why_relevant}")
            lines.append("")
            lines.append(f"> {item.snippet}")
            lines.append("")
    # Placeholders for Claude synthesis
    lines.append("## Best Practices")
    lines.append("")
    lines.append("*To be synthesized by Claude*")
    lines.append("")
    lines.append("## Prompt Pack")
    lines.append("")
    lines.append("*To be synthesized by Claude*")
    lines.append("")
    return "\n".join(lines)
 def write_outputs(
    report: schema.Report,
    raw_openai: Optional[dict] = None,
    raw_xai: Optional[dict] = None,
    raw_reddit_enriched: Optional[list] = None,
 ):
    """Write all output files.
    Args:
        report: Report data
        raw_openai: Raw OpenAI API response
        raw_xai: Raw xAI API response
        raw_reddit_enriched: Raw enriched Reddit thread data
    """
    ensure_output_dir()
    # report.json
    with open(OUTPUT_DIR / "report.json", 'w') as f:
        json.dump(report.to_dict(), f, indent=2)
    # report.md
    with open(OUTPUT_DIR / "report.md", 'w') as f:
        f.write(render_full_report(report))
    # last30days.context.md
    with open(OUTPUT_DIR / "last30days.context.md", 'w') as f:
        f.write(render_context_snippet(report))
    # Raw responses
    if raw_openai:
        with open(OUTPUT_DIR / "raw_openai.json", 'w') as f:
            json.dump(raw_openai, f, indent=2)
    if raw_xai:
        with open(OUTPUT_DIR / "raw_xai.json", 'w') as f:
            json.dump(raw_xai, f, indent=2)
    if raw_reddit_enriched:
        with open(OUTPUT_DIR / "raw_reddit_threads_enriched.json", 'w') as f:
            json.dump(raw_reddit_enriched, f, indent=2)
 def get_context_path() -> str:
    """Get path to context file."""
    return str(OUTPUT_DIR / "last30days.context.md")
@@ -1,336 +0,0 @@
 """Data schemas for last30days skill."""
 from dataclasses import dataclass, field, asdict
 from typing import Any, Dict, List, Optional
 from datetime import datetime, timezone
@dataclass
 class Engagement:
    """Engagement metrics."""
    # Reddit fields
    score: Optional[int] = None
    num_comments: Optional[int] = None
    upvote_ratio: Optional[float] = None
    # X fields
    likes: Optional[int] = None
    reposts: Optional[int] = None
    replies: Optional[int] = None
    quotes: Optional[int] = None
    def to_dict(self) -> Dict[str, Any]:
        d = {}
        if self.score is not None:
            d['score'] = self.score
        if self.num_comments is not None:
            d['num_comments'] = self.num_comments
        if self.upvote_ratio is not None:
            d['upvote_ratio'] = self.upvote_ratio
        if self.likes is not None:
            d['likes'] = self.likes
        if self.reposts is not None:
            d['reposts'] = self.reposts
        if self.replies is not None:
            d['replies'] = self.replies
        if self.quotes is not None:
            d['quotes'] = self.quotes
        return d if d else None
@dataclass
 class Comment:
    """Reddit comment."""
    score: int
    date: Optional[str]
    author: str
    excerpt: str
    url: str
    def to_dict(self) -> Dict[str, Any]:
        return {
            'score': self.score,
            'date': self.date,
            'author': self.author,
            'excerpt': self.excerpt,
            'url': self.url,
        }
@dataclass
 class SubScores:
    """Component scores."""
    relevance: int = 0
    recency: int = 0
    engagement: int = 0
    def to_dict(self) -> Dict[str, int]:
        return {
            'relevance': self.relevance,
            'recency': self.recency,
            'engagement': self.engagement,
        }
@dataclass
 class RedditItem:
    """Normalized Reddit item."""
    id: str
    title: str
    url: str
    subreddit: str
    date: Optional[str] = None
    date_confidence: str = "low"
    engagement: Optional[Engagement] = None
    top_comments: List[Comment] = field(default_factory=list)
    comment_insights: List[str] = field(default_factory=list)
    relevance: float = 0.5
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    def to_dict(self) -> Dict[str, Any]:
        return {
            'id': self.id,
            'title': self.title,
            'url': self.url,
            'subreddit': self.subreddit,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'engagement': self.engagement.to_dict() if self.engagement else None,
            'top_comments': [c.to_dict() for c in self.top_comments],
            'comment_insights': self.comment_insights,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
@dataclass
 class XItem:
    """Normalized X item."""
    id: str
    text: str
    url: str
    author_handle: str
    date: Optional[str] = None
    date_confidence: str = "low"
    engagement: Optional[Engagement] = None
    relevance: float = 0.5
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    def to_dict(self) -> Dict[str, Any]:
        return {
            'id': self.id,
            'text': self.text,
            'url': self.url,
            'author_handle': self.author_handle,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'engagement': self.engagement.to_dict() if self.engagement else None,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
@dataclass
 class WebSearchItem:
    """Normalized web search item (no engagement metrics)."""
    id: str
    title: str
    url: str
    source_domain: str  # e.g., "medium.com", "github.com"
    snippet: str
    date: Optional[str] = None
    date_confidence: str = "low"
    relevance: float = 0.5
    why_relevant: str = ""
    subs: SubScores = field(default_factory=SubScores)
    score: int = 0
    def to_dict(self) -> Dict[str, Any]:
        return {
            'id': self.id,
            'title': self.title,
            'url': self.url,
            'source_domain': self.source_domain,
            'snippet': self.snippet,
            'date': self.date,
            'date_confidence': self.date_confidence,
            'relevance': self.relevance,
            'why_relevant': self.why_relevant,
            'subs': self.subs.to_dict(),
            'score': self.score,
        }
@dataclass
 class Report:
    """Full research report."""
    topic: str
    range_from: str
    range_to: str
    generated_at: str
    mode: str  # 'reddit-only', 'x-only', 'both', 'web-only', etc.
    openai_model_used: Optional[str] = None
    xai_model_used: Optional[str] = None
    reddit: List[RedditItem] = field(default_factory=list)
    x: List[XItem] = field(default_factory=list)
    web: List[WebSearchItem] = field(default_factory=list)
    best_practices: List[str] = field(default_factory=list)
    prompt_pack: List[str] = field(default_factory=list)
    context_snippet_md: str = ""
    # Status tracking
    reddit_error: Optional[str] = None
    x_error: Optional[str] = None
    web_error: Optional[str] = None
    # Cache info
    from_cache: bool = False
    cache_age_hours: Optional[float] = None
    def to_dict(self) -> Dict[str, Any]:
        d = {
            'topic': self.topic,
            'range': {
                'from': self.range_from,
                'to': self.range_to,
            },
            'generated_at': self.generated_at,
            'mode': self.mode,
            'openai_model_used': self.openai_model_used,
            'xai_model_used': self.xai_model_used,
            'reddit': [r.to_dict() for r in self.reddit],
            'x': [x.to_dict() for x in self.x],
            'web': [w.to_dict() for w in self.web],
            'best_practices': self.best_practices,
            'prompt_pack': self.prompt_pack,
            'context_snippet_md': self.context_snippet_md,
        }
        if self.reddit_error:
            d['reddit_error'] = self.reddit_error
        if self.x_error:
            d['x_error'] = self.x_error
        if self.web_error:
            d['web_error'] = self.web_error
        if self.from_cache:
            d['from_cache'] = self.from_cache
        if self.cache_age_hours is not None:
            d['cache_age_hours'] = self.cache_age_hours
        return d
    @classmethod
    def from_dict(cls, data: Dict[str, Any]) -> "Report":
        """Create Report from serialized dict (handles cache format)."""
        # Handle range field conversion
        range_data = data.get('range', {})
        range_from = range_data.get('from', data.get('range_from', ''))
        range_to = range_data.get('to', data.get('range_to', ''))
        # Reconstruct Reddit items
        reddit_items = []
        for r in data.get('reddit', []):
            eng = None
            if r.get('engagement'):
                eng = Engagement(**r['engagement'])
            comments = [Comment(**c) for c in r.get('top_comments', [])]
            subs = SubScores(**r.get('subs', {})) if r.get('subs') else SubScores()
            reddit_items.append(RedditItem(
                id=r['id'],
                title=r['title'],
                url=r['url'],
                subreddit=r['subreddit'],
                date=r.get('date'),
                date_confidence=r.get('date_confidence', 'low'),
                engagement=eng,
                top_comments=comments,
                comment_insights=r.get('comment_insights', []),
                relevance=r.get('relevance', 0.5),
                why_relevant=r.get('why_relevant', ''),
                subs=subs,
                score=r.get('score', 0),
            ))
        # Reconstruct X items
        x_items = []
        for x in data.get('x', []):
            eng = None
            if x.get('engagement'):
                eng = Engagement(**x['engagement'])
            subs = SubScores(**x.get('subs', {})) if x.get('subs') else SubScores()
            x_items.append(XItem(
                id=x['id'],
                text=x['text'],
                url=x['url'],
                author_handle=x['author_handle'],
                date=x.get('date'),
                date_confidence=x.get('date_confidence', 'low'),
                engagement=eng,
                relevance=x.get('relevance', 0.5),
                why_relevant=x.get('why_relevant', ''),
                subs=subs,
                score=x.get('score', 0),
            ))
        # Reconstruct Web items
        web_items = []
        for w in data.get('web', []):
            subs = SubScores(**w.get('subs', {})) if w.get('subs') else SubScores()
            web_items.append(WebSearchItem(
                id=w['id'],
                title=w['title'],
                url=w['url'],
                source_domain=w.get('source_domain', ''),
                snippet=w.get('snippet', ''),
                date=w.get('date'),
                date_confidence=w.get('date_confidence', 'low'),
                relevance=w.get('relevance', 0.5),
                why_relevant=w.get('why_relevant', ''),
                subs=subs,
                score=w.get('score', 0),
            ))
        return cls(
            topic=data['topic'],
            range_from=range_from,
            range_to=range_to,
            generated_at=data['generated_at'],
            mode=data['mode'],
            openai_model_used=data.get('openai_model_used'),
            xai_model_used=data.get('xai_model_used'),
            reddit=reddit_items,
            x=x_items,
            web=web_items,
            best_practices=data.get('best_practices', []),
            prompt_pack=data.get('prompt_pack', []),
            context_snippet_md=data.get('context_snippet_md', ''),
            reddit_error=data.get('reddit_error'),
            x_error=data.get('x_error'),
            web_error=data.get('web_error'),
            from_cache=data.get('from_cache', False),
            cache_age_hours=data.get('cache_age_hours'),
        )
 def create_report(
    topic: str,
    from_date: str,
    to_date: str,
    mode: str,
    openai_model: Optional[str] = None,
    xai_model: Optional[str] = None,
 ) -> Report:
    """Create a new report with metadata."""
    return Report(
        topic=topic,
        range_from=from_date,
        range_to=to_date,
        generated_at=datetime.now(timezone.utc).isoformat(),
        mode=mode,
        openai_model_used=openai_model,
        xai_model_used=xai_model,
    )
@@ -1,299 +0,0 @@
 """Popularity-aware scoring for last30days skill."""
 import math
 from typing import List, Optional, Union
 from . import dates, schema
 # Score weights for Reddit/X (has engagement)
 WEIGHT_RELEVANCE = 0.45
 WEIGHT_RECENCY = 0.25
 WEIGHT_ENGAGEMENT = 0.30
 # WebSearch weights (no engagement, reweighted to 100%)
 WEBSEARCH_WEIGHT_RELEVANCE = 0.55
 WEBSEARCH_WEIGHT_RECENCY = 0.45
 WEBSEARCH_SOURCE_PENALTY = 15  # Points deducted for lacking engagement
 # Default engagement score for unknown
 DEFAULT_ENGAGEMENT = 35
 UNKNOWN_ENGAGEMENT_PENALTY = 10
 def log1p_safe(x: Optional[int]) -> float:
    """Safe log1p that handles None and negative values."""
    if x is None or x < 0:
        return 0.0
    return math.log1p(x)
 def compute_reddit_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for Reddit item.
    Formula: 0.55*log1p(score) + 0.40*log1p(num_comments) + 0.05*(upvote_ratio*10)
    """
    if engagement is None:
        return None
    if engagement.score is None and engagement.num_comments is None:
        return None
    score = log1p_safe(engagement.score)
    comments = log1p_safe(engagement.num_comments)
    ratio = (engagement.upvote_ratio or 0.5) * 10
    return 0.55 * score + 0.40 * comments + 0.05 * ratio
 def compute_x_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
    """Compute raw engagement score for X item.
    Formula: 0.55*log1p(likes) + 0.25*log1p(reposts) + 0.15*log1p(replies) + 0.05*log1p(quotes)
    """
    if engagement is None:
        return None
    if engagement.likes is None and engagement.reposts is None:
        return None
    likes = log1p_safe(engagement.likes)
    reposts = log1p_safe(engagement.reposts)
    replies = log1p_safe(engagement.replies)
    quotes = log1p_safe(engagement.quotes)
    return 0.55 * likes + 0.25 * reposts + 0.15 * replies + 0.05 * quotes
 def normalize_to_100(values: List[float], default: float = 50) -> List[float]:
    """Normalize a list of values to 0-100 scale.
    Args:
        values: Raw values (None values are preserved)
        default: Default value for None entries
    Returns:
        Normalized values
    """
    # Filter out None
    valid = [v for v in values if v is not None]
    if not valid:
        return [default if v is None else 50 for v in values]
    min_val = min(valid)
    max_val = max(valid)
    range_val = max_val - min_val
    if range_val == 0:
        return [50 if v is None else 50 for v in values]
    result = []
    for v in values:
        if v is None:
            result.append(None)
        else:
            normalized = ((v - min_val) / range_val) * 100
            result.append(normalized)
    return result
 def score_reddit_items(items: List[schema.RedditItem]) -> List[schema.RedditItem]:
    """Compute scores for Reddit items.
    Args:
        items: List of Reddit items
    Returns:
        Items with updated scores
    """
    if not items:
        return items
    # Compute raw engagement scores
    eng_raw = [compute_reddit_engagement_raw(item.engagement) for item in items]
    # Normalize engagement to 0-100
    eng_normalized = normalize_to_100(eng_raw)
    for i, item in enumerate(items):
        # Relevance subscore (model-provided, convert to 0-100)
        rel_score = int(item.relevance * 100)
        # Recency subscore
        rec_score = dates.recency_score(item.date)
        # Engagement subscore
        if eng_normalized[i] is not None:
            eng_score = int(eng_normalized[i])
        else:
            eng_score = DEFAULT_ENGAGEMENT
        # Store subscores
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=eng_score,
        )
        # Compute overall score
        overall = (
            WEIGHT_RELEVANCE * rel_score +
            WEIGHT_RECENCY * rec_score +
            WEIGHT_ENGAGEMENT * eng_score
        )
        # Apply penalty for unknown engagement
        if eng_raw[i] is None:
            overall -= UNKNOWN_ENGAGEMENT_PENALTY
        # Apply penalty for low date confidence
        if item.date_confidence == "low":
            overall -= 10
        elif item.date_confidence == "med":
            overall -= 5
        item.score = max(0, min(100, int(overall)))
    return items
 def score_x_items(items: List[schema.XItem]) -> List[schema.XItem]:
    """Compute scores for X items.
    Args:
        items: List of X items
    Returns:
        Items with updated scores
    """
    if not items:
        return items
    # Compute raw engagement scores
    eng_raw = [compute_x_engagement_raw(item.engagement) for item in items]
    # Normalize engagement to 0-100
    eng_normalized = normalize_to_100(eng_raw)
    for i, item in enumerate(items):
        # Relevance subscore (model-provided, convert to 0-100)
        rel_score = int(item.relevance * 100)
        # Recency subscore
        rec_score = dates.recency_score(item.date)
        # Engagement subscore
        if eng_normalized[i] is not None:
            eng_score = int(eng_normalized[i])
        else:
            eng_score = DEFAULT_ENGAGEMENT
        # Store subscores
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=eng_score,
        )
        # Compute overall score
        overall = (
            WEIGHT_RELEVANCE * rel_score +
            WEIGHT_RECENCY * rec_score +
            WEIGHT_ENGAGEMENT * eng_score
        )
        # Apply penalty for unknown engagement
        if eng_raw[i] is None:
            overall -= UNKNOWN_ENGAGEMENT_PENALTY
        # Apply penalty for low date confidence
        if item.date_confidence == "low":
            overall -= 10
        elif item.date_confidence == "med":
            overall -= 5
        item.score = max(0, min(100, int(overall)))
    return items
 def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
    """Compute scores for WebSearch items WITHOUT engagement metrics.
    Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
    This ensures WebSearch items rank below comparable Reddit/X items.
    Args:
        items: List of WebSearch items
    Returns:
        Items with updated scores
    """
    if not items:
        return items
    for item in items:
        # Relevance subscore (model-provided, convert to 0-100)
        rel_score = int(item.relevance * 100)
        # Recency subscore
        rec_score = dates.recency_score(item.date)
        # Store subscores (engagement is 0 for WebSearch - no data)
        item.subs = schema.SubScores(
            relevance=rel_score,
            recency=rec_score,
            engagement=0,  # Explicitly zero - no engagement data available
        )
        # Compute overall score using WebSearch weights
        overall = (
            WEBSEARCH_WEIGHT_RELEVANCE * rel_score +
            WEBSEARCH_WEIGHT_RECENCY * rec_score
        )
        # Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
        overall -= WEBSEARCH_SOURCE_PENALTY
        # Apply penalty for low date confidence
        if item.date_confidence == "low":
            overall -= 10
        elif item.date_confidence == "med":
            overall -= 5
        item.score = max(0, min(100, int(overall)))
    return items
 def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem]]) -> List:
    """Sort items by score (descending), then date, then source priority.
    Args:
        items: List of items to sort
    Returns:
        Sorted items
    """
    def sort_key(item):
        # Primary: score descending (negate for descending)
        score = -item.score
        # Secondary: date descending (recent first)
        date = item.date or "0000-00-00"
        date_key = -int(date.replace("-", ""))
        # Tertiary: source priority (Reddit > X > WebSearch)
        if isinstance(item, schema.RedditItem):
            source_priority = 0
        elif isinstance(item, schema.XItem):
            source_priority = 1
        else:  # WebSearchItem
            source_priority = 2
        # Quaternary: title/text for stability
        text = getattr(item, "title", "") or getattr(item, "text", "")
        return (score, date_key, source_priority, text)
    return sorted(items, key=sort_key)
@@ -1,229 +0,0 @@
 """Terminal UI utilities for last30days skill."""
 import os
 import sys
 import time
 import threading
 import random
 from typing import Optional
 # Check if we're in a real terminal (not captured by Claude Code)
 IS_TTY = sys.stderr.isatty()
 # ANSI color codes
 class Colors:
    PURPLE = '\033[95m'
    BLUE = '\033[94m'
    CYAN = '\033[96m'
    GREEN = '\033[92m'
    YELLOW = '\033[93m'
    RED = '\033[91m'
    BOLD = '\033[1m'
    DIM = '\033[2m'
    RESET = '\033[0m'
 BANNER = f"""{Colors.PURPLE}{Colors.BOLD}
  ██╗      █████╗ ███████╗████████╗██████╗  ██████╗ ██████╗  █████╗ ██╗   ██╗███████╗
  ██║     ██╔══██╗██╔════╝╚══██╔══╝╚════██╗██╔═████╗██╔══██╗██╔══██╗╚██╗ ██╔╝██╔════╝
  ██║     ███████║███████╗   ██║    █████╔╝██║██╔██║██║  ██║███████║ ╚████╔╝ ███████╗
  ██║     ██╔══██║╚════██║   ██║    ╚═══██╗████╔╝██║██║  ██║██╔══██║  ╚██╔╝  ╚════██║
  ███████╗██║  ██║███████║   ██║   ██████╔╝╚██████╔╝██████╔╝██║  ██║   ██║   ███████║
  ╚══════╝╚═╝  ╚═╝╚══════╝   ╚═╝   ╚═════╝  ╚═════╝ ╚═════╝ ╚═╝  ╚═╝   ╚═╝   ╚══════╝
 {Colors.RESET}{Colors.DIM}  30 days of research. 30 seconds of work.{Colors.RESET}
 """
 MINI_BANNER = f"""{Colors.PURPLE}{Colors.BOLD}/last30days{Colors.RESET} {Colors.DIM}· researching...{Colors.RESET}"""
 # Fun status messages for each phase
 REDDIT_MESSAGES = [
    "Diving into Reddit threads...",
    "Scanning subreddits for gold...",
    "Reading what Redditors are saying...",
    "Exploring the front page of the internet...",
    "Finding the good discussions...",
    "Upvoting mentally...",
    "Scrolling through comments...",
 ]
 X_MESSAGES = [
    "Checking what X is buzzing about...",
    "Reading the timeline...",
    "Finding the hot takes...",
    "Scanning tweets and threads...",
    "Discovering trending insights...",
    "Following the conversation...",
    "Reading between the posts...",
 ]
 ENRICHING_MESSAGES = [
    "Getting the juicy details...",
    "Fetching engagement metrics...",
    "Reading top comments...",
    "Extracting insights...",
    "Analyzing discussions...",
 ]
 PROCESSING_MESSAGES = [
    "Crunching the data...",
    "Scoring and ranking...",
    "Finding patterns...",
    "Removing duplicates...",
    "Organizing findings...",
 ]
 # Spinner frames
 SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏']
 DOTS_FRAMES = ['   ', '.  ', '.. ', '...']
 class Spinner:
    """Animated spinner for long-running operations."""
    def __init__(self, message: str = "Working", color: str = Colors.CYAN):
        self.message = message
        self.color = color
        self.running = False
        self.thread: Optional[threading.Thread] = None
        self.frame_idx = 0
        self.shown_static = False
    def _spin(self):
        while self.running:
            frame = SPINNER_FRAMES[self.frame_idx % len(SPINNER_FRAMES)]
            sys.stderr.write(f"\r{self.color}{frame}{Colors.RESET} {self.message}  ")
            sys.stderr.flush()
            self.frame_idx += 1
            time.sleep(0.08)
    def start(self):
        self.running = True
        if IS_TTY:
            # Real terminal - animate
            self.thread = threading.Thread(target=self._spin, daemon=True)
            self.thread.start()
        else:
            # Not a TTY (Claude Code) - just print once
            if not self.shown_static:
                sys.stderr.write(f"⏳ {self.message}\n")
                sys.stderr.flush()
                self.shown_static = True
    def update(self, message: str):
        self.message = message
        if not IS_TTY and not self.shown_static:
            # Print update in non-TTY mode
            sys.stderr.write(f"⏳ {message}\n")
            sys.stderr.flush()
    def stop(self, final_message: str = ""):
        self.running = False
        if self.thread:
            self.thread.join(timeout=0.2)
        if IS_TTY:
            # Clear the line in real terminal
            sys.stderr.write("\r" + " " * 80 + "\r")
        if final_message:
            sys.stderr.write(f"✓ {final_message}\n")
        sys.stderr.flush()
 class ProgressDisplay:
    """Progress display for research phases."""
    def __init__(self, topic: str, show_banner: bool = True):
        self.topic = topic
        self.spinner: Optional[Spinner] = None
        self.start_time = time.time()
        if show_banner:
            self._show_banner()
    def _show_banner(self):
        if IS_TTY:
            sys.stderr.write(MINI_BANNER + "\n")
            sys.stderr.write(f"{Colors.DIM}Topic: {Colors.RESET}{Colors.BOLD}{self.topic}{Colors.RESET}\n\n")
        else:
            # Simple text for non-TTY
            sys.stderr.write(f"/last30days · researching: {self.topic}\n")
        sys.stderr.flush()
    def start_reddit(self):
        msg = random.choice(REDDIT_MESSAGES)
        self.spinner = Spinner(f"{Colors.YELLOW}Reddit{Colors.RESET} {msg}", Colors.YELLOW)
        self.spinner.start()
    def end_reddit(self, count: int):
        if self.spinner:
            self.spinner.stop(f"{Colors.YELLOW}Reddit{Colors.RESET} Found {count} threads")
    def start_reddit_enrich(self, current: int, total: int):
        if self.spinner:
            self.spinner.stop()
        msg = random.choice(ENRICHING_MESSAGES)
        self.spinner = Spinner(f"{Colors.YELLOW}Reddit{Colors.RESET} [{current}/{total}] {msg}", Colors.YELLOW)
        self.spinner.start()
    def update_reddit_enrich(self, current: int, total: int):
        if self.spinner:
            msg = random.choice(ENRICHING_MESSAGES)
            self.spinner.update(f"{Colors.YELLOW}Reddit{Colors.RESET} [{current}/{total}] {msg}")
    def end_reddit_enrich(self):
        if self.spinner:
            self.spinner.stop(f"{Colors.YELLOW}Reddit{Colors.RESET} Enriched with engagement data")
    def start_x(self):
        msg = random.choice(X_MESSAGES)
        self.spinner = Spinner(f"{Colors.CYAN}X{Colors.RESET} {msg}", Colors.CYAN)
        self.spinner.start()
    def end_x(self, count: int):
        if self.spinner:
            self.spinner.stop(f"{Colors.CYAN}X{Colors.RESET} Found {count} posts")
    def start_processing(self):
        msg = random.choice(PROCESSING_MESSAGES)
        self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE)
        self.spinner.start()
    def end_processing(self):
        if self.spinner:
            self.spinner.stop()
    def show_complete(self, reddit_count: int, x_count: int):
        elapsed = time.time() - self.start_time
        if IS_TTY:
            sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
            sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n")
            sys.stderr.write(f"  {Colors.YELLOW}Reddit:{Colors.RESET} {reddit_count} threads  ")
            sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts\n\n")
        else:
            sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - Reddit: {reddit_count} threads, X: {x_count} posts\n")
        sys.stderr.flush()
    def show_cached(self, age_hours: float = None):
        if age_hours is not None:
            age_str = f" ({age_hours:.1f}h old)"
        else:
            age_str = ""
        sys.stderr.write(f"{Colors.GREEN}⚡{Colors.RESET} {Colors.DIM}Using cached results{age_str} - use --refresh for fresh data{Colors.RESET}\n\n")
        sys.stderr.flush()
    def show_error(self, message: str):
        sys.stderr.write(f"{Colors.RED}✗ Error:{Colors.RESET} {message}\n")
        sys.stderr.flush()
 def print_phase(phase: str, message: str):
    """Print a phase message."""
    colors = {
        "reddit": Colors.YELLOW,
        "x": Colors.CYAN,
        "process": Colors.PURPLE,
        "done": Colors.GREEN,
        "error": Colors.RED,
    }
    color = colors.get(phase, Colors.RESET)
    sys.stderr.write(f"{color}▸{Colors.RESET} {message}\n")
    sys.stderr.flush()
@@ -1,195 +0,0 @@
 """WebSearch module for last30days skill.
 NOTE: WebSearch uses Claude's built-in WebSearch tool, which runs INSIDE Claude Code.
 Unlike Reddit/X which use external APIs, WebSearch results are obtained by Claude
 directly and passed to this module for normalization and scoring.
 The typical flow is:
 1. Claude invokes WebSearch tool with the topic
 2. Claude passes results to parse_websearch_results()
 3. Results are normalized into WebSearchItem objects
 """
 import re
 from typing import Any, Dict, List, Optional
 from urllib.parse import urlparse
 from . import schema
 # Domains to exclude (Reddit and X are handled separately)
 EXCLUDED_DOMAINS = {
    "reddit.com",
    "www.reddit.com",
    "old.reddit.com",
    "twitter.com",
    "www.twitter.com",
    "x.com",
    "www.x.com",
    "mobile.twitter.com",
 }
 def extract_domain(url: str) -> str:
    """Extract the domain from a URL.
    Args:
        url: Full URL
    Returns:
        Domain string (e.g., "medium.com")
    """
    try:
        parsed = urlparse(url)
        domain = parsed.netloc.lower()
        # Remove www. prefix for cleaner display
        if domain.startswith("www."):
            domain = domain[4:]
        return domain
    except Exception:
        return ""
 def is_excluded_domain(url: str) -> bool:
    """Check if URL is from an excluded domain (Reddit/X).
    Args:
        url: URL to check
    Returns:
        True if URL should be excluded
    """
    try:
        parsed = urlparse(url)
        domain = parsed.netloc.lower()
        return domain in EXCLUDED_DOMAINS
    except Exception:
        return False
 def parse_websearch_results(
    results: List[Dict[str, Any]],
    topic: str,
 ) -> List[Dict[str, Any]]:
    """Parse WebSearch results into normalized format.
    This function expects results from Claude's WebSearch tool.
    Each result should have: title, url, snippet, and optionally date/relevance.
    Args:
        results: List of WebSearch result dicts
        topic: Original search topic (for context)
    Returns:
        List of normalized item dicts ready for WebSearchItem creation
    """
    items = []
    for i, result in enumerate(results):
        if not isinstance(result, dict):
            continue
        url = result.get("url", "")
        if not url:
            continue
        # Skip Reddit/X URLs (handled separately)
        if is_excluded_domain(url):
            continue
        title = str(result.get("title", "")).strip()
        snippet = str(result.get("snippet", result.get("description", ""))).strip()
        if not title and not snippet:
            continue
        # Parse date if provided
        date = result.get("date")
        date_confidence = "low"
        if date:
            # Validate date format
            if re.match(r'^\d{4}-\d{2}-\d{2}$', str(date)):
                date_confidence = "med"  # WebSearch dates are often approximate
            else:
                date = None
        # Get relevance if provided, default to 0.5
        relevance = result.get("relevance", 0.5)
        try:
            relevance = min(1.0, max(0.0, float(relevance)))
        except (TypeError, ValueError):
            relevance = 0.5
        item = {
            "id": f"W{i+1}",
            "title": title[:200],  # Truncate long titles
            "url": url,
            "source_domain": extract_domain(url),
            "snippet": snippet[:500],  # Truncate long snippets
            "date": date,
            "date_confidence": date_confidence,
            "relevance": relevance,
            "why_relevant": str(result.get("why_relevant", "")).strip(),
        }
        items.append(item)
    return items
 def normalize_websearch_items(
    items: List[Dict[str, Any]],
    from_date: str,
    to_date: str,
 ) -> List[schema.WebSearchItem]:
    """Convert parsed dicts to WebSearchItem objects.
    Args:
        items: List of parsed item dicts
        from_date: Start of date range (YYYY-MM-DD)
        to_date: End of date range (YYYY-MM-DD)
    Returns:
        List of WebSearchItem objects
    """
    result = []
    for item in items:
        web_item = schema.WebSearchItem(
            id=item["id"],
            title=item["title"],
            url=item["url"],
            source_domain=item["source_domain"],
            snippet=item["snippet"],
            date=item.get("date"),
            date_confidence=item.get("date_confidence", "low"),
            relevance=item.get("relevance", 0.5),
            why_relevant=item.get("why_relevant", ""),
        )
        result.append(web_item)
    return result
 def dedupe_websearch(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
    """Remove duplicate WebSearch items.
    Deduplication is based on URL.
    Args:
        items: List of WebSearchItem objects
    Returns:
        Deduplicated list
    """
    seen_urls = set()
    result = []
    for item in items:
        # Normalize URL for comparison
        url_key = item.url.lower().rstrip("/")
        if url_key not in seen_urls:
            seen_urls.add(url_key)
            result.append(item)
    return result
@@ -0,0 +1,8 @@
 interface:
  display_name: "Last 30 Days"
  short_description: "Research any topic across Reddit, X, YouTube, and the web from the last 30 days. Returns synthesized expert answers and copy-paste prompts."
  default_prompt: "Research this topic from the last 30 days across Reddit, X, YouTube, and web. Synthesize what people are actually saying, upvoting, and sharing right now."
  brand_color: "#FF6B35"
 policy:
  allow_implicit_invocation: true
@@ -0,0 +1,90 @@
 # Save shareable HTML brief
 This reference file is loaded by the main `SKILL.md` when the user asked for an HTML brief (either explicitly via `--emit=html` / `--emit:html` / `--html`, or in natural language - "give me a shareable HTML brief", "for Slack", "for Notion", "export as HTML", etc.). The detection happens in `SKILL.md` so that the common no-HTML path stays short; the implementation lives here.
 The contract: the synthesis still appears in chat as the primary output. The HTML is an additional artifact saved to disk for sharing. Both happen in the same turn.
 ## When to fire this flow
 - After you have already emitted the full chat response: badge, "What I learned:" (or comparison title), bold-lead-in paragraphs with citations, KEY PATTERNS list, engine footer pass-through, invitation block.
 - BEFORE the WAIT FOR USER'S RESPONSE pause.
 - ONLY if the user asked. Do NOT save HTML when the user didn't ask for it.
 ## How to fire it
 ```bash
 # 1. Write your synthesis prose VERBATIM to a temp file. The synthesis is the
 #    "What I learned:" prose label, the bold-lead-in paragraphs with their
 #    inline citations as you wrote them in chat, and the "KEY PATTERNS from
 #    the research:" numbered list. Do NOT include the badge or the engine
 #    footer in the temp file - the engine adds those when it renders the HTML.
 #    Use the EXACT text you just wrote in chat. Do not paraphrase, do not
 #    summarize, do not reorder. The HTML must read identically to the chat
 #    response in voice and citations.
 SYNTHESIS_FILE="/tmp/last30days-synthesis-${CLAUDE_SESSION_ID}.md"
 cat > "$SYNTHESIS_FILE" <<'SYNTHESIS_EOF'
 What I learned:
 **{First headline}** - {body with [name](url) inline citations}
 **{Second headline}** - {body}
 **{Third headline}** - {body}
 KEY PATTERNS from the research:
 1. {pattern} - per [@handle](url)
 2. {pattern} - per [r/sub](url)
 3. {pattern} - per [@handle](url)
 SYNTHESIS_EOF
 # 2. Convert the synthesis to a self-contained HTML file via the engine.
 #    The engine reuses the cache from your earlier engine run (same topic
 #    + plan), so this second invocation is typically <1s on cache hit.
 SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//')
 HTML_PATH="${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html"
 "${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "${TOPIC}" \
  --emit=html \
  --synthesis-file "$SYNTHESIS_FILE" \
  > "$HTML_PATH"
 # 3. Append ONE line to your already-emitted chat response, after the
 #    invitation block. Use a paperclip emoji as a visible signal that an
 #    artifact was produced:
 echo "📎 Shareable brief saved to $HTML_PATH"
 ```
 ## What ends up in the HTML file
 The engine's `--emit=html` renderer combines:
 - The badge (`🌐 last30days vX.Y.Z · synced YYYY-MM-DD`) at the top
 - A single inline metadata line (`{date range} · {active sources}`) below the badge
 - Your synthesis verbatim, with prose labels promoted to `<h2>` and bold lead-ins preserved
 - All `[name](url)` citations rendered as `<a>` tags
 - The engine footer (`✅ All agents reported back!` tree) preserved verbatim in monospace
 - A colophon with the topic and a re-run hint
 The renderer strips engine-internal noise that doesn't belong in a shareable artifact: the `# last30days vX.Y.Z: TOPIC` debug file header, the model-facing `> Safety note:` blockquote, and the `I'm now an expert on X` invitation block. Data quality warnings (degraded run, thin evidence, etc.) stay in the engine's stderr logs - they never leak into the share-ready file.
 ## Comparison mode
 Same flow when the topic is `X vs Y` (or `X vs Y vs Z`). The engine routes through `render_for_html_comparison` internally; you don't need to do anything special. The synthesis temp file should still contain the comparison-shaped synthesis you wrote in chat (`## Quick Verdict`, `## {Entity}` per entity, `## Head-to-Head` table, `## The Bottom Line`, `## The emerging stack` per LAW 4 comparison exception).
 ## Follow-up turn
 If the user runs `/last30days OpenClaw` normally, sees the synthesis in chat, and THEN says "save that as HTML" or "give me a shareable version" in a follow-up turn, do the same save flow on the synthesis you wrote in the previous turn. Do not re-research; the synthesis is already in the conversation history. Just write it to the temp file and call the engine with `--emit=html --synthesis-file`.
 ## What NOT to do
 - Do NOT save HTML if the user didn't ask. The sparse mode (no synthesis) produces a thin file; not useful as a shareable.
 - Do NOT add content to the temp file beyond your synthesis prose. The badge / footer / colophon come from the engine.
 - Do NOT change the file path convention. `${LAST30DAYS_MEMORY_DIR}/${SLUG}-brief.html` is the canonical location.
 - Do NOT silently overwrite an existing file without telling the user. If `$HTML_PATH` already exists from a prior run, the engine will pick a date-suffixed name (`{slug}-brief-YYYY-MM-DD.html`) automatically; just print whichever path the redirect produced.
 - Do NOT include the data quality warning text in the temp file or in your final chat line. Warnings are an engine-stderr concern, not an artifact concern.
 ## Edge cases
 - **Topic with shell-special characters** (quotes, ampersands): the temp filename uses a slugified version, but the engine receives the raw topic. The `cat <<'SYNTHESIS_EOF'` quoted heredoc form handles arbitrary content without expansion. Your synthesis text can include any character.
 - **Very long synthesis**: no upper bound. The engine handles long markdown bodies. Just paste verbatim.
 - **Synthesis with images or non-ASCII**: emoji and Unicode pass through. Image tags pass through as raw HTML; the renderer doesn't transform them. If you didn't include images in chat, don't add them here.
 - **No `${LAST30DAYS_MEMORY_DIR}` set**: defaults to `~/Documents/Last30Days/` per the SKILL.md `Configuration` section.
@@ -0,0 +1,264 @@
 #!/usr/bin/env python3
 """Morning briefing generator for last30days.
 Synthesizes accumulated findings into formatted briefings.
 The Python script collects the data; the agent (via SKILL.md) does the
 beautiful synthesis. This script provides the structured data.
 Usage:
    python3 briefing.py generate              # Daily briefing data
    python3 briefing.py generate --weekly     # Weekly digest data
    python3 briefing.py show [--date DATE]    # Show saved briefing
 """
 import argparse
 import json
 import sys
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 SCRIPT_DIR = Path(__file__).parent.resolve()
 sys.path.insert(0, str(SCRIPT_DIR))
 import store
 BRIEFS_DIR = Path.home() / ".local" / "share" / "last30days" / "briefs"
 def _parse_sqlite_utc_timestamp(value: str) -> datetime:
    return datetime.strptime(value, "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)
 def generate_daily(since: str = None) -> dict:
    """Generate daily briefing data.
    Returns structured data for the agent to synthesize into a beautiful briefing.
    """
    store.init_db()
    topics = store.list_topics()
    if not topics:
        return {
            "status": "no_topics",
            "message": "No watchlist topics yet. Add one with: last30days watch add \"your topic\"",
        }
    enabled = [t for t in topics if t["enabled"]]
    if not enabled:
        return {
            "status": "no_enabled",
            "message": "All topics are paused. Enable a topic to generate briefings.",
        }
    # Default: findings since yesterday
    if not since:
        since = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
    briefing_topics = []
    total_new = 0
    for topic in enabled:
        findings = store.get_new_findings(topic["id"], since)
        last_run = topic.get("last_run")
        last_status = topic.get("last_status", "unknown")
        # Calculate staleness
        stale = False
        hours_ago = None
        if last_run:
            try:
                run_dt = _parse_sqlite_utc_timestamp(last_run)
                hours_ago = (datetime.now(timezone.utc) - run_dt).total_seconds() / 3600
                stale = hours_ago > 36  # Stale if > 36 hours
            except (ValueError, TypeError):
                stale = True
        topic_data = {
            "name": topic["name"],
            "findings": findings,
            "new_count": len(findings),
            "last_run": last_run,
            "last_status": last_status,
            "stale": stale,
            "hours_ago": round(hours_ago, 1) if hours_ago else None,
        }
        # Extract top finding by engagement
        if findings:
            top = max(findings, key=lambda f: f.get("engagement_score", 0))
            topic_data["top_finding"] = {
                "title": top.get("source_title", ""),
                "source": top.get("source", ""),
                "author": top.get("author", ""),
                "engagement": top.get("engagement_score", 0),
                "content": top.get("content", "")[:300],
            }
        briefing_topics.append(topic_data)
        total_new += len(findings)
    # Cost info
    daily_cost = store.get_daily_cost()
    budget = float(store.get_setting("daily_budget", "5.00"))
    # Find the single top finding across all topics (for TL;DR)
    all_findings = []
    for t in briefing_topics:
        for f in t["findings"]:
            f["_topic"] = t["name"]
            all_findings.append(f)
    top_overall = None
    if all_findings:
        top_overall = max(all_findings, key=lambda f: f.get("engagement_score", 0))
    result = {
        "status": "ok",
        "date": datetime.now().strftime("%Y-%m-%d"),
        "since": since,
        "topics": briefing_topics,
        "total_new": total_new,
        "total_topics": len(briefing_topics),
        "top_finding": {
            "title": top_overall.get("source_title", ""),
            "topic": top_overall.get("_topic", ""),
            "engagement": top_overall.get("engagement_score", 0),
        } if top_overall else None,
        "cost": {
            "daily": daily_cost,
            "budget": budget,
        },
        "failed_topics": [
            t["name"] for t in briefing_topics if t["last_status"] == "failed"
        ],
    }
    # Save briefing data
    _save_briefing(result)
    return result
 def generate_weekly() -> dict:
    """Generate weekly digest data with trend analysis."""
    store.init_db()
    week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
    two_weeks_ago = (datetime.now() - timedelta(days=14)).strftime("%Y-%m-%d")
    topics = store.list_topics()
    if not topics:
        return {"status": "no_topics", "message": "No watchlist topics."}
    weekly_topics = []
    for topic in topics:
        if not topic["enabled"]:
            continue
        # This week's findings
        this_week = store.get_new_findings(topic["id"], week_ago)
        # Last week's findings (for comparison)
        conn = store._connect()
        try:
            last_week_rows = conn.execute(
                """SELECT * FROM findings
                   WHERE topic_id = ? AND first_seen >= ? AND first_seen < ? AND dismissed = 0
                   ORDER BY engagement_score DESC""",
                (topic["id"], two_weeks_ago, week_ago),
            ).fetchall()
            last_week = [dict(r) for r in last_week_rows]
        finally:
            conn.close()
        this_engagement = sum(f.get("engagement_score", 0) for f in this_week)
        last_engagement = sum(f.get("engagement_score", 0) for f in last_week)
        # Trend calculation
        if last_engagement > 0:
            engagement_change = ((this_engagement - last_engagement) / last_engagement) * 100
        else:
            engagement_change = 100 if this_engagement > 0 else 0
        weekly_topics.append({
            "name": topic["name"],
            "this_week_count": len(this_week),
            "last_week_count": len(last_week),
            "this_week_engagement": this_engagement,
            "last_week_engagement": last_engagement,
            "engagement_change_pct": round(engagement_change, 1),
            "top_findings": this_week[:5],  # Top 5 by engagement (already sorted)
        })
    result = {
        "status": "ok",
        "type": "weekly",
        "week_of": week_ago,
        "topics": weekly_topics,
    }
    _save_briefing(result, suffix="-weekly")
    return result
 def show_briefing(date: str = None) -> dict:
    """Load a saved briefing by date."""
    if not date:
        date = datetime.now().strftime("%Y-%m-%d")
    path = BRIEFS_DIR / f"{date}.json"
    if not path.exists():
        # Try weekly
        path = BRIEFS_DIR / f"{date}-weekly.json"
    if not path.exists():
        return {"status": "not_found", "message": f"No briefing found for {date}."}
    with open(path, encoding="utf-8") as f:
        return json.load(f)
 def _save_briefing(data: dict, suffix: str = ""):
    """Save briefing data to local archive."""
    BRIEFS_DIR.mkdir(parents=True, exist_ok=True)
    date = datetime.now().strftime("%Y-%m-%d")
    path = BRIEFS_DIR / f"{date}{suffix}.json"
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, default=str)
 def main():
    parser = argparse.ArgumentParser(description="Generate last30days briefings")
    sub = parser.add_subparsers(dest="command")
    # generate
    g = sub.add_parser("generate", help="Generate a briefing")
    g.add_argument("--weekly", action="store_true", help="Weekly digest")
    g.add_argument("--since", help="Findings since date (YYYY-MM-DD)")
    # show
    s = sub.add_parser("show", help="Show a saved briefing")
    s.add_argument("--date", help="Date (YYYY-MM-DD, default: today)")
    args = parser.parse_args()
    if args.command == "generate":
        if args.weekly:
            result = generate_weekly()
        else:
            result = generate_daily(since=args.since)
        print(json.dumps(result, indent=2, default=str))
    elif args.command == "show":
        result = show_briefing(date=args.date)
        print(json.dumps(result, indent=2, default=str))
    else:
        parser.print_help()
        sys.exit(1)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,39 @@
 #!/usr/bin/env bash
 # build-skill.sh - package this repo as a claude.ai-upload-ready .skill file
 # Usage: bash skills/last30days/scripts/build-skill.sh  (run from repo root)
 #
 # Produces dist/last30days.skill, a zip with a single top-level `last30days/`
 # directory containing SKILL.md and the scripts/ runtime from skills/last30days.
 # See
 # docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md.
 set -euo pipefail
 REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
 cd "$REPO_ROOT"
 if ! git diff --quiet || ! git diff --cached --quiet; then
  echo "error: working tree is dirty; commit or stash before building" >&2
  exit 1
 fi
 mkdir -p dist
 OUT="dist/last30days.skill"
 git archive --format=zip --prefix=last30days/ --output="$OUT" HEAD:skills/last30days
 COUNT=$(unzip -l "$OUT" | tail -1 | awk '{print $2}')
 SIZE=$(du -h "$OUT" | cut -f1)
 if [ "$COUNT" -gt 200 ]; then
  echo "error: $COUNT files in zip, claude.ai's cap is 200" >&2
  echo "       check .gitattributes export-ignore entries and this script's zip -d excludes" >&2
  exit 1
 fi
 SKILL_MD_COUNT=$(unzip -l "$OUT" | grep -c "SKILL.md" || true)
 if [ "$SKILL_MD_COUNT" -ne 1 ]; then
  echo "error: expected exactly one SKILL.md, found $SKILL_MD_COUNT" >&2
  exit 1
 fi
 echo "built $OUT ($COUNT files, $SIZE)"
 echo "upload via the claude.ai skill UI"
@@ -0,0 +1,61 @@
 #!/bin/bash
 # A/B test runner: public release vs private beta
 # Usage: bash skills/last30days/scripts/compare.sh "Kanye West"
 #
 # Runs /last30days (public release) and /last30days-beta (private beta)
 # sequentially with a 30s gap, saves raw results with distinct suffixes,
 # prints file paths for comparison.
 set -e
 if [ $# -eq 0 ]; then
  echo "Usage: bash skills/last30days/scripts/compare.sh <topic>"
  echo "  Example: bash skills/last30days/scripts/compare.sh Kevin Rose"
  exit 1
 fi
 TOPIC="$*"
 SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
 LAST30DAYS_MEMORY_DIR="${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}"
 DIR="$LAST30DAYS_MEMORY_DIR"
 DATE=$(date +%Y-%m-%d)
 echo "=============================================="
 echo " A/B Test: $TOPIC"
 echo " Date: $DATE"
 echo "=============================================="
 echo ""
 # Run 1: public release
 echo "[1/2] Running /last30days (public release)..."
 echo "  This takes 2-4 minutes..."
 claude -p --dangerously-skip-permissions "/last30days $TOPIC" > /dev/null 2>&1 || true
 RELEASE_FILE="$DIR/${SLUG}-raw.md"
 [ -f "$RELEASE_FILE" ] && echo "  Done: $RELEASE_FILE" || echo "  FAILED: no output file"
 echo ""
 echo "  Waiting 30s for API rate limits..."
 sleep 30
 # Run 2: private beta
 echo "[2/2] Running /last30days-beta (private beta)..."
 echo "  This takes 2-4 minutes..."
 claude -p --dangerously-skip-permissions "/last30days-beta $TOPIC" > /dev/null 2>&1 || true
 BETA_FILE="$DIR/${SLUG}-raw-beta.md"
 [ -f "$BETA_FILE" ] && echo "  Done: $BETA_FILE" || echo "  FAILED: no output file"
 echo ""
 echo "=============================================="
 echo " Both complete. Raw files:"
 echo "=============================================="
 echo ""
 ls -la "$DIR/${SLUG}-raw"*.md 2>/dev/null || echo "  (no files found - check if skills saved correctly)"
 echo ""
 echo "To compare, run in Claude Code:"
 echo "  Read and compare these raw research files, produce a detailed report:"
 echo "  $RELEASE_FILE"
 echo "  $BETA_FILE"
 echo ""
 echo "Beta output should start with a line like:"
 echo "  🧪 last30days-beta · branch <name> · synced $DATE"
 echo "If that line is missing, the beta badge regressed. See docs/plans/2026-04-17-005-*-plan.md."
 echo ""
@@ -0,0 +1,554 @@
 #!/usr/bin/env python3
 """Compare two last30days revisions on the v3 ranked candidate output."""
 from __future__ import annotations
 import argparse
 import json
 import math
 import os
 import subprocess
 import sys
 import tempfile
 from datetime import datetime
 from pathlib import Path
 from typing import Any
 from urllib.error import HTTPError, URLError
 from urllib.request import Request, urlopen
 sys.path.insert(0, str(Path(__file__).parent))
 from lib import env as envlib
 from lib import schema
 from lib.providers import GEMINI_FLASH_LITE
 SKILL_ROOT = Path(__file__).resolve().parents[1]
 REPO_ROOT = Path(__file__).resolve().parents[3]
 EVAL_TOPICS_FILE = REPO_ROOT / "fixtures" / "eval_topics.json"
 def _load_default_topics() -> list[tuple[str, str]]:
    if EVAL_TOPICS_FILE.exists():
        rows = json.loads(EVAL_TOPICS_FILE.read_text())
        return [(row["topic"], row["query_type"]) for row in rows]
    return [
        ("nano banana pro prompting", "product"),
        ("codex vs claude code", "comparison"),
        ("openclaw vs nanoclaw vs ironclaw", "comparison"),
        ("anthropic odds", "prediction"),
        ("kanye west", "breaking_news"),
        ("remotion animations for Claude Code", "how_to"),
    ]
 DEFAULT_TOPICS = _load_default_topics()
 DEFAULT_SEARCH = ""
 DEFAULT_JUDGE_MODEL = GEMINI_FLASH_LITE
 GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}"
 def stable_item_key(item: dict[str, Any]) -> str:
    return str(item.get("candidate_id") or item.get("url") or item.get("title") or "")
 def row_sources(row: dict[str, Any]) -> list[str]:
    candidate = schema.candidate_from_dict(row)
    return schema.candidate_sources(candidate)
 def row_best_date(row: dict[str, Any]) -> str | None:
    candidate = schema.candidate_from_dict(row)
    return schema.candidate_best_published_at(candidate)
 V2_SOURCE_KEYS = [
    ("reddit", "title"),
    ("x", "text"),
    ("youtube", "title"),
    ("tiktok", "text"),
    ("instagram", "text"),
    ("hackernews", "title"),
    ("bluesky", "text"),
    ("truthsocial", "text"),
    ("polymarket", "question"),
    ("web", "title"),
 ]
 def build_ranked_items(report: dict[str, Any], limit: int) -> list[dict[str, Any]]:
    # v3 format: ranked_candidates list
    if report.get("ranked_candidates"):
        ranked = []
        for row in report["ranked_candidates"][:limit]:
            candidate_sources = row_sources(row)
            ranked.append({
                "key": stable_item_key(row),
                "source": ", ".join(candidate_sources),
                "sources": candidate_sources,
                "url": str(row.get("url") or ""),
                "text": str(row.get("title") or ""),
                "date": row_best_date(row),
                "score": float(row.get("final_score") or 0.0),
            })
        return ranked
    # v2 format: per-source lists (reddit, x, youtube, etc.)
    all_items = []
    for source_key, text_field in V2_SOURCE_KEYS:
        for item in report.get(source_key) or []:
            if not isinstance(item, dict):
                continue
            all_items.append({
                "key": str(item.get("url") or item.get("id") or item.get(text_field) or ""),
                "source": source_key,
                "sources": [source_key],
                "url": str(item.get("url") or ""),
                "text": str(item.get(text_field) or item.get("title") or ""),
                "date": item.get("date"),
                "score": float(item.get("score") or 0.0),
            })
    all_items.sort(key=lambda x: x["score"], reverse=True)
    return all_items[:limit]
 def source_sets(report: dict[str, Any], limit: int) -> dict[str, set[str]]:
    grouped: dict[str, set[str]] = {}
    for item in build_ranked_items(report, limit):
        for source in item["sources"]:
            grouped.setdefault(source, set()).add(item["key"])
    return grouped
 def jaccard(left: set[str], right: set[str]) -> float:
    if not left and not right:
        return 1.0
    union = left | right
    if not union:
        return 1.0
    return len(left & right) / len(union)
 def retention(left: set[str], right: set[str]) -> float:
    if not left:
        return 1.0
    return len(left & right) / len(left)
 def precision_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int) -> float:
    top = ranking[:k]
    if not top:
        return 0.0
    return sum(1 for item in top if judgments.get(item["key"], 0) >= 2) / len(top)
 def ndcg_at_k(ranking: list[dict[str, Any]], judgments: dict[str, int], k: int, judged_pool: list[dict[str, Any]]) -> float:
    top = ranking[:k]
    if not top:
        return 0.0
    def dcg(grades: list[int]) -> float:
        total = 0.0
        for index, grade in enumerate(grades, start=1):
            total += (2**grade - 1) / math.log2(index + 1)
        return total
    actual = [judgments.get(item["key"], 0) for item in top]
    ideal = sorted((judgments.get(item["key"], 0) for item in judged_pool), reverse=True)[: len(top)]
    ideal_score = dcg(ideal)
    if ideal_score == 0:
        return 0.0
    return dcg(actual) / ideal_score
 def source_coverage_recall(ranking: list[dict[str, Any]], judged_pool: list[dict[str, Any]], judgments: dict[str, int]) -> float:
    good_sources = {
        source
        for item in judged_pool
        if judgments.get(item["key"], 0) >= 2
        for source in item["sources"]
    }
    if not good_sources:
        return 1.0
    hit_sources = {
        source
        for item in ranking
        if judgments.get(item["key"], 0) >= 2
        for source in item["sources"]
    }
    return len(hit_sources & good_sources) / len(good_sources)
 def resolve_google_judge_api_key(config: dict[str, Any]) -> str | None:
    return (
        os.environ.get("GOOGLE_API_KEY")
        or config.get("GOOGLE_API_KEY")
        or os.environ.get("GEMINI_API_KEY")
        or config.get("GEMINI_API_KEY")
        or os.environ.get("GOOGLE_GENAI_API_KEY")
        or config.get("GOOGLE_GENAI_API_KEY")
    )
 def extract_gemini_text(payload: dict[str, Any]) -> str:
    for candidate in payload.get("candidates") or []:
        content = candidate.get("content") or {}
        for part in content.get("parts") or []:
            if part.get("text"):
                return part["text"]
    raise ValueError("Gemini response did not contain text.")
 def call_gemini_judge(api_key: str, model: str, prompt: str) -> dict[str, Any]:
    body = {
        "contents": [{"parts": [{"text": prompt}]}],
        "generationConfig": {"temperature": 0, "responseMimeType": "application/json"},
    }
    request = Request(
        GEMINI_API_URL.format(model=model, api_key=api_key),
        data=json.dumps(body).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urlopen(request, timeout=120) as response:
            payload = json.loads(response.read().decode("utf-8"))
    except HTTPError as exc:
        detail = exc.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"Gemini HTTP {exc.code}: {detail}") from exc
    except URLError as exc:
        raise RuntimeError(f"Gemini request failed: {exc}") from exc
    return json.loads(extract_gemini_text(payload))
 def build_judge_prompt(topic: str, query_type: str, items: list[dict[str, Any]]) -> str:
    item_lines = []
    for item in items:
        item_lines.append(
            "\n".join([
                f"- id: {item['key']}",
                f"  source: {item['source']}",
                f"  title: {item['text'][:220]}",
                f"  url: {item['url']}",
                f"  date: {item.get('date') or 'unknown'}",
            ])
        )
    return f"""
 Judge search-result relevance for a last-30-days research tool.
 Topic: {topic}
 Query type: {query_type}
 Score each item on this 0-3 scale:
 - 0 = off-topic or clearly bad
 - 1 = weak or tangential
 - 2 = relevant and useful
 - 3 = highly relevant, one of the best results
 Return JSON only:
 {{
  "judgments": [
    {{"id": "ITEM_ID", "grade": 0}}
  ]
 }}
 Items:
 {chr(10).join(item_lines)}
 """.strip()
 def get_judgments(
    *,
    output_dir: Path,
    slug: str,
    topic: str,
    query_type: str,
    items: list[dict[str, Any]],
    judge_model: str,
    gemini_api_key: str | None,
 ) -> dict[str, int]:
    cache_file = output_dir / "judgments" / f"{slug}.json"
    cache_file.parent.mkdir(parents=True, exist_ok=True)
    if cache_file.exists():
        payload = json.loads(cache_file.read_text())
        return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}
    if not gemini_api_key or not items:
        return {}
    payload = call_gemini_judge(gemini_api_key, judge_model, build_judge_prompt(topic, query_type, items))
    cache_file.write_text(json.dumps(payload, indent=2))
    return {row["id"]: int(row["grade"]) for row in payload.get("judgments") or []}
 def create_eval_env() -> dict[str, str]:
    config = envlib.get_config()
    passthrough = {
        "PATH": os.environ.get("PATH", ""),
        "LANG": os.environ.get("LANG", "en_US.UTF-8"),
        "LC_ALL": os.environ.get("LC_ALL", ""),
        "TMPDIR": os.environ.get("TMPDIR", ""),
        "PYTHONUTF8": "1",
        "LAST30DAYS_CONFIG_DIR": "",
    }
    for key in (
        "GOOGLE_API_KEY",
        "GEMINI_API_KEY",
        "GOOGLE_GENAI_API_KEY",
        "OPENAI_API_KEY",
        "XAI_API_KEY",
        "SCRAPECREATORS_API_KEY",
        "BSKY_HANDLE",
        "BSKY_APP_PASSWORD",
        "TRUTHSOCIAL_TOKEN",
        "AUTH_TOKEN",
        "CT0",
    ):
        value = os.environ.get(key) or config.get(key)
        if value:
            passthrough[key] = value
    return passthrough
 def run_last30days(repo_dir: Path, topic: str, *, search: str, timeout_seconds: int, quick: bool, mock: bool, env: dict[str, str]) -> dict[str, Any]:
    engine = repo_dir / "skills" / "last30days" / "scripts" / "last30days.py"
    if not engine.exists():
        engine = repo_dir / "scripts" / "last30days.py"
    cmd = [sys.executable, str(engine), topic, "--emit=json"]
    if search:
        cmd.extend(["--search", search])
    if quick:
        cmd.append("--quick")
    if mock:
        cmd.append("--mock")
    result = subprocess.run(
        cmd,
        cwd=repo_dir,
        env=env,
        capture_output=True,
        text=True,
        timeout=timeout_seconds,
        check=False,
    )
    if result.returncode != 0:
        raise RuntimeError(f"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\n{result.stderr.strip()}")
    return json.loads(result.stdout)
 def create_worktree(rev: str) -> Path:
    worktree_dir = Path(tempfile.mkdtemp(prefix="last30days-eval-"))
    subprocess.run(
        ["git", "worktree", "add", "--detach", str(worktree_dir), rev],
        cwd=REPO_ROOT,
        check=True,
        capture_output=True,
        text=True,
    )
    return worktree_dir
 def resolve_repo_dir(label: str) -> tuple[Path, bool]:
    """Resolve a benchmark label into a repo directory and whether it is temporary."""
    if label == "WORKTREE":
        return REPO_ROOT, False
    return create_worktree(label), True
 def remove_worktree(path: Path) -> None:
    subprocess.run(
        ["git", "worktree", "remove", "--force", str(path)],
        cwd=REPO_ROOT,
        check=False,
        capture_output=True,
        text=True,
    )
    try:
        os.rmdir(path)
    except OSError:
        pass
 def summarize_topic(topic: str, query_type: str, baseline_report: dict[str, Any], candidate_report: dict[str, Any], judgments: dict[str, int], judged_pool: list[dict[str, Any]], limit: int) -> dict[str, Any]:
    baseline_ranked = build_ranked_items(baseline_report, limit)
    candidate_ranked = build_ranked_items(candidate_report, limit)
    baseline_sets = source_sets(baseline_report, limit)
    candidate_sets = source_sets(candidate_report, limit)
    overall_left = set().union(*baseline_sets.values()) if baseline_sets else set()
    overall_right = set().union(*candidate_sets.values()) if candidate_sets else set()
    sources = sorted(set(baseline_sets) | set(candidate_sets))
    return {
        "topic": topic,
        "query_type": query_type,
        "baseline": {
            "precision_at_5": precision_at_k(baseline_ranked, judgments, 5),
            "ndcg_at_5": ndcg_at_k(baseline_ranked, judgments, 5, judged_pool),
            "source_coverage_recall": source_coverage_recall(baseline_ranked, judged_pool, judgments),
        },
        "candidate": {
            "precision_at_5": precision_at_k(candidate_ranked, judgments, 5),
            "ndcg_at_5": ndcg_at_k(candidate_ranked, judgments, 5, judged_pool),
            "source_coverage_recall": source_coverage_recall(candidate_ranked, judged_pool, judgments),
        },
        "stability": {
            "overall_jaccard": jaccard(overall_left, overall_right),
            "overall_retention_vs_baseline": retention(overall_left, overall_right),
            "per_source": {
                source: {
                    "baseline_count": len(baseline_sets.get(source, set())),
                    "candidate_count": len(candidate_sets.get(source, set())),
                    "jaccard": jaccard(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
                    "retention_vs_baseline": retention(baseline_sets.get(source, set()), candidate_sets.get(source, set())),
                }
                for source in sources
            },
        },
    }
 def write_summary(output_dir: Path, baseline_label: str, candidate_label: str, summaries: list[dict[str, Any]]) -> None:
    output_dir.mkdir(parents=True, exist_ok=True)
    payload = {
        "generated_at": datetime.now().isoformat(timespec="seconds"),
        "baseline": baseline_label,
        "candidate": candidate_label,
        "topics": summaries,
    }
    (output_dir / "metrics.json").write_text(json.dumps(payload, indent=2))
    lines = [
        "# Search Quality Evaluation",
        "",
        f"- Baseline: `{baseline_label}`",
        f"- Candidate: `{candidate_label}`",
        f"- Generated: {payload['generated_at']}",
        "",
        "| Topic | Base P@5 | Cand P@5 | Base nDCG@5 | Cand nDCG@5 | Jaccard | Retention |",
        "|---|---:|---:|---:|---:|---:|---:|",
    ]
    for row in summaries:
        lines.append(
            "| {topic} | {bp:.2f} | {cp:.2f} | {bn:.2f} | {cn:.2f} | {jac:.2f} | {ret:.2f} |".format(
                topic=row["topic"],
                bp=row["baseline"]["precision_at_5"],
                cp=row["candidate"]["precision_at_5"],
                bn=row["baseline"]["ndcg_at_5"],
                cn=row["candidate"]["ndcg_at_5"],
                jac=row["stability"]["overall_jaccard"],
                ret=row["stability"]["overall_retention_vs_baseline"],
            )
        )
    (output_dir / "summary.md").write_text("\n".join(lines) + "\n")
 def write_failure_summary(
    output_dir: Path,
    baseline_label: str,
    candidate_label: str,
    summaries: list[dict[str, Any]],
    failures: list[dict[str, Any]],
 ) -> None:
    write_summary(output_dir, baseline_label, candidate_label, summaries)
    metrics_path = output_dir / "metrics.json"
    payload = json.loads(metrics_path.read_text()) if metrics_path.exists() else {
        "generated_at": datetime.now().isoformat(timespec="seconds"),
        "baseline": baseline_label,
        "candidate": candidate_label,
        "topics": [],
    }
    payload["failures"] = failures
    metrics_path.write_text(json.dumps(payload, indent=2))
    summary_path = output_dir / "summary.md"
    lines = summary_path.read_text().splitlines() if summary_path.exists() else ["# Search Quality Evaluation", ""]
    if failures:
        lines.extend([
            "",
            "## Failures",
            "",
        ])
        for failure in failures:
            lines.append(f"- `{failure['topic']}`: {failure['error']}")
    summary_path.write_text("\n".join(lines).rstrip() + "\n")
 def parse_topics_file(path: Path) -> list[tuple[str, str]]:
    rows = json.loads(path.read_text())
    return [(str(row["topic"]), str(row.get("query_type") or "general")) for row in rows]
 def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Compare two last30days revisions on ranked candidate quality")
    parser.add_argument("--baseline", default="HEAD~1")
    parser.add_argument("--candidate", default="WORKTREE")
    parser.add_argument("--search", default=DEFAULT_SEARCH)
    parser.add_argument("--output-dir", default="tmp/search-quality")
    parser.add_argument("--judge-model", default=DEFAULT_JUDGE_MODEL)
    parser.add_argument("--timeout", type=int, default=240)
    parser.add_argument("--limit", type=int, default=20)
    parser.add_argument("--mock", action="store_true")
    parser.add_argument("--quick", action="store_true")
    parser.add_argument("--topics-file")
    return parser
 def main() -> int:
    args = build_parser().parse_args()
    topics = parse_topics_file(Path(args.topics_file)) if args.topics_file else DEFAULT_TOPICS
    output_dir = Path(args.output_dir).resolve()
    config = envlib.get_config()
    gemini_api_key = resolve_google_judge_api_key(config)
    run_env = create_eval_env()
    baseline_dir, baseline_temp = resolve_repo_dir(args.baseline)
    candidate_dir, candidate_temp = resolve_repo_dir(args.candidate)
    try:
        summaries = []
        failures = []
        for topic, query_type in topics:
            try:
                baseline_report = run_last30days(
                    baseline_dir,
                    topic,
                    search=args.search,
                    timeout_seconds=args.timeout,
                    quick=args.quick,
                    mock=args.mock,
                    env=run_env,
                )
                candidate_report = run_last30days(
                    candidate_dir,
                    topic,
                    search=args.search,
                    timeout_seconds=args.timeout,
                    quick=args.quick,
                    mock=args.mock,
                    env=run_env,
                )
                judged_pool_map = {
                    item["key"]: item
                    for item in build_ranked_items(baseline_report, args.limit) + build_ranked_items(candidate_report, args.limit)
                }
                judged_pool = list(judged_pool_map.values())
                judgments = get_judgments(
                    output_dir=output_dir,
                    slug="".join(char.lower() if char.isalnum() else "-" for char in topic).strip("-"),
                    topic=topic,
                    query_type=query_type,
                    items=judged_pool,
                    judge_model=args.judge_model,
                    gemini_api_key=gemini_api_key,
                )
                summaries.append(summarize_topic(topic, query_type, baseline_report, candidate_report, judgments, judged_pool, args.limit))
            except Exception as exc:
                failures.append({"topic": topic, "query_type": query_type, "error": str(exc)})
        write_failure_summary(output_dir, args.baseline, args.candidate, summaries, failures)
    finally:
        if baseline_temp:
            remove_worktree(baseline_dir)
        if candidate_temp:
            remove_worktree(candidate_dir)
    result = {"output_dir": str(output_dir), "topics": len(topics), "failures": len(failures)}
    print(json.dumps(result, indent=2))
    return 1 if failures else 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,505 @@
 """Bird X search client for the v3.0.0 last30days pipeline.
 Uses a vendored subset of @steipete/bird v0.8.0 (MIT License) to search X
 via Twitter's GraphQL API. No external `bird` CLI binary needed - just Node.js.
 See scripts/lib/vendor/bird-search/package.json for authoritative version.
 """
 import json
 import os
 import shutil
 import sys
 import time
 from pathlib import Path
 from . import http, log, subproc
 from datetime import datetime
 from typing import Any, Dict, List, Optional, Tuple
 from .relevance import token_overlap_relevance as _compute_relevance
 # How many times to retry the bird-search subprocess when stdout is non-JSON
 # (typically an HTML anti-bot interstitial from Twitter's edge).
 MAX_JSON_DECODE_RETRIES = 2
 JSON_DECODE_RETRY_DELAY = 5.0  # seconds between retry attempts
 def _first_of(*values):
    """Return first value that is not None."""
    for v in values:
        if v is not None:
            return v
    return None
 # Path to the vendored bird-search wrapper
 _BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"
 # Depth configurations: number of results to request
 DEPTH_CONFIG = {
    "quick": 12,
    "default": 30,
    "deep": 60,
 }
 # Module-level credentials injected from .env config
 _credentials: Dict[str, str] = {}
 def set_credentials(auth_token: Optional[str], ct0: Optional[str]):
    """Inject AUTH_TOKEN/CT0 from .env config so Node subprocesses can use them."""
    if auth_token:
        _credentials['AUTH_TOKEN'] = auth_token
    if ct0:
        _credentials['CT0'] = ct0
 def _has_injected_credentials() -> bool:
    """Return True when both X session cookies were injected from config."""
    return bool(_credentials.get('AUTH_TOKEN') and _credentials.get('CT0'))
 def _has_process_credentials() -> bool:
    """Return True when AUTH_TOKEN/CT0 are present in process env."""
    return bool(os.environ.get("AUTH_TOKEN") and os.environ.get("CT0"))
 def _subprocess_env() -> Dict[str, str]:
    """Build env dict for Node subprocesses, merging injected credentials."""
    env = os.environ.copy()
    env.update(_credentials)
    # Hard-disable browser-cookie fallback so normal pipeline runs never hit
    # Safari/Chrome Keychain prompts during source detection or search.
    env["BIRD_DISABLE_BROWSER_COOKIES"] = "1"
    return env
 def _log(msg: str):
    log.source_log("Bird", msg, tty_only=False)
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for X search.
    X search is literal keyword AND matching — all words must appear.
    Aggressively strip question/meta/research words to keep only the
    core product/concept name (max 5 words).
    """
    from .query import extract_core_subject
    return extract_core_subject(topic, max_words=5, strip_suffixes=True)
 def is_bird_installed() -> bool:
    """Check if vendored Bird search module is available.
    Returns:
        True if bird-search.mjs exists and Node.js is in PATH.
    """
    if not _BIRD_SEARCH_MJS.exists():
        return False
    return shutil.which("node") is not None
 def is_bird_authenticated() -> Optional[str]:
    """Check if explicit X credentials are available.
    Returns:
        Auth source string if authenticated, None otherwise.
    """
    if not is_bird_installed():
        return None
    if _has_injected_credentials():
        return "env AUTH_TOKEN"
    if _has_process_credentials():
        return "env AUTH_TOKEN"
    return None
 def check_npm_available() -> bool:
    """Check if npm is available (kept for API compatibility).
    Returns:
        True if 'npm' command is available in PATH, False otherwise.
    """
    return shutil.which("npm") is not None
 def install_bird() -> Tuple[bool, str]:
    """No-op. Bird search is vendored in v3.0.0, no installation needed.
    Returns:
        Tuple of (success, message).
    """
    if is_bird_installed():
        return True, "Bird search is bundled with /last30days v3.0.0 - no installation needed."
    if not shutil.which("node"):
        return False, "Node.js 22+ is required for X search. Install Node.js first."
    return False, f"Vendored bird-search.mjs not found at {_BIRD_SEARCH_MJS}"
 def get_bird_status() -> Dict[str, Any]:
    """Get comprehensive Bird search status.
    Returns:
        Dict with keys: installed, authenticated, username, can_install
    """
    installed = is_bird_installed()
    auth_source = is_bird_authenticated() if installed else None
    return {
        "installed": installed,
        "authenticated": auth_source is not None,
        "username": auth_source,  # Now returns auth source (e.g., "Safari", "env AUTH_TOKEN")
        "can_install": True,  # Always vendored in v3.0.0
    }
 def _invoke_bird_subprocess(query: str, count: int, timeout: int):
    """Invoke the vendored bird-search.mjs subprocess once.
    Returns (result, error_dict). If error_dict is non-None, treat it as the
    final result and do not retry — those errors are terminal (timeout,
    spawn failure). If error_dict is None, the subprocess ran to completion
    and `result` is the SubprocResult; the caller decides whether to retry
    based on the result.stdout content.
    """
    cmd = [
        "node", str(_BIRD_SEARCH_MJS),
        query,
        "--count", str(count),
        "--json",
    ]
    pid_holder: list[int] = []
    def _register(pid: int) -> None:
        pid_holder.append(pid)
        try:
            from last30days import register_child_pid
            register_child_pid(pid)
        except ImportError:
            pass
    try:
        result = subproc.run_with_timeout(
            cmd,
            timeout=timeout,
            env=_subprocess_env(),
            on_pid=_register,
        )
    except subproc.SubprocTimeout:
        return None, {"error": f"Search timed out after {timeout}s", "items": []}
    except Exception as e:
        return None, {"error": str(e), "items": []}
    finally:
        if pid_holder:
            try:
                from last30days import unregister_child_pid
                unregister_child_pid(pid_holder[0])
            except Exception:
                pass
    return result, None
 def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
    """Run a search using the vendored bird-search.mjs module.
    Retries the subprocess on JSON-decode failure (typically a Twitter
    anti-bot HTML interstitial in stdout) up to MAX_JSON_DECODE_RETRIES
    times with JSON_DECODE_RETRY_DELAY seconds between attempts. Terminal
    errors (subprocess timeout, non-zero return code) are returned
    immediately without retry.
    Args:
        query: Full search query string (including since: filter)
        count: Number of results to request
        timeout: Timeout in seconds (per attempt)
    Returns:
        Raw Bird JSON response or error dict.
    """
    last_decode_error: Optional[str] = None
    for attempt in range(MAX_JSON_DECODE_RETRIES):
        result, terminal_error = _invoke_bird_subprocess(query, count, timeout)
        if terminal_error is not None:
            return terminal_error
        if result.returncode != 0:
            error = result.stderr.strip() or "Bird search failed"
            return {"error": error, "items": []}
        output = result.stdout.strip()
        if not output:
            return {"items": []}
        try:
            parsed = json.loads(output)
        except json.JSONDecodeError as e:
            # Twitter's edge sometimes serves an HTML anti-bot interstitial
            # in place of JSON. Tag the failure shape so it's distinguishable
            # from "no results" in logs, then retry the subprocess.
            looks_html = output.lstrip().lower().startswith(("<!doctype", "<html", "<"))
            attempt_num = attempt + 1
            log_msg = (
                f"Bird search returned non-JSON stdout "
                f"(looks_html={looks_html}, attempt {attempt_num}/{MAX_JSON_DECODE_RETRIES}, "
                f"first 80 chars: {output[:80]!r})"
            )
            last_decode_error = str(e)
            if attempt_num < MAX_JSON_DECODE_RETRIES:
                log.source_log(
                    "X/bird",
                    f"{log_msg}; retrying in {JSON_DECODE_RETRY_DELAY:.0f}s",
                )
                time.sleep(JSON_DECODE_RETRY_DELAY)
                continue
            log.source_log("X/bird", log_msg)
            return {
                "error": (
                    f"Invalid JSON response after {MAX_JSON_DECODE_RETRIES} attempts "
                    f"(likely Twitter anti-bot interstitial): {e}"
                ),
                "items": [],
            }
        if isinstance(parsed, list):
            return {"items": parsed}
        return parsed
    # Defensive fallthrough — loop should always return above.
    return {
        "error": f"Bird search exhausted retries: {last_decode_error}",
        "items": [],
    }
 def search_x(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
 ) -> Dict[str, Any]:
    """Search X using Bird CLI with automatic retry on 0 results.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD) - unused but kept for API compatibility
        depth: Research depth - "quick", "default", or "deep"
    Returns:
        Raw Bird JSON response or error dict.
    """
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    timeout = 30 if depth == "quick" else 45 if depth == "default" else 60
    # Extract core subject - X search is literal, not semantic
    core_topic = _extract_core_subject(topic)
    query = f"{core_topic} since:{from_date}"
    _log(f"Searching: {query}")
    response = _run_bird_search(query, count, timeout)
    # Check if we got results
    items = parse_bird_response(response, query=core_topic)
    # Retry with OR groups for multi-word queries (X supports OR operator)
    core_words = core_topic.split()
    if not items and len(core_words) >= 2:
        from .query import extract_compound_terms
        compounds = extract_compound_terms(topic)
        if compounds:
            # Build OR-group query: ("multi-agent" OR "agent simulation") since:DATE
            or_parts = ' OR '.join(f'"{t}"' for t in compounds[:3])
            _log(f"0 results for '{core_topic}', retrying with OR groups: {or_parts}")
            query = f"({or_parts}) since:{from_date}"
            response = _run_bird_search(query, count, timeout)
            items = parse_bird_response(response, query=core_topic)
    # Retry with fewer keywords if still 0 results and query has 3+ words
    if not items and len(core_words) > 2:
        shorter = ' '.join(core_words[:2])
        _log(f"0 results for '{core_topic}', retrying with '{shorter}'")
        query = f"{shorter} since:{from_date}"
        response = _run_bird_search(query, count, timeout)
        items = parse_bird_response(response, query=core_topic)
    # Last-chance retry: use strongest remaining token (often the product name)
    if not items and core_words:
        low_signal = {
            'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
            'best', 'top', 'latest', 'new', 'plugin', 'plugins',
            'skill', 'skills', 'tool', 'tools',
        }
        candidates = [w for w in core_words if w not in low_signal]
        if candidates:
            strongest = max(candidates, key=len)
            _log(f"0 results for '{core_topic}', retrying with strongest token '{strongest}'")
            query = f"{strongest} since:{from_date}"
            response = _run_bird_search(query, count, timeout)
    return response
 def search_handles(
    handles: List[str],
    topic: Optional[str],
    from_date: str,
    count_per: int = 5,
 ) -> List[Dict[str, Any]]:
    """Search specific X handles for topic-related content.
    Runs targeted Bird searches using `from:handle topic` syntax.
    Used in Phase 2 supplemental search after entity extraction.
    Args:
        handles: List of X handles to search (without @)
        topic: Search topic (core subject), or None for unfiltered search
        from_date: Start date (YYYY-MM-DD)
        count_per: Results to request per handle
    Returns:
        List of raw item dicts (same format as parse_bird_response output).
    """
    core_topic = _extract_core_subject(topic) if topic else None
    def _search_one_handle(handle: str) -> List[Dict[str, Any]]:
        handle = handle.lstrip("@")
        if core_topic:
            query = f"from:{handle} {core_topic} since:{from_date}"
        else:
            query = f"from:{handle} since:{from_date}"
        cmd = [
            "node", str(_BIRD_SEARCH_MJS),
            query,
            "--count", str(count_per),
            "--json",
        ]
        try:
            result = subproc.run_with_timeout(cmd, timeout=15, env=_subprocess_env())
        except subproc.SubprocTimeout:
            _log(f"Handle search timed out for @{handle}")
            return []
        except OSError as e:
            _log(f"Handle search error for @{handle}: {e}")
            return []
        if result.returncode != 0:
            _log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
            return []
        output = result.stdout.strip()
        if not output:
            return []
        try:
            response = json.loads(output)
        except json.JSONDecodeError:
            _log(f"Invalid JSON from handle search for @{handle}")
            return []
        return parse_bird_response(response, query=core_topic)
    from concurrent.futures import ThreadPoolExecutor, as_completed
    all_items: List[Dict[str, Any]] = []
    with ThreadPoolExecutor(max_workers=min(5, len(handles))) as executor:
        futures = {executor.submit(_search_one_handle, h): h for h in handles}
        for future in as_completed(futures):
            all_items.extend(future.result())
    return all_items
 def parse_bird_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
    """Parse Bird response to match xai_x output format.
    Args:
        response: Raw Bird JSON response
        query: Original search query for relevance scoring
    Returns:
        List of normalized item dicts matching xai_x.parse_x_response() format.
    """
    items = []
    # Check for errors
    if "error" in response and response["error"]:
        _log(f"Bird error: {response['error']}")
        return items
    # Bird returns a list of tweets directly or under a key
    raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))
    if not isinstance(raw_items, list):
        return items
    for i, tweet in enumerate(raw_items):
        if not isinstance(tweet, dict):
            continue
        # Extract URL - Bird uses permanent_url or we construct from id
        url = tweet.get("permanent_url") or tweet.get("url", "")
        if not url and tweet.get("id"):
            # Try different field structures Bird might use
            author = tweet.get("author", {}) or tweet.get("user", {})
            screen_name = author.get("username") or author.get("screen_name", "")
            if screen_name:
                url = f"https://x.com/{screen_name}/status/{tweet['id']}"
        if not url:
            continue
        # Parse date from created_at/createdAt (e.g., "Wed Jan 15 14:30:00 +0000 2026")
        date = None
        created_at = tweet.get("createdAt") or tweet.get("created_at", "")
        if created_at:
            try:
                # Try ISO format first (e.g., "2026-02-03T22:33:32Z")
                # Check for ISO date separator, not just "T" (which appears in "Tue")
                if len(created_at) > 10 and created_at[10] == "T":
                    dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
                else:
                    # Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
                    dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
                date = dt.strftime("%Y-%m-%d")
            except (ValueError, TypeError):
                pass
        # Extract user info (Bird uses author.username, older format uses user.screen_name)
        author = tweet.get("author", {}) or tweet.get("user", {})
        author_handle = author.get("username") or author.get("screen_name", "") or tweet.get("author_handle", "")
        # Build engagement dict (Bird uses camelCase: likeCount, retweetCount, etc.)
        engagement = {
            "likes": _first_of(tweet.get("likeCount"), tweet.get("like_count"), tweet.get("favorite_count")),
            "reposts": _first_of(tweet.get("retweetCount"), tweet.get("retweet_count")),
            "replies": _first_of(tweet.get("replyCount"), tweet.get("reply_count")),
            "quotes": _first_of(tweet.get("quoteCount"), tweet.get("quote_count")),
        }
        # Convert to int where possible
        for key in engagement:
            if engagement[key] is not None:
                try:
                    engagement[key] = int(engagement[key])
                except (ValueError, TypeError):
                    engagement[key] = None
        # Build normalized item
        item = {
            "id": f"X{i+1}",
            "text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
            "url": url,
            "author_handle": author_handle.lstrip("@"),
            "date": date,
            "engagement": engagement if any(v is not None for v in engagement.values()) else None,
            "why_relevant": "",  # Bird doesn't provide relevance explanations
            "relevance": _compute_relevance(query, str(tweet.get("text", ""))) if query else 0.7,
        }
        items.append(item)
    return items
@@ -0,0 +1,329 @@
 """Bluesky search via AT Protocol (requires app password).
 Uses bsky.social for auth and api.bsky.app for post search (the canonical
 authenticated AppView). The previous default `public.api.bsky.app` is the
 unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
 regardless of auth header (verified 2026-05-04). Override the search host
 via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
 Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
 19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
 The createSession endpoint accepts main-account passwords too, but they're
 bad hygiene (no scope, can't revoke individually).
 """
 import math
 import os
 import re
 import sys
 import time
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional
 from . import http, log
 BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
 _DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
 def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
    """Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
    Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
    .env file. The project's env.py loads .env into config but not into
    os.environ, so check both — same hybrid pattern as last30days.py for
    LAST30DAYS_STORE.
    Hardens user-supplied host values against three common mis-configurations:
    whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
    "api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
    embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
    we log a warning and fall back to the default rather than building an
    invalid URL with an opaque downstream error.
    """
    config = config or {}
    raw = (
        os.environ.get("BSKY_SEARCH_HOST")
        or config.get("BSKY_SEARCH_HOST")
        or _DEFAULT_BSKY_SEARCH_HOST
    )
    host = raw.strip().rstrip("/")
    # Strip embedded scheme so users who paste full URLs do not break the f-string.
    for prefix in ("https://", "http://"):
        if host.lower().startswith(prefix):
            host = host[len(prefix):]
            break
    if not host or "/" in host or " " in host:
        # Embedded path or whitespace remains — don't trust it. Default + log.
        if raw != _DEFAULT_BSKY_SEARCH_HOST:
            _log(
                f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
                f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
            )
        host = _DEFAULT_BSKY_SEARCH_HOST
    return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
 # App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
 # with three hyphens at fixed positions).
 _APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
 def _validate_app_password_format(value) -> bool:
    """Return True if value matches Bluesky's 19-char app-password format.
    False for non-strings (None, int, list) so callers passing config dict
    values directly don't crash. Detect-but-not-gate: the createSession
    endpoint also accepts main-account passwords, so failing this check is
    a hygiene smell, not a hard error.
    """
    if not isinstance(value, str):
        return False
    return bool(_APP_PASSWORD_RE.fullmatch(value))
 DEPTH_CONFIG = {
    "quick": 15,
    "default": 30,
    "deep": 60,
 }
 # Module-level token cache (valid for the lifetime of a single research run)
 _cached_token: Optional[str] = None
 _token_created_at: float = 0.0
 _session_error: Optional[str] = None
 _TOKEN_MAX_AGE_SECONDS = 5400  # 90 minutes (conservative, tokens last ~2 hours)
 def _log(msg: str):
    log.source_log("Bluesky", msg)
 def _create_session(handle: str, app_password: str) -> Optional[str]:
    """Create an AT Protocol session and return the access token.
    Args:
        handle: Bluesky handle (e.g. user.bsky.social)
        app_password: App password from bsky.app/settings/app-passwords
    Returns:
        Access JWT string, or None on failure. Sets _session_error on failure.
    """
    global _cached_token, _token_created_at, _session_error
    if _cached_token and (time.monotonic() - _token_created_at < _TOKEN_MAX_AGE_SECONDS):
        return _cached_token
    if _cached_token:
        _log("Session token expired, re-authenticating")
        _cached_token = None
        _token_created_at = 0.0
    try:
        response = http.request(
            "POST",
            BSKY_SESSION_URL,
            json_data={"identifier": handle, "password": app_password},
            timeout=15,
        )
        token = response.get("accessJwt")
        if token:
            _cached_token = token
            _token_created_at = time.monotonic()
            _session_error = None
            _log("Session created successfully")
            return token
        _log("No accessJwt in session response")
        _session_error = "No accessJwt in session response"
        return None
    except http.HTTPError as e:
        if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
            _session_error = "Cloudflare blocked the request (403 Forbidden). This is a network-level block, not an auth issue. Try a different network or VPN."
        elif e.status_code == 401:
            _session_error = "Invalid credentials (401 Unauthorized). Check BSKY_HANDLE and BSKY_APP_PASSWORD."
        else:
            _session_error = f"Session request failed: {e}"
        _log(f"Session creation failed: {_session_error}")
        return None
    except Exception as e:
        _session_error = f"Session request failed: {type(e).__name__}: {e}"
        _log(f"Session creation failed: {_session_error}")
        return None
 def _reset_session_cache() -> None:
    global _cached_token, _token_created_at, _session_error
    _cached_token = None
    _token_created_at = 0.0
    _session_error = None
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for Bluesky search."""
    from .query import extract_core_subject
    _BSKY_NOISE = frozenset({
        'best', 'top', 'good', 'great', 'awesome',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features', 'recommendations', 'advice',
    })
    return extract_core_subject(topic, noise=_BSKY_NOISE)
 def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    """Parse date from Bluesky post to YYYY-MM-DD.
    AT Protocol uses ISO 8601 format in indexedAt and createdAt fields.
    """
    for key in ("indexedAt", "createdAt"):
        val = item.get(key)
        if val and isinstance(val, str):
            try:
                dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
                return dt.strftime("%Y-%m-%d")
            except (ValueError, TypeError):
                pass
    return None
 def search_bluesky(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    config: Optional[Dict[str, Any]] = None,
 ) -> Dict[str, Any]:
    """Search Bluesky via AT Protocol API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        config: Config dict with BSKY_HANDLE and BSKY_APP_PASSWORD
    Returns:
        Dict with 'posts' list from AT Protocol response.
    """
    config = config or {}
    handle = config.get("BSKY_HANDLE", "")
    app_password = config.get("BSKY_APP_PASSWORD", "")
    if not handle or not app_password:
        return {"posts": [], "error": "Bluesky credentials not configured"}
    # One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
    # form. createSession accepts main-account passwords too — but main
    # passwords have no scope (full account access), can't be revoked
    # individually, and rotating them breaks every service that holds them.
    # We warn but do not gate, matching the project's detect-don't-block
    # philosophy elsewhere.
    if not _validate_app_password_format(app_password):
        _log(
            "BSKY_APP_PASSWORD does not look like an app password "
            "(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
            "account password — those work but are bad hygiene. Generate "
            "an app password at https://bsky.app/settings/app-passwords"
        )
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
    _log(f"Searching for '{core_topic}' (depth={depth}, limit={count})")
    from urllib.parse import urlencode
    params = {
        "q": core_topic,
        "limit": str(min(count, 100)),
        "sort": "top",
    }
    url = f"{_resolve_search_url(config)}?{urlencode(params)}"
    def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
        token = _create_session(handle, app_password)
        if not token:
            error_msg = _session_error or "Bluesky session creation failed (unknown error)"
            return None, error_msg
        try:
            response = http.request(
                "GET", url,
                headers={"Authorization": f"Bearer {token}"},
                timeout=30,
            )
            return response, None
        except http.HTTPError as e:
            _log(f"Search failed: {e}")
            if e.status_code == 401:
                _reset_session_cache()
                return None, "refresh"
            if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
                return None, "Bluesky search blocked by Cloudflare (403). This is a network-level block - try a different network or VPN."
            return None, f"Bluesky search failed: {e}"
        except Exception as e:
            _log(f"Search failed: {e}")
            return None, f"Bluesky search failed: {type(e).__name__}: {e}"
    response, error_msg = _auth_and_search()
    if error_msg == "refresh":
        _log("Session expired; recreating token and retrying once")
        response, error_msg = _auth_and_search()
    if error_msg:
        return {"posts": [], "error": error_msg}
    if response is None:
        return {"posts": [], "error": "Bluesky search failed (unknown error)"}
    posts = response.get("posts", [])
    _log(f"Found {len(posts)} posts")
    return response
 def parse_bluesky_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse AT Protocol response into normalized item dicts.
    Returns:
        List of item dicts ready for normalization.
    """
    posts = response.get("posts", [])
    items = []
    for i, post in enumerate(posts):
        record = post.get("record") or {}
        text = record.get("text") or ""
        author = post.get("author") or {}
        handle = author.get("handle") or ""
        display_name = author.get("displayName") or handle
        # Post URI -> URL
        # URI format: at://did:plc:xxx/app.bsky.feed.post/rkey
        uri = post.get("uri") or ""
        rkey = uri.rsplit("/", 1)[-1] if uri else ""
        url = f"https://bsky.app/profile/{handle}/post/{rkey}" if handle and rkey else ""
        likes = post.get("likeCount") or 0
        reposts = post.get("repostCount") or 0
        replies = post.get("replyCount") or 0
        quotes = post.get("quoteCount") or 0
        date_str = _parse_date(post) or _parse_date(record)
        # Relevance: position-based (AT Protocol sorts by relevance with sort=top)
        rank_score = max(0.3, 1.0 - (i * 0.02))
        engagement_boost = min(0.2, math.log1p(likes + reposts) / 40)
        relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
        items.append({
            "handle": handle,
            "display_name": display_name,
            "text": text,
            "url": url,
            "date": date_str,
            "engagement": {
                "likes": likes,
                "reposts": reposts,
                "replies": replies,
                "quotes": quotes,
            },
            "relevance": round(relevance, 2),
            "why_relevant": f"Bluesky: @{handle}: {text[:60]}" if text else f"Bluesky: {handle}",
        })
    return items
@@ -0,0 +1,283 @@
 """Category-peer subreddit map for Step 0.55 community resolution.
 When a topic is a product in a known category (AI image generation, AI coding
 agents, SaaS screen recording, etc.), brand-specific subreddits returned by
 WebSearch are insufficient: cross-product technique discussion lives in
 category-peer subs. This module classifies a topic into a category by matching
 compound-term patterns against the lowercased topic string, then returns the
 priority-ordered peer subreddit list for that category.
 The map is intentionally small, curated, and code-reviewed. Adding a new
 category is a code change; there is no user-editable override surface.
 False-positive guard: every pattern is either a multi-word compound (e.g.
 "image generation", "text to image") or a domain-specific single word
 (e.g. "midjourney", "stablediffusion"). Bare common nouns like "image",
 "ai", or "model" are never used as patterns.
 First-match-wins: categories are evaluated in declared order. Entries are
 sorted from most-specific to least-specific so narrower categories claim a
 topic before broader ones. For example, `ai_image_generation` appears
 before `ai_chat_model` so "gpt image 2" matches the image-gen category.
 """
 from __future__ import annotations
 from typing import List, Optional, TypedDict
 class _CategoryEntry(TypedDict):
    patterns: List[str]
    peer_subs: List[str]
 CATEGORY_PEERS: dict[str, _CategoryEntry] = {
    "ai_image_generation": {
        "patterns": [
            "image generation",
            "image gen",
            "text to image",
            "text-to-image",
            "gpt image",
            "gpt-image",
            "nano banana",
            "midjourney",
            "stable diffusion",
            "stablediffusion",
            "dall-e",
            "dalle",
            "flux.1",
            "flux schnell",
            "imagen",
            "seedance",
            "ideogram",
            "recraft",
        ],
        "peer_subs": [
            "StableDiffusion",
            "midjourney",
            "dalle2",
            "aiArt",
            "PromptEngineering",
            "MediaSynthesis",
        ],
    },
    "ai_video_generation": {
        "patterns": [
            "video generation",
            "text to video",
            "text-to-video",
            "sora",
            "veo 3",
            "veo3",
            "runway gen",
            "kling",
            "pika labs",
            "luma dream machine",
            "hailuo",
        ],
        "peer_subs": [
            "aivideo",
            "StableDiffusion",
            "runwayml",
            "singularity",
            "MediaSynthesis",
        ],
    },
    "ai_music_generation": {
        "patterns": [
            "music generation",
            "ai music",
            "suno",
            "udio",
            "riffusion",
            "stable audio",
        ],
        "peer_subs": [
            "SunoAI",
            "udiomusic",
            "aimusic",
            "artificial",
        ],
    },
    "ai_coding_agent": {
        "patterns": [
            "claude code",
            "cursor ide",
            "github copilot",
            "windsurf",
            "aider",
            "cline",
            "openclaw",
            "hermes agent",
            "continue.dev",
            "codeium",
            "sweep ai",
            "devin ai",
            "coding agent",
            "coding assistant",
        ],
        "peer_subs": [
            "ChatGPTCoding",
            "LocalLLaMA",
            "singularity",
            "PromptEngineering",
        ],
    },
    "ai_agent_framework": {
        "patterns": [
            "agent framework",
            "agentic framework",
            "langchain",
            "langgraph",
            "crewai",
            "autogen",
            "llamaindex",
            "dspy",
            "smolagents",
        ],
        "peer_subs": [
            "LangChain",
            "LocalLLaMA",
            "AI_Agents",
            "MachineLearning",
        ],
    },
    "ai_chat_model": {
        "patterns": [
            "gpt-5",
            "gpt-4",
            "claude opus",
            "claude sonnet",
            "claude haiku",
            "gemini pro",
            "gemini flash",
            "llama 3",
            "llama 4",
            "deepseek",
            "qwen",
            "mistral large",
            "grok",
        ],
        "peer_subs": [
            "LocalLLaMA",
            "ChatGPT",
            "ClaudeAI",
            "singularity",
            "artificial",
        ],
    },
    "saas_screen_recording": {
        "patterns": [
            "screen recording",
            "screen recorder",
            "loom video",
            "tella screen",
            "vidyard",
            "screen capture tool",
        ],
        "peer_subs": [
            "SaaS",
            "screenrecording",
            "productivity",
            "Entrepreneur",
        ],
    },
    "saas_productivity": {
        "patterns": [
            "notion app",
            "obsidian plugin",
            "obsidian app",
            "linear app",
            "asana",
            "clickup",
            "productivity app",
        ],
        "peer_subs": [
            "productivity",
            "SaaS",
            "ObsidianMD",
            "Notion",
        ],
    },
    "prediction_markets": {
        "patterns": [
            "polymarket",
            "kalshi",
            "prediction market",
            "event contracts",
            "manifold markets",
        ],
        "peer_subs": [
            "Polymarket",
            "Kalshi",
            "predictionmarkets",
        ],
    },
    "crypto_defi": {
        "patterns": [
            "defi protocol",
            "yield farming",
            "liquidity pool",
            "stablecoin",
            "ethereum layer",
            "layer 2",
            "l2 rollup",
        ],
        "peer_subs": [
            "defi",
            "ethfinance",
            "CryptoCurrency",
            "ethereum",
        ],
    },
    "dev_tool_cli": {
        "patterns": [
            "cli tool",
            "command line tool",
            "terminal app",
            "dev tool",
        ],
        "peer_subs": [
            "commandline",
            "programming",
            "webdev",
        ],
    },
 }
 def detect_category(topic: Optional[str]) -> Optional[str]:
    """Classify a topic into a known category by compound-term match.
    Returns the category id (e.g. "ai_image_generation") or None if no
    category's patterns match. Matching is case-insensitive substring over
    the lowercased topic. Declaration order wins (first-match-wins), so the
    map is ordered from most-specific to least-specific.
    A None or empty topic returns None. Classification never raises on
    normal string inputs; callers do not need to wrap in try/except for
    typical paths, though defensive callers may.
    """
    if not topic:
        return None
    lowered = topic.lower()
    for category_id, entry in CATEGORY_PEERS.items():
        for pattern in entry["patterns"]:
            if pattern in lowered:
                return category_id
    return None
 def peer_subs_for(category_id: Optional[str]) -> List[str]:
    """Return the priority-ordered peer subreddit list for a category.
    Returns an empty list for None or unknown category ids. The returned
    list is a fresh copy; callers may safely mutate it.
    """
    if not category_id:
        return []
    entry = CATEGORY_PEERS.get(category_id)
    if not entry:
        return []
    return list(entry["peer_subs"])
@@ -0,0 +1,322 @@
 """Chrome and Brave cookie extraction for macOS.
 Extracts cookies from Chromium-based browser SQLite databases using only
 stdlib modules and the system openssl CLI (ships with macOS). Zero pip
 dependencies.
 Chromium on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
 Chrome and Brave share the same algorithm; only the DB path and Keychain
 service name differ.
 This is NOT affected by Windows App-Bound Encryption (v20).
 """
 import hashlib
 import logging
 import shutil
 import sqlite3
 import subprocess
 import tempfile
 from pathlib import Path
 from typing import Optional
 logger = logging.getLogger(__name__)
 # Cookie DB locations on macOS
 CHROME_COOKIES_DB = Path.home() / "Library" / "Application Support" / "Google" / "Chrome" / "Default" / "Cookies"
 BRAVE_BASE_DIR = Path.home() / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser"
 # Chromium v10 encryption constants (shared by Chrome and Brave)
 CHROME_SALT = b"saltysalt"
 CHROME_PBKDF2_ITERATIONS = 1003
 CHROME_KEY_LENGTH = 16
 # IV is 16 space characters (0x20)
 CHROME_IV_HEX = "20" * 16
 def _get_chromium_encryption_key(service_name: str) -> Optional[bytes]:
    """Retrieve the encryption passphrase for a Chromium-based browser from macOS Keychain.
    Calls `security find-generic-password` which may trigger a system dialog
    on first access.
    Returns the raw passphrase bytes, or None on failure.
    """
    try:
        result = subprocess.run(
            ["security", "find-generic-password", "-w", "-s", service_name],
            capture_output=True,
            text=True,
            timeout=10,
        )
        if result.returncode != 0:
            logger.info("%s Keychain access denied or browser not installed: %s", service_name, result.stderr.strip())
            return None
        passphrase = result.stdout.strip()
        if not passphrase:
            logger.info("%s Keychain returned empty passphrase", service_name)
            return None
        return passphrase.encode("utf-8")
    except FileNotFoundError:
        logger.info("'security' command not found — not on macOS?")
        return None
    except subprocess.TimeoutExpired:
        logger.info("%s Keychain access timed out", service_name)
        return None
    except Exception as e:
        logger.info("Failed to get %s encryption key: %s", service_name, e)
        return None
 def _get_chrome_encryption_key() -> Optional[bytes]:
    return _get_chromium_encryption_key("Chrome Safe Storage")
 def _derive_aes_key(passphrase: bytes) -> bytes:
    """Derive 16-byte AES key from Chrome's Keychain passphrase via PBKDF2."""
    return hashlib.pbkdf2_hmac(
        "sha1",
        passphrase,
        CHROME_SALT,
        CHROME_PBKDF2_ITERATIONS,
        dklen=CHROME_KEY_LENGTH,
    )
 def _decrypt_v10_value(encrypted_value: bytes, aes_key: bytes, db_version: int) -> Optional[str]:
    """Decrypt a Chrome v10-encrypted cookie value.
    Uses system openssl CLI for AES-128-CBC decryption (zero pip deps).
    For Chrome 130+ (db_version >= 24), strips 32-byte SHA-256 prefix after decryption.
    Returns decrypted string or None on failure.
    """
    # Strip the 'v10' prefix
    ciphertext = encrypted_value[3:]
    if not ciphertext:
        return None
    hex_key = aes_key.hex()
    try:
        result = subprocess.run(
            [
                "openssl", "enc", "-aes-128-cbc", "-d",
                "-K", hex_key,
                "-iv", CHROME_IV_HEX,
                "-nopad",
            ],
            input=ciphertext,
            capture_output=True,
            timeout=5,
        )
        if result.returncode != 0:
            logger.debug("openssl decryption failed: %s", result.stderr.decode(errors="replace").strip())
            return None
        decrypted = result.stdout
        if not decrypted:
            return None
        # Remove PKCS7 padding
        decrypted = _remove_pkcs7_padding(decrypted)
        if decrypted is None:
            return None
        # Chrome 130+ (db version >= 24): strip 32-byte SHA-256 prefix
        if db_version >= 24 and len(decrypted) > 32:
            decrypted = decrypted[32:]
        return decrypted.decode("utf-8", errors="replace")
    except FileNotFoundError:
        logger.info("openssl not found — cannot decrypt Chrome cookies")
        return None
    except subprocess.TimeoutExpired:
        logger.info("openssl decryption timed out")
        return None
    except Exception as e:
        logger.debug("Chrome cookie decryption error: %s", e)
        return None
 def _remove_pkcs7_padding(data: bytes) -> Optional[bytes]:
    """Remove PKCS7 padding from decrypted data.
    The last byte indicates the number of padding bytes added.
    All padding bytes must have the same value.
    Returns unpadded data or None if padding is invalid.
    """
    if not data:
        return None
    pad_len = data[-1]
    if pad_len < 1 or pad_len > 16:
        return None
    # Verify all padding bytes match
    if data[-pad_len:] != bytes([pad_len]) * pad_len:
        return None
    return data[:-pad_len]
 def _get_db_version(cursor: sqlite3.Cursor) -> int:
    """Get Chrome cookie database version from the meta table.
    Returns 0 if meta table doesn't exist or version can't be read.
    """
    try:
        cursor.execute("SELECT value FROM meta WHERE key = 'version'")
        row = cursor.fetchone()
        if row:
            return int(row[0])
    except Exception:
        pass
    return 0
 def _extract_chromium_cookies_macos(
    db_path: Path,
    keychain_service: str,
    domain: str,
    cookie_names: list[str],
 ) -> Optional[dict[str, str]]:
    """Extract cookies from any Chromium-based browser on macOS.
    Copies the locked Cookies database to a temp file, reads specified cookies,
    and decrypts v10-encrypted values using the Keychain-stored key.
    Args:
        db_path: Path to the browser's Cookies SQLite file.
        keychain_service: macOS Keychain service name (e.g. "Chrome Safe Storage").
        domain: Cookie domain to match (e.g., ".twitter.com", ".x.com").
        cookie_names: List of cookie names to extract.
    Returns:
        Dict mapping cookie name to decrypted value, or None on failure.
        Only includes cookies that were successfully found and decrypted.
    """
    if not db_path.exists():
        logger.info("%s cookies database not found at %s", keychain_service, db_path)
        return None
    passphrase = _get_chromium_encryption_key(keychain_service)
    aes_key = _derive_aes_key(passphrase) if passphrase else None
    # Copy DB to temp file (browser locks the original while running)
    tmp_fd = None
    tmp_path = None
    try:
        tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
        shutil.copy2(str(db_path), tmp_path)
    except Exception as e:
        logger.info("Failed to copy %s cookies database: %s", keychain_service, e)
        if tmp_path:
            try:
                Path(tmp_path).unlink(missing_ok=True)
            except Exception:
                pass
        return None
    finally:
        if tmp_fd is not None:
            import os
            os.close(tmp_fd)
    try:
        conn = sqlite3.connect(tmp_path)
        cursor = conn.cursor()
        db_version = _get_db_version(cursor)
        logger.debug("%s cookie DB version: %d", keychain_service, db_version)
        placeholders = ",".join("?" for _ in cookie_names)
        query = (
            f"SELECT name, value, encrypted_value FROM cookies "
            f"WHERE host_key LIKE ? AND name IN ({placeholders})"
        )
        params = [f"%{domain}"] + list(cookie_names)
        cursor.execute(query, params)
        results: dict[str, str] = {}
        for name, value, encrypted_value in cursor.fetchall():
            if value:
                results[name] = value
                continue
            if encrypted_value and encrypted_value[:3] == b"v10":
                if aes_key is None:
                    logger.debug("Skipping encrypted cookie %s — no Keychain access", name)
                    continue
                decrypted = _decrypt_v10_value(encrypted_value, aes_key, db_version)
                if decrypted:
                    results[name] = decrypted
                else:
                    logger.debug("Failed to decrypt cookie %s", name)
            elif encrypted_value:
                logger.debug("Unknown encryption for cookie %s (prefix: %r)", name, encrypted_value[:3])
        conn.close()
        if not results:
            logger.info("No matching cookies found in %s for domain %s", keychain_service, domain)
            return None
        return results
    except sqlite3.Error as e:
        logger.info("Failed to read %s cookies database: %s", keychain_service, e)
        return None
    except Exception as e:
        logger.info("Unexpected error reading %s cookies: %s", keychain_service, e)
        return None
    finally:
        try:
            Path(tmp_path).unlink(missing_ok=True)
        except Exception:
            pass
 def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
    """Extract cookies from Chrome on macOS."""
    return _extract_chromium_cookies_macos(
        CHROME_COOKIES_DB, "Chrome Safe Storage", domain, cookie_names
    )
 def _find_brave_cookies_db() -> Optional[Path]:
    """Find Brave's Cookies database on macOS.
    Tries the Default profile first, then scans numbered Profile directories
    by most-recently-modified. Brave creates extra profiles as "Profile 1",
    "Profile 2", etc. alongside Default; the most recently used one is the
    likeliest to hold current cookies. Lexicographic sort would visit
    "Profile 10" before "Profile 2", which can return the wrong profile.
    """
    default = BRAVE_BASE_DIR / "Default" / "Cookies"
    if default.exists():
        return default
    try:
        candidates = [
            child for child in BRAVE_BASE_DIR.iterdir()
            if child.is_dir() and child.name.startswith("Profile ")
        ]
        for child in sorted(candidates, key=lambda p: p.stat().st_mtime, reverse=True):
            candidate = child / "Cookies"
            if candidate.exists():
                return candidate
    except OSError:
        pass
    return None
 def extract_brave_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
    """Extract cookies from Brave on macOS.
    Brave uses the same v10 AES-128-CBC encryption as Chrome; only the DB
    path and Keychain service name differ.
    """
    db_path = _find_brave_cookies_db()
    if db_path is None:
        logger.info("Brave cookies database not found under %s", BRAVE_BASE_DIR)
        return None
    return _extract_chromium_cookies_macos(db_path, "Brave Safe Storage", domain, cookie_names)
@@ -0,0 +1,271 @@
 """Candidate clustering and representative selection."""
 from __future__ import annotations
 import re
 from . import dedupe, schema
 CLUSTERABLE_INTENTS = {"breaking_news", "opinion", "comparison", "prediction"}
 # Words too common to signal shared topic between clusters.
 _ENTITY_STOPWORDS = frozenset({
    "the", "a", "an", "to", "for", "how", "is", "in", "of", "on", "and",
    "with", "from", "by", "at", "this", "that", "it", "what", "are", "do",
    "can", "his", "her", "he", "she", "its", "was", "has", "new", "just",
    "says", "said", "will", "about", "after", "now", "all", "been", "here",
    "not", "out", "up", "more", "also", "but", "who", "year", "first",
    "make", "being", "making", "over", "into", "than", "they", "their",
    "would", "could", "get", "got", "some", "like", "back", "going",
    "breaking", "https", "http", "www", "com",
 })
 def _candidate_text(candidate: schema.Candidate) -> str:
    return " ".join(part for part in [candidate.title, candidate.snippet] if part).strip()
 def _extract_entities(text: str) -> set[str]:
    """Extract significant words (proper nouns, numbers, capitalized words) from text.
    Used for cross-source cluster merging where phrasing differs but entities overlap.
    """
    # Normalize but preserve word boundaries
    words = re.sub(r"[^\w\s]", " ", text).split()
    entities = set()
    for word in words:
        lower = word.lower()
        if lower in _ENTITY_STOPWORDS or len(word) <= 2:
            continue
        # Keep words that are: capitalized, ALL CAPS, contain digits, or 4+ chars
        if word[0].isupper() or word.isupper() or any(c.isdigit() for c in word) or len(word) >= 4:
            entities.add(lower)
    return entities
 def _entity_overlap(entities_a: set[str], entities_b: set[str]) -> float:
    """Jaccard-style overlap on extracted entities."""
    if not entities_a or not entities_b:
        return 0.0
    intersection = entities_a & entities_b
    smaller = min(len(entities_a), len(entities_b))
    # Use overlap coefficient (intersection / min) instead of Jaccard,
    # because a short tweet about the same event as a long Reddit post
    # will have fewer total entities but high overlap with the larger set.
    return len(intersection) / smaller if smaller > 0 else 0.0
 def _mmr_representatives(
    candidates: list[schema.Candidate],
    text_cache: dict[str, dedupe._PreparedText],
    limit: int = 3,
    diversity_lambda: float = 0.75,
 ) -> list[str]:
    selected: list[schema.Candidate] = []
    remaining_set = {c.candidate_id for c in candidates}
    remaining = list(candidates)
    while remaining and len(selected) < limit:
        if not selected:
            best = max(remaining, key=lambda candidate: candidate.final_score)
            selected.append(best)
            remaining_set.discard(best.candidate_id)
            remaining = [c for c in remaining if c.candidate_id in remaining_set]
            continue
        selected_preps = [text_cache[c.candidate_id] for c in selected]
        def score(candidate: schema.Candidate) -> float:
            prep = text_cache[candidate.candidate_id]
            diversity_penalty = max(
                dedupe.prepared_similarity(prep, sp) for sp in selected_preps
            )
            return (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)
        best = max(remaining, key=score)
        selected.append(best)
        remaining_set.discard(best.candidate_id)
        remaining = [c for c in remaining if c.candidate_id in remaining_set]
    return [candidate.candidate_id for candidate in selected]
 def cluster_candidates(
    candidates: list[schema.Candidate],
    plan: schema.QueryPlan,
 ) -> list[schema.Cluster]:
    """Greedy clustering around high-ranked leaders."""
    if plan.intent not in CLUSTERABLE_INTENTS or plan.cluster_mode == "none":
        clusters = []
        for index, candidate in enumerate(candidates, start=1):
            cluster_id = f"cluster-{index}"
            candidate.cluster_id = cluster_id
            clusters.append(
                schema.Cluster(
                    cluster_id=cluster_id,
                    title=candidate.title,
                    candidate_ids=[candidate.candidate_id],
                    representative_ids=[candidate.candidate_id],
                    sources=sorted(schema.candidate_sources(candidate)),
                    score=candidate.final_score,
                    uncertainty=None,
                )
            )
        return clusters
    text_cache: dict[str, dedupe._PreparedText] = {
        c.candidate_id: dedupe._PreparedText(_candidate_text(c))
        for c in candidates
    }
    groups: list[list[schema.Candidate]] = []
    # Lower threshold for breaking_news: related articles share fewer exact
    # words but cover the same event.
    threshold = 0.42 if plan.intent == "breaking_news" else 0.48
    for candidate in candidates:
        assigned = False
        cand_prep = text_cache[candidate.candidate_id]
        for group in groups:
            leader = group[0]
            similarity = dedupe.prepared_similarity(cand_prep, text_cache[leader.candidate_id])
            if similarity >= threshold:
                group.append(candidate)
                assigned = True
                break
        if not assigned:
            groups.append([candidate])
    clusters: list[schema.Cluster] = []
    for index, group in enumerate(groups, start=1):
        group.sort(key=lambda candidate: candidate.final_score, reverse=True)
        cluster_id = f"cluster-{index}"
        representatives = _mmr_representatives(group, text_cache)
        for candidate in group:
            candidate.cluster_id = cluster_id
        clusters.append(
            schema.Cluster(
                cluster_id=cluster_id,
                title=group[0].title,
                candidate_ids=[candidate.candidate_id for candidate in group],
                representative_ids=representatives,
                sources=sorted({source for candidate in group for source in schema.candidate_sources(candidate)}),
                score=max(candidate.final_score for candidate in group),
                uncertainty=_cluster_uncertainty(group),
            )
        )
    # Second pass: merge small clusters that share entities across sources.
    clusters = _merge_entity_clusters(clusters, candidates)
    return sorted(clusters, key=lambda cluster: cluster.score, reverse=True)
 def _merge_entity_clusters(
    clusters: list[schema.Cluster],
    all_candidates: list[schema.Candidate],
 ) -> list[schema.Cluster]:
    """Merge small clusters that cover the same story across different sources.
    The initial greedy pass uses text similarity which misses cross-source
    matches where phrasing differs. This second pass looks at entity overlap
    (proper nouns, names, numbers) to catch cases like:
      - Reddit: "Kanye West to headline all three nights of Wireless Festival 2026"
      - X: "BREAKING: Kanye West (Ye) is making his massive UK comeback!"
    """
    if len(clusters) < 2:
        return clusters
    candidate_map = {c.candidate_id: c for c in all_candidates}
    # Build entity sets per cluster
    cluster_entities: list[set[str]] = []
    for cl in clusters:
        entities: set[str] = set()
        for cid in cl.candidate_ids:
            cand = candidate_map.get(cid)
            if cand:
                entities |= _extract_entities(_candidate_text(cand))
        cluster_entities.append(entities)
    # Only merge clusters with <= 3 items (don't merge already-large clusters)
    merged_into: dict[int, int] = {}  # index -> merge target index
    for i in range(len(clusters)):
        if i in merged_into or len(clusters[i].candidate_ids) > 3:
            continue
        for j in range(i + 1, len(clusters)):
            if j in merged_into or len(clusters[j].candidate_ids) > 3:
                continue
            # Require different sources to merge (same-source should already be grouped)
            sources_i = set(clusters[i].sources)
            sources_j = set(clusters[j].sources)
            if sources_i == sources_j and len(sources_i) == 1:
                continue
            # Prevent Polymarket clusters from merging with non-Polymarket
            # clusters. Prediction markets about "Sam Altman equity" should not
            # merge into a news cluster about "Sam Altman rivalry" just because
            # both mention the same entity.
            poly_i = "polymarket" in sources_i
            poly_j = "polymarket" in sources_j
            if poly_i != poly_j:
                continue
            overlap = _entity_overlap(cluster_entities[i], cluster_entities[j])
            if overlap >= 0.45:
                merged_into[j] = i
    if not merged_into:
        return clusters
    # Build merged cluster list
    result: list[schema.Cluster] = []
    for i, cl in enumerate(clusters):
        if i in merged_into:
            continue
        # Collect all clusters merged into this one
        merge_sources = [i] + [j for j, target in merged_into.items() if target == i]
        if len(merge_sources) == 1:
            result.append(cl)
            continue
        # Combine candidates from all merged clusters
        combined_cids: list[str] = []
        combined_sources: set[str] = set()
        best_score = 0.0
        for idx in merge_sources:
            combined_cids.extend(clusters[idx].candidate_ids)
            combined_sources.update(clusters[idx].sources)
            best_score = max(best_score, clusters[idx].score)
        # Pick representatives from combined pool
        combined_candidates = [candidate_map[cid] for cid in combined_cids if cid in candidate_map]
        combined_candidates.sort(key=lambda c: c.final_score, reverse=True)
        merge_text_cache = {
            c.candidate_id: dedupe._PreparedText(_candidate_text(c))
            for c in combined_candidates
        }
        reps = _mmr_representatives(combined_candidates, merge_text_cache)
        cluster_id = cl.cluster_id
        for cid in combined_cids:
            cand = candidate_map.get(cid)
            if cand:
                cand.cluster_id = cluster_id
        result.append(schema.Cluster(
            cluster_id=cluster_id,
            title=combined_candidates[0].title if combined_candidates else cl.title,
            candidate_ids=combined_cids,
            representative_ids=reps,
            sources=sorted(combined_sources),
            score=best_score,
            uncertainty=_cluster_uncertainty(combined_candidates),
        ))
    return result
 def _cluster_uncertainty(group: list[schema.Candidate]) -> str | None:
    sources = {source for candidate in group for source in schema.candidate_sources(candidate)}
    if len(sources) == 1:
        return "single-source"
    if max(candidate.final_score for candidate in group) < 55:
        return "thin-evidence"
    return None
@@ -0,0 +1,199 @@
 """Discover peer entities ("competitors") for a topic via web search.
 Mirrors the `resolve.auto_resolve()` pattern: fan out 2-3 web searches via
 `grounding.web_search()`, then extract capitalized entity candidates from
 titles and snippets with deterministic text mining. No LLM call — the
 hosting reasoning model can always override discovery via
 `--competitors-list`.
 Returned list is ordered by score (frequency across queries) and capped to
 the caller's requested count.
 """
 from __future__ import annotations
 import re
 import sys
 from collections import Counter
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from . import dates, grounding
 from .resolve import _has_backend
 # A "brand-shaped" token starts with uppercase OR is camelCase with an
 # uppercase letter later. Catches "Anthropic", "OpenAI", "xAI", "iPhone",
 # "eBay", "Hugging", "Face".
 _BRAND_TOKEN = (
    r"(?:[A-Z][A-Za-z0-9&.\-]*"
    r"|[a-z][A-Za-z0-9&.\-]*[A-Z][A-Za-z0-9&.\-]*)"
 )
 # A capitalized phrase of 1-4 brand tokens separated by whitespace.
 _CAPITALIZED_PHRASE = re.compile(
    rf"\b{_BRAND_TOKEN}(?:\s+{_BRAND_TOKEN}){{0,3}}\b"
 )
 # Title-case fillers common in listicle SERPs. Kept flat — extraction
 # rejects a candidate whose entire tokens are stopwords, not candidates
 # that merely contain one.
 _STOPWORD_TOKENS: frozenset[str] = frozenset(
    token.lower()
    for token in (
        # Listicle fillers
        "Top", "Best", "Worst", "Popular", "Leading", "Similar",
        "Alternatives", "Alternative", "Competitor", "Competitors",
        "vs", "Vs", "Versus", "Review", "Reviews", "Comparison",
        "Guide", "List", "Lists", "Full", "Complete", "Free", "Paid",
        "Tools", "Tool", "Options", "Rivals", "Rival", "Similar",
        "Pick", "Picks", "Ranking", "Ranked", "Recommended",
        # Grammar / time
        "The", "A", "An", "Of", "In", "For", "To", "With", "On", "At",
        "By", "From", "Is", "Are", "And", "Or", "But", "Than", "As",
        "This", "That", "These", "Those", "Our", "Your", "Their",
        "January", "February", "March", "April", "May", "June", "July",
        "August", "September", "October", "November", "December",
        # Years likely to appear as standalone tokens
        *(str(year) for year in range(2018, 2031)),
        # Miscellaneous SERP noise
        "AI", "Apps", "App", "Software", "Platform", "Service", "Startups",
        "Companies", "Company", "Products", "Product", "Brands", "Brand",
    )
 )
 def _log(msg: str) -> None:
    print(f"[Competitors] {msg}", file=sys.stderr)
 def _topic_tokens(topic: str) -> set[str]:
    """Return lowercase alphanumeric tokens of the topic for filtering."""
    return {tok for tok in re.findall(r"[A-Za-z0-9]+", topic.lower()) if tok}
 def _candidate_ok(candidate: str, topic_tokens: set[str]) -> bool:
    """Filter a candidate phrase against stopwords and topic overlap."""
    tokens = [t for t in re.findall(r"[A-Za-z0-9&.\-]+", candidate) if t]
    if not tokens:
        return False
    # Reject candidates made entirely of stopwords (e.g., "Top Alternatives").
    if all(tok.lower() in _STOPWORD_TOKENS for tok in tokens):
        return False
    # Reject candidates that overlap with the topic (e.g., topic="OpenAI"
    # should not return "OpenAI Alternatives" or "OpenAI").
    lower_tokens = {tok.lower() for tok in tokens}
    if lower_tokens & topic_tokens:
        return False
    # Reject too-short one-letter tokens like "I" or single digits.
    if len(tokens) == 1 and len(tokens[0]) < 2:
        return False
    return True
 def _normalize_candidate(candidate: str) -> str:
    """Collapse whitespace and strip trailing punctuation."""
    return re.sub(r"\s+", " ", candidate).strip(".,;:!?'\"()[] ")
 def _extract_peer_entities(
    items: list[dict], topic: str, limit: int,
 ) -> list[str]:
    """Score capitalized candidates across SERP items and return top `limit`.
    Scoring is bag-of-phrases frequency across all items in the input. Ties
    are broken by first-seen order so the output is deterministic.
    """
    topic_tokens = _topic_tokens(topic)
    counts: Counter[str] = Counter()
    first_seen: dict[str, int] = {}
    order = 0
    # Group candidates into a frequency map keyed by lowercased normalized
    # form so "xAI" and "xAI" count together regardless of case.
    canonical: dict[str, str] = {}
    for item in items:
        text = f"{item.get('title', '')} {item.get('snippet', '')}"
        for raw in _CAPITALIZED_PHRASE.findall(text):
            candidate = _normalize_candidate(raw)
            if not _candidate_ok(candidate, topic_tokens):
                continue
            key = candidate.lower()
            if key not in canonical:
                canonical[key] = candidate
                first_seen[key] = order
                order += 1
            counts[key] += 1
    ranked_keys = sorted(
        counts.keys(),
        key=lambda k: (-counts[k], first_seen[k]),
    )
    return [canonical[k] for k in ranked_keys[:limit]]
 def _queries_for(topic: str) -> dict[str, str]:
    return {
        "competitors": f"{topic} competitors",
        "alternatives": f"{topic} alternatives",
        "vs": f"{topic} vs",
    }
 def discover_competitors(
    topic: str,
    count: int,
    config: dict,
    *,
    lookback_days: int = 30,
 ) -> list[str]:
    """Discover `count` peer entities for `topic` via web search.
    Args:
        topic: The primary research topic.
        count: Desired number of competitor entities (1..N).
        config: Runtime config dict — expects the same shape as the engine
            config (BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / etc.).
        lookback_days: Date range for freshness. Defaults to 30.
    Returns:
        A list of up to `count` entity names, deduped and ordered by score.
        Empty list when no web backend is configured or every search fails
        or returns zero usable candidates.
    """
    if count < 1:
        return []
    if not _has_backend(config):
        _log("No web search backend available, skipping competitor discovery")
        return []
    date_range = dates.get_date_range(lookback_days)
    queries = _queries_for(topic)
    collected: list[dict] = []
    searches_run = 0
    def _search(label: str, query: str) -> tuple[str, list[dict]]:
        items, _artifact = grounding.web_search(query, date_range, config)
        return label, items
    with ThreadPoolExecutor(max_workers=len(queries)) as executor:
        futures = {
            executor.submit(_search, label, q): label
            for label, q in queries.items()
        }
        for future in as_completed(futures):
            label = futures[future]
            try:
                _label, items = future.result()
                collected.extend(items)
                searches_run += 1
            except Exception as exc:
                _log(f"Search failed for {label}: {exc}")
    if not collected:
        _log(f"No SERP results for {topic!r} across {searches_run}/{len(queries)} queries")
        return []
    entities = _extract_peer_entities(collected, topic, limit=count)
    _log(
        f"Discovered {len(entities)} competitor(s) for {topic!r} "
        f"from {searches_run}/{len(queries)} queries: {entities}"
    )
    return entities
@@ -0,0 +1,403 @@
 """Browser cookie extraction for last30days.
 Extracts cookies from local browser databases (Firefox, Chrome, Brave, Safari)
 to enable zero-config authentication for services like X/Twitter.
 Only uses Python stdlib — no external dependencies.
 """
 import configparser
 import functools
 import logging
 import platform
 import shutil
 import sqlite3
 import tempfile
 from pathlib import Path
 from typing import Dict, List, Optional
 logger = logging.getLogger(__name__)
@functools.lru_cache(maxsize=1)
 def _is_wsl() -> bool:
    """Detect if running under Windows Subsystem for Linux.
    Cached after the first call since /proc/version doesn't change at runtime.
    """
    try:
        return "microsoft" in Path("/proc/version").read_text().lower()
    except OSError:
        return False
 def _get_wsl_firefox_profiles_dir() -> Optional[Path]:
    """Find Firefox profiles directory on the Windows host from WSL.
    Scans /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox for real user
    directories (skips Public, Default, etc.).
    """
    mnt_users = Path("/mnt/c/Users")
    if not mnt_users.is_dir():
        return None
    skip = {"Public", "Default", "Default User", "All Users"}
    try:
        for user_dir in sorted(mnt_users.iterdir()):
            if user_dir.name in skip or not user_dir.is_dir():
                continue
            ff_dir = user_dir / "AppData" / "Roaming" / "Mozilla" / "Firefox"
            if ff_dir.is_dir():
                return ff_dir
    except OSError:
        pass
    return None
 def _get_firefox_profiles_dir() -> Optional[Path]:
    """Return the Firefox profiles directory for the current platform, or None."""
    system = platform.system()
    if system == "Darwin":
        path = Path.home() / "Library" / "Application Support" / "Firefox"
    elif system == "Linux":
        path = Path.home() / ".mozilla" / "firefox"
    else:
        # Windows: %APPDATA%\Mozilla\Firefox — best-effort
        appdata = Path.home() / "AppData" / "Roaming" / "Mozilla" / "Firefox"
        path = appdata
    return path if path.is_dir() else None
 def _find_default_profile(profiles_dir: Path) -> Optional[Path]:
    """Parse profiles.ini to find the default profile directory.
    Looks for a section with Default=1. Falls back to the first profile
    directory found on disk if profiles.ini is missing or malformed.
    """
    ini_path = profiles_dir / "profiles.ini"
    if ini_path.is_file():
        try:
            config = configparser.ConfigParser()
            config.read(str(ini_path), encoding="utf-8")
            # First pass: Install* section (Firefox >= 67 format, takes priority)
            for section in config.sections():
                if section.startswith("Install") and config.has_option(section, "Default"):
                    raw = config.get(section, "Default")
                    candidate = profiles_dir / raw
                    if candidate.is_dir():
                        return candidate
            # Second pass: Profile section with Default=1
            for section in config.sections():
                if section.startswith("Profile") and config.has_option(section, "Default") and config.get(section, "Default") == "1":
                    return _resolve_profile_path(profiles_dir, config, section)
            # Third pass: first Profile section that exists on disk
            for section in config.sections():
                if section.startswith("Profile"):
                    resolved = _resolve_profile_path(profiles_dir, config, section)
                    if resolved and resolved.is_dir():
                        return resolved
        except (configparser.Error, OSError) as exc:
            logger.debug("Failed to parse profiles.ini: %s", exc)
    # Fallback: scan directory for anything that looks like a profile
    return _fallback_find_profile(profiles_dir)
 def _resolve_profile_path(
    profiles_dir: Path, config: configparser.ConfigParser, section: str
 ) -> Optional[Path]:
    """Resolve a profile path from a ConfigParser section."""
    if not config.has_option(section, "Path"):
        return None
    raw_path = config.get(section, "Path")
    is_relative = config.has_option(section, "IsRelative") and config.get(section, "IsRelative") == "1"
    if is_relative:
        candidate = profiles_dir / raw_path
    else:
        candidate = Path(raw_path)
    return candidate if candidate.is_dir() else None
 def _fallback_find_profile(profiles_dir: Path) -> Optional[Path]:
    """Find the first directory that contains cookies.sqlite."""
    try:
        for child in sorted(profiles_dir.iterdir()):
            if child.is_dir() and (child / "cookies.sqlite").is_file():
                return child
    except OSError:
        pass
    return None
 def _query_cookies_db(
    db_path: Path, domain: str, cookie_names: List[str]
 ) -> Optional[Dict[str, str]]:
    """Copy the cookies database to a temp file and query it.
    Firefox locks cookies.sqlite while running, so we copy first.
    Returns {name: value} dict or None if no matching cookies found.
    """
    if not db_path.is_file():
        return None
    tmp_fd = None
    tmp_path = None
    try:
        tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
        shutil.copy2(str(db_path), tmp_path)
        conn = sqlite3.connect(tmp_path)
        try:
            # Build parameterized query — SQLite doesn't support array params,
            # so we build the IN clause with individual placeholders.
            placeholders = ",".join("?" for _ in cookie_names)
            query = (
                f"SELECT name, value FROM moz_cookies "
                f"WHERE host LIKE ? AND name IN ({placeholders})"
            )
            # domain pattern: match .x.com, x.com, etc.
            domain_pattern = f"%{domain}"
            params = [domain_pattern] + list(cookie_names)
            cursor = conn.execute(query, params)
            rows = cursor.fetchall()
        finally:
            conn.close()
        if not rows:
            return None
        return {name: value for name, value in rows}
    except (sqlite3.Error, OSError) as exc:
        logger.debug("Failed to query cookies database %s: %s", db_path, exc)
        return None
    finally:
        if tmp_path:
            try:
                Path(tmp_path).unlink(missing_ok=True)
            except OSError:
                pass
        if tmp_fd is not None:
            try:
                import os
                os.close(tmp_fd)
            except OSError:
                pass
 def _try_firefox_dir(profiles_dir: Path, domain: str, cookie_names: List[str]) -> Optional[Dict[str, str]]:
    """Try to extract cookies from a Firefox profiles directory."""
    profile_path = _find_default_profile(profiles_dir)
    if profile_path is None:
        logger.debug("No Firefox profile found in %s", profiles_dir)
        return None
    return _query_cookies_db(profile_path / "cookies.sqlite", domain, cookie_names)
 def extract_firefox_cookies(
    domain: str, cookie_names: List[str]
 ) -> Optional[Dict[str, str]]:
    """Extract cookies from Firefox for the given domain and cookie names.
    Finds the default Firefox profile, copies cookies.sqlite to a temp file
    (to avoid lock conflicts), and queries for the requested cookies.
    On WSL2, falls back to Windows Firefox if native Linux Firefox has no
    matching cookies. Windows Firefox cookies are unencrypted, so this works
    without DPAPI or any Windows-side helpers.
    Args:
        domain: The cookie domain to match (e.g. ".x.com"). Matched with LIKE %domain.
        cookie_names: List of cookie names to extract (e.g. ["auth_token", "ct0"]).
    Returns:
        Dict of {cookie_name: cookie_value} or None if extraction fails.
    """
    profiles_dir = _get_firefox_profiles_dir()
    if profiles_dir is not None:
        result = _try_firefox_dir(profiles_dir, domain, cookie_names)
        if result is not None:
            return result
    if platform.system() == "Linux" and _is_wsl():
        wsl_dir = _get_wsl_firefox_profiles_dir()
        if wsl_dir is not None:
            logger.debug("Trying Windows Firefox via WSL: %s", wsl_dir)
            return _try_firefox_dir(wsl_dir, domain, cookie_names)
    if profiles_dir is None:
        logger.debug("Firefox profiles directory not found")
    return None
 def extract_chrome_cookies(
    domain: str, cookie_names: List[str]
 ) -> Optional[Dict[str, str]]:
    """Extract cookies from Chrome for the given domain and cookie names.
    macOS only — uses Keychain + system openssl for AES-128-CBC decryption.
    Linux/Windows not supported (Chrome uses platform-specific encryption).
    Returns:
        Dict of {cookie_name: cookie_value} or None if extraction fails.
    """
    if platform.system() != "Darwin":
        logger.debug("Chrome cookie extraction only supported on macOS")
        return None
    try:
        from .chrome_cookies import extract_chrome_cookies_macos
        return extract_chrome_cookies_macos(domain, cookie_names)
    except Exception as exc:
        logger.debug("Chrome cookie extraction failed: %s", exc)
        return None
 def extract_brave_cookies(
    domain: str, cookie_names: List[str]
 ) -> Optional[Dict[str, str]]:
    """Extract cookies from Brave for the given domain and cookie names.
    macOS only — Brave uses the same v10 AES-128-CBC encryption as Chrome,
    with a different DB path and Keychain service name ("Brave Safe Storage").
    Tries the Default profile first, then scans numbered Profile directories.
    Returns:
        Dict of {cookie_name: cookie_value} or None if extraction fails.
    """
    if platform.system() != "Darwin":
        logger.debug("Brave cookie extraction only supported on macOS")
        return None
    try:
        from .chrome_cookies import extract_brave_cookies_macos
        return extract_brave_cookies_macos(domain, cookie_names)
    except Exception as exc:
        logger.debug("Brave cookie extraction failed: %s", exc)
        return None
 def extract_safari_cookies(
    domain: str, cookie_names: List[str]
 ) -> Optional[Dict[str, str]]:
    """Extract cookies from Safari for the given domain and cookie names.
    macOS only — parses the unencrypted binary cookie file.
    Returns:
        Dict of {cookie_name: cookie_value} or None if extraction fails.
    """
    if platform.system() != "Darwin":
        logger.debug("Safari cookie extraction only supported on macOS")
        return None
    try:
        from .safari_cookies import extract_safari_cookies_macos
        return extract_safari_cookies_macos(domain, cookie_names)
    except Exception as exc:
        logger.debug("Safari cookie extraction failed: %s", exc)
        return None
 def extract_cookies(
    browser: str, domain: str, cookie_names: list[str]
 ) -> Optional[dict[str, str]]:
    """Extract cookies from the specified browser.
    Args:
        browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
            'auto' tries browsers in platform-appropriate order:
            - macOS: Chrome -> Brave -> Firefox -> Safari
            - Linux: Firefox only
        domain: The cookie domain to match (e.g. ".x.com").
        cookie_names: List of cookie names to extract.
    Returns:
        Dict of {cookie_name: cookie_value} or None if extraction fails.
    """
    result = extract_cookies_with_source(browser, domain, cookie_names)
    if result is None:
        return None
    cookies, _browser_name = result
    return cookies
 def _extract_firefox_with_source(
    domain: str, cookie_names: List[str]
 ) -> Optional[tuple[Dict[str, str], str]]:
    """Extract Firefox cookies and report whether they came from native or WSL.
    Returns (cookies, "firefox") for native Linux/macOS Firefox, or
    (cookies, "firefox-wsl") for Windows Firefox accessed via WSL2.
    """
    profiles_dir = _get_firefox_profiles_dir()
    if profiles_dir is not None:
        result = _try_firefox_dir(profiles_dir, domain, cookie_names)
        if result is not None:
            return (result, "firefox")
    if platform.system() == "Linux" and _is_wsl():
        wsl_dir = _get_wsl_firefox_profiles_dir()
        if wsl_dir is not None:
            logger.debug("Trying Windows Firefox via WSL: %s", wsl_dir)
            result = _try_firefox_dir(wsl_dir, domain, cookie_names)
            if result is not None:
                return (result, "firefox-wsl")
    return None
 def extract_cookies_with_source(
    browser: str, domain: str, cookie_names: list[str]
 ) -> Optional[tuple[dict[str, str], str]]:
    """Extract cookies and report which browser they came from.
    Same as extract_cookies() but returns a (cookies, browser_name) tuple
    so callers can track the source.
    Args:
        browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
        domain: The cookie domain to match (e.g. ".x.com").
        cookie_names: List of cookie names to extract.
    Returns:
        Tuple of ({cookie_name: cookie_value}, browser_name) or None.
        browser_name is "firefox-wsl" when cookies came from Windows Firefox via WSL2.
    """
    extractors = {
        "firefox": extract_firefox_cookies,
        "chrome": extract_chrome_cookies,
        "brave": extract_brave_cookies,
        "safari": extract_safari_cookies,
    }
    if browser != "auto":
        if browser == "firefox":
            return _extract_firefox_with_source(domain, cookie_names)
        extractor = extractors.get(browser)
        if extractor is None:
            logger.warning("Unknown browser: %s", browser)
            return None
        result = extractor(domain, cookie_names)
        return (result, browser) if result is not None else None
    # Auto mode: try browsers in platform-appropriate order
    system = platform.system()
    if system == "Darwin":
        order = ["chrome", "brave", "firefox", "safari"]
    elif system == "Linux":
        order = ["firefox"]
    else:
        order = ["firefox"]
    for name in order:
        if name == "firefox":
            result = _extract_firefox_with_source(domain, cookie_names)
            if result is not None:
                return result
        else:
            result = extractors[name](domain, cookie_names)
            if result is not None:
                return (result, name)
    return None
@@ -41,7 +41,10 @@ def parse_date(date_str: Optional[str]) -> Optional[datetime]:
    for fmt in formats:
        try:
-            return datetime.strptime(date_str, fmt).replace(tzinfo=timezone.utc)
+            dt = datetime.strptime(date_str, fmt)
            if dt.tzinfo is not None:
                return dt.astimezone(timezone.utc)
            return dt.replace(tzinfo=timezone.utc)
        except ValueError:
            continue
@@ -78,14 +81,7 @@ def get_date_confidence(date_str: Optional[str], from_date: str, to_date: str) -
        start = datetime.strptime(from_date, "%Y-%m-%d").date()
        end = datetime.strptime(to_date, "%Y-%m-%d").date()
-        if start <= dt <= end:
+        return 'high' if start <= dt <= end else 'low'
            return 'high'
        elif dt < start:
            # Older than range
            return 'low'
        else:
            # Future date (suspicious)
            return 'low'
    except ValueError:
        return 'low'
@@ -0,0 +1,130 @@
 """Within-source near-duplicate detection."""
 from __future__ import annotations
 import re
 from . import schema
 STOPWORDS = frozenset(
    {
        "the",
        "a",
        "an",
        "to",
        "for",
        "how",
        "is",
        "in",
        "of",
        "on",
        "and",
        "with",
        "from",
        "by",
        "at",
        "this",
        "that",
        "it",
        "what",
        "are",
        "do",
        "can",
    }
 )
 def normalize_text(text: str) -> str:
    text = re.sub(r"[^\w\s]", " ", text.lower())
    return re.sub(r"\s+", " ", text).strip()
 def _ngrams_of_normalized(norm: str, n: int = 3) -> set[str]:
    if len(norm) < n:
        return {norm} if norm else set()
    return {norm[index:index + n] for index in range(len(norm) - n + 1)}
 def get_ngrams(text: str, n: int = 3) -> set[str]:
    return _ngrams_of_normalized(normalize_text(text), n)
 def jaccard_similarity(left: set[str], right: set[str]) -> float:
    if not left or not right:
        return 0.0
    union = left | right
    if not union:
        return 0.0
    return len(left & right) / len(union)
 def token_jaccard(text_a: str, text_b: str) -> float:
    tokens_a = {
        token
        for token in normalize_text(text_a).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    tokens_b = {
        token
        for token in normalize_text(text_b).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    return jaccard_similarity(tokens_a, tokens_b)
 def hybrid_similarity(text_a: str, text_b: str) -> float:
    return max(
        jaccard_similarity(get_ngrams(text_a), get_ngrams(text_b)),
        token_jaccard(text_a, text_b),
    )
 def _tokenize(normalized: str) -> frozenset[str]:
    return frozenset(
        tok for tok in normalized.split()
        if len(tok) > 1 and tok not in STOPWORDS
    )
 class _PreparedText:
    """Pre-computed text representations for fast repeated similarity checks."""
    __slots__ = ("ngrams", "tokens")
    def __init__(self, raw: str) -> None:
        norm = normalize_text(raw)
        self.ngrams = _ngrams_of_normalized(norm)
        self.tokens = _tokenize(norm)
 def prepared_similarity(a: _PreparedText, b: _PreparedText) -> float:
    return max(
        jaccard_similarity(a.ngrams, b.ngrams),
        jaccard_similarity(a.tokens, b.tokens),
    )
 def item_text(item: schema.SourceItem) -> str:
    parts = [item.title, item.body, item.author or "", item.container or ""]
    return " ".join(part for part in parts if part).strip()
 def dedupe_items(items: list[schema.SourceItem], threshold: float = 0.7) -> list[schema.SourceItem]:
    """Remove near-duplicates while keeping earlier, better-scored items."""
    kept: list[schema.SourceItem] = []
    kept_prepared: list[_PreparedText] = []
    for item in items:
        text = item_text(item)
        if not text:
            kept.append(item)
            continue
        prep = _PreparedText(text)
        is_duplicate = False
        for existing_prep in kept_prepared:
            if prepared_similarity(prep, existing_prep) >= threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            kept.append(item)
            kept_prepared.append(prep)
    return kept
@@ -0,0 +1,414 @@
 """Digg AI 1000 source for last30days.
 Shells out to ``digg-pp-cli`` (read-only, no auth required) to surface
 clustered stories curated from ~1000 high-signal AI accounts on X. Each
 cluster carries a published TLDR, a curatorial rank, and a list of X
 posts that can be fetched as inline quotes.
 Activation gate: this source is only available when ``digg-pp-cli`` is
 on PATH. ``pipeline.available_sources`` checks ``shutil.which`` before
 including ``digg`` in the source list. The functions below also detect
 the missing-binary case as a defensive fallback.
 Primary path: ``digg-pp-cli search <topic> --since 30d --agent --limit N``.
 Optional enrichment: ``digg-pp-cli posts <clusterUrlId> --agent --by rank
 --limit M`` for the top K clusters in default/deep depth, attaching the
 top-ranked X posts to each cluster's ``posts`` field.
 """
 from __future__ import annotations
 import json
 import shutil
 from datetime import datetime, timedelta, timezone
 from typing import Any, Dict, List, Optional
 from . import log, subproc
 from .relevance import token_overlap_relevance
 CLI_BIN = "digg-pp-cli"
 # Per-depth knobs.
 DEPTH_CONFIG = {
    "quick": 8,
    "default": 20,
    "deep": 40,
 }
 # How many top-ranked clusters get post enrichment, per depth. Quick mode
 # skips enrichment to keep latency low (clusters already carry a TLDR).
 ENRICH_CONFIG = {
    "quick": 0,
    "default": 3,
    "deep": 5,
 }
 # X posts pulled per enriched cluster. Matches the 5-comment cap used by
 # Reddit/HN/YouTube/TikTok/GitHub enrichment.
 POSTS_PER_CLUSTER = 5
 SEARCH_TIMEOUT = 30
 POSTS_TIMEOUT = 15
 def _log(msg: str) -> None:
    log.source_log("Digg", msg)
 def _is_available() -> bool:
    """True when the digg-pp-cli binary is on PATH."""
    return shutil.which(CLI_BIN) is not None
 def _today() -> datetime:
    return datetime.now(timezone.utc)
 def _parse_first_post_age(age: Optional[str], today: Optional[datetime] = None) -> Optional[str]:
    """Convert a digg firstPostAge token (e.g. '5d', '17d', '5h', '1w', '1m')
    into a YYYY-MM-DD string. Returns None when the value is outside the
    last-30-day window or cannot be parsed.
    Digg uses minutes-symbol-collision for 'months' (per agent-context:
    'Nh, Nd, Nw, Nm (e.g. 30d, 1w, 12h, 1m)'), so 'Nm' is months ~30 days.
    """
    if not age or not isinstance(age, str):
        return None
    age = age.strip().lower()
    if len(age) < 2:
        return None
    unit = age[-1]
    try:
        amount = int(age[:-1])
    except (ValueError, TypeError):
        return None
    if amount < 0:
        return None
    base = today or _today()
    if unit == "h":
        delta = timedelta(hours=amount)
    elif unit == "d":
        delta = timedelta(days=amount)
    elif unit == "w":
        delta = timedelta(weeks=amount)
    elif unit == "m":
        delta = timedelta(days=amount * 30)
    else:
        return None
    if delta > timedelta(days=30):
        return None
    point = base - delta
    return point.date().isoformat()
 def _build_search_args(query: str, limit: int) -> List[str]:
    return [
        CLI_BIN,
        "search",
        query,
        "--since",
        "30d",
        "--agent",
        "--limit",
        str(limit),
    ]
 def _build_posts_args(cluster_url_id: str, posts_per: int) -> List[str]:
    return [
        CLI_BIN,
        "posts",
        cluster_url_id,
        "--agent",
        "--by",
        "rank",
        "--limit",
        str(posts_per),
    ]
 def _run_cli(cmd: List[str], timeout: int) -> Dict[str, Any]:
    """Invoke digg-pp-cli and parse the JSON envelope.
    Returns ``{"results": [...]}`` on success, ``{"results": [], "error": "..."}``
    on failure. Never raises; the pipeline relies on shape consistency.
    """
    if not _is_available():
        return {"results": [], "error": f"{CLI_BIN} not on PATH"}
    try:
        result = subproc.run_with_timeout(cmd, timeout=timeout)
    except subproc.SubprocTimeout as exc:
        _log(f"Timeout: {exc}")
        return {"results": [], "error": str(exc)}
    except FileNotFoundError as exc:
        _log(f"Binary missing: {exc}")
        return {"results": [], "error": str(exc)}
    except OSError as exc:
        _log(f"Spawn failed: {exc}")
        return {"results": [], "error": str(exc)}
    if result.returncode != 0:
        snippet = (result.stderr or "").strip().splitlines()[:1]
        first = snippet[0] if snippet else f"exit {result.returncode}"
        _log(f"CLI exit {result.returncode}: {first}")
        return {"results": [], "error": first}
    stdout = result.stdout or ""
    if not stdout.strip():
        return {"results": []}
    try:
        data = json.loads(stdout)
    except json.JSONDecodeError as exc:
        _log(f"JSON decode failed: {exc}")
        return {"results": [], "error": f"json decode: {exc}"}
    if not isinstance(data, dict):
        return {"results": []}
    results = data.get("results")
    if not isinstance(results, list):
        return {"results": []}
    return data
 def search_digg(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
 ) -> Dict[str, Any]:
    """Search Digg AI 1000 clusters via digg-pp-cli.
    Args:
        topic: search query.
        from_date: YYYY-MM-DD start (advisory; --since 30d is the actual filter).
        to_date: YYYY-MM-DD end (advisory; same).
        depth: 'quick' | 'default' | 'deep'.
    Returns:
        Dict with ``results`` list. On failure, ``results`` is empty and an
        ``error`` key carries a one-line description.
    """
    limit = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    if not topic or not topic.strip():
        return {"results": []}
    cmd = _build_search_args(topic, limit)
    _log(f"search '{topic}' (limit={limit}, since=30d)")
    response = _run_cli(cmd, timeout=SEARCH_TIMEOUT)
    n = len(response.get("results") or [])
    _log(f"found {n} clusters")
    return response
 def _build_url(cluster_url_id: str) -> str:
    return f"https://di.gg/ai/{cluster_url_id}"
 def _rank_score(rank: Optional[int]) -> float:
    """Convert Digg rank (lower is better, top 50 are notable) into a
    positive engagement-style signal in [0, 50]. Anything off the top-50
    leaderboard contributes 0.
    """
    if rank is None:
        return 0.0
    try:
        r = int(rank)
    except (TypeError, ValueError):
        return 0.0
    if r < 1 or r > 50:
        return 0.0
    return float(51 - r)
 def parse_digg_response(
    response: Dict[str, Any],
    query: str = "",
 ) -> List[Dict[str, Any]]:
    """Parse a digg search envelope into normalized item dicts.
    Args:
        response: payload from ``search_digg``.
        query: original search query, used for token-overlap relevance.
    Returns:
        List of dicts ready for ``normalize._normalize_digg``.
    """
    raw = response.get("results") if isinstance(response, dict) else None
    if not isinstance(raw, list):
        return []
    items: List[Dict[str, Any]] = []
    for i, cluster in enumerate(raw):
        if not isinstance(cluster, dict):
            continue
        cluster_url_id = cluster.get("clusterUrlId")
        if not cluster_url_id:
            continue
        title = str(cluster.get("title") or "").strip()
        tldr = str(cluster.get("tldr") or "").strip()
        rank = cluster.get("rank")
        post_count = cluster.get("postCount") or 0
        unique_authors = cluster.get("uniqueAuthors") or 0
        first_post_age = cluster.get("firstPostAge")
        date_str = _parse_first_post_age(first_post_age)
        if date_str is None and first_post_age:
            # firstPostAge present but outside 30d -> drop; last30days contract.
            continue
        rank_decay = max(0.3, 1.0 - (i * 0.02))
        if query:
            content_score = token_overlap_relevance(query, f"{title} {tldr}".strip())
        else:
            content_score = 0.5
        rank_boost = min(0.2, _rank_score(rank) / 250.0)
        relevance = min(1.0, 0.55 * rank_decay + 0.35 * content_score + rank_boost)
        items.append(
            {
                "id": str(cluster_url_id),
                "title": title or f"Digg cluster {i + 1}",
                "url": _build_url(str(cluster_url_id)),
                "tldr": tldr,
                "author": "",
                "date": date_str,
                "engagement": {
                    "postCount": int(post_count) if isinstance(post_count, (int, float)) else 0,
                    "uniqueAuthors": int(unique_authors) if isinstance(unique_authors, (int, float)) else 0,
                    "rank": int(rank) if isinstance(rank, (int, float)) else None,
                    "rank_score": _rank_score(rank),
                },
                "first_post_age": first_post_age,
                "posts": [],
                "relevance": round(relevance, 2),
                "why_relevant": (
                    f"Digg cluster (rank {rank}, {post_count} posts, {unique_authors} authors)"
                    if rank is not None
                    else f"Digg cluster ({post_count} posts, {unique_authors} authors)"
                ),
            }
        )
    return items
 def _parse_post(raw_post: Dict[str, Any]) -> Optional[Dict[str, Any]]:
    """Reduce a digg post payload into the small dict render uses.
    We deliberately keep this minimal: an inline quote needs the author
    handle, the body, the post type, and the X URL.
    """
    if not isinstance(raw_post, dict):
        return None
    body = str(raw_post.get("body") or "").strip()
    if not body:
        return None
    author = raw_post.get("author") or {}
    if not isinstance(author, dict):
        author = {}
    username = str(author.get("username") or "").strip()
    if not username:
        return None
    x_url = str(raw_post.get("xUrl") or "").strip()
    if not x_url:
        return None
    return {
        "username": username,
        "display_name": str(author.get("display_name") or "").strip() or username,
        "category": str(author.get("category") or "").strip(),
        "rank": author.get("rank"),
        "body": body,
        "post_type": str(raw_post.get("post_type") or "tweet").strip(),
        "x_url": x_url,
        "posted_at": raw_post.get("posted_at"),
    }
 def fetch_top_posts(cluster_url_id: str, posts_per: int = POSTS_PER_CLUSTER) -> List[Dict[str, Any]]:
    """Fetch top-ranked X posts attached to a cluster.
    Returns an empty list on any failure (timeout, missing cluster, JSON
    error). Never raises.
    """
    if posts_per <= 0:
        return []
    cmd = _build_posts_args(cluster_url_id, posts_per)
    response = _run_cli(cmd, timeout=POSTS_TIMEOUT)
    raw = response.get("results") or []
    out: List[Dict[str, Any]] = []
    for entry in raw:
        post = _parse_post(entry)
        if post is not None:
            out.append(post)
    return out
 def enrich_with_top_posts(
    items: List[Dict[str, Any]],
    top_k: int = 3,
    posts_per: int = POSTS_PER_CLUSTER,
 ) -> List[Dict[str, Any]]:
    """Attach top X posts to the first ``top_k`` clusters by Digg rank order.
    Mutates and returns the same list. Items that already have posts, or
    whose ``postCount`` is 0, are skipped.
    """
    if top_k <= 0 or posts_per <= 0:
        return items
    enriched = 0
    for item in items:
        if enriched >= top_k:
            break
        if item.get("posts"):
            continue
        engagement = item.get("engagement") or {}
        if not engagement.get("postCount"):
            continue
        cluster_url_id = item.get("id")
        if not cluster_url_id:
            continue
        posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
        item["posts"] = posts
        enriched += 1
    if enriched:
        _log(f"enriched {enriched} clusters with X posts")
    return items
 def enrich_source_items(items: list, top_k: int = 3, posts_per: int = POSTS_PER_CLUSTER) -> list:
    """Attach top X posts to the first ``top_k`` SourceItems that survived dedupe.
    Reads ``metadata['clusterUrlId']`` and writes ``metadata['posts']`` in
    place. Skips items that already carry a non-empty ``metadata['posts']``,
    items whose engagement ``postCount`` is 0, and items whose source is not
    'digg'. Designed to run from `_finalize_items_by_source` so enrichment
    is spent on the items the brief actually shows.
    """
    if top_k <= 0 or posts_per <= 0:
        return items
    enriched = 0
    for item in items:
        if enriched >= top_k:
            break
        if getattr(item, "source", None) != "digg":
            continue
        metadata = getattr(item, "metadata", None) or {}
        if metadata.get("posts"):
            continue
        engagement = getattr(item, "engagement", None) or {}
        if not engagement.get("postCount"):
            continue
        cluster_url_id = metadata.get("clusterUrlId") or item.item_id
        if not cluster_url_id:
            continue
        posts = fetch_top_posts(str(cluster_url_id), posts_per=posts_per)
        if posts:
            metadata["posts"] = posts
            enriched += 1
    if enriched:
        _log(f"post-dedupe enriched {enriched} clusters with X posts")
    return items
@@ -0,0 +1,127 @@
 """Entity extraction from initial search results for supplemental searches."""
 import re
 from collections import Counter
 from typing import Any, Dict, List
 # Handles that appear too frequently to be useful for targeted search.
 # These are generic/platform accounts, not topic-specific voices.
 GENERIC_HANDLES = {
    "elonmusk", "openai", "google", "microsoft", "apple", "meta",
    "github", "youtube", "x", "twitter", "reddit", "wikipedia",
    "nytimes", "washingtonpost", "cnn", "bbc", "reuters",
    "verified", "jack", "sundarpichai",
 }
 def extract_entities(
    reddit_items: List[Dict[str, Any]],
    x_items: List[Dict[str, Any]],
    max_handles: int = 5,
    max_hashtags: int = 3,
    max_subreddits: int = 5,
 ) -> Dict[str, List[str]]:
    """Extract key entities from Phase 1 results for supplemental searches.
    Parses X results for @handles and #hashtags, Reddit results for subreddit
    names and cross-referenced communities.
    Args:
        reddit_items: Raw Reddit item dicts from Phase 1
        x_items: Raw X item dicts from Phase 1
        max_handles: Maximum handles to return
        max_hashtags: Maximum hashtags to return
        max_subreddits: Maximum subreddits to return
    Returns:
        Dict with keys: x_handles, x_hashtags, reddit_subreddits
    """
    handles = _extract_x_handles(x_items)
    hashtags = _extract_x_hashtags(x_items)
    subreddits = _extract_subreddits(reddit_items)
    return {
        "x_handles": handles[:max_handles],
        "x_hashtags": hashtags[:max_hashtags],
        "reddit_subreddits": subreddits[:max_subreddits],
    }
 def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank @handles from X results.
    Sources handles from:
    1. author_handle field (who posted)
    2. @mentions in post text (who they're talking about/to)
    Returns handles ranked by frequency, filtered for generic accounts.
    """
    handle_counts = Counter()
    for item in x_items:
        # Author handle
        author = item.get("author_handle", "").strip().lstrip("@").lower()
        if author and author not in GENERIC_HANDLES:
            handle_counts[author] += 1
        # @mentions in text
        text = item.get("text", "")
        mentions = re.findall(r'@(\w{1,15})', text)
        for mention in mentions:
            mention_lower = mention.lower()
            if mention_lower not in GENERIC_HANDLES:
                handle_counts[mention_lower] += 1
    # Return all handles ranked by frequency
    return [h for h, _ in handle_counts.most_common()]
 def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank #hashtags from X results.
    Returns hashtags ranked by frequency.
    """
    hashtag_counts = Counter()
    for item in x_items:
        text = item.get("text", "")
        tags = re.findall(r'#(\w{2,30})', text)
        for tag in tags:
            hashtag_counts[tag.lower()] += 1
    # Return all hashtags ranked by frequency
    return [f"#{t}" for t, _ in hashtag_counts.most_common()]
 def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank subreddits from Reddit results.
    Sources from:
    1. subreddit field on each result
    2. Cross-references in comment text (e.g., "check out r/localLLaMA")
    Returns subreddits ranked by frequency.
    """
    sub_counts = Counter()
    for item in reddit_items:
        # Primary subreddit
        sub = item.get("subreddit", "").strip().removeprefix("r/")
        if sub:
            sub_counts[sub] += 1
        # Cross-references in comment insights
        for insight in item.get("comment_insights", []):
            cross_refs = re.findall(r'r/(\w{2,30})', insight)
            for ref in cross_refs:
                sub_counts[ref] += 1
        # Cross-references in top comments
        for comment in item.get("top_comments", []):
            excerpt = comment.get("excerpt", "")
            cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
            for ref in cross_refs:
                sub_counts[ref] += 1
    # Return subreddits ranked by frequency
    return [sub for sub, _ in sub_counts.most_common()]
@@ -0,0 +1,744 @@
 """Environment and API key management for last30days skill."""
 from __future__ import annotations
 import base64
 import binascii
 import json
 import os
 import sys
 import time
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any, Literal
 # Allow override via environment variable for testing
 # Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
 # Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
 _config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
 if _config_override == "":
    # Empty string = no config file (clean mode)
    CONFIG_DIR = None
    CONFIG_FILE = None
 elif _config_override:
    CONFIG_DIR = Path(_config_override)
    CONFIG_FILE = CONFIG_DIR / ".env"
 else:
    CONFIG_DIR = Path.home() / ".config" / "last30days"
    CONFIG_FILE = CONFIG_DIR / ".env"
 CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))
 # macOS Keychain integration: items stored with this service prefix are picked
 # up automatically on Darwin as the lowest-priority credential source.
 # Example: `security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."`.
 KEYCHAIN_SERVICE_PREFIX = "last30days-"
 # Single source of truth for which credentials the Keychain loader looks up.
 # The setup-keychain.sh helper mirrors this list and is held in sync via
 # tests/test_env_keychain.py::test_keychain_keys_match_setup_script.
 KEYCHAIN_KEYS = (
    "OPENAI_API_KEY", "XAI_API_KEY", "GOOGLE_API_KEY", "GEMINI_API_KEY",
    "GOOGLE_GENAI_API_KEY", "SCRAPECREATORS_API_KEY", "APIFY_API_TOKEN",
    "AUTH_TOKEN", "CT0", "BSKY_HANDLE", "BSKY_APP_PASSWORD",
    "TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
    "OPENROUTER_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",
    "XIAOHONGSHU_API_BASE",
 )
 AuthSource = Literal["api_key", "codex", "none"]
 AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]
 AUTH_SOURCE_API_KEY: AuthSource = "api_key"
 AUTH_SOURCE_CODEX: AuthSource = "codex"
 AUTH_SOURCE_NONE: AuthSource = "none"
 AUTH_STATUS_OK: AuthStatus = "ok"
 AUTH_STATUS_MISSING: AuthStatus = "missing"
 AUTH_STATUS_EXPIRED: AuthStatus = "expired"
 AUTH_STATUS_MISSING_ACCOUNT_ID: AuthStatus = "missing_account_id"
@dataclass(frozen=True)
 class OpenAIAuth:
    token: str | None
    source: AuthSource
    status: AuthStatus
    account_id: str | None
    codex_auth_file: str
 def _check_file_permissions(path: Path) -> None:
    """Warn to stderr if a secrets file has overly permissive permissions."""
    if os.name == "nt":
        # Windows reports synthesized POSIX mode bits that do not reflect NTFS ACLs.
        return
    try:
        mode = path.stat().st_mode
        # Check if group or other can read (bits 0o044)
        if mode & 0o044:
            sys.stderr.write(
                f"[last30days] WARNING: {path} is readable by other users. "
                f"Run: chmod 600 {path}\n"
            )
            sys.stderr.flush()
    except OSError as exc:
        sys.stderr.write(f"[last30days] WARNING: could not stat {path}: {exc}\n")
        sys.stderr.flush()
 def load_env_file(path: Path) -> dict[str, str]:
    """Load environment variables from a file."""
    env = {}
    if not path or not path.exists():
        return env
    _check_file_permissions(path)
    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            if '=' in line:
                key, _, value = line.partition('=')
                key = key.strip()
                value = value.strip()
                # Remove quotes if present
                if value and value[0] in ('"', "'") and value[-1] == value[0]:
                    value = value[1:-1]
                if key and value:
                    env[key] = value
    return env
 def _load_keychain(keys: list[str]) -> dict[str, str]:
    """Load credentials from macOS Keychain (no-op on other platforms).
    Each key is looked up as a generic password with service name
    ``f"{KEYCHAIN_SERVICE_PREFIX}{key}"`` for the current user. Missing items
    and lookup failures are silent — Keychain is the lowest-priority source
    and is meant to be additive over `.env` files and process environment.
    """
    import platform
    if platform.system() != "Darwin":
        return {}
    import shutil
    security = shutil.which("security")
    if not security:
        return {}
    import subprocess
    import pwd
    # USER can be unset under sudo, in Docker without --env USER, or in some CI
    # runners; fall back to the OS user record so lookups still match items
    # stored by setup-keychain.sh (which uses $USER).
    user = os.environ.get("USER") or pwd.getpwuid(os.getuid()).pw_name
    env: dict[str, str] = {}
    for key in keys:
        try:
            result = subprocess.run(
                [security, "find-generic-password",
                 "-a", user,
                 "-s", f"{KEYCHAIN_SERVICE_PREFIX}{key}",
                 "-w"],
                capture_output=True, text=True, timeout=5,
            )
        except (subprocess.TimeoutExpired, OSError):
            continue
        if result.returncode == 0 and result.stdout.strip():
            env[key] = result.stdout.strip()
    return env
 def _decode_jwt_payload(token: str) -> dict[str, Any] | None:
    """Decode JWT payload without verification."""
    try:
        parts = token.split(".")
        if len(parts) < 2:
            return None
        payload_b64 = parts[1]
        pad = "=" * (-len(payload_b64) % 4)
        decoded = base64.urlsafe_b64decode(payload_b64 + pad)
        return json.loads(decoded.decode("utf-8"))
    except (json.JSONDecodeError, UnicodeDecodeError, binascii.Error, IndexError) as exc:
        sys.stderr.write(f"[last30days] WARNING: malformed JWT token: {exc}\n")
        sys.stderr.flush()
        return None
 def _token_expired(token: str, leeway_seconds: int = 60) -> bool:
    """Check if JWT token is expired."""
    payload = _decode_jwt_payload(token)
    if not payload:
        return False
    exp = payload.get("exp")
    if not exp:
        return False
    return exp <= (time.time() + leeway_seconds)
 def extract_chatgpt_account_id(access_token: str) -> str | None:
    """Extract chatgpt_account_id from JWT token."""
    payload = _decode_jwt_payload(access_token)
    if not payload:
        return None
    auth_claim = payload.get("https://api.openai.com/auth", {})
    if isinstance(auth_claim, dict):
        return auth_claim.get("chatgpt_account_id")
    return None
 def load_codex_auth(path: Path = CODEX_AUTH_FILE) -> dict[str, Any]:
    """Load Codex auth JSON."""
    if not path.exists():
        return {}
    try:
        with open(path, "r") as f:
            return json.load(f)
    except json.JSONDecodeError:
        sys.stderr.write(
            f"[last30days] WARNING: {path} exists but contains invalid JSON -- ignoring\n"
        )
        sys.stderr.flush()
        return {}
 def get_codex_access_token() -> tuple[str | None, str]:
    """Get Codex access token from auth.json.
    Returns:
        (token, status) where status is 'ok', 'missing', or 'expired'
    """
    auth = load_codex_auth()
    token = None
    if isinstance(auth, dict):
        tokens = auth.get("tokens") or {}
        if isinstance(tokens, dict):
            token = tokens.get("access_token")
        if not token:
            token = auth.get("access_token")
    if not token:
        return None, AUTH_STATUS_MISSING
    if _token_expired(token):
        return None, AUTH_STATUS_EXPIRED
    return token, AUTH_STATUS_OK
 def get_openai_auth(file_env: dict[str, str]) -> OpenAIAuth:
    """Resolve OpenAI auth from API key or Codex login."""
    api_key = os.environ.get('OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY')
    if api_key:
        return OpenAIAuth(
            token=api_key,
            source=AUTH_SOURCE_API_KEY,
            status=AUTH_STATUS_OK,
            account_id=None,
            codex_auth_file=str(CODEX_AUTH_FILE),
        )
    # Codex auth (chatgpt.com backend) intentionally skipped.
    # The endpoint is unstable and causes crashes when the token expires.
    # Users who want OpenAI should set OPENAI_API_KEY explicitly.
    return OpenAIAuth(
        token=None,
        source=AUTH_SOURCE_NONE,
        status=AUTH_STATUS_MISSING,
        account_id=None,
        codex_auth_file=str(CODEX_AUTH_FILE),
    )
 def _find_project_env() -> Path | None:
    """Find per-project .env by walking up from cwd.
    Searches for .claude/last30days.env in each parent directory,
    stopping at the user's home directory or filesystem root.
    """
    cwd = Path.cwd()
    for parent in [cwd, *cwd.parents]:
        candidate = parent / '.claude' / 'last30days.env'
        if candidate.exists():
            return candidate
        # Stop at filesystem root or home
        if parent == Path.home() or parent == parent.parent:
            break
    return None
 def get_config() -> dict[str, Any]:
    """Load configuration from multiple sources.
    Priority (highest wins):
      1. Environment variables (os.environ)
      2. .claude/last30days.env (per-project config)
      3. ~/.config/last30days/.env (global config)
      4. macOS Keychain items prefixed ``last30days-`` (Darwin only)
    """
    # Load from global config file
    file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
    # Load from per-project config (overrides global)
    project_env_path = _find_project_env()
    project_env = load_env_file(project_env_path) if project_env_path else {}
    # Merge file sources: project > global
    merged_env = {**file_env, **project_env}
    # Keychain is the lowest-priority source (Darwin only; no-op elsewhere).
    # Loaded before openai_auth so OPENAI_API_KEY can come from Keychain too.
    keychain_env = _load_keychain(list(KEYCHAIN_KEYS))
    merged_env = {**keychain_env, **merged_env}
    openai_auth = get_openai_auth(merged_env)
    # Build config: Codex/OpenAI auth + process.env > project .env > global .env
    config = {
        'OPENAI_API_KEY': openai_auth.token,
        'OPENAI_AUTH_SOURCE': openai_auth.source,
        'OPENAI_AUTH_STATUS': openai_auth.status,
        'OPENAI_CHATGPT_ACCOUNT_ID': openai_auth.account_id,
        'CODEX_AUTH_FILE': openai_auth.codex_auth_file,
    }
    keys = [
        ('XAI_API_KEY', None),
        ('GOOGLE_API_KEY', None),
        ('GEMINI_API_KEY', None),
        ('GOOGLE_GENAI_API_KEY', None),
        ('XIAOHONGSHU_API_BASE', None),
        ('LAST30DAYS_REASONING_PROVIDER', 'auto'),
        ('LAST30DAYS_PLANNER_MODEL', None),
        ('LAST30DAYS_RERANK_MODEL', None),
        ('LAST30DAYS_X_MODEL', None),
        ('LAST30DAYS_X_BACKEND', None),
        ('LAST30DAYS_STORE', None),
        ('OPENAI_MODEL_PIN', None),
        ('XAI_MODEL_PIN', None),
        ('SCRAPECREATORS_API_KEY', None),
        ('APIFY_API_TOKEN', None),
        ('AUTH_TOKEN', None),
        ('CT0', None),
        ('BSKY_HANDLE', None),
        ('BSKY_APP_PASSWORD', None),
        ('BSKY_SEARCH_HOST', None),
        ('TRUTHSOCIAL_TOKEN', None),
        ('BRAVE_API_KEY', None),
        ('EXA_API_KEY', None),
        ('SERPER_API_KEY', None),
        ('OPENROUTER_API_KEY', None),
        ('PARALLEL_API_KEY', None),
        ('XQUIK_API_KEY', None),
        ('FROM_BROWSER', None),
        ('SETUP_COMPLETE', None),
        ('INCLUDE_SOURCES', ''),
        ('EXCLUDE_SOURCES', ''),
        ('LAST30DAYS_YOUTUBE_SSH_HOST', None),
        ('LAST30DAYS_TRANSCRIPT_TIMEOUT', None),
    ]
    for key, default in keys:
        config[key] = os.environ.get(key) or merged_env.get(key, default)
    # Backward-compat: ScrapeCreators' own examples and tutorials use the
    # SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
    # CREATORS). Accept that form too so users who follow the vendor's docs
    # don't silently end up with has_scrapecreators=False. Canonical name
    # wins when both are set.
    if not config.get('SCRAPECREATORS_API_KEY'):
        legacy = os.environ.get('SCRAPE_CREATORS_API_KEY') or merged_env.get('SCRAPE_CREATORS_API_KEY')
        if legacy:
            config['SCRAPECREATORS_API_KEY'] = legacy
    # Multi-key rotation: comma-separated SCRAPECREATORS_API_KEY round-robins
    # via random.choice per run. Originally added in #268, accidentally dropped
    # in v3.0.6, restored here.
    sc_key_raw = config.get('SCRAPECREATORS_API_KEY') or ''
    if ',' in sc_key_raw:
        import random
        sc_keys = [k.strip() for k in sc_key_raw.split(',') if k.strip()]
        config['SCRAPECREATORS_API_KEY'] = random.choice(sc_keys) if sc_keys else ''
    # Track which config source was used (highest-priority file source wins
    # the label; keychain is only reported when nothing else is configured).
    if project_env_path:
        config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
    elif CONFIG_FILE and CONFIG_FILE.exists():
        config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
    elif keychain_env:
        config['_CONFIG_SOURCE'] = 'keychain'
    else:
        config['_CONFIG_SOURCE'] = 'env_only'
    # Extract browser credentials if configured
    browser_creds = extract_browser_credentials(config)
    for key, value in browser_creds.items():
        if not config.get(key):
            config[key] = value
            config[f"_{key}_SOURCE"] = "browser"
    return config
 # ---------------------------------------------------------------------------
 # Browser cookie extraction
 # ---------------------------------------------------------------------------
 COOKIE_DOMAINS: dict[str, dict[str, Any]] = {
    "x": {
        "domain": ".x.com",
        "cookies": ["auth_token", "ct0"],
        "mapping": {"auth_token": "AUTH_TOKEN", "ct0": "CT0"},
    },
    "truthsocial": {
        "domain": ".truthsocial.com",
        "cookies": ["_session_id"],
        "mapping": {"_session_id": "TRUTHSOCIAL_TOKEN"},
    },
 }
 def extract_browser_credentials(config: dict[str, Any]) -> dict[str, str]:
    """Extract auth cookies from local browsers.
    Default behavior (FROM_BROWSER unset): tries Firefox and Safari only.
    These read local files silently with no system dialogs.  Chrome is
    skipped because ``security find-generic-password`` triggers a macOS
    Keychain prompt that cannot be reliably suppressed.
    Set ``FROM_BROWSER=auto`` to also try Chrome (accepts the dialog),
    or ``FROM_BROWSER=off`` to disable extraction entirely.
    """
    from_browser = (config.get("FROM_BROWSER") or "").strip().lower()
    if from_browser == "off":
        return {}
    try:
        from . import cookie_extract
    except ImportError:
        return {}
    # Determine which browsers to try
    if from_browser in ("firefox", "chrome", "safari"):
        browsers = [from_browser]
    elif from_browser == "auto":
        browsers = ["firefox", "safari", "chrome"]
    else:
        # Default: silent browsers only (no Keychain dialog)
        browsers = ["firefox", "safari"]
    extracted: dict[str, str] = {}
    for _service, spec in COOKIE_DOMAINS.items():
        if all(config.get(env_key) for env_key in spec["mapping"].values()):
            continue
        for browser in browsers:
            try:
                cookies = cookie_extract.extract_cookies(browser, spec["domain"], spec["cookies"])
            except Exception:
                continue
            if cookies:
                for cookie_name, env_key in spec["mapping"].items():
                    if cookie_name in cookies and not config.get(env_key):
                        extracted[env_key] = cookies[cookie_name]
                break  # Found cookies for this service, stop trying browsers
    return extracted
 def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
    """Return (source, method) for X search, where method describes the auth origin."""
    if config.get("XAI_API_KEY"):
        return "xai", "xai"
    if config.get("AUTH_TOKEN") and config.get("CT0"):
        method = config.get("_AUTH_TOKEN_SOURCE", "env")
        return "bird", method
    # Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
    from . import xurl_x
    if xurl_x.is_available():
        return "xurl", "oauth2"
    return None, "none"
 def config_exists() -> bool:
    """Check if any configuration source exists."""
    if _find_project_env():
        return True
    if CONFIG_FILE:
        return CONFIG_FILE.exists()
    return False
 def get_reddit_source(config: dict[str, Any]) -> str | None:
    """Determine which Reddit backend to use.
    Returns: 'scrapecreators' or None
    """
    if config.get('SCRAPECREATORS_API_KEY'):
        return 'scrapecreators'
    return None
 def get_x_source(config: dict[str, Any]) -> str | None:
    """Determine the best available explicit X/Twitter source.
    Priority: explicit backend pin, then xAI, then Bird with explicit cookies.
    Browser-cookie probing is intentionally not used here. Automatic Keychain
    access causes popups during normal pipeline runs. Bird is only considered
    available when AUTH_TOKEN and CT0 are present explicitly.
    Args:
        config: Configuration dict from get_config()
    Returns:
        'bird' if Bird is installed and explicit cookies are configured,
        'xai' if XAI_API_KEY is configured,
        'xurl' if xurl CLI is installed and authenticated,
        None if no X source available.
    """
    # Import here to avoid circular dependency
    from . import bird_x
    preferred = (config.get('LAST30DAYS_X_BACKEND') or '').lower()
    has_bird_creds = bool(config.get('AUTH_TOKEN') and config.get('CT0'))
    if has_bird_creds:
        bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
    if preferred == 'xai':
        return 'xai' if config.get('XAI_API_KEY') else None
    if preferred == 'bird':
        return 'bird' if has_bird_creds and bird_x.is_bird_installed() else None
    if config.get('XAI_API_KEY'):
        return 'xai'
    if has_bird_creds and bird_x.is_bird_installed():
        return 'bird'
    # Fall back to xurl CLI (official X API v2, OAuth2, free developer app)
    from . import xurl_x
    if xurl_x.is_available():
        return 'xurl'
    return None
 def is_ytdlp_available() -> bool:
    """Check if yt-dlp is installed for YouTube search."""
    from . import youtube_yt
    return youtube_yt.is_ytdlp_installed()
 def is_youtube_comments_available(config: dict[str, Any]) -> bool:
    """Check if YouTube comment enrichment is available.
    Requires SCRAPECREATORS_API_KEY AND youtube_comments in INCLUDE_SOURCES.
    """
    if not config.get('SCRAPECREATORS_API_KEY'):
        return False
    include = _parse_include_sources(config)
    return 'youtube_comments' in include
 def is_tiktok_comments_available(config: dict[str, Any]) -> bool:
    """Check if TikTok comment enrichment is available.
    Requires SCRAPECREATORS_API_KEY AND tiktok_comments in INCLUDE_SOURCES.
    Mirrors the youtube_comments opt-in pattern.
    """
    if not config.get('SCRAPECREATORS_API_KEY'):
        return False
    include = _parse_include_sources(config)
    return 'tiktok_comments' in include
 def is_youtube_sc_available(config: dict[str, Any]) -> bool:
    """Check if ScrapeCreators YouTube search fallback is available.
    Used when yt-dlp is not installed or fails.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def is_hackernews_available() -> bool:
    """Check if Hacker News source is available.
    Always returns True - HN uses free Algolia API, no key needed.
    """
    return True
 def is_bluesky_available(config: dict[str, Any]) -> bool:
    """Check if Bluesky source is available.
    Requires BSKY_HANDLE and BSKY_APP_PASSWORD (app password from bsky.app/settings).
    """
    return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))
 def is_truthsocial_available(config: dict[str, Any]) -> bool:
    """Check if Truth Social source is available.
    Requires TRUTHSOCIAL_TOKEN (bearer token from browser dev tools).
    """
    return bool(config.get('TRUTHSOCIAL_TOKEN'))
 def is_polymarket_available() -> bool:
    """Check if Polymarket source is available.
    Always returns True - Gamma API is free, no key needed.
    """
    return True
 def is_tiktok_available(config: dict[str, Any]) -> bool:
    """Check if TikTok source is available (ScrapeCreators or legacy Apify).
    Returns True if SCRAPECREATORS_API_KEY or APIFY_API_TOKEN is set.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN'))
 def get_tiktok_token(config: dict[str, Any]) -> str:
    """Get TikTok API token, preferring ScrapeCreators over legacy Apify."""
    return config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN') or ''
 def _parse_include_sources(config: dict[str, Any]) -> set[str]:
    """Parse INCLUDE_SOURCES config value into a set of lowercase source names."""
    raw = config.get('INCLUDE_SOURCES') or ''
    return {s.strip().lower() for s in raw.split(',') if s.strip()}
 def is_threads_available(config: dict[str, Any]) -> bool:
    """Check if Threads source is available.
    Returns True when SCRAPECREATORS_API_KEY is set. Threads runs alongside
    TikTok and Instagram as part of the SC family — same key, same per-call
    cost shape, so the same default-on rule applies. Suppress via
    EXCLUDE_SOURCES=threads.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def is_instagram_available(config: dict[str, Any]) -> bool:
    """Check if Instagram source is available (ScrapeCreators).
    Returns True if SCRAPECREATORS_API_KEY is set.
    Instagram uses the same key as TikTok.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def get_instagram_token(config: dict[str, Any]) -> str:
    """Get Instagram API token (same ScrapeCreators key as TikTok)."""
    return config.get('SCRAPECREATORS_API_KEY') or ''
 def get_xiaohongshu_api_base(config: dict[str, Any]) -> str:
    """Get Xiaohongshu HTTP API base URL.
    Defaults to host.docker.internal so OpenClaw Docker can reach host service.
    """
    return (config.get('XIAOHONGSHU_API_BASE') or "http://host.docker.internal:18060").rstrip("/")
 def is_xiaohongshu_available(config: dict[str, Any]) -> bool:
    """Check whether Xiaohongshu HTTP API is reachable and logged in."""
    # Import here to avoid heavy imports at module load.
    from . import http
    base = get_xiaohongshu_api_base(config)
    try:
        # Keep health probe snappy, but allow one retry for transient hiccups.
        health = http.get(f"{base}/health", timeout=3, retries=2)
        if not isinstance(health, dict):
            return False
        if not health.get("success"):
            return False
        # Login probe can be slower on some deployments (browser/session checks),
        # so use a slightly longer timeout to avoid false negatives.
        login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=2)
        is_logged_in = (
            login.get("data", {}).get("is_logged_in")
            if isinstance(login, dict) else False
        )
        return bool(is_logged_in)
    except (OSError, http.HTTPError):
        return False
    except Exception as exc:
        sys.stderr.write(
            f"[last30days] WARNING: unexpected error checking Xiaohongshu: "
            f"{type(exc).__name__}: {exc}\n"
        )
        sys.stderr.flush()
        return False
 # Backward compat alias
 is_apify_available = is_tiktok_available
 def get_x_source_status(config: dict[str, Any]) -> dict[str, Any]:
    """Get detailed X source status for UI decisions.
    Returns:
        Dict with keys: source, bird_installed, bird_authenticated,
        bird_username, xai_available, can_install_bird
    """
    from . import bird_x
    if config.get('AUTH_TOKEN') and config.get('CT0'):
        bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))
    bird_status = bird_x.get_bird_status()
    xai_available = bool(config.get('XAI_API_KEY'))
    # Determine active source
    if bird_status["authenticated"]:
        source = 'bird'
    elif xai_available:
        source = 'xai'
    else:
        # Fall back to xurl CLI
        from . import xurl_x as _xurl_check
        source = 'xurl' if _xurl_check.is_available() else None
    from . import xurl_x as _xurl_x
    return {
        "source": source,
        "bird_installed": bird_status["installed"],
        "bird_authenticated": bird_status["authenticated"],
        "bird_username": bird_status["username"],
        "xai_available": xai_available,
        "xurl_available": _xurl_x.is_available(),
        "can_install_bird": bird_status["can_install"],
    }
 # Pinterest
 def is_pinterest_available(config: dict[str, Any]) -> bool:
    """Check if Pinterest source is available.
    Returns True when SCRAPECREATORS_API_KEY is set AND 'pinterest' is in
    INCLUDE_SOURCES (or requested_sources at the pipeline level).  Pinterest
    is opt-in because not every topic benefits from visual pin results.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))
 def get_pinterest_token(config: dict[str, Any]) -> str:
    """Get Pinterest API token (same ScrapeCreators key as TikTok/Instagram)."""
    return config.get('SCRAPECREATORS_API_KEY') or ''
 # Xquik
 def is_xquik_available(config: dict[str, Any]) -> bool:
    """Check if Xquik X search source is available.
    Requires XQUIK_API_KEY (API key from xquik.com).
    """
    return bool(config.get('XQUIK_API_KEY'))
 def get_xquik_token(config: dict[str, Any]) -> str:
    """Get Xquik API key."""
    return config.get('XQUIK_API_KEY') or ''
@@ -0,0 +1,85 @@
 """Parallel multi-entity fan-out for the --competitors flag.
 The orchestrator accepts a `main_runner()` for the topic and a
 `competitor_runner(entity)` for each peer. It parallelizes their execution
 via a `ThreadPoolExecutor` and collects per-entity Reports. Per-entity
 failures are logged and dropped; the run survives as long as the main topic
 plus at least one competitor succeed.
 This module owns no business logic about pipeline arguments — the caller
 (scripts/last30days.py main) builds the closures with the appropriate
 config, depth, and overrides for each entity.
 """
 from __future__ import annotations
 import sys
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Callable
 from . import schema
 # Sub-runs hit the same upstream APIs as the main topic. Cap parallelism so a
 # 6-way fan-out does not stampede a single backend's rate limit.
 MAX_PARALLEL_SUBRUNS = 6
 def _log(msg: str) -> None:
    print(f"[Fanout] {msg}", file=sys.stderr)
 def run_competitor_fanout(
    *,
    main_topic: str,
    main_runner: Callable[[], schema.Report],
    competitors: list[str],
    competitor_runner: Callable[[str], schema.Report],
 ) -> list[tuple[str, schema.Report]]:
    """Run main + competitor pipelines in parallel; return surviving reports.
    Args:
        main_topic: Display label for the user's primary topic.
        main_runner: Zero-arg callable returning the main topic's Report.
        competitors: Ordered list of competitor entity names.
        competitor_runner: Callable(entity_name) -> Report for each peer.
    Returns:
        Ordered list of (entity_name, Report) tuples for runs that succeeded.
        Empty list if every run raised; the caller decides how to surface
        partial-failure modes.
    """
    if not competitors:
        report = main_runner()
        return [(main_topic, report)]
    workers = min(len(competitors) + 1, MAX_PARALLEL_SUBRUNS)
    def _run_one(label: str, fn: Callable[[], schema.Report]) -> tuple[str, schema.Report | None, Exception | None]:
        try:
            return label, fn(), None
        except Exception as exc:
            return label, None, exc
    submissions: list[tuple[str, Callable[[], schema.Report]]] = [
        (main_topic, main_runner),
    ]
    for entity in competitors:
        submissions.append((entity, lambda e=entity: competitor_runner(e)))
    with ThreadPoolExecutor(max_workers=workers) as executor:
        futures = {
            executor.submit(_run_one, label, fn): label
            for label, fn in submissions
        }
        results: dict[str, schema.Report] = {}
        for future in as_completed(futures):
            label, report, exc = future.result()
            if exc is not None:
                _log(f"Sub-run failed for {label!r}: {type(exc).__name__}: {exc}")
                continue
            assert report is not None
            results[label] = report
    # Preserve the original submission order rather than completion order so
    # the comparison render is deterministic across runs.
    return [(label, results[label]) for label, _ in submissions if label in results]
@@ -0,0 +1,207 @@
 """Weighted reciprocal rank fusion for per-(subquery, source) streams."""
 from __future__ import annotations
 from urllib.parse import parse_qs, urlencode, urlparse, urlunparse
 from . import schema
 # Standard RRF smoothing constant (Cormack et al. 2009)
 RRF_K = 60
 def _candidate_sort_key(c: schema.Candidate) -> tuple:
    return (-c.rrf_score, -c.local_relevance, -c.freshness, schema.candidate_source_label(c), c.title)
 def _normalize_url(url: str) -> str:
    """Normalize URL for dedup: lowercase, strip www/old/m prefixes, remove tracking params."""
    parsed = urlparse(url.strip().lower())
    netloc = parsed.netloc
    for prefix in ("www.", "old.", "m."):
        if netloc.startswith(prefix):
            netloc = netloc[len(prefix):]
    # Strip tracking params
    params = parse_qs(parsed.query)
    clean_params = {k: v for k, v in params.items() if not k.startswith("utm_")}
    query = urlencode(clean_params, doseq=True)
    return urlunparse((parsed.scheme, netloc, parsed.path.rstrip("/"), "", query, ""))
 def candidate_key(item: schema.SourceItem) -> str:
    if item.url:
        return _normalize_url(item.url)
    return f"{item.source}:{item.item_id}"
 _DIVERSITY_RELEVANCE_THRESHOLD = 0.25
 # Per-author cap: no single author/handle should dominate the pool.
 _MAX_ITEMS_PER_AUTHOR = 3
 def _extract_author(candidate: schema.Candidate) -> str | None:
    """Return a normalized author key from a candidate's source items."""
    for item in candidate.source_items:
        if item.author:
            return item.author.strip().lower()
    return None
 def _apply_per_author_cap(
    candidates: list[schema.Candidate],
    max_per_author: int = _MAX_ITEMS_PER_AUTHOR,
 ) -> list[schema.Candidate]:
    """Keep at most *max_per_author* items from any single author.
    Candidates are assumed to already be sorted by quality (rrf_score etc.),
    so the first N encountered per author are the best ones.
    """
    author_counts: dict[str, int] = {}
    result: list[schema.Candidate] = []
    for c in candidates:
        author = _extract_author(c)
        if author is None:
            result.append(c)
            continue
        count = author_counts.get(author, 0)
        if count < max_per_author:
            result.append(c)
            author_counts[author] = count + 1
    return result
 def _diversify_pool(
    fused: list[schema.Candidate],
    pool_limit: int,
    min_per_source: int = 2,
 ) -> list[schema.Candidate]:
    """Ensure at least *min_per_source* items per qualifying source survive truncation.
    Sources only qualify for reserved slots if their best item exceeds
    the relevance threshold. Low-relevance sources compete on merit only.
    """
    max_relevance: dict[str, float] = {}
    for c in fused:
        current = max_relevance.get(c.source, 0.0)
        if c.local_relevance > current:
            max_relevance[c.source] = c.local_relevance
    reserved: dict[str, list[schema.Candidate]] = {}
    remainder: list[schema.Candidate] = []
    for c in fused:
        qualifies = max_relevance.get(c.source, 0.0) >= _DIVERSITY_RELEVANCE_THRESHOLD
        bucket = reserved.setdefault(c.source, [])
        if qualifies and len(bucket) < min_per_source:
            bucket.append(c)
        else:
            remainder.append(c)
    pool = [c for per_source in reserved.values() for c in per_source]
    seen = {c.candidate_id for c in pool}
    for c in remainder:
        if len(pool) >= pool_limit:
            break
        if c.candidate_id not in seen:
            pool.append(c)
    pool.sort(key=_candidate_sort_key)
    return pool[:pool_limit]
 def weighted_rrf(
    streams: dict[tuple[str, str], list[schema.SourceItem]],
    plan: schema.QueryPlan,
    *,
    pool_limit: int,
 ) -> list[schema.Candidate]:
    """Fuse ranked lists into a single candidate pool."""
    subqueries = {subquery.label: subquery for subquery in plan.subqueries}
    candidates: dict[str, schema.Candidate] = {}
    # Track (source, item_id) pairs already attached to each candidate for O(1) dedup.
    seen_source_items: dict[str, set[tuple[str, str]]] = {}
    for (label, source), items in streams.items():
        subquery = subqueries[label]
        weight = subquery.weight * plan.source_weights.get(source, 1.0)
        for rank, item in enumerate(items, start=1):
            key = candidate_key(item)
            score = weight / (RRF_K + rank)
            item_local_relevance = item.local_relevance if item.local_relevance is not None else float(item.metadata.get("local_relevance", item.relevance_hint))
            item_freshness = item.freshness if item.freshness is not None else int(item.metadata.get("freshness", 0))
            item_source_quality = item.source_quality if item.source_quality is not None else float(item.metadata.get("source_quality", 0.6))
            if key not in candidates:
                candidates[key] = schema.Candidate(
                    candidate_id=key,
                    item_id=item.item_id,
                    source=item.source,
                    title=item.title,
                    url=item.url,
                    snippet=item.snippet,
                    subquery_labels=[label],
                    native_ranks={f"{label}:{source}": rank},
                    local_relevance=item_local_relevance,
                    freshness=item_freshness,
                    engagement=item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score"),
                    source_quality=item_source_quality,
                    rrf_score=score,
                    sources=[item.source],
                    source_items=[item],
                    metadata={
                        "provenance": [
                            {
                                "source": source,
                                "subquery_label": label,
                                "native_rank": rank,
                                "item_id": item.item_id,
                            }
                        ]
                    },
                )
                seen_source_items[key] = {(item.source, item.item_id)}
                continue
            candidate = candidates[key]
            candidate.rrf_score += score
            previous_primary_score = (candidate.local_relevance * 100.0) + candidate.freshness + (candidate.source_quality * 10.0)
            incoming_primary_score = (item_local_relevance * 100.0) + item_freshness + (item_source_quality * 10.0)
            candidate.local_relevance = max(
                candidate.local_relevance,
                item_local_relevance,
            )
            candidate.freshness = max(candidate.freshness, item_freshness)
            item_eng = item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score")
            if candidate.engagement is None:
                candidate.engagement = item_eng
            elif item_eng is not None:
                candidate.engagement = max(candidate.engagement, item_eng)
            candidate.source_quality = max(
                candidate.source_quality,
                item_source_quality,
            )
            candidate.native_ranks[f"{label}:{source}"] = rank
            if label not in candidate.subquery_labels:
                candidate.subquery_labels.append(label)
            if item.source not in candidate.sources:
                candidate.sources.append(item.source)
            source_item_key = (item.source, item.item_id)
            if source_item_key not in seen_source_items[key]:
                seen_source_items[key].add(source_item_key)
                candidate.source_items.append(item)
            candidate.metadata.setdefault("provenance", []).append(
                {
                    "source": source,
                    "subquery_label": label,
                    "native_rank": rank,
                    "item_id": item.item_id,
                }
            )
            if incoming_primary_score > previous_primary_score:
                candidate.item_id = item.item_id
                candidate.source = item.source
                candidate.title = item.title
                candidate.snippet = item.snippet
            if len(candidate.snippet.split()) < len(item.snippet.split()):
                candidate.snippet = item.snippet
    fused = sorted(candidates.values(), key=_candidate_sort_key)
    fused = _apply_per_author_cap(fused)
    return _diversify_pool(fused, pool_limit)
@@ -0,0 +1,989 @@
 """GitHub Issues/PRs search via the public GitHub Search API.
 Uses api.github.com/search/issues for issue/PR discovery and
 per-item comment enrichment. Auth via GITHUB_TOKEN env var or
 `gh auth token` subprocess fallback.
 """
 import json
 import math
 import os
 import re
 import subprocess
 import sys
 import urllib.error
 import urllib.parse
 import urllib.request
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, Dict, List, Optional
 from . import dates, log
 from .query import extract_core_subject
 from .relevance import token_overlap_relevance
 SEARCH_URL = "https://api.github.com/search/issues"
 DEPTH_LIMITS = {
    "quick": 15,
    "default": 30,
    "deep": 60,
 }
 ENRICH_LIMITS = {
    "quick": 3,
    "default": 5,
    "deep": 8,
 }
 USER_AGENT = "last30days/3.0 (research tool)"
 def _log(msg: str):
    log.source_log("GitHub", msg, tty_only=False)
 def _resolve_token(token: Optional[str] = None) -> Optional[str]:
    """Resolve GitHub auth token from argument, env, or gh CLI."""
    if token:
        return token
    env_token = os.environ.get("GITHUB_TOKEN")
    if env_token:
        return env_token
    # Fallback: try gh CLI
    try:
        result = subprocess.run(
            ["gh", "auth", "token"],
            capture_output=True, text=True, timeout=5,
        )
        if result.returncode == 0 and result.stdout.strip():
            return result.stdout.strip()
    except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
        pass
    return None
 def resolve_token(token: Optional[str] = None) -> Optional[str]:
    """Public alias for ``_resolve_token``.
    The pipeline calls this once before ``search_github`` and
    ``enrich_with_comments`` so the ``gh auth token`` subprocess fallback
    only fires once per query when ``GITHUB_TOKEN`` is unset, instead of
    twice (once per call site).
    """
    return _resolve_token(token)
 def _fetch_json(
    url: str,
    token: Optional[str] = None,
    timeout: int = 15,
 ) -> Optional[Dict[str, Any]]:
    """Fetch JSON from GitHub API. Returns None on failure."""
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/vnd.github+json",
    }
    if token:
        headers["Authorization"] = f"Bearer {token}"
    req = urllib.request.Request(url, headers=headers)
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = resp.read().decode("utf-8")
            return json.loads(body)
    except urllib.error.HTTPError as e:
        if e.code == 403:
            _log(f"403 rate limited or forbidden: {url}")
            return None
        if e.code == 422:
            _log(f"422 unprocessable: {url}")
            return None
        _log(f"HTTP {e.code}: {e.reason}")
        return None
    except (urllib.error.URLError, OSError, TimeoutError) as e:
        _log(f"Network error: {e}")
        return None
    except json.JSONDecodeError as e:
        _log(f"JSON decode error: {e}")
        return None
 def _parse_repo_from_url(html_url: str) -> str:
    """Extract 'owner/repo' from a GitHub issue/PR URL."""
    parts = html_url.replace("https://github.com/", "").split("/")
    if len(parts) >= 2:
        return f"{parts[0]}/{parts[1]}"
    return ""
 def _parse_date(iso_str: Optional[str]) -> Optional[str]:
    """Parse a GitHub ISO 8601 datetime string and return YYYY-MM-DD.
    Returns None for non-date input. GitHub's API always emits ISO 8601
    (e.g. "2026-02-26T16:00:00Z"), but we defer to dates.parse_date() so
    garbage input gets rejected instead of silently sliced.
    """
    dt = dates.parse_date(iso_str)
    return dt.strftime("%Y-%m-%d") if dt else None
 def _compute_relevance(
    query: str,
    title: str,
    rank_index: int,
    reactions: int,
    comments: int,
 ) -> float:
    """Blend text relevance with engagement signals."""
    rank_score = max(0.3, 1.0 - (rank_index * 0.02))
    engagement_boost = min(0.2, math.log1p(reactions + comments) / 20)
    if query:
        content_score = token_overlap_relevance(query, title)
        relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
    else:
        relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
    return round(relevance, 2)
 def search_github(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: Optional[str] = None,
 ) -> Dict[str, Any]:
    """Search GitHub Issues and PRs (HTTP fetch only).
    Returns a raw envelope shaped like every other adapter's ``search_X``:
    ``{"items": [raw GitHub API items], "context": {core, from_date,
    to_date, count}}``. Normalization, date filtering, and sorting move
    to ``parse_github_response``; comment enrichment moves to
    ``enrich_with_comments``.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: Optional GitHub token (falls back to env/gh CLI)
    Returns:
        Dict envelope. Empty ``items`` list on any failure.
    """
    count = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
    core = extract_core_subject(topic)
    resolved_token = _resolve_token(token)
    if not resolved_token:
        _log("No GitHub token available (set GITHUB_TOKEN or install gh CLI)")
        return {
            "items": [],
            "error": "no token",
            "context": {
                "core": core,
                "from_date": from_date,
                "to_date": to_date,
                "count": count,
            },
        }
    _log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
    # Build search query with date filter
    q = f"{core} created:>{from_date}"
    params = {
        "q": q,
        "sort": "reactions",
        "order": "desc",
        "per_page": str(min(count, 100)),
    }
    url = f"{SEARCH_URL}?{urllib.parse.urlencode(params)}"
    data = _fetch_json(url, token=resolved_token, timeout=30)
    if not data:
        return {"items": [], "context": {"core": core, "from_date": from_date,
                                          "to_date": to_date, "count": count}}
    raw_items = data.get("items", [])
    _log(f"Found {len(raw_items)} issues/PRs")
    return {
        "items": raw_items,
        "context": {
            "core": core,
            "from_date": from_date,
            "to_date": to_date,
            "count": count,
        },
    }
 def parse_github_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Normalize a ``search_github`` envelope into the skill's item shape.
    Pure function: no I/O, no token, no enrichment. Applies the date
    filter using the search context and sorts by relevance.
    """
    if not isinstance(response, dict):
        return []
    raw_items = response.get("items") or []
    if not isinstance(raw_items, list):
        return []
    context = response.get("context") or {}
    core = context.get("core") or ""
    from_date = context.get("from_date") or ""
    to_date = context.get("to_date") or ""
    count = context.get("count") or DEPTH_LIMITS["default"]
    items: List[Dict[str, Any]] = []
    for i, item in enumerate(raw_items[:count]):
        html_url = item.get("html_url", "")
        repo = _parse_repo_from_url(html_url)
        title = item.get("title", "")
        body_text = item.get("body") or ""
        reactions_total = item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0
        comment_count = item.get("comments", 0)
        labels = [
            lbl.get("name", "") for lbl in (item.get("labels") or [])
            if isinstance(lbl, dict)
        ]
        state = item.get("state", "")
        is_pr = "pull_request" in item
        author = item.get("user", {}).get("login", "") if isinstance(item.get("user"), dict) else ""
        relevance = _compute_relevance(core, title, i, reactions_total, comment_count)
        items.append({
            "id": f"GH{i + 1}",
            "title": title,
            "url": html_url,
            "date": _parse_date(item.get("created_at")),
            "author": author,
            "source": "github",
            "score": reactions_total,
            "container": repo,
            "snippet": body_text[:300] if body_text else "",
            "relevance": relevance,
            "why_relevant": f"GitHub {'PR' if is_pr else 'issue'}: {title[:60]}",
            "engagement": {
                "reactions": reactions_total,
                "comments": comment_count,
            },
            "metadata": {
                "labels": labels,
                "state": state,
                "comment_count": comment_count,
                "reactions": reactions_total,
                "is_pr": is_pr,
            },
        })
    # Date filter
    if from_date and to_date:
        items = [
            item for item in items
            if item.get("date") is None or (from_date <= item["date"] <= to_date)
        ]
    items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
    return items
 def enrich_with_comments(
    items: List[Dict[str, Any]],
    depth: str = "default",
    token: Optional[str] = None,
 ) -> List[Dict[str, Any]]:
    """Fetch top comments for top-K items by reactions and attach to metadata.
    Mutates and returns ``items``. Resolves ``token`` via env/gh CLI when
    not supplied, matching ``search_github``'s fallback chain.
    """
    if not items:
        return items
    resolved_token = _resolve_token(token)
    if not resolved_token:
        _log("No GitHub token available for comment enrichment")
        return items
    return _enrich_top_items(items, depth, resolved_token)
 def _enrich_top_items(
    items: List[Dict[str, Any]],
    depth: str,
    token: str,
 ) -> List[Dict[str, Any]]:
    """Fetch comments for top N items by reactions."""
    if not items:
        return items
    limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
    by_reactions = sorted(
        range(len(items)),
        key=lambda i: items[i].get("score", 0),
        reverse=True,
    )
    to_enrich = by_reactions[:limit]
    _log(f"Enriching top {len(to_enrich)} items with comments")
    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = {
            executor.submit(
                _fetch_item_comments,
                items[idx]["url"],
                token,
            ): idx
            for idx in to_enrich
        }
        for future in as_completed(futures):
            idx = futures[future]
            try:
                comments = future.result(timeout=15)
                items[idx]["metadata"]["top_comments"] = comments
            except (KeyError, TypeError, OSError) as exc:
                _log(f"Comment enrichment failed for {items[idx].get('url', '?')}: {type(exc).__name__}: {exc}")
                items[idx]["metadata"]["top_comments"] = []
    return items
 def _fetch_item_comments(
    issue_url: str,
    token: str,
    max_comments: int = 5,
 ) -> List[Dict[str, Any]]:
    """Fetch comments for a GitHub issue/PR.
    Args:
        issue_url: HTML URL like https://github.com/owner/repo/issues/123
        token: GitHub auth token
        max_comments: Max comments to return
    Returns:
        List of comment dicts with score, excerpt, author.
    """
    path = issue_url.replace("https://github.com/", "")
    path = path.replace("/pull/", "/issues/")
    api_url = f"https://api.github.com/repos/{path}/comments?per_page={max_comments}&sort=reactions&direction=desc"
    data = _fetch_json(api_url, token=token, timeout=15)
    if not data or not isinstance(data, list):
        return []
    comments = []
    for c in data[:max_comments]:
        body = c.get("body") or ""
        excerpt = body[:300] + "..." if len(body) > 300 else body
        reactions = c.get("reactions", {})
        reaction_count = reactions.get("total_count", 0) if isinstance(reactions, dict) else 0
        author = c.get("user", {}).get("login", "") if isinstance(c.get("user"), dict) else ""
        comments.append({
            "score": reaction_count,
            "excerpt": excerpt,
            "author": author,
        })
    return comments
 # ---------------------------------------------------------------------------
 # Person-mode search: author-scoped queries, star enrichment, release notes
 # ---------------------------------------------------------------------------
 PERSON_DEPTH_LIMITS = {
    "quick": {"pr_pages": 1, "own_repos": 3, "external_repos": 5},
    "default": {"pr_pages": 1, "own_repos": 5, "external_repos": 10},
    "deep": {"pr_pages": 2, "own_repos": 5, "external_repos": 15},
 }
 def _fetch_readme_snippet(repo: str, token: str, max_chars: int = 500) -> Optional[str]:
    """Fetch README content for a repo, truncated to first ~max_chars."""
    url = f"https://api.github.com/repos/{repo}/readme"
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/vnd.github.raw+json",
    }
    if token:
        headers["Authorization"] = f"Bearer {token}"
    req = urllib.request.Request(url, headers=headers)
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            raw = resp.read().decode("utf-8", errors="replace")
    except (urllib.error.HTTPError, urllib.error.URLError, OSError, TimeoutError):
        return None
    if not raw:
        return None
    # Try to break at a paragraph boundary
    if len(raw) <= max_chars:
        return raw
    cut = raw[:max_chars]
    last_double_newline = cut.rfind("\n\n")
    if last_double_newline > max_chars // 3:
        return cut[:last_double_newline].rstrip()
    return cut.rstrip() + "..."
 def _fetch_latest_releases(
    repo: str, token: str, count: int = 3, max_body: int = 300,
 ) -> List[Dict[str, str]]:
    """Fetch latest releases for a repo."""
    url = f"https://api.github.com/repos/{repo}/releases?per_page={count}"
    data = _fetch_json(url, token=token, timeout=10)
    if not data or not isinstance(data, list):
        return []
    releases = []
    for r in data[:count]:
        tag = r.get("tag_name", "")
        date = _parse_date(r.get("published_at"))
        body = (r.get("body") or "")[:max_body]
        name = r.get("name") or tag
        releases.append({"tag": tag, "name": name, "date": date, "body": body})
    return releases
 def _fetch_top_issues(repo: str, token: str) -> Dict[str, Any]:
    """Fetch top feature request (by reactions) and top complaint (by comments)."""
    result: Dict[str, Any] = {}
    # Top feature request: issues with enhancement label, sorted by reactions
    feat_q = urllib.parse.quote(f"repo:{repo} is:issue is:open label:enhancement")
    feat_url = f"{SEARCH_URL}?q={feat_q}&sort=reactions&order=desc&per_page=1"
    feat_data = _fetch_json(feat_url, token=token, timeout=10)
    if feat_data and feat_data.get("items"):
        item = feat_data["items"][0]
        result["top_feature_request"] = {
            "title": item.get("title", ""),
            "reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
            "comments": item.get("comments", 0),
            "url": item.get("html_url", ""),
        }
    elif feat_data and feat_data.get("total_count", 0) == 0:
        # No enhancement label; fall back to top issue by reactions
        fallback_q = urllib.parse.quote(f"repo:{repo} is:issue is:open")
        fallback_url = f"{SEARCH_URL}?q={fallback_q}&sort=reactions&order=desc&per_page=1"
        fallback_data = _fetch_json(fallback_url, token=token, timeout=10)
        if fallback_data and fallback_data.get("items"):
            item = fallback_data["items"][0]
            result["top_feature_request"] = {
                "title": item.get("title", ""),
                "reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
                "comments": item.get("comments", 0),
                "url": item.get("html_url", ""),
            }
    # Top complaint: most-discussed open issue (by comments)
    bug_q = urllib.parse.quote(f"repo:{repo} is:issue is:open")
    bug_url = f"{SEARCH_URL}?q={bug_q}&sort=comments&order=desc&per_page=1"
    bug_data = _fetch_json(bug_url, token=token, timeout=10)
    if bug_data and bug_data.get("items"):
        item = bug_data["items"][0]
        result["top_complaint"] = {
            "title": item.get("title", ""),
            "reactions": item.get("reactions", {}).get("total_count", 0) if isinstance(item.get("reactions"), dict) else 0,
            "comments": item.get("comments", 0),
            "url": item.get("html_url", ""),
        }
    return result
 def _fetch_repo_info(repo: str, token: str) -> Optional[Dict[str, Any]]:
    """Fetch repo metadata (stars, forks, description, language)."""
    url = f"https://api.github.com/repos/{repo}"
    data = _fetch_json(url, token=token, timeout=10)
    if not data or not isinstance(data, dict):
        return None
    return {
        "stars": data.get("stargazers_count", 0),
        "forks": data.get("forks_count", 0),
        "description": (data.get("description") or "")[:200],
        "language": data.get("language") or "",
        "open_issues": data.get("open_issues_count", 0),
    }
 def _format_stars(n: int) -> str:
    """Format star count as human-readable (e.g., 349K, 2.9K, 42)."""
    if n >= 1_000_000:
        return f"{n / 1_000_000:.1f}M"
    if n >= 1_000:
        return f"{n / 1_000:.0f}K" if n >= 10_000 else f"{n / 1_000:.1f}K"
    return str(n)
 def search_github_person(
    username: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: Optional[str] = None,
 ) -> List[Dict[str, Any]]:
    """Person-mode GitHub search: author-scoped queries with star enrichment.
    Returns SourceItems for:
    - 1 velocity summary item
    - Per-repo items for top external repos (with stars + release notes)
    - Per-repo items for own repos (with stars + README + top issues + releases)
    """
    resolved_token = _resolve_token(token)
    if not resolved_token:
        _log("No GitHub token available for person-mode search")
        return []
    limits = PERSON_DEPTH_LIMITS.get(depth, PERSON_DEPTH_LIMITS["default"])
    _log(f"Person-mode search for @{username} (since {from_date})")
    # Phase 1: PR velocity via search API
    total_q = urllib.parse.quote(f"author:{username} type:pr created:>{from_date}")
    merged_q = urllib.parse.quote(f"author:{username} type:pr is:merged created:>{from_date}")
    total_url = f"{SEARCH_URL}?q={total_q}&per_page=1"
    merged_url = f"{SEARCH_URL}?q={merged_q}&sort=reactions&order=desc&per_page=100"
    total_data = _fetch_json(total_url, token=resolved_token, timeout=20)
    merged_data = _fetch_json(merged_url, token=resolved_token, timeout=20)
    total_prs = total_data.get("total_count", 0) if total_data else 0
    merged_count = merged_data.get("total_count", 0) if merged_data else 0
    merged_items = merged_data.get("items", []) if merged_data else []
    _log(f"Found {total_prs} total PRs, {merged_count} merged")
    if total_prs == 0 and merged_count == 0:
        _log("No PRs found, falling back to keyword search")
        return []
    # Phase 2: Group merged PRs by repo
    repo_pr_counts: Dict[str, int] = {}
    for item in merged_items:
        repo = _parse_repo_from_url(item.get("html_url", ""))
        if repo:
            repo_pr_counts[repo] = repo_pr_counts.get(repo, 0) + 1
    # Sort repos by PR count (most active first)
    sorted_repos = sorted(repo_pr_counts.items(), key=lambda x: x[1], reverse=True)
    # Phase 3: Fetch own repos
    own_repos_url = f"https://api.github.com/users/{username}/repos?sort=stars&per_page={limits['own_repos']}&direction=desc"
    own_repos_data = _fetch_json(own_repos_url, token=resolved_token, timeout=15)
    own_repo_names = set()
    own_repos_info: List[Dict[str, Any]] = []
    if own_repos_data and isinstance(own_repos_data, list):
        for r in own_repos_data:
            full_name = r.get("full_name", "")
            if full_name and not r.get("fork"):
                own_repo_names.add(full_name)
                own_repos_info.append({
                    "full_name": full_name,
                    "stars": r.get("stargazers_count", 0),
                    "forks": r.get("forks_count", 0),
                    "description": (r.get("description") or "")[:200],
                    "language": r.get("language") or "",
                    "open_issues": r.get("open_issues_count", 0),
                })
    # Separate external repos from own repos
    external_repos = [(repo, count) for repo, count in sorted_repos if repo not in own_repo_names]
    external_repos = external_repos[:limits["external_repos"]]
    # Phase 4: Parallel enrichment (star counts, releases, READMEs, top issues)
    items: List[Dict[str, Any]] = []
    idx = 0
    # Build velocity summary
    open_prs = total_prs - merged_count
    merge_rate = round(100 * merged_count / total_prs) if total_prs > 0 else 0
    num_repos = len(repo_pr_counts)
    velocity_text = (
        f"GitHub Person Profile: @{username}\n\n"
        f"CONTRIBUTION VELOCITY (last {(to_date > from_date) and 30 or 30} days)\n"
        f"- {merged_count} PRs merged across {num_repos} repos ({merge_rate}% merge rate)\n"
        f"- {total_prs} total PRs submitted, {open_prs} still open\n"
    )
    idx += 1
    items.append({
        "id": f"GH{idx}",
        "title": f"@{username}: {merged_count} PRs merged across {num_repos} repos ({merge_rate}% merge rate)",
        "url": f"https://github.com/{username}",
        "date": to_date,
        "author": username,
        "source": "github",
        "score": merged_count,
        "container": f"@{username}",
        "snippet": velocity_text,
        "relevance": 0.95,
        "why_relevant": f"GitHub profile: @{username} - {merged_count} PRs merged across {num_repos} repos",
        "engagement": {"reactions": merged_count, "comments": total_prs},
        "metadata": {
            "labels": ["person-profile", "velocity"],
            "state": "open",
            "comment_count": 0,
            "reactions": merged_count,
            "is_pr": False,
        },
    })
    # Phase 5: Enrich external repos (parallel: star counts + releases)
    _log(f"Enriching {len(external_repos)} external repos + {len(own_repos_info)} own repos")
    with ThreadPoolExecutor(max_workers=8) as executor:
        # External repo enrichment: stars + releases
        ext_futures = {}
        for repo, pr_count in external_repos:
            ext_futures[executor.submit(_enrich_external_repo, repo, resolved_token)] = (repo, pr_count)
        # Own repo enrichment: README + releases + top issues
        own_futures = {}
        for own_repo in own_repos_info:
            own_futures[executor.submit(_enrich_own_repo, own_repo["full_name"], resolved_token)] = own_repo
        # Collect external repo results
        for future in as_completed(ext_futures):
            repo, pr_count = ext_futures[future]
            try:
                enrichment = future.result(timeout=20)
            except Exception as exc:
                _log(f"External repo enrichment failed for {repo}: {exc}")
                enrichment = {}
            repo_info = enrichment.get("info")
            releases = enrichment.get("releases", [])
            stars = repo_info["stars"] if repo_info else 0
            stars_str = _format_stars(stars)
            desc = repo_info["description"] if repo_info else ""
            snippet_parts = [f"Contributed {pr_count} merged PRs to {repo} ({stars_str} stars)"]
            if desc:
                snippet_parts.append(f"  {desc}")
            if releases:
                for rel in releases[:2]:
                    body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
                    snippet_parts.append(f"  Latest release: {rel['name']} ({rel['date']}){body_preview}")
            idx += 1
            items.append({
                "id": f"GH{idx}",
                "title": f"{repo} ({stars_str} stars) - {pr_count} PRs merged",
                "url": f"https://github.com/{repo}",
                "date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
                "author": username,
                "source": "github",
                "score": stars,
                "container": repo,
                "snippet": "\n".join(snippet_parts),
                "relevance": min(0.9, 0.6 + math.log1p(stars) / 30 + min(0.15, pr_count / 20)),
                "why_relevant": f"GitHub contribution: {pr_count} PRs merged to {repo} ({stars_str} stars)",
                "engagement": {"reactions": stars, "comments": pr_count},
                "metadata": {
                    "labels": ["person-profile", "external-repo"],
                    "state": "open",
                    "comment_count": pr_count,
                    "reactions": stars,
                    "is_pr": False,
                },
            })
        # Collect own repo results
        for future in as_completed(own_futures):
            own_repo = own_futures[future]
            try:
                enrichment = future.result(timeout=25)
            except Exception as exc:
                _log(f"Own repo enrichment failed for {own_repo['full_name']}: {exc}")
                enrichment = {}
            repo_name = own_repo["full_name"]
            stars = own_repo["stars"]
            stars_str = _format_stars(stars)
            open_issues = own_repo["open_issues"]
            desc = own_repo["description"]
            readme = enrichment.get("readme")
            releases = enrichment.get("releases", [])
            top_issues = enrichment.get("top_issues", {})
            snippet_parts = [f"Own project: {repo_name} ({stars_str} stars, {open_issues} open issues)"]
            if desc:
                snippet_parts.append(f"  {desc}")
            if readme:
                snippet_parts.append(f"  README: {readme[:300]}")
            if releases:
                for rel in releases[:2]:
                    body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
                    snippet_parts.append(f"  Latest release: {rel['name']} ({rel['date']}){body_preview}")
            feat = top_issues.get("top_feature_request")
            if feat:
                snippet_parts.append(f"  Top feature request: \"{feat['title']}\" ({feat['reactions']} reactions, {feat['comments']} comments)")
            complaint = top_issues.get("top_complaint")
            if complaint:
                snippet_parts.append(f"  Top complaint: \"{complaint['title']}\" ({complaint['comments']} comments)")
            idx += 1
            items.append({
                "id": f"GH{idx}",
                "title": f"{repo_name} ({stars_str} stars) - own project, {open_issues} open issues",
                "url": f"https://github.com/{repo_name}",
                "date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
                "author": username,
                "source": "github",
                "score": stars,
                "container": repo_name,
                "snippet": "\n".join(snippet_parts),
                "relevance": min(0.95, 0.7 + math.log1p(stars) / 25),
                "why_relevant": f"GitHub own project: {repo_name} ({stars_str} stars)",
                "engagement": {"reactions": stars, "comments": open_issues},
                "metadata": {
                    "labels": ["person-profile", "own-repo"],
                    "state": "open",
                    "comment_count": open_issues,
                    "reactions": stars,
                    "is_pr": False,
                },
            })
    # Sort by relevance
    items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
    _log(f"Person-mode returned {len(items)} items")
    return items
 def _enrich_external_repo(repo: str, token: str) -> Dict[str, Any]:
    """Fetch star count + releases for an external repo."""
    info = _fetch_repo_info(repo, token)
    releases = _fetch_latest_releases(repo, token, count=3)
    return {"info": info, "releases": releases}
 def _enrich_own_repo(repo: str, token: str) -> Dict[str, Any]:
    """Fetch README + releases + top issues for an own repo."""
    readme = _fetch_readme_snippet(repo, token, max_chars=500)
    releases = _fetch_latest_releases(repo, token, count=3)
    top_issues = _fetch_top_issues(repo, token)
    return {"readme": readme, "releases": releases, "top_issues": top_issues}
 # ---------------------------------------------------------------------------
 # Project-mode search: fetch comprehensive data for specific repos
 # ---------------------------------------------------------------------------
 def search_github_project(
    repos: List[str],
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: Optional[str] = None,
 ) -> List[Dict[str, Any]]:
    """Project-mode GitHub search: fetch stars, README, releases, top issues for repos.
    Args:
        repos: List of 'owner/repo' strings.
        from_date: Start date (YYYY-MM-DD).
        to_date: End date (YYYY-MM-DD).
        depth: 'quick', 'default', or 'deep'.
        token: Optional GitHub token.
    Returns:
        List of SourceItems, one per repo.
    """
    resolved_token = _resolve_token(token)
    if not resolved_token:
        _log("No GitHub token available for project-mode search")
        return []
    _log(f"Project-mode search for {len(repos)} repos: {', '.join(repos)}")
    items: List[Dict[str, Any]] = []
    with ThreadPoolExecutor(max_workers=min(8, len(repos))) as executor:
        futures = {
            executor.submit(_enrich_project_repo, repo, resolved_token): repo
            for repo in repos
        }
        for idx, future in enumerate(as_completed(futures)):
            repo = futures[future]
            try:
                enrichment = future.result(timeout=25)
            except Exception as exc:
                _log(f"Project enrichment failed for {repo}: {exc}")
                continue
            info = enrichment.get("info")
            if not info:
                _log(f"No repo info for {repo}, skipping")
                continue
            readme = enrichment.get("readme")
            releases = enrichment.get("releases", [])
            top_issues = enrichment.get("top_issues", {})
            stars = info["stars"]
            stars_str = _format_stars(stars)
            open_issues = info["open_issues"]
            desc = info["description"]
            lang = info["language"]
            snippet_parts = [f"Project: {repo} ({stars_str} stars, {open_issues} open issues, {lang})"]
            if desc:
                snippet_parts.append(f"  {desc}")
            if readme:
                snippet_parts.append(f"  README: {readme[:400]}")
            if releases:
                for rel in releases[:2]:
                    body_preview = f" - {rel['body'][:150]}" if rel.get("body") else ""
                    snippet_parts.append(f"  Latest release: {rel['name']} ({rel['date']}){body_preview}")
            feat = top_issues.get("top_feature_request")
            if feat:
                snippet_parts.append(f"  Top feature request: \"{feat['title']}\" ({feat['reactions']} reactions, {feat['comments']} comments)")
            complaint = top_issues.get("top_complaint")
            if complaint:
                snippet_parts.append(f"  Top complaint: \"{complaint['title']}\" ({complaint['comments']} comments)")
            items.append({
                "id": f"GH{idx + 1}",
                "title": f"{repo} ({stars_str} stars) - {open_issues} open issues",
                "url": f"https://github.com/{repo}",
                "date": releases[0]["date"] if releases and releases[0].get("date") else to_date,
                "author": repo.split("/")[0],
                "source": "github",
                "score": stars,
                "container": repo,
                "snippet": "\n".join(snippet_parts),
                "relevance": min(0.95, 0.7 + math.log1p(stars) / 25),
                "why_relevant": f"GitHub project: {repo} ({stars_str} stars, live)",
                "engagement": {"reactions": stars, "comments": open_issues},
                "metadata": {
                    "labels": ["project-mode"],
                    "state": "open",
                    "comment_count": open_issues,
                    "reactions": stars,
                    "is_pr": False,
                    "github_stars": {repo: stars},
                },
            })
    items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
    _log(f"Project-mode returned {len(items)} items")
    return items
 def _enrich_project_repo(repo: str, token: str) -> Dict[str, Any]:
    """Fetch all project data for a repo: info + README + releases + top issues."""
    info = _fetch_repo_info(repo, token)
    readme = _fetch_readme_snippet(repo, token, max_chars=500)
    releases = _fetch_latest_releases(repo, token, count=3)
    top_issues = _fetch_top_issues(repo, token)
    return {"info": info, "readme": readme, "releases": releases, "top_issues": top_issues}
 # ---------------------------------------------------------------------------
 # Post-rerank star enrichment: annotate candidates with live star counts
 # ---------------------------------------------------------------------------
 _REPO_URL_PATTERN = re.compile(r"github\.com/([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)")
 _SKIP_PATHS = {"topics", "search", "orgs", "settings", "features", "about", "pricing", "enterprise", "explore", "marketplace", "sponsors"}
 def extract_repo_refs(candidates: List[Any]) -> List[str]:
    """Extract unique owner/repo strings from candidate URLs, titles, and snippets."""
    seen: set = set()
    repos: List[str] = []
    for c in candidates:
        texts = [
            getattr(c, "url", "") or "",
            getattr(c, "title", "") or "",
        ]
        # Also check evidence snippets if available
        evidence = getattr(c, "evidence", None)
        if evidence:
            texts.append(str(evidence))
        for text in texts:
            for match in _REPO_URL_PATTERN.findall(text):
                # Normalize: strip trailing .git, lowercase
                repo = match.rstrip(".git").lower()
                owner = repo.split("/")[0]
                if owner in _SKIP_PATHS:
                    continue
                if repo not in seen:
                    seen.add(repo)
                    repos.append(match)  # preserve original case
    return repos
 def enrich_candidates_with_stars(
    candidates: List[Any],
    token: Optional[str] = None,
    already_enriched: Optional[set] = None,
    max_repos: int = 10,
 ) -> int:
    """Annotate candidates with live GitHub star counts.
    Returns the number of repos enriched.
    """
    resolved_token = _resolve_token(token)
    if not resolved_token:
        return 0
    refs = extract_repo_refs(candidates)
    if not refs:
        return 0
    skip = already_enriched or set()
    to_fetch = [r for r in refs if r.lower() not in {s.lower() for s in skip}][:max_repos]
    if not to_fetch:
        return 0
    _log(f"Star enrichment: fetching {len(to_fetch)} repos")
    # Parallel fetch star counts
    star_map: Dict[str, int] = {}
    with ThreadPoolExecutor(max_workers=min(8, len(to_fetch))) as executor:
        futures = {executor.submit(_fetch_repo_info, repo, resolved_token): repo for repo in to_fetch}
        for future in as_completed(futures):
            repo = futures[future]
            try:
                info = future.result(timeout=10)
                if info:
                    star_map[repo.lower()] = info["stars"]
            except Exception:
                pass
    if not star_map:
        return 0
    # Annotate candidates
    enriched_count = 0
    for c in candidates:
        texts = [getattr(c, "url", "") or "", getattr(c, "title", "") or ""]
        evidence = getattr(c, "evidence", None)
        if evidence:
            texts.append(str(evidence))
        combined = " ".join(texts)
        for match in _REPO_URL_PATTERN.findall(combined):
            repo_lower = match.rstrip(".git").lower()
            if repo_lower in star_map:
                stars = star_map[repo_lower]
                stars_str = _format_stars(stars)
                # Add to metadata
                if not hasattr(c, "metadata") or c.metadata is None:
                    continue
                if "github_stars" not in c.metadata:
                    c.metadata["github_stars"] = {}
                c.metadata["github_stars"][match] = stars
                # Append to evidence if present
                if hasattr(c, "evidence") and c.evidence and f"(live:" not in c.evidence:
                    c.evidence = c.evidence + f" (live: {stars_str} stars)"
                enriched_count += 1
                break  # one annotation per candidate
    _log(f"Star enrichment: annotated {enriched_count} candidates")
    return enriched_count
@@ -0,0 +1,324 @@
 """Web search retrieval via Brave Search, Exa, and Serper."""
 from __future__ import annotations
 import sys
 import urllib.parse
 from datetime import datetime
 from urllib.parse import urlparse
 from . import dates, http
 # ---------------------------------------------------------------------------
 # Brave Search API
 # ---------------------------------------------------------------------------
 def brave_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
 ) -> tuple[list[dict], dict]:
    url = (
        "https://api.search.brave.com/res/v1/web/search?"
        + urllib.parse.urlencode(
            {
                "q": query,
                "count": count,
                "freshness": f"{date_range[0]}to{date_range[1]}",
            }
        )
    )
    data = http.request("GET", url, headers={"X-Subscription-Token": api_key}, timeout=15)
    items = []
    for i, r in enumerate((data.get("web", {}).get("results", []))[:count]):
        raw_date = r.get("page_age") or ""
        pub_date = _normalize_date(raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WB{i + 1}",
            "title": r.get("title", ""),
            "url": r.get("url", ""),
            "source_domain": _domain(r.get("url", "")),
            "snippet": r.get("description", ""),
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Brave web search",
        })
    artifact = {"label": "brave", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact
 # ---------------------------------------------------------------------------
 # Exa AI Search
 # ---------------------------------------------------------------------------
 def exa_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
 ) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://api.exa.ai/search",
        headers={"x-api-key": api_key},
        json_data={
            "query": query,
            "type": "auto",
            "numResults": count,
            "startPublishedDate": f"{date_range[0]}T00:00:00.000Z",
            "endPublishedDate": f"{date_range[1]}T23:59:59.999Z",
            "contents": {"text": {"maxCharacters": 2000}},
        },
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("results", []))[:count]):
        if not isinstance(r, dict):
            continue
        url = r.get("url", "")
        if not url:
            continue
        raw_date = r.get("publishedDate") or ""
        pub_date = _normalize_date(raw_date.split("T")[0] if "T" in raw_date else raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WE{i + 1}",
            "title": r.get("title", ""),
            "url": url,
            "source_domain": _domain(url),
            "snippet": (r.get("text") or "")[:500],
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Exa web search",
        })
    artifact = {"label": "exa", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact
 # ---------------------------------------------------------------------------
 # Serper (Google Search wrapper)
 # ---------------------------------------------------------------------------
 def serper_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
 ) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://google.serper.dev/search",
        headers={"X-API-KEY": api_key},
        json_data={
            "q": query,
            "num": count,
            "tbs": f"cdr:1,cd_min:{_serper_date_param(date_range[0])},cd_max:{_serper_date_param(date_range[1])}",
        },
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("organic", []))[:count]):
        raw_date = r.get("date") or ""
        pub_date = _parse_serper_date(raw_date)
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WS{i + 1}",
            "title": r.get("title", ""),
            "url": r.get("link", ""),
            "source_domain": _domain(r.get("link", "")),
            "snippet": r.get("snippet", ""),
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Serper web search",
        })
    artifact = {"label": "serper", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact
 # ---------------------------------------------------------------------------
 # Parallel AI Search
 # ---------------------------------------------------------------------------
 def parallel_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
 ) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://api.parallel.ai/v1/search",
        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
        json_data={
            "search_queries": [query],
            "advanced_settings": {"max_results": count},
        },
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("results", []))[:count]):
        if not isinstance(r, dict):
            continue
        url = r.get("url", "")
        if not url:
            continue
        raw_date = r.get("publish_date") or ""
        pub_date = _normalize_date(raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WP{i + 1}",
            "title": r.get("title", ""),
            "url": url,
            "source_domain": _domain(url),
            "snippet": ((r.get("excerpts") or [""])[0] or "")[:500],
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Parallel AI web search",
        })
    artifact = {"label": "parallel", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact
 def _parse_serper_date(raw: str) -> str | None:
    if not raw:
        return None
    normalized = _normalize_date(raw)
    if normalized:
        return normalized
    for fmt in ("%b %d, %Y", "%B %d, %Y", "%Y-%m-%d"):
        try:
            return datetime.strptime(raw.strip(), fmt).date().isoformat()
        except ValueError:
            continue
    return None
 # ---------------------------------------------------------------------------
 # Dispatcher
 # ---------------------------------------------------------------------------
 def web_search(
    query: str,
    date_range: tuple[str, str],
    config: dict,
    backend: str = "auto",
 ) -> tuple[list[dict], dict]:
    """Run web search with the specified or auto-detected backend."""
    if backend == "auto":
        if config.get("BRAVE_API_KEY"):
            backend = "brave"
        elif config.get("EXA_API_KEY"):
            backend = "exa"
        elif config.get("SERPER_API_KEY"):
            backend = "serper"
        elif config.get("PARALLEL_API_KEY"):
            backend = "parallel"
        else:
            return [], {}
    items: list[dict] = []
    artifact: dict = {}
    if backend == "brave":
        key = config.get("BRAVE_API_KEY")
        if not key:
            raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'")
        items, artifact = brave_search(query, date_range, key)
    elif backend == "exa":
        key = config.get("EXA_API_KEY")
        if not key:
            raise RuntimeError("EXA_API_KEY is required when web_backend='exa'")
        items, artifact = exa_search(query, date_range, key)
    elif backend == "serper":
        key = config.get("SERPER_API_KEY")
        if not key:
            raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'")
        items, artifact = serper_search(query, date_range, key)
    elif backend == "parallel":
        key = config.get("PARALLEL_API_KEY")
        if not key:
            raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'")
        items, artifact = parallel_search(query, date_range, key)
    elif backend != "none":
        raise ValueError(f"Unsupported web backend: {backend!r}")
    else:
        return [], {}
    if items and not _reddit_excluded(config):
        items = _enrich_reddit_items(items)
    return items, artifact
 def _reddit_excluded(config: dict) -> bool:
    """Return True when EXCLUDE_SOURCES contains 'reddit'.
    Respects the same suppression knob the pipeline uses for source gating,
    so a user who set EXCLUDE_SOURCES=reddit doesn't get Reddit content
    smuggled back in via web-search URLs.
    """
    raw = (config.get("EXCLUDE_SOURCES") or "").split(",")
    return any(s.strip().lower() == "reddit" for s in raw)
 def _enrich_reddit_items(items: list[dict]) -> list[dict]:
    """Enrich web search results that are Reddit URLs with thread body and comments.
    Claude Code's WebFetch blocks reddit.com, so the model can't retrieve
    Reddit content from web search results. This fetches it via the public
    JSON API (reddit.com/.../.json) which bypasses that restriction.
    Callers should gate this with EXCLUDE_SOURCES=reddit handling (see
    `_reddit_excluded`) so a user who explicitly excluded Reddit doesn't
    get Reddit content via web-search URLs.
    """
    from . import reddit_enrich
    from .reddit_enrich import RedditRateLimitError
    for item in items:
        url = item.get("url", "")
        if "reddit.com" not in url or "/comments/" not in url:
            continue
        try:
            thread_data = reddit_enrich.fetch_thread_data(url, timeout=8)
            if not thread_data:
                continue
            parsed = reddit_enrich.parse_thread_data(thread_data)
            # selftext lives under parsed["submission"], not at the top level
            selftext = (parsed.get("submission") or {}).get("selftext", "")
            if selftext:
                item["snippet"] = selftext[:2000]
            comments = parsed.get("comments", [])
            top = reddit_enrich.get_top_comments(comments)
            if top:
                item["top_comments"] = [
                    {"score": c.get("score", 0), "excerpt": (c.get("body") or "")[:200]}
                    for c in top[:5]
                ]
            item["enriched_via"] = "reddit_json_api"
        except RedditRateLimitError as exc:
            # Stop iterating to avoid flooding more 429s
            sys.stderr.write(f"[Web] Reddit rate-limited, halting enrichment: {exc}\n")
            break
        except Exception as exc:
            sys.stderr.write(f"[Web] Reddit enrichment failed for {url}: {exc}\n")
    return items
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
 def _normalize_date(value: object) -> str | None:
    if value is None:
        return None
    parsed = dates.parse_date(str(value).strip())
    if not parsed:
        return None
    return parsed.date().isoformat()
 def _serper_date_param(iso_date: str) -> str:
    """Convert YYYY-MM-DD to MM/DD/YYYY for Serper tbs parameter."""
    parts = iso_date.split("-")
    return f"{parts[1]}/{parts[2]}/{parts[0]}"
 def _in_date_range(pub_date: str | None, date_range: tuple[str, str]) -> bool:
    if not pub_date:
        return False
    return date_range[0] <= pub_date <= date_range[1]
 def _domain(url: str) -> str:
    return urlparse(url).netloc.strip().lower()
@@ -0,0 +1,338 @@
 """Hacker News search via Algolia API (free, no auth required).
 Uses hn.algolia.com/api/v1 for story discovery and comment enrichment.
 No API key needed - just HTTP calls via stdlib urllib.
 """
 import datetime
 import html
 import math
 import sys
 import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, Dict, List, Optional
 import re
 from . import http, log
 from .query import extract_core_subject
 from .relevance import token_overlap_relevance
 # Common HN prefixes that can cause false-positive keyword matches
 _HN_PREFIXES = re.compile(r"^(Tell HN|Show HN|Ask HN|Launch HN)\s*:\s*", re.IGNORECASE)
 ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
 ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
 ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"
 DEPTH_CONFIG = {
    "quick": 15,
    "default": 30,
    "deep": 60,
 }
 ENRICH_LIMITS = {
    "quick": 3,
    "default": 5,
    "deep": 10,
 }
 def _log(msg: str):
    log.source_log("HN", msg)
 def _date_to_unix(date_str: str) -> int:
    """Convert YYYY-MM-DD to Unix timestamp (start of day UTC)."""
    parts = date_str.split("-")
    year, month, day = int(parts[0]), int(parts[1]), int(parts[2])
    dt = datetime.datetime(year, month, day, tzinfo=datetime.timezone.utc)
    return int(dt.timestamp())
 def _unix_to_date(ts: int) -> str:
    """Convert Unix timestamp to YYYY-MM-DD."""
    dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
    return dt.strftime("%Y-%m-%d")
 def _strip_html(text: str) -> str:
    """Strip HTML tags and decode entities from HN comment text."""
    import re
    text = html.unescape(text)
    text = re.sub(r'<p>', '\n', text)
    text = re.sub(r'<[^>]+>', '', text)
    return text.strip()
 def search_hackernews(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
 ) -> Dict[str, Any]:
    """Search Hacker News via Algolia API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
    Returns:
        Dict with Algolia response (contains 'hits' list).
    """
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    from_ts = _date_to_unix(from_date)
    to_ts = _date_to_unix(to_date) + 86400  # Include the end date
    # Use extracted core subject instead of raw topic for cleaner Algolia matching
    core = extract_core_subject(topic)
    # Hyphens and commas tokenize awkwardly in Algolia; flatten them so themed
    # queries like "ts-bun-node" or "claude, personal agents" become plain words.
    core_flat = _flatten_query_for_algolia(core)
    _log(f"Searching for '{core_flat}' (raw: '{topic}', since {from_date}, count={count})")
    # Use relevance-sorted search with minimum engagement filter.
    # NOTE: restrictSearchableAttributes=title omitted intentionally — it would
    # miss Ask HN/Show HN threads where the topic appears in the body.
    params = {
        "query": core_flat,
        "tags": "story",
        "numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
        "hitsPerPage": str(count),
    }
    # Algolia defaults to AND across query tokens, so a 4-5 word theme query
    # matches no stories. Mark all-but-the-first token as optional so Algolia
    # ranks by how many tokens match instead of requiring every one.
    tokens = core_flat.split()
    if len(tokens) > 1:
        params["optionalWords"] = " ".join(tokens[1:])
    from urllib.parse import urlencode
    url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"
    try:
        response = http.request("GET", url, timeout=30)
    except http.HTTPError as e:
        _log(f"Search failed: {e}")
        return {"hits": [], "error": str(e)}
    except Exception as e:
        _log(f"Search failed: {e}")
        return {"hits": [], "error": str(e)}
    hits = response.get("hits", [])
    _log(f"Found {len(hits)} stories")
    return response
 _WORD_BOUNDARY_RE_CACHE: Dict[str, "re.Pattern[str]"] = {}
 def _flatten_query_for_algolia(text: str) -> str:
    """Normalise query for Algolia + post-filter comparison.
    Multi-keyword theme queries frequently contain commas (delimiters) or
    hyphens (compound terms like ``ts-bun-node``); both tokenize awkwardly.
    Flatten them to spaces and collapse runs of whitespace so the search
    parameter and the post-filter operate on the same shape.
    """
    return " ".join(text.replace(",", " ").replace("-", " ").split())
 def _title_matches_query(title: str, query: str, author: str = "") -> bool:
    """Check if any query token appears as a whole word in the title.
    Returns True when the query is empty (no filter), or when at least one
    query token matches as a whole word in the title after stripping
    "Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes.
    We previously required *every* token to appear (all-words), which killed
    every Algolia hit on multi-keyword themes like "claude, personal agents,
    agentic infra" because real HN titles never contain all five tokens
    verbatim. Relaxing to any-word matches Algolia's `optionalWords` behaviour
    in `search_hackernews`. Token-overlap relevance scoring at parse time
    demotes hits where only one weak token matched, so the loosened gate
    won't surface noise to the top of the ranking.
    Word-boundary matching (rather than naive substring) prevents short
    tokens like ``ai`` or ``ts`` from matching unrelated words like
    ``email`` or ``artists``.
    """
    if not query:
        return True
    stripped = _HN_PREFIXES.sub("", title).strip()
    check_text = stripped.lower()
    # Normalise the query the same way search_hackernews does so post-filter
    # tokens line up with what Algolia actually saw.
    query_words = [w for w in _flatten_query_for_algolia(query.lower()).split() if w]
    if not query_words:
        return True
    for word in query_words:
        pattern = _WORD_BOUNDARY_RE_CACHE.get(word)
        if pattern is None:
            pattern = re.compile(rf"\b{re.escape(word)}\b")
            _WORD_BOUNDARY_RE_CACHE[word] = pattern
        if pattern.search(check_text):
            return True
    return False
 def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
    """Parse Algolia response into normalized item dicts.
    Args:
        response: Algolia search response
        query: Original search query for token-overlap relevance scoring
    Returns:
        List of item dicts ready for normalization.
    """
    hits = response.get("hits", [])
    # Post-filter: remove items where query only matched an HN prefix like "Tell HN:"
    if query:
        before = len(hits)
        hits = [
            h for h in hits
            if _title_matches_query(h.get("title", ""), query, h.get("author", ""))
        ]
        dropped = before - len(hits)
        if dropped:
            _log(f"Prefix filter removed {dropped}/{before} false-positive hits for '{query}'")
    items = []
    for i, hit in enumerate(hits):
        object_id = hit.get("objectID", "")
        points = hit.get("points") or 0
        num_comments = hit.get("num_comments") or 0
        created_at_i = hit.get("created_at_i")
        date_str = None
        if created_at_i:
            date_str = _unix_to_date(created_at_i)
        # Article URL vs HN discussion URL
        article_url = hit.get("url") or ""
        hn_url = f"https://news.ycombinator.com/item?id={object_id}"
        # Relevance: blend Algolia rank with token-overlap content matching
        rank_score = max(0.3, 1.0 - (i * 0.02))  # 1.0 -> 0.3 over 35 items
        engagement_boost = min(0.2, math.log1p(points) / 40)
        if query:
            content_score = token_overlap_relevance(query, hit.get("title", ""))
            relevance = min(1.0, 0.6 * rank_score + 0.4 * content_score + engagement_boost)
        else:
            relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
        items.append({
            "id": object_id,
            "title": hit.get("title", ""),
            "url": article_url,
            "hn_url": hn_url,
            "author": hit.get("author", ""),
            "date": date_str,
            "engagement": {
                "points": points,
                "comments": num_comments,
            },
            "relevance": round(relevance, 2),
            "why_relevant": f"HN story about {hit.get('title', 'topic')[:60]}",
        })
    return items
 def _fetch_item_comments(object_id: str, max_comments: int = 5) -> Dict[str, Any]:
    """Fetch top-level comments for a story from Algolia items endpoint.
    Args:
        object_id: HN story ID
        max_comments: Max comments to return
    Returns:
        Dict with 'comments' list and 'comment_insights' list.
    """
    url = f"{ALGOLIA_ITEM_URL}/{object_id}"
    try:
        data = http.request("GET", url, timeout=15)
    except Exception as e:
        _log(f"Failed to fetch comments for {object_id}: {e}")
        return {"comments": [], "comment_insights": []}
    children = data.get("children", [])
    # Sort by points (highest first), filter to actual comments
    real_comments = [
        c for c in children
        if c.get("text") and c.get("author")
    ]
    real_comments.sort(key=lambda c: c.get("points") or 0, reverse=True)
    comments = []
    insights = []
    for c in real_comments[:max_comments]:
        text = _strip_html(c.get("text", ""))
        excerpt = text[:300] + "..." if len(text) > 300 else text
        comments.append({
            "author": c.get("author", ""),
            "text": excerpt,
            "points": c.get("points") or 0,
        })
        # First sentence as insight
        first_sentence = text.split(". ")[0].split("\n")[0][:200]
        if first_sentence:
            insights.append(first_sentence)
    return {"comments": comments, "comment_insights": insights}
 def enrich_top_stories(
    items: List[Dict[str, Any]],
    depth: str = "default",
 ) -> List[Dict[str, Any]]:
    """Fetch comments for top N stories by points.
    Args:
        items: Parsed HN items
        depth: Research depth (controls how many to enrich)
    Returns:
        Items with top_comments and comment_insights added.
    """
    if not items:
        return items
    limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
    # Sort by points to enrich the most popular stories
    by_points = sorted(
        range(len(items)),
        key=lambda i: items[i].get("engagement", {}).get("points", 0),
        reverse=True,
    )
    to_enrich = by_points[:limit]
    _log(f"Enriching top {len(to_enrich)} stories with comments")
    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = {
            executor.submit(
                _fetch_item_comments,
                items[idx]["id"],
            ): idx
            for idx in to_enrich
        }
        for future in as_completed(futures):
            idx = futures[future]
            try:
                result = future.result(timeout=15)
                items[idx]["top_comments"] = result["comments"]
                items[idx]["comment_insights"] = result["comment_insights"]
            except (KeyError, TypeError, OSError) as exc:
                _log(f"Comment enrichment failed for story {items[idx].get('id', '?')}: {type(exc).__name__}: {exc}")
                items[idx]["top_comments"] = []
                items[idx]["comment_insights"] = []
    return items
@@ -0,0 +1,674 @@
 """HTML rendering for shareable last30days reports."""
 from __future__ import annotations
 import html
 import re
 from datetime import date
 from . import render, schema
 PROSE_LABELS = [
    ("What I learned:", "What I learned"),
    ("KEY PATTERNS from the research:", "Key patterns from the research"),
 ]
 INVITATION_PATTERN = re.compile(r"^---\nI'm now an expert.*?Just ask\.$", re.MULTILINE | re.DOTALL)
 EVIDENCE_BLOCK_PATTERN = re.compile(r"<!-- EVIDENCE FOR SYNTHESIS.*?<!-- END EVIDENCE FOR SYNTHESIS -->", re.DOTALL)
 PASS_THROUGH_FOOTER_PATTERN = re.compile(r"<!-- PASS-THROUGH FOOTER.*?-->\n(.*?)<!-- END PASS-THROUGH FOOTER -->", re.DOTALL)
 CANONICAL_BOUNDARY_PATTERN = re.compile(r"\n?---\n# END OF last30days CANONICAL OUTPUT.*$", re.DOTALL)
 # render_for_html emits metadata as <!-- META: ... --> so it survives the
 # markdown converter (which escapes raw HTML inside paragraphs). Promoted to
 # a styled <div class="meta"> after conversion.
 META_MARKER_PATTERN = re.compile(r"<!--\s*META:\s*(.*?)\s*-->")
 CSS = """
 :root {
  --bg: #0e0e10;
  --bg-elev: #18181b;
  --fg: #fafafa;
  --fg-muted: #a1a1aa;
  --fg-subtle: #71717a;
  --accent: #a855f7;
  --accent-soft: #c4b5fd;
  --border: #27272a;
  --code-bg: #1a1a1d;
  --max-w: 720px;
 }
@media (prefers-color-scheme: light) {
  :root {
    --bg: #ffffff;
    --bg-elev: #fafafa;
    --fg: #18181b;
    --fg-muted: #52525b;
    --fg-subtle: #71717a;
    --accent: #7c3aed;
    --accent-soft: #6d28d9;
    --border: #e4e4e7;
    --code-bg: #f4f4f5;
  }
 }
 * { box-sizing: border-box; }
 html, body {
  margin: 0;
  padding: 0;
  background: var(--bg);
  color: var(--fg);
  font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, system-ui, sans-serif;
  font-size: 17px;
  line-height: 1.65;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
  text-rendering: optimizeLegibility;
 }
 body {
  max-width: var(--max-w);
  margin: 0 auto;
  padding: 4rem 1.5rem 6rem;
 }
 .badge {
  display: inline-block;
  padding: 0.4rem 0.85rem;
  margin-bottom: 2.5rem;
  background: var(--bg-elev);
  border: 1px solid var(--border);
  border-radius: 999px;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13px;
  font-weight: 500;
  color: var(--fg-muted);
  letter-spacing: 0;
 }
 .badge .accent { color: var(--accent); }
 .meta {
  margin: -1.5rem 0 2.5rem;
  color: var(--fg-subtle);
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13px;
  letter-spacing: 0.01em;
 }
 h1 {
  margin: 0 0 1.5rem;
  color: var(--fg);
  font-size: 30px;
  font-weight: 700;
  line-height: 1.2;
  letter-spacing: 0;
 }
 h2,
 .prose-label {
  margin: 2.75rem 0 1.25rem;
  color: var(--fg);
  font-size: 20px;
  font-weight: 600;
  line-height: 1.35;
  letter-spacing: 0;
 }
 .badge + h2,
 .badge + .prose-label { margin-top: 0.5rem; }
 h3 {
  margin: 2rem 0 0.85rem;
  color: var(--fg);
  font-size: 17px;
  font-weight: 600;
  line-height: 1.4;
  letter-spacing: 0;
 }
 p {
  margin: 0 0 1.4rem;
  color: var(--fg-muted);
 }
 p strong,
 li strong,
 td strong {
  color: var(--fg);
  font-weight: 600;
 }
 a {
  color: var(--accent);
  text-decoration: none;
  border-bottom: 1px solid transparent;
  transition: border-color 0.15s ease;
 }
 a:hover { border-bottom-color: var(--accent); }
 ul,
 ol {
  margin: 0 0 1.6rem;
  padding-left: 1.5rem;
  color: var(--fg-muted);
 }
 li {
  margin: 0.6rem 0;
  padding-left: 0.4rem;
 }
 li::marker {
  color: var(--accent);
  font-weight: 600;
 }
 blockquote {
  margin: 1.5rem 0;
  padding-left: 1rem;
  border-left: 3px solid var(--accent);
  color: var(--fg-muted);
 }
 hr {
  margin: 2.5rem 0;
  border: 0;
  border-top: 1px solid var(--border);
 }
 code {
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 0.92em;
  background: var(--code-bg);
  padding: 0.15rem 0.4rem;
  border-radius: 4px;
  color: var(--accent-soft);
 }
 pre {
  margin: 1.4rem 0;
  background: var(--code-bg);
  border: 1px solid var(--border);
  border-radius: 8px;
  padding: 1rem 1.25rem;
  overflow-x: auto;
  font-size: 14px;
  line-height: 1.6;
 }
 pre code {
  background: none;
  padding: 0;
  color: var(--fg);
 }
 table {
  width: 100%;
  border-collapse: collapse;
  margin: 1.5rem 0;
  font-size: 15px;
 }
 th,
 td {
  text-align: left;
  padding: 0.75rem 1rem;
  border-bottom: 1px solid var(--border);
  vertical-align: top;
 }
 th {
  color: var(--fg-muted);
  font-weight: 600;
  font-size: 13px;
  letter-spacing: 0;
  text-transform: uppercase;
 }
 td { color: var(--fg-muted); }
 td:first-child { color: var(--fg); font-weight: 500; }
 .engine-footer {
  margin: 3rem 0 2.5rem;
  padding: 1.25rem 1.5rem;
  background: var(--bg-elev);
  border: 1px solid var(--border);
  border-radius: 8px;
  color: var(--fg-muted);
 }
 .engine-footer pre {
  margin: 0;
  padding: 0;
  background: transparent;
  border: 0;
  border-radius: 0;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  font-size: 13.5px;
  font-weight: 400;
  line-height: 1.75;
  color: inherit;
  white-space: pre-wrap;
  word-break: break-word;
 }
 .colophon {
  margin-top: 4rem;
  padding-top: 2rem;
  border-top: 1px solid var(--border);
  color: var(--fg-subtle);
  font-size: 13px;
  font-family: 'JetBrains Mono', ui-monospace, 'SF Mono', 'Cascadia Code', Menlo, Consolas, monospace;
  line-height: 1.7;
 }
 .colophon .rerun {
  display: inline-block;
  padding: 0.15rem 0.5rem;
  margin-left: 0.25rem;
  background: var(--code-bg);
  border-radius: 4px;
  color: var(--accent-soft);
  font-size: 0.95em;
 }
@media print {
  :root {
    --bg: #ffffff;
    --bg-elev: #f5f5f5;
    --fg: #000000;
    --fg-muted: #1f2937;
    --fg-subtle: #4b5563;
    --accent: #6d28d9;
    --accent-soft: #6d28d9;
    --border: #d4d4d8;
    --code-bg: #f4f4f5;
  }
  @page { size: A4; margin: 1.5cm 2cm; }
  body {
    max-width: none;
    padding: 0;
    font-size: 11pt;
  }
  a {
    color: inherit;
    border-bottom: 0;
    text-decoration: underline;
  }
  a[href]::after {
    content: " (" attr(href) ")";
    font-size: 0.85em;
    color: var(--fg-subtle);
  }
  .engine-footer { page-break-inside: avoid; }
 }
@media (max-width: 600px) {
  body {
    padding: 2.5rem 1.25rem 4rem;
    font-size: 16px;
  }
  h1 { font-size: 25px; }
  .badge { font-size: 12px; }
  th, td { padding: 0.65rem 0.5rem; }
 }
 """.strip()
 HTML_TEMPLATE = """<!DOCTYPE html>
 <html lang="en">
 <head>
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <title>last30days · __TITLE__</title>
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&amp;family=JetBrains+Mono:wght@400;500&amp;display=swap" rel="stylesheet">
 <style>
 __CSS__
 </style>
 </head>
 <body>
 __BODY__
 __COLOPHON__
 </body>
 </html>
 """
 def render_html(
    report: schema.Report,
    *,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    _ = fun_level
    md = render.render_for_html(report, synthesis_md=synthesis_md, save_path=save_path)
    md = _strip_evidence_block(md)
    md = _strip_invitation(md)
    md = _strip_canonical_boundary(md)
    md = _promote_prose_labels(md)
    body = _markdown_to_html(md)
    body = _wrap_engine_footer(body)
    body = _promote_meta_marker(body)
    colophon = _build_colophon(report)
    return _wrap_in_template(body, colophon, report.topic)
 def render_html_comparison(
    entity_reports: list[tuple[str, schema.Report]],
    *,
    fun_level: str = "medium",
    save_path: str | None = None,
    synthesis_md: str | None = None,
 ) -> str:
    _ = fun_level
    md = render.render_for_html_comparison(
        entity_reports, synthesis_md=synthesis_md, save_path=save_path,
    )
    md = _strip_evidence_block(md)
    md = _strip_invitation(md)
    md = _strip_canonical_boundary(md)
    md = _promote_prose_labels(md)
    body = _markdown_to_html(md)
    body = _wrap_engine_footer(body)
    body = _promote_meta_marker(body)
    topic = " vs ".join(label for label, _ in entity_reports)
    colophon = _build_colophon(entity_reports[0][1], topic=topic)
    return _wrap_in_template(body, colophon, topic)
 def _strip_evidence_block(md: str) -> str:
    return EVIDENCE_BLOCK_PATTERN.sub("", md)
 def _strip_invitation(md: str) -> str:
    return INVITATION_PATTERN.sub("", md)
 def _strip_canonical_boundary(md: str) -> str:
    return CANONICAL_BOUNDARY_PATTERN.sub("", md)
 def _promote_prose_labels(md: str) -> str:
    for source, normalized in PROSE_LABELS:
        md = re.sub(
            rf"^{re.escape(source)}$",
            f"## {normalized}",
            md,
            flags=re.MULTILINE,
        )
    return md
 def _markdown_to_html(md: str) -> str:
    md, footers = _protect_engine_footers(md)
    global _ENGINE_FOOTER_STORE
    _ENGINE_FOOTER_STORE = footers
    # Strip HTML comments EXCEPT preserved markers used for post-processing
    # (META is promoted to <div class="meta"> after markdown conversion).
    md = re.sub(r"<!--(?!\s*META:).*?-->", "", md, flags=re.DOTALL)
    lines = md.splitlines()
    out: list[str] = []
    paragraph: list[str] = []
    list_type: str | None = None
    in_code = False
    code_lines: list[str] = []
    index = 0
    def flush_paragraph() -> None:
        nonlocal paragraph
        if paragraph:
            text = " ".join(part.strip() for part in paragraph).strip()
            if text:
                out.append(f"<p>{_inline_markdown(text)}</p>")
            paragraph = []
    def close_list() -> None:
        nonlocal list_type
        if list_type:
            out.append(f"</{list_type}>")
            list_type = None
    while index < len(lines):
        line = lines[index]
        stripped = line.strip()
        if in_code:
            if stripped.startswith("```"):
                out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
                code_lines = []
                in_code = False
            else:
                code_lines.append(line)
            index += 1
            continue
        if stripped.startswith("```"):
            flush_paragraph()
            close_list()
            in_code = True
            code_lines = []
            index += 1
            continue
        if stripped in footers:
            flush_paragraph()
            close_list()
            out.append(stripped)
            index += 1
            continue
        if not stripped:
            flush_paragraph()
            close_list()
            index += 1
            continue
        if stripped == "---":
            flush_paragraph()
            close_list()
            out.append("<hr>")
            index += 1
            continue
        if index + 1 < len(lines) and _is_table_row(stripped) and _is_table_separator(lines[index + 1].strip()):
            flush_paragraph()
            close_list()
            table_lines = [stripped]
            index += 2
            while index < len(lines) and _is_table_row(lines[index].strip()):
                table_lines.append(lines[index].strip())
                index += 1
            out.append(_render_table(table_lines))
            continue
        heading = re.match(r"^(#{1,4})\s+(.+)$", stripped)
        if heading:
            flush_paragraph()
            close_list()
            level = min(len(heading.group(1)), 3)
            out.append(f"<h{level}>{_inline_markdown(heading.group(2))}</h{level}>")
            index += 1
            continue
        if stripped.startswith(">"):
            flush_paragraph()
            close_list()
            quote_lines = []
            while index < len(lines) and lines[index].strip().startswith(">"):
                quote_lines.append(lines[index].strip().lstrip(">").strip())
                index += 1
            out.append(f"<blockquote>{_inline_markdown(' '.join(quote_lines))}</blockquote>")
            continue
        unordered = re.match(r"^[-*]\s+(.+)$", stripped)
        ordered = re.match(r"^\d+[.)]\s+(.+)$", stripped)
        if unordered or ordered:
            flush_paragraph()
            next_type = "ul" if unordered else "ol"
            if list_type != next_type:
                close_list()
                out.append(f"<{next_type}>")
                list_type = next_type
            item = unordered.group(1) if unordered else ordered.group(1)
            out.append(f"<li>{_inline_markdown(item)}</li>")
            index += 1
            continue
        if stripped.startswith("🌐 last30days"):
            flush_paragraph()
            close_list()
            badge_text = _inline_markdown(stripped.removeprefix("🌐").strip())
            out.append(f'<div class="badge"><span class="accent">🌐</span> {badge_text}</div>')
            index += 1
            continue
        paragraph.append(line)
        index += 1
    if in_code:
        out.append(f"<pre><code>{html.escape(chr(10).join(code_lines))}</code></pre>")
    flush_paragraph()
    close_list()
    return "\n".join(out).strip()
 def _protect_engine_footers(md: str) -> tuple[str, dict[str, str]]:
    footers: dict[str, str] = {}
    def replace(match: re.Match[str]) -> str:
        token = f"__LAST30DAYS_ENGINE_FOOTER_{len(footers)}__"
        footers[token] = match.group(1).strip("\n")
        return f"\n{token}\n"
    return PASS_THROUGH_FOOTER_PATTERN.sub(replace, md), footers
 def _wrap_engine_footer(body: str) -> str:
    def replace(match: re.Match[str]) -> str:
        footer = html.escape(_ENGINE_FOOTER_STORE.get(match.group(0), ""), quote=False)
        return f'<div class="engine-footer"><pre>{footer}</pre></div>'
    return re.sub(
        r"__LAST30DAYS_ENGINE_FOOTER_\d+__",
        replace,
        body,
    )
 def _promote_meta_marker(body: str) -> str:
    """Promote ``<!-- META: ... -->`` markers into a styled ``<div class="meta">``.
    The marker is preserved through the comment-strip pass (see
    _markdown_to_html exemption) but the markdown converter wraps it in
    ``<p>`` and HTML-escapes the angle brackets. After conversion the body
    contains shapes like:
      <p>&lt;!-- META: TEXT --&gt;</p>
      <p><!-- META: TEXT --></p>     (when not escaped)
    Both collapse to ``<div class="meta">TEXT</div>``.
    """
    def replace(match: re.Match[str]) -> str:
        text = match.group(1).strip()
        return f'<div class="meta">{text}</div>'
    # Escaped form (most common after markdown conversion)
    body = re.sub(
        r"<p>\s*&lt;!--\s*META:\s*(.*?)\s*--&gt;\s*</p>",
        replace,
        body,
    )
    body = re.sub(r"&lt;!--\s*META:\s*(.*?)\s*--&gt;", replace, body)
    # Unescaped form (paranoid fallback)
    body = re.sub(r"<p>\s*<!--\s*META:\s*(.*?)\s*-->\s*</p>", replace, body)
    body = re.sub(r"<!--\s*META:\s*(.*?)\s*-->", replace, body)
    return body
 _ENGINE_FOOTER_STORE: dict[str, str] = {}
 def _inline_markdown(text: str) -> str:
    escaped = html.escape(text, quote=True)
    code_tokens: dict[str, str] = {}
    def code_replace(match: re.Match[str]) -> str:
        token = f"__CODE_{len(code_tokens)}__"
        code_tokens[token] = f"<code>{match.group(1)}</code>"
        return token
    escaped = re.sub(r"`([^`]+)`", code_replace, escaped)
    escaped = re.sub(r"\*\*([^*]+)\*\*", r"<strong>\1</strong>", escaped)
    escaped = re.sub(
        r"\[([^\]]+)\]\(([^)\s]+)\)",
        r'<a href="\2">\1</a>',
        escaped,
    )
    for token, value in code_tokens.items():
        escaped = escaped.replace(token, value)
    return escaped
 def _is_table_row(line: str) -> bool:
    return "|" in line and len(_split_table_cells(line)) >= 2
 def _is_table_separator(line: str) -> bool:
    cells = _split_table_cells(line)
    return bool(cells) and all(re.fullmatch(r":?-{3,}:?", cell.strip()) for cell in cells)
 def _split_table_cells(line: str) -> list[str]:
    return [cell.strip() for cell in line.strip().strip("|").split("|")]
 def _render_table(rows: list[str]) -> str:
    header = _split_table_cells(rows[0])
    body_rows = [_split_table_cells(row) for row in rows[1:]]
    out = ["<table>", "<thead>", "<tr>"]
    out.extend(f"<th>{_inline_markdown(cell)}</th>" for cell in header)
    out.extend(["</tr>", "</thead>", "<tbody>"])
    for row in body_rows:
        out.append("<tr>")
        out.extend(f"<td>{_inline_markdown(cell)}</td>" for cell in row)
        out.append("</tr>")
    out.extend(["</tbody>", "</table>"])
    return "\n".join(out)
 def _build_colophon(report: schema.Report, *, topic: str | None = None) -> str:
    display_topic = topic or report.topic
    generated = _generated_date(report)
    version = render._skill_version()
    escaped_topic = html.escape(display_topic)
    rerun = html.escape(f"/last30days {display_topic}")
    return (
        '<div class="colophon">\n'
        f"  Generated {generated} by /last30days v{html.escape(version)} · topic: {escaped_topic}<br>\n"
        f'  Re-run for fresh data: <span class="rerun">{rerun}</span>\n'
        "</div>"
    )
 def _generated_date(report: schema.Report) -> str:
    if report.generated_at:
        return report.generated_at[:10]
    return date.today().strftime("%Y-%m-%d")
 def _wrap_in_template(body: str, colophon: str, title: str) -> str:
    return (
        HTML_TEMPLATE
        .replace("__TITLE__", html.escape(title))
        .replace("__CSS__", CSS)
        .replace("__BODY__", body)
        .replace("__COLOPHON__", colophon)
    )
@@ -0,0 +1,308 @@
 """HTTP utilities for last30days skill (stdlib only)."""
 import json
 import re
 import socket
 import sys
 import time
 import urllib.error
 import urllib.request
 from typing import Any, Dict, Optional, Union
 from urllib.parse import urlencode
 from . import log as _log
 DEFAULT_TIMEOUT = 30
 def log(msg: str):
    """Log debug message to stderr."""
    _log.debug(msg)
 MAX_RETRIES = 5
 MAX_429_RETRIES = 2
 RETRY_DELAY = 2.0
 # DNS resolution failures (gaierror) are transient — typically resolved by a
 # brief backoff and retry. Use a dedicated minimum attempt count + exponential
 # delays (1s, 2s, 4s) so callers that pass a small `retries` value still get a
 # meaningful chance to recover from a transient resolution failure.
 MIN_DNS_RETRIES = 3
 USER_AGENT = "last30days-skill/3.0 (Assistant Skill)"
 def _is_dns_failure(err: urllib.error.URLError) -> bool:
    """Return True if a URLError was caused by DNS resolution (gaierror)."""
    return isinstance(getattr(err, "reason", None), socket.gaierror)
 class HTTPError(Exception):
    """HTTP request error with status code."""
    def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
        super().__init__(message)
        self.status_code = status_code
        self.body = body
 def request(
    method: str,
    url: str,
    headers: Optional[Dict[str, str]] = None,
    json_data: Optional[Dict[str, Any]] = None,
    params: Optional[Dict[str, Any]] = None,
    timeout: int = DEFAULT_TIMEOUT,
    retries: int = MAX_RETRIES,
    max_429_retries: int = MAX_429_RETRIES,
    raw: bool = False,
 ) -> Union[Dict[str, Any], str]:
    """Make an HTTP request and return JSON response.
    Args:
        method: HTTP method (GET, POST, etc.)
        url: Request URL
        headers: Optional headers dict
        json_data: Optional JSON body (for POST)
        params: Optional query-string params. Values are stringified. None values
            are dropped. If ``url`` already has a query string, ``params`` is appended.
        timeout: Request timeout in seconds
        retries: Number of retries on failure
        max_429_retries: Maximum 429 retries before giving up (separate cap)
        raw: If True, return raw response text instead of parsed JSON
    Returns:
        Parsed JSON response as dict, or raw text string if raw=True.
    Raises:
        HTTPError: On request failure
    """
    headers = headers or {}
    headers.setdefault("User-Agent", USER_AGENT)
    if params:
        filtered = {k: str(v) for k, v in params.items() if v is not None}
        if filtered:
            separator = "&" if ("?" in url) else "?"
            url = f"{url}{separator}{urlencode(filtered)}"
    data = None
    if json_data is not None:
        data = json.dumps(json_data).encode('utf-8')
        headers.setdefault("Content-Type", "application/json")
    req = urllib.request.Request(url, data=data, headers=headers, method=method)
    safe_url = re.sub(r'([?&])(key|api_key|token|secret)=[^&]*', r'\1\2=***', url)
    log(f"{method} {safe_url}")
    last_error = None
    rate_limit_count = 0
    # DNS failures get a dedicated minimum attempt count + exponential backoff.
    # `effective_retries` is the actual loop bound; we expand it on the first
    # gaierror if the caller passed a smaller `retries` value than MIN_DNS_RETRIES.
    effective_retries = retries
    dns_attempts = 0
    attempt = 0
    while attempt < effective_retries:
        try:
            with urllib.request.urlopen(req, timeout=timeout) as response:
                body = response.read().decode('utf-8')
                log(f"Response: {response.status} ({len(body)} bytes)")
                if raw:
                    return body
                return json.loads(body) if body else {}
        except urllib.error.HTTPError as e:
            body = None
            try:
                body = e.read().decode('utf-8')
            except (OSError, UnicodeDecodeError):
                pass
            log(f"HTTP Error {e.code}: {e.reason}")
            if body:
                snippet = " ".join(body.split())
                log(f"Error body: {snippet[:200]}")
            last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)
            # Don't retry client errors (4xx) except rate limits
            if 400 <= e.code < 500 and e.code != 429:
                raise last_error
            # Cap 429 retries separately to avoid wasting latency
            if e.code == 429:
                rate_limit_count += 1
                if rate_limit_count >= max_429_retries:
                    raise last_error
            # HTTP errors respect the caller's original `retries`; only DNS
            # failures get the widened `effective_retries` budget.
            if attempt < retries - 1:
                if e.code == 429:
                    # Respect Retry-After header, fall back to exponential backoff
                    retry_after = e.headers.get("Retry-After") if hasattr(e, 'headers') else None
                    if retry_after:
                        try:
                            delay = float(retry_after)
                        except ValueError:
                            delay = RETRY_DELAY * (2 ** attempt) + 1
                    else:
                        delay = RETRY_DELAY * (2 ** attempt) + 1  # 3s, 5s, 9s...
                    log(f"Rate limited (429). Waiting {delay:.1f}s before retry {attempt + 2}/{retries}")
                else:
                    delay = RETRY_DELAY * (2 ** attempt)
                time.sleep(delay)
            else:
                # Caller's original retry budget exhausted; an earlier DNS
                # failure may have widened `effective_retries`, but that
                # widening is DNS-only — don't grant extra HTTP attempts.
                break
        except urllib.error.URLError as e:
            log(f"URL Error: {e.reason}")
            last_error = HTTPError(f"URL Error: {e.reason}")
            if _is_dns_failure(e):
                # DNS resolution failures are transient; expand the retry budget
                # to MIN_DNS_RETRIES if the caller passed fewer, and use
                # exponential backoff (1s, 2s, 4s, ...) instead of the linear
                # default. Counts DNS attempts separately so other URLError
                # causes don't bypass the regular retry budget.
                dns_attempts += 1
                if effective_retries < MIN_DNS_RETRIES:
                    log(
                        f"DNS resolution failed; expanding retry budget from "
                        f"{effective_retries} to {MIN_DNS_RETRIES}"
                    )
                    effective_retries = MIN_DNS_RETRIES
                if attempt < effective_retries - 1:
                    delay = 2 ** (dns_attempts - 1)  # 1s, 2s, 4s, 8s, ...
                    log(
                        f"DNS resolution failure (attempt {dns_attempts}); "
                        f"retrying in {delay:.1f}s"
                    )
                    time.sleep(delay)
            elif attempt < retries - 1:
                # Non-DNS URLError (e.g. ConnectionRefused) respects the
                # caller's original retry budget, not the DNS-widened bound.
                time.sleep(RETRY_DELAY * (attempt + 1))
            else:
                # Caller's original retry budget exhausted; an earlier DNS
                # failure widening `effective_retries` does not carry over
                # to non-DNS error paths.
                break
        except json.JSONDecodeError as e:
            log(f"JSON decode error: {e}")
            last_error = HTTPError(f"Invalid JSON response: {e}")
            raise last_error
        except (OSError, TimeoutError, ConnectionResetError) as e:
            # Handle socket-level errors (connection reset, timeout, etc.)
            log(f"Connection error: {type(e).__name__}: {e}")
            last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
            if attempt < retries - 1:
                # Socket errors respect the caller's original retry budget.
                time.sleep(RETRY_DELAY * (attempt + 1))
            else:
                # Original budget exhausted; DNS widening doesn't apply here.
                break
        attempt += 1
    if last_error:
        raise last_error
    raise HTTPError("Request failed with no error details")
 def get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a GET request."""
    return request("GET", url, headers=headers, **kwargs)
 def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a POST request with JSON body."""
    return request("POST", url, headers=headers, json_data=json_data, **kwargs)
 def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> str:
    """Make a POST request with JSON body and return raw text."""
    return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)
 BROWSER_USER_AGENT = (
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/124.0.0.0 Safari/537.36"
 )
 def get_text(
    url: str,
    timeout: int = DEFAULT_TIMEOUT,
    retries: int = 2,
    accept: str = "*/*",
    headers: Optional[Dict[str, str]] = None,
 ) -> Optional[str]:
    """Fetch a URL and return decoded text, or None on any failure.
    Keyless helper for Reddit RSS and shreddit HTML endpoints — the free path
    that replaced the now-403 ``.json`` endpoints. Sends a browser User-Agent
    and never raises: returns None on HTTP error, network failure, or timeout
    so tiered callers can fall through to the next source.
    Args:
        url: Request URL
        timeout: HTTP timeout per attempt in seconds
        retries: Number of retries on failure (kept low — these tiers fail fast)
        accept: Accept header value (e.g. "application/atom+xml", "text/html")
        headers: Optional extra headers merged over the defaults
    Returns:
        Decoded response body as text, or None on failure.
    """
    merged = {
        "User-Agent": BROWSER_USER_AGENT,
        "Accept": accept,
        "Accept-Language": "en-US,en;q=0.9",
    }
    if headers:
        merged.update(headers)
    try:
        return request(
            "GET", url, headers=merged, timeout=timeout, retries=retries, raw=True
        )
    except HTTPError as e:
        log(f"get_text failed ({e}): {url}")
        return None
 def scrapecreators_headers(token: str) -> Dict[str, str]:
    """Build ScrapeCreators request headers (x-api-key + JSON content type)."""
    return {
        "x-api-key": token,
        "Content-Type": "application/json",
    }
 def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
    """Fetch Reddit thread JSON.
    Args:
        path: Reddit path (e.g., /r/subreddit/comments/id/title)
        timeout: HTTP timeout per attempt in seconds
        retries: Number of retries on failure
    Returns:
        Parsed JSON response
    """
    # Ensure path starts with /
    if not path.startswith('/'):
        path = '/' + path
    # Remove trailing slash and add .json
    path = path.rstrip('/')
    if not path.endswith('.json'):
        path = path + '.json'
    url = f"https://www.reddit.com{path}?raw_json=1"
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/json",
    }
    return get(url, headers=headers, timeout=timeout, retries=retries)
@@ -0,0 +1,543 @@
 """Instagram Reels search via ScrapeCreators API for /last30days.
 Uses ScrapeCreators REST API to search Instagram Reels by keyword, extract
 engagement metrics (views, likes, comments), and fetch video transcripts.
 Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
 API docs: https://scrapecreators.com/docs
 """
 import os
 import re
 import sys
 from datetime import datetime
 from typing import Any, Dict, List, Optional, Set
 from . import dates, http, log
 from .relevance import token_overlap_relevance as _compute_relevance
 SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
 # Depth configurations: how many results to fetch / captions to extract
 DEPTH_CONFIG = {
    "quick":   {"results_per_page": 10, "max_captions": 3},
    "default": {"results_per_page": 20, "max_captions": 5},
    "deep":    {"results_per_page": 40, "max_captions": 8},
 }
 # Max words to keep from each caption
 CAPTION_MAX_WORDS = 500
 # Default transcript fetch timeout (seconds). SC's
 # /v2/instagram/media/transcript regularly takes >15s on real workloads,
 # so the default is generous; override via LAST30DAYS_TRANSCRIPT_TIMEOUT.
 DEFAULT_TRANSCRIPT_TIMEOUT = 30
 def _resolve_transcript_timeout(
    timeout: Optional[float] = None,
    config: Optional[Dict[str, Any]] = None,
 ) -> float:
    """Resolve the IG transcript-fetch timeout.
    Priority (highest wins):
      1. Explicit ``timeout`` kwarg
      2. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in os.environ
      3. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in caller-supplied config dict
      4. ``DEFAULT_TRANSCRIPT_TIMEOUT`` (30s)
    Mirrors the ``os.environ.get(X) or config.get(X)`` pattern used for
    LAST30DAYS_STORE in last30days.py so the env var works whether it's
    shell-exported or set in ~/.config/last30days/.env.
    """
    if timeout is not None:
        try:
            return float(timeout)
        except (TypeError, ValueError):
            pass
    raw = os.environ.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
    if not raw and config:
        raw = config.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
    if raw:
        try:
            return float(raw)
        except (TypeError, ValueError):
            pass
    return float(DEFAULT_TRANSCRIPT_TIMEOUT)
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for Instagram search."""
    from .query import extract_core_subject
    _INSTAGRAM_NOISE = frozenset({
        'best', 'top', 'good', 'great', 'awesome', 'killer',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features',
        'recommendations', 'advice',
        'prompt', 'prompts', 'prompting',
        'methods', 'strategies', 'approaches',
    })
    return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
 def _to_hashtag_form(query: str) -> str:
    """Collapse a multi-word query to hashtag form (no spaces, lowercase).
    SC's /v2/instagram/reels/search wraps Google Search and is documented
    to be flaky on multi-token queries. Single-token queries map to a
    hashtag page lookup which is the stable path. Used as a 500-retry
    fallback before the request bubbles up as a silent failure.
    """
    return ''.join(query.split()).lower()
 def _infer_query_intent(topic: str) -> str:
    """Tiny local intent classifier for Instagram query expansion."""
    text = topic.lower().strip()
    if re.search(r"\b(vs|versus|compare|difference between)\b", text):
        return "comparison"
    if re.search(r"\b(how to|tutorial|guide|setup|step by step|deploy|install)\b", text):
        return "how_to"
    if re.search(r"\b(thoughts on|worth it|should i|opinion|review)\b", text):
        return "opinion"
    if re.search(r"\b(pricing|feature|features|best .* for)\b", text):
        return "product"
    return "breaking_news"
 def expand_instagram_queries(topic: str, depth: str) -> List[str]:
    """Generate multiple Instagram search queries from a topic.
    Mirrors reddit.py's expand_reddit_queries() pattern:
    1. Extract core subject (strip noise words)
    2. Include original topic if different from core
    3. Add intent-specific OR-joined content-type variants
    4. Cap by depth: 1 for quick, 2 for default, 3 for deep
    Returns 1-3 query strings depending on depth.
    """
    core = _extract_core_subject(topic)
    queries = [core]
    # Include cleaned original topic as variant if different from core
    original_clean = topic.strip().rstrip('?!.')
    if core.lower() != original_clean.lower() and len(original_clean.split()) <= 8:
        queries.append(original_clean)
    qtype = _infer_query_intent(topic)
    # Intent-specific Instagram content-type variants
    if qtype == "breaking_news":
        queries.append(f"{core} reaction OR edit")
    elif qtype == "opinion":
        queries.append(f"{core} reaction OR edit")
    elif qtype == "product":
        queries.append(f"{core} review OR haul")
    elif qtype == "comparison":
        queries.append(f"{core} vs OR compared")
    elif qtype == "how_to":
        queries.append(f"{core} tutorial OR hack")
    else:
        queries.append(f"{core} reaction OR edit")
    # Deep depth: add viral content variant
    if depth == "deep":
        queries.append(f"{core} viral OR trending OR reel")
    # Cap by depth budget
    caps = {"quick": 1, "default": 2, "deep": 3}
    cap = caps.get(depth, 2)
    return queries[:cap]
 def _log(msg: str):
    log.source_log("Instagram", msg)
 def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    """Parse date from ScrapeCreators Instagram item to YYYY-MM-DD.
    Handles taken_at as ISO string (e.g. "2026-02-26T16:00:00.000Z")
    or unix timestamp.
    """
    ts = item.get("taken_at")
    if not ts:
        return None
    # Try ISO string first (ScrapeCreators reels/search returns this)
    if isinstance(ts, str):
        try:
            # Handle "2026-02-26T16:00:00.000Z" format
            dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
            return dt.strftime("%Y-%m-%d")
        except (ValueError, TypeError):
            pass
        # Try just the date portion
        if len(ts) >= 10:
            return ts[:10]
    # Fall back to unix timestamp
    try:
        return dates.timestamp_to_date(int(ts))
    except (ValueError, TypeError):
        pass
    return None
 def _extract_hashtags(caption_text: str) -> List[str]:
    """Extract hashtags from Instagram caption text."""
    if not caption_text:
        return []
    return re.findall(r'#(\w+)', caption_text)
 def _parse_items(raw_items: List[Dict[str, Any]], core_topic: str) -> List[Dict[str, Any]]:
    """Parse raw Instagram items into normalized dicts."""
    items = []
    for raw in raw_items:
        if not isinstance(raw, dict):
            continue
        # Extract reel ID and shortcode
        reel_pk = str(raw.get("id", raw.get("pk", "")))
        shortcode = raw.get("shortcode", raw.get("code", ""))
        # Caption text -- can be a string or dict depending on endpoint
        caption_obj = raw.get("caption", "")
        if isinstance(caption_obj, dict):
            text = caption_obj.get("text", "")
        elif isinstance(caption_obj, str):
            text = caption_obj
        else:
            text = raw.get("desc", raw.get("text", ""))
        # Engagement metrics
        play_count = raw.get("video_play_count") or raw.get("video_view_count") or raw.get("play_count") or 0
        like_count = raw.get("like_count") or 0
        comment_count = raw.get("comment_count") or 0
        # Author info -- 'owner' in reels/search, 'user' in user/reels
        owner_raw = raw.get("owner") or raw.get("user")
        if isinstance(owner_raw, dict):
            author_name = owner_raw.get("username", "")
        elif isinstance(owner_raw, str):
            author_name = owner_raw
        else:
            author_name = ""
        # Duration
        duration = raw.get("video_duration")
        # Date
        date_str = _parse_date(raw)
        # Hashtags from caption text
        hashtags = _extract_hashtags(text)
        # Compute relevance with hashtag boost
        relevance = _compute_relevance(core_topic, text, hashtags)
        # Build URL -- prefer API-provided url, fallback to shortcode
        url = raw.get("url", "")
        if not url and shortcode:
            url = f"https://www.instagram.com/reel/{shortcode}"
        items.append({
            "video_id": reel_pk,
            "text": text,
            "url": url,
            "author_name": author_name,
            "date": date_str,
            "engagement": {
                "views": play_count,
                "likes": like_count,
                "comments": comment_count,
            },
            "hashtags": hashtags,
            "duration": duration,
            "relevance": relevance,
            "why_relevant": f"Instagram: {text[:60]}" if text else f"Instagram: {core_topic}",
            "caption_snippet": "",  # populated by fetch_captions
        })
    return items
 def _user_reels(
    handle: str,
    token: str,
 ) -> List[Dict[str, Any]]:
    """Fetch an Instagram user's recent reels via ScrapeCreators.
    Args:
        handle: Instagram username (without @)
        token: ScrapeCreators API key
    Returns:
        List of raw Instagram reel dicts.
    """
    _log(f"User reels: @{handle}")
    reels_url = f"{SCRAPECREATORS_BASE}/v1/instagram/user/reels"
    try:
        data = http.get(
            reels_url,
            params={"handle": handle},
            headers=http.scrapecreators_headers(token),
            timeout=30,
            retries=2,
        )
    except Exception as e:
        _log(f"User reels error for @{handle}: {e}")
        return []
    raw_items = data.get("items") or data.get("reels") or data.get("data") or []
    _log(f"  -> {len(raw_items)} reels from @{handle}")
    return raw_items
 def search_instagram(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Search Instagram Reels via ScrapeCreators API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
    _log(f"Searching Instagram for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
    try:
        data = http.get(
            f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
            params={"query": core_topic},
            headers=http.scrapecreators_headers(token),
            timeout=30,
            retries=2,
        )
    except http.HTTPError as e:
        # SC's v2 reels search wraps Google Search and 500s frequently on
        # multi-token queries. Single tokens hit the stable hashtag-page
        # path. Retry once with hashtag form before bubbling up.
        if getattr(e, "status_code", None) == 500 and ' ' in core_topic:
            _log(f"IG search 500 on '{core_topic}', retrying with hashtag form")
            try:
                data = http.get(
                    f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
                    params={"query": _to_hashtag_form(core_topic)},
                    headers=http.scrapecreators_headers(token),
                    timeout=30,
                    retries=2,
                )
            except Exception as retry_e:
                _log(f"IG search retry failed: {retry_e}")
                return {"items": [], "error": f"{type(retry_e).__name__}: {retry_e}"}
        else:
            _log(f"ScrapeCreators error: {e}")
            return {"items": [], "error": f"{type(e).__name__}: {e}"}
    except Exception as e:
        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}
    # Items are in the 'reels' array (ScrapeCreators v2 response)
    raw_items = data.get("reels") or data.get("items") or data.get("data") or []
    # Limit to configured count
    raw_items = raw_items[:config["results_per_page"]]
    # Parse items
    items = _parse_items(raw_items, core_topic)
    # Hard date filter
    in_range = [i for i in items if i["date"] and from_date <= i["date"] <= to_date]
    out_of_range = len(items) - len(in_range)
    if in_range:
        items = in_range
        if out_of_range:
            _log(f"Filtered {out_of_range} reels outside date range")
    else:
        _log(f"No reels within date range, keeping all {len(items)}")
    # Sort by views descending
    items.sort(key=lambda x: x["engagement"]["views"], reverse=True)
    _log(f"Found {len(items)} Instagram reels")
    return {"items": items}
 def fetch_captions(
    video_items: List[Dict[str, Any]],
    token: str,
    depth: str = "default",
    timeout: Optional[float] = None,
    config: Optional[Dict[str, Any]] = None,
 ) -> Dict[str, str]:
    """Fetch transcripts for top N Instagram reels via ScrapeCreators.
    Strategy:
    1. Use the 'text' field (caption) as baseline
    2. For top N, call /v2/instagram/media/transcript for spoken-word captions
    Args:
        video_items: Items from search_instagram()
        token: ScrapeCreators API key
        depth: Depth level for caption limit
        timeout: Optional per-request transcript timeout in seconds. When
            None, resolves from LAST30DAYS_TRANSCRIPT_TIMEOUT (env or
            config), defaulting to DEFAULT_TRANSCRIPT_TIMEOUT (30s).
        config: Optional config dict (from env.get_config()) used as a
            fallback source for LAST30DAYS_TRANSCRIPT_TIMEOUT when the
            value is not exported in os.environ.
    Returns:
        Dict mapping video_id -> caption text (truncated to 500 words)
    """
    depth_cfg = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    max_captions = depth_cfg["max_captions"]
    transcript_timeout = _resolve_transcript_timeout(timeout, config)
    if not video_items or not token:
        return {}
    top_items = video_items[:max_captions]
    _log(f"Enriching captions for {len(top_items)} reels")
    captions = {}
    # First pass: use text field as caption (always available, free)
    for item in top_items:
        vid = item["video_id"]
        text = item.get("text", "")
        if text:
            words = text.split()
            if len(words) > CAPTION_MAX_WORDS:
                text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
            captions[vid] = text
    # Second pass: try to get spoken-word transcripts (1 credit each)
    for item in top_items:
        vid = item["video_id"]
        url = item.get("url", "")
        if not url:
            continue
        try:
            data = http.get(
                f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
                params={"url": url},
                headers=http.scrapecreators_headers(token),
                timeout=transcript_timeout,
                retries=1,
            )
            transcripts = data.get("transcripts") or []
            if transcripts and isinstance(transcripts, list):
                transcript_text = " ".join(
                    t.get("text", "") for t in transcripts
                    if isinstance(t, dict) and t.get("text")
                )
                if transcript_text:
                    words = transcript_text.split()
                    if len(words) > CAPTION_MAX_WORDS:
                        transcript_text = ' '.join(words[:CAPTION_MAX_WORDS]) + '...'
                    captions[vid] = transcript_text
        except Exception as e:
            _log(f"Transcript fetch failed for {vid}: {e}")
    got = sum(1 for v in captions.values() if v)
    _log(f"Got captions for {got}/{len(top_items)} reels")
    return captions
 def search_and_enrich(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
    ig_creators: List[str] | None = None,
 ) -> Dict[str, Any]:
    """Full Instagram search: find reels, then fetch captions for top results.
    Uses expand_instagram_queries() to generate multiple search queries,
    runs ScrapeCreators for each, and merges/deduplicates results by video ID.
    Args:
        topic: Search topic (raw topic, not planner's narrowed query)
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
        ig_creators: Optional list of Instagram creator handles to fetch reels from
    Returns:
        Dict with 'items' list. Each item has a 'caption_snippet' field.
    """
    core_topic = _extract_core_subject(topic)
    seen_ids: Set[str] = set()
    items: List[Dict[str, Any]] = []
    last_error = None
    # Step 0: Creator reels (high-signal, runs first)
    if ig_creators and token:
        for creator in ig_creators:
            raw_items = _user_reels(creator, token)
            parsed = _parse_items(raw_items, core_topic)
            for item in parsed:
                vid = item.get("video_id", "")
                if vid and vid not in seen_ids:
                    seen_ids.add(vid)
                    items.append(item)
    # Step 1: Multi-query keyword search — run ScrapeCreators for each expanded query
    queries = expand_instagram_queries(topic, depth)
    for q in queries:
        search_result = search_instagram(q, from_date, to_date, depth, token)
        if search_result.get("error"):
            last_error = search_result["error"]
        for item in search_result.get("items", []):
            vid = item.get("video_id", "")
            if vid and vid not in seen_ids:
                seen_ids.add(vid)
                items.append(item)
    # Sort merged results by views descending
    items.sort(key=lambda x: x.get("engagement", {}).get("views", 0), reverse=True)
    if not items:
        return {"items": [], "error": last_error}
    # Step 2: Fetch captions for top N
    captions = fetch_captions(items, token, depth)
    # Step 3: Attach captions to items
    for item in items:
        vid = item["video_id"]
        caption = captions.get(vid)
        if caption:
            item["caption_snippet"] = caption
    return {"items": items, "error": last_error}
 def parse_instagram_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse Instagram search response to normalized format.
    Returns:
        List of item dicts ready for normalization.
    """
    return response.get("items", [])
@@ -0,0 +1,28 @@
 """Shared logging utilities for last30days skill."""
 import os
 import sys
 DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")
 def debug(msg: str) -> None:
    """Log debug message to stderr (only when LAST30DAYS_DEBUG is set)."""
    if DEBUG:
        sys.stderr.write(f"[DEBUG] {msg}\n")
        sys.stderr.flush()
 def source_log(prefix: str, msg: str, *, tty_only: bool = True) -> None:
    """Log a source module message to stderr.
    Args:
        prefix: Source label (e.g. "Reddit", "Bird").
        msg: Message text.
        tty_only: If True, only log when stderr is a TTY (avoids cluttering
                  non-interactive output like Claude Code).
    """
    if tty_only and not sys.stderr.isatty():
        return
    sys.stderr.write(f"[{prefix}] {msg}\n")
    sys.stderr.flush()
@@ -0,0 +1,552 @@
 """Normalization of source-specific payloads into the v3 generic item model."""
 from __future__ import annotations
 from typing import Any
 from urllib.parse import urlparse
 from . import dates, schema
 def filter_by_date_range(
    items: list[schema.SourceItem],
    from_date: str,
    to_date: str,
    require_date: bool = False,
 ) -> list[schema.SourceItem]:
    """Keep only items within the requested window."""
    filtered: list[schema.SourceItem] = []
    for item in items:
        if not item.published_at:
            if not require_date:
                filtered.append(item)
            continue
        if item.published_at < from_date or item.published_at > to_date:
            continue
        filtered.append(item)
    return filtered
 def normalize_source_items(
    source: str,
    items: list[dict[str, Any]],
    from_date: str,
    to_date: str,
    freshness_mode: str = "balanced_recent",
 ) -> list[schema.SourceItem]:
    """Normalize raw source items, filter by date range, with evergreen fallback for how_to queries."""
    source = source.lower()
    normalizers = {
        "reddit": _normalize_reddit,
        "x": _normalize_x,
        "youtube": _normalize_youtube,
        "tiktok": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "TK", "TikTok post"),
        "instagram": lambda s, i, idx, fd, td: _normalize_shortform_video(s, i, idx, fd, td, "IG", "Instagram reel"),
        "hackernews": _normalize_hackernews,
        "bluesky": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "BS", "Bluesky post"),
        "truthsocial": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TS", "Truth Social post"),
        "threads": lambda s, i, idx, fd, td: _normalize_microblog(s, i, idx, fd, td, "TH", "Threads post"),
        "xquik": _normalize_x,
        "pinterest": _normalize_pinterest,
        "polymarket": _normalize_polymarket,
        "digg": _normalize_digg,
        "grounding": _normalize_grounding,
        "xiaohongshu": _normalize_grounding,
        "github": _normalize_github,
        "perplexity": _normalize_grounding,
    }
    normalizer = normalizers.get(source)
    if normalizer is None:
        raise ValueError(f"Unsupported source: {source}")
    normalized = [normalizer(source, item, index, from_date, to_date) for index, item in enumerate(items)]
    require_date = source == "grounding"
    filtered = filter_by_date_range(normalized, from_date, to_date, require_date=require_date)
    if filtered:
        return filtered
    if freshness_mode == "evergreen_ok" and source == "youtube":
        if require_date:
            return [item for item in normalized if item.published_at]
        return normalized
    return filtered
 def _remap_comments(
    raw: list[Any],
    score_keys: tuple[str, ...],
    excerpt_keys: tuple[str, ...],
 ) -> list[dict[str, Any]]:
    """Normalize comments from any source into the shared Reddit-compatible shape.
    Downstream code (signals._top_comment_score, render._top_comments_list,
    entity_extract, rerank) all expect `score` and `excerpt`. This helper maps
    per-source field names (YT: likes/text, TikTok: digg_count/text) onto that
    shape while preserving author/date/url passthrough.
    """
    out: list[dict[str, Any]] = []
    for raw_c in raw:
        if not isinstance(raw_c, dict):
            continue
        score = _first_present(raw_c, score_keys, default=0)
        excerpt = _first_present(raw_c, excerpt_keys, default="")
        try:
            score_int = int(score or 0)
        except (TypeError, ValueError):
            score_int = 0
        entry: dict[str, Any] = {
            "score": score_int,
            "excerpt": str(excerpt or "")[:400],
            "author": str(raw_c.get("author") or ""),
            "date": str(raw_c.get("date") or ""),
        }
        if raw_c.get("url"):
            entry["url"] = str(raw_c["url"])
        out.append(entry)
    return out
 def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> Any:
    for key in keys:
        if key in d and d[key] not in (None, ""):
            return d[key]
    return default
 def _join_comment_excerpts(
    top_comments: list[Any],
    key: str,
    limit: int = 3,
 ) -> str:
    """Space-join the `key` field from the first `limit` dict-shaped comments."""
    return " ".join(
        str(comment.get(key) or "").strip()
        for comment in top_comments[:limit]
        if isinstance(comment, dict)
    )
 def _domain_from_url(url: str) -> str | None:
    if not url:
        return None
    domain = urlparse(url).netloc.strip().lower()
    return domain or None
 def _date_confidence(item: dict[str, Any], from_date: str, to_date: str, default: str = "low") -> str:
    if item.get("date_confidence"):
        return str(item["date_confidence"])
    date_value = item.get("date")
    if not date_value:
        return default
    return dates.get_date_confidence(str(date_value), from_date, to_date)
 def _source_item(
    *,
    item_id: str,
    source: str,
    title: str,
    body: str,
    url: str,
    published_at: str | None,
    date_confidence: str,
    relevance_hint: float,
    why_relevant: str,
    author: str | None = None,
    container: str | None = None,
    engagement: dict[str, float | int] | None = None,
    snippet: str = "",
    metadata: dict[str, Any] | None = None,
 ) -> schema.SourceItem:
    return schema.SourceItem(
        item_id=item_id,
        source=source,
        title=title.strip() or body.strip()[:160] or item_id,
        body=body.strip(),
        url=url.strip(),
        author=(author or "").strip() or None,
        container=(container or "").strip() or None,
        published_at=published_at,
        date_confidence=date_confidence,
        engagement=engagement or {},
        relevance_hint=max(0.0, min(1.0, float(relevance_hint or 0.0))),
        why_relevant=why_relevant.strip(),
        snippet=snippet.strip(),
        metadata=metadata or {},
    )
 def _normalize_reddit(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    top_comments = item.get("top_comments") or []
    comment_text = _join_comment_excerpts(top_comments, "excerpt")
    body = "\n".join(
        part
        for part in [
            str(item.get("title") or "").strip(),
            str(item.get("selftext") or "").strip(),
            comment_text,
        ]
        if part
    )
    return _source_item(
        item_id=str(item.get("id") or f"R{index + 1}"),
        source=source,
        title=str(item.get("title") or ""),
        body=body,
        url=str(item.get("url") or ""),
        author=None,
        container=str(item.get("subreddit") or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=comment_text or str(item.get("selftext") or "")[:400],
        metadata={
            "top_comments": top_comments,
            "comment_insights": item.get("comment_insights") or [],
        },
    )
 def _normalize_x(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    text = str(item.get("text") or "").strip()
    return _source_item(
        item_id=str(item.get("id") or f"X{index + 1}"),
        source=source,
        title=text[:140] or f"X post {index + 1}",
        body=text,
        url=str(item.get("url") or ""),
        author=str(item.get("author_handle") or "").lstrip("@"),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
    )
 def _normalize_youtube(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    transcript = str(item.get("transcript_snippet") or "").strip()
    description = str(item.get("description") or "").strip()
    title = str(item.get("title") or "").strip()
    highlights = item.get("transcript_highlights") or []
    metadata: dict[str, Any] = {}
    if highlights:
        metadata["transcript_highlights"] = highlights
    if item.get("captions_disabled"):
        # Surfaced for quality_nudge: uploader disabled captions, so this
        # video should be subtracted from the degraded-transcript-ratio
        # denominator (it was never going to produce a transcript).
        metadata["captions_disabled"] = True
    metadata["top_comments"] = _remap_comments(
        item.get("top_comments") or [],
        score_keys=("score", "likes"),
        excerpt_keys=("excerpt", "text"),
    )
    return _source_item(
        item_id=str(item.get("video_id") or item.get("id") or f"YT{index + 1}"),
        source=source,
        title=title,
        body="\n".join(part for part in [title, description, transcript] if part),
        url=str(item.get("url") or ""),
        author=str(item.get("channel_name") or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=transcript,
        metadata=metadata,
    )
 def _normalize_shortform_video(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
    id_prefix: str,
    default_title: str,
 ) -> schema.SourceItem:
    """Shared normalizer for TikTok and Instagram (identical structure)."""
    caption = str(item.get("caption_snippet") or "").strip()
    text = str(item.get("text") or "").strip()
    return _source_item(
        item_id=str(item.get("id") or f"{id_prefix}{index + 1}"),
        source=source,
        title=text[:140] or caption[:140] or f"{default_title} {index + 1}",
        body="\n".join(part for part in [text, caption] if part),
        url=str(item.get("url") or ""),
        author=str(item.get("author_name") or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=caption,
        metadata={
            "hashtags": item.get("hashtags") or [],
            "top_comments": _remap_comments(
                item.get("top_comments") or [],
                # TikTok uses digg_count as the vote field; Instagram has no
                # comment fetcher today so the key is harmlessly absent.
                score_keys=("score", "digg_count", "likes"),
                excerpt_keys=("excerpt", "text"),
            ),
        },
    )
 def _normalize_pinterest(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    """Normalizer for Pinterest pins (visual content with descriptions).
    Saves are the primary engagement signal, analogous to likes/upvotes.
    """
    description = str(item.get("description") or "").strip()
    return _source_item(
        item_id=str(item.get("pin_id") or item.get("id") or f"PI{index + 1}"),
        source=source,
        title=description[:140] or f"Pinterest pin {index + 1}",
        body=description,
        url=str(item.get("url") or ""),
        author=str(item.get("author") or ""),
        container=str(item.get("board") or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="low"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=description[:400],
    )
 def _normalize_hackernews(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    top_comments = item.get("top_comments") or []
    comment_text = _join_comment_excerpts(top_comments, "text")
    title = str(item.get("title") or "").strip()
    body = "\n".join(part for part in [title, str(item.get("text") or "").strip(), comment_text] if part)
    return _source_item(
        item_id=str(item.get("id") or f"HN{index + 1}"),
        source=source,
        title=title or f"HN story {index + 1}",
        body=body,
        url=str(item.get("url") or item.get("hn_url") or ""),
        author=str(item.get("author") or ""),
        container="Hacker News",
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=comment_text,
        metadata={
            "hn_url": item.get("hn_url"),
            "top_comments": top_comments,
            "comment_insights": item.get("comment_insights") or [],
        },
    )
 def _normalize_microblog(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
    id_prefix: str,
    default_title: str,
 ) -> schema.SourceItem:
    """Shared normalizer for Bluesky and Truth Social (identical structure)."""
    text = str(item.get("text") or "").strip()
    return _source_item(
        item_id=str(item.get("id") or f"{id_prefix}{index + 1}"),
        source=source,
        title=text[:140] or f"{default_title} {index + 1}",
        body=text,
        url=str(item.get("url") or ""),
        author=str(item.get("handle") or item.get("author_handle") or "").lstrip("@"),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        metadata={"display_name": item.get("display_name")},
    )
 def _normalize_digg(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    """Normalizer for Digg AI 1000 clusters.
    Each cluster is one item. The TLDR carries the most useful body for
    rerank and synthesis. Top-ranked X posts attached at search time are
    passed through under metadata['posts'] so render can emit them as
    inline 'via Digg' quotes.
    """
    title = str(item.get("title") or "").strip()
    tldr = str(item.get("tldr") or "").strip()
    body = "\n\n".join(part for part in [title, tldr] if part)
    posts = item.get("posts") or []
    if not isinstance(posts, list):
        posts = []
    cluster_url_id = str(item.get("id") or f"DG{index + 1}")
    return _source_item(
        item_id=cluster_url_id,
        source=source,
        title=title or f"Digg cluster {index + 1}",
        body=body,
        url=str(item.get("url") or f"https://di.gg/ai/{cluster_url_id}"),
        author="",
        container="Digg",
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=tldr[:400],
        metadata={
            "clusterUrlId": cluster_url_id,
            "tldr": tldr,
            "rank": (item.get("engagement") or {}).get("rank"),
            "uniqueAuthors": (item.get("engagement") or {}).get("uniqueAuthors"),
            "postCount": (item.get("engagement") or {}).get("postCount"),
            "firstPostAge": item.get("first_post_age"),
            "posts": posts,
        },
    )
 def _normalize_polymarket(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    title = str(item.get("title") or "").strip()
    question = str(item.get("question") or "").strip()
    engagement = {
        "volume": item.get("volume1mo") or item.get("volume24hr") or 0,
        "liquidity": item.get("liquidity") or 0,
    }
    return _source_item(
        item_id=str(item.get("id") or f"PM{index + 1}"),
        source=source,
        title=title or question or f"Polymarket event {index + 1}",
        body="\n".join(part for part in [title, question, str(item.get("price_movement") or "")] if part),
        url=str(item.get("url") or ""),
        author=None,
        container="Polymarket",
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=engagement,
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=str(item.get("price_movement") or ""),
        metadata={
            "question": question,
            "end_date": item.get("end_date"),
            "outcome_prices": item.get("outcome_prices") or [],
            "outcomes_remaining": item.get("outcomes_remaining"),
        },
    )
 def _normalize_github(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    title = str(item.get("title") or "").strip()
    snippet_text = str(item.get("snippet") or "").strip()
    top_comments = item.get("metadata", {}).get("top_comments") or []
    comment_text = _join_comment_excerpts(top_comments, "excerpt")
    body = "\n".join(part for part in [title, snippet_text, comment_text] if part)
    metadata = item.get("metadata") or {}
    return _source_item(
        item_id=str(item.get("id") or f"GH{index + 1}"),
        source=source,
        title=title or f"GitHub item {index + 1}",
        body=body,
        url=str(item.get("url") or ""),
        author=str(item.get("author") or ""),
        container=str(item.get("container") or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date, default="high"),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=comment_text or snippet_text[:400],
        metadata={
            "top_comments": top_comments,
            "labels": metadata.get("labels") or [],
            "state": metadata.get("state", ""),
            "is_pr": metadata.get("is_pr", False),
        },
    )
 def _normalize_grounding(
    source: str,
    item: dict[str, Any],
    index: int,
    from_date: str,
    to_date: str,
 ) -> schema.SourceItem:
    title = str(item.get("title") or "").strip()
    snippet = str(item.get("snippet") or "").strip()
    url = str(item.get("url") or "").strip()
    return _source_item(
        item_id=str(item.get("id") or f"W{index + 1}"),
        source=source,
        title=title or _domain_from_url(url) or f"Web result {index + 1}",
        body="\n".join(part for part in [title, snippet] if part),
        url=url,
        author=None,
        container=str(item.get("source_domain") or _domain_from_url(url) or ""),
        published_at=item.get("date"),
        date_confidence=_date_confidence(item, from_date, to_date),
        engagement=item.get("engagement") or {},
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=snippet,
        metadata=item.get("metadata") or {},
    )
@@ -0,0 +1,164 @@
 """Perplexity Sonar Pro / Deep Research via OpenRouter API.
 Queries Perplexity models through OpenRouter for AI-synthesized research
 with citation annotations. Returns normalized items with synthesis text
 and individual citation entries.
 """
 from __future__ import annotations
 import sys
 from urllib.parse import urlparse
 from . import http, log
 OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
 MODEL_SONAR_PRO = "perplexity/sonar-pro"
 MODEL_DEEP_RESEARCH = "perplexity/sonar-deep-research"
 def _log(msg: str):
    log.source_log("Perplexity", msg)
 def _domain(url: str) -> str:
    return urlparse(url).netloc.strip().lower()
 def search(
    query: str,
    date_range: tuple[str, str],
    config: dict,
    deep: bool = False,
 ) -> tuple[list[dict], dict]:
    """Search via Perplexity Sonar Pro or Deep Research through OpenRouter.
    Args:
        query: Search topic
        date_range: (from_date, to_date) as YYYY-MM-DD strings
        config: Must contain OPENROUTER_API_KEY
        deep: Use Deep Research model (~$0.90/query) instead of Sonar Pro
    Returns:
        Tuple of (items list, artifact dict).
    """
    api_key = config.get("OPENROUTER_API_KEY")
    if not api_key:
        _log("No OPENROUTER_API_KEY configured, skipping")
        return [], {}
    from_date, to_date = date_range
    model = MODEL_DEEP_RESEARCH if deep else MODEL_SONAR_PRO
    timeout = 120 if deep else 30
    if deep:
        print("[Perplexity] Using Deep Research (~$0.90/query)", file=sys.stderr)
    prompt = (
        f"What has been happening with {query} between {from_date} and {to_date}? "
        "Include specific dates, names, numbers, and sources."
    )
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }
    json_data = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
    }
    _log(f"Querying {model} for '{query}' ({from_date} to {to_date})")
    try:
        data = http.post(OPENROUTER_URL, json_data, headers=headers, timeout=timeout)
    except http.HTTPError as e:
        if e.status_code == 401:
            _log("Invalid OpenRouter API key (401)")
        elif e.status_code == 429:
            _log("Rate limited by OpenRouter (429)")
        else:
            _log(f"HTTP error: {e}")
        return [], {}
    except Exception as e:
        _log(f"Request failed: {e}")
        return [], {}
    # Parse response
    choices = data.get("choices", [])
    if not choices:
        _log("No choices in response")
        return [], {}
    synthesis = choices[0].get("message", {}).get("content", "")
    if not synthesis:
        _log("Empty synthesis content")
        return [], {}
    # Extract citations from annotations
    annotations = choices[0].get("message", {}).get("annotations", [])
    citations = []
    for ann in annotations:
        url_citation = ann.get("url_citation", {})
        url = url_citation.get("url", "")
        title = url_citation.get("title", "")
        if url:
            citations.append({"url": url, "title": title})
    # Deduplicate citations by URL
    seen_urls = set()
    unique_citations = []
    for c in citations:
        if c["url"] not in seen_urls:
            seen_urls.add(c["url"])
            unique_citations.append(c)
    citations = unique_citations
    _log(f"Got synthesis ({len(synthesis)} chars) with {len(citations)} citations")
    # Build items list
    items = []
    # Primary item: the synthesis itself
    snippet = synthesis[:2000]
    items.append({
        "id": "PX1",
        "title": f"Perplexity {'Deep Research' if deep else 'Sonar Pro'}: {query}",
        "url": "",
        "source_domain": "perplexity.ai",
        "snippet": snippet,
        "date": to_date,
        "relevance": 0.9,
        "why_relevant": f"AI synthesis of recent activity for '{query}'",
        "engagement": {"citations": len(citations)},
        "metadata": {"citations": citations},
    })
    # Individual items for each citation
    for i, cit in enumerate(citations):
        items.append({
            "id": f"PX{i + 2}",
            "title": cit["title"] or _domain(cit["url"]),
            "url": cit["url"],
            "source_domain": _domain(cit["url"]),
            "snippet": "",
            "date": None,
            "relevance": 0.7,
            "why_relevant": f"Cited in Perplexity synthesis for '{query}'",
            "engagement": {"citations": 1},
            "metadata": {"citations": [cit]},
        })
    artifact = {
        "label": "perplexity",
        "model": model,
        "deep": deep,
        "query": query,
        "synthesisLength": len(synthesis),
        "citationCount": len(citations),
    }
    return items, artifact
@@ -0,0 +1,163 @@
 """Pinterest search via ScrapeCreators API for /last30days.
 Uses ScrapeCreators REST API to search Pinterest by keyword, extract
 engagement metrics (saves, comments), and return pin descriptions.
 Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
 API docs: https://scrapecreators.com/docs
 """
 import re
 import sys
 from typing import Any, Dict, List, Optional, Set
 from . import dates, http, log
 SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/pinterest"
 # Depth configurations: how many results to fetch
 DEPTH_CONFIG = {
    "quick":   {"results_per_page": 10},
    "default": {"results_per_page": 20},
    "deep":    {"results_per_page": 40},
 }
 from .relevance import token_overlap_relevance as _compute_relevance
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for Pinterest search."""
    from .query import extract_core_subject
    _PINTEREST_NOISE = frozenset({
        'best', 'top', 'good', 'great', 'awesome', 'killer',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features',
        'recommendations', 'advice',
        'prompt', 'prompts', 'prompting',
        'methods', 'strategies', 'approaches',
    })
    return extract_core_subject(topic, noise=_PINTEREST_NOISE)
 def _log(msg: str):
    log.source_log("Pinterest", msg)
 def _parse_items(raw_items: List[Dict[str, Any]], core_topic: str) -> List[Dict[str, Any]]:
    """Parse raw Pinterest items into normalized dicts.
    Pinterest pins are visual content with descriptions. Saves are the
    primary engagement signal (analogous to upvotes/likes on other platforms).
    """
    items = []
    for raw in raw_items:
        if not isinstance(raw, dict):
            continue
        pin_id = str(raw.get("id", raw.get("pin_id", "")))
        description = str(raw.get("description") or raw.get("title") or "")
        # Engagement metrics - saves are the primary signal
        save_count = raw.get("save_count") or raw.get("saves") or raw.get("repin_count") or 0
        comment_count = raw.get("comment_count") or raw.get("comments") or 0
        # Author info
        pinner = raw.get("pinner") or raw.get("creator") or raw.get("user") or {}
        if isinstance(pinner, dict):
            author_name = pinner.get("username") or pinner.get("full_name") or ""
        elif isinstance(pinner, str):
            author_name = pinner
        else:
            author_name = ""
        # URL
        url = raw.get("link") or raw.get("url") or ""
        if not url and pin_id:
            url = f"https://www.pinterest.com/pin/{pin_id}/"
        # Board info (container for pins)
        board = raw.get("board") or {}
        board_name = board.get("name", "") if isinstance(board, dict) else ""
        # Compute relevance
        relevance = _compute_relevance(core_topic, description, [])
        items.append({
            "pin_id": pin_id,
            "description": description,
            "url": url,
            "author": author_name,
            "board": board_name,
            "engagement": {
                "saves": save_count,
                "comments": comment_count,
            },
            "relevance": relevance,
            "why_relevant": f"Pinterest: {description[:60]}" if description else f"Pinterest: {core_topic}",
        })
    return items
 def parse_pinterest_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse Pinterest search response to normalized format.
    Returns:
        List of item dicts ready for normalization.
    """
    return response.get("items", [])
 def search_pinterest(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    token: str = None,
 ) -> Dict[str, Any]:
    """Search Pinterest via ScrapeCreators API.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        token: ScrapeCreators API key
    Returns:
        Dict with 'items' list and optional 'error'.
    """
    if not token:
        return {"items": [], "error": "No SCRAPECREATORS_API_KEY configured"}
    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)
    _log(f"Searching Pinterest for '{core_topic}' (depth={depth}, count={config['results_per_page']})")
    try:
        data = http.get(
            f"{SCRAPECREATORS_BASE}/search",
            params={"keyword": core_topic},
            headers=http.scrapecreators_headers(token),
            timeout=30,
            retries=2,
        )
    except Exception as e:
        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}
    # Extract items from response - try common SC response shapes
    raw_items = data.get("pins") or data.get("results") or data.get("data") or data.get("items") or []
    # Limit to configured count
    raw_items = raw_items[:config["results_per_page"]]
    # Parse items
    items = _parse_items(raw_items, core_topic)
    # Sort by saves descending (primary engagement signal)
    items.sort(key=lambda x: x["engagement"]["saves"], reverse=True)
    _log(f"Found {len(items)} Pinterest pins")
    return {"items": items}
@@ -0,0 +1,738 @@
 """LLM-first query planning with deterministic guards for risky queries."""
 from __future__ import annotations
 import json
 import re
 from . import http, providers, query, schema
 ALLOWED_INTENTS = {
    "factual",
    "product",
    "concept",
    "opinion",
    "how_to",
    "comparison",
    "breaking_news",
    "prediction",
 }
 ALLOWED_CLUSTER_MODES = {"none", "story", "workflow", "market", "debate"}
 QUICK_SOURCE_PRIORITY = {
    "factual": ["hackernews", "reddit", "x", "xquik", "youtube"],
    "product": ["youtube", "reddit", "x", "xquik", "tiktok"],
    "concept": ["hackernews", "reddit", "x", "xquik", "youtube"],
    "opinion": ["reddit", "x", "xquik", "youtube", "hackernews"],
    "how_to": ["youtube", "reddit", "x", "xquik", "hackernews"],
    "comparison": ["reddit", "x", "xquik", "hackernews", "youtube"],
    "breaking_news": ["x", "xquik", "reddit", "hackernews", "youtube", "polymarket"],
    "prediction": ["polymarket", "x", "xquik", "hackernews", "reddit", "youtube"],
 }
 SOURCE_PRIORITY = {
    "factual": ["hackernews", "reddit", "x", "youtube"],
    "product": ["youtube", "reddit", "x", "tiktok", "hackernews"],
    "concept": ["hackernews", "reddit", "x", "youtube"],
    "opinion": ["reddit", "x", "youtube", "hackernews"],
    "how_to": ["youtube", "reddit", "x", "hackernews"],
    "comparison": ["reddit", "x", "hackernews", "youtube"],
    "breaking_news": ["x", "reddit", "hackernews", "youtube", "polymarket"],
    "prediction": ["polymarket", "x", "hackernews", "reddit", "youtube"],
 }
 SOURCE_LIMITS = {
    "quick": {
        "factual": 2,
        "product": 2,
        "concept": 2,
        "opinion": 2,
        "how_to": 2,
        "comparison": 2,
        "breaking_news": 2,
        "prediction": 2,
    },
    # "default" intentionally absent: all available sources are searched
    # at default depth. Fusion and reranking handle quality. quick mode
    # uses tight budgets above for latency.
 }
 INTENT_SOURCE_EXCLUSIONS: dict[str, set[str]] = {
    "concept": {"polymarket"},
    "how_to": {"polymarket"},
 }
 SOURCE_CAPABILITIES = {
    "reddit": {"discussion", "social"},
    "x": {"discussion", "social"},
    "xquik": {"discussion", "social"},
    "youtube": {"video", "video_longform", "discussion"},
    "tiktok": {"video", "video_shortform", "social"},
    "instagram": {"video", "video_shortform", "social"},
    "hackernews": {"discussion", "link"},
    "bluesky": {"discussion", "social"},
    "truthsocial": {"discussion", "social"},
    "polymarket": {"market"},
    "digg": {"discussion", "social", "link"},
    "xiaohongshu": {"video", "video_shortform", "social"},
    "github": {"discussion", "link"},
    "grounding": {"web", "reference", "link"},
    "perplexity": {"web", "reference", "analysis"},
 }
 DEFAULT_INTENT_CAPABILITIES = {
    "comparison": {"discussion", "video", "web", "reference", "social", "link", "market"},
    "how_to": {"discussion", "video", "web", "reference", "link"},
 }
 def plan_query(
    *,
    topic: str,
    available_sources: list[str],
    requested_sources: list[str] | None,
    depth: str,
    provider: providers.ReasoningClient | None,
    model: str | None,
    context: str = "",
    internal_subrun: bool = False,
 ) -> schema.QueryPlan:
    """Create a query plan. Comparison queries with extractable entities use a
    deterministic plan; other intents prefer the configured reasoning provider.
    internal_subrun: when True, suppress the LAW 7 "No --plan passed" stderr
    warning. LAW 7 targets the hosting-reasoning-model path; competitor
    fan-out sub-runs are engine-internal and the warning is a false positive
    there. Default False preserves the warning on every user-facing invocation.
    """
    if _should_force_deterministic_plan(topic):
        return _fallback_plan(
            topic,
            available_sources,
            requested_sources,
            depth,
            note="deterministic-comparison-plan",
        )
    prompt = _build_prompt(topic, available_sources, requested_sources, depth)
    if context:
        prompt += f"\n\nCurrent context (from web search): {context}"
    if provider and model:
        try:
            raw = provider.generate_json(model, prompt)
            plan = _sanitize_plan(raw, topic, available_sources, requested_sources, depth)
            if plan.subqueries:
                return plan
        except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
            import sys
            print(f"[Planner] LLM planning failed, using deterministic fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
            return _fallback_plan(
                topic, available_sources, requested_sources, depth,
                note=f"fallback-plan (LLM error: {type(exc).__name__})",
            )
    # No --plan was passed and no engine-internal provider is configured.
    # The deterministic fallback path produces weaker plans than either the
    # --plan path or the engine-internal LLM planner path. Surface this so
    # the HOSTING reasoning model (Claude Code, Codex, Hermes, Gemini, etc.)
    # realizes it is supposed to generate the plan itself and pass --plan.
    # The word "provider" in engine messages refers to the engine's INTERNAL
    # planner credentials - NOT a prerequisite the caller needs. If you are
    # the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
    # Hermes Agent Use Cases failure mode.
    if not internal_subrun:
        import sys
        print(
            "[Planner] No --plan passed. If you are the reasoning model hosting "
            "this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime), "
            "YOU ARE the planner: generate a JSON query plan yourself and pass it "
            "via --plan. You do not need an API key or credentials; you ARE the "
            "LLM. The deterministic fallback below is the headless/cron path only. "
            "See LAW 7 in SKILL.md and Step 0.75 for the plan schema.",
            file=sys.stderr,
        )
    return _fallback_plan(topic, available_sources, requested_sources, depth)
 def _build_prompt(
    topic: str,
    available_sources: list[str],
    requested_sources: list[str] | None,
    depth: str,
 ) -> str:
    requested = ", ".join(requested_sources or ["auto"])
    available = ", ".join(available_sources)
    return f"""
 You are the query planner for a live last-30-days research pipeline.
 Topic: {topic}
 Depth: {depth}
 Available sources: {available}
 Requested sources: {requested}
 Return JSON only with this shape:
 {{
  "intent": "factual|product|concept|opinion|how_to|comparison|breaking_news|prediction",
  "freshness_mode": "strict_recent|balanced_recent|evergreen_ok",
  "cluster_mode": "none|story|workflow|market|debate",
  "source_weights": {{"source_name": 0.0}},
  "subqueries": [
    {{
      "label": "short label",
      "search_query": "keyword style query for search APIs",
      "ranking_query": "natural language rewrite for reranking",
      "sources": ["reddit", "x", "grounding"],
      "weight": 1.0
    }}
  ],
  "notes": ["optional short notes"]
 }}
 Rules:
 - emit 1 to 5 subqueries (how_to/opinion/product/breaking_news intents benefit from 4-5; factual/concept from 2)
 - every subquery must include both search_query and ranking_query
 - sources must be drawn from Available sources only
 - use cluster_mode=none for factual or many how-to queries
 - use strict_recent for breaking news and most predictions
 - use debate for comparison/opinion, market for prediction, workflow for how_to, story for breaking_news
 - search_query should be concise and keyword-heavy
 - ranking_query should read like a natural-language question
 - preserve exact proper nouns and entity strings from the topic
 - NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
 - NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
 - INTENT-MODIFIER HANDLING: when the topic contains one of {{use cases, use case, workflows, workflow, examples, tutorial, tutorials, review, reviews, comparison, applications, in practice, production, production use, how i use}}, STRIP that phrase from every search_query (keep its meaning in ranking_query). Emit 4-5 paraphrased subqueries that each express the intent differently (e.g., 'production', 'workflow OR pipeline', 'review OR experience', 'vs COMPETITOR', 'community discussion'). Broad retrieval, narrow ranking. This was the 2026-04-19 Hermes Agent Use Cases failure mode: the planner echoed "hermes agent use cases" as a literal search string and returned near-zero results because nobody posts that exact phrase.
 - DO NOT quote the user's full topic verbatim in search_query. Quote only multi-word proper nouns like "Hermes Agent", "Claude Code", "Nous Research". Bare keywords OR'd together retrieve more than exact-phrase searches.
 - search_query should match how content is TITLED on platforms
 - GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
 """.strip()
 def _sanitize_plan(
    raw: dict,
    topic: str,
    available_sources: list[str],
    requested_sources: list[str] | None,
    depth: str,
 ) -> schema.QueryPlan:
    intent_hint = str(raw.get("intent") or _infer_intent(topic)).strip()
    if intent_hint not in ALLOWED_INTENTS:
        intent_hint = _infer_intent(topic)
    requested = set(requested_sources or [])
    available = set(available_sources)
    eligible_sources = [
        source for source in available_sources
        if (not requested or source in requested)
    ]
    source_weights = {
        source: float(weight)
        for source, weight in (raw.get("source_weights") or {}).items()
        if source in available
    }
    if requested:
        source_weights = {source: weight for source, weight in source_weights.items() if source in requested}
    if not source_weights:
        source_weights = _default_source_weights(_infer_intent(topic), eligible_sources)
    # Ensure all eligible sources are available for subqueries. The LLM may
    # assign high weights to its preferred sources, but omitted sources still
    # participate with base weight so retrieval can overfetch and let fusion
    # decide quality.
    for source in eligible_sources:
        source_weights.setdefault(source, 1.0)
    if intent_hint in DEFAULT_INTENT_CAPABILITIES and depth != "quick":
        for source in _default_sources_for_intent(intent_hint, eligible_sources):
            source_weights.setdefault(source, 1.0)
    source_weights = _normalize_weights(source_weights)
    subqueries: list[schema.SubQuery] = []
    for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint, topic)], start=1):
        if not isinstance(subquery, dict):
            continue
        sources = [source for source in subquery.get("sources") or [] if source in source_weights]
        if requested:
            sources = [source for source in sources if source in requested]
        if not sources:
            sources = list(source_weights)
        search_query = str(subquery.get("search_query") or "").strip()
        ranking_query = str(subquery.get("ranking_query") or "").strip()
        if not search_query or not ranking_query:
            continue
        subqueries.append(
            schema.SubQuery(
                label=str(subquery.get("label") or f"q{index}").strip() or f"q{index}",
                search_query=search_query,
                ranking_query=ranking_query,
                sources=sources,
                weight=max(0.05, float(subquery.get("weight") or 1.0)),
            )
        )
    if depth == "quick" and subqueries:
        subqueries = subqueries[:1]
    if not subqueries:
        return _fallback_plan(topic, available_sources, requested_sources, depth)
    intent = intent_hint
    freshness_mode = str(raw.get("freshness_mode") or _default_freshness(intent)).strip()
    if intent == "how_to":
        freshness_mode = "evergreen_ok"
    cluster_mode = str(raw.get("cluster_mode") or _default_cluster_mode(intent)).strip()
    if cluster_mode not in ALLOWED_CLUSTER_MODES:
        cluster_mode = _default_cluster_mode(intent)
    return schema.QueryPlan(
        intent=intent,
        freshness_mode=freshness_mode,
        cluster_mode=cluster_mode,
        raw_topic=topic,
        subqueries=_normalize_subquery_weights(
            _trim_subqueries_for_depth(
                subqueries,
                intent,
                depth,
                eligible_sources,
                requested_sources=requested_sources,
            )
        ),
        source_weights=source_weights,
        notes=[str(note).strip() for note in raw.get("notes") or [] if str(note).strip()],
    )
 def _normalize_subquery_weights(subqueries: list[schema.SubQuery]) -> list[schema.SubQuery]:
    total = sum(subquery.weight for subquery in subqueries) or 1.0
    return [
        schema.SubQuery(
            label=subquery.label,
            search_query=subquery.search_query,
            ranking_query=subquery.ranking_query,
            sources=subquery.sources,
            weight=subquery.weight / total,
        )
        for subquery in subqueries
    ]
 def _normalize_weights(weights: dict[str, float]) -> dict[str, float]:
    total = sum(max(weight, 0.0) for weight in weights.values()) or 1.0
    return {
        source: max(weight, 0.0) / total
        for source, weight in weights.items()
    }
 def _trim_subqueries_for_depth(
    subqueries: list[schema.SubQuery],
    intent: str,
    depth: str,
    available_sources: list[str],
    requested_sources: list[str] | None = None,
 ) -> list[schema.SubQuery]:
    # At non-quick depth, expand sources: use capability routing for intents
    # that define it, or all available sources otherwise. The LLM planner may
    # assign narrow source lists; we override to let fusion decide quality.
    if depth != "quick":
        expanded_sources = _default_sources_for_intent(intent, available_sources)
        return [
            schema.SubQuery(
                label=subquery.label,
                search_query=subquery.search_query,
                ranking_query=subquery.ranking_query,
                sources=expanded_sources,
                weight=subquery.weight,
            )
            for subquery in subqueries
        ]
    limits = SOURCE_LIMITS.get(depth)
    if not limits:
        return subqueries
    priority_table = QUICK_SOURCE_PRIORITY if depth == "quick" else SOURCE_PRIORITY
    priority = priority_table.get(intent, priority_table["breaking_news"])
    limit = limits.get(intent, 3)
    ranked_sources = [source for source in priority if source in available_sources]
    if not ranked_sources:
        ranked_sources = list(available_sources)
    trimmed = []
    for subquery in subqueries:
        if depth in {"quick", "default"}:
            preferred_sources = ranked_sources[:limit]
            if requested_sources:
                requested = [
                    source
                    for source in requested_sources
                    if source in available_sources and source in subquery.sources
                ]
                for source in requested:
                    if source not in preferred_sources:
                        preferred_sources.append(source)
        else:
            preferred_sources = [source for source in ranked_sources if source in subquery.sources][:limit]
            if len(preferred_sources) < limit:
                for source in ranked_sources:
                    if source in preferred_sources:
                        continue
                    preferred_sources.append(source)
                    if len(preferred_sources) >= limit:
                        break
        trimmed.append(
            schema.SubQuery(
                label=subquery.label,
                search_query=subquery.search_query,
                ranking_query=subquery.ranking_query,
                sources=preferred_sources,
                weight=subquery.weight,
            )
        )
    return trimmed
 def _fallback_plan(
    topic: str,
    available_sources: list[str],
    requested_sources: list[str] | None,
    depth: str,
    note: str = "fallback-plan",
 ) -> schema.QueryPlan:
    intent = _infer_intent(topic)
    allowed_sources = requested_sources or available_sources
    source_weights = _default_source_weights(intent, allowed_sources)
    core = query.extract_core_subject(topic, max_words=6, strip_suffixes=True)
    base_search = _keyword_query(topic, core)
    base_ranking = _ranking_query(topic, core)
    subqueries = [schema.SubQuery(
        label="primary",
        search_query=base_search,
        ranking_query=base_ranking,
        sources=list(source_weights),
        weight=1.0,
    )]
    if depth != "quick" and intent == "comparison":
        entities = _comparison_entities(topic)
        if entities:
            for index, entity in enumerate(entities, start=1):
                subqueries.append(
                    schema.SubQuery(
                        label=f"entity-{index}",
                        search_query=entity,
                        ranking_query=f"What recent evidence from the last 30 days is most relevant to {entity} in the comparison '{topic}'?",
                        sources=list(source_weights),
                        weight=0.65,
                    )
                )
    elif depth != "quick" and intent == "prediction":
        subqueries.append(
            schema.SubQuery(
                label="odds",
                search_query=f"{base_search} odds forecast",
                ranking_query=f"What are the current odds, forecasts, or market signals about {topic}?",
                sources=[source for source in source_weights if source in {"polymarket", "grounding", "x", "reddit"}] or list(source_weights),
                weight=0.7,
            )
        )
    elif depth != "quick" and intent == "breaking_news":
        subqueries.append(
            schema.SubQuery(
                label="reaction",
                search_query=f"{base_search} reaction update",
                ranking_query=f"What new reactions or follow-up reporting from the last 30 days matter for {topic}?",
                sources=[source for source in source_weights if source in {"x", "reddit", "grounding", "hackernews"}] or list(source_weights),
                weight=0.7,
            )
        )
    # Intent-modifier fanout: when topic contains a phrase like "use cases",
    # "workflows", "examples", "review" (see _INTENT_MODIFIER_PATTERNS),
    # paraphrase the intent across 3 extra subqueries rather than echoing
    # the literal phrase. Fixes 2026-04-19 Hermes Agent Use Cases failure.
    # Excluded for comparison/prediction since those already have dedicated
    # fanout (entity-per-subquery / odds).
    if depth != "quick" and intent not in {"comparison", "prediction"} and _has_intent_modifier(topic):
        subqueries.extend(_intent_modifier_subqueries(topic, core, base_search, source_weights))
    return schema.QueryPlan(
        intent=intent,
        freshness_mode=_default_freshness(intent),
        cluster_mode=_default_cluster_mode(intent),
        raw_topic=topic,
        subqueries=_normalize_subquery_weights(
            _trim_subqueries_for_depth(
                subqueries[:_max_subqueries(intent, topic)],
                intent,
                depth,
                list(source_weights),
                requested_sources=requested_sources,
            )
        ),
        source_weights=_normalize_weights(source_weights),
        notes=[note],
    )
 def _infer_intent(topic: str) -> str:
    text = topic.lower().strip()
    if re.search(r"\b(vs|versus|compare|compared to|difference between)\b", text):
        return "comparison"
    # Slash-separated proper nouns: "React/Vue/Svelte" (not URLs, not acronyms like CI/CD or I/O)
    if not re.search(r"https?://", topic) and re.search(r"\b[A-Z][a-z]{2,}(?:/[A-Z][a-z]{2,})+\b", topic):
        return "comparison"
    if re.search(r"\b(odds|predict|prediction|forecast|chance|probability|will .* win)\b", text):
        return "prediction"
    if re.search(r"\b(how to|tutorial|guide|setup|step by step|deploy|install)\b", text):
        return "how_to"
    if re.search(r"\b(what is|what are|who is|who acquired|when did|parameter count|release date)\b", text):
        return "factual"
    if re.search(r"\b(thoughts on|worth it|should i|opinion|review)\b", text):
        return "opinion"
    if re.search(r"\b(latest|news|announced|just shipped|launched|released|update)\b", text):
        return "breaking_news"
    if re.search(r"\b(pricing|feature|features|best .* for|top .* for)\b", text):
        return "product"
    if re.search(r"\b(explain|concept|protocol|architecture|what does)\b", text):
        return "concept"
    if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
        return "breaking_news"
    # Recency signals take priority when nothing more specific matched.
    if re.search(r"\b(trending|this week|right now|today|this month)\b", text):
        return "breaking_news"
    # Default changed from "breaking_news" to "concept" on 2026-04-19 after
    # the Hermes Agent Use Cases failure: unclassified topics were getting
    # strict_recent freshness, which over-weighted the last 7 days and
    # under-weighted older relevant material. "concept" defaults to
    # evergreen_ok freshness, a safer posture for unknown topics.
    return "concept"
 def _default_freshness(intent: str) -> str:
    if intent in {"breaking_news", "prediction"}:
        return "strict_recent"
    if intent in {"concept", "how_to"}:
        return "evergreen_ok"
    return "balanced_recent"
 def _default_cluster_mode(intent: str) -> str:
    return {
        "breaking_news": "story",
        "comparison": "debate",
        "opinion": "debate",
        "prediction": "market",
        "how_to": "workflow",
        "factual": "none",
        "product": "none",
        "concept": "none",
    }.get(intent, "none")
 def _default_source_weights(intent: str, sources: list[str]) -> dict[str, float]:
    base = {source: 1.0 for source in sources}
    if intent == "prediction":
        for source, bonus in {"polymarket": 2.5, "x": 1.3}.items():
            if source in base:
                base[source] += bonus
    elif intent == "breaking_news":
        for source, bonus in {"x": 1.5, "reddit": 1.3, "hackernews": 0.8}.items():
            if source in base:
                base[source] += bonus
    elif intent == "how_to":
        for source, bonus in {"youtube": 2.0, "hackernews": 0.8}.items():
            if source in base:
                base[source] += bonus
    elif intent == "factual":
        for source, bonus in {"reddit": 0.8, "x": 0.5}.items():
            if source in base:
                base[source] += bonus
    return base
 def _keyword_query(topic: str, core: str) -> str:
    """Build a search_query string for the deterministic fallback.
    Quote ONLY title-cased multi-word proper nouns ("Hermes Agent",
    "Claude Code", "Nous Research") so platform search engines preserve the
    name as a phrase. Hyphenated compounds and lowercase terms are left as
    bare keywords, which broadens retrieval instead of narrowing it.
    Prior behavior quoted the entire compound including the user's typed
    topic, producing searches like `"Hermes Agent Actual Use Cases" hermes agent actual`
    that returned near-zero matches on X and Reddit because nobody posts
    that exact phrase. See 2026-04-19 Hermes Agent Use Cases failure.
    """
    compounds = query.extract_compound_terms(topic)
    # Only quote title-cased proper nouns (multi-word names). Hyphenated
    # compounds go unquoted so platform tokenizers can split and match.
    title_cased = [
        term for term in compounds
        if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
    ]
    quoted = " ".join(f'"{term}"' for term in title_cased[:2])
    keywords = [quoted.strip(), core.strip() or topic.strip()]
    return " ".join(part for part in keywords if part).strip()
 def _ranking_query(topic: str, core: str) -> str:
    if topic.strip().endswith("?"):
        return topic.strip()
    if core and core.lower() != topic.lower():
        return f"What recent evidence from the last 30 days is most relevant to {topic}, especially about {core}?"
    return f"What recent evidence from the last 30 days is most relevant to {topic}?"
 _TRAILING_CONTEXT = re.compile(
    r"\s+\b(?:for|in|on|at|to|with|about|from|by|during|since|after|before|using|via)\b.*$",
    re.I,
 )
 def _comparison_entities(topic: str) -> list[str]:
    # "difference between X and Y" -> "X vs Y" (replace "and" only in this context)
    normalized = re.sub(
        r"\bdifference between\s+(.+?)\s+and\s+",
        r"\1 vs ",
        topic,
        flags=re.I,
    )
    normalized = re.sub(r"\b(compared to)\b", " vs ", normalized, flags=re.I)
    parts = [
        part.strip(" \t\r\n?.,:;!()[]{}\"'")
        for part in re.split(r"\bvs\.?\b|\bversus\b|/", normalized, flags=re.I)
        if part.strip(" \t\r\n?.,:;!()[]{}\"'")
    ]
    # Strip trailing context from parts ("Svelte for frontend in 2026" -> "Svelte")
    if len(parts) >= 2:
        parts = [_TRAILING_CONTEXT.sub("", part).strip() or part for part in parts]
        deduped = []
        for part in parts:
            if part and part not in deduped:
                deduped.append(part)
        return deduped[:_max_subqueries("comparison")]
    return []
 def _should_force_deterministic_plan(topic: str) -> bool:
    return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
 _INTENT_MODIFIER_PATTERNS = (
    "use cases", "use case", "workflows", "workflow",
    "examples", "example", "tutorial", "tutorials",
    "review", "reviews", "comparison", "applications",
    "in practice", "production use", "production",
    "how i use",
 )
 def _has_intent_modifier(topic: str) -> bool:
    """Return True if the topic contains an intent modifier phrase.
    See 2026-04-19 Hermes Agent Use Cases failure: a literal "Hermes Agent
    use cases" search returns near-zero matches because nobody posts that
    exact phrase. Intent modifiers should be stripped from search_query
    and paraphrased across multiple subqueries.
    """
    text = topic.lower()
    return any(pattern in text for pattern in _INTENT_MODIFIER_PATTERNS)
 def _intent_modifier_subqueries(
    topic: str,
    core: str,
    base_search: str,
    source_weights: dict[str, float],
 ) -> list[schema.SubQuery]:
    """Produce paraphrased subqueries for intent-modifier topics.
    The deterministic fallback used to echo the user's literal phrase
    (e.g., "hermes agent use cases") into every search_query. This helper
    fans out 3 extra subqueries that each express the intent differently
    so retrieval pulls a broader corpus for reranking.
    """
    entity = core or topic.strip()
    sources = list(source_weights)
    return [
        schema.SubQuery(
            label="workflows",
            search_query=f"{entity} workflow pipeline",
            ranking_query=f"What real-world workflows or pipelines are people running with {entity}?",
            sources=sources,
            weight=0.6,
        ),
        schema.SubQuery(
            label="production",
            search_query=f"{entity} production real-world",
            ranking_query=f"What production deployments or real-world use cases of {entity} are people describing?",
            sources=sources,
            weight=0.55,
        ),
        schema.SubQuery(
            label="experience",
            search_query=f"{entity} experience review",
            ranking_query=f"What hands-on experience reports or reviews of {entity} exist in the last 30 days?",
            sources=sources,
            weight=0.5,
        ),
    ]
 def _max_subqueries(intent: str, topic: str | None = None) -> int:
    # how_to/opinion/product/breaking_news/prediction benefit from 4-5
    # paraphrased subqueries when the topic carries an intent modifier
    # (use cases, workflows, examples, review, etc.). See 2026-04-19
    # Hermes Agent Use Cases failure: prior cap of 3 produced near-literal
    # echoes of the topic instead of a paraphrase fanout.
    if intent == "comparison":
        return 4
    # Intent-modifier topics get headroom for paraphrase fanout even when
    # the intent itself is factual/concept. Without this, a "Hermes Agent
    # use cases" query (classified "concept" after the 2026-04-19 default
    # change) would be capped at 2 and drop the fanout.
    if topic and _has_intent_modifier(topic):
        return 5
    if intent in {"factual", "concept"}:
        return 2
    return 5
 def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
    if intent == "how_to":
        sources = _how_to_sources(available_sources)
    else:
        target_capabilities = DEFAULT_INTENT_CAPABILITIES.get(intent)
        if not target_capabilities:
            sources = list(available_sources)
        else:
            matched = [
                source
                for source in available_sources
                if SOURCE_CAPABILITIES.get(source, set()) & target_capabilities
            ]
            sources = matched or list(available_sources)
    excluded = INTENT_SOURCE_EXCLUSIONS.get(intent, set())
    if excluded:
        filtered = [s for s in sources if s not in excluded]
        return filtered or sources
    return sources
 def _how_to_sources(available_sources: list[str]) -> list[str]:
    """Pick one source per role: web/reference, video (prefer longform), discussion."""
    selected: set[str] = set()
    has_video = False
    # Order matters: web first, then longform video, generic video, discussion.
    role_capabilities = [
        {"web", "reference"},
        {"video_longform"},
        {"video"},
        {"discussion"},
    ]
    for role in role_capabilities:
        is_video_role = role & {"video", "video_longform"}
        if is_video_role and has_video:
            continue
        for source in available_sources:
            if source in selected:
                continue
            if SOURCE_CAPABILITIES.get(source, set()) & role:
                selected.add(source)
                if is_video_role:
                    has_video = True
                break
    # After core role-based selection, include remaining sources with any
    # how_to-relevant capability (video, discussion, web, reference, link).
    how_to_caps = DEFAULT_INTENT_CAPABILITIES.get("how_to", set())
    for source in available_sources:
        if source not in selected and SOURCE_CAPABILITIES.get(source, set()) & how_to_caps:
            selected.add(source)
    if not selected:
        return list(available_sources)
    return [source for source in available_sources if source in selected]
@@ -0,0 +1,786 @@
 """Polymarket prediction market search via Gamma API (free, no auth required).
 Uses gamma-api.polymarket.com for event/market discovery.
 No API key needed - public read-only API with generous rate limits (15K req/10s).
 """
 import json
 import math
 import re
 import sys
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from typing import Any, Dict, List, Optional
 from urllib.parse import quote_plus, urlencode
 from . import http, log
 from .relevance import LOW_SIGNAL_QUERY_TOKENS, token_overlap_relevance
 GAMMA_SEARCH_URL = "https://gamma-api.polymarket.com/public-search"
 # Pages to fetch per query (API returns 5 events per page, limit param is a no-op)
 DEPTH_CONFIG = {
    "quick": 1,
    "default": 3,
    "deep": 4,
 }
 # Max events to return after merge + dedup + re-ranking
 RESULT_CAP = {
    "quick": 5,
    "default": 15,
    "deep": 25,
 }
 def _log(msg: str):
    log.source_log("PM", msg)
 def _extract_core_subject(topic: str) -> str:
    """Extract core subject from topic string.
    Strips common prefixes like 'last 7 days', 'what are people saying about', etc.
    """
    topic = topic.strip()
    # Remove common leading phrases
    prefixes = [
        r"^last \d+ days?\s+",
        r"^what(?:'s| is| are) (?:people saying about|happening with|going on with)\s+",
        r"^how (?:is|are)\s+",
        r"^tell me about\s+",
        r"^research\s+",
    ]
    for pattern in prefixes:
        topic = re.sub(pattern, "", topic, flags=re.IGNORECASE)
    return topic.strip()
 def _expand_queries(topic: str) -> List[str]:
    """Generate search queries to cast a wider net.
    Strategy:
    - Always include the core subject
    - Add ALL individual words as standalone searches (not just first)
    - Include the full topic if different from core
    - Cap at 6 queries, dedupe
    """
    core = _extract_core_subject(topic)
    queries = [core]
    # Add ALL individual words as separate queries
    words = core.split()
    if len(words) >= 2:
        for word in words:
            if len(word) > 1 and word.lower() not in LOW_SIGNAL_QUERY_TOKENS and word.lower() not in _NOISE_WORDS:
                queries.append(word)
    # Add the full topic if different from core
    if topic.lower().strip() != core.lower():
        queries.append(topic.strip())
    # Dedupe while preserving order, cap at 6
    seen = set()
    unique = []
    for q in queries:
        q_lower = q.lower().strip()
        if q_lower and q_lower not in seen:
            seen.add(q_lower)
            unique.append(q.strip())
    return unique[:6]
 _GENERIC_TAGS = frozenset({"sports", "politics", "crypto", "science", "culture", "pop culture"})
 # Words that are too generic to serve as the sole topic-match signal.
 # If ALL core words from the topic are in this set, we skip filtering (can't meaningfully filter).
 # But if some words are informative and some are generic, we require at least one informative word.
 _NOISE_WORDS = frozenset({
    # Articles, prepositions, conjunctions
    "the", "a", "an", "in", "on", "at", "of", "for", "and", "or", "to", "is", "are",
    "was", "were", "will", "be", "by", "with", "from", "as", "it", "its", "not", "no",
    "but", "if", "so", "do", "has", "had", "have", "this", "that", "what", "who",
    # Directional / geographic terms that cause false matches
    "west", "east", "north", "south", "central", "southern", "northern", "eastern", "western",
    # Common sports / category terms
    "champion", "championship", "league", "division", "conference", "cup", "series",
    "team", "game", "match", "season", "win", "winner", "finals",
    # Common geographic / place nouns that cause false matches
    # "club" -> Athletic Club, Racing Club; "island" -> Epstein's Island, Rhode Island
    "club", "island", "city", "park", "hill", "lake", "bay", "beach", "valley",
    "river", "mountain", "county", "state", "village", "town", "point", "creek",
    "springs", "heights", "ridge", "bridge", "harbor", "port", "station", "center",
    "square", "field", "forest", "garden", "tower", "school", "church", "camp",
    "ranch", "crossing", "shore", "rock", "summit", "falls", "grove", "haven",
    # Generic tech terms that match too broadly on Polymarket
    # "cli" -> any CLI tool market; "mcp" -> protocol markets; "ai" -> every AI market
    "cli", "mcp", "protocol", "tool", "app", "code", "model", "ai", "api",
    "software", "plugin", "skill", "agent", "bot", "search", "research",
    # Generic prediction market terms
    "market", "odds", "prediction", "forecast", "chance", "probability",
    # Comparison-query conjunctions — should not count as informative filter tokens
    # when the topic is "X vs Y vs Z"
    "vs", "versus",
 })
 def _passes_topic_filter(topic: str, event_title: str) -> bool:
    """Check if event title contains enough informative words from the topic.
    Prevents noise like "Meek Mill" matching "Mill.com food recycler" by requiring
    proportional word overlap. For topics with 3+ informative words, at least 2 must
    match. For shorter topics, 1 match suffices (existing behavior).
    Returns True if the event should be kept, False if it should be filtered out.
    """
    core = _extract_core_subject(topic).lower()
    core_words = [w for w in re.sub(r"[^\w\s]", " ", core).split() if len(w) > 1]
    if not core_words:
        return True  # No words to check against
    # Split into informative vs generic
    informative = [w for w in core_words if w not in _NOISE_WORDS]
    # If ALL words are generic, we can't meaningfully filter — keep everything
    if not informative:
        return True
    # Normalize the title for matching
    title_lower = " ".join(re.sub(r"[^\w\s]", " ", event_title.lower()).split())
    title_words = set(title_lower.split())
    # Count how many informative words appear in the title
    match_count = 0
    for word in informative:
        # Check as whole word in the title word set
        if word in title_words:
            match_count += 1
            continue
        # Also check as substring for compound words (e.g., "kanye" in "kanyewest")
        if len(word) >= 4 and word in title_lower:
            match_count += 1
    # For topics with 3+ informative words, require at least 2 matches.
    # This prevents single-word false positives like "mill" in "Meek Mill"
    # when the topic is "Mill.com food recycler" (3 informative words).
    min_matches = 2 if len(informative) >= 3 else 1
    return match_count >= min_matches
 def _passes_any_informative_word(topic: str, event_title: str) -> bool:
    """Looser variant of _passes_topic_filter that keeps an item if ANY
    informative word from the topic appears in the title.
    Designed for post-merge validation of comparison topics (e.g., "OpenClaw vs
    Hermes vs Paperclip"), where a market mentioning just one of the entities
    is still on-topic. The stricter _passes_topic_filter (min_matches=2 for
    3+ informative words) is correct for single-entity topics like "Mill.com
    food recycler" but drops legitimate single-entity comparison results.
    """
    core = _extract_core_subject(topic).lower()
    core_words = [w for w in re.sub(r"[^\w\s]", " ", core).split() if len(w) > 1]
    if not core_words:
        return True
    informative = [w for w in core_words if w not in _NOISE_WORDS]
    if not informative:
        return True
    title_lower = " ".join(re.sub(r"[^\w\s]", " ", event_title.lower()).split())
    title_words = set(title_lower.split())
    for word in informative:
        if word in title_words:
            return True
        if len(word) >= 4 and word in title_lower:
            return True
    return False
 def filter_items_against_topic(topic: str, items: List[Any]) -> List[Any]:
    """Drop items whose title shares no informative word with the original topic.
    Called post-merge from pipeline.py so per-entity subquery results for
    comparison topics get re-validated against the ORIGINAL full topic before
    landing in the footer. Prevents noise like WTI crude oil or Elon tweet
    markets from surviving a loose "Hermes" single-entity subquery match.
    Uses the looser _passes_any_informative_word rule (ANY entity name match
    is sufficient) so a market mentioning just one of several compared entities
    still counts as on-topic.
    Accepts a list of either raw dicts (with 'title') or SourceItem-like objects
    (with .title attribute). Returns the filtered list in the same order.
    """
    if not topic:
        return items
    filtered = []
    for item in items:
        title = getattr(item, "title", None)
        if title is None and isinstance(item, dict):
            title = item.get("title", "")
        title = title or ""
        if _passes_any_informative_word(topic, title):
            filtered.append(item)
    dropped = len(items) - len(filtered)
    if dropped:
        _log(f"Post-merge topic filter dropped {dropped} Polymarket items against full topic '{topic}'")
    return filtered
 def filter_items_against_keywords(items: List[Any], keywords: List[str]) -> List[Any]:
    """Keep only items whose title contains at least one keyword (case-insensitive).
    Intended for disambiguating ambiguous single-token topics like 'Warriors'
    via --polymarket-keywords (e.g., 'nba,gsw,golden-state') to filter out
    Glasgow Warriors rugby, Honor of Kings Rogue Warriors markets that share
    the 'Warriors' token but are not the target entity.
    """
    if not keywords:
        return items
    normalized_keywords = [kw.strip().lower() for kw in keywords if kw and kw.strip()]
    if not normalized_keywords:
        return items
    filtered = []
    for item in items:
        title = getattr(item, "title", None)
        if title is None and isinstance(item, dict):
            title = item.get("title", "")
        title = (title or "").lower()
        if any(kw in title for kw in normalized_keywords):
            filtered.append(item)
    dropped = len(items) - len(filtered)
    if dropped:
        _log(
            f"Keyword filter dropped {dropped} Polymarket items; "
            f"kept {len(filtered)} matching {normalized_keywords}"
        )
    return filtered
 def _extract_domain_queries(topic: str, events: List[Dict]) -> List[str]:
    """Extract domain-indicator search terms from first-pass event tags.
    Uses structured tag metadata from Gamma API events to discover broader
    domain categories (e.g., 'NCAA CBB' from a Big 12 basketball event).
    Falls back to frequent title bigrams if no useful tags exist.
    """
    query_words = set(_extract_core_subject(topic).lower().split())
    # Collect tag labels from all first-pass events, count occurrences
    tag_counts: Dict[str, int] = {}
    for event in events:
        tags = event.get("tags") or []
        for tag in tags:
            label = tag.get("label", "") if isinstance(tag, dict) else str(tag)
            if not label:
                continue
            label_lower = label.lower()
            # Skip generic category tags and tags matching existing queries
            if label_lower in _GENERIC_TAGS:
                continue
            if label_lower in query_words:
                continue
            tag_counts[label] = tag_counts.get(label, 0) + 1
    # Sort by frequency, take top 2 that appear in 2+ events
    domain_queries = [
        label for label, count in sorted(tag_counts.items(), key=lambda x: -x[1])
        if count >= 2
    ][:2]
    return domain_queries
 def _infer_query_intent(topic: str) -> str:
    """Tiny local fallback for Polymarket search tuning only."""
    text = topic.lower().strip()
    if re.search(r"\b(predict|prediction|odds|forecast|chance|probability|will .* win)\b", text):
        return "prediction"
    return "breaking_news"
 def _search_single_query(query: str, page: int = 1) -> Dict[str, Any]:
    """Run a single search query against Gamma API."""
    params = {
        "q": query,
        "page": str(page),
        "events_status": "active",
        "keep_closed_markets": "0",
    }
    url = f"{GAMMA_SEARCH_URL}?{urlencode(params)}"
    try:
        response = http.request("GET", url, timeout=15, retries=2)
        return response
    except http.HTTPError as e:
        _log(f"Search failed for '{query}' page {page}: {e}")
        return {"events": [], "error": str(e)}
    except Exception as e:
        _log(f"Search failed for '{query}' page {page}: {e}")
        return {"events": [], "error": str(e)}
 def _run_queries_parallel(
    queries: List[str], pages: int, all_events: Dict, errors: List, start_idx: int = 0,
 ) -> None:
    """Run (query, page) combinations in parallel, merging into all_events."""
    with ThreadPoolExecutor(max_workers=min(8, len(queries) * pages)) as executor:
        futures = {}
        for i, q in enumerate(queries, start=start_idx):
            for p in range(1, pages + 1):
                future = executor.submit(_search_single_query, q, p)
                futures[future] = i
        for future in as_completed(futures):
            query_idx = futures[future]
            try:
                response = future.result(timeout=15)
                if response.get("error"):
                    errors.append(response["error"])
                events = response.get("events", [])
                for event in events:
                    event_id = event.get("id", "")
                    if not event_id:
                        continue
                    if event_id not in all_events:
                        all_events[event_id] = (event, query_idx)
                    elif query_idx < all_events[event_id][1]:
                        all_events[event_id] = (event, query_idx)
            except Exception as e:
                errors.append(str(e))
 def search_polymarket(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
 ) -> Dict[str, Any]:
    """Search Polymarket via Gamma API with two-pass query expansion.
    Pass 1: Run expanded queries in parallel, merge and dedupe by event ID.
    Pass 2: Extract domain-indicator terms from first-pass titles, search those.
    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD) - used for activity filtering
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
    Returns:
        Dict with 'events' list and optional 'error'.
    """
    pages = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    cap = RESULT_CAP.get(depth, RESULT_CAP["default"])
    queries = _expand_queries(topic)
    _log(f"Searching for '{topic}' with queries: {queries} (pages={pages})")
    # Pass 1: run expanded queries in parallel
    all_events: Dict[str, tuple] = {}
    errors: List[str] = []
    _run_queries_parallel(queries, pages, all_events, errors)
    # Pass 2: extract domain-indicator terms from first-pass titles and search
    first_pass_events = [ev for ev, _ in all_events.values()]
    domain_queries = _extract_domain_queries(topic, first_pass_events)
    # Filter out queries we already ran
    seen_queries = {q.lower() for q in queries}
    domain_queries = [dq for dq in domain_queries if dq.lower() not in seen_queries]
    if domain_queries:
        _log(f"Domain expansion queries: {domain_queries}")
        _run_queries_parallel(domain_queries, 1, all_events, errors, start_idx=len(queries))
    merged_events = [ev for ev, _ in sorted(all_events.values(), key=lambda x: x[1])]
    total_queries = len(queries) + len(domain_queries)
    _log(f"Found {len(merged_events)} unique events across {total_queries} queries")
    result = {"events": merged_events, "_cap": cap}
    if errors and not merged_events:
        result["error"] = "; ".join(errors[:2])
    return result
 def _format_price_movement(market: Dict[str, Any]) -> Optional[str]:
    """Pick the most significant price change and format it.
    Returns string like 'down 11.7% this month' or None if no significant change.
    """
    changes = [
        (abs(market.get("oneDayPriceChange") or 0), market.get("oneDayPriceChange"), "today"),
        (abs(market.get("oneWeekPriceChange") or 0), market.get("oneWeekPriceChange"), "this week"),
        (abs(market.get("oneMonthPriceChange") or 0), market.get("oneMonthPriceChange"), "this month"),
    ]
    # Pick the largest absolute change
    changes.sort(key=lambda x: x[0], reverse=True)
    abs_change, raw_change, period = changes[0]
    # Skip if change is less than 1% (noise)
    if abs_change < 0.01:
        return None
    direction = "up" if raw_change > 0 else "down"
    pct = abs_change * 100
    return f"{direction} {pct:.1f}% {period}"
 def _parse_outcome_prices(market: Dict[str, Any]) -> List[tuple]:
    """Parse outcomePrices JSON string into list of (outcome_name, price) tuples."""
    outcomes_raw = market.get("outcomes") or []
    prices_raw = market.get("outcomePrices")
    if not prices_raw:
        return []
    # Both outcomes and outcomePrices can be JSON-encoded strings
    try:
        if isinstance(outcomes_raw, str):
            outcomes = json.loads(outcomes_raw)
        else:
            outcomes = outcomes_raw
    except (json.JSONDecodeError, TypeError):
        outcomes = []
    try:
        if isinstance(prices_raw, str):
            prices = json.loads(prices_raw)
        else:
            prices = prices_raw
    except (json.JSONDecodeError, TypeError):
        return []
    result = []
    for i, price in enumerate(prices):
        try:
            p = float(price)
        except (ValueError, TypeError):
            continue
        name = outcomes[i] if i < len(outcomes) else f"Outcome {i+1}"
        result.append((name, p))
    return result
 def _shorten_question(question: str) -> str:
    """Extract a short display name from a market question.
    'Will Arizona win the 2026 NCAA Tournament?' -> 'Arizona'
    'Will Duke be a number 1 seed in the 2026 NCAA...' -> 'Duke'
    """
    q = question.strip().rstrip("?")
    # Common patterns: "Will X win/be/...", "X wins/loses..."
    m = re.match(r"^Will\s+(.+?)\s+(?:win|be|make|reach|have|lose|qualify|advance|strike|agree|pass|sign|get|become|remain|stay|leave|survive|next)\b", q, re.IGNORECASE)
    if m:
        return m.group(1).strip()
    m = re.match(r"^Will\s+(.+?)\s+", q, re.IGNORECASE)
    if m and len(m.group(1).split()) <= 4:
        return m.group(1).strip()
    # Fallback: truncate
    return question[:40] if len(question) > 40 else question
 def _compute_text_similarity(topic: str, title: str, outcomes: List[str] = None) -> float:
    """Score how well the event title (or outcome names) match the search topic.
    Returns 0.0-1.0. Exact title phrase match gets 1.0. Otherwise we reuse the
    shared query-centric relevance scorer and take the best title/outcome match.
    """
    core = _extract_core_subject(topic).lower()
    title_lower = title.lower()
    if not core:
        return 0.5
    # Full substring match in title
    if core in title_lower:
        return 1.0
    query_type = _infer_query_intent(topic)
    title_score = token_overlap_relevance(core, title)
    best_score = title_score
    if outcomes:
        for outcome_name in outcomes:
            outcome_lower = outcome_name.lower()
            outcome_score = token_overlap_relevance(core, outcome_name)
            if _strong_phrase_match(core, outcome_lower):
                outcome_score = max(outcome_score, 0.92 if len(outcome_lower.split()) >= 2 else 0.88)
            if title_score < 0.3:
                outcome_cap = 0.55 if query_type == "prediction" else 0.24
                outcome_score = min(outcome_cap, outcome_score)
            else:
                outcome_score = max(title_score, 0.75 * title_score + 0.25 * outcome_score)
            best_score = max(best_score, outcome_score)
    return round(best_score, 2)
 def _strong_phrase_match(core: str, candidate: str) -> bool:
    """Require real token matches, not accidental short substrings.
    This prevents binary outcomes like "No" from matching "nano" or similar
    short-string accidents.
    """
    candidate = " ".join(re.sub(r"[^\w\s]", " ", candidate.lower()).split())
    core = " ".join(re.sub(r"[^\w\s]", " ", core.lower()).split())
    if not candidate or not core:
        return False
    candidate_tokens = candidate.split()
    core_tokens = set(core.split())
    if len(candidate_tokens) >= 2:
        return candidate in core or core in candidate
    token = candidate_tokens[0]
    return len(token) > 2 and token in core_tokens
 def _safe_float(val, default=0.0) -> float:
    """Safely convert a value to float."""
    try:
        return float(val or default)
    except (ValueError, TypeError):
        return default
 def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List[Dict[str, Any]]:
    """Parse Gamma API response into normalized item dicts.
    Each event becomes one item showing its title and top markets.
    Args:
        response: Raw Gamma API response
        topic: Original search topic (for relevance scoring)
    Returns:
        List of item dicts ready for normalization.
    """
    events = response.get("events", [])
    items = []
    filtered_count = 0
    for i, event in enumerate(events):
        event_id = event.get("id", "")
        title = event.get("title", "")
        slug = event.get("slug", "")
        # Filter: skip closed/resolved events
        if event.get("closed", False):
            continue
        if not event.get("active", True):
            continue
        # Filter: skip events that don't match the topic's core subject
        # This prevents "NFC West" from matching a "Kanye West" search
        if topic and not _passes_topic_filter(topic, title):
            filtered_count += 1
            continue
        # Get markets for this event
        markets = event.get("markets", [])
        if not markets:
            continue
        # Filter to active, open markets with liquidity (excludes resolved markets)
        active_markets = []
        for m in markets:
            if m.get("closed", False):
                continue
            if not m.get("active", True):
                continue
            # Must have liquidity (resolved markets have 0 or None)
            try:
                liq = float(m.get("liquidity", 0) or 0)
            except (ValueError, TypeError):
                liq = 0
            if liq > 0:
                active_markets.append(m)
        if not active_markets:
            continue
        # Sort markets by volume (most liquid first)
        def market_volume(m):
            try:
                return float(m.get("volume", 0) or 0)
            except (ValueError, TypeError):
                return 0
        active_markets.sort(key=market_volume, reverse=True)
        # Take top market for the event
        top_market = active_markets[0]
        # Collect outcome names from ALL active markets (not just top) for similarity scoring
        # Filter to outcomes with price > 1% to avoid noise
        # Also extract subjects from market questions for neg-risk events (outcomes are Yes/No)
        all_outcome_names = []
        for m in active_markets:
            for name, price in _parse_outcome_prices(m):
                if price > 0.01 and name not in all_outcome_names:
                    all_outcome_names.append(name)
            # For neg-risk binary markets (Yes/No outcomes), the team/entity name
            # lives in the question, e.g., "Will Arizona win the NCAA Tournament?"
            question = m.get("question", "")
            if question and question != title:
                all_outcome_names.append(question)
        # Parse outcome prices - for multi-market events with Yes/No binary
        # sub-markets, synthesize from market questions to show actual
        # team/entity probabilities instead of a single market's Yes/No
        outcome_prices = _parse_outcome_prices(top_market)
        top_outcomes_are_binary = (
            len(outcome_prices) == 2
            and {n.lower() for n, _ in outcome_prices} == {"yes", "no"}
        )
        if top_outcomes_are_binary and len(active_markets) > 1:
            synth_outcomes = []
            for m in active_markets:
                q = m.get("question", "")
                if not q:
                    continue
                pairs = _parse_outcome_prices(m)
                yes_price = next((p for name, p in pairs if name.lower() == "yes"), None)
                if yes_price is not None and yes_price > 0.005:
                    synth_outcomes.append((q, yes_price))
            if synth_outcomes:
                synth_outcomes.sort(key=lambda x: x[1], reverse=True)
                outcome_prices = [(_shorten_question(q), p) for q, p in synth_outcomes]
        # Format price movement
        price_movement = _format_price_movement(top_market)
        # Volume and liquidity - prefer event-level (more stable), fall back to market-level
        event_volume1mo = _safe_float(event.get("volume1mo"))
        event_volume1wk = _safe_float(event.get("volume1wk"))
        event_liquidity = _safe_float(event.get("liquidity"))
        event_competitive = _safe_float(event.get("competitive"))
        volume24hr = _safe_float(event.get("volume24hr")) or _safe_float(top_market.get("volume24hr"))
        liquidity = event_liquidity or _safe_float(top_market.get("liquidity"))
        # Event URL
        url = f"https://polymarket.com/event/{slug}" if slug else f"https://polymarket.com/event/{event_id}"
        # Date: use updatedAt from event
        updated_at = event.get("updatedAt", "")
        date_str = None
        if updated_at:
            try:
                date_str = updated_at[:10]  # YYYY-MM-DD
            except (IndexError, TypeError):
                pass
        # End date for the market
        end_date = top_market.get("endDate")
        if end_date:
            try:
                end_date = end_date[:10]
            except (IndexError, TypeError):
                end_date = None
        # Semantic relevance should dominate. Market quality should refine
        # relevant matches, not rescue unrelated high-liquidity events.
        text_score = _compute_text_similarity(topic, title, all_outcome_names) if topic else 0.5
        # Volume signal: log-scaled monthly volume (most stable signal)
        vol_raw = event_volume1mo or event_volume1wk or volume24hr
        vol_score = min(1.0, math.log1p(vol_raw) / 16)  # ~$9M = 1.0
        # Liquidity signal
        liq_score = min(1.0, math.log1p(liquidity) / 14)  # ~$1.2M = 1.0
        # Price movement: daily weighted more than monthly
        day_change = abs(top_market.get("oneDayPriceChange") or 0) * 3
        week_change = abs(top_market.get("oneWeekPriceChange") or 0) * 2
        month_change = abs(top_market.get("oneMonthPriceChange") or 0)
        max_change = max(day_change, week_change, month_change)
        movement_score = min(1.0, max_change * 5)  # 20% change = 1.0
        # Competitive bonus: markets near 50/50 are more interesting
        competitive_score = event_competitive
        market_quality = (
            0.50 * vol_score +
            0.25 * liq_score +
            0.15 * movement_score +
            0.10 * competitive_score
        )
        relevance = min(1.0, text_score * (0.75 + 0.25 * market_quality))
        # Surface the topic-matching outcome to the front before truncating
        if topic and outcome_prices:
            core = _extract_core_subject(topic).lower()
            core_tokens = set(core.split())
            reordered = []
            rest = []
            for pair in outcome_prices:
                name_lower = pair[0].lower()
                # Match if full core is substring, or name is substring of core,
                # or any core token appears in the name (handles long question strings)
                if (core in name_lower or name_lower in core
                        or any(tok in name_lower for tok in core_tokens if len(tok) > 2)):
                    reordered.append(pair)
                else:
                    rest.append(pair)
            if reordered:
                outcome_prices = reordered + rest
        # Top 3 outcomes for multi-outcome markets
        top_outcomes = outcome_prices[:3]
        remaining = len(outcome_prices) - 3
        if remaining < 0:
            remaining = 0
        items.append({
            "event_id": event_id,
            "title": title,
            "question": top_market.get("question", title),
            "url": url,
            "outcome_prices": top_outcomes,
            "outcomes_remaining": remaining,
            "price_movement": price_movement,
            "volume24hr": volume24hr,
            "volume1mo": event_volume1mo,
            "liquidity": liquidity,
            "date": date_str,
            "end_date": end_date,
            "relevance": round(relevance, 2),
            "why_relevant": f"Prediction market: {title[:60]}",
        })
    if filtered_count:
        _log(f"Filtered {filtered_count} noise events (topic: '{topic}')")
    # Sort by relevance (quality-signal ranked) and apply cap
    items.sort(key=lambda x: x["relevance"], reverse=True)
    # Drop ALL results if nothing is genuinely on-topic.
    # If the best item's relevance is below the threshold, the Gamma API
    # returned only tangential matches (e.g., "Anthropic best AI model"
    # for a "CLI vs MCP" query). Better to show 0 than noise.
    _MIN_RELEVANCE = 0.15
    if items and items[0]["relevance"] < _MIN_RELEVANCE:
        _log(f"All {len(items)} Polymarket results below relevance threshold "
             f"({items[0]['relevance']:.2f} < {_MIN_RELEVANCE}), dropping all")
        return []
    # Per-item floor: drop individual noise items even if the best item passed
    _ITEM_MIN_RELEVANCE = 0.10
    before_count = len(items)
    items = [i for i in items if i["relevance"] >= _ITEM_MIN_RELEVANCE]
    dropped = before_count - len(items)
    if dropped:
        _log(f"Dropped {dropped} Polymarket items below per-item relevance floor ({_ITEM_MIN_RELEVANCE})")
    cap = response.get("_cap", len(items))
    return items[:cap]
@@ -0,0 +1,119 @@
 """Engine-side query-quality pre-flight.
 Detects Class 1 (demographic shopping) keyword-trap queries and returns a
 structured REFUSE message. The caller (scripts/last30days.py main()) writes
 the message to stderr and exits code 2. No pipeline work runs on a doomed
 query; the model sees the REFUSE on stderr and asks the user for the
 hobbies/relationship/budget context it needs.
 Patterns ported from SKILL.md Step 0.45 prose. Only Class 1 is implemented
 here because it has a verified failure mode on v3.0.8 (2026-04-18 'birthday
 gift for 40 year old' run returned r/todayilearned and unrelated drama
 posts).
 """
 from __future__ import annotations
 import re
 _CLASS_1_PATTERNS = [
    re.compile(
        r"^\s*(birthday\s+)?(gift|gifts|present|presents)\s+"
        r"(for|ideas\s+for)\s+(a\s+|my\s+)?\d+[\s-]?year[\s-]?old\b",
        re.IGNORECASE,
    ),
    re.compile(
        r"^\s*(best|top)\s+[\w\s-]+?\s+for\s+"
        r"(men|women|kids|guys|girls|teens|dads|moms|husbands|wives|brothers|sisters|friends)\b",
        re.IGNORECASE,
    ),
    re.compile(
        r"^\s*what\s+to\s+(buy|get|gift)\s+(for\s+)?(a\s+|my\s+)?"
        r"(\d+[\s-]?year[\s-]?old|husband|wife|dad|mom|brother|sister|friend|boss|coworker)\b",
        re.IGNORECASE,
    ),
    re.compile(
        r"^\s*(present|presents|gift|gifts)\s+for\s+(a\s+|my\s+)?"
        r"(husband|wife|dad|mom|brother|sister|friend|boss|coworker)\b",
        re.IGNORECASE,
    ),
 ]
 _QUALIFIER_PATTERNS = [
    re.compile(r"\$\d+"),
    re.compile(r"\bbudget\b", re.IGNORECASE),
    re.compile(r"\bwho\s+(loves|likes|is\s+into|enjoys)\b", re.IGNORECASE),
    re.compile(r"\bhobbies?\b", re.IGNORECASE),
    re.compile(r"\b(cooking|running|reading|gaming|golf|woodworking|coding|hiking|cycling|fishing|music)[\s-]?(obsessed|enthusiast|fan|lover)\b", re.IGNORECASE),
 ]
 _RELATIONSHIP_WORDS = {
    "husband", "wife", "dad", "mom", "father", "mother", "brother", "sister",
    "friend", "boss", "coworker", "son", "daughter", "grandma", "grandpa",
    "aunt", "uncle", "nephew", "niece", "partner", "boyfriend", "girlfriend",
 }
 _YEAR_OLD_NOUN = re.compile(r"\byear[\s-]?old\s+(\w+)", re.IGNORECASE)
 def _has_qualifier(topic: str) -> bool:
    """Return True if the topic contains hobbies/relationship/budget context.
    A Class 1 base pattern plus a qualifier means the user already filled in
    the specificity Step 0.45 would ask for. Skip the refuse-gate and let
    the engine run.
    Also skips when `{n} year old <activity-noun>` is present, but only when
    the noun is NOT a relationship word. 'year old runner' qualifies as an
    interest and skips; 'year old husband' is just another relationship
    reframing of the demographic query and does not skip.
    """
    if any(pattern.search(topic) for pattern in _QUALIFIER_PATTERNS):
        return True
    match = _YEAR_OLD_NOUN.search(topic)
    if match and match.group(1).lower() not in _RELATIONSHIP_WORDS:
        return True
    return False
 def check_class_1_trap(topic: str) -> str | None:
    """Return a REFUSE message string if the topic matches Class 1, else None.
    Class 1 is the demographic-shopping keyword trap. The literal phrase
    'birthday gift for 40 year old' is not the vocabulary of actual gift
    discussions on Reddit, X, or TikTok, so running the engine returns
    low-signal generic posts. Refuse up-front and ask for context.
    """
    if not topic:
        return None
    matched = any(pattern.search(topic) for pattern in _CLASS_1_PATTERNS)
    if not matched:
        return None
    if _has_qualifier(topic):
        return None
    return _refuse_message(topic.strip())
 def _refuse_message(topic: str) -> str:
    return (
        f'[last30days] REFUSE: topic "{topic}" matches Class 1 keyword-trap '
        "pattern (demographic shopping).\n"
        "\n"
        "The literal phrase is not the vocabulary of actual gift discussions "
        "on Reddit, X, or TikTok. Running the engine will return low-signal "
        "generic posts (the 2026-04-18 validation run returned "
        "r/todayilearned and unrelated drama).\n"
        "\n"
        "Ask the user for at least one of:\n"
        "  - hobbies (cooks / runs / reads / gaming / outdoors / golf / music)\n"
        "  - relationship (husband / dad / friend / boss / brother)\n"
        "  - budget range\n"
        "\n"
        "Then re-run with the enriched query. If the user insists 'just run it',\n"
        "re-invoke with LAST30DAYS_SKIP_PREFLIGHT=1 to bypass this gate.\n"
    )
--- a/Show More
+++ b/Show More