Compare commits

...

20 Commits

Author SHA1 Message Date
Matt Van Horn 122158415a chore(release): v3.3.2 (#485)
Security / Dependency audit (push) Has been cancelled
Security / Secret scan (push) Has been cancelled
Validate / tests (push) Has been cancelled
* chore(release): v3.3.2

* chore(release): sync uv.lock for 3.3.2

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-06-06 09:58:05 -07:00
Matt Van Horn 1bdc14878c fix(reddit): relevance-aware comment-enrichment slot selection in keyless path (#484)
* fix(reddit): relevance-aware comment-enrichment slot selection in keyless path

* docs(changelog): record relevance-aware enrichment fix under Unreleased

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-06-06 09:44:07 -07:00
Matt Van Horn 26da1e157c chore: remove dev artifacts from installer scan surface (#465)
* chore: remove dev artifacts from installer scan surface

Hermes (and other harnesses that clone raw GitHub instead of honoring
.clawhubignore) scan files that never reach an installed skill, producing
a wall of false-positive security findings. Remove the stale SKILL-original.md
backup, internal docs/plans and docs/test-results, and release-notes.md so
the scanned tree matches what actually ships.

These were already excluded from the ClawHub bundle via .clawhubignore and
from git archives via .gitattributes export-ignore. No runtime files change.

Refs #464

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: drop dangling SKILL-original.md reference in AGENTS.md

Greptile-flagged: the deletion left a 'kept for reference only' pointer to
the removed file. Refs #465

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 07:41:30 -07:00
Matt Van Horn 4aae93ee5d fix: remove duplicate /last30days command wrapper (#461) (#462)
* fix: remove duplicate command wrapper so plugin exposes only the skill (#461)

The plugin shipped both commands/last30days.md and the skill under the
same name, so /last30 surfaced two `last30days` entries with two
different descriptions. Remove the wrapper; the skill already carries
its own argument-hint, so the /last30days <topic> picker UX is unchanged.

Also corrects the README install note that claimed Claude Code dedupes
the slash command across install methods (it does not), and bumps
3.3.0 -> 3.3.1 across plugin.json, marketplace.json, gemini-extension.json,
and SKILL.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: bump pyproject.toml version to 3.3.1 (manifest contract)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update uv.lock for 3.3.1 version bump

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-30 00:33:16 -05:00
Matt Van Horn 8d3a9e4368 fix(reddit): restore free path via keyless RSS + shreddit scrape (.json is dead) (#457)
* test(reddit): add live RSS + shreddit comment fixtures

Captured from reddit.com on 2026-05-29 (search.rss listing + the
/svc/shreddit/comments partial), trimmed to a representative subset plus
two synthetic edge cases (deleted author, negative score) for offline
parser tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(http): add keyless get_text helper

Browser-UA text fetch for RSS/HTML endpoints; returns None on any HTTP or
network failure so tiered callers fall through cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless RSS discovery (search.rss + listing feeds)

Replaces the now-403 search.json with keyless Atom feeds, normalized to the
existing reddit_public post shape. Scores are placeholder zeros, backfilled
during shreddit enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): keyless shreddit comment scraper

Parses <shreddit-comment> elements from /svc/shreddit/comments/r/{sub}/t3_{id}
(score/author/created/permalink + thingId-anchored body) into top comments,
matching reddit_enrich output. Replaces the dead {thread}.json enrichment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): tiered keyless orchestrator

Tier 0 one-shot .json (residential bonus) -> Tier 1 RSS discovery ->
Tier 2 shreddit enrichment. Returns [] never raises, so the SC backup
still engages when every keyless tier is empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): route free path through keyless pipeline (.json is dead)

search_reddit_public is now a thin shim over reddit_keyless, so pipeline.py
and other callers need no change. Removes the dead .json enrichment helpers;
search/_parse_posts remain as the demoted Tier 0 attempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): request sort=top so true top comments land on page 1

Guarantees the highest-scored comments are captured even on large threads,
independent of Reddit's default comment sort. Local score re-sort remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(reddit): recover post upvote scores via keyless listing partials

The shreddit community-more-posts partial server-renders each post's score
and comment count (works for normal users, not IP-gated), unlike RSS or the
comments endpoint. Use it as a scored discovery source and to backfill scores
onto RSS-discovered posts (subreddits derived from results when not provided).
Ranking now uses real upvote score.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(reddit): listings backfill scores only on bare queries, not discovery

Caught running the full pipeline on a bare topic: deriving subreddits from
noisy RSS results and merging their top/hot listings flooded results with
high-upvote off-topic posts. Now derived-subreddit listings are used only to
backfill scores onto keyword-matched RSS posts; listing cards are merged as
discovery only when the caller explicitly provides subreddits (on-topic).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 14:43:56 -05:00
Trevin Chow 1e03af19e0 Merge pull request #423 from hnshah/ren/preserve-requested-quick-sources 2026-05-22 08:14:46 -07:00
Trevin Chow f032e25e51 Merge pull request #429 from josmithiii/docs/agents-md-install-propagation 2026-05-22 08:13:01 -07:00
Trevin Chow 84a19cf44d Merge pull request #438 from iliaal/refactor/github-search-parse-split 2026-05-22 08:10:40 -07:00
Trevin Chow 861462689e Merge pull request #444 from Yong-yuan-X/fix/centralize-test-path-setup 2026-05-22 08:09:20 -07:00
Yong-yuan-X e74b0e1e93 tests: centralize script path setup in conftest.py
Add a pytest-discovered tests/conftest.py for the last30days scripts path and
remove duplicate per-file sys.path.insert boilerplate from tests.

Normalize affected imports to rely on the shared scripts path and remove the
now-unneeded E402 suppressions.
2026-05-21 00:04:03 +08:00
Ilia Alshanetsky c5c0239dc9 refactor(github): resolve token once at pipeline boundary; pad no-token envelope
Greptile review (PR #438) flagged two issues:

1. search_github and enrich_with_comments both call _resolve_token,
   so when GITHUB_TOKEN is absent from config and env the gh-CLI
   subprocess (with its 5s timeout) fires twice per query.

2. The no-token early-return envelope `{"items": [], "error": "no token"}`
   was missing the `context` key that every other failure path includes,
   making the envelope shape inconsistent between the no-token and
   fetch-failure cases.

Fix 1: add public github.resolve_token(token) wrapping the existing
_resolve_token. Pipeline calls it once before search and enrich, so
both downstream calls receive an already-resolved (or already-None)
token and skip the fallback chain.

Fix 2: thread core/from_date/to_date/count through the no-token
envelope's `context` key, matching the fetch-failure envelope shape.
parse_github_response was already tolerant of the missing key, but
diagnostics callers that read response["context"]["..."] now get a
consistent dict in both error paths.

Reviewer's suggested code patch for issue 1 was a no-op (it kept the
same _resolve_token(token) call inside enrich_with_comments); the
underlying intent — resolve at the boundary — is what this commit
implements.
2026-05-19 12:35:40 -04:00
Ilia Alshanetsky 269dda9f6c refactor(github): split search_github / parse_github_response / enrich_with_comments
search_github returned a normalized List[dict] directly while every
other adapter follows search_X -> dict envelope, parse_X_response ->
list[dict]. The github branch in pipeline._retrieve_stream was the
only one that called search_* and returned (result, {}) without a
parse step. This blocked fixture-driven testing: there was no parse
function to feed a synthetic envelope to.

Split into three:

  search_github(...) -> Dict[str, Any]
    HTTP fetch only. Returns {"items": [raw items], "context": {core,
    from_date, to_date, count}}.

  parse_github_response(response) -> List[Dict[str, Any]]
    Pure function. Normalizes, date-filters, sorts by relevance.

  enrich_with_comments(items, depth, token) -> List[Dict[str, Any]]
    Public extraction of the old private _enrich_top_items. Resolves
    the token via env / gh CLI fallback so callers don't have to.

Pipeline now does the standard 3-call dance:

  response = github.search_github(...)
  items = github.parse_github_response(response)
  items = github.enrich_with_comments(items, depth=depth, token=token)

Keeping enrich_with_comments in parse_github_response would make parse
impure and force every fixture-driven test to either mock HTTP or
skip enrichment. Splitting it out matches the YouTube adapter's
pattern.
2026-05-19 12:18:47 -04:00
Julius Smith a35677da77 docs(agents): address Greptile review (stale Commands comment, duplicate Structure entry)
- Commands block's inline comment on `npx skills add` still said
  "symlink this repo into every detected harness's skill dir" — the
  exact misconception the PR set out to correct. Rewrite to describe
  the frozen-copy behavior and point at the Rules section for the
  full explanation.
- Structure section had SKILL.md listed twice (the original line 6
  entry plus a new line 13 entry added in this PR). Fold the
  SKILL-original.md context into line 6 and drop the duplicate.
2026-05-18 11:02:53 -07:00
Julius Smith a78ab69ffe docs(agents): correct install-propagation claim and fill in build/test gaps
AGENTS.md said "edits in the working tree propagate live to every harness"
after `npx skills add . -g -y`, but the install actually drops a real
(frozen-at-install-time) copy at ~/.agents/skills/<name>/ and per-host
symlinks point at *that copy*, not at the working tree. Clarify the
mechanism and offer two ways forward: re-run `npx skills add` to sync,
or replace the install copy with a working-tree symlink for live-edit.

Also fill in two gaps a fresh agent hits on entry:
- `uv run pytest` commands for the ~89-file test suite (no test runner
  was documented before)
- Python 3.12+ / `uv` / `.venv/` convention
- Brief doc map: CONFIGURATION.md, SKILL.md vs SKILL-original.md,
  CHANGELOG.md / release-notes.md, HERMES_SETUP.md
2026-05-18 07:45:27 -07:00
Hiten Shah 0bb01c2d6a test: cover requested sources in fallback quick plans 2026-05-18 07:43:10 -07:00
Hiten Shah 444e07d141 fix: preserve requested sources in quick plans 2026-05-17 15:54:45 -07:00
Hiten Shah 850c7e0185 chore: sync release manifest versions 2026-05-17 15:51:12 -07:00
Trevin Chow d53121f035 Merge pull request #420 from hnshah/ren/watchlist-delta 2026-05-17 10:35:41 -07:00
Hiten Shah 2502a19d46 fix(watchlist): clarify delta URL identity 2026-05-17 09:09:33 -07:00
Hiten Shah 0f280245ac feat(watchlist): show deltas between topic runs 2026-05-17 09:01:09 -07:00
131 changed files with 2298 additions and 4498 deletions
+1 -1
View File
@@ -11,7 +11,7 @@
{ {
"name": "last30days", "name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.3.0", "version": "3.3.2",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
"url": "https://github.com/mvanhorn" "url": "https://github.com/mvanhorn"
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "last30days", "name": "last30days",
"version": "3.3.0", "version": "3.3.2",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.", "description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": { "author": {
"name": "Matt Van Horn", "name": "Matt Van Horn",
-2
View File
@@ -23,13 +23,11 @@ assets/ export-ignore
# claude.ai-bundle-specific exclusions live in scripts/build-skill.sh. # claude.ai-bundle-specific exclusions live in scripts/build-skill.sh.
# Historical + repo-only manifests # Historical + repo-only manifests
SKILL-original.md export-ignore
SPEC.md export-ignore SPEC.md export-ignore
TASKS.md export-ignore TASKS.md export-ignore
test-run.log export-ignore test-run.log export-ignore
CONTRIBUTORS.md export-ignore CONTRIBUTORS.md export-ignore
HERMES_SETUP.md export-ignore HERMES_SETUP.md export-ignore
release-notes.md export-ignore
CHANGELOG.md export-ignore CHANGELOG.md export-ignore
uv.lock export-ignore uv.lock export-ignore
+15 -3
View File
@@ -3,12 +3,15 @@
Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation. Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
## Structure ## Structure
- `skills/last30days/SKILL.md` — canonical skill definition - `skills/last30days/SKILL.md` — canonical skill definition / runtime spec the model reads when the slash command fires
- `skills/last30days/scripts/last30days.py` — main research engine - `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules - `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client - `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
- `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`) - `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
- `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology - `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
- `CONFIGURATION.md` — user-facing knobs (env vars, flags, per-host install patterns); keep in sync per the rules below
- `CHANGELOG.md` — structured release history (launch copy lives in GitHub Releases)
- `HERMES_SETUP.md` — install instructions for the Hermes harness specifically
## Orientation ## Orientation
- This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into. - This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
@@ -20,11 +23,20 @@ Agent Skills package for researching any topic across Reddit, X, YouTube, and we
```bash ```bash
# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only) # Dev/fallback: direct engine invocation (scripting, cron, or engine testing only)
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # one-time: symlink this repo into every detected harness's skill dir npx skills add . -g -y # copies skill into ~/.agents/skills/<name>/ (frozen at install time); re-run to sync working-tree edits — see Rules below
# Tests (pytest, ~89 files under tests/, configured in pyproject.toml)
uv run pytest # full suite
uv run pytest tests/test_dedupe_v3.py # single file
uv run pytest tests/test_dedupe_v3.py -k some_case # single case
uv run pytest --cov # with coverage (skips lib/vendor/)
```
Python 3.12+ required. Use `uv` for the env; the venv lives at `.venv/`.
## Rules ## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports) - `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` creates symlinks from each detected harness's skill dir to this repo. Edits in the working tree propagate live to every harness — no re-deploy step needed. - One-time setup: `npx skills add . -g -y` copies the skill into `~/.agents/skills/<name>/` (real directory) and, for harnesses that support symlinked skill dirs, drops a per-host symlink pointing at that copy. **Working-tree edits do NOT propagate automatically** — the `~/.agents/skills/<name>/` copy is frozen at install time. To sync after edits, re-run `npx skills add . -g -y`. For live-edit on a dev machine, replace the install copy with a symlink to the working tree: `ln -sfn "$PWD/skills/last30days" ~/.agents/skills/last30days` (run from the repo root).
- Git remote: origin = public (`mvanhorn/last30days-skill`) - Git remote: origin = public (`mvanhorn/last30days-skill`)
## Security hygiene ## Security hygiene
+13
View File
@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [3.3.2] - 2026-06-06
### Fixed
- Keyless Reddit comment enrichment now spends its limited slots on entity-matching posts first (mirroring rerank's entity-miss demotion signal) instead of raw upvote order, so off-topic high-upvote threads from broad subreddits no longer consume the comment budget only to be demoted afterward ([#484](https://github.com/mvanhorn/last30days-skill/pull/484))
## [3.3.1] - 2026-05-30
### Fixed
- Removed the redundant `commands/last30days.md` wrapper so the plugin exposes only the skill ([#461](https://github.com/mvanhorn/last30days-skill/issues/461)). Previously the plugin shipped both a command wrapper and the skill under the same name, so `/last30` surfaced two `last30days` entries with two different descriptions. The skill already carries its own `argument-hint`, so the `/last30days <topic>` picker UX is unchanged.
- Corrected the README install note that claimed Claude Code dedupes the slash command across install methods; it does not, so having both the marketplace plugin and the `npx skills` copy active shows two entries.
## [3.3.0] - 2026-05-17 ## [3.3.0] - 2026-05-17
A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner. A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner.
+1 -1
View File
@@ -189,7 +189,7 @@ If you'd rather use the agent-skills install path on Claude Code, that's also su
npx skills add mvanhorn/last30days-skill -g -a claude-code npx skills add mvanhorn/last30days-skill -g -a claude-code
``` ```
The native plugin and the `npx skills` install can coexist; Claude Code dedupes the slash command. The native plugin and the `npx skills` install can coexist. Note that Claude Code does not dedupe across install methods: if you have both the marketplace plugin and the `npx skills` copy active, `/last30days` will show two entries. Use one install method per machine.
### Codex, Cursor, Copilot, Gemini CLI, and other Agent Skills hosts ### Codex, Cursor, Copilot, Gemini CLI, and other Agent Skills hosts
-391
View File
@@ -1,391 +0,0 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
context: fork
agent: Explore
disable-model-invocation: true
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
-9
View File
@@ -1,9 +0,0 @@
---
description: Research what people actually say about any topic in the last 30 days across Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web.
argument-hint: <topic> — e.g. "nvidia earnings reaction" or "best noise cancelling headphones"
allowed-tools: [Bash, Read, Write, AskUserQuestion, WebSearch]
---
Invoke the `last30days` skill with the user's arguments: $ARGUMENTS
Use the skill's canonical pipeline (plan → retrieve → normalize → fuse → rerank → cluster → render). If the user provided no arguments, ask them for a topic before proceeding.
@@ -1,306 +0,0 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: --competitors flag for auto-discovered comparison fan-out"
type: feat
status: active
date: 2026-04-22
---
# feat: --competitors flag for auto-discovered comparison fan-out
## Overview
Add a `--competitors` flag to the last30days engine that auto-discovers 2-4 peer entities for the topic, runs the full retrieval pipeline on each in parallel, and renders a multi-entity comparison. Invoking `last30days Kanye West --competitors` should resolve to "Kanye vs Drake vs Kendrick Lamar" and emit a comparison report covering all three. Invoking `last30days OpenAI --competitors` should resolve to "OpenAI vs Anthropic vs xAI vs Gemini" and emit a four-way comparison.
Discovery mirrors the existing `resolve.auto_resolve()` pattern used for X handles and subreddits at pipeline start — web search (Brave / Exa / Serper) plus deterministic extraction. Not an internal LLM call.
## Problem Frame
Users who want a comparison today must type "OpenAI vs Anthropic vs xAI" themselves. The `planner._comparison_entities()` path already handles explicit multi-entity topics and `render._render_comparison_scaffold()` already emits a 9-axis comparison table. What is missing is the discovery half — a user who types a single entity with `--competitors` should get the comparison for free.
This is also the natural next step after the Step 0.55 category-peer subreddit work (PR #305, merged 2026-04-22). That feature widens the subreddit set within a single topic; this feature widens the entity set into peer entities.
## Requirements Trace
- R1. New `--competitors` boolean flag that triggers competitor discovery and multi-entity fan-out.
- R2. New `--competitors-list="A,B,C"` to explicitly skip discovery (mirrors `--plan`, `--subreddits`, `--x-handle` overrides).
- R3. New `--competitors=N` short form to set competitor count inline (N in 1..6).
- R4. Default count is 3 competitors (original + 3 = 4-way comparison).
- R5. Competitor retrieval depth inherits the main run's depth (`--quick` / `--deep`); all entities run in parallel so wall clock stays close to a single run.
- R6. Discovery mirrors `resolve.auto_resolve()`: web search for peers, deterministic text extraction. No internal LLM dependency.
- R7. If no web search backend is configured and no `--competitors-list` was passed, engine emits a LAW 7-style stderr telling the host agent to pass `--competitors-list` and exits non-zero.
- R8. Output rendering is a single comparison report covering all entities, reusing the existing 9-axis scaffold from `render._render_comparison_scaffold()` where applicable.
## Scope Boundaries
- Synthesis prompt changes beyond wiring N reports into the existing comparison scaffold are out of scope.
- `--competitors` does not replace the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`; both paths coexist.
- No caching layer for discovery results in v1.
- No UI/SKILL.md rewrite of the entire comparison section; only the new flag is documented.
- No new web search backend.
### Deferred to Separate Tasks
- Caching of competitor lookups: separate follow-up once hit rate justifies it.
- Disambiguation UX for topics with multiple common entities ("Amazon" the company vs the river): separate brainstorm.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:168-249``build_parser()` argparse definitions. Existing depth flags (`--quick`, `--deep`) and override flags (`--plan`, `--subreddits`, `--x-handle`, `--auto-resolve`) set the convention to mirror.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the reference pattern: web search fan-out via `ThreadPoolExecutor`, per-query extraction functions, graceful empty-dict return when no backend is available.
- `scripts/lib/resolve.py:98-140``_extract_x_handle()` and sibling extractors show the deterministic text-mining style competitor extraction should mirror.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature is the fan-out target. One call per entity, each returning a `schema.Report`.
- `scripts/lib/planner.py:430-564` — Existing comparison-intent handling and `_comparison_entities()` entity extraction. The new flag feeds the same mental model but populates entities from discovery instead of from the topic string.
- `scripts/lib/render.py:333-392``_render_comparison_scaffold()` already emits a 9-axis markdown comparison table. The new multi-report renderer should reuse this helper by assembling a synthetic "A vs B vs C" topic header for it.
- `scripts/lib/grounding.py` + `scripts/lib/providers.py` — Web search backend resolution (Brave / Exa / Serper). Reused as-is.
### Institutional Learnings
- No existing `docs/solutions/` entries for competitor discovery or multi-entity fan-out.
- Recent plan `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md` established the precedent of deterministic peer expansion; this plan extends that idea from subreddits to entities.
### External References
- None gathered — local patterns are strong. `resolve.auto_resolve()` is a direct template.
## Key Technical Decisions
- **Discovery mirrors auto_resolve, not plan_query.** Web search + regex extraction, not an LLM call. Matches the user's explicit direction ("use the python brain the same way it searches for X handles"). Cheaper, no provider credential requirement, deterministic.
- **Orchestration lives in `last30days.py` main, not inside `pipeline.run()`.** The fan-out is a top-level concern — one pipeline run per entity, each independent. Keeps `pipeline.run()` single-entity and unchanged except for sharing a `ThreadPoolExecutor` factory.
- **Sub-runs inherit main depth and run in parallel.** Wall clock ≈ single run; token cost scales linearly with N. User-controlled via the existing `--quick`/`--deep` flags.
- **New module `scripts/lib/competitors.py` instead of adding to `resolve.py`.** Keeps resolve focused on single-entity entity-bundle discovery (handles/subreddits/github); competitors.py owns peer-entity discovery. Similar shape, different responsibility.
- **Multi-report render is additive in `render.py`.** New `render_comparison_multi(reports: list[Report]) -> str` composes a synthetic "A vs B vs C" topic and delegates to the existing scaffold + synthesis path where possible. No rewrite of the single-entity render path.
- **Default count = 3 competitors (4-way comparison).** Hard cap at 6.
- **LAW 7-style stderr when no backend and no list.** Matches how `planner.plan_query()` already tells the hosting agent to pass `--plan`.
## Open Questions
### Resolved During Planning
- **Discovery mechanism:** Web search via `grounding.web_search()`, not an internal LLM. User confirmed the auto_resolve pattern is the target.
- **Default competitor count:** 3 (original + 3 = 4-way).
- **Sub-run depth:** Inherit main depth, parallel execution.
- **Flag naming:** `--competitors` (standard argparse double-dash). `--competitors=N` for inline count. `--competitors-list="A,B,C"` to skip discovery.
### Deferred to Implementation
- Exact extraction heuristics for competitor names across Brave / Exa / Serper result shapes. The SERP text varies (listicles, comparison pages, "vs" pages); the initial implementation will start with listicle parsing plus a "X vs Y" pattern match, and harden against real results in the test phase.
- Handling of topic ambiguity ("Amazon", "Apple"). Initial behavior: trust whatever web search returns for the topic verbatim; disambiguation is a separate concern.
- Merge strategy when two entities return overlapping URLs (e.g., an "OpenAI vs Anthropic" article shows up in both runs). Likely dedupe at the clustering step, but defer the exact policy until we see how often it happens.
- Whether to expose competitor discovery artifacts (the raw web search results) as a debug emit. Follow the existing `--debug` conventions.
## Implementation Units
- [ ] **Unit 1: CLI flag parsing and validation**
**Goal:** Add `--competitors`, `--competitors=N`, and `--competitors-list` to the argparse surface, validate values, and thread them into the main orchestration.
**Requirements:** R1, R2, R3, R4
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py`
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Add three mutually cooperative flags near line 205 in `build_parser()`:
- `--competitors` with `nargs="?"` and `const=3` so bare `--competitors` defaults to 3, `--competitors=4` is honored, and `--competitors=0` is rejected
- `--competitors-list` free-text CSV
- Normalize in `main()`: if `--competitors-list` is present, skip discovery and use the list. If `--competitors` is set and no list, trigger discovery with count = the flag value. Clamp count to 1..6 with a stderr warning at boundary.
- Thread the resulting entity list into the orchestrator added in Unit 3.
**Patterns to follow:**
- `--plan` argument at `scripts/last30days.py:187` — same skip-discovery-when-explicit shape.
- `--subreddits` / `--x-handle` at `scripts/last30days.py:180,189` — same override semantics.
**Test scenarios:**
- Happy path: bare `--competitors` parses to count=3, empty list.
- Happy path: `--competitors=4` parses to count=4.
- Happy path: `--competitors-list="A,B,C"` parses to count=3, list=["A","B","C"], and is preferred over any discovery signal.
- Edge case: `--competitors=0` and `--competitors=-1` are rejected with a clear error.
- Edge case: `--competitors=99` clamps to 6 with a stderr warning.
- Edge case: `--competitors` combined with `--competitors-list` uses the list and logs that discovery was skipped.
- Edge case: `--competitors-list` value with whitespace ("A, B , C") normalizes correctly.
**Verification:**
- Running the binary with each flag variation produces the expected post-parse state without calling out to the network.
- [ ] **Unit 2: `scripts/lib/competitors.py` discovery module**
**Goal:** Discover peer entities for a topic using web search + deterministic extraction, mirroring `resolve.auto_resolve()`.
**Requirements:** R6, R7
**Dependencies:** None (pure module; wired by Unit 3)
**Files:**
- Create: `scripts/lib/competitors.py`
- Test: `tests/test_competitors.py`
**Approach:**
- Public entry point `discover_competitors(topic: str, count: int, config: dict) -> list[str]`.
- Early return `[]` when `_has_backend(config)` is false (reuse the helper from `resolve.py`; factor if needed).
- Fan out 2-3 web searches in a `ThreadPoolExecutor`:
- `"{topic} competitors"`
- `"{topic} alternatives"`
- `"{topic} vs"` (captures "X vs Y" articles)
- Feed results into a deterministic `_extract_peer_entities(results, topic)` that:
- Mines titles and snippets for capitalized noun phrases other than the topic itself
- Scores by frequency across results
- Filters stopwords and the topic's own tokens
- Returns top `count` unique entities ordered by score
- Emit a single-line stderr log mirroring the `resolve._log` format.
**Patterns to follow:**
- `scripts/lib/resolve.py:179-258` for the function shape, executor usage, and empty-result fallback.
- `scripts/lib/resolve.py:98-140` for extractor style (small, deterministic, no external state).
**Test scenarios:**
- Happy path: canned SERP fixtures for "OpenAI" return ["Anthropic", "xAI", "Google"] or close peers in the top 3.
- Happy path: canned SERP fixtures for "Kanye West" return rap peers (Drake, Kendrick) in the top 3.
- Edge case: empty SERP results return `[]` without raising.
- Edge case: extractor filters out the topic itself (case- and punctuation-insensitive).
- Edge case: near-duplicate entities ("OpenAI" vs "Open AI") dedupe to one slot.
- Error path: web search backend raises — the failure is logged and the function returns `[]`.
- Edge case: count=1 returns a single-element list; count=6 returns up to six entities.
**Verification:**
- Unit tests pass with fixtures committed under `tests/fixtures/competitors-*.json`.
- Manual run against a live backend for one topic confirms sensible output (recorded as a notes file, not a test assertion).
- [ ] **Unit 3: Parallel fan-out orchestrator**
**Goal:** Run `pipeline.run()` once per entity (topic + discovered competitors) in parallel, collect `schema.Report` per entity, and hand them to the comparison renderer.
**Requirements:** R5, R7
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py`
- Possibly create: `scripts/lib/fanout.py` if the orchestrator grows past ~60 lines
- Test: `tests/test_competitor_fanout.py`
**Approach:**
- After arg parsing and before the existing `pipeline.run()` call, branch on `args.competitors`:
- If a list was provided or discovery returned entities, build `entities = [topic, *competitors]`.
- Spawn one `pipeline.run()` per entity via `ThreadPoolExecutor(max_workers=len(entities))`, passing the same `config`, `depth`, and all sub-run-relevant args (mock, plan, etc.). Respect `--plan` — if a plan is passed it applies to the main topic only; competitors use the internal planner fallback for v1.
- Collect `{entity: Report}` mapping. A per-entity failure logs a stderr warning and drops that entity from the comparison; the run continues as long as 2 entities succeed.
- If fewer than 2 entities survive, exit with a clear error.
- LAW 7-style stderr:
- If `args.competitors` is set, no list was passed, no web search backend is configured, emit a LAW 7 stderr message pointing to the `--competitors-list` override and exit non-zero. Reuse the tone from `planner.plan_query()` fallback (`scripts/lib/planner.py:125-135`).
**Execution note:** Start with a failing integration test that exercises the full main → orchestrator → mocked pipeline.run path; the orchestrator is where bugs hide.
**Patterns to follow:**
- `scripts/lib/resolve.py:225-239` for ThreadPoolExecutor + as_completed + per-future error handling.
- `scripts/lib/pipeline.py:310+` for how ThreadPoolExecutor is already used inside a single run (same idiom, outer layer).
**Test scenarios:**
- Happy path: main + 2 competitors, all three `pipeline.run()` calls succeed (mocked), orchestrator returns 3 Reports.
- Happy path: discovery returns the competitor list; orchestrator fans out accordingly.
- Edge case: one of three competitor pipelines raises — the run continues with the surviving 2 and emits a warning.
- Edge case: all competitors fail but the main topic succeeds — orchestrator exits non-zero with a clear error rather than silently degrading to a single-entity render.
- Edge case: `--competitors` set, no backend, no list — orchestrator emits the LAW 7 stderr and exits non-zero before any pipeline call.
- Integration: wall-clock time for 3 mocked pipelines in parallel is close to the slowest single run, not the sum (timing assertion with generous margin).
**Verification:**
- End-to-end test with mocked `pipeline.run()` and mocked competitors discovery produces 3 Reports and hands them to a stubbed renderer.
- [ ] **Unit 4: Multi-report comparison renderer**
**Goal:** Compose N `schema.Report`s into a single comparison-mode output, reusing the existing 9-axis scaffold.
**Requirements:** R8
**Dependencies:** Unit 3
**Files:**
- Modify: `scripts/lib/render.py`
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- Add `render_comparison_multi(reports: list[schema.Report], *, emit: str) -> str`.
- Build a synthetic comparison topic: `f"{entity_a} vs {entity_b} vs {entity_c}"`.
- Reuse `_render_comparison_scaffold()` for the table skeleton. Each entity column is populated from its own Report's top clusters and citations.
- For the narrative synthesis block, concatenate per-entity highlights, clearly labeled by entity, under a shared "Comparison" header.
- Preserve existing emit modes (`compact`, `md`, `json`, `context`). In `json` emit, return a `{"entities": [...], "reports": [...]}` shape; single-Report consumers remain unaffected because the single-report render path is untouched.
**Patterns to follow:**
- `scripts/lib/render.py:333-392` (`_parse_comparison_entities`, `_render_comparison_scaffold`) — the scaffold is the contract.
- `scripts/lib/render.py` single-report rendering — for per-entity narrative blocks.
**Test scenarios:**
- Happy path: 3 Reports with distinct clusters render into a 3-column table and a "Comparison" section that mentions each entity at least once.
- Happy path: 2 Reports render as a 2-column table without breaking the scaffold.
- Edge case: a Report with an empty cluster list renders as "(no significant discussion this month)" in its column rather than crashing.
- Edge case: Reports with overlapping URLs (same article cited by two entities) dedupe citations at the footer but keep both column entries.
- Emit variants: `--emit=compact`, `--emit=md`, `--emit=json`, `--emit=context` each produce valid output with all entities represented.
- Integration: end-to-end snapshot test using fixture Reports, checked against a stored expected output (with a clear update path when the scaffold intentionally evolves).
**Verification:**
- Snapshot tests pass. Manual review of one real 3-way comparison confirms readability.
- [ ] **Unit 5: Docs, SKILL.md mention, and sync**
**Goal:** Document the new flag so the hosting agent and human users both know it exists, and run the sync script.
**Requirements:** R1-R8 (surfaces them to users)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md`
- Modify: `README.md` (brief flag reference)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- Add a compact "Competitor mode" subsection under the existing comparison docs in `SKILL.md`. Document the flag, the default count, the override flag, and the LAW 7 fallback stderr.
- Keep `README.md` addition to a single example line.
- CHANGELOG entry mirrors the voice of recent entries (imperative, outcome-first).
- Sync via `scripts/sync.sh` per CLAUDE.md rules so `~/.claude/`, `~/.agents/`, `~/.codex/` pick up the new SKILL.md.
**Test scenarios:**
- Test expectation: none — documentation and sync only. Verification is by inspection and by running `sync.sh` and confirming target directories updated.
**Verification:**
- `sync.sh` completes without errors.
- `SKILL.md` rendered preview mentions `--competitors` in the comparison section.
## System-Wide Impact
- **Interaction graph:** `last30days.py main()` now orchestrates multiple `pipeline.run()` calls instead of one. No other callers of `pipeline.run()` are affected (it remains single-entity).
- **Error propagation:** Per-entity failures degrade gracefully as long as ≥2 entities survive; fewer survivors exits non-zero. Discovery failure with `--competitors` and no list is fatal.
- **State lifecycle risks:** Each sub-run uses its own `pipeline.run()` state; no shared mutable config. The `config` dict is read-only in `pipeline.run()` today — verify before committing to shared-reference passing, else deep-copy per sub-run.
- **API surface parity:** `--competitors` coexists with the existing explicit "A vs B vs C" topic parsing in `planner._comparison_entities()`. Both produce comparable output formats; the only difference is where the entity list came from.
- **Integration coverage:** The fan-out orchestrator crosses CLI → discovery → N pipelines → render; integration tests in Unit 3 and Unit 4 must exercise the full path end to end, not just unit-level.
- **Unchanged invariants:** `pipeline.run()` signature and single-entity semantics are unchanged. The single-entity render path in `render.py` is unchanged. No changes to `planner.plan_query()`. No changes to existing flags.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Competitor discovery returns garbage entities for niche topics. | `--competitors-list` override lets the user (or hosting agent) correct it. Unit tests with edge-case fixtures. Log discovery output to stderr under `--debug`. |
| Token cost scales linearly with N sub-runs. | Default count capped at 3, hard max 6, inherit `--quick` to let users throttle. Wall clock stays parallel. Emit a cost hint to stderr when N ≥ 4. |
| Merge conflicts against the single-entity render path during refactoring. | Keep the multi-report renderer strictly additive; do not modify the single-Report code path. |
| Config dict mutation inside sub-runs could leak state between entities. | Verify read-only usage before sharing references. If any sub-component mutates, deep-copy per sub-run before spawning threads. |
| A SERP extractor that works on Brave fixtures breaks on Exa/Serper result shapes. | Test fixtures for all three backends. Extractor operates on a normalized shape from `grounding.web_search()` (already the case), not raw provider output. |
| Hosting agent (Claude Code, Codex) unaware of the new flag when it could usefully pass `--competitors-list`. | SKILL.md updated in Unit 5 documents the flag in the same style as `--plan` and `--auto-resolve`. |
## Documentation / Operational Notes
- Beta channel first: per `CLAUDE.md`, experimental changes go to `mvanhorn/last30days-skill-private` on the `/last30days-beta` command. Land this on the private repo first, shake out on real topics for a day or two, then cherry-pick to public.
- After land-merge: run `scripts/sync.sh` to deploy SKILL.md + scripts to `~/.claude/`, `~/.agents/`, `~/.codex/`.
- Release notes entry in CHANGELOG.md follows the v3.0.9 voice — outcome-first, one paragraph.
## Sources & References
- Related code: `scripts/lib/resolve.py:179` (`auto_resolve`), `scripts/lib/pipeline.py:162` (`pipeline.run`), `scripts/lib/planner.py:80` (`plan_query` LAW 7 fallback), `scripts/lib/render.py:333` (comparison scaffold)
- Related PRs: #305 (Step 0.55 category-peer subreddit expansion — the precedent for deterministic peer expansion, merged 2026-04-22)
- Related plan: `docs/plans/2026-04-22-001-fix-category-peer-subreddit-resolution-plan.md`
@@ -1,352 +0,0 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: per-entity resolution, default-2, and stale-path guard for --competitors"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md
---
# fix: per-entity resolution, default-2, and stale-path guard for --competitors
## Overview
Three test runs of v3.0.11 `--competitors` surfaced four real bugs plus one product tweak. This plan fixes all of them in a single follow-up:
1. Competitor sub-runs get no Step 0.55 resolution (no X handle, no subreddits, no GitHub repo). Drake / Kendrick / Travis ran with deterministic-fallback single-word queries while Kanye had the full targeting package. User called it "lazy" and was right.
2. Two of three test windows (Linear, Coinbase) never invoked the new flag at all. They loaded SKILL.md from `plugins/marketplaces/last30days-skill/` (a Claude-Code-managed git clone pinned to origin/main, which predates PR #308) instead of `plugins/cache/last30days-skill/last30days/3.0.11/`, so `--help` showed no `--competitors` flag and the model fell back to the manual comparison path.
3. Each competitor sub-run emits a scary `[Planner] No --plan passed... deterministic fallback` stderr line because LAW 7 targets the hosting-model path, not internal fan-out sub-runs.
4. Default competitor count is 3 (→ 4-way comparison). User wants default 2 (→ 3-way: original + 2 peers). Flag keeps `--competitors=N` to customize.
## Problem Frame
The 3 test runs (Kanye, Linear, Coinbase) showed a pattern:
| Window | Loaded SKILL.md from | Invoked --competitors? | Per-entity resolution? | Outcome |
|--------|----------------------|-----------------------|------------------------|---------|
| Kanye | cache/3.0.11/ (correct) | Yes | Only for main topic (Kanye) | Drake/Kendrick/Travis thin; Reddit 403 fallbacks |
| Linear | marketplaces/ (stale) | No — fell back to manual comparison | No | Thin run with noisy subreddits |
| Coinbase | marketplaces/ (stale) | No — fell back to manual comparison | Main only; keyword-search poisoned pool | Top subs: r/survivor, r/Airpodsmax (noise) |
Root causes:
- **Per-entity resolution gap:** `scripts/lib/fanout.py` calls `pipeline.run()` with topic + depth + web_backend + lookback_days only. It does not call `resolve.auto_resolve()` per entity, so sub-runs have no X handle, subreddit, or GitHub targeting. The original plan (`2026-04-22-002`) acknowledged this as a deliberate v1 simplification ("competitor sub-runs use planner defaults"). In practice this produces visibly asymmetric output and triggers downstream retrieval issues (403 fallbacks, keyword-search noise).
- **Stale-path loading:** Claude Code's skill loader alphabetizes `find` results with `marketplaces/` before `cache/`, and the model reads the first plausible SKILL.md it sees. SKILL.md line 823's `SKILL_ROOT` resolver is the correct path but only fires in engine-invocation blocks, not in the skill-load step.
- **LAW 7 in sub-runs:** LAW 7 exists because the *hosting reasoning model* is supposed to pass `--plan`. For competitor sub-runs, there is no hosting-model planning — it's an engine-internal fan-out. The warning is a false positive there.
## Requirements Trace
- R1. Default `--competitors` count is 2 peers (3-way comparison: original + 2).
- R2. Each competitor sub-run performs Step 0.55 resolution (X handle, subreddits, GitHub user/repos, news context) before its pipeline runs — not just the main topic.
- R3. Sub-runs do not emit the LAW 7 `No --plan passed` warning; they are internal fan-out, not hosting-model calls.
- R4. The rendered comparison output includes a visible "Resolved entities" block showing per-entity handles/subs/github for debug transparency (answers "did it resolve everyone?" without the user having to read stderr).
- R5. SKILL.md has a canonical-path self-check at the top: if the reader loaded it from anywhere other than `plugins/cache/last30days-skill/last30days/{VERSION}/`, re-read from the versioned path before proceeding.
- R6. Version bumps to 3.0.12; CHANGELOG entry; `scripts/sync.sh` deploys.
## Scope Boundaries
- No new discovery strategy. The web-search + regex extraction in `scripts/lib/competitors.py` stays as-is.
- No new CLI flags beyond the behavior changes above. Specifically: no per-entity override flags like `--competitor-handles`. The hosting-model escape hatch remains `--competitors-list`.
- No changes to the explicit `A vs B` comparison path (topic-string parsing in `planner._comparison_entities`).
- No marketplace-clone auto-restore fix — that's Claude Code harness behavior. This plan only guards against the symptom on the skill side.
### Deferred to Separate Tasks
- Caching of per-entity resolution results: separate follow-up once hit rate justifies it.
- Fan-out rate-limiting tuning (currently `max_workers=len(entities)+1`, capped at 6): defer until we see real-world quota exhaustion.
- Pre-flight cost hint when N ≥ 4 (noted in `2026-04-22-002` risks): defer.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py:205-219``--competitors` / `--competitors-list` argparse definition (const=3 today; changing to 2).
- `scripts/last30days.py:220-290``resolve_competitors_args()` validator; update `COMPETITORS_DEFAULT`.
- `scripts/last30days.py:438-520` — main() fan-out orchestration; currently passes only topic/depth to each `_competitor_runner`.
- `scripts/lib/fanout.py:40-95``run_competitor_fanout()` signature. The `competitor_runner` callable is where per-entity resolution needs to happen.
- `scripts/lib/resolve.py:179-258``auto_resolve()` is the exact per-entity resolver to reuse. Already does X handle + subreddits + GitHub user/repos + news context in parallel via ThreadPoolExecutor.
- `scripts/lib/planner.py:80-135``plan_query()` emits the LAW 7 stderr. A `quiet: bool` keyword or `internal_subrun: bool` flag will suppress it.
- `scripts/lib/pipeline.py:162-220``pipeline.run()` signature. Needs a new keyword to propagate quiet-mode down to the planner.
- `scripts/lib/render.py:render_comparison_multi` — where the "Resolved entities" block is inserted.
- `SKILL.md` line 823 — canonical `SKILL_ROOT` resolver already exists but fires in engine bash, not at skill-load time.
### Institutional Learnings
- `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md` acknowledged the per-entity-resolution gap as a v1 tradeoff. This plan closes that gap.
- Kanye run stderr: `[Planner] No --plan passed... deterministic fallback` × 3 (once per competitor sub-run). That's the LAW 7 noise R3 targets.
- Linear / Coinbase runs loaded `plugins/marketplaces/last30days-skill/CLAUDE.md` as the first hit. That's the stale-path issue R5 targets.
### External References
- None. All patterns are in-repo.
## Key Technical Decisions
- **Per-entity resolve happens inside fanout, not in SKILL.md.** The user-facing promise of `--competitors` is "one flag, engine does the work." Pushing resolution onto the hosting model creates another path-of-least-resistance trap (model skips it, output looks lazy). Auto-resolve inside each sub-run when a web backend is available makes the feature self-contained.
- **Stale-path guard is a SKILL.md self-check, not a code change.** We cannot stop Claude Code from auto-restoring the marketplace clone. But we can put a 3-line banner at the top of SKILL.md that forces any path-mismatched read to re-read from the versioned cache. Both the marketplace copy (once main catches up) and the cache copy carry the guard.
- **LAW 7 suppression is opt-in via `internal_subrun=True` keyword.** Do not remove the warning from the default path — it's load-bearing for the hosting-model contract. Add an explicit bypass for engine-internal fan-out only.
- **Default 2, hard max 6 unchanged.** "Original + 2" matches the Kanye/Drake/Kendrick mental model from the feature description. Still allow `--competitors=N` from 1 to 6.
- **Resolved block is inside the EVIDENCE envelope, not above it.** Keeps the rendered output structure stable for the synthesis contract (LAW 18). The block is context, not output.
- **Skip auto-resolve when `--mock` or no web backend.** Mirrors the existing `resolve.auto_resolve()` fast-fail and keeps the mock test path deterministic.
## Open Questions
### Resolved During Planning
- **Where does per-entity resolve live?** Inside `fanout.run_competitor_fanout`, not in `main()`. Each sub-run calls `auto_resolve()` just before `pipeline.run()`.
- **Should the hosting model still be able to override?** Yes — `--competitors-list` remains the escape hatch. When an explicit list is passed, the engine still does auto-resolve per entity; the user's list just skips discovery.
- **Should sub-runs run auto-resolve in parallel with each other?** Yes. The existing `ThreadPoolExecutor` in fanout already parallelizes sub-runs; auto-resolve happens inside each sub-run's thread, so resolve calls for different entities run concurrently.
- **Default count:** 2 peers (3-way). Confirmed.
### Deferred to Implementation
- Whether to expose a `--no-auto-resolve-competitors` flag for power users who want the fast, shallow behavior. Probably not needed v2; ship auto-resolve always-on and revisit if someone complains about cost.
- Whether to surface the per-entity resolution context back into the main topic's planner (cross-entity context sharing). Stays deferred.
- Whether the Resolved block should be collapsible or always inline. Start inline; revisit based on output length feedback.
## Implementation Units
- [ ] **Unit 1: Default `--competitors` to 2 peers**
**Goal:** Change the bare `--competitors` default from 3 to 2 per user feedback. `--competitors=N` still overrides; range 1..6 unchanged.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (`COMPETITORS_DEFAULT`, `--competitors` const, stderr messages if any reference 3)
- Modify: `SKILL.md` Competitor mode section ("discovered 2-6" wording, bare-flag default line)
- Modify: `README.md` auto-discovered example line (if it references count)
- Test: `tests/test_cli_competitors.py`
**Approach:**
- Change `COMPETITORS_DEFAULT = 3``2` in `scripts/last30days.py`.
- Change argparse `--competitors` `const=3``const=2`.
- Update any SKILL.md / README copy referencing "3 peers" to "2 peers" (default) or "2-6 peers" (range).
**Patterns to follow:**
- Existing default constants in `scripts/last30days.py` argparse block.
**Test scenarios:**
- Happy path: bare `--competitors` yields count=2, enabled=True, empty explicit_list.
- Edge case: `--competitors=3` still works (explicit override).
- Edge case: existing `test_bare_flag_defaults_to_three` test is updated to `test_bare_flag_defaults_to_two` and asserts count=2.
- Edge case: `--competitors=5` with a `--competitors-list` of length 2 still logs the mismatch warning and uses the list.
**Verification:**
- `pytest tests/test_cli_competitors.py -v` passes with the updated default.
- [ ] **Unit 2: Per-entity Step 0.55 resolution inside fanout**
**Goal:** Each competitor sub-run auto-resolves its own X handle, subreddits, GitHub user/repos, and news context via `resolve.auto_resolve()` before its `pipeline.run()` call — just like the main topic.
**Requirements:** R2
**Dependencies:** None (but Unit 3 should land together so sub-runs don't emit LAW 7 stderr while the resolution context is being passed)
**Files:**
- Modify: `scripts/lib/fanout.py`
- Modify: `scripts/last30days.py` (`_competitor_runner` closure builds the resolved args)
- Test: `tests/test_competitor_fanout.py`
- Test: `tests/test_competitors_resolve_integration.py` (new; covers the auto-resolve path)
**Approach:**
- `_competitor_runner(entity)` in main() does:
1. Call `resolve.auto_resolve(entity, config)` when `not args.mock` and a web backend is configured (reuse `_has_backend`).
2. Extract resolved x_handle, subreddits, github_user, github_repos, context.
3. Pass them to `pipeline.run()` for that sub-run.
4. Inject resolved context into a per-entity config copy (so `_auto_resolve_context` does not leak across sub-runs — deep-copy the config or use a local dict).
5. Store the resolved block on the Report's `artifacts` so the renderer can surface it (Unit 4).
- When `args.mock` is True or no backend is available, skip auto-resolve (fall through to planner defaults, matching the existing `auto_resolve()` early-return contract).
- Update `fanout.run_competitor_fanout` docstring to note that auto-resolve happens inside the caller-provided runner.
**Execution note:** Start with a failing integration test that exercises two-entity fanout + auto-resolve via a mocked `resolve.auto_resolve` and asserts that `pipeline.run` receives the resolved x_handle/subreddits for each entity.
**Patterns to follow:**
- `scripts/last30days.py` main topic branch (`if args.auto_resolve and not external_plan`) already calls `resolve.auto_resolve` and propagates results — mirror the shape for competitors.
- Config isolation: `scripts/lib/pipeline.py:162-220` reads config as-is; use `dict(config)` to avoid cross-sub-run mutation of `_auto_resolve_context`.
**Test scenarios:**
- Happy path: 3 entities, mocked `auto_resolve` returns distinct handles per entity; `pipeline.run` receives `x_handle=@drake` for Drake, `x_handle=@kendricklamar` for Kendrick, etc.
- Happy path: the main topic still uses the user-supplied `--x-handle` / `--subreddits` overrides (not overwritten by auto-resolve for the main). Competitors use their own auto-resolved values.
- Edge case: `--mock` skips auto-resolve entirely for all sub-runs (no `resolve.auto_resolve` calls).
- Edge case: `resolve.auto_resolve` returns empty dicts for one entity (low-signal topic) — the sub-run still executes with planner defaults; doesn't crash.
- Edge case: no web backend configured — auto-resolve returns empty for every entity, sub-runs fall through to planner defaults, no stack trace.
- Error path: `resolve.auto_resolve` raises — the sub-run logs a warning and continues with planner defaults (does not fail the whole comparison).
- Integration: config `_auto_resolve_context` from entity A does not leak into entity B's `pipeline.run`. Assert each sub-run gets its own context string.
**Verification:**
- New integration test passes.
- End-to-end smoke (mock mode + explicit list): each sub-run's stderr shows `[AutoResolve]` lines per entity with distinct values.
- [ ] **Unit 3: Suppress LAW 7 warning for engine-internal sub-runs**
**Goal:** The `[Planner] No --plan passed... deterministic fallback` warning does not fire during competitor sub-runs. LAW 7 is load-bearing for hosting-model contracts and must stay on the default path; this is an opt-in bypass for internal fan-out only.
**Requirements:** R3
**Dependencies:** Unit 2 (so the sub-run call site is already being modified)
**Files:**
- Modify: `scripts/lib/planner.py` (`plan_query` signature + conditional stderr)
- Modify: `scripts/lib/pipeline.py` (`run` signature + propagation)
- Modify: `scripts/last30days.py` or `scripts/lib/fanout.py` (pass `internal_subrun=True` for competitor runners)
- Test: `tests/test_planner_v3.py` (or new `tests/test_planner_quiet_mode.py`)
- Test: `tests/test_competitor_fanout.py` (assert sub-runs don't emit LAW 7 stderr)
**Approach:**
- Add a keyword `internal_subrun: bool = False` to `planner.plan_query`. When True, skip the two `print(..., file=sys.stderr)` blocks that emit the LAW 7 banner and the `[Planner] No --plan passed` capability message.
- Add the same keyword to `pipeline.run()`; pass through to `plan_query`.
- In main()/fanout, set `internal_subrun=True` for every competitor sub-run's pipeline.run call. The main topic's pipeline.run keeps the default (LAW 7 stays on for the hosting-model path).
- Also suppress the LAW 7-triggered degraded-run warning block in the render layer for sub-reports when the envelope is going to be merged into a comparison output (or accept that the block is per-entity and surfaces once per entity).
**Patterns to follow:**
- Existing keyword-only parameters on `pipeline.run` (`mock`, `x_handle`, etc.).
- `planner.plan_query` signature is already keyword-only.
**Test scenarios:**
- Happy path: `plan_query(..., internal_subrun=True, provider=None, model=None)` returns the deterministic fallback plan WITHOUT writing the LAW 7 stderr block.
- Happy path: `plan_query(...)` with default `internal_subrun=False` still writes the LAW 7 warning (unchanged behavior).
- Integration: end-to-end competitor fanout; assert captured stderr contains zero occurrences of `No --plan passed` and zero of `YOU ARE the planner`.
- Integration: main topic is not part of competitor mode; if the user invokes bare `/last30days OpenAI` without `--plan`, LAW 7 stderr fires exactly once (regression test).
**Verification:**
- Running the Kanye-style smoke test shows zero `[Planner] No --plan passed` lines for Drake / Kendrick / Travis sub-runs.
- [ ] **Unit 4: "Resolved entities" block in comparison output**
**Goal:** The rendered comparison output includes a visible block listing per-entity handles, subreddits, GitHub user, and resolved context. Answers "did it resolve everyone?" at a glance without reading stderr.
**Requirements:** R4
**Dependencies:** Unit 2 (needs resolved data on report artifacts)
**Files:**
- Modify: `scripts/lib/render.py` (`render_comparison_multi` and `render_comparison_multi_context`)
- Test: `tests/test_render_comparison_multi.py`
**Approach:**
- When each entity's `Report.artifacts` contains a `resolved` dict (populated by Unit 2), `render_comparison_multi` emits a `## Resolved Entities` block early in the EVIDENCE envelope:
```
## Resolved Entities
- **Kanye West**: X @kanyewest | Subs r/Kanye, r/hiphopheads | GitHub: — | Context: BULLY released, UK ban…
- **Drake**: X @Drake | Subs r/DrakeTheType, r/hiphopheads | GitHub: — | Context: ICEMAN rollout…
- **Kendrick Lamar**: X @kendricklamar | Subs r/KendrickLamar | GitHub: — | Context: Grammy wins, dormant…
```
- Missing fields render as `` not empty.
- When no entity has a `resolved` payload (mock mode, no web backend), omit the block entirely rather than emit an empty section.
- Context strings are truncated at 120 chars to keep the block scannable.
**Patterns to follow:**
- Existing `render_comparison_multi` envelope structure (lines ~395-480 in render.py).
- Existing per-entity evidence block format (`## {label}`) for consistency.
**Test scenarios:**
- Happy path: 3 entities each with a `resolved` artifact → block lists all 3 with their fields.
- Happy path: 2 entities, one with full resolution, one with partial (x_handle only) → missing fields render as ``.
- Edge case: no entity has a resolved artifact → block is omitted entirely.
- Edge case: context string > 120 chars → truncated with ellipsis.
- Integration: rendered output passes through the same EVIDENCE envelope comments and synthesis contract (LAW 18 unchanged).
**Verification:**
- Snapshot tests confirm the block appears in the right spot with the right formatting.
- End-to-end smoke shows a realistic 3-entity Resolved block in the rendered output.
- [ ] **Unit 5: SKILL.md canonical-path self-check**
**Goal:** A top-of-file SKILL.md directive forces any reader (Claude Code, Codex, Hermes, Gemini) to verify they loaded from `plugins/cache/last30days-skill/last30days/{VERSION}/SKILL.md` before proceeding. If loaded from `marketplaces/` or any other path, re-read from the pinned versioned cache.
**Requirements:** R5
**Dependencies:** None
**Files:**
- Modify: `SKILL.md` (prepend a STEP 0 block before the existing STEP 0 / LAW list)
**Approach:**
- Add a numbered first step at the top (before or bundled with existing "STEP 0: ToolSearch preload"):
```
## STEP 0: Canonical Path Self-Check (must run first)
Before reading anything else below, verify you loaded this SKILL.md from
the versioned cache, not the marketplace clone:
CANONICAL=$HOME/.claude/plugins/cache/last30days-skill/last30days/
CANONICAL_LATEST=$(ls -d "$CANONICAL"*/ 2>/dev/null | sort -V | tail -1)
If the SKILL.md you just read is not under $CANONICAL_LATEST, STOP. Re-read
$CANONICAL_LATEST/SKILL.md and restart from here. Marketplace clones
(`plugins/marketplaces/last30days-skill/`) are pinned to origin/main and
can be stale; the versioned cache is the ground truth.
```
- Reinforce in the existing LAW 7 block that `--help` output must be read from the same pinned `SKILL_ROOT` to avoid flag-list skew.
**Patterns to follow:**
- Existing STEP 0 ToolSearch preload (top of SKILL.md) for tone / imperative voice.
- Existing `SKILL_ROOT` resolver snippet (line ~823).
**Test scenarios:**
- Test expectation: none — SKILL.md is documentation; no unit test, verified by follow-up user invocation.
**Verification:**
- In a fresh Claude Code window, `/last30days Test --competitors` loads SKILL.md, the model executes the STEP 0 self-check, and (if it had loaded from marketplaces/) switches to the cache path before running `--help` or the engine. Observable via the model's announced reasoning / task list.
- [ ] **Unit 6: Version bump, CHANGELOG, sync**
**Goal:** Ship 3.0.12 and deploy to all local targets.
**Requirements:** R6
**Dependencies:** Units 1-5
**Files:**
- Modify: `.claude-plugin/plugin.json` (version 3.0.11 → 3.0.12)
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
**Approach:**
- CHANGELOG entry under `## [3.0.12]` dated 2026-04-22 covering the four fixes (Fixed: per-entity resolution; Fixed: LAW 7 sub-run noise; Changed: default count 3→2; Added: Resolved entities block; Added: canonical-path self-check in SKILL.md).
- `sync.sh` deploys to `~/.claude/plugins/cache/last30days-skill-private/...`, `~/.agents/`, `~/.codex/`, Hermes.
- Manual hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.12/` so the public `/last30days` slash command picks up the new version before PR merge (matches the 3.0.11 testing pattern).
**Test scenarios:**
- Test expectation: none — packaging only. Verification is by inspection.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns `3.0.12`.
- `sync.sh` exits 0 with "Import check: OK" for each target.
- Hot-copied 3.0.12 directory contains the new files and `/last30days` picks up the new version (highest-version resolver).
## System-Wide Impact
- **Interaction graph:** Fanout sub-runs now call `resolve.auto_resolve` per entity. Each sub-run is independent; no shared mutable state with other sub-runs or with the main topic.
- **Error propagation:** `auto_resolve` failures inside a sub-run log a warning and degrade to planner defaults; do not propagate up to abort the comparison. Same contract as today for the main topic.
- **State lifecycle risks:** Config dict is mutated by `auto_resolve` (via `config["_auto_resolve_context"]`). Must deep-copy per sub-run or scope context to a local mapping — otherwise two sub-runs' context strings race.
- **API surface parity:** `pipeline.run` gains a keyword (`internal_subrun`); callers that don't pass it get the existing behavior. `planner.plan_query` gains the same. Backward compatible.
- **Integration coverage:** New integration test for the fanout + auto-resolve + render chain. Existing snapshot tests update to include the Resolved block.
- **Unchanged invariants:** Single-entity `/last30days` invocations (no `--competitors`) behave identically. Explicit `A vs B` comparison topics behave identically. LAW 7 still fires on the default hosting-model path. `render_compact` path is untouched.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Auto-resolving per competitor triples the WebSearch call volume (4 queries × 3 competitors = 12 extra web searches). | Fast-fail when no backend; user can pass `--competitors-list` to skip discovery but still get auto-resolve. Cost note in CHANGELOG. |
| Config mutation across sub-runs via `_auto_resolve_context`. | Unit 2 deep-copies config per sub-run before each `auto_resolve` + `pipeline.run` call. Integration test asserts no cross-entity leak. |
| LAW 7 suppression leaks onto the hosting-model path via a wrong default. | Default `internal_subrun=False`. Only fanout's competitor sub-runs set True. Unit test asserts bare-topic invocation still emits LAW 7. |
| SKILL.md STEP 0 banner gets ignored by the model (same failure mode as line 823 today). | Put it in the guaranteed-read top band (before LAW 1, above all other content), imperative voice, concrete `STOP` verb. Still not bulletproof but strictly better than current. |
| Default count change breaks assumptions in downstream tools or existing user muscle memory. | Changelog calls it out as Changed; `--competitors=3` still works for users who want the old default. |
## Documentation / Operational Notes
- Beta channel first: merge behind `/last30days-beta` via the private repo before cherry-picking to public. Follows the same process as 3.0.11.
- Version 3.0.12 is a fix release; no marketing post required.
- After merge, add a line to the PR description pointing at this plan.
## Sources & References
- Origin plan: `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- Related PR: #308 (v3.0.11 shipping --competitors)
- Test windows that surfaced the bugs: Kanye, Linear, Coinbase (2026-04-22 session)
- Related code: `scripts/lib/fanout.py`, `scripts/lib/resolve.py` (`auto_resolve`), `scripts/lib/planner.py` (`plan_query`), `scripts/lib/render.py` (`render_comparison_multi`)
@@ -1,394 +0,0 @@
---
title: "fix: --competitors runs a full last30days per entity with hosting-model pre-resolve"
type: fix
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md
---
# fix: --competitors runs a full last30days per entity with hosting-model pre-resolve
## Overview
User intent confirmed 2026-04-22: `--competitors` should run a full single-entity `last30days` pipeline for the main topic AND for each discovered peer — three independent full-depth passes, each with its own Step 0.55 resolution, own X handle primary weight, own subreddit targeting, own GitHub repo scoping. Then merge them into the comparison output.
3.0.12 already built the N-parallel-pipelines orchestration (`scripts/lib/fanout.py`). What it got wrong: it tried to do per-entity Step 0.55 engine-side via `resolve.auto_resolve()`, which requires a web search backend key (BRAVE/EXA/SERPER/PARALLEL/OPENROUTER). Matt runs from Claude Code, which has its own WebSearch tool. The engine has none of those keys, so per-entity auto_resolve silently no-ops and all peer sub-runs fall through to deterministic single-word planner queries.
Four 2026-04-22 test runs (Warriors, Seattle, Arizona Wildcats, Kanye West) confirmed this via engine receipts:
- Compact Resolved Entities block shows peers as `X - | Subs - | GitHub - | Context: -`.
- Sub-run planner lines show `source=deterministic, subqueries=1` — the "I gave up and keyword-searched" shape.
- Engine footer keeps nudging `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY`, which is wrong advice for a Claude Code user who already has WebSearch.
- Kanye run leaked main topic's `--subreddits` into Drake's and Kendrick's sub-runs (regression bug).
The fix is to flip the resolution responsibility: the hosting model (Claude Code, Codex, Hermes, Gemini) does Step 0.55 via its own WebSearch tool for every entity, then passes the resolved targeting to the engine via a new `--competitors-plan` JSON flag. Engine fan-out remains — each peer still runs a full `pipeline.run()`. The difference is the peers now arrive with full targeting, equivalent to the main topic, so retrieval is apples-to-apples.
Why not just reuse vs-mode? vs-mode is a SINGLE `pipeline.run()` with a comparison-optimized plan. It pre-resolves Step 0.55 per entity but merges everything into one retrieval pool with lower-weight `--x-related` for peers, merged subreddits, and cross-entity keyword noise. That is not "three full passes." The user explicitly wants three full passes.
## Problem Frame
3.0.12's architecture was correct; its data dependency was wrong.
| Capability | 3.0.12 path | Target path (this plan) |
|---|---|---|
| Fan out to N parallel pipelines | Yes (`fanout.run_competitor_fanout`) | Same — keep |
| Per-entity Step 0.55 resolution | Engine-internal `resolve.auto_resolve()` — needs BRAVE/EXA/SERPER/PARALLEL key | Hosting model does it via its own WebSearch, passes to engine |
| Per-entity targeting threaded into `pipeline.run()` | Main topic only via outer flags; peers via auto_resolve (failing) or nothing | Main topic via outer flags; peers via `--competitors-plan` JSON |
| Footer nudge | Unconditional BRAVE/SERPER | Suppressed when `--plan` or `--competitors-plan` present |
| Resolved Entities block in raw save file | Stdout only | Also in `--save-dir` raw file |
| Override-leak from main into peers | Present (Kanye receipt) | Fixed via explicit per-entity kwargs scrub |
| Polymarket noise on ambiguous topics | Present (Warriors, Arizona receipts) | `--polymarket-keywords` + auto-skip for single-token-ambiguous |
The key architectural change is who owns per-entity resolution. The engine stops trying to do it itself; the hosting model does it upstream (it already has WebSearch) and passes results in.
This is the same pattern `--plan` already uses for the main topic: hosting model generates the plan via its own reasoning, passes it in, engine accepts. We apply the pattern to peers.
## Requirements Trace
- R1. New `--competitors-plan` JSON flag accepting per-entity targeting: `x_handle`, `x_related`, `subreddits`, `github_user`, `github_repos`, `context`. Implies `--competitors`. Per-entity values thread into that entity's `pipeline.run()`. Bypasses engine-internal `auto_resolve` for covered entities.
- R2. SKILL.md "Competitor mode" rewritten to make the hosting-model path canonical: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers) via WebSearch, (c) assemble `--competitors-plan` JSON, (d) invoke engine. Engine-internal auto_resolve remains as headless fallback.
- R3. The LAW 7-style stderr emitted when `--competitors` has no list, no plan, no backend is reframed: leads with "hosting reasoning model, use your WebSearch to run Step 0.55 per entity and pass `--competitors-plan`." Does not lead with BRAVE_API_KEY.
- R4. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` OR `--competitors-plan` was passed. Signal: hosting model is driving and already has WebSearch.
- R5. Override-leak fix: competitor sub-runs do not inherit main topic's `--subreddits`, `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`. Sub-runs use only their own per-entity targeting (from `--competitors-plan` if provided, else engine-internal auto_resolve if backend, else planner defaults).
- R6. The `## Resolved Entities` block is also appended to the saved raw file when `--save-dir` is in use. Each entity's effective targeting (whatever was actually passed to its `pipeline.run()`) is visible on audit.
- R6b. When `--save-dir` is in use with a comparison run, each entity's sub-run ALSO saves its own standalone raw file — same format as a single-entity run. `/last30days Kanye West --competitors` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md` (one per entity) plus the merged comparison file. Matches the historical vs-mode behavior when it ran as N passes.
- R7. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R8. Default `--competitors` count remains 2 (3-way: main + 2 peers). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to `scripts/lib/fanout.py` architecture. N parallel pipelines stays. Only the data each sub-run receives changes.
- No changes to the vs-mode (topic contains "vs" / "versus") behavior. That path is independent.
- No new emit modes. Comparison output format unchanged.
- No deprecation of `--competitors-list`. Stays as the minimum escape hatch for hosting models that skip per-entity Step 0.55 (names-only).
### Deferred to Separate Tasks
- Cache layer for hosting-model competitor resolution: separate plan once cost evidence exists.
- Cross-source disambiguation beyond Polymarket: separate plan.
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — `--competitors` / `--competitors-list` argparse block, `resolve_competitors_args` validator, `_main_runner` closure, `_competitor_runner` closure, the `[Competitors] --competitors requires...` stderr block. Primary file for this plan.
- `scripts/lib/fanout.py` — `run_competitor_fanout` orchestrator. Signature unchanged; `_competitor_runner` closure now builds kwargs from `--competitors-plan`.
- `scripts/lib/pipeline.py` — `pipeline.run()` signature; no changes required (all per-entity flags already exist as kwargs).
- `scripts/lib/planner.py` — existing `--plan` parsing and validation, pattern to mirror for `--competitors-plan`.
- `scripts/lib/render.py` `_render_resolved_entities_block` (added in 3.0.12) — already reads `report.artifacts["resolved"]`; no change needed.
- `scripts/last30days.py` `save_output` / `render.render_full` — the save path. Needs to include the Resolved Entities block for comparison runs.
- `scripts/lib/quality_nudge.py` — where the BRAVE/SERPER footer nudge is emitted. Needs a context-aware suppression check.
- `scripts/lib/polymarket.py` — source adapter. Entry point for `--polymarket-keywords` filter and single-token-ambiguous auto-skip.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built the initial fanout, deferred per-entity resolve as "v1 simplification."
- 3.0.12 plan (`2026-04-22-003`): tried to close the gap via engine-internal `auto_resolve`. Works only with backend keys. Fails silently without.
- 2026-04-22 test session receipts: confirmed all four fixes in this plan are real, reproducible bugs.
- User's architectural steer 2026-04-22: "runs a full last30days on all 3 topics" — this plan encodes that explicitly as N full `pipeline.run()` calls with pre-resolved targeting per entity.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **`--competitors-plan` is a single JSON flag, not a fan of separate flags.** Mirrors `--plan`. Stable schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Accept inline JSON or a file path (matches `--plan`).
- **Hosting-model-driven resolution is the documented default.** Engine-internal `auto_resolve` is the headless / cron fallback. SKILL.md routes hosting models to the JSON-flag path; engine keeps auto_resolve alive for BRAVE/EXA/SERPER users running CI.
- **Override-leak fix is call-site scrubbing, not a signature change.** `_competitor_runner` builds an explicit kwargs dict per entity from `_subrun_kwargs(entity, plan_entry)`. No closure-default fallthrough from main scope. The 3.0.12 `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag.
- **Footer nudge becomes context-aware.** Suppressed when `--plan` or `--competitors-plan` present. Not suppressed for bare `--competitors-list` or bare invocations. Headless cron without keys still sees the nudge.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known list of single-token-ambiguous names (states, common nouns). Stderr notes the skip so it is observable and overridable.
- **Per-entity sub-runs get the full `pipeline.run()` pass.** Same depth, same sources, same API cost per entity as a single-topic run. This is the explicit user intent — three full passes, not one merged pass.
## Open Questions
### Resolved During Planning
- **JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers (3-way comparison). Unchanged from 3.0.12.
- **Does engine-internal auto_resolve stay alive?** Yes, for entities not covered by `--competitors-plan` when a backend is configured. Headless/cron users with keys keep the current 3.0.12 behavior.
- **vs-mode or fanout?** Fanout. User's explicit ask: three full passes, not one merged pass. vs-mode merges into one pipeline with lower peer weighting, which is not what the user wants.
- **Does the save file need per-entity clusters?** Start with the Resolved block appended. Per-entity cluster sections can follow in a separate task; they are nice-to-have, not blocking.
### Deferred to Implementation
- Exact trace of override-leak source. Candidates: closure capture of `subreddits` in `_competitor_runner`, shared `_auto_resolve_context` leak, Reddit adapter inheriting global config. Test-first; trace at implementation time.
- Heuristic for "single-token-ambiguous topic" auto-skip. Start with a short hard-coded list (US state names, US city names, common nouns like "Warriors", "Suns", "Jets"); revisit after dogfood.
- Whether per-entity coverage warnings fire when `--competitors-plan` under-resolves an entity (e.g., only `x_handle`, no subreddits). Start with stderr logging; revisit UX.
## Implementation Units
- [ ] **Unit 1: `--competitors-plan` JSON flag + per-entity kwargs threading**
**Goal:** New CLI flag accepting per-entity targeting JSON. Each covered entity's `pipeline.run()` receives its own `x_handle` / `x_related` / `subreddits` / `github_user` / `github_repos` / `context`. Skips engine-internal `auto_resolve` for covered entities.
**Requirements:** R1, R5 (primary leak fix site)
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner`)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; verify)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON OR a file path (mirror `--plan`).
- Validation: parse JSON; must be a dict; each value must be a dict; unknown fields log warnings; malformed input exits 2.
- Schema per entity: optional fields `x_handle` (str), `x_related` (list), `subreddits` (list), `github_user` (str), `github_repos` (list), `context` (str).
- Case-insensitive matching against `--competitors-list` / discovered entities.
- Build `_subrun_kwargs(entity, plan_entry)` helper. Returns a complete, explicit kwargs dict for `pipeline.run()` with no closure-default fallthrough from main scope. This helper is the single source of truth for per-entity call args. It also fixes the override-leak (R5) by scrubbing all per-entity flags to None unless the plan (or auto_resolve) sets them.
- `_competitor_runner(entity)`:
1. Look up `plan_entry` from `--competitors-plan` (if any).
2. If plan covers entity fully, build kwargs from it; skip `auto_resolve`.
3. If plan partially covers or is absent, fall back to `auto_resolve` (3.0.12 behavior) when a backend is configured. Plan values win over auto_resolve values on conflict.
4. If neither plan nor backend, fall through to `pipeline.run()` with per-entity kwargs all None — engine uses planner defaults for that entity only (no leak).
- Deep-copy config per sub-run (already done in 3.0.12); merge per-entity `context` into `entity_config["_auto_resolve_context"]` only.
**Execution note:** Test-first for the override-leak regression (pass `--subreddits=A,B` on main + a peer, assert peer's `pipeline.run(subreddits=...)` is None or peer-specific).
**Patterns to follow:**
- `--plan` parsing at `scripts/last30days.py` (inline JSON or file path).
- 3.0.12's `_competitor_runner` closure for scope; extract the kwargs-build into `_subrun_kwargs` helper.
- `entity_config = dict(config)` deep-copy pattern from 3.0.12.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake": {"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's `pipeline.run` receives `x_handle="Drake"` and `subreddits=["Drizzy"]`; no `auto_resolve` call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered entities skip auto_resolve; third falls back to auto_resolve.
- Happy path: plan file path accepted like `--plan` file path.
- Happy path: case-insensitive entity match (`Drake` in plan, `drake` in list).
- Edge case: unknown fields in plan entry → logged, ignored, run continues.
- Edge case: plan entry for entity not in list → ignored with warning.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list not dict → exit 2.
- Regression (leak fix): main `--subreddits=A,B` + `--competitors-list "Drake"` + no plan → Drake's `pipeline.run` receives `subreddits=None` (no leak).
- Regression (leak fix): same for `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`.
- Regression (leak fix): main `--x-handle=kanyewest` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanyewest"`.
- Integration: full main + 2 peers run via `--competitors-plan`; assert each sub-run's effective kwargs match expected per-entity values.
**Verification:**
- All new and regression tests pass.
- Smoke run (mock mode + `--competitors-plan`): stderr shows `[Competitors] Drake: x=@Drake subs=Drizzy` line per entity; no `[AutoResolve]` calls for plan-covered entities; no leak of main topic's flags.
- [ ] **Unit 2: Reframe LAW 7-style stderr for hosting-model context**
**Goal:** When `--competitors` has no `--competitors-list`, no `--competitors-plan`, and no backend, stderr tells the hosting reasoning model to use its WebSearch tool for Step 0.55 per entity and pass `--competitors-plan`. Stops leading with BRAVE_API_KEY.
**Requirements:** R3
**Dependencies:** Unit 1 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the existing `[Competitors] --competitors requires...` block)
- Test: `tests/test_competitors_no_backend_message.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent runtime with a WebSearch tool), YOU should: (a) discover N peers via WebSearch, (b) run Step 0.55 per entity (main + peers), (c) assemble a `--competitors-plan` JSON, (d) re-invoke. Skip this step and quality degrades — peer entities will run with planner defaults."
2. "If you are running headless (cron, CI, no hosting model), set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list "A,B,C"` skips discovery but does not pre-resolve peers. Use only for quick tests."
- Exits non-zero as today.
**Patterns to follow:**
- Existing LAW 7 stderr in `planner.plan_query` for tone.
**Test scenarios:**
- Happy path: stderr leads with "If you are the hosting reasoning model" and names `--competitors-plan` before any backend key.
- Happy path: stderr explicitly names `--competitors-plan` as the preferred override.
- Happy path: stderr does NOT say "requires either a configured web search backend OR an explicit --competitors-list" (the current 3.0.12 wording).
**Verification:**
- Test asserts ordering and required phrases.
- [ ] **Unit 3: Suppress BRAVE/SERPER footer nudge when hosting-model-driven**
**Goal:** The `💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY` footer is suppressed when `--plan` or `--competitors-plan` was passed (signal: hosting model is driving and already has WebSearch).
**Requirements:** R4
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/lib/quality_nudge.py` (or wherever nudge is emitted; verify during implementation)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Locate the nudge emission point.
- Add a suppression check: if `--plan` OR `--competitors-plan` was passed, skip the nudge. Otherwise, current behavior.
- Don't suppress the nudge for bare `--competitors-list` alone — that path isn't necessarily hosting-model-driven.
**Test scenarios:**
- Happy path: `--plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-plan` passed, no backend → nudge does NOT fire.
- Happy path: `--competitors-list` only, no backend → nudge fires (current behavior).
- Happy path: no `--competitors`, no `--plan`, no backend → nudge fires (current behavior unchanged).
**Verification:**
- All four scenarios produce expected nudge presence/absence.
- [ ] **Unit 4: Per-entity save files + Resolved block in each**
**Goal:** When `--save-dir` is in use with a comparison run, each entity's sub-run saves its own standalone raw file (same format as a single-entity run), and each file includes the `## Resolved Entities` block so audits can see what targeting that entity received. Matches the historical vs-mode behavior when it was N passes.
**Requirements:** R6, R6b
**Dependencies:** Unit 1
**Files:**
- Modify: `scripts/last30days.py` (`save_output`, the save loop after fanout completes)
- Possibly modify: `scripts/lib/render.py` (`render_full` branch to include Resolved block when artifact is present)
- Test: `tests/test_save_raw_competitor_files.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]`. For each `(entity, entity_report)` tuple, call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)` — same path a single-entity run takes.
- Each saved file uses its entity's slug as the filename (`drake-raw.md`, `kendrick-lamar-raw.md`). Main topic keeps the existing `kanye-west-raw.md` filename.
- Each file includes its own `## Resolved Entities` block (single-entity variant: one row for that entity only). This makes each sub-run's file self-describing — you can see what targeting was used without opening the comparison file.
- The merged comparison output (stdout) still includes the 3-row Resolved Entities block.
- Optional: also save a comparison summary file (e.g., `kanye-west-comparison-raw.md`) holding the merged multi-entity render. Start with per-entity files only; comparison summary is a follow-up if stdout-plus-individual-files is insufficient.
- Single-entity runs unchanged (no additional files, no block change).
**Patterns to follow:**
- Existing `save_output` invocation for single-entity runs (line 501 of current `scripts/last30days.py`).
- Existing slug generation (`slugify(topic)`) for filename consistency.
- `_render_resolved_entities_block` from 3.0.12 for the single-entity variant.
**Test scenarios:**
- Happy path: `--competitors-list "Drake,Kendrick Lamar"` + `--save-dir=/tmp/x` → `/tmp/x/kanye-west-raw.md`, `/tmp/x/drake-raw.md`, `/tmp/x/kendrick-lamar-raw.md` all exist.
- Happy path: each peer file's first sections include that entity's Resolved Entities block with its own row only.
- Happy path: single-entity run with `--save-dir` → one file, unchanged from today's behavior.
- Edge case: entity slug collides with existing file → overwrite (matches single-entity behavior).
- Edge case: `--save-suffix=v3` → all 3 files get the suffix (`kanye-west-raw-v3.md`, `drake-raw-v3.md`, `kendrick-lamar-raw-v3.md`).
- Edge case: comparison run with one peer whose sub-run failed → that entity's file is NOT saved; others are.
- Integration: stderr after save shows three `[last30days] Saved output to <path>` lines, one per entity.
**Verification:**
- After `/last30days Kanye West --competitors-list "Drake,Kendrick Lamar" --save-dir=/tmp/x`: `ls /tmp/x/*-raw.md` shows 3 files. Each contains its entity's Resolved block.
- [ ] **Unit 5: SKILL.md "Competitor mode" rewrite — hosting-model Step 0.55 canonical**
**Goal:** SKILL.md documents the hosting-model-driven path as canonical: discover N peers via WebSearch, run Step 0.55 per entity, assemble `--competitors-plan`, invoke engine. Engine-internal `auto_resolve` is labeled the headless fallback.
**Requirements:** R2
**Dependencies:** Unit 1 (flag must exist before documented)
**Files:**
- Modify: `SKILL.md` (Competitor mode subsection)
- Modify: `README.md` (one-line example update)
**Approach:**
- Replace the 3.0.12 Competitor mode subsection with a clear flow:
1. User invokes with `--competitors` or `--competitors=N`.
2. Hosting model runs WebSearch for "[topic] competitors" / "[topic] alternatives" → picks top N peers.
3. Hosting model runs Step 0.55 for main + each peer (x_handle, subreddits, github_user, github_repos, context) — same protocol as vs-mode per SKILL.md §679.
4. Hosting model assembles a `--competitors-plan` JSON object.
5. Hosting model invokes the engine with `--competitors-list "A,B,C" --competitors-plan '{...}'`.
6. Engine fans out N full pipelines (main + peers), each with its own full Step 0.55-grade targeting. Each entity also saves its own `*-raw.md` file when `--save-dir` is set (three full passes → three save files, matching the historical vs-mode behavior). Comparison output merges them for display.
- Concrete JSON example in SKILL.md showing the schema.
- Failure-mode warning: a `## Resolved Entities` block with dashes for any entity means hosting model skipped Step 0.55 for that one. Re-run with corrected plan.
- "Headless fallback" sub-subsection: when BRAVE/EXA/SERPER/PARALLEL/OPENROUTER is set, engine's internal `auto_resolve` handles peers and `--competitors-plan` is optional.
**Patterns to follow:**
- SKILL.md "Step 0.55" section for per-entity resolve protocol.
- SKILL.md "If QUERY_TYPE = COMPARISON" section for the same-protocol-as-vs-mode reference.
- Tone of existing 3.0.12 Competitor mode prose.
**Test scenarios:**
- Test expectation: none — documentation. Verification is a fresh Claude Code window dogfood run.
**Verification:**
- `/last30days Kanye West --competitors` in a new window: hosting model does Step 0.55 for Kanye + 2 discovered peers; passes `--competitors-plan`; rendered Resolved block shows non-empty fields for all 3; top voices include at least one peer-specific handle.
- [ ] **Unit 6: Polymarket disambiguation guard**
**Goal:** Support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
**Requirements:** R7
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` argparse (`--polymarket-keywords`)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"` flag. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip rule: if topic is one token AND token matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords` provided, skip Polymarket with a stderr note.
- SKILL.md Step 0.55 protocol gets a small addition: for ambiguous topics, hosting model passes `--polymarket-keywords` with topic-specific qualifiers.
**Patterns to follow:**
- Existing Polymarket adapter match logic.
- Single-token detection heuristic.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr notes the skip.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs; matches filtered.
- Happy path: topic "OpenAI" (no ambiguity) → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke run → Polymarket footer absent OR filtered to nba/gsw markets.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R7
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG entry groups the fixes: Added `--competitors-plan` JSON flag for per-entity hosting-model pre-resolve. Fixed override-leak from main into peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when `--plan` / `--competitors-plan` is present. Added: Resolved Entities block persists to saved raw file. Added: `--polymarket-keywords` + auto-skip for ambiguous single-token topics.
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13 immediately.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` returns 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files with competitors.py, fanout.py, the updated SKILL.md, and plugin.json 3.0.13.
## System-Wide Impact
- **Interaction graph:** `_competitor_runner` becomes the single source of truth for sub-run kwargs via `_subrun_kwargs(entity, plan_entry)`. Every per-entity flag flows through one helper. No closure-default leaks.
- **Error propagation:** `--competitors-plan` JSON parse errors exit 2 with stderr (same as `--plan`). Per-entity plan entries with malformed values log warnings and fall back; don't abort the whole run.
- **State lifecycle risks:** `entity_config = dict(config)` already deep-copies for `_auto_resolve_context`; extend the isolation discipline to every per-entity flag. Verified in Unit 1 regression tests.
- **API surface parity:** `--competitors-plan` is additive. `--competitors` and `--competitors-list` unchanged. `--plan` unchanged. `--polymarket-keywords` additive.
- **Integration coverage:** New regression tests for override-leak. New integration test for plan-driven sub-run threading. New nudge-suppression test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. `planner.plan_query` LAW 7 behavior for the default path unchanged. Single-entity render path unchanged. vs-mode behavior unchanged.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model takes the lazy path and uses `--competitors-list` names-only. | Unit 2 stderr explicitly steers to `--competitors-plan` with Step 0.55 protocol named. Unit 5 SKILL.md docs. Resolved Entities dashes in output make the gap visible. |
| JSON gets verbose for the hosting model to construct repeatedly. | Schema is small (≤6 fields per entity). Hosting model already runs Step 0.55 for main topic in every comparison run; peers use the same protocol. One JSON block replaces N CLI flags. |
| Override-leak source is deeper than `_competitor_runner` closure. | Test-first per Unit 1. Receipts from 2026-04-22 Kanye run are reproducible. Trace methodically from call site. |
| Plan-covered entity bypasses auto_resolve but plan data is incomplete (e.g., no subreddits). | Hosting model's own SKILL.md contract says Step 0.55 must cover all fields. Stderr logs per-entity coverage so under-resolved entities are visible. Next-run correction, not engine-side rescue. |
| Polymarket auto-skip false-positives on legitimate ambiguous topics with real markets. | Conservative match (single-token + known list). `--polymarket-keywords` override is explicit and unambiguous. Stderr notes the skip. |
| Footer nudge suppression hides the message from headless users who genuinely need it. | Suppression only fires when `--plan` or `--competitors-plan` is present. Cron / CI runs that pass neither still see the nudge. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md (private repo `/last30days-beta`).
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG voice should call this out as the feedback-driven follow-up to 3.0.12. Reader should see "we tried engine-internal resolve in 3.0.12; it needs backend keys we don't have; we moved resolution to the hosting model in 3.0.13."
## Sources & References
- Origin plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Earlier plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts: Warriors, Seattle, Arizona Wildcats, Kanye West
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -1,454 +0,0 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: vs mode runs N full passes and --competitors is vs with auto-discovery"
type: feat
status: active
date: 2026-04-22
origin: docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded
---
# feat: vs mode runs N full passes and --competitors is vs with auto-discovery
## Overview
Architectural unification driven by user correction 2026-04-22: vs mode and `--competitors` are the same thing. A user typing `/last30days OpenAI vs Anthropic vs xAI` should get a full single-entity last30days pass for each of the three entities — three full pipelines, three saved `*-raw.md` files, merged into one comparison output. A user typing `/last30days OpenAI --competitors` should get the same output after the hosting model auto-picks 2 peers; i.e., `--competitors` is a thin shortcut that expands "topic + `--competitors`" into "topic vs peer1 vs peer2" and then runs the unified vs pipeline.
Current state diverges from this:
- **vs mode today**: one `pipeline.run()` with a comparison-optimized plan that merges all entities' targeting into a single retrieval pool. Lower-weight `--x-related` for peers, merged subreddits, cross-entity keyword noise. One saved file.
- **`--competitors` today (3.0.12)**: N parallel `pipeline.run()` calls via `scripts/lib/fanout.py`, but per-entity Step 0.55 depends on an engine-side web backend key Matt doesn't have. Silently degrades to planner defaults for peers. One saved file (main topic only). Override-leak from main into peers.
After this plan:
- **vs mode**: N parallel `pipeline.run()` calls, one per entity, each with its own full Step 0.55-grade targeting, each saving its own `*-raw.md`. Merged into one comparison output.
- **`--competitors`**: SKILL.md shortcut. Hosting model discovers N peers, builds `"topic vs peer1 vs peer2"`, and invokes the same vs pipeline. No separate orchestration path.
- **Same fanout machinery (`scripts/lib/fanout.py`)** serves both. One fix, both behaviors improve.
## Problem Frame
The product insight from 2026-04-22 test runs is simple: the user wants three full last30days reports plus a comparison merge. Not one comparison pass with N-way targeting merged into a single retrieval pool. Not one save file. Not "main gets Step 0.55, peers get planner defaults." Three full passes. Three save files. Merged output.
The historical vs mode did that (it ran as 3 passes, saving 3 files). SKILL.md §551 currently says:
> "When the user asks 'X vs Y', run ONE research pass with a comparison-optimized plan that covers both entities AND their rivalry. This replaces the old 3-pass approach (which took 13+ minutes and produced tangential content)."
That change was a latency optimization that removed the user-visible behavior the user wants. The fix is to revert the architectural direction: N passes per entity, in parallel rather than serial (parallelism lowers wall-clock to ~1× a single pass, not N×), with per-entity save files.
The 3.0.11 `--competitors` flag already introduced parallel N-pass machinery (`fanout.run_competitor_fanout`). The 3.0.12 follow-up tried to wire per-entity Step 0.55 into it but failed when no web backend was configured. The elegant move: stop maintaining two architectures. vs-mode and `--competitors` both use `fanout.py`. `--competitors` becomes a SKILL.md-level shortcut that discovers 2 peers and hands off to vs-mode.
Four 2026-04-22 test receipts (Warriors, Seattle, Arizona Wildcats, Kanye West) all confirmed the user's pain points:
- Peers thin because they ran without per-entity handle/sub targeting.
- Only one `*-raw.md` per run — no per-entity audit.
- Kanye peers leaked main topic's `--subreddits`.
- Engine footer nudging `BRAVE_API_KEY` to Claude Code users who already have WebSearch.
- Polymarket noise on ambiguous topics (Warriors → Glasgow rugby; Arizona → Diamondbacks).
This plan closes all of them by unifying the architecture and making hosting-model-driven Step 0.55 per entity the canonical path.
## Requirements Trace
- R1. vs mode (any topic containing ` vs ` / ` versus `) runs N full `pipeline.run()` calls in parallel, one per entity. Each sub-run uses its entity's own Step 0.55 targeting (from the hosting model's pre-resolution, passed via a new `--competitors-plan` JSON).
- R2. `--competitors` (and `--competitors=N`) becomes a SKILL.md-level shortcut: the hosting model (a) discovers N peers via WebSearch, (b) runs Step 0.55 per entity (main + peers), (c) rewrites the topic to `"main vs peer1 vs peer2"`, (d) invokes the engine with `--competitors-plan` containing each entity's targeting.
- R3. New `--competitors-plan` JSON flag. Schema: `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Implies vs mode when present with a single-entity topic. Applies per-entity targeting to each sub-run. Accepts inline JSON or a file path (matches `--plan`).
- R4. Each entity's sub-run saves its own `*-raw.md` file when `--save-dir` is in use. Example: `/last30days "Kanye West vs Drake vs Kendrick Lamar" --save-dir=~/Documents/Last30Days` produces `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`. Same filenames a single-entity run of each topic would produce. Matches historical vs-mode behavior.
- R5. Each per-entity saved file includes its own single-row `## Resolved Entities` block so the audit survives. The merged comparison stdout still shows the full 3-row block.
- R6. Override-leak fix: no main-topic flags (`--subreddits`, `--x-handle`, `--x-related`, `--tiktok-*`, `--ig-creators`, `--github-*`) leak into peer sub-runs. Every per-entity kwarg is scrubbed at the sub-run call site.
- R7. LAW 7-style stderr for `--competitors` invocations with no list, no plan, no backend is reframed for hosting-model context: leads with "use your WebSearch to discover peers, resolve Step 0.55 per entity, re-invoke with `topic vs peer1 vs peer2 --competitors-plan '...'`." Does not lead with BRAVE_API_KEY.
- R8. Footer nudge `💡 You can unlock native grounded web search with BRAVE_API_KEY...` is suppressed when `--plan` or `--competitors-plan` was passed.
- R9. Polymarket disambiguation: support `--polymarket-keywords "kw1,kw2"` to filter market matches; auto-skip Polymarket when topic is single-token-ambiguous and no override is provided.
- R10. Default `--competitors` count stays 2 peers (3-way comparison). Unchanged from 3.0.12.
## Scope Boundaries
- No changes to single-entity `pipeline.run()` semantics. Each sub-run in vs mode behaves identically to a bare `/last30days {entity}` invocation.
- No changes to the planner's comparison-intent logic for single-entity-containing topics. The `_should_force_deterministic_plan` shortcut for vs-topics routes to fanout, not to its current single-pipeline path.
- No new emit modes. Comparison output format unchanged.
- No removal of `--competitors-list`. Stays as a minimum escape hatch (names-only, no per-entity targeting) for scripted headless use.
- No removal of engine-internal `resolve.auto_resolve()` in fanout. Remains as headless / cron fallback for users with BRAVE/EXA/SERPER/PARALLEL/OPENROUTER keys. The dominant Claude Code path bypasses it via `--competitors-plan`.
### Deferred to Separate Tasks
- Explicit "head-to-head" rivalry pass in vs-mode (a supplemental subquery like `"A vs B"` that catches rivalry articles missing from pure entity-scoped passes). Start with N independent passes; add a head-to-head supplemental pass if the rivalry-content gap shows up in dogfood.
- Cache layer for hosting-model pre-resolution.
- Cross-source disambiguation (not just Polymarket).
- Latency knob for users who want the old one-pass vs behavior (probably not needed; parallel N-pass is ~1× wall clock).
## Context & Research
### Relevant Code and Patterns
- `scripts/last30days.py` — main(), `_main_runner`, `_competitor_runner`, the competitor enable/discovery branch. Primary file.
- `scripts/lib/fanout.py` — existing orchestrator (3.0.11). Reused as-is; `competitor_runner` closure is where per-entity kwargs apply.
- `scripts/lib/planner.py``_should_force_deterministic_plan` detects vs-topics via regex. Current path synthesizes ONE comparison plan; new path routes to fanout.
- `scripts/lib/render.py``render_comparison_multi` (3.0.12) + `_render_resolved_entities_block`. Both reused. `render_full` needs a per-entity variant when saving sub-run files.
- `scripts/last30days.py` `save_output` — where raw files are written. Needs to iterate per entity when competitor_reports artifact present.
- `scripts/lib/quality_nudge.py` — BRAVE/SERPER nudge emission.
- `scripts/lib/polymarket.py` — source adapter for `--polymarket-keywords` and ambiguous-topic auto-skip.
- SKILL.md §551 "If QUERY_TYPE = COMPARISON" and §679 per-entity Step 0.55 protocol — the hosting-model contract that drives per-entity pre-resolution for both vs mode and `--competitors`.
### Institutional Learnings
- 3.0.11 plan (`2026-04-22-002`): built fanout.
- 3.0.12 plan (`2026-04-22-003`): tried engine-internal per-entity auto_resolve; failed without backend keys.
- 3.0.13 plan draft (`2026-04-22-004-...superseded`): proposed `--competitors-plan` JSON + vs-mode-shortcut path but kept them separate. User's 2026-04-22 correction unifies them.
- 2026-04-22 test receipts: Warriors, Seattle, Arizona Wildcats, Kanye West runs all reproduced the per-entity resolve gap.
- User's architectural steer: "vs mode should work that way too" + "--competitors is just vs mode with auto-discovery." This plan encodes that.
### External References
- None. All patterns in-repo.
## Key Technical Decisions
- **Unify vs-mode and --competitors on one orchestrator.** `fanout.run_competitor_fanout` serves both. vs-mode is "topic contains ' vs '" detection → fanout. `--competitors` is "SKILL.md shortcut → hosting model rewrites topic to vs form → fanout." One code path.
- **Per-entity targeting via `--competitors-plan` JSON.** Schema `{entity_name: {x_handle, x_related, subreddits, github_user, github_repos, context}}`. Mirrors `--plan`. Applies to both vs-mode and `--competitors` paths. Hosting model passes it after running Step 0.55 per entity.
- **N save files, one per entity.** Each sub-run writes a `{entity-slug}-raw.md` file when `--save-dir` is set. Matches historical vs-mode behavior. Single-entity runs unchanged.
- **Revert the "one pass for latency" optimization that removed per-entity passes.** Parallel execution via `ThreadPoolExecutor` means wall-clock is ~max(per-entity-latency), not sum. The old latency concern (13+ minutes for 3 serial passes) does not apply to a parallel fan-out.
- **Override-leak fix at the call site.** `_subrun_kwargs(entity, plan_entry)` helper returns fully explicit per-entity kwargs; no closure-default fallthrough from main scope.
- **LAW 7 stderr reframed, not just updated.** Current message treats BRAVE_API_KEY as the solution. New message treats hosting-model Step 0.55 as the solution, with backend keys listed only as the headless fallback.
- **Polymarket disambiguation is additive and conservative.** `--polymarket-keywords` is explicit; auto-skip only fires for a known-ambiguous single-token list.
## Open Questions
### Resolved During Planning
- **vs mode N passes or single-pass?** N passes. User's architectural correction.
- **Should --competitors still be an engine flag at all?** Yes, kept for headless / cron contexts with backend keys. Dominant Claude Code path is SKILL.md shortcut → vs-mode fanout. Engine flag stays as compatibility surface.
- **`--competitors-plan` JSON or multi-flag?** JSON. Matches `--plan`.
- **Default count?** 2 peers → 3-way comparison. Unchanged.
- **Saved-file naming?** `{entity-slug}-raw.md` per entity, same as single-entity runs would produce.
### Deferred to Implementation
- Exact trace of override-leak path (closure capture vs shared config vs Reddit adapter fallback). Test-first per Unit 2; patch at the right layer.
- Heuristic for single-token-ambiguous Polymarket auto-skip. Start with a short hard-coded list; iterate.
- Whether to include a head-to-head rivalry supplemental pass in vs-mode. Ship N-independent passes first; revisit after dogfood if rivalry content is missing.
- Exact filename convention when the comparison merged output is saved (if saved at all). Not blocking — per-entity files are the primary save artifact.
## High-Level Technical Design
> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
```
User invokes:
/last30days "OpenAI vs Anthropic vs xAI"
OR
/last30days OpenAI --competitors (hosting model rewrites to vs form)
OR
/last30days OpenAI --competitors-list "Anthropic,xAI"
OR
/last30days "OpenAI vs Anthropic vs xAI" --competitors-plan '{...per-entity...}'
scripts/last30days.py main():
- Detect: topic has " vs " OR --competitors enabled
- If --competitors and no list/plan: emit LAW 7-style stderr with hosting-model instruction
- If --competitors with list or discovery: rewrite topic to vs form, continue
- Parse --competitors-plan JSON, map to entities
fanout.run_competitor_fanout (shared path):
- For each entity (main + peers):
- entity_config = dict(config) [deep copy to prevent leak]
- kwargs = _subrun_kwargs(entity, plan_entry) [explicit; no main-topic leak]
- If plan_entry missing a field AND backend available: auto_resolve() fill
- pipeline.run(topic=entity, **kwargs, internal_subrun=True)
- Parallel ThreadPoolExecutor
- Collect per-entity Reports
- Attach resolved targeting to each Report.artifacts["resolved"]
scripts/last30days.py after fanout:
- If --save-dir: save each entity's Report as {entity-slug}-raw.md
Each file includes its own single-row Resolved Entities block
- emit_comparison_output → render_comparison_multi (merged stdout)
Includes full N-row Resolved Entities block
```
## Implementation Units
- [ ] **Unit 1: vs-topic detection routes to fanout (not single-pipeline)**
**Goal:** A topic containing ` vs ` / ` versus ` triggers `fanout.run_competitor_fanout` with the parsed entities. Each entity runs a full `pipeline.run()`. Replace the current single-pipeline-with-comparison-plan behavior.
**Requirements:** R1
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (main() — detect vs-topic, route to fanout)
- Modify: `scripts/lib/planner.py` (remove / bypass the `_should_force_deterministic_plan` special case for vs topics; vs topics no longer go through `plan_query` as a single comparison plan)
- Test: `tests/test_vs_mode_fanout.py` (new)
**Approach:**
- Parse the incoming topic: if it contains ` vs ` or ` versus ` (case-insensitive), split into entities (reuse `planner._comparison_entities`-style logic or move that utility into main()).
- When vs-entities are detected, route to the same fanout branch `--competitors` uses today. The entity list comes from the topic string; no discovery step needed.
- Each entity runs `pipeline.run()` with its own plan (either from `--competitors-plan[entity]` or from the engine's per-entity fallback path).
- For back-compat, if the user passes both a vs-topic AND `--plan`, honor `--plan` for the main (first) entity and use per-entity defaults for peers unless `--competitors-plan` is also provided.
**Execution note:** Start with an integration test that runs `"A vs B"` via mock mode and asserts fanout was called with two entities + two pipeline.run calls.
**Patterns to follow:**
- 3.0.11 fanout wiring in `scripts/last30days.py`'s `--competitors` branch.
- `planner._comparison_entities` for the split logic.
**Test scenarios:**
- Happy path: topic `"A vs B"` → two pipeline.run calls, two Reports returned, merged render.
- Happy path: topic `"A vs B vs C"` → three pipeline.run calls.
- Happy path: topic `"A versus B"` → matches the same regex, two pipelines.
- Edge case: topic `"OpenAI vs"` (trailing empty entity) → treated as single-entity `"OpenAI"`, not vs mode.
- Edge case: topic contains "vs." (dot, no trailing space) → existing regex tolerates it; verify.
- Edge case: topic `"A vs B"` plus `--plan` → plan applies to first entity only, peers use per-entity defaults.
- Integration: full vs-mode run end-to-end in mock mode; verify rendered output, stderr has one `[Competitors] Comparing: A vs B vs ...` line.
**Verification:**
- Test assertions pass.
- Mock-mode smoke of `/last30days "OpenAI vs Anthropic"` shows fanout invocation, per-entity Reports, merged comparison output.
- [ ] **Unit 2: `--competitors-plan` JSON flag + `_subrun_kwargs` helper + override-leak fix**
**Goal:** New JSON flag threads per-entity targeting into each sub-run's `pipeline.run()`. A `_subrun_kwargs(entity, plan_entry)` helper is the single source of truth for per-entity kwargs, eliminating override-leak.
**Requirements:** R3, R6
**Dependencies:** None (can land alongside or before Unit 1)
**Files:**
- Modify: `scripts/last30days.py` (argparse + parse + `_competitor_runner` + `_subrun_kwargs` helper)
- Possibly modify: `scripts/lib/fanout.py` (no signature change expected; the competitor_runner contract is unchanged)
- Test: `tests/test_cli_competitors.py` (extend)
- Test: `tests/test_competitors_plan_threading.py` (new)
- Test: `tests/test_competitor_subrun_isolation.py` (new, regression)
**Approach:**
- Add `--competitors-plan` argparse flag. Accepts inline JSON or file path (mirror `--plan`).
- Validation: top-level dict; each value is a dict; unknown fields log warnings; malformed input exits 2. Case-insensitive entity matching.
- Schema: `{entity_name: {x_handle?, x_related?, subreddits?, github_user?, github_repos?, context?}}`.
- Build `_subrun_kwargs(entity, plan_entry)` — returns an explicit dict with every per-entity flag. No closure-default fallthrough. This is the leak fix.
- `_competitor_runner(entity)`:
1. Get `plan_entry` from `--competitors-plan` if present.
2. Build base kwargs with `_subrun_kwargs(entity, plan_entry)`.
3. Fill missing fields via `resolve.auto_resolve(entity, entity_config)` only if backend is configured (3.0.12 fallback path).
4. Call `pipeline.run(topic=entity, internal_subrun=True, **kwargs)`.
5. Attach `resolved` dict to `report.artifacts`.
- Verify no per-entity flag from main() leaks via closure. The helper is the only source of per-entity values.
**Execution note:** Test-first for the override-leak regression. Use the Kanye 2026-04-22 receipt as the failing test input (main `--subreddits=Kanye,hiphopheads` + `--competitors-list "Drake"` → assert Drake's pipeline.run receives `subreddits=None`).
**Patterns to follow:**
- `--plan` parsing block in `scripts/last30days.py`.
- 3.0.12's `entity_config = dict(config)` deep-copy pattern.
**Test scenarios:**
- Happy path: `--competitors-plan '{"Drake":{"x_handle":"Drake","subreddits":["Drizzy"]}}'` → Drake's pipeline.run receives `x_handle="Drake"`, `subreddits=["Drizzy"]`. No auto_resolve call for Drake.
- Happy path: plan covers 2 of 3 entities, backend configured → covered skip auto_resolve; third falls back.
- Happy path: plan file path accepted like `--plan`.
- Happy path: case-insensitive entity match.
- Edge case: unknown fields → warn, ignore.
- Edge case: plan entry for entity not in list → warn, ignore.
- Error path: malformed JSON → exit 2.
- Error path: top-level JSON is list → exit 2.
- Regression (leak): main `--subreddits=A,B` + `--competitors-list "X"` + no plan → X's pipeline.run gets `subreddits=None`.
- Regression (leak): same for `--x-handle`, `--x-related`, `--tiktok-hashtags`, `--tiktok-creators`, `--ig-creators`, `--github-user`, `--github-repo`.
- Regression (leak): main `--x-handle=kanye` + plan `{"Drake":{"x_handle":"Drake"}}` → Drake's sub-run gets `x_handle="Drake"`, NOT `"kanye"`.
**Verification:**
- All regression tests pass.
- Smoke run (mock mode + plan): stderr shows per-entity `[Competitors] {entity}: x=... subs=...` line; no leak from main topic's flags.
- [ ] **Unit 3: Per-entity save files**
**Goal:** When `--save-dir` is set in a vs-mode or `--competitors` run, each entity's sub-run saves its own `{entity-slug}-raw.md` file — same format as a single-entity run would produce.
**Requirements:** R4, R5
**Dependencies:** Unit 1, Unit 2
**Files:**
- Modify: `scripts/last30days.py` (`save_output` iteration after fanout)
- Modify: `scripts/lib/render.py` (`render_full` includes single-row Resolved Entities block when that entity's `artifacts["resolved"]` is present)
- Test: `tests/test_save_raw_per_entity.py` (new)
**Approach:**
- After fanout completes, iterate `report.artifacts["competitor_reports"]` (or equivalent). For each `(entity, entity_report)`:
- Call `save_output(entity_report, emit="md", save_dir=args.save_dir, suffix=args.save_suffix)`.
- Uses entity's `slugify(entity)` for the filename. Same pattern a single-entity run uses.
- Each saved file invokes `render_full` (or the save-variant). `render_full` now checks for `report.artifacts["resolved"]` and prepends a single-row Resolved Entities block.
- Stderr logs one `[last30days] Saved output to <path>` line per entity.
- Single-entity runs unchanged (no extra files, render_full unchanged for them).
**Patterns to follow:**
- Existing `save_output` invocation in main() for single-entity runs.
- `slugify(topic)` for filename.
- 3.0.12's `_render_resolved_entities_block` (reused, single-row mode).
**Test scenarios:**
- Happy path: `/last30days "A vs B vs C" --save-dir=/tmp/x``/tmp/x/a-raw.md`, `/tmp/x/b-raw.md`, `/tmp/x/c-raw.md` exist.
- Happy path: `--competitors-list "Drake,Kendrick" --save-dir=/tmp/x` on topic Kanye → three files: `kanye-west-raw.md`, `drake-raw.md`, `kendrick-lamar-raw.md`.
- Happy path: each file includes a single-row Resolved Entities block for its entity.
- Happy path: single-entity run with `--save-dir` → one file, no Resolved block (unchanged).
- Edge case: `--save-suffix=v3` → all N files get the suffix.
- Edge case: one entity sub-run failed → its file is NOT saved; the others are.
- Integration: `ls {save-dir}/*-raw.md` returns N files after a vs-mode run.
**Verification:**
- Test assertions pass.
- Manual vs-mode smoke saves N files.
- [ ] **Unit 4: LAW 7-style stderr reframe + footer-nudge suppression**
**Goal:** The `--competitors`-with-no-backend stderr tells the hosting model to do Step 0.55 per entity and pass `--competitors-plan`. The BRAVE/SERPER footer nudge is suppressed when `--plan` or `--competitors-plan` is present.
**Requirements:** R7, R8
**Dependencies:** Unit 2 (flag must exist)
**Files:**
- Modify: `scripts/last30days.py` (the `[Competitors] --competitors requires...` stderr block)
- Modify: `scripts/lib/quality_nudge.py` (or wherever footer nudge emits; verify during implementation)
- Test: `tests/test_competitors_no_backend_message.py` (new)
- Test: `tests/test_footer_nudge_suppression.py` (new)
**Approach:**
- Rewrite stderr in this order:
1. "If you are the hosting reasoning model (Claude Code, Codex, Hermes, Gemini, or any agent with WebSearch), the recommended path: (a) discover N peers via WebSearch, (b) run Step 0.55 for main + each peer, (c) re-invoke as `/last30days 'topic vs peer1 vs peer2' --competitors-plan '{...}'`. See SKILL.md 'Competitor mode'."
2. "Headless / cron path: set BRAVE_API_KEY / EXA_API_KEY / SERPER_API_KEY / PARALLEL_API_KEY / OPENROUTER_API_KEY and re-run."
3. "Minimum escape hatch: `--competitors-list 'A,B,C'` skips discovery but does not pre-resolve peers."
- Suppress footer nudge when `external_plan` OR `competitors_plan` was passed.
**Test scenarios:**
- Happy path: `--competitors` with no backend, no list, no plan → stderr leads with "If you are the hosting reasoning model" and references `--competitors-plan` before naming API keys.
- Happy path: `--plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-plan` passed → footer nudge does NOT fire.
- Happy path: `--competitors-list` only (no plan, no backend) → footer nudge still fires (hosting model didn't fully engage).
- Happy path: no `--competitors`, no `--plan` → footer nudge unchanged.
**Verification:**
- Tests pass.
- [ ] **Unit 5: Polymarket disambiguation guard**
**Goal:** `--polymarket-keywords "kw1,kw2"` filters market matches; auto-skip Polymarket on single-token-ambiguous topics without override.
**Requirements:** R9
**Dependencies:** None
**Files:**
- Modify: `scripts/last30days.py` (argparse)
- Modify: `scripts/lib/polymarket.py`
- Test: `tests/test_polymarket_disambiguation.py` (new)
**Approach:**
- Add `--polymarket-keywords "kw1,kw2"`. When provided, Polymarket adapter filters market titles to those whose normalized text contains at least one keyword.
- Auto-skip: if topic is one token AND matches a known-ambiguous list (US state names, US city names, common sports/color/animal words) AND no `--polymarket-keywords`, skip Polymarket with stderr note.
- SKILL.md update (small): mention `--polymarket-keywords` in Step 0.55 instructions for ambiguous topics.
**Test scenarios:**
- Happy path: topic "Warriors", no override → Polymarket skipped; stderr note.
- Happy path: topic "Warriors", `--polymarket-keywords "nba,gsw"` → Polymarket runs, filtered.
- Happy path: topic "OpenAI" → Polymarket runs as before.
- Happy path: topic "Arizona Wildcats" (multi-token) → Polymarket runs as before.
- Edge case: `--polymarket-keywords ""` → treated as empty, no filter.
**Verification:**
- Warriors smoke → Polymarket footer absent or filtered.
- [ ] **Unit 6: SKILL.md rewrite — vs mode is the canonical path, `--competitors` is a shortcut**
**Goal:** SKILL.md documents the unified architecture. vs mode runs N full passes. `--competitors` is a SKILL.md-level shortcut that discovers 2 peers and invokes vs mode with `--competitors-plan`.
**Requirements:** R1, R2, R10 (surfaces them)
**Dependencies:** Units 1-4
**Files:**
- Modify: `SKILL.md` (§551 "If QUERY_TYPE = COMPARISON" rewrite; Competitor mode subsection rewrite)
- Modify: `README.md` (one-line example)
**Approach:**
- Rewrite §551 to describe the N-pass architecture: "When the user asks 'X vs Y' (or 'X vs Y vs Z'), run Step 0.55 per entity, then invoke the engine. The engine fans out N full pipelines in parallel. Each entity gets its own single-entity-grade coverage. Wall clock is close to a single run."
- Remove the "ONE research pass with a comparison-optimized plan that replaces the old 3-pass approach" language.
- Add a `--competitors-plan` JSON example.
- Rewrite the Competitor mode subsection: "`--competitors` is a shortcut. The hosting model: (1) runs WebSearch to discover N=2 peers, (2) runs Step 0.55 for main + each peer, (3) rewrites topic to `'main vs peer1 vs peer2'`, (4) invokes engine with `--competitors-plan '{...}'`. Engine flag `--competitors` and `--competitors-list` remain for headless fallback."
- Cross-reference §679 (per-entity Step 0.55 protocol).
- Warning: a thin `## Resolved Entities` block (dashes for any entity) means the hosting model skipped Step 0.55 for that one.
**Patterns to follow:**
- Existing §679 per-entity Step 0.55 protocol for tone.
- 3.0.12 Competitor mode prose for terseness.
**Test scenarios:**
- Test expectation: none — documentation. Verification is dogfood.
**Verification:**
- `/last30days "OpenAI vs Anthropic vs xAI"` in a fresh Claude Code window produces 3 save files with populated Resolved blocks and non-dash per-entity targeting.
- `/last30days OpenAI --competitors` produces same after discovery step.
- [ ] **Unit 7: Version 3.0.13, CHANGELOG, sync, hot-copy**
**Goal:** Ship 3.0.13 to all local targets.
**Requirements:** Closes R1-R10
**Dependencies:** Units 1-6
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`
**Approach:**
- CHANGELOG: group the changes. "Changed: vs mode now runs N full passes in parallel, one per entity — reverting the one-pass optimization to restore per-entity depth. Added: --competitors-plan JSON for per-entity Step 0.55 targeting (applies to vs mode and --competitors). Changed: --competitors is now a SKILL.md shortcut for vs-with-discovery. Added: per-entity *-raw.md save files. Fixed: override-leak from main to peer sub-runs. Changed: LAW 7 stderr framing for hosting-model context. Changed: BRAVE/SERPER footer nudge suppressed when --plan / --competitors-plan present. Added: --polymarket-keywords + auto-skip for ambiguous topics."
- Beta channel first per CLAUDE.md.
- Hot-copy so public `/last30days` picks up 3.0.13.
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.13.
- `sync.sh` exits 0.
- Hot-copy contains the new files.
## System-Wide Impact
- **Interaction graph:** vs-mode and `--competitors` share one orchestrator (`fanout.run_competitor_fanout`). `_subrun_kwargs` is the single source of per-entity kwargs. Save loop iterates per entity.
- **Error propagation:** Per-entity sub-run failure → logged, dropped, continue (3.0.11 behavior unchanged). `--competitors-plan` JSON parse errors exit 2 (same shape as `--plan`).
- **State lifecycle risks:** `entity_config = dict(config)` deep-copy pattern extends to every per-entity flag (Unit 2 fix). No cross-entity context leak.
- **API surface parity:** `--competitors-plan` is additive. `--competitors`, `--competitors-list`, `--plan` unchanged. `--polymarket-keywords` additive. vs-mode keeps its topic-string surface.
- **Integration coverage:** New vs-mode-fanout integration test. New override-leak regression test. New plan-threading test. New nudge-suppression test. New per-entity-save test. New Polymarket disambiguation test.
- **Unchanged invariants:** `pipeline.run()` signature unchanged. Single-entity render path unchanged. LAW 7 on the default path unchanged (still fires when a single-entity run lacks `--plan`).
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| vs-mode N-pass latency feels slower for users who remember the one-pass shortcut. | Parallel execution keeps wall-clock ~= max(per-entity-latency), not sum. `--quick` on a vs-topic still applies to each sub-run. CHANGELOG calls out the revert + parallelism. |
| API cost scales linearly with N (per source). | Default count 2 caps it. Hard max 6 on `--competitors`. vs-mode users opted into N entities explicitly. |
| Rivalry content ("A vs B" articles) missed in N-independent passes. | Deferred to separate task (head-to-head supplemental pass). Start shipping and observe whether this is actually a gap. |
| Hosting model skips `--competitors-plan` and uses `--competitors-list` only. | Unit 4 stderr reframe steers explicitly. SKILL.md Unit 6 makes the plan-path canonical. Thin Resolved block in output makes skipped-Step-0.55 visible. |
| Override-leak fix misses a subtle closure path. | Unit 2 is test-first with the Kanye receipt as the failing input. Regression test asserts every per-entity flag is None unless plan provides it. |
## Documentation / Operational Notes
- Beta channel first per CLAUDE.md.
- After merge: hot-copy to `~/.claude/plugins/cache/last30days-skill/last30days/3.0.13/`.
- CHANGELOG explicitly frames the vs-mode change as an architectural revert-with-parallelism, not a regression to the old serial N-pass.
## Sources & References
- Superseded plan: `docs/plans/2026-04-22-004-fix-competitors-hosting-model-resolve-and-leak-plan.md.superseded`
- Previous plan (3.0.12): `docs/plans/2026-04-22-003-fix-competitors-per-entity-resolution-plan.md`
- Initial plan (3.0.11): `docs/plans/2026-04-22-002-feat-competitors-flag-comparison-fanout-plan.md`
- 2026-04-22 test session receipts (Warriors, Seattle, Arizona Wildcats, Kanye West)
- SKILL.md §551 + §679 — the per-entity Step 0.55 protocol the hosting model uses for both paths
- Related code: `scripts/lib/fanout.py`, `scripts/last30days.py` `_competitor_runner`, `scripts/lib/planner.py` vs-topic special-case, `scripts/lib/render.py` `_render_resolved_entities_block`, `scripts/lib/polymarket.py`, `scripts/lib/quality_nudge.py`
- Related PRs: #308 (3.0.11), #309 (3.0.12)
@@ -1,90 +0,0 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: comparison title says (/Last30Days) instead of (Last 30 Days)"
type: fix
status: active
date: 2026-04-22
---
# fix: comparison title says (/Last30Days) instead of (Last 30 Days)
## Overview
User feedback 2026-04-22 on the 3.0.13 release runs (Kanye vs Drake, Mercer Island, Figma): the comparison title currently reads `# Kanye West vs Drake: What the Community Says (Last 30 Days)`. It should read `# Kanye West vs Drake: What the Community Says (/Last30Days)` — attributing the output to the slash command rather than describing the date range generically.
Single-line change in SKILL.md, three occurrences. No code change.
## Requirements Trace
- R1. Comparison title pattern in SKILL.md changes from `(Last 30 Days)` to `(/Last30Days)` so synthesis outputs read `... What the Community Says (/Last30Days)`.
- R2. Both the rule statement (line 113) and the COMPARISON-exception statement (line 131) and the synthesis template example (line 1208) all use the new suffix.
- R3. Version bumps to 3.0.14, CHANGELOG entry, sync, hot-copy. Public cache picks up the new title pattern.
## Scope Boundaries
- No changes to the single-entity output title (no `(/Last30Days)` suffix there — only comparison topics carry it).
- No changes to engine code. Pure SKILL.md content.
- No changes to anything else surfaced in the test runs.
## Key Technical Decisions
- **Replace all three occurrences of the suffix string in one pass.** They are identical strings; changing one without the others would cause synthesis-time confusion when the model reaches a different reference.
- **Ship as 3.0.14, not 3.0.13.x.** Patch-level bump matches the small scope and keeps the release log clean.
## Implementation Units
- [ ] **Unit 1: Replace `(Last 30 Days)``(/Last30Days)` in SKILL.md**
**Goal:** All three SKILL.md references to the comparison title use the new suffix.
**Requirements:** R1, R2
**Files:**
- Modify: `SKILL.md`
**Approach:**
- `replace_all` swap of `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Three occurrences, no other strings overlap.
**Test scenarios:**
- Test expectation: none — pure documentation. Verification by inspection + dogfood run.
**Verification:**
- `grep -c "What the Community Says (/Last30Days)" SKILL.md` returns 3.
- `grep -c "What the Community Says (Last 30 Days)" SKILL.md` returns 0.
- [ ] **Unit 2: Version 3.0.14 + CHANGELOG + sync + hot-copy**
**Goal:** Ship 3.0.14 to all local targets.
**Requirements:** R3
**Dependencies:** Unit 1
**Files:**
- Modify: `.claude-plugin/plugin.json`
- Modify: `CHANGELOG.md`
- Run: `bash scripts/sync.sh`
- Hot-copy: `~/.claude/plugins/cache/last30days-skill/last30days/3.0.14/`
**Approach:**
- CHANGELOG: "Changed: comparison-mode title attribution — `What the Community Says (Last 30 Days)``What the Community Says (/Last30Days)`. Surfaces the slash-command identity instead of restating the date range."
**Test scenarios:**
- Test expectation: none — packaging.
**Verification:**
- `grep version .claude-plugin/plugin.json` → 3.0.14.
- Hot-copy contains the updated SKILL.md.
## Risks & Dependencies
| Risk | Mitigation |
|------|------------|
| Hosting model has the old title pattern memorized from a prior run and re-emits `(Last 30 Days)`. | SKILL.md is read top-to-bottom each invocation. STEP 0 canonical-path self-check (3.0.12) ensures the model loads the new SKILL.md, not the marketplace stale copy. |
## Sources & References
- 2026-04-22 dogfood runs (Kanye West vs Drake, Mercer Island --competitors, Figma --competitors)
- Related code: `SKILL.md` lines 113, 131, 1208
@@ -1,388 +0,0 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -1,310 +0,0 @@
# V1 vs V2 Comparison Analysis
**Date:** 2026-02-06
**Queries tested:** 4 (1 head-to-head, 3 V1-only)
**Scope:** Quick smoke test, not full 17-query matrix
---
## Part 1: Head-to-Head -- "kanye west" (NEWS Query)
### Dimension-by-Dimension Scoring
#### 1. Query Parsing Display
Does it show the `🔍 **{TOPIC}** · {QUERY_TYPE}` line before running tools?
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 1 | No parsing display at all. Output starts with "## What I learned:" -- jumps straight into synthesis. No acknowledgment of topic or query type before research. |
| V2 | 1 | No parsing display either. Output starts with "Here's what I found:" then "## What I learned:" -- same problem as V1. |
**Analysis:** Neither version actually rendered the query parsing display. V2 SKILL.md explicitly requires `🔍 **kanye west** · News` before any tools run, but the agent did not produce it. This is a V2 instruction that failed to land. Both score 1/5.
Possible cause: The parsing display is supposed to appear *before* tools are called -- it may have been shown during execution but not captured in the final output text. If so, both outputs represent only the post-research synthesis, not the full session. Regardless, based on what is in the output files, neither shows it.
---
#### 2. Source Coverage (Reddit/X/Web counts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | `Reddit: 0 relevant threads` / `X: 30 posts │ ~10 likes` / `Web: 20+ pages`. Two of three sources returned results. Reddit was zero. |
| V2 | 3 | `Reddit: 0 threads (no results this cycle)` / `X: 29 posts │ 33 likes │ 14 reposts` / `Web: 30+ pages`. Same pattern: two of three returned results. |
**Analysis:** Nearly identical coverage. Both got zero Reddit results (likely a script/API issue for this topic, not a SKILL.md problem). V2 has slightly more precise X metrics (33 likes, 14 reposts vs. V1's vague "~10 likes"). V2 has more web pages (30+ vs 20+). Both miss the 10+ Reddit threshold for a score of 4+.
---
#### 3. Citation Quality (sparse vs every-sentence)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 2 | No inline citations at all. The body text makes claims ("full-page Wall Street Journal apology," "Hellwatt Festival in Italy") but never attributes them to a specific source. The stats box lists "Washington Post, Billboard, AllHipHop" but the body has zero `per @handle` or `per Rolling Stone` attributions. |
| V2 | 5 | Every bold section ends with a sparse, clean citation. Examples: `"per Rolling Stone"`, `"per The Washington Post"`, `"per Billboard"`, `"per AllHipHop"`, `"per The News International"`. One citation per topic, never chained. Exactly what V2 SKILL.md specifies. |
**Analysis:** This is the single biggest quality gap between V1 and V2. V1's output reads like a Wikipedia summary -- informative but ungrounded. V2 reads like a researched briefing where every claim has a named source. V2 nails the "sparse citation" rule from its SKILL.md: `"cite 1 source per pattern, short format: 'per @handle' or 'per r/sub'"`.
V1 quote (no citation): `"He'll headline the new Hellwatt Festival in Italy (July 4-18, 2026)."`
V2 quote (cited): `"Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard."`
---
#### 4. Summary Structure (bold topic headers, organized sections)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Has a coherent narrative structure with a paragraph of synthesis, then a `**KEY THEMES:**` numbered list. But the opening is a single dense paragraph, not broken into scannable sections with bold headers. |
| V2 | 5 | Each storyline gets its own bold header: `**BULLY Album — March 20, 2026 via Gamma**`, `**Public Apology for Antisemitism**`, `**Hellwatt Festival in Italy**`, `**Health Concerns**`, `**Grammys Ban**`, `**Kim & Lewis Hamilton Buzz**`. Each is a standalone scannable unit with 1-3 sentences. |
**Analysis:** V2 follows the SKILL.md template exactly: `**{Topic 1}** — [1-2 sentences, per source]`. V1 uses a blob + list approach which is readable but less scannable. V2 is notably better for a user who wants to skim and find the story they care about.
V1 structure: 1 dense paragraph -> 5-item `KEY THEMES` list
V2 structure: 6 bold topic cards, each self-contained -> no KEY THEMES list (but doesn't need one because the structure itself is the organization)
---
#### 5. Stats Box Format (emoji tree vs plain text)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Uses `├─` tree format with emoji: `├─ 🟠 Reddit: 0 relevant threads` / `├─ 🔵 X: 30 posts` / `├─ 🌐 Web: 20+ pages` / `└─ Top voices:`. Minor deviation: says "0 relevant threads (filtered out noise)" instead of the V1 SKILL.md template "0 threads (no results this cycle)". Also omits the `🗣️` emoji on the Top voices line. |
| V2 | 5 | Perfect match to V2 SKILL.md template: `├─ 🟠 Reddit: 0 threads (no results this cycle)` / `├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)` / `├─ 🌐 Web: 30+ pages │ rollingstone.com, ...` / `└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex`. Includes `(via xAI)` notation, `🗣️` emoji, @handles with engagement counts. |
**Analysis:** V2 is tighter and matches its template exactly. V1 is close but has minor deviations (custom "filtered out noise" text, missing `🗣️` emoji, no @handles or engagement counts on Top voices). V2's inclusion of actual @handles with like counts (`@honest30bgfan_ (33 likes)`) adds credibility.
---
#### 6. Research Grounding (actual research vs generic knowledge)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 4 | Clearly grounded: mentions specific details like "Wall Street Journal apology (Jan 26, 2026)," "four-month-long manic episode," "frontal-lobe brain injury," "North West collaborated on 'Piercings on My Hand,'" "Monumental Plaza de Toros." These are specific enough to be from research, not pre-training. Minor generic leakage: the "KEY THEMES" list uses editorial framing ("Accountability arc," "Mental health transparency") that feels more like analysis than research extraction. |
| V2 | 5 | Every fact is specific and attributed: "12th studio album," "13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign," "earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded," "103,000-capacity RCF Arena." The AI-deepfaked vocals detail is a standout -- it is clearly from research, not something a model would know from pre-training. The Kim/Lewis Hamilton item (`"X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton"`) is explicitly sourced from X data, not general knowledge. |
**Analysis:** Both are well-grounded, but V2 has more "could only come from research" details. The deepfaked vocals story, the exact venue capacity, and the explicit X chatter observation are details that prove the synthesis is from the research output, not hallucinated.
---
#### 7. Prompt Quality (invitation to share vision, not dumping prompts)
| Version | Score | Evidence |
|---------|-------|----------|
| V1 | 3 | Ends with: `"Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in."` This is a follow-up invitation, but it is NOT the SKILL.md-specified invitation. It is topic-specific and conversational, which is nice, but it does not ask the user to "share your vision for what you want to create." It misses the prompt-generation angle entirely. |
| V2 | 5 | Ends with exactly: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice."` This matches the V2 SKILL.md template verbatim. It positions the skill correctly: not a news summarizer but a research-to-prompt pipeline. |
**Analysis:** V1's closing is friendly but off-brand. It treats the skill as a research tool, not a research-to-prompt tool. V2 correctly frames the next step as "tell me what to create and I'll write the prompt." This is a meaningful difference -- V1 would leave a user thinking they just got a summary, while V2 primes them to get a usable output.
---
### Head-to-Head Scorecard
| Dimension | V1 | V2 | Winner |
|-----------|----|----|--------|
| 1. Query Parsing Display | 1 | 1 | Tie (both failed) |
| 2. Source Coverage | 3 | 3 | Tie |
| 3. Citation Quality | 2 | 5 | **V2 (+3)** |
| 4. Summary Structure | 3 | 5 | **V2 (+2)** |
| 5. Stats Box Format | 4 | 5 | **V2 (+1)** |
| 6. Research Grounding | 4 | 5 | **V2 (+1)** |
| 7. Prompt Quality (invitation) | 3 | 5 | **V2 (+2)** |
| **TOTAL** | **20/35** | **29/35** | **V2 wins by 9 points** |
**V2 is clearly better.** The biggest gaps are citation quality (+3) and summary structure (+2). V2's output reads like a professional research briefing; V1's reads like a decent but unstructured summary.
---
## Part 2: V1-Only Outputs Analysis
### Output 1: "open claw" (GENERAL query)
**What V1 does well:**
- Strong research grounding. Mentions exact numbers: "145,000+ GitHub stars," "20,000+ forks," "700+ skills," "341 malicious skills." These are clearly from research.
- The KEY PATTERNS section is excellent: 5 well-organized patterns with community quotes (`"I give it sudo and let it configure everything"` vs `"prompt injection is terrifying when you give the bot access to your actual bank account"`).
- Good synthesis of the security vs. enthusiasm tension -- captures the community split accurately.
- Stats box uses the emoji tree format correctly with `├──` (though note: uses double-dash `──` instead of single `─`, minor inconsistency).
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display (`🔍 **open claw** · General`).
- No inline citations in the body text. The 5 KEY PATTERNS have no `per @handle` or `per r/sub` attribution. Which Reddit thread said "I give it sudo"? Which X post raised the security concern? We do not know.
- The stats box says `├── 🟠 Reddit: 25 threads │ ~750+ upvotes` -- the tilde and plus are imprecise. V2 SKILL.md wants exact parsed numbers.
- Top voices line lists subreddits and handles but no engagement counts: `@grok, @Starlink` -- are these the highest-engagement handles? No like counts shown.
- No bold topic headers in the body -- it is a single paragraph followed by a numbered list, not the `**{Topic}** — sentence, per source` format V2 requires.
**V1 Score (estimated):** 22/35
---
### Output 2: "nano banana pro prompting" (PROMPTING query)
**What V1 does well:**
- Correctly identifies two prompting styles (JSON structured vs. natural language "Creative Director") and explains when each works best. This is excellent PROMPTING-type synthesis.
- KEY PATTERNS are specific and actionable: "85mm lens at f/1.8," "three-point lighting with key at 45 degrees," "text rendering works -- keep text under 3 words for best results (75% success rate)." These are concrete tips a user can apply immediately.
- Research grounding is strong: cites specific upvote counts ("149-259 upvotes"), subreddit names (`r/nanobanana2pro`), and the Google AI blog.
- The invitation correctly targets Nano Banana Pro: `"Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro."`
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Stats box uses plain text dashes: `- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments` instead of the tree format `├─ 🟠 Reddit:`. Uses `|` pipe instead of `│` box-drawing character. V2 SKILL.md explicitly says: "NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji."
- No inline body citations. KEY PATTERNS mention Reddit upvote ranges but no specific `per @handle` attributions.
- Missing `✅ All agents reported back!` header -- just says "All agents reported back!" without the checkmark.
- Body structure is paragraph + numbered list, not bold topic headers.
**V1 Score (estimated):** 23/35 (slightly higher than open claw due to better actionability)
---
### Output 3: "how to best setup clawdbot" (HOW-TO query)
**What V1 does well:**
- This is the best V1 output of the batch. It goes beyond synthesis and actually delivers a **Quick-Start guide** with numbered steps, a **Security Hardening** checklist, and a **Budget Option** -- all grounded in research.
- Excellent research grounding: `"per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0"` -- this is an actual citation with an @handle!
- Specific, actionable recommendations: exact commands (`curl -fsSL https://clawd.bot/install.sh | bash`), specific model recommendations (Claude Opus 4.5 for best results, GLM 4.7 Flash for local), specific channel advice (Telegram first, WhatsApp QR code fails).
- Stats box is correct emoji tree format with engagement counts: `@aashatwt (452 likes), @recap_david (329 likes)`.
- Captures the naming confusion accurately: "Clawdbot -> Moltbot -> OpenClaw."
**What V1 is missing (per V2 SKILL.md features):**
- No query parsing display.
- Body text has no inline citations except the Budget Option section. The 5 KEY PATTERNS have no `per @handle` attribution.
- Bold topic headers are used only in the Quick-Start and Security sections, not in the KEY PATTERNS or intro.
- The output delivers the "answer" directly (setup guide) rather than waiting for the user's vision and offering to write a prompt. For a HOW-TO query this might be the right call, but it skips the SKILL.md flow of "show research -> invite vision -> write prompt."
**V1 Score (estimated):** 26/35 (best of the V1 outputs)
---
### Patterns Across All V1 Outputs
**Consistent strengths:**
1. Research grounding is solid across all three. V1 does not hallucinate -- the facts are clearly from the research output, not pre-training.
2. KEY PATTERNS lists are consistently useful and actionable.
3. Stats boxes are present in all outputs (though formatting varies).
4. The invitation/closing line is present in all outputs.
**Consistent weaknesses:**
1. **No query parsing display** in any output (0 for 4, including Kanye West).
2. **No inline citations** in the body text (except one @handle in the clawdbot output). The research feels real but is unattributed.
3. **Stats box formatting is inconsistent.** Open claw uses `├──` (double dash), nano banana pro uses `- 🟠` (plain dash + pipe), clawdbot uses `├─` (correct). Three different formats in three outputs.
4. **Body structure defaults to paragraph + numbered list** instead of bold topic headers. Only clawdbot partially uses bold headers (in the guide section, not the research section).
5. **No `(via Bird/xAI)` notation** on X stats in any output.
---
## Part 3: SKILL.md Feature Diff
### Features in V2 but NOT V1
| Feature | V2 Lines | Impact |
|---------|----------|--------|
| **Query parsing display** (`🔍 **{TOPIC}** · {QUERY_TYPE}`) | 40-53 | HIGH -- confirms to user the skill understood their request before spending time on research. |
| **Sparse citation rules** with BAD/GOOD examples | 186-193 | HIGH -- this is the #1 quality differentiator in the Kanye head-to-head. `"per @handle"` format, never chain multiple citations. |
| **Bold topic headers** template (`**{Topic 1}** — [1-2 sentences, per source]`) | 195-208 | HIGH -- makes output scannable. |
| **Strict stats template** with "NEVER use plain text dashes" instruction | 217-230 | MEDIUM -- prevents the formatting inconsistency seen across V1 outputs. |
| **RECOMMENDATIONS source attribution** (each item MUST have Sources: line with @handles) | 178-182 | MEDIUM -- only affects RECOMMENDATIONS queries. |
| **Reddit 0 results handling** (explicit instruction for what to write) | 229 | LOW -- edge case, but prevents ad-hoc text like V1's "filtered out noise." |
| **Bird CLI / xAI notation** in stats | 223 | LOW -- cosmetic transparency about data source. |
| **Step 2 phrasing: "DO WEBSEARCH WHILE SCRIPT RUNS"** | 71-73 | LOW -- execution optimization, no output impact. |
### Features in V1 but NOT V2
| Feature | V1 Lines | Impact | Should Restore? |
|---------|----------|--------|-----------------|
| **Use cases block** (4 examples in intro) | 12-17 | LOW | No |
| **Setup Check section** (3 modes, bash script, "keys are OPTIONAL") | 50-78 | MEDIUM for new users | Yes, for public release |
| **BAD/GOOD synthesis anti-pattern examples** | 172-191 | MEDIUM-HIGH | YES |
| **Self-check instruction** ("Re-read your 'What I learned' section...") | 269 | MEDIUM | YES |
| **Quality Checklist** (5-point checklist before delivering prompt) | 306-324 | HIGH | YES |
| **Prompt format anti-pattern** ("Research says JSON but you write prose") | 302 | MEDIUM | YES |
| **"IF USER ASKS FOR MORE OPTIONS"** section | 327-329 | LOW-MEDIUM | YES |
| **Web-only mode stats template + promo** | 248-259 | MEDIUM for no-key users | For public release |
| **TARGET_TOOL question template** (4 options) | 272-280 | LOW | No |
| **Context Memory: explicit "don't re-search" instructions** | 342-358 | MEDIUM | YES |
| **Output footer emoji + engagement counts** | 366-380 | LOW | YES |
### Features in BOTH (Shared)
| Feature | Notes |
|---------|-------|
| Parse User Intent (TOPIC, TARGET_TOOL, QUERY_TYPE) | Same 4 query types, same detection logic |
| "Don't ask about tool before research" rule | Identical |
| Research script execution command | Same `python3` command |
| WebSearch queries by QUERY_TYPE | Same search strategies |
| "Use user's exact terminology" instruction | V2 shorter but same intent |
| Judge Agent synthesis logic | Same 5-step weighting process |
| "Ground in actual research" instruction | Same core instruction, V1 has more examples |
| RECOMMENDATIONS: extract specific names | Same logic |
| Prompt format matching | Same instruction |
| Wait for user's vision | Same |
| Write ONE perfect prompt | Same structure |
| Context Memory | V2 shorter version |
| Output summary footer | Both have it, V1 has emoji |
| Depth options (quick/default/deep) | Same |
| "After each prompt: Stay in Expert Mode" | Same |
### Overall Assessment
**V2 is a clear upgrade in output formatting and citation quality.** The three features V2 adds (query parsing display, sparse citation rules, bold topic headers) directly address the three biggest weaknesses seen across all V1 outputs. The Kanye West head-to-head proves it: V2 scores 29/35 vs V1's 20/35.
**However, V2 dropped several quality guardrails from V1** that do not affect formatting but affect *correctness*: the self-check instruction, the anti-pattern examples, the quality checklist for prompts, and the "don't re-search" context memory rule. These are cheap to restore (under 25 lines total) and protect against subtle failure modes that may not show up in a 1-query test but will appear over dozens of uses.
---
## Part 4: Verdict
### Ship V2 or Not?
**Ship V2 -- but restore the guardrails first.**
V2 is unambiguously better on every formatting dimension. The citation quality improvement alone (V1: 2/5 -> V2: 5/5) makes it worth shipping. The bold topic headers and strict stats template fix the inconsistency problems visible across all V1 outputs.
But V2 dropped 6 guardrail features from V1 that cost almost nothing to include and protect against real failure modes. These should be restored before V2 goes public.
### Remaining Gaps
**Must fix before shipping (affects correctness):**
1. **Restore the quality checklist for prompts.** This is the test plan's #1 priority item. V1 had a 5-point checklist; V2 reduced it to one line. The checklist is what makes prompts feel polished -- it is the "that's a great prompt" mechanism. Add 8 lines.
2. **Restore BAD/GOOD anti-pattern examples.** V2 says "ground in actual research" but does not show what *bad* grounding looks like. V1's ClawdBot/Claude Code conflation example is exactly the kind of concrete negative example that prevents real failures. Add 5 lines.
3. **Restore self-check instruction.** One sentence: "Re-read your 'What I learned' section -- does it match what the research ACTUALLY says?" Zero cost, catches hallucination. Add 2 lines.
4. **Restore "don't re-search" context memory rule.** V2 only says "only do new research if user asks about a DIFFERENT topic." V1 explicitly bans re-searching and tells the agent to answer from existing research. Add 3 lines.
**Should fix (polish):**
5. Restore prompt format anti-pattern ("Research says JSON but you write prose"). Add 2 lines.
6. Restore "IF USER ASKS FOR MORE OPTIONS" section. Add 2 lines.
7. Add emoji + engagement counts back to the output summary footer. Edit 3 lines.
**Skip for now:**
8. Setup Check section -- add back for public release, not needed for execution.
9. Web-only mode stats template -- lower priority, most testers have API keys.
10. TARGET_TOOL question template -- agent handles this naturally.
### Query Parsing Display: Investigate
Both V1 and V2 scored 1/5 on query parsing display. V2 has the feature in its SKILL.md but the agent did not render it in the captured output. This could mean:
- The display was shown during execution but not captured (likely -- it appears before tools run, and the output files may only contain post-research content).
- The instruction is not strong enough and the agent skips it.
**Recommendation:** Verify in a live session whether the parsing display actually appears. If it does not, strengthen the instruction (e.g., "This line MUST be the first thing you output, before any tool calls").
### Total Effort
Restoring all 7 priority items: approximately 25 lines added to V2 SKILL.md. Under 15 minutes of work. The V2 formatting wins are substantial and proven; the V1 guardrails are small and proven. Combining both produces the best version.
### Final Score Summary
| | V1 (Kanye) | V2 (Kanye) | Delta |
|--|-----------|-----------|-------|
| Total | 20/35 | 29/35 | **V2 +9** |
| | V1 (Open Claw) | V1 (Nano Banana) | V1 (Clawdbot) | V1 Average |
|--|---------------|-----------------|--------------|------------|
| Estimated Total | 22/35 | 23/35 | 26/35 | **23.7/35** |
V2 at 29/35 beats every V1 output, including V1's best (clawdbot at 26/35).
**Decision: Ship V2 with guardrails restored.**
@@ -1,388 +0,0 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: "[topic] for [tool]" or "[topic]"
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
Use cases:
- **Prompting**: "photorealistic people in Nano Banana Pro", "Midjourney prompts", "ChatGPT image generation" → learn techniques, get copy-paste prompts
- **Recommendations**: "best Claude Code skills", "top AI tools" → get a LIST of specific things people mention
- **News**: "what's happening with OpenAI", "latest AI announcements" → current events and updates
- **General**: any topic you're curious about → understand what the community is saying
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
---
## Setup Check
The skill works in three modes based on available API keys:
1. **Full Mode** (both keys): Reddit + X + WebSearch - best results with engagement metrics
2. **Partial Mode** (one key): Reddit-only or X-only + WebSearch
3. **Web-Only Mode** (no keys): WebSearch only - still useful, but no engagement metrics
**API keys are OPTIONAL.** The skill will work without them using WebSearch fallback.
### First-Time Setup (Optional but Recommended)
If the user wants to add API keys for better results:
```bash
mkdir -p ~/.config/last30days
cat > ~/.config/last30days/.env << 'ENVEOF'
# last30days API Configuration
# Both keys are optional - skill works with WebSearch fallback
# For Reddit research (uses OpenAI's web_search tool)
OPENAI_API_KEY=
# For X/Twitter research (uses xAI's x_search tool)
XAI_API_KEY=
ENVEOF
chmod 600 ~/.config/last30days/.env
echo "Config created at ~/.config/last30days/.env"
echo "Edit to add your API keys for enhanced research."
```
**DO NOT stop if no keys are configured.** Proceed with web-only mode.
---
## Research Execution
**IMPORTANT: The script handles API key detection automatically.** Run it and check the output to determine mode.
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Show a promo banner if keys are missing (this is intentional marketing)
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
**Step 2: Check the output mode**
The script output will indicate the mode:
- **"Mode: both"** or **"Mode: reddit-only"** or **"Mode: x-only"**: Script found results, WebSearch is supplementary
- **"Mode: web-only"**: No API keys, Claude must do ALL research via WebSearch
**Step 3: Do WebSearch**
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- If user says "ChatGPT image prompting", search for "ChatGPT image prompting"
- Do NOT add "DALL-E", "GPT-4o", or other terms you think are related
- Your knowledge may be outdated - trust the user's terminology
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Step 3: Wait for background script to complete**
Use TaskOutput to get the script results before proceeding to synthesis.
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords? THIS IS CRITICAL.
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
**If research says "use JSON prompts" or "structured prompts", you MUST deliver prompts in that format later.**
---
## THEN: Show Summary + Invite Vision
**CRITICAL: Do NOT output any "Sources:" lists. The final display should be clean.**
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned:
```
🏆 Most mentioned:
1. [Specific name] - mentioned {n}x (r/sub, @handle, blog.com)
2. [Specific name] - mentioned {n}x (sources)
3. [Specific name] - mentioned {n}x (sources)
4. [Specific name] - mentioned {n}x (sources)
5. [Specific name] - mentioned {n}x (sources)
Notable mentions: [other specific things with 1-2 mentions]
```
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
```
What I learned:
[2-4 sentences synthesizing key insights FROM THE ACTUAL RESEARCH OUTPUT.]
KEY PATTERNS I'll use:
1. [Pattern from research]
2. [Pattern from research]
3. [Pattern from research]
```
**THEN - Stats (right before invitation):**
For **full/partial mode** (has API keys):
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {n} threads │ {sum} upvotes │ {sum} comments
├─ 🔵 X: {n} posts │ {sum} likes │ {sum} reposts
├─ 🌐 Web: {n} pages │ {domains}
└─ Top voices: r/{sub1}, r/{sub2} │ @{handle1}, @{handle2} │ {web_author} on {site}
```
For **web-only mode** (no API keys):
```
---
✅ Research complete!
├─ 🌐 Web: {n} pages │ {domains}
└─ Top sources: {author1} on {site1}, {author2} on {site2}
💡 Want engagement metrics? Add API keys to ~/.config/last30days/.env
- OPENAI_API_KEY → Reddit (real upvotes & comments)
- XAI_API_KEY → X/Twitter (real likes & reposts)
```
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
**Use real numbers from the research output.** The patterns should be actual insights from the research, not generic advice.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If the research was about ClawdBot (a self-hosted AI agent), your summary should be about ClawdBot, not Claude Code. If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**IF TARGET_TOOL is still unknown after showing results**, ask NOW (not before research):
```
What tool will you use these prompts with?
Options:
1. [Most relevant tool based on research - e.g., if research mentioned Figma/Sketch, offer those]
2. Nano Banana Pro (image generation)
3. ChatGPT / Claude (text/code)
4. Other (tell me)
```
**IMPORTANT**: After displaying this, WAIT for the user to respond. Don't dump generic prompts.
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
When they respond with their vision (e.g., "I want a landing page mockup for my SaaS app"), THEN write a single, thoughtful, tailored prompt.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT:**
- Research says "JSON prompts" → Write the prompt AS JSON
- Research says "structured parameters" → Use structured key: value format
- Research says "natural language" → Use conversational prose
- Research says "keyword lists" → Use comma-separated keywords
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS - if research said JSON, this is JSON. If research said natural language, this is prose. Match what works.]
---
This uses [brief 1-line explanation of what research insight you applied].
```
### Quality Checklist:
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
- **If they ask a question** - answer it from your research findings
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
For **full/partial mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
For **web-only mode**:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} web pages from {domains}
Want another prompt? Just tell me what you're creating next.
💡 Unlock Reddit & X data: Add API keys to ~/.config/last30days/.env
```
@@ -1,25 +0,0 @@
## What I learned:
The last 30 days have been one of Kanye West's most significant periods in years. The biggest story is his **full-page Wall Street Journal apology** (Jan 26, 2026) titled "To Those I've Hurt" — a comprehensive open letter addressing his antisemitic remarks and erratic behavior. He attributed his actions to a "four-month-long manic episode" and revealed that a car accident 25 years ago caused a frontal-lobe brain injury that wasn't properly diagnosed until 2023, leading to his bipolar type-1 diagnosis. Multiple sources describe wife **Bianca Censori** as instrumental in getting him to seek treatment during a manic depressive episode.
On the music side, he's announced **two upcoming albums**: *BULLY* (now scheduled for March 20, 2026) and *IN A PERFECT WORLD*. Daughter **North West** collaborated with him on a track called "Piercings on My Hand." He performed a massive show in Mexico City (Jan 30) at the Monumental Plaza de Toros, and he'll headline the new **Hellwatt Festival** in Italy (July 4-18, 2026).
The **2026 Grammys** remain contentious — he's reportedly not invited after last year's red carpet drama. Health concerns also surfaced after a January LA outing where he appeared "noticeably bloated and moving with sluggish energy."
**KEY THEMES:**
1. **Accountability arc** — The WSJ apology is being widely discussed as either genuine growth or calculated PR
2. **Mental health transparency** — Brain injury + bipolar diagnosis framing his past behavior
3. **Bianca Censori's role** — Increasingly seen as a stabilizing force (despite continued provocative public appearances)
4. **Music comeback** — Two albums announced, active touring in Latin America and Europe
5. **Grammy exile** — Still persona non grata at major award shows
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 relevant threads (filtered out noise)
├─ 🔵 X: 30 posts │ ~10 likes │ casual fan chatter + Bianca Censori discourse
├─ 🌐 Web: 20+ pages │ washingtonpost.com, billboard.com, allhiphop.com, euronews.com, rap-up.com
└─ Top voices: Washington Post, Billboard, AllHipHop, Euronews, The Mirror
---
Want to dive deeper into any of these threads — the apology, the new albums, the Grammys situation, or Bianca Censori? Just tell me what angle you're interested in.
@@ -1,24 +0,0 @@
## What I learned:
OpenClaw (formerly ClawdBot, formerly Moltbot) is an open-source autonomous AI agent created by Peter Steinberger that went massively viral in late January 2026 — hitting 145,000+ GitHub stars and 20,000+ forks. It runs locally on your machine, connects to messaging platforms (Telegram, WhatsApp, Discord, Signal, etc.), and uses LLMs like Claude, GPT, or Gemini to actually perform tasks on your computer — file management, scheduling, web browsing, purchases, and more. Think "JARVIS for everyone."
The community is simultaneously excited and alarmed. The "ClawHub" skill marketplace has 700+ skills but a VirusTotal/Koi Security audit found **341 malicious skills** doing data exfiltration and crypto theft. A high-severity RCE vulnerability was also disclosed. The #1 most downloaded skill on ClawHub was confirmed malware.
**KEY PATTERNS from the research:**
1. **Cost is the top complaint** — API usage burns money fast; many users recommend Gemini CLI (free 1,000 req/day) over Claude/GPT for OpenClaw to control spend
2. **Security is a dumpster fire** — Supply chain attacks via ClawHub skills, prompt injection risks when browsing the web, RCE vulnerabilities, and orgs are banning it at the EDR level
3. **Setup is nontrivial** — Even after install, most users get stuck on skills/tools/automation config; the `openclaw onboard` wizard is the recommended path
4. **The skill ecosystem is powerful but dangerous** — 700+ skills available, but no code signing, no real moderation, and active malware campaigns
5. **Community is split** — Enthusiasts love the autonomy ("I give it sudo and let it configure everything"), while security folks are terrified ("prompt injection is terrifying when you give the bot access to your actual bank account")
---
✅ All agents reported back!
├── 🟠 Reddit: 25 threads │ ~750+ upvotes │ ~600+ comments
├── 🔵 X: 27 posts │ 1,750+ likes │ 325+ reposts
├── 🌐 Web: 20+ pages │ Wikipedia, Bloomberg, The Register, Hacker News, Nature, VirusTotal, DigitalOcean, Codecademy, Tom's Hardware
└── Top voices: r/openclaw, r/cybersecurity, r/clawdbot, r/ArtificialInteligence │ @grok, @Starlink │ Nature, Bloomberg, ACM
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly.
@@ -1,27 +0,0 @@
Now I have a comprehensive picture. Let me synthesize everything.
**What I learned:**
Nano Banana Pro is fundamentally different from older image models — it's a "thinking" model that understands intent, physics, and composition. The community has converged on two distinct prompting styles, and **both work well** depending on the use case:
1. **JSON structured prompts** — Dominant on Reddit (r/nanobanana2pro) and X. Power users share elaborate JSON objects with nested fields for subject, appearance, lighting, composition, and style. These produce the most consistent, controlled results for portraits, editorials, and product shots.
2. **Natural language "Creative Director" prompts** — Recommended by Google's official guide. Full sentences describing the scene as if briefing a photographer. Better for cinematic shots, complex edits, and iterative refinement.
**KEY PATTERNS I'll use:**
1. **JSON format for maximum control** — The top Reddit posts (149-259 upvotes) all use structured JSON with nested objects: `subject > appearance > hair/skin/expression`, `lighting > key/fill/rim`, `camera > lens/angle/depth_of_field`
2. **Be a Creative Director, not a tag spammer** — Replace "4k, realistic, beautiful" with specific technical directions: "85mm lens at f/1.8", "three-point lighting with key at 45°", "shallow depth of field with bokeh"
3. **Provide context for WHY** — Adding purpose ("for a luxury fashion editorial", "for a high-end gourmet cookbook") helps the model infer appropriate styling, plating, lighting
4. **Edit, don't regenerate** — When 80% is right, ask for the specific change conversationally
5. **Text rendering works** — Keep text under 3 words for best results (75% success rate), specify font style and placement
---
All agents reported back!
- 🟠 Reddit: 5 threads | 638 upvotes | 66 comments
- 🔵 X: 29 posts | 312 likes | 117 reposts
- 🌐 Web: 10+ pages | blog.google, dev.to, eweek.com, imagine.art, github.com, leonardo.ai, medium.com
- Top voices: r/nanobanana2pro | @Xmira_belle, @doctorwasif, @chatgptpaglu | Google AI blog, DEV Community
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into Nano Banana Pro.
@@ -1,48 +0,0 @@
**What I learned:**
Clawdbot (now officially renamed **OpenClaw**) is a self-hosted, open-source personal AI assistant that connects to messaging apps (Telegram, WhatsApp, Discord, Slack, Signal, iMessage) and uses LLMs to act as an autonomous agent on your behalf. It's roughly one week old as a public project and the community is moving fast — the name has already changed twice (Clawdbot → Moltbot → OpenClaw).
**KEY PATTERNS from the research:**
1. **Security is the #1 concern** — Multiple Reddit threads and guides warn that many setups are wide open to the internet with zero auth. Run it on dedicated/isolated hardware (VM, Mac Mini, NAS), never your main machine. Block dangerous commands via allowlist. Never add it to group chats.
2. **Use the onboarding wizard, not Docker** — Docker was reported as unreliable (shims, bridges). The single-line install script (`curl -fsSL https://clawd.bot/install.sh | bash`) followed by `openclaw onboard` is the smoothest path. Docker is a "no-go" per multiple users.
3. **Model choice matters a lot** — Claude Opus 4.5 is recommended for best results (trained to resist prompt injection, ~99% resistance). For budget setups, Claude Haiku works at $0. Local models (Ollama/LM Studio) work but cause hallucination and token issues — GLM 4.7 Flash (30B) was recommended as the best local option.
4. **Telegram is the easiest channel to start with** — Most guides and users start with a Telegram bot token. WhatsApp QR code pairing fails frequently. Get Telegram working first, then expand.
5. **Sandbox and isolate** — Set `agents.defaults.sandbox.mode` to `"non-main"` for Docker sandboxing. Use a Linux VM or UTM on Mac to segregate traffic. Tailscale for secure remote access.
---
✅ All agents reported back!
├─ 🟠 Reddit: 19 threads │ 415+ upvotes │ 290+ comments
├─ 🔵 X: 29 posts │ 815+ likes │ 65+ reposts
├─ 🌐 Web: 10+ pages │ Medium, DataCamp, DEV, GitHub, QNAP, docs.openclaw.ai
└─ Top voices: r/clawdbot, r/LocalLLM, r/AskVibecoders │ @aashatwt (452 likes), @recap_david (329 likes) │ Lukas Niessen on Medium
---
**Here's the consensus best setup path based on what the community actually recommends:**
### Quick-Start (30 min)
1. **Get dedicated hardware** — Mac Mini, old laptop, NAS (QNAP has an official guide), or a VPS (DigitalOcean gist available). Don't use your daily driver.
2. **Install via script**: `curl -fsSL https://clawd.bot/install.sh | bash`
3. **Run the wizard**: `openclaw onboard` — it walks through gateway, workspace, channels, and skills
4. **Connect Telegram first** — create a bot via @BotFather, paste the token
5. **Set your API key** — Anthropic API key for Claude, or connect Ollama for local models
6. **Customize SOUL.md** — this controls your bot's personality and behavior
### Security Hardening (do this immediately)
- Bind gateway to localhost only, use Tailscale for remote access
- Set sandbox mode to `"non-main"`
- Create an explicit command allowlist (block `rm -rf`, `sudo`, `chmod`)
- Rotate API keys every 90 days
- Never add the bot to group chats
### Budget Option ($0)
Per @shynxbt: Use a free AWS VPS + Claude Haiku model + Telegram bot = fully functional for $0.
---
Want me to help you with a specific part of the setup, or do you have a particular use case in mind (home automation, CRM, coding assistant, etc.)?
@@ -1,332 +0,0 @@
---
name: last30days
description: Research a topic from the last 30 days on Reddit + X + Web, become an expert, and write copy-paste-ready prompts for the user's target tool.
argument-hint: '"[topic] for [tool]" or "[topic]"'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
---
# last30days: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent
Before doing anything, parse the user's input for:
1. **TOPIC**: What they want to learn about (e.g., "web app mockups", "Claude Code skills", "image generation")
2. **TARGET TOOL** (if specified): Where they'll use the prompts (e.g., "Nano Banana Pro", "ChatGPT", "Midjourney")
3. **QUERY TYPE**: What kind of research they want:
- **PROMPTING** - "X prompts", "prompting for X", "X best practices" → User wants to learn techniques and get copy-paste prompts
- **RECOMMENDATIONS** - "best X", "top X", "what X should I use", "recommended X" → User wants a LIST of specific things
- **NEWS** - "what's happening with X", "X news", "latest on X" → User wants current events/updates
- **GENERAL** - anything else → User wants broad understanding of the topic
Common patterns:
- `[topic] for [tool]` → "web mockups for Nano Banana Pro" → TOOL IS SPECIFIED
- `[topic] prompts for [tool]` → "UI design prompts for Midjourney" → TOOL IS SPECIFIED
- Just `[topic]` → "iOS design mockups" → TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" → QUERY_TYPE = RECOMMENDATIONS
- "what are the best [topic]" → QUERY_TYPE = RECOMMENDATIONS
**IMPORTANT: Do NOT ask about target tool before research.**
- If tool is specified in the query, use it
- If tool is NOT specified, run research first, then ask AFTER showing results
**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [RECOMMENDATIONS | NEWS | HOW-TO | GENERAL]`
**DISPLAY your parsing to the user.** Before running any tools, output a single line:
🔍 **{TOPIC}** · {QUERY_TYPE}
Searching Reddit, X, and the web for {natural language description of what you'll look for}...
Example outputs:
- 🔍 **kanye west** · News — Searching Reddit, X, and the web for the latest kanye west news and discussions...
- 🔍 **best MCP servers** · Recommendations — Searching Reddit, X, and the web for the most recommended MCP servers...
- 🔍 **nano banana pro prompting** · Prompting — Searching Reddit, X, and the web for nano banana pro prompting techniques and tips...
- 🔍 **open claw** · General — Searching Reddit, X, and the web for what people are saying about open claw...
If TARGET_TOOL is known, mention it: "...for nano banana pro prompting techniques to use in ChatGPT..."
This text MUST appear before you call any tools. It confirms to the user that you understood their request.
---
## Research Execution
**Step 1: Run the research script**
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "$ARGUMENTS" --emit=compact 2>&1
```
The script will automatically:
- Detect available API keys
- Run Reddit/X searches if keys exist
- Signal if WebSearch is needed
---
## STEP 2: DO WEBSEARCH WHILE SCRIPT RUNS
The script auto-detects sources (Bird CLI, API keys, etc). While waiting for it, do WebSearch.
For **ALL modes**, do WebSearch to supplement (or provide all data in web-only mode).
Choose search queries based on QUERY_TYPE:
**If RECOMMENDATIONS** ("best X", "top X", "what X should I use"):
- Search for: `best {TOPIC} recommendations`
- Search for: `{TOPIC} list examples`
- Search for: `most popular {TOPIC}`
- Goal: Find SPECIFIC NAMES of things, not generic advice
**If NEWS** ("what's happening with X", "X news"):
- Search for: `{TOPIC} news 2026`
- Search for: `{TOPIC} announcement update`
- Goal: Find current events and recent developments
**If PROMPTING** ("X prompts", "prompting for X"):
- Search for: `{TOPIC} prompts examples 2026`
- Search for: `{TOPIC} techniques tips`
- Goal: Find prompting techniques and examples to create copy-paste prompts
**If GENERAL** (default):
- Search for: `{TOPIC} 2026`
- Search for: `{TOPIC} discussion`
- Goal: Find what people are actually saying
For ALL query types:
- **USE THE USER'S EXACT TERMINOLOGY** - don't substitute or add tech names based on your knowledge
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- INCLUDE: blogs, tutorials, docs, news, GitHub repos
- **DO NOT output "Sources:" list** - this is noise, we'll show stats at the end
**Depth options** (passed through from user's command):
- `--quick` → Faster, fewer sources (8-12 each)
- (default) → Balanced (20-30 each)
- `--deep` → Comprehensive (50-70 Reddit, 40-60 X)
---
## Judge Agent: Synthesize All Sources
**After all searches complete, internally synthesize (don't display stats yet):**
The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight WebSearch sources LOWER (no engagement data)
3. Identify patterns that appear across ALL three sources (strongest signals)
4. Note any contradictions between sources
5. Extract the top 3-5 actionable insights
**Do NOT display stats here - they come at the end, right before the invitation.**
---
## FIRST: Internalize the Research
**CRITICAL: Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**
Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned (e.g., if research mentions "ClawdBot" or "@clawdbot", that's a DIFFERENT product than "Claude Code" - don't conflate them)
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about
**ANTI-PATTERN TO AVOID**: If user asks about "clawdbot skills" and research returns ClawdBot content (self-hosted AI agent), do NOT synthesize this as "Claude Code skills" just because both involve "skills". Read what the research actually says.
### If QUERY_TYPE = RECOMMENDATIONS
**CRITICAL: Extract SPECIFIC NAMES, not generic patterns.**
When user asks "best X" or "top X", they want a LIST of specific things:
- Scan research for specific product names, tool names, project names, skill names, etc.
- Count how many times each is mentioned
- Note which sources recommend each (Reddit thread, X post, blog)
- List them by popularity/mention count
**BAD synthesis for "best Claude Code skills":**
> "Skills are powerful. Keep them under 500 lines. Use progressive disclosure."
**GOOD synthesis for "best Claude Code skills":**
> "Most mentioned skills: /commit (5 mentions), remotion skill (4x), git-worktree (3x), /pr (3x). The Remotion announcement got 16K likes on X."
### For all QUERY_TYPEs
Identify from the ACTUAL RESEARCH OUTPUT:
- **PROMPT FORMAT** - Does research recommend JSON, structured params, natural language, keywords?
- The top 3-5 patterns/techniques that appeared across multiple sources
- Specific keywords, structures, or approaches mentioned BY THE SOURCES
- Common pitfalls mentioned BY THE SOURCES
---
## THEN: Show Summary + Invite Vision
**Display in this EXACT sequence:**
**FIRST - What I learned (based on QUERY_TYPE):**
**If RECOMMENDATIONS** - Show specific things mentioned with sources:
```
🏆 Most mentioned:
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, @handle2, r/sub, blog.com
[Tool Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle3, r/sub2, Complex
Notable mentions: [other specific things with 1-2 mentions]
```
**CRITICAL for RECOMMENDATIONS:**
- Each item MUST have a "Sources:" line with actual @handles from X posts (e.g., @LONGLIVE47, @ByDobson)
- Include subreddit names (r/hiphopheads) and web sources (Complex, Variety)
- Parse @handles from research output and include the highest-engagement ones
- Format naturally - tables work well for wide terminals, stacked cards for narrow
**If PROMPTING/NEWS/GENERAL** - Show synthesis and patterns:
CITATION RULE: Cite sources sparingly to prove research is real.
- In the "What I learned" intro: cite 1-2 top sources total, not every sentence
- In KEY PATTERNS: cite 1 source per pattern, short format: "per @handle" or "per r/sub"
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
**BAD:** "His album is set for March 20 (per @cocoabutterbf; Rolling Stone; HotNewHipHop; Complex)."
**GOOD:** "His album BULLY is set for March 20 via Gamma, per Rolling Stone."
```
What I learned:
**{Topic 1}** — [1-2 sentences about this storyline, per source]
**{Topic 2}** — [1-2 sentences, per source]
**{Topic 3}** — [1-2 sentences, per source]
KEY PATTERNS from the research:
1. [Pattern] — per @handle
2. [Pattern] — per r/sub
3. [Pattern] — per source
```
**THEN - Stats (right before invitation):**
**CRITICAL: Calculate actual totals from the research output.**
- Count posts/threads from each section
- Sum engagement: parse `[Xlikes, Yrt]` from each X post, `[Xpts, Ycmt]` from Reddit
- Identify top voices: highest-engagement @handles from X, most active subreddits
**Copy this EXACTLY, replacing only the {placeholders}:**
```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts (via Bird/xAI)
├─ 🌐 Web: {N} pages │ {domain1}, {domain2}, {domain3}
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
---
```
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
**LAST - Invitation:**
```
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into {TARGET_TOOL}.
```
---
## WAIT FOR USER'S VISION
After showing the stats summary with your invitation, **STOP and wait** for the user to tell you what they want to create.
---
## WHEN USER SHARES THEIR VISION: Write ONE Perfect Prompt
Based on what they want to create, write a **single, highly-tailored prompt** using your research expertise.
### CRITICAL: Match the FORMAT the research recommends
**If research says to use a specific prompt FORMAT, YOU MUST USE THAT FORMAT.**
**ANTI-PATTERN**: Research says "use JSON prompts with device specs" but you write plain prose. This defeats the entire purpose of the research.
### Quality Checklist (run before delivering):
- [ ] **FORMAT MATCHES RESEARCH** - If research said JSON/structured/etc, prompt IS that format
- [ ] Directly addresses what the user said they want to create
- [ ] Uses specific patterns/keywords discovered in research
- [ ] Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- [ ] Appropriate length and style for TARGET_TOOL
### Output Format:
```
Here's your prompt for {TARGET_TOOL}:
---
[The actual prompt IN THE FORMAT THE RESEARCH RECOMMENDS]
---
This uses [brief 1-line explanation of what research insight you applied].
```
---
## IF USER ASKS FOR MORE OPTIONS
Only if they ask for alternatives or more prompts, provide 2-3 variations. Don't dump a prompt pack unless requested.
---
## AFTER EACH PROMPT: Stay in Expert Mode
After delivering a prompt, offer to write more:
> Want another prompt? Just tell me what you're creating next.
---
## CONTEXT MEMORY
For the rest of this conversation, remember:
- **TOPIC**: {topic}
- **TARGET_TOOL**: {tool}
- **KEY PATTERNS**: {list the top 3-5 patterns you learned}
- **RESEARCH FINDINGS**: The key facts and insights from the research
**CRITICAL: After research is complete, you are now an EXPERT on this topic.**
When the user asks follow-up questions:
- **DO NOT run new WebSearches** - you already have the research
- **Answer from what you learned** - cite the Reddit threads, X posts, and web sources
- **If they ask for a prompt** - write one using your expertise
Only do new research if the user explicitly asks about a DIFFERENT topic.
---
## Output Summary Footer (After Each Prompt)
After delivering a prompt, end with:
```
---
📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} web pages
Want another prompt? Just tell me what you're creating next.
```
@@ -1,25 +0,0 @@
Here's what I found:
## What I learned:
**BULLY Album — March 20, 2026 via Gamma** — After years of delays (first announced September 2024), Kanye's 12th studio album finally has a firm release date. The 13-track project features Peso Pluma, Playboi Carti, and Ty Dolla Sign. Sonically it recalls *808s & Heartbreak* and *MBDTF* — Ye mostly sings rather than raps. Notably, earlier leak versions used AI-deepfaked vocals, which have reportedly been re-recorded with his real voice for the official release, per Rolling Stone.
**Public Apology for Antisemitism** — On January 26, Ye took out a full-page Wall Street Journal ad titled "To Those I've Hurt," publicly apologizing for his antisemitic remarks. He disclosed a previously undiagnosed brain injury from his 2002 car accident and attributed his behavior to a four-month manic episode fueled by bipolar disorder, psychosis, and paranoia. "I lost touch with reality," he wrote. The apology also extended to the Black community, per The Washington Post.
**Hellwatt Festival in Italy** — Ye is headlining a brand-new festival at the 103,000-capacity RCF Arena in Italy over three weekends from July 4-18, 2026 — his first-ever live concert in Italy, per Billboard.
**Health Concerns** — A rare January 2026 outing in LA with Bianca Censori reignited concern about Ye's physical appearance. Insiders point to medication side effects, frequent travel, and inconsistent routines, per AllHipHop.
**Grammys Ban** — Ye is reportedly not welcome at the 2026 Grammy Awards after clashing with organizers last year over his invitation terms, per The News International.
**Kim & Lewis Hamilton Buzz** — X chatter is heavily focused on Kim Kardashian's relationship with Lewis Hamilton, with users contrasting her new relationship against her marriage to Ye.
---
✅ All agents reported back!
├─ 🟠 Reddit: 0 threads (no results this cycle)
├─ 🔵 X: 29 posts │ 33 likes │ 14 reposts (via xAI)
├─ 🌐 Web: 30+ pages │ rollingstone.com, washingtonpost.com, complex.com, billboard.com, npr.org
└─ 🗣️ Top voices: @honest30bgfan_ (33 likes), @HipHopCrave_ │ Rolling Stone, Washington Post, Complex
---
Share your vision for what you want to create and I'll write a thoughtful prompt you can copy-paste directly into your tool of choice.
@@ -0,0 +1,8 @@
<!-- FIXTURE: captured live from reddit.com/svc/shreddit/community-more-posts/top/?name=technology&t=week on 2026-05-29; trimmed to 5 post cards (start-tag attrs only). -->
<div id="feed">
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tq0zk7/the_netherlands_just_blocked_a_us_company_from/" content-href="https://www.techspot.com/news/112552-netherlands-blocked-us-company-buying-app-dutch-citizens.html" view-context="SubredditFeed" comment-count="1743" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="0" award-count="23" award-id="award_obsessed_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/Obsessed_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-28T11:37:01.506000+0000" domain="techspot.com" id="t3_1tq0zk7" post-title="The Netherlands just blocked a US company from buying the app Dutch citizens use for everything" post-language="en" post-type="link" score="52692" upvote-ratio="0.9606269354736776" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1toe7m2/erin_brockovich_launches_map_of_over_4200_data/" content-href="https://www.newsweek.com/erin-brockovich-asks-americans-for-help-as-she-launches-data-center-map-11989813" view-context="SubredditFeed" comment-count="673" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="2" award-count="6" award-id="award_this_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/this_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T17:39:43.272000+0000" domain="newsweek.com" id="t3_1toe7m2" post-title="Erin Brockovich launches map of over 4,200 data centres in the US, appeals for local communities to report environmental impact and other costs" post-language="en" post-type="link" score="33567" upvote-ratio="0.973297166968053" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_fj9vsvfd" author="marketrent" icon="https://www.redditstatic.com/avatars/defaults/v2/avatar_default_1.png" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tollgz/majority_of_americans_support_ban_on_surveillance/" content-href="https://gizmodo.com/majority-of-americans-support-ban-on-surveillance-pricing-and-electronic-shelf-labels-2000762717" view-context="SubredditFeed" comment-count="1043" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="3" award-count="7" award-id="award_free_bravo" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/bravo_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-26T21:55:07.322000+0000" domain="gizmodo.com" id="t3_1tollgz" post-title="Majority of Americans Support Ban on Surveillance Pricing and Electronic Shelf Labels" post-language="en" post-type="link" score="29791" upvote-ratio="0.9815063671850003" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_98wao505" author="Plastic_Ninja_9014" icon="https://preview.redd.it/snoovatar/avatars/69af2b53-b0a1-4ab6-b119-d90f21c423fe-headshot.png?width=64&amp;height=64&amp;crop=smart&amp;auto=webp&amp;s=f3661eb511798004968f8b115a689dcee30f1428" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tp5qz2/tech_ceos_are_apparently_suffering_from_ai/" content-href="https://techcrunch.com/2026/05/27/tech-ceos-are-apparently-suffering-from-ai-psychosis/" view-context="SubredditFeed" comment-count="1653" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="4" award-count="6" award-id="award_free_regret_2" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/regret_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-27T13:33:49.280000+0000" domain="techcrunch.com" id="t3_1tp5qz2" post-title="Tech CEOs are apparently suffering from AI psychosis" post-language="en" post-type="link" score="26419" upvote-ratio="0.9605741880002646" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_cc0n0rs5" author="AdSpecialist6598" icon="https://styles.redditmedia.com/t5_4heieb/styles/profileIcon_snoob7abf9c5-a18e-4228-a419-5179810e11df-headshot-f.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=94f6b9715ca039332ed1714f3abe0842cef23b81" data-expected-lcp subreddit-name="technology"></shreddit-post>
<shreddit-post data-ks-item class="block relative cursor-pointer group bg-neutral-background focus-within:bg-neutral-background-hover hover:bg-neutral-background-hover xs:rounded-4 px-md py-2xs my-2xs nd:visible nd:pb-[var(--rem36)]" permalink="/r/technology/comments/1tn5g7s/pope_leo_issues_ai_encyclical_warning_that_opaque/" content-href="https://variety.com/2026/biz/global/pope-leo-ai-encyclical-algorithms-threaten-dehumanisation-1236758186/" view-context="SubredditFeed" comment-count="608" is-slim-card view-type="cardView" pdp-target="_self" feedIndex="6" award-count="7" award-id="award_hooray_3" award-icon-url="https://i.redd.it/snoovatar/snoo_assets/marketing/FTUE_40.png" moderation-verdict="" is-embeddable is-desktop-viewport is-awardable is-link-post created-timestamp="2026-05-25T10:45:04.093000+0000" domain="variety.com" id="t3_1tn5g7s" post-title="Pope Leo Issues AI Encyclical Warning That Opaque Algorithms Controlled by a Few Companies Can Bring New Forms of Dehumanisation" post-language="en" post-type="link" score="25835" upvote-ratio="0.9760626539506095" subreddit-id="t5_2qh16" subreddit-prefixed-name="r/technology" author-id="t2_1i1zizibn9" author="yourfavchoom" icon="https://styles.redditmedia.com/t5_dgdrt8/styles/profileIcon_k9x929ihm8rg1.png?width=64&amp;height=64&amp;frame=1&amp;auto=webp&amp;crop=64%3A64%2Csmart&amp;s=2e8a5042cccc4555167f98d28bc0de4e13fd3ca5" data-expected-lcp subreddit-name="technology"></shreddit-post>
</div>
+7
View File
@@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- FIXTURE: captured live from reddit.com/r/Rakuten/top.rss on 2026-05-29; trimmed to 5 entries. Atom shape identical to search.rss. --><feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/"><category term="Rakuten" label="r/Rakuten"/><updated>2026-05-29T14:14:32+00:00</updated><icon>https://www.redditstatic.com/icon.png/</icon><id>/r/Rakuten/top.rss?t=month</id><link rel="self" href="https://www.reddit.com/r/Rakuten/top.rss?t=month" type="application/atom+xml" /><link rel="alternate" href="https://www.reddit.com/r/Rakuten/top?t=month" type="text/html" /><subtitle>This is an unofficial subreddit for Rakuten Rewards, the cash back website. We are not affiliated with, endorsed by, or sponsored by Rakuten or any of its subsidiaries.</subtitle><title>top scoring links : Rakuten</title><entry><author><name>/u/InternetUser52</name><uri>https://www.reddit.com/user/InternetUser52</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I&amp;#39;m rich!!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/InternetUser52&quot;&gt; /u/InternetUser52 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/q8fgmxs29c2h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tiv013</id><link href="https://www.reddit.com/r/Rakuten/comments/1tiv013/lets_goo_002/" /><updated>2026-05-20T18:48:31+00:00</updated><published>2026-05-20T18:48:31+00:00</published><title>LETS GOO! $0.02!!!</title></entry>
<entry><author><name>/u/Immediate-Duck-6351</name><uri>https://www.reddit.com/user/Immediate-Duck-6351</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;I dont travel and Im buying a house in a few weeks so cash back is amazing 🙌 hoping to keep the pace in the next quarter so I can buy new kitchen appliances lol. &lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Immediate-Duck-6351&quot;&gt; /u/Immediate-Duck-6351 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/d2a4s0ipvb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te1fp8</id><link href="https://www.reddit.com/r/Rakuten/comments/1te1fp8/so_excited/" /><updated>2026-05-15T16:29:28+00:00</updated><published>2026-05-15T16:29:28+00:00</published><title>So excited 🥳</title></entry>
<entry><author><name>/u/gnibgnib</name><uri>https://www.reddit.com/user/gnibgnib</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;128k for the May transfer&lt;/p&gt; &lt;p&gt;41k pending for August &lt;/p&gt; &lt;p&gt;Got another 9k at Asics not showing but overall pretty happy with Rakuten&lt;/p&gt; &lt;p&gt;P2 was able to secure 85k for May transfer&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/gnibgnib&quot;&gt; /u/gnibgnib &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/gallery/1tb8674&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1tb8674</id><link href="https://www.reddit.com/r/Rakuten/comments/1tb8674/had_a_great_run_so_far_this_year_thanks_to_this/" /><updated>2026-05-12T17:17:19+00:00</updated><published>2026-05-12T17:17:19+00:00</published><title>Had a great run so far this year thanks to this sub!</title></entry>
<entry><author><name>/u/TravelVet93</name><uri>https://www.reddit.com/user/TravelVet93</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/TravelVet93&quot;&gt; /u/TravelVet93 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/x6b9whvupb1h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1te0hom</id><link href="https://www.reddit.com/r/Rakuten/comments/1te0hom/my_best_payout_so_far/" /><updated>2026-05-15T15:56:40+00:00</updated><published>2026-05-15T15:56:40+00:00</published><title>My best payout so far</title></entry>
<entry><author><name>/u/Beautiful-Piece-4252</name><uri>https://www.reddit.com/user/Beautiful-Piece-4252</uri></author><category term="Rakuten" label="r/Rakuten"/><content type="html">&lt;!-- SC_OFF --&gt;&lt;div class=&quot;md&quot;&gt;&lt;p&gt;The amount of $$ available in sign up bonuses is amazing. It&amp;#39;s kind of a part time job ensuring Rakuten captures everything, but my August and November payout should be sizeable. I&amp;#39;m new to this and it always seemed like a lot of work for little reward. I know it&amp;#39;s not sustainable, but wow!&lt;/p&gt; &lt;/div&gt;&lt;!-- SC_ON --&gt; &amp;#32; submitted by &amp;#32; &lt;a href=&quot;https://www.reddit.com/user/Beautiful-Piece-4252&quot;&gt; /u/Beautiful-Piece-4252 &lt;/a&gt; &lt;br/&gt; &lt;span&gt;&lt;a href=&quot;https://i.redd.it/1vqvajsci42h1.jpeg&quot;&gt;[link]&lt;/a&gt;&lt;/span&gt; &amp;#32; &lt;span&gt;&lt;a href=&quot;https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/&quot;&gt;[comments]&lt;/a&gt;&lt;/span&gt;</content><id>t3_1thsnm1</id><link href="https://www.reddit.com/r/Rakuten/comments/1thsnm1/how_can_this_be_real/" /><updated>2026-05-19T16:46:17+00:00</updated><published>2026-05-19T16:46:17+00:00</published><title>How can this be real?</title></entry>
</feed>
@@ -0,0 +1,29 @@
<!-- FIXTURE: captured live from reddit.com/svc/shreddit/comments/r/Rakuten/t3_1taeiw0 on 2026-05-29;
trimmed to 6 real comment elements (real attrs + real bodies) + 2 synthetic edge cases. -->
<shreddit-comment-tree-stats total-comments="14"></shreddit-comment-tree-stats>
<shreddit-comment-tree id="comment-tree" post-id="t3_1taeiw0">
<shreddit-comment created="2026-05-11T20:16:57.590000+0000" author="Obvious_Painting_881" thingId="t1_ol8tp8n" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8tp8n/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8tp8n-comment-rtjson-content" slot="comment"><div id="t1_ol8tp8n-post-rtjson-content" dir="auto"><p dir="auto">Where do you find $750? The highest available package for Total was $284.99 when I did the lifelock promotion. I did get the full 284.99 from Rakuten.</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-12T12:26:14.973000+0000" author="Stormtrooper149" thingId="t1_olcy1iv" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/olcy1iv/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_olcy1iv-comment-rtjson-content" slot="comment"><div id="t1_olcy1iv-post-rtjson-content" dir="auto"><p dir="auto">It went to pending ($712.49)</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-19T01:43:48.026000+0000" author="heythereyou01" thingId="t1_omlbiqg" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/omlbiqg/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_omlbiqg-comment-rtjson-content" slot="comment"><div id="t1_omlbiqg-post-rtjson-content" dir="auto"><p dir="auto">Hey I PMd. can I get the screenshot ?</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-11T20:21:16.398000+0000" author="Stormtrooper149" thingId="t1_ol8undb" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8undb/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8undb-comment-rtjson-content" slot="comment"><div id="t1_ol8undb-post-rtjson-content" dir="auto"><p dir="auto">Family plan</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-11T20:28:33.803000+0000" author="Obvious_Painting_881" thingId="t1_ol8w8w6" depth="2" permalink="/r/Rakuten/comments/1taeiw0/comment/ol8w8w6/" score="1" postId="t3_1taeiw0" content-type="text">
<div id="t1_ol8w8w6-comment-rtjson-content" slot="comment"><div id="t1_ol8w8w6-post-rtjson-content" dir="auto"><p dir="auto">Price seems to change every time I go to the page but I see only 249.99-369.99 for Total/Advanced. No where near your $750. Just saying the Total plan for 299.99 worked for me and I got 284.99 which is 95%.</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-12T02:33:48.200000+0000" author="jwegener" thingId="t1_olaqzjk" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/olaqzjk/" score="2" postId="t3_1taeiw0" content-type="text">
<div id="t1_olaqzjk-comment-rtjson-content" slot="comment"><div id="t1_olaqzjk-post-rtjson-content" dir="auto"><p dir="auto">I did that one. Lets pray</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-13T10:00:00.000000+0000" author="[deleted]" thingId="t1_synthdel" depth="0" permalink="/r/Rakuten/comments/1taeiw0/comment/synthdel/" score="5" postId="t3_1taeiw0" content-type="text">
<div id="t1_synthdel-comment-rtjson-content" slot="comment"><div id="t1_synthdel-post-rtjson-content" dir="auto"><p dir="auto">[removed]</p></div></div>
</shreddit-comment>
<shreddit-comment created="2026-05-13T11:00:00.000000+0000" author="NegScoreUser" thingId="t1_synthneg" depth="1" permalink="/r/Rakuten/comments/1taeiw0/comment/synthneg/" score="-7" postId="t3_1taeiw0" content-type="text">
<div id="t1_synthneg-comment-rtjson-content" slot="comment"><div id="t1_synthneg-post-rtjson-content" dir="auto"><p dir="auto">A downvoted but real reply with negative score for edge-case coverage.</p></div></div>
</shreddit-comment>
</shreddit-comment-tree>
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "last30days-skill", "name": "last30days-skill",
"version": "3.2.4", "version": "3.3.2",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.", "description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [ "settings": [
{ {
+1 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "last30days-skill" name = "last30days-skill"
version = "3.3.0" version = "3.3.2"
description = "Multi-source last-30-days research skill" description = "Multi-source last-30-days research skill"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
-90
View File
@@ -1,90 +0,0 @@
## v3.3.0 — install everywhere, ship the reliability sweep
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Digg, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## What's new in v3.3.0
### Install everywhere with one command
`npx skills add mvanhorn/last30days-skill -g -y` is now the canonical install path for **every harness** — Claude Code, OpenAI Codex CLI, Cursor, Gemini CLI, GitHub Copilot, Windsurf, and 50+ other Agent Skills hosts. The skill auto-detects each harness's skills directory and symlinks in place, so edits propagate live. No more per-harness manual paths in the README.
### New emit mode: `--emit=html`
Shareable, print-friendly HTML briefs. Drop the file in Slack, mail it to a stakeholder, or print it for the meeting. Same data as compact mode, structured for human reading.
### New source: Digg
Digg surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes directly into the brief. Auto-enabled when `digg-pp-cli` is on PATH. Footer line: `⛏️ Digg: N clusters │ K posts │ M authors`. No X auth required for the inline quotes.
### YouTube residential-IP routing (`LAST30DAYS_YOUTUBE_SSH_HOST`)
Running on a datacenter VPS (Hetzner, DigitalOcean, AWS, etc.)? YouTube's bot-wall fingerprints datacenter IP ranges before any cookie check. Set `LAST30DAYS_YOUTUBE_SSH_HOST=<ssh-alias>` and yt-dlp runs over SSH against a residential-IP host instead. One env var, no proxy service required.
### macOS Keychain credential source
When env vars and config files aren't set, the engine now reads credentials from the macOS Keychain. Stores secrets where macOS expects them; nothing on disk in plaintext.
### `EXCLUDE_SOURCES` env var
The inverse of `INCLUDE_SOURCES`. Useful for "everything except TikTok" or "everything except the slow ones."
## Reliability sweep
This release closes a long tail of platform-specific issues that have been accumulating:
- **Reddit**: subreddits starting with `r` no longer get mangled by `lstrip("r/")`. Browser-like headers + gzip handling fix urllib 403s on the public JSON endpoint. HTTP 402 now triggers the OpenAI/public-JSON fallback chain when ScrapeCreators credits are exhausted.
- **xAI**: empty or malformed responses now surface in `errors_by_source` instead of silently returning zero results.
- **Windows**: process cleanup no longer crashes on `os.killpg`. POSIX-style secret-permission warnings skipped. Save-path footer uses forward slashes.
- **Auth**: comma-separated `SCRAPECREATORS_API_KEY=key1,key2` rotation restored (accidentally dropped in v3.0.6).
- **YouTube + HN**: SC YouTube + multi-token HN searches unblocked. Transcript-fetch ratio surfaced.
- **HTTP**: retry budget expanded with exponential backoff on DNS failure. Parallel AI search aligned with current API schema.
- **OpenClaw**: now works without a ScrapeCreators key. Poll-timing initialized once.
## Multi-harness reframe
`AGENTS.md` is now the canonical project doc; `CLAUDE.md` points at it. The skill is positioned as a multi-harness Agent Skills package, not a Claude-Code-specific tool. SKILL.md's path resolution rewrote `SKILL_ROOT``SKILL_DIR`, removing ~80 lines of bash and fixing a real spec-vs-engine divergence bug.
## Breaking change
**`.codex-plugin/plugin.json` removed.** Codex native-plugin users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`. The `npx skills add` path now reaches every harness uniformly.
## Install
Any harness (recommended):
```
npx skills add mvanhorn/last30days-skill -g -y
```
Claude Code marketplace:
```
/plugin marketplace add mvanhorn/last30days-skill
```
OpenClaw:
```
clawhub install last30days-official
```
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
## Contributors
First-time contributors whose fixes shipped in v3.3.0 (most via PR triage salvage — the fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) — OpenClaw ScrapeCreators-key-optional fix
Plus every contributor who shipped one of the ~75 PRs merged this cycle. See [CHANGELOG.md](CHANGELOG.md) under `[3.3.0]` for the full PR list and `git log v3.2.0..v3.3.0` for the complete commit graph.
30 days of research. 30 seconds of work. Thirteen sources. Zero stale prompts.
+6 -6
View File
@@ -1,6 +1,6 @@
--- ---
name: last30days name: last30days
version: "3.3.0" version: "3.3.2"
description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web." description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react' argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -243,7 +243,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
--- ---
# last30days v3.3.0: Research Any Topic from the Last 30 Days # last30days v3.3.2: Research Any Topic from the Last 30 Days
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section. > **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
@@ -596,8 +596,8 @@ When the user asks "X vs Y" (or "X vs Y vs Z"), the engine fans out N full `pipe
# the Read tool result. Examples: # the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days # Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days # Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md # Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.2/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days # → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.2/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout # scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings). # packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>" SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
@@ -914,8 +914,8 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
# the Read tool result. Examples: # the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days # Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days # Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md # Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.2/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days # → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.2/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout # scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings). # packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>" SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
+88 -20
View File
@@ -62,6 +62,17 @@ def _resolve_token(token: Optional[str] = None) -> Optional[str]:
return None return None
def resolve_token(token: Optional[str] = None) -> Optional[str]:
"""Public alias for ``_resolve_token``.
The pipeline calls this once before ``search_github`` and
``enrich_with_comments`` so the ``gh auth token`` subprocess fallback
only fires once per query when ``GITHUB_TOKEN`` is unset, instead of
twice (once per call site).
"""
return _resolve_token(token)
def _fetch_json( def _fetch_json(
url: str, url: str,
token: Optional[str] = None, token: Optional[str] = None,
@@ -142,8 +153,14 @@ def search_github(
to_date: str, to_date: str,
depth: str = "default", depth: str = "default",
token: Optional[str] = None, token: Optional[str] = None,
) -> List[Dict[str, Any]]: ) -> Dict[str, Any]:
"""Search GitHub Issues and PRs. """Search GitHub Issues and PRs (HTTP fetch only).
Returns a raw envelope shaped like every other adapter's ``search_X``:
``{"items": [raw GitHub API items], "context": {core, from_date,
to_date, count}}``. Normalization, date filtering, and sorting move
to ``parse_github_response``; comment enrichment moves to
``enrich_with_comments``.
Args: Args:
topic: Search topic topic: Search topic
@@ -153,15 +170,23 @@ def search_github(
token: Optional GitHub token (falls back to env/gh CLI) token: Optional GitHub token (falls back to env/gh CLI)
Returns: Returns:
List of normalized item dicts. Empty list on any failure. Dict envelope. Empty ``items`` list on any failure.
""" """
count = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
core = extract_core_subject(topic)
resolved_token = _resolve_token(token) resolved_token = _resolve_token(token)
if not resolved_token: if not resolved_token:
_log("No GitHub token available (set GITHUB_TOKEN or install gh CLI)") _log("No GitHub token available (set GITHUB_TOKEN or install gh CLI)")
return [] return {
"items": [],
count = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"]) "error": "no token",
core = extract_core_subject(topic) "context": {
"core": core,
"from_date": from_date,
"to_date": to_date,
"count": count,
},
}
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})") _log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Build search query with date filter # Build search query with date filter
@@ -176,12 +201,41 @@ def search_github(
data = _fetch_json(url, token=resolved_token, timeout=30) data = _fetch_json(url, token=resolved_token, timeout=30)
if not data: if not data:
return [] return {"items": [], "context": {"core": core, "from_date": from_date,
"to_date": to_date, "count": count}}
raw_items = data.get("items", []) raw_items = data.get("items", [])
_log(f"Found {len(raw_items)} issues/PRs") _log(f"Found {len(raw_items)} issues/PRs")
items = [] return {
"items": raw_items,
"context": {
"core": core,
"from_date": from_date,
"to_date": to_date,
"count": count,
},
}
def parse_github_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Normalize a ``search_github`` envelope into the skill's item shape.
Pure function: no I/O, no token, no enrichment. Applies the date
filter using the search context and sorts by relevance.
"""
if not isinstance(response, dict):
return []
raw_items = response.get("items") or []
if not isinstance(raw_items, list):
return []
context = response.get("context") or {}
core = context.get("core") or ""
from_date = context.get("from_date") or ""
to_date = context.get("to_date") or ""
count = context.get("count") or DEPTH_LIMITS["default"]
items: List[Dict[str, Any]] = []
for i, item in enumerate(raw_items[:count]): for i, item in enumerate(raw_items[:count]):
html_url = item.get("html_url", "") html_url = item.get("html_url", "")
repo = _parse_repo_from_url(html_url) repo = _parse_repo_from_url(html_url)
@@ -224,20 +278,34 @@ def search_github(
}, },
}) })
# Enrich top items with comments
items = _enrich_top_items(items, depth, resolved_token)
# Date filter # Date filter
filtered = [] if from_date and to_date:
for item in items: items = [
d = item.get("date") item for item in items
if d is None or (from_date <= d <= to_date): if item.get("date") is None or (from_date <= item["date"] <= to_date)
filtered.append(item) ]
# Sort by relevance items.sort(key=lambda x: x.get("relevance", 0), reverse=True)
filtered.sort(key=lambda x: x.get("relevance", 0), reverse=True) return items
return filtered
def enrich_with_comments(
items: List[Dict[str, Any]],
depth: str = "default",
token: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Fetch top comments for top-K items by reactions and attach to metadata.
Mutates and returns ``items``. Resolves ``token`` via env/gh CLI when
not supplied, matching ``search_github``'s fallback chain.
"""
if not items:
return items
resolved_token = _resolve_token(token)
if not resolved_token:
_log("No GitHub token available for comment enrichment")
return items
return _enrich_top_items(items, depth, resolved_token)
def _enrich_top_items( def _enrich_top_items(
+47
View File
@@ -223,6 +223,53 @@ def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, st
return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs) return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)
BROWSER_USER_AGENT = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
def get_text(
url: str,
timeout: int = DEFAULT_TIMEOUT,
retries: int = 2,
accept: str = "*/*",
headers: Optional[Dict[str, str]] = None,
) -> Optional[str]:
"""Fetch a URL and return decoded text, or None on any failure.
Keyless helper for Reddit RSS and shreddit HTML endpoints the free path
that replaced the now-403 ``.json`` endpoints. Sends a browser User-Agent
and never raises: returns None on HTTP error, network failure, or timeout
so tiered callers can fall through to the next source.
Args:
url: Request URL
timeout: HTTP timeout per attempt in seconds
retries: Number of retries on failure (kept low these tiers fail fast)
accept: Accept header value (e.g. "application/atom+xml", "text/html")
headers: Optional extra headers merged over the defaults
Returns:
Decoded response body as text, or None on failure.
"""
merged = {
"User-Agent": BROWSER_USER_AGENT,
"Accept": accept,
"Accept-Language": "en-US,en;q=0.9",
}
if headers:
merged.update(headers)
try:
return request(
"GET", url, headers=merged, timeout=timeout, retries=retries, raw=True
)
except HTTPError as e:
log(f"get_text failed ({e}): {url}")
return None
def scrapecreators_headers(token: str) -> Dict[str, str]: def scrapecreators_headers(token: str) -> Dict[str, str]:
"""Build ScrapeCreators request headers (x-api-key + JSON content type).""" """Build ScrapeCreators request headers (x-api-key + JSON content type)."""
return { return {
+8 -2
View File
@@ -1007,8 +1007,14 @@ def _retrieve_stream(
result = polymarket.search_polymarket(subquery.search_query, from_date, to_date, depth=depth) result = polymarket.search_polymarket(subquery.search_query, from_date, to_date, depth=depth)
return polymarket.parse_polymarket_response(result, topic=subquery.search_query), {} return polymarket.parse_polymarket_response(result, topic=subquery.search_query), {}
if source == "github": if source == "github":
result = github.search_github(subquery.search_query, from_date, to_date, depth=depth, token=config.get("GITHUB_TOKEN")) # Resolve once at the pipeline boundary so search and enrich
return result, {} # share the result; otherwise each call would re-run the env
# lookup and gh-CLI subprocess fallback (up to 5s timeout each).
token = github.resolve_token(config.get("GITHUB_TOKEN"))
response = github.search_github(subquery.search_query, from_date, to_date, depth=depth, token=token)
items = github.parse_github_response(response)
items = github.enrich_with_comments(items, depth=depth, token=token)
return items, {}
if source == "pinterest": if source == "pinterest":
result = pinterest.search_pinterest( result = pinterest.search_pinterest(
subquery.search_query, from_date, to_date, subquery.search_query, from_date, to_date,
+26 -2
View File
@@ -274,7 +274,15 @@ def _sanitize_plan(
freshness_mode=freshness_mode, freshness_mode=freshness_mode,
cluster_mode=cluster_mode, cluster_mode=cluster_mode,
raw_topic=topic, raw_topic=topic,
subqueries=_normalize_subquery_weights(_trim_subqueries_for_depth(subqueries, intent, depth, eligible_sources)), subqueries=_normalize_subquery_weights(
_trim_subqueries_for_depth(
subqueries,
intent,
depth,
eligible_sources,
requested_sources=requested_sources,
)
),
source_weights=source_weights, source_weights=source_weights,
notes=[str(note).strip() for note in raw.get("notes") or [] if str(note).strip()], notes=[str(note).strip() for note in raw.get("notes") or [] if str(note).strip()],
) )
@@ -307,6 +315,7 @@ def _trim_subqueries_for_depth(
intent: str, intent: str,
depth: str, depth: str,
available_sources: list[str], available_sources: list[str],
requested_sources: list[str] | None = None,
) -> list[schema.SubQuery]: ) -> list[schema.SubQuery]:
# At non-quick depth, expand sources: use capability routing for intents # At non-quick depth, expand sources: use capability routing for intents
# that define it, or all available sources otherwise. The LLM planner may # that define it, or all available sources otherwise. The LLM planner may
@@ -336,6 +345,15 @@ def _trim_subqueries_for_depth(
for subquery in subqueries: for subquery in subqueries:
if depth in {"quick", "default"}: if depth in {"quick", "default"}:
preferred_sources = ranked_sources[:limit] preferred_sources = ranked_sources[:limit]
if requested_sources:
requested = [
source
for source in requested_sources
if source in available_sources and source in subquery.sources
]
for source in requested:
if source not in preferred_sources:
preferred_sources.append(source)
else: else:
preferred_sources = [source for source in ranked_sources if source in subquery.sources][:limit] preferred_sources = [source for source in ranked_sources if source in subquery.sources][:limit]
if len(preferred_sources) < limit: if len(preferred_sources) < limit:
@@ -428,7 +446,13 @@ def _fallback_plan(
cluster_mode=_default_cluster_mode(intent), cluster_mode=_default_cluster_mode(intent),
raw_topic=topic, raw_topic=topic,
subqueries=_normalize_subquery_weights( subqueries=_normalize_subquery_weights(
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent, topic)], intent, depth, list(source_weights)) _trim_subqueries_for_depth(
subqueries[:_max_subqueries(intent, topic)],
intent,
depth,
list(source_weights),
requested_sources=requested_sources,
)
), ),
source_weights=_normalize_weights(source_weights), source_weights=_normalize_weights(source_weights),
notes=[note], notes=[note],
@@ -0,0 +1,256 @@
"""Keyless Reddit pipeline: tiered free search + comment enrichment.
Replaces the dead ``.json`` free path. Discovery tiers, cheapest/most-likely
first; enrichment then runs on whatever was discovered:
Tier 0 one-shot legacy ``.json`` search demoted. Datacenter IPs get 403,
but a residential machine (where the skill usually runs) may still
get 200, so it is worth one cheap try. Honors the "brute-force .json"
intent without depending on it.
Tier 1 RSS discovery (reddit_rss) keyless, robust, the load-bearing path.
Tier 2 shreddit comment + count enrichment (reddit_shreddit) for top posts.
Returns ``[]`` (never raises) so ``pipeline.py`` can fall through to the
ScrapeCreators backup when every keyless tier comes up empty.
"""
import concurrent.futures
import sys
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Dict, List, Optional
from collections import Counter
from . import reddit_rss, reddit_shreddit, reddit_listing
ENRICH_LIMITS = reddit_shreddit.ENRICH_LIMITS
ENRICH_BUDGET = 45 # seconds total across all enrichment threads
MAX_ENRICH_WORKERS = 4
MAX_DERIVED_SUBS = 5 # subreddits derived from RSS results for score backfill
def _log(msg: str) -> None:
sys.stderr.write(f"[RedditKeyless] {msg}\n")
sys.stderr.flush()
def _tier0_json(topic: str, depth: str) -> List[Dict[str, Any]]:
"""One cheap global ``.json`` discovery attempt. Returns [] on the 403 wall."""
try:
from . import reddit_public
return reddit_public.search(topic, depth=depth) or []
except Exception as e: # never let the demoted tier sink the run
_log(f"Tier 0 (.json) unavailable: {e}")
return []
def _top_subreddits(posts: List[Dict[str, Any]], limit: int = MAX_DERIVED_SUBS) -> List[str]:
"""Most frequent subreddits across discovered posts (for score backfill)."""
counts = Counter(p.get("subreddit", "") for p in posts if p.get("subreddit"))
return [sub for sub, _ in counts.most_common(limit)]
def _apply_scores(post: Dict[str, Any], scored: Dict[str, int]) -> None:
post["score"] = scored["score"]
post["num_comments"] = scored["num_comments"]
post.setdefault("engagement", {})["score"] = scored["score"]
post["engagement"]["num_comments"] = scored["num_comments"]
def _discover(topic: str, depth: str, subreddits: Optional[List[str]]) -> List[Dict[str, Any]]:
# Tier 0: demoted one-shot .json (dead for normal users too, but free to try).
posts = _tier0_json(topic, depth)
if posts:
_log(f"Tier 0 (.json) returned {len(posts)} posts")
return posts
# Tier 1: keyless discovery. RSS gives breadth (incl. global keyword search);
# the listing partials give real upvote scores.
rss_posts = reddit_rss.search_rss(topic, depth=depth, subreddits=subreddits)
if subreddits:
# Targeted run: the caller chose these subreddits, so their listing cards
# are on-topic — include them as scored discovery AND as a score source.
listing_posts = reddit_listing.fetch_listings(subreddits, depth=depth, query=topic)
score_source = listing_posts
else:
# Bare global run: subreddits derived from noisy RSS results are NOT
# reliably on-topic, so their listings are used ONLY to backfill scores
# onto the keyword-matched RSS posts — never merged as discovery, which
# would flood results with high-upvote but irrelevant posts.
listing_posts = []
derived = _top_subreddits(rss_posts)
score_source = reddit_listing.fetch_listings(derived, depth=depth, query=topic)
_log(
f"Tier 1 (RSS) {len(rss_posts)} posts; "
f"{'listing discovery ' + str(len(listing_posts)) if subreddits else 'score-only'}; "
f"{len(score_source)} scored cards"
)
# Score lookup by post id, from the scored listing cards.
score_map: Dict[str, Dict[str, int]] = {}
for p in score_source:
pid = p.get("metadata", {}).get("post_id", "")
if pid:
score_map[pid] = {"score": p["score"], "num_comments": p["num_comments"]}
# Merge: scored listing posts first (targeted only), then RSS breadth,
# backfilled with real scores where the post appears in a listing.
merged: List[Dict[str, Any]] = []
seen: set = set()
for p in listing_posts:
if p["url"] not in seen:
seen.add(p["url"])
merged.append(p)
for p in rss_posts:
if p["url"] in seen:
continue
pid = reddit_listing._post_id(p["url"])
if pid in score_map:
_apply_scores(p, score_map[pid])
seen.add(p["url"])
merged.append(p)
return merged
def _enrich_one(post: Dict[str, Any]) -> Dict[str, Any]:
"""Attach shreddit comments + real comment count. Never raises."""
try:
data = reddit_shreddit.fetch_comments(post.get("url", ""))
if data.get("top_comments"):
post["top_comments"] = data["top_comments"]
if data.get("comment_insights"):
post["comment_insights"] = data["comment_insights"]
num = data.get("num_comments")
if num is not None:
post["num_comments"] = num
post.setdefault("engagement", {})["num_comments"] = num
except Exception:
pass # keep the post with whatever discovery gave us
return post
def _enrich(posts: List[Dict[str, Any]], depth: str) -> List[Dict[str, Any]]:
"""Enrich the top N posts with comments under a total time budget."""
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
to_enrich = posts[:limit]
rest = posts[limit:]
if not to_enrich:
return posts
result_map: Dict[int, Dict[str, Any]] = {}
try:
with ThreadPoolExecutor(max_workers=min(limit, MAX_ENRICH_WORKERS)) as executor:
futures = {
executor.submit(_enrich_one, post): i
for i, post in enumerate(to_enrich)
}
done, not_done = concurrent.futures.wait(futures, timeout=ENRICH_BUDGET)
for future in done:
idx = futures[future]
try:
result_map[idx] = future.result(timeout=0)
except Exception:
result_map[idx] = to_enrich[idx]
for future in not_done:
idx = futures[future]
result_map[idx] = to_enrich[idx]
future.cancel()
enriched = [result_map[i] for i in range(len(to_enrich))]
except Exception:
enriched = to_enrich
return enriched + rest
def _slot_priority(topic: str, posts: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Order posts for enrichment slots: entity-matching posts first.
Comment slots (ENRICH_LIMITS) are scarce; spending them on high-upvote
posts that rerank later demotes as entity misses starves the on-topic
posts the user actually sees (2026-06-06 "OpenClaw vs Hermes" run:
2,000+ upvote Gemma/GPU threads took every slot, then were demoted to
zero). Mirror rerank's demotion signal — the topic's stripped primary
entity contained in the post text so slots go to posts likely to
survive final ranking. Falls back to token-overlap relevance when the
topic yields no usable primary entity. Within each tier the incoming
(score-first) order is preserved. Never raises; on any failure the
incoming order is returned unchanged.
"""
try:
from . import relevance, rerank
def _post_text(post: Dict[str, Any]) -> str:
return f"{post.get('title') or ''} {post.get('selftext') or ''}"
entity = rerank._primary_entity(topic).lower()
if entity:
def _matches(post: Dict[str, Any]) -> bool:
return entity in _post_text(post).lower()
else:
prepared = relevance.PreparedQuery(topic)
def _matches(post: Dict[str, Any]) -> bool:
return relevance.token_overlap_relevance(prepared, _post_text(post)) > 0.24
matches: List[Dict[str, Any]] = []
misses: List[Dict[str, Any]] = []
for post in posts:
(matches if _matches(post) else misses).append(post)
return matches + misses
except Exception:
return posts
def search_and_enrich(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""Full keyless Reddit pipeline: discover (Tier 0/1) then enrich (Tier 2).
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
subreddits: Optional pre-resolved subreddit names (without r/)
Returns:
List of normalized item dicts matching the reddit_public output shape,
with top_comments/comment_insights attached on enriched posts.
Empty list when all keyless tiers fail (so SC backup can engage).
"""
posts = _discover(topic, depth, subreddits)
if not posts:
return []
# Date filter: keep posts in range or with unknown dates (mirrors reddit_public).
posts = [
p for p in posts
if p.get("date") is None or (from_date <= p["date"] <= to_date)
]
# Rank by real upvote score (from listing cards / backfill), then query
# relevance, then recency. Posts without a recovered score sort by the
# latter two — same behavior as before scores were available.
posts.sort(
key=lambda p: (
p.get("engagement", {}).get("score", 0) or 0,
p.get("relevance", 0) or 0,
p.get("date") or "",
),
reverse=True,
)
# Enrichment slot selection is relevance-aware: entity-matching posts
# claim the scarce comment slots first (score order preserved within
# each tier). The score-first sort above still governs within-tier order.
posts = _enrich(_slot_priority(topic, posts), depth)
for i, post in enumerate(posts):
post["id"] = f"R{i + 1}"
return posts
@@ -0,0 +1,183 @@
"""Keyless Reddit listing scrape via shreddit /svc partials — with real scores.
The subreddit listing partial
``/svc/shreddit/community-more-posts/{sort}/?name={sub}[&t={range}]`` serves
HTTP 200 with no API key and **server-renders each post's upvote score**, which
neither RSS nor the comments endpoint provides. Each post is a
``<shreddit-post>`` element whose start-tag attributes carry ``score``,
``comment-count``, ``post-title``, ``permalink``, ``author``, ``subreddit-name``
and ``created-timestamp``.
This is the keyless source of post-level upvotes. It works for normal users on
ordinary connections (verified), so reddit_keyless uses it both as a scored
discovery source and to backfill scores onto RSS-discovered posts.
"""
import html as _html
import re
import sys
from datetime import datetime, timezone
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from typing import Any, Dict, List, Optional
from . import http
from .relevance import token_overlap_relevance
# Listing sorts pulled per subreddit, by depth.
LISTING_SORTS = {
"quick": ["top"],
"default": ["top", "hot"],
"deep": ["top", "hot", "new"],
}
DEPTH_LIMITS = {"quick": 10, "default": 25, "deep": 50}
TIMEFRAME = "month"
MAX_WORKERS = 4
LISTING_TIMEOUT = 15
_POST_CARD = re.compile(r"<shreddit-post(?=[\s>])[^>]*>")
def _log(msg: str) -> None:
sys.stderr.write(f"[RedditListing] {msg}\n")
sys.stderr.flush()
def _attr(tag: str, name: str) -> Optional[str]:
m = re.search(rf'\b{name}="([^"]*)"', tag)
return _html.unescape(m.group(1)) if m else None
def _to_date(value: Optional[str]) -> Optional[str]:
if not value:
return None
try:
return datetime.fromisoformat(value.strip()).date().isoformat()
except (ValueError, TypeError):
return None
def _to_epoch(value: Optional[str]) -> Optional[float]:
if not value:
return None
try:
dt = datetime.fromisoformat(value.strip())
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.timestamp()
except (ValueError, TypeError):
return None
def _post_id(permalink: str) -> str:
m = re.search(r"/comments/([A-Za-z0-9]+)", permalink or "")
return m.group(1) if m else ""
def parse_cards(html_text: str, query: str = "") -> List[Dict[str, Any]]:
"""Parse <shreddit-post> cards into normalized post dicts with real scores."""
posts: List[Dict[str, Any]] = []
for m in _POST_CARD.finditer(html_text or ""):
tag = m.group(0)
permalink = _attr(tag, "permalink") or ""
if "/comments/" not in permalink:
continue
try:
score = int(_attr(tag, "score") or 0)
except ValueError:
score = 0
try:
num_comments = int(_attr(tag, "comment-count") or 0)
except ValueError:
num_comments = 0
title = _attr(tag, "post-title") or ""
author = _attr(tag, "author") or "[deleted]"
subreddit = _attr(tag, "subreddit-name") or ""
created = _attr(tag, "created-timestamp")
url = f"https://www.reddit.com{permalink}"
posts.append({
"id": "",
"title": title,
"url": url,
"score": score,
"num_comments": num_comments,
"subreddit": subreddit,
"created_utc": _to_epoch(created),
"author": author if author not in ("[deleted]", "[removed]") else "[deleted]",
"selftext": "",
"date": _to_date(created),
"engagement": {
"score": score,
"num_comments": num_comments,
"upvote_ratio": None,
},
"relevance": round(token_overlap_relevance(query, title), 3) if query else 0.0,
"why_relevant": "Reddit listing",
"metadata": {"post_id": _post_id(permalink)},
})
return posts
def _listing_url(subreddit: str, sort: str) -> str:
sub = subreddit.removeprefix("r/").strip()
url = f"https://www.reddit.com/svc/shreddit/community-more-posts/{sort}/?name={sub}"
if sort == "top":
url += f"&t={TIMEFRAME}"
return url
def _fetch_one(subreddit: str, sort: str, query: str) -> List[Dict[str, Any]]:
try:
text = http.get_text(_listing_url(subreddit, sort), timeout=LISTING_TIMEOUT,
accept="text/html")
return parse_cards(text, query) if text else []
except Exception as e:
_log(f"listing fetch failed r/{subreddit} {sort}: {e}")
return []
def fetch_listings(
subreddits: List[str],
depth: str = "default",
query: str = "",
) -> List[Dict[str, Any]]:
"""Fetch scored post cards across subreddits × depth-appropriate sorts.
Returns deduped normalized posts (with real scores), unranked/unsliced
the caller merges these with other sources, ranks, and slices.
"""
if not subreddits:
return []
sorts = LISTING_SORTS.get(depth, LISTING_SORTS["default"])
jobs = [(sub, sort) for sub in subreddits for sort in sorts]
all_posts: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=min(MAX_WORKERS, len(jobs)) or 1) as executor:
futures = {executor.submit(_fetch_one, sub, sort, query): (sub, sort)
for sub, sort in jobs}
for future in futures:
try:
all_posts.extend(future.result(timeout=LISTING_TIMEOUT + 5))
except (Exception, FuturesTimeoutError) as e:
_log(f"listing future failed: {e}")
seen: set = set()
unique: List[Dict[str, Any]] = []
for p in all_posts:
if p["url"] not in seen:
seen.add(p["url"])
unique.append(p)
return unique
def score_index(subreddits: List[str], depth: str = "default") -> Dict[str, Dict[str, int]]:
"""Build a {post_id: {score, num_comments}} map from subreddit listings.
Used to backfill real scores onto posts discovered via RSS, which carries
no engagement numbers.
"""
index: Dict[str, Dict[str, int]] = {}
for p in fetch_listings(subreddits, depth=depth):
pid = p.get("metadata", {}).get("post_id") or _post_id(p["url"])
if pid:
index[pid] = {"score": p["score"], "num_comments": p["num_comments"]}
return index
+25 -141
View File
@@ -1,9 +1,16 @@
"""Standalone Reddit public JSON search module. """Reddit public ``.json`` search module (demoted to keyless Tier 0).
Searches Reddit using the free public JSON endpoints (no API key required). Reddit's public ``.json`` endpoints now return HTTP 403 from most contexts
Promoted from last-resort fallback to robust primary free path. (shreddit anti-bot), so this is no longer the primary free path. The keyless
pipeline (see reddit_keyless.py) still calls ``search`` as a cheap one-shot
Tier 0 attempt a residential machine may occasionally get a 200 before
falling through to RSS discovery (reddit_rss.py) and shreddit comment
enrichment (reddit_shreddit.py).
Endpoints: ``search_reddit_public`` is retained as a compatibility shim that delegates to
the keyless pipeline, so existing callers (pipeline.py) need no change.
Endpoints (Tier 0):
- Global: https://www.reddit.com/search.json?q={query}&sort=relevance&t=month&limit={limit} - Global: https://www.reddit.com/search.json?q={query}&sort=relevance&t=month&limit={limit}
- Subreddit: https://www.reddit.com/r/{sub}/search.json?q={query}&restrict_sr=on&sort=relevance&t=month - Subreddit: https://www.reddit.com/r/{sub}/search.json?q={query}&restrict_sr=on&sort=relevance&t=month
@@ -18,7 +25,6 @@ import time
import urllib.error import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from typing import Any, Dict, List, Optional from typing import Any, Dict, List, Optional
@@ -35,13 +41,6 @@ DEPTH_LIMITS = {
"deep": 50, "deep": 50,
} }
# How many top posts to enrich with comments, by depth
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
MAX_RETRIES = 3 MAX_RETRIES = 3
BASE_BACKOFF = 2.0 # seconds BASE_BACKOFF = 2.0 # seconds
@@ -237,78 +236,6 @@ def search(
return unique[:limit] return unique[:limit]
def _enrich_post(item: Dict[str, Any], timeout: int = 10) -> Dict[str, Any]:
"""Enrich a single post with top comments. Never raises."""
try:
from . import reddit_enrich
thread_data = reddit_enrich.fetch_thread_data(item["url"], timeout=timeout)
if not thread_data:
return item
parsed = reddit_enrich.parse_thread_data(thread_data)
comments = parsed.get("comments", [])
top = reddit_enrich.get_top_comments(comments)
item["top_comments"] = [
{
"score": c.get("score", 0),
"excerpt": (c.get("body") or "")[:200],
"author": c.get("author", ""),
}
for c in top[:10]
]
except Exception:
# Never discard — keep post with empty metadata
pass
return item
def _enrich_posts(posts: List[Dict[str, Any]], depth: str = "default") -> List[Dict[str, Any]]:
"""Enrich top N posts with comment data using threads. Total budget 45s."""
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
to_enrich = posts[:limit]
rest = posts[limit:]
if not to_enrich:
return posts
enriched = []
try:
with ThreadPoolExecutor(max_workers=min(limit, 4)) as executor:
futures = {
executor.submit(_enrich_post, post, 10): i
for i, post in enumerate(to_enrich)
}
# Collect results with 45s total budget
import concurrent.futures
done, not_done = concurrent.futures.wait(futures, timeout=45)
# Build result list preserving order
result_map: Dict[int, Dict[str, Any]] = {}
for future in done:
idx = futures[future]
try:
result_map[idx] = future.result(timeout=0)
except Exception:
result_map[idx] = to_enrich[idx]
# Any not-done futures: keep original post
for future in not_done:
idx = futures[future]
result_map[idx] = to_enrich[idx]
future.cancel()
enriched = [result_map[i] for i in range(len(to_enrich))]
except Exception:
enriched = to_enrich
return enriched + rest
def _search_subreddit(sub: str, topic: str, depth: str, timeout: int = 15) -> List[Dict[str, Any]]:
"""Search a single subreddit. Never raises."""
try:
return search(topic, depth=depth, subreddit=sub, timeout=timeout)
except Exception as e:
_log(f"Subreddit search failed for r/{sub}: {e}")
return []
def search_reddit_public( def search_reddit_public(
topic: str, topic: str,
from_date: str, from_date: str,
@@ -316,12 +243,17 @@ def search_reddit_public(
depth: str = "default", depth: str = "default",
subreddits: Optional[List[str]] = None, subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]: ) -> List[Dict[str, Any]]:
"""High-level Reddit public search matching the openai_reddit interface. """High-level free Reddit search + enrichment (keyless).
When subreddits are provided (from agent planning), searches each targeted Thin compatibility shim over the tiered keyless pipeline: the legacy
sub first, then does global search, and deduplicates across both. This ``.json`` search/enrichment endpoints now return HTTP 403, so this delegates
mirrors the SC search_and_enrich() flow where pre-resolved subreddits get to ``reddit_keyless.search_and_enrich`` (Tier 0 one-shot ``.json``
priority. Tier 1 RSS discovery Tier 2 shreddit comment enrichment). The name and
signature are preserved so ``pipeline.py`` and other callers need no change
and the ScrapeCreators backup still engages when this returns empty.
The module-level ``search`` / ``_parse_posts`` helpers remain in use as the
keyless pipeline's demoted Tier 0 ``.json`` attempt.
Args: Args:
topic: Search topic topic: Search topic
@@ -332,57 +264,9 @@ def search_reddit_public(
Returns: Returns:
List of normalized item dicts matching ScrapeCreators output format. List of normalized item dicts matching ScrapeCreators output format.
Empty list on total failure (so SC backup can engage).
""" """
all_posts: List[Dict[str, Any]] = [] from . import reddit_keyless
return reddit_keyless.search_and_enrich(
# Phase 1: Search targeted subreddits in parallel (if provided) topic, from_date, to_date, depth=depth, subreddits=subreddits
if subreddits:
_log(f"Searching {len(subreddits)} targeted subreddits: {subreddits}")
workers = min(4, len(subreddits))
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {
executor.submit(_search_subreddit, sub, topic, depth): sub
for sub in subreddits
}
for future in futures:
sub = futures[future]
try:
sub_posts = future.result(timeout=30)
_log(f" -> {len(sub_posts)} results from r/{sub}")
all_posts.extend(sub_posts)
except (Exception, FuturesTimeoutError) as e:
_log(f" -> r/{sub} failed: {e}")
# Phase 2: Global search
global_posts = search(topic, depth=depth)
all_posts.extend(global_posts)
# Deduplicate by URL (targeted results keep priority since they come first)
seen_urls: set = set()
results: List[Dict[str, Any]] = []
for post in all_posts:
if post["url"] not in seen_urls:
seen_urls.add(post["url"])
results.append(post)
# Date filter: keep posts in range or with unknown dates
filtered = []
for item in results:
d = item.get("date")
if d is None or (from_date <= d <= to_date):
filtered.append(item)
# Sort by engagement (score desc)
filtered.sort(
key=lambda x: x.get("engagement", {}).get("score", 0),
reverse=True,
) )
# Enrich top posts with comments
filtered = _enrich_posts(filtered, depth=depth)
# Re-index IDs
for i, item in enumerate(filtered):
item["id"] = f"R{i + 1}"
return filtered
+224
View File
@@ -0,0 +1,224 @@
"""Keyless Reddit discovery via public RSS/Atom feeds.
Reddit's ``.json`` search endpoints now return HTTP 403 (shreddit anti-bot).
RSS feeds still serve HTTP 200 with no API key, so this module uses them for
post discovery, replacing ``reddit_public.search`` as the free search path.
Two feed families are combined and deduped:
- search: /search.rss?q=... and /r/{sub}/search.rss?q=...&restrict_sr=on
- listing: /r/{sub}/{top,hot}.rss?t=month
RSS entries carry no engagement score, so ``score``/``num_comments`` start at 0
and are backfilled during shreddit enrichment (see reddit_shreddit.py). Output
dicts match the normalized shape emitted by ``reddit_public._parse_posts`` so
downstream code (pipeline, renderer) is unaffected.
"""
import sys
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeoutError
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from urllib.parse import quote_plus
from . import http
from .relevance import token_overlap_relevance
ATOM = "{http://www.w3.org/2005/Atom}"
# Mirror reddit_public depth-aware limits so the two free paths behave alike.
DEPTH_LIMITS = {
"quick": 10,
"default": 25,
"deep": 50,
}
# Listing sorts pulled per subreddit (in addition to search), for volume.
LISTING_SORTS = {
"quick": ["top"],
"default": ["top", "hot"],
"deep": ["top", "hot", "new"],
}
MAX_WORKERS = 4
FEED_TIMEOUT = 15
def _log(msg: str) -> None:
sys.stderr.write(f"[RedditRSS] {msg}\n")
sys.stderr.flush()
def _iso_to_date(value: Optional[str]) -> Optional[str]:
"""Parse an ISO-8601 timestamp (e.g. 2026-05-20T18:48:31+00:00) to YYYY-MM-DD."""
if not value:
return None
try:
dt = datetime.fromisoformat(value.strip())
return dt.date().isoformat()
except (ValueError, TypeError):
return None
def _iso_to_epoch(value: Optional[str]) -> Optional[float]:
if not value:
return None
try:
dt = datetime.fromisoformat(value.strip())
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.timestamp()
except (ValueError, TypeError):
return None
def _subreddit_from(category: str, url: str) -> str:
"""Derive subreddit name from the entry category or, failing that, the URL."""
if category:
return category
# URL form: https://www.reddit.com/r/{sub}/comments/{id}/...
parts = url.split("/r/", 1)
if len(parts) == 2:
return parts[1].split("/", 1)[0]
return ""
def _parse_feed(xml_text: str, query: str = "") -> List[Dict[str, Any]]:
"""Parse an Atom feed string into normalized post dicts. Never raises."""
if not xml_text:
return []
try:
root = ET.fromstring(xml_text)
except ET.ParseError as e:
_log(f"feed parse error: {e}")
return []
posts: List[Dict[str, Any]] = []
for entry in root.iter(f"{ATOM}entry"):
link_el = entry.find(f"{ATOM}link")
url = link_el.get("href", "").strip() if link_el is not None else ""
if not url or "/comments/" not in url:
continue
title_el = entry.find(f"{ATOM}title")
title = (title_el.text or "").strip() if title_el is not None else ""
author = ""
author_el = entry.find(f"{ATOM}author/{ATOM}name")
if author_el is not None and author_el.text:
author = author_el.text.strip().removeprefix("/u/").removeprefix("u/")
if author in ("[deleted]", "[removed]", ""):
author = "[deleted]"
cat_el = entry.find(f"{ATOM}category")
category = cat_el.get("term", "").strip() if cat_el is not None else ""
subreddit = _subreddit_from(category, url)
updated_el = entry.find(f"{ATOM}updated")
updated = (updated_el.text or "").strip() if updated_el is not None else ""
content_el = entry.find(f"{ATOM}content")
selftext = ""
if content_el is not None and content_el.text:
# Strip the simplest HTML; renderer only needs an excerpt.
import re as _re
selftext = _re.sub(r"<[^>]+>", " ", content_el.text)
selftext = _re.sub(r"\s+", " ", selftext).strip()[:500]
relevance = round(token_overlap_relevance(query, title), 3) if query else 0.0
posts.append({
"id": "", # assigned after dedup
"title": title,
"url": url,
"score": 0, # backfilled by shreddit enrichment
"num_comments": 0, # backfilled by shreddit enrichment
"subreddit": subreddit,
"created_utc": _iso_to_epoch(updated),
"author": author,
"selftext": selftext,
"date": _iso_to_date(updated),
"engagement": {
"score": 0,
"num_comments": 0,
"upvote_ratio": None,
},
"relevance": relevance,
"why_relevant": "Reddit RSS",
"metadata": {},
})
return posts
def _build_urls(query: str, depth: str, subreddits: Optional[List[str]]) -> List[str]:
"""Build the keyless RSS feed URLs to fan out across."""
q = quote_plus(query)
urls: List[str] = [
f"https://www.reddit.com/search.rss?q={q}&sort=relevance&t=month"
]
for raw_sub in (subreddits or []):
sub = raw_sub.removeprefix("r/").strip()
if not sub:
continue
urls.append(
f"https://www.reddit.com/r/{sub}/search.rss"
f"?q={q}&restrict_sr=on&sort=relevance&t=month"
)
for sort in LISTING_SORTS.get(depth, LISTING_SORTS["default"]):
urls.append(f"https://www.reddit.com/r/{sub}/{sort}.rss?t=month")
return urls
def _fetch_feed(url: str, query: str) -> List[Dict[str, Any]]:
"""Fetch and parse one feed. Never raises."""
try:
text = http.get_text(url, timeout=FEED_TIMEOUT, accept="application/atom+xml")
return _parse_feed(text, query) if text else []
except Exception as e: # defensive: a single bad feed must not sink the run
_log(f"feed fetch failed for {url}: {e}")
return []
def search_rss(
query: str,
depth: str = "default",
subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""Discover Reddit posts for a query via keyless RSS feeds.
Args:
query: Search query string
depth: 'quick', 'default', or 'deep' controls result limit and feeds
subreddits: Optional pre-resolved subreddit names (without r/) to target
Returns:
List of normalized post dicts (deduped by URL, capped by depth),
with placeholder scores to be backfilled during enrichment.
Empty list on any failure.
"""
limit = DEPTH_LIMITS.get(depth, DEPTH_LIMITS["default"])
urls = _build_urls(query, depth, subreddits)
all_posts: List[Dict[str, Any]] = []
workers = min(MAX_WORKERS, len(urls)) or 1
with ThreadPoolExecutor(max_workers=workers) as executor:
futures = {executor.submit(_fetch_feed, url, query): url for url in urls}
for future in futures:
try:
all_posts.extend(future.result(timeout=FEED_TIMEOUT + 5))
except (Exception, FuturesTimeoutError) as e:
_log(f"feed future failed: {e}")
# Dedupe by URL (first occurrence wins).
seen: set = set()
unique: List[Dict[str, Any]] = []
for post in all_posts:
if post["url"] not in seen:
seen.add(post["url"])
unique.append(post)
for i, post in enumerate(unique):
post["id"] = f"R{i + 1}"
return unique[:limit]
@@ -0,0 +1,184 @@
"""Keyless Reddit comment enrichment via shreddit /svc endpoints.
Reddit's ``{thread}.json`` endpoint now returns HTTP 403. The shreddit partial
endpoint ``/svc/shreddit/comments/r/{sub}/t3_{id}`` still serves HTTP 200 HTML
with no API key, embedding each comment as a ``<shreddit-comment>`` custom
element whose start-tag attributes carry ``score`` / ``author`` / ``created`` /
``permalink``, and whose body lives in a ``<div id="{thingId}-post-rtjson-content">``
block. This module parses that markup into top comments, matching the
``top_comments`` / ``comment_insights`` shape produced by ``reddit_enrich`` so
the renderer is unaffected.
Limitation: the comments endpoint carries the real comment count
(``total-comments``) but not the post's upvote score, so post-level ``score``
cannot be recovered keylessly here (ScrapeCreators backup still provides it).
"""
import html as _html
import re
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional
from . import http
from . import reddit_enrich
# Up to N posts enriched per run, by depth (mirrors reddit_public.ENRICH_LIMITS).
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 8,
}
# Max comments returned per post (independent of how many posts get enriched).
MAX_COMMENTS = 10
SVC_TIMEOUT = 12
# Match the exact <shreddit-comment> element start tag, not <shreddit-comment-tree>
# or <shreddit-comment-tree-stats> (lookahead requires whitespace or '>').
_COMMENT_START = re.compile(r"<shreddit-comment(?=[\s>])[^>]*>")
_TOTAL_COMMENTS = re.compile(r'total-comments="(\d+)"')
_PARA = re.compile(r"<p[^>]*>(.*?)</p>", re.S)
_TAG = re.compile(r"<[^>]+>")
_WS = re.compile(r"\s+")
_NEXT_RTJSON = re.compile(r'id="t1_[A-Za-z0-9]+-(?:comment|post)-rtjson-content"')
def _log(msg: str) -> None:
sys.stderr.write(f"[RedditShreddit] {msg}\n")
sys.stderr.flush()
def extract_post_ref(url: str) -> Optional[tuple]:
"""Return (subreddit, post_id) from a Reddit thread URL, or None."""
m = re.search(r"/r/([^/]+)/comments/([A-Za-z0-9]+)", url or "")
if not m:
return None
return m.group(1), m.group(2)
def _svc_url(subreddit: str, post_id: str) -> str:
# sort=top guarantees Reddit front-loads the highest-scored comments on the
# first page, so the true top comments are captured even on huge threads
# (we still re-sort by score locally as a backstop).
return (
f"https://www.reddit.com/svc/shreddit/comments/r/{subreddit}/t3_{post_id}"
f"?sort=top"
)
def _attr(tag: str, name: str) -> str:
m = re.search(rf'\b{name}="([^"]*)"', tag)
return _html.unescape(m.group(1)) if m else ""
def _iso_to_date(value: str) -> Optional[str]:
if not value:
return None
try:
return datetime.fromisoformat(value.strip()).date().isoformat()
except (ValueError, TypeError):
return None
def _body_for(html_text: str, thing_id: str) -> str:
"""Extract a comment's text body, anchored on its unique thingId.
The body div id embeds the comment's thingId, so this assigns body→comment
correctly even for nested replies. The slice is bounded by the next
comment's rtjson anchor to avoid swallowing child-comment text.
"""
if not thing_id:
return ""
anchor = f'id="{thing_id}-post-rtjson-content"'
idx = html_text.find(anchor)
if idx == -1:
return ""
window = html_text[idx + len(anchor): idx + len(anchor) + 8000]
nxt = _NEXT_RTJSON.search(window)
if nxt:
window = window[: nxt.start()]
paras = _PARA.findall(window)
if not paras:
return ""
text = " ".join(_TAG.sub("", p) for p in paras)
return _WS.sub(" ", _html.unescape(text)).strip()
def parse_comments(html_text: str, limit: int = MAX_COMMENTS) -> List[Dict[str, Any]]:
"""Parse <shreddit-comment> elements into scored comment dicts (sorted desc)."""
comments: List[Dict[str, Any]] = []
for m in _COMMENT_START.finditer(html_text or ""):
tag = m.group(0)
author = _attr(tag, "author") or "[deleted]"
if author in ("[deleted]", "[removed]"):
continue
thing_id = _attr(tag, "thingId")
body = _body_for(html_text, thing_id)
if not body or body in ("[deleted]", "[removed]"):
continue
try:
score = int(_attr(tag, "score") or 0)
except ValueError:
score = 0
permalink = _attr(tag, "permalink")
comments.append({
"score": score,
"author": author,
"body": body[:300],
"excerpt": body[:200],
"permalink": permalink,
"date": _iso_to_date(_attr(tag, "created")),
"url": f"https://reddit.com{permalink}" if permalink else "",
})
comments.sort(key=lambda c: c.get("score", 0), reverse=True)
return comments[:limit]
def _total_comments(html_text: str) -> Optional[int]:
m = _TOTAL_COMMENTS.search(html_text or "")
return int(m.group(1)) if m else None
def fetch_comments(
post_url: str,
timeout: int = SVC_TIMEOUT,
) -> Dict[str, Any]:
"""Fetch and parse top comments for a Reddit post via the shreddit endpoint.
Args:
post_url: Reddit thread URL (/r/{sub}/comments/{id}/)
timeout: HTTP timeout in seconds
Returns:
Dict with 'top_comments' (list, reddit_enrich shape), 'comment_insights'
(list[str]), and 'num_comments' (int or None). Empty/None on any
failure never raises, so the caller can fall through to SC backup.
"""
ref = extract_post_ref(post_url)
if not ref:
return {"top_comments": [], "comment_insights": [], "num_comments": None}
sub, post_id = ref
html_text = http.get_text(_svc_url(sub, post_id), timeout=timeout, accept="text/html")
if not html_text:
return {"top_comments": [], "comment_insights": [], "num_comments": None}
comments = parse_comments(html_text, limit=MAX_COMMENTS)
insights = reddit_enrich.extract_comment_insights(comments)
return {
"top_comments": [
{
"score": c["score"],
"date": c["date"],
"author": c["author"],
"excerpt": c["excerpt"],
"url": c["url"],
}
for c in comments
],
"comment_insights": insights,
"num_comments": _total_comments(html_text),
}
+98
View File
@@ -360,6 +360,22 @@ def update_run(run_id: int, **kwargs):
conn.close() conn.close()
def get_latest_completed_runs(topic_id: int, limit: int = 2) -> List[Dict[str, Any]]:
"""Return newest completed runs for a topic."""
conn = _connect()
try:
rows = conn.execute(
"""SELECT * FROM research_runs
WHERE topic_id = ? AND status = 'completed'
ORDER BY datetime(run_date) DESC, id DESC
LIMIT ?""",
(topic_id, limit),
).fetchall()
return [dict(r) for r in rows]
finally:
conn.close()
# --- Findings --- # --- Findings ---
@@ -536,6 +552,88 @@ def get_sightings_for_run(topic_id: int, run_id: int) -> List[Dict[str, Any]]:
conn.close() conn.close()
def compute_topic_delta(topic_id: int) -> Dict[str, Any]:
"""Compare the latest completed watchlist run with the previous run."""
runs = get_latest_completed_runs(topic_id, limit=2)
topic = _get_topic_by_id(topic_id)
topic_name = topic["name"] if topic else str(topic_id)
if len(runs) < 2:
return {
"topic": topic_name,
"status": "insufficient_history",
"message": "Need at least two completed runs to compute a delta.",
}
current_run, previous_run = runs[0], runs[1]
current = _sightings_by_url(get_sightings_for_run(topic_id, current_run["id"]))
previous = _sightings_by_url(get_sightings_for_run(topic_id, previous_run["id"]))
current_urls = set(current)
previous_urls = set(previous)
new_urls = sorted(current_urls - previous_urls)
continued_urls = sorted(current_urls & previous_urls)
dropped_urls = sorted(previous_urls - current_urls)
findings = {
"new": [current[url] for url in new_urls],
"continued": [current[url] for url in continued_urls],
"dropped": [previous[url] for url in dropped_urls],
}
return {
"topic": topic_name,
"status": "ok",
"current_run_id": current_run["id"],
"previous_run_id": previous_run["id"],
"new": len(new_urls),
"continued": len(continued_urls),
"dropped": len(dropped_urls),
"sources": _delta_source_counts(findings),
"findings": findings,
}
def _get_topic_by_id(topic_id: int) -> Optional[Dict[str, Any]]:
conn = _connect()
try:
row = conn.execute("SELECT * FROM topics WHERE id = ?", (topic_id,)).fetchone()
return dict(row) if row else None
finally:
conn.close()
def _sightings_by_url(sightings: List[Dict[str, Any]]) -> Dict[str, Dict[str, Any]]:
"""Index sightings by stable URL identity for run-to-run delta comparisons.
URL-less sightings are intentionally excluded because there is no stable
cross-run identity to classify them as new, continued, or dropped.
"""
return {
sighting["source_url"]: sighting
for sighting in sightings
if sighting.get("source_url")
}
def _delta_source_counts(
findings: Dict[str, List[Dict[str, Any]]]
) -> Dict[str, Dict[str, int]]:
sources = sorted({
finding.get("source") or "unknown"
for group in findings.values()
for finding in group
})
counts = {
source: {"new": 0, "continued": 0, "dropped": 0}
for source in sources
}
for group_name, group in findings.items():
for finding in group:
source = finding.get("source") or "unknown"
counts[source][group_name] += 1
return counts
def get_new_findings( def get_new_findings(
topic_id: int, topic_id: int,
since: Optional[str] = None, since: Optional[str] = None,
+12
View File
@@ -111,6 +111,14 @@ def cmd_list(args):
}, default=str)) }, default=str))
def cmd_delta(args):
topic = store.get_topic(args.topic)
if not topic:
print(json.dumps({"error": f'Topic not found: "{args.topic}"'}))
sys.exit(1)
print(json.dumps(store.compute_topic_delta(topic["id"]), default=str))
def cmd_run_one(args): def cmd_run_one(args):
topic = store.get_topic(args.topic) topic = store.get_topic(args.topic)
if not topic: if not topic:
@@ -252,6 +260,10 @@ def build_parser() -> argparse.ArgumentParser:
list_parser = sub.add_parser("list") list_parser = sub.add_parser("list")
list_parser.set_defaults(func=cmd_list) list_parser.set_defaults(func=cmd_list)
delta = sub.add_parser("delta")
delta.add_argument("topic")
delta.set_defaults(func=cmd_delta)
run_one = sub.add_parser("run-one") run_one = sub.add_parser("run-one")
run_one.add_argument("topic") run_one.add_argument("topic")
run_one.set_defaults(func=cmd_run_one) run_one.set_defaults(func=cmd_run_one)
+4
View File
@@ -0,0 +1,4 @@
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "skills" / "last30days" / "scripts"))
-5
View File
@@ -5,11 +5,7 @@ comparisons, 'difference between X and Y' phrasing, trailing context
leaking into entities, degenerate inputs, and false-positive resistance. leaking into entities, degenerate inputs, and false-positive resistance.
""" """
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import planner from lib import planner
@@ -193,6 +189,5 @@ class TestNoiseWordEntities(unittest.TestCase):
entities = planner._comparison_entities("Swift vs Rust vs Go") entities = planner._comparison_entities("Swift vs Rust vs Go")
self.assertTrue(any("Go" in e for e in entities)) self.assertTrue(any("Go" in e for e in entities))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-6
View File
@@ -2,16 +2,12 @@ import json
import os import os
import shutil import shutil
import subprocess import subprocess
import sys
import textwrap import textwrap
import unittest import unittest
from pathlib import Path from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib.bird_x import parse_bird_response from lib.bird_x import parse_bird_response
REPO_ROOT = Path(__file__).resolve().parents[1] REPO_ROOT = Path(__file__).resolve().parents[1]
VENDORED_BIRD = REPO_ROOT / "skills" / "last30days" / "scripts" / "lib" / "vendor" / "bird-search" / "bird-search.mjs" VENDORED_BIRD = REPO_ROOT / "skills" / "last30days" / "scripts" / "lib" / "vendor" / "bird-search" / "bird-search.mjs"
@@ -31,7 +27,6 @@ class TestBirdXEngagementZero(unittest.TestCase):
self.assertEqual(0, items[0]["engagement"]["likes"]) self.assertEqual(0, items[0]["engagement"]["likes"])
self.assertEqual(5, items[0]["engagement"]["reposts"]) self.assertEqual(5, items[0]["engagement"]["reposts"])
@unittest.skipUnless(shutil.which("node"), "node is required for vendored Bird tests") @unittest.skipUnless(shutil.which("node"), "node is required for vendored Bird tests")
class TestVendoredBirdRuntime(unittest.TestCase): class TestVendoredBirdRuntime(unittest.TestCase):
def test_check_uses_env_credentials_without_browser_cookie_dependency(self): def test_check_uses_env_credentials_without_browser_cookie_dependency(self):
@@ -305,6 +300,5 @@ class TestRunBirdSearchJsonDecodeRetry(unittest.TestCase):
self.assertEqual(response, timeout_error) self.assertEqual(response, timeout_error)
mock_sleep.assert_not_called() mock_sleep.assert_not_called()
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,12 +1,9 @@
"""Tests for bluesky module.""" """Tests for bluesky module."""
import os import os
import sys
import unittest import unittest
from pathlib import Path
from unittest.mock import patch, MagicMock from unittest.mock import patch, MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import bluesky from lib import bluesky
@@ -357,6 +354,5 @@ class TestAppPasswordFormat(unittest.TestCase):
# Defensive: don't crash on iterables # Defensive: don't crash on iterables
self.assertFalse(bluesky._validate_app_password_format(["wfwp", "cq7o", "5six", "7wy5"])) self.assertFalse(bluesky._validate_app_password_format(["wfwp", "cq7o", "5six", "7wy5"]))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,11 +1,8 @@
import sys
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
import briefing import briefing
import store import store
@@ -52,6 +49,5 @@ class BriefingV3Tests(unittest.TestCase):
finally: finally:
briefing.BRIEFS_DIR = old_briefs_dir briefing.BRIEFS_DIR = old_briefs_dir
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -7,11 +7,7 @@ where prompting techniques actually live.
""" """
import re import re
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import categories from lib import categories
from lib.categories import CATEGORY_PEERS, detect_category, peer_subs_for from lib.categories import CATEGORY_PEERS, detect_category, peer_subs_for
@@ -149,6 +145,5 @@ class CategoryMapInvariants(unittest.TestCase):
self.assertGreaterEqual(len(CATEGORY_PEERS), 8) self.assertGreaterEqual(len(CATEGORY_PEERS), 8)
self.assertLessEqual(len(CATEGORY_PEERS), 20) self.assertLessEqual(len(CATEGORY_PEERS), 20)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-6
View File
@@ -13,17 +13,12 @@ Fixture reference: `tests/fixtures/prompting-gpt-image-2-resolved-block.md`.
""" """
import io import io
import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import resolve from lib import resolve
OPENAI_BRAND_SUBREDDIT_RESULTS = [ OPENAI_BRAND_SUBREDDIT_RESULTS = [
{ {
"title": "r/OpenAI community hub", "title": "r/OpenAI community hub",
@@ -140,6 +135,5 @@ class PromptingGptImage2RegressionGuard(unittest.TestCase):
self.assertIsNone(result["category"]) self.assertIsNone(result["category"])
self.assertNotIn("Matched category=", buf.getvalue()) self.assertNotIn("Matched category=", buf.getvalue())
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+22 -27
View File
@@ -1,19 +1,15 @@
"""Tests for Chrome cookie extraction on macOS.""" """Tests for Chrome cookie extraction on macOS."""
import hashlib import hashlib
import os
import sqlite3 import sqlite3
import subprocess import subprocess
import sys
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
import pytest import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days")) from lib.chrome_cookies import (
from scripts.lib.chrome_cookies import (
CHROME_COOKIES_DB, CHROME_COOKIES_DB,
CHROME_IV_HEX, CHROME_IV_HEX,
CHROME_KEY_LENGTH, CHROME_KEY_LENGTH,
@@ -27,7 +23,6 @@ from scripts.lib.chrome_cookies import (
extract_chrome_cookies_macos, extract_chrome_cookies_macos,
) )
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Helpers — create real encrypted cookie values using known key + system openssl # Helpers — create real encrypted cookie values using known key + system openssl
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -112,11 +107,11 @@ def _create_chrome_cookies_db(path: str, cookies: list[tuple], db_version: int =
conn.commit() conn.commit()
conn.close() conn.close()
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# PKCS7 padding tests # PKCS7 padding tests
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestPkcs7Padding: class TestPkcs7Padding:
def test_valid_padding_1(self): def test_valid_padding_1(self):
# 1 byte of padding # 1 byte of padding
@@ -143,11 +138,11 @@ class TestPkcs7Padding:
def test_empty_data(self): def test_empty_data(self):
assert _remove_pkcs7_padding(b"") is None assert _remove_pkcs7_padding(b"") is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Key derivation test # Key derivation test
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestKeyDerivation: class TestKeyDerivation:
def test_derive_aes_key_deterministic(self): def test_derive_aes_key_deterministic(self):
key1 = _derive_aes_key(b"my_passphrase") key1 = _derive_aes_key(b"my_passphrase")
@@ -160,11 +155,11 @@ class TestKeyDerivation:
key2 = _derive_aes_key(b"passphrase_b") key2 = _derive_aes_key(b"passphrase_b")
assert key1 != key2 assert key1 != key2
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Decryption test (real openssl, known key) # Decryption test (real openssl, known key)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestDecryption: class TestDecryption:
def test_decrypt_v10_roundtrip(self): def test_decrypt_v10_roundtrip(self):
"""Encrypt then decrypt — verifies the full pipeline works.""" """Encrypt then decrypt — verifies the full pipeline works."""
@@ -197,28 +192,28 @@ class TestDecryption:
"""v10 prefix with no ciphertext should return None.""" """v10 prefix with no ciphertext should return None."""
assert _decrypt_v10_value(b"v10", KNOWN_AES_KEY, db_version=20) is None assert _decrypt_v10_value(b"v10", KNOWN_AES_KEY, db_version=20) is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Chrome not installed → returns None # Chrome not installed → returns None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestChromeNotInstalled: class TestChromeNotInstalled:
def test_db_not_found(self): def test_db_not_found(self):
with mock.patch( with mock.patch(
"scripts.lib.chrome_cookies.CHROME_COOKIES_DB", "lib.chrome_cookies.CHROME_COOKIES_DB",
Path("/nonexistent/path/Cookies"), Path("/nonexistent/path/Cookies"),
): ):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"]) result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
assert result is None assert result is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Keychain access denied → returns None # Keychain access denied → returns None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestKeychainDenied: class TestKeychainDenied:
def test_security_command_fails(self): def test_security_command_fails(self):
with mock.patch("scripts.lib.chrome_cookies.subprocess.run") as mock_run: with mock.patch("lib.chrome_cookies.subprocess.run") as mock_run:
mock_run.return_value = subprocess.CompletedProcess( mock_run.return_value = subprocess.CompletedProcess(
args=[], returncode=44, stdout="", stderr="security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain." args=[], returncode=44, stdout="", stderr="security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain."
) )
@@ -226,27 +221,27 @@ class TestKeychainDenied:
assert result is None assert result is None
def test_security_command_not_found(self): def test_security_command_not_found(self):
with mock.patch("scripts.lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError): with mock.patch("lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError):
result = _get_chrome_encryption_key() result = _get_chrome_encryption_key()
assert result is None assert result is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# openssl not found → returns None # openssl not found → returns None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestOpensslNotFound: class TestOpensslNotFound:
def test_openssl_missing(self): def test_openssl_missing(self):
encrypted = _encrypt_value_v10("test", KNOWN_AES_KEY) encrypted = _encrypt_value_v10("test", KNOWN_AES_KEY)
with mock.patch("scripts.lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError): with mock.patch("lib.chrome_cookies.subprocess.run", side_effect=FileNotFoundError):
result = _decrypt_v10_value(encrypted, KNOWN_AES_KEY, db_version=20) result = _decrypt_v10_value(encrypted, KNOWN_AES_KEY, db_version=20)
assert result is None assert result is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Unencrypted cookie values → returned as-is # Unencrypted cookie values → returned as-is
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestUnencryptedCookies: class TestUnencryptedCookies:
def test_plain_value_returned(self, tmp_path): def test_plain_value_returned(self, tmp_path):
"""Unencrypted cookies (value column populated) returned without decryption.""" """Unencrypted cookies (value column populated) returned without decryption."""
@@ -256,18 +251,18 @@ class TestUnencryptedCookies:
(".x.com", "ct0", "plain_ct0_value", b""), (".x.com", "ct0", "plain_ct0_value", b""),
]) ])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)): with mock.patch("lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
# No keychain needed for unencrypted values # No keychain needed for unencrypted values
with mock.patch("scripts.lib.chrome_cookies._get_chrome_encryption_key", return_value=None): with mock.patch("lib.chrome_cookies._get_chrome_encryption_key", return_value=None):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"]) result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
assert result == {"auth_token": "plain_token_value", "ct0": "plain_ct0_value"} assert result == {"auth_token": "plain_token_value", "ct0": "plain_ct0_value"}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Full integration: mock DB with real v10 encryption, mock Keychain # Full integration: mock DB with real v10 encryption, mock Keychain
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFullExtraction: class TestFullExtraction:
def test_encrypted_cookies_extracted(self, tmp_path): def test_encrypted_cookies_extracted(self, tmp_path):
"""End-to-end: create DB with real v10-encrypted values, extract them.""" """End-to-end: create DB with real v10-encrypted values, extract them."""
@@ -284,9 +279,9 @@ class TestFullExtraction:
(".other.com", "other", "", b""), # unrelated cookie (".other.com", "other", "", b""), # unrelated cookie
]) ])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)): with mock.patch("lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch( with mock.patch(
"scripts.lib.chrome_cookies._get_chromium_encryption_key", "lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE, return_value=KNOWN_PASSPHRASE,
): ):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"]) result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
@@ -301,8 +296,8 @@ class TestFullExtraction:
(".other.com", "session", "val", b""), (".other.com", "session", "val", b""),
]) ])
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)): with mock.patch("lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch("scripts.lib.chrome_cookies._get_chrome_encryption_key", return_value=None): with mock.patch("lib.chrome_cookies._get_chrome_encryption_key", return_value=None):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"]) result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
assert result is None assert result is None
@@ -317,9 +312,9 @@ class TestFullExtraction:
(".x.com", "auth_token", "", encrypted_auth), (".x.com", "auth_token", "", encrypted_auth),
], db_version=24) ], db_version=24)
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)): with mock.patch("lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch( with mock.patch(
"scripts.lib.chrome_cookies._get_chromium_encryption_key", "lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE, return_value=KNOWN_PASSPHRASE,
): ):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"]) result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
@@ -327,11 +322,11 @@ class TestFullExtraction:
assert result is not None assert result is not None
assert result["auth_token"] == auth_val assert result["auth_token"] == auth_val
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# DB version detection # DB version detection
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestDbVersion: class TestDbVersion:
def test_reads_version_from_meta(self, tmp_path): def test_reads_version_from_meta(self, tmp_path):
db_path = str(tmp_path / "test.db") db_path = str(tmp_path / "test.db")
-7
View File
@@ -1,16 +1,10 @@
# ruff: noqa: E402
"""CLI parsing and validation for --competitors / --competitors-list.""" """CLI parsing and validation for --competitors / --competitors-list."""
from __future__ import annotations from __future__ import annotations
import io import io
import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
import last30days as cli import last30days as cli
@@ -132,6 +126,5 @@ class CompetitorsCliTests(unittest.TestCase):
cli.resolve_competitors_args(args) cli.resolve_competitors_args(args)
self.assertEqual(cm.exception.code, 2) self.assertEqual(cm.exception.code, 2)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+2 -6
View File
@@ -1,4 +1,3 @@
# ruff: noqa: E402
import json import json
import io import io
import shutil import shutil
@@ -11,13 +10,11 @@ from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
import last30days as cli import last30days as cli
from lib import schema from lib import schema
REPO_ROOT = Path(__file__).resolve().parents[1]
class CliV3Tests(unittest.TestCase): class CliV3Tests(unittest.TestCase):
def make_report(self) -> schema.Report: def make_report(self) -> schema.Report:
@@ -305,6 +302,5 @@ class CliV3Tests(unittest.TestCase):
) )
self.assertIn("[GitHub] Canonicalized repos:", stderr.getvalue()) self.assertIn("[GitHub] Canonicalized repos:", stderr.getvalue())
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import cluster, schema from lib import cluster, schema
@@ -196,6 +192,5 @@ class TestClusterUncertainty(unittest.TestCase):
result = cluster._cluster_uncertainty(candidates) result = cluster._cluster_uncertainty(candidates)
self.assertEqual("thin-evidence", result) self.assertEqual("thin-evidence", result)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-7
View File
@@ -1,20 +1,14 @@
# ruff: noqa: E402
"""Tests for scripts/lib/fanout.run_competitor_fanout.""" """Tests for scripts/lib/fanout.run_competitor_fanout."""
from __future__ import annotations from __future__ import annotations
import io import io
import sys
import threading import threading
import time import time
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
from lib import fanout from lib import fanout
@@ -155,6 +149,5 @@ class FanoutOrchestratorTests(unittest.TestCase):
) )
self.assertEqual([label for label, _ in results], ["OpenAI"]) self.assertEqual([label for label, _ in results], ["OpenAI"])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
@@ -1,4 +1,3 @@
# ruff: noqa: E402
"""Regression tests: main-topic flags must not leak into competitor sub-runs. """Regression tests: main-topic flags must not leak into competitor sub-runs.
Based on 2026-04-22 Kanye West --competitors receipt where Drake and Based on 2026-04-22 Kanye West --competitors receipt where Drake and
@@ -10,15 +9,10 @@ via closure capture, config mutation, or any other path.
from __future__ import annotations from __future__ import annotations
import io import io
import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
def _fake_report(topic: str): def _fake_report(topic: str):
class _R: class _R:
@@ -191,6 +185,5 @@ class SubRunIsolationTests(unittest.TestCase):
by_topic["Kendrick Lamar"]["config"].get("_auto_resolve_context", ""), by_topic["Kendrick Lamar"]["config"].get("_auto_resolve_context", ""),
) )
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-8
View File
@@ -1,18 +1,12 @@
# ruff: noqa: E402
"""Tests for scripts/lib/competitors.discover_competitors.""" """Tests for scripts/lib/competitors.discover_competitors."""
from __future__ import annotations from __future__ import annotations
import io import io
import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
from lib import competitors from lib import competitors
@@ -23,7 +17,6 @@ def _serp(items: list[tuple[str, str]]) -> list[dict]:
for title, snippet in items for title, snippet in items
] ]
OPENAI_SERP = _serp( OPENAI_SERP = _serp(
[ [
("OpenAI vs Anthropic vs xAI: which is better?", "xAI and Anthropic now compete directly with OpenAI."), ("OpenAI vs Anthropic vs xAI: which is better?", "xAI and Anthropic now compete directly with OpenAI."),
@@ -147,6 +140,5 @@ class CompetitorDiscoveryTests(unittest.TestCase):
results = self._run(OPENAI_SERP, "OpenAI", count=0) results = self._run(OPENAI_SERP, "OpenAI", count=0)
self.assertEqual(results, []) self.assertEqual(results, [])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-6
View File
@@ -1,20 +1,15 @@
# ruff: noqa: E402
"""Tests for --competitors-plan JSON parsing and per-entity kwargs threading.""" """Tests for --competitors-plan JSON parsing and per-entity kwargs threading."""
from __future__ import annotations from __future__ import annotations
import io import io
import json import json
import sys
import tempfile import tempfile
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
import last30days as cli import last30days as cli
@@ -175,6 +170,5 @@ class SubrunKwargsForTests(unittest.TestCase):
kwargs = cli.subrun_kwargs_for("X", {}, resolved=resolved) kwargs = cli.subrun_kwargs_for("X", {}, resolved=resolved)
self.assertEqual(kwargs["_context"], "Resolved context") self.assertEqual(kwargs["_context"], "Resolved context")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
@@ -1,4 +1,3 @@
# ruff: noqa: E402
"""Integration tests for per-entity Step 0.55 resolution inside competitor fan-out.""" """Integration tests for per-entity Step 0.55 resolution inside competitor fan-out."""
from __future__ import annotations from __future__ import annotations
@@ -7,12 +6,8 @@ import io
import sys import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
def _fake_report(topic: str): def _fake_report(topic: str):
"""Minimal Report stand-in for runner return values.""" """Minimal Report stand-in for runner return values."""
@@ -326,6 +321,5 @@ class PerEntityResolveTests(unittest.TestCase):
return [runner(c) for c in competitors] return [runner(c) for c in competitors]
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+19 -24
View File
@@ -2,24 +2,19 @@
import configparser import configparser
import sqlite3 import sqlite3
import sys
import textwrap import textwrap
from pathlib import Path
from typing import Dict, List, Optional, Tuple from typing import Dict, List, Optional, Tuple
from unittest.mock import patch from unittest.mock import patch
import pytest import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days")) from lib.cookie_extract import (
from scripts.lib.cookie_extract import (
extract_cookies, extract_cookies,
extract_firefox_cookies, extract_firefox_cookies,
_find_default_profile, _find_default_profile,
_get_firefox_profiles_dir, _get_firefox_profiles_dir,
) )
@pytest.fixture @pytest.fixture
def mock_firefox_env(tmp_path): def mock_firefox_env(tmp_path):
"""Create a mock Firefox profiles directory with cookies.sqlite. """Create a mock Firefox profiles directory with cookies.sqlite.
@@ -102,7 +97,7 @@ class TestExtractFirefoxCookies:
profiles_dir = mock_firefox_env() profiles_dir = mock_firefox_env()
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"]) result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
@@ -142,7 +137,7 @@ class TestExtractFirefoxCookies:
) )
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"]) result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
@@ -154,10 +149,10 @@ class TestExtractFirefoxCookies:
def test_firefox_not_installed(self): def test_firefox_not_installed(self):
"""Returns None when Firefox profiles directory doesn't exist.""" """Returns None when Firefox profiles directory doesn't exist."""
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=None, return_value=None,
), patch( ), patch(
"scripts.lib.cookie_extract._is_wsl", "lib.cookie_extract._is_wsl",
return_value=False, return_value=False,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token"]) result = extract_firefox_cookies(".x.com", ["auth_token"])
@@ -171,10 +166,10 @@ class TestExtractFirefoxCookies:
) )
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
), patch( ), patch(
"scripts.lib.cookie_extract._is_wsl", "lib.cookie_extract._is_wsl",
return_value=False, return_value=False,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"]) result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
@@ -192,10 +187,10 @@ class TestExtractFirefoxCookies:
) )
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
), patch( ), patch(
"scripts.lib.cookie_extract._is_wsl", "lib.cookie_extract._is_wsl",
return_value=False, return_value=False,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"]) result = extract_firefox_cookies(".x.com", ["auth_token", "ct0"])
@@ -214,7 +209,7 @@ class TestExtractFirefoxCookies:
) )
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
): ):
result = extract_firefox_cookies(".x.com", ["auth_token"]) result = extract_firefox_cookies(".x.com", ["auth_token"])
@@ -231,17 +226,17 @@ class TestExtractCookiesAuto:
profiles_dir = mock_firefox_env() profiles_dir = mock_firefox_env()
with ( with (
patch("scripts.lib.cookie_extract.platform.system", return_value="Darwin"), patch("lib.cookie_extract.platform.system", return_value="Darwin"),
patch( patch(
"scripts.lib.cookie_extract.extract_chrome_cookies", "lib.cookie_extract.extract_chrome_cookies",
return_value=None, return_value=None,
), ),
patch( patch(
"scripts.lib.cookie_extract.extract_safari_cookies", "lib.cookie_extract.extract_safari_cookies",
return_value=None, return_value=None,
), ),
patch( patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
), ),
): ):
@@ -257,9 +252,9 @@ class TestExtractCookiesAuto:
profiles_dir = mock_firefox_env() profiles_dir = mock_firefox_env()
with ( with (
patch("scripts.lib.cookie_extract.platform.system", return_value="Linux"), patch("lib.cookie_extract.platform.system", return_value="Linux"),
patch( patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
), ),
): ):
@@ -273,7 +268,7 @@ class TestExtractCookiesAuto:
profiles_dir = mock_firefox_env() profiles_dir = mock_firefox_env()
with patch( with patch(
"scripts.lib.cookie_extract._get_firefox_profiles_dir", "lib.cookie_extract._get_firefox_profiles_dir",
return_value=profiles_dir, return_value=profiles_dir,
): ):
result = extract_cookies("firefox", ".x.com", ["auth_token"]) result = extract_cookies("firefox", ".x.com", ["auth_token"])
@@ -289,7 +284,7 @@ class TestExtractCookiesAuto:
def test_chrome_delegates_to_chrome_module(self): def test_chrome_delegates_to_chrome_module(self):
"""Chrome extraction delegates to chrome_cookies module.""" """Chrome extraction delegates to chrome_cookies module."""
with patch( with patch(
"scripts.lib.cookie_extract.extract_chrome_cookies", "lib.cookie_extract.extract_chrome_cookies",
return_value={"auth_token": "chrome_tok"}, return_value={"auth_token": "chrome_tok"},
): ):
result = extract_cookies("chrome", ".x.com", ["auth_token"]) result = extract_cookies("chrome", ".x.com", ["auth_token"])
@@ -298,7 +293,7 @@ class TestExtractCookiesAuto:
def test_safari_delegates_to_safari_module(self): def test_safari_delegates_to_safari_module(self):
"""Safari extraction delegates to safari_cookies module.""" """Safari extraction delegates to safari_cookies module."""
with patch( with patch(
"scripts.lib.cookie_extract.extract_safari_cookies", "lib.cookie_extract.extract_safari_cookies",
return_value={"auth_token": "safari_tok"}, return_value={"auth_token": "safari_tok"},
): ):
result = extract_cookies("safari", ".x.com", ["auth_token"]) result = extract_cookies("safari", ".x.com", ["auth_token"])
-4
View File
@@ -1,12 +1,9 @@
"""Tests for dates module.""" """Tests for dates module."""
import sys
import unittest import unittest
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from pathlib import Path
# Add lib to path # Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import dates from lib import dates
@@ -109,6 +106,5 @@ class TestRecencyScore(unittest.TestCase):
result = dates.recency_score(None) result = dates.recency_score(None)
self.assertEqual(result, 0) self.assertEqual(result, 0)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,9 +1,5 @@
import sys
import unittest import unittest
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import dates from lib import dates
@@ -58,6 +54,5 @@ class DatesV3Tests(unittest.TestCase):
self.assertEqual(100, dates.recency_score(future)) self.assertEqual(100, dates.recency_score(future))
self.assertEqual(0, dates.recency_score(None)) self.assertEqual(0, dates.recency_score(None))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+7 -12
View File
@@ -1,10 +1,6 @@
"""Unit tests for dedupe.py: text normalization, similarity metrics, and deduplication.""" """Unit tests for dedupe.py: text normalization, similarity metrics, and deduplication."""
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import dedupe from lib import dedupe
from lib.schema import SourceItem from lib.schema import SourceItem
@@ -16,11 +12,11 @@ def _item(title: str, body: str = "", source: str = "reddit", item_id: str = "t1
url="https://example.com", engagement={}, metadata={}, url="https://example.com", engagement={}, metadata={},
) )
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# normalize_text # normalize_text
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestNormalizeText(unittest.TestCase): class TestNormalizeText(unittest.TestCase):
def test_lowercases(self): def test_lowercases(self):
@@ -35,11 +31,11 @@ class TestNormalizeText(unittest.TestCase):
def test_empty_string(self): def test_empty_string(self):
self.assertEqual(dedupe.normalize_text(""), "") self.assertEqual(dedupe.normalize_text(""), "")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# get_ngrams # get_ngrams
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestGetNgrams(unittest.TestCase): class TestGetNgrams(unittest.TestCase):
def test_simple_trigrams(self): def test_simple_trigrams(self):
@@ -58,11 +54,11 @@ class TestGetNgrams(unittest.TestCase):
ngrams = dedupe.get_ngrams("A!B") ngrams = dedupe.get_ngrams("A!B")
self.assertEqual(ngrams, {"a b"}) self.assertEqual(ngrams, {"a b"})
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# jaccard_similarity # jaccard_similarity
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestJaccardSimilarity(unittest.TestCase): class TestJaccardSimilarity(unittest.TestCase):
def test_identical_sets(self): def test_identical_sets(self):
@@ -81,11 +77,11 @@ class TestJaccardSimilarity(unittest.TestCase):
def test_both_empty(self): def test_both_empty(self):
self.assertAlmostEqual(dedupe.jaccard_similarity(set(), set()), 0.0) self.assertAlmostEqual(dedupe.jaccard_similarity(set(), set()), 0.0)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# token_jaccard # token_jaccard
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestTokenJaccard(unittest.TestCase): class TestTokenJaccard(unittest.TestCase):
def test_identical_texts(self): def test_identical_texts(self):
@@ -105,11 +101,11 @@ class TestTokenJaccard(unittest.TestCase):
# "am" is len 2, "great"/"terrible" are content # "am" is len 2, "great"/"terrible" are content
self.assertGreater(result, 0.0) self.assertGreater(result, 0.0)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# hybrid_similarity # hybrid_similarity
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestHybridSimilarity(unittest.TestCase): class TestHybridSimilarity(unittest.TestCase):
def test_identical_texts(self): def test_identical_texts(self):
@@ -131,11 +127,11 @@ class TestHybridSimilarity(unittest.TestCase):
max(ngram_sim, token_sim), max(ngram_sim, token_sim),
) )
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# item_text # item_text
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestItemText(unittest.TestCase): class TestItemText(unittest.TestCase):
def test_combines_fields(self): def test_combines_fields(self):
@@ -159,11 +155,11 @@ class TestItemText(unittest.TestCase):
self.assertIn("john", text) self.assertIn("john", text)
self.assertIn("r/python", text) self.assertIn("r/python", text)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# dedupe_items # dedupe_items
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestDedupeItems(unittest.TestCase): class TestDedupeItems(unittest.TestCase):
def test_keeps_unique_items(self): def test_keeps_unique_items(self):
@@ -212,6 +208,5 @@ class TestDedupeItems(unittest.TestCase):
result_loose = dedupe.dedupe_items(items, threshold=0.3) result_loose = dedupe.dedupe_items(items, threshold=0.3)
self.assertEqual(len(result_loose), 1) self.assertEqual(len(result_loose), 1)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+11 -16
View File
@@ -5,21 +5,17 @@ from __future__ import annotations
import json import json
import os import os
import shutil import shutil
import sys
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from pathlib import Path
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
import pytest import pytest
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts")) from lib import digg
from lib import subproc
from lib import digg # noqa: E402
from lib import subproc # noqa: E402
# === Helpers === # === Helpers ===
def _cluster( def _cluster(
cluster_url_id: str = "abc123xy", cluster_url_id: str = "abc123xy",
title: str = "Sample cluster", title: str = "Sample cluster",
@@ -66,9 +62,9 @@ def _post(
def _stdout_for(payload: dict) -> subproc.SubprocResult: def _stdout_for(payload: dict) -> subproc.SubprocResult:
return subproc.SubprocResult(returncode=0, stdout=json.dumps(payload), stderr="") return subproc.SubprocResult(returncode=0, stdout=json.dumps(payload), stderr="")
# === _parse_first_post_age === # === _parse_first_post_age ===
def test_parse_first_post_age_days(): def test_parse_first_post_age_days():
today = datetime(2026, 5, 9, tzinfo=timezone.utc) today = datetime(2026, 5, 9, tzinfo=timezone.utc)
assert digg._parse_first_post_age("5d", today=today) == "2026-05-04" assert digg._parse_first_post_age("5d", today=today) == "2026-05-04"
@@ -104,9 +100,9 @@ def test_parse_first_post_age_invalid():
assert digg._parse_first_post_age("d") is None assert digg._parse_first_post_age("d") is None
assert digg._parse_first_post_age("-3d") is None assert digg._parse_first_post_age("-3d") is None
# === parse_digg_response === # === parse_digg_response ===
def test_parse_response_happy_path(): def test_parse_response_happy_path():
response = { response = {
"results": [ "results": [
@@ -193,9 +189,9 @@ def test_parse_response_engagement_rank_score():
assert by_id["top"]["engagement"]["rank_score"] == 50.0 assert by_id["top"]["engagement"]["rank_score"] == 50.0
assert by_id["off-leaderboard"]["engagement"]["rank_score"] == 0.0 assert by_id["off-leaderboard"]["engagement"]["rank_score"] == 0.0
# === _parse_post === # === _parse_post ===
def test_parse_post_happy(): def test_parse_post_happy():
out = digg._parse_post(_post(username="adam", body="Hello world")) out = digg._parse_post(_post(username="adam", body="Hello world"))
assert out is not None assert out is not None
@@ -210,9 +206,9 @@ def test_parse_post_drops_missing_body_or_handle_or_url():
assert digg._parse_post({"author": {"username": "x"}, "body": "txt", "xUrl": ""}) is None assert digg._parse_post({"author": {"username": "x"}, "body": "txt", "xUrl": ""}) is None
assert digg._parse_post(None) is None # type: ignore[arg-type] assert digg._parse_post(None) is None # type: ignore[arg-type]
# === _run_cli / search_digg with stubbed subprocess === # === _run_cli / search_digg with stubbed subprocess ===
def test_search_digg_binary_missing_returns_empty(monkeypatch): def test_search_digg_binary_missing_returns_empty(monkeypatch):
monkeypatch.setattr(digg.shutil, "which", lambda _: None) monkeypatch.setattr(digg.shutil, "which", lambda _: None)
out = digg.search_digg("anything", "2026-04-09", "2026-05-09") out = digg.search_digg("anything", "2026-04-09", "2026-05-09")
@@ -280,9 +276,9 @@ def test_search_digg_empty_query_short_circuits(monkeypatch):
assert out["results"] == [] assert out["results"] == []
called.assert_not_called() called.assert_not_called()
# === enrich_with_top_posts === # === enrich_with_top_posts ===
def test_enrich_with_top_posts_attaches_posts(monkeypatch): def test_enrich_with_top_posts_attaches_posts(monkeypatch):
monkeypatch.setattr(digg.shutil, "which", lambda _: "/fake/path") monkeypatch.setattr(digg.shutil, "which", lambda _: "/fake/path")
@@ -357,9 +353,9 @@ def test_enrich_top_k_zero_skips_all(monkeypatch):
digg.enrich_with_top_posts(items, top_k=0) digg.enrich_with_top_posts(items, top_k=0)
fake.assert_not_called() fake.assert_not_called()
# === enrich_source_items (post-dedupe path) === # === enrich_source_items (post-dedupe path) ===
class _FakeSourceItem: class _FakeSourceItem:
def __init__(self, source, item_id, engagement, metadata): def __init__(self, source, item_id, engagement, metadata):
self.source = source self.source = source
@@ -410,14 +406,14 @@ def test_enrich_source_items_falls_back_to_item_id(monkeypatch):
digg.enrich_source_items(items, top_k=1) digg.enrich_source_items(items, top_k=1)
assert captured["cluster_id"] == "fallbackid" assert captured["cluster_id"] == "fallbackid"
# === Live tests (opt-in) === # === Live tests (opt-in) ===
LIVE = os.environ.get("LAST30DAYS_DIGG_LIVE", "").lower() in ("1", "true", "yes") LIVE = os.environ.get("LAST30DAYS_DIGG_LIVE", "").lower() in ("1", "true", "yes")
HAVE_BIN = shutil.which(digg.CLI_BIN) is not None HAVE_BIN = shutil.which(digg.CLI_BIN) is not None
@pytest.mark.skipif(not (LIVE and HAVE_BIN), reason="LAST30DAYS_DIGG_LIVE not set or digg-pp-cli missing") @pytest.mark.skipif(not (LIVE and HAVE_BIN), reason="LAST30DAYS_DIGG_LIVE not set or digg-pp-cli missing")
class TestLiveDigg: class TestLiveDigg:
def test_search_returns_clusters(self): def test_search_returns_clusters(self):
out = digg.search_digg("claude code", "2026-04-09", "2026-05-09", depth="quick") out = digg.search_digg("claude code", "2026-04-09", "2026-05-09", depth="quick")
@@ -451,6 +447,5 @@ class TestLiveDigg:
posts = digg.fetch_top_posts("notarealclusterid", posts_per=2) posts = digg.fetch_top_posts("notarealclusterid", posts_per=2)
assert posts == [] assert posts == []
if __name__ == "__main__": if __name__ == "__main__":
pytest.main([__file__, "-v"]) pytest.main([__file__, "-v"])
-4
View File
@@ -1,11 +1,8 @@
"""Tests for entity_extract module.""" """Tests for entity_extract module."""
import sys
import unittest import unittest
from pathlib import Path
# Add lib to path # Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import entity_extract from lib import entity_extract
@@ -162,6 +159,5 @@ class TestExtractEntities(unittest.TestCase):
result = entity_extract.extract_entities([], []) result = entity_extract.extract_entities([], [])
self.assertSetEqual(set(result.keys()), {"x_handles", "x_hashtags", "reddit_subreddits"}) self.assertSetEqual(set(result.keys()), {"x_handles", "x_hashtags", "reddit_subreddits"})
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import entity_extract from lib import entity_extract
@@ -53,6 +49,5 @@ class TestExtractSubreddits(unittest.TestCase):
def test_empty_items(self): def test_empty_items(self):
self.assertEqual([], entity_extract._extract_subreddits([])) self.assertEqual([], entity_extract._extract_subreddits([]))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,14 +1,10 @@
"""Tests for browser cookie extraction integration in env.py.""" """Tests for browser cookie extraction integration in env.py."""
import os import os
import sys
from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
import pytest import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib.env import extract_browser_credentials, COOKIE_DOMAINS from lib.env import extract_browser_credentials, COOKIE_DOMAINS
+1 -6
View File
@@ -1,9 +1,4 @@
import sys from lib import env
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days"))
from scripts.lib import env
def test_include_sources_defaults_to_empty_string(monkeypatch, tmp_path): def test_include_sources_defaults_to_empty_string(monkeypatch, tmp_path):
+1 -8
View File
@@ -12,19 +12,15 @@ from __future__ import annotations
import re import re
import subprocess import subprocess
import sys
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
import pytest import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts")) from lib import env
from lib import env # noqa: E402
SETUP_KEYCHAIN_SH = Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts" / "setup-keychain.sh" SETUP_KEYCHAIN_SH = Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts" / "setup-keychain.sh"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# _load_keychain unit tests # _load_keychain unit tests
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -93,12 +89,10 @@ def test_load_keychain_skips_empty_stdout():
mock.patch("subprocess.run", return_value=_run_result(0, "")): mock.patch("subprocess.run", return_value=_run_result(0, "")):
assert env._load_keychain(["XAI_API_KEY"]) == {} assert env._load_keychain(["XAI_API_KEY"]) == {}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# get_config integration tests # get_config integration tests
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@pytest.fixture @pytest.fixture
def clean_env(monkeypatch, tmp_path): def clean_env(monkeypatch, tmp_path):
"""Hide every key get_config might touch and point CONFIG_FILE at a """Hide every key get_config might touch and point CONFIG_FILE at a
@@ -154,7 +148,6 @@ def test_get_config_openai_key_can_come_from_keychain(clean_env):
assert cfg["OPENAI_API_KEY"] == "sk-from-kc" assert cfg["OPENAI_API_KEY"] == "sk-from-kc"
assert cfg["OPENAI_AUTH_SOURCE"] == "api_key" assert cfg["OPENAI_AUTH_SOURCE"] == "api_key"
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Drift guard: lib/env.py KEYCHAIN_KEYS and setup-keychain.sh ALL_KEYS must # Drift guard: lib/env.py KEYCHAIN_KEYS and setup-keychain.sh ALL_KEYS must
# stay in lockstep. A mismatch means users storing a key via the helper script # stay in lockstep. A mismatch means users storing a key via the helper script
-4
View File
@@ -1,11 +1,8 @@
import os import os
import sys
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import bird_x, env from lib import bird_x, env
@@ -69,6 +66,5 @@ class ThreadsAvailabilityTests(unittest.TestCase):
"INCLUDE_SOURCES": "", "INCLUDE_SOURCES": "",
})) }))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,13 +1,10 @@
import json import json
import os import os
import sys
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest import mock from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
import evaluate_search_quality as evaluator import evaluate_search_quality as evaluator
@@ -202,6 +199,5 @@ class EvaluatorV3Tests(unittest.TestCase):
self.assertIn("| topic a | 0.10 | 0.30 |", summary) self.assertIn("| topic a | 0.10 | 0.30 |", summary)
self.assertEqual("HEAD~1", metrics["baseline"]) self.assertEqual("HEAD~1", metrics["baseline"])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-3
View File
@@ -1,4 +1,3 @@
# ruff: noqa: E402
"""Tests for the BRAVE/SERPER web-promo suppression when hosting-model-driven.""" """Tests for the BRAVE/SERPER web-promo suppression when hosting-model-driven."""
from __future__ import annotations from __future__ import annotations
@@ -11,7 +10,6 @@ import unittest
from pathlib import Path from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1] REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
def _engine() -> Path: def _engine() -> Path:
@@ -90,6 +88,5 @@ class FooterNudgeSuppressionTests(unittest.TestCase):
msg="web promo should be suppressed when --plan is passed", msg="web promo should be suppressed when --plan is passed",
) )
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-7
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import fusion, schema from lib import fusion, schema
@@ -52,7 +48,6 @@ class FusionV3Tests(unittest.TestCase):
self.assertEqual({"reddit", "x"}, set(merged.sources)) self.assertEqual({"reddit", "x"}, set(merged.sources))
self.assertEqual(2, len(merged.source_items)) self.assertEqual(2, len(merged.source_items))
def test_diversify_pool_guarantees_min_per_qualifying_source(self): def test_diversify_pool_guarantees_min_per_qualifying_source(self):
"""Every qualifying source (local_relevance >= 0.25) gets at least 2 """Every qualifying source (local_relevance >= 0.25) gets at least 2
items in the fused pool. items in the fused pool.
@@ -118,7 +113,6 @@ class FusionV3Tests(unittest.TestCase):
f"Source '{src}' has {source_counts.get(src, 0)} items, expected >= 2", f"Source '{src}' has {source_counts.get(src, 0)} items, expected >= 2",
) )
def test_diversify_pool_denies_slots_for_low_relevance_source(self): def test_diversify_pool_denies_slots_for_low_relevance_source(self):
"""Sources with best local_relevance < 0.25 do not get reserved slots. """Sources with best local_relevance < 0.25 do not get reserved slots.
@@ -439,6 +433,5 @@ class TestUrlNormalization(unittest.TestCase):
_normalize_url("https://reddit.com/r/test"), _normalize_url("https://reddit.com/r/test"),
) )
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+134 -17
View File
@@ -1,12 +1,9 @@
"""Tests for GitHub source module.""" """Tests for GitHub source module."""
import json import json
import sys
import unittest import unittest
from pathlib import Path
from unittest.mock import patch, MagicMock from unittest.mock import patch, MagicMock
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import github from lib import github
@@ -76,13 +73,29 @@ class TestParseDate(unittest.TestCase):
class TestSearchGithub(unittest.TestCase): class TestSearchGithub(unittest.TestCase):
@patch.dict("os.environ", {}, clear=True) @patch.dict("os.environ", {}, clear=True)
@patch("subprocess.run", side_effect=FileNotFoundError) @patch("subprocess.run", side_effect=FileNotFoundError)
def test_no_token_returns_empty(self, mock_run): def test_no_token_returns_empty_envelope(self, mock_run):
result = github.search_github("react", "2026-03-01", "2026-03-31", token=None) result = github.search_github("react", "2026-03-01", "2026-03-31", token=None)
self.assertEqual(result, []) self.assertEqual(result.get("items", []), [])
self.assertIn("error", result)
# Envelope shape must match the fetch-failure path: context is
# always present so callers (and parse_github_response) can read
# diagnostic fields without branching on which failure mode hit.
self.assertIn("context", result)
self.assertEqual(result["context"]["from_date"], "2026-03-01")
self.assertEqual(result["context"]["to_date"], "2026-03-31")
def test_resolve_token_public_alias(self):
"""resolve_token is the public entry point pipeline uses; _resolve_token stays
private. Both should return the same value for the same input."""
self.assertEqual(
github.resolve_token("explicit-token"),
github._resolve_token("explicit-token"),
)
self.assertEqual(github.resolve_token("explicit-token"), "explicit-token")
@patch.object(github, "_fetch_json") @patch.object(github, "_fetch_json")
@patch.object(github, "_resolve_token", return_value="test-token") @patch.object(github, "_resolve_token", return_value="test-token")
def test_search_returns_items(self, mock_token, mock_fetch): def test_search_returns_raw_envelope(self, mock_token, mock_fetch):
mock_fetch.return_value = { mock_fetch.return_value = {
"total_count": 1, "total_count": 1,
"items": [ "items": [
@@ -99,9 +112,15 @@ class TestSearchGithub(unittest.TestCase):
}, },
], ],
} }
result = github.search_github("react", "2026-03-01", "2026-03-31") # Search returns raw envelope; parse normalizes.
self.assertEqual(len(result), 1) response = github.search_github("react", "2026-03-01", "2026-03-31")
item = result[0] self.assertEqual(len(response["items"]), 1)
self.assertEqual(response["items"][0]["title"], "React Server Components bug")
self.assertEqual(response["context"]["from_date"], "2026-03-01")
items = github.parse_github_response(response)
self.assertEqual(len(items), 1)
item = items[0]
self.assertEqual(item["source"], "github") self.assertEqual(item["source"], "github")
self.assertEqual(item["container"], "facebook/react") self.assertEqual(item["container"], "facebook/react")
self.assertEqual(item["title"], "React Server Components bug") self.assertEqual(item["title"], "React Server Components bug")
@@ -117,10 +136,11 @@ class TestSearchGithub(unittest.TestCase):
@patch.object(github, "_fetch_json", return_value=None) @patch.object(github, "_fetch_json", return_value=None)
@patch.object(github, "_resolve_token", return_value="test-token") @patch.object(github, "_resolve_token", return_value="test-token")
def test_rate_limit_returns_empty(self, mock_token, mock_fetch): def test_rate_limit_returns_empty_envelope(self, mock_token, mock_fetch):
"""403 rate limit returns empty list gracefully.""" """403 rate limit returns envelope with empty items list."""
result = github.search_github("react", "2026-03-01", "2026-03-31") response = github.search_github("react", "2026-03-01", "2026-03-31")
self.assertEqual(result, []) self.assertEqual(response["items"], [])
self.assertEqual(github.parse_github_response(response), [])
@patch.object(github, "_fetch_json") @patch.object(github, "_fetch_json")
@patch.object(github, "_resolve_token", return_value="test-token") @patch.object(github, "_resolve_token", return_value="test-token")
@@ -142,9 +162,107 @@ class TestSearchGithub(unittest.TestCase):
}, },
], ],
} }
result = github.search_github("next.js", "2026-03-01", "2026-03-31") response = github.search_github("next.js", "2026-03-01", "2026-03-31")
self.assertEqual(len(result), 1) items = github.parse_github_response(response)
self.assertTrue(result[0]["metadata"]["is_pr"]) self.assertEqual(len(items), 1)
self.assertTrue(items[0]["metadata"]["is_pr"])
class TestParseGithubResponse(unittest.TestCase):
"""Fixture-driven parse tests: feed a synthetic search_github envelope to
parse_github_response and assert normalized output.
This contract (search returns dict envelope, parse turns it into a list)
matches every other source adapter. Before this refactor, search_github
returned a bare list and there was no parse step, blocking fixture tests.
"""
_RAW_ENVELOPE = {
"items": [
{
"html_url": "https://github.com/facebook/react/issues/42",
"title": "React Server Components bug",
"body": "There is a bug when using RSC with streaming...",
"created_at": "2026-03-15T10:00:00Z",
"state": "open",
"comments": 12,
"reactions": {"total_count": 8},
"labels": [{"name": "bug"}, {"name": "rsc"}],
"user": {"login": "testuser"},
},
{
"html_url": "https://github.com/vercel/next.js/pull/99",
"title": "Add streaming support",
"body": "This PR adds...",
"created_at": "2026-03-20T10:00:00Z",
"state": "open",
"comments": 5,
"reactions": {"total_count": 3},
"labels": [],
"user": {"login": "dev"},
"pull_request": {"url": "..."},
},
],
"context": {
"core": "react",
"from_date": "2026-03-01",
"to_date": "2026-03-31",
"count": 25,
},
}
def test_normalizes_items(self):
items = github.parse_github_response(self._RAW_ENVELOPE)
self.assertEqual(len(items), 2)
by_url = {i["url"]: i for i in items}
issue = by_url["https://github.com/facebook/react/issues/42"]
self.assertEqual(issue["source"], "github")
self.assertEqual(issue["container"], "facebook/react")
self.assertEqual(issue["title"], "React Server Components bug")
self.assertEqual(issue["date"], "2026-03-15")
self.assertEqual(issue["author"], "testuser")
self.assertEqual(issue["engagement"]["reactions"], 8)
self.assertEqual(issue["engagement"]["comments"], 12)
self.assertFalse(issue["metadata"]["is_pr"])
def test_detects_pr(self):
items = github.parse_github_response(self._RAW_ENVELOPE)
pr = next(i for i in items if "/pull/" in i["url"])
self.assertTrue(pr["metadata"]["is_pr"])
def test_date_filter_drops_outside_window(self):
envelope = {
"items": [
{
"html_url": "https://github.com/foo/bar/issues/1",
"title": "Too old",
"created_at": "2026-01-15T10:00:00Z",
"comments": 0, "reactions": {"total_count": 0},
"labels": [], "user": {"login": "x"},
},
{
"html_url": "https://github.com/foo/bar/issues/2",
"title": "In window",
"created_at": "2026-03-15T10:00:00Z",
"comments": 0, "reactions": {"total_count": 0},
"labels": [], "user": {"login": "x"},
},
],
"context": {"core": "foo", "from_date": "2026-03-01",
"to_date": "2026-03-31", "count": 25},
}
items = github.parse_github_response(envelope)
self.assertEqual(len(items), 1)
self.assertEqual(items[0]["title"], "In window")
def test_sorts_by_relevance(self):
items = github.parse_github_response(self._RAW_ENVELOPE)
scores = [i.get("relevance", 0) for i in items]
self.assertEqual(scores, sorted(scores, reverse=True))
def test_empty_envelope(self):
self.assertEqual(github.parse_github_response({"items": []}), [])
self.assertEqual(github.parse_github_response({}), [])
class TestComputeRelevance(unittest.TestCase): class TestComputeRelevance(unittest.TestCase):
@@ -158,6 +276,5 @@ class TestComputeRelevance(unittest.TestCase):
low = github._compute_relevance("react", "React", 20, 0, 0) low = github._compute_relevance("react", "React", 20, 0, 0)
self.assertGreater(high, low) self.assertGreater(high, low)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,10 +1,6 @@
import sys
import unittest import unittest
from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import grounding from lib import grounding
@@ -338,6 +334,5 @@ class RedditEnrichItemsTests(unittest.TestCase):
msg=f"Expected a rate-limit stderr message, got: {captured_stderr!r}", msg=f"Expected a rate-limit stderr message, got: {captured_stderr!r}",
) )
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+17 -17
View File
@@ -1,20 +1,16 @@
"""Tests for hackernews.py - HN search via Algolia API.""" """Tests for hackernews.py - HN search via Algolia API."""
import json import json
import sys
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import Mock, patch from unittest.mock import Mock, patch
import pytest import pytest
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import hackernews from lib import hackernews
# === Helper Functions === # === Helper Functions ===
def create_mock_hit( def create_mock_hit(
object_id="12345", object_id="12345",
title="Test HN Story", title="Test HN Story",
@@ -40,9 +36,9 @@ def create_mock_hit(
"url": url, "url": url,
} }
# === Tests for _date_to_unix() === # === Tests for _date_to_unix() ===
def test_date_to_unix_basic(): def test_date_to_unix_basic():
"""Test converting YYYY-MM-DD to Unix timestamp.""" """Test converting YYYY-MM-DD to Unix timestamp."""
result = hackernews._date_to_unix("2026-01-01") result = hackernews._date_to_unix("2026-01-01")
@@ -59,9 +55,9 @@ def test_date_to_unix_leap_day():
expected = datetime(2024, 2, 29, tzinfo=timezone.utc).timestamp() expected = datetime(2024, 2, 29, tzinfo=timezone.utc).timestamp()
assert result == int(expected) assert result == int(expected)
# === Tests for _unix_to_date() === # === Tests for _unix_to_date() ===
def test_unix_to_date_basic(): def test_unix_to_date_basic():
"""Test converting Unix timestamp to YYYY-MM-DD.""" """Test converting Unix timestamp to YYYY-MM-DD."""
ts = int(datetime(2026, 1, 15, tzinfo=timezone.utc).timestamp()) ts = int(datetime(2026, 1, 15, tzinfo=timezone.utc).timestamp())
@@ -77,9 +73,9 @@ def test_unix_to_date_with_time():
assert result == "2026-01-15" assert result == "2026-01-15"
# === Tests for _strip_html() === # === Tests for _strip_html() ===
def test_strip_html_basic(): def test_strip_html_basic():
"""Test HTML stripping and entity decoding.""" """Test HTML stripping and entity decoding."""
html_text = "<p>Hello &amp; goodbye</p>" html_text = "<p>Hello &amp; goodbye</p>"
@@ -113,9 +109,9 @@ def test_strip_html_entities():
# Entities are decoded # Entities are decoded
assert "&" in result or "test" in result assert "&" in result or "test" in result
# === Tests for _title_matches_query() === # === Tests for _title_matches_query() ===
def test_title_matches_query_basic(): def test_title_matches_query_basic():
"""Test basic query matching.""" """Test basic query matching."""
title = "New AI framework for developers" title = "New AI framework for developers"
@@ -199,10 +195,11 @@ def test_title_matches_query_flattens_hyphens_and_commas():
# query 'rust, go, zig' flattens; title contains 'go' # query 'rust, go, zig' flattens; title contains 'go'
assert hackernews._title_matches_query("Go 1.24 generics update", "rust, go, zig") is True assert hackernews._title_matches_query("Go 1.24 generics update", "rust, go, zig") is True
# === Tests for search_hackernews() === # === Tests for search_hackernews() ===
@patch('lib.hackernews.http.request') @patch('lib.hackernews.http.request')
def test_search_hackernews_basic(mock_request): def test_search_hackernews_basic(mock_request):
"""Test basic HN search.""" """Test basic HN search."""
mock_request.return_value = { mock_request.return_value = {
@@ -221,8 +218,9 @@ def test_search_hackernews_basic(mock_request):
assert len(result["hits"]) == 1 assert len(result["hits"]) == 1
assert mock_request.called assert mock_request.called
@patch('lib.hackernews.http.request') @patch('lib.hackernews.http.request')
def test_search_hackernews_depth_config(mock_request): def test_search_hackernews_depth_config(mock_request):
"""Test that depth parameter controls hit count.""" """Test that depth parameter controls hit count."""
mock_request.return_value = {"hits": [], "nbHits": 0} mock_request.return_value = {"hits": [], "nbHits": 0}
@@ -235,8 +233,9 @@ def test_search_hackernews_depth_config(mock_request):
assert "hitsPerPage=15" in url assert "hitsPerPage=15" in url
@patch('lib.hackernews.http.request') @patch('lib.hackernews.http.request')
def test_search_hackernews_date_filtering(mock_request): def test_search_hackernews_date_filtering(mock_request):
"""Test that date range is applied correctly.""" """Test that date range is applied correctly."""
mock_request.return_value = {"hits": [], "nbHits": 0} mock_request.return_value = {"hits": [], "nbHits": 0}
@@ -250,8 +249,9 @@ def test_search_hackernews_date_filtering(mock_request):
assert "numericFilters" in url assert "numericFilters" in url
assert "created_at_i" in url assert "created_at_i" in url
@patch('lib.hackernews.http.request') @patch('lib.hackernews.http.request')
def test_search_hackernews_http_error_handling(mock_request): def test_search_hackernews_http_error_handling(mock_request):
"""Test graceful handling of HTTP errors.""" """Test graceful handling of HTTP errors."""
from lib.http import HTTPError from lib.http import HTTPError
@@ -263,8 +263,9 @@ def test_search_hackernews_http_error_handling(mock_request):
assert result["hits"] == [] assert result["hits"] == []
assert "error" in result assert "error" in result
@patch('lib.hackernews.http.request') @patch('lib.hackernews.http.request')
def test_search_hackernews_engagement_filter(mock_request): def test_search_hackernews_engagement_filter(mock_request):
"""Test that low-engagement stories are filtered.""" """Test that low-engagement stories are filtered."""
mock_request.return_value = {"hits": [], "nbHits": 0} mock_request.return_value = {"hits": [], "nbHits": 0}
@@ -277,9 +278,9 @@ def test_search_hackernews_engagement_filter(mock_request):
# Should filter for points > 2 (URL-encoded) # Should filter for points > 2 (URL-encoded)
assert "points" in url and "%3E2" in url assert "points" in url and "%3E2" in url
# === Tests for parse_hackernews_response() === # === Tests for parse_hackernews_response() ===
def test_parse_hackernews_response_basic(): def test_parse_hackernews_response_basic():
"""Test parsing basic Algolia response.""" """Test parsing basic Algolia response."""
response = { response = {
@@ -402,9 +403,9 @@ def test_parse_hackernews_response_empty_response():
assert items == [] assert items == []
# === Tests for engagement scoring === # === Tests for engagement scoring ===
def test_engagement_score_calculation(): def test_engagement_score_calculation():
"""Test that engagement dict contains points and comments.""" """Test that engagement dict contains points and comments."""
response = { response = {
@@ -435,6 +436,5 @@ def test_engagement_score_zero_values():
assert engagement["points"] == 0 assert engagement["points"] == 0
assert engagement["comments"] == 0 assert engagement["comments"] == 0
if __name__ == "__main__": if __name__ == "__main__":
pytest.main([__file__, "-v"]) pytest.main([__file__, "-v"])
-6
View File
@@ -1,17 +1,12 @@
# ruff: noqa: E402
"""Tests for the HTML emit renderer.""" """Tests for the HTML emit renderer."""
from __future__ import annotations from __future__ import annotations
import sys
import tempfile import tempfile
import unittest import unittest
from html.parser import HTMLParser from html.parser import HTMLParser
from pathlib import Path from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
import last30days as cli import last30days as cli
from lib import html_render, schema from lib import html_render, schema
@@ -297,6 +292,5 @@ class HtmlCliIntegrationTests(unittest.TestCase):
self.assertIn("comparing 2: OpenClaw, Hermes", saved) self.assertIn("comparing 2: OpenClaw, Hermes", saved)
self.assertNotIn("last30days · OpenClaw</title>", saved) self.assertNotIn("last30days · OpenClaw</title>", saved)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,11 +1,7 @@
import sys
import urllib.error import urllib.error
import unittest import unittest
from pathlib import Path
from unittest.mock import patch, MagicMock from unittest.mock import patch, MagicMock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import http from lib import http
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib.instagram import _parse_items from lib.instagram import _parse_items
@@ -63,6 +59,5 @@ class TestExpandInstagramQueries(unittest.TestCase):
queries = expand_instagram_queries("Kanye West", "quick") queries = expand_instagram_queries("Kanye West", "quick")
self.assertEqual(len(queries), 1) self.assertEqual(len(queries), 1)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-4
View File
@@ -1,13 +1,10 @@
"""Tests for instagram.py — ScrapeCreators Instagram search module.""" """Tests for instagram.py — ScrapeCreators Instagram search module."""
import os import os
import sys
import unittest import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
# Add lib to path # Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import instagram from lib import instagram
from lib.relevance import tokenize as _tokenize from lib.relevance import tokenize as _tokenize
@@ -253,6 +250,5 @@ class TestTranscriptTimeoutConfig(unittest.TestCase):
kwargs = mock_http_get.call_args.kwargs kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 30.0) self.assertEqual(kwargs["timeout"], 30.0)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+16 -22
View File
@@ -5,11 +5,7 @@ tests exercise transitively but don't assert on directly. A regression in
any of these functions would silently degrade output quality. any of these functions would silently degrade output quality.
""" """
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import planner, rerank, render, signals, schema from lib import planner, rerank, render, signals, schema
@@ -34,11 +30,11 @@ def _candidate(source: str = "reddit", **kwargs) -> schema.Candidate:
defaults.update(kwargs) defaults.update(kwargs)
return schema.Candidate(**defaults) return schema.Candidate(**defaults)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# rerank._fallback_tuple # rerank._fallback_tuple
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFallbackTuple(unittest.TestCase): class TestFallbackTuple(unittest.TestCase):
def test_returns_score_and_explanation(self): def test_returns_score_and_explanation(self):
@@ -58,11 +54,11 @@ class TestFallbackTuple(unittest.TestCase):
low = _candidate(local_relevance=0.1, freshness=50, source_quality=0.7) low = _candidate(local_relevance=0.1, freshness=50, source_quality=0.7)
self.assertGreater(rerank._fallback_tuple(high)[0], rerank._fallback_tuple(low)[0]) self.assertGreater(rerank._fallback_tuple(high)[0], rerank._fallback_tuple(low)[0])
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# rerank._normalized_rrf # rerank._normalized_rrf
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestNormalizedRrf(unittest.TestCase): class TestNormalizedRrf(unittest.TestCase):
def test_zero_input(self): def test_zero_input(self):
@@ -77,11 +73,11 @@ class TestNormalizedRrf(unittest.TestCase):
result = rerank._normalized_rrf(1.0) result = rerank._normalized_rrf(1.0)
self.assertLessEqual(result, 100.0) self.assertLessEqual(result, 100.0)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._assess_data_freshness # render._assess_data_freshness
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestAssessDataFreshness(unittest.TestCase): class TestAssessDataFreshness(unittest.TestCase):
def _report(self, items_by_source: dict) -> schema.Report: def _report(self, items_by_source: dict) -> schema.Report:
@@ -120,11 +116,11 @@ class TestAssessDataFreshness(unittest.TestCase):
result = render._assess_data_freshness(report) result = render._assess_data_freshness(report)
self.assertIsNone(result) self.assertIsNone(result)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._format_date # render._format_date
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFormatDate(unittest.TestCase): class TestFormatDate(unittest.TestCase):
def test_high_confidence_clean(self): def test_high_confidence_clean(self):
@@ -138,11 +134,11 @@ class TestFormatDate(unittest.TestCase):
def test_none_item(self): def test_none_item(self):
self.assertIn("unknown", render._format_date(None).lower()) self.assertIn("unknown", render._format_date(None).lower())
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._format_actor # render._format_actor
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFormatActor(unittest.TestCase): class TestFormatActor(unittest.TestCase):
def test_reddit_subreddit(self): def test_reddit_subreddit(self):
@@ -157,11 +153,11 @@ class TestFormatActor(unittest.TestCase):
item = _item(source="youtube", author="Fireship") item = _item(source="youtube", author="Fireship")
self.assertEqual(render._format_actor(item), "Fireship") self.assertEqual(render._format_actor(item), "Fireship")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._format_engagement # render._format_engagement
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFormatEngagement(unittest.TestCase): class TestFormatEngagement(unittest.TestCase):
def test_reddit_format(self): def test_reddit_format(self):
@@ -174,11 +170,11 @@ class TestFormatEngagement(unittest.TestCase):
item = _item(engagement={}) item = _item(engagement={})
self.assertIsNone(render._format_engagement(item)) self.assertIsNone(render._format_engagement(item))
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._format_corroboration # render._format_corroboration
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFormatCorroboration(unittest.TestCase): class TestFormatCorroboration(unittest.TestCase):
def test_multi_source(self): def test_multi_source(self):
@@ -191,11 +187,11 @@ class TestFormatCorroboration(unittest.TestCase):
c = _candidate(sources=["reddit"]) c = _candidate(sources=["reddit"])
self.assertIsNone(render._format_corroboration(c)) self.assertIsNone(render._format_corroboration(c))
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._format_explanation # render._format_explanation
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFormatExplanation(unittest.TestCase): class TestFormatExplanation(unittest.TestCase):
def test_hides_fallback_sentinel(self): def test_hides_fallback_sentinel(self):
@@ -206,11 +202,11 @@ class TestFormatExplanation(unittest.TestCase):
c = _candidate(explanation="Directly compares frameworks") c = _candidate(explanation="Directly compares frameworks")
self.assertEqual(render._format_explanation(c), "Directly compares frameworks") self.assertEqual(render._format_explanation(c), "Directly compares frameworks")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._fmt_pairs and _format_number # render._fmt_pairs and _format_number
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestFmtPairs(unittest.TestCase): class TestFmtPairs(unittest.TestCase):
def test_basic(self): def test_basic(self):
@@ -231,11 +227,11 @@ class TestFormatNumber(unittest.TestCase):
def test_small_integer(self): def test_small_integer(self):
self.assertEqual(render._format_number(42), "42") self.assertEqual(render._format_number(42), "42")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# render._truncate # render._truncate
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestTruncate(unittest.TestCase): class TestTruncate(unittest.TestCase):
def test_short_text(self): def test_short_text(self):
@@ -246,11 +242,11 @@ class TestTruncate(unittest.TestCase):
self.assertTrue(result.endswith("...")) self.assertTrue(result.endswith("..."))
self.assertEqual(len(result), 50) self.assertEqual(len(result), 50)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# planner._normalize_subquery_weights # planner._normalize_subquery_weights
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestNormalizeSubqueryWeights(unittest.TestCase): class TestNormalizeSubqueryWeights(unittest.TestCase):
def test_sums_to_one(self): def test_sums_to_one(self):
@@ -270,11 +266,11 @@ class TestNormalizeSubqueryWeights(unittest.TestCase):
normed = planner._normalize_subquery_weights(sqs) normed = planner._normalize_subquery_weights(sqs)
self.assertAlmostEqual(normed[0].weight / normed[1].weight, 4.0) self.assertAlmostEqual(normed[0].weight / normed[1].weight, 4.0)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# planner._normalize_weights # planner._normalize_weights
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestNormalizeWeights(unittest.TestCase): class TestNormalizeWeights(unittest.TestCase):
def test_sums_to_one(self): def test_sums_to_one(self):
@@ -285,11 +281,11 @@ class TestNormalizeWeights(unittest.TestCase):
result = planner._normalize_weights({"a": 2.0, "b": -1.0}) result = planner._normalize_weights({"a": 2.0, "b": -1.0})
self.assertAlmostEqual(result["b"], 0.0) self.assertAlmostEqual(result["b"], 0.0)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# planner._trim_subqueries_for_depth # planner._trim_subqueries_for_depth
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestTrimSubqueriesForDepth(unittest.TestCase): class TestTrimSubqueriesForDepth(unittest.TestCase):
def _sq(self, label: str = "primary", sources: list[str] = None) -> schema.SubQuery: def _sq(self, label: str = "primary", sources: list[str] = None) -> schema.SubQuery:
@@ -318,11 +314,11 @@ class TestTrimSubqueriesForDepth(unittest.TestCase):
# Deep comparison should also use capability expansion, not trim # Deep comparison should also use capability expansion, not trim
self.assertGreaterEqual(len(result[0].sources), 4) self.assertGreaterEqual(len(result[0].sources), 4)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# signals.annotate_stream # signals.annotate_stream
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestAnnotateStream(unittest.TestCase): class TestAnnotateStream(unittest.TestCase):
def test_attaches_metadata(self): def test_attaches_metadata(self):
@@ -345,11 +341,11 @@ class TestAnnotateStream(unittest.TestCase):
annotated = signals.annotate_stream(items, "test query", "balanced_recent") annotated = signals.annotate_stream(items, "test query", "balanced_recent")
self.assertEqual(annotated[0].item_id, "high") self.assertEqual(annotated[0].item_id, "high")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# signals.prune_low_relevance # signals.prune_low_relevance
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestPruneLowRelevance(unittest.TestCase): class TestPruneLowRelevance(unittest.TestCase):
def test_removes_low_relevance_items(self): def test_removes_low_relevance_items(self):
@@ -369,11 +365,11 @@ class TestPruneLowRelevance(unittest.TestCase):
result = signals.prune_low_relevance(items, minimum=0.1) result = signals.prune_low_relevance(items, minimum=0.1)
self.assertEqual(len(result), 1) # fallback keeps all self.assertEqual(len(result), 1) # fallback keeps all
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Bug fixes found by PR review agents # Bug fixes found by PR review agents
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestDaysAgoZeroFalsy(unittest.TestCase): class TestDaysAgoZeroFalsy(unittest.TestCase):
"""render._assess_data_freshness must not treat days_ago=0 as falsy.""" """render._assess_data_freshness must not treat days_ago=0 as falsy."""
@@ -455,7 +451,6 @@ class TestGenericEngagementFormatter(unittest.TestCase):
# Should contain numeric values, not dict keys as numbers # Should contain numeric values, not dict keys as numbers
self.assertIn("500", result) self.assertIn("500", result)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
@@ -521,7 +516,6 @@ class TestDefaultDepthDoesNotCapSources(unittest.TestCase):
self.assertLessEqual(len(plan.subqueries[0].sources), 3) self.assertLessEqual(len(plan.subqueries[0].sources), 3)
class TestRerankWeightBalance(unittest.TestCase): class TestRerankWeightBalance(unittest.TestCase):
"""Reranker weight must dominate over RRF when candidates have divergent quality.""" """Reranker weight must dominate over RRF when candidates have divergent quality."""
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import normalize from lib import normalize
@@ -225,6 +221,5 @@ class NormalizeV3Tests(unittest.TestCase):
) )
self.assertEqual([], normalized) self.assertEqual([], normalized)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,11 +1,7 @@
import sys
import threading import threading
import unittest import unittest
from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import pipeline from lib import pipeline
from lib import http from lib import http
from lib import schema from lib import schema
@@ -1009,6 +1005,5 @@ class TestExcludeSourcesEndToEnd(unittest.TestCase):
self.assertNotIn("tiktok", sources) self.assertNotIn("tiktok", sources)
self.assertNotIn("instagram", sources) self.assertNotIn("instagram", sources)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-7
View File
@@ -1,16 +1,10 @@
# ruff: noqa: E402
"""Tests for planner.plan_query internal_subrun quiet mode.""" """Tests for planner.plan_query internal_subrun quiet mode."""
from __future__ import annotations from __future__ import annotations
import io import io
import sys
import unittest import unittest
from contextlib import redirect_stderr from contextlib import redirect_stderr
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
from lib import planner from lib import planner
@@ -50,6 +44,5 @@ class PlannerQuietModeTests(unittest.TestCase):
# no planner-error indication. # no planner-error indication.
self.assertGreater(len(plan.subqueries), 0) self.assertGreater(len(plan.subqueries), 0)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+35 -5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import planner from lib import planner
@@ -96,6 +92,41 @@ class PlannerV3Tests(unittest.TestCase):
self.assertEqual(1, len(plan.subqueries)) self.assertEqual(1, len(plan.subqueries))
self.assertEqual(["reddit", "x"], plan.subqueries[0].sources) self.assertEqual(["reddit", "x"], plan.subqueries[0].sources)
def test_quick_mode_preserves_explicit_requested_sources(self):
raw = {
"intent": "product",
"freshness_mode": "balanced_recent",
"cluster_mode": "debate",
"subqueries": [
{
"label": "primary",
"search_query": "AI coding agents",
"ranking_query": "What are people saying about AI coding agents?",
"sources": ["reddit", "youtube", "grounding", "digg"],
"weight": 1.0,
}
],
}
plan = planner._sanitize_plan(
raw,
"AI coding agents",
["reddit", "youtube", "grounding", "digg"],
["reddit", "youtube", "grounding", "digg"],
"quick",
)
self.assertIn("digg", plan.subqueries[0].sources)
def test_quick_mode_preserves_explicit_requested_sources_in_fallback_plan(self):
plan = planner.plan_query(
topic="AI coding agents",
available_sources=["reddit", "youtube", "github"],
requested_sources=["reddit", "github"],
depth="quick",
provider=None,
model=None,
)
self.assertIn("github", plan.subqueries[0].sources)
def test_default_comparison_uses_all_capable_sources(self): def test_default_comparison_uses_all_capable_sources(self):
plan = planner.plan_query( plan = planner.plan_query(
topic="codex vs claude code", topic="codex vs claude code",
@@ -452,6 +483,5 @@ class FallbackDefaultsTests(unittest.TestCase):
self.assertIn("LLM planning failed", output) self.assertIn("LLM planning failed", output)
self.assertNotIn("No --plan passed", output) self.assertNotIn("No --plan passed", output)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+1 -5
View File
@@ -1,16 +1,13 @@
import json import json
import sys
import tomllib import tomllib
import unittest import unittest
from pathlib import Path from pathlib import Path
from lib.skill_meta import read_skill_version
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days" SKILL_ROOT = ROOT / "skills" / "last30days"
sys.path.insert(0, str(SKILL_ROOT / "scripts"))
from lib.skill_meta import read_skill_version # noqa: E402
def _json(path: Path) -> dict: def _json(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8")) return json.loads(path.read_text(encoding="utf-8"))
@@ -60,6 +57,5 @@ class TestPluginContract(unittest.TestCase):
self.assertEqual([], offenders) self.assertEqual([], offenders)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+14 -18
View File
@@ -1,19 +1,15 @@
"""Tests for polymarket.py - Polymarket prediction market search.""" """Tests for polymarket.py - Polymarket prediction market search."""
import json import json
import sys
from pathlib import Path
from unittest.mock import Mock, patch from unittest.mock import Mock, patch
import pytest import pytest
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import polymarket from lib import polymarket
# === Helper Functions === # === Helper Functions ===
def create_mock_event( def create_mock_event(
event_id="evt-123", event_id="evt-123",
title="Test Event", title="Test Event",
@@ -60,9 +56,9 @@ def create_mock_market(
"liquidity": liquidity, "liquidity": liquidity,
} }
# === Tests for _extract_core_subject() === # === Tests for _extract_core_subject() ===
def test_extract_core_subject_basic(): def test_extract_core_subject_basic():
"""Test basic subject extraction.""" """Test basic subject extraction."""
result = polymarket._extract_core_subject("AI frameworks") result = polymarket._extract_core_subject("AI frameworks")
@@ -86,9 +82,9 @@ def test_extract_core_subject_multiple_prefixes():
result = polymarket._extract_core_subject("research AI models") result = polymarket._extract_core_subject("research AI models")
assert result == "AI models" assert result == "AI models"
# === Tests for _expand_queries() === # === Tests for _expand_queries() ===
def test_expand_queries_basic(): def test_expand_queries_basic():
"""Test basic query expansion.""" """Test basic query expansion."""
queries = polymarket._expand_queries("AI framework") queries = polymarket._expand_queries("AI framework")
@@ -132,9 +128,9 @@ def test_expand_queries_cap_at_six():
assert len(queries) <= 6 assert len(queries) <= 6
# === Tests for _passes_topic_filter() === # === Tests for _passes_topic_filter() ===
def test_passes_topic_filter_match(): def test_passes_topic_filter_match():
"""Test that matching events pass the filter.""" """Test that matching events pass the filter."""
assert polymarket._passes_topic_filter("AI safety", "AI Safety Conference 2026") is True assert polymarket._passes_topic_filter("AI safety", "AI Safety Conference 2026") is True
@@ -194,9 +190,9 @@ def test_passes_topic_filter_multi_word_edge_exactly_three():
"Tesla stock price", "Tesla quarterly earnings" "Tesla stock price", "Tesla quarterly earnings"
) is False # only "tesla" matches, needs 2 ) is False # only "tesla" matches, needs 2
# === Tests for _parse_outcome_prices() === # === Tests for _parse_outcome_prices() ===
def test_parse_outcome_prices_basic(): def test_parse_outcome_prices_basic():
"""Test basic outcome price parsing.""" """Test basic outcome price parsing."""
market = { market = {
@@ -259,9 +255,9 @@ def test_parse_outcome_prices_invalid_json():
assert result == [] assert result == []
# === Tests for _format_price_movement() === # === Tests for _format_price_movement() ===
def test_format_price_movement_one_day(): def test_format_price_movement_one_day():
"""Test formatting one-day price movement.""" """Test formatting one-day price movement."""
market = { market = {
@@ -322,9 +318,9 @@ def test_format_price_movement_missing_data():
assert result is None assert result is None
# === Tests for _shorten_question() === # === Tests for _shorten_question() ===
def test_shorten_question_will_pattern(): def test_shorten_question_will_pattern():
"""Test shortening 'Will X...' questions.""" """Test shortening 'Will X...' questions."""
result = polymarket._shorten_question("Will Arizona win the NCAA Tournament?") result = polymarket._shorten_question("Will Arizona win the NCAA Tournament?")
@@ -354,9 +350,9 @@ def test_shorten_question_long():
assert len(result) <= 40 assert len(result) <= 40
# === Tests for search_polymarket() === # === Tests for search_polymarket() ===
def test_search_polymarket_result_cap(): def test_search_polymarket_result_cap():
"""Test that result cap configuration exists.""" """Test that result cap configuration exists."""
assert "quick" in polymarket.RESULT_CAP assert "quick" in polymarket.RESULT_CAP
@@ -385,8 +381,9 @@ def test_search_polymarket_query_expansion():
# Should expand to multiple queries # Should expand to multiple queries
assert len(queries) >= 2 assert len(queries) >= 2
@patch('lib.polymarket.http.post') @patch('lib.polymarket.http.post')
def test_search_polymarket_http_error_handling(mock_post): def test_search_polymarket_http_error_handling(mock_post):
"""Test graceful handling of HTTP errors.""" """Test graceful handling of HTTP errors."""
from lib.http import HTTPError from lib.http import HTTPError
@@ -397,9 +394,9 @@ def test_search_polymarket_http_error_handling(mock_post):
# Should return structure with error # Should return structure with error
assert "events" in result or "error" in result assert "events" in result or "error" in result
# === Tests for parse_polymarket_response() === # === Tests for parse_polymarket_response() ===
def test_parse_polymarket_response_basic(): def test_parse_polymarket_response_basic():
"""Test basic response parsing.""" """Test basic response parsing."""
response = { response = {
@@ -472,9 +469,9 @@ def test_parse_polymarket_response_engagement():
# Check for volume or liquidity fields # Check for volume or liquidity fields
assert "volume24hr" in items[0] or "liquidity" in items[0] or isinstance(items[0], dict) assert "volume24hr" in items[0] or "liquidity" in items[0] or isinstance(items[0], dict)
# === Tests for engagement scoring === # === Tests for engagement scoring ===
def test_engagement_with_volume(): def test_engagement_with_volume():
"""Test engagement calculation with volume.""" """Test engagement calculation with volume."""
response = { response = {
@@ -491,9 +488,9 @@ def test_engagement_with_volume():
# volume24hr should be captured # volume24hr should be captured
assert "volume24hr" in engagement or isinstance(engagement, dict) assert "volume24hr" in engagement or isinstance(engagement, dict)
# === Tests for noise-word query skipping === # === Tests for noise-word query skipping ===
def test_expand_queries_skips_noise_words(): def test_expand_queries_skips_noise_words():
"""Noise words like 'west' should not become standalone queries.""" """Noise words like 'west' should not become standalone queries."""
queries = polymarket._expand_queries("kanye west") queries = polymarket._expand_queries("kanye west")
@@ -520,9 +517,9 @@ def test_expand_queries_all_noise_words_keeps_phrase():
lowered = [q.lower() for q in queries] lowered = [q.lower() for q in queries]
assert "north" not in lowered or "north west" in lowered # only as part of phrase assert "north" not in lowered or "north west" in lowered # only as part of phrase
# === Tests for per-item relevance floor === # === Tests for per-item relevance floor ===
def test_per_item_relevance_floor_drops_zero_items(): def test_per_item_relevance_floor_drops_zero_items():
"""Items with relevance 0.0 should be dropped even if best item is high.""" """Items with relevance 0.0 should be dropped even if best item is high."""
# Simulate the filtering logic directly # Simulate the filtering logic directly
@@ -559,6 +556,5 @@ def test_per_item_relevance_floor_no_drops_when_all_high():
filtered = [i for i in items if i["relevance"] >= 0.10] filtered = [i for i in items if i["relevance"] >= 0.10]
assert len(filtered) == 3 assert len(filtered) == 3
if __name__ == "__main__": if __name__ == "__main__":
pytest.main([__file__, "-v"]) pytest.main([__file__, "-v"])
-7
View File
@@ -1,14 +1,8 @@
# ruff: noqa: E402
"""Tests for --polymarket-keywords filter and filter_items_against_keywords.""" """Tests for --polymarket-keywords filter and filter_items_against_keywords."""
from __future__ import annotations from __future__ import annotations
import sys
import unittest import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(REPO_ROOT / "skills" / "last30days" / "scripts"))
from lib import polymarket from lib import polymarket
@@ -69,6 +63,5 @@ class FilterItemsAgainstKeywordsTests(unittest.TestCase):
out = polymarket.filter_items_against_keywords(items, ["nba", "gsw"]) out = polymarket.filter_items_against_keywords(items, ["nba", "gsw"])
self.assertEqual(out, []) self.assertEqual(out, [])
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -6,11 +6,7 @@ public v3.0.8 and still returned junk for queries like 'birthday gift for
cannot bypass by skipping SKILL.md. cannot bypass by skipping SKILL.md.
""" """
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import preflight from lib import preflight
@@ -123,6 +119,5 @@ class TestRefuseMessage(unittest.TestCase):
assert msg is not None assert msg is not None
self.assertIn("birthday gift for 40 year old", msg) self.assertIn("birthday gift for 40 year old", msg)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,9 +1,5 @@
import json import json
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import providers from lib import providers
@@ -162,6 +158,5 @@ class TestParseCodexStream(unittest.TestCase):
def test_done_only_stream(self): def test_done_only_stream(self):
self.assertEqual({}, providers._parse_codex_stream("data: [DONE]\n\n")) self.assertEqual({}, providers._parse_codex_stream("data: [DONE]\n\n"))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+2 -7
View File
@@ -5,19 +5,14 @@ HN, Polymarket, Reddit (always active), X, YouTube.
ScrapeCreators adds TikTok + Instagram as bonus sources, not core. ScrapeCreators adds TikTok + Instagram as bonus sources, not core.
""" """
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
import pytest import pytest
from unittest.mock import patch from unittest.mock import patch
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Helpers # Helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def _base_config(**overrides): def _base_config(**overrides):
"""Return a minimal config dict.""" """Return a minimal config dict."""
config = { config = {
@@ -52,11 +47,11 @@ def _compute(config_overrides=None, result_overrides=None, ytdlp_installed=False
with patch.object(youtube_yt, "is_ytdlp_installed", return_value=ytdlp_installed): with patch.object(youtube_yt, "is_ytdlp_installed", return_value=ytdlp_installed):
return compute_quality_score(config, results) return compute_quality_score(config, results)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Tests # Tests
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestBaseline: class TestBaseline:
"""HN + Polymarket + Reddit always active (no X, no YT) -> 60%.""" """HN + Polymarket + Reddit always active (no X, no YT) -> 60%."""
-6
View File
@@ -1,10 +1,6 @@
"""Tests for query.py — shared query utilities.""" """Tests for query.py — shared query utilities."""
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib.query import NOISE_WORDS, extract_compound_terms, extract_core_subject from lib.query import NOISE_WORDS, extract_compound_terms, extract_core_subject
@@ -126,7 +122,6 @@ class TestNoiseWordsCompleteness(unittest.TestCase):
self.assertIn(w, NOISE_WORDS) self.assertIn(w, NOISE_WORDS)
class TestExtractCompoundTerms(unittest.TestCase): class TestExtractCompoundTerms(unittest.TestCase):
"""Tests for extract_compound_terms().""" """Tests for extract_compound_terms()."""
@@ -148,6 +143,5 @@ class TestExtractCompoundTerms(unittest.TestCase):
self.assertIn("vc-backed", terms) self.assertIn("vc-backed", terms)
self.assertIn("start-up", terms) self.assertIn("start-up", terms)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import query from lib import query
@@ -39,6 +35,5 @@ class QueryV3Tests(unittest.TestCase):
self.assertIn("Claude Code", terms) self.assertIn("Claude Code", terms)
self.assertIn("React Native", terms) self.assertIn("React Native", terms)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-5
View File
@@ -1,8 +1,4 @@
import sys
import unittest import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib.reddit import ( from lib.reddit import (
_extract_date, _extract_date,
@@ -264,6 +260,5 @@ class TestEnrichmentBudget(unittest.TestCase):
enriched = [i for i in result if i.get("top_comments")] enriched = [i for i in result if i.get("top_comments")]
self.assertEqual(len(enriched), 0) self.assertEqual(len(enriched), 0)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
-3
View File
@@ -1,12 +1,10 @@
"""Tests for reddit_enrich.py — comment enrichment and parsing.""" """Tests for reddit_enrich.py — comment enrichment and parsing."""
import json import json
import sys
import unittest import unittest
from pathlib import Path from pathlib import Path
# Add lib to path # Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import reddit_enrich from lib import reddit_enrich
@@ -124,6 +122,5 @@ class TestExtractCommentInsights(unittest.TestCase):
insights = reddit_enrich.extract_comment_insights(comments, limit=3) insights = reddit_enrich.extract_comment_insights(comments, limit=3)
self.assertLessEqual(len(insights), 3) self.assertLessEqual(len(insights), 3)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+266
View File
@@ -0,0 +1,266 @@
"""Tests for scripts/lib/reddit_keyless.py — tiered keyless Reddit pipeline."""
from unittest import mock
from lib import reddit_keyless
def _post(i, date="2026-05-20", rel=0.0):
url = f"https://www.reddit.com/r/test/comments/{i:06d}/post_{i}/"
return {
"id": "", "title": f"Post {i}", "url": url, "score": 0, "num_comments": 0,
"subreddit": "test", "created_utc": None, "author": "u", "selftext": "",
"date": date, "engagement": {"score": 0, "num_comments": 0, "upvote_ratio": None},
"relevance": rel, "why_relevant": "Reddit RSS", "metadata": {},
}
def _scored(i, score, ncmt=0):
p = _post(i)
p["score"] = score
p["num_comments"] = ncmt
p["engagement"]["score"] = score
p["engagement"]["num_comments"] = ncmt
p["why_relevant"] = "Reddit listing"
p["metadata"] = {"post_id": f"{i:06d}"}
return p
class TestDiscoveryTierOrder:
"""Tier 0 (.json) is tried first; RSS + scored listings are the keyless path."""
def test_tier0_success_skips_keyless(self):
with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[_post(1)]) as t0, \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss") as rss, \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings") as lst:
out = reddit_keyless._discover("topic", "default", None)
assert len(out) == 1
t0.assert_called_once()
rss.assert_not_called()
lst.assert_not_called()
def test_tier0_empty_falls_to_keyless(self):
with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[_post(1), _post(2)]) as rss, \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[]):
out = reddit_keyless._discover("topic", "default", ["test"])
assert len(out) == 2
rss.assert_called_once()
def test_listing_scores_backfill_rss_posts(self):
# RSS finds post 1 (no score); listing card for post 1 carries the score.
rss_post = _post(1)
listing_post = _scored(1, score=52692, ncmt=1743)
with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[listing_post]):
out = reddit_keyless._discover("topic", "default", ["test"])
# listing post (scored) is kept; RSS dup of same url is dropped
assert len(out) == 1
assert out[0]["engagement"]["score"] == 52692
assert out[0]["num_comments"] == 1743
def test_scores_flow_to_distinct_rss_posts(self):
# Distinct RSS post whose id matches a listing card gets backfilled.
rss_post = _post(7) # url .../000007/...
listing_post = _scored(7, score=999)
listing_post["url"] = "https://www.reddit.com/r/test/comments/zzzzzz/other/"
with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[listing_post]):
out = reddit_keyless._discover("topic", "default", ["test"])
backfilled = [p for p in out if p["url"] == rss_post["url"]][0]
assert backfilled["engagement"]["score"] == 999
def test_bare_query_does_not_merge_listing_discovery(self):
# No subreddits provided: derived-subreddit listings must NOT be added as
# results (avoids flooding with off-topic high-upvote posts) — only used
# to backfill scores onto the keyword-matched RSS posts.
rss_post = _post(1) # on-topic keyword match
offtopic_listing = _scored(99, score=88888) # high score, unrelated sub
offtopic_listing["url"] = "https://www.reddit.com/r/random/comments/zzz999/x/"
with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless, "_top_subreddits", return_value=["random"]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[offtopic_listing]):
out = reddit_keyless._discover("topic", "default", None)
urls = [p["url"] for p in out]
assert rss_post["url"] in urls
assert offtopic_listing["url"] not in urls # not merged as discovery
def test_tier0_never_raises(self):
with mock.patch("lib.reddit_public.search", side_effect=Exception("boom")), \
mock.patch.object(reddit_keyless.reddit_rss, "search_rss", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings", return_value=[]):
assert reddit_keyless._discover("t", "default", None) == []
class TestSearchAndEnrich:
"""Full pipeline: discover -> date filter -> rank -> enrich -> reindex."""
def _patch_enrich_passthrough(self):
return mock.patch.object(
reddit_keyless.reddit_shreddit, "fetch_comments",
return_value={"top_comments": [], "comment_insights": [], "num_comments": None},
)
def test_returns_empty_when_no_discovery(self):
with mock.patch.object(reddit_keyless, "_discover", return_value=[]):
assert reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31") == []
def test_date_filter_keeps_in_range_and_unknown(self):
posts = [_post(1, date="2026-05-10"), _post(2, date="2020-01-01"),
_post(3, date=None)]
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
self._patch_enrich_passthrough():
out = reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31")
titles = {p["title"] for p in out}
assert "Post 1" in titles and "Post 3" in titles
assert "Post 2" not in titles
def test_reindexes_ids(self):
posts = [_post(1), _post(2), _post(3)]
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
self._patch_enrich_passthrough():
out = reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31")
assert [p["id"] for p in out] == ["R1", "R2", "R3"]
def test_enrichment_attaches_comments(self):
posts = [_post(1)]
enriched = {
"top_comments": [{"score": 9, "date": "2026-05-19", "author": "a",
"excerpt": "great", "url": "https://reddit.com/x"}],
"comment_insights": ["great point about X"],
"num_comments": 14,
}
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
mock.patch.object(reddit_keyless.reddit_shreddit, "fetch_comments",
return_value=enriched):
out = reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31")
assert out[0]["top_comments"][0]["score"] == 9
assert out[0]["num_comments"] == 14
assert out[0]["engagement"]["num_comments"] == 14
def test_enrichment_failure_keeps_posts(self):
posts = [_post(i) for i in range(8)]
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
mock.patch.object(reddit_keyless.reddit_shreddit, "fetch_comments",
side_effect=Exception("svc down")):
out = reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31")
assert len(out) == 8 # all posts retained despite enrichment failure
def test_only_top_n_enriched_by_depth(self):
posts = [_post(i, rel=1.0 - i / 100) for i in range(10)]
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
mock.patch.object(reddit_keyless.reddit_shreddit, "fetch_comments",
return_value={"top_comments": [], "comment_insights": [],
"num_comments": None}) as fc:
reddit_keyless.search_and_enrich("t", "2026-05-01", "2026-05-31", depth="quick")
# quick depth enriches only top 3 posts
assert fc.call_count == reddit_keyless.ENRICH_LIMITS["quick"]
class TestSlotPriority:
"""Enrichment slot selection prefers entity-matching posts (R1-R3)."""
@staticmethod
def _titled(i, title, score=0, selftext=""):
p = _post(i)
p["title"] = title
p["selftext"] = selftext
p["score"] = score
p["engagement"]["score"] = score
return p
def test_on_topic_low_score_beats_off_topic_high_score(self):
# 3 off-topic monsters + 2 on-topic small threads; quick depth = 3 slots.
posts = [
self._titled(1, "Stop asking what model to run", score=2662),
self._titled(2, "RTX 4090 PSA", score=2068),
self._titled(3, "Gemma 4 release", score=997),
self._titled(4, "My OpenClaw self-migrated", score=73),
self._titled(5, "Using openclaw with Claude API key is so expensive", score=47),
]
enriched_urls = []
def _capture(url):
enriched_urls.append(url)
return {"top_comments": [], "comment_insights": [], "num_comments": None}
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
mock.patch.object(reddit_keyless.reddit_shreddit, "fetch_comments",
side_effect=_capture):
reddit_keyless.search_and_enrich(
"openclaw", "2026-05-01", "2026-05-31", depth="quick")
assert posts[3]["url"] in enriched_urls
assert posts[4]["url"] in enriched_urls
assert len(enriched_urls) == reddit_keyless.ENRICH_LIMITS["quick"]
def test_multiword_topic_uses_substring_not_token_overlap(self):
# "claude tips" clears token overlap for "Claude Code" but rerank
# demotes it; the partition must mirror rerank's substring test.
token_only = self._titled(1, "claude tips", score=500)
full_entity = self._titled(2, "Claude Code best setup", score=5)
out = reddit_keyless._slot_priority("Claude Code", [token_only, full_entity])
assert out[0] is full_entity
assert out[1] is token_only
def test_intent_modifier_stripped_from_topic(self):
on_topic = self._titled(1, "Hermes Agent v0.13 is great", score=1)
off_topic = self._titled(2, "Hermes Birkin unboxing", score=900)
out = reddit_keyless._slot_priority("Hermes Agent review", [off_topic, on_topic])
assert out[0] is on_topic
def test_all_miss_keeps_score_order_and_full_slots(self):
posts = [self._titled(i, f"Gemma thread {i}", score=1000 - i) for i in range(5)]
out = reddit_keyless._slot_priority("openclaw", posts)
assert out == posts # order unchanged
with mock.patch.object(reddit_keyless, "_discover", return_value=posts), \
mock.patch.object(reddit_keyless.reddit_shreddit, "fetch_comments",
return_value={"top_comments": [], "comment_insights": [],
"num_comments": None}) as fc:
reddit_keyless.search_and_enrich(
"openclaw", "2026-05-01", "2026-05-31", depth="quick")
assert fc.call_count == reddit_keyless.ENRICH_LIMITS["quick"]
def test_same_tier_order_preserved(self):
posts = [self._titled(i, f"openclaw thread {i}", score=100 - i) for i in range(4)]
out = reddit_keyless._slot_priority("openclaw", posts)
assert out == posts
def test_empty_entity_falls_back_to_token_overlap(self):
# Pure intent-modifier topic yields no primary entity; fallback path
# must not raise and must keep every post.
posts = [self._titled(1, "Post one"), self._titled(2, "review of things")]
out = reddit_keyless._slot_priority("review", posts)
assert len(out) == 2
assert {p["url"] for p in out} == {p["url"] for p in posts}
def test_selftext_match_lands_in_match_tier(self):
body_match = self._titled(1, "Need help with my setup", score=2,
selftext="my openclaw agent keeps asking for ssh keys")
off_topic = self._titled(2, "Gemma 4 with QAT", score=700)
out = reddit_keyless._slot_priority("openclaw", [off_topic, body_match])
assert out[0] is body_match
def test_none_score_posts_do_not_break_partition(self):
p1 = self._titled(1, "openclaw tips")
p1["engagement"]["score"] = None
p2 = self._titled(2, "Gemma news")
p2["engagement"]["score"] = None
out = reddit_keyless._slot_priority("openclaw", [p2, p1])
assert out[0] is p1
def test_partition_never_raises(self):
posts = [self._titled(1, "openclaw tips", score=1)]
with mock.patch("lib.rerank._primary_entity", side_effect=Exception("boom")):
out = reddit_keyless._slot_priority("openclaw", posts)
assert out == posts
+85
View File
@@ -0,0 +1,85 @@
"""Tests for scripts/lib/reddit_listing.py — keyless scored listing scrape."""
from pathlib import Path
from unittest import mock
from lib import reddit_listing as rl
FIXTURE = Path(__file__).resolve().parent.parent / "fixtures" / "reddit_listing_cards_sample.html"
def _html():
return FIXTURE.read_text(encoding="utf-8")
class TestParseCards:
"""parse_cards reads <shreddit-post> cards into scored post dicts."""
def test_parses_five_cards(self):
posts = rl.parse_cards(_html(), query="netherlands")
assert len(posts) == 5
def test_real_score_and_count(self):
posts = rl.parse_cards(_html())
top = posts[0]
assert top["score"] == 52692 # the real upvote count
assert top["engagement"]["score"] == 52692
assert top["num_comments"] == 1743
assert top["engagement"]["num_comments"] == 1743
def test_normalized_shape(self):
post = rl.parse_cards(_html())[0]
required = {"id", "title", "url", "score", "num_comments", "subreddit",
"created_utc", "author", "selftext", "date",
"engagement", "relevance", "why_relevant", "metadata"}
assert required.issubset(set(post.keys()))
assert post["why_relevant"] == "Reddit listing"
assert post["metadata"]["post_id"] # post id captured for backfill
def test_fields_populated(self):
post = rl.parse_cards(_html())[0]
assert post["title"]
assert post["author"] == "AdSpecialist6598"
assert post["subreddit"] == "technology"
assert "/comments/" in post["url"]
assert post["date"] and len(post["date"]) == 10
def test_empty_html_returns_empty(self):
assert rl.parse_cards("") == []
assert rl.parse_cards("<div>no cards</div>") == []
class TestListingUrl:
def test_top_includes_timeframe(self):
u = rl._listing_url("technology", "top")
assert "community-more-posts/top/" in u and "name=technology" in u and "t=month" in u
def test_hot_no_timeframe(self):
u = rl._listing_url("r/technology", "hot")
assert "community-more-posts/hot/" in u and "name=technology" in u and "t=" not in u
assert ".json" not in u
class TestFetchListings:
def test_dedupes_across_sorts(self):
with mock.patch.object(rl.http, "get_text", return_value=_html()):
posts = rl.fetch_listings(["technology"], depth="default")
urls = [p["url"] for p in posts]
assert len(urls) == len(set(urls)) # top + hot return same cards -> deduped
def test_no_subreddits_returns_empty(self):
assert rl.fetch_listings([], depth="default") == []
def test_all_fetches_fail_returns_empty(self):
with mock.patch.object(rl.http, "get_text", return_value=None):
assert rl.fetch_listings(["technology"]) == []
class TestScoreIndex:
def test_builds_post_id_to_score_map(self):
with mock.patch.object(rl.http, "get_text", return_value=_html()):
idx = rl.score_index(["technology"], depth="quick")
assert idx # non-empty
first = next(iter(idx.values()))
assert set(first.keys()) == {"score", "num_comments"}
assert any(v["score"] == 52692 for v in idx.values())
+18 -81
View File
@@ -5,19 +5,16 @@ import urllib.error
from unittest import mock from unittest import mock
import pytest import pytest
import sys
import os
# Ensure lib is importable # Ensure lib is importable
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "skills", "last30days", "scripts"))
from lib import reddit_public from lib import reddit_public
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Fixtures / helpers # Fixtures / helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def _make_reddit_listing(posts): def _make_reddit_listing(posts):
"""Build a Reddit listing JSON structure from a list of post dicts.""" """Build a Reddit listing JSON structure from a list of post dicts."""
children = [] children = []
@@ -38,7 +35,6 @@ def _make_reddit_listing(posts):
}) })
return {"data": {"children": children}} return {"data": {"children": children}}
SAMPLE_LISTING = _make_reddit_listing([ SAMPLE_LISTING = _make_reddit_listing([
{ {
"title": "Claude Code is amazing", "title": "Claude Code is amazing",
@@ -72,11 +68,11 @@ def _mock_urlopen_ok(listing_data):
resp.__exit__ = mock.MagicMock(return_value=False) resp.__exit__ = mock.MagicMock(return_value=False)
return resp return resp
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Tests # Tests
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestSearchReturnsCorrectFields: class TestSearchReturnsCorrectFields:
"""Search query returns parsed results with correct fields.""" """Search query returns parsed results with correct fields."""
@@ -329,84 +325,25 @@ class TestMissingSubreddit:
results = reddit_public.search("test", subreddit="nonexistent") results = reddit_public.search("test", subreddit="nonexistent")
assert results == [] assert results == []
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Tests for comment enrichment (Unit 2) # search_reddit_public is now a thin shim over the keyless pipeline.
# Full discovery + enrichment behavior is covered in test_reddit_keyless.py.
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class TestEnrichmentIntegration:
"""search_reddit_public enriches top posts with comments."""
@mock.patch("lib.reddit_public._enrich_post") class TestSearchRedditPublicDelegatesToKeyless:
@mock.patch("lib.reddit_public.urllib.request.urlopen") """search_reddit_public delegates to reddit_keyless.search_and_enrich."""
def test_search_enriches_top_5_by_default(self, mock_urlopen, mock_enrich):
listing = _make_reddit_listing([
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
"score": 100 - i, "created_utc": 1711670400}
for i in range(10)
])
mock_urlopen.return_value = _mock_urlopen_ok(listing)
mock_enrich.side_effect = lambda item, timeout=10: item # pass-through
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31") def test_delegates_with_all_args(self):
with mock.patch("lib.reddit_keyless.search_and_enrich") as mock_keyless:
mock_keyless.return_value = [{"id": "R1", "title": "x"}]
results = reddit_public.search_reddit_public(
"test", "2024-03-01", "2024-03-31",
depth="quick", subreddits=["ClaudeAI"],
)
assert len(results) == 10 assert results == [{"id": "R1", "title": "x"}]
# Default depth enriches top 5 mock_keyless.assert_called_once_with(
assert mock_enrich.call_count == 5 "test", "2024-03-01", "2024-03-31",
depth="quick", subreddits=["ClaudeAI"],
@mock.patch("lib.reddit_public._enrich_post") )
@mock.patch("lib.reddit_public.urllib.request.urlopen")
def test_enrichment_timeout_keeps_posts(self, mock_urlopen, mock_enrich):
listing = _make_reddit_listing([
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
"score": 100 - i, "created_utc": 1711670400}
for i in range(10)
])
mock_urlopen.return_value = _mock_urlopen_ok(listing)
# Some enrichments raise, some succeed
call_count = {"n": 0}
def _side_effect(item, timeout=10):
call_count["n"] += 1
if call_count["n"] % 2 == 0:
raise TimeoutError("enrichment timed out")
return item
mock_enrich.side_effect = _side_effect
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
# All 10 posts should still be returned
assert len(results) == 10
@mock.patch("lib.reddit_public._enrich_post")
@mock.patch("lib.reddit_public.urllib.request.urlopen")
def test_all_enrichment_fails_all_posts_returned(self, mock_urlopen, mock_enrich):
listing = _make_reddit_listing([
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
"score": 100 - i, "created_utc": 1711670400}
for i in range(10)
])
mock_urlopen.return_value = _mock_urlopen_ok(listing)
mock_enrich.side_effect = Exception("total failure")
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
# All posts returned despite enrichment failure
assert len(results) == 10
@mock.patch("lib.reddit_public._enrich_post")
@mock.patch("lib.reddit_public.urllib.request.urlopen")
def test_quick_depth_enriches_top_3(self, mock_urlopen, mock_enrich):
listing = _make_reddit_listing([
{"title": f"Post {i}", "permalink": f"/r/test/comments/{i:06d}/post_{i}/",
"score": 100 - i, "created_utc": 1711670400}
for i in range(10)
])
mock_urlopen.return_value = _mock_urlopen_ok(listing)
mock_enrich.side_effect = lambda item, timeout=10: item
results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31", depth="quick")
assert len(results) == 10
# Quick depth enriches only top 3
assert mock_enrich.call_count == 3
+96
View File
@@ -0,0 +1,96 @@
"""Tests for scripts/lib/reddit_rss.py — keyless Reddit RSS discovery."""
from pathlib import Path
from unittest import mock
from lib import reddit_rss
FIXTURE = Path(__file__).resolve().parent.parent / "fixtures" / "reddit_search_rss_sample.xml"
def _feed_text():
return FIXTURE.read_text(encoding="utf-8")
class TestParseFeed:
"""_parse_feed turns Atom entries into normalized post dicts."""
def test_parses_entries(self):
posts = reddit_rss._parse_feed(_feed_text(), query="lifelock")
assert len(posts) == 5
for p in posts:
assert p["title"]
assert "/comments/" in p["url"]
assert p["url"].startswith("https://www.reddit.com/")
def test_normalized_shape_matches_scrapecreators(self):
post = reddit_rss._parse_feed(_feed_text(), query="x")[0]
required = {"id", "title", "url", "score", "num_comments", "subreddit",
"created_utc", "author", "selftext", "date",
"engagement", "relevance", "why_relevant", "metadata"}
assert required.issubset(set(post.keys()))
assert set(post["engagement"].keys()) == {"score", "num_comments", "upvote_ratio"}
assert post["why_relevant"] == "Reddit RSS"
def test_score_is_placeholder_zero(self):
# RSS carries no engagement score; it is backfilled during enrichment.
for p in reddit_rss._parse_feed(_feed_text(), query="x"):
assert p["score"] == 0
assert p["engagement"]["score"] == 0
def test_subreddit_derivation(self):
post = reddit_rss._parse_feed(_feed_text(), query="x")[0]
assert post["subreddit"] == "Rakuten"
def test_date_parsed_to_iso(self):
post = reddit_rss._parse_feed(_feed_text(), query="x")[0]
assert post["date"] and len(post["date"]) == 10 # YYYY-MM-DD
assert isinstance(post["created_utc"], float)
def test_author_strips_u_prefix(self):
authors = [p["author"] for p in reddit_rss._parse_feed(_feed_text(), query="x")]
assert all(not a.startswith("/u/") and not a.startswith("u/") for a in authors)
def test_empty_and_malformed_feed_never_raises(self):
assert reddit_rss._parse_feed("", query="x") == []
assert reddit_rss._parse_feed("<not xml", query="x") == []
assert reddit_rss._parse_feed("<feed></feed>", query="x") == []
def test_entry_without_comments_link_skipped(self):
feed = (
'<feed xmlns="http://www.w3.org/2005/Atom"><entry>'
'<title>Subreddit itself</title>'
'<link href="https://www.reddit.com/r/test/" />'
'<updated>2026-05-20T00:00:00+00:00</updated></entry></feed>'
)
assert reddit_rss._parse_feed(feed, query="x") == []
class TestSearchRss:
"""search_rss fans out, dedupes, assigns IDs, and honors depth limits."""
def test_dedupe_and_ids(self):
# Same feed returned for every URL -> deduped to 5 unique posts.
with mock.patch.object(reddit_rss.http, "get_text", return_value=_feed_text()):
posts = reddit_rss.search_rss("lifelock", depth="default",
subreddits=["Rakuten", "ConsumerAdvice"])
urls = [p["url"] for p in posts]
assert len(urls) == len(set(urls)) # no duplicates
assert [p["id"] for p in posts] == [f"R{i+1}" for i in range(len(posts))]
def test_depth_limit_quick(self):
with mock.patch.object(reddit_rss.http, "get_text", return_value=_feed_text()):
posts = reddit_rss.search_rss("lifelock", depth="quick")
assert len(posts) <= reddit_rss.DEPTH_LIMITS["quick"]
def test_all_feeds_fail_returns_empty(self):
with mock.patch.object(reddit_rss.http, "get_text", return_value=None):
posts = reddit_rss.search_rss("lifelock", subreddits=["Rakuten"])
assert posts == []
def test_builds_keyless_rss_urls(self):
urls = reddit_rss._build_urls("life lock", "default", ["Rakuten"])
assert any("search.rss?q=life+lock" in u and "/r/" not in u.split("?")[0] for u in urls)
assert any("/r/Rakuten/search.rss" in u and "restrict_sr=on" in u for u in urls)
assert any("/r/Rakuten/top.rss" in u for u in urls)
assert all(".json" not in u for u in urls) # never the dead endpoint
-4
View File
@@ -1,11 +1,8 @@
"""Tests for reddit.py — ScrapeCreators Reddit search module.""" """Tests for reddit.py — ScrapeCreators Reddit search module."""
import sys
import unittest import unittest
from pathlib import Path
# Add lib to path # Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
from lib import reddit from lib import reddit
@@ -176,6 +173,5 @@ class TestPostRelevance(unittest.TestCase):
) )
self.assertGreater(score, 0.7) self.assertGreater(score, 0.7)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+103
View File
@@ -0,0 +1,103 @@
"""Tests for scripts/lib/reddit_shreddit.py — keyless shreddit comment scrape."""
from pathlib import Path
from unittest import mock
from lib import reddit_shreddit as rs
FIXTURE = Path(__file__).resolve().parent.parent / "fixtures" / "reddit_shreddit_comments_sample.html"
def _html():
return FIXTURE.read_text(encoding="utf-8")
class TestExtractPostRef:
def test_extracts_sub_and_id(self):
ref = rs.extract_post_ref("https://www.reddit.com/r/Rakuten/comments/1taeiw0/title/")
assert ref == ("Rakuten", "1taeiw0")
def test_non_thread_url_returns_none(self):
assert rs.extract_post_ref("https://www.reddit.com/r/Rakuten/") is None
assert rs.extract_post_ref("") is None
def test_svc_url_shape(self):
# sort=top guarantees the highest-scored comments land on page 1.
assert rs._svc_url("Rakuten", "1taeiw0") == (
"https://www.reddit.com/svc/shreddit/comments/r/Rakuten/t3_1taeiw0?sort=top"
)
class TestParseComments:
"""parse_comments reads <shreddit-comment> elements into scored dicts."""
def test_happy_path(self):
comments = rs.parse_comments(_html())
assert len(comments) >= 1
for c in comments:
assert isinstance(c["score"], int)
assert c["author"] and c["author"] not in ("[deleted]", "[removed]")
assert c["body"]
def test_sorted_by_score_desc(self):
scores = [c["score"] for c in rs.parse_comments(_html())]
assert scores == sorted(scores, reverse=True)
def test_deleted_and_removed_filtered(self):
authors = [c["author"] for c in rs.parse_comments(_html())]
assert "[deleted]" not in authors and "[removed]" not in authors
def test_negative_score_retained(self):
scores = [c["score"] for c in rs.parse_comments(_html())]
assert -7 in scores # synthetic downvoted-but-real comment
def test_limit_honored(self):
assert len(rs.parse_comments(_html(), limit=2)) == 2
def test_body_text_extracted(self):
bodies = [c["body"] for c in rs.parse_comments(_html())]
assert any("$750" in b or "pending" in b for b in bodies)
def test_comment_url_built(self):
for c in rs.parse_comments(_html()):
if c["url"]:
assert c["url"].startswith("https://reddit.com/r/")
def test_empty_html_returns_empty(self):
assert rs.parse_comments("") == []
assert rs.parse_comments("<html>no comments here</html>") == []
class TestTotalComments:
def test_reads_total(self):
assert rs._total_comments(_html()) == 14
def test_missing_returns_none(self):
assert rs._total_comments("<html></html>") is None
class TestFetchComments:
"""fetch_comments wires URL -> svc fetch -> parse, never raising."""
def test_happy_path(self):
url = "https://www.reddit.com/r/Rakuten/comments/1taeiw0/title/"
with mock.patch.object(rs.http, "get_text", return_value=_html()) as m:
out = rs.fetch_comments(url)
# svc endpoint, not .json
assert "/svc/shreddit/comments/" in m.call_args[0][0]
assert ".json" not in m.call_args[0][0]
assert out["num_comments"] == 14
assert len(out["top_comments"]) >= 1
first = out["top_comments"][0]
assert {"score", "date", "author", "excerpt", "url"} <= set(first.keys())
assert isinstance(out["comment_insights"], list)
def test_bad_url_returns_empty(self):
out = rs.fetch_comments("https://www.reddit.com/r/Rakuten/")
assert out["top_comments"] == [] and out["num_comments"] is None
def test_fetch_failure_returns_empty(self):
url = "https://www.reddit.com/r/Rakuten/comments/1taeiw0/title/"
with mock.patch.object(rs.http, "get_text", return_value=None):
out = rs.fetch_comments(url)
assert out["top_comments"] == [] and out["num_comments"] is None

Some files were not shown because too many files have changed in this diff Show More