diff --git a/SKILL.md b/SKILL.md index 14ebd2b..b7f03e4 100644 --- a/SKILL.md +++ b/SKILL.md @@ -596,6 +596,43 @@ The first search finds subreddits. The second gives you current events context ( Extract 3-5 subreddit names from the results. Store as `RESOLVED_SUBREDDITS` (comma-separated, no r/ prefix). +**2a. Category-peer expansion (MANDATORY for product topics).** If the topic is a product in a recognizable category (AI image generation, AI video generation, AI coding agents, AI music, AI chat models, SaaS screen recording, prediction markets, etc.), the brand-specific subreddits that WebSearch returned are INSUFFICIENT. Add 2-3 peer subreddits from the category. Peer subs are where cross-product technique discussion actually lives. Missing them is the 2026-04-22 `GPT Image 2` failure mode: the model resolved `r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering` (all OpenAI-brand) and missed `r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt` where prompting techniques are actually shared. The user had to manually prompt "check image generation reddits too" to get a usable run. + +Canonical category peers (single source of truth; `scripts/lib/categories.py` mirrors this for the `--auto-resolve` engine path): + +| Category | Trigger keywords | Peer subs (priority order) | +|----------|------------------|---------------------------| +| `ai_image_generation` | image generation, text to image, GPT Image, Nano Banana, Midjourney, Stable Diffusion, DALL-E, Flux.1, Imagen, Seedance, Ideogram, Recraft | `StableDiffusion, midjourney, dalle2, aiArt, PromptEngineering, MediaSynthesis` | +| `ai_video_generation` | video generation, text to video, Sora, Veo 3, Runway Gen, Kling, Pika Labs, Luma Dream Machine, Hailuo | `aivideo, StableDiffusion, runwayml, singularity, MediaSynthesis` | +| `ai_music_generation` | music generation, ai music, Suno, Udio, Riffusion, Stable Audio | `SunoAI, udiomusic, aimusic, artificial` | +| `ai_coding_agent` | Claude Code, Cursor IDE, GitHub Copilot, Windsurf, Aider, Cline, OpenClaw, Hermes Agent, Continue.dev, Codeium, Devin | `ChatGPTCoding, LocalLLaMA, singularity, PromptEngineering` | +| `ai_agent_framework` | agent framework, LangChain, LangGraph, CrewAI, AutoGen, LlamaIndex, DSPy, smolagents | `LangChain, LocalLLaMA, AI_Agents, MachineLearning` | +| `ai_chat_model` | GPT-5/4, Claude Opus/Sonnet/Haiku, Gemini Pro/Flash, Llama 3/4, DeepSeek, Qwen, Mistral Large, Grok | `LocalLLaMA, ChatGPT, ClaudeAI, singularity, artificial` | +| `saas_screen_recording` | screen recording, screen recorder, Loom video, Tella screen, Vidyard | `SaaS, screenrecording, productivity, Entrepreneur` | +| `saas_productivity` | Notion app, Obsidian, Linear app, Asana, ClickUp, productivity app | `productivity, SaaS, ObsidianMD, Notion` | +| `prediction_markets` | Polymarket, Kalshi, prediction market, event contracts, Manifold Markets | `Polymarket, Kalshi, predictionmarkets` | +| `crypto_defi` | DeFi protocol, yield farming, liquidity pool, stablecoin, layer 2, L2 rollup | `defi, ethfinance, CryptoCurrency, ethereum` | + +**Merging rule.** Start with WebSearch-returned subs. Append 2-3 category peers in the priority order shown. Dedupe case-insensitively (don't list `midjourney` twice if WebSearch already returned it). Cap total at 10: if adding all peers would exceed the cap, keep every WebSearch-returned sub (they are the freshest signal) and drop peers from the end of the priority list. + +**Extrapolation.** If the topic is a product in a category NOT listed in the table (new AI tool, niche SaaS), use the same spirit: pick the 2-3 most active cross-product communities where technique discussion happens. A new image-gen tool still gets `r/StableDiffusion, r/midjourney, r/aiArt`. A new code editor still gets `r/ChatGPTCoding, r/LocalLLaMA`. + +**Worked example — the failing query.** Topic: `Prompting GPT Image 2`. + +Before (the 2026-04-22 failure mode): +``` +Resolved: +- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/artificial +``` + +After (with category-peer expansion): +``` +Resolved: +- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt (+ ai_image_generation peers) +``` + +The parenthetical `(+ ai_image_generation peers)` is the observable contract of the new Resolved block format. See Step 0.55 self-check below. + **3. TikTok hashtags + creators** - **INFER these from your topic knowledge. Do NOT WebSearch for "{PERSON} TikTok account" - most people/CEOs don't have TikTok, and the search is wasted.** - **Hashtags:** Infer 2-3 from the topic name + category. Examples: "Kanye West" → `kanyewest,ye,bully`. "Claude Code" → `claudecode,aiagent,aicoding`. "Sam Altman" → `samaltman,openai,chatgpt`. @@ -658,17 +695,19 @@ Passing the resolved block visibly (per-entity, all 4 types each) is the observa **If you can't infer targeting for a platform, skip that flag -- the Python engine will fall back to keyword search.** +**Step 0.55 self-check: category-peer coverage.** Before emitting the Resolved block, re-read your resolved subreddit list. Does the topic match any category in the Section 2a table (or fit the spirit of one — AI image gen, AI coding, AI music, etc.)? If YES: does your list include AT LEAST 2 peer subs from that category? If NO, widen the list NOW — do not run the engine yet. The observable contract is the `(+ {category_id} peers)` annotation on the Reddit line in the Resolved block. Its absence on a product-in-a-known-category topic is a Step 0.55 regression — the named 2026-04-22 failure mode. Person topics, music artists, news stories, and topics outside any category are exempt; omit the annotation. + **After resolving all handles and communities, display what you found before moving on.** This shows the user that intelligent pre-research happened: ``` Resolved: - X: @{HANDLE} (+ @{COMPANY}, @{COMMENTATOR}) -- Reddit: r/{sub1}, r/{sub2}, r/{sub3} +- Reddit: r/{sub1}, r/{sub2}, r/{sub3}, r/{peer1}, r/{peer2} (+ {category_id} peers) - TikTok: #{hashtag1}, #{hashtag2} - YouTube: {query1}, {query2} ``` -Only show lines for platforms where something was resolved. Skip empty lines. This display replaces the old "Parsed intent" block with something more useful. +Only show lines for platforms where something was resolved. Skip empty lines. On the Reddit line, the trailing `(+ {category_id} peers)` annotation appears when Step 0.55 Section 2a added category-peer subs. Omit the annotation when the topic had no matching category. This display replaces the old "Parsed intent" block with something more useful. --- diff --git a/scripts/lib/categories.py b/scripts/lib/categories.py new file mode 100644 index 0000000..f21f07c --- /dev/null +++ b/scripts/lib/categories.py @@ -0,0 +1,283 @@ +"""Category-peer subreddit map for Step 0.55 community resolution. + +When a topic is a product in a known category (AI image generation, AI coding +agents, SaaS screen recording, etc.), brand-specific subreddits returned by +WebSearch are insufficient: cross-product technique discussion lives in +category-peer subs. This module classifies a topic into a category by matching +compound-term patterns against the lowercased topic string, then returns the +priority-ordered peer subreddit list for that category. + +The map is intentionally small, curated, and code-reviewed. Adding a new +category is a code change; there is no user-editable override surface. + +False-positive guard: every pattern is either a multi-word compound (e.g. +"image generation", "text to image") or a domain-specific single word +(e.g. "midjourney", "stablediffusion"). Bare common nouns like "image", +"ai", or "model" are never used as patterns. + +First-match-wins: categories are evaluated in declared order. Entries are +sorted from most-specific to least-specific so narrower categories claim a +topic before broader ones. For example, `ai_image_generation` appears +before `ai_chat_model` so "gpt image 2" matches the image-gen category. +""" + +from __future__ import annotations + +from typing import List, Optional, TypedDict + + +class _CategoryEntry(TypedDict): + patterns: List[str] + peer_subs: List[str] + + +CATEGORY_PEERS: dict[str, _CategoryEntry] = { + "ai_image_generation": { + "patterns": [ + "image generation", + "image gen", + "text to image", + "text-to-image", + "gpt image", + "gpt-image", + "nano banana", + "midjourney", + "stable diffusion", + "stablediffusion", + "dall-e", + "dalle", + "flux.1", + "flux schnell", + "imagen", + "seedance", + "ideogram", + "recraft", + ], + "peer_subs": [ + "StableDiffusion", + "midjourney", + "dalle2", + "aiArt", + "PromptEngineering", + "MediaSynthesis", + ], + }, + "ai_video_generation": { + "patterns": [ + "video generation", + "text to video", + "text-to-video", + "sora", + "veo 3", + "veo3", + "runway gen", + "kling", + "pika labs", + "luma dream machine", + "hailuo", + ], + "peer_subs": [ + "aivideo", + "StableDiffusion", + "runwayml", + "singularity", + "MediaSynthesis", + ], + }, + "ai_music_generation": { + "patterns": [ + "music generation", + "ai music", + "suno", + "udio", + "riffusion", + "stable audio", + ], + "peer_subs": [ + "SunoAI", + "udiomusic", + "aimusic", + "artificial", + ], + }, + "ai_coding_agent": { + "patterns": [ + "claude code", + "cursor ide", + "github copilot", + "windsurf", + "aider", + "cline", + "openclaw", + "hermes agent", + "continue.dev", + "codeium", + "sweep ai", + "devin ai", + "coding agent", + "coding assistant", + ], + "peer_subs": [ + "ChatGPTCoding", + "LocalLLaMA", + "singularity", + "PromptEngineering", + ], + }, + "ai_agent_framework": { + "patterns": [ + "agent framework", + "agentic framework", + "langchain", + "langgraph", + "crewai", + "autogen", + "llamaindex", + "dspy", + "smolagents", + ], + "peer_subs": [ + "LangChain", + "LocalLLaMA", + "AI_Agents", + "MachineLearning", + ], + }, + "ai_chat_model": { + "patterns": [ + "gpt-5", + "gpt-4", + "claude opus", + "claude sonnet", + "claude haiku", + "gemini pro", + "gemini flash", + "llama 3", + "llama 4", + "deepseek", + "qwen", + "mistral large", + "grok", + ], + "peer_subs": [ + "LocalLLaMA", + "ChatGPT", + "ClaudeAI", + "singularity", + "artificial", + ], + }, + "saas_screen_recording": { + "patterns": [ + "screen recording", + "screen recorder", + "loom video", + "tella screen", + "vidyard", + "screen capture tool", + ], + "peer_subs": [ + "SaaS", + "screenrecording", + "productivity", + "Entrepreneur", + ], + }, + "saas_productivity": { + "patterns": [ + "notion app", + "obsidian plugin", + "obsidian app", + "linear app", + "asana", + "clickup", + "productivity app", + ], + "peer_subs": [ + "productivity", + "SaaS", + "ObsidianMD", + "Notion", + ], + }, + "prediction_markets": { + "patterns": [ + "polymarket", + "kalshi", + "prediction market", + "event contracts", + "manifold markets", + ], + "peer_subs": [ + "Polymarket", + "Kalshi", + "predictionmarkets", + ], + }, + "crypto_defi": { + "patterns": [ + "defi protocol", + "yield farming", + "liquidity pool", + "stablecoin", + "ethereum layer", + "layer 2", + "l2 rollup", + ], + "peer_subs": [ + "defi", + "ethfinance", + "CryptoCurrency", + "ethereum", + ], + }, + "dev_tool_cli": { + "patterns": [ + "cli tool", + "command line tool", + "terminal app", + "dev tool", + ], + "peer_subs": [ + "commandline", + "programming", + "webdev", + ], + }, +} + + +def detect_category(topic: Optional[str]) -> Optional[str]: + """Classify a topic into a known category by compound-term match. + + Returns the category id (e.g. "ai_image_generation") or None if no + category's patterns match. Matching is case-insensitive substring over + the lowercased topic. Declaration order wins (first-match-wins), so the + map is ordered from most-specific to least-specific. + + A None or empty topic returns None. Classification never raises on + normal string inputs; callers do not need to wrap in try/except for + typical paths, though defensive callers may. + """ + if not topic: + return None + lowered = topic.lower() + for category_id, entry in CATEGORY_PEERS.items(): + for pattern in entry["patterns"]: + if pattern in lowered: + return category_id + return None + + +def peer_subs_for(category_id: Optional[str]) -> List[str]: + """Return the priority-ordered peer subreddit list for a category. + + Returns an empty list for None or unknown category ids. The returned + list is a fresh copy; callers may safely mutate it. + """ + if not category_id: + return [] + entry = CATEGORY_PEERS.get(category_id) + if not entry: + return [] + return list(entry["peer_subs"]) diff --git a/scripts/lib/resolve.py b/scripts/lib/resolve.py index 8e945ce..6bc7748 100644 --- a/scripts/lib/resolve.py +++ b/scripts/lib/resolve.py @@ -11,14 +11,64 @@ import re import sys from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime, timezone +from typing import Optional -from . import dates, grounding +from . import categories, dates, grounding + +MAX_SUBS = 10 def _log(msg: str) -> None: print(f"[Resolve] {msg}", file=sys.stderr) +def _merge_category_peers(topic: str, subreddits: list[str]) -> tuple[list[str], Optional[str]]: + """Extend the WebSearch-extracted subreddit list with category peers. + + Classifies the topic, fetches the category's peer subs, dedupes + case-insensitively against the existing list, and appends missing + peers in priority order. Caps the final list at MAX_SUBS, preserving + every WebSearch-returned sub (they are the freshest signal) and + trimming from the peer-additions end. + + Returns a tuple of (merged_subs, matched_category_id_or_None). + Emits a [Resolve] Matched category log line only when peers were + actually added (not when every peer was already in the WebSearch set). + + Classification failures degrade to "no match" — the unwidened list + is returned and a warning is logged. + """ + try: + category = categories.detect_category(topic) + except Exception as exc: + _log(f"Category classification failed: {exc}") + return list(subreddits)[:MAX_SUBS], None + + if category is None: + return list(subreddits)[:MAX_SUBS], None + + peers = categories.peer_subs_for(category) + if not peers: + return list(subreddits)[:MAX_SUBS], category + + existing_lower = {s.lower() for s in subreddits} + merged = list(subreddits) + added: list[str] = [] + for peer in peers: + if len(merged) >= MAX_SUBS: + break + if peer.lower() in existing_lower: + continue + merged.append(peer) + existing_lower.add(peer.lower()) + added.append(peer) + + if added: + _log(f"Matched category={category}, adding peers: {', '.join(added)}") + + return merged, category + + def _has_backend(config: dict) -> bool: """Check if any web search backend is available.""" return bool( @@ -134,10 +184,19 @@ def auto_resolve(topic: str, config: dict) -> dict: config: Dict with API keys (BRAVE_API_KEY, EXA_API_KEY, SERPER_API_KEY). Returns: - Dict with keys: subreddits, x_handle, context, searches_run. - Returns empty result if no web search backend is available. + Dict with keys: subreddits, x_handle, github_user, github_repos, + context, category, searches_run. Returns empty result if no web + search backend is available. """ - empty = {"subreddits": [], "x_handle": "", "context": "", "searches_run": 0} + empty = { + "subreddits": [], + "x_handle": "", + "github_user": "", + "github_repos": [], + "context": "", + "category": None, + "searches_run": 0, + } if not _has_backend(config): _log("No web search backend available, skipping resolve") @@ -184,7 +243,9 @@ def auto_resolve(topic: str, config: dict) -> dict: github_repos = _extract_github_repos(results.get("github", [])) context = _build_context_summary(results.get("news", [])) - _log(f"Resolved {len(subreddits)} subreddits, x_handle={x_handle!r}, github_user={github_user!r}, github_repos={github_repos!r}, context_len={len(context)}") + subreddits, category = _merge_category_peers(topic, subreddits) + + _log(f"Resolved {len(subreddits)} subreddits, x_handle={x_handle!r}, github_user={github_user!r}, github_repos={github_repos!r}, context_len={len(context)}, category={category!r}") return { "subreddits": subreddits, @@ -192,5 +253,6 @@ def auto_resolve(topic: str, config: dict) -> dict: "github_user": github_user, "github_repos": github_repos, "context": context, + "category": category, "searches_run": searches_run, } diff --git a/tests/fixtures/prompting-gpt-image-2-resolved-block.md b/tests/fixtures/prompting-gpt-image-2-resolved-block.md new file mode 100644 index 0000000..8aaea27 --- /dev/null +++ b/tests/fixtures/prompting-gpt-image-2-resolved-block.md @@ -0,0 +1,60 @@ +# Fixture: `Prompting GPT Image 2` Resolved-block regression + +Documentation-grade fixture. Captures the pre-fix and post-fix shape of the +Step 0.55 Resolved block for the topic `Prompting GPT Image 2`. Not parsed +by test code — read by reviewers when evaluating regressions in +`scripts/lib/categories.py` or the SKILL.md Step 0.55 block. + +The live assertion lives in `tests/test_category_integration.py`. This +markdown fixture exists so reviewers can eyeball expected behavior without +running pytest. + +## Failing run (2026-04-22, pre-fix) + +User ran `/last30days Prompting GPT Image 2`. Step 0.55 WebSearch returned +OpenAI-brand communities. The model resolved exactly those. + +``` +Resolved: +- X: @OpenAI (+ @sama, @openaidevs) +- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/artificial, r/ChatGPTpromptengineering +- TikTok: #gptimage2, #openai, #aiart +``` + +Engine run returned thin results. User manually intervened with "make sure +to check image generatorion reddits too" and re-ran with the image-gen +peer subs added. + +## Expected run (post-fix, no user intervention) + +After Step 0.55 Section 2a (category-peer expansion) and Unit 2's engine-side +merge in `auto_resolve`, the same topic produces: + +``` +Resolved: +- X: @OpenAI (+ @sama, @openaidevs) +- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt (+ ai_image_generation peers) +- TikTok: #gptimage2, #openai, #aiart +``` + +The peer subs (`StableDiffusion, midjourney, dalle2, aiArt`) appear alongside +the WebSearch-returned brand subs. The `(+ ai_image_generation peers)` +annotation is the observable contract — its absence on a product-in-a-known- +category topic is a Step 0.55 regression. + +## Guards + +- `tests/test_categories.py::DetectCategoryHappyPath::test_prompting_gpt_image_2_matches_image_generation` +- `tests/test_resolve.py::MergeCategoryPeersHappyPath::test_image_gen_topic_appends_peers` +- `tests/test_resolve.py::AutoResolveCategoryIntegration::test_auto_resolve_returns_category_key` +- `tests/test_category_integration.py` — end-to-end over `auto_resolve` with + a stubbed WebSearch that mimics the original failing response. + +## When to update this fixture + +- Category map changed (a peer sub was reordered, added, or removed). +- The observable Resolved-block annotation format changed. +- A new category was added that affects this topic. + +Do not update casually. This file is the pre/post record of the 2026-04-22 +failure. diff --git a/tests/test_categories.py b/tests/test_categories.py new file mode 100644 index 0000000..848abaa --- /dev/null +++ b/tests/test_categories.py @@ -0,0 +1,154 @@ +"""Unit tests for scripts/lib/categories.py — the Step 0.55 category-peer map. + +Guards the 2026-04-22 `Prompting GPT Image 2` failure mode: the original bug +was that Step 0.55 resolved only brand-adjacent subs (r/OpenAI, r/ChatGPT) +and missed the category peers (r/StableDiffusion, r/midjourney, r/dalle2) +where prompting techniques actually live. +""" + +import re +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) + +from lib import categories +from lib.categories import CATEGORY_PEERS, detect_category, peer_subs_for + + +class DetectCategoryHappyPath(unittest.TestCase): + def test_prompting_gpt_image_2_matches_image_generation(self): + self.assertEqual( + detect_category("Prompting GPT Image 2"), + "ai_image_generation", + ) + + def test_claude_code_matches_coding_agent(self): + self.assertEqual( + detect_category("Claude Code skills"), + "ai_coding_agent", + ) + + def test_suno_matches_music_generation(self): + self.assertEqual(detect_category("Suno v4 review"), "ai_music_generation") + + def test_polymarket_matches_prediction_markets(self): + self.assertEqual( + detect_category("Polymarket election odds"), + "prediction_markets", + ) + + def test_sora_matches_video_generation(self): + self.assertEqual(detect_category("Sora 2 prompts"), "ai_video_generation") + + +class PeerSubsForHappyPath(unittest.TestCase): + def test_image_generation_peer_subs_priority_order(self): + subs = peer_subs_for("ai_image_generation") + self.assertIn("StableDiffusion", subs) + self.assertIn("midjourney", subs) + self.assertIn("dalle2", subs) + self.assertLess(subs.index("StableDiffusion"), subs.index("midjourney")) + self.assertLess(subs.index("midjourney"), subs.index("dalle2")) + + def test_unknown_category_returns_empty_list(self): + self.assertEqual(peer_subs_for("unknown_category"), []) + + def test_none_category_returns_empty_list(self): + self.assertEqual(peer_subs_for(None), []) + + def test_returned_list_is_fresh_copy(self): + first = peer_subs_for("ai_image_generation") + first.append("MutatedSub") + second = peer_subs_for("ai_image_generation") + self.assertNotIn("MutatedSub", second) + + +class DetectCategoryEdgeCases(unittest.TestCase): + def test_case_insensitive_match(self): + self.assertEqual( + detect_category("STABLE DIFFUSION walkthrough"), + "ai_image_generation", + ) + + def test_non_category_topic_returns_none(self): + self.assertIsNone(detect_category("Kanye West")) + + def test_bare_image_word_does_not_trigger_image_generation(self): + # Compound-term guard: "image" alone is not a pattern; only + # multi-word compounds or domain-specific brand names match. + self.assertIsNone(detect_category("image editing on my phone")) + + def test_bare_ai_word_does_not_trigger_any_category(self): + self.assertIsNone(detect_category("ai news today")) + + def test_empty_topic_returns_none(self): + self.assertIsNone(detect_category("")) + + def test_none_topic_returns_none(self): + self.assertIsNone(detect_category(None)) + + def test_first_match_wins_image_gen_before_chat_model(self): + # "gpt image 2" contains "gpt image" (ai_image_generation) and the + # substring "gpt" could resemble gpt-N chat-model patterns. The + # narrower category wins because it is declared earlier. + self.assertEqual( + detect_category("gpt image 2 review"), + "ai_image_generation", + ) + + +class CategoryMapInvariants(unittest.TestCase): + """Regression guards on the map itself — catch accidental bare-word patterns.""" + + # Common nouns that would produce false positives if used as bare patterns. + FORBIDDEN_BARE_PATTERNS = frozenset({ + "image", "video", "music", "ai", "model", "agent", "chat", + "code", "cli", "app", "tool", "defi", + }) + + def test_no_category_has_a_bare_common_noun_pattern(self): + offenders = [] + for category_id, entry in CATEGORY_PEERS.items(): + for pattern in entry["patterns"]: + if pattern.strip() in self.FORBIDDEN_BARE_PATTERNS: + offenders.append((category_id, pattern)) + self.assertEqual( + offenders, + [], + msg=( + "Bare common-noun patterns cause false positives. " + f"Offenders: {offenders}. Patterns must be compound " + "(e.g. 'image generation') or domain-specific " + "(e.g. 'midjourney')." + ), + ) + + def test_every_category_has_at_least_one_compound_or_brand_pattern(self): + multi_word_or_brand = re.compile(r"(\s|-|\.)|^[a-z][a-z0-9]{3,}$") + for category_id, entry in CATEGORY_PEERS.items(): + patterns = entry["patterns"] + self.assertTrue(patterns, f"{category_id} has no patterns") + has_strong = any(multi_word_or_brand.search(p) for p in patterns) + self.assertTrue( + has_strong, + f"{category_id} needs at least one multi-word or brand pattern", + ) + + def test_every_category_has_at_least_two_peer_subs(self): + for category_id, entry in CATEGORY_PEERS.items(): + self.assertGreaterEqual( + len(entry["peer_subs"]), + 2, + f"{category_id} should list at least 2 peer subs", + ) + + def test_category_count_is_in_expected_range(self): + # Sanity check: the map is intentionally small and curated. + self.assertGreaterEqual(len(CATEGORY_PEERS), 8) + self.assertLessEqual(len(CATEGORY_PEERS), 20) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_category_integration.py b/tests/test_category_integration.py new file mode 100644 index 0000000..513c193 --- /dev/null +++ b/tests/test_category_integration.py @@ -0,0 +1,145 @@ +"""End-to-end regression test for the 2026-04-22 `Prompting GPT Image 2` bug. + +Guards the failing run's Resolved-block shape end-to-end: stubs +`grounding.web_search` to return the OpenAI-only subs that caused the +original failure, then asserts that `auto_resolve` now returns the widened +list and emits the expected stderr trace. + +If this test starts failing after a `scripts/lib/categories.py` edit, either +the fix regressed or the map intentionally dropped the `ai_image_generation` +category — update the test deliberately. + +Fixture reference: `tests/fixtures/prompting-gpt-image-2-resolved-block.md`. +""" + +import io +import sys +import unittest +from contextlib import redirect_stderr +from pathlib import Path +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) + +from lib import resolve + + +OPENAI_BRAND_SUBREDDIT_RESULTS = [ + { + "title": "r/OpenAI community hub", + "snippet": "Discussion at r/ChatGPT and r/singularity about GPT Image 2.", + "url": "https://reddit.com/r/OpenAI/", + }, + { + "title": "r/ChatGPTpromptengineering prompt collection", + "snippet": "Also see r/artificial for broader AI chatter.", + "url": "", + }, +] + +EMPTY_RESULTS: list[dict] = [] + + +def _fake_websearch(label_to_items: dict[str, list[dict]]): + def _search(query, date_range, config): + if "subreddit" in query: + return label_to_items.get("subreddit", EMPTY_RESULTS), {} + if "news" in query: + return label_to_items.get("news", EMPTY_RESULTS), {} + if "handle" in query: + return label_to_items.get("x_handle", EMPTY_RESULTS), {} + if "github" in query: + return label_to_items.get("github", EMPTY_RESULTS), {} + return EMPTY_RESULTS, {} + + return _search + + +class PromptingGptImage2RegressionGuard(unittest.TestCase): + """The named 2026-04-22 failure mode. Resolved block must include peers.""" + + @patch("lib.resolve.grounding.web_search") + def test_auto_resolve_widens_to_image_gen_peers(self, mock_search): + mock_search.side_effect = _fake_websearch({ + "subreddit": OPENAI_BRAND_SUBREDDIT_RESULTS, + }) + + result = resolve.auto_resolve( + "Prompting GPT Image 2", + {"BRAVE_API_KEY": "fake"}, + ) + + subs_lower = [s.lower() for s in result["subreddits"]] + + # Original WebSearch-returned brand subs preserved + self.assertIn("openai", subs_lower) + self.assertIn("chatgpt", subs_lower) + self.assertIn("singularity", subs_lower) + + # At least three of the image-gen peers were added + expected_peers = {"stablediffusion", "midjourney", "dalle2", "aiart", "promptengineering"} + found_peers = expected_peers.intersection(subs_lower) + self.assertGreaterEqual( + len(found_peers), + 3, + f"Expected at least 3 image-gen peer subs, found: {found_peers}. " + f"Actual subs: {result['subreddits']}", + ) + + self.assertEqual(result["category"], "ai_image_generation") + + @patch("lib.resolve.grounding.web_search") + def test_stderr_contains_category_match_log_line(self, mock_search): + mock_search.side_effect = _fake_websearch({ + "subreddit": OPENAI_BRAND_SUBREDDIT_RESULTS, + }) + + buf = io.StringIO() + with redirect_stderr(buf): + resolve.auto_resolve( + "Prompting GPT Image 2", + {"BRAVE_API_KEY": "fake"}, + ) + + self.assertIn("Matched category=ai_image_generation", buf.getvalue()) + + @patch("lib.resolve.grounding.web_search") + def test_cap_enforced_end_to_end(self, mock_search): + # Synthesize a subreddit response with 9 brand subs + many_subs_items = [ + {"title": f"r/Brand{i}", "snippet": "", "url": ""} + for i in range(9) + ] + mock_search.side_effect = _fake_websearch({ + "subreddit": many_subs_items, + }) + + result = resolve.auto_resolve( + "Prompting GPT Image 2", + {"BRAVE_API_KEY": "fake"}, + ) + + self.assertLessEqual(len(result["subreddits"]), resolve.MAX_SUBS) + # The first WebSearch sub is still present (brand subs never evicted) + self.assertIn("Brand0", result["subreddits"]) + + @patch("lib.resolve.grounding.web_search") + def test_uncategorized_topic_does_not_inject_peers(self, mock_search): + mock_search.side_effect = _fake_websearch({ + "subreddit": [{"title": "r/Kanye is wild", "snippet": "", "url": ""}], + }) + + buf = io.StringIO() + with redirect_stderr(buf): + result = resolve.auto_resolve( + "Kanye West latest album", + {"BRAVE_API_KEY": "fake"}, + ) + + self.assertEqual(result["subreddits"], ["Kanye"]) + self.assertIsNone(result["category"]) + self.assertNotIn("Matched category=", buf.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_resolve.py b/tests/test_resolve.py index 7690263..f497fde 100644 --- a/tests/test_resolve.py +++ b/tests/test_resolve.py @@ -1,11 +1,14 @@ +import io import sys import unittest +from contextlib import redirect_stderr from pathlib import Path from unittest.mock import patch sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) from lib import resolve +from lib.resolve import MAX_SUBS, _merge_category_peers class TestHasBackend(unittest.TestCase): @@ -137,8 +140,8 @@ class TestAutoResolve(unittest.TestCase): self.assertEqual(result["subreddits"], ["technology", "gadgets"]) self.assertEqual(result["x_handle"], "techco") self.assertIn("breakthrough", result["context"]) - self.assertEqual(result["searches_run"], 3) - self.assertEqual(mock_search.call_count, 3) + self.assertEqual(result["searches_run"], 4) + self.assertEqual(mock_search.call_count, 4) @patch("lib.resolve.grounding.web_search") def test_search_failure_graceful(self, mock_search): @@ -167,8 +170,180 @@ class TestAutoResolve(unittest.TestCase): self.assertEqual(result["subreddits"], ["cooking"]) # News search failed, so context is empty self.assertEqual(result["context"], "") - # 2 out of 3 succeeded - self.assertEqual(result["searches_run"], 2) + # 3 out of 4 succeeded (subreddit, x_handle, github; news failed) + self.assertEqual(result["searches_run"], 3) + + +class MergeCategoryPeersHappyPath(unittest.TestCase): + def test_image_gen_topic_appends_peers(self): + merged, category = _merge_category_peers( + "Prompting GPT Image 2", + ["OpenAI", "ChatGPT", "singularity"], + ) + self.assertEqual(category, "ai_image_generation") + self.assertIn("OpenAI", merged) + self.assertIn("ChatGPT", merged) + self.assertIn("singularity", merged) + self.assertIn("StableDiffusion", merged) + self.assertIn("midjourney", merged) + self.assertIn("dalle2", merged) + + def test_preserves_websearch_order_then_appends_peers(self): + merged, _ = _merge_category_peers( + "Prompting GPT Image 2", + ["OpenAI", "ChatGPT"], + ) + self.assertEqual(merged[0], "OpenAI") + self.assertEqual(merged[1], "ChatGPT") + self.assertEqual(merged[2], "StableDiffusion") + + def test_emits_stderr_log_when_peers_added(self): + buf = io.StringIO() + with redirect_stderr(buf): + _merge_category_peers( + "Prompting GPT Image 2", + ["OpenAI", "ChatGPT"], + ) + output = buf.getvalue() + self.assertIn("Matched category=ai_image_generation", output) + self.assertIn("StableDiffusion", output) + + +class MergeCategoryPeersDedupe(unittest.TestCase): + def test_peer_already_in_websearch_not_duplicated(self): + merged, _ = _merge_category_peers( + "midjourney v7 prompts", + ["midjourney", "aiArt"], + ) + self.assertEqual( + sum(1 for s in merged if s.lower() == "midjourney"), + 1, + ) + + def test_dedupe_is_case_insensitive(self): + merged, _ = _merge_category_peers( + "Prompting GPT Image 2", + ["STABLEDIFFUSION"], + ) + lower = [s.lower() for s in merged] + self.assertEqual(lower.count("stablediffusion"), 1) + + def test_no_log_when_all_peers_already_present(self): + buf = io.StringIO() + with redirect_stderr(buf): + _merge_category_peers( + "Prompting GPT Image 2", + [ + "StableDiffusion", + "midjourney", + "dalle2", + "aiArt", + "PromptEngineering", + "MediaSynthesis", + ], + ) + self.assertNotIn("Matched category=", buf.getvalue()) + + +class MergeCategoryPeersEdgeCases(unittest.TestCase): + def test_topic_with_no_category_returns_unchanged(self): + merged, category = _merge_category_peers( + "Kanye West", + ["Kanye", "hiphopheads"], + ) + self.assertIsNone(category) + self.assertEqual(merged, ["Kanye", "hiphopheads"]) + + def test_empty_subreddit_list_with_category_still_adds_peers(self): + merged, category = _merge_category_peers("Prompting GPT Image 2", []) + self.assertEqual(category, "ai_image_generation") + self.assertIn("StableDiffusion", merged) + + def test_empty_topic_returns_unchanged(self): + merged, category = _merge_category_peers("", ["foo", "bar"]) + self.assertIsNone(category) + self.assertEqual(merged, ["foo", "bar"]) + + def test_none_topic_returns_unchanged(self): + merged, category = _merge_category_peers(None, ["foo", "bar"]) + self.assertIsNone(category) + self.assertEqual(merged, ["foo", "bar"]) + + def test_no_log_when_topic_has_no_category(self): + buf = io.StringIO() + with redirect_stderr(buf): + _merge_category_peers("Kanye West", ["Kanye"]) + self.assertNotIn("Matched category=", buf.getvalue()) + + +class MergeCategoryPeersCap(unittest.TestCase): + def test_cap_is_enforced_at_max_subs(self): + websearch_subs = [f"Sub{i}" for i in range(9)] + merged, _ = _merge_category_peers( + "Prompting GPT Image 2", + websearch_subs, + ) + self.assertEqual(len(merged), MAX_SUBS) + for s in websearch_subs: + self.assertIn(s, merged) + self.assertEqual(len(merged) - len(websearch_subs), 1) + self.assertEqual(merged[9], "StableDiffusion") + + def test_cap_preserves_highest_priority_peer_when_trimming(self): + websearch_subs = [f"Sub{i}" for i in range(8)] + merged, _ = _merge_category_peers( + "Prompting GPT Image 2", + websearch_subs, + ) + self.assertEqual(len(merged), MAX_SUBS) + self.assertEqual(merged[8], "StableDiffusion") + self.assertEqual(merged[9], "midjourney") + + +class MergeCategoryPeersClassificationFailure(unittest.TestCase): + def test_classification_error_returns_unwidened_list_and_logs(self): + original = resolve.categories.detect_category + + def boom(_topic): + raise RuntimeError("synthetic classifier failure") + + resolve.categories.detect_category = boom + try: + buf = io.StringIO() + with redirect_stderr(buf): + merged, category = _merge_category_peers( + "Prompting GPT Image 2", + ["OpenAI"], + ) + self.assertEqual(merged, ["OpenAI"]) + self.assertIsNone(category) + self.assertIn("Category classification failed", buf.getvalue()) + finally: + resolve.categories.detect_category = original + + +class AutoResolveCategoryIntegration(unittest.TestCase): + @patch("lib.resolve.grounding.web_search") + def test_auto_resolve_returns_category_key(self, mock_search): + def side_effect(query, date_range, config): + if "subreddit" in query: + return [ + {"title": "r/OpenAI", "snippet": "r/ChatGPT r/singularity", "url": ""}, + ], {} + return [], {} + + mock_search.side_effect = side_effect + result = resolve.auto_resolve( + "Prompting GPT Image 2", + {"BRAVE_API_KEY": "fake"}, + ) + self.assertEqual(result["category"], "ai_image_generation") + self.assertIn("StableDiffusion", result["subreddits"]) + self.assertIn("OpenAI", result["subreddits"]) + + def test_no_backend_returns_category_none(self): + result = resolve.auto_resolve("test topic", {}) + self.assertIsNone(result["category"]) if __name__ == "__main__":