Matt Van Horn
4e91f4e754
* feat(resolve): category-peer subreddit map for Step 0.55 Introduces scripts/lib/categories.py with a curated category->peer-subs map and wires scripts/lib/resolve.py auto_resolve() to merge peers into the WebSearch-extracted subreddit list. Named 2026-04-22 failure mode: a "Prompting GPT Image 2" run resolved only r/OpenAI + r/ChatGPT and missed r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt where prompting techniques actually live. Map is static, curated, ~11 categories (ai_image_generation, ai_video_generation, ai_music_generation, ai_coding_agent, ai_agent_framework, ai_chat_model, saas_screen_recording, saas_productivity, prediction_markets, crypto_defi, dev_tool_cli). First-match-wins ordering from most-specific to least-specific. Compound-term patterns only (no bare common nouns like "image", "ai"). auto_resolve now: - calls detect_category(topic) after _extract_subreddits - merges peer_subs case-insensitively, caps at MAX_SUBS (10) - preserves every WebSearch-returned sub (freshest signal) - emits [Resolve] Matched category=<id>, adding peers: <list> on stderr only when peers were actually added - returns new "category" key in the result dict for observability - wraps classifier in try/except so failures degrade to unwidened list Includes drive-by: test_full_resolve / test_partial_failure searches_run expectations bumped from 3->4 / 2->3 to match the current queries dict (subreddit + news + x_handle + github). * feat(skill): Step 0.55 category-peer expansion and self-check Adds Section 2a (category-peer expansion, MANDATORY for product topics) and the Step 0.55 self-check checkpoint that fires immediately before the Resolved block displays. Structural mirror of the engine-side categories.py map: same categories, same peer subs, same priority order. The model-side path now: - Applies category-peer expansion to the WebSearch-resolved subs on every product-in-a-known-category run. - Emits the (+ <category_id> peers) annotation on the Reddit line of the Resolved block as the observable contract. Absence on a product-in-a-known-category topic is a Step 0.55 regression. - Runs a self-check before emitting Resolved: "does the resolved list include at least 2 peer subs for the matched category? if not, widen NOW and do not run the engine yet." Mirror of the Python map lives inside Step 0.55 as a table for the model to pattern-match against; extrapolation to unlisted categories is explicitly allowed. Worked example (the exact failing query) appears below the table so reviewers can see before/after at a glance. Both changes land inside the existing Step 0.55 block. No new top-level section, no new LAW. LAWs 1-6 wording unchanged. * test: end-to-end regression for GPT Image 2 failure mode Stubs grounding.web_search to return the OpenAI-only subs that caused the 2026-04-22 failure, then asserts that auto_resolve widens to include the image-gen peers and emits the [Resolve] Matched category=ai_image_generation stderr line. Covers the cap boundary and the uncategorized-topic no-op path. Fixture tests/fixtures/prompting-gpt-image-2-resolved-block.md is documentation-grade (not parsed by tests) and shows the pre-fix vs post-fix Resolved block shape so reviewers can evaluate future categories.py edits against the original bug. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
61 lines
2.4 KiB
Markdown
61 lines
2.4 KiB
Markdown
# Fixture: `Prompting GPT Image 2` Resolved-block regression
|
|
|
|
Documentation-grade fixture. Captures the pre-fix and post-fix shape of the
|
|
Step 0.55 Resolved block for the topic `Prompting GPT Image 2`. Not parsed
|
|
by test code — read by reviewers when evaluating regressions in
|
|
`scripts/lib/categories.py` or the SKILL.md Step 0.55 block.
|
|
|
|
The live assertion lives in `tests/test_category_integration.py`. This
|
|
markdown fixture exists so reviewers can eyeball expected behavior without
|
|
running pytest.
|
|
|
|
## Failing run (2026-04-22, pre-fix)
|
|
|
|
User ran `/last30days Prompting GPT Image 2`. Step 0.55 WebSearch returned
|
|
OpenAI-brand communities. The model resolved exactly those.
|
|
|
|
```
|
|
Resolved:
|
|
- X: @OpenAI (+ @sama, @openaidevs)
|
|
- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/artificial, r/ChatGPTpromptengineering
|
|
- TikTok: #gptimage2, #openai, #aiart
|
|
```
|
|
|
|
Engine run returned thin results. User manually intervened with "make sure
|
|
to check image generatorion reddits too" and re-ran with the image-gen
|
|
peer subs added.
|
|
|
|
## Expected run (post-fix, no user intervention)
|
|
|
|
After Step 0.55 Section 2a (category-peer expansion) and Unit 2's engine-side
|
|
merge in `auto_resolve`, the same topic produces:
|
|
|
|
```
|
|
Resolved:
|
|
- X: @OpenAI (+ @sama, @openaidevs)
|
|
- Reddit: r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering, r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt (+ ai_image_generation peers)
|
|
- TikTok: #gptimage2, #openai, #aiart
|
|
```
|
|
|
|
The peer subs (`StableDiffusion, midjourney, dalle2, aiArt`) appear alongside
|
|
the WebSearch-returned brand subs. The `(+ ai_image_generation peers)`
|
|
annotation is the observable contract — its absence on a product-in-a-known-
|
|
category topic is a Step 0.55 regression.
|
|
|
|
## Guards
|
|
|
|
- `tests/test_categories.py::DetectCategoryHappyPath::test_prompting_gpt_image_2_matches_image_generation`
|
|
- `tests/test_resolve.py::MergeCategoryPeersHappyPath::test_image_gen_topic_appends_peers`
|
|
- `tests/test_resolve.py::AutoResolveCategoryIntegration::test_auto_resolve_returns_category_key`
|
|
- `tests/test_category_integration.py` — end-to-end over `auto_resolve` with
|
|
a stubbed WebSearch that mimics the original failing response.
|
|
|
|
## When to update this fixture
|
|
|
|
- Category map changed (a peer sub was reordered, added, or removed).
|
|
- The observable Resolved-block annotation format changed.
|
|
- A new category was added that affects this topic.
|
|
|
|
Do not update casually. This file is the pre/post record of the 2026-04-22
|
|
failure.
|