Compare commits

..

11 Commits

Author SHA1 Message Date
Matt Van Horn 5864c687a3 fix: hoist inline-link citation into LAW 8
Four live test runs on 2026-04-20 (Matt Van Horn, Peter Steinberger,
Best Headphones, OpenClaw vs Hermes) confirmed PR #289's citation rule
was deployed (diff IN SYNC, grep found it) but consistently skipped on
first-pass synthesis. Agent's own root cause, repeated verbatim across
all four runs: "SKILL.md is 45K tokens and fails a single Read. I read
offsets 1-200, 200-600, 600-1000, then stopped and ran the engine. The
inline-link rule lives at line 1224 of a 1523-line file. I never
reached it."

This is the exact failure mode the VOICE CONTRACT LAW block at line 97
was created to prevent. LAWs 1-7 were hoisted in v3.0.7/3.0.8 because
the file is too long to read top-to-bottom before synthesis. The
inline-link rule in PR #289 was added at line 1224 and never joined the
LAWs, so it lives below the chunked-read window and reliably gets
skipped. Same pattern as v3.0.6 (invented titles), disaster #2 (stripped
bold), disaster #3 (trailing Sources), and the 2026-04-19 Hermes
evidence-dump disaster. Same fix pattern: add the rule to the LAWs
block with the established anatomy.

Changes:

- Add LAW 8 at line 167, inside the VOICE CONTRACT LAW block. Full
  LAW-style shape: loud one-line rule, "applies to every query type",
  mechanism sentence, plain-text fallback clause, BAD/BAD/BAD/GOOD/
  FALLBACK example set, named incident reference (2026-04-20 inline-
  links saga), post-synthesis self-check.
- Update preamble at line 101 from "These five rules" to "These LAWs"
  (stale since LAWs 6-7 were added; fixed in the same commit).
- Convert the old CITATION PRIORITY / URL FORMATTING block at line 1218
  into a short pointer to LAW 8 plus the citation-priority ordering list
  (which is a preference, not the correctness rule, so it can live
  lower). Narrative BAD/GOOD examples stay in place with a back-ref
  line: "(These narrative examples illustrate LAW 8 from the VOICE
  CONTRACT.)"
- Single source of truth preserved: rule text lives exactly once in the
  LAWs block; lower references point back.

Does not touch: LAWs 1-7, LAW numbering, deterministic engine footer,
PASS-THROUGH FOOTER boundaries, comparison scaffold, mandatory badge,
em-dash/en-dash prohibition, no-## header rule, or any other structural
contract.

Verification pending: one fresh Cmd-Q session in Ghostty, then
/last30days Matt Van Horn to confirm first-pass inline links without a
correction round.

Plan: docs/plans/2026-04-20-005-fix-hoist-citation-law-plan.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 09:28:46 -07:00
Matt Van Horn 790e5bc26a feat: inline markdown links on narrative citations
Citation rule inverted: every @handle, r/sub, publication, YouTube channel,
TikTok/Instagram creator, and Polymarket market cited in "What I learned"
and KEY PATTERNS is now an inline markdown link [name](url). URLs come
from the raw research dump. Claude Code renders [text](url) as blue
CMD-clickable text with the URL hidden.

Raw URL strings remain forbidden. Plain text is the fallback only when
the raw data has no URL for a specific source. Broken empty links
[name]() are explicitly called out as bad.

Scope:
- Updates CITATION PRIORITY to show each item as a markdown link.
- Updates URL FORMATTING rule: was "NEVER paste raw URLs", now "every
  citation is [name](url), never a raw URL string".
- Updates BAD/GOOD narrative examples to show linked @handles and r/subs.
- Updates the What-I-learned / KEY-PATTERNS template placeholders.
- Adds one sentence noting the engine-emitted stats footer (LAW 5) is
  pass-through only - agent does NOT format its links.

Does not touch: LAWs 1-7, deterministic engine footer, PASS-THROUGH
FOOTER boundaries, comparison scaffold, badge rules, em-dash/en-dash
prohibition, no-## header rule, or any other existing structural
enforcement.

Net change: +29 / -24 lines, one contiguous SKILL.md region.

Context: prior attempt (PR #286, closed) branched off a stale main and
accumulated three failed prompt-enforcement amendments on top. This
commit is a fresh start against current main, applying only the minimal
link-rule edit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 08:33:22 -07:00
Matt Van Horn 1da9c601c3 Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth
fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
2026-04-19 11:09:56 -07:00
Matt Van Horn 4388fed46a fix: rewrite 'no LLM provider' stderr to stop the capability-constraint misread
PR #285 introduced the stderr warning "No --plan and no LLM provider
configured. Using deterministic fallback..." The 2026-04-19 Run 1
agent self-debug said it read that as "I don't have a key, I can't do
LLM stuff, I have to accept fallback" - which is the exact wrong
mental model. The word "provider" referred to the engine's INTERNAL
planner credentials, but the agent parsed it as "I need credentials
to plan at all."

Rewritten to say plainly: YOU are the reasoning model hosting this
skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime);
YOU ARE the planner; you do not need an API key or credentials - you
ARE the LLM. The --plan flag exists precisely so a reasoning model
generates its own plan upstream and passes it to the engine. The
deterministic fallback is the headless/cron path only.

Runtime enumeration is explicit so agents on every supported runtime
recognize themselves - this skill ships to Claude Code, Codex, Hermes,
and ~/.agents via sync.sh.

Tests: updated test_fallback_logs_warning_when_no_provider to assert
the new language (YOU ARE the planner, runtime names present) and
assert the old misleading phrasing is absent. Renamed the companion
test for clarity.
2026-04-19 10:28:48 -07:00
Matt Van Horn a7d6ef051a fix: expand entity-grounding haystack to transcripts + top comments
PR #285's entity grounding checked only title + snippet. That missed:

- YouTube videos where the entity is mentioned in transcript but not
  in title (false demotion of on-topic content)
- Reddit posts where the entity is in top comments but not in title
  (false demotion of on-topic discussion)

And it also wasn't strong enough to reliably demote items like the
2026-04-19 Nate Herk "Managed Agents" video - which had no Hermes
anywhere - because the -25 penalty on rerank_score composed to only
-15 on final_score via the 0.60 weight, and engagement bonus partially
offset that.

Two fixes:

1. _candidate_haystack() now joins title + snippet +
   metadata[transcript_snippet] + metadata[transcript_highlights] +
   metadata[top_comments][*].excerpt/text + metadata[comment_insights].
   Catches entity mentions wherever they actually live. Guarded with
   isinstance checks so malformed metadata doesn't raise.
2. ENTITY_MISS_FINAL_PENALTY (20.0) applied directly in _final_score
   when candidate.explanation contains "entity-miss". This lands the
   full penalty weight on the composite signal that cluster-scoring
   consumes, instead of being diluted by the rerank_score weight.
   Combined effect: entity-miss gap grows from ~15 to ~35 points.

Tests: 8 new scenarios covering transcript match, transcript highlight
match, top-comment match, comment-insight match, empty-text skip,
no-primary-entity no-op, and the dual-penalty composition check.
2026-04-19 10:28:36 -07:00
Matt Van Horn b7df5ecd2d fix: emit user-visible DEGRADED RUN WARNING on bare named-entity calls
The stderr [Planner] warning from PR #285 doesn't reach the user because
Claude and other reasoning agents hide stderr from their synthesis. The
2026-04-19 Hermes Agent Use Cases Run 1 produced source=deterministic
and the user never saw it.

Adds a user-visible stdout block that the model's LAW 5 pass-through
contract forces into the response. Fires only when plan_source is
deterministic AND no pre-research flags were passed AND the topic is
pre-research-eligible (named entity). Cron jobs on abstract topics
don't trigger it.

Position: BEFORE the EVIDENCE FOR SYNTHESIS envelope so the model sees
it as the first non-badge content. Wrapped in a new USER-VISIBLE BANNER
envelope matching the EVIDENCE/PASS-THROUGH envelope pattern from Unit 1
of PR #285.

Runtime-agnostic language: explicitly enumerates Claude Code, Codex,
Hermes, Gemini so the hosting reasoning model recognizes itself
regardless of runtime.

pipeline.py now persists plan_source to report.artifacts so the
renderer can consume it. Adds 7 tests covering fire conditions,
suppression conditions (external/llm plan source, flags present,
abstract topic), and correct position relative to the evidence envelope.
2026-04-19 10:28:21 -07:00
Matt Van Horn a0d61b0dc6 fix: add LAW 7 - YOU ARE the planner, --plan mandatory on named entities
Run 1 of /last30days Hermes Agent use cases on 2026-04-19 called the engine
bare despite SKILL.md already having a detailed Step 0.75 (YOU are the
planner) and a PRECONDITION GATE requiring --plan. Those lived at lines
647 and 729 - the model didn't reach them before invoking Bash.

LAW 7 hoists the rule into the OUTPUT CONTRACT block at the top (same
placement pattern as LAW 6), so it is the first thing the model reads.
Runtime-agnostic language: Claude Code, Codex, Hermes, Gemini, or any
agent runtime. Named failure mode with the misread diagnosis: "provider"
in engine messages refers to the engine's INTERNAL planner credentials,
NOT a prerequisite the caller needs - if you are the hosting reasoning
model, YOU are the provider.

Concrete self-check: re-read pending Bash command; if no --plan and topic
is a named entity, STOP and generate a plan.
2026-04-19 10:28:08 -07:00
Matt Van Horn 5f218aaac5 fix: always log planner subqueries to stderr
The prior pipeline.py only logged the planner outcome when an external
--plan was passed ("[Planner] Using external plan (N subqueries)").
The internal LLM planner and the deterministic fallback ran silently,
so retrieval-breadth failures were invisible without --debug.

After plan finalization, emit a unified trace:

  [Planner] Plan: intent=X, freshness=Y, cluster_mode=Z, subqueries=N, source=external|llm|deterministic
  [Planner]   sq1 label=... search="..." sources=[...]
  [Planner]   sq2 ...

Stderr only; does not touch the user-facing stdout synthesis. The
source= annotation distinguishes --plan (external), provider-backed
(llm), and deterministic paths — so when the 2026-04-19 Hermes Agent
Use Cases failure mode recurs, the trace tells the user which path ran
and what subqueries it produced.

Tests: added test_planner_trace_always_fires_on_mock_run which captures
stderr on a mock pipeline run and asserts the summary + per-subquery
lines appear.
2026-04-19 09:24:52 -07:00
Matt Van Horn a709d66e2a fix: demote reranker candidates that miss the primary entity
The 2026-04-19 Hermes Agent Use Cases run had a Nate Herk YouTube video
titled "I Tested Claude's New Managed Agents" score 51 and rank #2
with zero Hermes content. The reranker had intent-specific scoring hints
but no entity-grounding check, so topic-vicinity matches (one offhand
OpenClaw mention) drifted to the top.

Add _primary_entity(topic) that strips intent-modifier suffixes ("use
cases", "workflows", etc.) so "Hermes Agent use cases" yields
primary_entity="Hermes Agent". Pass the entity through to both the LLM
and fallback scoring paths.

Fallback path: if primary_entity is not found (case-insensitive) in
title + snippet, subtract ENTITY_MISS_PENALTY (25 pts). Skip the
demotion for candidates with no text at all (image-only TikToks etc.)
to avoid false negatives on thin-text sources.

LLM path: add a "Primary entity grounding" hint to _build_prompt when
primary_entity is non-empty. Instructs the LLM to score candidates
without the entity at <=30.

Tests: 24 rerank tests pass, including 8 new entity-grounding tests.
2026-04-19 09:24:43 -07:00
Matt Van Horn 4d9f29d2ed fix: broaden planner retrieval and fix deterministic fallback defaults
Topics with suffixes like "use cases", "workflows", "review",
"examples" were previously echoed near-verbatim into search_query,
returning near-zero matches because nobody posts the literal phrase
(2026-04-19 Hermes Agent Use Cases failure).

Unit 2 — planner breadth:

1. Planner prompt rule: STRIP intent-modifier phrases from search_query
   (keep them in ranking_query). Paraphrase across 4-5 subqueries that
   each express the intent differently.
2. Planner prompt rule: quote only multi-word proper nouns like
   "Hermes Agent", not the user's full topic.
3. Raise _max_subqueries cap from 3 to 5 for how_to / opinion / product /
   breaking_news / prediction. Comparison stays at 4; factual / concept
   stay at 2 unless the topic carries an intent modifier.
4. Deterministic fallback: when intent is non-{comparison,prediction}
   and topic contains an intent modifier, append 3 paraphrased
   subqueries (workflows, production, experience).

Unit 3 — deterministic fallback defaults:

5. _infer_intent default changed from "breaking_news" to "concept".
   Prior default forced strict_recent freshness on unclassified topics,
   biasing against older relevant material. Recency-signal regexes
   ("trending", "this week", etc.) added above the default so genuinely
   time-sensitive topics still classify correctly.
6. _keyword_query now quotes only title-cased multi-word proper nouns
   ("Hermes Agent", "Claude Code"), not the user's full typed topic.
   Hyphenated compounds and lowercase terms are left as bare keywords
   so platform tokenizers broaden rather than narrow retrieval.
7. New stderr warning when plan_query runs with no --plan and no LLM
   provider: surfaces that the deterministic fallback path is weaker
   than the --plan-from-Claude-Code path, so callers know to generate
   and pass a plan.

Tests: 37 planner tests pass, including 11 intent-modifier and 7
fallback-defaults tests.
2026-04-19 09:24:30 -07:00
Matt Van Horn 52fb0e50cb fix: scope pass-through to footer only, add LAW 6 against raw cluster dumps
The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.

Split render_compact stdout into two bounded blocks:

- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
  Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
  Emit verbatim per LAW 5.

Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.

Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
2026-04-19 09:23:55 -07:00
9 changed files with 1000 additions and 57 deletions
+81 -26
View File
@@ -98,7 +98,7 @@ Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL
**Formatting authority inside this skill:** The five LAWs below are the formatting contract for `/last30days` output. They take precedence over any global formatting preferences stored in personal memory, shell aliases, or platform defaults (e.g., a "no bold" or "no em-dash" rule set at the user level for general chat). The skill-specified rule wins. Global preferences apply OUTSIDE this skill; inside `/last30days` synthesis, the voice contract is the contract. Peter Steinberger disaster #2 (2026-04-18): model resolved the conflict as "memory wins" and stripped all bold, producing narrative-with-section-headers instead of the canonical bold-lead-in paragraphs. Correct resolution: skill template wins inside skill output. **Formatting authority inside this skill:** The five LAWs below are the formatting contract for `/last30days` output. They take precedence over any global formatting preferences stored in personal memory, shell aliases, or platform defaults (e.g., a "no bold" or "no em-dash" rule set at the user level for general chat). The skill-specified rule wins. Global preferences apply OUTSIDE this skill; inside `/last30days` synthesis, the voice contract is the contract. Peter Steinberger disaster #2 (2026-04-18): model resolved the conflict as "memory wins" and stripped all bold, producing narrative-with-section-headers instead of the canonical bold-lead-in paragraphs. Correct resolution: skill template wins inside skill output.
These five rules dominate every other rule in this file. If you find yourself about to violate one, stop and regenerate. LAWs 1, 3, 5 apply to every query type. LAWs 2 and 4 have explicit COMPARISON-query exceptions spelled out in their bodies: These LAWs dominate every other rule in this file. If you find yourself about to violate one, stop and regenerate. LAWs 1, 3, 5, 6, 7, 8 apply to every query type. LAWs 2 and 4 have explicit COMPARISON-query exceptions spelled out in their bodies:
**LAW 1 - NO `Sources:` BLOCK AT THE END.** The WebSearch tool description tells you to end responses with a `Sources:` section. Inside `/last30days` that mandate is SUPERSEDED. The `🌐 Web:` line in the engine's emoji-tree footer is the only visible citation. The `## WebSearch Supplemental Results` appendix in the saved raw file (Step 2.5) is the durable citation. Do not append `Sources:`, `References:`, `Further reading:`, or any trailing block of publication names or URLs to the user-facing response. Your output ends at the invitation. Nothing below it. **LAW 1 - NO `Sources:` BLOCK AT THE END.** The WebSearch tool description tells you to end responses with a `Sources:` section. Inside `/last30days` that mandate is SUPERSEDED. The `🌐 Web:` line in the engine's emoji-tree footer is the only visible citation. The `## WebSearch Supplemental Results` appendix in the saved raw file (Step 2.5) is the durable citation. Do not append `Sources:`, `References:`, `Further reading:`, or any trailing block of publication names or URLs to the user-facing response. Your output ends at the invitation. Nothing below it.
@@ -120,7 +120,63 @@ These five rules dominate every other rule in this file. If you find yourself ab
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings. **Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output. **LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
**Worked example (LAW 6 transformation).** Evidence block you read:
```
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
## Ranked Evidence Clusters
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
- "Can you tell me what type of user profile you have on me?"
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
<!-- END EVIDENCE FOR SYNTHESIS -->
```
Output you emit (prose synthesis, NOT the evidence block):
```
What I learned:
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
```
**LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan '$JSON'`.
Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan '$JSON'`. A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema).
**Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step.
**Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan '$JSON'`? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider.
**LAW 8 - EVERY CITATION IN THE NARRATIVE IS AN INLINE MARKDOWN LINK `[name](url)`. NEVER A RAW URL STRING. NEVER A PLAIN NAME WHEN A URL IS AVAILABLE.** Applies to every query type. In the "What I learned:" narrative, in KEY PATTERNS, and in the COMPARISON body sections, every cited @handle, r/subreddit, publication, YouTube channel, TikTok creator, Instagram creator, and Polymarket market is wrapped as `[name](url)` at first mention. The URL comes from the raw research dump — every engine item carries a URL; WebSearch supplements carry URLs in their own output. Claude Code renders `[text](url)` as blue CMD-clickable text; the URL is hidden in the rendering, only the link text shows. The stats footer (emoji-tree block) is engine-emitted per LAW 5 and passes through verbatim — do NOT reformat its links yourself.
**Plain-text fallback:** if the raw data genuinely has no URL for a specific source, fall back to plain text for that one citation only. Never emit a broken empty link like `[Rolling Stone]()` or `[@handle]()`. Default assumption: URL exists; plain text is the exception.
**BAD (raw URL):** `per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/`
**BAD (plain name when URL is available):** `per Rolling Stone`, `per @honest30bgfan_`, `r/hiphopheads`
**BAD (broken empty link):** `per [Rolling Stone]()`
**GOOD:** `per [Rolling Stone](https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/)`, `per [@honest30bgfan_](https://x.com/honest30bgfan_)`, `[r/hiphopheads](https://reddit.com/r/hiphopheads)`
**FALLBACK (URL genuinely missing):** `per Rolling Stone`
**Observed LAW 8 need (2026-04-20 inline-links saga):** the citation rule existed in SKILL.md but was placed in the CITATION PRIORITY block around line 1224 - below the chunked-read window. Four consecutive test runs (Matt Van Horn, Peter Steinberger, Best Headphones, OpenClaw vs Hermes) confirmed the rule was deployed (diff IN SYNC, grep found the text) but was skipped on every synthesis because the model read lines 1-1000 and stopped. The model's own self-diagnosis, repeated verbatim four times: "I never reached line 1224." LAW 8 hoists the rule into the same guaranteed-loaded band as LAWs 1-7 so it enters context on every run. Same pattern that solved v3.0.6 (invented titles), disaster #2 (stripped bold), disaster #3 (trailing Sources), and the Hermes 2026-04-19 evidence-dump disaster.
**Post-synthesis self-check (do this BEFORE emitting your response):** scan your drafted "What I learned:" and KEY PATTERNS for the `[name](url)` pattern. Count how many inline markdown links appear. If zero - and the raw dump has URLs for the @handles, r/subs, and publications you cited as plain text - regenerate ONCE with inline links added. Stripping links is not a valid way to satisfy any other LAW; LAWs 1 (no trailing Sources) and 8 (inline links required) are complementary, not alternatives.
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail. End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
@@ -1179,30 +1235,27 @@ CITATION RULE: Cite sources sparingly to prove research is real.
- Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box - Do NOT include engagement metrics in citations (likes, upvotes) - save those for stats box
- Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one. - Do NOT chain multiple citations: "per @x, @y, @z" is too much. Pick the strongest one.
CITATION PRIORITY (most to least preferred): **URL formatting is governed by LAW 8** in the VOICE CONTRACT block above. Every citation in the narrative body is an inline markdown link `[name](url)`; raw URL strings are forbidden; plain-text fallback only when the raw data has no URL for that specific source. Re-read LAW 8 now if you skipped it. The stats footer is engine-emitted per LAW 5 and passes through verbatim.
1. @handles from X - "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit - "per r/subreddit" (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title) CITATION PRIORITY (most to least preferred), with each example showing the LAW 8 inline-link shape:
3. YouTube channels - "per [channel name] on YouTube" (transcript-backed insights) 1. @handles from X - `per [@handle](https://x.com/handle)` (these prove the tool's unique value)
4. TikTok creators - "per @creator on TikTok" (viral/trending signal) 2. r/subreddits from Reddit - `per [r/subreddit](https://reddit.com/r/subreddit)` (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title)
5. Instagram creators - "per @creator on Instagram" (influencer/creator signal) 3. YouTube channels - `per [channel name](https://youtube.com/@channel) on YouTube` (transcript-backed insights)
6. HN discussions - "per HN" or "per hn/username" (developer community signal) 4. TikTok creators - `per [@creator](https://tiktok.com/@creator) on TikTok` (viral/trending signal)
7. Polymarket - "Polymarket has X at Y% (up/down Z%)" with specific odds and movement 5. Instagram creators - `per [@creator](https://instagram.com/creator) on Instagram` (influencer/creator signal)
8. Web sources - ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact 6. HN discussions - `per [HN](https://news.ycombinator.com/item?id=N)` or `per [hn/username](https://news.ycombinator.com/user?id=username)` (developer community signal)
7. Polymarket - `[Polymarket](https://polymarket.com/event/...) has X at Y% (up/down Z%)` with specific odds and movement
8. Web sources - ONLY when Reddit/X/YouTube/TikTok/Instagram/HN/Polymarket don't cover that specific fact; link the publication: `per [Rolling Stone](https://rollingstone.com/...)`
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote. The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post. When both a web article and an X post cover the same fact, cite the X post.
URL FORMATTING: NEVER paste raw URLs anywhere in the output - not in synthesis, not in stats, not in sources. (These narrative examples illustrate LAW 8 from the VOICE CONTRACT.)
- **BAD:** "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- **GOOD:** "per Rolling Stone"
- **BAD stats line:** `🌐 Web: 10 pages - https://later.com/blog/..., https://buffer.com/...`
- **GOOD stats line:** `🌐 Web: 10 pages - Later, Buffer, CNN, SocialBee`
Use the publication/site name, not the URL. The user doesn't need links - they need clean, readable text.
**BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)." **BAD:** "His album is set for March 20 (per Rolling Stone; Billboard; Complex)."
**GOOD:** "His album BULLY drops March 20 - fans on X are split on the tracklist, per @honest30bgfan_" **GOOD:** "His album BULLY drops March 20 - fans on X are split on the tracklist, per [@honest30bgfan_](https://x.com/honest30bgfan_)"
**GOOD:** "Ye's apology got massive traction on r/hiphopheads" **GOOD:** "Ye's apology got massive traction on [r/hiphopheads](https://reddit.com/r/hiphopheads)"
**OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per Billboard" **OK** (web, only when Reddit/X don't have it): "The Hellwatt Festival runs July 4-18 at RCF Arena, per [Billboard](https://www.billboard.com/music/music-news/hellwatt-festival-2026-lineup-...)"
**Lead with people, not publications.** Start each topic with what Reddit/X **Lead with people, not publications.** Start each topic with what Reddit/X
users are saying/feeling, then add web context only if needed. The user came users are saying/feeling, then add web context only if needed. The user came
@@ -1221,18 +1274,20 @@ here for the conversation, not the press release.
What I learned: What I learned:
**{Headline summarizing topic 1}** - [1-2 sentences about what people are saying, per @handle or r/sub] **{Headline summarizing topic 1}** - [1-2 sentences about what people are saying, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Headline summarizing topic 2}** - [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 2}** - [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
**{Headline summarizing topic 3}** - [1-2 sentences, per @handle or r/sub] **{Headline summarizing topic 3}** - [1-2 sentences, per [@handle](https://x.com/handle) or [r/sub](https://reddit.com/r/sub)]
KEY PATTERNS from the research: KEY PATTERNS from the research:
1. [Pattern] - per @handle 1. [Pattern] - per [@handle](https://x.com/handle)
2. [Pattern] - per r/sub 2. [Pattern] - per [r/sub](https://reddit.com/r/sub)
3. [Pattern] - per @handle 3. [Pattern] - per [@handle](https://x.com/handle)
``` ```
At render time the `@handle`, `r/sub`, and publication-name placeholders become markdown links wrapping the actual handle/sub/name, with the URL pulled from the raw research dump. Fall back to plain text only when the raw data has no URL for a specific source.
Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Europe is banning him one country at a time"), not generic ("Album release", "Tour updates"). Headlines should be specific and newsy ("BULLY dropped and it's dominating", "Europe is banning him one country at a time"), not generic ("Album release", "Tour updates").
**THEN - Quality Nudge (if present in the output):** **THEN - Quality Nudge (if present in the output):**
+34 -1
View File
@@ -204,7 +204,7 @@ def run(
plan = planner._sanitize_plan( plan = planner._sanitize_plan(
external_plan, topic, available, requested_sources, depth, external_plan, topic, available, requested_sources, depth,
) )
print(f"[Planner] Using external plan ({len(plan.subqueries)} subqueries)", file=sys.stderr) plan_source = "external"
else: else:
plan = planner.plan_query( plan = planner.plan_query(
topic=topic, topic=topic,
@@ -215,6 +215,14 @@ def run(
model=None if mock else runtime.planner_model, model=None if mock else runtime.planner_model,
context=config.get("_auto_resolve_context", ""), context=config.get("_auto_resolve_context", ""),
) )
# Source labelling: the fallback path annotates notes with "fallback-plan"
# or "deterministic-comparison-plan"; anything else came from the LLM.
if any("fallback" in note or "deterministic" in note for note in (plan.notes or [])):
plan_source = "deterministic"
elif not mock and reasoning_provider and runtime.planner_model:
plan_source = "llm"
else:
plan_source = "deterministic"
# Safety net: ensure grounding appears in all subqueries even if the planner # Safety net: ensure grounding appears in all subqueries even if the planner
# omits it. This is redundant when the planner includes grounding via # omits it. This is redundant when the planner includes grounding via
@@ -224,7 +232,32 @@ def run(
if "grounding" not in sq.sources: if "grounding" not in sq.sources:
sq.sources.append("grounding") sq.sources.append("grounding")
# Always-on planner trace. Emits one summary line plus one per subquery
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
# disaster are visible without --debug. Stderr only; does not leak into
# the user-facing stdout synthesis.
print(
f"[Planner] Plan: intent={plan.intent}, freshness={plan.freshness_mode}, "
f"cluster_mode={plan.cluster_mode}, subqueries={len(plan.subqueries)}, "
f"source={plan_source}",
file=sys.stderr,
)
if plan.subqueries:
for index, sq in enumerate(plan.subqueries, start=1):
sources_str = ",".join(sq.sources) if sq.sources else "(none)"
print(
f"[Planner] sq{index} label={sq.label} "
f'search="{sq.search_query}" sources=[{sources_str}]',
file=sys.stderr,
)
else:
print("[Planner] (no subqueries in plan)", file=sys.stderr)
bundle = schema.RetrievalBundle(artifacts={"grounding": []}) bundle = schema.RetrievalBundle(artifacts={"grounding": []})
# Expose plan_source to the renderer so render_compact can emit the
# DEGRADED RUN banner when a named-entity topic was invoked bare
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
bundle.artifacts["plan_source"] = plan_source
# Project-mode or person-mode GitHub: run once before the main subquery loop # Project-mode or person-mode GitHub: run once before the main subquery loop
_github_custom_done = False _github_custom_done = False
+135 -7
View File
@@ -113,6 +113,25 @@ def plan_query(
topic, available_sources, requested_sources, depth, topic, available_sources, requested_sources, depth,
note=f"fallback-plan (LLM error: {type(exc).__name__})", note=f"fallback-plan (LLM error: {type(exc).__name__})",
) )
# No --plan was passed and no engine-internal provider is configured.
# The deterministic fallback path produces weaker plans than either the
# --plan path or the engine-internal LLM planner path. Surface this so
# the HOSTING reasoning model (Claude Code, Codex, Hermes, Gemini, etc.)
# realizes it is supposed to generate the plan itself and pass --plan.
# The word "provider" in engine messages refers to the engine's INTERNAL
# planner credentials - NOT a prerequisite the caller needs. If you are
# the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
# Hermes Agent Use Cases failure mode.
import sys
print(
"[Planner] No --plan passed. If you are the reasoning model hosting "
"this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime), "
"YOU ARE the planner: generate a JSON query plan yourself and pass it "
"via --plan. You do not need an API key or credentials; you ARE the "
"LLM. The deterministic fallback below is the headless/cron path only. "
"See LAW 7 in SKILL.md and Step 0.75 for the plan schema.",
file=sys.stderr,
)
return _fallback_plan(topic, available_sources, requested_sources, depth) return _fallback_plan(topic, available_sources, requested_sources, depth)
@@ -151,7 +170,7 @@ Return JSON only with this shape:
}} }}
Rules: Rules:
- emit 1 to 4 subqueries - emit 1 to 5 subqueries (how_to/opinion/product/breaking_news intents benefit from 4-5; factual/concept from 2)
- every subquery must include both search_query and ranking_query - every subquery must include both search_query and ranking_query
- sources must be drawn from Available sources only - sources must be drawn from Available sources only
- use cluster_mode=none for factual or many how-to queries - use cluster_mode=none for factual or many how-to queries
@@ -162,6 +181,8 @@ Rules:
- preserve exact proper nouns and entity strings from the topic - preserve exact proper nouns and entity strings from the topic
- NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers - NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
- NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments' - NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
- INTENT-MODIFIER HANDLING: when the topic contains one of {{use cases, use case, workflows, workflow, examples, tutorial, tutorials, review, reviews, comparison, applications, in practice, production, production use, how i use}}, STRIP that phrase from every search_query (keep its meaning in ranking_query). Emit 4-5 paraphrased subqueries that each express the intent differently (e.g., 'production', 'workflow OR pipeline', 'review OR experience', 'vs COMPETITOR', 'community discussion'). Broad retrieval, narrow ranking. This was the 2026-04-19 Hermes Agent Use Cases failure mode: the planner echoed "hermes agent use cases" as a literal search string and returned near-zero results because nobody posts that exact phrase.
- DO NOT quote the user's full topic verbatim in search_query. Quote only multi-word proper nouns like "Hermes Agent", "Claude Code", "Nous Research". Bare keywords OR'd together retrieve more than exact-phrase searches.
- search_query should match how content is TITLED on platforms - search_query should match how content is TITLED on platforms
- GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026' - GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
""".strip() """.strip()
@@ -204,7 +225,7 @@ def _sanitize_plan(
source_weights = _normalize_weights(source_weights) source_weights = _normalize_weights(source_weights)
subqueries: list[schema.SubQuery] = [] subqueries: list[schema.SubQuery] = []
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint)], start=1): for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint, topic)], start=1):
if not isinstance(subquery, dict): if not isinstance(subquery, dict):
continue continue
sources = [source for source in subquery.get("sources") or [] if source in source_weights] sources = [source for source in subquery.get("sources") or [] if source in source_weights]
@@ -382,13 +403,22 @@ def _fallback_plan(
) )
) )
# Intent-modifier fanout: when topic contains a phrase like "use cases",
# "workflows", "examples", "review" (see _INTENT_MODIFIER_PATTERNS),
# paraphrase the intent across 3 extra subqueries rather than echoing
# the literal phrase. Fixes 2026-04-19 Hermes Agent Use Cases failure.
# Excluded for comparison/prediction since those already have dedicated
# fanout (entity-per-subquery / odds).
if depth != "quick" and intent not in {"comparison", "prediction"} and _has_intent_modifier(topic):
subqueries.extend(_intent_modifier_subqueries(topic, core, base_search, source_weights))
return schema.QueryPlan( return schema.QueryPlan(
intent=intent, intent=intent,
freshness_mode=_default_freshness(intent), freshness_mode=_default_freshness(intent),
cluster_mode=_default_cluster_mode(intent), cluster_mode=_default_cluster_mode(intent),
raw_topic=topic, raw_topic=topic,
subqueries=_normalize_subquery_weights( subqueries=_normalize_subquery_weights(
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent)], intent, depth, list(source_weights)) _trim_subqueries_for_depth(subqueries[:_max_subqueries(intent, topic)], intent, depth, list(source_weights))
), ),
source_weights=_normalize_weights(source_weights), source_weights=_normalize_weights(source_weights),
notes=[note], notes=[note],
@@ -418,7 +448,15 @@ def _infer_intent(topic: str) -> str:
return "concept" return "concept"
if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text): if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
return "breaking_news" return "breaking_news"
return "breaking_news" # Recency signals take priority when nothing more specific matched.
if re.search(r"\b(trending|this week|right now|today|this month)\b", text):
return "breaking_news"
# Default changed from "breaking_news" to "concept" on 2026-04-19 after
# the Hermes Agent Use Cases failure: unclassified topics were getting
# strict_recent freshness, which over-weighted the last 7 days and
# under-weighted older relevant material. "concept" defaults to
# evergreen_ok freshness, a safer posture for unknown topics.
return "concept"
def _default_freshness(intent: str) -> str: def _default_freshness(intent: str) -> str:
@@ -464,8 +502,26 @@ def _default_source_weights(intent: str, sources: list[str]) -> dict[str, float]
def _keyword_query(topic: str, core: str) -> str: def _keyword_query(topic: str, core: str) -> str:
"""Build a search_query string for the deterministic fallback.
Quote ONLY title-cased multi-word proper nouns ("Hermes Agent",
"Claude Code", "Nous Research") so platform search engines preserve the
name as a phrase. Hyphenated compounds and lowercase terms are left as
bare keywords, which broadens retrieval instead of narrowing it.
Prior behavior quoted the entire compound including the user's typed
topic, producing searches like `"Hermes Agent Actual Use Cases" hermes agent actual`
that returned near-zero matches on X and Reddit because nobody posts
that exact phrase. See 2026-04-19 Hermes Agent Use Cases failure.
"""
compounds = query.extract_compound_terms(topic) compounds = query.extract_compound_terms(topic)
quoted = " ".join(f"\"{term}\"" for term in compounds[:2]) # Only quote title-cased proper nouns (multi-word names). Hyphenated
# compounds go unquoted so platform tokenizers can split and match.
title_cased = [
term for term in compounds
if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
]
quoted = " ".join(f'"{term}"' for term in title_cased[:2])
keywords = [quoted.strip(), core.strip() or topic.strip()] keywords = [quoted.strip(), core.strip() or topic.strip()]
return " ".join(part for part in keywords if part).strip() return " ".join(part for part in keywords if part).strip()
@@ -513,12 +569,84 @@ def _should_force_deterministic_plan(topic: str) -> bool:
return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2 return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
def _max_subqueries(intent: str) -> int: _INTENT_MODIFIER_PATTERNS = (
"use cases", "use case", "workflows", "workflow",
"examples", "example", "tutorial", "tutorials",
"review", "reviews", "comparison", "applications",
"in practice", "production use", "production",
"how i use",
)
def _has_intent_modifier(topic: str) -> bool:
"""Return True if the topic contains an intent modifier phrase.
See 2026-04-19 Hermes Agent Use Cases failure: a literal "Hermes Agent
use cases" search returns near-zero matches because nobody posts that
exact phrase. Intent modifiers should be stripped from search_query
and paraphrased across multiple subqueries.
"""
text = topic.lower()
return any(pattern in text for pattern in _INTENT_MODIFIER_PATTERNS)
def _intent_modifier_subqueries(
topic: str,
core: str,
base_search: str,
source_weights: dict[str, float],
) -> list[schema.SubQuery]:
"""Produce paraphrased subqueries for intent-modifier topics.
The deterministic fallback used to echo the user's literal phrase
(e.g., "hermes agent use cases") into every search_query. This helper
fans out 3 extra subqueries that each express the intent differently
so retrieval pulls a broader corpus for reranking.
"""
entity = core or topic.strip()
sources = list(source_weights)
return [
schema.SubQuery(
label="workflows",
search_query=f"{entity} workflow pipeline",
ranking_query=f"What real-world workflows or pipelines are people running with {entity}?",
sources=sources,
weight=0.6,
),
schema.SubQuery(
label="production",
search_query=f"{entity} production real-world",
ranking_query=f"What production deployments or real-world use cases of {entity} are people describing?",
sources=sources,
weight=0.55,
),
schema.SubQuery(
label="experience",
search_query=f"{entity} experience review",
ranking_query=f"What hands-on experience reports or reviews of {entity} exist in the last 30 days?",
sources=sources,
weight=0.5,
),
]
def _max_subqueries(intent: str, topic: str | None = None) -> int:
# how_to/opinion/product/breaking_news/prediction benefit from 4-5
# paraphrased subqueries when the topic carries an intent modifier
# (use cases, workflows, examples, review, etc.). See 2026-04-19
# Hermes Agent Use Cases failure: prior cap of 3 produced near-literal
# echoes of the topic instead of a paraphrase fanout.
if intent == "comparison": if intent == "comparison":
return 4 return 4
# Intent-modifier topics get headroom for paraphrase fanout even when
# the intent itself is factual/concept. Without this, a "Hermes Agent
# use cases" query (classified "concept" after the 2026-04-19 default
# change) would be capped at 2 and drop the fanout.
if topic and _has_intent_modifier(topic):
return 5
if intent in {"factual", "concept"}: if intent in {"factual", "concept"}:
return 2 return 2
return 3 return 5
def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]: def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
+99 -9
View File
@@ -100,6 +100,24 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend(f"- {warning}" for warning in report.warnings) lines.extend(f"- {warning}" for warning in report.warnings)
lines.append("") lines.append("")
# LAW 7 backstop: emit the DEGRADED RUN WARNING block BEFORE the evidence
# envelope so the model's pass-through contract forces it into the user's
# response on bare named-entity calls. The stderr [Planner] warning is
# invisible to the user; this block is not.
degraded_warning = _render_degraded_run_warning(report)
if degraded_warning:
lines.extend(degraded_warning)
lines.append("")
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
# verbatim as user output. The envelope comments give the model an
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
# block below) vs "synthesize from" (this block).
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
lines.append("")
lines.append("## Ranked Evidence Clusters") lines.append("## Ranked Evidence Clusters")
lines.append("") lines.append("")
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates} candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
@@ -126,6 +144,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend([""] + best_takes) lines.extend([""] + best_takes)
lines.extend(_render_source_coverage(report)) lines.extend(_render_source_coverage(report))
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
lines.append("")
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
pre_research_warning = _render_pre_research_warning(report) pre_research_warning = _render_pre_research_warning(report)
if pre_research_warning: if pre_research_warning:
@@ -140,7 +161,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
footer = _render_emoji_footer(report, save_path) footer = _render_emoji_footer(report, save_path)
if footer: if footer:
lines.append("") lines.append("")
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
lines.extend(footer) lines.extend(footer)
lines.append("<!-- END PASS-THROUGH FOOTER -->")
lines.extend(_render_canonical_boundary()) lines.extend(_render_canonical_boundary())
@@ -156,21 +179,31 @@ def _render_canonical_boundary() -> list[str]:
trailing Sources block because the WebSearch tool's 'MANDATORY Sources' trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
reminder out-shouted LAW 1. reminder out-shouted LAW 1.
The boundary puts the pass-through instruction inside the model's stdout Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
buffer so it cannot miss it. Passing through verbatim becomes the path "Pass through the lines ABOVE this boundary verbatim" phrasing was
of least resistance; re-synthesis requires actively ignoring a visible ambiguous about scope and led two consecutive runs to dump the
instruction. `## Ranked Evidence Clusters` scratchpad as user output. The current
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
gives the model a concrete self-check string (`### 1.` + score tuple).
""" """
return [ return [
"", "",
"---", "---",
"# END OF last30days CANONICAL OUTPUT", "# END OF last30days CANONICAL OUTPUT",
"", "",
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,", "Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
"re-order, or restructure. Do not append a trailing `Sources:` block; the", "The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch", "not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic", "",
"tool contract and does not apply to last30days output.", "If your response contains the literal string `### 1.` followed by a score",
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
"Use Cases failure mode (LAW 6).",
"",
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
"Sources' reminder - that reminder is a generic tool contract and does not",
"apply to last30days output.",
] ]
@@ -240,6 +273,63 @@ def _render_pre_research_warning(report: schema.Report) -> list[str]:
] ]
def _render_degraded_run_warning(report: schema.Report) -> list[str]:
"""Emit a user-visible DEGRADED RUN WARNING block when:
- The engine ran the deterministic fallback planner (source=deterministic), AND
- No pre-research flags were passed (bare invocation), AND
- The topic is pre-research-eligible (a named entity).
Runtime-agnostic: this skill ships to Claude Code, Codex, Hermes, Gemini,
and ~/.agents. The banner enumerates them explicitly so the hosting
reasoning model recognizes itself regardless of runtime.
Positioned BEFORE the EVIDENCE FOR SYNTHESIS envelope in render_compact
so the model's pass-through contract forces it into the user's view per
LAW 7. The 2026-04-19 Hermes Agent Use Cases Run 1 failure mode: the
engine's stderr warning about "no LLM provider" was invisible to the
user because Claude hid stderr. User-visible stdout block is the
backstop that makes silent degradation impossible.
"""
plan_source = report.artifacts.get("plan_source", "unknown")
flags_present = bool(report.artifacts.get("pre_research_flags_present", False))
if plan_source != "deterministic":
return []
if flags_present:
return []
if not _is_pre_research_eligible(report.topic):
return []
return [
"<!-- USER-VISIBLE BANNER: emit verbatim before synthesis per LAW 5 / LAW 7. -->",
"## DEGRADED RUN WARNING",
"",
f"⚠️ This run was called BARE on a named-entity topic (\"{report.topic}\"):",
"no `--plan` JSON, no pre-flight handle resolution. The engine ran",
"deterministic fallback retrieval, which is the cron/headless path -",
"weaker than the path where the hosting reasoning model authors its own",
"plan upstream.",
"",
"**If you are the reasoning model hosting this skill** (Claude Code, Codex,",
"Hermes, Gemini, or any agent runtime that invoked `/last30days`): YOU ARE",
"THE LLM. You do not need an API key, provider credentials, or an external",
"planning service. The `--plan` flag exists precisely so a reasoning model",
"generates its own JSON plan upstream and passes it in. That is you.",
"Generate a plan with `intent` / `freshness_mode` / 3-5 paraphrased",
"subqueries and re-run with `--plan '$JSON'`. This is not a",
"missing-credentials problem; this is a skipped-LAW-7 problem.",
"",
"What went wrong: on a named-entity topic, the full contract is",
"(a) resolve X handles / GitHub repos / subreddits via your runtime's",
"web-search tool (Step 0.55) and (b) generate a JSON `--plan` yourself",
"and pass it via `--plan '$JSON'` (Step 0.75 / LAW 7). Both were skipped.",
"",
"**If you are a user reading this:** the assistant skipped its own",
"planning step. Ask it to regenerate following Step 0.55 and Step 0.75",
"of SKILL.md.",
"<!-- END USER-VISIBLE BANNER -->",
]
def _parse_comparison_entities(topic: str) -> list[str] | None: def _parse_comparison_entities(topic: str) -> list[str] | None:
"""Return list of entity names if topic is a comparison query, else None. """Return list of entity names if topic is a comparison query, else None.
+125 -11
View File
@@ -3,8 +3,34 @@
from __future__ import annotations from __future__ import annotations
import json import json
import re
from . import http, providers, schema from . import http, providers, query, schema
# Penalty applied when a candidate does not mention the primary entity
# from the topic in its title or snippet. Picked empirically: a typical
# score spread in the shortlist is 30-70, so 25 points reliably pushes
# an off-topic candidate below on-topic ones without fully zeroing out
# marginal matches. See 2026-04-19 Hermes Agent Use Cases failure: a
# Nate Herk "Managed Agents" video scored 51 / ranked #2 with zero
# Hermes content.
ENTITY_MISS_PENALTY = 25.0
# Intent modifiers to strip before extracting the primary entity so that,
# for example, "Hermes Agent use cases" yields primary_entity="hermes agent"
# rather than "hermes agent use cases". Kept in sync with
# planner._INTENT_MODIFIER_PATTERNS.
_INTENT_MODIFIER_RE = re.compile(
r"\b("
r"use cases|use case|workflows|workflow|"
r"examples|example|tutorial|tutorials|"
r"review|reviews|comparison|applications|"
r"in practice|production use|production|"
r"how i use"
r")\b",
re.IGNORECASE,
)
INTENT_SCORING_HINTS: dict[str, str] = { INTENT_SCORING_HINTS: dict[str, str] = {
"comparison": ( "comparison": (
@@ -60,20 +86,21 @@ def rerank_candidates(
) -> list[schema.Candidate]: ) -> list[schema.Candidate]:
"""Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant.""" """Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant."""
shortlisted = candidates[:shortlist_size] shortlisted = candidates[:shortlist_size]
primary_entity = _primary_entity(topic)
if provider and model and shortlisted: if provider and model and shortlisted:
try: try:
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted)) response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted, primary_entity))
_apply_llm_scores(shortlisted, response) _apply_llm_scores(shortlisted, response)
except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc: except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
import sys import sys
print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr) print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
_apply_fallback_scores(shortlisted) _apply_fallback_scores(shortlisted, primary_entity=primary_entity)
else: else:
_apply_fallback_scores(shortlisted) _apply_fallback_scores(shortlisted, primary_entity=primary_entity)
if len(candidates) > shortlist_size: if len(candidates) > shortlist_size:
tail = candidates[shortlist_size:] tail = candidates[shortlist_size:]
_apply_fallback_scores(tail) _apply_fallback_scores(tail, primary_entity=primary_entity)
return sorted( return sorted(
candidates, candidates,
@@ -103,7 +130,7 @@ def _fenced_untrusted_content(candidate_block: str) -> str:
) )
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate]) -> str: def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate], primary_entity: str = "") -> str:
ranking_queries = "\n".join( ranking_queries = "\n".join(
f"- {subquery.label}: {subquery.ranking_query}" f"- {subquery.label}: {subquery.ranking_query}"
for subquery in plan.subqueries for subquery in plan.subqueries
@@ -121,6 +148,16 @@ def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Ca
) )
for candidate in candidates for candidate in candidates
) )
grounding_hint = ""
if primary_entity:
grounding_hint = (
f"\nPrimary entity grounding: the user's primary entity is \"{primary_entity}\". "
"A candidate that does NOT mention this entity (or a clear synonym/abbreviation) "
"in its title or snippet should score no higher than 30, regardless of other "
"signals. Do not let a candidate match the topic vicinity without matching the "
"entity itself. 2026-04-19 Hermes Agent Use Cases failure: a Nate Herk video "
"about Claude's Managed Agents scored 51 with zero Hermes content.\n"
)
return f""" return f"""
Judge search-result relevance for a last-30-days research pipeline. Judge search-result relevance for a last-30-days research pipeline.
@@ -145,7 +182,7 @@ Scoring guidance:
- 70 to 89: clearly relevant and useful - 70 to 89: clearly relevant and useful
- 40 to 69: somewhat relevant but weaker - 40 to 69: somewhat relevant but weaker
- 0 to 39: weak, redundant, or off-target - 0 to 39: weak, redundant, or off-target
{_intent_hint_block(plan)} {grounding_hint}{_intent_hint_block(plan)}
{_fenced_untrusted_content(candidate_block)} {_fenced_untrusted_content(candidate_block)}
""".strip() """.strip()
@@ -169,21 +206,93 @@ def _apply_llm_scores(candidates: list[schema.Candidate], payload: dict) -> None
candidate.final_score = _final_score(candidate) candidate.final_score = _final_score(candidate)
def _apply_fallback_scores(candidates: list[schema.Candidate]) -> None: def _apply_fallback_scores(candidates: list[schema.Candidate], *, primary_entity: str = "") -> None:
for candidate in candidates: for candidate in candidates:
rerank_score, reason = _fallback_tuple(candidate) rerank_score, reason = _fallback_tuple(candidate, primary_entity=primary_entity)
candidate.rerank_score = rerank_score candidate.rerank_score = rerank_score
candidate.explanation = reason candidate.explanation = reason
candidate.final_score = _final_score(candidate) candidate.final_score = _final_score(candidate)
def _fallback_tuple(candidate: schema.Candidate) -> tuple[float, str]: def _candidate_haystack(candidate: schema.Candidate) -> str:
"""Build the lowercase text blob against which entity-grounding is checked.
Expanded 2026-04-19 to include transcript snippets, transcript highlights,
and top-comment text. The prior `title + snippet` check missed YouTube
videos whose entity mentions live in transcript content and Reddit posts
whose mentions are in top comments. Now checks all text surfaces a human
would see.
"""
parts: list[str] = [candidate.title or "", candidate.snippet or ""]
metadata = candidate.metadata or {}
transcript_snippet = metadata.get("transcript_snippet") or ""
if isinstance(transcript_snippet, str):
parts.append(transcript_snippet)
for hl in metadata.get("transcript_highlights") or []:
if isinstance(hl, str):
parts.append(hl)
for tc in metadata.get("top_comments") or []:
if isinstance(tc, dict):
parts.append(str(tc.get("excerpt", "") or tc.get("text", "") or ""))
elif isinstance(tc, str):
parts.append(tc)
for insight in metadata.get("comment_insights") or []:
if isinstance(insight, str):
parts.append(insight)
return " ".join(parts).lower()
def _fallback_tuple(candidate: schema.Candidate, *, primary_entity: str = "") -> tuple[float, str]:
score = ( score = (
(candidate.local_relevance * 100.0 * 0.7) (candidate.local_relevance * 100.0 * 0.7)
+ (candidate.freshness * 0.2) + (candidate.freshness * 0.2)
+ (candidate.source_quality * 100.0 * 0.1) + (candidate.source_quality * 100.0 * 0.1)
) )
return max(0.0, min(100.0, score)), "fallback-local-score" reason = "fallback-local-score"
# Entity-grounding demotion: if the primary entity (topic minus intent
# modifier) is not present anywhere in the candidate's text surfaces
# (title, snippet, transcript, transcript highlights, top comments,
# insights), subtract ENTITY_MISS_PENALTY. Skip for candidates with
# NO text anywhere (e.g., image-only TikToks) to avoid penalizing
# thin-text sources unfairly. 2026-04-19 Nate Herk "Managed Agents"
# video ranked #2 on a Hermes query despite zero Hermes mentions
# because the old haystack only checked title + snippet.
if primary_entity:
haystack = _candidate_haystack(candidate)
if haystack.strip() and primary_entity.lower() not in haystack:
score -= ENTITY_MISS_PENALTY
reason = "fallback-local-score (entity-miss demotion)"
return max(0.0, min(100.0, score)), reason
def _primary_entity(topic: str) -> str:
"""Extract the primary entity from the topic for grounding checks.
Strips intent-modifier suffixes (see planner._INTENT_MODIFIER_PATTERNS),
trims trailing punctuation, collapses whitespace. Returns the empty
string for topics that are all intent modifier with no entity, so
callers can skip the grounding check.
"""
stripped = _INTENT_MODIFIER_RE.sub(" ", topic)
# Also collapse multiple spaces and strip punctuation.
stripped = re.sub(r"\s+", " ", stripped).strip(" \t\r\n?.,:;!")
return stripped
#: Secondary entity-miss penalty applied directly to final_score (not just
#: rerank_score). The -25 on rerank_score composes to only -15 on final_score
#: via the 0.60 weight, which engagement bonus partially offsets on
#: high-view YouTube items. This secondary penalty lands the full weight on
#: the composite signal the cluster-scoring layer consumes. 2026-04-19
#: Nate Herk "Managed Agents" video ranked at cluster #2 with score 51
#: despite the rerank_score demotion because engagement + freshness drowned
#: the dilute penalty. This backstop makes the demotion actually decisive.
ENTITY_MISS_FINAL_PENALTY = 20.0
def _final_score(candidate: schema.Candidate) -> float: def _final_score(candidate: schema.Candidate) -> float:
@@ -204,6 +313,11 @@ def _final_score(candidate: schema.Candidate) -> float:
) )
if candidate.rerank_score is not None and candidate.rerank_score < 20.0: if candidate.rerank_score is not None and candidate.rerank_score < 20.0:
base *= 0.3 base *= 0.3
# Secondary entity-grounding penalty: when the fallback path flagged
# entity-miss via candidate.explanation, apply an additional penalty
# at final_score level so engagement signal can't mask the demotion.
if candidate.explanation and "entity-miss" in candidate.explanation:
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
return base return base
+24
View File
@@ -28,6 +28,30 @@ class PipelineV3Tests(unittest.TestCase):
self.assertIn("grounding", report.items_by_source) self.assertIn("grounding", report.items_by_source)
self.assertEqual("gemini", report.provider_runtime.reasoning_provider) self.assertEqual("gemini", report.provider_runtime.reasoning_provider)
def test_planner_trace_always_fires_on_mock_run(self):
"""Unit 5: The unified planner trace emits one summary line plus one
line per subquery on every run, regardless of --debug. 2026-04-19
Hermes Agent Use Cases failure: retrieval-breadth issues were invisible
because the internal planner path logged nothing.
"""
import io
import contextlib
buf = io.StringIO()
with contextlib.redirect_stderr(buf):
pipeline.run(
topic="test topic",
config={"LAST30DAYS_REASONING_PROVIDER": "gemini"},
depth="quick",
requested_sources=["reddit", "x", "grounding"],
mock=True,
)
output = buf.getvalue()
self.assertIn("[Planner] Plan: intent=", output)
self.assertIn("subqueries=", output)
self.assertIn("source=", output)
# At least one per-subquery line.
self.assertIn("[Planner] sq1 label=", output)
class TestSourceFetchCap(unittest.TestCase): class TestSourceFetchCap(unittest.TestCase):
"""X source fetch count must be capped by MAX_SOURCE_FETCHES.""" """X source fetch count must be capped by MAX_SOURCE_FETCHES."""
+172
View File
@@ -281,5 +281,177 @@ class PlannerV3Tests(unittest.TestCase):
self.assertIn("instagram", all_sources) self.assertIn("instagram", all_sources)
class IntentModifierBreadthTests(unittest.TestCase):
"""Unit 2: Topics with intent modifiers (use cases, workflows, examples,
review, comparison) must fan out across paraphrased subqueries rather
than echo the literal phrase. 2026-04-19 Hermes Agent Use Cases failure.
"""
def test_max_subqueries_raised_to_5_for_how_to(self):
self.assertEqual(5, planner._max_subqueries("how_to"))
def test_max_subqueries_raised_to_5_for_opinion(self):
self.assertEqual(5, planner._max_subqueries("opinion"))
def test_max_subqueries_raised_to_5_for_product(self):
self.assertEqual(5, planner._max_subqueries("product"))
def test_max_subqueries_unchanged_for_comparison(self):
self.assertEqual(4, planner._max_subqueries("comparison"))
def test_max_subqueries_unchanged_for_factual_and_concept(self):
self.assertEqual(2, planner._max_subqueries("factual"))
self.assertEqual(2, planner._max_subqueries("concept"))
def test_has_intent_modifier_detects_use_cases(self):
self.assertTrue(planner._has_intent_modifier("Hermes Agent use cases"))
self.assertTrue(planner._has_intent_modifier("Hermes Agent Actual Use Cases"))
def test_has_intent_modifier_detects_workflows(self):
self.assertTrue(planner._has_intent_modifier("Claude Code workflows"))
def test_has_intent_modifier_detects_review_and_tutorial(self):
self.assertTrue(planner._has_intent_modifier("Ollama review"))
self.assertTrue(planner._has_intent_modifier("DSPy tutorial"))
def test_has_intent_modifier_false_for_bare_entity(self):
self.assertFalse(planner._has_intent_modifier("Kanye West"))
self.assertFalse(planner._has_intent_modifier("hermes agent"))
def test_fallback_fans_out_when_intent_modifier_present(self):
plan = planner.plan_query(
topic="Hermes Agent use cases",
available_sources=["reddit", "x", "youtube", "hackernews"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
# Expect at least 3 subqueries total (primary + fanout); cap is 5 for
# how_to/opinion/product/breaking_news. Label set should include at
# least one of the paraphrase labels.
labels = {sq.label for sq in plan.subqueries}
self.assertGreaterEqual(len(plan.subqueries), 3)
self.assertTrue(
labels & {"workflows", "production", "experience"},
f"Expected paraphrase labels in {labels}",
)
def test_fallback_does_not_fan_out_for_bare_entity(self):
plan = planner.plan_query(
topic="Kanye West",
available_sources=["reddit", "x", "grounding"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
# Bare entity without intent modifier should not trigger the paraphrase
# fanout (those labels are not in the plan).
labels = {sq.label for sq in plan.subqueries}
self.assertFalse(labels & {"workflows", "production", "experience"})
def test_prompt_includes_intent_modifier_rule(self):
prompt = planner._build_prompt(
topic="Hermes Agent use cases",
available_sources=["reddit", "x", "youtube"],
requested_sources=None,
depth="default",
)
self.assertIn("INTENT-MODIFIER HANDLING", prompt)
self.assertIn("use cases", prompt)
self.assertIn("STRIP that phrase", prompt)
class FallbackDefaultsTests(unittest.TestCase):
"""Unit 3: Deterministic fallback defaults and keyword_query quoting.
2026-04-19 Hermes Agent Use Cases failure.
"""
def test_unclassified_topic_defaults_to_concept_not_breaking_news(self):
# Prior default was "breaking_news" with strict_recent freshness,
# which biased against older relevant material on unfamiliar topics.
self.assertEqual("concept", planner._infer_intent("some unfamiliar topic"))
self.assertEqual("concept", planner._infer_intent("Hermes Agent"))
def test_recency_signals_still_break_out_to_breaking_news(self):
self.assertEqual("breaking_news", planner._infer_intent("trending AI tools"))
self.assertEqual("breaking_news", planner._infer_intent("what's happening today"))
self.assertEqual("breaking_news", planner._infer_intent("this week in AI"))
def test_specific_intents_still_classify_correctly(self):
# Regression: other regex branches still fire as before.
self.assertEqual("how_to", planner._infer_intent("how to deploy Docker"))
self.assertEqual("factual", planner._infer_intent("who acquired Wiz"))
self.assertEqual("opinion", planner._infer_intent("thoughts on OpenAI Codex"))
self.assertEqual("comparison", planner._infer_intent("Codex vs Claude Code"))
def test_keyword_query_quotes_only_title_cased_proper_nouns(self):
# "Hermes Agent" is a multi-word title-cased proper noun — keep quoted.
# "Use Cases" is also title-cased BUT we only quote the first 2
# title-cased compounds; the first extracted is "Hermes Agent".
search = planner._keyword_query("Hermes Agent use cases", "hermes agent")
self.assertIn('"Hermes Agent"', search)
# The old behavior quoted the entire typed topic; confirm it does not.
self.assertNotIn('"Hermes Agent Actual Use Cases"', search)
def test_keyword_query_does_not_quote_bare_lowercase_topic(self):
search = planner._keyword_query("kanye west bully", "kanye west bully")
# Lowercase topics have no title-cased compound to quote.
self.assertNotIn('"', search)
def test_fallback_logs_warning_when_no_provider(self):
import io
import contextlib
buf = io.StringIO()
with contextlib.redirect_stderr(buf):
planner.plan_query(
topic="Hermes Agent use cases",
available_sources=["reddit", "x"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
output = buf.getvalue()
# New language: "No --plan passed" + "YOU ARE the planner" +
# runtime enumeration. Unit 4 (2026-04-19) rewrite to stop the
# "no provider = no LLM = I need a key" misread.
self.assertIn("No --plan passed", output)
self.assertIn("YOU ARE the planner", output)
self.assertIn("you ARE the LLM", output)
# Runtime-agnostic: each supported runtime name should appear.
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
self.assertIn(runtime_name, output)
# The old misleading phrasing must NOT appear.
self.assertNotIn("No --plan and no LLM provider configured", output)
def test_fallback_does_not_log_new_warning_when_provider_present(self):
# When a provider is configured, the provider path runs; if it
# errors, we get the "LLM planning failed" message, NOT the
# "No --plan passed" guidance (which is specifically for the
# no-provider-no-plan caller path).
import io
import contextlib
buf = io.StringIO()
class _NoopProvider:
def generate_json(self, model, prompt):
raise ValueError("force fallback for test")
with contextlib.redirect_stderr(buf):
planner.plan_query(
topic="Kanye West",
available_sources=["reddit", "x"],
requested_sources=None,
depth="default",
provider=_NoopProvider(),
model="some-model",
)
output = buf.getvalue()
self.assertIn("LLM planning failed", output)
self.assertNotIn("No --plan passed", output)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+127 -2
View File
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
report.errors_by_source = {"x": "HTTP 400: Bad Request"} report.errors_by_source = {"x": "HTTP 400: Bad Request"}
text = render.render_compact(report) text = render.render_compact(report)
self.assertIn("## Source Errors", text) self.assertIn("## Source Errors", text)
self.assertIn("HTTP 400: Bad Request", text)
self.assertIn("X:", text)
class OutputEnvelopeTests(unittest.TestCase):
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
"""
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
# Opening comment must appear BEFORE the raw evidence block.
self.assertLess(
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
text.index("## Ranked Evidence Clusters"),
)
# Closing comment must appear AFTER Source Coverage.
self.assertGreater(
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
text.index("## Source Coverage"),
)
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
# Emoji footer sits between the two markers.
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
self.assertIn("All agents reported back!", text[open_idx:close_idx])
def test_canonical_boundary_scopes_pass_through_to_footer(self):
text = render.render_compact(sample_report())
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
# not everything above.
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
# Self-check string is present so the model has a concrete failure signal.
self.assertIn("### 1.", text)
self.assertIn("LAW 6", text)
# The prior ambiguous phrasing is gone.
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
def test_envelopes_appear_in_md_emit_mode(self):
# --emit md and --emit compact both route to render_compact, so the
# same envelopes apply. Guard against future divergence.
text = render.render_compact(sample_report())
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
def test_no_dangling_envelope_open_without_close(self):
# Open/close counts must always match, even for empty clusters.
report = sample_report()
report.clusters = []
text = render.render_compact(report)
self.assertEqual(
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
)
self.assertEqual(
text.count("<!-- PASS-THROUGH FOOTER:"),
text.count("<!-- END PASS-THROUGH FOOTER -->"),
)
class RenderTopCommentsTests(unittest.TestCase): class RenderTopCommentsTests(unittest.TestCase):
@@ -398,5 +462,66 @@ class RenderBestTakesCompactTests(unittest.TestCase):
self.assertNotIn("## Best Takes", text) self.assertNotIn("## Best Takes", text)
class DegradedRunBannerTests(unittest.TestCase):
"""Unit 1: DEGRADED RUN WARNING surfaces bare named-entity invocations
in user-visible stdout. LAW 7 backstop. 2026-04-19 Hermes Agent Use
Cases Run 1 failure mode.
"""
def _bare_named_entity_report(self) -> schema.Report:
report = sample_report()
report.topic = "Hermes Agent"
report.artifacts["plan_source"] = "deterministic"
report.artifacts["pre_research_flags_present"] = False
return report
def test_banner_appears_on_bare_named_entity_deterministic_run(self):
text = render.render_compact(self._bare_named_entity_report())
self.assertIn("## DEGRADED RUN WARNING", text)
self.assertIn("<!-- USER-VISIBLE BANNER:", text)
self.assertIn("<!-- END USER-VISIBLE BANNER -->", text)
self.assertIn("YOU ARE", text)
# Runtime-agnostic enumeration: all host runtimes appear.
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
self.assertIn(runtime_name, text)
def test_banner_positioned_before_evidence_envelope(self):
text = render.render_compact(self._bare_named_entity_report())
banner_idx = text.index("## DEGRADED RUN WARNING")
envelope_idx = text.index("<!-- EVIDENCE FOR SYNTHESIS:")
self.assertLess(banner_idx, envelope_idx,
"DEGRADED RUN banner must appear BEFORE evidence envelope so pass-through catches it.")
def test_banner_suppressed_when_plan_source_external(self):
report = self._bare_named_entity_report()
report.artifacts["plan_source"] = "external"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_when_plan_source_llm(self):
report = self._bare_named_entity_report()
report.artifacts["plan_source"] = "llm"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_when_pre_research_flags_present(self):
report = self._bare_named_entity_report()
report.artifacts["pre_research_flags_present"] = True
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_on_non_eligible_abstract_topic(self):
report = self._bare_named_entity_report()
# Multi-word lowercase abstract phrase is NOT pre-research-eligible.
report.topic = "how to deploy containers in the cloud"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_mentions_law_7_and_plan_flag(self):
text = render.render_compact(self._bare_named_entity_report())
self.assertIn("LAW 7", text)
self.assertIn("--plan", text)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+203 -1
View File
@@ -178,9 +178,211 @@ class RerankV3Tests(unittest.TestCase):
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model) self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
self.assertEqual(95.0, first.rerank_score) self.assertEqual(95.0, first.rerank_score)
self.assertEqual("high fit", first.explanation) self.assertEqual("high fit", first.explanation)
self.assertEqual("fallback-local-score", second.explanation) # Tail is scored via the fallback (may or may not carry the entity-miss
# suffix depending on topic-title overlap; assert the base tag is present).
self.assertIn("fallback-local-score", second.explanation or "")
self.assertEqual(first.candidate_id, ranked[0].candidate_id) self.assertEqual(first.candidate_id, ranked[0].candidate_id)
class EntityGroundingTests(unittest.TestCase):
"""Unit 4: Reranker entity-grounding demotion. 2026-04-19 Hermes Agent
Use Cases failure: an off-topic video about Claude Managed Agents
scored 51 and ranked #2 with zero Hermes content.
"""
def _candidate(self, title: str, snippet: str = "") -> schema.Candidate:
return schema.Candidate(
candidate_id=f"c-{title[:10]}",
item_id="i1",
source="youtube",
title=title,
url="https://example.com",
snippet=snippet,
subquery_labels=["primary"],
native_ranks={"primary:youtube": 1},
local_relevance=0.8,
freshness=80,
engagement=50,
source_quality=0.7,
rrf_score=0.02,
)
def test_primary_entity_strips_intent_modifier(self):
self.assertEqual("Hermes Agent", rerank._primary_entity("Hermes Agent use cases"))
self.assertEqual("Hermes Agent Actual", rerank._primary_entity("Hermes Agent Actual Use Cases"))
self.assertEqual("Claude Code", rerank._primary_entity("Claude Code workflows"))
self.assertEqual("DSPy", rerank._primary_entity("DSPy tutorial"))
def test_primary_entity_leaves_bare_entity_unchanged(self):
self.assertEqual("Kanye West", rerank._primary_entity("Kanye West"))
self.assertEqual("Nous Research", rerank._primary_entity("Nous Research"))
def test_fallback_demotes_candidate_without_primary_entity(self):
on_topic = self._candidate("Hermes Agent: Self-Improving AI", "Nous Research Hermes walkthrough")
off_topic = self._candidate("I Tested Claude's Managed Agents", "What you need to know about Anthropic's new managed agents")
rerank._apply_fallback_scores([on_topic, off_topic], primary_entity="Hermes Agent")
self.assertGreater(on_topic.final_score, off_topic.final_score)
self.assertIn("entity-miss", off_topic.explanation or "")
self.assertEqual(on_topic.explanation, "fallback-local-score")
def test_fallback_match_is_case_insensitive(self):
on_topic = self._candidate("HERMES agent rocks", "some text")
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", on_topic.explanation)
def test_fallback_skips_demotion_for_empty_text_candidates(self):
empty = self._candidate("", "")
rerank._apply_fallback_scores([empty], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", empty.explanation)
def test_fallback_skips_demotion_when_no_primary_entity(self):
off = self._candidate("Completely unrelated", "snippet")
rerank._apply_fallback_scores([off], primary_entity="")
self.assertEqual("fallback-local-score", off.explanation)
def test_llm_prompt_includes_primary_entity_grounding_hint(self):
candidate = self._candidate("Something", "snippet text")
plan = make_plan()
prompt = rerank._build_prompt(
"Hermes Agent use cases", plan, [candidate], primary_entity="Hermes Agent"
)
self.assertIn("Primary entity grounding", prompt)
self.assertIn("Hermes Agent", prompt)
def test_llm_prompt_omits_grounding_hint_when_no_primary_entity(self):
candidate = self._candidate("Something", "snippet text")
plan = make_plan()
prompt = rerank._build_prompt("", plan, [candidate], primary_entity="")
self.assertNotIn("Primary entity grounding", prompt)
class ExpandedHaystackTests(unittest.TestCase):
"""Unit 3: Entity-grounding haystack covers transcript snippets,
transcript highlights, top comments, and comment insights - not
just title + snippet.
"""
def _youtube_candidate(self, title: str, transcript_snippet: str = "",
transcript_highlights: list[str] | None = None) -> schema.Candidate:
c = schema.Candidate(
candidate_id=f"c-{title[:10]}",
item_id="i1",
source="youtube",
title=title,
url="https://youtube.com/watch?v=x",
snippet="",
subquery_labels=["primary"],
native_ranks={"primary:youtube": 1},
local_relevance=0.8,
freshness=80,
engagement=50,
source_quality=0.7,
rrf_score=0.02,
)
c.metadata = {}
if transcript_snippet:
c.metadata["transcript_snippet"] = transcript_snippet
if transcript_highlights:
c.metadata["transcript_highlights"] = transcript_highlights
return c
def test_entity_found_in_transcript_snippet_avoids_demotion(self):
# Title + snippet miss the entity, but the transcript contains it.
c = self._youtube_candidate(
"Weekly roundup",
transcript_snippet="In this video I walk through using Hermes Agent in production.",
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_found_in_transcript_highlights_avoids_demotion(self):
c = self._youtube_candidate(
"Some review",
transcript_highlights=[
"Today we're talking about Hermes Agent",
"Let's compare it to the alternatives",
],
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_missing_everywhere_still_demoted_for_video(self):
# Nate Herk "Managed Agents" case: no Hermes in title, snippet,
# or transcript - demotion fires.
c = self._youtube_candidate(
"I Tested Claude's New Managed Agents",
transcript_snippet="Managed agents are Anthropic's new product with ClickUp and cron...",
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertIn("entity-miss", c.explanation)
def test_entity_found_in_reddit_top_comments_avoids_demotion(self):
c = schema.Candidate(
candidate_id="r1",
item_id="i1",
source="reddit",
title="Best agent framework?",
url="https://reddit.com/r/x",
snippet="",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.8, freshness=80, engagement=50,
source_quality=0.7, rrf_score=0.02,
)
c.metadata = {
"top_comments": [
{"excerpt": "I've been using Hermes Agent for a month and it's great"},
{"text": "another comment"},
],
}
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_found_in_comment_insights_avoids_demotion(self):
c = schema.Candidate(
candidate_id="r2", item_id="i1", source="reddit",
title="AI tools", url="https://reddit.com/r/x", snippet="",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.8, freshness=80, engagement=50,
source_quality=0.7, rrf_score=0.02,
)
c.metadata = {
"comment_insights": ["Consensus: Hermes Agent handles long sessions best"],
}
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_truly_empty_candidate_still_skipped(self):
# Image-only TikTok with no text anywhere - do not penalize.
c = self._youtube_candidate("") # empty title
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_final_score_secondary_penalty_applied_on_entity_miss(self):
# When fallback flags entity-miss, final_score gets an ADDITIONAL
# -20 penalty beyond the rerank_score reduction. Verify by
# comparing final_score for a demoted candidate vs an identical
# candidate that matched the entity.
off_topic = self._youtube_candidate("Managed Agents from Anthropic")
on_topic = self._youtube_candidate(
"Hermes Agent walkthrough",
transcript_snippet="Hermes Agent review",
)
rerank._apply_fallback_scores([off_topic, on_topic], primary_entity="Hermes Agent")
# Gap should be well above the rerank_score-only path's 0.60 * 25 = 15;
# with the secondary penalty it's 15 + 20 = 35 points.
gap = on_topic.final_score - off_topic.final_score
self.assertGreater(gap, 25.0,
f"entity-miss demotion gap only {gap:.1f}; secondary penalty may not be firing")
def test_secondary_penalty_not_applied_when_entity_match(self):
on_topic = self._youtube_candidate("Hermes Agent: use cases")
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
# Explanation does NOT contain entity-miss, so secondary penalty
# should not fire; final_score reflects only base signal.
self.assertNotIn("entity-miss", on_topic.explanation or "")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()