Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth

fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
This commit is contained in:
Matt Van Horn
2026-04-19 11:09:56 -07:00
committed by GitHub
9 changed files with 962 additions and 32 deletions
+43 -1
View File
@@ -120,7 +120,49 @@ These five rules dominate every other rule in this file. If you find yourself ab
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings. **Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output. **LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
**Worked example (LAW 6 transformation).** Evidence block you read:
```
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
## Ranked Evidence Clusters
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
- "Can you tell me what type of user profile you have on me?"
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
<!-- END EVIDENCE FOR SYNTHESIS -->
```
Output you emit (prose synthesis, NOT the evidence block):
```
What I learned:
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
```
**LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan '$JSON'`.
Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan '$JSON'`. A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema).
**Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step.
**Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan '$JSON'`? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider.
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail. End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
+34 -1
View File
@@ -204,7 +204,7 @@ def run(
plan = planner._sanitize_plan( plan = planner._sanitize_plan(
external_plan, topic, available, requested_sources, depth, external_plan, topic, available, requested_sources, depth,
) )
print(f"[Planner] Using external plan ({len(plan.subqueries)} subqueries)", file=sys.stderr) plan_source = "external"
else: else:
plan = planner.plan_query( plan = planner.plan_query(
topic=topic, topic=topic,
@@ -215,6 +215,14 @@ def run(
model=None if mock else runtime.planner_model, model=None if mock else runtime.planner_model,
context=config.get("_auto_resolve_context", ""), context=config.get("_auto_resolve_context", ""),
) )
# Source labelling: the fallback path annotates notes with "fallback-plan"
# or "deterministic-comparison-plan"; anything else came from the LLM.
if any("fallback" in note or "deterministic" in note for note in (plan.notes or [])):
plan_source = "deterministic"
elif not mock and reasoning_provider and runtime.planner_model:
plan_source = "llm"
else:
plan_source = "deterministic"
# Safety net: ensure grounding appears in all subqueries even if the planner # Safety net: ensure grounding appears in all subqueries even if the planner
# omits it. This is redundant when the planner includes grounding via # omits it. This is redundant when the planner includes grounding via
@@ -224,7 +232,32 @@ def run(
if "grounding" not in sq.sources: if "grounding" not in sq.sources:
sq.sources.append("grounding") sq.sources.append("grounding")
# Always-on planner trace. Emits one summary line plus one per subquery
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
# disaster are visible without --debug. Stderr only; does not leak into
# the user-facing stdout synthesis.
print(
f"[Planner] Plan: intent={plan.intent}, freshness={plan.freshness_mode}, "
f"cluster_mode={plan.cluster_mode}, subqueries={len(plan.subqueries)}, "
f"source={plan_source}",
file=sys.stderr,
)
if plan.subqueries:
for index, sq in enumerate(plan.subqueries, start=1):
sources_str = ",".join(sq.sources) if sq.sources else "(none)"
print(
f"[Planner] sq{index} label={sq.label} "
f'search="{sq.search_query}" sources=[{sources_str}]',
file=sys.stderr,
)
else:
print("[Planner] (no subqueries in plan)", file=sys.stderr)
bundle = schema.RetrievalBundle(artifacts={"grounding": []}) bundle = schema.RetrievalBundle(artifacts={"grounding": []})
# Expose plan_source to the renderer so render_compact can emit the
# DEGRADED RUN banner when a named-entity topic was invoked bare
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
bundle.artifacts["plan_source"] = plan_source
# Project-mode or person-mode GitHub: run once before the main subquery loop # Project-mode or person-mode GitHub: run once before the main subquery loop
_github_custom_done = False _github_custom_done = False
+135 -7
View File
@@ -113,6 +113,25 @@ def plan_query(
topic, available_sources, requested_sources, depth, topic, available_sources, requested_sources, depth,
note=f"fallback-plan (LLM error: {type(exc).__name__})", note=f"fallback-plan (LLM error: {type(exc).__name__})",
) )
# No --plan was passed and no engine-internal provider is configured.
# The deterministic fallback path produces weaker plans than either the
# --plan path or the engine-internal LLM planner path. Surface this so
# the HOSTING reasoning model (Claude Code, Codex, Hermes, Gemini, etc.)
# realizes it is supposed to generate the plan itself and pass --plan.
# The word "provider" in engine messages refers to the engine's INTERNAL
# planner credentials - NOT a prerequisite the caller needs. If you are
# the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
# Hermes Agent Use Cases failure mode.
import sys
print(
"[Planner] No --plan passed. If you are the reasoning model hosting "
"this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime), "
"YOU ARE the planner: generate a JSON query plan yourself and pass it "
"via --plan. You do not need an API key or credentials; you ARE the "
"LLM. The deterministic fallback below is the headless/cron path only. "
"See LAW 7 in SKILL.md and Step 0.75 for the plan schema.",
file=sys.stderr,
)
return _fallback_plan(topic, available_sources, requested_sources, depth) return _fallback_plan(topic, available_sources, requested_sources, depth)
@@ -151,7 +170,7 @@ Return JSON only with this shape:
}} }}
Rules: Rules:
- emit 1 to 4 subqueries - emit 1 to 5 subqueries (how_to/opinion/product/breaking_news intents benefit from 4-5; factual/concept from 2)
- every subquery must include both search_query and ranking_query - every subquery must include both search_query and ranking_query
- sources must be drawn from Available sources only - sources must be drawn from Available sources only
- use cluster_mode=none for factual or many how-to queries - use cluster_mode=none for factual or many how-to queries
@@ -162,6 +181,8 @@ Rules:
- preserve exact proper nouns and entity strings from the topic - preserve exact proper nouns and entity strings from the topic
- NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers - NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
- NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments' - NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
- INTENT-MODIFIER HANDLING: when the topic contains one of {{use cases, use case, workflows, workflow, examples, tutorial, tutorials, review, reviews, comparison, applications, in practice, production, production use, how i use}}, STRIP that phrase from every search_query (keep its meaning in ranking_query). Emit 4-5 paraphrased subqueries that each express the intent differently (e.g., 'production', 'workflow OR pipeline', 'review OR experience', 'vs COMPETITOR', 'community discussion'). Broad retrieval, narrow ranking. This was the 2026-04-19 Hermes Agent Use Cases failure mode: the planner echoed "hermes agent use cases" as a literal search string and returned near-zero results because nobody posts that exact phrase.
- DO NOT quote the user's full topic verbatim in search_query. Quote only multi-word proper nouns like "Hermes Agent", "Claude Code", "Nous Research". Bare keywords OR'd together retrieve more than exact-phrase searches.
- search_query should match how content is TITLED on platforms - search_query should match how content is TITLED on platforms
- GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026' - GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
""".strip() """.strip()
@@ -204,7 +225,7 @@ def _sanitize_plan(
source_weights = _normalize_weights(source_weights) source_weights = _normalize_weights(source_weights)
subqueries: list[schema.SubQuery] = [] subqueries: list[schema.SubQuery] = []
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint)], start=1): for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint, topic)], start=1):
if not isinstance(subquery, dict): if not isinstance(subquery, dict):
continue continue
sources = [source for source in subquery.get("sources") or [] if source in source_weights] sources = [source for source in subquery.get("sources") or [] if source in source_weights]
@@ -382,13 +403,22 @@ def _fallback_plan(
) )
) )
# Intent-modifier fanout: when topic contains a phrase like "use cases",
# "workflows", "examples", "review" (see _INTENT_MODIFIER_PATTERNS),
# paraphrase the intent across 3 extra subqueries rather than echoing
# the literal phrase. Fixes 2026-04-19 Hermes Agent Use Cases failure.
# Excluded for comparison/prediction since those already have dedicated
# fanout (entity-per-subquery / odds).
if depth != "quick" and intent not in {"comparison", "prediction"} and _has_intent_modifier(topic):
subqueries.extend(_intent_modifier_subqueries(topic, core, base_search, source_weights))
return schema.QueryPlan( return schema.QueryPlan(
intent=intent, intent=intent,
freshness_mode=_default_freshness(intent), freshness_mode=_default_freshness(intent),
cluster_mode=_default_cluster_mode(intent), cluster_mode=_default_cluster_mode(intent),
raw_topic=topic, raw_topic=topic,
subqueries=_normalize_subquery_weights( subqueries=_normalize_subquery_weights(
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent)], intent, depth, list(source_weights)) _trim_subqueries_for_depth(subqueries[:_max_subqueries(intent, topic)], intent, depth, list(source_weights))
), ),
source_weights=_normalize_weights(source_weights), source_weights=_normalize_weights(source_weights),
notes=[note], notes=[note],
@@ -418,7 +448,15 @@ def _infer_intent(topic: str) -> str:
return "concept" return "concept"
if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text): if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
return "breaking_news" return "breaking_news"
return "breaking_news" # Recency signals take priority when nothing more specific matched.
if re.search(r"\b(trending|this week|right now|today|this month)\b", text):
return "breaking_news"
# Default changed from "breaking_news" to "concept" on 2026-04-19 after
# the Hermes Agent Use Cases failure: unclassified topics were getting
# strict_recent freshness, which over-weighted the last 7 days and
# under-weighted older relevant material. "concept" defaults to
# evergreen_ok freshness, a safer posture for unknown topics.
return "concept"
def _default_freshness(intent: str) -> str: def _default_freshness(intent: str) -> str:
@@ -464,8 +502,26 @@ def _default_source_weights(intent: str, sources: list[str]) -> dict[str, float]
def _keyword_query(topic: str, core: str) -> str: def _keyword_query(topic: str, core: str) -> str:
"""Build a search_query string for the deterministic fallback.
Quote ONLY title-cased multi-word proper nouns ("Hermes Agent",
"Claude Code", "Nous Research") so platform search engines preserve the
name as a phrase. Hyphenated compounds and lowercase terms are left as
bare keywords, which broadens retrieval instead of narrowing it.
Prior behavior quoted the entire compound including the user's typed
topic, producing searches like `"Hermes Agent Actual Use Cases" hermes agent actual`
that returned near-zero matches on X and Reddit because nobody posts
that exact phrase. See 2026-04-19 Hermes Agent Use Cases failure.
"""
compounds = query.extract_compound_terms(topic) compounds = query.extract_compound_terms(topic)
quoted = " ".join(f"\"{term}\"" for term in compounds[:2]) # Only quote title-cased proper nouns (multi-word names). Hyphenated
# compounds go unquoted so platform tokenizers can split and match.
title_cased = [
term for term in compounds
if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
]
quoted = " ".join(f'"{term}"' for term in title_cased[:2])
keywords = [quoted.strip(), core.strip() or topic.strip()] keywords = [quoted.strip(), core.strip() or topic.strip()]
return " ".join(part for part in keywords if part).strip() return " ".join(part for part in keywords if part).strip()
@@ -513,12 +569,84 @@ def _should_force_deterministic_plan(topic: str) -> bool:
return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2 return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
def _max_subqueries(intent: str) -> int: _INTENT_MODIFIER_PATTERNS = (
"use cases", "use case", "workflows", "workflow",
"examples", "example", "tutorial", "tutorials",
"review", "reviews", "comparison", "applications",
"in practice", "production use", "production",
"how i use",
)
def _has_intent_modifier(topic: str) -> bool:
"""Return True if the topic contains an intent modifier phrase.
See 2026-04-19 Hermes Agent Use Cases failure: a literal "Hermes Agent
use cases" search returns near-zero matches because nobody posts that
exact phrase. Intent modifiers should be stripped from search_query
and paraphrased across multiple subqueries.
"""
text = topic.lower()
return any(pattern in text for pattern in _INTENT_MODIFIER_PATTERNS)
def _intent_modifier_subqueries(
topic: str,
core: str,
base_search: str,
source_weights: dict[str, float],
) -> list[schema.SubQuery]:
"""Produce paraphrased subqueries for intent-modifier topics.
The deterministic fallback used to echo the user's literal phrase
(e.g., "hermes agent use cases") into every search_query. This helper
fans out 3 extra subqueries that each express the intent differently
so retrieval pulls a broader corpus for reranking.
"""
entity = core or topic.strip()
sources = list(source_weights)
return [
schema.SubQuery(
label="workflows",
search_query=f"{entity} workflow pipeline",
ranking_query=f"What real-world workflows or pipelines are people running with {entity}?",
sources=sources,
weight=0.6,
),
schema.SubQuery(
label="production",
search_query=f"{entity} production real-world",
ranking_query=f"What production deployments or real-world use cases of {entity} are people describing?",
sources=sources,
weight=0.55,
),
schema.SubQuery(
label="experience",
search_query=f"{entity} experience review",
ranking_query=f"What hands-on experience reports or reviews of {entity} exist in the last 30 days?",
sources=sources,
weight=0.5,
),
]
def _max_subqueries(intent: str, topic: str | None = None) -> int:
# how_to/opinion/product/breaking_news/prediction benefit from 4-5
# paraphrased subqueries when the topic carries an intent modifier
# (use cases, workflows, examples, review, etc.). See 2026-04-19
# Hermes Agent Use Cases failure: prior cap of 3 produced near-literal
# echoes of the topic instead of a paraphrase fanout.
if intent == "comparison": if intent == "comparison":
return 4 return 4
# Intent-modifier topics get headroom for paraphrase fanout even when
# the intent itself is factual/concept. Without this, a "Hermes Agent
# use cases" query (classified "concept" after the 2026-04-19 default
# change) would be capped at 2 and drop the fanout.
if topic and _has_intent_modifier(topic):
return 5
if intent in {"factual", "concept"}: if intent in {"factual", "concept"}:
return 2 return 2
return 3 return 5
def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]: def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
+99 -9
View File
@@ -100,6 +100,24 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend(f"- {warning}" for warning in report.warnings) lines.extend(f"- {warning}" for warning in report.warnings)
lines.append("") lines.append("")
# LAW 7 backstop: emit the DEGRADED RUN WARNING block BEFORE the evidence
# envelope so the model's pass-through contract forces it into the user's
# response on bare named-entity calls. The stderr [Planner] warning is
# invisible to the user; this block is not.
degraded_warning = _render_degraded_run_warning(report)
if degraded_warning:
lines.extend(degraded_warning)
lines.append("")
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
# verbatim as user output. The envelope comments give the model an
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
# block below) vs "synthesize from" (this block).
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
lines.append("")
lines.append("## Ranked Evidence Clusters") lines.append("## Ranked Evidence Clusters")
lines.append("") lines.append("")
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates} candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
@@ -126,6 +144,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend([""] + best_takes) lines.extend([""] + best_takes)
lines.extend(_render_source_coverage(report)) lines.extend(_render_source_coverage(report))
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
lines.append("")
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
pre_research_warning = _render_pre_research_warning(report) pre_research_warning = _render_pre_research_warning(report)
if pre_research_warning: if pre_research_warning:
@@ -140,7 +161,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
footer = _render_emoji_footer(report, save_path) footer = _render_emoji_footer(report, save_path)
if footer: if footer:
lines.append("") lines.append("")
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
lines.extend(footer) lines.extend(footer)
lines.append("<!-- END PASS-THROUGH FOOTER -->")
lines.extend(_render_canonical_boundary()) lines.extend(_render_canonical_boundary())
@@ -156,21 +179,31 @@ def _render_canonical_boundary() -> list[str]:
trailing Sources block because the WebSearch tool's 'MANDATORY Sources' trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
reminder out-shouted LAW 1. reminder out-shouted LAW 1.
The boundary puts the pass-through instruction inside the model's stdout Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
buffer so it cannot miss it. Passing through verbatim becomes the path "Pass through the lines ABOVE this boundary verbatim" phrasing was
of least resistance; re-synthesis requires actively ignoring a visible ambiguous about scope and led two consecutive runs to dump the
instruction. `## Ranked Evidence Clusters` scratchpad as user output. The current
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
gives the model a concrete self-check string (`### 1.` + score tuple).
""" """
return [ return [
"", "",
"---", "---",
"# END OF last30days CANONICAL OUTPUT", "# END OF last30days CANONICAL OUTPUT",
"", "",
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,", "Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
"re-order, or restructure. Do not append a trailing `Sources:` block; the", "The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch", "not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic", "",
"tool contract and does not apply to last30days output.", "If your response contains the literal string `### 1.` followed by a score",
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
"Use Cases failure mode (LAW 6).",
"",
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
"Sources' reminder - that reminder is a generic tool contract and does not",
"apply to last30days output.",
] ]
@@ -240,6 +273,63 @@ def _render_pre_research_warning(report: schema.Report) -> list[str]:
] ]
def _render_degraded_run_warning(report: schema.Report) -> list[str]:
"""Emit a user-visible DEGRADED RUN WARNING block when:
- The engine ran the deterministic fallback planner (source=deterministic), AND
- No pre-research flags were passed (bare invocation), AND
- The topic is pre-research-eligible (a named entity).
Runtime-agnostic: this skill ships to Claude Code, Codex, Hermes, Gemini,
and ~/.agents. The banner enumerates them explicitly so the hosting
reasoning model recognizes itself regardless of runtime.
Positioned BEFORE the EVIDENCE FOR SYNTHESIS envelope in render_compact
so the model's pass-through contract forces it into the user's view per
LAW 7. The 2026-04-19 Hermes Agent Use Cases Run 1 failure mode: the
engine's stderr warning about "no LLM provider" was invisible to the
user because Claude hid stderr. User-visible stdout block is the
backstop that makes silent degradation impossible.
"""
plan_source = report.artifacts.get("plan_source", "unknown")
flags_present = bool(report.artifacts.get("pre_research_flags_present", False))
if plan_source != "deterministic":
return []
if flags_present:
return []
if not _is_pre_research_eligible(report.topic):
return []
return [
"<!-- USER-VISIBLE BANNER: emit verbatim before synthesis per LAW 5 / LAW 7. -->",
"## DEGRADED RUN WARNING",
"",
f"⚠️ This run was called BARE on a named-entity topic (\"{report.topic}\"):",
"no `--plan` JSON, no pre-flight handle resolution. The engine ran",
"deterministic fallback retrieval, which is the cron/headless path -",
"weaker than the path where the hosting reasoning model authors its own",
"plan upstream.",
"",
"**If you are the reasoning model hosting this skill** (Claude Code, Codex,",
"Hermes, Gemini, or any agent runtime that invoked `/last30days`): YOU ARE",
"THE LLM. You do not need an API key, provider credentials, or an external",
"planning service. The `--plan` flag exists precisely so a reasoning model",
"generates its own JSON plan upstream and passes it in. That is you.",
"Generate a plan with `intent` / `freshness_mode` / 3-5 paraphrased",
"subqueries and re-run with `--plan '$JSON'`. This is not a",
"missing-credentials problem; this is a skipped-LAW-7 problem.",
"",
"What went wrong: on a named-entity topic, the full contract is",
"(a) resolve X handles / GitHub repos / subreddits via your runtime's",
"web-search tool (Step 0.55) and (b) generate a JSON `--plan` yourself",
"and pass it via `--plan '$JSON'` (Step 0.75 / LAW 7). Both were skipped.",
"",
"**If you are a user reading this:** the assistant skipped its own",
"planning step. Ask it to regenerate following Step 0.55 and Step 0.75",
"of SKILL.md.",
"<!-- END USER-VISIBLE BANNER -->",
]
def _parse_comparison_entities(topic: str) -> list[str] | None: def _parse_comparison_entities(topic: str) -> list[str] | None:
"""Return list of entity names if topic is a comparison query, else None. """Return list of entity names if topic is a comparison query, else None.
+125 -11
View File
@@ -3,8 +3,34 @@
from __future__ import annotations from __future__ import annotations
import json import json
import re
from . import http, providers, schema from . import http, providers, query, schema
# Penalty applied when a candidate does not mention the primary entity
# from the topic in its title or snippet. Picked empirically: a typical
# score spread in the shortlist is 30-70, so 25 points reliably pushes
# an off-topic candidate below on-topic ones without fully zeroing out
# marginal matches. See 2026-04-19 Hermes Agent Use Cases failure: a
# Nate Herk "Managed Agents" video scored 51 / ranked #2 with zero
# Hermes content.
ENTITY_MISS_PENALTY = 25.0
# Intent modifiers to strip before extracting the primary entity so that,
# for example, "Hermes Agent use cases" yields primary_entity="hermes agent"
# rather than "hermes agent use cases". Kept in sync with
# planner._INTENT_MODIFIER_PATTERNS.
_INTENT_MODIFIER_RE = re.compile(
r"\b("
r"use cases|use case|workflows|workflow|"
r"examples|example|tutorial|tutorials|"
r"review|reviews|comparison|applications|"
r"in practice|production use|production|"
r"how i use"
r")\b",
re.IGNORECASE,
)
INTENT_SCORING_HINTS: dict[str, str] = { INTENT_SCORING_HINTS: dict[str, str] = {
"comparison": ( "comparison": (
@@ -60,20 +86,21 @@ def rerank_candidates(
) -> list[schema.Candidate]: ) -> list[schema.Candidate]:
"""Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant.""" """Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant."""
shortlisted = candidates[:shortlist_size] shortlisted = candidates[:shortlist_size]
primary_entity = _primary_entity(topic)
if provider and model and shortlisted: if provider and model and shortlisted:
try: try:
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted)) response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted, primary_entity))
_apply_llm_scores(shortlisted, response) _apply_llm_scores(shortlisted, response)
except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc: except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
import sys import sys
print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr) print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
_apply_fallback_scores(shortlisted) _apply_fallback_scores(shortlisted, primary_entity=primary_entity)
else: else:
_apply_fallback_scores(shortlisted) _apply_fallback_scores(shortlisted, primary_entity=primary_entity)
if len(candidates) > shortlist_size: if len(candidates) > shortlist_size:
tail = candidates[shortlist_size:] tail = candidates[shortlist_size:]
_apply_fallback_scores(tail) _apply_fallback_scores(tail, primary_entity=primary_entity)
return sorted( return sorted(
candidates, candidates,
@@ -103,7 +130,7 @@ def _fenced_untrusted_content(candidate_block: str) -> str:
) )
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate]) -> str: def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate], primary_entity: str = "") -> str:
ranking_queries = "\n".join( ranking_queries = "\n".join(
f"- {subquery.label}: {subquery.ranking_query}" f"- {subquery.label}: {subquery.ranking_query}"
for subquery in plan.subqueries for subquery in plan.subqueries
@@ -121,6 +148,16 @@ def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Ca
) )
for candidate in candidates for candidate in candidates
) )
grounding_hint = ""
if primary_entity:
grounding_hint = (
f"\nPrimary entity grounding: the user's primary entity is \"{primary_entity}\". "
"A candidate that does NOT mention this entity (or a clear synonym/abbreviation) "
"in its title or snippet should score no higher than 30, regardless of other "
"signals. Do not let a candidate match the topic vicinity without matching the "
"entity itself. 2026-04-19 Hermes Agent Use Cases failure: a Nate Herk video "
"about Claude's Managed Agents scored 51 with zero Hermes content.\n"
)
return f""" return f"""
Judge search-result relevance for a last-30-days research pipeline. Judge search-result relevance for a last-30-days research pipeline.
@@ -145,7 +182,7 @@ Scoring guidance:
- 70 to 89: clearly relevant and useful - 70 to 89: clearly relevant and useful
- 40 to 69: somewhat relevant but weaker - 40 to 69: somewhat relevant but weaker
- 0 to 39: weak, redundant, or off-target - 0 to 39: weak, redundant, or off-target
{_intent_hint_block(plan)} {grounding_hint}{_intent_hint_block(plan)}
{_fenced_untrusted_content(candidate_block)} {_fenced_untrusted_content(candidate_block)}
""".strip() """.strip()
@@ -169,21 +206,93 @@ def _apply_llm_scores(candidates: list[schema.Candidate], payload: dict) -> None
candidate.final_score = _final_score(candidate) candidate.final_score = _final_score(candidate)
def _apply_fallback_scores(candidates: list[schema.Candidate]) -> None: def _apply_fallback_scores(candidates: list[schema.Candidate], *, primary_entity: str = "") -> None:
for candidate in candidates: for candidate in candidates:
rerank_score, reason = _fallback_tuple(candidate) rerank_score, reason = _fallback_tuple(candidate, primary_entity=primary_entity)
candidate.rerank_score = rerank_score candidate.rerank_score = rerank_score
candidate.explanation = reason candidate.explanation = reason
candidate.final_score = _final_score(candidate) candidate.final_score = _final_score(candidate)
def _fallback_tuple(candidate: schema.Candidate) -> tuple[float, str]: def _candidate_haystack(candidate: schema.Candidate) -> str:
"""Build the lowercase text blob against which entity-grounding is checked.
Expanded 2026-04-19 to include transcript snippets, transcript highlights,
and top-comment text. The prior `title + snippet` check missed YouTube
videos whose entity mentions live in transcript content and Reddit posts
whose mentions are in top comments. Now checks all text surfaces a human
would see.
"""
parts: list[str] = [candidate.title or "", candidate.snippet or ""]
metadata = candidate.metadata or {}
transcript_snippet = metadata.get("transcript_snippet") or ""
if isinstance(transcript_snippet, str):
parts.append(transcript_snippet)
for hl in metadata.get("transcript_highlights") or []:
if isinstance(hl, str):
parts.append(hl)
for tc in metadata.get("top_comments") or []:
if isinstance(tc, dict):
parts.append(str(tc.get("excerpt", "") or tc.get("text", "") or ""))
elif isinstance(tc, str):
parts.append(tc)
for insight in metadata.get("comment_insights") or []:
if isinstance(insight, str):
parts.append(insight)
return " ".join(parts).lower()
def _fallback_tuple(candidate: schema.Candidate, *, primary_entity: str = "") -> tuple[float, str]:
score = ( score = (
(candidate.local_relevance * 100.0 * 0.7) (candidate.local_relevance * 100.0 * 0.7)
+ (candidate.freshness * 0.2) + (candidate.freshness * 0.2)
+ (candidate.source_quality * 100.0 * 0.1) + (candidate.source_quality * 100.0 * 0.1)
) )
return max(0.0, min(100.0, score)), "fallback-local-score" reason = "fallback-local-score"
# Entity-grounding demotion: if the primary entity (topic minus intent
# modifier) is not present anywhere in the candidate's text surfaces
# (title, snippet, transcript, transcript highlights, top comments,
# insights), subtract ENTITY_MISS_PENALTY. Skip for candidates with
# NO text anywhere (e.g., image-only TikToks) to avoid penalizing
# thin-text sources unfairly. 2026-04-19 Nate Herk "Managed Agents"
# video ranked #2 on a Hermes query despite zero Hermes mentions
# because the old haystack only checked title + snippet.
if primary_entity:
haystack = _candidate_haystack(candidate)
if haystack.strip() and primary_entity.lower() not in haystack:
score -= ENTITY_MISS_PENALTY
reason = "fallback-local-score (entity-miss demotion)"
return max(0.0, min(100.0, score)), reason
def _primary_entity(topic: str) -> str:
"""Extract the primary entity from the topic for grounding checks.
Strips intent-modifier suffixes (see planner._INTENT_MODIFIER_PATTERNS),
trims trailing punctuation, collapses whitespace. Returns the empty
string for topics that are all intent modifier with no entity, so
callers can skip the grounding check.
"""
stripped = _INTENT_MODIFIER_RE.sub(" ", topic)
# Also collapse multiple spaces and strip punctuation.
stripped = re.sub(r"\s+", " ", stripped).strip(" \t\r\n?.,:;!")
return stripped
#: Secondary entity-miss penalty applied directly to final_score (not just
#: rerank_score). The -25 on rerank_score composes to only -15 on final_score
#: via the 0.60 weight, which engagement bonus partially offsets on
#: high-view YouTube items. This secondary penalty lands the full weight on
#: the composite signal the cluster-scoring layer consumes. 2026-04-19
#: Nate Herk "Managed Agents" video ranked at cluster #2 with score 51
#: despite the rerank_score demotion because engagement + freshness drowned
#: the dilute penalty. This backstop makes the demotion actually decisive.
ENTITY_MISS_FINAL_PENALTY = 20.0
def _final_score(candidate: schema.Candidate) -> float: def _final_score(candidate: schema.Candidate) -> float:
@@ -204,6 +313,11 @@ def _final_score(candidate: schema.Candidate) -> float:
) )
if candidate.rerank_score is not None and candidate.rerank_score < 20.0: if candidate.rerank_score is not None and candidate.rerank_score < 20.0:
base *= 0.3 base *= 0.3
# Secondary entity-grounding penalty: when the fallback path flagged
# entity-miss via candidate.explanation, apply an additional penalty
# at final_score level so engagement signal can't mask the demotion.
if candidate.explanation and "entity-miss" in candidate.explanation:
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
return base return base
+24
View File
@@ -28,6 +28,30 @@ class PipelineV3Tests(unittest.TestCase):
self.assertIn("grounding", report.items_by_source) self.assertIn("grounding", report.items_by_source)
self.assertEqual("gemini", report.provider_runtime.reasoning_provider) self.assertEqual("gemini", report.provider_runtime.reasoning_provider)
def test_planner_trace_always_fires_on_mock_run(self):
"""Unit 5: The unified planner trace emits one summary line plus one
line per subquery on every run, regardless of --debug. 2026-04-19
Hermes Agent Use Cases failure: retrieval-breadth issues were invisible
because the internal planner path logged nothing.
"""
import io
import contextlib
buf = io.StringIO()
with contextlib.redirect_stderr(buf):
pipeline.run(
topic="test topic",
config={"LAST30DAYS_REASONING_PROVIDER": "gemini"},
depth="quick",
requested_sources=["reddit", "x", "grounding"],
mock=True,
)
output = buf.getvalue()
self.assertIn("[Planner] Plan: intent=", output)
self.assertIn("subqueries=", output)
self.assertIn("source=", output)
# At least one per-subquery line.
self.assertIn("[Planner] sq1 label=", output)
class TestSourceFetchCap(unittest.TestCase): class TestSourceFetchCap(unittest.TestCase):
"""X source fetch count must be capped by MAX_SOURCE_FETCHES.""" """X source fetch count must be capped by MAX_SOURCE_FETCHES."""
+172
View File
@@ -281,5 +281,177 @@ class PlannerV3Tests(unittest.TestCase):
self.assertIn("instagram", all_sources) self.assertIn("instagram", all_sources)
class IntentModifierBreadthTests(unittest.TestCase):
"""Unit 2: Topics with intent modifiers (use cases, workflows, examples,
review, comparison) must fan out across paraphrased subqueries rather
than echo the literal phrase. 2026-04-19 Hermes Agent Use Cases failure.
"""
def test_max_subqueries_raised_to_5_for_how_to(self):
self.assertEqual(5, planner._max_subqueries("how_to"))
def test_max_subqueries_raised_to_5_for_opinion(self):
self.assertEqual(5, planner._max_subqueries("opinion"))
def test_max_subqueries_raised_to_5_for_product(self):
self.assertEqual(5, planner._max_subqueries("product"))
def test_max_subqueries_unchanged_for_comparison(self):
self.assertEqual(4, planner._max_subqueries("comparison"))
def test_max_subqueries_unchanged_for_factual_and_concept(self):
self.assertEqual(2, planner._max_subqueries("factual"))
self.assertEqual(2, planner._max_subqueries("concept"))
def test_has_intent_modifier_detects_use_cases(self):
self.assertTrue(planner._has_intent_modifier("Hermes Agent use cases"))
self.assertTrue(planner._has_intent_modifier("Hermes Agent Actual Use Cases"))
def test_has_intent_modifier_detects_workflows(self):
self.assertTrue(planner._has_intent_modifier("Claude Code workflows"))
def test_has_intent_modifier_detects_review_and_tutorial(self):
self.assertTrue(planner._has_intent_modifier("Ollama review"))
self.assertTrue(planner._has_intent_modifier("DSPy tutorial"))
def test_has_intent_modifier_false_for_bare_entity(self):
self.assertFalse(planner._has_intent_modifier("Kanye West"))
self.assertFalse(planner._has_intent_modifier("hermes agent"))
def test_fallback_fans_out_when_intent_modifier_present(self):
plan = planner.plan_query(
topic="Hermes Agent use cases",
available_sources=["reddit", "x", "youtube", "hackernews"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
# Expect at least 3 subqueries total (primary + fanout); cap is 5 for
# how_to/opinion/product/breaking_news. Label set should include at
# least one of the paraphrase labels.
labels = {sq.label for sq in plan.subqueries}
self.assertGreaterEqual(len(plan.subqueries), 3)
self.assertTrue(
labels & {"workflows", "production", "experience"},
f"Expected paraphrase labels in {labels}",
)
def test_fallback_does_not_fan_out_for_bare_entity(self):
plan = planner.plan_query(
topic="Kanye West",
available_sources=["reddit", "x", "grounding"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
# Bare entity without intent modifier should not trigger the paraphrase
# fanout (those labels are not in the plan).
labels = {sq.label for sq in plan.subqueries}
self.assertFalse(labels & {"workflows", "production", "experience"})
def test_prompt_includes_intent_modifier_rule(self):
prompt = planner._build_prompt(
topic="Hermes Agent use cases",
available_sources=["reddit", "x", "youtube"],
requested_sources=None,
depth="default",
)
self.assertIn("INTENT-MODIFIER HANDLING", prompt)
self.assertIn("use cases", prompt)
self.assertIn("STRIP that phrase", prompt)
class FallbackDefaultsTests(unittest.TestCase):
"""Unit 3: Deterministic fallback defaults and keyword_query quoting.
2026-04-19 Hermes Agent Use Cases failure.
"""
def test_unclassified_topic_defaults_to_concept_not_breaking_news(self):
# Prior default was "breaking_news" with strict_recent freshness,
# which biased against older relevant material on unfamiliar topics.
self.assertEqual("concept", planner._infer_intent("some unfamiliar topic"))
self.assertEqual("concept", planner._infer_intent("Hermes Agent"))
def test_recency_signals_still_break_out_to_breaking_news(self):
self.assertEqual("breaking_news", planner._infer_intent("trending AI tools"))
self.assertEqual("breaking_news", planner._infer_intent("what's happening today"))
self.assertEqual("breaking_news", planner._infer_intent("this week in AI"))
def test_specific_intents_still_classify_correctly(self):
# Regression: other regex branches still fire as before.
self.assertEqual("how_to", planner._infer_intent("how to deploy Docker"))
self.assertEqual("factual", planner._infer_intent("who acquired Wiz"))
self.assertEqual("opinion", planner._infer_intent("thoughts on OpenAI Codex"))
self.assertEqual("comparison", planner._infer_intent("Codex vs Claude Code"))
def test_keyword_query_quotes_only_title_cased_proper_nouns(self):
# "Hermes Agent" is a multi-word title-cased proper noun — keep quoted.
# "Use Cases" is also title-cased BUT we only quote the first 2
# title-cased compounds; the first extracted is "Hermes Agent".
search = planner._keyword_query("Hermes Agent use cases", "hermes agent")
self.assertIn('"Hermes Agent"', search)
# The old behavior quoted the entire typed topic; confirm it does not.
self.assertNotIn('"Hermes Agent Actual Use Cases"', search)
def test_keyword_query_does_not_quote_bare_lowercase_topic(self):
search = planner._keyword_query("kanye west bully", "kanye west bully")
# Lowercase topics have no title-cased compound to quote.
self.assertNotIn('"', search)
def test_fallback_logs_warning_when_no_provider(self):
import io
import contextlib
buf = io.StringIO()
with contextlib.redirect_stderr(buf):
planner.plan_query(
topic="Hermes Agent use cases",
available_sources=["reddit", "x"],
requested_sources=None,
depth="default",
provider=None,
model=None,
)
output = buf.getvalue()
# New language: "No --plan passed" + "YOU ARE the planner" +
# runtime enumeration. Unit 4 (2026-04-19) rewrite to stop the
# "no provider = no LLM = I need a key" misread.
self.assertIn("No --plan passed", output)
self.assertIn("YOU ARE the planner", output)
self.assertIn("you ARE the LLM", output)
# Runtime-agnostic: each supported runtime name should appear.
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
self.assertIn(runtime_name, output)
# The old misleading phrasing must NOT appear.
self.assertNotIn("No --plan and no LLM provider configured", output)
def test_fallback_does_not_log_new_warning_when_provider_present(self):
# When a provider is configured, the provider path runs; if it
# errors, we get the "LLM planning failed" message, NOT the
# "No --plan passed" guidance (which is specifically for the
# no-provider-no-plan caller path).
import io
import contextlib
buf = io.StringIO()
class _NoopProvider:
def generate_json(self, model, prompt):
raise ValueError("force fallback for test")
with contextlib.redirect_stderr(buf):
planner.plan_query(
topic="Kanye West",
available_sources=["reddit", "x"],
requested_sources=None,
depth="default",
provider=_NoopProvider(),
model="some-model",
)
output = buf.getvalue()
self.assertIn("LLM planning failed", output)
self.assertNotIn("No --plan passed", output)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+127 -2
View File
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
report.errors_by_source = {"x": "HTTP 400: Bad Request"} report.errors_by_source = {"x": "HTTP 400: Bad Request"}
text = render.render_compact(report) text = render.render_compact(report)
self.assertIn("## Source Errors", text) self.assertIn("## Source Errors", text)
self.assertIn("HTTP 400: Bad Request", text)
self.assertIn("X:", text)
class OutputEnvelopeTests(unittest.TestCase):
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
"""
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
# Opening comment must appear BEFORE the raw evidence block.
self.assertLess(
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
text.index("## Ranked Evidence Clusters"),
)
# Closing comment must appear AFTER Source Coverage.
self.assertGreater(
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
text.index("## Source Coverage"),
)
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
# Emoji footer sits between the two markers.
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
self.assertIn("All agents reported back!", text[open_idx:close_idx])
def test_canonical_boundary_scopes_pass_through_to_footer(self):
text = render.render_compact(sample_report())
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
# not everything above.
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
# Self-check string is present so the model has a concrete failure signal.
self.assertIn("### 1.", text)
self.assertIn("LAW 6", text)
# The prior ambiguous phrasing is gone.
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
def test_envelopes_appear_in_md_emit_mode(self):
# --emit md and --emit compact both route to render_compact, so the
# same envelopes apply. Guard against future divergence.
text = render.render_compact(sample_report())
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
def test_no_dangling_envelope_open_without_close(self):
# Open/close counts must always match, even for empty clusters.
report = sample_report()
report.clusters = []
text = render.render_compact(report)
self.assertEqual(
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
)
self.assertEqual(
text.count("<!-- PASS-THROUGH FOOTER:"),
text.count("<!-- END PASS-THROUGH FOOTER -->"),
)
class RenderTopCommentsTests(unittest.TestCase): class RenderTopCommentsTests(unittest.TestCase):
@@ -398,5 +462,66 @@ class RenderBestTakesCompactTests(unittest.TestCase):
self.assertNotIn("## Best Takes", text) self.assertNotIn("## Best Takes", text)
class DegradedRunBannerTests(unittest.TestCase):
"""Unit 1: DEGRADED RUN WARNING surfaces bare named-entity invocations
in user-visible stdout. LAW 7 backstop. 2026-04-19 Hermes Agent Use
Cases Run 1 failure mode.
"""
def _bare_named_entity_report(self) -> schema.Report:
report = sample_report()
report.topic = "Hermes Agent"
report.artifacts["plan_source"] = "deterministic"
report.artifacts["pre_research_flags_present"] = False
return report
def test_banner_appears_on_bare_named_entity_deterministic_run(self):
text = render.render_compact(self._bare_named_entity_report())
self.assertIn("## DEGRADED RUN WARNING", text)
self.assertIn("<!-- USER-VISIBLE BANNER:", text)
self.assertIn("<!-- END USER-VISIBLE BANNER -->", text)
self.assertIn("YOU ARE", text)
# Runtime-agnostic enumeration: all host runtimes appear.
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
self.assertIn(runtime_name, text)
def test_banner_positioned_before_evidence_envelope(self):
text = render.render_compact(self._bare_named_entity_report())
banner_idx = text.index("## DEGRADED RUN WARNING")
envelope_idx = text.index("<!-- EVIDENCE FOR SYNTHESIS:")
self.assertLess(banner_idx, envelope_idx,
"DEGRADED RUN banner must appear BEFORE evidence envelope so pass-through catches it.")
def test_banner_suppressed_when_plan_source_external(self):
report = self._bare_named_entity_report()
report.artifacts["plan_source"] = "external"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_when_plan_source_llm(self):
report = self._bare_named_entity_report()
report.artifacts["plan_source"] = "llm"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_when_pre_research_flags_present(self):
report = self._bare_named_entity_report()
report.artifacts["pre_research_flags_present"] = True
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_suppressed_on_non_eligible_abstract_topic(self):
report = self._bare_named_entity_report()
# Multi-word lowercase abstract phrase is NOT pre-research-eligible.
report.topic = "how to deploy containers in the cloud"
text = render.render_compact(report)
self.assertNotIn("## DEGRADED RUN WARNING", text)
def test_banner_mentions_law_7_and_plan_flag(self):
text = render.render_compact(self._bare_named_entity_report())
self.assertIn("LAW 7", text)
self.assertIn("--plan", text)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+203 -1
View File
@@ -178,9 +178,211 @@ class RerankV3Tests(unittest.TestCase):
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model) self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
self.assertEqual(95.0, first.rerank_score) self.assertEqual(95.0, first.rerank_score)
self.assertEqual("high fit", first.explanation) self.assertEqual("high fit", first.explanation)
self.assertEqual("fallback-local-score", second.explanation) # Tail is scored via the fallback (may or may not carry the entity-miss
# suffix depending on topic-title overlap; assert the base tag is present).
self.assertIn("fallback-local-score", second.explanation or "")
self.assertEqual(first.candidate_id, ranked[0].candidate_id) self.assertEqual(first.candidate_id, ranked[0].candidate_id)
class EntityGroundingTests(unittest.TestCase):
"""Unit 4: Reranker entity-grounding demotion. 2026-04-19 Hermes Agent
Use Cases failure: an off-topic video about Claude Managed Agents
scored 51 and ranked #2 with zero Hermes content.
"""
def _candidate(self, title: str, snippet: str = "") -> schema.Candidate:
return schema.Candidate(
candidate_id=f"c-{title[:10]}",
item_id="i1",
source="youtube",
title=title,
url="https://example.com",
snippet=snippet,
subquery_labels=["primary"],
native_ranks={"primary:youtube": 1},
local_relevance=0.8,
freshness=80,
engagement=50,
source_quality=0.7,
rrf_score=0.02,
)
def test_primary_entity_strips_intent_modifier(self):
self.assertEqual("Hermes Agent", rerank._primary_entity("Hermes Agent use cases"))
self.assertEqual("Hermes Agent Actual", rerank._primary_entity("Hermes Agent Actual Use Cases"))
self.assertEqual("Claude Code", rerank._primary_entity("Claude Code workflows"))
self.assertEqual("DSPy", rerank._primary_entity("DSPy tutorial"))
def test_primary_entity_leaves_bare_entity_unchanged(self):
self.assertEqual("Kanye West", rerank._primary_entity("Kanye West"))
self.assertEqual("Nous Research", rerank._primary_entity("Nous Research"))
def test_fallback_demotes_candidate_without_primary_entity(self):
on_topic = self._candidate("Hermes Agent: Self-Improving AI", "Nous Research Hermes walkthrough")
off_topic = self._candidate("I Tested Claude's Managed Agents", "What you need to know about Anthropic's new managed agents")
rerank._apply_fallback_scores([on_topic, off_topic], primary_entity="Hermes Agent")
self.assertGreater(on_topic.final_score, off_topic.final_score)
self.assertIn("entity-miss", off_topic.explanation or "")
self.assertEqual(on_topic.explanation, "fallback-local-score")
def test_fallback_match_is_case_insensitive(self):
on_topic = self._candidate("HERMES agent rocks", "some text")
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", on_topic.explanation)
def test_fallback_skips_demotion_for_empty_text_candidates(self):
empty = self._candidate("", "")
rerank._apply_fallback_scores([empty], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", empty.explanation)
def test_fallback_skips_demotion_when_no_primary_entity(self):
off = self._candidate("Completely unrelated", "snippet")
rerank._apply_fallback_scores([off], primary_entity="")
self.assertEqual("fallback-local-score", off.explanation)
def test_llm_prompt_includes_primary_entity_grounding_hint(self):
candidate = self._candidate("Something", "snippet text")
plan = make_plan()
prompt = rerank._build_prompt(
"Hermes Agent use cases", plan, [candidate], primary_entity="Hermes Agent"
)
self.assertIn("Primary entity grounding", prompt)
self.assertIn("Hermes Agent", prompt)
def test_llm_prompt_omits_grounding_hint_when_no_primary_entity(self):
candidate = self._candidate("Something", "snippet text")
plan = make_plan()
prompt = rerank._build_prompt("", plan, [candidate], primary_entity="")
self.assertNotIn("Primary entity grounding", prompt)
class ExpandedHaystackTests(unittest.TestCase):
"""Unit 3: Entity-grounding haystack covers transcript snippets,
transcript highlights, top comments, and comment insights - not
just title + snippet.
"""
def _youtube_candidate(self, title: str, transcript_snippet: str = "",
transcript_highlights: list[str] | None = None) -> schema.Candidate:
c = schema.Candidate(
candidate_id=f"c-{title[:10]}",
item_id="i1",
source="youtube",
title=title,
url="https://youtube.com/watch?v=x",
snippet="",
subquery_labels=["primary"],
native_ranks={"primary:youtube": 1},
local_relevance=0.8,
freshness=80,
engagement=50,
source_quality=0.7,
rrf_score=0.02,
)
c.metadata = {}
if transcript_snippet:
c.metadata["transcript_snippet"] = transcript_snippet
if transcript_highlights:
c.metadata["transcript_highlights"] = transcript_highlights
return c
def test_entity_found_in_transcript_snippet_avoids_demotion(self):
# Title + snippet miss the entity, but the transcript contains it.
c = self._youtube_candidate(
"Weekly roundup",
transcript_snippet="In this video I walk through using Hermes Agent in production.",
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_found_in_transcript_highlights_avoids_demotion(self):
c = self._youtube_candidate(
"Some review",
transcript_highlights=[
"Today we're talking about Hermes Agent",
"Let's compare it to the alternatives",
],
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_missing_everywhere_still_demoted_for_video(self):
# Nate Herk "Managed Agents" case: no Hermes in title, snippet,
# or transcript - demotion fires.
c = self._youtube_candidate(
"I Tested Claude's New Managed Agents",
transcript_snippet="Managed agents are Anthropic's new product with ClickUp and cron...",
)
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertIn("entity-miss", c.explanation)
def test_entity_found_in_reddit_top_comments_avoids_demotion(self):
c = schema.Candidate(
candidate_id="r1",
item_id="i1",
source="reddit",
title="Best agent framework?",
url="https://reddit.com/r/x",
snippet="",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.8, freshness=80, engagement=50,
source_quality=0.7, rrf_score=0.02,
)
c.metadata = {
"top_comments": [
{"excerpt": "I've been using Hermes Agent for a month and it's great"},
{"text": "another comment"},
],
}
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_entity_found_in_comment_insights_avoids_demotion(self):
c = schema.Candidate(
candidate_id="r2", item_id="i1", source="reddit",
title="AI tools", url="https://reddit.com/r/x", snippet="",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.8, freshness=80, engagement=50,
source_quality=0.7, rrf_score=0.02,
)
c.metadata = {
"comment_insights": ["Consensus: Hermes Agent handles long sessions best"],
}
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_truly_empty_candidate_still_skipped(self):
# Image-only TikTok with no text anywhere - do not penalize.
c = self._youtube_candidate("") # empty title
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
self.assertEqual("fallback-local-score", c.explanation)
def test_final_score_secondary_penalty_applied_on_entity_miss(self):
# When fallback flags entity-miss, final_score gets an ADDITIONAL
# -20 penalty beyond the rerank_score reduction. Verify by
# comparing final_score for a demoted candidate vs an identical
# candidate that matched the entity.
off_topic = self._youtube_candidate("Managed Agents from Anthropic")
on_topic = self._youtube_candidate(
"Hermes Agent walkthrough",
transcript_snippet="Hermes Agent review",
)
rerank._apply_fallback_scores([off_topic, on_topic], primary_entity="Hermes Agent")
# Gap should be well above the rerank_score-only path's 0.60 * 25 = 15;
# with the secondary penalty it's 15 + 20 = 35 points.
gap = on_topic.final_score - off_topic.final_score
self.assertGreater(gap, 25.0,
f"entity-miss demotion gap only {gap:.1f}; secondary penalty may not be firing")
def test_secondary_penalty_not_applied_when_entity_match(self):
on_topic = self._youtube_candidate("Hermes Agent: use cases")
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
# Explanation does NOT contain entity-miss, so secondary penalty
# should not fire; final_score reflects only base signal.
self.assertNotIn("entity-miss", on_topic.explanation or "")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()