Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth
fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
This commit is contained in:
Matt Van Horn
@@ -120,7 +120,49 @@ These five rules dominate every other rule in this file. If you find yourself ab
|
||||
|
||||
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
|
||||
|
||||
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
||||
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
||||
|
||||
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
|
||||
|
||||
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
|
||||
|
||||
**Worked example (LAW 6 transformation).** Evidence block you read:
|
||||
|
||||
```
|
||||
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
|
||||
## Ranked Evidence Clusters
|
||||
|
||||
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
|
||||
|
||||
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
|
||||
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
|
||||
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||
- "Can you tell me what type of user profile you have on me?"
|
||||
|
||||
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
|
||||
|
||||
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
|
||||
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||
<!-- END EVIDENCE FOR SYNTHESIS -->
|
||||
```
|
||||
|
||||
Output you emit (prose synthesis, NOT the evidence block):
|
||||
|
||||
```
|
||||
What I learned:
|
||||
|
||||
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||
|
||||
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||
```
|
||||
|
||||
**LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan '$JSON'`.
|
||||
|
||||
Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan '$JSON'`. A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema).
|
||||
|
||||
**Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step.
|
||||
|
||||
**Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan '$JSON'`? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider.
|
||||
|
||||
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
|
||||
|
||||
|
||||
+34
-1
@@ -204,7 +204,7 @@ def run(
|
||||
plan = planner._sanitize_plan(
|
||||
external_plan, topic, available, requested_sources, depth,
|
||||
)
|
||||
print(f"[Planner] Using external plan ({len(plan.subqueries)} subqueries)", file=sys.stderr)
|
||||
plan_source = "external"
|
||||
else:
|
||||
plan = planner.plan_query(
|
||||
topic=topic,
|
||||
@@ -215,6 +215,14 @@ def run(
|
||||
model=None if mock else runtime.planner_model,
|
||||
context=config.get("_auto_resolve_context", ""),
|
||||
)
|
||||
# Source labelling: the fallback path annotates notes with "fallback-plan"
|
||||
# or "deterministic-comparison-plan"; anything else came from the LLM.
|
||||
if any("fallback" in note or "deterministic" in note for note in (plan.notes or [])):
|
||||
plan_source = "deterministic"
|
||||
elif not mock and reasoning_provider and runtime.planner_model:
|
||||
plan_source = "llm"
|
||||
else:
|
||||
plan_source = "deterministic"
|
||||
|
||||
# Safety net: ensure grounding appears in all subqueries even if the planner
|
||||
# omits it. This is redundant when the planner includes grounding via
|
||||
@@ -224,7 +232,32 @@ def run(
|
||||
if "grounding" not in sq.sources:
|
||||
sq.sources.append("grounding")
|
||||
|
||||
# Always-on planner trace. Emits one summary line plus one per subquery
|
||||
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
|
||||
# disaster are visible without --debug. Stderr only; does not leak into
|
||||
# the user-facing stdout synthesis.
|
||||
print(
|
||||
f"[Planner] Plan: intent={plan.intent}, freshness={plan.freshness_mode}, "
|
||||
f"cluster_mode={plan.cluster_mode}, subqueries={len(plan.subqueries)}, "
|
||||
f"source={plan_source}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if plan.subqueries:
|
||||
for index, sq in enumerate(plan.subqueries, start=1):
|
||||
sources_str = ",".join(sq.sources) if sq.sources else "(none)"
|
||||
print(
|
||||
f"[Planner] sq{index} label={sq.label} "
|
||||
f'search="{sq.search_query}" sources=[{sources_str}]',
|
||||
file=sys.stderr,
|
||||
)
|
||||
else:
|
||||
print("[Planner] (no subqueries in plan)", file=sys.stderr)
|
||||
|
||||
bundle = schema.RetrievalBundle(artifacts={"grounding": []})
|
||||
# Expose plan_source to the renderer so render_compact can emit the
|
||||
# DEGRADED RUN banner when a named-entity topic was invoked bare
|
||||
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
|
||||
bundle.artifacts["plan_source"] = plan_source
|
||||
|
||||
# Project-mode or person-mode GitHub: run once before the main subquery loop
|
||||
_github_custom_done = False
|
||||
|
||||
+134
-6
@@ -113,6 +113,25 @@ def plan_query(
|
||||
topic, available_sources, requested_sources, depth,
|
||||
note=f"fallback-plan (LLM error: {type(exc).__name__})",
|
||||
)
|
||||
# No --plan was passed and no engine-internal provider is configured.
|
||||
# The deterministic fallback path produces weaker plans than either the
|
||||
# --plan path or the engine-internal LLM planner path. Surface this so
|
||||
# the HOSTING reasoning model (Claude Code, Codex, Hermes, Gemini, etc.)
|
||||
# realizes it is supposed to generate the plan itself and pass --plan.
|
||||
# The word "provider" in engine messages refers to the engine's INTERNAL
|
||||
# planner credentials - NOT a prerequisite the caller needs. If you are
|
||||
# the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
|
||||
# Hermes Agent Use Cases failure mode.
|
||||
import sys
|
||||
print(
|
||||
"[Planner] No --plan passed. If you are the reasoning model hosting "
|
||||
"this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime), "
|
||||
"YOU ARE the planner: generate a JSON query plan yourself and pass it "
|
||||
"via --plan. You do not need an API key or credentials; you ARE the "
|
||||
"LLM. The deterministic fallback below is the headless/cron path only. "
|
||||
"See LAW 7 in SKILL.md and Step 0.75 for the plan schema.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return _fallback_plan(topic, available_sources, requested_sources, depth)
|
||||
|
||||
|
||||
@@ -151,7 +170,7 @@ Return JSON only with this shape:
|
||||
}}
|
||||
|
||||
Rules:
|
||||
- emit 1 to 4 subqueries
|
||||
- emit 1 to 5 subqueries (how_to/opinion/product/breaking_news intents benefit from 4-5; factual/concept from 2)
|
||||
- every subquery must include both search_query and ranking_query
|
||||
- sources must be drawn from Available sources only
|
||||
- use cluster_mode=none for factual or many how-to queries
|
||||
@@ -162,6 +181,8 @@ Rules:
|
||||
- preserve exact proper nouns and entity strings from the topic
|
||||
- NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
|
||||
- NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
|
||||
- INTENT-MODIFIER HANDLING: when the topic contains one of {{use cases, use case, workflows, workflow, examples, tutorial, tutorials, review, reviews, comparison, applications, in practice, production, production use, how i use}}, STRIP that phrase from every search_query (keep its meaning in ranking_query). Emit 4-5 paraphrased subqueries that each express the intent differently (e.g., 'production', 'workflow OR pipeline', 'review OR experience', 'vs COMPETITOR', 'community discussion'). Broad retrieval, narrow ranking. This was the 2026-04-19 Hermes Agent Use Cases failure mode: the planner echoed "hermes agent use cases" as a literal search string and returned near-zero results because nobody posts that exact phrase.
|
||||
- DO NOT quote the user's full topic verbatim in search_query. Quote only multi-word proper nouns like "Hermes Agent", "Claude Code", "Nous Research". Bare keywords OR'd together retrieve more than exact-phrase searches.
|
||||
- search_query should match how content is TITLED on platforms
|
||||
- GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
|
||||
""".strip()
|
||||
@@ -204,7 +225,7 @@ def _sanitize_plan(
|
||||
source_weights = _normalize_weights(source_weights)
|
||||
|
||||
subqueries: list[schema.SubQuery] = []
|
||||
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint)], start=1):
|
||||
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint, topic)], start=1):
|
||||
if not isinstance(subquery, dict):
|
||||
continue
|
||||
sources = [source for source in subquery.get("sources") or [] if source in source_weights]
|
||||
@@ -382,13 +403,22 @@ def _fallback_plan(
|
||||
)
|
||||
)
|
||||
|
||||
# Intent-modifier fanout: when topic contains a phrase like "use cases",
|
||||
# "workflows", "examples", "review" (see _INTENT_MODIFIER_PATTERNS),
|
||||
# paraphrase the intent across 3 extra subqueries rather than echoing
|
||||
# the literal phrase. Fixes 2026-04-19 Hermes Agent Use Cases failure.
|
||||
# Excluded for comparison/prediction since those already have dedicated
|
||||
# fanout (entity-per-subquery / odds).
|
||||
if depth != "quick" and intent not in {"comparison", "prediction"} and _has_intent_modifier(topic):
|
||||
subqueries.extend(_intent_modifier_subqueries(topic, core, base_search, source_weights))
|
||||
|
||||
return schema.QueryPlan(
|
||||
intent=intent,
|
||||
freshness_mode=_default_freshness(intent),
|
||||
cluster_mode=_default_cluster_mode(intent),
|
||||
raw_topic=topic,
|
||||
subqueries=_normalize_subquery_weights(
|
||||
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent)], intent, depth, list(source_weights))
|
||||
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent, topic)], intent, depth, list(source_weights))
|
||||
),
|
||||
source_weights=_normalize_weights(source_weights),
|
||||
notes=[note],
|
||||
@@ -418,7 +448,15 @@ def _infer_intent(topic: str) -> str:
|
||||
return "concept"
|
||||
if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
|
||||
return "breaking_news"
|
||||
# Recency signals take priority when nothing more specific matched.
|
||||
if re.search(r"\b(trending|this week|right now|today|this month)\b", text):
|
||||
return "breaking_news"
|
||||
# Default changed from "breaking_news" to "concept" on 2026-04-19 after
|
||||
# the Hermes Agent Use Cases failure: unclassified topics were getting
|
||||
# strict_recent freshness, which over-weighted the last 7 days and
|
||||
# under-weighted older relevant material. "concept" defaults to
|
||||
# evergreen_ok freshness, a safer posture for unknown topics.
|
||||
return "concept"
|
||||
|
||||
|
||||
def _default_freshness(intent: str) -> str:
|
||||
@@ -464,8 +502,26 @@ def _default_source_weights(intent: str, sources: list[str]) -> dict[str, float]
|
||||
|
||||
|
||||
def _keyword_query(topic: str, core: str) -> str:
|
||||
"""Build a search_query string for the deterministic fallback.
|
||||
|
||||
Quote ONLY title-cased multi-word proper nouns ("Hermes Agent",
|
||||
"Claude Code", "Nous Research") so platform search engines preserve the
|
||||
name as a phrase. Hyphenated compounds and lowercase terms are left as
|
||||
bare keywords, which broadens retrieval instead of narrowing it.
|
||||
|
||||
Prior behavior quoted the entire compound including the user's typed
|
||||
topic, producing searches like `"Hermes Agent Actual Use Cases" hermes agent actual`
|
||||
that returned near-zero matches on X and Reddit because nobody posts
|
||||
that exact phrase. See 2026-04-19 Hermes Agent Use Cases failure.
|
||||
"""
|
||||
compounds = query.extract_compound_terms(topic)
|
||||
quoted = " ".join(f"\"{term}\"" for term in compounds[:2])
|
||||
# Only quote title-cased proper nouns (multi-word names). Hyphenated
|
||||
# compounds go unquoted so platform tokenizers can split and match.
|
||||
title_cased = [
|
||||
term for term in compounds
|
||||
if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
|
||||
]
|
||||
quoted = " ".join(f'"{term}"' for term in title_cased[:2])
|
||||
keywords = [quoted.strip(), core.strip() or topic.strip()]
|
||||
return " ".join(part for part in keywords if part).strip()
|
||||
|
||||
@@ -513,12 +569,84 @@ def _should_force_deterministic_plan(topic: str) -> bool:
|
||||
return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
|
||||
|
||||
|
||||
def _max_subqueries(intent: str) -> int:
|
||||
_INTENT_MODIFIER_PATTERNS = (
|
||||
"use cases", "use case", "workflows", "workflow",
|
||||
"examples", "example", "tutorial", "tutorials",
|
||||
"review", "reviews", "comparison", "applications",
|
||||
"in practice", "production use", "production",
|
||||
"how i use",
|
||||
)
|
||||
|
||||
|
||||
def _has_intent_modifier(topic: str) -> bool:
|
||||
"""Return True if the topic contains an intent modifier phrase.
|
||||
|
||||
See 2026-04-19 Hermes Agent Use Cases failure: a literal "Hermes Agent
|
||||
use cases" search returns near-zero matches because nobody posts that
|
||||
exact phrase. Intent modifiers should be stripped from search_query
|
||||
and paraphrased across multiple subqueries.
|
||||
"""
|
||||
text = topic.lower()
|
||||
return any(pattern in text for pattern in _INTENT_MODIFIER_PATTERNS)
|
||||
|
||||
|
||||
def _intent_modifier_subqueries(
|
||||
topic: str,
|
||||
core: str,
|
||||
base_search: str,
|
||||
source_weights: dict[str, float],
|
||||
) -> list[schema.SubQuery]:
|
||||
"""Produce paraphrased subqueries for intent-modifier topics.
|
||||
|
||||
The deterministic fallback used to echo the user's literal phrase
|
||||
(e.g., "hermes agent use cases") into every search_query. This helper
|
||||
fans out 3 extra subqueries that each express the intent differently
|
||||
so retrieval pulls a broader corpus for reranking.
|
||||
"""
|
||||
entity = core or topic.strip()
|
||||
sources = list(source_weights)
|
||||
return [
|
||||
schema.SubQuery(
|
||||
label="workflows",
|
||||
search_query=f"{entity} workflow pipeline",
|
||||
ranking_query=f"What real-world workflows or pipelines are people running with {entity}?",
|
||||
sources=sources,
|
||||
weight=0.6,
|
||||
),
|
||||
schema.SubQuery(
|
||||
label="production",
|
||||
search_query=f"{entity} production real-world",
|
||||
ranking_query=f"What production deployments or real-world use cases of {entity} are people describing?",
|
||||
sources=sources,
|
||||
weight=0.55,
|
||||
),
|
||||
schema.SubQuery(
|
||||
label="experience",
|
||||
search_query=f"{entity} experience review",
|
||||
ranking_query=f"What hands-on experience reports or reviews of {entity} exist in the last 30 days?",
|
||||
sources=sources,
|
||||
weight=0.5,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _max_subqueries(intent: str, topic: str | None = None) -> int:
|
||||
# how_to/opinion/product/breaking_news/prediction benefit from 4-5
|
||||
# paraphrased subqueries when the topic carries an intent modifier
|
||||
# (use cases, workflows, examples, review, etc.). See 2026-04-19
|
||||
# Hermes Agent Use Cases failure: prior cap of 3 produced near-literal
|
||||
# echoes of the topic instead of a paraphrase fanout.
|
||||
if intent == "comparison":
|
||||
return 4
|
||||
# Intent-modifier topics get headroom for paraphrase fanout even when
|
||||
# the intent itself is factual/concept. Without this, a "Hermes Agent
|
||||
# use cases" query (classified "concept" after the 2026-04-19 default
|
||||
# change) would be capped at 2 and drop the fanout.
|
||||
if topic and _has_intent_modifier(topic):
|
||||
return 5
|
||||
if intent in {"factual", "concept"}:
|
||||
return 2
|
||||
return 3
|
||||
return 5
|
||||
|
||||
|
||||
def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
|
||||
|
||||
+99
-9
@@ -100,6 +100,24 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
||||
lines.extend(f"- {warning}" for warning in report.warnings)
|
||||
lines.append("")
|
||||
|
||||
# LAW 7 backstop: emit the DEGRADED RUN WARNING block BEFORE the evidence
|
||||
# envelope so the model's pass-through contract forces it into the user's
|
||||
# response on bare named-entity calls. The stderr [Planner] warning is
|
||||
# invisible to the user; this block is not.
|
||||
degraded_warning = _render_degraded_run_warning(report)
|
||||
if degraded_warning:
|
||||
lines.extend(degraded_warning)
|
||||
lines.append("")
|
||||
|
||||
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
|
||||
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
|
||||
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
|
||||
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
|
||||
# verbatim as user output. The envelope comments give the model an
|
||||
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
|
||||
# block below) vs "synthesize from" (this block).
|
||||
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
|
||||
lines.append("")
|
||||
lines.append("## Ranked Evidence Clusters")
|
||||
lines.append("")
|
||||
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
||||
@@ -126,6 +144,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
||||
lines.extend([""] + best_takes)
|
||||
|
||||
lines.extend(_render_source_coverage(report))
|
||||
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
|
||||
lines.append("")
|
||||
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
|
||||
|
||||
pre_research_warning = _render_pre_research_warning(report)
|
||||
if pre_research_warning:
|
||||
@@ -140,7 +161,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
||||
footer = _render_emoji_footer(report, save_path)
|
||||
if footer:
|
||||
lines.append("")
|
||||
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
|
||||
lines.extend(footer)
|
||||
lines.append("<!-- END PASS-THROUGH FOOTER -->")
|
||||
|
||||
lines.extend(_render_canonical_boundary())
|
||||
|
||||
@@ -156,21 +179,31 @@ def _render_canonical_boundary() -> list[str]:
|
||||
trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
|
||||
reminder out-shouted LAW 1.
|
||||
|
||||
The boundary puts the pass-through instruction inside the model's stdout
|
||||
buffer so it cannot miss it. Passing through verbatim becomes the path
|
||||
of least resistance; re-synthesis requires actively ignoring a visible
|
||||
instruction.
|
||||
Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
|
||||
"Pass through the lines ABOVE this boundary verbatim" phrasing was
|
||||
ambiguous about scope and led two consecutive runs to dump the
|
||||
`## Ranked Evidence Clusters` scratchpad as user output. The current
|
||||
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
|
||||
gives the model a concrete self-check string (`### 1.` + score tuple).
|
||||
"""
|
||||
return [
|
||||
"",
|
||||
"---",
|
||||
"# END OF last30days CANONICAL OUTPUT",
|
||||
"",
|
||||
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,",
|
||||
"re-order, or restructure. Do not append a trailing `Sources:` block; the",
|
||||
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch",
|
||||
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic",
|
||||
"tool contract and does not apply to last30days output.",
|
||||
"Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
|
||||
"The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
|
||||
"not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
|
||||
"",
|
||||
"If your response contains the literal string `### 1.` followed by a score",
|
||||
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
|
||||
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
|
||||
"Use Cases failure mode (LAW 6).",
|
||||
"",
|
||||
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
|
||||
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
|
||||
"Sources' reminder - that reminder is a generic tool contract and does not",
|
||||
"apply to last30days output.",
|
||||
]
|
||||
|
||||
|
||||
@@ -240,6 +273,63 @@ def _render_pre_research_warning(report: schema.Report) -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def _render_degraded_run_warning(report: schema.Report) -> list[str]:
|
||||
"""Emit a user-visible DEGRADED RUN WARNING block when:
|
||||
- The engine ran the deterministic fallback planner (source=deterministic), AND
|
||||
- No pre-research flags were passed (bare invocation), AND
|
||||
- The topic is pre-research-eligible (a named entity).
|
||||
|
||||
Runtime-agnostic: this skill ships to Claude Code, Codex, Hermes, Gemini,
|
||||
and ~/.agents. The banner enumerates them explicitly so the hosting
|
||||
reasoning model recognizes itself regardless of runtime.
|
||||
|
||||
Positioned BEFORE the EVIDENCE FOR SYNTHESIS envelope in render_compact
|
||||
so the model's pass-through contract forces it into the user's view per
|
||||
LAW 7. The 2026-04-19 Hermes Agent Use Cases Run 1 failure mode: the
|
||||
engine's stderr warning about "no LLM provider" was invisible to the
|
||||
user because Claude hid stderr. User-visible stdout block is the
|
||||
backstop that makes silent degradation impossible.
|
||||
"""
|
||||
plan_source = report.artifacts.get("plan_source", "unknown")
|
||||
flags_present = bool(report.artifacts.get("pre_research_flags_present", False))
|
||||
if plan_source != "deterministic":
|
||||
return []
|
||||
if flags_present:
|
||||
return []
|
||||
if not _is_pre_research_eligible(report.topic):
|
||||
return []
|
||||
|
||||
return [
|
||||
"<!-- USER-VISIBLE BANNER: emit verbatim before synthesis per LAW 5 / LAW 7. -->",
|
||||
"## DEGRADED RUN WARNING",
|
||||
"",
|
||||
f"⚠️ This run was called BARE on a named-entity topic (\"{report.topic}\"):",
|
||||
"no `--plan` JSON, no pre-flight handle resolution. The engine ran",
|
||||
"deterministic fallback retrieval, which is the cron/headless path -",
|
||||
"weaker than the path where the hosting reasoning model authors its own",
|
||||
"plan upstream.",
|
||||
"",
|
||||
"**If you are the reasoning model hosting this skill** (Claude Code, Codex,",
|
||||
"Hermes, Gemini, or any agent runtime that invoked `/last30days`): YOU ARE",
|
||||
"THE LLM. You do not need an API key, provider credentials, or an external",
|
||||
"planning service. The `--plan` flag exists precisely so a reasoning model",
|
||||
"generates its own JSON plan upstream and passes it in. That is you.",
|
||||
"Generate a plan with `intent` / `freshness_mode` / 3-5 paraphrased",
|
||||
"subqueries and re-run with `--plan '$JSON'`. This is not a",
|
||||
"missing-credentials problem; this is a skipped-LAW-7 problem.",
|
||||
"",
|
||||
"What went wrong: on a named-entity topic, the full contract is",
|
||||
"(a) resolve X handles / GitHub repos / subreddits via your runtime's",
|
||||
"web-search tool (Step 0.55) and (b) generate a JSON `--plan` yourself",
|
||||
"and pass it via `--plan '$JSON'` (Step 0.75 / LAW 7). Both were skipped.",
|
||||
"",
|
||||
"**If you are a user reading this:** the assistant skipped its own",
|
||||
"planning step. Ask it to regenerate following Step 0.55 and Step 0.75",
|
||||
"of SKILL.md.",
|
||||
"<!-- END USER-VISIBLE BANNER -->",
|
||||
]
|
||||
|
||||
|
||||
def _parse_comparison_entities(topic: str) -> list[str] | None:
|
||||
"""Return list of entity names if topic is a comparison query, else None.
|
||||
|
||||
|
||||
+125
-11
@@ -3,8 +3,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from . import http, providers, schema
|
||||
from . import http, providers, query, schema
|
||||
|
||||
|
||||
# Penalty applied when a candidate does not mention the primary entity
|
||||
# from the topic in its title or snippet. Picked empirically: a typical
|
||||
# score spread in the shortlist is 30-70, so 25 points reliably pushes
|
||||
# an off-topic candidate below on-topic ones without fully zeroing out
|
||||
# marginal matches. See 2026-04-19 Hermes Agent Use Cases failure: a
|
||||
# Nate Herk "Managed Agents" video scored 51 / ranked #2 with zero
|
||||
# Hermes content.
|
||||
ENTITY_MISS_PENALTY = 25.0
|
||||
|
||||
# Intent modifiers to strip before extracting the primary entity so that,
|
||||
# for example, "Hermes Agent use cases" yields primary_entity="hermes agent"
|
||||
# rather than "hermes agent use cases". Kept in sync with
|
||||
# planner._INTENT_MODIFIER_PATTERNS.
|
||||
_INTENT_MODIFIER_RE = re.compile(
|
||||
r"\b("
|
||||
r"use cases|use case|workflows|workflow|"
|
||||
r"examples|example|tutorial|tutorials|"
|
||||
r"review|reviews|comparison|applications|"
|
||||
r"in practice|production use|production|"
|
||||
r"how i use"
|
||||
r")\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
INTENT_SCORING_HINTS: dict[str, str] = {
|
||||
"comparison": (
|
||||
@@ -60,20 +86,21 @@ def rerank_candidates(
|
||||
) -> list[schema.Candidate]:
|
||||
"""Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant."""
|
||||
shortlisted = candidates[:shortlist_size]
|
||||
primary_entity = _primary_entity(topic)
|
||||
if provider and model and shortlisted:
|
||||
try:
|
||||
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted))
|
||||
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted, primary_entity))
|
||||
_apply_llm_scores(shortlisted, response)
|
||||
except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
|
||||
import sys
|
||||
print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
|
||||
_apply_fallback_scores(shortlisted)
|
||||
_apply_fallback_scores(shortlisted, primary_entity=primary_entity)
|
||||
else:
|
||||
_apply_fallback_scores(shortlisted)
|
||||
_apply_fallback_scores(shortlisted, primary_entity=primary_entity)
|
||||
|
||||
if len(candidates) > shortlist_size:
|
||||
tail = candidates[shortlist_size:]
|
||||
_apply_fallback_scores(tail)
|
||||
_apply_fallback_scores(tail, primary_entity=primary_entity)
|
||||
|
||||
return sorted(
|
||||
candidates,
|
||||
@@ -103,7 +130,7 @@ def _fenced_untrusted_content(candidate_block: str) -> str:
|
||||
)
|
||||
|
||||
|
||||
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate]) -> str:
|
||||
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate], primary_entity: str = "") -> str:
|
||||
ranking_queries = "\n".join(
|
||||
f"- {subquery.label}: {subquery.ranking_query}"
|
||||
for subquery in plan.subqueries
|
||||
@@ -121,6 +148,16 @@ def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Ca
|
||||
)
|
||||
for candidate in candidates
|
||||
)
|
||||
grounding_hint = ""
|
||||
if primary_entity:
|
||||
grounding_hint = (
|
||||
f"\nPrimary entity grounding: the user's primary entity is \"{primary_entity}\". "
|
||||
"A candidate that does NOT mention this entity (or a clear synonym/abbreviation) "
|
||||
"in its title or snippet should score no higher than 30, regardless of other "
|
||||
"signals. Do not let a candidate match the topic vicinity without matching the "
|
||||
"entity itself. 2026-04-19 Hermes Agent Use Cases failure: a Nate Herk video "
|
||||
"about Claude's Managed Agents scored 51 with zero Hermes content.\n"
|
||||
)
|
||||
return f"""
|
||||
Judge search-result relevance for a last-30-days research pipeline.
|
||||
|
||||
@@ -145,7 +182,7 @@ Scoring guidance:
|
||||
- 70 to 89: clearly relevant and useful
|
||||
- 40 to 69: somewhat relevant but weaker
|
||||
- 0 to 39: weak, redundant, or off-target
|
||||
{_intent_hint_block(plan)}
|
||||
{grounding_hint}{_intent_hint_block(plan)}
|
||||
{_fenced_untrusted_content(candidate_block)}
|
||||
""".strip()
|
||||
|
||||
@@ -169,21 +206,93 @@ def _apply_llm_scores(candidates: list[schema.Candidate], payload: dict) -> None
|
||||
candidate.final_score = _final_score(candidate)
|
||||
|
||||
|
||||
def _apply_fallback_scores(candidates: list[schema.Candidate]) -> None:
|
||||
def _apply_fallback_scores(candidates: list[schema.Candidate], *, primary_entity: str = "") -> None:
|
||||
for candidate in candidates:
|
||||
rerank_score, reason = _fallback_tuple(candidate)
|
||||
rerank_score, reason = _fallback_tuple(candidate, primary_entity=primary_entity)
|
||||
candidate.rerank_score = rerank_score
|
||||
candidate.explanation = reason
|
||||
candidate.final_score = _final_score(candidate)
|
||||
|
||||
|
||||
def _fallback_tuple(candidate: schema.Candidate) -> tuple[float, str]:
|
||||
def _candidate_haystack(candidate: schema.Candidate) -> str:
|
||||
"""Build the lowercase text blob against which entity-grounding is checked.
|
||||
|
||||
Expanded 2026-04-19 to include transcript snippets, transcript highlights,
|
||||
and top-comment text. The prior `title + snippet` check missed YouTube
|
||||
videos whose entity mentions live in transcript content and Reddit posts
|
||||
whose mentions are in top comments. Now checks all text surfaces a human
|
||||
would see.
|
||||
"""
|
||||
parts: list[str] = [candidate.title or "", candidate.snippet or ""]
|
||||
metadata = candidate.metadata or {}
|
||||
|
||||
transcript_snippet = metadata.get("transcript_snippet") or ""
|
||||
if isinstance(transcript_snippet, str):
|
||||
parts.append(transcript_snippet)
|
||||
|
||||
for hl in metadata.get("transcript_highlights") or []:
|
||||
if isinstance(hl, str):
|
||||
parts.append(hl)
|
||||
|
||||
for tc in metadata.get("top_comments") or []:
|
||||
if isinstance(tc, dict):
|
||||
parts.append(str(tc.get("excerpt", "") or tc.get("text", "") or ""))
|
||||
elif isinstance(tc, str):
|
||||
parts.append(tc)
|
||||
|
||||
for insight in metadata.get("comment_insights") or []:
|
||||
if isinstance(insight, str):
|
||||
parts.append(insight)
|
||||
|
||||
return " ".join(parts).lower()
|
||||
|
||||
|
||||
def _fallback_tuple(candidate: schema.Candidate, *, primary_entity: str = "") -> tuple[float, str]:
|
||||
score = (
|
||||
(candidate.local_relevance * 100.0 * 0.7)
|
||||
+ (candidate.freshness * 0.2)
|
||||
+ (candidate.source_quality * 100.0 * 0.1)
|
||||
)
|
||||
return max(0.0, min(100.0, score)), "fallback-local-score"
|
||||
reason = "fallback-local-score"
|
||||
# Entity-grounding demotion: if the primary entity (topic minus intent
|
||||
# modifier) is not present anywhere in the candidate's text surfaces
|
||||
# (title, snippet, transcript, transcript highlights, top comments,
|
||||
# insights), subtract ENTITY_MISS_PENALTY. Skip for candidates with
|
||||
# NO text anywhere (e.g., image-only TikToks) to avoid penalizing
|
||||
# thin-text sources unfairly. 2026-04-19 Nate Herk "Managed Agents"
|
||||
# video ranked #2 on a Hermes query despite zero Hermes mentions
|
||||
# because the old haystack only checked title + snippet.
|
||||
if primary_entity:
|
||||
haystack = _candidate_haystack(candidate)
|
||||
if haystack.strip() and primary_entity.lower() not in haystack:
|
||||
score -= ENTITY_MISS_PENALTY
|
||||
reason = "fallback-local-score (entity-miss demotion)"
|
||||
return max(0.0, min(100.0, score)), reason
|
||||
|
||||
|
||||
def _primary_entity(topic: str) -> str:
|
||||
"""Extract the primary entity from the topic for grounding checks.
|
||||
|
||||
Strips intent-modifier suffixes (see planner._INTENT_MODIFIER_PATTERNS),
|
||||
trims trailing punctuation, collapses whitespace. Returns the empty
|
||||
string for topics that are all intent modifier with no entity, so
|
||||
callers can skip the grounding check.
|
||||
"""
|
||||
stripped = _INTENT_MODIFIER_RE.sub(" ", topic)
|
||||
# Also collapse multiple spaces and strip punctuation.
|
||||
stripped = re.sub(r"\s+", " ", stripped).strip(" \t\r\n?.,:;!")
|
||||
return stripped
|
||||
|
||||
|
||||
#: Secondary entity-miss penalty applied directly to final_score (not just
|
||||
#: rerank_score). The -25 on rerank_score composes to only -15 on final_score
|
||||
#: via the 0.60 weight, which engagement bonus partially offsets on
|
||||
#: high-view YouTube items. This secondary penalty lands the full weight on
|
||||
#: the composite signal the cluster-scoring layer consumes. 2026-04-19
|
||||
#: Nate Herk "Managed Agents" video ranked at cluster #2 with score 51
|
||||
#: despite the rerank_score demotion because engagement + freshness drowned
|
||||
#: the dilute penalty. This backstop makes the demotion actually decisive.
|
||||
ENTITY_MISS_FINAL_PENALTY = 20.0
|
||||
|
||||
|
||||
def _final_score(candidate: schema.Candidate) -> float:
|
||||
@@ -204,6 +313,11 @@ def _final_score(candidate: schema.Candidate) -> float:
|
||||
)
|
||||
if candidate.rerank_score is not None and candidate.rerank_score < 20.0:
|
||||
base *= 0.3
|
||||
# Secondary entity-grounding penalty: when the fallback path flagged
|
||||
# entity-miss via candidate.explanation, apply an additional penalty
|
||||
# at final_score level so engagement signal can't mask the demotion.
|
||||
if candidate.explanation and "entity-miss" in candidate.explanation:
|
||||
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
|
||||
return base
|
||||
|
||||
|
||||
|
||||
@@ -28,6 +28,30 @@ class PipelineV3Tests(unittest.TestCase):
|
||||
self.assertIn("grounding", report.items_by_source)
|
||||
self.assertEqual("gemini", report.provider_runtime.reasoning_provider)
|
||||
|
||||
def test_planner_trace_always_fires_on_mock_run(self):
|
||||
"""Unit 5: The unified planner trace emits one summary line plus one
|
||||
line per subquery on every run, regardless of --debug. 2026-04-19
|
||||
Hermes Agent Use Cases failure: retrieval-breadth issues were invisible
|
||||
because the internal planner path logged nothing.
|
||||
"""
|
||||
import io
|
||||
import contextlib
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stderr(buf):
|
||||
pipeline.run(
|
||||
topic="test topic",
|
||||
config={"LAST30DAYS_REASONING_PROVIDER": "gemini"},
|
||||
depth="quick",
|
||||
requested_sources=["reddit", "x", "grounding"],
|
||||
mock=True,
|
||||
)
|
||||
output = buf.getvalue()
|
||||
self.assertIn("[Planner] Plan: intent=", output)
|
||||
self.assertIn("subqueries=", output)
|
||||
self.assertIn("source=", output)
|
||||
# At least one per-subquery line.
|
||||
self.assertIn("[Planner] sq1 label=", output)
|
||||
|
||||
|
||||
class TestSourceFetchCap(unittest.TestCase):
|
||||
"""X source fetch count must be capped by MAX_SOURCE_FETCHES."""
|
||||
|
||||
@@ -281,5 +281,177 @@ class PlannerV3Tests(unittest.TestCase):
|
||||
self.assertIn("instagram", all_sources)
|
||||
|
||||
|
||||
class IntentModifierBreadthTests(unittest.TestCase):
|
||||
"""Unit 2: Topics with intent modifiers (use cases, workflows, examples,
|
||||
review, comparison) must fan out across paraphrased subqueries rather
|
||||
than echo the literal phrase. 2026-04-19 Hermes Agent Use Cases failure.
|
||||
"""
|
||||
|
||||
def test_max_subqueries_raised_to_5_for_how_to(self):
|
||||
self.assertEqual(5, planner._max_subqueries("how_to"))
|
||||
|
||||
def test_max_subqueries_raised_to_5_for_opinion(self):
|
||||
self.assertEqual(5, planner._max_subqueries("opinion"))
|
||||
|
||||
def test_max_subqueries_raised_to_5_for_product(self):
|
||||
self.assertEqual(5, planner._max_subqueries("product"))
|
||||
|
||||
def test_max_subqueries_unchanged_for_comparison(self):
|
||||
self.assertEqual(4, planner._max_subqueries("comparison"))
|
||||
|
||||
def test_max_subqueries_unchanged_for_factual_and_concept(self):
|
||||
self.assertEqual(2, planner._max_subqueries("factual"))
|
||||
self.assertEqual(2, planner._max_subqueries("concept"))
|
||||
|
||||
def test_has_intent_modifier_detects_use_cases(self):
|
||||
self.assertTrue(planner._has_intent_modifier("Hermes Agent use cases"))
|
||||
self.assertTrue(planner._has_intent_modifier("Hermes Agent Actual Use Cases"))
|
||||
|
||||
def test_has_intent_modifier_detects_workflows(self):
|
||||
self.assertTrue(planner._has_intent_modifier("Claude Code workflows"))
|
||||
|
||||
def test_has_intent_modifier_detects_review_and_tutorial(self):
|
||||
self.assertTrue(planner._has_intent_modifier("Ollama review"))
|
||||
self.assertTrue(planner._has_intent_modifier("DSPy tutorial"))
|
||||
|
||||
def test_has_intent_modifier_false_for_bare_entity(self):
|
||||
self.assertFalse(planner._has_intent_modifier("Kanye West"))
|
||||
self.assertFalse(planner._has_intent_modifier("hermes agent"))
|
||||
|
||||
def test_fallback_fans_out_when_intent_modifier_present(self):
|
||||
plan = planner.plan_query(
|
||||
topic="Hermes Agent use cases",
|
||||
available_sources=["reddit", "x", "youtube", "hackernews"],
|
||||
requested_sources=None,
|
||||
depth="default",
|
||||
provider=None,
|
||||
model=None,
|
||||
)
|
||||
# Expect at least 3 subqueries total (primary + fanout); cap is 5 for
|
||||
# how_to/opinion/product/breaking_news. Label set should include at
|
||||
# least one of the paraphrase labels.
|
||||
labels = {sq.label for sq in plan.subqueries}
|
||||
self.assertGreaterEqual(len(plan.subqueries), 3)
|
||||
self.assertTrue(
|
||||
labels & {"workflows", "production", "experience"},
|
||||
f"Expected paraphrase labels in {labels}",
|
||||
)
|
||||
|
||||
def test_fallback_does_not_fan_out_for_bare_entity(self):
|
||||
plan = planner.plan_query(
|
||||
topic="Kanye West",
|
||||
available_sources=["reddit", "x", "grounding"],
|
||||
requested_sources=None,
|
||||
depth="default",
|
||||
provider=None,
|
||||
model=None,
|
||||
)
|
||||
# Bare entity without intent modifier should not trigger the paraphrase
|
||||
# fanout (those labels are not in the plan).
|
||||
labels = {sq.label for sq in plan.subqueries}
|
||||
self.assertFalse(labels & {"workflows", "production", "experience"})
|
||||
|
||||
def test_prompt_includes_intent_modifier_rule(self):
|
||||
prompt = planner._build_prompt(
|
||||
topic="Hermes Agent use cases",
|
||||
available_sources=["reddit", "x", "youtube"],
|
||||
requested_sources=None,
|
||||
depth="default",
|
||||
)
|
||||
self.assertIn("INTENT-MODIFIER HANDLING", prompt)
|
||||
self.assertIn("use cases", prompt)
|
||||
self.assertIn("STRIP that phrase", prompt)
|
||||
|
||||
|
||||
class FallbackDefaultsTests(unittest.TestCase):
|
||||
"""Unit 3: Deterministic fallback defaults and keyword_query quoting.
|
||||
2026-04-19 Hermes Agent Use Cases failure.
|
||||
"""
|
||||
|
||||
def test_unclassified_topic_defaults_to_concept_not_breaking_news(self):
|
||||
# Prior default was "breaking_news" with strict_recent freshness,
|
||||
# which biased against older relevant material on unfamiliar topics.
|
||||
self.assertEqual("concept", planner._infer_intent("some unfamiliar topic"))
|
||||
self.assertEqual("concept", planner._infer_intent("Hermes Agent"))
|
||||
|
||||
def test_recency_signals_still_break_out_to_breaking_news(self):
|
||||
self.assertEqual("breaking_news", planner._infer_intent("trending AI tools"))
|
||||
self.assertEqual("breaking_news", planner._infer_intent("what's happening today"))
|
||||
self.assertEqual("breaking_news", planner._infer_intent("this week in AI"))
|
||||
|
||||
def test_specific_intents_still_classify_correctly(self):
|
||||
# Regression: other regex branches still fire as before.
|
||||
self.assertEqual("how_to", planner._infer_intent("how to deploy Docker"))
|
||||
self.assertEqual("factual", planner._infer_intent("who acquired Wiz"))
|
||||
self.assertEqual("opinion", planner._infer_intent("thoughts on OpenAI Codex"))
|
||||
self.assertEqual("comparison", planner._infer_intent("Codex vs Claude Code"))
|
||||
|
||||
def test_keyword_query_quotes_only_title_cased_proper_nouns(self):
|
||||
# "Hermes Agent" is a multi-word title-cased proper noun — keep quoted.
|
||||
# "Use Cases" is also title-cased BUT we only quote the first 2
|
||||
# title-cased compounds; the first extracted is "Hermes Agent".
|
||||
search = planner._keyword_query("Hermes Agent use cases", "hermes agent")
|
||||
self.assertIn('"Hermes Agent"', search)
|
||||
# The old behavior quoted the entire typed topic; confirm it does not.
|
||||
self.assertNotIn('"Hermes Agent Actual Use Cases"', search)
|
||||
|
||||
def test_keyword_query_does_not_quote_bare_lowercase_topic(self):
|
||||
search = planner._keyword_query("kanye west bully", "kanye west bully")
|
||||
# Lowercase topics have no title-cased compound to quote.
|
||||
self.assertNotIn('"', search)
|
||||
|
||||
def test_fallback_logs_warning_when_no_provider(self):
|
||||
import io
|
||||
import contextlib
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stderr(buf):
|
||||
planner.plan_query(
|
||||
topic="Hermes Agent use cases",
|
||||
available_sources=["reddit", "x"],
|
||||
requested_sources=None,
|
||||
depth="default",
|
||||
provider=None,
|
||||
model=None,
|
||||
)
|
||||
output = buf.getvalue()
|
||||
# New language: "No --plan passed" + "YOU ARE the planner" +
|
||||
# runtime enumeration. Unit 4 (2026-04-19) rewrite to stop the
|
||||
# "no provider = no LLM = I need a key" misread.
|
||||
self.assertIn("No --plan passed", output)
|
||||
self.assertIn("YOU ARE the planner", output)
|
||||
self.assertIn("you ARE the LLM", output)
|
||||
# Runtime-agnostic: each supported runtime name should appear.
|
||||
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
|
||||
self.assertIn(runtime_name, output)
|
||||
# The old misleading phrasing must NOT appear.
|
||||
self.assertNotIn("No --plan and no LLM provider configured", output)
|
||||
|
||||
def test_fallback_does_not_log_new_warning_when_provider_present(self):
|
||||
# When a provider is configured, the provider path runs; if it
|
||||
# errors, we get the "LLM planning failed" message, NOT the
|
||||
# "No --plan passed" guidance (which is specifically for the
|
||||
# no-provider-no-plan caller path).
|
||||
import io
|
||||
import contextlib
|
||||
buf = io.StringIO()
|
||||
|
||||
class _NoopProvider:
|
||||
def generate_json(self, model, prompt):
|
||||
raise ValueError("force fallback for test")
|
||||
|
||||
with contextlib.redirect_stderr(buf):
|
||||
planner.plan_query(
|
||||
topic="Kanye West",
|
||||
available_sources=["reddit", "x"],
|
||||
requested_sources=None,
|
||||
depth="default",
|
||||
provider=_NoopProvider(),
|
||||
model="some-model",
|
||||
)
|
||||
output = buf.getvalue()
|
||||
self.assertIn("LLM planning failed", output)
|
||||
self.assertNotIn("No --plan passed", output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+127
-2
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
|
||||
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
|
||||
text = render.render_compact(report)
|
||||
self.assertIn("## Source Errors", text)
|
||||
self.assertIn("HTTP 400: Bad Request", text)
|
||||
self.assertIn("X:", text)
|
||||
|
||||
|
||||
class OutputEnvelopeTests(unittest.TestCase):
|
||||
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
|
||||
|
||||
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
|
||||
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
|
||||
"""
|
||||
|
||||
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
|
||||
text = render.render_compact(sample_report())
|
||||
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
|
||||
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
|
||||
# Opening comment must appear BEFORE the raw evidence block.
|
||||
self.assertLess(
|
||||
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||
text.index("## Ranked Evidence Clusters"),
|
||||
)
|
||||
# Closing comment must appear AFTER Source Coverage.
|
||||
self.assertGreater(
|
||||
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||
text.index("## Source Coverage"),
|
||||
)
|
||||
|
||||
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
|
||||
text = render.render_compact(sample_report())
|
||||
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
|
||||
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
|
||||
# Emoji footer sits between the two markers.
|
||||
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
|
||||
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
|
||||
self.assertIn("All agents reported back!", text[open_idx:close_idx])
|
||||
|
||||
def test_canonical_boundary_scopes_pass_through_to_footer(self):
|
||||
text = render.render_compact(sample_report())
|
||||
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
|
||||
# not everything above.
|
||||
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
|
||||
# Self-check string is present so the model has a concrete failure signal.
|
||||
self.assertIn("### 1.", text)
|
||||
self.assertIn("LAW 6", text)
|
||||
# The prior ambiguous phrasing is gone.
|
||||
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
|
||||
|
||||
def test_envelopes_appear_in_md_emit_mode(self):
|
||||
# --emit md and --emit compact both route to render_compact, so the
|
||||
# same envelopes apply. Guard against future divergence.
|
||||
text = render.render_compact(sample_report())
|
||||
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
|
||||
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
|
||||
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
|
||||
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
|
||||
|
||||
def test_no_dangling_envelope_open_without_close(self):
|
||||
# Open/close counts must always match, even for empty clusters.
|
||||
report = sample_report()
|
||||
report.clusters = []
|
||||
text = render.render_compact(report)
|
||||
self.assertEqual(
|
||||
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||
)
|
||||
self.assertEqual(
|
||||
text.count("<!-- PASS-THROUGH FOOTER:"),
|
||||
text.count("<!-- END PASS-THROUGH FOOTER -->"),
|
||||
)
|
||||
|
||||
|
||||
class RenderTopCommentsTests(unittest.TestCase):
|
||||
@@ -398,5 +462,66 @@ class RenderBestTakesCompactTests(unittest.TestCase):
|
||||
self.assertNotIn("## Best Takes", text)
|
||||
|
||||
|
||||
class DegradedRunBannerTests(unittest.TestCase):
|
||||
"""Unit 1: DEGRADED RUN WARNING surfaces bare named-entity invocations
|
||||
in user-visible stdout. LAW 7 backstop. 2026-04-19 Hermes Agent Use
|
||||
Cases Run 1 failure mode.
|
||||
"""
|
||||
|
||||
def _bare_named_entity_report(self) -> schema.Report:
|
||||
report = sample_report()
|
||||
report.topic = "Hermes Agent"
|
||||
report.artifacts["plan_source"] = "deterministic"
|
||||
report.artifacts["pre_research_flags_present"] = False
|
||||
return report
|
||||
|
||||
def test_banner_appears_on_bare_named_entity_deterministic_run(self):
|
||||
text = render.render_compact(self._bare_named_entity_report())
|
||||
self.assertIn("## DEGRADED RUN WARNING", text)
|
||||
self.assertIn("<!-- USER-VISIBLE BANNER:", text)
|
||||
self.assertIn("<!-- END USER-VISIBLE BANNER -->", text)
|
||||
self.assertIn("YOU ARE", text)
|
||||
# Runtime-agnostic enumeration: all host runtimes appear.
|
||||
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
|
||||
self.assertIn(runtime_name, text)
|
||||
|
||||
def test_banner_positioned_before_evidence_envelope(self):
|
||||
text = render.render_compact(self._bare_named_entity_report())
|
||||
banner_idx = text.index("## DEGRADED RUN WARNING")
|
||||
envelope_idx = text.index("<!-- EVIDENCE FOR SYNTHESIS:")
|
||||
self.assertLess(banner_idx, envelope_idx,
|
||||
"DEGRADED RUN banner must appear BEFORE evidence envelope so pass-through catches it.")
|
||||
|
||||
def test_banner_suppressed_when_plan_source_external(self):
|
||||
report = self._bare_named_entity_report()
|
||||
report.artifacts["plan_source"] = "external"
|
||||
text = render.render_compact(report)
|
||||
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||
|
||||
def test_banner_suppressed_when_plan_source_llm(self):
|
||||
report = self._bare_named_entity_report()
|
||||
report.artifacts["plan_source"] = "llm"
|
||||
text = render.render_compact(report)
|
||||
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||
|
||||
def test_banner_suppressed_when_pre_research_flags_present(self):
|
||||
report = self._bare_named_entity_report()
|
||||
report.artifacts["pre_research_flags_present"] = True
|
||||
text = render.render_compact(report)
|
||||
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||
|
||||
def test_banner_suppressed_on_non_eligible_abstract_topic(self):
|
||||
report = self._bare_named_entity_report()
|
||||
# Multi-word lowercase abstract phrase is NOT pre-research-eligible.
|
||||
report.topic = "how to deploy containers in the cloud"
|
||||
text = render.render_compact(report)
|
||||
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||
|
||||
def test_banner_mentions_law_7_and_plan_flag(self):
|
||||
text = render.render_compact(self._bare_named_entity_report())
|
||||
self.assertIn("LAW 7", text)
|
||||
self.assertIn("--plan", text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
+203
-1
@@ -178,9 +178,211 @@ class RerankV3Tests(unittest.TestCase):
|
||||
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
|
||||
self.assertEqual(95.0, first.rerank_score)
|
||||
self.assertEqual("high fit", first.explanation)
|
||||
self.assertEqual("fallback-local-score", second.explanation)
|
||||
# Tail is scored via the fallback (may or may not carry the entity-miss
|
||||
# suffix depending on topic-title overlap; assert the base tag is present).
|
||||
self.assertIn("fallback-local-score", second.explanation or "")
|
||||
self.assertEqual(first.candidate_id, ranked[0].candidate_id)
|
||||
|
||||
|
||||
class EntityGroundingTests(unittest.TestCase):
|
||||
"""Unit 4: Reranker entity-grounding demotion. 2026-04-19 Hermes Agent
|
||||
Use Cases failure: an off-topic video about Claude Managed Agents
|
||||
scored 51 and ranked #2 with zero Hermes content.
|
||||
"""
|
||||
|
||||
def _candidate(self, title: str, snippet: str = "") -> schema.Candidate:
|
||||
return schema.Candidate(
|
||||
candidate_id=f"c-{title[:10]}",
|
||||
item_id="i1",
|
||||
source="youtube",
|
||||
title=title,
|
||||
url="https://example.com",
|
||||
snippet=snippet,
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:youtube": 1},
|
||||
local_relevance=0.8,
|
||||
freshness=80,
|
||||
engagement=50,
|
||||
source_quality=0.7,
|
||||
rrf_score=0.02,
|
||||
)
|
||||
|
||||
def test_primary_entity_strips_intent_modifier(self):
|
||||
self.assertEqual("Hermes Agent", rerank._primary_entity("Hermes Agent use cases"))
|
||||
self.assertEqual("Hermes Agent Actual", rerank._primary_entity("Hermes Agent Actual Use Cases"))
|
||||
self.assertEqual("Claude Code", rerank._primary_entity("Claude Code workflows"))
|
||||
self.assertEqual("DSPy", rerank._primary_entity("DSPy tutorial"))
|
||||
|
||||
def test_primary_entity_leaves_bare_entity_unchanged(self):
|
||||
self.assertEqual("Kanye West", rerank._primary_entity("Kanye West"))
|
||||
self.assertEqual("Nous Research", rerank._primary_entity("Nous Research"))
|
||||
|
||||
def test_fallback_demotes_candidate_without_primary_entity(self):
|
||||
on_topic = self._candidate("Hermes Agent: Self-Improving AI", "Nous Research Hermes walkthrough")
|
||||
off_topic = self._candidate("I Tested Claude's Managed Agents", "What you need to know about Anthropic's new managed agents")
|
||||
rerank._apply_fallback_scores([on_topic, off_topic], primary_entity="Hermes Agent")
|
||||
self.assertGreater(on_topic.final_score, off_topic.final_score)
|
||||
self.assertIn("entity-miss", off_topic.explanation or "")
|
||||
self.assertEqual(on_topic.explanation, "fallback-local-score")
|
||||
|
||||
def test_fallback_match_is_case_insensitive(self):
|
||||
on_topic = self._candidate("HERMES agent rocks", "some text")
|
||||
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", on_topic.explanation)
|
||||
|
||||
def test_fallback_skips_demotion_for_empty_text_candidates(self):
|
||||
empty = self._candidate("", "")
|
||||
rerank._apply_fallback_scores([empty], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", empty.explanation)
|
||||
|
||||
def test_fallback_skips_demotion_when_no_primary_entity(self):
|
||||
off = self._candidate("Completely unrelated", "snippet")
|
||||
rerank._apply_fallback_scores([off], primary_entity="")
|
||||
self.assertEqual("fallback-local-score", off.explanation)
|
||||
|
||||
def test_llm_prompt_includes_primary_entity_grounding_hint(self):
|
||||
candidate = self._candidate("Something", "snippet text")
|
||||
plan = make_plan()
|
||||
prompt = rerank._build_prompt(
|
||||
"Hermes Agent use cases", plan, [candidate], primary_entity="Hermes Agent"
|
||||
)
|
||||
self.assertIn("Primary entity grounding", prompt)
|
||||
self.assertIn("Hermes Agent", prompt)
|
||||
|
||||
def test_llm_prompt_omits_grounding_hint_when_no_primary_entity(self):
|
||||
candidate = self._candidate("Something", "snippet text")
|
||||
plan = make_plan()
|
||||
prompt = rerank._build_prompt("", plan, [candidate], primary_entity="")
|
||||
self.assertNotIn("Primary entity grounding", prompt)
|
||||
|
||||
|
||||
class ExpandedHaystackTests(unittest.TestCase):
|
||||
"""Unit 3: Entity-grounding haystack covers transcript snippets,
|
||||
transcript highlights, top comments, and comment insights - not
|
||||
just title + snippet.
|
||||
"""
|
||||
|
||||
def _youtube_candidate(self, title: str, transcript_snippet: str = "",
|
||||
transcript_highlights: list[str] | None = None) -> schema.Candidate:
|
||||
c = schema.Candidate(
|
||||
candidate_id=f"c-{title[:10]}",
|
||||
item_id="i1",
|
||||
source="youtube",
|
||||
title=title,
|
||||
url="https://youtube.com/watch?v=x",
|
||||
snippet="",
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:youtube": 1},
|
||||
local_relevance=0.8,
|
||||
freshness=80,
|
||||
engagement=50,
|
||||
source_quality=0.7,
|
||||
rrf_score=0.02,
|
||||
)
|
||||
c.metadata = {}
|
||||
if transcript_snippet:
|
||||
c.metadata["transcript_snippet"] = transcript_snippet
|
||||
if transcript_highlights:
|
||||
c.metadata["transcript_highlights"] = transcript_highlights
|
||||
return c
|
||||
|
||||
def test_entity_found_in_transcript_snippet_avoids_demotion(self):
|
||||
# Title + snippet miss the entity, but the transcript contains it.
|
||||
c = self._youtube_candidate(
|
||||
"Weekly roundup",
|
||||
transcript_snippet="In this video I walk through using Hermes Agent in production.",
|
||||
)
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", c.explanation)
|
||||
|
||||
def test_entity_found_in_transcript_highlights_avoids_demotion(self):
|
||||
c = self._youtube_candidate(
|
||||
"Some review",
|
||||
transcript_highlights=[
|
||||
"Today we're talking about Hermes Agent",
|
||||
"Let's compare it to the alternatives",
|
||||
],
|
||||
)
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", c.explanation)
|
||||
|
||||
def test_entity_missing_everywhere_still_demoted_for_video(self):
|
||||
# Nate Herk "Managed Agents" case: no Hermes in title, snippet,
|
||||
# or transcript - demotion fires.
|
||||
c = self._youtube_candidate(
|
||||
"I Tested Claude's New Managed Agents",
|
||||
transcript_snippet="Managed agents are Anthropic's new product with ClickUp and cron...",
|
||||
)
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertIn("entity-miss", c.explanation)
|
||||
|
||||
def test_entity_found_in_reddit_top_comments_avoids_demotion(self):
|
||||
c = schema.Candidate(
|
||||
candidate_id="r1",
|
||||
item_id="i1",
|
||||
source="reddit",
|
||||
title="Best agent framework?",
|
||||
url="https://reddit.com/r/x",
|
||||
snippet="",
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:reddit": 1},
|
||||
local_relevance=0.8, freshness=80, engagement=50,
|
||||
source_quality=0.7, rrf_score=0.02,
|
||||
)
|
||||
c.metadata = {
|
||||
"top_comments": [
|
||||
{"excerpt": "I've been using Hermes Agent for a month and it's great"},
|
||||
{"text": "another comment"},
|
||||
],
|
||||
}
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", c.explanation)
|
||||
|
||||
def test_entity_found_in_comment_insights_avoids_demotion(self):
|
||||
c = schema.Candidate(
|
||||
candidate_id="r2", item_id="i1", source="reddit",
|
||||
title="AI tools", url="https://reddit.com/r/x", snippet="",
|
||||
subquery_labels=["primary"],
|
||||
native_ranks={"primary:reddit": 1},
|
||||
local_relevance=0.8, freshness=80, engagement=50,
|
||||
source_quality=0.7, rrf_score=0.02,
|
||||
)
|
||||
c.metadata = {
|
||||
"comment_insights": ["Consensus: Hermes Agent handles long sessions best"],
|
||||
}
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", c.explanation)
|
||||
|
||||
def test_truly_empty_candidate_still_skipped(self):
|
||||
# Image-only TikTok with no text anywhere - do not penalize.
|
||||
c = self._youtube_candidate("") # empty title
|
||||
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||
self.assertEqual("fallback-local-score", c.explanation)
|
||||
|
||||
def test_final_score_secondary_penalty_applied_on_entity_miss(self):
|
||||
# When fallback flags entity-miss, final_score gets an ADDITIONAL
|
||||
# -20 penalty beyond the rerank_score reduction. Verify by
|
||||
# comparing final_score for a demoted candidate vs an identical
|
||||
# candidate that matched the entity.
|
||||
off_topic = self._youtube_candidate("Managed Agents from Anthropic")
|
||||
on_topic = self._youtube_candidate(
|
||||
"Hermes Agent walkthrough",
|
||||
transcript_snippet="Hermes Agent review",
|
||||
)
|
||||
rerank._apply_fallback_scores([off_topic, on_topic], primary_entity="Hermes Agent")
|
||||
# Gap should be well above the rerank_score-only path's 0.60 * 25 = 15;
|
||||
# with the secondary penalty it's 15 + 20 = 35 points.
|
||||
gap = on_topic.final_score - off_topic.final_score
|
||||
self.assertGreater(gap, 25.0,
|
||||
f"entity-miss demotion gap only {gap:.1f}; secondary penalty may not be firing")
|
||||
|
||||
def test_secondary_penalty_not_applied_when_entity_match(self):
|
||||
on_topic = self._youtube_candidate("Hermes Agent: use cases")
|
||||
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
|
||||
# Explanation does NOT contain entity-miss, so secondary penalty
|
||||
# should not fire; final_score reflects only base signal.
|
||||
self.assertNotIn("entity-miss", on_topic.explanation or "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user